You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@solr.apache.org by Jason Gerlowski <ge...@gmail.com> on 2021/11/25 04:40:19 UTC
JAX-RS APIs in Solr
Solr's custom annotation framework ('@Endpoint', '@Command', etc.) has
cropped up a few times over the past week or two. [1] [2]. Having them
on top of mind, I've been wondering - is there a reason we use our own
annotations here instead of something off the shelf?
What we have works well enough, but anything homegrown comes with more
maintenance burden than we'd have if we used something off the shelf.
There are plenty of well-used, active projects out there whose whole
purpose is facilitating the whole "annotation based API" thing
(Jersey, Restlet, RESTEasy, etc.) - why not use one of them?
Does anyone know of any technical reasons why we can't go this route?
Or have any subjective reasons why we shouldn't? Or any context on
why we wrote our own Endpoint, Command, JsonProperty annotations
originally?
FWIW, Eric Pugh and I spiked out a small POC recently, and got
Jersey+Jackson working for a few simple APIs without too much trouble.
[3] Obviously nothing production-ready there, and there's still a lot
of open questions (e.g. how would javabin be supported?), but we both
came away convinced that it seemed feasible, at least. Best of all,
APIs using our current homegrown annotation framework the switchover
seems blessedly straightforward, and it doesn't look like Jersey
(which we chose mostly arbitrarily) bloats our dist all that much.
Curious if anyone has thoughts or context on how we ended up with the
annotation setup we use today!
Best,
Jason
[1] https://issues.apache.org/jira/browse/SOLR-15182 (and children)
[2] http://mail-archives.apache.org/mod_mbox/solr-dev/202111.mbox/%3CCABEwPvENL41Pm6%2BOmjXb6Sx5N2XjUtnbWhgKOZSrnLjWBA8tcA%40mail.gmail.com%3E
[3] https://github.com/gerlowskija/solr/tree/jersey_jaxrs_jackson_solr_apis.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: JAX-RS APIs in Solr
Posted by Jason Gerlowski <ge...@gmail.com>.
Hey all,
Thanks for all the feedback since I last chimed in here; a lot to
think through for sure.
> This also has associated ways to: register end points and configuration, manage lifecycle, collect metrics, enforce security, logging, tracing, etc.
Thanks Noble - this list is super helpful in suggesting specific
things to dig into in terms of gauging viability for replacement.
Some list items require no explanation ("register end points") but on
a few others I'm less sure what you're referring to. Could you expand
a bit on what role SolrRequestHandler plays for "logging", "tracing",
and "collects metrics"?
Best,
Jason
On Mon, Dec 6, 2021 at 2:04 PM Noble Paul <no...@gmail.com> wrote:
>
> I'm +1 for modernization
> I just wanted to highlight the hurdles we need to overcome to reach there
>
> SolrRequestHandler is the current framework that serves all requests in Solr. This also has associated ways to
>
> register end points and configuration
> manage lifecycle
> collect metrics
> enforce security
> logging
> tracing
> etc etc
>
> Introducing a new framework will have to deal with all of the above. For instance the JAX-RS is used you will start using web.xml to register your JAX-RS endpoints.
>
> --Noble
>
>
>
> On Mon, Dec 6, 2021 at 6:11 AM David Smiley <ds...@apache.org> wrote:
>>
>> Just a simple +1 of support to modernization efforts in general. It's encouraging to see that Jason & Eric had some fun together on this. Modernization, I think, helps with the fun of any open-source project, and thus helps keep everyone interested in continuing and reviewing interest in Solr. If we/others feel we can't make fundamental changes, then I think our interests (and that of contributors) will wane. Personally, I really enjoy refactoring codebases, even if it may not seem sexy to some.
>>
>> I don't think we can rush into a SIP before more research/POC is tried. It's too abstract at this stage. We don't even know what framework to use yet ;-)
>
>
>
> --
> -----------------------------------------------------
> Noble Paul
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: JAX-RS APIs in Solr
Posted by David Smiley <ds...@apache.org>.
I'm excited to see what this looks like!
~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley
On Mon, Aug 8, 2022 at 12:11 PM Jason Gerlowski <ge...@gmail.com>
wrote:
> Hey all,
>
> I spent some time last week digging into OpenAPI and spiking out how
> it could be integrated into Solr. I came away very impressed. It
> opens a lot of really cool doors: auto-generated clients and docs,
> "Swagger UI", backcompat/api-breakage detection, etc. There's a LOT
> to gain.
>
> Another takeaway from my spike though was that OpenAPI is much more
> "do-able" in JAX-RS projects. It's possible to create
> annotation-drive OpenAPI specs without JAX-RS, but it requires more
> explicit (and duplicative) documentation of each API's inputs and
> ouputs, which probably isn't maintainable in a project as large as
> ours.
>
> With that in mind, I'm planning on returning to the JAX-RS spike Eric
> and I did months back, and updating it to cover some of the issues
> this thread brought to light. In particular: security integration and
> serving collection/core-specific APIs. If that pans out, we can
> figure out next steps from there. (A SIP? A JIRA?)
>
> Before I start that effort though, I figured it'd be worth
> double-checking that there are no -1's/vetos on the idea sight-unseen?
> If so, let me know!
>
> Best,
>
> Jason
>
>
> On Fri, Jan 7, 2022 at 5:36 PM Eric Pugh
> <ep...@opensourceconnections.com> wrote:
> >
> > I wanted to share that our hack day did happen, but it went a bit
> sideways as we spent the first half of the day experimenting with how to
> support CORS in Solr. So API related, but not JAX-RS API specific. The
> second half of the day got consumed by $dayjob.
> >
> > We’re going to pick it up again next month, and dig into trying out how
> existing Solr security would work.
> >
> > https://github.com/gerlowskija/solr/tree/cors_stuff if you want to see.
> >
> > Eric
> >
> >
> > On Dec 9, 2021, at 10:06 AM, Eric Pugh <ep...@opensourceconnections.com>
> wrote:
> >
> > Thank everyone for the input, it’s been a productive conversation!
> >
> > Jason and I are planning on another hack day Jan 7th to take some of the
> feedback, and work more on how our spike can help meet some concerns/show
> promise, so we’ll report back then!
> >
> > We’re planning on zooming during US East Coast hours, and I’ll drop the
> Zoom invite in the ASF Slack for anyone who wants to join in and say hi!
> >
> > Eric
> >
> >
> >
> > On Dec 7, 2021, at 3:47 PM, Mark Miller <ma...@gmail.com> wrote:
> >
> > Two cents from the peanut gallery:
> >
> > I’ve looked at this before. My opinion:
> >
> > Our stuff was a just terrible, take your pick on the api version.
> Reasons are numerous.
> >
> > Custom end points is an anti feature. Even worse for cloud.
> >
> > JAX-RS looked ridiculously sensible.
> >
> >
> > --
> > - MRM
> >
> >
> > _______________________
> > Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 |
> http://www.opensourceconnections.com | My Free/Busy
> > Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
> > This e-mail and all contents, including attachments, is considered to be
> Company Confidential unless explicitly stated otherwise, regardless of
> whether attachments are marked as such.
> >
> >
> > _______________________
> > Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 |
> http://www.opensourceconnections.com | My Free/Busy
> > Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
> > This e-mail and all contents, including attachments, is considered to be
> Company Confidential unless explicitly stated otherwise, regardless of
> whether attachments are marked as such.
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org
>
>
Re: JAX-RS APIs in Solr
Posted by Jason Gerlowski <ge...@gmail.com>.
Hey all,
I've got an updated stab at a JAX-RS integration using Jersey:
https://github.com/apache/solr/pull/975. That PR solves most of the
concerns mentioned above (authc/authz integration, per-core API
registration). It's still definitely in "draft" territory: logging
needs cleaned up, it needs javadocs and tests, etc. But it should be
complete enough to give a realistic sense of the pros/cons.
Some notes on the implementation:
* Jersey invocation happens in the same ServletFilter as everything
else (SolrDispatchFilter).
** I initially hoped to put it in its own Servlet, but gave up when I
had to copy big chunks of SDF and V2HttpCall anyways. Gus is right
that we should really break SDF down into smaller composable filters,
but I didn't have the time (or knowledge, really) to jump into that
all here.
* JAX-RS resources are registered in an "Application" and then
looked-up and invoked by a "ApplicationHandler".
** ApplicationHandler is analogous to Solr's "ApiBag" class, and it
has the same lifecycle/span (both on CoreContainer and on individual
SolrCore instances)
* Authorization is done via Jersey's "Request Filter" functionality,
see SolrRequestAuthorizer.
* Jersey supports a few response "mime types" natively, but offers an
interface ("MessageBodyWriter") to add others. This PR tests that out
by adding a javabin implementation in "JavabinWriter". It's def not
perfect but should serve as a PoC.
* For testing purposes, I implemented one "admin" API
(ListConfigSetsAPI) and one "core" API
(SomeCoreHandler/SomeCoreResource) to be Jersey-based. Planning to
add a few more in short order, but there's at least something to test
out-of-the-box (e.g. curl http://localhost:8983/api/cluster/configs)
Overall, while it definitely wasn't trivial, in my opinion the
integration code wasn't as bad as it might've been. And especially
with some of the OpenAPI tie-in benefits in mind, JAX-RS is worth
pursuing.
If no one sees any grounds to veto, my next-steps would probably be to
file a JIRA and start hardening the spike and filling gaps (plugin
framework support, etc.) with a mind to commit down the road once
those are cleared away.
Excited to hear what folks think!
Best,
Jason
On Tue, Aug 9, 2022 at 5:03 PM Colvin Cowie <co...@gmail.com> wrote:
>
> Yes that's true, if you only want to generate clients and not the server
> then you can definitely have an easier time of things.
>
> On Tue, 9 Aug 2022 at 14:21, Jason Gerlowski <ge...@gmail.com> wrote:
>
> > Thanks for the insight Colvin! Spec-driving-the-code sounds appealing
> > in a lot of ways (I'd love to have defaults handled uniformly across
> > Solr, for instance), but I think Jan's right that it's probably
> > infeasible for Solr.
> >
> > And "point taken" about expecting some issues with the tooling!
> >
> > Best,
> >
> > Jason
> >
> > On Tue, Aug 9, 2022 at 6:10 AM Jan Høydahl <ja...@cominvent.com> wrote:
> > >
> > > I have recent experience from a SpringBoot / Kotlin application where we
> > use code-first annotation driven OpenAPI definition. I think that will be
> > the only feasible way for Solr.
> > >
> > > The sweet thing about that approach is that the OpenAPI definitions and
> > documentation is all done inline with the code using @Tag and @Operaton
> > annotations, and there is no drift. You can then generate the OpenAPI spec
> > JSON from code to do (client) code generation. Swagger UI comes for free in
> > the app itself with a simple dependency.
> > >
> > > Jan
> > >
> > > > 9. aug. 2022 kl. 11:45 skrev Colvin Cowie <colvin.cowie.dev@gmail.com
> > >:
> > > >
> > > > I spent the first few months of this year adopting OpenAPI 3 spec in
> > our
> > > > product, which has a fairly large API.
> > > >
> > > > We previously generated a Swagger 1.0 spec from our Java code, now
> > we're
> > > > generating our POJOs and JAX-RS endpoints from the OA3 spec.
> > > >
> > > > Aside from needing to get off outdated Swagger 1 tools, we were
> > motivated
> > > > to go spec-first to close the gap between our API designs "on paper"
> > and
> > > > the actual implementation, be more rigorous with our API definitions,
> > and
> > > > to (hopefully) speed up / simplify development in the future.
> > > >
> > > > Migration for us was complicated by the fact that our existing
> > codebase was
> > > > inconsistent when it came to things like default initialization of
> > fields
> > > > in POJOs, use of primitives/Objects, and List vs Collection etc. While
> > the
> > > > code generator is consistent in what it produces (naturally). Our
> > codebase
> > > > is also fairly tightly coupled between transport classes and model
> > classes
> > > > as we eschewed the use of models for a lot of it. We did modify the
> > > > templates and add extra x- vendor attributes to our spec to control the
> > > > generation of certain things where we needed to though, so we were
> > able to
> > > > maintain some of our existing idiosyncrasies when necessary.
> > > >
> > > > We were also bitten by some bugs/inconsistencies in the OA3 tooling we
> > used
> > > > - even though OA3 is a specification, everyone seems to have a slightly
> > > > different interpretation or level of support for some parts of it.
> > > > The main pain point for us is around support for polymorphic types /
> > > > inheritance (
> > > >
> > https://swagger.io/docs/specification/data-models/oneof-anyof-allof-not/),
> > > > where modelling them one way works for one viewer but not another, or
> > > > doesn't work for code generators, or api-diffing and so on.
> > > >
> > > > We use
> > > >
> > > > - https://www.apicur.io/apicurito to edit the spec
> > > > - https://github.com/OpenAPITools/openapi-generator to generate
> > Java code
> > > > - https://github.com/rapi-doc/RapiDoc as our viewer, we like
> > > > https://redocly.com/redoc too
> > > > - https://github.com/OpenAPITools/openapi-diff to check for
> > (breaking)
> > > > changes
> > > >
> > > > Getting to where we are was quite painful - it would definitely be a
> > lot
> > > > easier to adopt a spec-first approach in a brand new project - but now
> > that
> > > > we're here it was probably worth it for the consistency and API-first
> > view
> > > > it brings.
> > > >
> > > > So I hope I don't put you off trying it, but just be aware that some
> > things
> > > > won't work as well as you might expect them to and you might find
> > yourself
> > > > contributing to some of these OpenAPI projects to get things to work
> > the
> > > > way you need them to :)
> > > >
> > > > Colvin
> > > >
> > > >
> > > > On Tue, 9 Aug 2022 at 08:30, Jan Høydahl <ja...@cominvent.com>
> > wrote:
> > > >
> > > >> Makes sense to explore at this point to figure out best order of
> > > >> development. And with the door open to slightly adapt some URL
> > patterns in
> > > >> v2, should it be required, it's less likely that roadblocks would
> > derail
> > > >> the entire effort.
> > > >>
> > > >> Jan
> > > >>
> > > >>> 8. aug. 2022 kl. 18:10 skrev Jason Gerlowski <gerlowskija@gmail.com
> > >:
> > > >>>
> > > >>> Hey all,
> > > >>>
> > > >>> I spent some time last week digging into OpenAPI and spiking out how
> > > >>> it could be integrated into Solr. I came away very impressed. It
> > > >>> opens a lot of really cool doors: auto-generated clients and docs,
> > > >>> "Swagger UI", backcompat/api-breakage detection, etc. There's a LOT
> > > >>> to gain.
> > > >>>
> > > >>> Another takeaway from my spike though was that OpenAPI is much more
> > > >>> "do-able" in JAX-RS projects. It's possible to create
> > > >>> annotation-drive OpenAPI specs without JAX-RS, but it requires more
> > > >>> explicit (and duplicative) documentation of each API's inputs and
> > > >>> ouputs, which probably isn't maintainable in a project as large as
> > > >>> ours.
> > > >>>
> > > >>> With that in mind, I'm planning on returning to the JAX-RS spike Eric
> > > >>> and I did months back, and updating it to cover some of the issues
> > > >>> this thread brought to light. In particular: security integration
> > and
> > > >>> serving collection/core-specific APIs. If that pans out, we can
> > > >>> figure out next steps from there. (A SIP? A JIRA?)
> > > >>>
> > > >>> Before I start that effort though, I figured it'd be worth
> > > >>> double-checking that there are no -1's/vetos on the idea
> > sight-unseen?
> > > >>> If so, let me know!
> > > >>>
> > > >>> Best,
> > > >>>
> > > >>> Jason
> > > >>>
> > > >>>
> > > >>> On Fri, Jan 7, 2022 at 5:36 PM Eric Pugh
> > > >>> <ep...@opensourceconnections.com> wrote:
> > > >>>>
> > > >>>> I wanted to share that our hack day did happen, but it went a bit
> > > >> sideways as we spent the first half of the day experimenting with how
> > to
> > > >> support CORS in Solr. So API related, but not JAX-RS API specific.
> > The
> > > >> second half of the day got consumed by $dayjob.
> > > >>>>
> > > >>>> We’re going to pick it up again next month, and dig into trying out
> > how
> > > >> existing Solr security would work.
> > > >>>>
> > > >>>> https://github.com/gerlowskija/solr/tree/cors_stuff if you want to
> > see.
> > > >>>>
> > > >>>> Eric
> > > >>>>
> > > >>>>
> > > >>>> On Dec 9, 2021, at 10:06 AM, Eric Pugh <
> > epugh@opensourceconnections.com>
> > > >> wrote:
> > > >>>>
> > > >>>> Thank everyone for the input, it’s been a productive conversation!
> > > >>>>
> > > >>>> Jason and I are planning on another hack day Jan 7th to take some of
> > > >> the feedback, and work more on how our spike can help meet some
> > > >> concerns/show promise, so we’ll report back then!
> > > >>>>
> > > >>>> We’re planning on zooming during US East Coast hours, and I’ll drop
> > the
> > > >> Zoom invite in the ASF Slack for anyone who wants to join in and say
> > hi!
> > > >>>>
> > > >>>> Eric
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> On Dec 7, 2021, at 3:47 PM, Mark Miller <ma...@gmail.com>
> > wrote:
> > > >>>>
> > > >>>> Two cents from the peanut gallery:
> > > >>>>
> > > >>>> I’ve looked at this before. My opinion:
> > > >>>>
> > > >>>> Our stuff was a just terrible, take your pick on the api version.
> > > >> Reasons are numerous.
> > > >>>>
> > > >>>> Custom end points is an anti feature. Even worse for cloud.
> > > >>>>
> > > >>>> JAX-RS looked ridiculously sensible.
> > > >>>>
> > > >>>>
> > > >>>> --
> > > >>>> - MRM
> > > >>>>
> > > >>>>
> > > >>>> _______________________
> > > >>>> Eric Pugh | Founder & CEO | OpenSource Connections, LLC |
> > 434.466.1467
> > > >> | http://www.opensourceconnections.com | My Free/Busy
> > > >>>> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
> > > >>>> This e-mail and all contents, including attachments, is considered
> > to
> > > >> be Company Confidential unless explicitly stated otherwise,
> > regardless of
> > > >> whether attachments are marked as such.
> > > >>>>
> > > >>>>
> > > >>>> _______________________
> > > >>>> Eric Pugh | Founder & CEO | OpenSource Connections, LLC |
> > 434.466.1467
> > > >> | http://www.opensourceconnections.com | My Free/Busy
> > > >>>> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
> > > >>>> This e-mail and all contents, including attachments, is considered
> > to
> > > >> be Company Confidential unless explicitly stated otherwise,
> > regardless of
> > > >> whether attachments are marked as such.
> > > >>>>
> > > >>>
> > > >>> ---------------------------------------------------------------------
> > > >>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > >>> For additional commands, e-mail: dev-help@solr.apache.org
> > > >>>
> > > >>
> > > >>
> > > >> ---------------------------------------------------------------------
> > > >> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > >> For additional commands, e-mail: dev-help@solr.apache.org
> > > >>
> > > >>
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > For additional commands, e-mail: dev-help@solr.apache.org
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > For additional commands, e-mail: dev-help@solr.apache.org
> >
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: JAX-RS APIs in Solr
Posted by Colvin Cowie <co...@gmail.com>.
Yes that's true, if you only want to generate clients and not the server
then you can definitely have an easier time of things.
On Tue, 9 Aug 2022 at 14:21, Jason Gerlowski <ge...@gmail.com> wrote:
> Thanks for the insight Colvin! Spec-driving-the-code sounds appealing
> in a lot of ways (I'd love to have defaults handled uniformly across
> Solr, for instance), but I think Jan's right that it's probably
> infeasible for Solr.
>
> And "point taken" about expecting some issues with the tooling!
>
> Best,
>
> Jason
>
> On Tue, Aug 9, 2022 at 6:10 AM Jan Høydahl <ja...@cominvent.com> wrote:
> >
> > I have recent experience from a SpringBoot / Kotlin application where we
> use code-first annotation driven OpenAPI definition. I think that will be
> the only feasible way for Solr.
> >
> > The sweet thing about that approach is that the OpenAPI definitions and
> documentation is all done inline with the code using @Tag and @Operaton
> annotations, and there is no drift. You can then generate the OpenAPI spec
> JSON from code to do (client) code generation. Swagger UI comes for free in
> the app itself with a simple dependency.
> >
> > Jan
> >
> > > 9. aug. 2022 kl. 11:45 skrev Colvin Cowie <colvin.cowie.dev@gmail.com
> >:
> > >
> > > I spent the first few months of this year adopting OpenAPI 3 spec in
> our
> > > product, which has a fairly large API.
> > >
> > > We previously generated a Swagger 1.0 spec from our Java code, now
> we're
> > > generating our POJOs and JAX-RS endpoints from the OA3 spec.
> > >
> > > Aside from needing to get off outdated Swagger 1 tools, we were
> motivated
> > > to go spec-first to close the gap between our API designs "on paper"
> and
> > > the actual implementation, be more rigorous with our API definitions,
> and
> > > to (hopefully) speed up / simplify development in the future.
> > >
> > > Migration for us was complicated by the fact that our existing
> codebase was
> > > inconsistent when it came to things like default initialization of
> fields
> > > in POJOs, use of primitives/Objects, and List vs Collection etc. While
> the
> > > code generator is consistent in what it produces (naturally). Our
> codebase
> > > is also fairly tightly coupled between transport classes and model
> classes
> > > as we eschewed the use of models for a lot of it. We did modify the
> > > templates and add extra x- vendor attributes to our spec to control the
> > > generation of certain things where we needed to though, so we were
> able to
> > > maintain some of our existing idiosyncrasies when necessary.
> > >
> > > We were also bitten by some bugs/inconsistencies in the OA3 tooling we
> used
> > > - even though OA3 is a specification, everyone seems to have a slightly
> > > different interpretation or level of support for some parts of it.
> > > The main pain point for us is around support for polymorphic types /
> > > inheritance (
> > >
> https://swagger.io/docs/specification/data-models/oneof-anyof-allof-not/),
> > > where modelling them one way works for one viewer but not another, or
> > > doesn't work for code generators, or api-diffing and so on.
> > >
> > > We use
> > >
> > > - https://www.apicur.io/apicurito to edit the spec
> > > - https://github.com/OpenAPITools/openapi-generator to generate
> Java code
> > > - https://github.com/rapi-doc/RapiDoc as our viewer, we like
> > > https://redocly.com/redoc too
> > > - https://github.com/OpenAPITools/openapi-diff to check for
> (breaking)
> > > changes
> > >
> > > Getting to where we are was quite painful - it would definitely be a
> lot
> > > easier to adopt a spec-first approach in a brand new project - but now
> that
> > > we're here it was probably worth it for the consistency and API-first
> view
> > > it brings.
> > >
> > > So I hope I don't put you off trying it, but just be aware that some
> things
> > > won't work as well as you might expect them to and you might find
> yourself
> > > contributing to some of these OpenAPI projects to get things to work
> the
> > > way you need them to :)
> > >
> > > Colvin
> > >
> > >
> > > On Tue, 9 Aug 2022 at 08:30, Jan Høydahl <ja...@cominvent.com>
> wrote:
> > >
> > >> Makes sense to explore at this point to figure out best order of
> > >> development. And with the door open to slightly adapt some URL
> patterns in
> > >> v2, should it be required, it's less likely that roadblocks would
> derail
> > >> the entire effort.
> > >>
> > >> Jan
> > >>
> > >>> 8. aug. 2022 kl. 18:10 skrev Jason Gerlowski <gerlowskija@gmail.com
> >:
> > >>>
> > >>> Hey all,
> > >>>
> > >>> I spent some time last week digging into OpenAPI and spiking out how
> > >>> it could be integrated into Solr. I came away very impressed. It
> > >>> opens a lot of really cool doors: auto-generated clients and docs,
> > >>> "Swagger UI", backcompat/api-breakage detection, etc. There's a LOT
> > >>> to gain.
> > >>>
> > >>> Another takeaway from my spike though was that OpenAPI is much more
> > >>> "do-able" in JAX-RS projects. It's possible to create
> > >>> annotation-drive OpenAPI specs without JAX-RS, but it requires more
> > >>> explicit (and duplicative) documentation of each API's inputs and
> > >>> ouputs, which probably isn't maintainable in a project as large as
> > >>> ours.
> > >>>
> > >>> With that in mind, I'm planning on returning to the JAX-RS spike Eric
> > >>> and I did months back, and updating it to cover some of the issues
> > >>> this thread brought to light. In particular: security integration
> and
> > >>> serving collection/core-specific APIs. If that pans out, we can
> > >>> figure out next steps from there. (A SIP? A JIRA?)
> > >>>
> > >>> Before I start that effort though, I figured it'd be worth
> > >>> double-checking that there are no -1's/vetos on the idea
> sight-unseen?
> > >>> If so, let me know!
> > >>>
> > >>> Best,
> > >>>
> > >>> Jason
> > >>>
> > >>>
> > >>> On Fri, Jan 7, 2022 at 5:36 PM Eric Pugh
> > >>> <ep...@opensourceconnections.com> wrote:
> > >>>>
> > >>>> I wanted to share that our hack day did happen, but it went a bit
> > >> sideways as we spent the first half of the day experimenting with how
> to
> > >> support CORS in Solr. So API related, but not JAX-RS API specific.
> The
> > >> second half of the day got consumed by $dayjob.
> > >>>>
> > >>>> We’re going to pick it up again next month, and dig into trying out
> how
> > >> existing Solr security would work.
> > >>>>
> > >>>> https://github.com/gerlowskija/solr/tree/cors_stuff if you want to
> see.
> > >>>>
> > >>>> Eric
> > >>>>
> > >>>>
> > >>>> On Dec 9, 2021, at 10:06 AM, Eric Pugh <
> epugh@opensourceconnections.com>
> > >> wrote:
> > >>>>
> > >>>> Thank everyone for the input, it’s been a productive conversation!
> > >>>>
> > >>>> Jason and I are planning on another hack day Jan 7th to take some of
> > >> the feedback, and work more on how our spike can help meet some
> > >> concerns/show promise, so we’ll report back then!
> > >>>>
> > >>>> We’re planning on zooming during US East Coast hours, and I’ll drop
> the
> > >> Zoom invite in the ASF Slack for anyone who wants to join in and say
> hi!
> > >>>>
> > >>>> Eric
> > >>>>
> > >>>>
> > >>>>
> > >>>> On Dec 7, 2021, at 3:47 PM, Mark Miller <ma...@gmail.com>
> wrote:
> > >>>>
> > >>>> Two cents from the peanut gallery:
> > >>>>
> > >>>> I’ve looked at this before. My opinion:
> > >>>>
> > >>>> Our stuff was a just terrible, take your pick on the api version.
> > >> Reasons are numerous.
> > >>>>
> > >>>> Custom end points is an anti feature. Even worse for cloud.
> > >>>>
> > >>>> JAX-RS looked ridiculously sensible.
> > >>>>
> > >>>>
> > >>>> --
> > >>>> - MRM
> > >>>>
> > >>>>
> > >>>> _______________________
> > >>>> Eric Pugh | Founder & CEO | OpenSource Connections, LLC |
> 434.466.1467
> > >> | http://www.opensourceconnections.com | My Free/Busy
> > >>>> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
> > >>>> This e-mail and all contents, including attachments, is considered
> to
> > >> be Company Confidential unless explicitly stated otherwise,
> regardless of
> > >> whether attachments are marked as such.
> > >>>>
> > >>>>
> > >>>> _______________________
> > >>>> Eric Pugh | Founder & CEO | OpenSource Connections, LLC |
> 434.466.1467
> > >> | http://www.opensourceconnections.com | My Free/Busy
> > >>>> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
> > >>>> This e-mail and all contents, including attachments, is considered
> to
> > >> be Company Confidential unless explicitly stated otherwise,
> regardless of
> > >> whether attachments are marked as such.
> > >>>>
> > >>>
> > >>> ---------------------------------------------------------------------
> > >>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > >>> For additional commands, e-mail: dev-help@solr.apache.org
> > >>>
> > >>
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > >> For additional commands, e-mail: dev-help@solr.apache.org
> > >>
> > >>
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > For additional commands, e-mail: dev-help@solr.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org
>
>
Re: JAX-RS APIs in Solr
Posted by Jason Gerlowski <ge...@gmail.com>.
Thanks for the insight Colvin! Spec-driving-the-code sounds appealing
in a lot of ways (I'd love to have defaults handled uniformly across
Solr, for instance), but I think Jan's right that it's probably
infeasible for Solr.
And "point taken" about expecting some issues with the tooling!
Best,
Jason
On Tue, Aug 9, 2022 at 6:10 AM Jan Høydahl <ja...@cominvent.com> wrote:
>
> I have recent experience from a SpringBoot / Kotlin application where we use code-first annotation driven OpenAPI definition. I think that will be the only feasible way for Solr.
>
> The sweet thing about that approach is that the OpenAPI definitions and documentation is all done inline with the code using @Tag and @Operaton annotations, and there is no drift. You can then generate the OpenAPI spec JSON from code to do (client) code generation. Swagger UI comes for free in the app itself with a simple dependency.
>
> Jan
>
> > 9. aug. 2022 kl. 11:45 skrev Colvin Cowie <co...@gmail.com>:
> >
> > I spent the first few months of this year adopting OpenAPI 3 spec in our
> > product, which has a fairly large API.
> >
> > We previously generated a Swagger 1.0 spec from our Java code, now we're
> > generating our POJOs and JAX-RS endpoints from the OA3 spec.
> >
> > Aside from needing to get off outdated Swagger 1 tools, we were motivated
> > to go spec-first to close the gap between our API designs "on paper" and
> > the actual implementation, be more rigorous with our API definitions, and
> > to (hopefully) speed up / simplify development in the future.
> >
> > Migration for us was complicated by the fact that our existing codebase was
> > inconsistent when it came to things like default initialization of fields
> > in POJOs, use of primitives/Objects, and List vs Collection etc. While the
> > code generator is consistent in what it produces (naturally). Our codebase
> > is also fairly tightly coupled between transport classes and model classes
> > as we eschewed the use of models for a lot of it. We did modify the
> > templates and add extra x- vendor attributes to our spec to control the
> > generation of certain things where we needed to though, so we were able to
> > maintain some of our existing idiosyncrasies when necessary.
> >
> > We were also bitten by some bugs/inconsistencies in the OA3 tooling we used
> > - even though OA3 is a specification, everyone seems to have a slightly
> > different interpretation or level of support for some parts of it.
> > The main pain point for us is around support for polymorphic types /
> > inheritance (
> > https://swagger.io/docs/specification/data-models/oneof-anyof-allof-not/),
> > where modelling them one way works for one viewer but not another, or
> > doesn't work for code generators, or api-diffing and so on.
> >
> > We use
> >
> > - https://www.apicur.io/apicurito to edit the spec
> > - https://github.com/OpenAPITools/openapi-generator to generate Java code
> > - https://github.com/rapi-doc/RapiDoc as our viewer, we like
> > https://redocly.com/redoc too
> > - https://github.com/OpenAPITools/openapi-diff to check for (breaking)
> > changes
> >
> > Getting to where we are was quite painful - it would definitely be a lot
> > easier to adopt a spec-first approach in a brand new project - but now that
> > we're here it was probably worth it for the consistency and API-first view
> > it brings.
> >
> > So I hope I don't put you off trying it, but just be aware that some things
> > won't work as well as you might expect them to and you might find yourself
> > contributing to some of these OpenAPI projects to get things to work the
> > way you need them to :)
> >
> > Colvin
> >
> >
> > On Tue, 9 Aug 2022 at 08:30, Jan Høydahl <ja...@cominvent.com> wrote:
> >
> >> Makes sense to explore at this point to figure out best order of
> >> development. And with the door open to slightly adapt some URL patterns in
> >> v2, should it be required, it's less likely that roadblocks would derail
> >> the entire effort.
> >>
> >> Jan
> >>
> >>> 8. aug. 2022 kl. 18:10 skrev Jason Gerlowski <ge...@gmail.com>:
> >>>
> >>> Hey all,
> >>>
> >>> I spent some time last week digging into OpenAPI and spiking out how
> >>> it could be integrated into Solr. I came away very impressed. It
> >>> opens a lot of really cool doors: auto-generated clients and docs,
> >>> "Swagger UI", backcompat/api-breakage detection, etc. There's a LOT
> >>> to gain.
> >>>
> >>> Another takeaway from my spike though was that OpenAPI is much more
> >>> "do-able" in JAX-RS projects. It's possible to create
> >>> annotation-drive OpenAPI specs without JAX-RS, but it requires more
> >>> explicit (and duplicative) documentation of each API's inputs and
> >>> ouputs, which probably isn't maintainable in a project as large as
> >>> ours.
> >>>
> >>> With that in mind, I'm planning on returning to the JAX-RS spike Eric
> >>> and I did months back, and updating it to cover some of the issues
> >>> this thread brought to light. In particular: security integration and
> >>> serving collection/core-specific APIs. If that pans out, we can
> >>> figure out next steps from there. (A SIP? A JIRA?)
> >>>
> >>> Before I start that effort though, I figured it'd be worth
> >>> double-checking that there are no -1's/vetos on the idea sight-unseen?
> >>> If so, let me know!
> >>>
> >>> Best,
> >>>
> >>> Jason
> >>>
> >>>
> >>> On Fri, Jan 7, 2022 at 5:36 PM Eric Pugh
> >>> <ep...@opensourceconnections.com> wrote:
> >>>>
> >>>> I wanted to share that our hack day did happen, but it went a bit
> >> sideways as we spent the first half of the day experimenting with how to
> >> support CORS in Solr. So API related, but not JAX-RS API specific. The
> >> second half of the day got consumed by $dayjob.
> >>>>
> >>>> We’re going to pick it up again next month, and dig into trying out how
> >> existing Solr security would work.
> >>>>
> >>>> https://github.com/gerlowskija/solr/tree/cors_stuff if you want to see.
> >>>>
> >>>> Eric
> >>>>
> >>>>
> >>>> On Dec 9, 2021, at 10:06 AM, Eric Pugh <ep...@opensourceconnections.com>
> >> wrote:
> >>>>
> >>>> Thank everyone for the input, it’s been a productive conversation!
> >>>>
> >>>> Jason and I are planning on another hack day Jan 7th to take some of
> >> the feedback, and work more on how our spike can help meet some
> >> concerns/show promise, so we’ll report back then!
> >>>>
> >>>> We’re planning on zooming during US East Coast hours, and I’ll drop the
> >> Zoom invite in the ASF Slack for anyone who wants to join in and say hi!
> >>>>
> >>>> Eric
> >>>>
> >>>>
> >>>>
> >>>> On Dec 7, 2021, at 3:47 PM, Mark Miller <ma...@gmail.com> wrote:
> >>>>
> >>>> Two cents from the peanut gallery:
> >>>>
> >>>> I’ve looked at this before. My opinion:
> >>>>
> >>>> Our stuff was a just terrible, take your pick on the api version.
> >> Reasons are numerous.
> >>>>
> >>>> Custom end points is an anti feature. Even worse for cloud.
> >>>>
> >>>> JAX-RS looked ridiculously sensible.
> >>>>
> >>>>
> >>>> --
> >>>> - MRM
> >>>>
> >>>>
> >>>> _______________________
> >>>> Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467
> >> | http://www.opensourceconnections.com | My Free/Busy
> >>>> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
> >>>> This e-mail and all contents, including attachments, is considered to
> >> be Company Confidential unless explicitly stated otherwise, regardless of
> >> whether attachments are marked as such.
> >>>>
> >>>>
> >>>> _______________________
> >>>> Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467
> >> | http://www.opensourceconnections.com | My Free/Busy
> >>>> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
> >>>> This e-mail and all contents, including attachments, is considered to
> >> be Company Confidential unless explicitly stated otherwise, regardless of
> >> whether attachments are marked as such.
> >>>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> >>> For additional commands, e-mail: dev-help@solr.apache.org
> >>>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> >> For additional commands, e-mail: dev-help@solr.apache.org
> >>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: JAX-RS APIs in Solr
Posted by Jan Høydahl <ja...@cominvent.com>.
I have recent experience from a SpringBoot / Kotlin application where we use code-first annotation driven OpenAPI definition. I think that will be the only feasible way for Solr.
The sweet thing about that approach is that the OpenAPI definitions and documentation is all done inline with the code using @Tag and @Operaton annotations, and there is no drift. You can then generate the OpenAPI spec JSON from code to do (client) code generation. Swagger UI comes for free in the app itself with a simple dependency.
Jan
> 9. aug. 2022 kl. 11:45 skrev Colvin Cowie <co...@gmail.com>:
>
> I spent the first few months of this year adopting OpenAPI 3 spec in our
> product, which has a fairly large API.
>
> We previously generated a Swagger 1.0 spec from our Java code, now we're
> generating our POJOs and JAX-RS endpoints from the OA3 spec.
>
> Aside from needing to get off outdated Swagger 1 tools, we were motivated
> to go spec-first to close the gap between our API designs "on paper" and
> the actual implementation, be more rigorous with our API definitions, and
> to (hopefully) speed up / simplify development in the future.
>
> Migration for us was complicated by the fact that our existing codebase was
> inconsistent when it came to things like default initialization of fields
> in POJOs, use of primitives/Objects, and List vs Collection etc. While the
> code generator is consistent in what it produces (naturally). Our codebase
> is also fairly tightly coupled between transport classes and model classes
> as we eschewed the use of models for a lot of it. We did modify the
> templates and add extra x- vendor attributes to our spec to control the
> generation of certain things where we needed to though, so we were able to
> maintain some of our existing idiosyncrasies when necessary.
>
> We were also bitten by some bugs/inconsistencies in the OA3 tooling we used
> - even though OA3 is a specification, everyone seems to have a slightly
> different interpretation or level of support for some parts of it.
> The main pain point for us is around support for polymorphic types /
> inheritance (
> https://swagger.io/docs/specification/data-models/oneof-anyof-allof-not/),
> where modelling them one way works for one viewer but not another, or
> doesn't work for code generators, or api-diffing and so on.
>
> We use
>
> - https://www.apicur.io/apicurito to edit the spec
> - https://github.com/OpenAPITools/openapi-generator to generate Java code
> - https://github.com/rapi-doc/RapiDoc as our viewer, we like
> https://redocly.com/redoc too
> - https://github.com/OpenAPITools/openapi-diff to check for (breaking)
> changes
>
> Getting to where we are was quite painful - it would definitely be a lot
> easier to adopt a spec-first approach in a brand new project - but now that
> we're here it was probably worth it for the consistency and API-first view
> it brings.
>
> So I hope I don't put you off trying it, but just be aware that some things
> won't work as well as you might expect them to and you might find yourself
> contributing to some of these OpenAPI projects to get things to work the
> way you need them to :)
>
> Colvin
>
>
> On Tue, 9 Aug 2022 at 08:30, Jan Høydahl <ja...@cominvent.com> wrote:
>
>> Makes sense to explore at this point to figure out best order of
>> development. And with the door open to slightly adapt some URL patterns in
>> v2, should it be required, it's less likely that roadblocks would derail
>> the entire effort.
>>
>> Jan
>>
>>> 8. aug. 2022 kl. 18:10 skrev Jason Gerlowski <ge...@gmail.com>:
>>>
>>> Hey all,
>>>
>>> I spent some time last week digging into OpenAPI and spiking out how
>>> it could be integrated into Solr. I came away very impressed. It
>>> opens a lot of really cool doors: auto-generated clients and docs,
>>> "Swagger UI", backcompat/api-breakage detection, etc. There's a LOT
>>> to gain.
>>>
>>> Another takeaway from my spike though was that OpenAPI is much more
>>> "do-able" in JAX-RS projects. It's possible to create
>>> annotation-drive OpenAPI specs without JAX-RS, but it requires more
>>> explicit (and duplicative) documentation of each API's inputs and
>>> ouputs, which probably isn't maintainable in a project as large as
>>> ours.
>>>
>>> With that in mind, I'm planning on returning to the JAX-RS spike Eric
>>> and I did months back, and updating it to cover some of the issues
>>> this thread brought to light. In particular: security integration and
>>> serving collection/core-specific APIs. If that pans out, we can
>>> figure out next steps from there. (A SIP? A JIRA?)
>>>
>>> Before I start that effort though, I figured it'd be worth
>>> double-checking that there are no -1's/vetos on the idea sight-unseen?
>>> If so, let me know!
>>>
>>> Best,
>>>
>>> Jason
>>>
>>>
>>> On Fri, Jan 7, 2022 at 5:36 PM Eric Pugh
>>> <ep...@opensourceconnections.com> wrote:
>>>>
>>>> I wanted to share that our hack day did happen, but it went a bit
>> sideways as we spent the first half of the day experimenting with how to
>> support CORS in Solr. So API related, but not JAX-RS API specific. The
>> second half of the day got consumed by $dayjob.
>>>>
>>>> We’re going to pick it up again next month, and dig into trying out how
>> existing Solr security would work.
>>>>
>>>> https://github.com/gerlowskija/solr/tree/cors_stuff if you want to see.
>>>>
>>>> Eric
>>>>
>>>>
>>>> On Dec 9, 2021, at 10:06 AM, Eric Pugh <ep...@opensourceconnections.com>
>> wrote:
>>>>
>>>> Thank everyone for the input, it’s been a productive conversation!
>>>>
>>>> Jason and I are planning on another hack day Jan 7th to take some of
>> the feedback, and work more on how our spike can help meet some
>> concerns/show promise, so we’ll report back then!
>>>>
>>>> We’re planning on zooming during US East Coast hours, and I’ll drop the
>> Zoom invite in the ASF Slack for anyone who wants to join in and say hi!
>>>>
>>>> Eric
>>>>
>>>>
>>>>
>>>> On Dec 7, 2021, at 3:47 PM, Mark Miller <ma...@gmail.com> wrote:
>>>>
>>>> Two cents from the peanut gallery:
>>>>
>>>> I’ve looked at this before. My opinion:
>>>>
>>>> Our stuff was a just terrible, take your pick on the api version.
>> Reasons are numerous.
>>>>
>>>> Custom end points is an anti feature. Even worse for cloud.
>>>>
>>>> JAX-RS looked ridiculously sensible.
>>>>
>>>>
>>>> --
>>>> - MRM
>>>>
>>>>
>>>> _______________________
>>>> Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467
>> | http://www.opensourceconnections.com | My Free/Busy
>>>> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
>>>> This e-mail and all contents, including attachments, is considered to
>> be Company Confidential unless explicitly stated otherwise, regardless of
>> whether attachments are marked as such.
>>>>
>>>>
>>>> _______________________
>>>> Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467
>> | http://www.opensourceconnections.com | My Free/Busy
>>>> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
>>>> This e-mail and all contents, including attachments, is considered to
>> be Company Confidential unless explicitly stated otherwise, regardless of
>> whether attachments are marked as such.
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>>> For additional commands, e-mail: dev-help@solr.apache.org
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>> For additional commands, e-mail: dev-help@solr.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: JAX-RS APIs in Solr
Posted by Colvin Cowie <co...@gmail.com>.
I spent the first few months of this year adopting OpenAPI 3 spec in our
product, which has a fairly large API.
We previously generated a Swagger 1.0 spec from our Java code, now we're
generating our POJOs and JAX-RS endpoints from the OA3 spec.
Aside from needing to get off outdated Swagger 1 tools, we were motivated
to go spec-first to close the gap between our API designs "on paper" and
the actual implementation, be more rigorous with our API definitions, and
to (hopefully) speed up / simplify development in the future.
Migration for us was complicated by the fact that our existing codebase was
inconsistent when it came to things like default initialization of fields
in POJOs, use of primitives/Objects, and List vs Collection etc. While the
code generator is consistent in what it produces (naturally). Our codebase
is also fairly tightly coupled between transport classes and model classes
as we eschewed the use of models for a lot of it. We did modify the
templates and add extra x- vendor attributes to our spec to control the
generation of certain things where we needed to though, so we were able to
maintain some of our existing idiosyncrasies when necessary.
We were also bitten by some bugs/inconsistencies in the OA3 tooling we used
- even though OA3 is a specification, everyone seems to have a slightly
different interpretation or level of support for some parts of it.
The main pain point for us is around support for polymorphic types /
inheritance (
https://swagger.io/docs/specification/data-models/oneof-anyof-allof-not/),
where modelling them one way works for one viewer but not another, or
doesn't work for code generators, or api-diffing and so on.
We use
- https://www.apicur.io/apicurito to edit the spec
- https://github.com/OpenAPITools/openapi-generator to generate Java code
- https://github.com/rapi-doc/RapiDoc as our viewer, we like
https://redocly.com/redoc too
- https://github.com/OpenAPITools/openapi-diff to check for (breaking)
changes
Getting to where we are was quite painful - it would definitely be a lot
easier to adopt a spec-first approach in a brand new project - but now that
we're here it was probably worth it for the consistency and API-first view
it brings.
So I hope I don't put you off trying it, but just be aware that some things
won't work as well as you might expect them to and you might find yourself
contributing to some of these OpenAPI projects to get things to work the
way you need them to :)
Colvin
On Tue, 9 Aug 2022 at 08:30, Jan Høydahl <ja...@cominvent.com> wrote:
> Makes sense to explore at this point to figure out best order of
> development. And with the door open to slightly adapt some URL patterns in
> v2, should it be required, it's less likely that roadblocks would derail
> the entire effort.
>
> Jan
>
> > 8. aug. 2022 kl. 18:10 skrev Jason Gerlowski <ge...@gmail.com>:
> >
> > Hey all,
> >
> > I spent some time last week digging into OpenAPI and spiking out how
> > it could be integrated into Solr. I came away very impressed. It
> > opens a lot of really cool doors: auto-generated clients and docs,
> > "Swagger UI", backcompat/api-breakage detection, etc. There's a LOT
> > to gain.
> >
> > Another takeaway from my spike though was that OpenAPI is much more
> > "do-able" in JAX-RS projects. It's possible to create
> > annotation-drive OpenAPI specs without JAX-RS, but it requires more
> > explicit (and duplicative) documentation of each API's inputs and
> > ouputs, which probably isn't maintainable in a project as large as
> > ours.
> >
> > With that in mind, I'm planning on returning to the JAX-RS spike Eric
> > and I did months back, and updating it to cover some of the issues
> > this thread brought to light. In particular: security integration and
> > serving collection/core-specific APIs. If that pans out, we can
> > figure out next steps from there. (A SIP? A JIRA?)
> >
> > Before I start that effort though, I figured it'd be worth
> > double-checking that there are no -1's/vetos on the idea sight-unseen?
> > If so, let me know!
> >
> > Best,
> >
> > Jason
> >
> >
> > On Fri, Jan 7, 2022 at 5:36 PM Eric Pugh
> > <ep...@opensourceconnections.com> wrote:
> >>
> >> I wanted to share that our hack day did happen, but it went a bit
> sideways as we spent the first half of the day experimenting with how to
> support CORS in Solr. So API related, but not JAX-RS API specific. The
> second half of the day got consumed by $dayjob.
> >>
> >> We’re going to pick it up again next month, and dig into trying out how
> existing Solr security would work.
> >>
> >> https://github.com/gerlowskija/solr/tree/cors_stuff if you want to see.
> >>
> >> Eric
> >>
> >>
> >> On Dec 9, 2021, at 10:06 AM, Eric Pugh <ep...@opensourceconnections.com>
> wrote:
> >>
> >> Thank everyone for the input, it’s been a productive conversation!
> >>
> >> Jason and I are planning on another hack day Jan 7th to take some of
> the feedback, and work more on how our spike can help meet some
> concerns/show promise, so we’ll report back then!
> >>
> >> We’re planning on zooming during US East Coast hours, and I’ll drop the
> Zoom invite in the ASF Slack for anyone who wants to join in and say hi!
> >>
> >> Eric
> >>
> >>
> >>
> >> On Dec 7, 2021, at 3:47 PM, Mark Miller <ma...@gmail.com> wrote:
> >>
> >> Two cents from the peanut gallery:
> >>
> >> I’ve looked at this before. My opinion:
> >>
> >> Our stuff was a just terrible, take your pick on the api version.
> Reasons are numerous.
> >>
> >> Custom end points is an anti feature. Even worse for cloud.
> >>
> >> JAX-RS looked ridiculously sensible.
> >>
> >>
> >> --
> >> - MRM
> >>
> >>
> >> _______________________
> >> Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467
> | http://www.opensourceconnections.com | My Free/Busy
> >> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
> >> This e-mail and all contents, including attachments, is considered to
> be Company Confidential unless explicitly stated otherwise, regardless of
> whether attachments are marked as such.
> >>
> >>
> >> _______________________
> >> Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467
> | http://www.opensourceconnections.com | My Free/Busy
> >> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
> >> This e-mail and all contents, including attachments, is considered to
> be Company Confidential unless explicitly stated otherwise, regardless of
> whether attachments are marked as such.
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > For additional commands, e-mail: dev-help@solr.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org
>
>
Re: JAX-RS APIs in Solr
Posted by Jan Høydahl <ja...@cominvent.com>.
Makes sense to explore at this point to figure out best order of development. And with the door open to slightly adapt some URL patterns in v2, should it be required, it's less likely that roadblocks would derail the entire effort.
Jan
> 8. aug. 2022 kl. 18:10 skrev Jason Gerlowski <ge...@gmail.com>:
>
> Hey all,
>
> I spent some time last week digging into OpenAPI and spiking out how
> it could be integrated into Solr. I came away very impressed. It
> opens a lot of really cool doors: auto-generated clients and docs,
> "Swagger UI", backcompat/api-breakage detection, etc. There's a LOT
> to gain.
>
> Another takeaway from my spike though was that OpenAPI is much more
> "do-able" in JAX-RS projects. It's possible to create
> annotation-drive OpenAPI specs without JAX-RS, but it requires more
> explicit (and duplicative) documentation of each API's inputs and
> ouputs, which probably isn't maintainable in a project as large as
> ours.
>
> With that in mind, I'm planning on returning to the JAX-RS spike Eric
> and I did months back, and updating it to cover some of the issues
> this thread brought to light. In particular: security integration and
> serving collection/core-specific APIs. If that pans out, we can
> figure out next steps from there. (A SIP? A JIRA?)
>
> Before I start that effort though, I figured it'd be worth
> double-checking that there are no -1's/vetos on the idea sight-unseen?
> If so, let me know!
>
> Best,
>
> Jason
>
>
> On Fri, Jan 7, 2022 at 5:36 PM Eric Pugh
> <ep...@opensourceconnections.com> wrote:
>>
>> I wanted to share that our hack day did happen, but it went a bit sideways as we spent the first half of the day experimenting with how to support CORS in Solr. So API related, but not JAX-RS API specific. The second half of the day got consumed by $dayjob.
>>
>> We’re going to pick it up again next month, and dig into trying out how existing Solr security would work.
>>
>> https://github.com/gerlowskija/solr/tree/cors_stuff if you want to see.
>>
>> Eric
>>
>>
>> On Dec 9, 2021, at 10:06 AM, Eric Pugh <ep...@opensourceconnections.com> wrote:
>>
>> Thank everyone for the input, it’s been a productive conversation!
>>
>> Jason and I are planning on another hack day Jan 7th to take some of the feedback, and work more on how our spike can help meet some concerns/show promise, so we’ll report back then!
>>
>> We’re planning on zooming during US East Coast hours, and I’ll drop the Zoom invite in the ASF Slack for anyone who wants to join in and say hi!
>>
>> Eric
>>
>>
>>
>> On Dec 7, 2021, at 3:47 PM, Mark Miller <ma...@gmail.com> wrote:
>>
>> Two cents from the peanut gallery:
>>
>> I’ve looked at this before. My opinion:
>>
>> Our stuff was a just terrible, take your pick on the api version. Reasons are numerous.
>>
>> Custom end points is an anti feature. Even worse for cloud.
>>
>> JAX-RS looked ridiculously sensible.
>>
>>
>> --
>> - MRM
>>
>>
>> _______________________
>> Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | http://www.opensourceconnections.com | My Free/Busy
>> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
>> This e-mail and all contents, including attachments, is considered to be Company Confidential unless explicitly stated otherwise, regardless of whether attachments are marked as such.
>>
>>
>> _______________________
>> Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | http://www.opensourceconnections.com | My Free/Busy
>> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
>> This e-mail and all contents, including attachments, is considered to be Company Confidential unless explicitly stated otherwise, regardless of whether attachments are marked as such.
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: JAX-RS APIs in Solr
Posted by Jason Gerlowski <ge...@gmail.com>.
Hey all,
I spent some time last week digging into OpenAPI and spiking out how
it could be integrated into Solr. I came away very impressed. It
opens a lot of really cool doors: auto-generated clients and docs,
"Swagger UI", backcompat/api-breakage detection, etc. There's a LOT
to gain.
Another takeaway from my spike though was that OpenAPI is much more
"do-able" in JAX-RS projects. It's possible to create
annotation-drive OpenAPI specs without JAX-RS, but it requires more
explicit (and duplicative) documentation of each API's inputs and
ouputs, which probably isn't maintainable in a project as large as
ours.
With that in mind, I'm planning on returning to the JAX-RS spike Eric
and I did months back, and updating it to cover some of the issues
this thread brought to light. In particular: security integration and
serving collection/core-specific APIs. If that pans out, we can
figure out next steps from there. (A SIP? A JIRA?)
Before I start that effort though, I figured it'd be worth
double-checking that there are no -1's/vetos on the idea sight-unseen?
If so, let me know!
Best,
Jason
On Fri, Jan 7, 2022 at 5:36 PM Eric Pugh
<ep...@opensourceconnections.com> wrote:
>
> I wanted to share that our hack day did happen, but it went a bit sideways as we spent the first half of the day experimenting with how to support CORS in Solr. So API related, but not JAX-RS API specific. The second half of the day got consumed by $dayjob.
>
> We’re going to pick it up again next month, and dig into trying out how existing Solr security would work.
>
> https://github.com/gerlowskija/solr/tree/cors_stuff if you want to see.
>
> Eric
>
>
> On Dec 9, 2021, at 10:06 AM, Eric Pugh <ep...@opensourceconnections.com> wrote:
>
> Thank everyone for the input, it’s been a productive conversation!
>
> Jason and I are planning on another hack day Jan 7th to take some of the feedback, and work more on how our spike can help meet some concerns/show promise, so we’ll report back then!
>
> We’re planning on zooming during US East Coast hours, and I’ll drop the Zoom invite in the ASF Slack for anyone who wants to join in and say hi!
>
> Eric
>
>
>
> On Dec 7, 2021, at 3:47 PM, Mark Miller <ma...@gmail.com> wrote:
>
> Two cents from the peanut gallery:
>
> I’ve looked at this before. My opinion:
>
> Our stuff was a just terrible, take your pick on the api version. Reasons are numerous.
>
> Custom end points is an anti feature. Even worse for cloud.
>
> JAX-RS looked ridiculously sensible.
>
>
> --
> - MRM
>
>
> _______________________
> Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | http://www.opensourceconnections.com | My Free/Busy
> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
> This e-mail and all contents, including attachments, is considered to be Company Confidential unless explicitly stated otherwise, regardless of whether attachments are marked as such.
>
>
> _______________________
> Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | http://www.opensourceconnections.com | My Free/Busy
> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
> This e-mail and all contents, including attachments, is considered to be Company Confidential unless explicitly stated otherwise, regardless of whether attachments are marked as such.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: JAX-RS APIs in Solr
Posted by Eric Pugh <ep...@opensourceconnections.com>.
I wanted to share that our hack day did happen, but it went a bit sideways as we spent the first half of the day experimenting with how to support CORS in Solr. So API related, but not JAX-RS API specific. The second half of the day got consumed by $dayjob.
We’re going to pick it up again next month, and dig into trying out how existing Solr security would work.
https://github.com/gerlowskija/solr/tree/cors_stuff <https://github.com/gerlowskija/solr/tree/cors_stuff> if you want to see.
Eric
> On Dec 9, 2021, at 10:06 AM, Eric Pugh <ep...@opensourceconnections.com> wrote:
>
> Thank everyone for the input, it’s been a productive conversation!
>
> Jason and I are planning on another hack day Jan 7th to take some of the feedback, and work more on how our spike can help meet some concerns/show promise, so we’ll report back then!
>
> We’re planning on zooming during US East Coast hours, and I’ll drop the Zoom invite in the ASF Slack for anyone who wants to join in and say hi!
>
> Eric
>
>
>
>> On Dec 7, 2021, at 3:47 PM, Mark Miller <markrmiller@gmail.com <ma...@gmail.com>> wrote:
>>
>> Two cents from the peanut gallery:
>>
>> I’ve looked at this before. My opinion:
>>
>> Our stuff was a just terrible, take your pick on the api version. Reasons are numerous.
>>
>> Custom end points is an anti feature. Even worse for cloud.
>>
>> JAX-RS looked ridiculously sensible.
>>
>>
>> --
>> - MRM
>
> _______________________
> Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | http://www.opensourceconnections.com <http://www.opensourceconnections.com/> | My Free/Busy <http://tinyurl.com/eric-cal>
> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed <https://www.packtpub.com/big-data-and-business-intelligence/apache-solr-enterprise-search-server-third-edition-raw>
> This e-mail and all contents, including attachments, is considered to be Company Confidential unless explicitly stated otherwise, regardless of whether attachments are marked as such.
>
_______________________
Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | http://www.opensourceconnections.com <http://www.opensourceconnections.com/> | My Free/Busy <http://tinyurl.com/eric-cal>
Co-Author: Apache Solr Enterprise Search Server, 3rd Ed <https://www.packtpub.com/big-data-and-business-intelligence/apache-solr-enterprise-search-server-third-edition-raw>
This e-mail and all contents, including attachments, is considered to be Company Confidential unless explicitly stated otherwise, regardless of whether attachments are marked as such.
Re: JAX-RS APIs in Solr
Posted by Eric Pugh <ep...@opensourceconnections.com>.
Thank everyone for the input, it’s been a productive conversation!
Jason and I are planning on another hack day Jan 7th to take some of the feedback, and work more on how our spike can help meet some concerns/show promise, so we’ll report back then!
We’re planning on zooming during US East Coast hours, and I’ll drop the Zoom invite in the ASF Slack for anyone who wants to join in and say hi!
Eric
> On Dec 7, 2021, at 3:47 PM, Mark Miller <ma...@gmail.com> wrote:
>
> Two cents from the peanut gallery:
>
> I’ve looked at this before. My opinion:
>
> Our stuff was a just terrible, take your pick on the api version. Reasons are numerous.
>
> Custom end points is an anti feature. Even worse for cloud.
>
> JAX-RS looked ridiculously sensible.
>
>
> --
> - MRM
_______________________
Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | http://www.opensourceconnections.com <http://www.opensourceconnections.com/> | My Free/Busy <http://tinyurl.com/eric-cal>
Co-Author: Apache Solr Enterprise Search Server, 3rd Ed <https://www.packtpub.com/big-data-and-business-intelligence/apache-solr-enterprise-search-server-third-edition-raw>
This e-mail and all contents, including attachments, is considered to be Company Confidential unless explicitly stated otherwise, regardless of whether attachments are marked as such.
Re: JAX-RS APIs in Solr
Posted by Mark Miller <ma...@gmail.com>.
Two cents from the peanut gallery:
I’ve looked at this before. My opinion:
Our stuff was a just terrible, take your pick on the api version. Reasons
are numerous.
Custom end points is an anti feature. Even worse for cloud.
JAX-RS looked ridiculously sensible.
--
- MRM
Re: JAX-RS APIs in Solr
Posted by Gus Heck <gu...@gmail.com>.
some thoughts inline...
On Mon, Dec 6, 2021 at 2:04 PM Noble Paul <no...@gmail.com> wrote:
> I'm +1 for modernization
> I just wanted to highlight the hurdles we need to overcome to reach there
>
> SolrRequestHandler is the current framework that serves all requests in
> Solr. This also has associated ways to
>
> - register end points and configuration
>
> The way we construct our URL's makes this a bit difficult since they are
written /context/object/action rather than context/action/object. If
designed from scratch, /context/action/object would have been be preferable
because the ation would naturally map to a servlet or other service and the
path-info to the thing to act on. This ancient design decision is a hurdle
for applying filters (due to the limited pattens afforded by servlet spec
12.2) with specificity, and for any alternative implementations to mimic.
>
> - manage lifecycle
>
> Most frameworks and certainly every servlet/filter/listener has a built in
lifecycle that we should leverage.
>
> - collect metrics
>
> I haven't looked but I expect there's some need to connect the request
thread with the metrics system and clear out the association upon request
completion, likely a job for a Servlet filter, which could then wrap any of
our services.
>
> - enforce security
>
> Security probably ought to have a few parts:
- Authentication should be standardized to a servlet filter that can be
wrapped around any of our services however they are implemented. The filter
would be in charge of populating a principal, and connecting it with a
system to apply roles from which permissions can be calculated (and
appropriately cache this information in the session so we don't calculate
it on every request... unless there is a use case for permission updates
faster than session expiration, but I suggest not, or at least not by
default).
- A filter inside of (after) the authn filter that checks authz for path
based permissions, possibly also inspectint the request parameters and/or
payload to calculate things like collection or handler and verify that the
user has the required permission. Note that there might be several such
filters, possibly one inferring read/write/update/delete from the request
method, that only wraps API's that consistently utilize http verbs... or
whatever
- In some cases explicit checks in code at critical points to guard
specific functionality might be required, a good security framework will
allow this too. i.e. if (SecurityUtils.getSubject().isPermitted(permission))
{ /* do stuff */ } else { throw AuthzException("or whatever") }
> - logging
>
> MDC context setup/teardown per request for logging should be a Servlet
Filter wrapping any of our services running in jetty.
>
> - tracing
>
> Also a candidate for a servlet filter that can wrap any of our services
>
> - etc etc
>
> The more of this stuff we can extract from SolrDispatchServlet and into
independent filters, the easier it is to just wrap JAX-RS or other new
technologies (assuming we aren't targeting async requests in JAX-RS wherein
things get a bit more complicated.) The fact we have lots of servlet
related code and run inside a servlet container (jetty) means we should be
biased towards JAX-RS implementations that are based on servlets (I think
Jersey is one, quick googling indicates there is a Jersey-Servlet class )
> Introducing a new framework will have to deal with all of the above. For
> instance the JAX-RS is used you will start using web.xml to register your
> JAX-RS endpoints.
>
> --Noble
>
>
Re: JAX-RS APIs in Solr
Posted by Noble Paul <no...@gmail.com>.
I'm +1 for modernization
I just wanted to highlight the hurdles we need to overcome to reach there
SolrRequestHandler is the current framework that serves all requests in
Solr. This also has associated ways to
- register end points and configuration
- manage lifecycle
- collect metrics
- enforce security
- logging
- tracing
- etc etc
Introducing a new framework will have to deal with all of the above. For
instance the JAX-RS is used you will start using web.xml to register your
JAX-RS endpoints.
--Noble
On Mon, Dec 6, 2021 at 6:11 AM David Smiley <ds...@apache.org> wrote:
> Just a simple +1 of support to modernization efforts in general. It's
> encouraging to see that Jason & Eric had some fun together on this.
> Modernization, I think, helps with the fun of any open-source project, and
> thus helps keep everyone interested in continuing and reviewing interest in
> Solr. If we/others feel we can't make fundamental changes, then I think
> our interests (and that of contributors) will wane. Personally, I really
> enjoy refactoring codebases, even if it may not seem sexy to some.
>
> I don't think we can rush into a SIP before more research/POC is tried.
> It's too abstract at this stage. We don't even know what framework to use
> yet ;-)
>
--
-----------------------------------------------------
Noble Paul
Re: JAX-RS APIs in Solr
Posted by Ishan Chattopadhyaya <ic...@gmail.com>.
> We don't even know what framework to use yet ;-)
We should define what we want to achieve before picking the framework. PoCs
are fine at this point for evaluating various options, but we shouldn't let
any framework dictate what we should or shouldn't do for our users.
On Mon, 6 Dec, 2021, 12:41 am David Smiley, <ds...@apache.org> wrote:
> Just a simple +1 of support to modernization efforts in general. It's
> encouraging to see that Jason & Eric had some fun together on this.
> Modernization, I think, helps with the fun of any open-source project, and
> thus helps keep everyone interested in continuing and reviewing interest in
> Solr. If we/others feel we can't make fundamental changes, then I think
> our interests (and that of contributors) will wane. Personally, I really
> enjoy refactoring codebases, even if it may not seem sexy to some.
>
> I don't think we can rush into a SIP before more research/POC is tried.
> It's too abstract at this stage. We don't even know what framework to use
> yet ;-)
>
Re: JAX-RS APIs in Solr
Posted by David Smiley <ds...@apache.org>.
Just a simple +1 of support to modernization efforts in general. It's
encouraging to see that Jason & Eric had some fun together on this.
Modernization, I think, helps with the fun of any open-source project, and
thus helps keep everyone interested in continuing and reviewing interest in
Solr. If we/others feel we can't make fundamental changes, then I think
our interests (and that of contributors) will wane. Personally, I really
enjoy refactoring codebases, even if it may not seem sexy to some.
I don't think we can rush into a SIP before more research/POC is tried.
It's too abstract at this stage. We don't even know what framework to use
yet ;-)
Re: JAX-RS APIs in Solr
Posted by Jan Høydahl <ja...@cominvent.com>.
Looks like Jersey can build endpoints and APIs programmatically too, see https://eclipse-ee4j.github.io/jersey.github.io/documentation/latest3x/resource-builder.html
So I guess it can be done by hooking in at the right places if we want.
It would be really encouraging if we could modernize this part of Solr, we have too many custom hard-to-grasp concepts everywhere that fends off new developers.
Jan
> 30. nov. 2021 kl. 16:37 skrev Ishan Chattopadhyaya <ic...@gmail.com>:
>
>
>
> On Tue, Nov 30, 2021 at 8:44 PM Jason Gerlowski <gerlowskija@gmail.com <ma...@gmail.com>> wrote:
> > We need to make an API work on a per core basis and cores can come up and go down randomly
>
> Ah, yeah. I'll admit I was overlooking some of the wrinkles around
> custom request handlers etc. That is a problem for switching to some
> off-the-shelf framework for sure. It may still be possible, but it's
> definitely a big hurdle. I wonder how commonly users still take
> advantage of the custom requestHandler stuff these days?
>
> Here's YASA using annotations:
> https://github.com/yasa-org/yasa/blob/master/yasa-solr-plugin/src/main/java/io/github/kezhenxu94/YasaHandler.java#L45 <https://github.com/yasa-org/yasa/blob/master/yasa-solr-plugin/src/main/java/io/github/kezhenxu94/YasaHandler.java#L45>
>
> I haven't
> run across many customers who used them extensively, but maybe that's
> self-selecting in some way?
>
> > If SolrJ has a Jackson dependency, it can have a conflict (different versions) with the Jackson used by the client app
>
> SolrJ has a handful of dependencies - is there a reason we take this
> approach with Jackson, but not with any of SolrJ's other deps?
> (apache-httpcomponents, apache-commons, zookeeper-client, etc.)
>
> Granted, I get that we want SolrJ to be as painless as possible for
> its users, and jar-conflicts are a part of that and really suck. But
> there are good tools out there for handling jar conflicts, and
> reimplementing chunks of library functionality just to avoid a
> gradle-dep strikes me as unsustainable from a maintenance/community
> perspective. Not to mention that forking/mimicking Jackson code
> deprives our users of any efficiency/security improvements that
> Jackson might get tomorrow or next week or next year.
>
> Unless there's some reason specific to Jackson I guess I just don't
> get it. But if I'm the minority opinion on that, fair enough I guess.
>
> On Tue, Nov 30, 2021 at 5:05 AM Noble Paul <noble.paul@gmail.com <ma...@gmail.com>> wrote:
> >
> > The @JsonProperty annotations are added because it's a dependency in SolrJ as well
> > If SolrJ has a Jackson dependency, it can have a conflict (different versions) with the Jackson used by the client app.
> >
> > On Tue, Nov 30, 2021, 5:21 PM Noble Paul <noble.paul@gmail.com <ma...@gmail.com>> wrote:
> >>
> >> Also keep in mind that the same endpoint can be accessed with a core name and a collection name prefixes.
> >>
> >> On Tue, Nov 30, 2021 at 3:55 PM Noble Paul <noble.paul@gmail.com <ma...@gmail.com>> wrote:
> >>>
> >>> True Gus, Almost every framework works outside of SolrDispatchFilter+ HttpSolrCall. A lot of our initializations occur there.
> >>>
> >>> We need to make an API work on a per core basis and cores can come up and go down randomly. So we need to register these endpoints on a core.
> >>>
> >>> I'm not sure if any framework can achieve the same.
> >>>
> >>> On Tue, Nov 30, 2021 at 5:39 AM Gus Heck <gus.heck@gmail.com <ma...@gmail.com>> wrote:
> >>>>
> >>>> IIRC last time I looked restlet had the unsavory property of existing outside of the SolrDispatchFilter, unlike everything else which made for special cases because several things that probably ought to be their own siervlet filters are glommed into SolrDispatchFilter, like security, tracing and MDC setup/teardown per request. Restlet wouldn't be so bad if one could just wrap such filters around it too...
> >>>>
> >>>> On Mon, Nov 29, 2021 at 9:42 AM Jason Gerlowski <gerlowskija@gmail.com <ma...@gmail.com>> wrote:
> >>>>>
> >>>>> > These are minor improvements compared to a full rewrite of the entire framework
> >>>>>
> >>>>> If you think data type support is minor, fair enough. But to clarify
> >>>>> I'm not suggesting a rewrite - I'm suggesting using something that
> >>>>> already exists off the shelf. Jersey (e.g.) itself provides the
> >>>>> framework - there would be no "rewrite".
> >>>>>
> >>>>> re: past restlet use
> >>>>>
> >>>>> > It was not playing well with our security framework. The framework was not working well with Solr APIs
> >>>>>
> >>>>> Ah, very interesting! Security isn't something Eric or I tackled in
> >>>>> our little spike branch, but it's definitely a concern. Do you
> >>>>> remember the specific concerns? Or recall where any of the discussion
> >>>>> around this happened?
> >>>>>
> >>>>> Without the context of that past discussion, it seems like the
> >>>>> "PermissionNameProvider" interface could be implemented just as well
> >>>>> by a class with (e.g.) Jersey annotations as one with our own custom
> >>>>> annotations. Certainly there'd need to be some
> >>>>> RuleBasedAuthorizationPlugin changes or other integration code, but
> >>>>> nothing that feels insurmountable.
> >>>>>
> >>>>> Maybe I can try spiking it out soon and find the issues myself, but
> >>>>> it'd be much easier if someone happens to remember and can save me the
> >>>>> trouble :-p
> >>>>>
> >>>>> Best,
> >>>>>
> >>>>> Jason
> >>>>>
> >>>>> On Fri, Nov 26, 2021 at 12:05 AM Noble Paul <noble.paul@gmail.com <ma...@gmail.com>> wrote:
> >>>>> >
> >>>>> > The Annotations framework was written after playing with other frameworks. There were many shortcomings which were hard to overcome.
> >>>>> >
> >>>>> > The best example is a per collection API . How do you register an endpoint for a collection/core ?
> >>>>> >
> >>>>> > On Fri, Nov 26, 2021 at 3:42 PM Noble Paul <noble.paul@gmail.com <ma...@gmail.com>> wrote:
> >>>>> >>
> >>>>> >>
> >>>>> >>
> >>>>> >> On Fri, Nov 26, 2021 at 1:03 AM Jason Gerlowski <gerlowskija@gmail.com <ma...@gmail.com>> wrote:
> >>>>> >>>
> >>>>> >>> > Is there some problem with our annotations that we hope to solve using third party dependencies?
> >>>>> >>>
> >>>>> >>> I guess so yeah. Third-party deps are just fuller, more robust
> >>>>> >>> solutions, whereas our annotations still need support added now and
> >>>>> >>> then for even primitive data types like "long" (see SOLR-15619).
> >>>>> >>
> >>>>> >> These are minor improvements compared to a full rewrite of the entire framework
> >>>>> >>
> >>>>> >>
> >>>>> >>>
> >>>>> >>> Every JIRA spent doing basic stuff like that is time away from
> >>>>> >>> improving Solr in some other way.
> >>>>> >>>
> >>>>> >>> So there are feature-gap/capabilities arguments for moving to a
> >>>>> >>> third-party dep, sure. But, even if our annotations did everything
> >>>>> >>> Jersey+Jackson do today, I think switching would still be worth it.
> >>>>> >>> Every LOC in our code base brings along with it some maintenance cost:
> >>>>> >>> it might have bugs, needs tested, takes time for new contributors to
> >>>>> >>> "grok", etc. Using off-the-shelf here would nuke a whole bunch of
> >>>>> >>> that. If off-the-shelf is available for some given functionality, we
> >>>>> >>> should need a compelling reason to NOT use it.
> >>>>> >>>
> >>>>> >>> Lastly, I think there's an "approachability" argument for using
> >>>>> >>> off-the-shelf. Thousands of developers out there are familiar with
> >>>>> >>> (e.g.) Jersey, compared to maybe 15 or 20 (in the world) familiar with
> >>>>> >>> Solr's custom annotations. Using a well-known technology like Jersey
> >>>>> >>> would make Solr all the easier to approach and contribute to for that
> >>>>> >>> pool of developers.
> >>>>> >>>
> >>>>> >>> > By the way, we have used Restlet in the past and that has been a regrettable decision.
> >>>>> >>
> >>>>> >>
> >>>>> >>>
> >>>>> >>> Ah, yeah, that's just the context I'm missing. Anyone have a pointer
> >>>>> >>> to related discussions, or remember what made this "regrettable"? All
> >>>>> >>> the theoretical benefits in the world don't matter much if we've
> >>>>> >>> already tried something like this in the past and decided against it.
> >>>>> >>
> >>>>> >>
> >>>>> >> It was not playing well with our security framework. The framework was not working well with Solr APIs
> >>>>> >>
> >>>>> >>>
> >>>>> >>> (Unrelated - Happy Thanksgiving all!)
> >>>>> >>>
> >>>>> >>> Best,
> >>>>> >>>
> >>>>> >>> Jason
> >>>>> >>>
> >>>>> >>> On Thu, Nov 25, 2021 at 7:32 AM Noble Paul <noble.paul@gmail.com <ma...@gmail.com>> wrote:
> >>>>> >>> >
> >>>>> >>> > Have you gone through an API written using the @EndPoint annotation?
> >>>>> >>> >
> >>>>> >>> > I strongly recommend that you do
> >>>>> >>> >
> >>>>> >>> > On Thu, Nov 25, 2021, 11:30 PM Eric Pugh <epugh@opensourceconnections.com <ma...@opensourceconnections.com>> wrote:
> >>>>> >>> >>
> >>>>> >>> >> I have found our V2 API code to be very impenetrable to understand. Part of it is how the code is intertwined with support for V1, however it’s also because there aren’t really resources to go look at to understand how it should work! Maintaining the API should be very simple work, as they just exist as a translation. The home grown stuff may make sense if you are a super knowledgable Solr developer, but if you are just a new person, it’s a lot harder to contribute.
> >>>>> >>> >>
> >>>>> >>> >> I was interested in the Jersey stuff because I’ve seen lots of projects use it very successfully, and if I want to implement something, well, there are lots of blogs and resources out there!
> >>>>> >>> >>
> >>>>> >>> >> Can anyone recap briefly why we dropped RESTlet? And what lessons learned there might apply to adopting Jersey for API support? Looking at https://issues.apache.org/jira/browse/SOLR-14659 <https://issues.apache.org/jira/browse/SOLR-14659>, it was partly deprecated because we were not using it to support all the API, only the ManagedResource ones, and https://issues.apache.org/jira/browse/SOLR-14766 <https://issues.apache.org/jira/browse/SOLR-14766> suggests that RESTlet maybe was no longer being updated? One reason why we spiked out Jersey was because of the broad support in the Java world! Looking at how much work we have to do in the V2 API world, we need a much broader pool of developers contributing to get there!
> >>>>> >>> >>
> >>>>> >>> >> Related, are there specific features/aspects of our annotations that enable things in Solr that couldn’t be done otherwise?
> >>>>> >>> >>
> >>>>> >>> >> On Nov 25, 2021, at 2:12 AM, Ishan Chattopadhyaya <ichattopadhyaya@gmail.com <ma...@gmail.com>> wrote:
> >>>>> >>> >>
> >>>>> >>> >> Is there some problem with our annotations that we hope to solve using third party dependencies?
> >>>>> >>> >> By the way, we have used Restlet in the past and that has been a regrettable decision.
> >>>>> >>> >>
> >>>>> >>> >> On Thu, Nov 25, 2021 at 10:10 AM Jason Gerlowski <gerlowskija@gmail.com <ma...@gmail.com>> wrote:
> >>>>> >>> >>>
> >>>>> >>> >>> Solr's custom annotation framework ('@Endpoint', '@Command', etc.) has
> >>>>> >>> >>> cropped up a few times over the past week or two. [1] [2]. Having them
> >>>>> >>> >>> on top of mind, I've been wondering - is there a reason we use our own
> >>>>> >>> >>> annotations here instead of something off the shelf?
> >>>>> >>> >>>
> >>>>> >>> >>> What we have works well enough, but anything homegrown comes with more
> >>>>> >>> >>> maintenance burden than we'd have if we used something off the shelf.
> >>>>> >>> >>> There are plenty of well-used, active projects out there whose whole
> >>>>> >>> >>> purpose is facilitating the whole "annotation based API" thing
> >>>>> >>> >>> (Jersey, Restlet, RESTEasy, etc.) - why not use one of them?
> >>>>> >>> >>>
> >>>>> >>> >>> Does anyone know of any technical reasons why we can't go this route?
> >>>>> >>> >>> Or have any subjective reasons why we shouldn't? Or any context on
> >>>>> >>> >>> why we wrote our own Endpoint, Command, JsonProperty annotations
> >>>>> >>> >>> originally?
> >>>>> >>> >>>
> >>>>> >>> >>> FWIW, Eric Pugh and I spiked out a small POC recently, and got
> >>>>> >>> >>> Jersey+Jackson working for a few simple APIs without too much trouble.
> >>>>> >>> >>> [3] Obviously nothing production-ready there, and there's still a lot
> >>>>> >>> >>> of open questions (e.g. how would javabin be supported?), but we both
> >>>>> >>> >>> came away convinced that it seemed feasible, at least. Best of all,
> >>>>> >>> >>> APIs using our current homegrown annotation framework the switchover
> >>>>> >>> >>> seems blessedly straightforward, and it doesn't look like Jersey
> >>>>> >>> >>> (which we chose mostly arbitrarily) bloats our dist all that much.
> >>>>> >>> >>>
> >>>>> >>> >>> Curious if anyone has thoughts or context on how we ended up with the
> >>>>> >>> >>> annotation setup we use today!
> >>>>> >>> >>>
> >>>>> >>> >>> Best,
> >>>>> >>> >>>
> >>>>> >>> >>> Jason
> >>>>> >>> >>>
> >>>>> >>> >>> [1] https://issues.apache.org/jira/browse/SOLR-15182 <https://issues.apache.org/jira/browse/SOLR-15182> (and children)
> >>>>> >>> >>> [2] http://mail-archives.apache.org/mod_mbox/solr-dev/202111.mbox/%3CCABEwPvENL41Pm6%2BOmjXb6Sx5N2XjUtnbWhgKOZSrnLjWBA8tcA%40mail.gmail.com%3E <http://mail-archives.apache.org/mod_mbox/solr-dev/202111.mbox/%3CCABEwPvENL41Pm6%2BOmjXb6Sx5N2XjUtnbWhgKOZSrnLjWBA8tcA%40mail.gmail.com%3E>
> >>>>> >>> >>> [3] https://github.com/gerlowskija/solr/tree/jersey_jaxrs_jackson_solr_apis <https://github.com/gerlowskija/solr/tree/jersey_jaxrs_jackson_solr_apis>.
> >>>>> >>> >>>
> >>>>> >>> >>> ---------------------------------------------------------------------
> >>>>> >>> >>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org <ma...@solr.apache.org>
> >>>>> >>> >>> For additional commands, e-mail: dev-help@solr.apache.org <ma...@solr.apache.org>
> >>>>> >>> >>>
> >>>>> >>> >>
> >>>>> >>> >> _______________________
> >>>>> >>> >> Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | http://www.opensourceconnections.com <http://www.opensourceconnections.com/> | My Free/Busy
> >>>>> >>> >> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
> >>>>> >>> >> This e-mail and all contents, including attachments, is considered to be Company Confidential unless explicitly stated otherwise, regardless of whether attachments are marked as such.
> >>>>> >>> >>
> >>>>> >>>
> >>>>> >>> ---------------------------------------------------------------------
> >>>>> >>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org <ma...@solr.apache.org>
> >>>>> >>> For additional commands, e-mail: dev-help@solr.apache.org <ma...@solr.apache.org>
> >>>>> >>>
> >>>>> >>
> >>>>> >>
> >>>>> >> --
> >>>>> >> -----------------------------------------------------
> >>>>> >> Noble Paul
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> > --
> >>>>> > -----------------------------------------------------
> >>>>> > Noble Paul
> >>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org <ma...@solr.apache.org>
> >>>>> For additional commands, e-mail: dev-help@solr.apache.org <ma...@solr.apache.org>
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> http://www.needhamsoftware.com <http://www.needhamsoftware.com/> (work)
> >>>> http://www.the111shift.com <http://www.the111shift.com/> (play)
> >>>
> >>>
> >>>
> >>> --
> >>> -----------------------------------------------------
> >>> Noble Paul
> >>
> >>
> >>
> >> --
> >> -----------------------------------------------------
> >> Noble Paul
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org <ma...@solr.apache.org>
> For additional commands, e-mail: dev-help@solr.apache.org <ma...@solr.apache.org>
>
Re: JAX-RS APIs in Solr
Posted by Ishan Chattopadhyaya <ic...@gmail.com>.
On Tue, Nov 30, 2021 at 8:44 PM Jason Gerlowski <ge...@gmail.com>
wrote:
> > We need to make an API work on a per core basis and cores can come up
> and go down randomly
>
> Ah, yeah. I'll admit I was overlooking some of the wrinkles around
> custom request handlers etc. That is a problem for switching to some
> off-the-shelf framework for sure. It may still be possible, but it's
> definitely a big hurdle. I wonder how commonly users still take
> advantage of the custom requestHandler stuff these days?
Here's YASA using annotations:
https://github.com/yasa-org/yasa/blob/master/yasa-solr-plugin/src/main/java/io/github/kezhenxu94/YasaHandler.java#L45
> I haven't
> run across many customers who used them extensively, but maybe that's
> self-selecting in some way?
>
> > If SolrJ has a Jackson dependency, it can have a conflict (different
> versions) with the Jackson used by the client app
>
> SolrJ has a handful of dependencies - is there a reason we take this
> approach with Jackson, but not with any of SolrJ's other deps?
> (apache-httpcomponents, apache-commons, zookeeper-client, etc.)
>
> Granted, I get that we want SolrJ to be as painless as possible for
> its users, and jar-conflicts are a part of that and really suck. But
> there are good tools out there for handling jar conflicts, and
> reimplementing chunks of library functionality just to avoid a
> gradle-dep strikes me as unsustainable from a maintenance/community
> perspective. Not to mention that forking/mimicking Jackson code
> deprives our users of any efficiency/security improvements that
> Jackson might get tomorrow or next week or next year.
>
> Unless there's some reason specific to Jackson I guess I just don't
> get it. But if I'm the minority opinion on that, fair enough I guess.
>
> On Tue, Nov 30, 2021 at 5:05 AM Noble Paul <no...@gmail.com> wrote:
> >
> > The @JsonProperty annotations are added because it's a dependency in
> SolrJ as well
> > If SolrJ has a Jackson dependency, it can have a conflict (different
> versions) with the Jackson used by the client app.
> >
> > On Tue, Nov 30, 2021, 5:21 PM Noble Paul <no...@gmail.com> wrote:
> >>
> >> Also keep in mind that the same endpoint can be accessed with a core
> name and a collection name prefixes.
> >>
> >> On Tue, Nov 30, 2021 at 3:55 PM Noble Paul <no...@gmail.com>
> wrote:
> >>>
> >>> True Gus, Almost every framework works outside of SolrDispatchFilter+
> HttpSolrCall. A lot of our initializations occur there.
> >>>
> >>> We need to make an API work on a per core basis and cores can come up
> and go down randomly. So we need to register these endpoints on a core.
> >>>
> >>> I'm not sure if any framework can achieve the same.
> >>>
> >>> On Tue, Nov 30, 2021 at 5:39 AM Gus Heck <gu...@gmail.com> wrote:
> >>>>
> >>>> IIRC last time I looked restlet had the unsavory property of existing
> outside of the SolrDispatchFilter, unlike everything else which made for
> special cases because several things that probably ought to be their own
> siervlet filters are glommed into SolrDispatchFilter, like security,
> tracing and MDC setup/teardown per request. Restlet wouldn't be so bad if
> one could just wrap such filters around it too...
> >>>>
> >>>> On Mon, Nov 29, 2021 at 9:42 AM Jason Gerlowski <
> gerlowskija@gmail.com> wrote:
> >>>>>
> >>>>> > These are minor improvements compared to a full rewrite of the
> entire framework
> >>>>>
> >>>>> If you think data type support is minor, fair enough. But to clarify
> >>>>> I'm not suggesting a rewrite - I'm suggesting using something that
> >>>>> already exists off the shelf. Jersey (e.g.) itself provides the
> >>>>> framework - there would be no "rewrite".
> >>>>>
> >>>>> re: past restlet use
> >>>>>
> >>>>> > It was not playing well with our security framework. The framework
> was not working well with Solr APIs
> >>>>>
> >>>>> Ah, very interesting! Security isn't something Eric or I tackled in
> >>>>> our little spike branch, but it's definitely a concern. Do you
> >>>>> remember the specific concerns? Or recall where any of the
> discussion
> >>>>> around this happened?
> >>>>>
> >>>>> Without the context of that past discussion, it seems like the
> >>>>> "PermissionNameProvider" interface could be implemented just as well
> >>>>> by a class with (e.g.) Jersey annotations as one with our own custom
> >>>>> annotations. Certainly there'd need to be some
> >>>>> RuleBasedAuthorizationPlugin changes or other integration code, but
> >>>>> nothing that feels insurmountable.
> >>>>>
> >>>>> Maybe I can try spiking it out soon and find the issues myself, but
> >>>>> it'd be much easier if someone happens to remember and can save me
> the
> >>>>> trouble :-p
> >>>>>
> >>>>> Best,
> >>>>>
> >>>>> Jason
> >>>>>
> >>>>> On Fri, Nov 26, 2021 at 12:05 AM Noble Paul <no...@gmail.com>
> wrote:
> >>>>> >
> >>>>> > The Annotations framework was written after playing with other
> frameworks. There were many shortcomings which were hard to overcome.
> >>>>> >
> >>>>> > The best example is a per collection API . How do you register an
> endpoint for a collection/core ?
> >>>>> >
> >>>>> > On Fri, Nov 26, 2021 at 3:42 PM Noble Paul <no...@gmail.com>
> wrote:
> >>>>> >>
> >>>>> >>
> >>>>> >>
> >>>>> >> On Fri, Nov 26, 2021 at 1:03 AM Jason Gerlowski <
> gerlowskija@gmail.com> wrote:
> >>>>> >>>
> >>>>> >>> > Is there some problem with our annotations that we hope to
> solve using third party dependencies?
> >>>>> >>>
> >>>>> >>> I guess so yeah. Third-party deps are just fuller, more robust
> >>>>> >>> solutions, whereas our annotations still need support added now
> and
> >>>>> >>> then for even primitive data types like "long" (see SOLR-15619).
> >>>>> >>
> >>>>> >> These are minor improvements compared to a full rewrite of the
> entire framework
> >>>>> >>
> >>>>> >>
> >>>>> >>>
> >>>>> >>> Every JIRA spent doing basic stuff like that is time away from
> >>>>> >>> improving Solr in some other way.
> >>>>> >>>
> >>>>> >>> So there are feature-gap/capabilities arguments for moving to a
> >>>>> >>> third-party dep, sure. But, even if our annotations did
> everything
> >>>>> >>> Jersey+Jackson do today, I think switching would still be worth
> it.
> >>>>> >>> Every LOC in our code base brings along with it some maintenance
> cost:
> >>>>> >>> it might have bugs, needs tested, takes time for new
> contributors to
> >>>>> >>> "grok", etc. Using off-the-shelf here would nuke a whole bunch
> of
> >>>>> >>> that. If off-the-shelf is available for some given
> functionality, we
> >>>>> >>> should need a compelling reason to NOT use it.
> >>>>> >>>
> >>>>> >>> Lastly, I think there's an "approachability" argument for using
> >>>>> >>> off-the-shelf. Thousands of developers out there are familiar
> with
> >>>>> >>> (e.g.) Jersey, compared to maybe 15 or 20 (in the world)
> familiar with
> >>>>> >>> Solr's custom annotations. Using a well-known technology like
> Jersey
> >>>>> >>> would make Solr all the easier to approach and contribute to for
> that
> >>>>> >>> pool of developers.
> >>>>> >>>
> >>>>> >>> > By the way, we have used Restlet in the past and that has been
> a regrettable decision.
> >>>>> >>
> >>>>> >>
> >>>>> >>>
> >>>>> >>> Ah, yeah, that's just the context I'm missing. Anyone have a
> pointer
> >>>>> >>> to related discussions, or remember what made this
> "regrettable"? All
> >>>>> >>> the theoretical benefits in the world don't matter much if we've
> >>>>> >>> already tried something like this in the past and decided
> against it.
> >>>>> >>
> >>>>> >>
> >>>>> >> It was not playing well with our security framework. The
> framework was not working well with Solr APIs
> >>>>> >>
> >>>>> >>>
> >>>>> >>> (Unrelated - Happy Thanksgiving all!)
> >>>>> >>>
> >>>>> >>> Best,
> >>>>> >>>
> >>>>> >>> Jason
> >>>>> >>>
> >>>>> >>> On Thu, Nov 25, 2021 at 7:32 AM Noble Paul <no...@gmail.com>
> wrote:
> >>>>> >>> >
> >>>>> >>> > Have you gone through an API written using the @EndPoint
> annotation?
> >>>>> >>> >
> >>>>> >>> > I strongly recommend that you do
> >>>>> >>> >
> >>>>> >>> > On Thu, Nov 25, 2021, 11:30 PM Eric Pugh <
> epugh@opensourceconnections.com> wrote:
> >>>>> >>> >>
> >>>>> >>> >> I have found our V2 API code to be very impenetrable to
> understand. Part of it is how the code is intertwined with support for
> V1, however it’s also because there aren’t really resources to go look at
> to understand how it should work! Maintaining the API should be very
> simple work, as they just exist as a translation. The home grown stuff
> may make sense if you are a super knowledgable Solr developer, but if you
> are just a new person, it’s a lot harder to contribute.
> >>>>> >>> >>
> >>>>> >>> >> I was interested in the Jersey stuff because I’ve seen lots
> of projects use it very successfully, and if I want to implement something,
> well, there are lots of blogs and resources out there!
> >>>>> >>> >>
> >>>>> >>> >> Can anyone recap briefly why we dropped RESTlet? And what
> lessons learned there might apply to adopting Jersey for API support?
> Looking at https://issues.apache.org/jira/browse/SOLR-14659, it was
> partly deprecated because we were not using it to support all the API, only
> the ManagedResource ones, and
> https://issues.apache.org/jira/browse/SOLR-14766 suggests that RESTlet
> maybe was no longer being updated? One reason why we spiked out Jersey
> was because of the broad support in the Java world! Looking at how much
> work we have to do in the V2 API world, we need a much broader pool of
> developers contributing to get there!
> >>>>> >>> >>
> >>>>> >>> >> Related, are there specific features/aspects of our
> annotations that enable things in Solr that couldn’t be done otherwise?
> >>>>> >>> >>
> >>>>> >>> >> On Nov 25, 2021, at 2:12 AM, Ishan Chattopadhyaya <
> ichattopadhyaya@gmail.com> wrote:
> >>>>> >>> >>
> >>>>> >>> >> Is there some problem with our annotations that we hope to
> solve using third party dependencies?
> >>>>> >>> >> By the way, we have used Restlet in the past and that has
> been a regrettable decision.
> >>>>> >>> >>
> >>>>> >>> >> On Thu, Nov 25, 2021 at 10:10 AM Jason Gerlowski <
> gerlowskija@gmail.com> wrote:
> >>>>> >>> >>>
> >>>>> >>> >>> Solr's custom annotation framework ('@Endpoint', '@Command',
> etc.) has
> >>>>> >>> >>> cropped up a few times over the past week or two. [1] [2].
> Having them
> >>>>> >>> >>> on top of mind, I've been wondering - is there a reason we
> use our own
> >>>>> >>> >>> annotations here instead of something off the shelf?
> >>>>> >>> >>>
> >>>>> >>> >>> What we have works well enough, but anything homegrown comes
> with more
> >>>>> >>> >>> maintenance burden than we'd have if we used something off
> the shelf.
> >>>>> >>> >>> There are plenty of well-used, active projects out there
> whose whole
> >>>>> >>> >>> purpose is facilitating the whole "annotation based API"
> thing
> >>>>> >>> >>> (Jersey, Restlet, RESTEasy, etc.) - why not use one of them?
> >>>>> >>> >>>
> >>>>> >>> >>> Does anyone know of any technical reasons why we can't go
> this route?
> >>>>> >>> >>> Or have any subjective reasons why we shouldn't? Or any
> context on
> >>>>> >>> >>> why we wrote our own Endpoint, Command, JsonProperty
> annotations
> >>>>> >>> >>> originally?
> >>>>> >>> >>>
> >>>>> >>> >>> FWIW, Eric Pugh and I spiked out a small POC recently, and
> got
> >>>>> >>> >>> Jersey+Jackson working for a few simple APIs without too
> much trouble.
> >>>>> >>> >>> [3] Obviously nothing production-ready there, and there's
> still a lot
> >>>>> >>> >>> of open questions (e.g. how would javabin be supported?),
> but we both
> >>>>> >>> >>> came away convinced that it seemed feasible, at least. Best
> of all,
> >>>>> >>> >>> APIs using our current homegrown annotation framework the
> switchover
> >>>>> >>> >>> seems blessedly straightforward, and it doesn't look like
> Jersey
> >>>>> >>> >>> (which we chose mostly arbitrarily) bloats our dist all that
> much.
> >>>>> >>> >>>
> >>>>> >>> >>> Curious if anyone has thoughts or context on how we ended up
> with the
> >>>>> >>> >>> annotation setup we use today!
> >>>>> >>> >>>
> >>>>> >>> >>> Best,
> >>>>> >>> >>>
> >>>>> >>> >>> Jason
> >>>>> >>> >>>
> >>>>> >>> >>> [1] https://issues.apache.org/jira/browse/SOLR-15182 (and
> children)
> >>>>> >>> >>> [2]
> http://mail-archives.apache.org/mod_mbox/solr-dev/202111.mbox/%3CCABEwPvENL41Pm6%2BOmjXb6Sx5N2XjUtnbWhgKOZSrnLjWBA8tcA%40mail.gmail.com%3E
> >>>>> >>> >>> [3]
> https://github.com/gerlowskija/solr/tree/jersey_jaxrs_jackson_solr_apis.
> >>>>> >>> >>>
> >>>>> >>> >>>
> ---------------------------------------------------------------------
> >>>>> >>> >>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> >>>>> >>> >>> For additional commands, e-mail: dev-help@solr.apache.org
> >>>>> >>> >>>
> >>>>> >>> >>
> >>>>> >>> >> _______________________
> >>>>> >>> >> Eric Pugh | Founder & CEO | OpenSource Connections, LLC |
> 434.466.1467 | http://www.opensourceconnections.com | My Free/Busy
> >>>>> >>> >> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
> >>>>> >>> >> This e-mail and all contents, including attachments, is
> considered to be Company Confidential unless explicitly stated otherwise,
> regardless of whether attachments are marked as such.
> >>>>> >>> >>
> >>>>> >>>
> >>>>> >>>
> ---------------------------------------------------------------------
> >>>>> >>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> >>>>> >>> For additional commands, e-mail: dev-help@solr.apache.org
> >>>>> >>>
> >>>>> >>
> >>>>> >>
> >>>>> >> --
> >>>>> >> -----------------------------------------------------
> >>>>> >> Noble Paul
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> > --
> >>>>> > -----------------------------------------------------
> >>>>> > Noble Paul
> >>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> >>>>> For additional commands, e-mail: dev-help@solr.apache.org
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> http://www.needhamsoftware.com (work)
> >>>> http://www.the111shift.com (play)
> >>>
> >>>
> >>>
> >>> --
> >>> -----------------------------------------------------
> >>> Noble Paul
> >>
> >>
> >>
> >> --
> >> -----------------------------------------------------
> >> Noble Paul
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org
>
>
Re: JAX-RS APIs in Solr
Posted by Ishan Chattopadhyaya <ic...@gmail.com>.
Hi Tim,
You bring up great suggestions for improvement to Solr, I really think we
should do something to address all of those as soon as possible. However,
you misunderstand the comments here and mischaracterize Noble's intentions
here.
This discussion was about Jason discussing "Solr's custom annotation
framework ('@Endpoint', '@Command', etc.)" and using JAX-RS in that context
and Jason is actively seeking others' opinion and historical context. At a
broader level, it seems to me that Jason is discussing about
rewriting/reimplementing existing V2 APIs with JAX-RS. Noble answered all
the problems he experienced with Restlet specifically because Jason
specifically asked for his opinion on problems with Restlet. Noble is NOT
advocating against "modernization". He was providing historic context.
If someone is interested in my candid opinion for historical context, let
me dive a little bit on my perspective of history: [*footnote1*; optional
read]
You're suggesting, however, that we explore improvements to our APIs to
support OpenAPI, OIDC, Shiro etc. Those are great suggestions, and MUST
HAPPEN. But, they are orthogonal to this current thread. When you talk
about “an excuse to block adoption of other API frameworks.”, you miss the
point. The discussion shouldn’t be about which frameworks to use, but
instead a separate and broader discussion (SIP) about what all we want to
achieve and how, and then selecting the right framework for it.
While it seems that your focus is only on improving how our Admin APIs
look, I’d like to suggest that the broader discussion should address
regular query/indexing APIs as well, since that’s what majority of users
use Solr for. [Anecdote: *footnote2*]. To add to that, we should also focus
on supporting grpc/protobuf, Amazon Eon and many other modern API things.
No one in their right mind would want to “block adoption” for any of this.
I think we should immediately start a SIP for the next generation of modern
APIs (V3) and discuss modernization from a wholistic point of view, instead
of just this limited-scope PoC discussed here.
Regards,
Ishan
*[footnote1]* (1) when the smartest of the committers were getting lost to
the causes of Hadoop, Spark etc, companies were paying my employers to
shove in features as per their timeline. Authentication & Authorization
framework is an example, where there were requirements to build a
authc/authz framework for a Hadoop focused company so they can integrate
Apache Ranger into Solr. We did so, and I doubt that ranger integration
ever proved useful to Solr users, and we needed at least a default
implementation of the security framework that we built for them to address
most users’ needs. That is what we have today. And I agree that allowing
everything by default is a poor choice. (2) when V2 APIs were conceived and
implemented, there was no OpenAPI specs people were using, and there wasn’t
as much collaboration as we have today. I remember only Noble focused on
this, and hence we are limited by or benefited by whatever he proposed back
in the day. Was it perfect? No. Was it better than V1? Hell yes! Could we
(community) have benefited from greater collaboration? I certainly feel so.
Same when Steve introduced Restlet for Schema APIs.
*[footnote2]* It is so confusing currently for a user to use faceting using
SolrJ; many users probably don’t realize they are using legacy impl and not
JSON impl. JSON Faceting request and response should be strongly typed for
a good Solr client to function well. We can look at Elasticsearch for
inspiration, their client is very good IMHO.
On Fri, Dec 3, 2021 at 10:02 PM Timothy Potter <th...@gmail.com> wrote:
> Why do we keep talking about Restlet here? Restlet is a stale (at
> best) project, so not sure why it keeps coming up in a discussion
> about modernizing our API? Integration with it in Solr was introduced
> nearly 10 years ago, time to move on and stop using that as an excuse
> to block adoption of other API frameworks.
>
> Surely anyone committed to this effort, after doing even the most
> cursory due diligence, will not land on Restlet as the way forward.
> Jersey is a likely candidate, I'm sure there are others?
>
> I actually don't care all that much about which annotations we use to
> describe our APIs (of course standards are better). What's more
> important for me, is that we can auto-generate an OpenAPI spec for our
> APIs. That opens up a world of automated tooling for our users to
> interact with our APIs. Using JAX-RS / Jersey probably helps get us
> closer to an automated way to generate an OpenAPI spec but there are
> other ways to get there as well.
>
> In general, it sounds like the objections to modernizing our API fall
> into three buckets: 1) our bad experience with Restlet, 2) performance
> is better with our homegrown code, 3) security is too complicated with
> another framework.
>
> Performance overhead added by a framework like Jersey (if there is
> any) for Admin APIs is mostly irrelevant. You think Jersey is going to
> be slower at mapping requests to Java method calls? Pretty sure they
> have that part figured out at this point.
>
> As for Solr's security, it also needs to be replaced with a modern
> framework like Shiro or pac4j or similar. The permissions are
> basically impossible to understand, even for most committers. This
> whole idea of allow by default instead of deny by default goes against
> every security best practice in the industry. So arguing that we need
> some homegrown API framework to support Solr's obscure authz scheme
> seems a bit backwards to me. Once you start using standard API
> frameworks like JAX-RS, you start getting security integration too,
> e.g. https://github.com/pac4j/jax-rs-pac4j. Moreover, we can't keep up
> with all the innovation around security; heck, SolrJ doesn't even
> support OIDC yet. I'm all for scraping both homegrown frameworks and
> starting fresh with mature open source frameworks that have all the
> kinks worked out over many years with many projects using them.
>
> If you want to talk about stress, try building a non-trivial app with
> Solr's current API's, that'll induce some stress for sure!
>
> Tim
>
>
>
> On Thu, Dec 2, 2021 at 10:35 PM Noble Paul <no...@gmail.com> wrote:
> >
> > I do not deny that a standardized framework may be helpful. I'm already
> suffering from the PTSD of restlet integration. The integration code was
> extremely complex and a security nightmare.It was added without any perf
> tests too. It took many years to get rid of that
> >
> >
> >
> > On Fri, Dec 3, 2021 at 1:37 AM Jason Gerlowski <ge...@gmail.com>
> wrote:
> >>
> >> > 99% of Solr users never see the java code that implements an API let
> alone writing a new API.
> >>
> >> We may be talking past each other here a bit Noble. The JAX-RS
> >> benefits I've been trying to describe are to benefit us developers,
> >> not Solr users. I agree that 99% of our users won't know or care
> >> about it, but who cares. The goal is to make _our_ lives as
> >> maintainers easier going into the future. Jan described it in a word
> >> better than I've been struggling to: it's about "modernization".
> >>
> >> Best,
> >>
> >> Jason
> >>
> >> On Thu, Dec 2, 2021 at 3:34 AM Jan Høydahl <ja...@cominvent.com>
> wrote:
> >> >
> >> > Agree, Ishan.
> >> >
> >> > Let's welcome and cheer initiatives to modernize this old codebase.
> >> > Diving in deep and un-tangling DispatchFilter like Gus is attempting
> is brave.
> >> > And testing JAX-RS to modernize APIs and make the codebas easier to
> grok is also not for the faint hearted.
> >> >
> >> > I hope both initiatives end up as a breath of fresh air for Solr
> developers. And perhaps users may get their hands on auto-generated
> OpenAPI docs for Solr as well!
> >> > Theoretically foundations like this may make it feasible in the
> future to auto-generate client code from the spec,so that full Admin API
> client support in e.g. Go or Rust may be achievable without hand-coding.
> But that is still far away :)
> >> >
> >> > Jan
> >> >
> >> > 2. des. 2021 kl. 05:52 skrev Ishan Chattopadhyaya <
> ichattopadhyaya@gmail.com>:
> >> >
> >> > > We are discussing this as if moving to an external framework is
> going to be smooth. It's not.
> >> >
> >> > Noble, we won't know before someone has given it a try. My hunch
> suggests it won't be smooth, but we can decide on that once we see a
> concrete solution that 1) works well for all existing APIs (per core or
> global) without hacks/workarounds, and 2) has no security implications, and
> 3) doesn't clutter our codebase, and 4) doesn't affect our runtime
> performance.
> >> >
> >> > On Thu, Dec 2, 2021 at 4:22 AM Noble Paul <no...@gmail.com>
> wrote:
> >> >>
> >> >>
> >> >>
> >> >> On Thu, Dec 2, 2021 at 2:46 AM Jason Gerlowski <
> gerlowskija@gmail.com> wrote:
> >> >>>
> >> >>> > There are no known issues with the current system
> >> >>>
> >> >>> Maybe you're using hyperbole to emphasise a point, but let's steer
> >> >>> this discussion away from straw-men and caricatures. I mentioned
> >> >>> specific known issues in a previous reply as a direct response to
> your
> >> >>> question about them. The incomplete support for data types
> >> >>> (SOLR-15619 and SOLR-15796). The inflexible PermissionNameProvider
> >> >>> integration (SOLR-15823). If you consider those "minor", fair
> enough,
> >> >>> and you're welcome to that opinion. But saying that they don't
> exist
> >> >>> is a mischaracterization that wastes time and muddies water.
> >> >>
> >> >>
> >> >> We are discussing this as if moving to an external framework is
> going to be smooth. It's not. I have seen how hard the Restlet integration
> was and how it stood out like a sore thumb. The amount of code required to
> integrate Solr APIs into restlet components was 10x more than the amount of
> code written for the new framework and it still was incomplete .
> Integrating anything to Solr is not simple and it's going to be an ongoing
> effort to ensure that all of the functionalities of a SolrRequestHandler
> are exposed to the new framework. Every single component that we have today
> has ongoing improvements and bug fixes. Using that as an excuse to do a
> complete rewrite is probably unwise.
> >> >>
> >> >>
> >> >>>
> >> >>> > Solr is not a general purpose web server where people write APIs
> every day
> >> >>>
> >> >>> Totally agree - 99% of the time someone opens an API file in Solr,
> >> >>> they're reading, not writing. So if JAX-RS didn't help that case,
> it
> >> >>> wouldn't be worth the effort of switching. But IMO it helps that
> >> >>> "read" case a lot.
> >> >>
> >> >>
> >> >> 99% of Solr users never see the java code that implements an API let
> alone writing a new API. They wouldn't care if an API is implemented using
> a Servlet API, a custom framework or something else. They only care about
> the input and output. New end points are rare.
> >> >>>
> >> >>>
> >> >>> JAX-RS defined APIs expose inputs as strongly-typed parameters
> >> >>> (compare to the weakly-typed SolrQueryRequest/SolrParams used
> today).
> >> >>> JAX-RS enumerates every input to the API in one place. JAX-RS APIs
> >> >>> can return any serializable type as a response (compare to the
> >> >>> near-omnipresent NamedList today).
> >> >>>
> >> >>> (The annotation framework's PayloadObj class gets us a bit of this
> >> >>> benefit, but not all of it and only on POST requests afaik.)
> >> >>>
> >> >>> This is all nice stuff at API-write time, but where it really helps
> is
> >> >>> in making the code clearer and simpler whenever someone goes to read
> >> >>> it later. No wondering what parameters an API actually takes. No
> >> >>> puzzling out what a bunch of NamedList operations spread throughout
> >> >>> API execution actually produce in the end. etc.
> >> >>>
> >> >>> > The only reason why anyone would need to use this framework is to
> write a custom request handler.
> >> >>>
> >> >>> ...except for the majority of people who come in contact with the
> >> >>> framework by reading an existing Solr API? This ties into your
> point
> >> >>> above about APIs being written infrequently but read all the time.
> >> >>> Improving readability is the big gain here, and I think JAX-RS
> offers
> >> >>> real improvements in that regard.
> >> >>
> >> >>
> >> >> We had no framework for the last 15+ years and V1 API (the only
> actually used API) still uses no frameworks. Even the annotation framework
> is not used much even though readability is a lot better in the annotation
> based system. Readability has not been an overriding concern for V1 APIs
> for sure
> >> >>
> >> >>> Also, to clarify, I'm not asserting that no one configures custom
> >> >>> requestHandlers. I was wondering aloud how common it was, as an
> open
> >> >>> question.
> >> >>>
> >> >> Your point IS right. IRL, most of our users do not write request
> handlers . They write search components , query parsers and other plugins.
> Because we are a search engine and not a web server. I have never seen a
> user complaining about the V1 API of writing a request handler. They may
> actually complain about how hard it is to deploy one. Our users are much
> smarter than we give them credit for.
> >> >>
> >> >>
> >> >>>
> >> >>> > We did not fork Jackson. We are using Jackson itself.
> >> >>>
> >> >>> Ah, maybe you're right, "fork" is probably an unfair word. I looked
> >> >>> for the serialization code, and I'll admit there is less than I
> >> >>> thought involved here and I misunderstood how it works. So that's
> my
> >> >>> bad. The JSON serialization/deserialization discussion is prob a
> >> >>> digression from the core contention about whether JAX-RS would be
> >> >>> helpful. I was curious about your earlier response and couldn't
> >> >>> resist following up here but I should've done so elsewhere to avoid
> >> >>> the distraction.
> >> >>>
> >> >>> Best,
> >> >>>
> >> >>>
> >> >>> Jason
> >> >>>
> >> >>> On Tue, Nov 30, 2021 at 4:12 PM Noble Paul <no...@gmail.com>
> wrote:
> >> >>> >
> >> >>> > The annotation framework is just a single class and it is made to
> work well with Solr instead of changing Solr to suit the needs of some
> external framework. There are no known issues with the current system and
> there are unknown issues that you'll face introducing a new framework.
> >> >>> >
> >> >>> > Solr is not a general purpose web server where people write APIs
> every day. It's a finished product where people use the publicly available
> APIs . We should focus our efforts on making our APIs work well .
> >> >>> >
> >> >>> > On Wed, Dec 1, 2021, 2:14 AM Jason Gerlowski <
> gerlowskija@gmail.com> wrote:
> >> >>> >>
> >> >>> >> > We need to make an API work on a per core basis and cores can
> come up and go down randomly
> >> >>> >>
> >> >>> >> Ah, yeah. I'll admit I was overlooking some of the wrinkles
> around
> >> >>> >> custom request handlers etc. That is a problem for switching to
> some
> >> >>> >> off-the-shelf framework for sure. It may still be possible, but
> it's
> >> >>> >> definitely a big hurdle. I wonder how commonly users still take
> >> >>> >> advantage of the custom requestHandler stuff these days? I
> haven't
> >> >>> >> run across many customers who used them extensively, but maybe
> that's
> >> >>> >> self-selecting in some way?
> >> >>> >
> >> >>> >
> >> >>> > The only reason why anyone would need to use this framework is to
> write a custom request handler. You're also saying nobody writes custom
> request handlers. So, what problem are you trying to solve if you think
> nobody uses them?
> >> >>> >
> >> >>> >
> >> >>> >>
> >> >>> >> > If SolrJ has a Jackson dependency, it can have a conflict
> (different versions) with the Jackson used by the client app
> >> >>> >>
> >> >>> >> SolrJ has a handful of dependencies - is there a reason we take
> this
> >> >>> >> approach with Jackson, but not with any of SolrJ's other deps?
> >> >>> >> (apache-httpcomponents, apache-commons, zookeeper-client, etc.)
> >> >>> >>
> >> >>> >> Granted, I get that we want SolrJ to be as painless as possible
> for
> >> >>> >> its users, and jar-conflicts are a part of that and really
> suck. But
> >> >>> >> there are good tools out there for handling jar conflicts, and
> >> >>> >> reimplementing chunks of library functionality just to avoid a
> >> >>> >> gradle-dep strikes me as unsustainable from a
> maintenance/community
> >> >>> >> perspective. Not to mention that forking/mimicking Jackson code
> >> >>> >> deprives our users of any efficiency/security improvements that
> >> >>> >> Jackson might get tomorrow or next week or next year.
> >> >>> >
> >> >>> >
> >> >>> > We did not fork Jackson. We are using Jackson itself. Jackson
> supports custom annotations. There was a huge discussion why we should do
> it this way.
> >> >>> >>
> >> >>> >>
> >> >>> >> Unless there's some reason specific to Jackson I guess I just
> don't
> >> >>> >> get it. But if I'm the minority opinion on that, fair enough I
> guess.
> >> >>> >>
> >> >>> >> On Tue, Nov 30, 2021 at 5:05 AM Noble Paul <no...@gmail.com>
> wrote:
> >> >>> >> >
> >> >>> >> > The @JsonProperty annotations are added because it's a
> dependency in SolrJ as well
> >> >>> >> > If SolrJ has a Jackson dependency, it can have a conflict
> (different versions) with the Jackson used by the client app.
> >> >>> >> >
> >> >>> >> > On Tue, Nov 30, 2021, 5:21 PM Noble Paul <no...@gmail.com>
> wrote:
> >> >>> >> >>
> >> >>> >> >> Also keep in mind that the same endpoint can be accessed with
> a core name and a collection name prefixes.
> >> >>> >> >>
> >> >>> >> >> On Tue, Nov 30, 2021 at 3:55 PM Noble Paul <
> noble.paul@gmail.com> wrote:
> >> >>> >> >>>
> >> >>> >> >>> True Gus, Almost every framework works outside of
> SolrDispatchFilter+ HttpSolrCall. A lot of our initializations occur there.
> >> >>> >> >>>
> >> >>> >> >>> We need to make an API work on a per core basis and cores
> can come up and go down randomly. So we need to register these endpoints on
> a core.
> >> >>> >> >>>
> >> >>> >> >>> I'm not sure if any framework can achieve the same.
> >> >>> >> >>>
> >> >>> >> >>> On Tue, Nov 30, 2021 at 5:39 AM Gus Heck <gu...@gmail.com>
> wrote:
> >> >>> >> >>>>
> >> >>> >> >>>> IIRC last time I looked restlet had the unsavory property
> of existing outside of the SolrDispatchFilter, unlike everything else which
> made for special cases because several things that probably ought to be
> their own siervlet filters are glommed into SolrDispatchFilter, like
> security, tracing and MDC setup/teardown per request. Restlet wouldn't be
> so bad if one could just wrap such filters around it too...
> >> >>> >> >>>>
> >> >>> >> >>>> On Mon, Nov 29, 2021 at 9:42 AM Jason Gerlowski <
> gerlowskija@gmail.com> wrote:
> >> >>> >> >>>>>
> >> >>> >> >>>>> > These are minor improvements compared to a full rewrite
> of the entire framework
> >> >>> >> >>>>>
> >> >>> >> >>>>> If you think data type support is minor, fair enough. But
> to clarify
> >> >>> >> >>>>> I'm not suggesting a rewrite - I'm suggesting using
> something that
> >> >>> >> >>>>> already exists off the shelf. Jersey (e.g.) itself
> provides the
> >> >>> >> >>>>> framework - there would be no "rewrite".
> >> >>> >> >>>>>
> >> >>> >> >>>>> re: past restlet use
> >> >>> >> >>>>>
> >> >>> >> >>>>> > It was not playing well with our security framework. The
> framework was not working well with Solr APIs
> >> >>> >> >>>>>
> >> >>> >> >>>>> Ah, very interesting! Security isn't something Eric or I
> tackled in
> >> >>> >> >>>>> our little spike branch, but it's definitely a concern.
> Do you
> >> >>> >> >>>>> remember the specific concerns? Or recall where any of
> the discussion
> >> >>> >> >>>>> around this happened?
> >> >>> >> >>>>>
> >> >>> >> >>>>> Without the context of that past discussion, it seems like
> the
> >> >>> >> >>>>> "PermissionNameProvider" interface could be implemented
> just as well
> >> >>> >> >>>>> by a class with (e.g.) Jersey annotations as one with our
> own custom
> >> >>> >> >>>>> annotations. Certainly there'd need to be some
> >> >>> >> >>>>> RuleBasedAuthorizationPlugin changes or other integration
> code, but
> >> >>> >> >>>>> nothing that feels insurmountable.
> >> >>> >> >>>>>
> >> >>> >> >>>>> Maybe I can try spiking it out soon and find the issues
> myself, but
> >> >>> >> >>>>> it'd be much easier if someone happens to remember and can
> save me the
> >> >>> >> >>>>> trouble :-p
> >> >>> >> >>>>>
> >> >>> >> >>>>> Best,
> >> >>> >> >>>>>
> >> >>> >> >>>>> Jason
> >> >>> >> >>>>>
> >> >>> >> >>>>> On Fri, Nov 26, 2021 at 12:05 AM Noble Paul <
> noble.paul@gmail.com> wrote:
> >> >>> >> >>>>> >
> >> >>> >> >>>>> > The Annotations framework was written after playing with
> other frameworks. There were many shortcomings which were hard to overcome.
> >> >>> >> >>>>> >
> >> >>> >> >>>>> > The best example is a per collection API . How do you
> register an endpoint for a collection/core ?
> >> >>> >> >>>>> >
> >> >>> >> >>>>> > On Fri, Nov 26, 2021 at 3:42 PM Noble Paul <
> noble.paul@gmail.com> wrote:
> >> >>> >> >>>>> >>
> >> >>> >> >>>>> >>
> >> >>> >> >>>>> >>
> >> >>> >> >>>>> >> On Fri, Nov 26, 2021 at 1:03 AM Jason Gerlowski <
> gerlowskija@gmail.com> wrote:
> >> >>> >> >>>>> >>>
> >> >>> >> >>>>> >>> > Is there some problem with our annotations that we
> hope to solve using third party dependencies?
> >> >>> >> >>>>> >>>
> >> >>> >> >>>>> >>> I guess so yeah. Third-party deps are just fuller,
> more robust
> >> >>> >> >>>>> >>> solutions, whereas our annotations still need support
> added now and
> >> >>> >> >>>>> >>> then for even primitive data types like "long" (see
> SOLR-15619).
> >> >>> >> >>>>> >>
> >> >>> >> >>>>> >> These are minor improvements compared to a full rewrite
> of the entire framework
> >> >>> >> >>>>> >>
> >> >>> >> >>>>> >>
> >> >>> >> >>>>> >>>
> >> >>> >> >>>>> >>> Every JIRA spent doing basic stuff like that is time
> away from
> >> >>> >> >>>>> >>> improving Solr in some other way.
> >> >>> >> >>>>> >>>
> >> >>> >> >>>>> >>> So there are feature-gap/capabilities arguments for
> moving to a
> >> >>> >> >>>>> >>> third-party dep, sure. But, even if our annotations
> did everything
> >> >>> >> >>>>> >>> Jersey+Jackson do today, I think switching would still
> be worth it.
> >> >>> >> >>>>> >>> Every LOC in our code base brings along with it some
> maintenance cost:
> >> >>> >> >>>>> >>> it might have bugs, needs tested, takes time for new
> contributors to
> >> >>> >> >>>>> >>> "grok", etc. Using off-the-shelf here would nuke a
> whole bunch of
> >> >>> >> >>>>> >>> that. If off-the-shelf is available for some given
> functionality, we
> >> >>> >> >>>>> >>> should need a compelling reason to NOT use it.
> >> >>> >> >>>>> >>>
> >> >>> >> >>>>> >>> Lastly, I think there's an "approachability" argument
> for using
> >> >>> >> >>>>> >>> off-the-shelf. Thousands of developers out there are
> familiar with
> >> >>> >> >>>>> >>> (e.g.) Jersey, compared to maybe 15 or 20 (in the
> world) familiar with
> >> >>> >> >>>>> >>> Solr's custom annotations. Using a well-known
> technology like Jersey
> >> >>> >> >>>>> >>> would make Solr all the easier to approach and
> contribute to for that
> >> >>> >> >>>>> >>> pool of developers.
> >> >>> >> >>>>> >>>
> >> >>> >> >>>>> >>> > By the way, we have used Restlet in the past and
> that has been a regrettable decision.
> >> >>> >> >>>>> >>
> >> >>> >> >>>>> >>
> >> >>> >> >>>>> >>>
> >> >>> >> >>>>> >>> Ah, yeah, that's just the context I'm missing. Anyone
> have a pointer
> >> >>> >> >>>>> >>> to related discussions, or remember what made this
> "regrettable"? All
> >> >>> >> >>>>> >>> the theoretical benefits in the world don't matter
> much if we've
> >> >>> >> >>>>> >>> already tried something like this in the past and
> decided against it.
> >> >>> >> >>>>> >>
> >> >>> >> >>>>> >>
> >> >>> >> >>>>> >> It was not playing well with our security framework.
> The framework was not working well with Solr APIs
> >> >>> >> >>>>> >>
> >> >>> >> >>>>> >>>
> >> >>> >> >>>>> >>> (Unrelated - Happy Thanksgiving all!)
> >> >>> >> >>>>> >>>
> >> >>> >> >>>>> >>> Best,
> >> >>> >> >>>>> >>>
> >> >>> >> >>>>> >>> Jason
> >> >>> >> >>>>> >>>
> >> >>> >> >>>>> >>> On Thu, Nov 25, 2021 at 7:32 AM Noble Paul <
> noble.paul@gmail.com> wrote:
> >> >>> >> >>>>> >>> >
> >> >>> >> >>>>> >>> > Have you gone through an API written using the
> @EndPoint annotation?
> >> >>> >> >>>>> >>> >
> >> >>> >> >>>>> >>> > I strongly recommend that you do
> >> >>> >> >>>>> >>> >
> >> >>> >> >>>>> >>> > On Thu, Nov 25, 2021, 11:30 PM Eric Pugh <
> epugh@opensourceconnections.com> wrote:
> >> >>> >> >>>>> >>> >>
> >> >>> >> >>>>> >>> >> I have found our V2 API code to be very
> impenetrable to understand. Part of it is how the code is intertwined
> with support for V1, however it’s also because there aren’t really
> resources to go look at to understand how it should work! Maintaining the
> API should be very simple work, as they just exist as a translation. The
> home grown stuff may make sense if you are a super knowledgable Solr
> developer, but if you are just a new person, it’s a lot harder to
> contribute.
> >> >>> >> >>>>> >>> >>
> >> >>> >> >>>>> >>> >> I was interested in the Jersey stuff because I’ve
> seen lots of projects use it very successfully, and if I want to implement
> something, well, there are lots of blogs and resources out there!
> >> >>> >> >>>>> >>> >>
> >> >>> >> >>>>> >>> >> Can anyone recap briefly why we dropped RESTlet?
> And what lessons learned there might apply to adopting Jersey for API
> support? Looking at https://issues.apache.org/jira/browse/SOLR-14659,
> it was partly deprecated because we were not using it to support all the
> API, only the ManagedResource ones, and
> https://issues.apache.org/jira/browse/SOLR-14766 suggests that RESTlet
> maybe was no longer being updated? One reason why we spiked out Jersey
> was because of the broad support in the Java world! Looking at how much
> work we have to do in the V2 API world, we need a much broader pool of
> developers contributing to get there!
> >> >>> >> >>>>> >>> >>
> >> >>> >> >>>>> >>> >> Related, are there specific features/aspects of our
> annotations that enable things in Solr that couldn’t be done otherwise?
> >> >>> >> >>>>> >>> >>
> >> >>> >> >>>>> >>> >> On Nov 25, 2021, at 2:12 AM, Ishan Chattopadhyaya <
> ichattopadhyaya@gmail.com> wrote:
> >> >>> >> >>>>> >>> >>
> >> >>> >> >>>>> >>> >> Is there some problem with our annotations that we
> hope to solve using third party dependencies?
> >> >>> >> >>>>> >>> >> By the way, we have used Restlet in the past and
> that has been a regrettable decision.
> >> >>> >> >>>>> >>> >>
> >> >>> >> >>>>> >>> >> On Thu, Nov 25, 2021 at 10:10 AM Jason Gerlowski <
> gerlowskija@gmail.com> wrote:
> >> >>> >> >>>>> >>> >>>
> >> >>> >> >>>>> >>> >>> Solr's custom annotation framework ('@Endpoint',
> '@Command', etc.) has
> >> >>> >> >>>>> >>> >>> cropped up a few times over the past week or two.
> [1] [2]. Having them
> >> >>> >> >>>>> >>> >>> on top of mind, I've been wondering - is there a
> reason we use our own
> >> >>> >> >>>>> >>> >>> annotations here instead of something off the
> shelf?
> >> >>> >> >>>>> >>> >>>
> >> >>> >> >>>>> >>> >>> What we have works well enough, but anything
> homegrown comes with more
> >> >>> >> >>>>> >>> >>> maintenance burden than we'd have if we used
> something off the shelf.
> >> >>> >> >>>>> >>> >>> There are plenty of well-used, active projects out
> there whose whole
> >> >>> >> >>>>> >>> >>> purpose is facilitating the whole "annotation
> based API" thing
> >> >>> >> >>>>> >>> >>> (Jersey, Restlet, RESTEasy, etc.) - why not use
> one of them?
> >> >>> >> >>>>> >>> >>>
> >> >>> >> >>>>> >>> >>> Does anyone know of any technical reasons why we
> can't go this route?
> >> >>> >> >>>>> >>> >>> Or have any subjective reasons why we shouldn't?
> Or any context on
> >> >>> >> >>>>> >>> >>> why we wrote our own Endpoint, Command,
> JsonProperty annotations
> >> >>> >> >>>>> >>> >>> originally?
> >> >>> >> >>>>> >>> >>>
> >> >>> >> >>>>> >>> >>> FWIW, Eric Pugh and I spiked out a small POC
> recently, and got
> >> >>> >> >>>>> >>> >>> Jersey+Jackson working for a few simple APIs
> without too much trouble.
> >> >>> >> >>>>> >>> >>> [3] Obviously nothing production-ready there, and
> there's still a lot
> >> >>> >> >>>>> >>> >>> of open questions (e.g. how would javabin be
> supported?), but we both
> >> >>> >> >>>>> >>> >>> came away convinced that it seemed feasible, at
> least. Best of all,
> >> >>> >> >>>>> >>> >>> APIs using our current homegrown annotation
> framework the switchover
> >> >>> >> >>>>> >>> >>> seems blessedly straightforward, and it doesn't
> look like Jersey
> >> >>> >> >>>>> >>> >>> (which we chose mostly arbitrarily) bloats our
> dist all that much.
> >> >>> >> >>>>> >>> >>>
> >> >>> >> >>>>> >>> >>> Curious if anyone has thoughts or context on how
> we ended up with the
> >> >>> >> >>>>> >>> >>> annotation setup we use today!
> >> >>> >> >>>>> >>> >>>
> >> >>> >> >>>>> >>> >>> Best,
> >> >>> >> >>>>> >>> >>>
> >> >>> >> >>>>> >>> >>> Jason
> >> >>> >> >>>>> >>> >>>
> >> >>> >> >>>>> >>> >>> [1]
> https://issues.apache.org/jira/browse/SOLR-15182 (and children)
> >> >>> >> >>>>> >>> >>> [2]
> http://mail-archives.apache.org/mod_mbox/solr-dev/202111.mbox/%3CCABEwPvENL41Pm6%2BOmjXb6Sx5N2XjUtnbWhgKOZSrnLjWBA8tcA%40mail.gmail.com%3E
> >> >>> >> >>>>> >>> >>> [3]
> https://github.com/gerlowskija/solr/tree/jersey_jaxrs_jackson_solr_apis.
> >> >>> >> >>>>> >>> >>>
> >> >>> >> >>>>> >>> >>>
> ---------------------------------------------------------------------
> >> >>> >> >>>>> >>> >>> To unsubscribe, e-mail:
> dev-unsubscribe@solr.apache.org
> >> >>> >> >>>>> >>> >>> For additional commands, e-mail:
> dev-help@solr.apache.org
> >> >>> >> >>>>> >>> >>>
> >> >>> >> >>>>> >>> >>
> >> >>> >> >>>>> >>> >> _______________________
> >> >>> >> >>>>> >>> >> Eric Pugh | Founder & CEO | OpenSource Connections,
> LLC | 434.466.1467 | http://www.opensourceconnections.com | My Free/Busy
> >> >>> >> >>>>> >>> >> Co-Author: Apache Solr Enterprise Search Server,
> 3rd Ed
> >> >>> >> >>>>> >>> >> This e-mail and all contents, including
> attachments, is considered to be Company Confidential unless explicitly
> stated otherwise, regardless of whether attachments are marked as such.
> >> >>> >> >>>>> >>> >>
> >> >>> >> >>>>> >>>
> >> >>> >> >>>>> >>>
> ---------------------------------------------------------------------
> >> >>> >> >>>>> >>> To unsubscribe, e-mail:
> dev-unsubscribe@solr.apache.org
> >> >>> >> >>>>> >>> For additional commands, e-mail:
> dev-help@solr.apache.org
> >> >>> >> >>>>> >>>
> >> >>> >> >>>>> >>
> >> >>> >> >>>>> >>
> >> >>> >> >>>>> >> --
> >> >>> >> >>>>> >> -----------------------------------------------------
> >> >>> >> >>>>> >> Noble Paul
> >> >>> >> >>>>> >
> >> >>> >> >>>>> >
> >> >>> >> >>>>> >
> >> >>> >> >>>>> > --
> >> >>> >> >>>>> > -----------------------------------------------------
> >> >>> >> >>>>> > Noble Paul
> >> >>> >> >>>>>
> >> >>> >> >>>>>
> ---------------------------------------------------------------------
> >> >>> >> >>>>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> >> >>> >> >>>>> For additional commands, e-mail: dev-help@solr.apache.org
> >> >>> >> >>>>>
> >> >>> >> >>>>
> >> >>> >> >>>>
> >> >>> >> >>>> --
> >> >>> >> >>>> http://www.needhamsoftware.com (work)
> >> >>> >> >>>> http://www.the111shift.com (play)
> >> >>> >> >>>
> >> >>> >> >>>
> >> >>> >> >>>
> >> >>> >> >>> --
> >> >>> >> >>> -----------------------------------------------------
> >> >>> >> >>> Noble Paul
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >> --
> >> >>> >> >> -----------------------------------------------------
> >> >>> >> >> Noble Paul
> >> >>> >>
> >> >>> >>
> ---------------------------------------------------------------------
> >> >>> >> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> >> >>> >> For additional commands, e-mail: dev-help@solr.apache.org
> >> >>> >>
> >> >>>
> >> >>>
> ---------------------------------------------------------------------
> >> >>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> >> >>> For additional commands, e-mail: dev-help@solr.apache.org
> >> >>>
> >> >>
> >> >>
> >> >> --
> >> >> -----------------------------------------------------
> >> >> Noble Paul
> >> >
> >> >
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> >> For additional commands, e-mail: dev-help@solr.apache.org
> >>
> >
> >
> > --
> > -----------------------------------------------------
> > Noble Paul
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org
>
>
Re: JAX-RS APIs in Solr
Posted by Noble Paul <no...@gmail.com>.
On Sat, Dec 4, 2021 at 3:32 AM Timothy Potter <th...@gmail.com> wrote:
> Why do we keep talking about Restlet here? Restlet is a stale (at
> best) project, so not sure why it keeps coming up in a discussion
> about modernizing our API? Integration with it in Solr was introduced
> nearly 10 years ago, time to move on and stop using that as an excuse
> to block adoption of other API frameworks.
>
> Surely anyone committed to this effort, after doing even the most
> cursory due diligence, will not land on Restlet as the way forward.
> Jersey is a likely candidate, I'm sure there are others?
>
Jason was asking about adopting an external framework and he suggested
many. I didn’t raise a concern with restlet or any other framework . My
concern was about the way the Restlet framework was integrated into Solr.
Any new framework will have similar challenges. 99.9% of Solr calls are
made to /select and /update paths and they are the most important pieces of
the Solr UX. The collection admin commands are usually used by scripts and
people are even fine with V1 APIs. So. Any framework that we are going to
use will have to co-exist with the existing SolrRequestHandler framework.
Restlet integration was difficult because it had to work with the existing
APIs.
The challenges with using any publicly available frameworks is that the
input/output of Solr is not strongly typed. Most frameworks assume that the
request/response is strongly typed
>
> I actually don't care all that much about which annotations we use to
> describe our APIs (of course standards are better). What's more
> important for me, is that we can auto-generate an OpenAPI spec for our
> APIs. That opens up a world of automated tooling for our users to
> interact with our APIs. Using JAX-RS / Jersey probably helps get us
> closer to an automated way to generate an OpenAPI spec but there are
> other ways to get there as well.
>
> In general, it sounds like the objections to modernizing our API fall
> into three buckets: 1) our bad experience with Restlet, 2) performance
> is better with our homegrown code, 3) security is too complicated with
> another framework.
>
> Performance overhead added by a framework like Jersey (if there is
> any) for Admin APIs is mostly irrelevant. You think Jersey is going to
> be slower at mapping requests to Java method calls? Pretty sure they
> have that part figured out at this point.
>
> As for Solr's security, it also needs to be replaced with a modern
> framework like Shiro or pac4j or similar. The permissions are
> basically impossible to understand, even for most committers. This
> whole idea of allow by default instead of deny by default goes against
> every security best practice in the industry. So arguing that we need
> some homegrown API framework to support Solr's obscure authz scheme
> seems a bit backwards to me. Once you start using standard API
> frameworks like JAX-RS, you start getting security integration too,
> e.g. https://github.com/pac4j/jax-rs-pac4j. Moreover, we can't keep up
> with all the innovation around security; heck, SolrJ doesn't even
> support OIDC yet. I'm all for scraping both homegrown frameworks and
> starting fresh with mature open source frameworks that have all the
> kinks worked out over many years with many projects using them.
>
Solr’s security framework is an API . You do not have to use the default
shipped implementation. You can implement anything and use it. However ,
it’s not possible to have one security framework for JAX-RS and another for
/select and /update
It has been there for the past decade and I don’t have a problem if you
wish to completely replace it with something else
>
> If you want to talk about stress, try building a non-trivial app with
> Solr's current API's, that'll induce some stress for sure!
>
> Tim
>
>
>
> On Thu, Dec 2, 2021 at 10:35 PM Noble Paul <no...@gmail.com> wrote:
> >
> > I do not deny that a standardized framework may be helpful. I'm already
> suffering from the PTSD of restlet integration. The integration code was
> extremely complex and a security nightmare.It was added without any perf
> tests too. It took many years to get rid of that
> >
> >
> >
> > On Fri, Dec 3, 2021 at 1:37 AM Jason Gerlowski <ge...@gmail.com>
> wrote:
> >>
> >> > 99% of Solr users never see the java code that implements an API let
> alone writing a new API.
> >>
> >> We may be talking past each other here a bit Noble. The JAX-RS
> >> benefits I've been trying to describe are to benefit us developers,
> >> not Solr users. I agree that 99% of our users won't know or care
> >> about it, but who cares. The goal is to make _our_ lives as
> >> maintainers easier going into the future. Jan described it in a word
> >> better than I've been struggling to: it's about "modernization".
> >>
> >> Best,
> >>
> >> Jason
> >>
> >> On Thu, Dec 2, 2021 at 3:34 AM Jan Høydahl <ja...@cominvent.com>
> wrote:
> >> >
> >> > Agree, Ishan.
> >> >
> >> > Let's welcome and cheer initiatives to modernize this old codebase.
> >> > Diving in deep and un-tangling DispatchFilter like Gus is attempting
> is brave.
> >> > And testing JAX-RS to modernize APIs and make the codebas easier to
> grok is also not for the faint hearted.
> >> >
> >> > I hope both initiatives end up as a breath of fresh air for Solr
> developers. And perhaps users may get their hands on auto-generated
> OpenAPI docs for Solr as well!
> >> > Theoretically foundations like this may make it feasible in the
> future to auto-generate client code from the spec,so that full Admin API
> client support in e.g. Go or Rust may be achievable without hand-coding.
> But that is still far away :)
> >> >
> >> > Jan
> >> >
> >> > 2. des. 2021 kl. 05:52 skrev Ishan Chattopadhyaya <
> ichattopadhyaya@gmail.com>:
> >> >
> >> > > We are discussing this as if moving to an external framework is
> going to be smooth. It's not.
> >> >
> >> > Noble, we won't know before someone has given it a try. My hunch
> suggests it won't be smooth, but we can decide on that once we see a
> concrete solution that 1) works well for all existing APIs (per core or
> global) without hacks/workarounds, and 2) has no security implications, and
> 3) doesn't clutter our codebase, and 4) doesn't affect our runtime
> performance.
> >> >
> >> > On Thu, Dec 2, 2021 at 4:22 AM Noble Paul <no...@gmail.com>
> wrote:
> >> >>
> >> >>
> >> >>
> >> >> On Thu, Dec 2, 2021 at 2:46 AM Jason Gerlowski <
> gerlowskija@gmail.com> wrote:
> >> >>>
> >> >>> > There are no known issues with the current system
> >> >>>
> >> >>> Maybe you're using hyperbole to emphasise a point, but let's steer
> >> >>> this discussion away from straw-men and caricatures. I mentioned
> >> >>> specific known issues in a previous reply as a direct response to
> your
> >> >>> question about them. The incomplete support for data types
> >> >>> (SOLR-15619 and SOLR-15796). The inflexible PermissionNameProvider
> >> >>> integration (SOLR-15823). If you consider those "minor", fair
> enough,
> >> >>> and you're welcome to that opinion. But saying that they don't
> exist
> >> >>> is a mischaracterization that wastes time and muddies water.
> >> >>
> >> >>
> >> >> We are discussing this as if moving to an external framework is
> going to be smooth. It's not. I have seen how hard the Restlet integration
> was and how it stood out like a sore thumb. The amount of code required to
> integrate Solr APIs into restlet components was 10x more than the amount of
> code written for the new framework and it still was incomplete .
> Integrating anything to Solr is not simple and it's going to be an ongoing
> effort to ensure that all of the functionalities of a SolrRequestHandler
> are exposed to the new framework. Every single component that we have today
> has ongoing improvements and bug fixes. Using that as an excuse to do a
> complete rewrite is probably unwise.
> >> >>
> >> >>
> >> >>>
> >> >>> > Solr is not a general purpose web server where people write APIs
> every day
> >> >>>
> >> >>> Totally agree - 99% of the time someone opens an API file in Solr,
> >> >>> they're reading, not writing. So if JAX-RS didn't help that case,
> it
> >> >>> wouldn't be worth the effort of switching. But IMO it helps that
> >> >>> "read" case a lot.
> >> >>
> >> >>
> >> >> 99% of Solr users never see the java code that implements an API let
> alone writing a new API. They wouldn't care if an API is implemented using
> a Servlet API, a custom framework or something else. They only care about
> the input and output. New end points are rare.
> >> >>>
> >> >>>
> >> >>> JAX-RS defined APIs expose inputs as strongly-typed parameters
> >> >>> (compare to the weakly-typed SolrQueryRequest/SolrParams used
> today).
> >> >>> JAX-RS enumerates every input to the API in one place. JAX-RS APIs
> >> >>> can return any serializable type as a response (compare to the
> >> >>> near-omnipresent NamedList today).
> >> >>>
> >> >>> (The annotation framework's PayloadObj class gets us a bit of this
> >> >>> benefit, but not all of it and only on POST requests afaik.)
> >> >>>
> >> >>> This is all nice stuff at API-write time, but where it really helps
> is
> >> >>> in making the code clearer and simpler whenever someone goes to read
> >> >>> it later. No wondering what parameters an API actually takes. No
> >> >>> puzzling out what a bunch of NamedList operations spread throughout
> >> >>> API execution actually produce in the end. etc.
> >> >>>
> >> >>> > The only reason why anyone would need to use this framework is to
> write a custom request handler.
> >> >>>
> >> >>> ...except for the majority of people who come in contact with the
> >> >>> framework by reading an existing Solr API? This ties into your
> point
> >> >>> above about APIs being written infrequently but read all the time.
> >> >>> Improving readability is the big gain here, and I think JAX-RS
> offers
> >> >>> real improvements in that regard.
> >> >>
> >> >>
> >> >> We had no framework for the last 15+ years and V1 API (the only
> actually used API) still uses no frameworks. Even the annotation framework
> is not used much even though readability is a lot better in the annotation
> based system. Readability has not been an overriding concern for V1 APIs
> for sure
> >> >>
> >> >>> Also, to clarify, I'm not asserting that no one configures custom
> >> >>> requestHandlers. I was wondering aloud how common it was, as an
> open
> >> >>> question.
> >> >>>
> >> >> Your point IS right. IRL, most of our users do not write request
> handlers . They write search components , query parsers and other plugins.
> Because we are a search engine and not a web server. I have never seen a
> user complaining about the V1 API of writing a request handler. They may
> actually complain about how hard it is to deploy one. Our users are much
> smarter than we give them credit for.
> >> >>
> >> >>
> >> >>>
> >> >>> > We did not fork Jackson. We are using Jackson itself.
> >> >>>
> >> >>> Ah, maybe you're right, "fork" is probably an unfair word. I looked
> >> >>> for the serialization code, and I'll admit there is less than I
> >> >>> thought involved here and I misunderstood how it works. So that's
> my
> >> >>> bad. The JSON serialization/deserialization discussion is prob a
> >> >>> digression from the core contention about whether JAX-RS would be
> >> >>> helpful. I was curious about your earlier response and couldn't
> >> >>> resist following up here but I should've done so elsewhere to avoid
> >> >>> the distraction.
> >> >>>
> >> >>> Best,
> >> >>>
> >> >>>
> >> >>> Jason
> >> >>>
> >> >>> On Tue, Nov 30, 2021 at 4:12 PM Noble Paul <no...@gmail.com>
> wrote:
> >> >>> >
> >> >>> > The annotation framework is just a single class and it is made to
> work well with Solr instead of changing Solr to suit the needs of some
> external framework. There are no known issues with the current system and
> there are unknown issues that you'll face introducing a new framework.
> >> >>> >
> >> >>> > Solr is not a general purpose web server where people write APIs
> every day. It's a finished product where people use the publicly available
> APIs . We should focus our efforts on making our APIs work well .
> >> >>> >
> >> >>> > On Wed, Dec 1, 2021, 2:14 AM Jason Gerlowski <
> gerlowskija@gmail.com> wrote:
> >> >>> >>
> >> >>> >> > We need to make an API work on a per core basis and cores can
> come up and go down randomly
> >> >>> >>
> >> >>> >> Ah, yeah. I'll admit I was overlooking some of the wrinkles
> around
> >> >>> >> custom request handlers etc. That is a problem for switching to
> some
> >> >>> >> off-the-shelf framework for sure. It may still be possible, but
> it's
> >> >>> >> definitely a big hurdle. I wonder how commonly users still take
> >> >>> >> advantage of the custom requestHandler stuff these days? I
> haven't
> >> >>> >> run across many customers who used them extensively, but maybe
> that's
> >> >>> >> self-selecting in some way?
> >> >>> >
> >> >>> >
> >> >>> > The only reason why anyone would need to use this framework is to
> write a custom request handler. You're also saying nobody writes custom
> request handlers. So, what problem are you trying to solve if you think
> nobody uses them?
> >> >>> >
> >> >>> >
> >> >>> >>
> >> >>> >> > If SolrJ has a Jackson dependency, it can have a conflict
> (different versions) with the Jackson used by the client app
> >> >>> >>
> >> >>> >> SolrJ has a handful of dependencies - is there a reason we take
> this
> >> >>> >> approach with Jackson, but not with any of SolrJ's other deps?
> >> >>> >> (apache-httpcomponents, apache-commons, zookeeper-client, etc.)
> >> >>> >>
> >> >>> >> Granted, I get that we want SolrJ to be as painless as possible
> for
> >> >>> >> its users, and jar-conflicts are a part of that and really
> suck. But
> >> >>> >> there are good tools out there for handling jar conflicts, and
> >> >>> >> reimplementing chunks of library functionality just to avoid a
> >> >>> >> gradle-dep strikes me as unsustainable from a
> maintenance/community
> >> >>> >> perspective. Not to mention that forking/mimicking Jackson code
> >> >>> >> deprives our users of any efficiency/security improvements that
> >> >>> >> Jackson might get tomorrow or next week or next year.
> >> >>> >
> >> >>> >
> >> >>> > We did not fork Jackson. We are using Jackson itself. Jackson
> supports custom annotations. There was a huge discussion why we should do
> it this way.
> >> >>> >>
> >> >>> >>
> >> >>> >> Unless there's some reason specific to Jackson I guess I just
> don't
> >> >>> >> get it. But if I'm the minority opinion on that, fair enough I
> guess.
> >> >>> >>
> >> >>> >> On Tue, Nov 30, 2021 at 5:05 AM Noble Paul <no...@gmail.com>
> wrote:
> >> >>> >> >
> >> >>> >> > The @JsonProperty annotations are added because it's a
> dependency in SolrJ as well
> >> >>> >> > If SolrJ has a Jackson dependency, it can have a conflict
> (different versions) with the Jackson used by the client app.
> >> >>> >> >
> >> >>> >> > On Tue, Nov 30, 2021, 5:21 PM Noble Paul <no...@gmail.com>
> wrote:
> >> >>> >> >>
> >> >>> >> >> Also keep in mind that the same endpoint can be accessed with
> a core name and a collection name prefixes.
> >> >>> >> >>
> >> >>> >> >> On Tue, Nov 30, 2021 at 3:55 PM Noble Paul <
> noble.paul@gmail.com> wrote:
> >> >>> >> >>>
> >> >>> >> >>> True Gus, Almost every framework works outside of
> SolrDispatchFilter+ HttpSolrCall. A lot of our initializations occur there.
> >> >>> >> >>>
> >> >>> >> >>> We need to make an API work on a per core basis and cores
> can come up and go down randomly. So we need to register these endpoints on
> a core.
> >> >>> >> >>>
> >> >>> >> >>> I'm not sure if any framework can achieve the same.
> >> >>> >> >>>
> >> >>> >> >>> On Tue, Nov 30, 2021 at 5:39 AM Gus Heck <gu...@gmail.com>
> wrote:
> >> >>> >> >>>>
> >> >>> >> >>>> IIRC last time I looked restlet had the unsavory property
> of existing outside of the SolrDispatchFilter, unlike everything else which
> made for special cases because several things that probably ought to be
> their own siervlet filters are glommed into SolrDispatchFilter, like
> security, tracing and MDC setup/teardown per request. Restlet wouldn't be
> so bad if one could just wrap such filters around it too...
> >> >>> >> >>>>
> >> >>> >> >>>> On Mon, Nov 29, 2021 at 9:42 AM Jason Gerlowski <
> gerlowskija@gmail.com> wrote:
> >> >>> >> >>>>>
> >> >>> >> >>>>> > These are minor improvements compared to a full rewrite
> of the entire framework
> >> >>> >> >>>>>
> >> >>> >> >>>>> If you think data type support is minor, fair enough. But
> to clarify
> >> >>> >> >>>>> I'm not suggesting a rewrite - I'm suggesting using
> something that
> >> >>> >> >>>>> already exists off the shelf. Jersey (e.g.) itself
> provides the
> >> >>> >> >>>>> framework - there would be no "rewrite".
> >> >>> >> >>>>>
> >> >>> >> >>>>> re: past restlet use
> >> >>> >> >>>>>
> >> >>> >> >>>>> > It was not playing well with our security framework. The
> framework was not working well with Solr APIs
> >> >>> >> >>>>>
> >> >>> >> >>>>> Ah, very interesting! Security isn't something Eric or I
> tackled in
> >> >>> >> >>>>> our little spike branch, but it's definitely a concern.
> Do you
> >> >>> >> >>>>> remember the specific concerns? Or recall where any of
> the discussion
> >> >>> >> >>>>> around this happened?
> >> >>> >> >>>>>
> >> >>> >> >>>>> Without the context of that past discussion, it seems like
> the
> >> >>> >> >>>>> "PermissionNameProvider" interface could be implemented
> just as well
> >> >>> >> >>>>> by a class with (e.g.) Jersey annotations as one with our
> own custom
> >> >>> >> >>>>> annotations. Certainly there'd need to be some
> >> >>> >> >>>>> RuleBasedAuthorizationPlugin changes or other integration
> code, but
> >> >>> >> >>>>> nothing that feels insurmountable.
> >> >>> >> >>>>>
> >> >>> >> >>>>> Maybe I can try spiking it out soon and find the issues
> myself, but
> >> >>> >> >>>>> it'd be much easier if someone happens to remember and can
> save me the
> >> >>> >> >>>>> trouble :-p
> >> >>> >> >>>>>
> >> >>> >> >>>>> Best,
> >> >>> >> >>>>>
> >> >>> >> >>>>> Jason
> >> >>> >> >>>>>
> >> >>> >> >>>>> On Fri, Nov 26, 2021 at 12:05 AM Noble Paul <
> noble.paul@gmail.com> wrote:
> >> >>> >> >>>>> >
> >> >>> >> >>>>> > The Annotations framework was written after playing with
> other frameworks. There were many shortcomings which were hard to overcome.
> >> >>> >> >>>>> >
> >> >>> >> >>>>> > The best example is a per collection API . How do you
> register an endpoint for a collection/core ?
> >> >>> >> >>>>> >
> >> >>> >> >>>>> > On Fri, Nov 26, 2021 at 3:42 PM Noble Paul <
> noble.paul@gmail.com> wrote:
> >> >>> >> >>>>> >>
> >> >>> >> >>>>> >>
> >> >>> >> >>>>> >>
> >> >>> >> >>>>> >> On Fri, Nov 26, 2021 at 1:03 AM Jason Gerlowski <
> gerlowskija@gmail.com> wrote:
> >> >>> >> >>>>> >>>
> >> >>> >> >>>>> >>> > Is there some problem with our annotations that we
> hope to solve using third party dependencies?
> >> >>> >> >>>>> >>>
> >> >>> >> >>>>> >>> I guess so yeah. Third-party deps are just fuller,
> more robust
> >> >>> >> >>>>> >>> solutions, whereas our annotations still need support
> added now and
> >> >>> >> >>>>> >>> then for even primitive data types like "long" (see
> SOLR-15619).
> >> >>> >> >>>>> >>
> >> >>> >> >>>>> >> These are minor improvements compared to a full rewrite
> of the entire framework
> >> >>> >> >>>>> >>
> >> >>> >> >>>>> >>
> >> >>> >> >>>>> >>>
> >> >>> >> >>>>> >>> Every JIRA spent doing basic stuff like that is time
> away from
> >> >>> >> >>>>> >>> improving Solr in some other way.
> >> >>> >> >>>>> >>>
> >> >>> >> >>>>> >>> So there are feature-gap/capabilities arguments for
> moving to a
> >> >>> >> >>>>> >>> third-party dep, sure. But, even if our annotations
> did everything
> >> >>> >> >>>>> >>> Jersey+Jackson do today, I think switching would still
> be worth it.
> >> >>> >> >>>>> >>> Every LOC in our code base brings along with it some
> maintenance cost:
> >> >>> >> >>>>> >>> it might have bugs, needs tested, takes time for new
> contributors to
> >> >>> >> >>>>> >>> "grok", etc. Using off-the-shelf here would nuke a
> whole bunch of
> >> >>> >> >>>>> >>> that. If off-the-shelf is available for some given
> functionality, we
> >> >>> >> >>>>> >>> should need a compelling reason to NOT use it.
> >> >>> >> >>>>> >>>
> >> >>> >> >>>>> >>> Lastly, I think there's an "approachability" argument
> for using
> >> >>> >> >>>>> >>> off-the-shelf. Thousands of developers out there are
> familiar with
> >> >>> >> >>>>> >>> (e.g.) Jersey, compared to maybe 15 or 20 (in the
> world) familiar with
> >> >>> >> >>>>> >>> Solr's custom annotations. Using a well-known
> technology like Jersey
> >> >>> >> >>>>> >>> would make Solr all the easier to approach and
> contribute to for that
> >> >>> >> >>>>> >>> pool of developers.
> >> >>> >> >>>>> >>>
> >> >>> >> >>>>> >>> > By the way, we have used Restlet in the past and
> that has been a regrettable decision.
> >> >>> >> >>>>> >>
> >> >>> >> >>>>> >>
> >> >>> >> >>>>> >>>
> >> >>> >> >>>>> >>> Ah, yeah, that's just the context I'm missing. Anyone
> have a pointer
> >> >>> >> >>>>> >>> to related discussions, or remember what made this
> "regrettable"? All
> >> >>> >> >>>>> >>> the theoretical benefits in the world don't matter
> much if we've
> >> >>> >> >>>>> >>> already tried something like this in the past and
> decided against it.
> >> >>> >> >>>>> >>
> >> >>> >> >>>>> >>
> >> >>> >> >>>>> >> It was not playing well with our security framework.
> The framework was not working well with Solr APIs
> >> >>> >> >>>>> >>
> >> >>> >> >>>>> >>>
> >> >>> >> >>>>> >>> (Unrelated - Happy Thanksgiving all!)
> >> >>> >> >>>>> >>>
> >> >>> >> >>>>> >>> Best,
> >> >>> >> >>>>> >>>
> >> >>> >> >>>>> >>> Jason
> >> >>> >> >>>>> >>>
> >> >>> >> >>>>> >>> On Thu, Nov 25, 2021 at 7:32 AM Noble Paul <
> noble.paul@gmail.com> wrote:
> >> >>> >> >>>>> >>> >
> >> >>> >> >>>>> >>> > Have you gone through an API written using the
> @EndPoint annotation?
> >> >>> >> >>>>> >>> >
> >> >>> >> >>>>> >>> > I strongly recommend that you do
> >> >>> >> >>>>> >>> >
> >> >>> >> >>>>> >>> > On Thu, Nov 25, 2021, 11:30 PM Eric Pugh <
> epugh@opensourceconnections.com> wrote:
> >> >>> >> >>>>> >>> >>
> >> >>> >> >>>>> >>> >> I have found our V2 API code to be very
> impenetrable to understand. Part of it is how the code is intertwined
> with support for V1, however it’s also because there aren’t really
> resources to go look at to understand how it should work! Maintaining the
> API should be very simple work, as they just exist as a translation. The
> home grown stuff may make sense if you are a super knowledgable Solr
> developer, but if you are just a new person, it’s a lot harder to
> contribute.
> >> >>> >> >>>>> >>> >>
> >> >>> >> >>>>> >>> >> I was interested in the Jersey stuff because I’ve
> seen lots of projects use it very successfully, and if I want to implement
> something, well, there are lots of blogs and resources out there!
> >> >>> >> >>>>> >>> >>
> >> >>> >> >>>>> >>> >> Can anyone recap briefly why we dropped RESTlet?
> And what lessons learned there might apply to adopting Jersey for API
> support? Looking at https://issues.apache.org/jira/browse/SOLR-14659,
> it was partly deprecated because we were not using it to support all the
> API, only the ManagedResource ones, and
> https://issues.apache.org/jira/browse/SOLR-14766 suggests that RESTlet
> maybe was no longer being updated? One reason why we spiked out Jersey
> was because of the broad support in the Java world! Looking at how much
> work we have to do in the V2 API world, we need a much broader pool of
> developers contributing to get there!
> >> >>> >> >>>>> >>> >>
> >> >>> >> >>>>> >>> >> Related, are there specific features/aspects of our
> annotations that enable things in Solr that couldn’t be done otherwise?
> >> >>> >> >>>>> >>> >>
> >> >>> >> >>>>> >>> >> On Nov 25, 2021, at 2:12 AM, Ishan Chattopadhyaya <
> ichattopadhyaya@gmail.com> wrote:
> >> >>> >> >>>>> >>> >>
> >> >>> >> >>>>> >>> >> Is there some problem with our annotations that we
> hope to solve using third party dependencies?
> >> >>> >> >>>>> >>> >> By the way, we have used Restlet in the past and
> that has been a regrettable decision.
> >> >>> >> >>>>> >>> >>
> >> >>> >> >>>>> >>> >> On Thu, Nov 25, 2021 at 10:10 AM Jason Gerlowski <
> gerlowskija@gmail.com> wrote:
> >> >>> >> >>>>> >>> >>>
> >> >>> >> >>>>> >>> >>> Solr's custom annotation framework ('@Endpoint',
> '@Command', etc.) has
> >> >>> >> >>>>> >>> >>> cropped up a few times over the past week or two.
> [1] [2]. Having them
> >> >>> >> >>>>> >>> >>> on top of mind, I've been wondering - is there a
> reason we use our own
> >> >>> >> >>>>> >>> >>> annotations here instead of something off the
> shelf?
> >> >>> >> >>>>> >>> >>>
> >> >>> >> >>>>> >>> >>> What we have works well enough, but anything
> homegrown comes with more
> >> >>> >> >>>>> >>> >>> maintenance burden than we'd have if we used
> something off the shelf.
> >> >>> >> >>>>> >>> >>> There are plenty of well-used, active projects out
> there whose whole
> >> >>> >> >>>>> >>> >>> purpose is facilitating the whole "annotation
> based API" thing
> >> >>> >> >>>>> >>> >>> (Jersey, Restlet, RESTEasy, etc.) - why not use
> one of them?
> >> >>> >> >>>>> >>> >>>
> >> >>> >> >>>>> >>> >>> Does anyone know of any technical reasons why we
> can't go this route?
> >> >>> >> >>>>> >>> >>> Or have any subjective reasons why we shouldn't?
> Or any context on
> >> >>> >> >>>>> >>> >>> why we wrote our own Endpoint, Command,
> JsonProperty annotations
> >> >>> >> >>>>> >>> >>> originally?
> >> >>> >> >>>>> >>> >>>
> >> >>> >> >>>>> >>> >>> FWIW, Eric Pugh and I spiked out a small POC
> recently, and got
> >> >>> >> >>>>> >>> >>> Jersey+Jackson working for a few simple APIs
> without too much trouble.
> >> >>> >> >>>>> >>> >>> [3] Obviously nothing production-ready there, and
> there's still a lot
> >> >>> >> >>>>> >>> >>> of open questions (e.g. how would javabin be
> supported?), but we both
> >> >>> >> >>>>> >>> >>> came away convinced that it seemed feasible, at
> least. Best of all,
> >> >>> >> >>>>> >>> >>> APIs using our current homegrown annotation
> framework the switchover
> >> >>> >> >>>>> >>> >>> seems blessedly straightforward, and it doesn't
> look like Jersey
> >> >>> >> >>>>> >>> >>> (which we chose mostly arbitrarily) bloats our
> dist all that much.
> >> >>> >> >>>>> >>> >>>
> >> >>> >> >>>>> >>> >>> Curious if anyone has thoughts or context on how
> we ended up with the
> >> >>> >> >>>>> >>> >>> annotation setup we use today!
> >> >>> >> >>>>> >>> >>>
> >> >>> >> >>>>> >>> >>> Best,
> >> >>> >> >>>>> >>> >>>
> >> >>> >> >>>>> >>> >>> Jason
> >> >>> >> >>>>> >>> >>>
> >> >>> >> >>>>> >>> >>> [1]
> https://issues.apache.org/jira/browse/SOLR-15182 (and children)
> >> >>> >> >>>>> >>> >>> [2]
> http://mail-archives.apache.org/mod_mbox/solr-dev/202111.mbox/%3CCABEwPvENL41Pm6%2BOmjXb6Sx5N2XjUtnbWhgKOZSrnLjWBA8tcA%40mail.gmail.com%3E
> >> >>> >> >>>>> >>> >>> [3]
> https://github.com/gerlowskija/solr/tree/jersey_jaxrs_jackson_solr_apis.
> >> >>> >> >>>>> >>> >>>
> >> >>> >> >>>>> >>> >>>
> ---------------------------------------------------------------------
> >> >>> >> >>>>> >>> >>> To unsubscribe, e-mail:
> dev-unsubscribe@solr.apache.org
> >> >>> >> >>>>> >>> >>> For additional commands, e-mail:
> dev-help@solr.apache.org
> >> >>> >> >>>>> >>> >>>
> >> >>> >> >>>>> >>> >>
> >> >>> >> >>>>> >>> >> _______________________
> >> >>> >> >>>>> >>> >> Eric Pugh | Founder & CEO | OpenSource Connections,
> LLC | 434.466.1467 | http://www.opensourceconnections.com | My Free/Busy
> >> >>> >> >>>>> >>> >> Co-Author: Apache Solr Enterprise Search Server,
> 3rd Ed
> >> >>> >> >>>>> >>> >> This e-mail and all contents, including
> attachments, is considered to be Company Confidential unless explicitly
> stated otherwise, regardless of whether attachments are marked as such.
> >> >>> >> >>>>> >>> >>
> >> >>> >> >>>>> >>>
> >> >>> >> >>>>> >>>
> ---------------------------------------------------------------------
> >> >>> >> >>>>> >>> To unsubscribe, e-mail:
> dev-unsubscribe@solr.apache.org
> >> >>> >> >>>>> >>> For additional commands, e-mail:
> dev-help@solr.apache.org
> >> >>> >> >>>>> >>>
> >> >>> >> >>>>> >>
> >> >>> >> >>>>> >>
> >> >>> >> >>>>> >> --
> >> >>> >> >>>>> >> -----------------------------------------------------
> >> >>> >> >>>>> >> Noble Paul
> >> >>> >> >>>>> >
> >> >>> >> >>>>> >
> >> >>> >> >>>>> >
> >> >>> >> >>>>> > --
> >> >>> >> >>>>> > -----------------------------------------------------
> >> >>> >> >>>>> > Noble Paul
> >> >>> >> >>>>>
> >> >>> >> >>>>>
> ---------------------------------------------------------------------
> >> >>> >> >>>>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> >> >>> >> >>>>> For additional commands, e-mail: dev-help@solr.apache.org
> >> >>> >> >>>>>
> >> >>> >> >>>>
> >> >>> >> >>>>
> >> >>> >> >>>> --
> >> >>> >> >>>> http://www.needhamsoftware.com (work)
> >> >>> >> >>>> http://www.the111shift.com (play)
> >> >>> >> >>>
> >> >>> >> >>>
> >> >>> >> >>>
> >> >>> >> >>> --
> >> >>> >> >>> -----------------------------------------------------
> >> >>> >> >>> Noble Paul
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >> --
> >> >>> >> >> -----------------------------------------------------
> >> >>> >> >> Noble Paul
> >> >>> >>
> >> >>> >>
> ---------------------------------------------------------------------
> >> >>> >> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> >> >>> >> For additional commands, e-mail: dev-help@solr.apache.org
> >> >>> >>
> >> >>>
> >> >>>
> ---------------------------------------------------------------------
> >> >>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> >> >>> For additional commands, e-mail: dev-help@solr.apache.org
> >> >>>
> >> >>
> >> >>
> >> >> --
> >> >> -----------------------------------------------------
> >> >> Noble Paul
> >> >
> >> >
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> >> For additional commands, e-mail: dev-help@solr.apache.org
> >>
> >
> >
> > --
> > -----------------------------------------------------
> > Noble Paul
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org
>
>
--
-----------------------------------------------------
Noble Paul
Re: JAX-RS APIs in Solr
Posted by Ishan Chattopadhyaya <ic...@gmail.com>.
FYI, I misspelt "Amazon Ion" to "Amazon Eon". I meant this:
https://en.wikipedia.org/wiki/Ion_(serialization_format)
On Mon, Dec 6, 2021 at 6:56 PM Eric Pugh <ep...@opensourceconnections.com>
wrote:
> I was actually poking around trying to find out who actually in the Ruby
> world uses the wt=ruby, and all the projects that intersect with Solr seems
> to do wt=json ;-)
>
>
> On Dec 6, 2021, at 8:17 AM, Jan Høydahl <ja...@cominvent.com> wrote:
>
> I'm happy to see that there is agreement in a shift towards standard..
>
> Bwt - in v3, lets get rid of "wt" parameter and instead use standard HTTP
> Accept header to tell Solr whether you want the response as JSON or XML.
> Maybe we are even ready to deprecate XML, PYTHON, XSLT, Excel..
> See https://issues.apache.org/jira/browse/SOLR-10998
>
> Jan
>
> 3. des. 2021 kl. 17:31 skrev Timothy Potter <th...@gmail.com>:
>
> Why do we keep talking about Restlet here? Restlet is a stale (at
> best) project, so not sure why it keeps coming up in a discussion
> about modernizing our API? Integration with it in Solr was introduced
> nearly 10 years ago, time to move on and stop using that as an excuse
> to block adoption of other API frameworks.
>
> Surely anyone committed to this effort, after doing even the most
> cursory due diligence, will not land on Restlet as the way forward.
> Jersey is a likely candidate, I'm sure there are others?
>
> I actually don't care all that much about which annotations we use to
> describe our APIs (of course standards are better). What's more
> important for me, is that we can auto-generate an OpenAPI spec for our
> APIs. That opens up a world of automated tooling for our users to
> interact with our APIs. Using JAX-RS / Jersey probably helps get us
> closer to an automated way to generate an OpenAPI spec but there are
> other ways to get there as well.
>
> In general, it sounds like the objections to modernizing our API fall
> into three buckets: 1) our bad experience with Restlet, 2) performance
> is better with our homegrown code, 3) security is too complicated with
> another framework.
>
> Performance overhead added by a framework like Jersey (if there is
> any) for Admin APIs is mostly irrelevant. You think Jersey is going to
> be slower at mapping requests to Java method calls? Pretty sure they
> have that part figured out at this point.
>
> As for Solr's security, it also needs to be replaced with a modern
> framework like Shiro or pac4j or similar. The permissions are
> basically impossible to understand, even for most committers. This
> whole idea of allow by default instead of deny by default goes against
> every security best practice in the industry. So arguing that we need
> some homegrown API framework to support Solr's obscure authz scheme
> seems a bit backwards to me. Once you start using standard API
> frameworks like JAX-RS, you start getting security integration too,
> e.g. https://github.com/pac4j/jax-rs-pac4j. Moreover, we can't keep up
> with all the innovation around security; heck, SolrJ doesn't even
> support OIDC yet. I'm all for scraping both homegrown frameworks and
> starting fresh with mature open source frameworks that have all the
> kinks worked out over many years with many projects using them.
>
> If you want to talk about stress, try building a non-trivial app with
> Solr's current API's, that'll induce some stress for sure!
>
> Tim
>
>
>
> On Thu, Dec 2, 2021 at 10:35 PM Noble Paul <no...@gmail.com> wrote:
>
>
> I do not deny that a standardized framework may be helpful. I'm already
> suffering from the PTSD of restlet integration. The integration code was
> extremely complex and a security nightmare.It was added without any perf
> tests too. It took many years to get rid of that
>
>
>
> On Fri, Dec 3, 2021 at 1:37 AM Jason Gerlowski <ge...@gmail.com>
> wrote:
>
>
> 99% of Solr users never see the java code that implements an API let alone
> writing a new API.
>
>
> We may be talking past each other here a bit Noble. The JAX-RS
> benefits I've been trying to describe are to benefit us developers,
> not Solr users. I agree that 99% of our users won't know or care
> about it, but who cares. The goal is to make _our_ lives as
> maintainers easier going into the future. Jan described it in a word
> better than I've been struggling to: it's about "modernization".
>
> Best,
>
> Jason
>
> On Thu, Dec 2, 2021 at 3:34 AM Jan Høydahl <ja...@cominvent.com> wrote:
>
>
> Agree, Ishan.
>
> Let's welcome and cheer initiatives to modernize this old codebase.
> Diving in deep and un-tangling DispatchFilter like Gus is attempting is
> brave.
> And testing JAX-RS to modernize APIs and make the codebas easier to grok
> is also not for the faint hearted.
>
> I hope both initiatives end up as a breath of fresh air for Solr
> developers. And perhaps users may get their hands on auto-generated OpenAPI
> docs for Solr as well!
> Theoretically foundations like this may make it feasible in the future to
> auto-generate client code from the spec,so that full Admin API client
> support in e.g. Go or Rust may be achievable without hand-coding. But that
> is still far away :)
>
> Jan
>
> 2. des. 2021 kl. 05:52 skrev Ishan Chattopadhyaya <
> ichattopadhyaya@gmail.com>:
>
> We are discussing this as if moving to an external framework is going to
> be smooth. It's not.
>
>
> Noble, we won't know before someone has given it a try. My hunch suggests
> it won't be smooth, but we can decide on that once we see a concrete
> solution that 1) works well for all existing APIs (per core or global)
> without hacks/workarounds, and 2) has no security implications, and 3)
> doesn't clutter our codebase, and 4) doesn't affect our runtime performance.
>
> On Thu, Dec 2, 2021 at 4:22 AM Noble Paul <no...@gmail.com> wrote:
>
>
>
>
> On Thu, Dec 2, 2021 at 2:46 AM Jason Gerlowski <ge...@gmail.com>
> wrote:
>
>
> There are no known issues with the current system
>
>
> Maybe you're using hyperbole to emphasise a point, but let's steer
> this discussion away from straw-men and caricatures. I mentioned
> specific known issues in a previous reply as a direct response to your
> question about them. The incomplete support for data types
> (SOLR-15619 and SOLR-15796). The inflexible PermissionNameProvider
> integration (SOLR-15823). If you consider those "minor", fair enough,
> and you're welcome to that opinion. But saying that they don't exist
> is a mischaracterization that wastes time and muddies water.
>
>
>
> We are discussing this as if moving to an external framework is going to
> be smooth. It's not. I have seen how hard the Restlet integration was and
> how it stood out like a sore thumb. The amount of code required to
> integrate Solr APIs into restlet components was 10x more than the amount of
> code written for the new framework and it still was incomplete .
> Integrating anything to Solr is not simple and it's going to be an ongoing
> effort to ensure that all of the functionalities of a SolrRequestHandler
> are exposed to the new framework. Every single component that we have today
> has ongoing improvements and bug fixes. Using that as an excuse to do a
> complete rewrite is probably unwise.
>
>
>
> Solr is not a general purpose web server where people write APIs every day
>
>
> Totally agree - 99% of the time someone opens an API file in Solr,
> they're reading, not writing. So if JAX-RS didn't help that case, it
> wouldn't be worth the effort of switching. But IMO it helps that
> "read" case a lot.
>
>
>
> 99% of Solr users never see the java code that implements an API let alone
> writing a new API. They wouldn't care if an API is implemented using a
> Servlet API, a custom framework or something else. They only care about
> the input and output. New end points are rare.
>
>
>
> JAX-RS defined APIs expose inputs as strongly-typed parameters
> (compare to the weakly-typed SolrQueryRequest/SolrParams used today).
> JAX-RS enumerates every input to the API in one place. JAX-RS APIs
> can return any serializable type as a response (compare to the
> near-omnipresent NamedList today).
>
> (The annotation framework's PayloadObj class gets us a bit of this
> benefit, but not all of it and only on POST requests afaik.)
>
> This is all nice stuff at API-write time, but where it really helps is
> in making the code clearer and simpler whenever someone goes to read
> it later. No wondering what parameters an API actually takes. No
> puzzling out what a bunch of NamedList operations spread throughout
> API execution actually produce in the end. etc.
>
> The only reason why anyone would need to use this framework is to write a
> custom request handler.
>
>
> ...except for the majority of people who come in contact with the
> framework by reading an existing Solr API? This ties into your point
> above about APIs being written infrequently but read all the time.
> Improving readability is the big gain here, and I think JAX-RS offers
> real improvements in that regard.
>
>
>
> We had no framework for the last 15+ years and V1 API (the only actually
> used API) still uses no frameworks. Even the annotation framework is not
> used much even though readability is a lot better in the annotation based
> system. Readability has not been an overriding concern for V1 APIs for sure
>
> Also, to clarify, I'm not asserting that no one configures custom
> requestHandlers. I was wondering aloud how common it was, as an open
> question.
>
> Your point IS right. IRL, most of our users do not write request handlers
> . They write search components , query parsers and other plugins. Because
> we are a search engine and not a web server. I have never seen a user
> complaining about the V1 API of writing a request handler. They may
> actually complain about how hard it is to deploy one. Our users are much
> smarter than we give them credit for.
>
>
>
> We did not fork Jackson. We are using Jackson itself.
>
>
> Ah, maybe you're right, "fork" is probably an unfair word. I looked
> for the serialization code, and I'll admit there is less than I
> thought involved here and I misunderstood how it works. So that's my
> bad. The JSON serialization/deserialization discussion is prob a
> digression from the core contention about whether JAX-RS would be
> helpful. I was curious about your earlier response and couldn't
> resist following up here but I should've done so elsewhere to avoid
> the distraction.
>
> Best,
>
>
> Jason
>
> On Tue, Nov 30, 2021 at 4:12 PM Noble Paul <no...@gmail.com> wrote:
>
>
> The annotation framework is just a single class and it is made to work
> well with Solr instead of changing Solr to suit the needs of some external
> framework. There are no known issues with the current system and there are
> unknown issues that you'll face introducing a new framework.
>
> Solr is not a general purpose web server where people write APIs every
> day. It's a finished product where people use the publicly available APIs .
> We should focus our efforts on making our APIs work well .
>
> On Wed, Dec 1, 2021, 2:14 AM Jason Gerlowski <ge...@gmail.com>
> wrote:
>
>
> We need to make an API work on a per core basis and cores can come up and
> go down randomly
>
>
> Ah, yeah. I'll admit I was overlooking some of the wrinkles around
> custom request handlers etc. That is a problem for switching to some
> off-the-shelf framework for sure. It may still be possible, but it's
> definitely a big hurdle. I wonder how commonly users still take
> advantage of the custom requestHandler stuff these days? I haven't
> run across many customers who used them extensively, but maybe that's
> self-selecting in some way?
>
>
>
> The only reason why anyone would need to use this framework is to write a
> custom request handler. You're also saying nobody writes custom request
> handlers. So, what problem are you trying to solve if you think nobody uses
> them?
>
>
>
> If SolrJ has a Jackson dependency, it can have a conflict (different
> versions) with the Jackson used by the client app
>
>
> SolrJ has a handful of dependencies - is there a reason we take this
> approach with Jackson, but not with any of SolrJ's other deps?
> (apache-httpcomponents, apache-commons, zookeeper-client, etc.)
>
> Granted, I get that we want SolrJ to be as painless as possible for
> its users, and jar-conflicts are a part of that and really suck. But
> there are good tools out there for handling jar conflicts, and
> reimplementing chunks of library functionality just to avoid a
> gradle-dep strikes me as unsustainable from a maintenance/community
> perspective. Not to mention that forking/mimicking Jackson code
> deprives our users of any efficiency/security improvements that
> Jackson might get tomorrow or next week or next year.
>
>
>
> We did not fork Jackson. We are using Jackson itself. Jackson supports
> custom annotations. There was a huge discussion why we should do it this
> way.
>
>
>
> Unless there's some reason specific to Jackson I guess I just don't
> get it. But if I'm the minority opinion on that, fair enough I guess.
>
> On Tue, Nov 30, 2021 at 5:05 AM Noble Paul <no...@gmail.com> wrote:
>
>
> The @JsonProperty annotations are added because it's a dependency in SolrJ
> as well
> If SolrJ has a Jackson dependency, it can have a conflict (different
> versions) with the Jackson used by the client app.
>
> On Tue, Nov 30, 2021, 5:21 PM Noble Paul <no...@gmail.com> wrote:
>
>
> Also keep in mind that the same endpoint can be accessed with a core name
> and a collection name prefixes.
>
> On Tue, Nov 30, 2021 at 3:55 PM Noble Paul <no...@gmail.com> wrote:
>
>
> True Gus, Almost every framework works outside of SolrDispatchFilter+
> HttpSolrCall. A lot of our initializations occur there.
>
> We need to make an API work on a per core basis and cores can come up and
> go down randomly. So we need to register these endpoints on a core.
>
> I'm not sure if any framework can achieve the same.
>
> On Tue, Nov 30, 2021 at 5:39 AM Gus Heck <gu...@gmail.com> wrote:
>
>
> IIRC last time I looked restlet had the unsavory property of existing
> outside of the SolrDispatchFilter, unlike everything else which made for
> special cases because several things that probably ought to be their own
> siervlet filters are glommed into SolrDispatchFilter, like security,
> tracing and MDC setup/teardown per request. Restlet wouldn't be so bad if
> one could just wrap such filters around it too...
>
> On Mon, Nov 29, 2021 at 9:42 AM Jason Gerlowski <ge...@gmail.com>
> wrote:
>
>
> These are minor improvements compared to a full rewrite of the entire
> framework
>
>
> If you think data type support is minor, fair enough. But to clarify
> I'm not suggesting a rewrite - I'm suggesting using something that
> already exists off the shelf. Jersey (e.g.) itself provides the
> framework - there would be no "rewrite".
>
> re: past restlet use
>
> It was not playing well with our security framework. The framework was not
> working well with Solr APIs
>
>
> Ah, very interesting! Security isn't something Eric or I tackled in
> our little spike branch, but it's definitely a concern. Do you
> remember the specific concerns? Or recall where any of the discussion
> around this happened?
>
> Without the context of that past discussion, it seems like the
> "PermissionNameProvider" interface could be implemented just as well
> by a class with (e.g.) Jersey annotations as one with our own custom
> annotations. Certainly there'd need to be some
> RuleBasedAuthorizationPlugin changes or other integration code, but
> nothing that feels insurmountable.
>
> Maybe I can try spiking it out soon and find the issues myself, but
> it'd be much easier if someone happens to remember and can save me the
> trouble :-p
>
> Best,
>
> Jason
>
> On Fri, Nov 26, 2021 at 12:05 AM Noble Paul <no...@gmail.com> wrote:
>
>
> The Annotations framework was written after playing with other frameworks.
> There were many shortcomings which were hard to overcome.
>
> The best example is a per collection API . How do you register an endpoint
> for a collection/core ?
>
> On Fri, Nov 26, 2021 at 3:42 PM Noble Paul <no...@gmail.com> wrote:
>
>
>
>
> On Fri, Nov 26, 2021 at 1:03 AM Jason Gerlowski <ge...@gmail.com>
> wrote:
>
>
> Is there some problem with our annotations that we hope to solve using
> third party dependencies?
>
>
> I guess so yeah. Third-party deps are just fuller, more robust
> solutions, whereas our annotations still need support added now and
> then for even primitive data types like "long" (see SOLR-15619).
>
>
> These are minor improvements compared to a full rewrite of the entire
> framework
>
>
>
> Every JIRA spent doing basic stuff like that is time away from
> improving Solr in some other way.
>
> So there are feature-gap/capabilities arguments for moving to a
> third-party dep, sure. But, even if our annotations did everything
> Jersey+Jackson do today, I think switching would still be worth it.
> Every LOC in our code base brings along with it some maintenance cost:
> it might have bugs, needs tested, takes time for new contributors to
> "grok", etc. Using off-the-shelf here would nuke a whole bunch of
> that. If off-the-shelf is available for some given functionality, we
> should need a compelling reason to NOT use it.
>
> Lastly, I think there's an "approachability" argument for using
> off-the-shelf. Thousands of developers out there are familiar with
> (e.g.) Jersey, compared to maybe 15 or 20 (in the world) familiar with
> Solr's custom annotations. Using a well-known technology like Jersey
> would make Solr all the easier to approach and contribute to for that
> pool of developers.
>
> By the way, we have used Restlet in the past and that has been a
> regrettable decision.
>
>
>
>
> Ah, yeah, that's just the context I'm missing. Anyone have a pointer
> to related discussions, or remember what made this "regrettable"? All
> the theoretical benefits in the world don't matter much if we've
> already tried something like this in the past and decided against it.
>
>
>
> It was not playing well with our security framework. The framework was not
> working well with Solr APIs
>
>
> (Unrelated - Happy Thanksgiving all!)
>
> Best,
>
> Jason
>
> On Thu, Nov 25, 2021 at 7:32 AM Noble Paul <no...@gmail.com> wrote:
>
>
> Have you gone through an API written using the @EndPoint annotation?
>
> I strongly recommend that you do
>
> On Thu, Nov 25, 2021, 11:30 PM Eric Pugh <ep...@opensourceconnections.com>
> wrote:
>
>
> I have found our V2 API code to be very impenetrable to understand. Part
> of it is how the code is intertwined with support for V1, however it’s also
> because there aren’t really resources to go look at to understand how it
> should work! Maintaining the API should be very simple work, as they just
> exist as a translation. The home grown stuff may make sense if you are a
> super knowledgable Solr developer, but if you are just a new person, it’s a
> lot harder to contribute.
>
> I was interested in the Jersey stuff because I’ve seen lots of projects
> use it very successfully, and if I want to implement something, well, there
> are lots of blogs and resources out there!
>
> Can anyone recap briefly why we dropped RESTlet? And what lessons
> learned there might apply to adopting Jersey for API support? Looking at
> https://issues.apache.org/jira/browse/SOLR-14659, it was partly
> deprecated because we were not using it to support all the API, only the
> ManagedResource ones, and https://issues.apache.org/jira/browse/SOLR-14766
> suggests that RESTlet maybe was no longer being updated? One reason why
> we spiked out Jersey was because of the broad support in the Java world!
> Looking at how much work we have to do in the V2 API world, we need a
> much broader pool of developers contributing to get there!
>
> Related, are there specific features/aspects of our annotations that
> enable things in Solr that couldn’t be done otherwise?
>
> On Nov 25, 2021, at 2:12 AM, Ishan Chattopadhyaya <
> ichattopadhyaya@gmail.com> wrote:
>
> Is there some problem with our annotations that we hope to solve using
> third party dependencies?
> By the way, we have used Restlet in the past and that has been a
> regrettable decision.
>
> On Thu, Nov 25, 2021 at 10:10 AM Jason Gerlowski <ge...@gmail.com>
> wrote:
>
>
> Solr's custom annotation framework ('@Endpoint', '@Command', etc.) has
> cropped up a few times over the past week or two. [1] [2]. Having them
> on top of mind, I've been wondering - is there a reason we use our own
> annotations here instead of something off the shelf?
>
> What we have works well enough, but anything homegrown comes with more
> maintenance burden than we'd have if we used something off the shelf.
> There are plenty of well-used, active projects out there whose whole
> purpose is facilitating the whole "annotation based API" thing
> (Jersey, Restlet, RESTEasy, etc.) - why not use one of them?
>
> Does anyone know of any technical reasons why we can't go this route?
> Or have any subjective reasons why we shouldn't? Or any context on
> why we wrote our own Endpoint, Command, JsonProperty annotations
> originally?
>
> FWIW, Eric Pugh and I spiked out a small POC recently, and got
> Jersey+Jackson working for a few simple APIs without too much trouble.
> [3] Obviously nothing production-ready there, and there's still a lot
> of open questions (e.g. how would javabin be supported?), but we both
> came away convinced that it seemed feasible, at least. Best of all,
> APIs using our current homegrown annotation framework the switchover
> seems blessedly straightforward, and it doesn't look like Jersey
> (which we chose mostly arbitrarily) bloats our dist all that much.
>
> Curious if anyone has thoughts or context on how we ended up with the
> annotation setup we use today!
>
> Best,
>
> Jason
>
> [1] https://issues.apache.org/jira/browse/SOLR-15182 (and children)
> [2]
> http://mail-archives.apache.org/mod_mbox/solr-dev/202111.mbox/%3CCABEwPvENL41Pm6%2BOmjXb6Sx5N2XjUtnbWhgKOZSrnLjWBA8tcA%40mail.gmail.com%3E
> [3]
> https://github.com/gerlowskija/solr/tree/jersey_jaxrs_jackson_solr_apis.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> <de...@solr.apache.org>
> For additional commands, e-mail: dev-help@solr.apache.org
> <de...@solr.apache.org>
>
>
> _______________________
> Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 |
> http://www.opensourceconnections.com | My Free/Busy
> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
> This e-mail and all contents, including attachments, is considered to be
> Company Confidential unless explicitly stated otherwise, regardless of
> whether attachments are marked as such.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> <de...@solr.apache.org>
> For additional commands, e-mail: dev-help@solr.apache.org
> <de...@solr.apache.org>
>
>
>
> --
> -----------------------------------------------------
> Noble Paul
>
>
>
>
> --
> -----------------------------------------------------
> Noble Paul
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> <de...@solr.apache.org>
> For additional commands, e-mail: dev-help@solr.apache.org
> <de...@solr.apache.org>
>
>
>
> --
> http://www.needhamsoftware.com (work)
> http://www.the111shift.com (play)
>
>
>
>
> --
> -----------------------------------------------------
> Noble Paul
>
>
>
>
> --
> -----------------------------------------------------
> Noble Paul
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> <de...@solr.apache.org>
> For additional commands, e-mail: dev-help@solr.apache.org
> <de...@solr.apache.org>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> <de...@solr.apache.org>
> For additional commands, e-mail: dev-help@solr.apache.org
> <de...@solr.apache.org>
>
>
>
> --
> -----------------------------------------------------
> Noble Paul
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> <de...@solr.apache.org>
> For additional commands, e-mail: dev-help@solr.apache.org
> <de...@solr.apache.org>
>
>
>
> --
> -----------------------------------------------------
> Noble Paul
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> <de...@solr.apache.org>
> For additional commands, e-mail: dev-help@solr.apache.org
> <de...@solr.apache.org>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> <de...@solr.apache.org>
> For additional commands, e-mail: dev-help@solr.apache.org
> <de...@solr.apache.org>
>
>
> _______________________
> *Eric Pugh **| *Founder & CEO | OpenSource Connections, LLC | 434.466.1467
> | http://www.opensourceconnections.com | My Free/Busy
> <http://tinyurl.com/eric-cal>
> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
> <https://www.packtpub.com/big-data-and-business-intelligence/apache-solr-enterprise-search-server-third-edition-raw>
> This e-mail and all contents, including attachments, is considered to be
> Company Confidential unless explicitly stated otherwise, regardless
> of whether attachments are marked as such.
>
>
Re: JAX-RS APIs in Solr
Posted by Eric Pugh <ep...@opensourceconnections.com>.
I was actually poking around trying to find out who actually in the Ruby world uses the wt=ruby, and all the projects that intersect with Solr seems to do wt=json ;-)
> On Dec 6, 2021, at 8:17 AM, Jan Høydahl <ja...@cominvent.com> wrote:
>
> I'm happy to see that there is agreement in a shift towards standard..
>
> Bwt - in v3, lets get rid of "wt" parameter and instead use standard HTTP Accept header to tell Solr whether you want the response as JSON or XML. Maybe we are even ready to deprecate XML, PYTHON, XSLT, Excel..
> See https://issues.apache.org/jira/browse/SOLR-10998
>
> Jan
>
>> 3. des. 2021 kl. 17:31 skrev Timothy Potter <th...@gmail.com>:
>>
>> Why do we keep talking about Restlet here? Restlet is a stale (at
>> best) project, so not sure why it keeps coming up in a discussion
>> about modernizing our API? Integration with it in Solr was introduced
>> nearly 10 years ago, time to move on and stop using that as an excuse
>> to block adoption of other API frameworks.
>>
>> Surely anyone committed to this effort, after doing even the most
>> cursory due diligence, will not land on Restlet as the way forward.
>> Jersey is a likely candidate, I'm sure there are others?
>>
>> I actually don't care all that much about which annotations we use to
>> describe our APIs (of course standards are better). What's more
>> important for me, is that we can auto-generate an OpenAPI spec for our
>> APIs. That opens up a world of automated tooling for our users to
>> interact with our APIs. Using JAX-RS / Jersey probably helps get us
>> closer to an automated way to generate an OpenAPI spec but there are
>> other ways to get there as well.
>>
>> In general, it sounds like the objections to modernizing our API fall
>> into three buckets: 1) our bad experience with Restlet, 2) performance
>> is better with our homegrown code, 3) security is too complicated with
>> another framework.
>>
>> Performance overhead added by a framework like Jersey (if there is
>> any) for Admin APIs is mostly irrelevant. You think Jersey is going to
>> be slower at mapping requests to Java method calls? Pretty sure they
>> have that part figured out at this point.
>>
>> As for Solr's security, it also needs to be replaced with a modern
>> framework like Shiro or pac4j or similar. The permissions are
>> basically impossible to understand, even for most committers. This
>> whole idea of allow by default instead of deny by default goes against
>> every security best practice in the industry. So arguing that we need
>> some homegrown API framework to support Solr's obscure authz scheme
>> seems a bit backwards to me. Once you start using standard API
>> frameworks like JAX-RS, you start getting security integration too,
>> e.g. https://github.com/pac4j/jax-rs-pac4j. Moreover, we can't keep up
>> with all the innovation around security; heck, SolrJ doesn't even
>> support OIDC yet. I'm all for scraping both homegrown frameworks and
>> starting fresh with mature open source frameworks that have all the
>> kinks worked out over many years with many projects using them.
>>
>> If you want to talk about stress, try building a non-trivial app with
>> Solr's current API's, that'll induce some stress for sure!
>>
>> Tim
>>
>>
>>
>> On Thu, Dec 2, 2021 at 10:35 PM Noble Paul <no...@gmail.com> wrote:
>>>
>>> I do not deny that a standardized framework may be helpful. I'm already suffering from the PTSD of restlet integration. The integration code was extremely complex and a security nightmare.It was added without any perf tests too. It took many years to get rid of that
>>>
>>>
>>>
>>> On Fri, Dec 3, 2021 at 1:37 AM Jason Gerlowski <ge...@gmail.com> wrote:
>>>>
>>>>> 99% of Solr users never see the java code that implements an API let alone writing a new API.
>>>>
>>>> We may be talking past each other here a bit Noble. The JAX-RS
>>>> benefits I've been trying to describe are to benefit us developers,
>>>> not Solr users. I agree that 99% of our users won't know or care
>>>> about it, but who cares. The goal is to make _our_ lives as
>>>> maintainers easier going into the future. Jan described it in a word
>>>> better than I've been struggling to: it's about "modernization".
>>>>
>>>> Best,
>>>>
>>>> Jason
>>>>
>>>> On Thu, Dec 2, 2021 at 3:34 AM Jan Høydahl <ja...@cominvent.com> wrote:
>>>>>
>>>>> Agree, Ishan.
>>>>>
>>>>> Let's welcome and cheer initiatives to modernize this old codebase.
>>>>> Diving in deep and un-tangling DispatchFilter like Gus is attempting is brave.
>>>>> And testing JAX-RS to modernize APIs and make the codebas easier to grok is also not for the faint hearted.
>>>>>
>>>>> I hope both initiatives end up as a breath of fresh air for Solr developers. And perhaps users may get their hands on auto-generated OpenAPI docs for Solr as well!
>>>>> Theoretically foundations like this may make it feasible in the future to auto-generate client code from the spec,so that full Admin API client support in e.g. Go or Rust may be achievable without hand-coding. But that is still far away :)
>>>>>
>>>>> Jan
>>>>>
>>>>> 2. des. 2021 kl. 05:52 skrev Ishan Chattopadhyaya <ic...@gmail.com>:
>>>>>
>>>>>> We are discussing this as if moving to an external framework is going to be smooth. It's not.
>>>>>
>>>>> Noble, we won't know before someone has given it a try. My hunch suggests it won't be smooth, but we can decide on that once we see a concrete solution that 1) works well for all existing APIs (per core or global) without hacks/workarounds, and 2) has no security implications, and 3) doesn't clutter our codebase, and 4) doesn't affect our runtime performance.
>>>>>
>>>>> On Thu, Dec 2, 2021 at 4:22 AM Noble Paul <no...@gmail.com> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Dec 2, 2021 at 2:46 AM Jason Gerlowski <ge...@gmail.com> wrote:
>>>>>>>
>>>>>>>> There are no known issues with the current system
>>>>>>>
>>>>>>> Maybe you're using hyperbole to emphasise a point, but let's steer
>>>>>>> this discussion away from straw-men and caricatures. I mentioned
>>>>>>> specific known issues in a previous reply as a direct response to your
>>>>>>> question about them. The incomplete support for data types
>>>>>>> (SOLR-15619 and SOLR-15796). The inflexible PermissionNameProvider
>>>>>>> integration (SOLR-15823). If you consider those "minor", fair enough,
>>>>>>> and you're welcome to that opinion. But saying that they don't exist
>>>>>>> is a mischaracterization that wastes time and muddies water.
>>>>>>
>>>>>>
>>>>>> We are discussing this as if moving to an external framework is going to be smooth. It's not. I have seen how hard the Restlet integration was and how it stood out like a sore thumb. The amount of code required to integrate Solr APIs into restlet components was 10x more than the amount of code written for the new framework and it still was incomplete . Integrating anything to Solr is not simple and it's going to be an ongoing effort to ensure that all of the functionalities of a SolrRequestHandler are exposed to the new framework. Every single component that we have today has ongoing improvements and bug fixes. Using that as an excuse to do a complete rewrite is probably unwise.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>> Solr is not a general purpose web server where people write APIs every day
>>>>>>>
>>>>>>> Totally agree - 99% of the time someone opens an API file in Solr,
>>>>>>> they're reading, not writing. So if JAX-RS didn't help that case, it
>>>>>>> wouldn't be worth the effort of switching. But IMO it helps that
>>>>>>> "read" case a lot.
>>>>>>
>>>>>>
>>>>>> 99% of Solr users never see the java code that implements an API let alone writing a new API. They wouldn't care if an API is implemented using a Servlet API, a custom framework or something else. They only care about the input and output. New end points are rare.
>>>>>>>
>>>>>>>
>>>>>>> JAX-RS defined APIs expose inputs as strongly-typed parameters
>>>>>>> (compare to the weakly-typed SolrQueryRequest/SolrParams used today).
>>>>>>> JAX-RS enumerates every input to the API in one place. JAX-RS APIs
>>>>>>> can return any serializable type as a response (compare to the
>>>>>>> near-omnipresent NamedList today).
>>>>>>>
>>>>>>> (The annotation framework's PayloadObj class gets us a bit of this
>>>>>>> benefit, but not all of it and only on POST requests afaik.)
>>>>>>>
>>>>>>> This is all nice stuff at API-write time, but where it really helps is
>>>>>>> in making the code clearer and simpler whenever someone goes to read
>>>>>>> it later. No wondering what parameters an API actually takes. No
>>>>>>> puzzling out what a bunch of NamedList operations spread throughout
>>>>>>> API execution actually produce in the end. etc.
>>>>>>>
>>>>>>>> The only reason why anyone would need to use this framework is to write a custom request handler.
>>>>>>>
>>>>>>> ...except for the majority of people who come in contact with the
>>>>>>> framework by reading an existing Solr API? This ties into your point
>>>>>>> above about APIs being written infrequently but read all the time.
>>>>>>> Improving readability is the big gain here, and I think JAX-RS offers
>>>>>>> real improvements in that regard.
>>>>>>
>>>>>>
>>>>>> We had no framework for the last 15+ years and V1 API (the only actually used API) still uses no frameworks. Even the annotation framework is not used much even though readability is a lot better in the annotation based system. Readability has not been an overriding concern for V1 APIs for sure
>>>>>>
>>>>>>> Also, to clarify, I'm not asserting that no one configures custom
>>>>>>> requestHandlers. I was wondering aloud how common it was, as an open
>>>>>>> question.
>>>>>>>
>>>>>> Your point IS right. IRL, most of our users do not write request handlers . They write search components , query parsers and other plugins. Because we are a search engine and not a web server. I have never seen a user complaining about the V1 API of writing a request handler. They may actually complain about how hard it is to deploy one. Our users are much smarter than we give them credit for.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>> We did not fork Jackson. We are using Jackson itself.
>>>>>>>
>>>>>>> Ah, maybe you're right, "fork" is probably an unfair word. I looked
>>>>>>> for the serialization code, and I'll admit there is less than I
>>>>>>> thought involved here and I misunderstood how it works. So that's my
>>>>>>> bad. The JSON serialization/deserialization discussion is prob a
>>>>>>> digression from the core contention about whether JAX-RS would be
>>>>>>> helpful. I was curious about your earlier response and couldn't
>>>>>>> resist following up here but I should've done so elsewhere to avoid
>>>>>>> the distraction.
>>>>>>>
>>>>>>> Best,
>>>>>>>
>>>>>>>
>>>>>>> Jason
>>>>>>>
>>>>>>> On Tue, Nov 30, 2021 at 4:12 PM Noble Paul <no...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> The annotation framework is just a single class and it is made to work well with Solr instead of changing Solr to suit the needs of some external framework. There are no known issues with the current system and there are unknown issues that you'll face introducing a new framework.
>>>>>>>>
>>>>>>>> Solr is not a general purpose web server where people write APIs every day. It's a finished product where people use the publicly available APIs . We should focus our efforts on making our APIs work well .
>>>>>>>>
>>>>>>>> On Wed, Dec 1, 2021, 2:14 AM Jason Gerlowski <ge...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> We need to make an API work on a per core basis and cores can come up and go down randomly
>>>>>>>>>
>>>>>>>>> Ah, yeah. I'll admit I was overlooking some of the wrinkles around
>>>>>>>>> custom request handlers etc. That is a problem for switching to some
>>>>>>>>> off-the-shelf framework for sure. It may still be possible, but it's
>>>>>>>>> definitely a big hurdle. I wonder how commonly users still take
>>>>>>>>> advantage of the custom requestHandler stuff these days? I haven't
>>>>>>>>> run across many customers who used them extensively, but maybe that's
>>>>>>>>> self-selecting in some way?
>>>>>>>>
>>>>>>>>
>>>>>>>> The only reason why anyone would need to use this framework is to write a custom request handler. You're also saying nobody writes custom request handlers. So, what problem are you trying to solve if you think nobody uses them?
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>>> If SolrJ has a Jackson dependency, it can have a conflict (different versions) with the Jackson used by the client app
>>>>>>>>>
>>>>>>>>> SolrJ has a handful of dependencies - is there a reason we take this
>>>>>>>>> approach with Jackson, but not with any of SolrJ's other deps?
>>>>>>>>> (apache-httpcomponents, apache-commons, zookeeper-client, etc.)
>>>>>>>>>
>>>>>>>>> Granted, I get that we want SolrJ to be as painless as possible for
>>>>>>>>> its users, and jar-conflicts are a part of that and really suck. But
>>>>>>>>> there are good tools out there for handling jar conflicts, and
>>>>>>>>> reimplementing chunks of library functionality just to avoid a
>>>>>>>>> gradle-dep strikes me as unsustainable from a maintenance/community
>>>>>>>>> perspective. Not to mention that forking/mimicking Jackson code
>>>>>>>>> deprives our users of any efficiency/security improvements that
>>>>>>>>> Jackson might get tomorrow or next week or next year.
>>>>>>>>
>>>>>>>>
>>>>>>>> We did not fork Jackson. We are using Jackson itself. Jackson supports custom annotations. There was a huge discussion why we should do it this way.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Unless there's some reason specific to Jackson I guess I just don't
>>>>>>>>> get it. But if I'm the minority opinion on that, fair enough I guess.
>>>>>>>>>
>>>>>>>>> On Tue, Nov 30, 2021 at 5:05 AM Noble Paul <no...@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>> The @JsonProperty annotations are added because it's a dependency in SolrJ as well
>>>>>>>>>> If SolrJ has a Jackson dependency, it can have a conflict (different versions) with the Jackson used by the client app.
>>>>>>>>>>
>>>>>>>>>> On Tue, Nov 30, 2021, 5:21 PM Noble Paul <no...@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Also keep in mind that the same endpoint can be accessed with a core name and a collection name prefixes.
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Nov 30, 2021 at 3:55 PM Noble Paul <no...@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> True Gus, Almost every framework works outside of SolrDispatchFilter+ HttpSolrCall. A lot of our initializations occur there.
>>>>>>>>>>>>
>>>>>>>>>>>> We need to make an API work on a per core basis and cores can come up and go down randomly. So we need to register these endpoints on a core.
>>>>>>>>>>>>
>>>>>>>>>>>> I'm not sure if any framework can achieve the same.
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Nov 30, 2021 at 5:39 AM Gus Heck <gu...@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> IIRC last time I looked restlet had the unsavory property of existing outside of the SolrDispatchFilter, unlike everything else which made for special cases because several things that probably ought to be their own siervlet filters are glommed into SolrDispatchFilter, like security, tracing and MDC setup/teardown per request. Restlet wouldn't be so bad if one could just wrap such filters around it too...
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Nov 29, 2021 at 9:42 AM Jason Gerlowski <ge...@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> These are minor improvements compared to a full rewrite of the entire framework
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> If you think data type support is minor, fair enough. But to clarify
>>>>>>>>>>>>>> I'm not suggesting a rewrite - I'm suggesting using something that
>>>>>>>>>>>>>> already exists off the shelf. Jersey (e.g.) itself provides the
>>>>>>>>>>>>>> framework - there would be no "rewrite".
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> re: past restlet use
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> It was not playing well with our security framework. The framework was not working well with Solr APIs
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Ah, very interesting! Security isn't something Eric or I tackled in
>>>>>>>>>>>>>> our little spike branch, but it's definitely a concern. Do you
>>>>>>>>>>>>>> remember the specific concerns? Or recall where any of the discussion
>>>>>>>>>>>>>> around this happened?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Without the context of that past discussion, it seems like the
>>>>>>>>>>>>>> "PermissionNameProvider" interface could be implemented just as well
>>>>>>>>>>>>>> by a class with (e.g.) Jersey annotations as one with our own custom
>>>>>>>>>>>>>> annotations. Certainly there'd need to be some
>>>>>>>>>>>>>> RuleBasedAuthorizationPlugin changes or other integration code, but
>>>>>>>>>>>>>> nothing that feels insurmountable.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Maybe I can try spiking it out soon and find the issues myself, but
>>>>>>>>>>>>>> it'd be much easier if someone happens to remember and can save me the
>>>>>>>>>>>>>> trouble :-p
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Jason
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Nov 26, 2021 at 12:05 AM Noble Paul <no...@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The Annotations framework was written after playing with other frameworks. There were many shortcomings which were hard to overcome.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The best example is a per collection API . How do you register an endpoint for a collection/core ?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Nov 26, 2021 at 3:42 PM Noble Paul <no...@gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, Nov 26, 2021 at 1:03 AM Jason Gerlowski <ge...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Is there some problem with our annotations that we hope to solve using third party dependencies?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I guess so yeah. Third-party deps are just fuller, more robust
>>>>>>>>>>>>>>>>> solutions, whereas our annotations still need support added now and
>>>>>>>>>>>>>>>>> then for even primitive data types like "long" (see SOLR-15619).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> These are minor improvements compared to a full rewrite of the entire framework
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Every JIRA spent doing basic stuff like that is time away from
>>>>>>>>>>>>>>>>> improving Solr in some other way.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> So there are feature-gap/capabilities arguments for moving to a
>>>>>>>>>>>>>>>>> third-party dep, sure. But, even if our annotations did everything
>>>>>>>>>>>>>>>>> Jersey+Jackson do today, I think switching would still be worth it.
>>>>>>>>>>>>>>>>> Every LOC in our code base brings along with it some maintenance cost:
>>>>>>>>>>>>>>>>> it might have bugs, needs tested, takes time for new contributors to
>>>>>>>>>>>>>>>>> "grok", etc. Using off-the-shelf here would nuke a whole bunch of
>>>>>>>>>>>>>>>>> that. If off-the-shelf is available for some given functionality, we
>>>>>>>>>>>>>>>>> should need a compelling reason to NOT use it.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Lastly, I think there's an "approachability" argument for using
>>>>>>>>>>>>>>>>> off-the-shelf. Thousands of developers out there are familiar with
>>>>>>>>>>>>>>>>> (e.g.) Jersey, compared to maybe 15 or 20 (in the world) familiar with
>>>>>>>>>>>>>>>>> Solr's custom annotations. Using a well-known technology like Jersey
>>>>>>>>>>>>>>>>> would make Solr all the easier to approach and contribute to for that
>>>>>>>>>>>>>>>>> pool of developers.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> By the way, we have used Restlet in the past and that has been a regrettable decision.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Ah, yeah, that's just the context I'm missing. Anyone have a pointer
>>>>>>>>>>>>>>>>> to related discussions, or remember what made this "regrettable"? All
>>>>>>>>>>>>>>>>> the theoretical benefits in the world don't matter much if we've
>>>>>>>>>>>>>>>>> already tried something like this in the past and decided against it.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> It was not playing well with our security framework. The framework was not working well with Solr APIs
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> (Unrelated - Happy Thanksgiving all!)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Jason
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Thu, Nov 25, 2021 at 7:32 AM Noble Paul <no...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Have you gone through an API written using the @EndPoint annotation?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I strongly recommend that you do
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Thu, Nov 25, 2021, 11:30 PM Eric Pugh <ep...@opensourceconnections.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I have found our V2 API code to be very impenetrable to understand. Part of it is how the code is intertwined with support for V1, however it’s also because there aren’t really resources to go look at to understand how it should work! Maintaining the API should be very simple work, as they just exist as a translation. The home grown stuff may make sense if you are a super knowledgable Solr developer, but if you are just a new person, it’s a lot harder to contribute.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I was interested in the Jersey stuff because I’ve seen lots of projects use it very successfully, and if I want to implement something, well, there are lots of blogs and resources out there!
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Can anyone recap briefly why we dropped RESTlet? And what lessons learned there might apply to adopting Jersey for API support? Looking at https://issues.apache.org/jira/browse/SOLR-14659, it was partly deprecated because we were not using it to support all the API, only the ManagedResource ones, and https://issues.apache.org/jira/browse/SOLR-14766 suggests that RESTlet maybe was no longer being updated? One reason why we spiked out Jersey was because of the broad support in the Java world! Looking at how much work we have to do in the V2 API world, we need a much broader pool of developers contributing to get there!
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Related, are there specific features/aspects of our annotations that enable things in Solr that couldn’t be done otherwise?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Nov 25, 2021, at 2:12 AM, Ishan Chattopadhyaya <ic...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Is there some problem with our annotations that we hope to solve using third party dependencies?
>>>>>>>>>>>>>>>>>>> By the way, we have used Restlet in the past and that has been a regrettable decision.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Thu, Nov 25, 2021 at 10:10 AM Jason Gerlowski <ge...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Solr's custom annotation framework ('@Endpoint', '@Command', etc.) has
>>>>>>>>>>>>>>>>>>>> cropped up a few times over the past week or two. [1] [2]. Having them
>>>>>>>>>>>>>>>>>>>> on top of mind, I've been wondering - is there a reason we use our own
>>>>>>>>>>>>>>>>>>>> annotations here instead of something off the shelf?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> What we have works well enough, but anything homegrown comes with more
>>>>>>>>>>>>>>>>>>>> maintenance burden than we'd have if we used something off the shelf.
>>>>>>>>>>>>>>>>>>>> There are plenty of well-used, active projects out there whose whole
>>>>>>>>>>>>>>>>>>>> purpose is facilitating the whole "annotation based API" thing
>>>>>>>>>>>>>>>>>>>> (Jersey, Restlet, RESTEasy, etc.) - why not use one of them?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Does anyone know of any technical reasons why we can't go this route?
>>>>>>>>>>>>>>>>>>>> Or have any subjective reasons why we shouldn't? Or any context on
>>>>>>>>>>>>>>>>>>>> why we wrote our own Endpoint, Command, JsonProperty annotations
>>>>>>>>>>>>>>>>>>>> originally?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> FWIW, Eric Pugh and I spiked out a small POC recently, and got
>>>>>>>>>>>>>>>>>>>> Jersey+Jackson working for a few simple APIs without too much trouble.
>>>>>>>>>>>>>>>>>>>> [3] Obviously nothing production-ready there, and there's still a lot
>>>>>>>>>>>>>>>>>>>> of open questions (e.g. how would javabin be supported?), but we both
>>>>>>>>>>>>>>>>>>>> came away convinced that it seemed feasible, at least. Best of all,
>>>>>>>>>>>>>>>>>>>> APIs using our current homegrown annotation framework the switchover
>>>>>>>>>>>>>>>>>>>> seems blessedly straightforward, and it doesn't look like Jersey
>>>>>>>>>>>>>>>>>>>> (which we chose mostly arbitrarily) bloats our dist all that much.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Curious if anyone has thoughts or context on how we ended up with the
>>>>>>>>>>>>>>>>>>>> annotation setup we use today!
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Jason
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/SOLR-15182 (and children)
>>>>>>>>>>>>>>>>>>>> [2] http://mail-archives.apache.org/mod_mbox/solr-dev/202111.mbox/%3CCABEwPvENL41Pm6%2BOmjXb6Sx5N2XjUtnbWhgKOZSrnLjWBA8tcA%40mail.gmail.com%3E
>>>>>>>>>>>>>>>>>>>> [3] https://github.com/gerlowskija/solr/tree/jersey_jaxrs_jackson_solr_apis.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>>>>>>>>>>>>>>>>>>>> For additional commands, e-mail: dev-help@solr.apache.org
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> _______________________
>>>>>>>>>>>>>>>>>>> Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | http://www.opensourceconnections.com | My Free/Busy
>>>>>>>>>>>>>>>>>>> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
>>>>>>>>>>>>>>>>>>> This e-mail and all contents, including attachments, is considered to be Company Confidential unless explicitly stated otherwise, regardless of whether attachments are marked as such.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>>>>>>>>>>>>>>>>> For additional commands, e-mail: dev-help@solr.apache.org
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> -----------------------------------------------------
>>>>>>>>>>>>>>>> Noble Paul
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> -----------------------------------------------------
>>>>>>>>>>>>>>> Noble Paul
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>>>>>>>>>>>>>> For additional commands, e-mail: dev-help@solr.apache.org
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> http://www.needhamsoftware.com (work)
>>>>>>>>>>>>> http://www.the111shift.com (play)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> -----------------------------------------------------
>>>>>>>>>>>> Noble Paul
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> -----------------------------------------------------
>>>>>>>>>>> Noble Paul
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>>>>>>>>> For additional commands, e-mail: dev-help@solr.apache.org
>>>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>>>>>>> For additional commands, e-mail: dev-help@solr.apache.org
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> -----------------------------------------------------
>>>>>> Noble Paul
>>>>>
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>>>> For additional commands, e-mail: dev-help@solr.apache.org
>>>>
>>>
>>>
>>> --
>>> -----------------------------------------------------
>>> Noble Paul
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>> For additional commands, e-mail: dev-help@solr.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org
>
_______________________
Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | http://www.opensourceconnections.com <http://www.opensourceconnections.com/> | My Free/Busy <http://tinyurl.com/eric-cal>
Co-Author: Apache Solr Enterprise Search Server, 3rd Ed <https://www.packtpub.com/big-data-and-business-intelligence/apache-solr-enterprise-search-server-third-edition-raw>
This e-mail and all contents, including attachments, is considered to be Company Confidential unless explicitly stated otherwise, regardless of whether attachments are marked as such.
Re: JAX-RS APIs in Solr
Posted by Jan Høydahl <ja...@cominvent.com>.
I'm happy to see that there is agreement in a shift towards standard..
Bwt - in v3, lets get rid of "wt" parameter and instead use standard HTTP Accept header to tell Solr whether you want the response as JSON or XML. Maybe we are even ready to deprecate XML, PYTHON, XSLT, Excel..
See https://issues.apache.org/jira/browse/SOLR-10998
Jan
> 3. des. 2021 kl. 17:31 skrev Timothy Potter <th...@gmail.com>:
>
> Why do we keep talking about Restlet here? Restlet is a stale (at
> best) project, so not sure why it keeps coming up in a discussion
> about modernizing our API? Integration with it in Solr was introduced
> nearly 10 years ago, time to move on and stop using that as an excuse
> to block adoption of other API frameworks.
>
> Surely anyone committed to this effort, after doing even the most
> cursory due diligence, will not land on Restlet as the way forward.
> Jersey is a likely candidate, I'm sure there are others?
>
> I actually don't care all that much about which annotations we use to
> describe our APIs (of course standards are better). What's more
> important for me, is that we can auto-generate an OpenAPI spec for our
> APIs. That opens up a world of automated tooling for our users to
> interact with our APIs. Using JAX-RS / Jersey probably helps get us
> closer to an automated way to generate an OpenAPI spec but there are
> other ways to get there as well.
>
> In general, it sounds like the objections to modernizing our API fall
> into three buckets: 1) our bad experience with Restlet, 2) performance
> is better with our homegrown code, 3) security is too complicated with
> another framework.
>
> Performance overhead added by a framework like Jersey (if there is
> any) for Admin APIs is mostly irrelevant. You think Jersey is going to
> be slower at mapping requests to Java method calls? Pretty sure they
> have that part figured out at this point.
>
> As for Solr's security, it also needs to be replaced with a modern
> framework like Shiro or pac4j or similar. The permissions are
> basically impossible to understand, even for most committers. This
> whole idea of allow by default instead of deny by default goes against
> every security best practice in the industry. So arguing that we need
> some homegrown API framework to support Solr's obscure authz scheme
> seems a bit backwards to me. Once you start using standard API
> frameworks like JAX-RS, you start getting security integration too,
> e.g. https://github.com/pac4j/jax-rs-pac4j. Moreover, we can't keep up
> with all the innovation around security; heck, SolrJ doesn't even
> support OIDC yet. I'm all for scraping both homegrown frameworks and
> starting fresh with mature open source frameworks that have all the
> kinks worked out over many years with many projects using them.
>
> If you want to talk about stress, try building a non-trivial app with
> Solr's current API's, that'll induce some stress for sure!
>
> Tim
>
>
>
> On Thu, Dec 2, 2021 at 10:35 PM Noble Paul <no...@gmail.com> wrote:
>>
>> I do not deny that a standardized framework may be helpful. I'm already suffering from the PTSD of restlet integration. The integration code was extremely complex and a security nightmare.It was added without any perf tests too. It took many years to get rid of that
>>
>>
>>
>> On Fri, Dec 3, 2021 at 1:37 AM Jason Gerlowski <ge...@gmail.com> wrote:
>>>
>>>> 99% of Solr users never see the java code that implements an API let alone writing a new API.
>>>
>>> We may be talking past each other here a bit Noble. The JAX-RS
>>> benefits I've been trying to describe are to benefit us developers,
>>> not Solr users. I agree that 99% of our users won't know or care
>>> about it, but who cares. The goal is to make _our_ lives as
>>> maintainers easier going into the future. Jan described it in a word
>>> better than I've been struggling to: it's about "modernization".
>>>
>>> Best,
>>>
>>> Jason
>>>
>>> On Thu, Dec 2, 2021 at 3:34 AM Jan Høydahl <ja...@cominvent.com> wrote:
>>>>
>>>> Agree, Ishan.
>>>>
>>>> Let's welcome and cheer initiatives to modernize this old codebase.
>>>> Diving in deep and un-tangling DispatchFilter like Gus is attempting is brave.
>>>> And testing JAX-RS to modernize APIs and make the codebas easier to grok is also not for the faint hearted.
>>>>
>>>> I hope both initiatives end up as a breath of fresh air for Solr developers. And perhaps users may get their hands on auto-generated OpenAPI docs for Solr as well!
>>>> Theoretically foundations like this may make it feasible in the future to auto-generate client code from the spec,so that full Admin API client support in e.g. Go or Rust may be achievable without hand-coding. But that is still far away :)
>>>>
>>>> Jan
>>>>
>>>> 2. des. 2021 kl. 05:52 skrev Ishan Chattopadhyaya <ic...@gmail.com>:
>>>>
>>>>> We are discussing this as if moving to an external framework is going to be smooth. It's not.
>>>>
>>>> Noble, we won't know before someone has given it a try. My hunch suggests it won't be smooth, but we can decide on that once we see a concrete solution that 1) works well for all existing APIs (per core or global) without hacks/workarounds, and 2) has no security implications, and 3) doesn't clutter our codebase, and 4) doesn't affect our runtime performance.
>>>>
>>>> On Thu, Dec 2, 2021 at 4:22 AM Noble Paul <no...@gmail.com> wrote:
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Dec 2, 2021 at 2:46 AM Jason Gerlowski <ge...@gmail.com> wrote:
>>>>>>
>>>>>>> There are no known issues with the current system
>>>>>>
>>>>>> Maybe you're using hyperbole to emphasise a point, but let's steer
>>>>>> this discussion away from straw-men and caricatures. I mentioned
>>>>>> specific known issues in a previous reply as a direct response to your
>>>>>> question about them. The incomplete support for data types
>>>>>> (SOLR-15619 and SOLR-15796). The inflexible PermissionNameProvider
>>>>>> integration (SOLR-15823). If you consider those "minor", fair enough,
>>>>>> and you're welcome to that opinion. But saying that they don't exist
>>>>>> is a mischaracterization that wastes time and muddies water.
>>>>>
>>>>>
>>>>> We are discussing this as if moving to an external framework is going to be smooth. It's not. I have seen how hard the Restlet integration was and how it stood out like a sore thumb. The amount of code required to integrate Solr APIs into restlet components was 10x more than the amount of code written for the new framework and it still was incomplete . Integrating anything to Solr is not simple and it's going to be an ongoing effort to ensure that all of the functionalities of a SolrRequestHandler are exposed to the new framework. Every single component that we have today has ongoing improvements and bug fixes. Using that as an excuse to do a complete rewrite is probably unwise.
>>>>>
>>>>>
>>>>>>
>>>>>>> Solr is not a general purpose web server where people write APIs every day
>>>>>>
>>>>>> Totally agree - 99% of the time someone opens an API file in Solr,
>>>>>> they're reading, not writing. So if JAX-RS didn't help that case, it
>>>>>> wouldn't be worth the effort of switching. But IMO it helps that
>>>>>> "read" case a lot.
>>>>>
>>>>>
>>>>> 99% of Solr users never see the java code that implements an API let alone writing a new API. They wouldn't care if an API is implemented using a Servlet API, a custom framework or something else. They only care about the input and output. New end points are rare.
>>>>>>
>>>>>>
>>>>>> JAX-RS defined APIs expose inputs as strongly-typed parameters
>>>>>> (compare to the weakly-typed SolrQueryRequest/SolrParams used today).
>>>>>> JAX-RS enumerates every input to the API in one place. JAX-RS APIs
>>>>>> can return any serializable type as a response (compare to the
>>>>>> near-omnipresent NamedList today).
>>>>>>
>>>>>> (The annotation framework's PayloadObj class gets us a bit of this
>>>>>> benefit, but not all of it and only on POST requests afaik.)
>>>>>>
>>>>>> This is all nice stuff at API-write time, but where it really helps is
>>>>>> in making the code clearer and simpler whenever someone goes to read
>>>>>> it later. No wondering what parameters an API actually takes. No
>>>>>> puzzling out what a bunch of NamedList operations spread throughout
>>>>>> API execution actually produce in the end. etc.
>>>>>>
>>>>>>> The only reason why anyone would need to use this framework is to write a custom request handler.
>>>>>>
>>>>>> ...except for the majority of people who come in contact with the
>>>>>> framework by reading an existing Solr API? This ties into your point
>>>>>> above about APIs being written infrequently but read all the time.
>>>>>> Improving readability is the big gain here, and I think JAX-RS offers
>>>>>> real improvements in that regard.
>>>>>
>>>>>
>>>>> We had no framework for the last 15+ years and V1 API (the only actually used API) still uses no frameworks. Even the annotation framework is not used much even though readability is a lot better in the annotation based system. Readability has not been an overriding concern for V1 APIs for sure
>>>>>
>>>>>> Also, to clarify, I'm not asserting that no one configures custom
>>>>>> requestHandlers. I was wondering aloud how common it was, as an open
>>>>>> question.
>>>>>>
>>>>> Your point IS right. IRL, most of our users do not write request handlers . They write search components , query parsers and other plugins. Because we are a search engine and not a web server. I have never seen a user complaining about the V1 API of writing a request handler. They may actually complain about how hard it is to deploy one. Our users are much smarter than we give them credit for.
>>>>>
>>>>>
>>>>>>
>>>>>>> We did not fork Jackson. We are using Jackson itself.
>>>>>>
>>>>>> Ah, maybe you're right, "fork" is probably an unfair word. I looked
>>>>>> for the serialization code, and I'll admit there is less than I
>>>>>> thought involved here and I misunderstood how it works. So that's my
>>>>>> bad. The JSON serialization/deserialization discussion is prob a
>>>>>> digression from the core contention about whether JAX-RS would be
>>>>>> helpful. I was curious about your earlier response and couldn't
>>>>>> resist following up here but I should've done so elsewhere to avoid
>>>>>> the distraction.
>>>>>>
>>>>>> Best,
>>>>>>
>>>>>>
>>>>>> Jason
>>>>>>
>>>>>> On Tue, Nov 30, 2021 at 4:12 PM Noble Paul <no...@gmail.com> wrote:
>>>>>>>
>>>>>>> The annotation framework is just a single class and it is made to work well with Solr instead of changing Solr to suit the needs of some external framework. There are no known issues with the current system and there are unknown issues that you'll face introducing a new framework.
>>>>>>>
>>>>>>> Solr is not a general purpose web server where people write APIs every day. It's a finished product where people use the publicly available APIs . We should focus our efforts on making our APIs work well .
>>>>>>>
>>>>>>> On Wed, Dec 1, 2021, 2:14 AM Jason Gerlowski <ge...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> We need to make an API work on a per core basis and cores can come up and go down randomly
>>>>>>>>
>>>>>>>> Ah, yeah. I'll admit I was overlooking some of the wrinkles around
>>>>>>>> custom request handlers etc. That is a problem for switching to some
>>>>>>>> off-the-shelf framework for sure. It may still be possible, but it's
>>>>>>>> definitely a big hurdle. I wonder how commonly users still take
>>>>>>>> advantage of the custom requestHandler stuff these days? I haven't
>>>>>>>> run across many customers who used them extensively, but maybe that's
>>>>>>>> self-selecting in some way?
>>>>>>>
>>>>>>>
>>>>>>> The only reason why anyone would need to use this framework is to write a custom request handler. You're also saying nobody writes custom request handlers. So, what problem are you trying to solve if you think nobody uses them?
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>> If SolrJ has a Jackson dependency, it can have a conflict (different versions) with the Jackson used by the client app
>>>>>>>>
>>>>>>>> SolrJ has a handful of dependencies - is there a reason we take this
>>>>>>>> approach with Jackson, but not with any of SolrJ's other deps?
>>>>>>>> (apache-httpcomponents, apache-commons, zookeeper-client, etc.)
>>>>>>>>
>>>>>>>> Granted, I get that we want SolrJ to be as painless as possible for
>>>>>>>> its users, and jar-conflicts are a part of that and really suck. But
>>>>>>>> there are good tools out there for handling jar conflicts, and
>>>>>>>> reimplementing chunks of library functionality just to avoid a
>>>>>>>> gradle-dep strikes me as unsustainable from a maintenance/community
>>>>>>>> perspective. Not to mention that forking/mimicking Jackson code
>>>>>>>> deprives our users of any efficiency/security improvements that
>>>>>>>> Jackson might get tomorrow or next week or next year.
>>>>>>>
>>>>>>>
>>>>>>> We did not fork Jackson. We are using Jackson itself. Jackson supports custom annotations. There was a huge discussion why we should do it this way.
>>>>>>>>
>>>>>>>>
>>>>>>>> Unless there's some reason specific to Jackson I guess I just don't
>>>>>>>> get it. But if I'm the minority opinion on that, fair enough I guess.
>>>>>>>>
>>>>>>>> On Tue, Nov 30, 2021 at 5:05 AM Noble Paul <no...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>> The @JsonProperty annotations are added because it's a dependency in SolrJ as well
>>>>>>>>> If SolrJ has a Jackson dependency, it can have a conflict (different versions) with the Jackson used by the client app.
>>>>>>>>>
>>>>>>>>> On Tue, Nov 30, 2021, 5:21 PM Noble Paul <no...@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>> Also keep in mind that the same endpoint can be accessed with a core name and a collection name prefixes.
>>>>>>>>>>
>>>>>>>>>> On Tue, Nov 30, 2021 at 3:55 PM Noble Paul <no...@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>> True Gus, Almost every framework works outside of SolrDispatchFilter+ HttpSolrCall. A lot of our initializations occur there.
>>>>>>>>>>>
>>>>>>>>>>> We need to make an API work on a per core basis and cores can come up and go down randomly. So we need to register these endpoints on a core.
>>>>>>>>>>>
>>>>>>>>>>> I'm not sure if any framework can achieve the same.
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Nov 30, 2021 at 5:39 AM Gus Heck <gu...@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> IIRC last time I looked restlet had the unsavory property of existing outside of the SolrDispatchFilter, unlike everything else which made for special cases because several things that probably ought to be their own siervlet filters are glommed into SolrDispatchFilter, like security, tracing and MDC setup/teardown per request. Restlet wouldn't be so bad if one could just wrap such filters around it too...
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Nov 29, 2021 at 9:42 AM Jason Gerlowski <ge...@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> These are minor improvements compared to a full rewrite of the entire framework
>>>>>>>>>>>>>
>>>>>>>>>>>>> If you think data type support is minor, fair enough. But to clarify
>>>>>>>>>>>>> I'm not suggesting a rewrite - I'm suggesting using something that
>>>>>>>>>>>>> already exists off the shelf. Jersey (e.g.) itself provides the
>>>>>>>>>>>>> framework - there would be no "rewrite".
>>>>>>>>>>>>>
>>>>>>>>>>>>> re: past restlet use
>>>>>>>>>>>>>
>>>>>>>>>>>>>> It was not playing well with our security framework. The framework was not working well with Solr APIs
>>>>>>>>>>>>>
>>>>>>>>>>>>> Ah, very interesting! Security isn't something Eric or I tackled in
>>>>>>>>>>>>> our little spike branch, but it's definitely a concern. Do you
>>>>>>>>>>>>> remember the specific concerns? Or recall where any of the discussion
>>>>>>>>>>>>> around this happened?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Without the context of that past discussion, it seems like the
>>>>>>>>>>>>> "PermissionNameProvider" interface could be implemented just as well
>>>>>>>>>>>>> by a class with (e.g.) Jersey annotations as one with our own custom
>>>>>>>>>>>>> annotations. Certainly there'd need to be some
>>>>>>>>>>>>> RuleBasedAuthorizationPlugin changes or other integration code, but
>>>>>>>>>>>>> nothing that feels insurmountable.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Maybe I can try spiking it out soon and find the issues myself, but
>>>>>>>>>>>>> it'd be much easier if someone happens to remember and can save me the
>>>>>>>>>>>>> trouble :-p
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Jason
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Nov 26, 2021 at 12:05 AM Noble Paul <no...@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The Annotations framework was written after playing with other frameworks. There were many shortcomings which were hard to overcome.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The best example is a per collection API . How do you register an endpoint for a collection/core ?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Nov 26, 2021 at 3:42 PM Noble Paul <no...@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Nov 26, 2021 at 1:03 AM Jason Gerlowski <ge...@gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Is there some problem with our annotations that we hope to solve using third party dependencies?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I guess so yeah. Third-party deps are just fuller, more robust
>>>>>>>>>>>>>>>> solutions, whereas our annotations still need support added now and
>>>>>>>>>>>>>>>> then for even primitive data types like "long" (see SOLR-15619).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> These are minor improvements compared to a full rewrite of the entire framework
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Every JIRA spent doing basic stuff like that is time away from
>>>>>>>>>>>>>>>> improving Solr in some other way.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> So there are feature-gap/capabilities arguments for moving to a
>>>>>>>>>>>>>>>> third-party dep, sure. But, even if our annotations did everything
>>>>>>>>>>>>>>>> Jersey+Jackson do today, I think switching would still be worth it.
>>>>>>>>>>>>>>>> Every LOC in our code base brings along with it some maintenance cost:
>>>>>>>>>>>>>>>> it might have bugs, needs tested, takes time for new contributors to
>>>>>>>>>>>>>>>> "grok", etc. Using off-the-shelf here would nuke a whole bunch of
>>>>>>>>>>>>>>>> that. If off-the-shelf is available for some given functionality, we
>>>>>>>>>>>>>>>> should need a compelling reason to NOT use it.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Lastly, I think there's an "approachability" argument for using
>>>>>>>>>>>>>>>> off-the-shelf. Thousands of developers out there are familiar with
>>>>>>>>>>>>>>>> (e.g.) Jersey, compared to maybe 15 or 20 (in the world) familiar with
>>>>>>>>>>>>>>>> Solr's custom annotations. Using a well-known technology like Jersey
>>>>>>>>>>>>>>>> would make Solr all the easier to approach and contribute to for that
>>>>>>>>>>>>>>>> pool of developers.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> By the way, we have used Restlet in the past and that has been a regrettable decision.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Ah, yeah, that's just the context I'm missing. Anyone have a pointer
>>>>>>>>>>>>>>>> to related discussions, or remember what made this "regrettable"? All
>>>>>>>>>>>>>>>> the theoretical benefits in the world don't matter much if we've
>>>>>>>>>>>>>>>> already tried something like this in the past and decided against it.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> It was not playing well with our security framework. The framework was not working well with Solr APIs
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> (Unrelated - Happy Thanksgiving all!)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Jason
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Thu, Nov 25, 2021 at 7:32 AM Noble Paul <no...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Have you gone through an API written using the @EndPoint annotation?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I strongly recommend that you do
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Thu, Nov 25, 2021, 11:30 PM Eric Pugh <ep...@opensourceconnections.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I have found our V2 API code to be very impenetrable to understand. Part of it is how the code is intertwined with support for V1, however it’s also because there aren’t really resources to go look at to understand how it should work! Maintaining the API should be very simple work, as they just exist as a translation. The home grown stuff may make sense if you are a super knowledgable Solr developer, but if you are just a new person, it’s a lot harder to contribute.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I was interested in the Jersey stuff because I’ve seen lots of projects use it very successfully, and if I want to implement something, well, there are lots of blogs and resources out there!
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Can anyone recap briefly why we dropped RESTlet? And what lessons learned there might apply to adopting Jersey for API support? Looking at https://issues.apache.org/jira/browse/SOLR-14659, it was partly deprecated because we were not using it to support all the API, only the ManagedResource ones, and https://issues.apache.org/jira/browse/SOLR-14766 suggests that RESTlet maybe was no longer being updated? One reason why we spiked out Jersey was because of the broad support in the Java world! Looking at how much work we have to do in the V2 API world, we need a much broader pool of developers contributing to get there!
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Related, are there specific features/aspects of our annotations that enable things in Solr that couldn’t be done otherwise?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Nov 25, 2021, at 2:12 AM, Ishan Chattopadhyaya <ic...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Is there some problem with our annotations that we hope to solve using third party dependencies?
>>>>>>>>>>>>>>>>>> By the way, we have used Restlet in the past and that has been a regrettable decision.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Thu, Nov 25, 2021 at 10:10 AM Jason Gerlowski <ge...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Solr's custom annotation framework ('@Endpoint', '@Command', etc.) has
>>>>>>>>>>>>>>>>>>> cropped up a few times over the past week or two. [1] [2]. Having them
>>>>>>>>>>>>>>>>>>> on top of mind, I've been wondering - is there a reason we use our own
>>>>>>>>>>>>>>>>>>> annotations here instead of something off the shelf?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> What we have works well enough, but anything homegrown comes with more
>>>>>>>>>>>>>>>>>>> maintenance burden than we'd have if we used something off the shelf.
>>>>>>>>>>>>>>>>>>> There are plenty of well-used, active projects out there whose whole
>>>>>>>>>>>>>>>>>>> purpose is facilitating the whole "annotation based API" thing
>>>>>>>>>>>>>>>>>>> (Jersey, Restlet, RESTEasy, etc.) - why not use one of them?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Does anyone know of any technical reasons why we can't go this route?
>>>>>>>>>>>>>>>>>>> Or have any subjective reasons why we shouldn't? Or any context on
>>>>>>>>>>>>>>>>>>> why we wrote our own Endpoint, Command, JsonProperty annotations
>>>>>>>>>>>>>>>>>>> originally?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> FWIW, Eric Pugh and I spiked out a small POC recently, and got
>>>>>>>>>>>>>>>>>>> Jersey+Jackson working for a few simple APIs without too much trouble.
>>>>>>>>>>>>>>>>>>> [3] Obviously nothing production-ready there, and there's still a lot
>>>>>>>>>>>>>>>>>>> of open questions (e.g. how would javabin be supported?), but we both
>>>>>>>>>>>>>>>>>>> came away convinced that it seemed feasible, at least. Best of all,
>>>>>>>>>>>>>>>>>>> APIs using our current homegrown annotation framework the switchover
>>>>>>>>>>>>>>>>>>> seems blessedly straightforward, and it doesn't look like Jersey
>>>>>>>>>>>>>>>>>>> (which we chose mostly arbitrarily) bloats our dist all that much.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Curious if anyone has thoughts or context on how we ended up with the
>>>>>>>>>>>>>>>>>>> annotation setup we use today!
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Jason
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/SOLR-15182 (and children)
>>>>>>>>>>>>>>>>>>> [2] http://mail-archives.apache.org/mod_mbox/solr-dev/202111.mbox/%3CCABEwPvENL41Pm6%2BOmjXb6Sx5N2XjUtnbWhgKOZSrnLjWBA8tcA%40mail.gmail.com%3E
>>>>>>>>>>>>>>>>>>> [3] https://github.com/gerlowskija/solr/tree/jersey_jaxrs_jackson_solr_apis.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>>>>>>>>>>>>>>>>>>> For additional commands, e-mail: dev-help@solr.apache.org
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> _______________________
>>>>>>>>>>>>>>>>>> Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | http://www.opensourceconnections.com | My Free/Busy
>>>>>>>>>>>>>>>>>> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
>>>>>>>>>>>>>>>>>> This e-mail and all contents, including attachments, is considered to be Company Confidential unless explicitly stated otherwise, regardless of whether attachments are marked as such.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>>>>>>>>>>>>>>>> For additional commands, e-mail: dev-help@solr.apache.org
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> -----------------------------------------------------
>>>>>>>>>>>>>>> Noble Paul
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> -----------------------------------------------------
>>>>>>>>>>>>>> Noble Paul
>>>>>>>>>>>>>
>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>>>>>>>>>>>>> For additional commands, e-mail: dev-help@solr.apache.org
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> http://www.needhamsoftware.com (work)
>>>>>>>>>>>> http://www.the111shift.com (play)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> -----------------------------------------------------
>>>>>>>>>>> Noble Paul
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> -----------------------------------------------------
>>>>>>>>>> Noble Paul
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>>>>>>>> For additional commands, e-mail: dev-help@solr.apache.org
>>>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>>>>>> For additional commands, e-mail: dev-help@solr.apache.org
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> -----------------------------------------------------
>>>>> Noble Paul
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>>> For additional commands, e-mail: dev-help@solr.apache.org
>>>
>>
>>
>> --
>> -----------------------------------------------------
>> Noble Paul
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: JAX-RS APIs in Solr
Posted by Gus Heck <gu...@gmail.com>.
+1 to pretty much everything Tim said, especially not re-inventing wheels.
Let's use the light weight drop forged aluminum ones that come with steel
belted tubeless radials from the store, not the ones we carve out of our
own block of rubber attached to a bit of steel plate with assorted screws.
We might have to build a hub with a standard bolt pattern to do it but
probably worth it in the long run.
On Fri, Dec 3, 2021 at 11:32 AM Timothy Potter <th...@gmail.com> wrote:
> Why do we keep talking about Restlet here? Restlet is a stale (at
> best) project, so not sure why it keeps coming up in a discussion
> about modernizing our API? Integration with it in Solr was introduced
> nearly 10 years ago, time to move on and stop using that as an excuse
> to block adoption of other API frameworks.
>
> Surely anyone committed to this effort, after doing even the most
> cursory due diligence, will not land on Restlet as the way forward.
> Jersey is a likely candidate, I'm sure there are others?
>
> I actually don't care all that much about which annotations we use to
> describe our APIs (of course standards are better). What's more
> important for me, is that we can auto-generate an OpenAPI spec for our
> APIs. That opens up a world of automated tooling for our users to
> interact with our APIs. Using JAX-RS / Jersey probably helps get us
> closer to an automated way to generate an OpenAPI spec but there are
> other ways to get there as well.
>
> In general, it sounds like the objections to modernizing our API fall
> into three buckets: 1) our bad experience with Restlet, 2) performance
> is better with our homegrown code, 3) security is too complicated with
> another framework.
>
> Performance overhead added by a framework like Jersey (if there is
> any) for Admin APIs is mostly irrelevant. You think Jersey is going to
> be slower at mapping requests to Java method calls? Pretty sure they
> have that part figured out at this point.
>
> As for Solr's security, it also needs to be replaced with a modern
> framework like Shiro or pac4j or similar. The permissions are
> basically impossible to understand, even for most committers. This
> whole idea of allow by default instead of deny by default goes against
> every security best practice in the industry. So arguing that we need
> some homegrown API framework to support Solr's obscure authz scheme
> seems a bit backwards to me. Once you start using standard API
> frameworks like JAX-RS, you start getting security integration too,
> e.g. https://github.com/pac4j/jax-rs-pac4j. Moreover, we can't keep up
> with all the innovation around security; heck, SolrJ doesn't even
> support OIDC yet. I'm all for scraping both homegrown frameworks and
> starting fresh with mature open source frameworks that have all the
> kinks worked out over many years with many projects using them.
>
> If you want to talk about stress, try building a non-trivial app with
> Solr's current API's, that'll induce some stress for sure!
>
> Tim
>
>
>
> On Thu, Dec 2, 2021 at 10:35 PM Noble Paul <no...@gmail.com> wrote:
> >
> > I do not deny that a standardized framework may be helpful. I'm already
> suffering from the PTSD of restlet integration. The integration code was
> extremely complex and a security nightmare.It was added without any perf
> tests too. It took many years to get rid of that
> >
> >
> >
> > On Fri, Dec 3, 2021 at 1:37 AM Jason Gerlowski <ge...@gmail.com>
> wrote:
> >>
> >> > 99% of Solr users never see the java code that implements an API let
> alone writing a new API.
> >>
> >> We may be talking past each other here a bit Noble. The JAX-RS
> >> benefits I've been trying to describe are to benefit us developers,
> >> not Solr users. I agree that 99% of our users won't know or care
> >> about it, but who cares. The goal is to make _our_ lives as
> >> maintainers easier going into the future. Jan described it in a word
> >> better than I've been struggling to: it's about "modernization".
> >>
> >> Best,
> >>
> >> Jason
> >>
> >> On Thu, Dec 2, 2021 at 3:34 AM Jan Høydahl <ja...@cominvent.com>
> wrote:
> >> >
> >> > Agree, Ishan.
> >> >
> >> > Let's welcome and cheer initiatives to modernize this old codebase.
> >> > Diving in deep and un-tangling DispatchFilter like Gus is attempting
> is brave.
> >> > And testing JAX-RS to modernize APIs and make the codebas easier to
> grok is also not for the faint hearted.
> >> >
> >> > I hope both initiatives end up as a breath of fresh air for Solr
> developers. And perhaps users may get their hands on auto-generated
> OpenAPI docs for Solr as well!
> >> > Theoretically foundations like this may make it feasible in the
> future to auto-generate client code from the spec,so that full Admin API
> client support in e.g. Go or Rust may be achievable without hand-coding.
> But that is still far away :)
> >> >
> >> > Jan
> >> >
> >> > 2. des. 2021 kl. 05:52 skrev Ishan Chattopadhyaya <
> ichattopadhyaya@gmail.com>:
> >> >
> >> > > We are discussing this as if moving to an external framework is
> going to be smooth. It's not.
> >> >
> >> > Noble, we won't know before someone has given it a try. My hunch
> suggests it won't be smooth, but we can decide on that once we see a
> concrete solution that 1) works well for all existing APIs (per core or
> global) without hacks/workarounds, and 2) has no security implications, and
> 3) doesn't clutter our codebase, and 4) doesn't affect our runtime
> performance.
> >> >
> >> > On Thu, Dec 2, 2021 at 4:22 AM Noble Paul <no...@gmail.com>
> wrote:
> >> >>
> >> >>
> >> >>
> >> >> On Thu, Dec 2, 2021 at 2:46 AM Jason Gerlowski <
> gerlowskija@gmail.com> wrote:
> >> >>>
> >> >>> > There are no known issues with the current system
> >> >>>
> >> >>> Maybe you're using hyperbole to emphasise a point, but let's steer
> >> >>> this discussion away from straw-men and caricatures. I mentioned
> >> >>> specific known issues in a previous reply as a direct response to
> your
> >> >>> question about them. The incomplete support for data types
> >> >>> (SOLR-15619 and SOLR-15796). The inflexible PermissionNameProvider
> >> >>> integration (SOLR-15823). If you consider those "minor", fair
> enough,
> >> >>> and you're welcome to that opinion. But saying that they don't
> exist
> >> >>> is a mischaracterization that wastes time and muddies water.
> >> >>
> >> >>
> >> >> We are discussing this as if moving to an external framework is
> going to be smooth. It's not. I have seen how hard the Restlet integration
> was and how it stood out like a sore thumb. The amount of code required to
> integrate Solr APIs into restlet components was 10x more than the amount of
> code written for the new framework and it still was incomplete .
> Integrating anything to Solr is not simple and it's going to be an ongoing
> effort to ensure that all of the functionalities of a SolrRequestHandler
> are exposed to the new framework. Every single component that we have today
> has ongoing improvements and bug fixes. Using that as an excuse to do a
> complete rewrite is probably unwise.
> >> >>
> >> >>
> >> >>>
> >> >>> > Solr is not a general purpose web server where people write APIs
> every day
> >> >>>
> >> >>> Totally agree - 99% of the time someone opens an API file in Solr,
> >> >>> they're reading, not writing. So if JAX-RS didn't help that case,
> it
> >> >>> wouldn't be worth the effort of switching. But IMO it helps that
> >> >>> "read" case a lot.
> >> >>
> >> >>
> >> >> 99% of Solr users never see the java code that implements an API let
> alone writing a new API. They wouldn't care if an API is implemented using
> a Servlet API, a custom framework or something else. They only care about
> the input and output. New end points are rare.
> >> >>>
> >> >>>
> >> >>> JAX-RS defined APIs expose inputs as strongly-typed parameters
> >> >>> (compare to the weakly-typed SolrQueryRequest/SolrParams used
> today).
> >> >>> JAX-RS enumerates every input to the API in one place. JAX-RS APIs
> >> >>> can return any serializable type as a response (compare to the
> >> >>> near-omnipresent NamedList today).
> >> >>>
> >> >>> (The annotation framework's PayloadObj class gets us a bit of this
> >> >>> benefit, but not all of it and only on POST requests afaik.)
> >> >>>
> >> >>> This is all nice stuff at API-write time, but where it really helps
> is
> >> >>> in making the code clearer and simpler whenever someone goes to read
> >> >>> it later. No wondering what parameters an API actually takes. No
> >> >>> puzzling out what a bunch of NamedList operations spread throughout
> >> >>> API execution actually produce in the end. etc.
> >> >>>
> >> >>> > The only reason why anyone would need to use this framework is to
> write a custom request handler.
> >> >>>
> >> >>> ...except for the majority of people who come in contact with the
> >> >>> framework by reading an existing Solr API? This ties into your
> point
> >> >>> above about APIs being written infrequently but read all the time.
> >> >>> Improving readability is the big gain here, and I think JAX-RS
> offers
> >> >>> real improvements in that regard.
> >> >>
> >> >>
> >> >> We had no framework for the last 15+ years and V1 API (the only
> actually used API) still uses no frameworks. Even the annotation framework
> is not used much even though readability is a lot better in the annotation
> based system. Readability has not been an overriding concern for V1 APIs
> for sure
> >> >>
> >> >>> Also, to clarify, I'm not asserting that no one configures custom
> >> >>> requestHandlers. I was wondering aloud how common it was, as an
> open
> >> >>> question.
> >> >>>
> >> >> Your point IS right. IRL, most of our users do not write request
> handlers . They write search components , query parsers and other plugins.
> Because we are a search engine and not a web server. I have never seen a
> user complaining about the V1 API of writing a request handler. They may
> actually complain about how hard it is to deploy one. Our users are much
> smarter than we give them credit for.
> >> >>
> >> >>
> >> >>>
> >> >>> > We did not fork Jackson. We are using Jackson itself.
> >> >>>
> >> >>> Ah, maybe you're right, "fork" is probably an unfair word. I looked
> >> >>> for the serialization code, and I'll admit there is less than I
> >> >>> thought involved here and I misunderstood how it works. So that's
> my
> >> >>> bad. The JSON serialization/deserialization discussion is prob a
> >> >>> digression from the core contention about whether JAX-RS would be
> >> >>> helpful. I was curious about your earlier response and couldn't
> >> >>> resist following up here but I should've done so elsewhere to avoid
> >> >>> the distraction.
> >> >>>
> >> >>> Best,
> >> >>>
> >> >>>
> >> >>> Jason
> >> >>>
> >> >>> On Tue, Nov 30, 2021 at 4:12 PM Noble Paul <no...@gmail.com>
> wrote:
> >> >>> >
> >> >>> > The annotation framework is just a single class and it is made to
> work well with Solr instead of changing Solr to suit the needs of some
> external framework. There are no known issues with the current system and
> there are unknown issues that you'll face introducing a new framework.
> >> >>> >
> >> >>> > Solr is not a general purpose web server where people write APIs
> every day. It's a finished product where people use the publicly available
> APIs . We should focus our efforts on making our APIs work well .
> >> >>> >
> >> >>> > On Wed, Dec 1, 2021, 2:14 AM Jason Gerlowski <
> gerlowskija@gmail.com> wrote:
> >> >>> >>
> >> >>> >> > We need to make an API work on a per core basis and cores can
> come up and go down randomly
> >> >>> >>
> >> >>> >> Ah, yeah. I'll admit I was overlooking some of the wrinkles
> around
> >> >>> >> custom request handlers etc. That is a problem for switching to
> some
> >> >>> >> off-the-shelf framework for sure. It may still be possible, but
> it's
> >> >>> >> definitely a big hurdle. I wonder how commonly users still take
> >> >>> >> advantage of the custom requestHandler stuff these days? I
> haven't
> >> >>> >> run across many customers who used them extensively, but maybe
> that's
> >> >>> >> self-selecting in some way?
> >> >>> >
> >> >>> >
> >> >>> > The only reason why anyone would need to use this framework is to
> write a custom request handler. You're also saying nobody writes custom
> request handlers. So, what problem are you trying to solve if you think
> nobody uses them?
> >> >>> >
> >> >>> >
> >> >>> >>
> >> >>> >> > If SolrJ has a Jackson dependency, it can have a conflict
> (different versions) with the Jackson used by the client app
> >> >>> >>
> >> >>> >> SolrJ has a handful of dependencies - is there a reason we take
> this
> >> >>> >> approach with Jackson, but not with any of SolrJ's other deps?
> >> >>> >> (apache-httpcomponents, apache-commons, zookeeper-client, etc.)
> >> >>> >>
> >> >>> >> Granted, I get that we want SolrJ to be as painless as possible
> for
> >> >>> >> its users, and jar-conflicts are a part of that and really
> suck. But
> >> >>> >> there are good tools out there for handling jar conflicts, and
> >> >>> >> reimplementing chunks of library functionality just to avoid a
> >> >>> >> gradle-dep strikes me as unsustainable from a
> maintenance/community
> >> >>> >> perspective. Not to mention that forking/mimicking Jackson code
> >> >>> >> deprives our users of any efficiency/security improvements that
> >> >>> >> Jackson might get tomorrow or next week or next year.
> >> >>> >
> >> >>> >
> >> >>> > We did not fork Jackson. We are using Jackson itself. Jackson
> supports custom annotations. There was a huge discussion why we should do
> it this way.
> >> >>> >>
> >> >>> >>
> >> >>> >> Unless there's some reason specific to Jackson I guess I just
> don't
> >> >>> >> get it. But if I'm the minority opinion on that, fair enough I
> guess.
> >> >>> >>
> >> >>> >> On Tue, Nov 30, 2021 at 5:05 AM Noble Paul <no...@gmail.com>
> wrote:
> >> >>> >> >
> >> >>> >> > The @JsonProperty annotations are added because it's a
> dependency in SolrJ as well
> >> >>> >> > If SolrJ has a Jackson dependency, it can have a conflict
> (different versions) with the Jackson used by the client app.
> >> >>> >> >
> >> >>> >> > On Tue, Nov 30, 2021, 5:21 PM Noble Paul <no...@gmail.com>
> wrote:
> >> >>> >> >>
> >> >>> >> >> Also keep in mind that the same endpoint can be accessed with
> a core name and a collection name prefixes.
> >> >>> >> >>
> >> >>> >> >> On Tue, Nov 30, 2021 at 3:55 PM Noble Paul <
> noble.paul@gmail.com> wrote:
> >> >>> >> >>>
> >> >>> >> >>> True Gus, Almost every framework works outside of
> SolrDispatchFilter+ HttpSolrCall. A lot of our initializations occur there.
> >> >>> >> >>>
> >> >>> >> >>> We need to make an API work on a per core basis and cores
> can come up and go down randomly. So we need to register these endpoints on
> a core.
> >> >>> >> >>>
> >> >>> >> >>> I'm not sure if any framework can achieve the same.
> >> >>> >> >>>
> >> >>> >> >>> On Tue, Nov 30, 2021 at 5:39 AM Gus Heck <gu...@gmail.com>
> wrote:
> >> >>> >> >>>>
> >> >>> >> >>>> IIRC last time I looked restlet had the unsavory property
> of existing outside of the SolrDispatchFilter, unlike everything else which
> made for special cases because several things that probably ought to be
> their own siervlet filters are glommed into SolrDispatchFilter, like
> security, tracing and MDC setup/teardown per request. Restlet wouldn't be
> so bad if one could just wrap such filters around it too...
> >> >>> >> >>>>
> >> >>> >> >>>> On Mon, Nov 29, 2021 at 9:42 AM Jason Gerlowski <
> gerlowskija@gmail.com> wrote:
> >> >>> >> >>>>>
> >> >>> >> >>>>> > These are minor improvements compared to a full rewrite
> of the entire framework
> >> >>> >> >>>>>
> >> >>> >> >>>>> If you think data type support is minor, fair enough. But
> to clarify
> >> >>> >> >>>>> I'm not suggesting a rewrite - I'm suggesting using
> something that
> >> >>> >> >>>>> already exists off the shelf. Jersey (e.g.) itself
> provides the
> >> >>> >> >>>>> framework - there would be no "rewrite".
> >> >>> >> >>>>>
> >> >>> >> >>>>> re: past restlet use
> >> >>> >> >>>>>
> >> >>> >> >>>>> > It was not playing well with our security framework. The
> framework was not working well with Solr APIs
> >> >>> >> >>>>>
> >> >>> >> >>>>> Ah, very interesting! Security isn't something Eric or I
> tackled in
> >> >>> >> >>>>> our little spike branch, but it's definitely a concern.
> Do you
> >> >>> >> >>>>> remember the specific concerns? Or recall where any of
> the discussion
> >> >>> >> >>>>> around this happened?
> >> >>> >> >>>>>
> >> >>> >> >>>>> Without the context of that past discussion, it seems like
> the
> >> >>> >> >>>>> "PermissionNameProvider" interface could be implemented
> just as well
> >> >>> >> >>>>> by a class with (e.g.) Jersey annotations as one with our
> own custom
> >> >>> >> >>>>> annotations. Certainly there'd need to be some
> >> >>> >> >>>>> RuleBasedAuthorizationPlugin changes or other integration
> code, but
> >> >>> >> >>>>> nothing that feels insurmountable.
> >> >>> >> >>>>>
> >> >>> >> >>>>> Maybe I can try spiking it out soon and find the issues
> myself, but
> >> >>> >> >>>>> it'd be much easier if someone happens to remember and can
> save me the
> >> >>> >> >>>>> trouble :-p
> >> >>> >> >>>>>
> >> >>> >> >>>>> Best,
> >> >>> >> >>>>>
> >> >>> >> >>>>> Jason
> >> >>> >> >>>>>
> >> >>> >> >>>>> On Fri, Nov 26, 2021 at 12:05 AM Noble Paul <
> noble.paul@gmail.com> wrote:
> >> >>> >> >>>>> >
> >> >>> >> >>>>> > The Annotations framework was written after playing with
> other frameworks. There were many shortcomings which were hard to overcome.
> >> >>> >> >>>>> >
> >> >>> >> >>>>> > The best example is a per collection API . How do you
> register an endpoint for a collection/core ?
> >> >>> >> >>>>> >
> >> >>> >> >>>>> > On Fri, Nov 26, 2021 at 3:42 PM Noble Paul <
> noble.paul@gmail.com> wrote:
> >> >>> >> >>>>> >>
> >> >>> >> >>>>> >>
> >> >>> >> >>>>> >>
> >> >>> >> >>>>> >> On Fri, Nov 26, 2021 at 1:03 AM Jason Gerlowski <
> gerlowskija@gmail.com> wrote:
> >> >>> >> >>>>> >>>
> >> >>> >> >>>>> >>> > Is there some problem with our annotations that we
> hope to solve using third party dependencies?
> >> >>> >> >>>>> >>>
> >> >>> >> >>>>> >>> I guess so yeah. Third-party deps are just fuller,
> more robust
> >> >>> >> >>>>> >>> solutions, whereas our annotations still need support
> added now and
> >> >>> >> >>>>> >>> then for even primitive data types like "long" (see
> SOLR-15619).
> >> >>> >> >>>>> >>
> >> >>> >> >>>>> >> These are minor improvements compared to a full rewrite
> of the entire framework
> >> >>> >> >>>>> >>
> >> >>> >> >>>>> >>
> >> >>> >> >>>>> >>>
> >> >>> >> >>>>> >>> Every JIRA spent doing basic stuff like that is time
> away from
> >> >>> >> >>>>> >>> improving Solr in some other way.
> >> >>> >> >>>>> >>>
> >> >>> >> >>>>> >>> So there are feature-gap/capabilities arguments for
> moving to a
> >> >>> >> >>>>> >>> third-party dep, sure. But, even if our annotations
> did everything
> >> >>> >> >>>>> >>> Jersey+Jackson do today, I think switching would still
> be worth it.
> >> >>> >> >>>>> >>> Every LOC in our code base brings along with it some
> maintenance cost:
> >> >>> >> >>>>> >>> it might have bugs, needs tested, takes time for new
> contributors to
> >> >>> >> >>>>> >>> "grok", etc. Using off-the-shelf here would nuke a
> whole bunch of
> >> >>> >> >>>>> >>> that. If off-the-shelf is available for some given
> functionality, we
> >> >>> >> >>>>> >>> should need a compelling reason to NOT use it.
> >> >>> >> >>>>> >>>
> >> >>> >> >>>>> >>> Lastly, I think there's an "approachability" argument
> for using
> >> >>> >> >>>>> >>> off-the-shelf. Thousands of developers out there are
> familiar with
> >> >>> >> >>>>> >>> (e.g.) Jersey, compared to maybe 15 or 20 (in the
> world) familiar with
> >> >>> >> >>>>> >>> Solr's custom annotations. Using a well-known
> technology like Jersey
> >> >>> >> >>>>> >>> would make Solr all the easier to approach and
> contribute to for that
> >> >>> >> >>>>> >>> pool of developers.
> >> >>> >> >>>>> >>>
> >> >>> >> >>>>> >>> > By the way, we have used Restlet in the past and
> that has been a regrettable decision.
> >> >>> >> >>>>> >>
> >> >>> >> >>>>> >>
> >> >>> >> >>>>> >>>
> >> >>> >> >>>>> >>> Ah, yeah, that's just the context I'm missing. Anyone
> have a pointer
> >> >>> >> >>>>> >>> to related discussions, or remember what made this
> "regrettable"? All
> >> >>> >> >>>>> >>> the theoretical benefits in the world don't matter
> much if we've
> >> >>> >> >>>>> >>> already tried something like this in the past and
> decided against it.
> >> >>> >> >>>>> >>
> >> >>> >> >>>>> >>
> >> >>> >> >>>>> >> It was not playing well with our security framework.
> The framework was not working well with Solr APIs
> >> >>> >> >>>>> >>
> >> >>> >> >>>>> >>>
> >> >>> >> >>>>> >>> (Unrelated - Happy Thanksgiving all!)
> >> >>> >> >>>>> >>>
> >> >>> >> >>>>> >>> Best,
> >> >>> >> >>>>> >>>
> >> >>> >> >>>>> >>> Jason
> >> >>> >> >>>>> >>>
> >> >>> >> >>>>> >>> On Thu, Nov 25, 2021 at 7:32 AM Noble Paul <
> noble.paul@gmail.com> wrote:
> >> >>> >> >>>>> >>> >
> >> >>> >> >>>>> >>> > Have you gone through an API written using the
> @EndPoint annotation?
> >> >>> >> >>>>> >>> >
> >> >>> >> >>>>> >>> > I strongly recommend that you do
> >> >>> >> >>>>> >>> >
> >> >>> >> >>>>> >>> > On Thu, Nov 25, 2021, 11:30 PM Eric Pugh <
> epugh@opensourceconnections.com> wrote:
> >> >>> >> >>>>> >>> >>
> >> >>> >> >>>>> >>> >> I have found our V2 API code to be very
> impenetrable to understand. Part of it is how the code is intertwined
> with support for V1, however it’s also because there aren’t really
> resources to go look at to understand how it should work! Maintaining the
> API should be very simple work, as they just exist as a translation. The
> home grown stuff may make sense if you are a super knowledgable Solr
> developer, but if you are just a new person, it’s a lot harder to
> contribute.
> >> >>> >> >>>>> >>> >>
> >> >>> >> >>>>> >>> >> I was interested in the Jersey stuff because I’ve
> seen lots of projects use it very successfully, and if I want to implement
> something, well, there are lots of blogs and resources out there!
> >> >>> >> >>>>> >>> >>
> >> >>> >> >>>>> >>> >> Can anyone recap briefly why we dropped RESTlet?
> And what lessons learned there might apply to adopting Jersey for API
> support? Looking at https://issues.apache.org/jira/browse/SOLR-14659,
> it was partly deprecated because we were not using it to support all the
> API, only the ManagedResource ones, and
> https://issues.apache.org/jira/browse/SOLR-14766 suggests that RESTlet
> maybe was no longer being updated? One reason why we spiked out Jersey
> was because of the broad support in the Java world! Looking at how much
> work we have to do in the V2 API world, we need a much broader pool of
> developers contributing to get there!
> >> >>> >> >>>>> >>> >>
> >> >>> >> >>>>> >>> >> Related, are there specific features/aspects of our
> annotations that enable things in Solr that couldn’t be done otherwise?
> >> >>> >> >>>>> >>> >>
> >> >>> >> >>>>> >>> >> On Nov 25, 2021, at 2:12 AM, Ishan Chattopadhyaya <
> ichattopadhyaya@gmail.com> wrote:
> >> >>> >> >>>>> >>> >>
> >> >>> >> >>>>> >>> >> Is there some problem with our annotations that we
> hope to solve using third party dependencies?
> >> >>> >> >>>>> >>> >> By the way, we have used Restlet in the past and
> that has been a regrettable decision.
> >> >>> >> >>>>> >>> >>
> >> >>> >> >>>>> >>> >> On Thu, Nov 25, 2021 at 10:10 AM Jason Gerlowski <
> gerlowskija@gmail.com> wrote:
> >> >>> >> >>>>> >>> >>>
> >> >>> >> >>>>> >>> >>> Solr's custom annotation framework ('@Endpoint',
> '@Command', etc.) has
> >> >>> >> >>>>> >>> >>> cropped up a few times over the past week or two.
> [1] [2]. Having them
> >> >>> >> >>>>> >>> >>> on top of mind, I've been wondering - is there a
> reason we use our own
> >> >>> >> >>>>> >>> >>> annotations here instead of something off the
> shelf?
> >> >>> >> >>>>> >>> >>>
> >> >>> >> >>>>> >>> >>> What we have works well enough, but anything
> homegrown comes with more
> >> >>> >> >>>>> >>> >>> maintenance burden than we'd have if we used
> something off the shelf.
> >> >>> >> >>>>> >>> >>> There are plenty of well-used, active projects out
> there whose whole
> >> >>> >> >>>>> >>> >>> purpose is facilitating the whole "annotation
> based API" thing
> >> >>> >> >>>>> >>> >>> (Jersey, Restlet, RESTEasy, etc.) - why not use
> one of them?
> >> >>> >> >>>>> >>> >>>
> >> >>> >> >>>>> >>> >>> Does anyone know of any technical reasons why we
> can't go this route?
> >> >>> >> >>>>> >>> >>> Or have any subjective reasons why we shouldn't?
> Or any context on
> >> >>> >> >>>>> >>> >>> why we wrote our own Endpoint, Command,
> JsonProperty annotations
> >> >>> >> >>>>> >>> >>> originally?
> >> >>> >> >>>>> >>> >>>
> >> >>> >> >>>>> >>> >>> FWIW, Eric Pugh and I spiked out a small POC
> recently, and got
> >> >>> >> >>>>> >>> >>> Jersey+Jackson working for a few simple APIs
> without too much trouble.
> >> >>> >> >>>>> >>> >>> [3] Obviously nothing production-ready there, and
> there's still a lot
> >> >>> >> >>>>> >>> >>> of open questions (e.g. how would javabin be
> supported?), but we both
> >> >>> >> >>>>> >>> >>> came away convinced that it seemed feasible, at
> least. Best of all,
> >> >>> >> >>>>> >>> >>> APIs using our current homegrown annotation
> framework the switchover
> >> >>> >> >>>>> >>> >>> seems blessedly straightforward, and it doesn't
> look like Jersey
> >> >>> >> >>>>> >>> >>> (which we chose mostly arbitrarily) bloats our
> dist all that much.
> >> >>> >> >>>>> >>> >>>
> >> >>> >> >>>>> >>> >>> Curious if anyone has thoughts or context on how
> we ended up with the
> >> >>> >> >>>>> >>> >>> annotation setup we use today!
> >> >>> >> >>>>> >>> >>>
> >> >>> >> >>>>> >>> >>> Best,
> >> >>> >> >>>>> >>> >>>
> >> >>> >> >>>>> >>> >>> Jason
> >> >>> >> >>>>> >>> >>>
> >> >>> >> >>>>> >>> >>> [1]
> https://issues.apache.org/jira/browse/SOLR-15182 (and children)
> >> >>> >> >>>>> >>> >>> [2]
> http://mail-archives.apache.org/mod_mbox/solr-dev/202111.mbox/%3CCABEwPvENL41Pm6%2BOmjXb6Sx5N2XjUtnbWhgKOZSrnLjWBA8tcA%40mail.gmail.com%3E
> >> >>> >> >>>>> >>> >>> [3]
> https://github.com/gerlowskija/solr/tree/jersey_jaxrs_jackson_solr_apis.
> >> >>> >> >>>>> >>> >>>
> >> >>> >> >>>>> >>> >>>
> ---------------------------------------------------------------------
> >> >>> >> >>>>> >>> >>> To unsubscribe, e-mail:
> dev-unsubscribe@solr.apache.org
> >> >>> >> >>>>> >>> >>> For additional commands, e-mail:
> dev-help@solr.apache.org
> >> >>> >> >>>>> >>> >>>
> >> >>> >> >>>>> >>> >>
> >> >>> >> >>>>> >>> >> _______________________
> >> >>> >> >>>>> >>> >> Eric Pugh | Founder & CEO | OpenSource Connections,
> LLC | 434.466.1467 | http://www.opensourceconnections.com | My Free/Busy
> >> >>> >> >>>>> >>> >> Co-Author: Apache Solr Enterprise Search Server,
> 3rd Ed
> >> >>> >> >>>>> >>> >> This e-mail and all contents, including
> attachments, is considered to be Company Confidential unless explicitly
> stated otherwise, regardless of whether attachments are marked as such.
> >> >>> >> >>>>> >>> >>
> >> >>> >> >>>>> >>>
> >> >>> >> >>>>> >>>
> ---------------------------------------------------------------------
> >> >>> >> >>>>> >>> To unsubscribe, e-mail:
> dev-unsubscribe@solr.apache.org
> >> >>> >> >>>>> >>> For additional commands, e-mail:
> dev-help@solr.apache.org
> >> >>> >> >>>>> >>>
> >> >>> >> >>>>> >>
> >> >>> >> >>>>> >>
> >> >>> >> >>>>> >> --
> >> >>> >> >>>>> >> -----------------------------------------------------
> >> >>> >> >>>>> >> Noble Paul
> >> >>> >> >>>>> >
> >> >>> >> >>>>> >
> >> >>> >> >>>>> >
> >> >>> >> >>>>> > --
> >> >>> >> >>>>> > -----------------------------------------------------
> >> >>> >> >>>>> > Noble Paul
> >> >>> >> >>>>>
> >> >>> >> >>>>>
> ---------------------------------------------------------------------
> >> >>> >> >>>>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> >> >>> >> >>>>> For additional commands, e-mail: dev-help@solr.apache.org
> >> >>> >> >>>>>
> >> >>> >> >>>>
> >> >>> >> >>>>
> >> >>> >> >>>> --
> >> >>> >> >>>> http://www.needhamsoftware.com (work)
> >> >>> >> >>>> http://www.the111shift.com (play)
> >> >>> >> >>>
> >> >>> >> >>>
> >> >>> >> >>>
> >> >>> >> >>> --
> >> >>> >> >>> -----------------------------------------------------
> >> >>> >> >>> Noble Paul
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >> --
> >> >>> >> >> -----------------------------------------------------
> >> >>> >> >> Noble Paul
> >> >>> >>
> >> >>> >>
> ---------------------------------------------------------------------
> >> >>> >> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> >> >>> >> For additional commands, e-mail: dev-help@solr.apache.org
> >> >>> >>
> >> >>>
> >> >>>
> ---------------------------------------------------------------------
> >> >>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> >> >>> For additional commands, e-mail: dev-help@solr.apache.org
> >> >>>
> >> >>
> >> >>
> >> >> --
> >> >> -----------------------------------------------------
> >> >> Noble Paul
> >> >
> >> >
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> >> For additional commands, e-mail: dev-help@solr.apache.org
> >>
> >
> >
> > --
> > -----------------------------------------------------
> > Noble Paul
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org
>
>
--
http://www.needhamsoftware.com (work)
http://www.the111shift.com (play)
Re: JAX-RS APIs in Solr
Posted by Timothy Potter <th...@gmail.com>.
Why do we keep talking about Restlet here? Restlet is a stale (at
best) project, so not sure why it keeps coming up in a discussion
about modernizing our API? Integration with it in Solr was introduced
nearly 10 years ago, time to move on and stop using that as an excuse
to block adoption of other API frameworks.
Surely anyone committed to this effort, after doing even the most
cursory due diligence, will not land on Restlet as the way forward.
Jersey is a likely candidate, I'm sure there are others?
I actually don't care all that much about which annotations we use to
describe our APIs (of course standards are better). What's more
important for me, is that we can auto-generate an OpenAPI spec for our
APIs. That opens up a world of automated tooling for our users to
interact with our APIs. Using JAX-RS / Jersey probably helps get us
closer to an automated way to generate an OpenAPI spec but there are
other ways to get there as well.
In general, it sounds like the objections to modernizing our API fall
into three buckets: 1) our bad experience with Restlet, 2) performance
is better with our homegrown code, 3) security is too complicated with
another framework.
Performance overhead added by a framework like Jersey (if there is
any) for Admin APIs is mostly irrelevant. You think Jersey is going to
be slower at mapping requests to Java method calls? Pretty sure they
have that part figured out at this point.
As for Solr's security, it also needs to be replaced with a modern
framework like Shiro or pac4j or similar. The permissions are
basically impossible to understand, even for most committers. This
whole idea of allow by default instead of deny by default goes against
every security best practice in the industry. So arguing that we need
some homegrown API framework to support Solr's obscure authz scheme
seems a bit backwards to me. Once you start using standard API
frameworks like JAX-RS, you start getting security integration too,
e.g. https://github.com/pac4j/jax-rs-pac4j. Moreover, we can't keep up
with all the innovation around security; heck, SolrJ doesn't even
support OIDC yet. I'm all for scraping both homegrown frameworks and
starting fresh with mature open source frameworks that have all the
kinks worked out over many years with many projects using them.
If you want to talk about stress, try building a non-trivial app with
Solr's current API's, that'll induce some stress for sure!
Tim
On Thu, Dec 2, 2021 at 10:35 PM Noble Paul <no...@gmail.com> wrote:
>
> I do not deny that a standardized framework may be helpful. I'm already suffering from the PTSD of restlet integration. The integration code was extremely complex and a security nightmare.It was added without any perf tests too. It took many years to get rid of that
>
>
>
> On Fri, Dec 3, 2021 at 1:37 AM Jason Gerlowski <ge...@gmail.com> wrote:
>>
>> > 99% of Solr users never see the java code that implements an API let alone writing a new API.
>>
>> We may be talking past each other here a bit Noble. The JAX-RS
>> benefits I've been trying to describe are to benefit us developers,
>> not Solr users. I agree that 99% of our users won't know or care
>> about it, but who cares. The goal is to make _our_ lives as
>> maintainers easier going into the future. Jan described it in a word
>> better than I've been struggling to: it's about "modernization".
>>
>> Best,
>>
>> Jason
>>
>> On Thu, Dec 2, 2021 at 3:34 AM Jan Høydahl <ja...@cominvent.com> wrote:
>> >
>> > Agree, Ishan.
>> >
>> > Let's welcome and cheer initiatives to modernize this old codebase.
>> > Diving in deep and un-tangling DispatchFilter like Gus is attempting is brave.
>> > And testing JAX-RS to modernize APIs and make the codebas easier to grok is also not for the faint hearted.
>> >
>> > I hope both initiatives end up as a breath of fresh air for Solr developers. And perhaps users may get their hands on auto-generated OpenAPI docs for Solr as well!
>> > Theoretically foundations like this may make it feasible in the future to auto-generate client code from the spec,so that full Admin API client support in e.g. Go or Rust may be achievable without hand-coding. But that is still far away :)
>> >
>> > Jan
>> >
>> > 2. des. 2021 kl. 05:52 skrev Ishan Chattopadhyaya <ic...@gmail.com>:
>> >
>> > > We are discussing this as if moving to an external framework is going to be smooth. It's not.
>> >
>> > Noble, we won't know before someone has given it a try. My hunch suggests it won't be smooth, but we can decide on that once we see a concrete solution that 1) works well for all existing APIs (per core or global) without hacks/workarounds, and 2) has no security implications, and 3) doesn't clutter our codebase, and 4) doesn't affect our runtime performance.
>> >
>> > On Thu, Dec 2, 2021 at 4:22 AM Noble Paul <no...@gmail.com> wrote:
>> >>
>> >>
>> >>
>> >> On Thu, Dec 2, 2021 at 2:46 AM Jason Gerlowski <ge...@gmail.com> wrote:
>> >>>
>> >>> > There are no known issues with the current system
>> >>>
>> >>> Maybe you're using hyperbole to emphasise a point, but let's steer
>> >>> this discussion away from straw-men and caricatures. I mentioned
>> >>> specific known issues in a previous reply as a direct response to your
>> >>> question about them. The incomplete support for data types
>> >>> (SOLR-15619 and SOLR-15796). The inflexible PermissionNameProvider
>> >>> integration (SOLR-15823). If you consider those "minor", fair enough,
>> >>> and you're welcome to that opinion. But saying that they don't exist
>> >>> is a mischaracterization that wastes time and muddies water.
>> >>
>> >>
>> >> We are discussing this as if moving to an external framework is going to be smooth. It's not. I have seen how hard the Restlet integration was and how it stood out like a sore thumb. The amount of code required to integrate Solr APIs into restlet components was 10x more than the amount of code written for the new framework and it still was incomplete . Integrating anything to Solr is not simple and it's going to be an ongoing effort to ensure that all of the functionalities of a SolrRequestHandler are exposed to the new framework. Every single component that we have today has ongoing improvements and bug fixes. Using that as an excuse to do a complete rewrite is probably unwise.
>> >>
>> >>
>> >>>
>> >>> > Solr is not a general purpose web server where people write APIs every day
>> >>>
>> >>> Totally agree - 99% of the time someone opens an API file in Solr,
>> >>> they're reading, not writing. So if JAX-RS didn't help that case, it
>> >>> wouldn't be worth the effort of switching. But IMO it helps that
>> >>> "read" case a lot.
>> >>
>> >>
>> >> 99% of Solr users never see the java code that implements an API let alone writing a new API. They wouldn't care if an API is implemented using a Servlet API, a custom framework or something else. They only care about the input and output. New end points are rare.
>> >>>
>> >>>
>> >>> JAX-RS defined APIs expose inputs as strongly-typed parameters
>> >>> (compare to the weakly-typed SolrQueryRequest/SolrParams used today).
>> >>> JAX-RS enumerates every input to the API in one place. JAX-RS APIs
>> >>> can return any serializable type as a response (compare to the
>> >>> near-omnipresent NamedList today).
>> >>>
>> >>> (The annotation framework's PayloadObj class gets us a bit of this
>> >>> benefit, but not all of it and only on POST requests afaik.)
>> >>>
>> >>> This is all nice stuff at API-write time, but where it really helps is
>> >>> in making the code clearer and simpler whenever someone goes to read
>> >>> it later. No wondering what parameters an API actually takes. No
>> >>> puzzling out what a bunch of NamedList operations spread throughout
>> >>> API execution actually produce in the end. etc.
>> >>>
>> >>> > The only reason why anyone would need to use this framework is to write a custom request handler.
>> >>>
>> >>> ...except for the majority of people who come in contact with the
>> >>> framework by reading an existing Solr API? This ties into your point
>> >>> above about APIs being written infrequently but read all the time.
>> >>> Improving readability is the big gain here, and I think JAX-RS offers
>> >>> real improvements in that regard.
>> >>
>> >>
>> >> We had no framework for the last 15+ years and V1 API (the only actually used API) still uses no frameworks. Even the annotation framework is not used much even though readability is a lot better in the annotation based system. Readability has not been an overriding concern for V1 APIs for sure
>> >>
>> >>> Also, to clarify, I'm not asserting that no one configures custom
>> >>> requestHandlers. I was wondering aloud how common it was, as an open
>> >>> question.
>> >>>
>> >> Your point IS right. IRL, most of our users do not write request handlers . They write search components , query parsers and other plugins. Because we are a search engine and not a web server. I have never seen a user complaining about the V1 API of writing a request handler. They may actually complain about how hard it is to deploy one. Our users are much smarter than we give them credit for.
>> >>
>> >>
>> >>>
>> >>> > We did not fork Jackson. We are using Jackson itself.
>> >>>
>> >>> Ah, maybe you're right, "fork" is probably an unfair word. I looked
>> >>> for the serialization code, and I'll admit there is less than I
>> >>> thought involved here and I misunderstood how it works. So that's my
>> >>> bad. The JSON serialization/deserialization discussion is prob a
>> >>> digression from the core contention about whether JAX-RS would be
>> >>> helpful. I was curious about your earlier response and couldn't
>> >>> resist following up here but I should've done so elsewhere to avoid
>> >>> the distraction.
>> >>>
>> >>> Best,
>> >>>
>> >>>
>> >>> Jason
>> >>>
>> >>> On Tue, Nov 30, 2021 at 4:12 PM Noble Paul <no...@gmail.com> wrote:
>> >>> >
>> >>> > The annotation framework is just a single class and it is made to work well with Solr instead of changing Solr to suit the needs of some external framework. There are no known issues with the current system and there are unknown issues that you'll face introducing a new framework.
>> >>> >
>> >>> > Solr is not a general purpose web server where people write APIs every day. It's a finished product where people use the publicly available APIs . We should focus our efforts on making our APIs work well .
>> >>> >
>> >>> > On Wed, Dec 1, 2021, 2:14 AM Jason Gerlowski <ge...@gmail.com> wrote:
>> >>> >>
>> >>> >> > We need to make an API work on a per core basis and cores can come up and go down randomly
>> >>> >>
>> >>> >> Ah, yeah. I'll admit I was overlooking some of the wrinkles around
>> >>> >> custom request handlers etc. That is a problem for switching to some
>> >>> >> off-the-shelf framework for sure. It may still be possible, but it's
>> >>> >> definitely a big hurdle. I wonder how commonly users still take
>> >>> >> advantage of the custom requestHandler stuff these days? I haven't
>> >>> >> run across many customers who used them extensively, but maybe that's
>> >>> >> self-selecting in some way?
>> >>> >
>> >>> >
>> >>> > The only reason why anyone would need to use this framework is to write a custom request handler. You're also saying nobody writes custom request handlers. So, what problem are you trying to solve if you think nobody uses them?
>> >>> >
>> >>> >
>> >>> >>
>> >>> >> > If SolrJ has a Jackson dependency, it can have a conflict (different versions) with the Jackson used by the client app
>> >>> >>
>> >>> >> SolrJ has a handful of dependencies - is there a reason we take this
>> >>> >> approach with Jackson, but not with any of SolrJ's other deps?
>> >>> >> (apache-httpcomponents, apache-commons, zookeeper-client, etc.)
>> >>> >>
>> >>> >> Granted, I get that we want SolrJ to be as painless as possible for
>> >>> >> its users, and jar-conflicts are a part of that and really suck. But
>> >>> >> there are good tools out there for handling jar conflicts, and
>> >>> >> reimplementing chunks of library functionality just to avoid a
>> >>> >> gradle-dep strikes me as unsustainable from a maintenance/community
>> >>> >> perspective. Not to mention that forking/mimicking Jackson code
>> >>> >> deprives our users of any efficiency/security improvements that
>> >>> >> Jackson might get tomorrow or next week or next year.
>> >>> >
>> >>> >
>> >>> > We did not fork Jackson. We are using Jackson itself. Jackson supports custom annotations. There was a huge discussion why we should do it this way.
>> >>> >>
>> >>> >>
>> >>> >> Unless there's some reason specific to Jackson I guess I just don't
>> >>> >> get it. But if I'm the minority opinion on that, fair enough I guess.
>> >>> >>
>> >>> >> On Tue, Nov 30, 2021 at 5:05 AM Noble Paul <no...@gmail.com> wrote:
>> >>> >> >
>> >>> >> > The @JsonProperty annotations are added because it's a dependency in SolrJ as well
>> >>> >> > If SolrJ has a Jackson dependency, it can have a conflict (different versions) with the Jackson used by the client app.
>> >>> >> >
>> >>> >> > On Tue, Nov 30, 2021, 5:21 PM Noble Paul <no...@gmail.com> wrote:
>> >>> >> >>
>> >>> >> >> Also keep in mind that the same endpoint can be accessed with a core name and a collection name prefixes.
>> >>> >> >>
>> >>> >> >> On Tue, Nov 30, 2021 at 3:55 PM Noble Paul <no...@gmail.com> wrote:
>> >>> >> >>>
>> >>> >> >>> True Gus, Almost every framework works outside of SolrDispatchFilter+ HttpSolrCall. A lot of our initializations occur there.
>> >>> >> >>>
>> >>> >> >>> We need to make an API work on a per core basis and cores can come up and go down randomly. So we need to register these endpoints on a core.
>> >>> >> >>>
>> >>> >> >>> I'm not sure if any framework can achieve the same.
>> >>> >> >>>
>> >>> >> >>> On Tue, Nov 30, 2021 at 5:39 AM Gus Heck <gu...@gmail.com> wrote:
>> >>> >> >>>>
>> >>> >> >>>> IIRC last time I looked restlet had the unsavory property of existing outside of the SolrDispatchFilter, unlike everything else which made for special cases because several things that probably ought to be their own siervlet filters are glommed into SolrDispatchFilter, like security, tracing and MDC setup/teardown per request. Restlet wouldn't be so bad if one could just wrap such filters around it too...
>> >>> >> >>>>
>> >>> >> >>>> On Mon, Nov 29, 2021 at 9:42 AM Jason Gerlowski <ge...@gmail.com> wrote:
>> >>> >> >>>>>
>> >>> >> >>>>> > These are minor improvements compared to a full rewrite of the entire framework
>> >>> >> >>>>>
>> >>> >> >>>>> If you think data type support is minor, fair enough. But to clarify
>> >>> >> >>>>> I'm not suggesting a rewrite - I'm suggesting using something that
>> >>> >> >>>>> already exists off the shelf. Jersey (e.g.) itself provides the
>> >>> >> >>>>> framework - there would be no "rewrite".
>> >>> >> >>>>>
>> >>> >> >>>>> re: past restlet use
>> >>> >> >>>>>
>> >>> >> >>>>> > It was not playing well with our security framework. The framework was not working well with Solr APIs
>> >>> >> >>>>>
>> >>> >> >>>>> Ah, very interesting! Security isn't something Eric or I tackled in
>> >>> >> >>>>> our little spike branch, but it's definitely a concern. Do you
>> >>> >> >>>>> remember the specific concerns? Or recall where any of the discussion
>> >>> >> >>>>> around this happened?
>> >>> >> >>>>>
>> >>> >> >>>>> Without the context of that past discussion, it seems like the
>> >>> >> >>>>> "PermissionNameProvider" interface could be implemented just as well
>> >>> >> >>>>> by a class with (e.g.) Jersey annotations as one with our own custom
>> >>> >> >>>>> annotations. Certainly there'd need to be some
>> >>> >> >>>>> RuleBasedAuthorizationPlugin changes or other integration code, but
>> >>> >> >>>>> nothing that feels insurmountable.
>> >>> >> >>>>>
>> >>> >> >>>>> Maybe I can try spiking it out soon and find the issues myself, but
>> >>> >> >>>>> it'd be much easier if someone happens to remember and can save me the
>> >>> >> >>>>> trouble :-p
>> >>> >> >>>>>
>> >>> >> >>>>> Best,
>> >>> >> >>>>>
>> >>> >> >>>>> Jason
>> >>> >> >>>>>
>> >>> >> >>>>> On Fri, Nov 26, 2021 at 12:05 AM Noble Paul <no...@gmail.com> wrote:
>> >>> >> >>>>> >
>> >>> >> >>>>> > The Annotations framework was written after playing with other frameworks. There were many shortcomings which were hard to overcome.
>> >>> >> >>>>> >
>> >>> >> >>>>> > The best example is a per collection API . How do you register an endpoint for a collection/core ?
>> >>> >> >>>>> >
>> >>> >> >>>>> > On Fri, Nov 26, 2021 at 3:42 PM Noble Paul <no...@gmail.com> wrote:
>> >>> >> >>>>> >>
>> >>> >> >>>>> >>
>> >>> >> >>>>> >>
>> >>> >> >>>>> >> On Fri, Nov 26, 2021 at 1:03 AM Jason Gerlowski <ge...@gmail.com> wrote:
>> >>> >> >>>>> >>>
>> >>> >> >>>>> >>> > Is there some problem with our annotations that we hope to solve using third party dependencies?
>> >>> >> >>>>> >>>
>> >>> >> >>>>> >>> I guess so yeah. Third-party deps are just fuller, more robust
>> >>> >> >>>>> >>> solutions, whereas our annotations still need support added now and
>> >>> >> >>>>> >>> then for even primitive data types like "long" (see SOLR-15619).
>> >>> >> >>>>> >>
>> >>> >> >>>>> >> These are minor improvements compared to a full rewrite of the entire framework
>> >>> >> >>>>> >>
>> >>> >> >>>>> >>
>> >>> >> >>>>> >>>
>> >>> >> >>>>> >>> Every JIRA spent doing basic stuff like that is time away from
>> >>> >> >>>>> >>> improving Solr in some other way.
>> >>> >> >>>>> >>>
>> >>> >> >>>>> >>> So there are feature-gap/capabilities arguments for moving to a
>> >>> >> >>>>> >>> third-party dep, sure. But, even if our annotations did everything
>> >>> >> >>>>> >>> Jersey+Jackson do today, I think switching would still be worth it.
>> >>> >> >>>>> >>> Every LOC in our code base brings along with it some maintenance cost:
>> >>> >> >>>>> >>> it might have bugs, needs tested, takes time for new contributors to
>> >>> >> >>>>> >>> "grok", etc. Using off-the-shelf here would nuke a whole bunch of
>> >>> >> >>>>> >>> that. If off-the-shelf is available for some given functionality, we
>> >>> >> >>>>> >>> should need a compelling reason to NOT use it.
>> >>> >> >>>>> >>>
>> >>> >> >>>>> >>> Lastly, I think there's an "approachability" argument for using
>> >>> >> >>>>> >>> off-the-shelf. Thousands of developers out there are familiar with
>> >>> >> >>>>> >>> (e.g.) Jersey, compared to maybe 15 or 20 (in the world) familiar with
>> >>> >> >>>>> >>> Solr's custom annotations. Using a well-known technology like Jersey
>> >>> >> >>>>> >>> would make Solr all the easier to approach and contribute to for that
>> >>> >> >>>>> >>> pool of developers.
>> >>> >> >>>>> >>>
>> >>> >> >>>>> >>> > By the way, we have used Restlet in the past and that has been a regrettable decision.
>> >>> >> >>>>> >>
>> >>> >> >>>>> >>
>> >>> >> >>>>> >>>
>> >>> >> >>>>> >>> Ah, yeah, that's just the context I'm missing. Anyone have a pointer
>> >>> >> >>>>> >>> to related discussions, or remember what made this "regrettable"? All
>> >>> >> >>>>> >>> the theoretical benefits in the world don't matter much if we've
>> >>> >> >>>>> >>> already tried something like this in the past and decided against it.
>> >>> >> >>>>> >>
>> >>> >> >>>>> >>
>> >>> >> >>>>> >> It was not playing well with our security framework. The framework was not working well with Solr APIs
>> >>> >> >>>>> >>
>> >>> >> >>>>> >>>
>> >>> >> >>>>> >>> (Unrelated - Happy Thanksgiving all!)
>> >>> >> >>>>> >>>
>> >>> >> >>>>> >>> Best,
>> >>> >> >>>>> >>>
>> >>> >> >>>>> >>> Jason
>> >>> >> >>>>> >>>
>> >>> >> >>>>> >>> On Thu, Nov 25, 2021 at 7:32 AM Noble Paul <no...@gmail.com> wrote:
>> >>> >> >>>>> >>> >
>> >>> >> >>>>> >>> > Have you gone through an API written using the @EndPoint annotation?
>> >>> >> >>>>> >>> >
>> >>> >> >>>>> >>> > I strongly recommend that you do
>> >>> >> >>>>> >>> >
>> >>> >> >>>>> >>> > On Thu, Nov 25, 2021, 11:30 PM Eric Pugh <ep...@opensourceconnections.com> wrote:
>> >>> >> >>>>> >>> >>
>> >>> >> >>>>> >>> >> I have found our V2 API code to be very impenetrable to understand. Part of it is how the code is intertwined with support for V1, however it’s also because there aren’t really resources to go look at to understand how it should work! Maintaining the API should be very simple work, as they just exist as a translation. The home grown stuff may make sense if you are a super knowledgable Solr developer, but if you are just a new person, it’s a lot harder to contribute.
>> >>> >> >>>>> >>> >>
>> >>> >> >>>>> >>> >> I was interested in the Jersey stuff because I’ve seen lots of projects use it very successfully, and if I want to implement something, well, there are lots of blogs and resources out there!
>> >>> >> >>>>> >>> >>
>> >>> >> >>>>> >>> >> Can anyone recap briefly why we dropped RESTlet? And what lessons learned there might apply to adopting Jersey for API support? Looking at https://issues.apache.org/jira/browse/SOLR-14659, it was partly deprecated because we were not using it to support all the API, only the ManagedResource ones, and https://issues.apache.org/jira/browse/SOLR-14766 suggests that RESTlet maybe was no longer being updated? One reason why we spiked out Jersey was because of the broad support in the Java world! Looking at how much work we have to do in the V2 API world, we need a much broader pool of developers contributing to get there!
>> >>> >> >>>>> >>> >>
>> >>> >> >>>>> >>> >> Related, are there specific features/aspects of our annotations that enable things in Solr that couldn’t be done otherwise?
>> >>> >> >>>>> >>> >>
>> >>> >> >>>>> >>> >> On Nov 25, 2021, at 2:12 AM, Ishan Chattopadhyaya <ic...@gmail.com> wrote:
>> >>> >> >>>>> >>> >>
>> >>> >> >>>>> >>> >> Is there some problem with our annotations that we hope to solve using third party dependencies?
>> >>> >> >>>>> >>> >> By the way, we have used Restlet in the past and that has been a regrettable decision.
>> >>> >> >>>>> >>> >>
>> >>> >> >>>>> >>> >> On Thu, Nov 25, 2021 at 10:10 AM Jason Gerlowski <ge...@gmail.com> wrote:
>> >>> >> >>>>> >>> >>>
>> >>> >> >>>>> >>> >>> Solr's custom annotation framework ('@Endpoint', '@Command', etc.) has
>> >>> >> >>>>> >>> >>> cropped up a few times over the past week or two. [1] [2]. Having them
>> >>> >> >>>>> >>> >>> on top of mind, I've been wondering - is there a reason we use our own
>> >>> >> >>>>> >>> >>> annotations here instead of something off the shelf?
>> >>> >> >>>>> >>> >>>
>> >>> >> >>>>> >>> >>> What we have works well enough, but anything homegrown comes with more
>> >>> >> >>>>> >>> >>> maintenance burden than we'd have if we used something off the shelf.
>> >>> >> >>>>> >>> >>> There are plenty of well-used, active projects out there whose whole
>> >>> >> >>>>> >>> >>> purpose is facilitating the whole "annotation based API" thing
>> >>> >> >>>>> >>> >>> (Jersey, Restlet, RESTEasy, etc.) - why not use one of them?
>> >>> >> >>>>> >>> >>>
>> >>> >> >>>>> >>> >>> Does anyone know of any technical reasons why we can't go this route?
>> >>> >> >>>>> >>> >>> Or have any subjective reasons why we shouldn't? Or any context on
>> >>> >> >>>>> >>> >>> why we wrote our own Endpoint, Command, JsonProperty annotations
>> >>> >> >>>>> >>> >>> originally?
>> >>> >> >>>>> >>> >>>
>> >>> >> >>>>> >>> >>> FWIW, Eric Pugh and I spiked out a small POC recently, and got
>> >>> >> >>>>> >>> >>> Jersey+Jackson working for a few simple APIs without too much trouble.
>> >>> >> >>>>> >>> >>> [3] Obviously nothing production-ready there, and there's still a lot
>> >>> >> >>>>> >>> >>> of open questions (e.g. how would javabin be supported?), but we both
>> >>> >> >>>>> >>> >>> came away convinced that it seemed feasible, at least. Best of all,
>> >>> >> >>>>> >>> >>> APIs using our current homegrown annotation framework the switchover
>> >>> >> >>>>> >>> >>> seems blessedly straightforward, and it doesn't look like Jersey
>> >>> >> >>>>> >>> >>> (which we chose mostly arbitrarily) bloats our dist all that much.
>> >>> >> >>>>> >>> >>>
>> >>> >> >>>>> >>> >>> Curious if anyone has thoughts or context on how we ended up with the
>> >>> >> >>>>> >>> >>> annotation setup we use today!
>> >>> >> >>>>> >>> >>>
>> >>> >> >>>>> >>> >>> Best,
>> >>> >> >>>>> >>> >>>
>> >>> >> >>>>> >>> >>> Jason
>> >>> >> >>>>> >>> >>>
>> >>> >> >>>>> >>> >>> [1] https://issues.apache.org/jira/browse/SOLR-15182 (and children)
>> >>> >> >>>>> >>> >>> [2] http://mail-archives.apache.org/mod_mbox/solr-dev/202111.mbox/%3CCABEwPvENL41Pm6%2BOmjXb6Sx5N2XjUtnbWhgKOZSrnLjWBA8tcA%40mail.gmail.com%3E
>> >>> >> >>>>> >>> >>> [3] https://github.com/gerlowskija/solr/tree/jersey_jaxrs_jackson_solr_apis.
>> >>> >> >>>>> >>> >>>
>> >>> >> >>>>> >>> >>> ---------------------------------------------------------------------
>> >>> >> >>>>> >>> >>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>> >>> >> >>>>> >>> >>> For additional commands, e-mail: dev-help@solr.apache.org
>> >>> >> >>>>> >>> >>>
>> >>> >> >>>>> >>> >>
>> >>> >> >>>>> >>> >> _______________________
>> >>> >> >>>>> >>> >> Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | http://www.opensourceconnections.com | My Free/Busy
>> >>> >> >>>>> >>> >> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
>> >>> >> >>>>> >>> >> This e-mail and all contents, including attachments, is considered to be Company Confidential unless explicitly stated otherwise, regardless of whether attachments are marked as such.
>> >>> >> >>>>> >>> >>
>> >>> >> >>>>> >>>
>> >>> >> >>>>> >>> ---------------------------------------------------------------------
>> >>> >> >>>>> >>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>> >>> >> >>>>> >>> For additional commands, e-mail: dev-help@solr.apache.org
>> >>> >> >>>>> >>>
>> >>> >> >>>>> >>
>> >>> >> >>>>> >>
>> >>> >> >>>>> >> --
>> >>> >> >>>>> >> -----------------------------------------------------
>> >>> >> >>>>> >> Noble Paul
>> >>> >> >>>>> >
>> >>> >> >>>>> >
>> >>> >> >>>>> >
>> >>> >> >>>>> > --
>> >>> >> >>>>> > -----------------------------------------------------
>> >>> >> >>>>> > Noble Paul
>> >>> >> >>>>>
>> >>> >> >>>>> ---------------------------------------------------------------------
>> >>> >> >>>>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>> >>> >> >>>>> For additional commands, e-mail: dev-help@solr.apache.org
>> >>> >> >>>>>
>> >>> >> >>>>
>> >>> >> >>>>
>> >>> >> >>>> --
>> >>> >> >>>> http://www.needhamsoftware.com (work)
>> >>> >> >>>> http://www.the111shift.com (play)
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>> --
>> >>> >> >>> -----------------------------------------------------
>> >>> >> >>> Noble Paul
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> --
>> >>> >> >> -----------------------------------------------------
>> >>> >> >> Noble Paul
>> >>> >>
>> >>> >> ---------------------------------------------------------------------
>> >>> >> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>> >>> >> For additional commands, e-mail: dev-help@solr.apache.org
>> >>> >>
>> >>>
>> >>> ---------------------------------------------------------------------
>> >>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>> >>> For additional commands, e-mail: dev-help@solr.apache.org
>> >>>
>> >>
>> >>
>> >> --
>> >> -----------------------------------------------------
>> >> Noble Paul
>> >
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>> For additional commands, e-mail: dev-help@solr.apache.org
>>
>
>
> --
> -----------------------------------------------------
> Noble Paul
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: JAX-RS APIs in Solr
Posted by Noble Paul <no...@gmail.com>.
I do not deny that a standardized framework may be helpful. I'm already
suffering from the PTSD of restlet integration. The integration code was
extremely complex and a security nightmare.It was added without any perf
tests too. It took many years to get rid of that
On Fri, Dec 3, 2021 at 1:37 AM Jason Gerlowski <ge...@gmail.com>
wrote:
> > 99% of Solr users never see the java code that implements an API let
> alone writing a new API.
>
> We may be talking past each other here a bit Noble. The JAX-RS
> benefits I've been trying to describe are to benefit us developers,
> not Solr users. I agree that 99% of our users won't know or care
> about it, but who cares. The goal is to make _our_ lives as
> maintainers easier going into the future. Jan described it in a word
> better than I've been struggling to: it's about "modernization".
>
> Best,
>
> Jason
>
> On Thu, Dec 2, 2021 at 3:34 AM Jan Høydahl <ja...@cominvent.com> wrote:
> >
> > Agree, Ishan.
> >
> > Let's welcome and cheer initiatives to modernize this old codebase.
> > Diving in deep and un-tangling DispatchFilter like Gus is attempting is
> brave.
> > And testing JAX-RS to modernize APIs and make the codebas easier to grok
> is also not for the faint hearted.
> >
> > I hope both initiatives end up as a breath of fresh air for Solr
> developers. And perhaps users may get their hands on auto-generated
> OpenAPI docs for Solr as well!
> > Theoretically foundations like this may make it feasible in the future
> to auto-generate client code from the spec,so that full Admin API client
> support in e.g. Go or Rust may be achievable without hand-coding. But that
> is still far away :)
> >
> > Jan
> >
> > 2. des. 2021 kl. 05:52 skrev Ishan Chattopadhyaya <
> ichattopadhyaya@gmail.com>:
> >
> > > We are discussing this as if moving to an external framework is going
> to be smooth. It's not.
> >
> > Noble, we won't know before someone has given it a try. My hunch
> suggests it won't be smooth, but we can decide on that once we see a
> concrete solution that 1) works well for all existing APIs (per core or
> global) without hacks/workarounds, and 2) has no security implications, and
> 3) doesn't clutter our codebase, and 4) doesn't affect our runtime
> performance.
> >
> > On Thu, Dec 2, 2021 at 4:22 AM Noble Paul <no...@gmail.com> wrote:
> >>
> >>
> >>
> >> On Thu, Dec 2, 2021 at 2:46 AM Jason Gerlowski <ge...@gmail.com>
> wrote:
> >>>
> >>> > There are no known issues with the current system
> >>>
> >>> Maybe you're using hyperbole to emphasise a point, but let's steer
> >>> this discussion away from straw-men and caricatures. I mentioned
> >>> specific known issues in a previous reply as a direct response to your
> >>> question about them. The incomplete support for data types
> >>> (SOLR-15619 and SOLR-15796). The inflexible PermissionNameProvider
> >>> integration (SOLR-15823). If you consider those "minor", fair enough,
> >>> and you're welcome to that opinion. But saying that they don't exist
> >>> is a mischaracterization that wastes time and muddies water.
> >>
> >>
> >> We are discussing this as if moving to an external framework is going
> to be smooth. It's not. I have seen how hard the Restlet integration was
> and how it stood out like a sore thumb. The amount of code required to
> integrate Solr APIs into restlet components was 10x more than the amount of
> code written for the new framework and it still was incomplete .
> Integrating anything to Solr is not simple and it's going to be an ongoing
> effort to ensure that all of the functionalities of a SolrRequestHandler
> are exposed to the new framework. Every single component that we have today
> has ongoing improvements and bug fixes. Using that as an excuse to do a
> complete rewrite is probably unwise.
> >>
> >>
> >>>
> >>> > Solr is not a general purpose web server where people write APIs
> every day
> >>>
> >>> Totally agree - 99% of the time someone opens an API file in Solr,
> >>> they're reading, not writing. So if JAX-RS didn't help that case, it
> >>> wouldn't be worth the effort of switching. But IMO it helps that
> >>> "read" case a lot.
> >>
> >>
> >> 99% of Solr users never see the java code that implements an API let
> alone writing a new API. They wouldn't care if an API is implemented using
> a Servlet API, a custom framework or something else. They only care about
> the input and output. New end points are rare.
> >>>
> >>>
> >>> JAX-RS defined APIs expose inputs as strongly-typed parameters
> >>> (compare to the weakly-typed SolrQueryRequest/SolrParams used today).
> >>> JAX-RS enumerates every input to the API in one place. JAX-RS APIs
> >>> can return any serializable type as a response (compare to the
> >>> near-omnipresent NamedList today).
> >>>
> >>> (The annotation framework's PayloadObj class gets us a bit of this
> >>> benefit, but not all of it and only on POST requests afaik.)
> >>>
> >>> This is all nice stuff at API-write time, but where it really helps is
> >>> in making the code clearer and simpler whenever someone goes to read
> >>> it later. No wondering what parameters an API actually takes. No
> >>> puzzling out what a bunch of NamedList operations spread throughout
> >>> API execution actually produce in the end. etc.
> >>>
> >>> > The only reason why anyone would need to use this framework is to
> write a custom request handler.
> >>>
> >>> ...except for the majority of people who come in contact with the
> >>> framework by reading an existing Solr API? This ties into your point
> >>> above about APIs being written infrequently but read all the time.
> >>> Improving readability is the big gain here, and I think JAX-RS offers
> >>> real improvements in that regard.
> >>
> >>
> >> We had no framework for the last 15+ years and V1 API (the only
> actually used API) still uses no frameworks. Even the annotation framework
> is not used much even though readability is a lot better in the annotation
> based system. Readability has not been an overriding concern for V1 APIs
> for sure
> >>
> >>> Also, to clarify, I'm not asserting that no one configures custom
> >>> requestHandlers. I was wondering aloud how common it was, as an open
> >>> question.
> >>>
> >> Your point IS right. IRL, most of our users do not write request
> handlers . They write search components , query parsers and other plugins.
> Because we are a search engine and not a web server. I have never seen a
> user complaining about the V1 API of writing a request handler. They may
> actually complain about how hard it is to deploy one. Our users are much
> smarter than we give them credit for.
> >>
> >>
> >>>
> >>> > We did not fork Jackson. We are using Jackson itself.
> >>>
> >>> Ah, maybe you're right, "fork" is probably an unfair word. I looked
> >>> for the serialization code, and I'll admit there is less than I
> >>> thought involved here and I misunderstood how it works. So that's my
> >>> bad. The JSON serialization/deserialization discussion is prob a
> >>> digression from the core contention about whether JAX-RS would be
> >>> helpful. I was curious about your earlier response and couldn't
> >>> resist following up here but I should've done so elsewhere to avoid
> >>> the distraction.
> >>>
> >>> Best,
> >>>
> >>>
> >>> Jason
> >>>
> >>> On Tue, Nov 30, 2021 at 4:12 PM Noble Paul <no...@gmail.com>
> wrote:
> >>> >
> >>> > The annotation framework is just a single class and it is made to
> work well with Solr instead of changing Solr to suit the needs of some
> external framework. There are no known issues with the current system and
> there are unknown issues that you'll face introducing a new framework.
> >>> >
> >>> > Solr is not a general purpose web server where people write APIs
> every day. It's a finished product where people use the publicly available
> APIs . We should focus our efforts on making our APIs work well .
> >>> >
> >>> > On Wed, Dec 1, 2021, 2:14 AM Jason Gerlowski <ge...@gmail.com>
> wrote:
> >>> >>
> >>> >> > We need to make an API work on a per core basis and cores can
> come up and go down randomly
> >>> >>
> >>> >> Ah, yeah. I'll admit I was overlooking some of the wrinkles around
> >>> >> custom request handlers etc. That is a problem for switching to
> some
> >>> >> off-the-shelf framework for sure. It may still be possible, but
> it's
> >>> >> definitely a big hurdle. I wonder how commonly users still take
> >>> >> advantage of the custom requestHandler stuff these days? I haven't
> >>> >> run across many customers who used them extensively, but maybe
> that's
> >>> >> self-selecting in some way?
> >>> >
> >>> >
> >>> > The only reason why anyone would need to use this framework is to
> write a custom request handler. You're also saying nobody writes custom
> request handlers. So, what problem are you trying to solve if you think
> nobody uses them?
> >>> >
> >>> >
> >>> >>
> >>> >> > If SolrJ has a Jackson dependency, it can have a conflict
> (different versions) with the Jackson used by the client app
> >>> >>
> >>> >> SolrJ has a handful of dependencies - is there a reason we take this
> >>> >> approach with Jackson, but not with any of SolrJ's other deps?
> >>> >> (apache-httpcomponents, apache-commons, zookeeper-client, etc.)
> >>> >>
> >>> >> Granted, I get that we want SolrJ to be as painless as possible for
> >>> >> its users, and jar-conflicts are a part of that and really suck.
> But
> >>> >> there are good tools out there for handling jar conflicts, and
> >>> >> reimplementing chunks of library functionality just to avoid a
> >>> >> gradle-dep strikes me as unsustainable from a maintenance/community
> >>> >> perspective. Not to mention that forking/mimicking Jackson code
> >>> >> deprives our users of any efficiency/security improvements that
> >>> >> Jackson might get tomorrow or next week or next year.
> >>> >
> >>> >
> >>> > We did not fork Jackson. We are using Jackson itself. Jackson
> supports custom annotations. There was a huge discussion why we should do
> it this way.
> >>> >>
> >>> >>
> >>> >> Unless there's some reason specific to Jackson I guess I just don't
> >>> >> get it. But if I'm the minority opinion on that, fair enough I
> guess.
> >>> >>
> >>> >> On Tue, Nov 30, 2021 at 5:05 AM Noble Paul <no...@gmail.com>
> wrote:
> >>> >> >
> >>> >> > The @JsonProperty annotations are added because it's a dependency
> in SolrJ as well
> >>> >> > If SolrJ has a Jackson dependency, it can have a conflict
> (different versions) with the Jackson used by the client app.
> >>> >> >
> >>> >> > On Tue, Nov 30, 2021, 5:21 PM Noble Paul <no...@gmail.com>
> wrote:
> >>> >> >>
> >>> >> >> Also keep in mind that the same endpoint can be accessed with a
> core name and a collection name prefixes.
> >>> >> >>
> >>> >> >> On Tue, Nov 30, 2021 at 3:55 PM Noble Paul <no...@gmail.com>
> wrote:
> >>> >> >>>
> >>> >> >>> True Gus, Almost every framework works outside of
> SolrDispatchFilter+ HttpSolrCall. A lot of our initializations occur there.
> >>> >> >>>
> >>> >> >>> We need to make an API work on a per core basis and cores can
> come up and go down randomly. So we need to register these endpoints on a
> core.
> >>> >> >>>
> >>> >> >>> I'm not sure if any framework can achieve the same.
> >>> >> >>>
> >>> >> >>> On Tue, Nov 30, 2021 at 5:39 AM Gus Heck <gu...@gmail.com>
> wrote:
> >>> >> >>>>
> >>> >> >>>> IIRC last time I looked restlet had the unsavory property of
> existing outside of the SolrDispatchFilter, unlike everything else which
> made for special cases because several things that probably ought to be
> their own siervlet filters are glommed into SolrDispatchFilter, like
> security, tracing and MDC setup/teardown per request. Restlet wouldn't be
> so bad if one could just wrap such filters around it too...
> >>> >> >>>>
> >>> >> >>>> On Mon, Nov 29, 2021 at 9:42 AM Jason Gerlowski <
> gerlowskija@gmail.com> wrote:
> >>> >> >>>>>
> >>> >> >>>>> > These are minor improvements compared to a full rewrite of
> the entire framework
> >>> >> >>>>>
> >>> >> >>>>> If you think data type support is minor, fair enough. But to
> clarify
> >>> >> >>>>> I'm not suggesting a rewrite - I'm suggesting using something
> that
> >>> >> >>>>> already exists off the shelf. Jersey (e.g.) itself provides
> the
> >>> >> >>>>> framework - there would be no "rewrite".
> >>> >> >>>>>
> >>> >> >>>>> re: past restlet use
> >>> >> >>>>>
> >>> >> >>>>> > It was not playing well with our security framework. The
> framework was not working well with Solr APIs
> >>> >> >>>>>
> >>> >> >>>>> Ah, very interesting! Security isn't something Eric or I
> tackled in
> >>> >> >>>>> our little spike branch, but it's definitely a concern. Do
> you
> >>> >> >>>>> remember the specific concerns? Or recall where any of the
> discussion
> >>> >> >>>>> around this happened?
> >>> >> >>>>>
> >>> >> >>>>> Without the context of that past discussion, it seems like the
> >>> >> >>>>> "PermissionNameProvider" interface could be implemented just
> as well
> >>> >> >>>>> by a class with (e.g.) Jersey annotations as one with our own
> custom
> >>> >> >>>>> annotations. Certainly there'd need to be some
> >>> >> >>>>> RuleBasedAuthorizationPlugin changes or other integration
> code, but
> >>> >> >>>>> nothing that feels insurmountable.
> >>> >> >>>>>
> >>> >> >>>>> Maybe I can try spiking it out soon and find the issues
> myself, but
> >>> >> >>>>> it'd be much easier if someone happens to remember and can
> save me the
> >>> >> >>>>> trouble :-p
> >>> >> >>>>>
> >>> >> >>>>> Best,
> >>> >> >>>>>
> >>> >> >>>>> Jason
> >>> >> >>>>>
> >>> >> >>>>> On Fri, Nov 26, 2021 at 12:05 AM Noble Paul <
> noble.paul@gmail.com> wrote:
> >>> >> >>>>> >
> >>> >> >>>>> > The Annotations framework was written after playing with
> other frameworks. There were many shortcomings which were hard to overcome.
> >>> >> >>>>> >
> >>> >> >>>>> > The best example is a per collection API . How do you
> register an endpoint for a collection/core ?
> >>> >> >>>>> >
> >>> >> >>>>> > On Fri, Nov 26, 2021 at 3:42 PM Noble Paul <
> noble.paul@gmail.com> wrote:
> >>> >> >>>>> >>
> >>> >> >>>>> >>
> >>> >> >>>>> >>
> >>> >> >>>>> >> On Fri, Nov 26, 2021 at 1:03 AM Jason Gerlowski <
> gerlowskija@gmail.com> wrote:
> >>> >> >>>>> >>>
> >>> >> >>>>> >>> > Is there some problem with our annotations that we hope
> to solve using third party dependencies?
> >>> >> >>>>> >>>
> >>> >> >>>>> >>> I guess so yeah. Third-party deps are just fuller, more
> robust
> >>> >> >>>>> >>> solutions, whereas our annotations still need support
> added now and
> >>> >> >>>>> >>> then for even primitive data types like "long" (see
> SOLR-15619).
> >>> >> >>>>> >>
> >>> >> >>>>> >> These are minor improvements compared to a full rewrite of
> the entire framework
> >>> >> >>>>> >>
> >>> >> >>>>> >>
> >>> >> >>>>> >>>
> >>> >> >>>>> >>> Every JIRA spent doing basic stuff like that is time away
> from
> >>> >> >>>>> >>> improving Solr in some other way.
> >>> >> >>>>> >>>
> >>> >> >>>>> >>> So there are feature-gap/capabilities arguments for
> moving to a
> >>> >> >>>>> >>> third-party dep, sure. But, even if our annotations did
> everything
> >>> >> >>>>> >>> Jersey+Jackson do today, I think switching would still be
> worth it.
> >>> >> >>>>> >>> Every LOC in our code base brings along with it some
> maintenance cost:
> >>> >> >>>>> >>> it might have bugs, needs tested, takes time for new
> contributors to
> >>> >> >>>>> >>> "grok", etc. Using off-the-shelf here would nuke a whole
> bunch of
> >>> >> >>>>> >>> that. If off-the-shelf is available for some given
> functionality, we
> >>> >> >>>>> >>> should need a compelling reason to NOT use it.
> >>> >> >>>>> >>>
> >>> >> >>>>> >>> Lastly, I think there's an "approachability" argument for
> using
> >>> >> >>>>> >>> off-the-shelf. Thousands of developers out there are
> familiar with
> >>> >> >>>>> >>> (e.g.) Jersey, compared to maybe 15 or 20 (in the world)
> familiar with
> >>> >> >>>>> >>> Solr's custom annotations. Using a well-known technology
> like Jersey
> >>> >> >>>>> >>> would make Solr all the easier to approach and contribute
> to for that
> >>> >> >>>>> >>> pool of developers.
> >>> >> >>>>> >>>
> >>> >> >>>>> >>> > By the way, we have used Restlet in the past and that
> has been a regrettable decision.
> >>> >> >>>>> >>
> >>> >> >>>>> >>
> >>> >> >>>>> >>>
> >>> >> >>>>> >>> Ah, yeah, that's just the context I'm missing. Anyone
> have a pointer
> >>> >> >>>>> >>> to related discussions, or remember what made this
> "regrettable"? All
> >>> >> >>>>> >>> the theoretical benefits in the world don't matter much
> if we've
> >>> >> >>>>> >>> already tried something like this in the past and decided
> against it.
> >>> >> >>>>> >>
> >>> >> >>>>> >>
> >>> >> >>>>> >> It was not playing well with our security framework. The
> framework was not working well with Solr APIs
> >>> >> >>>>> >>
> >>> >> >>>>> >>>
> >>> >> >>>>> >>> (Unrelated - Happy Thanksgiving all!)
> >>> >> >>>>> >>>
> >>> >> >>>>> >>> Best,
> >>> >> >>>>> >>>
> >>> >> >>>>> >>> Jason
> >>> >> >>>>> >>>
> >>> >> >>>>> >>> On Thu, Nov 25, 2021 at 7:32 AM Noble Paul <
> noble.paul@gmail.com> wrote:
> >>> >> >>>>> >>> >
> >>> >> >>>>> >>> > Have you gone through an API written using the
> @EndPoint annotation?
> >>> >> >>>>> >>> >
> >>> >> >>>>> >>> > I strongly recommend that you do
> >>> >> >>>>> >>> >
> >>> >> >>>>> >>> > On Thu, Nov 25, 2021, 11:30 PM Eric Pugh <
> epugh@opensourceconnections.com> wrote:
> >>> >> >>>>> >>> >>
> >>> >> >>>>> >>> >> I have found our V2 API code to be very impenetrable
> to understand. Part of it is how the code is intertwined with support for
> V1, however it’s also because there aren’t really resources to go look at
> to understand how it should work! Maintaining the API should be very
> simple work, as they just exist as a translation. The home grown stuff
> may make sense if you are a super knowledgable Solr developer, but if you
> are just a new person, it’s a lot harder to contribute.
> >>> >> >>>>> >>> >>
> >>> >> >>>>> >>> >> I was interested in the Jersey stuff because I’ve seen
> lots of projects use it very successfully, and if I want to implement
> something, well, there are lots of blogs and resources out there!
> >>> >> >>>>> >>> >>
> >>> >> >>>>> >>> >> Can anyone recap briefly why we dropped RESTlet? And
> what lessons learned there might apply to adopting Jersey for API support?
> Looking at https://issues.apache.org/jira/browse/SOLR-14659, it was
> partly deprecated because we were not using it to support all the API, only
> the ManagedResource ones, and
> https://issues.apache.org/jira/browse/SOLR-14766 suggests that RESTlet
> maybe was no longer being updated? One reason why we spiked out Jersey
> was because of the broad support in the Java world! Looking at how much
> work we have to do in the V2 API world, we need a much broader pool of
> developers contributing to get there!
> >>> >> >>>>> >>> >>
> >>> >> >>>>> >>> >> Related, are there specific features/aspects of our
> annotations that enable things in Solr that couldn’t be done otherwise?
> >>> >> >>>>> >>> >>
> >>> >> >>>>> >>> >> On Nov 25, 2021, at 2:12 AM, Ishan Chattopadhyaya <
> ichattopadhyaya@gmail.com> wrote:
> >>> >> >>>>> >>> >>
> >>> >> >>>>> >>> >> Is there some problem with our annotations that we
> hope to solve using third party dependencies?
> >>> >> >>>>> >>> >> By the way, we have used Restlet in the past and that
> has been a regrettable decision.
> >>> >> >>>>> >>> >>
> >>> >> >>>>> >>> >> On Thu, Nov 25, 2021 at 10:10 AM Jason Gerlowski <
> gerlowskija@gmail.com> wrote:
> >>> >> >>>>> >>> >>>
> >>> >> >>>>> >>> >>> Solr's custom annotation framework ('@Endpoint',
> '@Command', etc.) has
> >>> >> >>>>> >>> >>> cropped up a few times over the past week or two. [1]
> [2]. Having them
> >>> >> >>>>> >>> >>> on top of mind, I've been wondering - is there a
> reason we use our own
> >>> >> >>>>> >>> >>> annotations here instead of something off the shelf?
> >>> >> >>>>> >>> >>>
> >>> >> >>>>> >>> >>> What we have works well enough, but anything
> homegrown comes with more
> >>> >> >>>>> >>> >>> maintenance burden than we'd have if we used
> something off the shelf.
> >>> >> >>>>> >>> >>> There are plenty of well-used, active projects out
> there whose whole
> >>> >> >>>>> >>> >>> purpose is facilitating the whole "annotation based
> API" thing
> >>> >> >>>>> >>> >>> (Jersey, Restlet, RESTEasy, etc.) - why not use one
> of them?
> >>> >> >>>>> >>> >>>
> >>> >> >>>>> >>> >>> Does anyone know of any technical reasons why we
> can't go this route?
> >>> >> >>>>> >>> >>> Or have any subjective reasons why we shouldn't? Or
> any context on
> >>> >> >>>>> >>> >>> why we wrote our own Endpoint, Command, JsonProperty
> annotations
> >>> >> >>>>> >>> >>> originally?
> >>> >> >>>>> >>> >>>
> >>> >> >>>>> >>> >>> FWIW, Eric Pugh and I spiked out a small POC
> recently, and got
> >>> >> >>>>> >>> >>> Jersey+Jackson working for a few simple APIs without
> too much trouble.
> >>> >> >>>>> >>> >>> [3] Obviously nothing production-ready there, and
> there's still a lot
> >>> >> >>>>> >>> >>> of open questions (e.g. how would javabin be
> supported?), but we both
> >>> >> >>>>> >>> >>> came away convinced that it seemed feasible, at
> least. Best of all,
> >>> >> >>>>> >>> >>> APIs using our current homegrown annotation framework
> the switchover
> >>> >> >>>>> >>> >>> seems blessedly straightforward, and it doesn't look
> like Jersey
> >>> >> >>>>> >>> >>> (which we chose mostly arbitrarily) bloats our dist
> all that much.
> >>> >> >>>>> >>> >>>
> >>> >> >>>>> >>> >>> Curious if anyone has thoughts or context on how we
> ended up with the
> >>> >> >>>>> >>> >>> annotation setup we use today!
> >>> >> >>>>> >>> >>>
> >>> >> >>>>> >>> >>> Best,
> >>> >> >>>>> >>> >>>
> >>> >> >>>>> >>> >>> Jason
> >>> >> >>>>> >>> >>>
> >>> >> >>>>> >>> >>> [1] https://issues.apache.org/jira/browse/SOLR-15182
> (and children)
> >>> >> >>>>> >>> >>> [2]
> http://mail-archives.apache.org/mod_mbox/solr-dev/202111.mbox/%3CCABEwPvENL41Pm6%2BOmjXb6Sx5N2XjUtnbWhgKOZSrnLjWBA8tcA%40mail.gmail.com%3E
> >>> >> >>>>> >>> >>> [3]
> https://github.com/gerlowskija/solr/tree/jersey_jaxrs_jackson_solr_apis.
> >>> >> >>>>> >>> >>>
> >>> >> >>>>> >>> >>>
> ---------------------------------------------------------------------
> >>> >> >>>>> >>> >>> To unsubscribe, e-mail:
> dev-unsubscribe@solr.apache.org
> >>> >> >>>>> >>> >>> For additional commands, e-mail:
> dev-help@solr.apache.org
> >>> >> >>>>> >>> >>>
> >>> >> >>>>> >>> >>
> >>> >> >>>>> >>> >> _______________________
> >>> >> >>>>> >>> >> Eric Pugh | Founder & CEO | OpenSource Connections,
> LLC | 434.466.1467 | http://www.opensourceconnections.com | My Free/Busy
> >>> >> >>>>> >>> >> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
> >>> >> >>>>> >>> >> This e-mail and all contents, including attachments,
> is considered to be Company Confidential unless explicitly stated
> otherwise, regardless of whether attachments are marked as such.
> >>> >> >>>>> >>> >>
> >>> >> >>>>> >>>
> >>> >> >>>>> >>>
> ---------------------------------------------------------------------
> >>> >> >>>>> >>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> >>> >> >>>>> >>> For additional commands, e-mail: dev-help@solr.apache.org
> >>> >> >>>>> >>>
> >>> >> >>>>> >>
> >>> >> >>>>> >>
> >>> >> >>>>> >> --
> >>> >> >>>>> >> -----------------------------------------------------
> >>> >> >>>>> >> Noble Paul
> >>> >> >>>>> >
> >>> >> >>>>> >
> >>> >> >>>>> >
> >>> >> >>>>> > --
> >>> >> >>>>> > -----------------------------------------------------
> >>> >> >>>>> > Noble Paul
> >>> >> >>>>>
> >>> >> >>>>>
> ---------------------------------------------------------------------
> >>> >> >>>>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> >>> >> >>>>> For additional commands, e-mail: dev-help@solr.apache.org
> >>> >> >>>>>
> >>> >> >>>>
> >>> >> >>>>
> >>> >> >>>> --
> >>> >> >>>> http://www.needhamsoftware.com (work)
> >>> >> >>>> http://www.the111shift.com (play)
> >>> >> >>>
> >>> >> >>>
> >>> >> >>>
> >>> >> >>> --
> >>> >> >>> -----------------------------------------------------
> >>> >> >>> Noble Paul
> >>> >> >>
> >>> >> >>
> >>> >> >>
> >>> >> >> --
> >>> >> >> -----------------------------------------------------
> >>> >> >> Noble Paul
> >>> >>
> >>> >>
> ---------------------------------------------------------------------
> >>> >> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> >>> >> For additional commands, e-mail: dev-help@solr.apache.org
> >>> >>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> >>> For additional commands, e-mail: dev-help@solr.apache.org
> >>>
> >>
> >>
> >> --
> >> -----------------------------------------------------
> >> Noble Paul
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org
>
>
--
-----------------------------------------------------
Noble Paul
Re: JAX-RS APIs in Solr
Posted by Jason Gerlowski <ge...@gmail.com>.
> 99% of Solr users never see the java code that implements an API let alone writing a new API.
We may be talking past each other here a bit Noble. The JAX-RS
benefits I've been trying to describe are to benefit us developers,
not Solr users. I agree that 99% of our users won't know or care
about it, but who cares. The goal is to make _our_ lives as
maintainers easier going into the future. Jan described it in a word
better than I've been struggling to: it's about "modernization".
Best,
Jason
On Thu, Dec 2, 2021 at 3:34 AM Jan Høydahl <ja...@cominvent.com> wrote:
>
> Agree, Ishan.
>
> Let's welcome and cheer initiatives to modernize this old codebase.
> Diving in deep and un-tangling DispatchFilter like Gus is attempting is brave.
> And testing JAX-RS to modernize APIs and make the codebas easier to grok is also not for the faint hearted.
>
> I hope both initiatives end up as a breath of fresh air for Solr developers. And perhaps users may get their hands on auto-generated OpenAPI docs for Solr as well!
> Theoretically foundations like this may make it feasible in the future to auto-generate client code from the spec,so that full Admin API client support in e.g. Go or Rust may be achievable without hand-coding. But that is still far away :)
>
> Jan
>
> 2. des. 2021 kl. 05:52 skrev Ishan Chattopadhyaya <ic...@gmail.com>:
>
> > We are discussing this as if moving to an external framework is going to be smooth. It's not.
>
> Noble, we won't know before someone has given it a try. My hunch suggests it won't be smooth, but we can decide on that once we see a concrete solution that 1) works well for all existing APIs (per core or global) without hacks/workarounds, and 2) has no security implications, and 3) doesn't clutter our codebase, and 4) doesn't affect our runtime performance.
>
> On Thu, Dec 2, 2021 at 4:22 AM Noble Paul <no...@gmail.com> wrote:
>>
>>
>>
>> On Thu, Dec 2, 2021 at 2:46 AM Jason Gerlowski <ge...@gmail.com> wrote:
>>>
>>> > There are no known issues with the current system
>>>
>>> Maybe you're using hyperbole to emphasise a point, but let's steer
>>> this discussion away from straw-men and caricatures. I mentioned
>>> specific known issues in a previous reply as a direct response to your
>>> question about them. The incomplete support for data types
>>> (SOLR-15619 and SOLR-15796). The inflexible PermissionNameProvider
>>> integration (SOLR-15823). If you consider those "minor", fair enough,
>>> and you're welcome to that opinion. But saying that they don't exist
>>> is a mischaracterization that wastes time and muddies water.
>>
>>
>> We are discussing this as if moving to an external framework is going to be smooth. It's not. I have seen how hard the Restlet integration was and how it stood out like a sore thumb. The amount of code required to integrate Solr APIs into restlet components was 10x more than the amount of code written for the new framework and it still was incomplete . Integrating anything to Solr is not simple and it's going to be an ongoing effort to ensure that all of the functionalities of a SolrRequestHandler are exposed to the new framework. Every single component that we have today has ongoing improvements and bug fixes. Using that as an excuse to do a complete rewrite is probably unwise.
>>
>>
>>>
>>> > Solr is not a general purpose web server where people write APIs every day
>>>
>>> Totally agree - 99% of the time someone opens an API file in Solr,
>>> they're reading, not writing. So if JAX-RS didn't help that case, it
>>> wouldn't be worth the effort of switching. But IMO it helps that
>>> "read" case a lot.
>>
>>
>> 99% of Solr users never see the java code that implements an API let alone writing a new API. They wouldn't care if an API is implemented using a Servlet API, a custom framework or something else. They only care about the input and output. New end points are rare.
>>>
>>>
>>> JAX-RS defined APIs expose inputs as strongly-typed parameters
>>> (compare to the weakly-typed SolrQueryRequest/SolrParams used today).
>>> JAX-RS enumerates every input to the API in one place. JAX-RS APIs
>>> can return any serializable type as a response (compare to the
>>> near-omnipresent NamedList today).
>>>
>>> (The annotation framework's PayloadObj class gets us a bit of this
>>> benefit, but not all of it and only on POST requests afaik.)
>>>
>>> This is all nice stuff at API-write time, but where it really helps is
>>> in making the code clearer and simpler whenever someone goes to read
>>> it later. No wondering what parameters an API actually takes. No
>>> puzzling out what a bunch of NamedList operations spread throughout
>>> API execution actually produce in the end. etc.
>>>
>>> > The only reason why anyone would need to use this framework is to write a custom request handler.
>>>
>>> ...except for the majority of people who come in contact with the
>>> framework by reading an existing Solr API? This ties into your point
>>> above about APIs being written infrequently but read all the time.
>>> Improving readability is the big gain here, and I think JAX-RS offers
>>> real improvements in that regard.
>>
>>
>> We had no framework for the last 15+ years and V1 API (the only actually used API) still uses no frameworks. Even the annotation framework is not used much even though readability is a lot better in the annotation based system. Readability has not been an overriding concern for V1 APIs for sure
>>
>>> Also, to clarify, I'm not asserting that no one configures custom
>>> requestHandlers. I was wondering aloud how common it was, as an open
>>> question.
>>>
>> Your point IS right. IRL, most of our users do not write request handlers . They write search components , query parsers and other plugins. Because we are a search engine and not a web server. I have never seen a user complaining about the V1 API of writing a request handler. They may actually complain about how hard it is to deploy one. Our users are much smarter than we give them credit for.
>>
>>
>>>
>>> > We did not fork Jackson. We are using Jackson itself.
>>>
>>> Ah, maybe you're right, "fork" is probably an unfair word. I looked
>>> for the serialization code, and I'll admit there is less than I
>>> thought involved here and I misunderstood how it works. So that's my
>>> bad. The JSON serialization/deserialization discussion is prob a
>>> digression from the core contention about whether JAX-RS would be
>>> helpful. I was curious about your earlier response and couldn't
>>> resist following up here but I should've done so elsewhere to avoid
>>> the distraction.
>>>
>>> Best,
>>>
>>>
>>> Jason
>>>
>>> On Tue, Nov 30, 2021 at 4:12 PM Noble Paul <no...@gmail.com> wrote:
>>> >
>>> > The annotation framework is just a single class and it is made to work well with Solr instead of changing Solr to suit the needs of some external framework. There are no known issues with the current system and there are unknown issues that you'll face introducing a new framework.
>>> >
>>> > Solr is not a general purpose web server where people write APIs every day. It's a finished product where people use the publicly available APIs . We should focus our efforts on making our APIs work well .
>>> >
>>> > On Wed, Dec 1, 2021, 2:14 AM Jason Gerlowski <ge...@gmail.com> wrote:
>>> >>
>>> >> > We need to make an API work on a per core basis and cores can come up and go down randomly
>>> >>
>>> >> Ah, yeah. I'll admit I was overlooking some of the wrinkles around
>>> >> custom request handlers etc. That is a problem for switching to some
>>> >> off-the-shelf framework for sure. It may still be possible, but it's
>>> >> definitely a big hurdle. I wonder how commonly users still take
>>> >> advantage of the custom requestHandler stuff these days? I haven't
>>> >> run across many customers who used them extensively, but maybe that's
>>> >> self-selecting in some way?
>>> >
>>> >
>>> > The only reason why anyone would need to use this framework is to write a custom request handler. You're also saying nobody writes custom request handlers. So, what problem are you trying to solve if you think nobody uses them?
>>> >
>>> >
>>> >>
>>> >> > If SolrJ has a Jackson dependency, it can have a conflict (different versions) with the Jackson used by the client app
>>> >>
>>> >> SolrJ has a handful of dependencies - is there a reason we take this
>>> >> approach with Jackson, but not with any of SolrJ's other deps?
>>> >> (apache-httpcomponents, apache-commons, zookeeper-client, etc.)
>>> >>
>>> >> Granted, I get that we want SolrJ to be as painless as possible for
>>> >> its users, and jar-conflicts are a part of that and really suck. But
>>> >> there are good tools out there for handling jar conflicts, and
>>> >> reimplementing chunks of library functionality just to avoid a
>>> >> gradle-dep strikes me as unsustainable from a maintenance/community
>>> >> perspective. Not to mention that forking/mimicking Jackson code
>>> >> deprives our users of any efficiency/security improvements that
>>> >> Jackson might get tomorrow or next week or next year.
>>> >
>>> >
>>> > We did not fork Jackson. We are using Jackson itself. Jackson supports custom annotations. There was a huge discussion why we should do it this way.
>>> >>
>>> >>
>>> >> Unless there's some reason specific to Jackson I guess I just don't
>>> >> get it. But if I'm the minority opinion on that, fair enough I guess.
>>> >>
>>> >> On Tue, Nov 30, 2021 at 5:05 AM Noble Paul <no...@gmail.com> wrote:
>>> >> >
>>> >> > The @JsonProperty annotations are added because it's a dependency in SolrJ as well
>>> >> > If SolrJ has a Jackson dependency, it can have a conflict (different versions) with the Jackson used by the client app.
>>> >> >
>>> >> > On Tue, Nov 30, 2021, 5:21 PM Noble Paul <no...@gmail.com> wrote:
>>> >> >>
>>> >> >> Also keep in mind that the same endpoint can be accessed with a core name and a collection name prefixes.
>>> >> >>
>>> >> >> On Tue, Nov 30, 2021 at 3:55 PM Noble Paul <no...@gmail.com> wrote:
>>> >> >>>
>>> >> >>> True Gus, Almost every framework works outside of SolrDispatchFilter+ HttpSolrCall. A lot of our initializations occur there.
>>> >> >>>
>>> >> >>> We need to make an API work on a per core basis and cores can come up and go down randomly. So we need to register these endpoints on a core.
>>> >> >>>
>>> >> >>> I'm not sure if any framework can achieve the same.
>>> >> >>>
>>> >> >>> On Tue, Nov 30, 2021 at 5:39 AM Gus Heck <gu...@gmail.com> wrote:
>>> >> >>>>
>>> >> >>>> IIRC last time I looked restlet had the unsavory property of existing outside of the SolrDispatchFilter, unlike everything else which made for special cases because several things that probably ought to be their own siervlet filters are glommed into SolrDispatchFilter, like security, tracing and MDC setup/teardown per request. Restlet wouldn't be so bad if one could just wrap such filters around it too...
>>> >> >>>>
>>> >> >>>> On Mon, Nov 29, 2021 at 9:42 AM Jason Gerlowski <ge...@gmail.com> wrote:
>>> >> >>>>>
>>> >> >>>>> > These are minor improvements compared to a full rewrite of the entire framework
>>> >> >>>>>
>>> >> >>>>> If you think data type support is minor, fair enough. But to clarify
>>> >> >>>>> I'm not suggesting a rewrite - I'm suggesting using something that
>>> >> >>>>> already exists off the shelf. Jersey (e.g.) itself provides the
>>> >> >>>>> framework - there would be no "rewrite".
>>> >> >>>>>
>>> >> >>>>> re: past restlet use
>>> >> >>>>>
>>> >> >>>>> > It was not playing well with our security framework. The framework was not working well with Solr APIs
>>> >> >>>>>
>>> >> >>>>> Ah, very interesting! Security isn't something Eric or I tackled in
>>> >> >>>>> our little spike branch, but it's definitely a concern. Do you
>>> >> >>>>> remember the specific concerns? Or recall where any of the discussion
>>> >> >>>>> around this happened?
>>> >> >>>>>
>>> >> >>>>> Without the context of that past discussion, it seems like the
>>> >> >>>>> "PermissionNameProvider" interface could be implemented just as well
>>> >> >>>>> by a class with (e.g.) Jersey annotations as one with our own custom
>>> >> >>>>> annotations. Certainly there'd need to be some
>>> >> >>>>> RuleBasedAuthorizationPlugin changes or other integration code, but
>>> >> >>>>> nothing that feels insurmountable.
>>> >> >>>>>
>>> >> >>>>> Maybe I can try spiking it out soon and find the issues myself, but
>>> >> >>>>> it'd be much easier if someone happens to remember and can save me the
>>> >> >>>>> trouble :-p
>>> >> >>>>>
>>> >> >>>>> Best,
>>> >> >>>>>
>>> >> >>>>> Jason
>>> >> >>>>>
>>> >> >>>>> On Fri, Nov 26, 2021 at 12:05 AM Noble Paul <no...@gmail.com> wrote:
>>> >> >>>>> >
>>> >> >>>>> > The Annotations framework was written after playing with other frameworks. There were many shortcomings which were hard to overcome.
>>> >> >>>>> >
>>> >> >>>>> > The best example is a per collection API . How do you register an endpoint for a collection/core ?
>>> >> >>>>> >
>>> >> >>>>> > On Fri, Nov 26, 2021 at 3:42 PM Noble Paul <no...@gmail.com> wrote:
>>> >> >>>>> >>
>>> >> >>>>> >>
>>> >> >>>>> >>
>>> >> >>>>> >> On Fri, Nov 26, 2021 at 1:03 AM Jason Gerlowski <ge...@gmail.com> wrote:
>>> >> >>>>> >>>
>>> >> >>>>> >>> > Is there some problem with our annotations that we hope to solve using third party dependencies?
>>> >> >>>>> >>>
>>> >> >>>>> >>> I guess so yeah. Third-party deps are just fuller, more robust
>>> >> >>>>> >>> solutions, whereas our annotations still need support added now and
>>> >> >>>>> >>> then for even primitive data types like "long" (see SOLR-15619).
>>> >> >>>>> >>
>>> >> >>>>> >> These are minor improvements compared to a full rewrite of the entire framework
>>> >> >>>>> >>
>>> >> >>>>> >>
>>> >> >>>>> >>>
>>> >> >>>>> >>> Every JIRA spent doing basic stuff like that is time away from
>>> >> >>>>> >>> improving Solr in some other way.
>>> >> >>>>> >>>
>>> >> >>>>> >>> So there are feature-gap/capabilities arguments for moving to a
>>> >> >>>>> >>> third-party dep, sure. But, even if our annotations did everything
>>> >> >>>>> >>> Jersey+Jackson do today, I think switching would still be worth it.
>>> >> >>>>> >>> Every LOC in our code base brings along with it some maintenance cost:
>>> >> >>>>> >>> it might have bugs, needs tested, takes time for new contributors to
>>> >> >>>>> >>> "grok", etc. Using off-the-shelf here would nuke a whole bunch of
>>> >> >>>>> >>> that. If off-the-shelf is available for some given functionality, we
>>> >> >>>>> >>> should need a compelling reason to NOT use it.
>>> >> >>>>> >>>
>>> >> >>>>> >>> Lastly, I think there's an "approachability" argument for using
>>> >> >>>>> >>> off-the-shelf. Thousands of developers out there are familiar with
>>> >> >>>>> >>> (e.g.) Jersey, compared to maybe 15 or 20 (in the world) familiar with
>>> >> >>>>> >>> Solr's custom annotations. Using a well-known technology like Jersey
>>> >> >>>>> >>> would make Solr all the easier to approach and contribute to for that
>>> >> >>>>> >>> pool of developers.
>>> >> >>>>> >>>
>>> >> >>>>> >>> > By the way, we have used Restlet in the past and that has been a regrettable decision.
>>> >> >>>>> >>
>>> >> >>>>> >>
>>> >> >>>>> >>>
>>> >> >>>>> >>> Ah, yeah, that's just the context I'm missing. Anyone have a pointer
>>> >> >>>>> >>> to related discussions, or remember what made this "regrettable"? All
>>> >> >>>>> >>> the theoretical benefits in the world don't matter much if we've
>>> >> >>>>> >>> already tried something like this in the past and decided against it.
>>> >> >>>>> >>
>>> >> >>>>> >>
>>> >> >>>>> >> It was not playing well with our security framework. The framework was not working well with Solr APIs
>>> >> >>>>> >>
>>> >> >>>>> >>>
>>> >> >>>>> >>> (Unrelated - Happy Thanksgiving all!)
>>> >> >>>>> >>>
>>> >> >>>>> >>> Best,
>>> >> >>>>> >>>
>>> >> >>>>> >>> Jason
>>> >> >>>>> >>>
>>> >> >>>>> >>> On Thu, Nov 25, 2021 at 7:32 AM Noble Paul <no...@gmail.com> wrote:
>>> >> >>>>> >>> >
>>> >> >>>>> >>> > Have you gone through an API written using the @EndPoint annotation?
>>> >> >>>>> >>> >
>>> >> >>>>> >>> > I strongly recommend that you do
>>> >> >>>>> >>> >
>>> >> >>>>> >>> > On Thu, Nov 25, 2021, 11:30 PM Eric Pugh <ep...@opensourceconnections.com> wrote:
>>> >> >>>>> >>> >>
>>> >> >>>>> >>> >> I have found our V2 API code to be very impenetrable to understand. Part of it is how the code is intertwined with support for V1, however it’s also because there aren’t really resources to go look at to understand how it should work! Maintaining the API should be very simple work, as they just exist as a translation. The home grown stuff may make sense if you are a super knowledgable Solr developer, but if you are just a new person, it’s a lot harder to contribute.
>>> >> >>>>> >>> >>
>>> >> >>>>> >>> >> I was interested in the Jersey stuff because I’ve seen lots of projects use it very successfully, and if I want to implement something, well, there are lots of blogs and resources out there!
>>> >> >>>>> >>> >>
>>> >> >>>>> >>> >> Can anyone recap briefly why we dropped RESTlet? And what lessons learned there might apply to adopting Jersey for API support? Looking at https://issues.apache.org/jira/browse/SOLR-14659, it was partly deprecated because we were not using it to support all the API, only the ManagedResource ones, and https://issues.apache.org/jira/browse/SOLR-14766 suggests that RESTlet maybe was no longer being updated? One reason why we spiked out Jersey was because of the broad support in the Java world! Looking at how much work we have to do in the V2 API world, we need a much broader pool of developers contributing to get there!
>>> >> >>>>> >>> >>
>>> >> >>>>> >>> >> Related, are there specific features/aspects of our annotations that enable things in Solr that couldn’t be done otherwise?
>>> >> >>>>> >>> >>
>>> >> >>>>> >>> >> On Nov 25, 2021, at 2:12 AM, Ishan Chattopadhyaya <ic...@gmail.com> wrote:
>>> >> >>>>> >>> >>
>>> >> >>>>> >>> >> Is there some problem with our annotations that we hope to solve using third party dependencies?
>>> >> >>>>> >>> >> By the way, we have used Restlet in the past and that has been a regrettable decision.
>>> >> >>>>> >>> >>
>>> >> >>>>> >>> >> On Thu, Nov 25, 2021 at 10:10 AM Jason Gerlowski <ge...@gmail.com> wrote:
>>> >> >>>>> >>> >>>
>>> >> >>>>> >>> >>> Solr's custom annotation framework ('@Endpoint', '@Command', etc.) has
>>> >> >>>>> >>> >>> cropped up a few times over the past week or two. [1] [2]. Having them
>>> >> >>>>> >>> >>> on top of mind, I've been wondering - is there a reason we use our own
>>> >> >>>>> >>> >>> annotations here instead of something off the shelf?
>>> >> >>>>> >>> >>>
>>> >> >>>>> >>> >>> What we have works well enough, but anything homegrown comes with more
>>> >> >>>>> >>> >>> maintenance burden than we'd have if we used something off the shelf.
>>> >> >>>>> >>> >>> There are plenty of well-used, active projects out there whose whole
>>> >> >>>>> >>> >>> purpose is facilitating the whole "annotation based API" thing
>>> >> >>>>> >>> >>> (Jersey, Restlet, RESTEasy, etc.) - why not use one of them?
>>> >> >>>>> >>> >>>
>>> >> >>>>> >>> >>> Does anyone know of any technical reasons why we can't go this route?
>>> >> >>>>> >>> >>> Or have any subjective reasons why we shouldn't? Or any context on
>>> >> >>>>> >>> >>> why we wrote our own Endpoint, Command, JsonProperty annotations
>>> >> >>>>> >>> >>> originally?
>>> >> >>>>> >>> >>>
>>> >> >>>>> >>> >>> FWIW, Eric Pugh and I spiked out a small POC recently, and got
>>> >> >>>>> >>> >>> Jersey+Jackson working for a few simple APIs without too much trouble.
>>> >> >>>>> >>> >>> [3] Obviously nothing production-ready there, and there's still a lot
>>> >> >>>>> >>> >>> of open questions (e.g. how would javabin be supported?), but we both
>>> >> >>>>> >>> >>> came away convinced that it seemed feasible, at least. Best of all,
>>> >> >>>>> >>> >>> APIs using our current homegrown annotation framework the switchover
>>> >> >>>>> >>> >>> seems blessedly straightforward, and it doesn't look like Jersey
>>> >> >>>>> >>> >>> (which we chose mostly arbitrarily) bloats our dist all that much.
>>> >> >>>>> >>> >>>
>>> >> >>>>> >>> >>> Curious if anyone has thoughts or context on how we ended up with the
>>> >> >>>>> >>> >>> annotation setup we use today!
>>> >> >>>>> >>> >>>
>>> >> >>>>> >>> >>> Best,
>>> >> >>>>> >>> >>>
>>> >> >>>>> >>> >>> Jason
>>> >> >>>>> >>> >>>
>>> >> >>>>> >>> >>> [1] https://issues.apache.org/jira/browse/SOLR-15182 (and children)
>>> >> >>>>> >>> >>> [2] http://mail-archives.apache.org/mod_mbox/solr-dev/202111.mbox/%3CCABEwPvENL41Pm6%2BOmjXb6Sx5N2XjUtnbWhgKOZSrnLjWBA8tcA%40mail.gmail.com%3E
>>> >> >>>>> >>> >>> [3] https://github.com/gerlowskija/solr/tree/jersey_jaxrs_jackson_solr_apis.
>>> >> >>>>> >>> >>>
>>> >> >>>>> >>> >>> ---------------------------------------------------------------------
>>> >> >>>>> >>> >>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>>> >> >>>>> >>> >>> For additional commands, e-mail: dev-help@solr.apache.org
>>> >> >>>>> >>> >>>
>>> >> >>>>> >>> >>
>>> >> >>>>> >>> >> _______________________
>>> >> >>>>> >>> >> Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | http://www.opensourceconnections.com | My Free/Busy
>>> >> >>>>> >>> >> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
>>> >> >>>>> >>> >> This e-mail and all contents, including attachments, is considered to be Company Confidential unless explicitly stated otherwise, regardless of whether attachments are marked as such.
>>> >> >>>>> >>> >>
>>> >> >>>>> >>>
>>> >> >>>>> >>> ---------------------------------------------------------------------
>>> >> >>>>> >>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>>> >> >>>>> >>> For additional commands, e-mail: dev-help@solr.apache.org
>>> >> >>>>> >>>
>>> >> >>>>> >>
>>> >> >>>>> >>
>>> >> >>>>> >> --
>>> >> >>>>> >> -----------------------------------------------------
>>> >> >>>>> >> Noble Paul
>>> >> >>>>> >
>>> >> >>>>> >
>>> >> >>>>> >
>>> >> >>>>> > --
>>> >> >>>>> > -----------------------------------------------------
>>> >> >>>>> > Noble Paul
>>> >> >>>>>
>>> >> >>>>> ---------------------------------------------------------------------
>>> >> >>>>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>>> >> >>>>> For additional commands, e-mail: dev-help@solr.apache.org
>>> >> >>>>>
>>> >> >>>>
>>> >> >>>>
>>> >> >>>> --
>>> >> >>>> http://www.needhamsoftware.com (work)
>>> >> >>>> http://www.the111shift.com (play)
>>> >> >>>
>>> >> >>>
>>> >> >>>
>>> >> >>> --
>>> >> >>> -----------------------------------------------------
>>> >> >>> Noble Paul
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >> --
>>> >> >> -----------------------------------------------------
>>> >> >> Noble Paul
>>> >>
>>> >> ---------------------------------------------------------------------
>>> >> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>>> >> For additional commands, e-mail: dev-help@solr.apache.org
>>> >>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>>> For additional commands, e-mail: dev-help@solr.apache.org
>>>
>>
>>
>> --
>> -----------------------------------------------------
>> Noble Paul
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: JAX-RS APIs in Solr
Posted by Jan Høydahl <ja...@cominvent.com>.
Agree, Ishan.
Let's welcome and cheer initiatives to modernize this old codebase.
Diving in deep and un-tangling DispatchFilter like Gus is attempting is brave.
And testing JAX-RS to modernize APIs and make the codebas easier to grok is also not for the faint hearted.
I hope both initiatives end up as a breath of fresh air for Solr developers. And perhaps users may get their hands on auto-generated OpenAPI docs for Solr as well!
Theoretically foundations like this may make it feasible in the future to auto-generate client code from the spec,so that full Admin API client support in e.g. Go or Rust may be achievable without hand-coding. But that is still far away :)
Jan
> 2. des. 2021 kl. 05:52 skrev Ishan Chattopadhyaya <ic...@gmail.com>:
>
> > We are discussing this as if moving to an external framework is going to be smooth. It's not.
>
> Noble, we won't know before someone has given it a try. My hunch suggests it won't be smooth, but we can decide on that once we see a concrete solution that 1) works well for all existing APIs (per core or global) without hacks/workarounds, and 2) has no security implications, and 3) doesn't clutter our codebase, and 4) doesn't affect our runtime performance.
>
> On Thu, Dec 2, 2021 at 4:22 AM Noble Paul <noble.paul@gmail.com <ma...@gmail.com>> wrote:
>
>
> On Thu, Dec 2, 2021 at 2:46 AM Jason Gerlowski <gerlowskija@gmail.com <ma...@gmail.com>> wrote:
> > There are no known issues with the current system
>
> Maybe you're using hyperbole to emphasise a point, but let's steer
> this discussion away from straw-men and caricatures. I mentioned
> specific known issues in a previous reply as a direct response to your
> question about them. The incomplete support for data types
> (SOLR-15619 and SOLR-15796). The inflexible PermissionNameProvider
> integration (SOLR-15823). If you consider those "minor", fair enough,
> and you're welcome to that opinion. But saying that they don't exist
> is a mischaracterization that wastes time and muddies water.
>
> We are discussing this as if moving to an external framework is going to be smooth. It's not. I have seen how hard the Restlet integration was and how it stood out like a sore thumb. The amount of code required to integrate Solr APIs into restlet components was 10x more than the amount of code written for the new framework and it still was incomplete . Integrating anything to Solr is not simple and it's going to be an ongoing effort to ensure that all of the functionalities of a SolrRequestHandler are exposed to the new framework. Every single component that we have today has ongoing improvements and bug fixes. Using that as an excuse to do a complete rewrite is probably unwise.
>
>
>
> > Solr is not a general purpose web server where people write APIs every day
>
> Totally agree - 99% of the time someone opens an API file in Solr,
> they're reading, not writing. So if JAX-RS didn't help that case, it
> wouldn't be worth the effort of switching. But IMO it helps that
> "read" case a lot.
>
> 99% of Solr users never see the java code that implements an API let alone writing a new API. They wouldn't care if an API is implemented using a Servlet API, a custom framework or something else. They only care about the input and output. New end points are rare.
>
> JAX-RS defined APIs expose inputs as strongly-typed parameters
> (compare to the weakly-typed SolrQueryRequest/SolrParams used today).
> JAX-RS enumerates every input to the API in one place. JAX-RS APIs
> can return any serializable type as a response (compare to the
> near-omnipresent NamedList today).
>
> (The annotation framework's PayloadObj class gets us a bit of this
> benefit, but not all of it and only on POST requests afaik.)
>
> This is all nice stuff at API-write time, but where it really helps is
> in making the code clearer and simpler whenever someone goes to read
> it later. No wondering what parameters an API actually takes. No
> puzzling out what a bunch of NamedList operations spread throughout
> API execution actually produce in the end. etc.
>
> > The only reason why anyone would need to use this framework is to write a custom request handler.
>
> ...except for the majority of people who come in contact with the
> framework by reading an existing Solr API? This ties into your point
> above about APIs being written infrequently but read all the time.
> Improving readability is the big gain here, and I think JAX-RS offers
> real improvements in that regard.
>
> We had no framework for the last 15+ years and V1 API (the only actually used API) still uses no frameworks. Even the annotation framework is not used much even though readability is a lot better in the annotation based system. Readability has not been an overriding concern for V1 APIs for sure
>
> Also, to clarify, I'm not asserting that no one configures custom
> requestHandlers. I was wondering aloud how common it was, as an open
> question.
>
> Your point IS right. IRL, most of our users do not write request handlers . They write search components , query parsers and other plugins. Because we are a search engine and not a web server. I have never seen a user complaining about the V1 API of writing a request handler. They may actually complain about how hard it is to deploy one. Our users are much smarter than we give them credit for.
>
>
> > We did not fork Jackson. We are using Jackson itself.
>
> Ah, maybe you're right, "fork" is probably an unfair word. I looked
> for the serialization code, and I'll admit there is less than I
> thought involved here and I misunderstood how it works. So that's my
> bad. The JSON serialization/deserialization discussion is prob a
> digression from the core contention about whether JAX-RS would be
> helpful. I was curious about your earlier response and couldn't
> resist following up here but I should've done so elsewhere to avoid
> the distraction.
>
> Best,
>
>
> Jason
>
> On Tue, Nov 30, 2021 at 4:12 PM Noble Paul <noble.paul@gmail.com <ma...@gmail.com>> wrote:
> >
> > The annotation framework is just a single class and it is made to work well with Solr instead of changing Solr to suit the needs of some external framework. There are no known issues with the current system and there are unknown issues that you'll face introducing a new framework.
> >
> > Solr is not a general purpose web server where people write APIs every day. It's a finished product where people use the publicly available APIs . We should focus our efforts on making our APIs work well .
> >
> > On Wed, Dec 1, 2021, 2:14 AM Jason Gerlowski <gerlowskija@gmail.com <ma...@gmail.com>> wrote:
> >>
> >> > We need to make an API work on a per core basis and cores can come up and go down randomly
> >>
> >> Ah, yeah. I'll admit I was overlooking some of the wrinkles around
> >> custom request handlers etc. That is a problem for switching to some
> >> off-the-shelf framework for sure. It may still be possible, but it's
> >> definitely a big hurdle. I wonder how commonly users still take
> >> advantage of the custom requestHandler stuff these days? I haven't
> >> run across many customers who used them extensively, but maybe that's
> >> self-selecting in some way?
> >
> >
> > The only reason why anyone would need to use this framework is to write a custom request handler. You're also saying nobody writes custom request handlers. So, what problem are you trying to solve if you think nobody uses them?
> >
> >
> >>
> >> > If SolrJ has a Jackson dependency, it can have a conflict (different versions) with the Jackson used by the client app
> >>
> >> SolrJ has a handful of dependencies - is there a reason we take this
> >> approach with Jackson, but not with any of SolrJ's other deps?
> >> (apache-httpcomponents, apache-commons, zookeeper-client, etc.)
> >>
> >> Granted, I get that we want SolrJ to be as painless as possible for
> >> its users, and jar-conflicts are a part of that and really suck. But
> >> there are good tools out there for handling jar conflicts, and
> >> reimplementing chunks of library functionality just to avoid a
> >> gradle-dep strikes me as unsustainable from a maintenance/community
> >> perspective. Not to mention that forking/mimicking Jackson code
> >> deprives our users of any efficiency/security improvements that
> >> Jackson might get tomorrow or next week or next year.
> >
> >
> > We did not fork Jackson. We are using Jackson itself. Jackson supports custom annotations. There was a huge discussion why we should do it this way.
> >>
> >>
> >> Unless there's some reason specific to Jackson I guess I just don't
> >> get it. But if I'm the minority opinion on that, fair enough I guess.
> >>
> >> On Tue, Nov 30, 2021 at 5:05 AM Noble Paul <noble.paul@gmail.com <ma...@gmail.com>> wrote:
> >> >
> >> > The @JsonProperty annotations are added because it's a dependency in SolrJ as well
> >> > If SolrJ has a Jackson dependency, it can have a conflict (different versions) with the Jackson used by the client app.
> >> >
> >> > On Tue, Nov 30, 2021, 5:21 PM Noble Paul <noble.paul@gmail.com <ma...@gmail.com>> wrote:
> >> >>
> >> >> Also keep in mind that the same endpoint can be accessed with a core name and a collection name prefixes.
> >> >>
> >> >> On Tue, Nov 30, 2021 at 3:55 PM Noble Paul <noble.paul@gmail.com <ma...@gmail.com>> wrote:
> >> >>>
> >> >>> True Gus, Almost every framework works outside of SolrDispatchFilter+ HttpSolrCall. A lot of our initializations occur there.
> >> >>>
> >> >>> We need to make an API work on a per core basis and cores can come up and go down randomly. So we need to register these endpoints on a core.
> >> >>>
> >> >>> I'm not sure if any framework can achieve the same.
> >> >>>
> >> >>> On Tue, Nov 30, 2021 at 5:39 AM Gus Heck <gus.heck@gmail.com <ma...@gmail.com>> wrote:
> >> >>>>
> >> >>>> IIRC last time I looked restlet had the unsavory property of existing outside of the SolrDispatchFilter, unlike everything else which made for special cases because several things that probably ought to be their own siervlet filters are glommed into SolrDispatchFilter, like security, tracing and MDC setup/teardown per request. Restlet wouldn't be so bad if one could just wrap such filters around it too...
> >> >>>>
> >> >>>> On Mon, Nov 29, 2021 at 9:42 AM Jason Gerlowski <gerlowskija@gmail.com <ma...@gmail.com>> wrote:
> >> >>>>>
> >> >>>>> > These are minor improvements compared to a full rewrite of the entire framework
> >> >>>>>
> >> >>>>> If you think data type support is minor, fair enough. But to clarify
> >> >>>>> I'm not suggesting a rewrite - I'm suggesting using something that
> >> >>>>> already exists off the shelf. Jersey (e.g.) itself provides the
> >> >>>>> framework - there would be no "rewrite".
> >> >>>>>
> >> >>>>> re: past restlet use
> >> >>>>>
> >> >>>>> > It was not playing well with our security framework. The framework was not working well with Solr APIs
> >> >>>>>
> >> >>>>> Ah, very interesting! Security isn't something Eric or I tackled in
> >> >>>>> our little spike branch, but it's definitely a concern. Do you
> >> >>>>> remember the specific concerns? Or recall where any of the discussion
> >> >>>>> around this happened?
> >> >>>>>
> >> >>>>> Without the context of that past discussion, it seems like the
> >> >>>>> "PermissionNameProvider" interface could be implemented just as well
> >> >>>>> by a class with (e.g.) Jersey annotations as one with our own custom
> >> >>>>> annotations. Certainly there'd need to be some
> >> >>>>> RuleBasedAuthorizationPlugin changes or other integration code, but
> >> >>>>> nothing that feels insurmountable.
> >> >>>>>
> >> >>>>> Maybe I can try spiking it out soon and find the issues myself, but
> >> >>>>> it'd be much easier if someone happens to remember and can save me the
> >> >>>>> trouble :-p
> >> >>>>>
> >> >>>>> Best,
> >> >>>>>
> >> >>>>> Jason
> >> >>>>>
> >> >>>>> On Fri, Nov 26, 2021 at 12:05 AM Noble Paul <noble.paul@gmail.com <ma...@gmail.com>> wrote:
> >> >>>>> >
> >> >>>>> > The Annotations framework was written after playing with other frameworks. There were many shortcomings which were hard to overcome.
> >> >>>>> >
> >> >>>>> > The best example is a per collection API . How do you register an endpoint for a collection/core ?
> >> >>>>> >
> >> >>>>> > On Fri, Nov 26, 2021 at 3:42 PM Noble Paul <noble.paul@gmail.com <ma...@gmail.com>> wrote:
> >> >>>>> >>
> >> >>>>> >>
> >> >>>>> >>
> >> >>>>> >> On Fri, Nov 26, 2021 at 1:03 AM Jason Gerlowski <gerlowskija@gmail.com <ma...@gmail.com>> wrote:
> >> >>>>> >>>
> >> >>>>> >>> > Is there some problem with our annotations that we hope to solve using third party dependencies?
> >> >>>>> >>>
> >> >>>>> >>> I guess so yeah. Third-party deps are just fuller, more robust
> >> >>>>> >>> solutions, whereas our annotations still need support added now and
> >> >>>>> >>> then for even primitive data types like "long" (see SOLR-15619).
> >> >>>>> >>
> >> >>>>> >> These are minor improvements compared to a full rewrite of the entire framework
> >> >>>>> >>
> >> >>>>> >>
> >> >>>>> >>>
> >> >>>>> >>> Every JIRA spent doing basic stuff like that is time away from
> >> >>>>> >>> improving Solr in some other way.
> >> >>>>> >>>
> >> >>>>> >>> So there are feature-gap/capabilities arguments for moving to a
> >> >>>>> >>> third-party dep, sure. But, even if our annotations did everything
> >> >>>>> >>> Jersey+Jackson do today, I think switching would still be worth it.
> >> >>>>> >>> Every LOC in our code base brings along with it some maintenance cost:
> >> >>>>> >>> it might have bugs, needs tested, takes time for new contributors to
> >> >>>>> >>> "grok", etc. Using off-the-shelf here would nuke a whole bunch of
> >> >>>>> >>> that. If off-the-shelf is available for some given functionality, we
> >> >>>>> >>> should need a compelling reason to NOT use it.
> >> >>>>> >>>
> >> >>>>> >>> Lastly, I think there's an "approachability" argument for using
> >> >>>>> >>> off-the-shelf. Thousands of developers out there are familiar with
> >> >>>>> >>> (e.g.) Jersey, compared to maybe 15 or 20 (in the world) familiar with
> >> >>>>> >>> Solr's custom annotations. Using a well-known technology like Jersey
> >> >>>>> >>> would make Solr all the easier to approach and contribute to for that
> >> >>>>> >>> pool of developers.
> >> >>>>> >>>
> >> >>>>> >>> > By the way, we have used Restlet in the past and that has been a regrettable decision.
> >> >>>>> >>
> >> >>>>> >>
> >> >>>>> >>>
> >> >>>>> >>> Ah, yeah, that's just the context I'm missing. Anyone have a pointer
> >> >>>>> >>> to related discussions, or remember what made this "regrettable"? All
> >> >>>>> >>> the theoretical benefits in the world don't matter much if we've
> >> >>>>> >>> already tried something like this in the past and decided against it.
> >> >>>>> >>
> >> >>>>> >>
> >> >>>>> >> It was not playing well with our security framework. The framework was not working well with Solr APIs
> >> >>>>> >>
> >> >>>>> >>>
> >> >>>>> >>> (Unrelated - Happy Thanksgiving all!)
> >> >>>>> >>>
> >> >>>>> >>> Best,
> >> >>>>> >>>
> >> >>>>> >>> Jason
> >> >>>>> >>>
> >> >>>>> >>> On Thu, Nov 25, 2021 at 7:32 AM Noble Paul <noble.paul@gmail.com <ma...@gmail.com>> wrote:
> >> >>>>> >>> >
> >> >>>>> >>> > Have you gone through an API written using the @EndPoint annotation?
> >> >>>>> >>> >
> >> >>>>> >>> > I strongly recommend that you do
> >> >>>>> >>> >
> >> >>>>> >>> > On Thu, Nov 25, 2021, 11:30 PM Eric Pugh <epugh@opensourceconnections.com <ma...@opensourceconnections.com>> wrote:
> >> >>>>> >>> >>
> >> >>>>> >>> >> I have found our V2 API code to be very impenetrable to understand. Part of it is how the code is intertwined with support for V1, however it’s also because there aren’t really resources to go look at to understand how it should work! Maintaining the API should be very simple work, as they just exist as a translation. The home grown stuff may make sense if you are a super knowledgable Solr developer, but if you are just a new person, it’s a lot harder to contribute.
> >> >>>>> >>> >>
> >> >>>>> >>> >> I was interested in the Jersey stuff because I’ve seen lots of projects use it very successfully, and if I want to implement something, well, there are lots of blogs and resources out there!
> >> >>>>> >>> >>
> >> >>>>> >>> >> Can anyone recap briefly why we dropped RESTlet? And what lessons learned there might apply to adopting Jersey for API support? Looking at https://issues.apache.org/jira/browse/SOLR-14659 <https://issues.apache.org/jira/browse/SOLR-14659>, it was partly deprecated because we were not using it to support all the API, only the ManagedResource ones, and https://issues.apache.org/jira/browse/SOLR-14766 <https://issues.apache.org/jira/browse/SOLR-14766> suggests that RESTlet maybe was no longer being updated? One reason why we spiked out Jersey was because of the broad support in the Java world! Looking at how much work we have to do in the V2 API world, we need a much broader pool of developers contributing to get there!
> >> >>>>> >>> >>
> >> >>>>> >>> >> Related, are there specific features/aspects of our annotations that enable things in Solr that couldn’t be done otherwise?
> >> >>>>> >>> >>
> >> >>>>> >>> >> On Nov 25, 2021, at 2:12 AM, Ishan Chattopadhyaya <ichattopadhyaya@gmail.com <ma...@gmail.com>> wrote:
> >> >>>>> >>> >>
> >> >>>>> >>> >> Is there some problem with our annotations that we hope to solve using third party dependencies?
> >> >>>>> >>> >> By the way, we have used Restlet in the past and that has been a regrettable decision.
> >> >>>>> >>> >>
> >> >>>>> >>> >> On Thu, Nov 25, 2021 at 10:10 AM Jason Gerlowski <gerlowskija@gmail.com <ma...@gmail.com>> wrote:
> >> >>>>> >>> >>>
> >> >>>>> >>> >>> Solr's custom annotation framework ('@Endpoint', '@Command', etc.) has
> >> >>>>> >>> >>> cropped up a few times over the past week or two. [1] [2]. Having them
> >> >>>>> >>> >>> on top of mind, I've been wondering - is there a reason we use our own
> >> >>>>> >>> >>> annotations here instead of something off the shelf?
> >> >>>>> >>> >>>
> >> >>>>> >>> >>> What we have works well enough, but anything homegrown comes with more
> >> >>>>> >>> >>> maintenance burden than we'd have if we used something off the shelf.
> >> >>>>> >>> >>> There are plenty of well-used, active projects out there whose whole
> >> >>>>> >>> >>> purpose is facilitating the whole "annotation based API" thing
> >> >>>>> >>> >>> (Jersey, Restlet, RESTEasy, etc.) - why not use one of them?
> >> >>>>> >>> >>>
> >> >>>>> >>> >>> Does anyone know of any technical reasons why we can't go this route?
> >> >>>>> >>> >>> Or have any subjective reasons why we shouldn't? Or any context on
> >> >>>>> >>> >>> why we wrote our own Endpoint, Command, JsonProperty annotations
> >> >>>>> >>> >>> originally?
> >> >>>>> >>> >>>
> >> >>>>> >>> >>> FWIW, Eric Pugh and I spiked out a small POC recently, and got
> >> >>>>> >>> >>> Jersey+Jackson working for a few simple APIs without too much trouble.
> >> >>>>> >>> >>> [3] Obviously nothing production-ready there, and there's still a lot
> >> >>>>> >>> >>> of open questions (e.g. how would javabin be supported?), but we both
> >> >>>>> >>> >>> came away convinced that it seemed feasible, at least. Best of all,
> >> >>>>> >>> >>> APIs using our current homegrown annotation framework the switchover
> >> >>>>> >>> >>> seems blessedly straightforward, and it doesn't look like Jersey
> >> >>>>> >>> >>> (which we chose mostly arbitrarily) bloats our dist all that much.
> >> >>>>> >>> >>>
> >> >>>>> >>> >>> Curious if anyone has thoughts or context on how we ended up with the
> >> >>>>> >>> >>> annotation setup we use today!
> >> >>>>> >>> >>>
> >> >>>>> >>> >>> Best,
> >> >>>>> >>> >>>
> >> >>>>> >>> >>> Jason
> >> >>>>> >>> >>>
> >> >>>>> >>> >>> [1] https://issues.apache.org/jira/browse/SOLR-15182 <https://issues.apache.org/jira/browse/SOLR-15182> (and children)
> >> >>>>> >>> >>> [2] http://mail-archives.apache.org/mod_mbox/solr-dev/202111.mbox/%3CCABEwPvENL41Pm6%2BOmjXb6Sx5N2XjUtnbWhgKOZSrnLjWBA8tcA%40mail.gmail.com%3E <http://mail-archives.apache.org/mod_mbox/solr-dev/202111.mbox/%3CCABEwPvENL41Pm6%2BOmjXb6Sx5N2XjUtnbWhgKOZSrnLjWBA8tcA%40mail.gmail.com%3E>
> >> >>>>> >>> >>> [3] https://github.com/gerlowskija/solr/tree/jersey_jaxrs_jackson_solr_apis <https://github.com/gerlowskija/solr/tree/jersey_jaxrs_jackson_solr_apis>.
> >> >>>>> >>> >>>
> >> >>>>> >>> >>> ---------------------------------------------------------------------
> >> >>>>> >>> >>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org <ma...@solr.apache.org>
> >> >>>>> >>> >>> For additional commands, e-mail: dev-help@solr.apache.org <ma...@solr.apache.org>
> >> >>>>> >>> >>>
> >> >>>>> >>> >>
> >> >>>>> >>> >> _______________________
> >> >>>>> >>> >> Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | http://www.opensourceconnections.com <http://www.opensourceconnections.com/> | My Free/Busy
> >> >>>>> >>> >> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
> >> >>>>> >>> >> This e-mail and all contents, including attachments, is considered to be Company Confidential unless explicitly stated otherwise, regardless of whether attachments are marked as such.
> >> >>>>> >>> >>
> >> >>>>> >>>
> >> >>>>> >>> ---------------------------------------------------------------------
> >> >>>>> >>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org <ma...@solr.apache.org>
> >> >>>>> >>> For additional commands, e-mail: dev-help@solr.apache.org <ma...@solr.apache.org>
> >> >>>>> >>>
> >> >>>>> >>
> >> >>>>> >>
> >> >>>>> >> --
> >> >>>>> >> -----------------------------------------------------
> >> >>>>> >> Noble Paul
> >> >>>>> >
> >> >>>>> >
> >> >>>>> >
> >> >>>>> > --
> >> >>>>> > -----------------------------------------------------
> >> >>>>> > Noble Paul
> >> >>>>>
> >> >>>>> ---------------------------------------------------------------------
> >> >>>>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org <ma...@solr.apache.org>
> >> >>>>> For additional commands, e-mail: dev-help@solr.apache.org <ma...@solr.apache.org>
> >> >>>>>
> >> >>>>
> >> >>>>
> >> >>>> --
> >> >>>> http://www.needhamsoftware.com <http://www.needhamsoftware.com/> (work)
> >> >>>> http://www.the111shift.com <http://www.the111shift.com/> (play)
> >> >>>
> >> >>>
> >> >>>
> >> >>> --
> >> >>> -----------------------------------------------------
> >> >>> Noble Paul
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> -----------------------------------------------------
> >> >> Noble Paul
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org <ma...@solr.apache.org>
> >> For additional commands, e-mail: dev-help@solr.apache.org <ma...@solr.apache.org>
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org <ma...@solr.apache.org>
> For additional commands, e-mail: dev-help@solr.apache.org <ma...@solr.apache.org>
>
>
>
> --
> -----------------------------------------------------
> Noble Paul
Re: JAX-RS APIs in Solr
Posted by Ishan Chattopadhyaya <ic...@gmail.com>.
> We are discussing this as if moving to an external framework is going to
be smooth. It's not.
Noble, we won't know before someone has given it a try. My hunch suggests
it won't be smooth, but we can decide on that once we see a concrete
solution that 1) works well for all existing APIs (per core or global)
without hacks/workarounds, and 2) has no security implications, and 3)
doesn't clutter our codebase, and 4) doesn't affect our runtime performance.
On Thu, Dec 2, 2021 at 4:22 AM Noble Paul <no...@gmail.com> wrote:
>
>
> On Thu, Dec 2, 2021 at 2:46 AM Jason Gerlowski <ge...@gmail.com>
> wrote:
>
>> > There are no known issues with the current system
>>
>> Maybe you're using hyperbole to emphasise a point, but let's steer
>> this discussion away from straw-men and caricatures. I mentioned
>> specific known issues in a previous reply as a direct response to your
>> question about them. The incomplete support for data types
>> (SOLR-15619 and SOLR-15796). The inflexible PermissionNameProvider
>> integration (SOLR-15823). If you consider those "minor", fair enough,
>> and you're welcome to that opinion. But saying that they don't exist
>> is a mischaracterization that wastes time and muddies water.
>>
>
> We are discussing this as if moving to an external framework is going to
> be smooth. It's not. I have seen how hard the Restlet integration was and
> how it stood out like a sore thumb. The amount of code required to
> integrate Solr APIs into restlet components was 10x more than the amount of
> code written for the new framework and it still was incomplete
> . Integrating anything to Solr is not simple and it's going to be an
> ongoing effort to ensure that all of the functionalities of a
> SolrRequestHandler are exposed to the new framework. Every single component
> that we have today has ongoing improvements and bug fixes. Using that as an
> excuse to do a complete rewrite is probably unwise.
>
>
>
>> > Solr is not a general purpose web server where people write APIs every
>> day
>>
>> Totally agree - 99% of the time someone opens an API file in Solr,
>> they're reading, not writing. So if JAX-RS didn't help that case, it
>> wouldn't be worth the effort of switching. But IMO it helps that
>> "read" case a lot.
>>
>
> 99% of Solr users never see the java code that implements an API let alone
> writing a new API. They wouldn't care if an API is implemented using a
> Servlet API, a custom framework or something else. They only care about
> the input and output. New end points are rare.
>
>>
>> JAX-RS defined APIs expose inputs as strongly-typed parameters
>> (compare to the weakly-typed SolrQueryRequest/SolrParams used today).
>> JAX-RS enumerates every input to the API in one place. JAX-RS APIs
>> can return any serializable type as a response (compare to the
>> near-omnipresent NamedList today).
>>
>> (The annotation framework's PayloadObj class gets us a bit of this
>> benefit, but not all of it and only on POST requests afaik.)
>>
>> This is all nice stuff at API-write time, but where it really helps is
>> in making the code clearer and simpler whenever someone goes to read
>> it later. No wondering what parameters an API actually takes. No
>> puzzling out what a bunch of NamedList operations spread throughout
>> API execution actually produce in the end. etc.
>>
>> > The only reason why anyone would need to use this framework is to write
>> a custom request handler.
>>
>> ...except for the majority of people who come in contact with the
>> framework by reading an existing Solr API? This ties into your point
>> above about APIs being written infrequently but read all the time.
>> Improving readability is the big gain here, and I think JAX-RS offers
>> real improvements in that regard.
>>
>
> We had no framework for the last 15+ years and V1 API (the only actually
> used API) still uses no frameworks. Even the annotation framework is not
> used much even though readability is a lot better in the annotation based
> system. Readability has not been an overriding concern for V1 APIs for sure
>
> Also, to clarify, I'm not asserting that no one configures custom
>> requestHandlers. I was wondering aloud how common it was, as an open
>> question.
>>
>> Your point IS right. IRL, most of our users do not write request handlers
> . They write search components , query parsers and other plugins. Because
> we are a search engine and not a web server. I have never seen a user
> complaining about the V1 API of writing a request handler. They may
> actually complain about how hard it is to deploy one. Our users are much
> smarter than we give them credit for.
>
>
>
>> > We did not fork Jackson. We are using Jackson itself.
>>
>> Ah, maybe you're right, "fork" is probably an unfair word. I looked
>> for the serialization code, and I'll admit there is less than I
>> thought involved here and I misunderstood how it works. So that's my
>> bad. The JSON serialization/deserialization discussion is prob a
>> digression from the core contention about whether JAX-RS would be
>> helpful. I was curious about your earlier response and couldn't
>> resist following up here but I should've done so elsewhere to avoid
>> the distraction.
>>
>> Best,
>>
>>
>> Jason
>>
>> On Tue, Nov 30, 2021 at 4:12 PM Noble Paul <no...@gmail.com> wrote:
>> >
>> > The annotation framework is just a single class and it is made to work
>> well with Solr instead of changing Solr to suit the needs of some external
>> framework. There are no known issues with the current system and there are
>> unknown issues that you'll face introducing a new framework.
>> >
>> > Solr is not a general purpose web server where people write APIs every
>> day. It's a finished product where people use the publicly available APIs .
>> We should focus our efforts on making our APIs work well .
>> >
>> > On Wed, Dec 1, 2021, 2:14 AM Jason Gerlowski <ge...@gmail.com>
>> wrote:
>> >>
>> >> > We need to make an API work on a per core basis and cores can come
>> up and go down randomly
>> >>
>> >> Ah, yeah. I'll admit I was overlooking some of the wrinkles around
>> >> custom request handlers etc. That is a problem for switching to some
>> >> off-the-shelf framework for sure. It may still be possible, but it's
>> >> definitely a big hurdle. I wonder how commonly users still take
>> >> advantage of the custom requestHandler stuff these days? I haven't
>> >> run across many customers who used them extensively, but maybe that's
>> >> self-selecting in some way?
>> >
>> >
>> > The only reason why anyone would need to use this framework is to write
>> a custom request handler. You're also saying nobody writes custom request
>> handlers. So, what problem are you trying to solve if you think nobody uses
>> them?
>> >
>> >
>> >>
>> >> > If SolrJ has a Jackson dependency, it can have a conflict (different
>> versions) with the Jackson used by the client app
>> >>
>> >> SolrJ has a handful of dependencies - is there a reason we take this
>> >> approach with Jackson, but not with any of SolrJ's other deps?
>> >> (apache-httpcomponents, apache-commons, zookeeper-client, etc.)
>> >>
>> >> Granted, I get that we want SolrJ to be as painless as possible for
>> >> its users, and jar-conflicts are a part of that and really suck. But
>> >> there are good tools out there for handling jar conflicts, and
>> >> reimplementing chunks of library functionality just to avoid a
>> >> gradle-dep strikes me as unsustainable from a maintenance/community
>> >> perspective. Not to mention that forking/mimicking Jackson code
>> >> deprives our users of any efficiency/security improvements that
>> >> Jackson might get tomorrow or next week or next year.
>> >
>> >
>> > We did not fork Jackson. We are using Jackson itself. Jackson supports
>> custom annotations. There was a huge discussion why we should do it this
>> way.
>> >>
>> >>
>> >> Unless there's some reason specific to Jackson I guess I just don't
>> >> get it. But if I'm the minority opinion on that, fair enough I guess.
>> >>
>> >> On Tue, Nov 30, 2021 at 5:05 AM Noble Paul <no...@gmail.com>
>> wrote:
>> >> >
>> >> > The @JsonProperty annotations are added because it's a dependency in
>> SolrJ as well
>> >> > If SolrJ has a Jackson dependency, it can have a conflict (different
>> versions) with the Jackson used by the client app.
>> >> >
>> >> > On Tue, Nov 30, 2021, 5:21 PM Noble Paul <no...@gmail.com>
>> wrote:
>> >> >>
>> >> >> Also keep in mind that the same endpoint can be accessed with a
>> core name and a collection name prefixes.
>> >> >>
>> >> >> On Tue, Nov 30, 2021 at 3:55 PM Noble Paul <no...@gmail.com>
>> wrote:
>> >> >>>
>> >> >>> True Gus, Almost every framework works outside of
>> SolrDispatchFilter+ HttpSolrCall. A lot of our initializations occur there.
>> >> >>>
>> >> >>> We need to make an API work on a per core basis and cores can come
>> up and go down randomly. So we need to register these endpoints on a core.
>> >> >>>
>> >> >>> I'm not sure if any framework can achieve the same.
>> >> >>>
>> >> >>> On Tue, Nov 30, 2021 at 5:39 AM Gus Heck <gu...@gmail.com>
>> wrote:
>> >> >>>>
>> >> >>>> IIRC last time I looked restlet had the unsavory property of
>> existing outside of the SolrDispatchFilter, unlike everything else which
>> made for special cases because several things that probably ought to be
>> their own siervlet filters are glommed into SolrDispatchFilter, like
>> security, tracing and MDC setup/teardown per request. Restlet wouldn't be
>> so bad if one could just wrap such filters around it too...
>> >> >>>>
>> >> >>>> On Mon, Nov 29, 2021 at 9:42 AM Jason Gerlowski <
>> gerlowskija@gmail.com> wrote:
>> >> >>>>>
>> >> >>>>> > These are minor improvements compared to a full rewrite of the
>> entire framework
>> >> >>>>>
>> >> >>>>> If you think data type support is minor, fair enough. But to
>> clarify
>> >> >>>>> I'm not suggesting a rewrite - I'm suggesting using something
>> that
>> >> >>>>> already exists off the shelf. Jersey (e.g.) itself provides the
>> >> >>>>> framework - there would be no "rewrite".
>> >> >>>>>
>> >> >>>>> re: past restlet use
>> >> >>>>>
>> >> >>>>> > It was not playing well with our security framework. The
>> framework was not working well with Solr APIs
>> >> >>>>>
>> >> >>>>> Ah, very interesting! Security isn't something Eric or I
>> tackled in
>> >> >>>>> our little spike branch, but it's definitely a concern. Do you
>> >> >>>>> remember the specific concerns? Or recall where any of the
>> discussion
>> >> >>>>> around this happened?
>> >> >>>>>
>> >> >>>>> Without the context of that past discussion, it seems like the
>> >> >>>>> "PermissionNameProvider" interface could be implemented just as
>> well
>> >> >>>>> by a class with (e.g.) Jersey annotations as one with our own
>> custom
>> >> >>>>> annotations. Certainly there'd need to be some
>> >> >>>>> RuleBasedAuthorizationPlugin changes or other integration code,
>> but
>> >> >>>>> nothing that feels insurmountable.
>> >> >>>>>
>> >> >>>>> Maybe I can try spiking it out soon and find the issues myself,
>> but
>> >> >>>>> it'd be much easier if someone happens to remember and can save
>> me the
>> >> >>>>> trouble :-p
>> >> >>>>>
>> >> >>>>> Best,
>> >> >>>>>
>> >> >>>>> Jason
>> >> >>>>>
>> >> >>>>> On Fri, Nov 26, 2021 at 12:05 AM Noble Paul <
>> noble.paul@gmail.com> wrote:
>> >> >>>>> >
>> >> >>>>> > The Annotations framework was written after playing with other
>> frameworks. There were many shortcomings which were hard to overcome.
>> >> >>>>> >
>> >> >>>>> > The best example is a per collection API . How do you register
>> an endpoint for a collection/core ?
>> >> >>>>> >
>> >> >>>>> > On Fri, Nov 26, 2021 at 3:42 PM Noble Paul <
>> noble.paul@gmail.com> wrote:
>> >> >>>>> >>
>> >> >>>>> >>
>> >> >>>>> >>
>> >> >>>>> >> On Fri, Nov 26, 2021 at 1:03 AM Jason Gerlowski <
>> gerlowskija@gmail.com> wrote:
>> >> >>>>> >>>
>> >> >>>>> >>> > Is there some problem with our annotations that we hope to
>> solve using third party dependencies?
>> >> >>>>> >>>
>> >> >>>>> >>> I guess so yeah. Third-party deps are just fuller, more
>> robust
>> >> >>>>> >>> solutions, whereas our annotations still need support added
>> now and
>> >> >>>>> >>> then for even primitive data types like "long" (see
>> SOLR-15619).
>> >> >>>>> >>
>> >> >>>>> >> These are minor improvements compared to a full rewrite of
>> the entire framework
>> >> >>>>> >>
>> >> >>>>> >>
>> >> >>>>> >>>
>> >> >>>>> >>> Every JIRA spent doing basic stuff like that is time away
>> from
>> >> >>>>> >>> improving Solr in some other way.
>> >> >>>>> >>>
>> >> >>>>> >>> So there are feature-gap/capabilities arguments for moving
>> to a
>> >> >>>>> >>> third-party dep, sure. But, even if our annotations did
>> everything
>> >> >>>>> >>> Jersey+Jackson do today, I think switching would still be
>> worth it.
>> >> >>>>> >>> Every LOC in our code base brings along with it some
>> maintenance cost:
>> >> >>>>> >>> it might have bugs, needs tested, takes time for new
>> contributors to
>> >> >>>>> >>> "grok", etc. Using off-the-shelf here would nuke a whole
>> bunch of
>> >> >>>>> >>> that. If off-the-shelf is available for some given
>> functionality, we
>> >> >>>>> >>> should need a compelling reason to NOT use it.
>> >> >>>>> >>>
>> >> >>>>> >>> Lastly, I think there's an "approachability" argument for
>> using
>> >> >>>>> >>> off-the-shelf. Thousands of developers out there are
>> familiar with
>> >> >>>>> >>> (e.g.) Jersey, compared to maybe 15 or 20 (in the world)
>> familiar with
>> >> >>>>> >>> Solr's custom annotations. Using a well-known technology
>> like Jersey
>> >> >>>>> >>> would make Solr all the easier to approach and contribute to
>> for that
>> >> >>>>> >>> pool of developers.
>> >> >>>>> >>>
>> >> >>>>> >>> > By the way, we have used Restlet in the past and that has
>> been a regrettable decision.
>> >> >>>>> >>
>> >> >>>>> >>
>> >> >>>>> >>>
>> >> >>>>> >>> Ah, yeah, that's just the context I'm missing. Anyone have
>> a pointer
>> >> >>>>> >>> to related discussions, or remember what made this
>> "regrettable"? All
>> >> >>>>> >>> the theoretical benefits in the world don't matter much if
>> we've
>> >> >>>>> >>> already tried something like this in the past and decided
>> against it.
>> >> >>>>> >>
>> >> >>>>> >>
>> >> >>>>> >> It was not playing well with our security framework. The
>> framework was not working well with Solr APIs
>> >> >>>>> >>
>> >> >>>>> >>>
>> >> >>>>> >>> (Unrelated - Happy Thanksgiving all!)
>> >> >>>>> >>>
>> >> >>>>> >>> Best,
>> >> >>>>> >>>
>> >> >>>>> >>> Jason
>> >> >>>>> >>>
>> >> >>>>> >>> On Thu, Nov 25, 2021 at 7:32 AM Noble Paul <
>> noble.paul@gmail.com> wrote:
>> >> >>>>> >>> >
>> >> >>>>> >>> > Have you gone through an API written using the @EndPoint
>> annotation?
>> >> >>>>> >>> >
>> >> >>>>> >>> > I strongly recommend that you do
>> >> >>>>> >>> >
>> >> >>>>> >>> > On Thu, Nov 25, 2021, 11:30 PM Eric Pugh <
>> epugh@opensourceconnections.com> wrote:
>> >> >>>>> >>> >>
>> >> >>>>> >>> >> I have found our V2 API code to be very impenetrable to
>> understand. Part of it is how the code is intertwined with support for
>> V1, however it’s also because there aren’t really resources to go look at
>> to understand how it should work! Maintaining the API should be very
>> simple work, as they just exist as a translation. The home grown stuff
>> may make sense if you are a super knowledgable Solr developer, but if you
>> are just a new person, it’s a lot harder to contribute.
>> >> >>>>> >>> >>
>> >> >>>>> >>> >> I was interested in the Jersey stuff because I’ve seen
>> lots of projects use it very successfully, and if I want to implement
>> something, well, there are lots of blogs and resources out there!
>> >> >>>>> >>> >>
>> >> >>>>> >>> >> Can anyone recap briefly why we dropped RESTlet? And
>> what lessons learned there might apply to adopting Jersey for API support?
>> Looking at https://issues.apache.org/jira/browse/SOLR-14659, it was
>> partly deprecated because we were not using it to support all the API, only
>> the ManagedResource ones, and
>> https://issues.apache.org/jira/browse/SOLR-14766 suggests that RESTlet
>> maybe was no longer being updated? One reason why we spiked out Jersey
>> was because of the broad support in the Java world! Looking at how much
>> work we have to do in the V2 API world, we need a much broader pool of
>> developers contributing to get there!
>> >> >>>>> >>> >>
>> >> >>>>> >>> >> Related, are there specific features/aspects of our
>> annotations that enable things in Solr that couldn’t be done otherwise?
>> >> >>>>> >>> >>
>> >> >>>>> >>> >> On Nov 25, 2021, at 2:12 AM, Ishan Chattopadhyaya <
>> ichattopadhyaya@gmail.com> wrote:
>> >> >>>>> >>> >>
>> >> >>>>> >>> >> Is there some problem with our annotations that we hope
>> to solve using third party dependencies?
>> >> >>>>> >>> >> By the way, we have used Restlet in the past and that has
>> been a regrettable decision.
>> >> >>>>> >>> >>
>> >> >>>>> >>> >> On Thu, Nov 25, 2021 at 10:10 AM Jason Gerlowski <
>> gerlowskija@gmail.com> wrote:
>> >> >>>>> >>> >>>
>> >> >>>>> >>> >>> Solr's custom annotation framework ('@Endpoint',
>> '@Command', etc.) has
>> >> >>>>> >>> >>> cropped up a few times over the past week or two. [1]
>> [2]. Having them
>> >> >>>>> >>> >>> on top of mind, I've been wondering - is there a reason
>> we use our own
>> >> >>>>> >>> >>> annotations here instead of something off the shelf?
>> >> >>>>> >>> >>>
>> >> >>>>> >>> >>> What we have works well enough, but anything homegrown
>> comes with more
>> >> >>>>> >>> >>> maintenance burden than we'd have if we used something
>> off the shelf.
>> >> >>>>> >>> >>> There are plenty of well-used, active projects out there
>> whose whole
>> >> >>>>> >>> >>> purpose is facilitating the whole "annotation based API"
>> thing
>> >> >>>>> >>> >>> (Jersey, Restlet, RESTEasy, etc.) - why not use one of
>> them?
>> >> >>>>> >>> >>>
>> >> >>>>> >>> >>> Does anyone know of any technical reasons why we can't
>> go this route?
>> >> >>>>> >>> >>> Or have any subjective reasons why we shouldn't? Or any
>> context on
>> >> >>>>> >>> >>> why we wrote our own Endpoint, Command, JsonProperty
>> annotations
>> >> >>>>> >>> >>> originally?
>> >> >>>>> >>> >>>
>> >> >>>>> >>> >>> FWIW, Eric Pugh and I spiked out a small POC recently,
>> and got
>> >> >>>>> >>> >>> Jersey+Jackson working for a few simple APIs without too
>> much trouble.
>> >> >>>>> >>> >>> [3] Obviously nothing production-ready there, and
>> there's still a lot
>> >> >>>>> >>> >>> of open questions (e.g. how would javabin be
>> supported?), but we both
>> >> >>>>> >>> >>> came away convinced that it seemed feasible, at least.
>> Best of all,
>> >> >>>>> >>> >>> APIs using our current homegrown annotation framework
>> the switchover
>> >> >>>>> >>> >>> seems blessedly straightforward, and it doesn't look
>> like Jersey
>> >> >>>>> >>> >>> (which we chose mostly arbitrarily) bloats our dist all
>> that much.
>> >> >>>>> >>> >>>
>> >> >>>>> >>> >>> Curious if anyone has thoughts or context on how we
>> ended up with the
>> >> >>>>> >>> >>> annotation setup we use today!
>> >> >>>>> >>> >>>
>> >> >>>>> >>> >>> Best,
>> >> >>>>> >>> >>>
>> >> >>>>> >>> >>> Jason
>> >> >>>>> >>> >>>
>> >> >>>>> >>> >>> [1] https://issues.apache.org/jira/browse/SOLR-15182
>> (and children)
>> >> >>>>> >>> >>> [2]
>> http://mail-archives.apache.org/mod_mbox/solr-dev/202111.mbox/%3CCABEwPvENL41Pm6%2BOmjXb6Sx5N2XjUtnbWhgKOZSrnLjWBA8tcA%40mail.gmail.com%3E
>> >> >>>>> >>> >>> [3]
>> https://github.com/gerlowskija/solr/tree/jersey_jaxrs_jackson_solr_apis.
>> >> >>>>> >>> >>>
>> >> >>>>> >>> >>>
>> ---------------------------------------------------------------------
>> >> >>>>> >>> >>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>> >> >>>>> >>> >>> For additional commands, e-mail:
>> dev-help@solr.apache.org
>> >> >>>>> >>> >>>
>> >> >>>>> >>> >>
>> >> >>>>> >>> >> _______________________
>> >> >>>>> >>> >> Eric Pugh | Founder & CEO | OpenSource Connections, LLC |
>> 434.466.1467 | http://www.opensourceconnections.com | My Free/Busy
>> >> >>>>> >>> >> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
>> >> >>>>> >>> >> This e-mail and all contents, including attachments, is
>> considered to be Company Confidential unless explicitly stated otherwise,
>> regardless of whether attachments are marked as such.
>> >> >>>>> >>> >>
>> >> >>>>> >>>
>> >> >>>>> >>>
>> ---------------------------------------------------------------------
>> >> >>>>> >>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>> >> >>>>> >>> For additional commands, e-mail: dev-help@solr.apache.org
>> >> >>>>> >>>
>> >> >>>>> >>
>> >> >>>>> >>
>> >> >>>>> >> --
>> >> >>>>> >> -----------------------------------------------------
>> >> >>>>> >> Noble Paul
>> >> >>>>> >
>> >> >>>>> >
>> >> >>>>> >
>> >> >>>>> > --
>> >> >>>>> > -----------------------------------------------------
>> >> >>>>> > Noble Paul
>> >> >>>>>
>> >> >>>>>
>> ---------------------------------------------------------------------
>> >> >>>>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>> >> >>>>> For additional commands, e-mail: dev-help@solr.apache.org
>> >> >>>>>
>> >> >>>>
>> >> >>>>
>> >> >>>> --
>> >> >>>> http://www.needhamsoftware.com (work)
>> >> >>>> http://www.the111shift.com (play)
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> --
>> >> >>> -----------------------------------------------------
>> >> >>> Noble Paul
>> >> >>
>> >> >>
>> >> >>
>> >> >> --
>> >> >> -----------------------------------------------------
>> >> >> Noble Paul
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>> >> For additional commands, e-mail: dev-help@solr.apache.org
>> >>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>> For additional commands, e-mail: dev-help@solr.apache.org
>>
>>
>
> --
> -----------------------------------------------------
> Noble Paul
>
Re: JAX-RS APIs in Solr
Posted by Noble Paul <no...@gmail.com>.
On Thu, Dec 2, 2021 at 2:46 AM Jason Gerlowski <ge...@gmail.com>
wrote:
> > There are no known issues with the current system
>
> Maybe you're using hyperbole to emphasise a point, but let's steer
> this discussion away from straw-men and caricatures. I mentioned
> specific known issues in a previous reply as a direct response to your
> question about them. The incomplete support for data types
> (SOLR-15619 and SOLR-15796). The inflexible PermissionNameProvider
> integration (SOLR-15823). If you consider those "minor", fair enough,
> and you're welcome to that opinion. But saying that they don't exist
> is a mischaracterization that wastes time and muddies water.
>
We are discussing this as if moving to an external framework is going to be
smooth. It's not. I have seen how hard the Restlet integration was and how
it stood out like a sore thumb. The amount of code required to integrate
Solr APIs into restlet components was 10x more than the amount of code
written for the new framework and it still was incomplete
. Integrating anything to Solr is not simple and it's going to be an
ongoing effort to ensure that all of the functionalities of a
SolrRequestHandler are exposed to the new framework. Every single component
that we have today has ongoing improvements and bug fixes. Using that as an
excuse to do a complete rewrite is probably unwise.
> > Solr is not a general purpose web server where people write APIs every
> day
>
> Totally agree - 99% of the time someone opens an API file in Solr,
> they're reading, not writing. So if JAX-RS didn't help that case, it
> wouldn't be worth the effort of switching. But IMO it helps that
> "read" case a lot.
>
99% of Solr users never see the java code that implements an API let alone
writing a new API. They wouldn't care if an API is implemented using a
Servlet API, a custom framework or something else. They only care about
the input and output. New end points are rare.
>
> JAX-RS defined APIs expose inputs as strongly-typed parameters
> (compare to the weakly-typed SolrQueryRequest/SolrParams used today).
> JAX-RS enumerates every input to the API in one place. JAX-RS APIs
> can return any serializable type as a response (compare to the
> near-omnipresent NamedList today).
>
> (The annotation framework's PayloadObj class gets us a bit of this
> benefit, but not all of it and only on POST requests afaik.)
>
> This is all nice stuff at API-write time, but where it really helps is
> in making the code clearer and simpler whenever someone goes to read
> it later. No wondering what parameters an API actually takes. No
> puzzling out what a bunch of NamedList operations spread throughout
> API execution actually produce in the end. etc.
>
> > The only reason why anyone would need to use this framework is to write
> a custom request handler.
>
> ...except for the majority of people who come in contact with the
> framework by reading an existing Solr API? This ties into your point
> above about APIs being written infrequently but read all the time.
> Improving readability is the big gain here, and I think JAX-RS offers
> real improvements in that regard.
>
We had no framework for the last 15+ years and V1 API (the only actually
used API) still uses no frameworks. Even the annotation framework is not
used much even though readability is a lot better in the annotation based
system. Readability has not been an overriding concern for V1 APIs for sure
Also, to clarify, I'm not asserting that no one configures custom
> requestHandlers. I was wondering aloud how common it was, as an open
> question.
>
> Your point IS right. IRL, most of our users do not write request handlers
. They write search components , query parsers and other plugins. Because
we are a search engine and not a web server. I have never seen a user
complaining about the V1 API of writing a request handler. They may
actually complain about how hard it is to deploy one. Our users are much
smarter than we give them credit for.
> > We did not fork Jackson. We are using Jackson itself.
>
> Ah, maybe you're right, "fork" is probably an unfair word. I looked
> for the serialization code, and I'll admit there is less than I
> thought involved here and I misunderstood how it works. So that's my
> bad. The JSON serialization/deserialization discussion is prob a
> digression from the core contention about whether JAX-RS would be
> helpful. I was curious about your earlier response and couldn't
> resist following up here but I should've done so elsewhere to avoid
> the distraction.
>
> Best,
>
>
> Jason
>
> On Tue, Nov 30, 2021 at 4:12 PM Noble Paul <no...@gmail.com> wrote:
> >
> > The annotation framework is just a single class and it is made to work
> well with Solr instead of changing Solr to suit the needs of some external
> framework. There are no known issues with the current system and there are
> unknown issues that you'll face introducing a new framework.
> >
> > Solr is not a general purpose web server where people write APIs every
> day. It's a finished product where people use the publicly available APIs .
> We should focus our efforts on making our APIs work well .
> >
> > On Wed, Dec 1, 2021, 2:14 AM Jason Gerlowski <ge...@gmail.com>
> wrote:
> >>
> >> > We need to make an API work on a per core basis and cores can come up
> and go down randomly
> >>
> >> Ah, yeah. I'll admit I was overlooking some of the wrinkles around
> >> custom request handlers etc. That is a problem for switching to some
> >> off-the-shelf framework for sure. It may still be possible, but it's
> >> definitely a big hurdle. I wonder how commonly users still take
> >> advantage of the custom requestHandler stuff these days? I haven't
> >> run across many customers who used them extensively, but maybe that's
> >> self-selecting in some way?
> >
> >
> > The only reason why anyone would need to use this framework is to write
> a custom request handler. You're also saying nobody writes custom request
> handlers. So, what problem are you trying to solve if you think nobody uses
> them?
> >
> >
> >>
> >> > If SolrJ has a Jackson dependency, it can have a conflict (different
> versions) with the Jackson used by the client app
> >>
> >> SolrJ has a handful of dependencies - is there a reason we take this
> >> approach with Jackson, but not with any of SolrJ's other deps?
> >> (apache-httpcomponents, apache-commons, zookeeper-client, etc.)
> >>
> >> Granted, I get that we want SolrJ to be as painless as possible for
> >> its users, and jar-conflicts are a part of that and really suck. But
> >> there are good tools out there for handling jar conflicts, and
> >> reimplementing chunks of library functionality just to avoid a
> >> gradle-dep strikes me as unsustainable from a maintenance/community
> >> perspective. Not to mention that forking/mimicking Jackson code
> >> deprives our users of any efficiency/security improvements that
> >> Jackson might get tomorrow or next week or next year.
> >
> >
> > We did not fork Jackson. We are using Jackson itself. Jackson supports
> custom annotations. There was a huge discussion why we should do it this
> way.
> >>
> >>
> >> Unless there's some reason specific to Jackson I guess I just don't
> >> get it. But if I'm the minority opinion on that, fair enough I guess.
> >>
> >> On Tue, Nov 30, 2021 at 5:05 AM Noble Paul <no...@gmail.com>
> wrote:
> >> >
> >> > The @JsonProperty annotations are added because it's a dependency in
> SolrJ as well
> >> > If SolrJ has a Jackson dependency, it can have a conflict (different
> versions) with the Jackson used by the client app.
> >> >
> >> > On Tue, Nov 30, 2021, 5:21 PM Noble Paul <no...@gmail.com>
> wrote:
> >> >>
> >> >> Also keep in mind that the same endpoint can be accessed with a core
> name and a collection name prefixes.
> >> >>
> >> >> On Tue, Nov 30, 2021 at 3:55 PM Noble Paul <no...@gmail.com>
> wrote:
> >> >>>
> >> >>> True Gus, Almost every framework works outside of
> SolrDispatchFilter+ HttpSolrCall. A lot of our initializations occur there.
> >> >>>
> >> >>> We need to make an API work on a per core basis and cores can come
> up and go down randomly. So we need to register these endpoints on a core.
> >> >>>
> >> >>> I'm not sure if any framework can achieve the same.
> >> >>>
> >> >>> On Tue, Nov 30, 2021 at 5:39 AM Gus Heck <gu...@gmail.com>
> wrote:
> >> >>>>
> >> >>>> IIRC last time I looked restlet had the unsavory property of
> existing outside of the SolrDispatchFilter, unlike everything else which
> made for special cases because several things that probably ought to be
> their own siervlet filters are glommed into SolrDispatchFilter, like
> security, tracing and MDC setup/teardown per request. Restlet wouldn't be
> so bad if one could just wrap such filters around it too...
> >> >>>>
> >> >>>> On Mon, Nov 29, 2021 at 9:42 AM Jason Gerlowski <
> gerlowskija@gmail.com> wrote:
> >> >>>>>
> >> >>>>> > These are minor improvements compared to a full rewrite of the
> entire framework
> >> >>>>>
> >> >>>>> If you think data type support is minor, fair enough. But to
> clarify
> >> >>>>> I'm not suggesting a rewrite - I'm suggesting using something that
> >> >>>>> already exists off the shelf. Jersey (e.g.) itself provides the
> >> >>>>> framework - there would be no "rewrite".
> >> >>>>>
> >> >>>>> re: past restlet use
> >> >>>>>
> >> >>>>> > It was not playing well with our security framework. The
> framework was not working well with Solr APIs
> >> >>>>>
> >> >>>>> Ah, very interesting! Security isn't something Eric or I tackled
> in
> >> >>>>> our little spike branch, but it's definitely a concern. Do you
> >> >>>>> remember the specific concerns? Or recall where any of the
> discussion
> >> >>>>> around this happened?
> >> >>>>>
> >> >>>>> Without the context of that past discussion, it seems like the
> >> >>>>> "PermissionNameProvider" interface could be implemented just as
> well
> >> >>>>> by a class with (e.g.) Jersey annotations as one with our own
> custom
> >> >>>>> annotations. Certainly there'd need to be some
> >> >>>>> RuleBasedAuthorizationPlugin changes or other integration code,
> but
> >> >>>>> nothing that feels insurmountable.
> >> >>>>>
> >> >>>>> Maybe I can try spiking it out soon and find the issues myself,
> but
> >> >>>>> it'd be much easier if someone happens to remember and can save
> me the
> >> >>>>> trouble :-p
> >> >>>>>
> >> >>>>> Best,
> >> >>>>>
> >> >>>>> Jason
> >> >>>>>
> >> >>>>> On Fri, Nov 26, 2021 at 12:05 AM Noble Paul <no...@gmail.com>
> wrote:
> >> >>>>> >
> >> >>>>> > The Annotations framework was written after playing with other
> frameworks. There were many shortcomings which were hard to overcome.
> >> >>>>> >
> >> >>>>> > The best example is a per collection API . How do you register
> an endpoint for a collection/core ?
> >> >>>>> >
> >> >>>>> > On Fri, Nov 26, 2021 at 3:42 PM Noble Paul <
> noble.paul@gmail.com> wrote:
> >> >>>>> >>
> >> >>>>> >>
> >> >>>>> >>
> >> >>>>> >> On Fri, Nov 26, 2021 at 1:03 AM Jason Gerlowski <
> gerlowskija@gmail.com> wrote:
> >> >>>>> >>>
> >> >>>>> >>> > Is there some problem with our annotations that we hope to
> solve using third party dependencies?
> >> >>>>> >>>
> >> >>>>> >>> I guess so yeah. Third-party deps are just fuller, more
> robust
> >> >>>>> >>> solutions, whereas our annotations still need support added
> now and
> >> >>>>> >>> then for even primitive data types like "long" (see
> SOLR-15619).
> >> >>>>> >>
> >> >>>>> >> These are minor improvements compared to a full rewrite of the
> entire framework
> >> >>>>> >>
> >> >>>>> >>
> >> >>>>> >>>
> >> >>>>> >>> Every JIRA spent doing basic stuff like that is time away from
> >> >>>>> >>> improving Solr in some other way.
> >> >>>>> >>>
> >> >>>>> >>> So there are feature-gap/capabilities arguments for moving to
> a
> >> >>>>> >>> third-party dep, sure. But, even if our annotations did
> everything
> >> >>>>> >>> Jersey+Jackson do today, I think switching would still be
> worth it.
> >> >>>>> >>> Every LOC in our code base brings along with it some
> maintenance cost:
> >> >>>>> >>> it might have bugs, needs tested, takes time for new
> contributors to
> >> >>>>> >>> "grok", etc. Using off-the-shelf here would nuke a whole
> bunch of
> >> >>>>> >>> that. If off-the-shelf is available for some given
> functionality, we
> >> >>>>> >>> should need a compelling reason to NOT use it.
> >> >>>>> >>>
> >> >>>>> >>> Lastly, I think there's an "approachability" argument for
> using
> >> >>>>> >>> off-the-shelf. Thousands of developers out there are
> familiar with
> >> >>>>> >>> (e.g.) Jersey, compared to maybe 15 or 20 (in the world)
> familiar with
> >> >>>>> >>> Solr's custom annotations. Using a well-known technology
> like Jersey
> >> >>>>> >>> would make Solr all the easier to approach and contribute to
> for that
> >> >>>>> >>> pool of developers.
> >> >>>>> >>>
> >> >>>>> >>> > By the way, we have used Restlet in the past and that has
> been a regrettable decision.
> >> >>>>> >>
> >> >>>>> >>
> >> >>>>> >>>
> >> >>>>> >>> Ah, yeah, that's just the context I'm missing. Anyone have a
> pointer
> >> >>>>> >>> to related discussions, or remember what made this
> "regrettable"? All
> >> >>>>> >>> the theoretical benefits in the world don't matter much if
> we've
> >> >>>>> >>> already tried something like this in the past and decided
> against it.
> >> >>>>> >>
> >> >>>>> >>
> >> >>>>> >> It was not playing well with our security framework. The
> framework was not working well with Solr APIs
> >> >>>>> >>
> >> >>>>> >>>
> >> >>>>> >>> (Unrelated - Happy Thanksgiving all!)
> >> >>>>> >>>
> >> >>>>> >>> Best,
> >> >>>>> >>>
> >> >>>>> >>> Jason
> >> >>>>> >>>
> >> >>>>> >>> On Thu, Nov 25, 2021 at 7:32 AM Noble Paul <
> noble.paul@gmail.com> wrote:
> >> >>>>> >>> >
> >> >>>>> >>> > Have you gone through an API written using the @EndPoint
> annotation?
> >> >>>>> >>> >
> >> >>>>> >>> > I strongly recommend that you do
> >> >>>>> >>> >
> >> >>>>> >>> > On Thu, Nov 25, 2021, 11:30 PM Eric Pugh <
> epugh@opensourceconnections.com> wrote:
> >> >>>>> >>> >>
> >> >>>>> >>> >> I have found our V2 API code to be very impenetrable to
> understand. Part of it is how the code is intertwined with support for
> V1, however it’s also because there aren’t really resources to go look at
> to understand how it should work! Maintaining the API should be very
> simple work, as they just exist as a translation. The home grown stuff
> may make sense if you are a super knowledgable Solr developer, but if you
> are just a new person, it’s a lot harder to contribute.
> >> >>>>> >>> >>
> >> >>>>> >>> >> I was interested in the Jersey stuff because I’ve seen
> lots of projects use it very successfully, and if I want to implement
> something, well, there are lots of blogs and resources out there!
> >> >>>>> >>> >>
> >> >>>>> >>> >> Can anyone recap briefly why we dropped RESTlet? And
> what lessons learned there might apply to adopting Jersey for API support?
> Looking at https://issues.apache.org/jira/browse/SOLR-14659, it was
> partly deprecated because we were not using it to support all the API, only
> the ManagedResource ones, and
> https://issues.apache.org/jira/browse/SOLR-14766 suggests that RESTlet
> maybe was no longer being updated? One reason why we spiked out Jersey
> was because of the broad support in the Java world! Looking at how much
> work we have to do in the V2 API world, we need a much broader pool of
> developers contributing to get there!
> >> >>>>> >>> >>
> >> >>>>> >>> >> Related, are there specific features/aspects of our
> annotations that enable things in Solr that couldn’t be done otherwise?
> >> >>>>> >>> >>
> >> >>>>> >>> >> On Nov 25, 2021, at 2:12 AM, Ishan Chattopadhyaya <
> ichattopadhyaya@gmail.com> wrote:
> >> >>>>> >>> >>
> >> >>>>> >>> >> Is there some problem with our annotations that we hope to
> solve using third party dependencies?
> >> >>>>> >>> >> By the way, we have used Restlet in the past and that has
> been a regrettable decision.
> >> >>>>> >>> >>
> >> >>>>> >>> >> On Thu, Nov 25, 2021 at 10:10 AM Jason Gerlowski <
> gerlowskija@gmail.com> wrote:
> >> >>>>> >>> >>>
> >> >>>>> >>> >>> Solr's custom annotation framework ('@Endpoint',
> '@Command', etc.) has
> >> >>>>> >>> >>> cropped up a few times over the past week or two. [1]
> [2]. Having them
> >> >>>>> >>> >>> on top of mind, I've been wondering - is there a reason
> we use our own
> >> >>>>> >>> >>> annotations here instead of something off the shelf?
> >> >>>>> >>> >>>
> >> >>>>> >>> >>> What we have works well enough, but anything homegrown
> comes with more
> >> >>>>> >>> >>> maintenance burden than we'd have if we used something
> off the shelf.
> >> >>>>> >>> >>> There are plenty of well-used, active projects out there
> whose whole
> >> >>>>> >>> >>> purpose is facilitating the whole "annotation based API"
> thing
> >> >>>>> >>> >>> (Jersey, Restlet, RESTEasy, etc.) - why not use one of
> them?
> >> >>>>> >>> >>>
> >> >>>>> >>> >>> Does anyone know of any technical reasons why we can't go
> this route?
> >> >>>>> >>> >>> Or have any subjective reasons why we shouldn't? Or any
> context on
> >> >>>>> >>> >>> why we wrote our own Endpoint, Command, JsonProperty
> annotations
> >> >>>>> >>> >>> originally?
> >> >>>>> >>> >>>
> >> >>>>> >>> >>> FWIW, Eric Pugh and I spiked out a small POC recently,
> and got
> >> >>>>> >>> >>> Jersey+Jackson working for a few simple APIs without too
> much trouble.
> >> >>>>> >>> >>> [3] Obviously nothing production-ready there, and
> there's still a lot
> >> >>>>> >>> >>> of open questions (e.g. how would javabin be supported?),
> but we both
> >> >>>>> >>> >>> came away convinced that it seemed feasible, at least.
> Best of all,
> >> >>>>> >>> >>> APIs using our current homegrown annotation framework the
> switchover
> >> >>>>> >>> >>> seems blessedly straightforward, and it doesn't look like
> Jersey
> >> >>>>> >>> >>> (which we chose mostly arbitrarily) bloats our dist all
> that much.
> >> >>>>> >>> >>>
> >> >>>>> >>> >>> Curious if anyone has thoughts or context on how we ended
> up with the
> >> >>>>> >>> >>> annotation setup we use today!
> >> >>>>> >>> >>>
> >> >>>>> >>> >>> Best,
> >> >>>>> >>> >>>
> >> >>>>> >>> >>> Jason
> >> >>>>> >>> >>>
> >> >>>>> >>> >>> [1] https://issues.apache.org/jira/browse/SOLR-15182
> (and children)
> >> >>>>> >>> >>> [2]
> http://mail-archives.apache.org/mod_mbox/solr-dev/202111.mbox/%3CCABEwPvENL41Pm6%2BOmjXb6Sx5N2XjUtnbWhgKOZSrnLjWBA8tcA%40mail.gmail.com%3E
> >> >>>>> >>> >>> [3]
> https://github.com/gerlowskija/solr/tree/jersey_jaxrs_jackson_solr_apis.
> >> >>>>> >>> >>>
> >> >>>>> >>> >>>
> ---------------------------------------------------------------------
> >> >>>>> >>> >>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> >> >>>>> >>> >>> For additional commands, e-mail: dev-help@solr.apache.org
> >> >>>>> >>> >>>
> >> >>>>> >>> >>
> >> >>>>> >>> >> _______________________
> >> >>>>> >>> >> Eric Pugh | Founder & CEO | OpenSource Connections, LLC |
> 434.466.1467 | http://www.opensourceconnections.com | My Free/Busy
> >> >>>>> >>> >> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
> >> >>>>> >>> >> This e-mail and all contents, including attachments, is
> considered to be Company Confidential unless explicitly stated otherwise,
> regardless of whether attachments are marked as such.
> >> >>>>> >>> >>
> >> >>>>> >>>
> >> >>>>> >>>
> ---------------------------------------------------------------------
> >> >>>>> >>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> >> >>>>> >>> For additional commands, e-mail: dev-help@solr.apache.org
> >> >>>>> >>>
> >> >>>>> >>
> >> >>>>> >>
> >> >>>>> >> --
> >> >>>>> >> -----------------------------------------------------
> >> >>>>> >> Noble Paul
> >> >>>>> >
> >> >>>>> >
> >> >>>>> >
> >> >>>>> > --
> >> >>>>> > -----------------------------------------------------
> >> >>>>> > Noble Paul
> >> >>>>>
> >> >>>>>
> ---------------------------------------------------------------------
> >> >>>>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> >> >>>>> For additional commands, e-mail: dev-help@solr.apache.org
> >> >>>>>
> >> >>>>
> >> >>>>
> >> >>>> --
> >> >>>> http://www.needhamsoftware.com (work)
> >> >>>> http://www.the111shift.com (play)
> >> >>>
> >> >>>
> >> >>>
> >> >>> --
> >> >>> -----------------------------------------------------
> >> >>> Noble Paul
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> -----------------------------------------------------
> >> >> Noble Paul
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> >> For additional commands, e-mail: dev-help@solr.apache.org
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org
>
>
--
-----------------------------------------------------
Noble Paul
Re: JAX-RS APIs in Solr
Posted by Jason Gerlowski <ge...@gmail.com>.
> There are no known issues with the current system
Maybe you're using hyperbole to emphasise a point, but let's steer
this discussion away from straw-men and caricatures. I mentioned
specific known issues in a previous reply as a direct response to your
question about them. The incomplete support for data types
(SOLR-15619 and SOLR-15796). The inflexible PermissionNameProvider
integration (SOLR-15823). If you consider those "minor", fair enough,
and you're welcome to that opinion. But saying that they don't exist
is a mischaracterization that wastes time and muddies water.
> Solr is not a general purpose web server where people write APIs every day
Totally agree - 99% of the time someone opens an API file in Solr,
they're reading, not writing. So if JAX-RS didn't help that case, it
wouldn't be worth the effort of switching. But IMO it helps that
"read" case a lot.
JAX-RS defined APIs expose inputs as strongly-typed parameters
(compare to the weakly-typed SolrQueryRequest/SolrParams used today).
JAX-RS enumerates every input to the API in one place. JAX-RS APIs
can return any serializable type as a response (compare to the
near-omnipresent NamedList today).
(The annotation framework's PayloadObj class gets us a bit of this
benefit, but not all of it and only on POST requests afaik.)
This is all nice stuff at API-write time, but where it really helps is
in making the code clearer and simpler whenever someone goes to read
it later. No wondering what parameters an API actually takes. No
puzzling out what a bunch of NamedList operations spread throughout
API execution actually produce in the end. etc.
> The only reason why anyone would need to use this framework is to write a custom request handler.
...except for the majority of people who come in contact with the
framework by reading an existing Solr API? This ties into your point
above about APIs being written infrequently but read all the time.
Improving readability is the big gain here, and I think JAX-RS offers
real improvements in that regard.
Also, to clarify, I'm not asserting that no one configures custom
requestHandlers. I was wondering aloud how common it was, as an open
question.
> We did not fork Jackson. We are using Jackson itself.
Ah, maybe you're right, "fork" is probably an unfair word. I looked
for the serialization code, and I'll admit there is less than I
thought involved here and I misunderstood how it works. So that's my
bad. The JSON serialization/deserialization discussion is prob a
digression from the core contention about whether JAX-RS would be
helpful. I was curious about your earlier response and couldn't
resist following up here but I should've done so elsewhere to avoid
the distraction.
Best,
Jason
On Tue, Nov 30, 2021 at 4:12 PM Noble Paul <no...@gmail.com> wrote:
>
> The annotation framework is just a single class and it is made to work well with Solr instead of changing Solr to suit the needs of some external framework. There are no known issues with the current system and there are unknown issues that you'll face introducing a new framework.
>
> Solr is not a general purpose web server where people write APIs every day. It's a finished product where people use the publicly available APIs . We should focus our efforts on making our APIs work well .
>
> On Wed, Dec 1, 2021, 2:14 AM Jason Gerlowski <ge...@gmail.com> wrote:
>>
>> > We need to make an API work on a per core basis and cores can come up and go down randomly
>>
>> Ah, yeah. I'll admit I was overlooking some of the wrinkles around
>> custom request handlers etc. That is a problem for switching to some
>> off-the-shelf framework for sure. It may still be possible, but it's
>> definitely a big hurdle. I wonder how commonly users still take
>> advantage of the custom requestHandler stuff these days? I haven't
>> run across many customers who used them extensively, but maybe that's
>> self-selecting in some way?
>
>
> The only reason why anyone would need to use this framework is to write a custom request handler. You're also saying nobody writes custom request handlers. So, what problem are you trying to solve if you think nobody uses them?
>
>
>>
>> > If SolrJ has a Jackson dependency, it can have a conflict (different versions) with the Jackson used by the client app
>>
>> SolrJ has a handful of dependencies - is there a reason we take this
>> approach with Jackson, but not with any of SolrJ's other deps?
>> (apache-httpcomponents, apache-commons, zookeeper-client, etc.)
>>
>> Granted, I get that we want SolrJ to be as painless as possible for
>> its users, and jar-conflicts are a part of that and really suck. But
>> there are good tools out there for handling jar conflicts, and
>> reimplementing chunks of library functionality just to avoid a
>> gradle-dep strikes me as unsustainable from a maintenance/community
>> perspective. Not to mention that forking/mimicking Jackson code
>> deprives our users of any efficiency/security improvements that
>> Jackson might get tomorrow or next week or next year.
>
>
> We did not fork Jackson. We are using Jackson itself. Jackson supports custom annotations. There was a huge discussion why we should do it this way.
>>
>>
>> Unless there's some reason specific to Jackson I guess I just don't
>> get it. But if I'm the minority opinion on that, fair enough I guess.
>>
>> On Tue, Nov 30, 2021 at 5:05 AM Noble Paul <no...@gmail.com> wrote:
>> >
>> > The @JsonProperty annotations are added because it's a dependency in SolrJ as well
>> > If SolrJ has a Jackson dependency, it can have a conflict (different versions) with the Jackson used by the client app.
>> >
>> > On Tue, Nov 30, 2021, 5:21 PM Noble Paul <no...@gmail.com> wrote:
>> >>
>> >> Also keep in mind that the same endpoint can be accessed with a core name and a collection name prefixes.
>> >>
>> >> On Tue, Nov 30, 2021 at 3:55 PM Noble Paul <no...@gmail.com> wrote:
>> >>>
>> >>> True Gus, Almost every framework works outside of SolrDispatchFilter+ HttpSolrCall. A lot of our initializations occur there.
>> >>>
>> >>> We need to make an API work on a per core basis and cores can come up and go down randomly. So we need to register these endpoints on a core.
>> >>>
>> >>> I'm not sure if any framework can achieve the same.
>> >>>
>> >>> On Tue, Nov 30, 2021 at 5:39 AM Gus Heck <gu...@gmail.com> wrote:
>> >>>>
>> >>>> IIRC last time I looked restlet had the unsavory property of existing outside of the SolrDispatchFilter, unlike everything else which made for special cases because several things that probably ought to be their own siervlet filters are glommed into SolrDispatchFilter, like security, tracing and MDC setup/teardown per request. Restlet wouldn't be so bad if one could just wrap such filters around it too...
>> >>>>
>> >>>> On Mon, Nov 29, 2021 at 9:42 AM Jason Gerlowski <ge...@gmail.com> wrote:
>> >>>>>
>> >>>>> > These are minor improvements compared to a full rewrite of the entire framework
>> >>>>>
>> >>>>> If you think data type support is minor, fair enough. But to clarify
>> >>>>> I'm not suggesting a rewrite - I'm suggesting using something that
>> >>>>> already exists off the shelf. Jersey (e.g.) itself provides the
>> >>>>> framework - there would be no "rewrite".
>> >>>>>
>> >>>>> re: past restlet use
>> >>>>>
>> >>>>> > It was not playing well with our security framework. The framework was not working well with Solr APIs
>> >>>>>
>> >>>>> Ah, very interesting! Security isn't something Eric or I tackled in
>> >>>>> our little spike branch, but it's definitely a concern. Do you
>> >>>>> remember the specific concerns? Or recall where any of the discussion
>> >>>>> around this happened?
>> >>>>>
>> >>>>> Without the context of that past discussion, it seems like the
>> >>>>> "PermissionNameProvider" interface could be implemented just as well
>> >>>>> by a class with (e.g.) Jersey annotations as one with our own custom
>> >>>>> annotations. Certainly there'd need to be some
>> >>>>> RuleBasedAuthorizationPlugin changes or other integration code, but
>> >>>>> nothing that feels insurmountable.
>> >>>>>
>> >>>>> Maybe I can try spiking it out soon and find the issues myself, but
>> >>>>> it'd be much easier if someone happens to remember and can save me the
>> >>>>> trouble :-p
>> >>>>>
>> >>>>> Best,
>> >>>>>
>> >>>>> Jason
>> >>>>>
>> >>>>> On Fri, Nov 26, 2021 at 12:05 AM Noble Paul <no...@gmail.com> wrote:
>> >>>>> >
>> >>>>> > The Annotations framework was written after playing with other frameworks. There were many shortcomings which were hard to overcome.
>> >>>>> >
>> >>>>> > The best example is a per collection API . How do you register an endpoint for a collection/core ?
>> >>>>> >
>> >>>>> > On Fri, Nov 26, 2021 at 3:42 PM Noble Paul <no...@gmail.com> wrote:
>> >>>>> >>
>> >>>>> >>
>> >>>>> >>
>> >>>>> >> On Fri, Nov 26, 2021 at 1:03 AM Jason Gerlowski <ge...@gmail.com> wrote:
>> >>>>> >>>
>> >>>>> >>> > Is there some problem with our annotations that we hope to solve using third party dependencies?
>> >>>>> >>>
>> >>>>> >>> I guess so yeah. Third-party deps are just fuller, more robust
>> >>>>> >>> solutions, whereas our annotations still need support added now and
>> >>>>> >>> then for even primitive data types like "long" (see SOLR-15619).
>> >>>>> >>
>> >>>>> >> These are minor improvements compared to a full rewrite of the entire framework
>> >>>>> >>
>> >>>>> >>
>> >>>>> >>>
>> >>>>> >>> Every JIRA spent doing basic stuff like that is time away from
>> >>>>> >>> improving Solr in some other way.
>> >>>>> >>>
>> >>>>> >>> So there are feature-gap/capabilities arguments for moving to a
>> >>>>> >>> third-party dep, sure. But, even if our annotations did everything
>> >>>>> >>> Jersey+Jackson do today, I think switching would still be worth it.
>> >>>>> >>> Every LOC in our code base brings along with it some maintenance cost:
>> >>>>> >>> it might have bugs, needs tested, takes time for new contributors to
>> >>>>> >>> "grok", etc. Using off-the-shelf here would nuke a whole bunch of
>> >>>>> >>> that. If off-the-shelf is available for some given functionality, we
>> >>>>> >>> should need a compelling reason to NOT use it.
>> >>>>> >>>
>> >>>>> >>> Lastly, I think there's an "approachability" argument for using
>> >>>>> >>> off-the-shelf. Thousands of developers out there are familiar with
>> >>>>> >>> (e.g.) Jersey, compared to maybe 15 or 20 (in the world) familiar with
>> >>>>> >>> Solr's custom annotations. Using a well-known technology like Jersey
>> >>>>> >>> would make Solr all the easier to approach and contribute to for that
>> >>>>> >>> pool of developers.
>> >>>>> >>>
>> >>>>> >>> > By the way, we have used Restlet in the past and that has been a regrettable decision.
>> >>>>> >>
>> >>>>> >>
>> >>>>> >>>
>> >>>>> >>> Ah, yeah, that's just the context I'm missing. Anyone have a pointer
>> >>>>> >>> to related discussions, or remember what made this "regrettable"? All
>> >>>>> >>> the theoretical benefits in the world don't matter much if we've
>> >>>>> >>> already tried something like this in the past and decided against it.
>> >>>>> >>
>> >>>>> >>
>> >>>>> >> It was not playing well with our security framework. The framework was not working well with Solr APIs
>> >>>>> >>
>> >>>>> >>>
>> >>>>> >>> (Unrelated - Happy Thanksgiving all!)
>> >>>>> >>>
>> >>>>> >>> Best,
>> >>>>> >>>
>> >>>>> >>> Jason
>> >>>>> >>>
>> >>>>> >>> On Thu, Nov 25, 2021 at 7:32 AM Noble Paul <no...@gmail.com> wrote:
>> >>>>> >>> >
>> >>>>> >>> > Have you gone through an API written using the @EndPoint annotation?
>> >>>>> >>> >
>> >>>>> >>> > I strongly recommend that you do
>> >>>>> >>> >
>> >>>>> >>> > On Thu, Nov 25, 2021, 11:30 PM Eric Pugh <ep...@opensourceconnections.com> wrote:
>> >>>>> >>> >>
>> >>>>> >>> >> I have found our V2 API code to be very impenetrable to understand. Part of it is how the code is intertwined with support for V1, however it’s also because there aren’t really resources to go look at to understand how it should work! Maintaining the API should be very simple work, as they just exist as a translation. The home grown stuff may make sense if you are a super knowledgable Solr developer, but if you are just a new person, it’s a lot harder to contribute.
>> >>>>> >>> >>
>> >>>>> >>> >> I was interested in the Jersey stuff because I’ve seen lots of projects use it very successfully, and if I want to implement something, well, there are lots of blogs and resources out there!
>> >>>>> >>> >>
>> >>>>> >>> >> Can anyone recap briefly why we dropped RESTlet? And what lessons learned there might apply to adopting Jersey for API support? Looking at https://issues.apache.org/jira/browse/SOLR-14659, it was partly deprecated because we were not using it to support all the API, only the ManagedResource ones, and https://issues.apache.org/jira/browse/SOLR-14766 suggests that RESTlet maybe was no longer being updated? One reason why we spiked out Jersey was because of the broad support in the Java world! Looking at how much work we have to do in the V2 API world, we need a much broader pool of developers contributing to get there!
>> >>>>> >>> >>
>> >>>>> >>> >> Related, are there specific features/aspects of our annotations that enable things in Solr that couldn’t be done otherwise?
>> >>>>> >>> >>
>> >>>>> >>> >> On Nov 25, 2021, at 2:12 AM, Ishan Chattopadhyaya <ic...@gmail.com> wrote:
>> >>>>> >>> >>
>> >>>>> >>> >> Is there some problem with our annotations that we hope to solve using third party dependencies?
>> >>>>> >>> >> By the way, we have used Restlet in the past and that has been a regrettable decision.
>> >>>>> >>> >>
>> >>>>> >>> >> On Thu, Nov 25, 2021 at 10:10 AM Jason Gerlowski <ge...@gmail.com> wrote:
>> >>>>> >>> >>>
>> >>>>> >>> >>> Solr's custom annotation framework ('@Endpoint', '@Command', etc.) has
>> >>>>> >>> >>> cropped up a few times over the past week or two. [1] [2]. Having them
>> >>>>> >>> >>> on top of mind, I've been wondering - is there a reason we use our own
>> >>>>> >>> >>> annotations here instead of something off the shelf?
>> >>>>> >>> >>>
>> >>>>> >>> >>> What we have works well enough, but anything homegrown comes with more
>> >>>>> >>> >>> maintenance burden than we'd have if we used something off the shelf.
>> >>>>> >>> >>> There are plenty of well-used, active projects out there whose whole
>> >>>>> >>> >>> purpose is facilitating the whole "annotation based API" thing
>> >>>>> >>> >>> (Jersey, Restlet, RESTEasy, etc.) - why not use one of them?
>> >>>>> >>> >>>
>> >>>>> >>> >>> Does anyone know of any technical reasons why we can't go this route?
>> >>>>> >>> >>> Or have any subjective reasons why we shouldn't? Or any context on
>> >>>>> >>> >>> why we wrote our own Endpoint, Command, JsonProperty annotations
>> >>>>> >>> >>> originally?
>> >>>>> >>> >>>
>> >>>>> >>> >>> FWIW, Eric Pugh and I spiked out a small POC recently, and got
>> >>>>> >>> >>> Jersey+Jackson working for a few simple APIs without too much trouble.
>> >>>>> >>> >>> [3] Obviously nothing production-ready there, and there's still a lot
>> >>>>> >>> >>> of open questions (e.g. how would javabin be supported?), but we both
>> >>>>> >>> >>> came away convinced that it seemed feasible, at least. Best of all,
>> >>>>> >>> >>> APIs using our current homegrown annotation framework the switchover
>> >>>>> >>> >>> seems blessedly straightforward, and it doesn't look like Jersey
>> >>>>> >>> >>> (which we chose mostly arbitrarily) bloats our dist all that much.
>> >>>>> >>> >>>
>> >>>>> >>> >>> Curious if anyone has thoughts or context on how we ended up with the
>> >>>>> >>> >>> annotation setup we use today!
>> >>>>> >>> >>>
>> >>>>> >>> >>> Best,
>> >>>>> >>> >>>
>> >>>>> >>> >>> Jason
>> >>>>> >>> >>>
>> >>>>> >>> >>> [1] https://issues.apache.org/jira/browse/SOLR-15182 (and children)
>> >>>>> >>> >>> [2] http://mail-archives.apache.org/mod_mbox/solr-dev/202111.mbox/%3CCABEwPvENL41Pm6%2BOmjXb6Sx5N2XjUtnbWhgKOZSrnLjWBA8tcA%40mail.gmail.com%3E
>> >>>>> >>> >>> [3] https://github.com/gerlowskija/solr/tree/jersey_jaxrs_jackson_solr_apis.
>> >>>>> >>> >>>
>> >>>>> >>> >>> ---------------------------------------------------------------------
>> >>>>> >>> >>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>> >>>>> >>> >>> For additional commands, e-mail: dev-help@solr.apache.org
>> >>>>> >>> >>>
>> >>>>> >>> >>
>> >>>>> >>> >> _______________________
>> >>>>> >>> >> Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | http://www.opensourceconnections.com | My Free/Busy
>> >>>>> >>> >> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
>> >>>>> >>> >> This e-mail and all contents, including attachments, is considered to be Company Confidential unless explicitly stated otherwise, regardless of whether attachments are marked as such.
>> >>>>> >>> >>
>> >>>>> >>>
>> >>>>> >>> ---------------------------------------------------------------------
>> >>>>> >>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>> >>>>> >>> For additional commands, e-mail: dev-help@solr.apache.org
>> >>>>> >>>
>> >>>>> >>
>> >>>>> >>
>> >>>>> >> --
>> >>>>> >> -----------------------------------------------------
>> >>>>> >> Noble Paul
>> >>>>> >
>> >>>>> >
>> >>>>> >
>> >>>>> > --
>> >>>>> > -----------------------------------------------------
>> >>>>> > Noble Paul
>> >>>>>
>> >>>>> ---------------------------------------------------------------------
>> >>>>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>> >>>>> For additional commands, e-mail: dev-help@solr.apache.org
>> >>>>>
>> >>>>
>> >>>>
>> >>>> --
>> >>>> http://www.needhamsoftware.com (work)
>> >>>> http://www.the111shift.com (play)
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> -----------------------------------------------------
>> >>> Noble Paul
>> >>
>> >>
>> >>
>> >> --
>> >> -----------------------------------------------------
>> >> Noble Paul
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>> For additional commands, e-mail: dev-help@solr.apache.org
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: JAX-RS APIs in Solr
Posted by Noble Paul <no...@gmail.com>.
The annotation framework is just a single class and it is made to work well
with Solr instead of changing Solr to suit the needs of some external
framework. There are no known issues with the current system and there are
unknown issues that you'll face introducing a new framework.
Solr is not a general purpose web server where people write APIs every day.
It's a finished product where people use the publicly available APIs . We
should focus our efforts on making our APIs work well .
On Wed, Dec 1, 2021, 2:14 AM Jason Gerlowski <ge...@gmail.com> wrote:
> > We need to make an API work on a per core basis and cores can come up
> and go down randomly
>
> Ah, yeah. I'll admit I was overlooking some of the wrinkles around
> custom request handlers etc. That is a problem for switching to some
> off-the-shelf framework for sure. It may still be possible, but it's
> definitely a big hurdle. I wonder how commonly users still take
> advantage of the custom requestHandler stuff these days? I haven't
> run across many customers who used them extensively, but maybe that's
> self-selecting in some way?
>
The only reason why anyone would need to use this framework is to write a
custom request handler. You're also saying nobody writes custom request
handlers. So, what problem are you trying to solve if you think nobody uses
them?
> > If SolrJ has a Jackson dependency, it can have a conflict (different
> versions) with the Jackson used by the client app
>
> SolrJ has a handful of dependencies - is there a reason we take this
> approach with Jackson, but not with any of SolrJ's other deps?
> (apache-httpcomponents, apache-commons, zookeeper-client, etc.)
>
> Granted, I get that we want SolrJ to be as painless as possible for
> its users, and jar-conflicts are a part of that and really suck. But
> there are good tools out there for handling jar conflicts, and
> reimplementing chunks of library functionality just to avoid a
> gradle-dep strikes me as unsustainable from a maintenance/community
> perspective. Not to mention that forking/mimicking Jackson code
> deprives our users of any efficiency/security improvements that
> Jackson might get tomorrow or next week or next year.
>
We did not fork Jackson. We are using Jackson itself. Jackson supports
custom annotations. There was a huge discussion why we should do it this
way.
>
> Unless there's some reason specific to Jackson I guess I just don't
> get it. But if I'm the minority opinion on that, fair enough I guess.
>
> On Tue, Nov 30, 2021 at 5:05 AM Noble Paul <no...@gmail.com> wrote:
> >
> > The @JsonProperty annotations are added because it's a dependency in
> SolrJ as well
> > If SolrJ has a Jackson dependency, it can have a conflict (different
> versions) with the Jackson used by the client app.
> >
> > On Tue, Nov 30, 2021, 5:21 PM Noble Paul <no...@gmail.com> wrote:
> >>
> >> Also keep in mind that the same endpoint can be accessed with a core
> name and a collection name prefixes.
> >>
> >> On Tue, Nov 30, 2021 at 3:55 PM Noble Paul <no...@gmail.com>
> wrote:
> >>>
> >>> True Gus, Almost every framework works outside of SolrDispatchFilter+
> HttpSolrCall. A lot of our initializations occur there.
> >>>
> >>> We need to make an API work on a per core basis and cores can come up
> and go down randomly. So we need to register these endpoints on a core.
> >>>
> >>> I'm not sure if any framework can achieve the same.
> >>>
> >>> On Tue, Nov 30, 2021 at 5:39 AM Gus Heck <gu...@gmail.com> wrote:
> >>>>
> >>>> IIRC last time I looked restlet had the unsavory property of existing
> outside of the SolrDispatchFilter, unlike everything else which made for
> special cases because several things that probably ought to be their own
> siervlet filters are glommed into SolrDispatchFilter, like security,
> tracing and MDC setup/teardown per request. Restlet wouldn't be so bad if
> one could just wrap such filters around it too...
> >>>>
> >>>> On Mon, Nov 29, 2021 at 9:42 AM Jason Gerlowski <
> gerlowskija@gmail.com> wrote:
> >>>>>
> >>>>> > These are minor improvements compared to a full rewrite of the
> entire framework
> >>>>>
> >>>>> If you think data type support is minor, fair enough. But to clarify
> >>>>> I'm not suggesting a rewrite - I'm suggesting using something that
> >>>>> already exists off the shelf. Jersey (e.g.) itself provides the
> >>>>> framework - there would be no "rewrite".
> >>>>>
> >>>>> re: past restlet use
> >>>>>
> >>>>> > It was not playing well with our security framework. The framework
> was not working well with Solr APIs
> >>>>>
> >>>>> Ah, very interesting! Security isn't something Eric or I tackled in
> >>>>> our little spike branch, but it's definitely a concern. Do you
> >>>>> remember the specific concerns? Or recall where any of the
> discussion
> >>>>> around this happened?
> >>>>>
> >>>>> Without the context of that past discussion, it seems like the
> >>>>> "PermissionNameProvider" interface could be implemented just as well
> >>>>> by a class with (e.g.) Jersey annotations as one with our own custom
> >>>>> annotations. Certainly there'd need to be some
> >>>>> RuleBasedAuthorizationPlugin changes or other integration code, but
> >>>>> nothing that feels insurmountable.
> >>>>>
> >>>>> Maybe I can try spiking it out soon and find the issues myself, but
> >>>>> it'd be much easier if someone happens to remember and can save me
> the
> >>>>> trouble :-p
> >>>>>
> >>>>> Best,
> >>>>>
> >>>>> Jason
> >>>>>
> >>>>> On Fri, Nov 26, 2021 at 12:05 AM Noble Paul <no...@gmail.com>
> wrote:
> >>>>> >
> >>>>> > The Annotations framework was written after playing with other
> frameworks. There were many shortcomings which were hard to overcome.
> >>>>> >
> >>>>> > The best example is a per collection API . How do you register an
> endpoint for a collection/core ?
> >>>>> >
> >>>>> > On Fri, Nov 26, 2021 at 3:42 PM Noble Paul <no...@gmail.com>
> wrote:
> >>>>> >>
> >>>>> >>
> >>>>> >>
> >>>>> >> On Fri, Nov 26, 2021 at 1:03 AM Jason Gerlowski <
> gerlowskija@gmail.com> wrote:
> >>>>> >>>
> >>>>> >>> > Is there some problem with our annotations that we hope to
> solve using third party dependencies?
> >>>>> >>>
> >>>>> >>> I guess so yeah. Third-party deps are just fuller, more robust
> >>>>> >>> solutions, whereas our annotations still need support added now
> and
> >>>>> >>> then for even primitive data types like "long" (see SOLR-15619).
> >>>>> >>
> >>>>> >> These are minor improvements compared to a full rewrite of the
> entire framework
> >>>>> >>
> >>>>> >>
> >>>>> >>>
> >>>>> >>> Every JIRA spent doing basic stuff like that is time away from
> >>>>> >>> improving Solr in some other way.
> >>>>> >>>
> >>>>> >>> So there are feature-gap/capabilities arguments for moving to a
> >>>>> >>> third-party dep, sure. But, even if our annotations did
> everything
> >>>>> >>> Jersey+Jackson do today, I think switching would still be worth
> it.
> >>>>> >>> Every LOC in our code base brings along with it some maintenance
> cost:
> >>>>> >>> it might have bugs, needs tested, takes time for new
> contributors to
> >>>>> >>> "grok", etc. Using off-the-shelf here would nuke a whole bunch
> of
> >>>>> >>> that. If off-the-shelf is available for some given
> functionality, we
> >>>>> >>> should need a compelling reason to NOT use it.
> >>>>> >>>
> >>>>> >>> Lastly, I think there's an "approachability" argument for using
> >>>>> >>> off-the-shelf. Thousands of developers out there are familiar
> with
> >>>>> >>> (e.g.) Jersey, compared to maybe 15 or 20 (in the world)
> familiar with
> >>>>> >>> Solr's custom annotations. Using a well-known technology like
> Jersey
> >>>>> >>> would make Solr all the easier to approach and contribute to for
> that
> >>>>> >>> pool of developers.
> >>>>> >>>
> >>>>> >>> > By the way, we have used Restlet in the past and that has been
> a regrettable decision.
> >>>>> >>
> >>>>> >>
> >>>>> >>>
> >>>>> >>> Ah, yeah, that's just the context I'm missing. Anyone have a
> pointer
> >>>>> >>> to related discussions, or remember what made this
> "regrettable"? All
> >>>>> >>> the theoretical benefits in the world don't matter much if we've
> >>>>> >>> already tried something like this in the past and decided
> against it.
> >>>>> >>
> >>>>> >>
> >>>>> >> It was not playing well with our security framework. The
> framework was not working well with Solr APIs
> >>>>> >>
> >>>>> >>>
> >>>>> >>> (Unrelated - Happy Thanksgiving all!)
> >>>>> >>>
> >>>>> >>> Best,
> >>>>> >>>
> >>>>> >>> Jason
> >>>>> >>>
> >>>>> >>> On Thu, Nov 25, 2021 at 7:32 AM Noble Paul <no...@gmail.com>
> wrote:
> >>>>> >>> >
> >>>>> >>> > Have you gone through an API written using the @EndPoint
> annotation?
> >>>>> >>> >
> >>>>> >>> > I strongly recommend that you do
> >>>>> >>> >
> >>>>> >>> > On Thu, Nov 25, 2021, 11:30 PM Eric Pugh <
> epugh@opensourceconnections.com> wrote:
> >>>>> >>> >>
> >>>>> >>> >> I have found our V2 API code to be very impenetrable to
> understand. Part of it is how the code is intertwined with support for
> V1, however it’s also because there aren’t really resources to go look at
> to understand how it should work! Maintaining the API should be very
> simple work, as they just exist as a translation. The home grown stuff
> may make sense if you are a super knowledgable Solr developer, but if you
> are just a new person, it’s a lot harder to contribute.
> >>>>> >>> >>
> >>>>> >>> >> I was interested in the Jersey stuff because I’ve seen lots
> of projects use it very successfully, and if I want to implement something,
> well, there are lots of blogs and resources out there!
> >>>>> >>> >>
> >>>>> >>> >> Can anyone recap briefly why we dropped RESTlet? And what
> lessons learned there might apply to adopting Jersey for API support?
> Looking at https://issues.apache.org/jira/browse/SOLR-14659, it was
> partly deprecated because we were not using it to support all the API, only
> the ManagedResource ones, and
> https://issues.apache.org/jira/browse/SOLR-14766 suggests that RESTlet
> maybe was no longer being updated? One reason why we spiked out Jersey
> was because of the broad support in the Java world! Looking at how much
> work we have to do in the V2 API world, we need a much broader pool of
> developers contributing to get there!
> >>>>> >>> >>
> >>>>> >>> >> Related, are there specific features/aspects of our
> annotations that enable things in Solr that couldn’t be done otherwise?
> >>>>> >>> >>
> >>>>> >>> >> On Nov 25, 2021, at 2:12 AM, Ishan Chattopadhyaya <
> ichattopadhyaya@gmail.com> wrote:
> >>>>> >>> >>
> >>>>> >>> >> Is there some problem with our annotations that we hope to
> solve using third party dependencies?
> >>>>> >>> >> By the way, we have used Restlet in the past and that has
> been a regrettable decision.
> >>>>> >>> >>
> >>>>> >>> >> On Thu, Nov 25, 2021 at 10:10 AM Jason Gerlowski <
> gerlowskija@gmail.com> wrote:
> >>>>> >>> >>>
> >>>>> >>> >>> Solr's custom annotation framework ('@Endpoint', '@Command',
> etc.) has
> >>>>> >>> >>> cropped up a few times over the past week or two. [1] [2].
> Having them
> >>>>> >>> >>> on top of mind, I've been wondering - is there a reason we
> use our own
> >>>>> >>> >>> annotations here instead of something off the shelf?
> >>>>> >>> >>>
> >>>>> >>> >>> What we have works well enough, but anything homegrown comes
> with more
> >>>>> >>> >>> maintenance burden than we'd have if we used something off
> the shelf.
> >>>>> >>> >>> There are plenty of well-used, active projects out there
> whose whole
> >>>>> >>> >>> purpose is facilitating the whole "annotation based API"
> thing
> >>>>> >>> >>> (Jersey, Restlet, RESTEasy, etc.) - why not use one of them?
> >>>>> >>> >>>
> >>>>> >>> >>> Does anyone know of any technical reasons why we can't go
> this route?
> >>>>> >>> >>> Or have any subjective reasons why we shouldn't? Or any
> context on
> >>>>> >>> >>> why we wrote our own Endpoint, Command, JsonProperty
> annotations
> >>>>> >>> >>> originally?
> >>>>> >>> >>>
> >>>>> >>> >>> FWIW, Eric Pugh and I spiked out a small POC recently, and
> got
> >>>>> >>> >>> Jersey+Jackson working for a few simple APIs without too
> much trouble.
> >>>>> >>> >>> [3] Obviously nothing production-ready there, and there's
> still a lot
> >>>>> >>> >>> of open questions (e.g. how would javabin be supported?),
> but we both
> >>>>> >>> >>> came away convinced that it seemed feasible, at least. Best
> of all,
> >>>>> >>> >>> APIs using our current homegrown annotation framework the
> switchover
> >>>>> >>> >>> seems blessedly straightforward, and it doesn't look like
> Jersey
> >>>>> >>> >>> (which we chose mostly arbitrarily) bloats our dist all that
> much.
> >>>>> >>> >>>
> >>>>> >>> >>> Curious if anyone has thoughts or context on how we ended up
> with the
> >>>>> >>> >>> annotation setup we use today!
> >>>>> >>> >>>
> >>>>> >>> >>> Best,
> >>>>> >>> >>>
> >>>>> >>> >>> Jason
> >>>>> >>> >>>
> >>>>> >>> >>> [1] https://issues.apache.org/jira/browse/SOLR-15182 (and
> children)
> >>>>> >>> >>> [2]
> http://mail-archives.apache.org/mod_mbox/solr-dev/202111.mbox/%3CCABEwPvENL41Pm6%2BOmjXb6Sx5N2XjUtnbWhgKOZSrnLjWBA8tcA%40mail.gmail.com%3E
> >>>>> >>> >>> [3]
> https://github.com/gerlowskija/solr/tree/jersey_jaxrs_jackson_solr_apis.
> >>>>> >>> >>>
> >>>>> >>> >>>
> ---------------------------------------------------------------------
> >>>>> >>> >>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> >>>>> >>> >>> For additional commands, e-mail: dev-help@solr.apache.org
> >>>>> >>> >>>
> >>>>> >>> >>
> >>>>> >>> >> _______________________
> >>>>> >>> >> Eric Pugh | Founder & CEO | OpenSource Connections, LLC |
> 434.466.1467 | http://www.opensourceconnections.com | My Free/Busy
> >>>>> >>> >> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
> >>>>> >>> >> This e-mail and all contents, including attachments, is
> considered to be Company Confidential unless explicitly stated otherwise,
> regardless of whether attachments are marked as such.
> >>>>> >>> >>
> >>>>> >>>
> >>>>> >>>
> ---------------------------------------------------------------------
> >>>>> >>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> >>>>> >>> For additional commands, e-mail: dev-help@solr.apache.org
> >>>>> >>>
> >>>>> >>
> >>>>> >>
> >>>>> >> --
> >>>>> >> -----------------------------------------------------
> >>>>> >> Noble Paul
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> > --
> >>>>> > -----------------------------------------------------
> >>>>> > Noble Paul
> >>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> >>>>> For additional commands, e-mail: dev-help@solr.apache.org
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> http://www.needhamsoftware.com (work)
> >>>> http://www.the111shift.com (play)
> >>>
> >>>
> >>>
> >>> --
> >>> -----------------------------------------------------
> >>> Noble Paul
> >>
> >>
> >>
> >> --
> >> -----------------------------------------------------
> >> Noble Paul
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org
>
>
Re: JAX-RS APIs in Solr
Posted by Jason Gerlowski <ge...@gmail.com>.
> We need to make an API work on a per core basis and cores can come up and go down randomly
Ah, yeah. I'll admit I was overlooking some of the wrinkles around
custom request handlers etc. That is a problem for switching to some
off-the-shelf framework for sure. It may still be possible, but it's
definitely a big hurdle. I wonder how commonly users still take
advantage of the custom requestHandler stuff these days? I haven't
run across many customers who used them extensively, but maybe that's
self-selecting in some way?
> If SolrJ has a Jackson dependency, it can have a conflict (different versions) with the Jackson used by the client app
SolrJ has a handful of dependencies - is there a reason we take this
approach with Jackson, but not with any of SolrJ's other deps?
(apache-httpcomponents, apache-commons, zookeeper-client, etc.)
Granted, I get that we want SolrJ to be as painless as possible for
its users, and jar-conflicts are a part of that and really suck. But
there are good tools out there for handling jar conflicts, and
reimplementing chunks of library functionality just to avoid a
gradle-dep strikes me as unsustainable from a maintenance/community
perspective. Not to mention that forking/mimicking Jackson code
deprives our users of any efficiency/security improvements that
Jackson might get tomorrow or next week or next year.
Unless there's some reason specific to Jackson I guess I just don't
get it. But if I'm the minority opinion on that, fair enough I guess.
On Tue, Nov 30, 2021 at 5:05 AM Noble Paul <no...@gmail.com> wrote:
>
> The @JsonProperty annotations are added because it's a dependency in SolrJ as well
> If SolrJ has a Jackson dependency, it can have a conflict (different versions) with the Jackson used by the client app.
>
> On Tue, Nov 30, 2021, 5:21 PM Noble Paul <no...@gmail.com> wrote:
>>
>> Also keep in mind that the same endpoint can be accessed with a core name and a collection name prefixes.
>>
>> On Tue, Nov 30, 2021 at 3:55 PM Noble Paul <no...@gmail.com> wrote:
>>>
>>> True Gus, Almost every framework works outside of SolrDispatchFilter+ HttpSolrCall. A lot of our initializations occur there.
>>>
>>> We need to make an API work on a per core basis and cores can come up and go down randomly. So we need to register these endpoints on a core.
>>>
>>> I'm not sure if any framework can achieve the same.
>>>
>>> On Tue, Nov 30, 2021 at 5:39 AM Gus Heck <gu...@gmail.com> wrote:
>>>>
>>>> IIRC last time I looked restlet had the unsavory property of existing outside of the SolrDispatchFilter, unlike everything else which made for special cases because several things that probably ought to be their own siervlet filters are glommed into SolrDispatchFilter, like security, tracing and MDC setup/teardown per request. Restlet wouldn't be so bad if one could just wrap such filters around it too...
>>>>
>>>> On Mon, Nov 29, 2021 at 9:42 AM Jason Gerlowski <ge...@gmail.com> wrote:
>>>>>
>>>>> > These are minor improvements compared to a full rewrite of the entire framework
>>>>>
>>>>> If you think data type support is minor, fair enough. But to clarify
>>>>> I'm not suggesting a rewrite - I'm suggesting using something that
>>>>> already exists off the shelf. Jersey (e.g.) itself provides the
>>>>> framework - there would be no "rewrite".
>>>>>
>>>>> re: past restlet use
>>>>>
>>>>> > It was not playing well with our security framework. The framework was not working well with Solr APIs
>>>>>
>>>>> Ah, very interesting! Security isn't something Eric or I tackled in
>>>>> our little spike branch, but it's definitely a concern. Do you
>>>>> remember the specific concerns? Or recall where any of the discussion
>>>>> around this happened?
>>>>>
>>>>> Without the context of that past discussion, it seems like the
>>>>> "PermissionNameProvider" interface could be implemented just as well
>>>>> by a class with (e.g.) Jersey annotations as one with our own custom
>>>>> annotations. Certainly there'd need to be some
>>>>> RuleBasedAuthorizationPlugin changes or other integration code, but
>>>>> nothing that feels insurmountable.
>>>>>
>>>>> Maybe I can try spiking it out soon and find the issues myself, but
>>>>> it'd be much easier if someone happens to remember and can save me the
>>>>> trouble :-p
>>>>>
>>>>> Best,
>>>>>
>>>>> Jason
>>>>>
>>>>> On Fri, Nov 26, 2021 at 12:05 AM Noble Paul <no...@gmail.com> wrote:
>>>>> >
>>>>> > The Annotations framework was written after playing with other frameworks. There were many shortcomings which were hard to overcome.
>>>>> >
>>>>> > The best example is a per collection API . How do you register an endpoint for a collection/core ?
>>>>> >
>>>>> > On Fri, Nov 26, 2021 at 3:42 PM Noble Paul <no...@gmail.com> wrote:
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> On Fri, Nov 26, 2021 at 1:03 AM Jason Gerlowski <ge...@gmail.com> wrote:
>>>>> >>>
>>>>> >>> > Is there some problem with our annotations that we hope to solve using third party dependencies?
>>>>> >>>
>>>>> >>> I guess so yeah. Third-party deps are just fuller, more robust
>>>>> >>> solutions, whereas our annotations still need support added now and
>>>>> >>> then for even primitive data types like "long" (see SOLR-15619).
>>>>> >>
>>>>> >> These are minor improvements compared to a full rewrite of the entire framework
>>>>> >>
>>>>> >>
>>>>> >>>
>>>>> >>> Every JIRA spent doing basic stuff like that is time away from
>>>>> >>> improving Solr in some other way.
>>>>> >>>
>>>>> >>> So there are feature-gap/capabilities arguments for moving to a
>>>>> >>> third-party dep, sure. But, even if our annotations did everything
>>>>> >>> Jersey+Jackson do today, I think switching would still be worth it.
>>>>> >>> Every LOC in our code base brings along with it some maintenance cost:
>>>>> >>> it might have bugs, needs tested, takes time for new contributors to
>>>>> >>> "grok", etc. Using off-the-shelf here would nuke a whole bunch of
>>>>> >>> that. If off-the-shelf is available for some given functionality, we
>>>>> >>> should need a compelling reason to NOT use it.
>>>>> >>>
>>>>> >>> Lastly, I think there's an "approachability" argument for using
>>>>> >>> off-the-shelf. Thousands of developers out there are familiar with
>>>>> >>> (e.g.) Jersey, compared to maybe 15 or 20 (in the world) familiar with
>>>>> >>> Solr's custom annotations. Using a well-known technology like Jersey
>>>>> >>> would make Solr all the easier to approach and contribute to for that
>>>>> >>> pool of developers.
>>>>> >>>
>>>>> >>> > By the way, we have used Restlet in the past and that has been a regrettable decision.
>>>>> >>
>>>>> >>
>>>>> >>>
>>>>> >>> Ah, yeah, that's just the context I'm missing. Anyone have a pointer
>>>>> >>> to related discussions, or remember what made this "regrettable"? All
>>>>> >>> the theoretical benefits in the world don't matter much if we've
>>>>> >>> already tried something like this in the past and decided against it.
>>>>> >>
>>>>> >>
>>>>> >> It was not playing well with our security framework. The framework was not working well with Solr APIs
>>>>> >>
>>>>> >>>
>>>>> >>> (Unrelated - Happy Thanksgiving all!)
>>>>> >>>
>>>>> >>> Best,
>>>>> >>>
>>>>> >>> Jason
>>>>> >>>
>>>>> >>> On Thu, Nov 25, 2021 at 7:32 AM Noble Paul <no...@gmail.com> wrote:
>>>>> >>> >
>>>>> >>> > Have you gone through an API written using the @EndPoint annotation?
>>>>> >>> >
>>>>> >>> > I strongly recommend that you do
>>>>> >>> >
>>>>> >>> > On Thu, Nov 25, 2021, 11:30 PM Eric Pugh <ep...@opensourceconnections.com> wrote:
>>>>> >>> >>
>>>>> >>> >> I have found our V2 API code to be very impenetrable to understand. Part of it is how the code is intertwined with support for V1, however it’s also because there aren’t really resources to go look at to understand how it should work! Maintaining the API should be very simple work, as they just exist as a translation. The home grown stuff may make sense if you are a super knowledgable Solr developer, but if you are just a new person, it’s a lot harder to contribute.
>>>>> >>> >>
>>>>> >>> >> I was interested in the Jersey stuff because I’ve seen lots of projects use it very successfully, and if I want to implement something, well, there are lots of blogs and resources out there!
>>>>> >>> >>
>>>>> >>> >> Can anyone recap briefly why we dropped RESTlet? And what lessons learned there might apply to adopting Jersey for API support? Looking at https://issues.apache.org/jira/browse/SOLR-14659, it was partly deprecated because we were not using it to support all the API, only the ManagedResource ones, and https://issues.apache.org/jira/browse/SOLR-14766 suggests that RESTlet maybe was no longer being updated? One reason why we spiked out Jersey was because of the broad support in the Java world! Looking at how much work we have to do in the V2 API world, we need a much broader pool of developers contributing to get there!
>>>>> >>> >>
>>>>> >>> >> Related, are there specific features/aspects of our annotations that enable things in Solr that couldn’t be done otherwise?
>>>>> >>> >>
>>>>> >>> >> On Nov 25, 2021, at 2:12 AM, Ishan Chattopadhyaya <ic...@gmail.com> wrote:
>>>>> >>> >>
>>>>> >>> >> Is there some problem with our annotations that we hope to solve using third party dependencies?
>>>>> >>> >> By the way, we have used Restlet in the past and that has been a regrettable decision.
>>>>> >>> >>
>>>>> >>> >> On Thu, Nov 25, 2021 at 10:10 AM Jason Gerlowski <ge...@gmail.com> wrote:
>>>>> >>> >>>
>>>>> >>> >>> Solr's custom annotation framework ('@Endpoint', '@Command', etc.) has
>>>>> >>> >>> cropped up a few times over the past week or two. [1] [2]. Having them
>>>>> >>> >>> on top of mind, I've been wondering - is there a reason we use our own
>>>>> >>> >>> annotations here instead of something off the shelf?
>>>>> >>> >>>
>>>>> >>> >>> What we have works well enough, but anything homegrown comes with more
>>>>> >>> >>> maintenance burden than we'd have if we used something off the shelf.
>>>>> >>> >>> There are plenty of well-used, active projects out there whose whole
>>>>> >>> >>> purpose is facilitating the whole "annotation based API" thing
>>>>> >>> >>> (Jersey, Restlet, RESTEasy, etc.) - why not use one of them?
>>>>> >>> >>>
>>>>> >>> >>> Does anyone know of any technical reasons why we can't go this route?
>>>>> >>> >>> Or have any subjective reasons why we shouldn't? Or any context on
>>>>> >>> >>> why we wrote our own Endpoint, Command, JsonProperty annotations
>>>>> >>> >>> originally?
>>>>> >>> >>>
>>>>> >>> >>> FWIW, Eric Pugh and I spiked out a small POC recently, and got
>>>>> >>> >>> Jersey+Jackson working for a few simple APIs without too much trouble.
>>>>> >>> >>> [3] Obviously nothing production-ready there, and there's still a lot
>>>>> >>> >>> of open questions (e.g. how would javabin be supported?), but we both
>>>>> >>> >>> came away convinced that it seemed feasible, at least. Best of all,
>>>>> >>> >>> APIs using our current homegrown annotation framework the switchover
>>>>> >>> >>> seems blessedly straightforward, and it doesn't look like Jersey
>>>>> >>> >>> (which we chose mostly arbitrarily) bloats our dist all that much.
>>>>> >>> >>>
>>>>> >>> >>> Curious if anyone has thoughts or context on how we ended up with the
>>>>> >>> >>> annotation setup we use today!
>>>>> >>> >>>
>>>>> >>> >>> Best,
>>>>> >>> >>>
>>>>> >>> >>> Jason
>>>>> >>> >>>
>>>>> >>> >>> [1] https://issues.apache.org/jira/browse/SOLR-15182 (and children)
>>>>> >>> >>> [2] http://mail-archives.apache.org/mod_mbox/solr-dev/202111.mbox/%3CCABEwPvENL41Pm6%2BOmjXb6Sx5N2XjUtnbWhgKOZSrnLjWBA8tcA%40mail.gmail.com%3E
>>>>> >>> >>> [3] https://github.com/gerlowskija/solr/tree/jersey_jaxrs_jackson_solr_apis.
>>>>> >>> >>>
>>>>> >>> >>> ---------------------------------------------------------------------
>>>>> >>> >>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>>>>> >>> >>> For additional commands, e-mail: dev-help@solr.apache.org
>>>>> >>> >>>
>>>>> >>> >>
>>>>> >>> >> _______________________
>>>>> >>> >> Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | http://www.opensourceconnections.com | My Free/Busy
>>>>> >>> >> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
>>>>> >>> >> This e-mail and all contents, including attachments, is considered to be Company Confidential unless explicitly stated otherwise, regardless of whether attachments are marked as such.
>>>>> >>> >>
>>>>> >>>
>>>>> >>> ---------------------------------------------------------------------
>>>>> >>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>>>>> >>> For additional commands, e-mail: dev-help@solr.apache.org
>>>>> >>>
>>>>> >>
>>>>> >>
>>>>> >> --
>>>>> >> -----------------------------------------------------
>>>>> >> Noble Paul
>>>>> >
>>>>> >
>>>>> >
>>>>> > --
>>>>> > -----------------------------------------------------
>>>>> > Noble Paul
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>>>>> For additional commands, e-mail: dev-help@solr.apache.org
>>>>>
>>>>
>>>>
>>>> --
>>>> http://www.needhamsoftware.com (work)
>>>> http://www.the111shift.com (play)
>>>
>>>
>>>
>>> --
>>> -----------------------------------------------------
>>> Noble Paul
>>
>>
>>
>> --
>> -----------------------------------------------------
>> Noble Paul
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: JAX-RS APIs in Solr
Posted by Noble Paul <no...@gmail.com>.
The @JsonProperty annotations are added because it's a dependency in SolrJ
as well
If SolrJ has a Jackson dependency, it can have a conflict (different
versions) with the Jackson used by the client app.
On Tue, Nov 30, 2021, 5:21 PM Noble Paul <no...@gmail.com> wrote:
> Also keep in mind that the same endpoint can be accessed with a core name
> and a collection name prefixes.
>
> On Tue, Nov 30, 2021 at 3:55 PM Noble Paul <no...@gmail.com> wrote:
>
>> True Gus, Almost every framework works outside of SolrDispatchFilter+
>> HttpSolrCall. A lot of our initializations occur there.
>>
>> We need to make an API work on a per core basis and cores can come up and
>> go down randomly. So we need to register these endpoints on a core.
>>
>> I'm not sure if any framework can achieve the same.
>>
>> On Tue, Nov 30, 2021 at 5:39 AM Gus Heck <gu...@gmail.com> wrote:
>>
>>> IIRC last time I looked restlet had the unsavory property of existing
>>> outside of the SolrDispatchFilter, unlike everything else which made for
>>> special cases because several things that probably ought to be their own
>>> siervlet filters are glommed into SolrDispatchFilter, like security,
>>> tracing and MDC setup/teardown per request. Restlet wouldn't be so bad if
>>> one could just wrap such filters around it too...
>>>
>>> On Mon, Nov 29, 2021 at 9:42 AM Jason Gerlowski <ge...@gmail.com>
>>> wrote:
>>>
>>>> > These are minor improvements compared to a full rewrite of the entire
>>>> framework
>>>>
>>>> If you think data type support is minor, fair enough. But to clarify
>>>> I'm not suggesting a rewrite - I'm suggesting using something that
>>>> already exists off the shelf. Jersey (e.g.) itself provides the
>>>> framework - there would be no "rewrite".
>>>>
>>>> re: past restlet use
>>>>
>>>> > It was not playing well with our security framework. The framework
>>>> was not working well with Solr APIs
>>>>
>>>> Ah, very interesting! Security isn't something Eric or I tackled in
>>>> our little spike branch, but it's definitely a concern. Do you
>>>> remember the specific concerns? Or recall where any of the discussion
>>>> around this happened?
>>>>
>>>> Without the context of that past discussion, it seems like the
>>>> "PermissionNameProvider" interface could be implemented just as well
>>>> by a class with (e.g.) Jersey annotations as one with our own custom
>>>> annotations. Certainly there'd need to be some
>>>> RuleBasedAuthorizationPlugin changes or other integration code, but
>>>> nothing that feels insurmountable.
>>>>
>>>> Maybe I can try spiking it out soon and find the issues myself, but
>>>> it'd be much easier if someone happens to remember and can save me the
>>>> trouble :-p
>>>>
>>>> Best,
>>>>
>>>> Jason
>>>>
>>>> On Fri, Nov 26, 2021 at 12:05 AM Noble Paul <no...@gmail.com>
>>>> wrote:
>>>> >
>>>> > The Annotations framework was written after playing with other
>>>> frameworks. There were many shortcomings which were hard to overcome.
>>>> >
>>>> > The best example is a per collection API . How do you register an
>>>> endpoint for a collection/core ?
>>>> >
>>>> > On Fri, Nov 26, 2021 at 3:42 PM Noble Paul <no...@gmail.com>
>>>> wrote:
>>>> >>
>>>> >>
>>>> >>
>>>> >> On Fri, Nov 26, 2021 at 1:03 AM Jason Gerlowski <
>>>> gerlowskija@gmail.com> wrote:
>>>> >>>
>>>> >>> > Is there some problem with our annotations that we hope to solve
>>>> using third party dependencies?
>>>> >>>
>>>> >>> I guess so yeah. Third-party deps are just fuller, more robust
>>>> >>> solutions, whereas our annotations still need support added now and
>>>> >>> then for even primitive data types like "long" (see SOLR-15619).
>>>> >>
>>>> >> These are minor improvements compared to a full rewrite of the
>>>> entire framework
>>>> >>
>>>> >>
>>>> >>>
>>>> >>> Every JIRA spent doing basic stuff like that is time away from
>>>> >>> improving Solr in some other way.
>>>> >>>
>>>> >>> So there are feature-gap/capabilities arguments for moving to a
>>>> >>> third-party dep, sure. But, even if our annotations did everything
>>>> >>> Jersey+Jackson do today, I think switching would still be worth it.
>>>> >>> Every LOC in our code base brings along with it some maintenance
>>>> cost:
>>>> >>> it might have bugs, needs tested, takes time for new contributors to
>>>> >>> "grok", etc. Using off-the-shelf here would nuke a whole bunch of
>>>> >>> that. If off-the-shelf is available for some given functionality,
>>>> we
>>>> >>> should need a compelling reason to NOT use it.
>>>> >>>
>>>> >>> Lastly, I think there's an "approachability" argument for using
>>>> >>> off-the-shelf. Thousands of developers out there are familiar with
>>>> >>> (e.g.) Jersey, compared to maybe 15 or 20 (in the world) familiar
>>>> with
>>>> >>> Solr's custom annotations. Using a well-known technology like
>>>> Jersey
>>>> >>> would make Solr all the easier to approach and contribute to for
>>>> that
>>>> >>> pool of developers.
>>>> >>>
>>>> >>> > By the way, we have used Restlet in the past and that has been a
>>>> regrettable decision.
>>>> >>
>>>> >>
>>>> >>>
>>>> >>> Ah, yeah, that's just the context I'm missing. Anyone have a
>>>> pointer
>>>> >>> to related discussions, or remember what made this "regrettable"?
>>>> All
>>>> >>> the theoretical benefits in the world don't matter much if we've
>>>> >>> already tried something like this in the past and decided against
>>>> it.
>>>> >>
>>>> >>
>>>> >> It was not playing well with our security framework. The framework
>>>> was not working well with Solr APIs
>>>> >>
>>>> >>>
>>>> >>> (Unrelated - Happy Thanksgiving all!)
>>>> >>>
>>>> >>> Best,
>>>> >>>
>>>> >>> Jason
>>>> >>>
>>>> >>> On Thu, Nov 25, 2021 at 7:32 AM Noble Paul <no...@gmail.com>
>>>> wrote:
>>>> >>> >
>>>> >>> > Have you gone through an API written using the @EndPoint
>>>> annotation?
>>>> >>> >
>>>> >>> > I strongly recommend that you do
>>>> >>> >
>>>> >>> > On Thu, Nov 25, 2021, 11:30 PM Eric Pugh <
>>>> epugh@opensourceconnections.com> wrote:
>>>> >>> >>
>>>> >>> >> I have found our V2 API code to be very impenetrable to
>>>> understand. Part of it is how the code is intertwined with support for
>>>> V1, however it’s also because there aren’t really resources to go look at
>>>> to understand how it should work! Maintaining the API should be very
>>>> simple work, as they just exist as a translation. The home grown stuff
>>>> may make sense if you are a super knowledgable Solr developer, but if you
>>>> are just a new person, it’s a lot harder to contribute.
>>>> >>> >>
>>>> >>> >> I was interested in the Jersey stuff because I’ve seen lots of
>>>> projects use it very successfully, and if I want to implement something,
>>>> well, there are lots of blogs and resources out there!
>>>> >>> >>
>>>> >>> >> Can anyone recap briefly why we dropped RESTlet? And what
>>>> lessons learned there might apply to adopting Jersey for API support?
>>>> Looking at https://issues.apache.org/jira/browse/SOLR-14659, it was
>>>> partly deprecated because we were not using it to support all the API, only
>>>> the ManagedResource ones, and
>>>> https://issues.apache.org/jira/browse/SOLR-14766 suggests that RESTlet
>>>> maybe was no longer being updated? One reason why we spiked out Jersey
>>>> was because of the broad support in the Java world! Looking at how much
>>>> work we have to do in the V2 API world, we need a much broader pool of
>>>> developers contributing to get there!
>>>> >>> >>
>>>> >>> >> Related, are there specific features/aspects of our annotations
>>>> that enable things in Solr that couldn’t be done otherwise?
>>>> >>> >>
>>>> >>> >> On Nov 25, 2021, at 2:12 AM, Ishan Chattopadhyaya <
>>>> ichattopadhyaya@gmail.com> wrote:
>>>> >>> >>
>>>> >>> >> Is there some problem with our annotations that we hope to solve
>>>> using third party dependencies?
>>>> >>> >> By the way, we have used Restlet in the past and that has been a
>>>> regrettable decision.
>>>> >>> >>
>>>> >>> >> On Thu, Nov 25, 2021 at 10:10 AM Jason Gerlowski <
>>>> gerlowskija@gmail.com> wrote:
>>>> >>> >>>
>>>> >>> >>> Solr's custom annotation framework ('@Endpoint', '@Command',
>>>> etc.) has
>>>> >>> >>> cropped up a few times over the past week or two. [1] [2].
>>>> Having them
>>>> >>> >>> on top of mind, I've been wondering - is there a reason we use
>>>> our own
>>>> >>> >>> annotations here instead of something off the shelf?
>>>> >>> >>>
>>>> >>> >>> What we have works well enough, but anything homegrown comes
>>>> with more
>>>> >>> >>> maintenance burden than we'd have if we used something off the
>>>> shelf.
>>>> >>> >>> There are plenty of well-used, active projects out there whose
>>>> whole
>>>> >>> >>> purpose is facilitating the whole "annotation based API" thing
>>>> >>> >>> (Jersey, Restlet, RESTEasy, etc.) - why not use one of them?
>>>> >>> >>>
>>>> >>> >>> Does anyone know of any technical reasons why we can't go this
>>>> route?
>>>> >>> >>> Or have any subjective reasons why we shouldn't? Or any
>>>> context on
>>>> >>> >>> why we wrote our own Endpoint, Command, JsonProperty annotations
>>>> >>> >>> originally?
>>>> >>> >>>
>>>> >>> >>> FWIW, Eric Pugh and I spiked out a small POC recently, and got
>>>> >>> >>> Jersey+Jackson working for a few simple APIs without too much
>>>> trouble.
>>>> >>> >>> [3] Obviously nothing production-ready there, and there's
>>>> still a lot
>>>> >>> >>> of open questions (e.g. how would javabin be supported?), but
>>>> we both
>>>> >>> >>> came away convinced that it seemed feasible, at least. Best of
>>>> all,
>>>> >>> >>> APIs using our current homegrown annotation framework the
>>>> switchover
>>>> >>> >>> seems blessedly straightforward, and it doesn't look like Jersey
>>>> >>> >>> (which we chose mostly arbitrarily) bloats our dist all that
>>>> much.
>>>> >>> >>>
>>>> >>> >>> Curious if anyone has thoughts or context on how we ended up
>>>> with the
>>>> >>> >>> annotation setup we use today!
>>>> >>> >>>
>>>> >>> >>> Best,
>>>> >>> >>>
>>>> >>> >>> Jason
>>>> >>> >>>
>>>> >>> >>> [1] https://issues.apache.org/jira/browse/SOLR-15182 (and
>>>> children)
>>>> >>> >>> [2]
>>>> http://mail-archives.apache.org/mod_mbox/solr-dev/202111.mbox/%3CCABEwPvENL41Pm6%2BOmjXb6Sx5N2XjUtnbWhgKOZSrnLjWBA8tcA%40mail.gmail.com%3E
>>>> >>> >>> [3]
>>>> https://github.com/gerlowskija/solr/tree/jersey_jaxrs_jackson_solr_apis
>>>> .
>>>> >>> >>>
>>>> >>> >>>
>>>> ---------------------------------------------------------------------
>>>> >>> >>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>>>> >>> >>> For additional commands, e-mail: dev-help@solr.apache.org
>>>> >>> >>>
>>>> >>> >>
>>>> >>> >> _______________________
>>>> >>> >> Eric Pugh | Founder & CEO | OpenSource Connections, LLC |
>>>> 434.466.1467 | http://www.opensourceconnections.com | My Free/Busy
>>>> >>> >> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
>>>> >>> >> This e-mail and all contents, including attachments, is
>>>> considered to be Company Confidential unless explicitly stated otherwise,
>>>> regardless of whether attachments are marked as such.
>>>> >>> >>
>>>> >>>
>>>> >>>
>>>> ---------------------------------------------------------------------
>>>> >>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>>>> >>> For additional commands, e-mail: dev-help@solr.apache.org
>>>> >>>
>>>> >>
>>>> >>
>>>> >> --
>>>> >> -----------------------------------------------------
>>>> >> Noble Paul
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > -----------------------------------------------------
>>>> > Noble Paul
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>>>> For additional commands, e-mail: dev-help@solr.apache.org
>>>>
>>>>
>>>
>>> --
>>> http://www.needhamsoftware.com (work)
>>> http://www.the111shift.com (play)
>>>
>>
>>
>> --
>> -----------------------------------------------------
>> Noble Paul
>>
>
>
> --
> -----------------------------------------------------
> Noble Paul
>
Re: JAX-RS APIs in Solr
Posted by Noble Paul <no...@gmail.com>.
Also keep in mind that the same endpoint can be accessed with a core name
and a collection name prefixes.
On Tue, Nov 30, 2021 at 3:55 PM Noble Paul <no...@gmail.com> wrote:
> True Gus, Almost every framework works outside of SolrDispatchFilter+
> HttpSolrCall. A lot of our initializations occur there.
>
> We need to make an API work on a per core basis and cores can come up and
> go down randomly. So we need to register these endpoints on a core.
>
> I'm not sure if any framework can achieve the same.
>
> On Tue, Nov 30, 2021 at 5:39 AM Gus Heck <gu...@gmail.com> wrote:
>
>> IIRC last time I looked restlet had the unsavory property of existing
>> outside of the SolrDispatchFilter, unlike everything else which made for
>> special cases because several things that probably ought to be their own
>> siervlet filters are glommed into SolrDispatchFilter, like security,
>> tracing and MDC setup/teardown per request. Restlet wouldn't be so bad if
>> one could just wrap such filters around it too...
>>
>> On Mon, Nov 29, 2021 at 9:42 AM Jason Gerlowski <ge...@gmail.com>
>> wrote:
>>
>>> > These are minor improvements compared to a full rewrite of the entire
>>> framework
>>>
>>> If you think data type support is minor, fair enough. But to clarify
>>> I'm not suggesting a rewrite - I'm suggesting using something that
>>> already exists off the shelf. Jersey (e.g.) itself provides the
>>> framework - there would be no "rewrite".
>>>
>>> re: past restlet use
>>>
>>> > It was not playing well with our security framework. The framework was
>>> not working well with Solr APIs
>>>
>>> Ah, very interesting! Security isn't something Eric or I tackled in
>>> our little spike branch, but it's definitely a concern. Do you
>>> remember the specific concerns? Or recall where any of the discussion
>>> around this happened?
>>>
>>> Without the context of that past discussion, it seems like the
>>> "PermissionNameProvider" interface could be implemented just as well
>>> by a class with (e.g.) Jersey annotations as one with our own custom
>>> annotations. Certainly there'd need to be some
>>> RuleBasedAuthorizationPlugin changes or other integration code, but
>>> nothing that feels insurmountable.
>>>
>>> Maybe I can try spiking it out soon and find the issues myself, but
>>> it'd be much easier if someone happens to remember and can save me the
>>> trouble :-p
>>>
>>> Best,
>>>
>>> Jason
>>>
>>> On Fri, Nov 26, 2021 at 12:05 AM Noble Paul <no...@gmail.com>
>>> wrote:
>>> >
>>> > The Annotations framework was written after playing with other
>>> frameworks. There were many shortcomings which were hard to overcome.
>>> >
>>> > The best example is a per collection API . How do you register an
>>> endpoint for a collection/core ?
>>> >
>>> > On Fri, Nov 26, 2021 at 3:42 PM Noble Paul <no...@gmail.com>
>>> wrote:
>>> >>
>>> >>
>>> >>
>>> >> On Fri, Nov 26, 2021 at 1:03 AM Jason Gerlowski <
>>> gerlowskija@gmail.com> wrote:
>>> >>>
>>> >>> > Is there some problem with our annotations that we hope to solve
>>> using third party dependencies?
>>> >>>
>>> >>> I guess so yeah. Third-party deps are just fuller, more robust
>>> >>> solutions, whereas our annotations still need support added now and
>>> >>> then for even primitive data types like "long" (see SOLR-15619).
>>> >>
>>> >> These are minor improvements compared to a full rewrite of the entire
>>> framework
>>> >>
>>> >>
>>> >>>
>>> >>> Every JIRA spent doing basic stuff like that is time away from
>>> >>> improving Solr in some other way.
>>> >>>
>>> >>> So there are feature-gap/capabilities arguments for moving to a
>>> >>> third-party dep, sure. But, even if our annotations did everything
>>> >>> Jersey+Jackson do today, I think switching would still be worth it.
>>> >>> Every LOC in our code base brings along with it some maintenance
>>> cost:
>>> >>> it might have bugs, needs tested, takes time for new contributors to
>>> >>> "grok", etc. Using off-the-shelf here would nuke a whole bunch of
>>> >>> that. If off-the-shelf is available for some given functionality, we
>>> >>> should need a compelling reason to NOT use it.
>>> >>>
>>> >>> Lastly, I think there's an "approachability" argument for using
>>> >>> off-the-shelf. Thousands of developers out there are familiar with
>>> >>> (e.g.) Jersey, compared to maybe 15 or 20 (in the world) familiar
>>> with
>>> >>> Solr's custom annotations. Using a well-known technology like Jersey
>>> >>> would make Solr all the easier to approach and contribute to for that
>>> >>> pool of developers.
>>> >>>
>>> >>> > By the way, we have used Restlet in the past and that has been a
>>> regrettable decision.
>>> >>
>>> >>
>>> >>>
>>> >>> Ah, yeah, that's just the context I'm missing. Anyone have a pointer
>>> >>> to related discussions, or remember what made this "regrettable"?
>>> All
>>> >>> the theoretical benefits in the world don't matter much if we've
>>> >>> already tried something like this in the past and decided against it.
>>> >>
>>> >>
>>> >> It was not playing well with our security framework. The framework
>>> was not working well with Solr APIs
>>> >>
>>> >>>
>>> >>> (Unrelated - Happy Thanksgiving all!)
>>> >>>
>>> >>> Best,
>>> >>>
>>> >>> Jason
>>> >>>
>>> >>> On Thu, Nov 25, 2021 at 7:32 AM Noble Paul <no...@gmail.com>
>>> wrote:
>>> >>> >
>>> >>> > Have you gone through an API written using the @EndPoint
>>> annotation?
>>> >>> >
>>> >>> > I strongly recommend that you do
>>> >>> >
>>> >>> > On Thu, Nov 25, 2021, 11:30 PM Eric Pugh <
>>> epugh@opensourceconnections.com> wrote:
>>> >>> >>
>>> >>> >> I have found our V2 API code to be very impenetrable to
>>> understand. Part of it is how the code is intertwined with support for
>>> V1, however it’s also because there aren’t really resources to go look at
>>> to understand how it should work! Maintaining the API should be very
>>> simple work, as they just exist as a translation. The home grown stuff
>>> may make sense if you are a super knowledgable Solr developer, but if you
>>> are just a new person, it’s a lot harder to contribute.
>>> >>> >>
>>> >>> >> I was interested in the Jersey stuff because I’ve seen lots of
>>> projects use it very successfully, and if I want to implement something,
>>> well, there are lots of blogs and resources out there!
>>> >>> >>
>>> >>> >> Can anyone recap briefly why we dropped RESTlet? And what
>>> lessons learned there might apply to adopting Jersey for API support?
>>> Looking at https://issues.apache.org/jira/browse/SOLR-14659, it was
>>> partly deprecated because we were not using it to support all the API, only
>>> the ManagedResource ones, and
>>> https://issues.apache.org/jira/browse/SOLR-14766 suggests that RESTlet
>>> maybe was no longer being updated? One reason why we spiked out Jersey
>>> was because of the broad support in the Java world! Looking at how much
>>> work we have to do in the V2 API world, we need a much broader pool of
>>> developers contributing to get there!
>>> >>> >>
>>> >>> >> Related, are there specific features/aspects of our annotations
>>> that enable things in Solr that couldn’t be done otherwise?
>>> >>> >>
>>> >>> >> On Nov 25, 2021, at 2:12 AM, Ishan Chattopadhyaya <
>>> ichattopadhyaya@gmail.com> wrote:
>>> >>> >>
>>> >>> >> Is there some problem with our annotations that we hope to solve
>>> using third party dependencies?
>>> >>> >> By the way, we have used Restlet in the past and that has been a
>>> regrettable decision.
>>> >>> >>
>>> >>> >> On Thu, Nov 25, 2021 at 10:10 AM Jason Gerlowski <
>>> gerlowskija@gmail.com> wrote:
>>> >>> >>>
>>> >>> >>> Solr's custom annotation framework ('@Endpoint', '@Command',
>>> etc.) has
>>> >>> >>> cropped up a few times over the past week or two. [1] [2].
>>> Having them
>>> >>> >>> on top of mind, I've been wondering - is there a reason we use
>>> our own
>>> >>> >>> annotations here instead of something off the shelf?
>>> >>> >>>
>>> >>> >>> What we have works well enough, but anything homegrown comes
>>> with more
>>> >>> >>> maintenance burden than we'd have if we used something off the
>>> shelf.
>>> >>> >>> There are plenty of well-used, active projects out there whose
>>> whole
>>> >>> >>> purpose is facilitating the whole "annotation based API" thing
>>> >>> >>> (Jersey, Restlet, RESTEasy, etc.) - why not use one of them?
>>> >>> >>>
>>> >>> >>> Does anyone know of any technical reasons why we can't go this
>>> route?
>>> >>> >>> Or have any subjective reasons why we shouldn't? Or any context
>>> on
>>> >>> >>> why we wrote our own Endpoint, Command, JsonProperty annotations
>>> >>> >>> originally?
>>> >>> >>>
>>> >>> >>> FWIW, Eric Pugh and I spiked out a small POC recently, and got
>>> >>> >>> Jersey+Jackson working for a few simple APIs without too much
>>> trouble.
>>> >>> >>> [3] Obviously nothing production-ready there, and there's still
>>> a lot
>>> >>> >>> of open questions (e.g. how would javabin be supported?), but we
>>> both
>>> >>> >>> came away convinced that it seemed feasible, at least. Best of
>>> all,
>>> >>> >>> APIs using our current homegrown annotation framework the
>>> switchover
>>> >>> >>> seems blessedly straightforward, and it doesn't look like Jersey
>>> >>> >>> (which we chose mostly arbitrarily) bloats our dist all that
>>> much.
>>> >>> >>>
>>> >>> >>> Curious if anyone has thoughts or context on how we ended up
>>> with the
>>> >>> >>> annotation setup we use today!
>>> >>> >>>
>>> >>> >>> Best,
>>> >>> >>>
>>> >>> >>> Jason
>>> >>> >>>
>>> >>> >>> [1] https://issues.apache.org/jira/browse/SOLR-15182 (and
>>> children)
>>> >>> >>> [2]
>>> http://mail-archives.apache.org/mod_mbox/solr-dev/202111.mbox/%3CCABEwPvENL41Pm6%2BOmjXb6Sx5N2XjUtnbWhgKOZSrnLjWBA8tcA%40mail.gmail.com%3E
>>> >>> >>> [3]
>>> https://github.com/gerlowskija/solr/tree/jersey_jaxrs_jackson_solr_apis.
>>> >>> >>>
>>> >>> >>>
>>> ---------------------------------------------------------------------
>>> >>> >>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>>> >>> >>> For additional commands, e-mail: dev-help@solr.apache.org
>>> >>> >>>
>>> >>> >>
>>> >>> >> _______________________
>>> >>> >> Eric Pugh | Founder & CEO | OpenSource Connections, LLC |
>>> 434.466.1467 | http://www.opensourceconnections.com | My Free/Busy
>>> >>> >> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
>>> >>> >> This e-mail and all contents, including attachments, is
>>> considered to be Company Confidential unless explicitly stated otherwise,
>>> regardless of whether attachments are marked as such.
>>> >>> >>
>>> >>>
>>> >>> ---------------------------------------------------------------------
>>> >>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>>> >>> For additional commands, e-mail: dev-help@solr.apache.org
>>> >>>
>>> >>
>>> >>
>>> >> --
>>> >> -----------------------------------------------------
>>> >> Noble Paul
>>> >
>>> >
>>> >
>>> > --
>>> > -----------------------------------------------------
>>> > Noble Paul
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>>> For additional commands, e-mail: dev-help@solr.apache.org
>>>
>>>
>>
>> --
>> http://www.needhamsoftware.com (work)
>> http://www.the111shift.com (play)
>>
>
>
> --
> -----------------------------------------------------
> Noble Paul
>
--
-----------------------------------------------------
Noble Paul
Re: JAX-RS APIs in Solr
Posted by Noble Paul <no...@gmail.com>.
True Gus, Almost every framework works outside of SolrDispatchFilter+
HttpSolrCall. A lot of our initializations occur there.
We need to make an API work on a per core basis and cores can come up and
go down randomly. So we need to register these endpoints on a core.
I'm not sure if any framework can achieve the same.
On Tue, Nov 30, 2021 at 5:39 AM Gus Heck <gu...@gmail.com> wrote:
> IIRC last time I looked restlet had the unsavory property of existing
> outside of the SolrDispatchFilter, unlike everything else which made for
> special cases because several things that probably ought to be their own
> siervlet filters are glommed into SolrDispatchFilter, like security,
> tracing and MDC setup/teardown per request. Restlet wouldn't be so bad if
> one could just wrap such filters around it too...
>
> On Mon, Nov 29, 2021 at 9:42 AM Jason Gerlowski <ge...@gmail.com>
> wrote:
>
>> > These are minor improvements compared to a full rewrite of the entire
>> framework
>>
>> If you think data type support is minor, fair enough. But to clarify
>> I'm not suggesting a rewrite - I'm suggesting using something that
>> already exists off the shelf. Jersey (e.g.) itself provides the
>> framework - there would be no "rewrite".
>>
>> re: past restlet use
>>
>> > It was not playing well with our security framework. The framework was
>> not working well with Solr APIs
>>
>> Ah, very interesting! Security isn't something Eric or I tackled in
>> our little spike branch, but it's definitely a concern. Do you
>> remember the specific concerns? Or recall where any of the discussion
>> around this happened?
>>
>> Without the context of that past discussion, it seems like the
>> "PermissionNameProvider" interface could be implemented just as well
>> by a class with (e.g.) Jersey annotations as one with our own custom
>> annotations. Certainly there'd need to be some
>> RuleBasedAuthorizationPlugin changes or other integration code, but
>> nothing that feels insurmountable.
>>
>> Maybe I can try spiking it out soon and find the issues myself, but
>> it'd be much easier if someone happens to remember and can save me the
>> trouble :-p
>>
>> Best,
>>
>> Jason
>>
>> On Fri, Nov 26, 2021 at 12:05 AM Noble Paul <no...@gmail.com> wrote:
>> >
>> > The Annotations framework was written after playing with other
>> frameworks. There were many shortcomings which were hard to overcome.
>> >
>> > The best example is a per collection API . How do you register an
>> endpoint for a collection/core ?
>> >
>> > On Fri, Nov 26, 2021 at 3:42 PM Noble Paul <no...@gmail.com>
>> wrote:
>> >>
>> >>
>> >>
>> >> On Fri, Nov 26, 2021 at 1:03 AM Jason Gerlowski <ge...@gmail.com>
>> wrote:
>> >>>
>> >>> > Is there some problem with our annotations that we hope to solve
>> using third party dependencies?
>> >>>
>> >>> I guess so yeah. Third-party deps are just fuller, more robust
>> >>> solutions, whereas our annotations still need support added now and
>> >>> then for even primitive data types like "long" (see SOLR-15619).
>> >>
>> >> These are minor improvements compared to a full rewrite of the entire
>> framework
>> >>
>> >>
>> >>>
>> >>> Every JIRA spent doing basic stuff like that is time away from
>> >>> improving Solr in some other way.
>> >>>
>> >>> So there are feature-gap/capabilities arguments for moving to a
>> >>> third-party dep, sure. But, even if our annotations did everything
>> >>> Jersey+Jackson do today, I think switching would still be worth it.
>> >>> Every LOC in our code base brings along with it some maintenance cost:
>> >>> it might have bugs, needs tested, takes time for new contributors to
>> >>> "grok", etc. Using off-the-shelf here would nuke a whole bunch of
>> >>> that. If off-the-shelf is available for some given functionality, we
>> >>> should need a compelling reason to NOT use it.
>> >>>
>> >>> Lastly, I think there's an "approachability" argument for using
>> >>> off-the-shelf. Thousands of developers out there are familiar with
>> >>> (e.g.) Jersey, compared to maybe 15 or 20 (in the world) familiar with
>> >>> Solr's custom annotations. Using a well-known technology like Jersey
>> >>> would make Solr all the easier to approach and contribute to for that
>> >>> pool of developers.
>> >>>
>> >>> > By the way, we have used Restlet in the past and that has been a
>> regrettable decision.
>> >>
>> >>
>> >>>
>> >>> Ah, yeah, that's just the context I'm missing. Anyone have a pointer
>> >>> to related discussions, or remember what made this "regrettable"? All
>> >>> the theoretical benefits in the world don't matter much if we've
>> >>> already tried something like this in the past and decided against it.
>> >>
>> >>
>> >> It was not playing well with our security framework. The framework was
>> not working well with Solr APIs
>> >>
>> >>>
>> >>> (Unrelated - Happy Thanksgiving all!)
>> >>>
>> >>> Best,
>> >>>
>> >>> Jason
>> >>>
>> >>> On Thu, Nov 25, 2021 at 7:32 AM Noble Paul <no...@gmail.com>
>> wrote:
>> >>> >
>> >>> > Have you gone through an API written using the @EndPoint annotation?
>> >>> >
>> >>> > I strongly recommend that you do
>> >>> >
>> >>> > On Thu, Nov 25, 2021, 11:30 PM Eric Pugh <
>> epugh@opensourceconnections.com> wrote:
>> >>> >>
>> >>> >> I have found our V2 API code to be very impenetrable to
>> understand. Part of it is how the code is intertwined with support for
>> V1, however it’s also because there aren’t really resources to go look at
>> to understand how it should work! Maintaining the API should be very
>> simple work, as they just exist as a translation. The home grown stuff
>> may make sense if you are a super knowledgable Solr developer, but if you
>> are just a new person, it’s a lot harder to contribute.
>> >>> >>
>> >>> >> I was interested in the Jersey stuff because I’ve seen lots of
>> projects use it very successfully, and if I want to implement something,
>> well, there are lots of blogs and resources out there!
>> >>> >>
>> >>> >> Can anyone recap briefly why we dropped RESTlet? And what
>> lessons learned there might apply to adopting Jersey for API support?
>> Looking at https://issues.apache.org/jira/browse/SOLR-14659, it was
>> partly deprecated because we were not using it to support all the API, only
>> the ManagedResource ones, and
>> https://issues.apache.org/jira/browse/SOLR-14766 suggests that RESTlet
>> maybe was no longer being updated? One reason why we spiked out Jersey
>> was because of the broad support in the Java world! Looking at how much
>> work we have to do in the V2 API world, we need a much broader pool of
>> developers contributing to get there!
>> >>> >>
>> >>> >> Related, are there specific features/aspects of our annotations
>> that enable things in Solr that couldn’t be done otherwise?
>> >>> >>
>> >>> >> On Nov 25, 2021, at 2:12 AM, Ishan Chattopadhyaya <
>> ichattopadhyaya@gmail.com> wrote:
>> >>> >>
>> >>> >> Is there some problem with our annotations that we hope to solve
>> using third party dependencies?
>> >>> >> By the way, we have used Restlet in the past and that has been a
>> regrettable decision.
>> >>> >>
>> >>> >> On Thu, Nov 25, 2021 at 10:10 AM Jason Gerlowski <
>> gerlowskija@gmail.com> wrote:
>> >>> >>>
>> >>> >>> Solr's custom annotation framework ('@Endpoint', '@Command',
>> etc.) has
>> >>> >>> cropped up a few times over the past week or two. [1] [2]. Having
>> them
>> >>> >>> on top of mind, I've been wondering - is there a reason we use
>> our own
>> >>> >>> annotations here instead of something off the shelf?
>> >>> >>>
>> >>> >>> What we have works well enough, but anything homegrown comes with
>> more
>> >>> >>> maintenance burden than we'd have if we used something off the
>> shelf.
>> >>> >>> There are plenty of well-used, active projects out there whose
>> whole
>> >>> >>> purpose is facilitating the whole "annotation based API" thing
>> >>> >>> (Jersey, Restlet, RESTEasy, etc.) - why not use one of them?
>> >>> >>>
>> >>> >>> Does anyone know of any technical reasons why we can't go this
>> route?
>> >>> >>> Or have any subjective reasons why we shouldn't? Or any context
>> on
>> >>> >>> why we wrote our own Endpoint, Command, JsonProperty annotations
>> >>> >>> originally?
>> >>> >>>
>> >>> >>> FWIW, Eric Pugh and I spiked out a small POC recently, and got
>> >>> >>> Jersey+Jackson working for a few simple APIs without too much
>> trouble.
>> >>> >>> [3] Obviously nothing production-ready there, and there's still
>> a lot
>> >>> >>> of open questions (e.g. how would javabin be supported?), but we
>> both
>> >>> >>> came away convinced that it seemed feasible, at least. Best of
>> all,
>> >>> >>> APIs using our current homegrown annotation framework the
>> switchover
>> >>> >>> seems blessedly straightforward, and it doesn't look like Jersey
>> >>> >>> (which we chose mostly arbitrarily) bloats our dist all that much.
>> >>> >>>
>> >>> >>> Curious if anyone has thoughts or context on how we ended up with
>> the
>> >>> >>> annotation setup we use today!
>> >>> >>>
>> >>> >>> Best,
>> >>> >>>
>> >>> >>> Jason
>> >>> >>>
>> >>> >>> [1] https://issues.apache.org/jira/browse/SOLR-15182 (and
>> children)
>> >>> >>> [2]
>> http://mail-archives.apache.org/mod_mbox/solr-dev/202111.mbox/%3CCABEwPvENL41Pm6%2BOmjXb6Sx5N2XjUtnbWhgKOZSrnLjWBA8tcA%40mail.gmail.com%3E
>> >>> >>> [3]
>> https://github.com/gerlowskija/solr/tree/jersey_jaxrs_jackson_solr_apis.
>> >>> >>>
>> >>> >>>
>> ---------------------------------------------------------------------
>> >>> >>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>> >>> >>> For additional commands, e-mail: dev-help@solr.apache.org
>> >>> >>>
>> >>> >>
>> >>> >> _______________________
>> >>> >> Eric Pugh | Founder & CEO | OpenSource Connections, LLC |
>> 434.466.1467 | http://www.opensourceconnections.com | My Free/Busy
>> >>> >> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
>> >>> >> This e-mail and all contents, including attachments, is considered
>> to be Company Confidential unless explicitly stated otherwise, regardless
>> of whether attachments are marked as such.
>> >>> >>
>> >>>
>> >>> ---------------------------------------------------------------------
>> >>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>> >>> For additional commands, e-mail: dev-help@solr.apache.org
>> >>>
>> >>
>> >>
>> >> --
>> >> -----------------------------------------------------
>> >> Noble Paul
>> >
>> >
>> >
>> > --
>> > -----------------------------------------------------
>> > Noble Paul
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>> For additional commands, e-mail: dev-help@solr.apache.org
>>
>>
>
> --
> http://www.needhamsoftware.com (work)
> http://www.the111shift.com (play)
>
--
-----------------------------------------------------
Noble Paul
Re: JAX-RS APIs in Solr
Posted by Gus Heck <gu...@gmail.com>.
IIRC last time I looked restlet had the unsavory property of existing
outside of the SolrDispatchFilter, unlike everything else which made for
special cases because several things that probably ought to be their own
siervlet filters are glommed into SolrDispatchFilter, like security,
tracing and MDC setup/teardown per request. Restlet wouldn't be so bad if
one could just wrap such filters around it too...
On Mon, Nov 29, 2021 at 9:42 AM Jason Gerlowski <ge...@gmail.com>
wrote:
> > These are minor improvements compared to a full rewrite of the entire
> framework
>
> If you think data type support is minor, fair enough. But to clarify
> I'm not suggesting a rewrite - I'm suggesting using something that
> already exists off the shelf. Jersey (e.g.) itself provides the
> framework - there would be no "rewrite".
>
> re: past restlet use
>
> > It was not playing well with our security framework. The framework was
> not working well with Solr APIs
>
> Ah, very interesting! Security isn't something Eric or I tackled in
> our little spike branch, but it's definitely a concern. Do you
> remember the specific concerns? Or recall where any of the discussion
> around this happened?
>
> Without the context of that past discussion, it seems like the
> "PermissionNameProvider" interface could be implemented just as well
> by a class with (e.g.) Jersey annotations as one with our own custom
> annotations. Certainly there'd need to be some
> RuleBasedAuthorizationPlugin changes or other integration code, but
> nothing that feels insurmountable.
>
> Maybe I can try spiking it out soon and find the issues myself, but
> it'd be much easier if someone happens to remember and can save me the
> trouble :-p
>
> Best,
>
> Jason
>
> On Fri, Nov 26, 2021 at 12:05 AM Noble Paul <no...@gmail.com> wrote:
> >
> > The Annotations framework was written after playing with other
> frameworks. There were many shortcomings which were hard to overcome.
> >
> > The best example is a per collection API . How do you register an
> endpoint for a collection/core ?
> >
> > On Fri, Nov 26, 2021 at 3:42 PM Noble Paul <no...@gmail.com> wrote:
> >>
> >>
> >>
> >> On Fri, Nov 26, 2021 at 1:03 AM Jason Gerlowski <ge...@gmail.com>
> wrote:
> >>>
> >>> > Is there some problem with our annotations that we hope to solve
> using third party dependencies?
> >>>
> >>> I guess so yeah. Third-party deps are just fuller, more robust
> >>> solutions, whereas our annotations still need support added now and
> >>> then for even primitive data types like "long" (see SOLR-15619).
> >>
> >> These are minor improvements compared to a full rewrite of the entire
> framework
> >>
> >>
> >>>
> >>> Every JIRA spent doing basic stuff like that is time away from
> >>> improving Solr in some other way.
> >>>
> >>> So there are feature-gap/capabilities arguments for moving to a
> >>> third-party dep, sure. But, even if our annotations did everything
> >>> Jersey+Jackson do today, I think switching would still be worth it.
> >>> Every LOC in our code base brings along with it some maintenance cost:
> >>> it might have bugs, needs tested, takes time for new contributors to
> >>> "grok", etc. Using off-the-shelf here would nuke a whole bunch of
> >>> that. If off-the-shelf is available for some given functionality, we
> >>> should need a compelling reason to NOT use it.
> >>>
> >>> Lastly, I think there's an "approachability" argument for using
> >>> off-the-shelf. Thousands of developers out there are familiar with
> >>> (e.g.) Jersey, compared to maybe 15 or 20 (in the world) familiar with
> >>> Solr's custom annotations. Using a well-known technology like Jersey
> >>> would make Solr all the easier to approach and contribute to for that
> >>> pool of developers.
> >>>
> >>> > By the way, we have used Restlet in the past and that has been a
> regrettable decision.
> >>
> >>
> >>>
> >>> Ah, yeah, that's just the context I'm missing. Anyone have a pointer
> >>> to related discussions, or remember what made this "regrettable"? All
> >>> the theoretical benefits in the world don't matter much if we've
> >>> already tried something like this in the past and decided against it.
> >>
> >>
> >> It was not playing well with our security framework. The framework was
> not working well with Solr APIs
> >>
> >>>
> >>> (Unrelated - Happy Thanksgiving all!)
> >>>
> >>> Best,
> >>>
> >>> Jason
> >>>
> >>> On Thu, Nov 25, 2021 at 7:32 AM Noble Paul <no...@gmail.com>
> wrote:
> >>> >
> >>> > Have you gone through an API written using the @EndPoint annotation?
> >>> >
> >>> > I strongly recommend that you do
> >>> >
> >>> > On Thu, Nov 25, 2021, 11:30 PM Eric Pugh <
> epugh@opensourceconnections.com> wrote:
> >>> >>
> >>> >> I have found our V2 API code to be very impenetrable to
> understand. Part of it is how the code is intertwined with support for
> V1, however it’s also because there aren’t really resources to go look at
> to understand how it should work! Maintaining the API should be very
> simple work, as they just exist as a translation. The home grown stuff
> may make sense if you are a super knowledgable Solr developer, but if you
> are just a new person, it’s a lot harder to contribute.
> >>> >>
> >>> >> I was interested in the Jersey stuff because I’ve seen lots of
> projects use it very successfully, and if I want to implement something,
> well, there are lots of blogs and resources out there!
> >>> >>
> >>> >> Can anyone recap briefly why we dropped RESTlet? And what lessons
> learned there might apply to adopting Jersey for API support? Looking at
> https://issues.apache.org/jira/browse/SOLR-14659, it was partly
> deprecated because we were not using it to support all the API, only the
> ManagedResource ones, and https://issues.apache.org/jira/browse/SOLR-14766
> suggests that RESTlet maybe was no longer being updated? One reason why
> we spiked out Jersey was because of the broad support in the Java world!
> Looking at how much work we have to do in the V2 API world, we need a much
> broader pool of developers contributing to get there!
> >>> >>
> >>> >> Related, are there specific features/aspects of our annotations
> that enable things in Solr that couldn’t be done otherwise?
> >>> >>
> >>> >> On Nov 25, 2021, at 2:12 AM, Ishan Chattopadhyaya <
> ichattopadhyaya@gmail.com> wrote:
> >>> >>
> >>> >> Is there some problem with our annotations that we hope to solve
> using third party dependencies?
> >>> >> By the way, we have used Restlet in the past and that has been a
> regrettable decision.
> >>> >>
> >>> >> On Thu, Nov 25, 2021 at 10:10 AM Jason Gerlowski <
> gerlowskija@gmail.com> wrote:
> >>> >>>
> >>> >>> Solr's custom annotation framework ('@Endpoint', '@Command', etc.)
> has
> >>> >>> cropped up a few times over the past week or two. [1] [2]. Having
> them
> >>> >>> on top of mind, I've been wondering - is there a reason we use our
> own
> >>> >>> annotations here instead of something off the shelf?
> >>> >>>
> >>> >>> What we have works well enough, but anything homegrown comes with
> more
> >>> >>> maintenance burden than we'd have if we used something off the
> shelf.
> >>> >>> There are plenty of well-used, active projects out there whose
> whole
> >>> >>> purpose is facilitating the whole "annotation based API" thing
> >>> >>> (Jersey, Restlet, RESTEasy, etc.) - why not use one of them?
> >>> >>>
> >>> >>> Does anyone know of any technical reasons why we can't go this
> route?
> >>> >>> Or have any subjective reasons why we shouldn't? Or any context on
> >>> >>> why we wrote our own Endpoint, Command, JsonProperty annotations
> >>> >>> originally?
> >>> >>>
> >>> >>> FWIW, Eric Pugh and I spiked out a small POC recently, and got
> >>> >>> Jersey+Jackson working for a few simple APIs without too much
> trouble.
> >>> >>> [3] Obviously nothing production-ready there, and there's still a
> lot
> >>> >>> of open questions (e.g. how would javabin be supported?), but we
> both
> >>> >>> came away convinced that it seemed feasible, at least. Best of
> all,
> >>> >>> APIs using our current homegrown annotation framework the
> switchover
> >>> >>> seems blessedly straightforward, and it doesn't look like Jersey
> >>> >>> (which we chose mostly arbitrarily) bloats our dist all that much.
> >>> >>>
> >>> >>> Curious if anyone has thoughts or context on how we ended up with
> the
> >>> >>> annotation setup we use today!
> >>> >>>
> >>> >>> Best,
> >>> >>>
> >>> >>> Jason
> >>> >>>
> >>> >>> [1] https://issues.apache.org/jira/browse/SOLR-15182 (and
> children)
> >>> >>> [2]
> http://mail-archives.apache.org/mod_mbox/solr-dev/202111.mbox/%3CCABEwPvENL41Pm6%2BOmjXb6Sx5N2XjUtnbWhgKOZSrnLjWBA8tcA%40mail.gmail.com%3E
> >>> >>> [3]
> https://github.com/gerlowskija/solr/tree/jersey_jaxrs_jackson_solr_apis.
> >>> >>>
> >>> >>>
> ---------------------------------------------------------------------
> >>> >>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> >>> >>> For additional commands, e-mail: dev-help@solr.apache.org
> >>> >>>
> >>> >>
> >>> >> _______________________
> >>> >> Eric Pugh | Founder & CEO | OpenSource Connections, LLC |
> 434.466.1467 | http://www.opensourceconnections.com | My Free/Busy
> >>> >> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
> >>> >> This e-mail and all contents, including attachments, is considered
> to be Company Confidential unless explicitly stated otherwise, regardless
> of whether attachments are marked as such.
> >>> >>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> >>> For additional commands, e-mail: dev-help@solr.apache.org
> >>>
> >>
> >>
> >> --
> >> -----------------------------------------------------
> >> Noble Paul
> >
> >
> >
> > --
> > -----------------------------------------------------
> > Noble Paul
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org
>
>
--
http://www.needhamsoftware.com (work)
http://www.the111shift.com (play)
Re: JAX-RS APIs in Solr
Posted by Jason Gerlowski <ge...@gmail.com>.
> These are minor improvements compared to a full rewrite of the entire framework
If you think data type support is minor, fair enough. But to clarify
I'm not suggesting a rewrite - I'm suggesting using something that
already exists off the shelf. Jersey (e.g.) itself provides the
framework - there would be no "rewrite".
re: past restlet use
> It was not playing well with our security framework. The framework was not working well with Solr APIs
Ah, very interesting! Security isn't something Eric or I tackled in
our little spike branch, but it's definitely a concern. Do you
remember the specific concerns? Or recall where any of the discussion
around this happened?
Without the context of that past discussion, it seems like the
"PermissionNameProvider" interface could be implemented just as well
by a class with (e.g.) Jersey annotations as one with our own custom
annotations. Certainly there'd need to be some
RuleBasedAuthorizationPlugin changes or other integration code, but
nothing that feels insurmountable.
Maybe I can try spiking it out soon and find the issues myself, but
it'd be much easier if someone happens to remember and can save me the
trouble :-p
Best,
Jason
On Fri, Nov 26, 2021 at 12:05 AM Noble Paul <no...@gmail.com> wrote:
>
> The Annotations framework was written after playing with other frameworks. There were many shortcomings which were hard to overcome.
>
> The best example is a per collection API . How do you register an endpoint for a collection/core ?
>
> On Fri, Nov 26, 2021 at 3:42 PM Noble Paul <no...@gmail.com> wrote:
>>
>>
>>
>> On Fri, Nov 26, 2021 at 1:03 AM Jason Gerlowski <ge...@gmail.com> wrote:
>>>
>>> > Is there some problem with our annotations that we hope to solve using third party dependencies?
>>>
>>> I guess so yeah. Third-party deps are just fuller, more robust
>>> solutions, whereas our annotations still need support added now and
>>> then for even primitive data types like "long" (see SOLR-15619).
>>
>> These are minor improvements compared to a full rewrite of the entire framework
>>
>>
>>>
>>> Every JIRA spent doing basic stuff like that is time away from
>>> improving Solr in some other way.
>>>
>>> So there are feature-gap/capabilities arguments for moving to a
>>> third-party dep, sure. But, even if our annotations did everything
>>> Jersey+Jackson do today, I think switching would still be worth it.
>>> Every LOC in our code base brings along with it some maintenance cost:
>>> it might have bugs, needs tested, takes time for new contributors to
>>> "grok", etc. Using off-the-shelf here would nuke a whole bunch of
>>> that. If off-the-shelf is available for some given functionality, we
>>> should need a compelling reason to NOT use it.
>>>
>>> Lastly, I think there's an "approachability" argument for using
>>> off-the-shelf. Thousands of developers out there are familiar with
>>> (e.g.) Jersey, compared to maybe 15 or 20 (in the world) familiar with
>>> Solr's custom annotations. Using a well-known technology like Jersey
>>> would make Solr all the easier to approach and contribute to for that
>>> pool of developers.
>>>
>>> > By the way, we have used Restlet in the past and that has been a regrettable decision.
>>
>>
>>>
>>> Ah, yeah, that's just the context I'm missing. Anyone have a pointer
>>> to related discussions, or remember what made this "regrettable"? All
>>> the theoretical benefits in the world don't matter much if we've
>>> already tried something like this in the past and decided against it.
>>
>>
>> It was not playing well with our security framework. The framework was not working well with Solr APIs
>>
>>>
>>> (Unrelated - Happy Thanksgiving all!)
>>>
>>> Best,
>>>
>>> Jason
>>>
>>> On Thu, Nov 25, 2021 at 7:32 AM Noble Paul <no...@gmail.com> wrote:
>>> >
>>> > Have you gone through an API written using the @EndPoint annotation?
>>> >
>>> > I strongly recommend that you do
>>> >
>>> > On Thu, Nov 25, 2021, 11:30 PM Eric Pugh <ep...@opensourceconnections.com> wrote:
>>> >>
>>> >> I have found our V2 API code to be very impenetrable to understand. Part of it is how the code is intertwined with support for V1, however it’s also because there aren’t really resources to go look at to understand how it should work! Maintaining the API should be very simple work, as they just exist as a translation. The home grown stuff may make sense if you are a super knowledgable Solr developer, but if you are just a new person, it’s a lot harder to contribute.
>>> >>
>>> >> I was interested in the Jersey stuff because I’ve seen lots of projects use it very successfully, and if I want to implement something, well, there are lots of blogs and resources out there!
>>> >>
>>> >> Can anyone recap briefly why we dropped RESTlet? And what lessons learned there might apply to adopting Jersey for API support? Looking at https://issues.apache.org/jira/browse/SOLR-14659, it was partly deprecated because we were not using it to support all the API, only the ManagedResource ones, and https://issues.apache.org/jira/browse/SOLR-14766 suggests that RESTlet maybe was no longer being updated? One reason why we spiked out Jersey was because of the broad support in the Java world! Looking at how much work we have to do in the V2 API world, we need a much broader pool of developers contributing to get there!
>>> >>
>>> >> Related, are there specific features/aspects of our annotations that enable things in Solr that couldn’t be done otherwise?
>>> >>
>>> >> On Nov 25, 2021, at 2:12 AM, Ishan Chattopadhyaya <ic...@gmail.com> wrote:
>>> >>
>>> >> Is there some problem with our annotations that we hope to solve using third party dependencies?
>>> >> By the way, we have used Restlet in the past and that has been a regrettable decision.
>>> >>
>>> >> On Thu, Nov 25, 2021 at 10:10 AM Jason Gerlowski <ge...@gmail.com> wrote:
>>> >>>
>>> >>> Solr's custom annotation framework ('@Endpoint', '@Command', etc.) has
>>> >>> cropped up a few times over the past week or two. [1] [2]. Having them
>>> >>> on top of mind, I've been wondering - is there a reason we use our own
>>> >>> annotations here instead of something off the shelf?
>>> >>>
>>> >>> What we have works well enough, but anything homegrown comes with more
>>> >>> maintenance burden than we'd have if we used something off the shelf.
>>> >>> There are plenty of well-used, active projects out there whose whole
>>> >>> purpose is facilitating the whole "annotation based API" thing
>>> >>> (Jersey, Restlet, RESTEasy, etc.) - why not use one of them?
>>> >>>
>>> >>> Does anyone know of any technical reasons why we can't go this route?
>>> >>> Or have any subjective reasons why we shouldn't? Or any context on
>>> >>> why we wrote our own Endpoint, Command, JsonProperty annotations
>>> >>> originally?
>>> >>>
>>> >>> FWIW, Eric Pugh and I spiked out a small POC recently, and got
>>> >>> Jersey+Jackson working for a few simple APIs without too much trouble.
>>> >>> [3] Obviously nothing production-ready there, and there's still a lot
>>> >>> of open questions (e.g. how would javabin be supported?), but we both
>>> >>> came away convinced that it seemed feasible, at least. Best of all,
>>> >>> APIs using our current homegrown annotation framework the switchover
>>> >>> seems blessedly straightforward, and it doesn't look like Jersey
>>> >>> (which we chose mostly arbitrarily) bloats our dist all that much.
>>> >>>
>>> >>> Curious if anyone has thoughts or context on how we ended up with the
>>> >>> annotation setup we use today!
>>> >>>
>>> >>> Best,
>>> >>>
>>> >>> Jason
>>> >>>
>>> >>> [1] https://issues.apache.org/jira/browse/SOLR-15182 (and children)
>>> >>> [2] http://mail-archives.apache.org/mod_mbox/solr-dev/202111.mbox/%3CCABEwPvENL41Pm6%2BOmjXb6Sx5N2XjUtnbWhgKOZSrnLjWBA8tcA%40mail.gmail.com%3E
>>> >>> [3] https://github.com/gerlowskija/solr/tree/jersey_jaxrs_jackson_solr_apis.
>>> >>>
>>> >>> ---------------------------------------------------------------------
>>> >>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>>> >>> For additional commands, e-mail: dev-help@solr.apache.org
>>> >>>
>>> >>
>>> >> _______________________
>>> >> Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | http://www.opensourceconnections.com | My Free/Busy
>>> >> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
>>> >> This e-mail and all contents, including attachments, is considered to be Company Confidential unless explicitly stated otherwise, regardless of whether attachments are marked as such.
>>> >>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>>> For additional commands, e-mail: dev-help@solr.apache.org
>>>
>>
>>
>> --
>> -----------------------------------------------------
>> Noble Paul
>
>
>
> --
> -----------------------------------------------------
> Noble Paul
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: JAX-RS APIs in Solr
Posted by Noble Paul <no...@gmail.com>.
The Annotations framework was written after playing with other frameworks.
There were many shortcomings which were hard to overcome.
The best example is a per collection API . How do you register an endpoint
for a collection/core ?
On Fri, Nov 26, 2021 at 3:42 PM Noble Paul <no...@gmail.com> wrote:
>
>
> On Fri, Nov 26, 2021 at 1:03 AM Jason Gerlowski <ge...@gmail.com>
> wrote:
>
>> > Is there some problem with our annotations that we hope to solve using
>> third party dependencies?
>>
>> I guess so yeah. Third-party deps are just fuller, more robust
>> solutions, whereas our annotations still need support added now and
>> then for even primitive data types like "long" (see SOLR-15619).
>>
> These are minor improvements compared to a full rewrite of the entire
> framework
>
>
>
>> Every JIRA spent doing basic stuff like that is time away from
>> improving Solr in some other way.
>>
>> So there are feature-gap/capabilities arguments for moving to a
>> third-party dep, sure. But, even if our annotations did everything
>> Jersey+Jackson do today, I think switching would still be worth it.
>> Every LOC in our code base brings along with it some maintenance cost:
>> it might have bugs, needs tested, takes time for new contributors to
>> "grok", etc. Using off-the-shelf here would nuke a whole bunch of
>> that. If off-the-shelf is available for some given functionality, we
>> should need a compelling reason to NOT use it.
>>
> Lastly, I think there's an "approachability" argument for using
>> off-the-shelf. Thousands of developers out there are familiar with
>> (e.g.) Jersey, compared to maybe 15 or 20 (in the world) familiar with
>> Solr's custom annotations. Using a well-known technology like Jersey
>> would make Solr all the easier to approach and contribute to for that
>> pool of developers.
>>
>> > By the way, we have used Restlet in the past and that has been a
>> regrettable decision.
>>
>
>
>> Ah, yeah, that's just the context I'm missing. Anyone have a pointer
>> to related discussions, or remember what made this "regrettable"? All
>> the theoretical benefits in the world don't matter much if we've
>> already tried something like this in the past and decided against it.
>>
>
> It was not playing well with our security framework. The framework was not
> working well with Solr APIs
>
>
>> (Unrelated - Happy Thanksgiving all!)
>>
>> Best,
>>
>> Jason
>>
>> On Thu, Nov 25, 2021 at 7:32 AM Noble Paul <no...@gmail.com> wrote:
>> >
>> > Have you gone through an API written using the @EndPoint annotation?
>> >
>> > I strongly recommend that you do
>> >
>> > On Thu, Nov 25, 2021, 11:30 PM Eric Pugh <
>> epugh@opensourceconnections.com> wrote:
>> >>
>> >> I have found our V2 API code to be very impenetrable to understand.
>> Part of it is how the code is intertwined with support for V1, however
>> it’s also because there aren’t really resources to go look at to understand
>> how it should work! Maintaining the API should be very simple work, as
>> they just exist as a translation. The home grown stuff may make sense if
>> you are a super knowledgable Solr developer, but if you are just a new
>> person, it’s a lot harder to contribute.
>> >>
>> >> I was interested in the Jersey stuff because I’ve seen lots of
>> projects use it very successfully, and if I want to implement something,
>> well, there are lots of blogs and resources out there!
>> >>
>> >> Can anyone recap briefly why we dropped RESTlet? And what lessons
>> learned there might apply to adopting Jersey for API support? Looking at
>> https://issues.apache.org/jira/browse/SOLR-14659, it was partly
>> deprecated because we were not using it to support all the API, only the
>> ManagedResource ones, and
>> https://issues.apache.org/jira/browse/SOLR-14766 suggests that RESTlet
>> maybe was no longer being updated? One reason why we spiked out Jersey
>> was because of the broad support in the Java world! Looking at how much
>> work we have to do in the V2 API world, we need a much broader pool of
>> developers contributing to get there!
>> >>
>> >> Related, are there specific features/aspects of our annotations that
>> enable things in Solr that couldn’t be done otherwise?
>> >>
>> >> On Nov 25, 2021, at 2:12 AM, Ishan Chattopadhyaya <
>> ichattopadhyaya@gmail.com> wrote:
>> >>
>> >> Is there some problem with our annotations that we hope to solve using
>> third party dependencies?
>> >> By the way, we have used Restlet in the past and that has been a
>> regrettable decision.
>> >>
>> >> On Thu, Nov 25, 2021 at 10:10 AM Jason Gerlowski <
>> gerlowskija@gmail.com> wrote:
>> >>>
>> >>> Solr's custom annotation framework ('@Endpoint', '@Command', etc.) has
>> >>> cropped up a few times over the past week or two. [1] [2]. Having them
>> >>> on top of mind, I've been wondering - is there a reason we use our own
>> >>> annotations here instead of something off the shelf?
>> >>>
>> >>> What we have works well enough, but anything homegrown comes with more
>> >>> maintenance burden than we'd have if we used something off the shelf.
>> >>> There are plenty of well-used, active projects out there whose whole
>> >>> purpose is facilitating the whole "annotation based API" thing
>> >>> (Jersey, Restlet, RESTEasy, etc.) - why not use one of them?
>> >>>
>> >>> Does anyone know of any technical reasons why we can't go this route?
>> >>> Or have any subjective reasons why we shouldn't? Or any context on
>> >>> why we wrote our own Endpoint, Command, JsonProperty annotations
>> >>> originally?
>> >>>
>> >>> FWIW, Eric Pugh and I spiked out a small POC recently, and got
>> >>> Jersey+Jackson working for a few simple APIs without too much trouble.
>> >>> [3] Obviously nothing production-ready there, and there's still a lot
>> >>> of open questions (e.g. how would javabin be supported?), but we both
>> >>> came away convinced that it seemed feasible, at least. Best of all,
>> >>> APIs using our current homegrown annotation framework the switchover
>> >>> seems blessedly straightforward, and it doesn't look like Jersey
>> >>> (which we chose mostly arbitrarily) bloats our dist all that much.
>> >>>
>> >>> Curious if anyone has thoughts or context on how we ended up with the
>> >>> annotation setup we use today!
>> >>>
>> >>> Best,
>> >>>
>> >>> Jason
>> >>>
>> >>> [1] https://issues.apache.org/jira/browse/SOLR-15182 (and children)
>> >>> [2]
>> http://mail-archives.apache.org/mod_mbox/solr-dev/202111.mbox/%3CCABEwPvENL41Pm6%2BOmjXb6Sx5N2XjUtnbWhgKOZSrnLjWBA8tcA%40mail.gmail.com%3E
>> >>> [3]
>> https://github.com/gerlowskija/solr/tree/jersey_jaxrs_jackson_solr_apis.
>> >>>
>> >>> ---------------------------------------------------------------------
>> >>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>> >>> For additional commands, e-mail: dev-help@solr.apache.org
>> >>>
>> >>
>> >> _______________________
>> >> Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467
>> | http://www.opensourceconnections.com | My Free/Busy
>> >> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
>> >> This e-mail and all contents, including attachments, is considered to
>> be Company Confidential unless explicitly stated otherwise, regardless of
>> whether attachments are marked as such.
>> >>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>> For additional commands, e-mail: dev-help@solr.apache.org
>>
>>
>
> --
> -----------------------------------------------------
> Noble Paul
>
--
-----------------------------------------------------
Noble Paul
Re: JAX-RS APIs in Solr
Posted by Noble Paul <no...@gmail.com>.
On Fri, Nov 26, 2021 at 1:03 AM Jason Gerlowski <ge...@gmail.com>
wrote:
> > Is there some problem with our annotations that we hope to solve using
> third party dependencies?
>
> I guess so yeah. Third-party deps are just fuller, more robust
> solutions, whereas our annotations still need support added now and
> then for even primitive data types like "long" (see SOLR-15619).
>
These are minor improvements compared to a full rewrite of the entire
framework
> Every JIRA spent doing basic stuff like that is time away from
> improving Solr in some other way.
>
> So there are feature-gap/capabilities arguments for moving to a
> third-party dep, sure. But, even if our annotations did everything
> Jersey+Jackson do today, I think switching would still be worth it.
> Every LOC in our code base brings along with it some maintenance cost:
> it might have bugs, needs tested, takes time for new contributors to
> "grok", etc. Using off-the-shelf here would nuke a whole bunch of
> that. If off-the-shelf is available for some given functionality, we
> should need a compelling reason to NOT use it.
>
Lastly, I think there's an "approachability" argument for using
> off-the-shelf. Thousands of developers out there are familiar with
> (e.g.) Jersey, compared to maybe 15 or 20 (in the world) familiar with
> Solr's custom annotations. Using a well-known technology like Jersey
> would make Solr all the easier to approach and contribute to for that
> pool of developers.
>
> > By the way, we have used Restlet in the past and that has been a
> regrettable decision.
>
> Ah, yeah, that's just the context I'm missing. Anyone have a pointer
> to related discussions, or remember what made this "regrettable"? All
> the theoretical benefits in the world don't matter much if we've
> already tried something like this in the past and decided against it.
>
It was not playing well with our security framework. The framework was not
working well with Solr APIs
> (Unrelated - Happy Thanksgiving all!)
>
> Best,
>
> Jason
>
> On Thu, Nov 25, 2021 at 7:32 AM Noble Paul <no...@gmail.com> wrote:
> >
> > Have you gone through an API written using the @EndPoint annotation?
> >
> > I strongly recommend that you do
> >
> > On Thu, Nov 25, 2021, 11:30 PM Eric Pugh <
> epugh@opensourceconnections.com> wrote:
> >>
> >> I have found our V2 API code to be very impenetrable to understand.
> Part of it is how the code is intertwined with support for V1, however
> it’s also because there aren’t really resources to go look at to understand
> how it should work! Maintaining the API should be very simple work, as
> they just exist as a translation. The home grown stuff may make sense if
> you are a super knowledgable Solr developer, but if you are just a new
> person, it’s a lot harder to contribute.
> >>
> >> I was interested in the Jersey stuff because I’ve seen lots of projects
> use it very successfully, and if I want to implement something, well, there
> are lots of blogs and resources out there!
> >>
> >> Can anyone recap briefly why we dropped RESTlet? And what lessons
> learned there might apply to adopting Jersey for API support? Looking at
> https://issues.apache.org/jira/browse/SOLR-14659, it was partly
> deprecated because we were not using it to support all the API, only the
> ManagedResource ones, and https://issues.apache.org/jira/browse/SOLR-14766
> suggests that RESTlet maybe was no longer being updated? One reason why
> we spiked out Jersey was because of the broad support in the Java world!
> Looking at how much work we have to do in the V2 API world, we need a much
> broader pool of developers contributing to get there!
> >>
> >> Related, are there specific features/aspects of our annotations that
> enable things in Solr that couldn’t be done otherwise?
> >>
> >> On Nov 25, 2021, at 2:12 AM, Ishan Chattopadhyaya <
> ichattopadhyaya@gmail.com> wrote:
> >>
> >> Is there some problem with our annotations that we hope to solve using
> third party dependencies?
> >> By the way, we have used Restlet in the past and that has been a
> regrettable decision.
> >>
> >> On Thu, Nov 25, 2021 at 10:10 AM Jason Gerlowski <ge...@gmail.com>
> wrote:
> >>>
> >>> Solr's custom annotation framework ('@Endpoint', '@Command', etc.) has
> >>> cropped up a few times over the past week or two. [1] [2]. Having them
> >>> on top of mind, I've been wondering - is there a reason we use our own
> >>> annotations here instead of something off the shelf?
> >>>
> >>> What we have works well enough, but anything homegrown comes with more
> >>> maintenance burden than we'd have if we used something off the shelf.
> >>> There are plenty of well-used, active projects out there whose whole
> >>> purpose is facilitating the whole "annotation based API" thing
> >>> (Jersey, Restlet, RESTEasy, etc.) - why not use one of them?
> >>>
> >>> Does anyone know of any technical reasons why we can't go this route?
> >>> Or have any subjective reasons why we shouldn't? Or any context on
> >>> why we wrote our own Endpoint, Command, JsonProperty annotations
> >>> originally?
> >>>
> >>> FWIW, Eric Pugh and I spiked out a small POC recently, and got
> >>> Jersey+Jackson working for a few simple APIs without too much trouble.
> >>> [3] Obviously nothing production-ready there, and there's still a lot
> >>> of open questions (e.g. how would javabin be supported?), but we both
> >>> came away convinced that it seemed feasible, at least. Best of all,
> >>> APIs using our current homegrown annotation framework the switchover
> >>> seems blessedly straightforward, and it doesn't look like Jersey
> >>> (which we chose mostly arbitrarily) bloats our dist all that much.
> >>>
> >>> Curious if anyone has thoughts or context on how we ended up with the
> >>> annotation setup we use today!
> >>>
> >>> Best,
> >>>
> >>> Jason
> >>>
> >>> [1] https://issues.apache.org/jira/browse/SOLR-15182 (and children)
> >>> [2]
> http://mail-archives.apache.org/mod_mbox/solr-dev/202111.mbox/%3CCABEwPvENL41Pm6%2BOmjXb6Sx5N2XjUtnbWhgKOZSrnLjWBA8tcA%40mail.gmail.com%3E
> >>> [3]
> https://github.com/gerlowskija/solr/tree/jersey_jaxrs_jackson_solr_apis.
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> >>> For additional commands, e-mail: dev-help@solr.apache.org
> >>>
> >>
> >> _______________________
> >> Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467
> | http://www.opensourceconnections.com | My Free/Busy
> >> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
> >> This e-mail and all contents, including attachments, is considered to
> be Company Confidential unless explicitly stated otherwise, regardless of
> whether attachments are marked as such.
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org
>
>
--
-----------------------------------------------------
Noble Paul
Re: JAX-RS APIs in Solr
Posted by Jason Gerlowski <ge...@gmail.com>.
> Is there some problem with our annotations that we hope to solve using third party dependencies?
I guess so yeah. Third-party deps are just fuller, more robust
solutions, whereas our annotations still need support added now and
then for even primitive data types like "long" (see SOLR-15619).
Every JIRA spent doing basic stuff like that is time away from
improving Solr in some other way.
So there are feature-gap/capabilities arguments for moving to a
third-party dep, sure. But, even if our annotations did everything
Jersey+Jackson do today, I think switching would still be worth it.
Every LOC in our code base brings along with it some maintenance cost:
it might have bugs, needs tested, takes time for new contributors to
"grok", etc. Using off-the-shelf here would nuke a whole bunch of
that. If off-the-shelf is available for some given functionality, we
should need a compelling reason to NOT use it.
Lastly, I think there's an "approachability" argument for using
off-the-shelf. Thousands of developers out there are familiar with
(e.g.) Jersey, compared to maybe 15 or 20 (in the world) familiar with
Solr's custom annotations. Using a well-known technology like Jersey
would make Solr all the easier to approach and contribute to for that
pool of developers.
> By the way, we have used Restlet in the past and that has been a regrettable decision.
Ah, yeah, that's just the context I'm missing. Anyone have a pointer
to related discussions, or remember what made this "regrettable"? All
the theoretical benefits in the world don't matter much if we've
already tried something like this in the past and decided against it.
(Unrelated - Happy Thanksgiving all!)
Best,
Jason
On Thu, Nov 25, 2021 at 7:32 AM Noble Paul <no...@gmail.com> wrote:
>
> Have you gone through an API written using the @EndPoint annotation?
>
> I strongly recommend that you do
>
> On Thu, Nov 25, 2021, 11:30 PM Eric Pugh <ep...@opensourceconnections.com> wrote:
>>
>> I have found our V2 API code to be very impenetrable to understand. Part of it is how the code is intertwined with support for V1, however it’s also because there aren’t really resources to go look at to understand how it should work! Maintaining the API should be very simple work, as they just exist as a translation. The home grown stuff may make sense if you are a super knowledgable Solr developer, but if you are just a new person, it’s a lot harder to contribute.
>>
>> I was interested in the Jersey stuff because I’ve seen lots of projects use it very successfully, and if I want to implement something, well, there are lots of blogs and resources out there!
>>
>> Can anyone recap briefly why we dropped RESTlet? And what lessons learned there might apply to adopting Jersey for API support? Looking at https://issues.apache.org/jira/browse/SOLR-14659, it was partly deprecated because we were not using it to support all the API, only the ManagedResource ones, and https://issues.apache.org/jira/browse/SOLR-14766 suggests that RESTlet maybe was no longer being updated? One reason why we spiked out Jersey was because of the broad support in the Java world! Looking at how much work we have to do in the V2 API world, we need a much broader pool of developers contributing to get there!
>>
>> Related, are there specific features/aspects of our annotations that enable things in Solr that couldn’t be done otherwise?
>>
>> On Nov 25, 2021, at 2:12 AM, Ishan Chattopadhyaya <ic...@gmail.com> wrote:
>>
>> Is there some problem with our annotations that we hope to solve using third party dependencies?
>> By the way, we have used Restlet in the past and that has been a regrettable decision.
>>
>> On Thu, Nov 25, 2021 at 10:10 AM Jason Gerlowski <ge...@gmail.com> wrote:
>>>
>>> Solr's custom annotation framework ('@Endpoint', '@Command', etc.) has
>>> cropped up a few times over the past week or two. [1] [2]. Having them
>>> on top of mind, I've been wondering - is there a reason we use our own
>>> annotations here instead of something off the shelf?
>>>
>>> What we have works well enough, but anything homegrown comes with more
>>> maintenance burden than we'd have if we used something off the shelf.
>>> There are plenty of well-used, active projects out there whose whole
>>> purpose is facilitating the whole "annotation based API" thing
>>> (Jersey, Restlet, RESTEasy, etc.) - why not use one of them?
>>>
>>> Does anyone know of any technical reasons why we can't go this route?
>>> Or have any subjective reasons why we shouldn't? Or any context on
>>> why we wrote our own Endpoint, Command, JsonProperty annotations
>>> originally?
>>>
>>> FWIW, Eric Pugh and I spiked out a small POC recently, and got
>>> Jersey+Jackson working for a few simple APIs without too much trouble.
>>> [3] Obviously nothing production-ready there, and there's still a lot
>>> of open questions (e.g. how would javabin be supported?), but we both
>>> came away convinced that it seemed feasible, at least. Best of all,
>>> APIs using our current homegrown annotation framework the switchover
>>> seems blessedly straightforward, and it doesn't look like Jersey
>>> (which we chose mostly arbitrarily) bloats our dist all that much.
>>>
>>> Curious if anyone has thoughts or context on how we ended up with the
>>> annotation setup we use today!
>>>
>>> Best,
>>>
>>> Jason
>>>
>>> [1] https://issues.apache.org/jira/browse/SOLR-15182 (and children)
>>> [2] http://mail-archives.apache.org/mod_mbox/solr-dev/202111.mbox/%3CCABEwPvENL41Pm6%2BOmjXb6Sx5N2XjUtnbWhgKOZSrnLjWBA8tcA%40mail.gmail.com%3E
>>> [3] https://github.com/gerlowskija/solr/tree/jersey_jaxrs_jackson_solr_apis.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>>> For additional commands, e-mail: dev-help@solr.apache.org
>>>
>>
>> _______________________
>> Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | http://www.opensourceconnections.com | My Free/Busy
>> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
>> This e-mail and all contents, including attachments, is considered to be Company Confidential unless explicitly stated otherwise, regardless of whether attachments are marked as such.
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: JAX-RS APIs in Solr
Posted by Noble Paul <no...@gmail.com>.
Have you gone through an API written using the @EndPoint annotation?
I strongly recommend that you do
On Thu, Nov 25, 2021, 11:30 PM Eric Pugh <ep...@opensourceconnections.com>
wrote:
> I have found our V2 API code to be very impenetrable to understand. Part
> of it is how the code is intertwined with support for V1, however it’s also
> because there aren’t really resources to go look at to understand how it
> should work! Maintaining the API should be very simple work, as they just
> exist as a translation. The home grown stuff may make sense if you are a
> super knowledgable Solr developer, but if you are just a new person, it’s a
> lot harder to contribute.
>
> I was interested in the Jersey stuff because I’ve seen lots of projects
> use it very successfully, and if I want to implement something, well, there
> are lots of blogs and resources out there!
>
> Can anyone recap briefly why we dropped RESTlet? And what lessons
> learned there might apply to adopting Jersey for API support? Looking at
> https://issues.apache.org/jira/browse/SOLR-14659, it was partly
> deprecated because we were not using it to support all the API, only the
> ManagedResource ones, and https://issues.apache.org/jira/browse/SOLR-14766
> suggests that RESTlet maybe was no longer being updated? One reason why
> we spiked out Jersey was because of the broad support in the Java world!
> Looking at how much work we have to do in the V2 API world, we need a much
> broader pool of developers contributing to get there!
>
> Related, are there specific features/aspects of our annotations that
> enable things in Solr that couldn’t be done otherwise?
>
> On Nov 25, 2021, at 2:12 AM, Ishan Chattopadhyaya <
> ichattopadhyaya@gmail.com> wrote:
>
> Is there some problem with our annotations that we hope to solve using
> third party dependencies?
> By the way, we have used Restlet in the past and that has been a
> regrettable decision.
>
> On Thu, Nov 25, 2021 at 10:10 AM Jason Gerlowski <ge...@gmail.com>
> wrote:
>
>> Solr's custom annotation framework ('@Endpoint', '@Command', etc.) has
>> cropped up a few times over the past week or two. [1] [2]. Having them
>> on top of mind, I've been wondering - is there a reason we use our own
>> annotations here instead of something off the shelf?
>>
>> What we have works well enough, but anything homegrown comes with more
>> maintenance burden than we'd have if we used something off the shelf.
>> There are plenty of well-used, active projects out there whose whole
>> purpose is facilitating the whole "annotation based API" thing
>> (Jersey, Restlet, RESTEasy, etc.) - why not use one of them?
>>
>> Does anyone know of any technical reasons why we can't go this route?
>> Or have any subjective reasons why we shouldn't? Or any context on
>> why we wrote our own Endpoint, Command, JsonProperty annotations
>> originally?
>>
>> FWIW, Eric Pugh and I spiked out a small POC recently, and got
>> Jersey+Jackson working for a few simple APIs without too much trouble.
>> [3] Obviously nothing production-ready there, and there's still a lot
>> of open questions (e.g. how would javabin be supported?), but we both
>> came away convinced that it seemed feasible, at least. Best of all,
>> APIs using our current homegrown annotation framework the switchover
>> seems blessedly straightforward, and it doesn't look like Jersey
>> (which we chose mostly arbitrarily) bloats our dist all that much.
>>
>> Curious if anyone has thoughts or context on how we ended up with the
>> annotation setup we use today!
>>
>> Best,
>>
>> Jason
>>
>> [1] https://issues.apache.org/jira/browse/SOLR-15182 (and children)
>> [2]
>> http://mail-archives.apache.org/mod_mbox/solr-dev/202111.mbox/%3CCABEwPvENL41Pm6%2BOmjXb6Sx5N2XjUtnbWhgKOZSrnLjWBA8tcA%40mail.gmail.com%3E
>> [3]
>> https://github.com/gerlowskija/solr/tree/jersey_jaxrs_jackson_solr_apis.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>> For additional commands, e-mail: dev-help@solr.apache.org
>>
>>
> _______________________
> *Eric Pugh **| *Founder & CEO | OpenSource Connections, LLC | 434.466.1467
> | http://www.opensourceconnections.com | My Free/Busy
> <http://tinyurl.com/eric-cal>
> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
> <https://www.packtpub.com/big-data-and-business-intelligence/apache-solr-enterprise-search-server-third-edition-raw>
> This e-mail and all contents, including attachments, is considered to be
> Company Confidential unless explicitly stated otherwise, regardless
> of whether attachments are marked as such.
>
>
Re: JAX-RS APIs in Solr
Posted by Eric Pugh <ep...@opensourceconnections.com>.
I have found our V2 API code to be very impenetrable to understand. Part of it is how the code is intertwined with support for V1, however it’s also because there aren’t really resources to go look at to understand how it should work! Maintaining the API should be very simple work, as they just exist as a translation. The home grown stuff may make sense if you are a super knowledgable Solr developer, but if you are just a new person, it’s a lot harder to contribute.
I was interested in the Jersey stuff because I’ve seen lots of projects use it very successfully, and if I want to implement something, well, there are lots of blogs and resources out there!
Can anyone recap briefly why we dropped RESTlet? And what lessons learned there might apply to adopting Jersey for API support? Looking at https://issues.apache.org/jira/browse/SOLR-14659, it was partly deprecated because we were not using it to support all the API, only the ManagedResource ones, and https://issues.apache.org/jira/browse/SOLR-14766 suggests that RESTlet maybe was no longer being updated? One reason why we spiked out Jersey was because of the broad support in the Java world! Looking at how much work we have to do in the V2 API world, we need a much broader pool of developers contributing to get there!
Related, are there specific features/aspects of our annotations that enable things in Solr that couldn’t be done otherwise?
> On Nov 25, 2021, at 2:12 AM, Ishan Chattopadhyaya <ic...@gmail.com> wrote:
>
> Is there some problem with our annotations that we hope to solve using third party dependencies?
> By the way, we have used Restlet in the past and that has been a regrettable decision.
>
> On Thu, Nov 25, 2021 at 10:10 AM Jason Gerlowski <gerlowskija@gmail.com <ma...@gmail.com>> wrote:
> Solr's custom annotation framework ('@Endpoint', '@Command', etc.) has
> cropped up a few times over the past week or two. [1] [2]. Having them
> on top of mind, I've been wondering - is there a reason we use our own
> annotations here instead of something off the shelf?
>
> What we have works well enough, but anything homegrown comes with more
> maintenance burden than we'd have if we used something off the shelf.
> There are plenty of well-used, active projects out there whose whole
> purpose is facilitating the whole "annotation based API" thing
> (Jersey, Restlet, RESTEasy, etc.) - why not use one of them?
>
> Does anyone know of any technical reasons why we can't go this route?
> Or have any subjective reasons why we shouldn't? Or any context on
> why we wrote our own Endpoint, Command, JsonProperty annotations
> originally?
>
> FWIW, Eric Pugh and I spiked out a small POC recently, and got
> Jersey+Jackson working for a few simple APIs without too much trouble.
> [3] Obviously nothing production-ready there, and there's still a lot
> of open questions (e.g. how would javabin be supported?), but we both
> came away convinced that it seemed feasible, at least. Best of all,
> APIs using our current homegrown annotation framework the switchover
> seems blessedly straightforward, and it doesn't look like Jersey
> (which we chose mostly arbitrarily) bloats our dist all that much.
>
> Curious if anyone has thoughts or context on how we ended up with the
> annotation setup we use today!
>
> Best,
>
> Jason
>
> [1] https://issues.apache.org/jira/browse/SOLR-15182 <https://issues.apache.org/jira/browse/SOLR-15182> (and children)
> [2] http://mail-archives.apache.org/mod_mbox/solr-dev/202111.mbox/%3CCABEwPvENL41Pm6%2BOmjXb6Sx5N2XjUtnbWhgKOZSrnLjWBA8tcA%40mail.gmail.com%3E <http://mail-archives.apache.org/mod_mbox/solr-dev/202111.mbox/%3CCABEwPvENL41Pm6%2BOmjXb6Sx5N2XjUtnbWhgKOZSrnLjWBA8tcA%40mail.gmail.com%3E>
> [3] https://github.com/gerlowskija/solr/tree/jersey_jaxrs_jackson_solr_apis <https://github.com/gerlowskija/solr/tree/jersey_jaxrs_jackson_solr_apis>.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org <ma...@solr.apache.org>
> For additional commands, e-mail: dev-help@solr.apache.org <ma...@solr.apache.org>
>
_______________________
Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | http://www.opensourceconnections.com <http://www.opensourceconnections.com/> | My Free/Busy <http://tinyurl.com/eric-cal>
Co-Author: Apache Solr Enterprise Search Server, 3rd Ed <https://www.packtpub.com/big-data-and-business-intelligence/apache-solr-enterprise-search-server-third-edition-raw>
This e-mail and all contents, including attachments, is considered to be Company Confidential unless explicitly stated otherwise, regardless of whether attachments are marked as such.
Re: JAX-RS APIs in Solr
Posted by Ishan Chattopadhyaya <ic...@gmail.com>.
Is there some problem with our annotations that we hope to solve using
third party dependencies?
By the way, we have used Restlet in the past and that has been a
regrettable decision.
On Thu, Nov 25, 2021 at 10:10 AM Jason Gerlowski <ge...@gmail.com>
wrote:
> Solr's custom annotation framework ('@Endpoint', '@Command', etc.) has
> cropped up a few times over the past week or two. [1] [2]. Having them
> on top of mind, I've been wondering - is there a reason we use our own
> annotations here instead of something off the shelf?
>
> What we have works well enough, but anything homegrown comes with more
> maintenance burden than we'd have if we used something off the shelf.
> There are plenty of well-used, active projects out there whose whole
> purpose is facilitating the whole "annotation based API" thing
> (Jersey, Restlet, RESTEasy, etc.) - why not use one of them?
>
> Does anyone know of any technical reasons why we can't go this route?
> Or have any subjective reasons why we shouldn't? Or any context on
> why we wrote our own Endpoint, Command, JsonProperty annotations
> originally?
>
> FWIW, Eric Pugh and I spiked out a small POC recently, and got
> Jersey+Jackson working for a few simple APIs without too much trouble.
> [3] Obviously nothing production-ready there, and there's still a lot
> of open questions (e.g. how would javabin be supported?), but we both
> came away convinced that it seemed feasible, at least. Best of all,
> APIs using our current homegrown annotation framework the switchover
> seems blessedly straightforward, and it doesn't look like Jersey
> (which we chose mostly arbitrarily) bloats our dist all that much.
>
> Curious if anyone has thoughts or context on how we ended up with the
> annotation setup we use today!
>
> Best,
>
> Jason
>
> [1] https://issues.apache.org/jira/browse/SOLR-15182 (and children)
> [2]
> http://mail-archives.apache.org/mod_mbox/solr-dev/202111.mbox/%3CCABEwPvENL41Pm6%2BOmjXb6Sx5N2XjUtnbWhgKOZSrnLjWBA8tcA%40mail.gmail.com%3E
> [3]
> https://github.com/gerlowskija/solr/tree/jersey_jaxrs_jackson_solr_apis.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org
>
>