You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Jan Høydahl <ja...@cominvent.com> on 2020/02/17 11:44:36 UTC
SolrJ has 29 compile-time dependencies!
Hi
According to https://mvnrepository.com/artifact/org.apache.solr/solr-solrj/8.4.1 SolrJ now has 29 compile-time dependencies. Those are the ones explicitly mentioned in ivy.xml and I believe that the number would be even higher if we used transitive dependencies.
That means that if you want to include SolrJ in a small app for just searching Solr, you get a ton of dependencies in your project that you may not need and that increase the chance of collision with other libs in your all.
So I want to raise the question whether it is time to take some action here.
Otions may include:
Get rid of unneeded deps
Explicitly exclude deps from gradle build that we know we do not need
Modularize SolrJ into a solrj-core and solrj-xxx, where solrj-core would be the minimum anyone would need to do the basics
Look into shading select libs that often cause collisions
Let the discussion begin :)
Jan
Re: SolrJ has 29 compile-time dependencies!
Posted by Dawid Weiss <da...@gmail.com>.
Yes, this is insane. The problem here is that solrj is treated as a
bin of dependencies further downstream in consuming projects. It
should not be this way, really. It should be a minimal set of
dependencies.
> Get rid of unneeded deps
> Explicitly exclude deps from gradle build that we know we do not need
I don't think it would require exclusions if the dependencies are
declared the right way. Right now gradle reflects what ant does (I
don't think it's correct, it's just consistent with ant; see
https://issues.apache.org/jira/browse/LUCENE-9140):
> Look into shading select libs that often cause collisions
This is possible but *hard* to manage in practice. I know because we
do such things in a few internal projects. It's really a nightmare to
manage builds and make external tools (like IDEs) work consistently
with such shaded dependencies. I think it'd be better to have a sane
set of dependencies and a separate project which would repackage
(shade) what's needed, if it's really a problem.
D.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: SolrJ has 29 compile-time dependencies!
Posted by Tim Casey <tc...@gmail.com>.
If you have a 'small' application and you want search, you can embed a
lucent index. This is a lower impact dependency graph.
Solr is a service. I would suspect the only dependencies *required* are
client dependencies. And even those might boil down to a shell equivalent
curl + json parser.
As a service, you might have a lot more dependencies. Caching,
journaling, flushing,'languaging' in addition to all of the
communication/server stuff happening. you might have spidering,
re-reindexing and other transitive dependencies in the overall footprint.
Then, you could add the underlying cloud support libraries. A service is a
complicated thing.
On Mon, Feb 17, 2020 at 2:30 PM Robert Muir <rc...@gmail.com> wrote:
> Yes why would this library have any dependencies at all? In trunk java 11
> is a minimum, and JDK11+ has HTTP client capable of HTTP/2, https, etc. So
> seeing both netty and jetty client libraries makes no sense at all.
>
>
> https://docs.oracle.com/en/java/javase/11/docs/api/java.net.http/java/net/http/package-summary.html
>
>
> On Mon, Feb 17, 2020 at 6:44 AM Jan Høydahl <ja...@cominvent.com> wrote:
>
>> Hi
>>
>> According to
>> https://mvnrepository.com/artifact/org.apache.solr/solr-solrj/8.4.1
>> SolrJ now has 29 compile-time dependencies. Those are the ones explicitly
>> mentioned in ivy.xml and I believe that the number would be even higher if
>> we used transitive dependencies.
>> That means that if you want to include SolrJ in a small app for just
>> searching Solr, you get a ton of dependencies in your project that you may
>> not need and that increase the chance of collision with other libs in your
>> all.
>>
>> So I want to raise the question whether it is time to take some action
>> here.
>>
>> Otions may include:
>>
>> - Get rid of unneeded deps
>> - Explicitly exclude deps from gradle build that we know we do not
>> need
>> - Modularize SolrJ into a solrj-core and solrj-xxx, where solrj-core
>> would be the minimum anyone would need to do the basics
>> - Look into shading select libs that often cause collisions
>>
>>
>> Let the discussion begin :)
>>
>> Jan
>>
>
Re: SolrJ has 29 compile-time dependencies!
Posted by Noble Paul <no...@gmail.com>.
I agree with the general consensus here. There should be a minimal
SolrJ client that has just one jar and zero dependencies
On Thu, Feb 20, 2020 at 2:54 AM Jan Høydahl <ja...@cominvent.com> wrote:
>
> I added a comment to https://issues.apache.org/jira/browse/SOLR-599 "Lightweight SolrJ client" From June 2008 (!)
> Let’s continue discussion there?
>
> Jan
>
> 19. feb. 2020 kl. 16:19 skrev David Smiley <da...@gmail.com>:
>
> I definitely care about this issue. I like the idea of splitting off a solr-zk module. Few people know that CloudSolrClient can do a pure HTTP mode to get cluster state indirectly from ZK via Solr, thus no necessity of talking with ZK and what that entails architecturally (security concerns).
>
> Whenever someone proposes changes to SolrJ impacting dependencies, we need to be super clear about that; needs it’s own issue or dev list message announcement. Code reviews will help. I can’t stand dependency creep in SolrJ!
>
> (FYI I have almost no internet connectivity this week. )
>
> On Mon, Feb 17, 2020 at 7:26 PM Jan Høydahl <ja...@cominvent.com> wrote:
>>
>> The netty deps are there solely because they are needed by zookeeper client when talking to zookeeper over the new https transport with netty.
>> So a quick win would be to make a solrj-zk.jar which depends on zookeeper-client.jar along with netty and its other dependencies, and thus keep solrj-core clean of these.
>>
>> It would be great to be able to choose what HTTP lib to use in SolrJ. Obviously you want Jetty-HTTP2-client to do fancy things such as bring-your-own-client with interceptors and what not, but for the simple case we should be just fine with Java11 client in solrj-core.
>>
>> Looking at the solrj module there are 756 Java classes in there. So we use SolrJ as some common module and perhaps it is a bit too easy to just throw new stuff in there,, not really considering this is all going to end up in every user’s classpath + dependencies?
>>
>> Jan
>>
>> 17. feb. 2020 kl. 23:24 skrev Robert Muir <rc...@gmail.com>:
>>
>> Yes why would this library have any dependencies at all? In trunk java 11 is a minimum, and JDK11+ has HTTP client capable of HTTP/2, https, etc. So seeing both netty and jetty client libraries makes no sense at all.
>>
>> https://docs.oracle.com/en/java/javase/11/docs/api/java.net.http/java/net/http/package-summary.html
>>
>>
>> On Mon, Feb 17, 2020 at 6:44 AM Jan Høydahl <ja...@cominvent.com> wrote:
>>>
>>> Hi
>>>
>>> According to https://mvnrepository.com/artifact/org.apache.solr/solr-solrj/8.4.1 SolrJ now has 29 compile-time dependencies. Those are the ones explicitly mentioned in ivy.xml and I believe that the number would be even higher if we used transitive dependencies.
>>> That means that if you want to include SolrJ in a small app for just searching Solr, you get a ton of dependencies in your project that you may not need and that increase the chance of collision with other libs in your all.
>>>
>>> So I want to raise the question whether it is time to take some action here.
>>>
>>> Otions may include:
>>>
>>> Get rid of unneeded deps
>>> Explicitly exclude deps from gradle build that we know we do not need
>>> Modularize SolrJ into a solrj-core and solrj-xxx, where solrj-core would be the minimum anyone would need to do the basics
>>> Look into shading select libs that often cause collisions
>>>
>>>
>>> Let the discussion begin :)
>>>
>>> Jan
>>
>>
> --
> Sent from Gmail Mobile
>
>
--
-----------------------------------------------------
Noble Paul
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: SolrJ has 29 compile-time dependencies!
Posted by Jan Høydahl <ja...@cominvent.com>.
I added a comment to https://issues.apache.org/jira/browse/SOLR-599 "Lightweight SolrJ client" From June 2008 (!)
Let’s continue discussion there?
Jan
> 19. feb. 2020 kl. 16:19 skrev David Smiley <da...@gmail.com>:
>
> I definitely care about this issue. I like the idea of splitting off a solr-zk module. Few people know that CloudSolrClient can do a pure HTTP mode to get cluster state indirectly from ZK via Solr, thus no necessity of talking with ZK and what that entails architecturally (security concerns).
>
> Whenever someone proposes changes to SolrJ impacting dependencies, we need to be super clear about that; needs it’s own issue or dev list message announcement. Code reviews will help. I can’t stand dependency creep in SolrJ!
>
> (FYI I have almost no internet connectivity this week. )
>
> On Mon, Feb 17, 2020 at 7:26 PM Jan Høydahl <jan.asf@cominvent.com <ma...@cominvent.com>> wrote:
> The netty deps are there solely because they are needed by zookeeper client when talking to zookeeper over the new https transport with netty.
> So a quick win would be to make a solrj-zk.jar which depends on zookeeper-client.jar along with netty and its other dependencies, and thus keep solrj-core clean of these.
>
> It would be great to be able to choose what HTTP lib to use in SolrJ. Obviously you want Jetty-HTTP2-client to do fancy things such as bring-your-own-client with interceptors and what not, but for the simple case we should be just fine with Java11 client in solrj-core.
>
> Looking at the solrj module there are 756 Java classes in there. So we use SolrJ as some common module and perhaps it is a bit too easy to just throw new stuff in there,, not really considering this is all going to end up in every user’s classpath + dependencies?
>
> Jan
>
>> 17. feb. 2020 kl. 23:24 skrev Robert Muir <rcmuir@gmail.com <ma...@gmail.com>>:
>>
>> Yes why would this library have any dependencies at all? In trunk java 11 is a minimum, and JDK11+ has HTTP client capable of HTTP/2, https, etc. So seeing both netty and jetty client libraries makes no sense at all.
>>
>> https://docs.oracle.com/en/java/javase/11/docs/api/java.net.http/java/net/http/package-summary.html <https://docs.oracle.com/en/java/javase/11/docs/api/java.net.http/java/net/http/package-summary.html>
>>
>>
>> On Mon, Feb 17, 2020 at 6:44 AM Jan Høydahl <jan.asf@cominvent.com <ma...@cominvent.com>> wrote:
>> Hi
>>
>> According to https://mvnrepository.com/artifact/org.apache.solr/solr-solrj/8.4.1 <https://mvnrepository.com/artifact/org.apache.solr/solr-solrj/8.4.1> SolrJ now has 29 compile-time dependencies. Those are the ones explicitly mentioned in ivy.xml and I believe that the number would be even higher if we used transitive dependencies.
>> That means that if you want to include SolrJ in a small app for just searching Solr, you get a ton of dependencies in your project that you may not need and that increase the chance of collision with other libs in your all.
>>
>> So I want to raise the question whether it is time to take some action here.
>>
>> Otions may include:
>> Get rid of unneeded deps
>> Explicitly exclude deps from gradle build that we know we do not need
>> Modularize SolrJ into a solrj-core and solrj-xxx, where solrj-core would be the minimum anyone would need to do the basics
>> Look into shading select libs that often cause collisions
>>
>> Let the discussion begin :)
>>
>> Jan
>
> --
> Sent from Gmail Mobile
Re: SolrJ has 29 compile-time dependencies!
Posted by David Smiley <da...@gmail.com>.
I definitely care about this issue. I like the idea of splitting off a
solr-zk module. Few people know that CloudSolrClient can do a pure HTTP
mode to get cluster state indirectly from ZK via Solr, thus no necessity of
talking with ZK and what that entails architecturally (security concerns).
Whenever someone proposes changes to SolrJ impacting dependencies, we need
to be super clear about that; needs it’s own issue or dev list message
announcement. Code reviews will help. I can’t stand dependency creep in
SolrJ!
(FYI I have almost no internet connectivity this week. )
On Mon, Feb 17, 2020 at 7:26 PM Jan Høydahl <ja...@cominvent.com> wrote:
> The netty deps are there solely because they are needed by zookeeper
> client when talking to zookeeper over the new https transport with netty.
> So a quick win would be to make a solrj-zk.jar which depends on
> zookeeper-client.jar along with netty and its other dependencies, and thus
> keep solrj-core clean of these.
>
> It would be great to be able to choose what HTTP lib to use in SolrJ.
> Obviously you want Jetty-HTTP2-client to do fancy things such as
> bring-your-own-client with interceptors and what not, but for the simple
> case we should be just fine with Java11 client in solrj-core.
>
> Looking at the solrj module there are 756 Java classes in there. So we use
> SolrJ as some common module and perhaps it is a bit too easy to just throw
> new stuff in there,, not really considering this is all going to end up in
> every user’s classpath + dependencies?
>
> Jan
>
> 17. feb. 2020 kl. 23:24 skrev Robert Muir <rc...@gmail.com>:
>
> Yes why would this library have any dependencies at all? In trunk java 11
> is a minimum, and JDK11+ has HTTP client capable of HTTP/2, https, etc. So
> seeing both netty and jetty client libraries makes no sense at all.
>
>
> https://docs.oracle.com/en/java/javase/11/docs/api/java.net.http/java/net/http/package-summary.html
>
>
> On Mon, Feb 17, 2020 at 6:44 AM Jan Høydahl <ja...@cominvent.com> wrote:
>
>> Hi
>>
>> According to
>> https://mvnrepository.com/artifact/org.apache.solr/solr-solrj/8.4.1
>> SolrJ now has 29 compile-time dependencies. Those are the ones explicitly
>> mentioned in ivy.xml and I believe that the number would be even higher if
>> we used transitive dependencies.
>> That means that if you want to include SolrJ in a small app for just
>> searching Solr, you get a ton of dependencies in your project that you may
>> not need and that increase the chance of collision with other libs in your
>> all.
>>
>> So I want to raise the question whether it is time to take some action
>> here.
>>
>> Otions may include:
>>
>> - Get rid of unneeded deps
>> - Explicitly exclude deps from gradle build that we know we do not
>> need
>> - Modularize SolrJ into a solrj-core and solrj-xxx, where solrj-core
>> would be the minimum anyone would need to do the basics
>> - Look into shading select libs that often cause collisions
>>
>>
>> Let the discussion begin :)
>>
>> Jan
>>
>
> --
Sent from Gmail Mobile
Re: SolrJ has 29 compile-time dependencies!
Posted by Jan Høydahl <ja...@cominvent.com>.
The netty deps are there solely because they are needed by zookeeper client when talking to zookeeper over the new https transport with netty.
So a quick win would be to make a solrj-zk.jar which depends on zookeeper-client.jar along with netty and its other dependencies, and thus keep solrj-core clean of these.
It would be great to be able to choose what HTTP lib to use in SolrJ. Obviously you want Jetty-HTTP2-client to do fancy things such as bring-your-own-client with interceptors and what not, but for the simple case we should be just fine with Java11 client in solrj-core.
Looking at the solrj module there are 756 Java classes in there. So we use SolrJ as some common module and perhaps it is a bit too easy to just throw new stuff in there,, not really considering this is all going to end up in every user’s classpath + dependencies?
Jan
> 17. feb. 2020 kl. 23:24 skrev Robert Muir <rc...@gmail.com>:
>
> Yes why would this library have any dependencies at all? In trunk java 11 is a minimum, and JDK11+ has HTTP client capable of HTTP/2, https, etc. So seeing both netty and jetty client libraries makes no sense at all.
>
> https://docs.oracle.com/en/java/javase/11/docs/api/java.net.http/java/net/http/package-summary.html <https://docs.oracle.com/en/java/javase/11/docs/api/java.net.http/java/net/http/package-summary.html>
>
>
> On Mon, Feb 17, 2020 at 6:44 AM Jan Høydahl <jan.asf@cominvent.com <ma...@cominvent.com>> wrote:
> Hi
>
> According to https://mvnrepository.com/artifact/org.apache.solr/solr-solrj/8.4.1 <https://mvnrepository.com/artifact/org.apache.solr/solr-solrj/8.4.1> SolrJ now has 29 compile-time dependencies. Those are the ones explicitly mentioned in ivy.xml and I believe that the number would be even higher if we used transitive dependencies.
> That means that if you want to include SolrJ in a small app for just searching Solr, you get a ton of dependencies in your project that you may not need and that increase the chance of collision with other libs in your all.
>
> So I want to raise the question whether it is time to take some action here.
>
> Otions may include:
> Get rid of unneeded deps
> Explicitly exclude deps from gradle build that we know we do not need
> Modularize SolrJ into a solrj-core and solrj-xxx, where solrj-core would be the minimum anyone would need to do the basics
> Look into shading select libs that often cause collisions
>
> Let the discussion begin :)
>
> Jan
Re: SolrJ has 29 compile-time dependencies!
Posted by Robert Muir <rc...@gmail.com>.
Yes why would this library have any dependencies at all? In trunk java 11
is a minimum, and JDK11+ has HTTP client capable of HTTP/2, https, etc. So
seeing both netty and jetty client libraries makes no sense at all.
https://docs.oracle.com/en/java/javase/11/docs/api/java.net.http/java/net/http/package-summary.html
On Mon, Feb 17, 2020 at 6:44 AM Jan Høydahl <ja...@cominvent.com> wrote:
> Hi
>
> According to
> https://mvnrepository.com/artifact/org.apache.solr/solr-solrj/8.4.1 SolrJ
> now has 29 compile-time dependencies. Those are the ones explicitly
> mentioned in ivy.xml and I believe that the number would be even higher if
> we used transitive dependencies.
> That means that if you want to include SolrJ in a small app for just
> searching Solr, you get a ton of dependencies in your project that you may
> not need and that increase the chance of collision with other libs in your
> all.
>
> So I want to raise the question whether it is time to take some action
> here.
>
> Otions may include:
>
> - Get rid of unneeded deps
> - Explicitly exclude deps from gradle build that we know we do not need
> - Modularize SolrJ into a solrj-core and solrj-xxx, where solrj-core
> would be the minimum anyone would need to do the basics
> - Look into shading select libs that often cause collisions
>
>
> Let the discussion begin :)
>
> Jan
>