You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@pekko.apache.org by "pjfanning (via GitHub)" <gi...@apache.org> on 2024/02/13 13:49:52 UTC
[PR] [DRAFT] support Solr9 [incubator-pekko-connectors]
pjfanning opened a new pull request, #481:
URL: https://github.com/apache/incubator-pekko-connectors/pull/481
* WIP - not ready for review
* relates to #480
* temporarily drop support for getIdField (removed from Solr 9 with no replacement)
* will try to add back a MethodHandle to call getIdField if an older Solr client is used
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] slightly better support for Solr9 - but not there yet [incubator-pekko-connectors]
Posted by "pjfanning (via GitHub)" <gi...@apache.org>.
pjfanning commented on PR #481:
URL: https://github.com/apache/incubator-pekko-connectors/pull/481#issuecomment-1942224430
Sticking with solr client v8 because solr v9 needs Java 11.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] support Solr9 [incubator-pekko-connectors]
Posted by "pjfanning (via GitHub)" <gi...@apache.org>.
pjfanning commented on code in PR #481:
URL: https://github.com/apache/incubator-pekko-connectors/pull/481#discussion_r1488136072
##########
project/Dependencies.scala:
##########
Review Comment:
thanks @epugh -- looks like solrj 9.5 removed one the the new methods that I started to use
csc.getZkStateReader
##########
project/Dependencies.scala:
##########
Review Comment:
thanks @epugh -- looks like solrj 9.5 removed one of the new methods that I started to use
csc.getZkStateReader
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] slightly better support for Solr9 - but not there yet [incubator-pekko-connectors]
Posted by "pjfanning (via GitHub)" <gi...@apache.org>.
pjfanning commented on PR #481:
URL: https://github.com/apache/incubator-pekko-connectors/pull/481#issuecomment-1942231254
> > Sticking with solr client v8 because solr v9 needs Java 11.
>
> what version of Java does Pekko need? Solr 10 will probably be Java 17 minimum version, will that be an issue for Pekko?
We still support Java 8. I might create my own Solr 9 based Pekko connector in a fork.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] slightly better support for Solr9 - but not there yet [incubator-pekko-connectors]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on PR #481:
URL: https://github.com/apache/incubator-pekko-connectors/pull/481#issuecomment-1942227686
> Sticking with solr client v8 because solr v9 needs Java 11.
what version of Java does Pekko need? Solr 10 will probably be Java 17 minimum version, will that be an issue for Pekko?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] slightly better support for Solr 9 client - (but full support won't be possible until we use Java 11 to build) [incubator-pekko-connectors]
Posted by "pjfanning (via GitHub)" <gi...@apache.org>.
pjfanning commented on code in PR #481:
URL: https://github.com/apache/incubator-pekko-connectors/pull/481#discussion_r1495484578
##########
solr/src/main/scala/org/apache/pekko/stream/connectors/solr/impl/SolrFlowStage.scala:
##########
@@ -114,8 +113,12 @@ private final class SolrFlowLogic[T, C](
message.routingFieldValue.foreach { routingFieldValue =>
val routingField = client match {
- case csc: CloudSolrClient =>
- Option(csc.getIdField)
+ case csc: CloudSolrClient => {
+ val docCollection = Option(csc.getZkStateReader.getCollection(collection))
+ docCollection.flatMap { dc =>
+ Option(dc.getRouter.getRouteField(dc))
+ }
Review Comment:
The whole point is the worry that this Java code could return null. The use of `Option` is deliberate here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] [DRAFT] support Solr9 [incubator-pekko-connectors]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #481:
URL: https://github.com/apache/incubator-pekko-connectors/pull/481#discussion_r1488046734
##########
project/Dependencies.scala:
##########
Review Comment:
9.5 was just released ;-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] slightly better support for Solr 9 client - (but full support won't be possible until we use Java 11 to build) [incubator-pekko-connectors]
Posted by "pjfanning (via GitHub)" <gi...@apache.org>.
pjfanning merged PR #481:
URL: https://github.com/apache/incubator-pekko-connectors/pull/481
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] slightly better support for Solr9 - but not there yet [incubator-pekko-connectors]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on PR #481:
URL: https://github.com/apache/incubator-pekko-connectors/pull/481#issuecomment-1942292717
BTW, are you familiar with https://github.com/inoio/solrs ??
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] support Solr9 [incubator-pekko-connectors]
Posted by "pjfanning (via GitHub)" <gi...@apache.org>.
pjfanning commented on PR #481:
URL: https://github.com/apache/incubator-pekko-connectors/pull/481#issuecomment-1941950086
Apologies - this seemed to be building ok but now I am finding that some of the test code doesn't compile - so I'll need to investigate that. I will need to get back to this later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] slightly better support for Solr 9 client - (but full support won't be possible until we use Java 11 to build) [incubator-pekko-connectors]
Posted by "pjfanning (via GitHub)" <gi...@apache.org>.
pjfanning commented on PR #481:
URL: https://github.com/apache/incubator-pekko-connectors/pull/481#issuecomment-1953132048
Is this something that can be merged?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] slightly better support for Solr 9 client - (but full support won't be possible until we use Java 11 to build) [incubator-pekko-connectors]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #481:
URL: https://github.com/apache/incubator-pekko-connectors/pull/481#discussion_r1495263312
##########
solr/src/main/scala/org/apache/pekko/stream/connectors/solr/impl/SolrFlowStage.scala:
##########
@@ -114,8 +113,12 @@ private final class SolrFlowLogic[T, C](
message.routingFieldValue.foreach { routingFieldValue =>
val routingField = client match {
- case csc: CloudSolrClient =>
- Option(csc.getIdField)
+ case csc: CloudSolrClient => {
+ val docCollection = Option(csc.getZkStateReader.getCollection(collection))
+ docCollection.flatMap { dc =>
+ Option(dc.getRouter.getRouteField(dc))
+ }
Review Comment:
Agreed, unless you are expecting values like `csc.getZkStateReader.getCollection(collection)` to be `null`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] slightly better support for Solr 9 client - (but full support won't be possible until we use Java 11 to build) [incubator-pekko-connectors]
Posted by "samueleresca (via GitHub)" <gi...@apache.org>.
samueleresca commented on code in PR #481:
URL: https://github.com/apache/incubator-pekko-connectors/pull/481#discussion_r1495084621
##########
solr/src/main/scala/org/apache/pekko/stream/connectors/solr/impl/SolrFlowStage.scala:
##########
@@ -114,8 +113,12 @@ private final class SolrFlowLogic[T, C](
message.routingFieldValue.foreach { routingFieldValue =>
val routingField = client match {
- case csc: CloudSolrClient =>
- Option(csc.getIdField)
+ case csc: CloudSolrClient => {
+ val docCollection = Option(csc.getZkStateReader.getClusterState.getCollectionOrNull(collection))
Review Comment:
minor: I think you can directly access the docCollection using `csc.getZkStateReader.getCollection`. The underlying call is checking for the `clusterState` and calls the `getCollectionOrNull`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] slightly better support for Solr 9 client - (but full support won't be possible until we use Java 11 to build) [incubator-pekko-connectors]
Posted by "pjfanning (via GitHub)" <gi...@apache.org>.
pjfanning commented on code in PR #481:
URL: https://github.com/apache/incubator-pekko-connectors/pull/481#discussion_r1495091144
##########
solr/src/main/scala/org/apache/pekko/stream/connectors/solr/impl/SolrFlowStage.scala:
##########
@@ -114,8 +113,12 @@ private final class SolrFlowLogic[T, C](
message.routingFieldValue.foreach { routingFieldValue =>
val routingField = client match {
- case csc: CloudSolrClient =>
- Option(csc.getIdField)
+ case csc: CloudSolrClient => {
+ val docCollection = Option(csc.getZkStateReader.getClusterState.getCollectionOrNull(collection))
Review Comment:
thanks - change made
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] slightly better support for Solr 9 client - (but full support won't be possible until we use Java 11 to build) [incubator-pekko-connectors]
Posted by "Roiocam (via GitHub)" <gi...@apache.org>.
Roiocam commented on code in PR #481:
URL: https://github.com/apache/incubator-pekko-connectors/pull/481#discussion_r1495141542
##########
solr/src/main/scala/org/apache/pekko/stream/connectors/solr/impl/SolrFlowStage.scala:
##########
@@ -114,8 +113,12 @@ private final class SolrFlowLogic[T, C](
message.routingFieldValue.foreach { routingFieldValue =>
val routingField = client match {
- case csc: CloudSolrClient =>
- Option(csc.getIdField)
+ case csc: CloudSolrClient => {
+ val docCollection = Option(csc.getZkStateReader.getCollection(collection))
+ docCollection.flatMap { dc =>
+ Option(dc.getRouter.getRouteField(dc))
+ }
Review Comment:
```suggestion
case csc: CloudSolrClient => {
val docCollection = Some(csc.getZkStateReader.getCollection(collection))
docCollection.flatMap { dc =>
Some(dc.getRouter.getRouteField(dc))
}
```
`Some` and `None` is a group of subclasses of `Option`, in Scala, we recommend using `Some` rather than directly use `Option`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] slightly better support for Solr9 - but not there yet [incubator-pekko-connectors]
Posted by "pjfanning (via GitHub)" <gi...@apache.org>.
pjfanning commented on PR #481:
URL: https://github.com/apache/incubator-pekko-connectors/pull/481#issuecomment-1942311432
> BTW, are you familiar with https://github.com/inoio/solrs ??
Thanks @epugh that could be an option. I'll have a look at it in next few weeks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org