You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by "nicolo-rinaldi (via GitHub)" <gi...@apache.org> on 2026/04/28 09:46:40 UTC
[PR] Text to vector refactoring [solr]
nicolo-rinaldi opened a new pull request, #4375:
URL: https://github.com/apache/solr/pull/4375
<!--
_(If you are a project committer then you may remove some/all of the following template.)_
Before creating a pull request, please file an issue in the ASF Jira system for Solr:
* https://issues.apache.org/jira/projects/SOLR
For something minor (i.e. that wouldn't be worth putting in release notes), you can skip JIRA.
To create a Jira issue, you will need to create an account there first.
The title of the PR should reference the Jira issue number in the form:
* SOLR-####: <short description of problem or changes>
SOLR must be fully capitalized. A short description helps people scanning pull requests for items they can work on.
Properly referencing the issue in the title ensures that Jira is correctly updated with code review comments and commits. -->
# Description
This refactoring is needed to prepare the code for the addition of the new module "Document Enrichment with LLMs" already in [PR #4229](https://github.com/apache/solr/pull/4259/), to avoid duplicated code as much as possible.
# Solution
Abstract classes have been created for both the model and the store/rest packages. Also, renamed the exception to generalize across models. Implemented actual instances of the new abstract classes for the Text-to-vector module.
Claude code has been used as an assistant for coding this PR.
# Tests
No tests have been added, since only interfaces and abstract classes have been developed.
# Checklist
Please review the following and check all that apply:
- [x] I have reviewed the guidelines for [How to Contribute](https://github.com/apache/solr/blob/main/CONTRIBUTING.md) and my code conforms to the standards described there to the best of my ability.
- [ ] I have created a Jira issue and added the issue ID to my pull request title.
- [ ] I have given Solr maintainers [access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork) to contribute to my PR branch. (optional but recommended, not available for branches on forks living under an organisation)
- [x] I have developed this patch against the `main` branch.
- [x] I have run `./gradlew check`.
- [ ] I have added tests for my changes.
- [ ] I have added documentation for the [Reference Guide](https://github.com/apache/solr/tree/main/solr/solr-ref-guide)
- [ ] I have added a [changelog entry](https://github.com/apache/solr/blob/main/dev-docs/changelog.adoc) for my change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Text to vector refactoring [solr]
Posted by "nicolo-rinaldi (via GitHub)" <gi...@apache.org>.
nicolo-rinaldi commented on code in PR #4375:
URL: https://github.com/apache/solr/pull/4375#discussion_r3189196908
##########
solr/modules/language-models/src/java/org/apache/solr/languagemodels/store/rest/ManagedModelStore.java:
##########
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.languagemodels.store.rest;
+
+import java.lang.invoke.MethodHandles;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import net.jcip.annotations.ThreadSafe;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.core.SolrResourceLoader;
+import org.apache.solr.languagemodels.model.SolrLanguageModel;
+import org.apache.solr.languagemodels.store.LanguageModelException;
+import org.apache.solr.languagemodels.store.LanguageModelStore;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.rest.BaseSolrResource;
+import org.apache.solr.rest.ManagedResource;
+import org.apache.solr.rest.ManagedResourceStorage;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Abstract base class for {@link ManagedResource} wrappers that expose a {@link LanguageModelStore}
+ * via the REST API. Concrete subclasses supply the REST endpoint and the model instantiation logic.
+ */
+@ThreadSafe
+public abstract class ManagedModelStore<M extends SolrLanguageModel> extends ManagedResource
Review Comment:
Renamed into `ModelT`
##########
solr/modules/language-models/src/java/org/apache/solr/languagemodels/store/rest/ManagedModelStore.java:
##########
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.languagemodels.store.rest;
+
+import java.lang.invoke.MethodHandles;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import net.jcip.annotations.ThreadSafe;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.core.SolrResourceLoader;
+import org.apache.solr.languagemodels.model.SolrLanguageModel;
+import org.apache.solr.languagemodels.store.LanguageModelException;
+import org.apache.solr.languagemodels.store.LanguageModelStore;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.rest.BaseSolrResource;
+import org.apache.solr.rest.ManagedResource;
+import org.apache.solr.rest.ManagedResourceStorage;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Abstract base class for {@link ManagedResource} wrappers that expose a {@link LanguageModelStore}
+ * via the REST API. Concrete subclasses supply the REST endpoint and the model instantiation logic.
+ */
+@ThreadSafe
+public abstract class ManagedModelStore<M extends SolrLanguageModel> extends ManagedResource
Review Comment:
Renamed to `ModelT`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Text to vector refactoring [solr]
Posted by "nicolo-rinaldi (via GitHub)" <gi...@apache.org>.
nicolo-rinaldi commented on code in PR #4375:
URL: https://github.com/apache/solr/pull/4375#discussion_r3217562220
##########
changelog/unreleased/PR#4375-language-model-refactoring.yml:
##########
@@ -0,0 +1,9 @@
+title: Refactor of the language-model module to accomodate new "Document Enrichment with LLMs" feature avoiding code duplication.
Review Comment:
Changed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Text to vector refactoring [solr]
Posted by "aruggero (via GitHub)" <gi...@apache.org>.
aruggero commented on code in PR #4375:
URL: https://github.com/apache/solr/pull/4375#discussion_r3166414102
##########
solr/modules/language-models/src/java/org/apache/solr/languagemodels/store/LanguageModelStore.java:
##########
@@ -14,53 +14,52 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.solr.languagemodels.textvectorisation.store;
+package org.apache.solr.languagemodels.store;
import java.util.ArrayList;
import java.util.Collections;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
-import org.apache.solr.languagemodels.textvectorisation.model.SolrTextToVectorModel;
+import org.apache.solr.languagemodels.model.SolrLanguageModel;
-/** Simple store to manage CRUD operations on the {@link SolrTextToVectorModel} */
-public class TextToVectorModelStore {
+/** Generic store to manage CRUD operations on models that extend {@link SolrLanguageModel} */
Review Comment:
To discuss
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Text to vector refactoring [solr]
Posted by "nicolo-rinaldi (via GitHub)" <gi...@apache.org>.
nicolo-rinaldi commented on code in PR #4375:
URL: https://github.com/apache/solr/pull/4375#discussion_r3166821192
##########
solr/modules/language-models/src/test/org/apache/solr/languagemodels/textvectorisation/store/rest/TestModelManagerPersistence.java:
##########
@@ -39,12 +39,12 @@ public void cleanup() throws Exception {
}
@Test
- public void testModelAreStoredCompact() throws Exception {
+ public void testModelAreStored() throws Exception {
Review Comment:
Yes, it can be removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Text to vector refactoring [solr]
Posted by "aruggero (via GitHub)" <gi...@apache.org>.
aruggero commented on code in PR #4375:
URL: https://github.com/apache/solr/pull/4375#discussion_r3166407267
##########
solr/modules/language-models/src/test/org/apache/solr/languagemodels/textvectorisation/store/rest/TestModelManagerPersistence.java:
##########
@@ -39,12 +39,12 @@ public void cleanup() throws Exception {
}
@Test
- public void testModelAreStoredCompact() throws Exception {
+ public void testModelAreStored() throws Exception {
Review Comment:
So does it make sense to have this test?
Was this made just to check if compact?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Text to vector refactoring [solr]
Posted by "nicolo-rinaldi (via GitHub)" <gi...@apache.org>.
nicolo-rinaldi commented on code in PR #4375:
URL: https://github.com/apache/solr/pull/4375#discussion_r3166843986
##########
solr/modules/language-models/src/java/org/apache/solr/languagemodels/store/rest/package-info.java:
##########
@@ -0,0 +1,19 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/** Contains model store related classes. */
Review Comment:
I replied to the wrong comment. I changed it now, adding "rest"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Text to vector refactoring [solr]
Posted by "aruggero (via GitHub)" <gi...@apache.org>.
aruggero commented on code in PR #4375:
URL: https://github.com/apache/solr/pull/4375#discussion_r3166398635
##########
solr/modules/language-models/src/java/org/apache/solr/languagemodels/textvectorisation/model/SolrTextToVectorModel.java:
##########
Review Comment:
To discuss
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Text to vector refactoring [solr]
Posted by "alessandrobenedetti (via GitHub)" <gi...@apache.org>.
alessandrobenedetti merged PR #4375:
URL: https://github.com/apache/solr/pull/4375
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Text to vector refactoring [solr]
Posted by "alessandrobenedetti (via GitHub)" <gi...@apache.org>.
alessandrobenedetti commented on code in PR #4375:
URL: https://github.com/apache/solr/pull/4375#discussion_r3207728707
##########
solr/modules/language-models/src/test/org/apache/solr/languagemodels/textvectorisation/store/rest/TestModelManagerPersistence.java:
##########
@@ -38,15 +38,6 @@ public void cleanup() throws Exception {
afterTest();
}
- @Test
Review Comment:
Why has this test been removed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Text to vector refactoring [solr]
Posted by "aruggero (via GitHub)" <gi...@apache.org>.
aruggero commented on code in PR #4375:
URL: https://github.com/apache/solr/pull/4375#discussion_r3166546430
##########
solr/modules/language-models/src/java/org/apache/solr/languagemodels/store/rest/ManagedModelStore.java:
##########
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.languagemodels.store.rest;
+
+import java.lang.invoke.MethodHandles;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import net.jcip.annotations.ThreadSafe;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.core.SolrResourceLoader;
+import org.apache.solr.languagemodels.model.SolrLanguageModel;
+import org.apache.solr.languagemodels.store.LanguageModelException;
+import org.apache.solr.languagemodels.store.LanguageModelStore;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.rest.BaseSolrResource;
+import org.apache.solr.rest.ManagedResource;
+import org.apache.solr.rest.ManagedResourceStorage;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Abstract base class for {@link ManagedResource} wrappers that expose a {@link LanguageModelStore}
+ * via the REST API. Concrete subclasses supply the REST endpoint and the model instantiation logic.
+ */
+@ThreadSafe
+public abstract class ManagedModelStore<M extends SolrLanguageModel> extends ManagedResource
Review Comment:
Maybe LM? to discuss
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Text to vector refactoring [solr]
Posted by "aruggero (via GitHub)" <gi...@apache.org>.
aruggero commented on code in PR #4375:
URL: https://github.com/apache/solr/pull/4375#discussion_r3166371919
##########
solr/modules/language-models/src/java/org/apache/solr/languagemodels/store/rest/ManagedLanguageModelStore.java:
##########
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.languagemodels.store.rest;
+
+import java.lang.invoke.MethodHandles;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import net.jcip.annotations.ThreadSafe;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.core.SolrResourceLoader;
+import org.apache.solr.languagemodels.model.SolrLanguageModel;
+import org.apache.solr.languagemodels.store.LanguageModelException;
+import org.apache.solr.languagemodels.store.LanguageModelStore;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.rest.BaseSolrResource;
+import org.apache.solr.rest.ManagedResource;
+import org.apache.solr.rest.ManagedResourceStorage;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Abstract base class for {@link ManagedResource} wrappers that expose a {@link LanguageModelStore}
+ * via the REST API. Concrete subclasses supply the REST endpoint and the model instantiation logic.
+ */
+@ThreadSafe
+public abstract class ManagedModelStore<M extends SolrLanguageModel> extends ManagedResource
+ implements ManagedResource.ChildResourceSupport {
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
Review Comment:
What is the standard for it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Text to vector refactoring [solr]
Posted by "nicolo-rinaldi (via GitHub)" <gi...@apache.org>.
nicolo-rinaldi commented on code in PR #4375:
URL: https://github.com/apache/solr/pull/4375#discussion_r3161389353
##########
solr/modules/language-models/src/java/org/apache/solr/languagemodels/model/SolrLanguageModel.java:
##########
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.languagemodels.model;
+
+import java.util.Map;
+
+/**
+ * Base class for Solr-managed wrappers around langchain4j used in {@code language-models} module
Review Comment:
Changed
##########
solr/modules/language-models/src/java/org/apache/solr/languagemodels/textvectorisation/store/rest/ManagedTextToVectorModelStore.java:
##########
@@ -93,108 +56,9 @@ public static SolrTextToVectorModel fromModelMap(
(Map<String, Object>) embeddingModel.get(PARAMS_KEY));
}
- private static LinkedHashMap<String, Object> toModelMap(SolrTextToVectorModel model) {
- final LinkedHashMap<String, Object> modelMap = new LinkedHashMap<>(5, 1.0f);
- modelMap.put(NAME_KEY, model.getName());
- modelMap.put(CLASS_KEY, model.getEmbeddingModelClassName());
- modelMap.put(PARAMS_KEY, model.getParams());
- return modelMap;
- }
-
- private final TextToVectorModelStore store;
- private Object managedData;
-
public ManagedTextToVectorModelStore(
String resourceId, SolrResourceLoader loader, ManagedResourceStorage.StorageIO storageIO)
throws SolrException {
super(resourceId, loader, storageIO);
- store = new TextToVectorModelStore();
- }
-
- @Override
- protected ManagedResourceStorage createStorage(
Review Comment:
See the comment in `TestModelManagerPersistence`
##########
solr/modules/language-models/src/java/org/apache/solr/languagemodels/store/rest/ManagedModelStore.java:
##########
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.languagemodels.store.rest;
+
+import java.lang.invoke.MethodHandles;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import net.jcip.annotations.ThreadSafe;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.core.SolrResourceLoader;
+import org.apache.solr.languagemodels.model.SolrLanguageModel;
+import org.apache.solr.languagemodels.store.LanguageModelException;
+import org.apache.solr.languagemodels.store.LanguageModelStore;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.rest.BaseSolrResource;
+import org.apache.solr.rest.ManagedResource;
+import org.apache.solr.rest.ManagedResourceStorage;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Abstract base class for {@link ManagedResource} wrappers that expose a {@link LanguageModelStore}
+ * via the REST API. Concrete subclasses supply the REST endpoint and the model instantiation logic.
+ */
+@ThreadSafe
+public abstract class ManagedModelStore<M extends SolrLanguageModel> extends ManagedResource
+ implements ManagedResource.ChildResourceSupport {
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
Review Comment:
I kept it here due to the fact that most of the logic is in the function that are already in the abstract class
##########
solr/modules/language-models/src/java/org/apache/solr/languagemodels/store/rest/ManagedModelStore.java:
##########
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.languagemodels.store.rest;
+
+import java.lang.invoke.MethodHandles;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import net.jcip.annotations.ThreadSafe;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.core.SolrResourceLoader;
+import org.apache.solr.languagemodels.model.SolrLanguageModel;
+import org.apache.solr.languagemodels.store.LanguageModelException;
+import org.apache.solr.languagemodels.store.LanguageModelStore;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.rest.BaseSolrResource;
+import org.apache.solr.rest.ManagedResource;
+import org.apache.solr.rest.ManagedResourceStorage;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Abstract base class for {@link ManagedResource} wrappers that expose a {@link LanguageModelStore}
+ * via the REST API. Concrete subclasses supply the REST endpoint and the model instantiation logic.
+ */
+@ThreadSafe
+public abstract class ManagedModelStore<M extends SolrLanguageModel> extends ManagedResource
+ implements ManagedResource.ChildResourceSupport {
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ private static final String MODELS_JSON_FIELD = "models";
+
+ protected static final String CLASS_KEY = "class";
+ protected static final String NAME_KEY = "name";
+ protected static final String PARAMS_KEY = "params";
+
+ private final LanguageModelStore<M> store;
+ private Object managedData;
+
+ protected ManagedModelStore(
+ String resourceId, SolrResourceLoader loader, ManagedResourceStorage.StorageIO storageIO)
+ throws SolrException {
+ super(resourceId, loader, storageIO);
+ store = new LanguageModelStore<>();
+ }
+
+ /**
+ * Creates a model instance from the JSON map persisted in the managed resource storage.
+ *
+ * @param loader the resource loader for the current core
+ * @param modelMap a map containing {@code "class"}, {@code "name"}, and {@code "params"} keys
+ * @return the instantiated model
+ */
+ protected abstract M fromModelMap(SolrResourceLoader loader, Map<String, Object> modelMap);
Review Comment:
A static function cannot be declared as abstract
##########
solr/modules/language-models/src/java/org/apache/solr/languagemodels/store/LanguageModelStore.java:
##########
@@ -14,53 +14,52 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.solr.languagemodels.textvectorisation.store;
+package org.apache.solr.languagemodels.store;
import java.util.ArrayList;
import java.util.Collections;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
-import org.apache.solr.languagemodels.textvectorisation.model.SolrTextToVectorModel;
+import org.apache.solr.languagemodels.model.SolrLanguageModel;
-/** Simple store to manage CRUD operations on the {@link SolrTextToVectorModel} */
-public class TextToVectorModelStore {
+/** Generic store to manage CRUD operations on models that extend {@link SolrLanguageModel} */
Review Comment:
We have a different endpoint for each module, so I wanted to keep them different
##########
solr/modules/language-models/src/java/org/apache/solr/languagemodels/textvectorisation/store/rest/ManagedTextToVectorModelStore.java:
##########
@@ -70,21 +45,9 @@ public static ManagedTextToVectorModelStore getManagedModelStore(SolrCore core)
return (ManagedTextToVectorModelStore) core.getRestManager().getManagedResource(REST_END_POINT);
}
- /**
- * Returns the available models as a list of Maps objects. After an update the managed resources
- * needs to return the resources in this format in order to store in json somewhere (zookeeper,
- * disk...)
- *
- * @return the available models as a list of Maps objects
- */
- private static List<Object> modelsAsManagedResources(List<SolrTextToVectorModel> models) {
- return models.stream()
- .map(ManagedTextToVectorModelStore::toModelMap)
- .collect(Collectors.toList());
- }
-
+ @Override
@SuppressWarnings("unchecked")
- public static SolrTextToVectorModel fromModelMap(
+ protected SolrTextToVectorModel fromModelMap(
Review Comment:
Static methods cannot be overridden. I moved this to protected since it's accessed only by its base class
##########
solr/modules/language-models/src/java/org/apache/solr/languagemodels/textvectorisation/model/SolrTextToVectorModel.java:
##########
Review Comment:
Since each model might have different parameter, I wanted to keep this separated
##########
solr/modules/language-models/src/java/org/apache/solr/languagemodels/store/rest/package-info.java:
##########
@@ -0,0 +1,19 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/** Contains model store related classes. */
Review Comment:
Yes, this is leftover from a copy-paste of the license. Changed
##########
solr/modules/language-models/src/test/org/apache/solr/languagemodels/textvectorisation/store/rest/TestModelManagerPersistence.java:
##########
@@ -39,12 +39,12 @@ public void cleanup() throws Exception {
}
@Test
- public void testModelAreStoredCompact() throws Exception {
+ public void testModelAreStored() throws Exception {
Review Comment:
Before it was saved as a compact file, this was done due to a (possibly) copy-paste leftover of this module from the LTR module, where it makes sense to save the files related to the models as compact, due to the possible high number of features. In both these contributions (text-to-vector and following document enrichment), it doesn't make sense to apply the compression to reduce space. This is also a consequence of the fact that the method
```
@Override
protected ManagedResourceStorage createStorage(
ManagedResourceStorage.StorageIO storageIO, SolrResourceLoader loader) throws SolrException {
return new ManagedResourceStorage.JsonStorage(storageIO, loader, -1);
}
```
from `ManagedTextToVectorStore` is no longer overridden. "-1" means "save in a compact format".
##########
solr/modules/language-models/src/java/org/apache/solr/languagemodels/store/rest/ManagedModelStore.java:
##########
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.languagemodels.store.rest;
+
+import java.lang.invoke.MethodHandles;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import net.jcip.annotations.ThreadSafe;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.core.SolrResourceLoader;
+import org.apache.solr.languagemodels.model.SolrLanguageModel;
+import org.apache.solr.languagemodels.store.LanguageModelException;
+import org.apache.solr.languagemodels.store.LanguageModelStore;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.rest.BaseSolrResource;
+import org.apache.solr.rest.ManagedResource;
+import org.apache.solr.rest.ManagedResourceStorage;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Abstract base class for {@link ManagedResource} wrappers that expose a {@link LanguageModelStore}
+ * via the REST API. Concrete subclasses supply the REST endpoint and the model instantiation logic.
+ */
+@ThreadSafe
+public abstract class ManagedModelStore<M extends SolrLanguageModel> extends ManagedResource
Review Comment:
Generics are usually uppercase single letters that can be followed by a number. Do you still want me to rename this variable?
##########
solr/modules/language-models/src/java/org/apache/solr/languagemodels/store/rest/ManagedModelStore.java:
##########
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.languagemodels.store.rest;
+
+import java.lang.invoke.MethodHandles;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import net.jcip.annotations.ThreadSafe;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.core.SolrResourceLoader;
+import org.apache.solr.languagemodels.model.SolrLanguageModel;
+import org.apache.solr.languagemodels.store.LanguageModelException;
+import org.apache.solr.languagemodels.store.LanguageModelStore;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.rest.BaseSolrResource;
+import org.apache.solr.rest.ManagedResource;
+import org.apache.solr.rest.ManagedResourceStorage;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Abstract base class for {@link ManagedResource} wrappers that expose a {@link LanguageModelStore}
+ * via the REST API. Concrete subclasses supply the REST endpoint and the model instantiation logic.
+ */
+@ThreadSafe
+public abstract class ManagedModelStore<M extends SolrLanguageModel> extends ManagedResource
+ implements ManagedResource.ChildResourceSupport {
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ private static final String MODELS_JSON_FIELD = "models";
+
+ protected static final String CLASS_KEY = "class";
+ protected static final String NAME_KEY = "name";
+ protected static final String PARAMS_KEY = "params";
+
+ private final LanguageModelStore<M> store;
+ private Object managedData;
+
+ protected ManagedModelStore(
+ String resourceId, SolrResourceLoader loader, ManagedResourceStorage.StorageIO storageIO)
+ throws SolrException {
+ super(resourceId, loader, storageIO);
+ store = new LanguageModelStore<>();
+ }
+
+ /**
+ * Creates a model instance from the JSON map persisted in the managed resource storage.
+ *
+ * @param loader the resource loader for the current core
+ * @param modelMap a map containing {@code "class"}, {@code "name"}, and {@code "params"} keys
+ * @return the instantiated model
+ */
+ protected abstract M fromModelMap(SolrResourceLoader loader, Map<String, Object> modelMap);
+
+ private static LinkedHashMap<String, Object> toModelMap(SolrLanguageModel model) {
+ final LinkedHashMap<String, Object> modelMap = new LinkedHashMap<>(3, 1.0f);
+ modelMap.put(NAME_KEY, model.getName());
+ modelMap.put(CLASS_KEY, model.getModelClassName());
+ modelMap.put(PARAMS_KEY, model.getParams());
+ return modelMap;
+ }
+
+ @Override
+ protected void onManagedDataLoadedFromStorage(NamedList<?> managedInitArgs, Object managedData)
+ throws SolrException {
+ store.clear();
+ this.managedData = managedData;
+ }
+
+ public void loadStoredModels() {
+ log.info("------ managed models ~ loading ------");
+ if ((managedData != null) && (managedData instanceof List)) {
+ @SuppressWarnings("unchecked")
+ final List<Map<String, Object>> models = (List<Map<String, Object>>) managedData;
+ for (final Map<String, Object> model : models) {
+ addModelFromMap(model);
+ }
+ }
+ }
+
+ private void addModelFromMap(Map<String, Object> modelMap) {
+ try {
+ addModel(fromModelMap(solrResourceLoader, modelMap));
+ } catch (final LanguageModelException e) {
+ throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, e);
+ }
+ }
+
+ public void addModel(M model) throws SolrException {
+ try {
+ if (log.isInfoEnabled()) {
+ log.info("adding model {}", model.getName());
+ }
+ store.addModel(model);
+ } catch (final LanguageModelException e) {
+ throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, e);
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ @Override
+ protected Object applyUpdatesToManagedData(Object updates) {
+ if (updates instanceof List) {
+ final List<Map<String, Object>> models = (List<Map<String, Object>>) updates;
+ for (final Map<String, Object> model : models) {
+ addModelFromMap(model);
+ }
+ }
+ if (updates instanceof Map) {
+ addModelFromMap((Map<String, Object>) updates);
+ }
+ return modelsAsManagedResources(store.getModels());
+ }
+
+ @Override
+ public void doDeleteChild(BaseSolrResource endpoint, String childId) {
+ store.delete(childId);
+ storeManagedData(applyUpdatesToManagedData(null));
+ }
+
+ @Override
+ public void doGet(BaseSolrResource endpoint, String childId) {
+ final SolrQueryResponse response = endpoint.getSolrResponse();
+ response.add(MODELS_JSON_FIELD, modelsAsManagedResources(store.getModels()));
+ }
+
+ public M getModel(String modelName) {
+ return store.getModel(modelName);
+ }
+
+ private static List<Object> modelsAsManagedResources(List<? extends SolrLanguageModel> models) {
+ return models.stream().map(ManagedModelStore::toModelMap).collect(Collectors.toList());
+ }
+
+ @Override
+ public String toString() {
+ return getClass().getSimpleName() + " [store=" + store + "]";
Review Comment:
Since we have inheritance, I wanted the name of the implementation of the class and not the abstract one. The implementation right now produces something similar to the following string
```
ManagedTextToVectorModelStore [store=LanguageModelStore [availableModels=[my-bert, ada-002]]]
```
if invoked from the `ManagedTextToVectorModelStore` class.
##########
solr/modules/language-models/src/java/org/apache/solr/languagemodels/store/rest/ManagedModelStore.java:
##########
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.languagemodels.store.rest;
+
+import java.lang.invoke.MethodHandles;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import net.jcip.annotations.ThreadSafe;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.core.SolrResourceLoader;
+import org.apache.solr.languagemodels.model.SolrLanguageModel;
+import org.apache.solr.languagemodels.store.LanguageModelException;
+import org.apache.solr.languagemodels.store.LanguageModelStore;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.rest.BaseSolrResource;
+import org.apache.solr.rest.ManagedResource;
+import org.apache.solr.rest.ManagedResourceStorage;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Abstract base class for {@link ManagedResource} wrappers that expose a {@link LanguageModelStore}
+ * via the REST API. Concrete subclasses supply the REST endpoint and the model instantiation logic.
+ */
+@ThreadSafe
+public abstract class ManagedModelStore<M extends SolrLanguageModel> extends ManagedResource
+ implements ManagedResource.ChildResourceSupport {
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ private static final String MODELS_JSON_FIELD = "models";
+
+ protected static final String CLASS_KEY = "class";
+ protected static final String NAME_KEY = "name";
+ protected static final String PARAMS_KEY = "params";
+
+ private final LanguageModelStore<M> store;
+ private Object managedData;
+
+ protected ManagedModelStore(
+ String resourceId, SolrResourceLoader loader, ManagedResourceStorage.StorageIO storageIO)
+ throws SolrException {
+ super(resourceId, loader, storageIO);
+ store = new LanguageModelStore<>();
+ }
+
+ /**
+ * Creates a model instance from the JSON map persisted in the managed resource storage.
+ *
+ * @param loader the resource loader for the current core
+ * @param modelMap a map containing {@code "class"}, {@code "name"}, and {@code "params"} keys
+ * @return the instantiated model
+ */
+ protected abstract M fromModelMap(SolrResourceLoader loader, Map<String, Object> modelMap);
+
+ private static LinkedHashMap<String, Object> toModelMap(SolrLanguageModel model) {
+ final LinkedHashMap<String, Object> modelMap = new LinkedHashMap<>(3, 1.0f);
+ modelMap.put(NAME_KEY, model.getName());
+ modelMap.put(CLASS_KEY, model.getModelClassName());
+ modelMap.put(PARAMS_KEY, model.getParams());
+ return modelMap;
+ }
+
+ @Override
+ protected void onManagedDataLoadedFromStorage(NamedList<?> managedInitArgs, Object managedData)
+ throws SolrException {
+ store.clear();
+ this.managedData = managedData;
+ }
+
+ public void loadStoredModels() {
+ log.info("------ managed models ~ loading ------");
+ if ((managedData != null) && (managedData instanceof List)) {
+ @SuppressWarnings("unchecked")
+ final List<Map<String, Object>> models = (List<Map<String, Object>>) managedData;
+ for (final Map<String, Object> model : models) {
+ addModelFromMap(model);
+ }
+ }
+ }
+
+ private void addModelFromMap(Map<String, Object> modelMap) {
+ try {
+ addModel(fromModelMap(solrResourceLoader, modelMap));
+ } catch (final LanguageModelException e) {
+ throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, e);
+ }
+ }
+
+ public void addModel(M model) throws SolrException {
+ try {
+ if (log.isInfoEnabled()) {
+ log.info("adding model {}", model.getName());
+ }
+ store.addModel(model);
+ } catch (final LanguageModelException e) {
+ throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, e);
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ @Override
+ protected Object applyUpdatesToManagedData(Object updates) {
+ if (updates instanceof List) {
+ final List<Map<String, Object>> models = (List<Map<String, Object>>) updates;
+ for (final Map<String, Object> model : models) {
+ addModelFromMap(model);
+ }
+ }
+ if (updates instanceof Map) {
+ addModelFromMap((Map<String, Object>) updates);
+ }
+ return modelsAsManagedResources(store.getModels());
+ }
+
+ @Override
+ public void doDeleteChild(BaseSolrResource endpoint, String childId) {
+ store.delete(childId);
+ storeManagedData(applyUpdatesToManagedData(null));
+ }
+
+ @Override
+ public void doGet(BaseSolrResource endpoint, String childId) {
+ final SolrQueryResponse response = endpoint.getSolrResponse();
+ response.add(MODELS_JSON_FIELD, modelsAsManagedResources(store.getModels()));
+ }
+
+ public M getModel(String modelName) {
+ return store.getModel(modelName);
+ }
+
+ private static List<Object> modelsAsManagedResources(List<? extends SolrLanguageModel> models) {
+ return models.stream().map(ManagedModelStore::toModelMap).collect(Collectors.toList());
+ }
+
+ @Override
+ public String toString() {
+ return getClass().getSimpleName() + " [store=" + store + "]";
Review Comment:
I changed class name to `ManagedLanguageModelStore`
##########
solr/modules/language-models/src/java/org/apache/solr/languagemodels/store/LanguageModelStore.java:
##########
@@ -14,53 +14,52 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.solr.languagemodels.textvectorisation.store;
+package org.apache.solr.languagemodels.store;
import java.util.ArrayList;
import java.util.Collections;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
-import org.apache.solr.languagemodels.textvectorisation.model.SolrTextToVectorModel;
+import org.apache.solr.languagemodels.model.SolrLanguageModel;
-/** Simple store to manage CRUD operations on the {@link SolrTextToVectorModel} */
-public class TextToVectorModelStore {
+/** Generic store to manage CRUD operations on models that extend {@link SolrLanguageModel} */
+public class LanguageModelStore<M extends SolrLanguageModel> {
Review Comment:
Same as before
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Text to vector refactoring [solr]
Posted by "aruggero (via GitHub)" <gi...@apache.org>.
aruggero commented on code in PR #4375:
URL: https://github.com/apache/solr/pull/4375#discussion_r3159860946
##########
solr/modules/language-models/src/java/org/apache/solr/languagemodels/model/SolrLanguageModel.java:
##########
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.languagemodels.model;
+
+import java.util.Map;
+
+/**
+ * Base class for Solr-managed wrappers around langchain4j used in {@code language-models} module
Review Comment:
Abstract base class
##########
solr/modules/language-models/src/java/org/apache/solr/languagemodels/model/package-info.java:
##########
@@ -16,4 +16,4 @@
*/
Review Comment:
Is this still a "store" related package?
##########
solr/modules/language-models/src/java/org/apache/solr/languagemodels/store/rest/ManagedModelStore.java:
##########
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.languagemodels.store.rest;
+
+import java.lang.invoke.MethodHandles;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import net.jcip.annotations.ThreadSafe;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.core.SolrResourceLoader;
+import org.apache.solr.languagemodels.model.SolrLanguageModel;
+import org.apache.solr.languagemodels.store.LanguageModelException;
+import org.apache.solr.languagemodels.store.LanguageModelStore;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.rest.BaseSolrResource;
+import org.apache.solr.rest.ManagedResource;
+import org.apache.solr.rest.ManagedResourceStorage;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Abstract base class for {@link ManagedResource} wrappers that expose a {@link LanguageModelStore}
+ * via the REST API. Concrete subclasses supply the REST endpoint and the model instantiation logic.
+ */
+@ThreadSafe
+public abstract class ManagedModelStore<M extends SolrLanguageModel> extends ManagedResource
+ implements ManagedResource.ChildResourceSupport {
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ private static final String MODELS_JSON_FIELD = "models";
+
+ protected static final String CLASS_KEY = "class";
+ protected static final String NAME_KEY = "name";
+ protected static final String PARAMS_KEY = "params";
+
+ private final LanguageModelStore<M> store;
+ private Object managedData;
+
+ protected ManagedModelStore(
+ String resourceId, SolrResourceLoader loader, ManagedResourceStorage.StorageIO storageIO)
+ throws SolrException {
+ super(resourceId, loader, storageIO);
+ store = new LanguageModelStore<>();
+ }
+
+ /**
+ * Creates a model instance from the JSON map persisted in the managed resource storage.
+ *
+ * @param loader the resource loader for the current core
+ * @param modelMap a map containing {@code "class"}, {@code "name"}, and {@code "params"} keys
+ * @return the instantiated model
+ */
+ protected abstract M fromModelMap(SolrResourceLoader loader, Map<String, Object> modelMap);
Review Comment:
Why registerManagedTextToVectorModelStore and getManagedModelStore are not abstract like this method?
##########
solr/modules/language-models/src/java/org/apache/solr/languagemodels/textvectorisation/store/rest/ManagedTextToVectorModelStore.java:
##########
@@ -70,21 +45,9 @@ public static ManagedTextToVectorModelStore getManagedModelStore(SolrCore core)
return (ManagedTextToVectorModelStore) core.getRestManager().getManagedResource(REST_END_POINT);
}
- /**
- * Returns the available models as a list of Maps objects. After an update the managed resources
- * needs to return the resources in this format in order to store in json somewhere (zookeeper,
- * disk...)
- *
- * @return the available models as a list of Maps objects
- */
- private static List<Object> modelsAsManagedResources(List<SolrTextToVectorModel> models) {
- return models.stream()
- .map(ManagedTextToVectorModelStore::toModelMap)
- .collect(Collectors.toList());
- }
-
+ @Override
@SuppressWarnings("unchecked")
- public static SolrTextToVectorModel fromModelMap(
+ protected SolrTextToVectorModel fromModelMap(
Review Comment:
Why?
##########
solr/modules/language-models/src/java/org/apache/solr/languagemodels/textvectorisation/model/SolrTextToVectorModel.java:
##########
Review Comment:
Would it be worth also moving the common part of this logic to SolrLanguageModel and generalizing it?
##########
solr/modules/language-models/src/java/org/apache/solr/languagemodels/store/rest/ManagedModelStore.java:
##########
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.languagemodels.store.rest;
+
+import java.lang.invoke.MethodHandles;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import net.jcip.annotations.ThreadSafe;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.core.SolrResourceLoader;
+import org.apache.solr.languagemodels.model.SolrLanguageModel;
+import org.apache.solr.languagemodels.store.LanguageModelException;
+import org.apache.solr.languagemodels.store.LanguageModelStore;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.rest.BaseSolrResource;
+import org.apache.solr.rest.ManagedResource;
+import org.apache.solr.rest.ManagedResourceStorage;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Abstract base class for {@link ManagedResource} wrappers that expose a {@link LanguageModelStore}
+ * via the REST API. Concrete subclasses supply the REST endpoint and the model instantiation logic.
+ */
+@ThreadSafe
+public abstract class ManagedModelStore<M extends SolrLanguageModel> extends ManagedResource
+ implements ManagedResource.ChildResourceSupport {
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
Review Comment:
Why is the logger here, while for SolrLanguageModel it is in the classes that extend the abstract class?
##########
solr/modules/language-models/src/java/org/apache/solr/languagemodels/store/rest/package-info.java:
##########
@@ -0,0 +1,19 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/** Contains model store related classes. */
Review Comment:
Isn't this rest specific?
##########
solr/modules/language-models/src/test/org/apache/solr/languagemodels/textvectorisation/store/rest/TestModelManagerPersistence.java:
##########
@@ -39,12 +39,12 @@ public void cleanup() throws Exception {
}
@Test
- public void testModelAreStoredCompact() throws Exception {
+ public void testModelAreStored() throws Exception {
Review Comment:
What does this mean?
##########
solr/modules/language-models/src/java/org/apache/solr/languagemodels/store/rest/ManagedModelStore.java:
##########
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.languagemodels.store.rest;
+
+import java.lang.invoke.MethodHandles;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import net.jcip.annotations.ThreadSafe;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.core.SolrResourceLoader;
+import org.apache.solr.languagemodels.model.SolrLanguageModel;
+import org.apache.solr.languagemodels.store.LanguageModelException;
+import org.apache.solr.languagemodels.store.LanguageModelStore;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.rest.BaseSolrResource;
+import org.apache.solr.rest.ManagedResource;
+import org.apache.solr.rest.ManagedResourceStorage;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Abstract base class for {@link ManagedResource} wrappers that expose a {@link LanguageModelStore}
+ * via the REST API. Concrete subclasses supply the REST endpoint and the model instantiation logic.
+ */
+@ThreadSafe
+public abstract class ManagedModelStore<M extends SolrLanguageModel> extends ManagedResource
Review Comment:
Is there a better name variable instead of "M"?
##########
solr/modules/language-models/src/java/org/apache/solr/languagemodels/store/rest/ManagedModelStore.java:
##########
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.languagemodels.store.rest;
+
+import java.lang.invoke.MethodHandles;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import net.jcip.annotations.ThreadSafe;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.core.SolrResourceLoader;
+import org.apache.solr.languagemodels.model.SolrLanguageModel;
+import org.apache.solr.languagemodels.store.LanguageModelException;
+import org.apache.solr.languagemodels.store.LanguageModelStore;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.rest.BaseSolrResource;
+import org.apache.solr.rest.ManagedResource;
+import org.apache.solr.rest.ManagedResourceStorage;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Abstract base class for {@link ManagedResource} wrappers that expose a {@link LanguageModelStore}
+ * via the REST API. Concrete subclasses supply the REST endpoint and the model instantiation logic.
+ */
+@ThreadSafe
+public abstract class ManagedModelStore<M extends SolrLanguageModel> extends ManagedResource
+ implements ManagedResource.ChildResourceSupport {
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ private static final String MODELS_JSON_FIELD = "models";
+
+ protected static final String CLASS_KEY = "class";
+ protected static final String NAME_KEY = "name";
+ protected static final String PARAMS_KEY = "params";
+
+ private final LanguageModelStore<M> store;
+ private Object managedData;
+
+ protected ManagedModelStore(
+ String resourceId, SolrResourceLoader loader, ManagedResourceStorage.StorageIO storageIO)
+ throws SolrException {
+ super(resourceId, loader, storageIO);
+ store = new LanguageModelStore<>();
+ }
+
+ /**
+ * Creates a model instance from the JSON map persisted in the managed resource storage.
+ *
+ * @param loader the resource loader for the current core
+ * @param modelMap a map containing {@code "class"}, {@code "name"}, and {@code "params"} keys
+ * @return the instantiated model
+ */
+ protected abstract M fromModelMap(SolrResourceLoader loader, Map<String, Object> modelMap);
+
+ private static LinkedHashMap<String, Object> toModelMap(SolrLanguageModel model) {
+ final LinkedHashMap<String, Object> modelMap = new LinkedHashMap<>(3, 1.0f);
+ modelMap.put(NAME_KEY, model.getName());
+ modelMap.put(CLASS_KEY, model.getModelClassName());
+ modelMap.put(PARAMS_KEY, model.getParams());
+ return modelMap;
+ }
+
+ @Override
+ protected void onManagedDataLoadedFromStorage(NamedList<?> managedInitArgs, Object managedData)
+ throws SolrException {
+ store.clear();
+ this.managedData = managedData;
+ }
+
+ public void loadStoredModels() {
+ log.info("------ managed models ~ loading ------");
+ if ((managedData != null) && (managedData instanceof List)) {
+ @SuppressWarnings("unchecked")
+ final List<Map<String, Object>> models = (List<Map<String, Object>>) managedData;
+ for (final Map<String, Object> model : models) {
+ addModelFromMap(model);
+ }
+ }
+ }
+
+ private void addModelFromMap(Map<String, Object> modelMap) {
+ try {
+ addModel(fromModelMap(solrResourceLoader, modelMap));
+ } catch (final LanguageModelException e) {
+ throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, e);
+ }
+ }
+
+ public void addModel(M model) throws SolrException {
+ try {
+ if (log.isInfoEnabled()) {
+ log.info("adding model {}", model.getName());
+ }
+ store.addModel(model);
+ } catch (final LanguageModelException e) {
+ throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, e);
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ @Override
+ protected Object applyUpdatesToManagedData(Object updates) {
+ if (updates instanceof List) {
+ final List<Map<String, Object>> models = (List<Map<String, Object>>) updates;
+ for (final Map<String, Object> model : models) {
+ addModelFromMap(model);
+ }
+ }
+ if (updates instanceof Map) {
+ addModelFromMap((Map<String, Object>) updates);
+ }
+ return modelsAsManagedResources(store.getModels());
+ }
+
+ @Override
+ public void doDeleteChild(BaseSolrResource endpoint, String childId) {
+ store.delete(childId);
+ storeManagedData(applyUpdatesToManagedData(null));
+ }
+
+ @Override
+ public void doGet(BaseSolrResource endpoint, String childId) {
+ final SolrQueryResponse response = endpoint.getSolrResponse();
+ response.add(MODELS_JSON_FIELD, modelsAsManagedResources(store.getModels()));
+ }
+
+ public M getModel(String modelName) {
+ return store.getModel(modelName);
+ }
+
+ private static List<Object> modelsAsManagedResources(List<? extends SolrLanguageModel> models) {
+ return models.stream().map(ManagedModelStore::toModelMap).collect(Collectors.toList());
+ }
+
+ @Override
+ public String toString() {
+ return getClass().getSimpleName() + " [store=" + store + "]";
Review Comment:
Why do we not have "ManagedLargeLanguageModelStore" anymore here? or something similar
##########
solr/modules/language-models/src/java/org/apache/solr/languagemodels/store/LanguageModelStore.java:
##########
@@ -14,53 +14,52 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.solr.languagemodels.textvectorisation.store;
+package org.apache.solr.languagemodels.store;
import java.util.ArrayList;
import java.util.Collections;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
-import org.apache.solr.languagemodels.textvectorisation.model.SolrTextToVectorModel;
+import org.apache.solr.languagemodels.model.SolrLanguageModel;
-/** Simple store to manage CRUD operations on the {@link SolrTextToVectorModel} */
-public class TextToVectorModelStore {
+/** Generic store to manage CRUD operations on models that extend {@link SolrLanguageModel} */
Review Comment:
Is this class necessary? Can't we just have ManagedModelStore with everything inside?
##########
solr/modules/language-models/src/java/org/apache/solr/languagemodels/store/LanguageModelStore.java:
##########
@@ -14,53 +14,52 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.solr.languagemodels.textvectorisation.store;
+package org.apache.solr.languagemodels.store;
import java.util.ArrayList;
import java.util.Collections;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
-import org.apache.solr.languagemodels.textvectorisation.model.SolrTextToVectorModel;
+import org.apache.solr.languagemodels.model.SolrLanguageModel;
-/** Simple store to manage CRUD operations on the {@link SolrTextToVectorModel} */
-public class TextToVectorModelStore {
+/** Generic store to manage CRUD operations on models that extend {@link SolrLanguageModel} */
+public class LanguageModelStore<M extends SolrLanguageModel> {
Review Comment:
Again, a better name instead of "M"?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Text to vector refactoring [solr]
Posted by "aruggero (via GitHub)" <gi...@apache.org>.
aruggero commented on code in PR #4375:
URL: https://github.com/apache/solr/pull/4375#discussion_r3166393659
##########
solr/modules/language-models/src/java/org/apache/solr/languagemodels/store/rest/package-info.java:
##########
@@ -0,0 +1,19 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/** Contains model store related classes. */
Review Comment:
Where?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Text to vector refactoring [solr]
Posted by "nicolo-rinaldi (via GitHub)" <gi...@apache.org>.
nicolo-rinaldi commented on code in PR #4375:
URL: https://github.com/apache/solr/pull/4375#discussion_r3189199052
##########
solr/modules/language-models/src/java/org/apache/solr/languagemodels/store/LanguageModelStore.java:
##########
@@ -14,53 +14,52 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.solr.languagemodels.textvectorisation.store;
+package org.apache.solr.languagemodels.store;
import java.util.ArrayList;
import java.util.Collections;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
-import org.apache.solr.languagemodels.textvectorisation.model.SolrTextToVectorModel;
+import org.apache.solr.languagemodels.model.SolrLanguageModel;
-/** Simple store to manage CRUD operations on the {@link SolrTextToVectorModel} */
-public class TextToVectorModelStore {
+/** Generic store to manage CRUD operations on models that extend {@link SolrLanguageModel} */
+public class LanguageModelStore<M extends SolrLanguageModel> {
Review Comment:
Renamed to `ModelT`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Text to vector refactoring [solr]
Posted by "nicolo-rinaldi (via GitHub)" <gi...@apache.org>.
nicolo-rinaldi commented on code in PR #4375:
URL: https://github.com/apache/solr/pull/4375#discussion_r3189931134
##########
solr/modules/language-models/src/java/org/apache/solr/languagemodels/store/rest/ManagedLanguageModelStore.java:
##########
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.languagemodels.store.rest;
+
+import java.lang.invoke.MethodHandles;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import net.jcip.annotations.ThreadSafe;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.core.SolrResourceLoader;
+import org.apache.solr.languagemodels.model.SolrLanguageModel;
+import org.apache.solr.languagemodels.store.LanguageModelException;
+import org.apache.solr.languagemodels.store.LanguageModelStore;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.rest.BaseSolrResource;
+import org.apache.solr.rest.ManagedResource;
+import org.apache.solr.rest.ManagedResourceStorage;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Abstract base class for {@link ManagedResource} wrappers that expose a {@link LanguageModelStore}
+ * via the REST API. Concrete subclasses supply the REST endpoint and the model instantiation logic.
+ */
+@ThreadSafe
+public abstract class ManagedModelStore<M extends SolrLanguageModel> extends ManagedResource
+ implements ManagedResource.ChildResourceSupport {
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
Review Comment:
The loggers in Solr are overridden when used in the same way:
```
private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
```
An example of this is `RequestHandlerBase`, which has its own logger, and `CollectionsHandler` opens another logger for the specific class. Since the logger is not used in `SolrLanguageModel`, I decided to avoid adding an unused variable. This way of working seems to be the standard in Solr
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Text to vector refactoring [solr]
Posted by "alessandrobenedetti (via GitHub)" <gi...@apache.org>.
alessandrobenedetti commented on code in PR #4375:
URL: https://github.com/apache/solr/pull/4375#discussion_r3217356571
##########
changelog/unreleased/PR#4375-language-model-refactoring.yml:
##########
@@ -0,0 +1,9 @@
+title: Refactor of the language-model module to accomodate new "Document Enrichment with LLMs" feature avoiding code duplication.
Review Comment:
mmm no need to specify 'new "Document Enrichment with LLMs" feature', it doesn't exist yet, no one except us have visibility on it, would just be confusing.
Replace that with ' new upcoming LLM oriented features'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org