You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hugegraph.apache.org by "Creeprime (via GitHub)" <gi...@apache.org> on 2025/06/04 05:12:23 UTC

[PR] feat(llm): support storing vector data for a graph instance by model type/name [incubator-hugegraph-ai]

Creeprime opened a new pull request, #265:
URL: https://github.com/apache/incubator-hugegraph-ai/pull/265

   ## Overview
   This PR implements independent data directory management when switching between different embedding models. Each embedding model will store its results in a separate directory, and during queries, the corresponding model’s embedding data will be automatically identified and loaded, ensuring that multiple models can coexist without data overwriting.
   
   ## Main Changes
   1. **`vector_index` Module Method Adjustments**  
      - Updated methods such as `from_index_file`, `to_index_file`, and `clean` to include a `model_name` parameter, enabling model-based independent storage paths.  
      - Backward compatibility is maintained: if `model_name` is not provided, the original behavior remains unchanged.
   
   2. **Index Operation Class Path Concatenation Refactoring**  
      - In classes like `BuildSemanticIndex`, modified directory concatenation so that the current `embedding_type` is automatically used to generate subdirectories without manual specification.
   
   3. **Embedding Class Name Unification**  
      - Unified the model name field across all embedding classes as `model_name`, and updated related invocation logic to ensure consistency with the index module.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org


Re: [PR] feat(llm): support storing vector data for a graph instance by model type/name [incubator-hugegraph-ai]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #265:
URL: https://github.com/apache/incubator-hugegraph-ai/pull/265#discussion_r2126678179


##########
hugegraph-llm/src/hugegraph_llm/utils/graph_index_utils.py:
##########
@@ -46,7 +48,7 @@ def get_graph_index_info():
 
 
 def clean_all_graph_index():
-    VectorIndex.clean(str(os.path.join(resource_path, huge_settings.graph_name, "graph_vids")))
+    VectorIndex.clean(str(os.path.join(resource_path, huge_settings.graph_name, "graph_vids", llm_settings.embedding_type)),Embeddings().get_embedding())

Review Comment:
   The call to VectorIndex.clean is passing the entire embedding object instead of its model name. Use 'getattr(Embeddings().get_embedding(), "model_name", null)' to extract the model name.
   ```suggestion
       VectorIndex.clean(str(os.path.join(resource_path, huge_settings.graph_name, "graph_vids", llm_settings.embedding_type)), getattr(Embeddings().get_embedding(), "model_name", None))
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org


Re: [PR] feat(llm): support storing vector data for a graph instance by model type/name [incubator-hugegraph-ai]

Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin merged PR #265:
URL: https://github.com/apache/incubator-hugegraph-ai/pull/265


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org


Re: [PR] feat(llm): support storing vector data for a graph instance by model type/name [incubator-hugegraph-ai]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #265:
URL: https://github.com/apache/incubator-hugegraph-ai/pull/265#discussion_r2257044111


##########
hugegraph-llm/src/hugegraph_llm/operators/index_op/gremlin_example_index_query.py:
##########
@@ -22,26 +22,31 @@
 
 import pandas as pd
 
-from hugegraph_llm.config import resource_path
-from hugegraph_llm.indices.vector_index import VectorIndex
+from hugegraph_llm.config import resource_path, llm_settings
+from hugegraph_llm.indices.vector_index import VectorIndex, INDEX_FILE_NAME, PROPERTIES_FILE_NAME
 from hugegraph_llm.models.embeddings.base import BaseEmbedding
 from hugegraph_llm.models.embeddings.init_embedding import Embeddings
+from hugegraph_llm.utils.embedding_utils import get_embeddings_parallel, get_filename_prefix, get_index_folder_name
 from hugegraph_llm.utils.log import log
-from hugegraph_llm.utils.embedding_utils import get_embeddings_parallel
 
 
 class GremlinExampleIndexQuery:
     def __init__(self, embedding: BaseEmbedding = None, num_examples: int = 1):
         self.embedding = embedding or Embeddings().get_embedding()
         self.num_examples = num_examples
-        self.index_dir = os.path.join(resource_path, "gremlin_examples")
+        self.folder_name = get_index_folder_name(llm_settings.graph_name, llm_settings.graph_space)

Review Comment:
   Using `llm_settings.graph_name` and `llm_settings.graph_space` instead of `huge_settings.graph_name` and `huge_settings.graph_space` is inconsistent with other files and may cause incorrect folder paths. Should use `huge_settings` like other index operations.
   ```suggestion
           self.folder_name = get_index_folder_name(huge_settings.graph_name, huge_settings.graph_space)
   ```



##########
hugegraph-llm/src/hugegraph_llm/operators/index_op/build_vector_index.py:
##########
@@ -20,18 +20,23 @@
 import os
 from typing import Dict, Any
 
-from hugegraph_llm.config import huge_settings, resource_path
+from hugegraph_llm.config import huge_settings, resource_path, llm_settings
 from hugegraph_llm.indices.vector_index import VectorIndex
 from hugegraph_llm.models.embeddings.base import BaseEmbedding
+from hugegraph_llm.utils.embedding_utils import get_embeddings_parallel, get_filename_prefix
 from hugegraph_llm.utils.log import log
-from hugegraph_llm.utils.embedding_utils import get_embeddings_parallel
 
 
 class BuildVectorIndex:
     def __init__(self, embedding: BaseEmbedding):
         self.embedding = embedding
-        self.index_dir = str(os.path.join(resource_path, huge_settings.graph_name, "chunks"))
-        self.vector_index = VectorIndex.from_index_file(self.index_dir)
+        self.folder_name = "_".join(filter(None, [huge_settings.graph_space, huge_settings.graph_name]))

Review Comment:
   This folder name generation logic is inconsistent with the `get_index_folder_name` utility function used elsewhere. Consider using the utility function for consistency: `self.folder_name = get_index_folder_name(huge_settings.graph_name, huge_settings.graph_space)`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org


Re: [PR] feat(llm): support storing vector data for a graph instance by model type/name [incubator-hugegraph-ai]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #265:
URL: https://github.com/apache/incubator-hugegraph-ai/pull/265#discussion_r2247728738


##########
hugegraph-llm/src/hugegraph_llm/operators/index_op/vector_index_query.py:
##########
@@ -19,18 +19,21 @@
 import os
 from typing import Dict, Any
 
-from hugegraph_llm.config import resource_path, huge_settings
+from hugegraph_llm.config import resource_path, huge_settings, llm_settings
 from hugegraph_llm.indices.vector_index import VectorIndex
 from hugegraph_llm.models.embeddings.base import BaseEmbedding
+from hugegraph_llm.utils.embedding_utils import get_filename_prefix, get_index_folder_name
 from hugegraph_llm.utils.log import log
 
 
 class VectorIndexQuery:
     def __init__(self, embedding: BaseEmbedding, topk: int = 3):
         self.embedding = embedding
         self.topk = topk
-        self.index_dir = str(os.path.join(resource_path, huge_settings.graph_name, "chunks"))
-        self.vector_index = VectorIndex.from_index_file(self.index_dir)
+        self.folder_name = get_index_folder_name(huge_settings.graph_name, huge_settings.graph_space)
+        self.index_dir = str(os.path.join(resource_path, self.folder_name, "chunks"))
+        self.filename_prefix = get_filename_prefix(llm_settings.embedding_type, getattr(embedding, "model_name", None))
+        self.vector_index = VectorIndex.from_index_file(self.index_dir,self.filename_prefix)

Review Comment:
   Missing space after comma. Should be `VectorIndex.from_index_file(self.index_dir, self.filename_prefix)` for consistent formatting.
   ```suggestion
           self.vector_index = VectorIndex.from_index_file(self.index_dir, self.filename_prefix)
   ```



##########
hugegraph-llm/src/hugegraph_llm/utils/embedding_utils.py:
##########
@@ -71,3 +71,21 @@ async def get_embeddings_parallel(embedding: BaseEmbedding, vids: list[str]) ->
             embeddings.extend(batch_embeddings)
 
     return embeddings
+
+
+def get_filename_prefix(embedding_type: str = None, model_name: str = None) -> str:
+    """Generate filename based on model name."""
+    if not model_name or model_name.strip() == "" or not embedding_type or embedding_type.strip() == "":

Review Comment:
   [nitpick] The condition can be simplified using truthiness checks. Consider: `if not (model_name and model_name.strip() and embedding_type and embedding_type.strip()):`
   ```suggestion
       if not (model_name and model_name.strip() and embedding_type and embedding_type.strip()):
   ```



##########
hugegraph-llm/src/hugegraph_llm/utils/embedding_utils.py:
##########
@@ -71,3 +71,21 @@ async def get_embeddings_parallel(embedding: BaseEmbedding, vids: list[str]) ->
             embeddings.extend(batch_embeddings)
 
     return embeddings
+
+
+def get_filename_prefix(embedding_type: str = None, model_name: str = None) -> str:
+    """Generate filename based on model name."""
+    if not model_name or model_name.strip() == "" or not embedding_type or embedding_type.strip() == "":
+        return ""
+    # Sanitize model_name to prevent path traversal or invalid filename chars
+    safe_embedding_type = embedding_type.replace("/", "_").replace("\\", "_").strip()
+    safe_model_name = model_name.replace("/", "_").replace("\\", "_").strip()
+    return f"{safe_embedding_type}_{safe_model_name}"
+
+
+def get_index_folder_name(graph_name: str, space_name: str = None) -> str:
+    if not space_name or space_name.strip() == "":

Review Comment:
   [nitpick] Inconsistent empty string checking pattern. Consider using the same pattern as line 78 or simplify to `if not (space_name and space_name.strip()):`
   ```suggestion
       if not (space_name and space_name.strip()):
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org


Re: [PR] feat(llm): support storing vector data for a graph instance by model type/name [incubator-hugegraph-ai]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #265:
URL: https://github.com/apache/incubator-hugegraph-ai/pull/265#discussion_r2257139549


##########
hugegraph-llm/src/hugegraph_llm/utils/embedding_utils.py:
##########
@@ -71,3 +71,21 @@ async def get_embeddings_parallel(embedding: BaseEmbedding, vids: list[str]) ->
             embeddings.extend(batch_embeddings)
 
     return embeddings
+
+
+def get_filename_prefix(embedding_type: str = None, model_name: str = None) -> str:
+    """Generate filename based on model name."""
+    if not (model_name and model_name.strip() and embedding_type and embedding_type.strip()):
+        return ""
+    # Sanitize model_name to prevent path traversal or invalid filename chars
+    safe_embedding_type = embedding_type.replace("/", "_").replace("\\", "_").strip()
+    safe_model_name = model_name.replace("/", "_").replace("\\", "_").strip()
+    return f"{safe_embedding_type}_{safe_model_name}"
+
+
+def get_index_folder_name(graph_name: str, space_name: str = None) -> str:
+    if not (space_name and space_name.strip()):

Review Comment:
   [nitpick] This condition pattern is repeated from the function above. Consider extracting a helper function like `is_valid_string(value)` to reduce code duplication.
   ```suggestion
       if not is_valid_string(space_name):
   ```



##########
hugegraph-llm/src/hugegraph_llm/utils/embedding_utils.py:
##########
@@ -71,3 +71,21 @@ async def get_embeddings_parallel(embedding: BaseEmbedding, vids: list[str]) ->
             embeddings.extend(batch_embeddings)
 
     return embeddings
+
+
+def get_filename_prefix(embedding_type: str = None, model_name: str = None) -> str:
+    """Generate filename based on model name."""
+    if not (model_name and model_name.strip() and embedding_type and embedding_type.strip()):

Review Comment:
   [nitpick] The condition combines multiple checks in a complex way. Consider splitting this into separate, more readable conditions or using a helper function to validate parameters.
   ```suggestion
   def is_valid_param(param: str) -> bool:
       return param is not None and bool(param.strip())
   
   def get_filename_prefix(embedding_type: str = None, model_name: str = None) -> str:
       """Generate filename based on model name."""
       if not (is_valid_param(model_name) and is_valid_param(embedding_type)):
   ```



##########
hugegraph-llm/src/hugegraph_llm/indices/vector_index.py:
##########
@@ -37,30 +37,57 @@ def __init__(self, embed_dim: int = 1024):
         self.properties = []
 
     @staticmethod
-    def from_index_file(dir_path: str, record_miss: bool = True) -> "VectorIndex":
-        index_file = os.path.join(dir_path, INDEX_FILE_NAME)
-        properties_file = os.path.join(dir_path, PROPERTIES_FILE_NAME)
+    def from_index_file(dir_path: str, filename_prefix: str = None, record_miss: bool = True) -> "VectorIndex":
+        """Load index from files, supporting model-specific filenames.
+
+        This method loads a Faiss index and its corresponding properties from a directory.
+        It handles model-specific filenames by using the get_model_filename utility.

Review Comment:
   The docstring mentions 'get_model_filename utility' but this function doesn't exist in the code. The actual implementation uses inline filename construction.
   ```suggestion
           It handles model-specific filenames by constructing them inline using f-strings.
   ```



##########
hugegraph-llm/src/hugegraph_llm/demo/rag_demo/text2gremlin_block.py:
##########
@@ -45,10 +47,27 @@ def store_schema(schema, question, gremlin_prompt):
 
 
 def build_example_vector_index(temp_file) -> dict:
+    folder_name = get_index_folder_name(huge_settings.graph_name, huge_settings.graph_space)
+    index_path = os.path.join(resource_path, folder_name, "gremlin_examples")
+    if not os.path.exists(index_path):
+        os.makedirs(index_path)
     if temp_file is None:
         full_path = os.path.join(resource_path, "demo", "text2gremlin.csv")
     else:
         full_path = temp_file.name
+        name, ext = os.path.splitext(temp_file.name)
+        timestamp = datetime.now().strftime("%Y%m%d%H%M%S")
+        _, file_name = os.path.split(f"{name}_{timestamp}{ext}")
+        log.info("Copying file to: %s", file_name)
+        target_file = os.path.join(resource_path, folder_name, "gremlin_examples", file_name)
+        try:
+            import shutil

Review Comment:
   Import statements should be placed at the top of the file, not inside functions. Move the shutil import to the top with other imports.
   ```suggestion
   
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org


Re: [PR] feat(llm): support storing vector data for a graph instance by model type/name [incubator-hugegraph-ai]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #265:
URL: https://github.com/apache/incubator-hugegraph-ai/pull/265#discussion_r2135275502


##########
hugegraph-llm/src/hugegraph_llm/indices/vector_index.py:
##########
@@ -37,12 +38,21 @@ def __init__(self, embed_dim: int = 1024):
         self.properties = []
 
     @staticmethod
-    def from_index_file(dir_path: str) -> "VectorIndex":
-        index_file = os.path.join(dir_path, INDEX_FILE_NAME)
-        properties_file = os.path.join(dir_path, PROPERTIES_FILE_NAME)
+    def from_index_file(dir_path: str, model_name: str = None) -> "VectorIndex":
+        """Load index from files,Supporting model-specific filenames"""

Review Comment:
   Typo in docstring: add a space after the comma (‘files, Supporting …’).
   ```suggestion
           """Load index from files, Supporting model-specific filenames"""
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org