You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hugegraph.apache.org by "weijinglin (via GitHub)" <gi...@apache.org> on 2025/09/30 03:11:51 UTC

[PR] [Refactor] Add Fixed Workflow Execution Engine: Flow, Node, and Scheduler Architecture [incubator-hugegraph-ai]

weijinglin opened a new pull request, #302:
URL: https://github.com/apache/incubator-hugegraph-ai/pull/302

   (no comment)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org


Re: [PR] [Refactor] Add Fixed Workflow Execution Engine: Flow, Node, and Scheduler Architecture [incubator-hugegraph-ai]

Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on code in PR #302:
URL: https://github.com/apache/incubator-hugegraph-ai/pull/302#discussion_r2447065027


##########
hugegraph-llm/README.md:
##########
@@ -248,6 +170,79 @@ The system supports both English and Chinese prompts. To switch languages:
 
 **LLM Provider Support**: This project uses [LiteLLM](https://docs.litellm.ai/docs/providers) for multi-provider LLM support.
 
+### Programmatic Examples (new workflow engine)
+
+If you previously used high-level classes like `RAGPipeline` or `KgBuilder`, the project now exposes stable flows through the `Scheduler` API. Use `SchedulerSingleton.get_instance().schedule_flow(...)` to invoke workflows programmatically. Below are concise, working examples that match the new architecture.
+
+1) RAG (graph-only) query example
+
+```python
+from hugegraph_llm.flows.scheduler import SchedulerSingleton
+
+scheduler = SchedulerSingleton.get_instance()
+res = scheduler.schedule_flow(
+  "rag_graph_only",
+  query="Tell me about Al Pacino.",
+  graph_only_answer=True,
+  vector_only_answer=False,
+  raw_answer=False,
+  gremlin_tmpl_num=-1,
+  gremlin_prompt=None,
+)
+
+print(res.get("graph_only_answer"))
+```
+
+2) RAG (vector-only) query example
+
+```python
+from hugegraph_llm.flows.scheduler import SchedulerSingleton
+
+scheduler = SchedulerSingleton.get_instance()
+res = scheduler.schedule_flow(
+  "rag_vector_only",
+  query="Summarize the career of Ada Lovelace.",
+  vector_only_answer=True,
+  vector_search=True
+)
+
+print(res.get("vector_only_answer"))
+```
+
+3) Text -> Gremlin (text2gremlin) example
+
+```python
+from hugegraph_llm.flows.scheduler import SchedulerSingleton
+
+scheduler = SchedulerSingleton.get_instance()
+response = scheduler.schedule_flow(
+  "text2gremlin",
+  "find people who worked with Alan Turing",
+  2,  # example_num
+  "hugegraph",  # schema_input (graph name or schema)
+  None,  # gremlin_prompt_input (optional)
+  ["template_gremlin", "raw_gremlin"],
+)
+
+print(response.get("template_gremlin"))
+```
+
+4) Build example index (used by text2gremlin examples)
+
+```python
+from hugegraph_llm.flows.scheduler import SchedulerSingleton
+
+examples = [{"id": "natural language query", "gremlin": "g.V().hasLabel('person').valueMap()"}]
+res = SchedulerSingleton.get_instance().schedule_flow("build_examples_index", examples)
+print(res)
+```
+

Review Comment:
   ⚠️ **Important: Migration guide incomplete**
   
   While the migration note is helpful, it lacks concrete examples:
   - No side-by-side comparison of old vs new API
   - Doesn't explain parameter mapping
   - Missing error handling differences
   
   **Recommendation**: Add concrete migration examples:
   
   ```python
   # OLD (deprecated)
   from hugegraph_llm.operators.graph_rag_task import RAGPipeline
   graph_rag = RAGPipeline()
   result = (graph_rag
       .extract_keywords(text="Tell me about Al Pacino.")
       .keywords_to_vid()
       .query_graphdb(max_deep=2, max_graph_items=30)
       .merge_dedup_rerank()
       .synthesize_answer()
       .run())
   
   # NEW (recommended)
   from hugegraph_llm.flows.scheduler import SchedulerSingleton
   scheduler = SchedulerSingleton.get_instance()
   result = scheduler.schedule_flow(
       "rag_graph_only",
       query="Tell me about Al Pacino.",
       graph_only_answer=True,
       max_graph_items=30
   )
   ```



##########
hugegraph-llm/README.md:
##########
@@ -248,6 +170,79 @@ The system supports both English and Chinese prompts. To switch languages:
 
 **LLM Provider Support**: This project uses [LiteLLM](https://docs.litellm.ai/docs/providers) for multi-provider LLM support.
 
+### Programmatic Examples (new workflow engine)
+
+If you previously used high-level classes like `RAGPipeline` or `KgBuilder`, the project now exposes stable flows through the `Scheduler` API. Use `SchedulerSingleton.get_instance().schedule_flow(...)` to invoke workflows programmatically. Below are concise, working examples that match the new architecture.
+
+1) RAG (graph-only) query example
+
+```python
+from hugegraph_llm.flows.scheduler import SchedulerSingleton
+
+scheduler = SchedulerSingleton.get_instance()
+res = scheduler.schedule_flow(

Review Comment:
   ⚠️ **Important: Hardcoded flow names reduce maintainability**
   
   Flow names like `"rag_graph_only"`, `"rag_vector_only"`, `"text2gremlin"` are hardcoded strings throughout the examples. This is error-prone and makes refactoring difficult.
   
   **Recommendation**: Define flow names as constants or enums:
   
   ```python
   # In flows/__init__.py or flows/constants.py
   class FlowName:
       RAG_GRAPH_ONLY = "rag_graph_only"
       RAG_VECTOR_ONLY = "rag_vector_only"
       TEXT2GREMLIN = "text2gremlin"
       BUILD_EXAMPLES_INDEX = "build_examples_index"
   
   # Usage in examples:
   from hugegraph_llm.flows import FlowName
   res = scheduler.schedule_flow(
       FlowName.RAG_GRAPH_ONLY,
       query="Tell me about Al Pacino.",
       ...
   )
   ```
   
   This provides IDE autocomplete, type safety, and easier maintenance.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org


Re: [PR] [Refactor] Add Fixed Workflow Execution Engine: Flow, Node, and Scheduler Architecture [incubator-hugegraph-ai]

Posted by "weijinglin (via GitHub)" <gi...@apache.org>.
weijinglin commented on code in PR #302:
URL: https://github.com/apache/incubator-hugegraph-ai/pull/302#discussion_r2444637351


##########
hugegraph-llm/pyproject.toml:
##########
@@ -85,3 +86,4 @@ allow-direct-references = true
 
 [tool.uv.sources]
 hugegraph-python-client = { workspace = true }
+pycgraph = { git = "https://github.com/ChunelFeng/CGraph.git", subdirectory = "python", rev = "main", marker = "sys_platform == 'linux'"  }

Review Comment:
   We can using this method to pull the dependency for CGraph in different platforms. We pull the source code & build dependency on linux platforms. We will pull the dependency from official library (like uv add PyCGraph) on Mac and Windows.  



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org


Re: [PR] [Refactor] Add Fixed Workflow Execution Engine: Flow, Node, and Scheduler Architecture [incubator-hugegraph-ai]

Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on code in PR #302:
URL: https://github.com/apache/incubator-hugegraph-ai/pull/302#discussion_r2447067940


##########
hugegraph-llm/src/hugegraph_llm/api/rag_api.py:
##########
@@ -102,15 +122,11 @@ def graph_rag_recall_api(req: GraphRAGRequest):
                 near_neighbor_first=req.near_neighbor_first,
                 custom_related_information=req.custom_priority_info,
                 gremlin_prompt=req.gremlin_prompt or prompt.gremlin_generate_prompt,
-                get_vertex_only=req.get_vertex_only
+                get_vertex_only=req.get_vertex_only,
             )
 
             if req.get_vertex_only:

Review Comment:
   ⚠️ **Important: Function extraction improves maintainability**
   
   Good refactoring to extract vertex details logic! However, consider a few enhancements:
   
   1. The function name `get_vertex_details` doesn't indicate it's a utility
   2. Error handling in this extracted function isn't clear from the diff
   
   **Recommendation**:
   - Ensure `get_vertex_details` has proper docstrings explaining parameters and return values
   - Add type hints for better IDE support
   - Handle cases where `match_vids` is empty or malformed



##########
hugegraph-llm/src/hugegraph_llm/api/models/rag_requests.py:
##########
@@ -24,10 +24,10 @@
 
 
 class GraphConfigRequest(BaseModel):
-    url: str = Query('127.0.0.1:8080', description="hugegraph client url.")
-    graph: str = Query('hugegraph', description="hugegraph client name.")
-    user: str = Query('', description="hugegraph client user.")
-    pwd: str = Query('', description="hugegraph client pwd.")
+    url: str = Query("127.0.0.1:8080", description="hugegraph client url.")

Review Comment:
   🧹 **Minor: Code formatting improvements applied**
   
   Good job using consistent formatting tools (likely black/ruff). The line length and quote style are now consistent across the codebase. This improves readability and reduces merge conflicts.
   
   Consider adding a pre-commit hook or CI check to enforce this formatting automatically for future contributions.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org


Re: [PR] [Refactor] Add Fixed Workflow Execution Engine: Flow, Node, and Scheduler Architecture [incubator-hugegraph-ai]

Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on code in PR #302:
URL: https://github.com/apache/incubator-hugegraph-ai/pull/302#discussion_r2444559237


##########
hugegraph-llm/pyproject.toml:
##########
@@ -17,7 +17,7 @@
 
 [project]
 name = "hugegraph-llm"
-version = "1.5.0"
+version = "1.7.0"

Review Comment:
   ⚠️ **Important: Version Bump Justification**
   
   The version jumps from 1.5.0 to 1.7.0 (skipping 1.6.0):
   ```python
   -version = "1.5.0"
   +version = "1.7.0"
   ```
   
   For a refactoring PR, this is an unusual version jump. Could you clarify:
   1. Why skip 1.6.0?
   2. Does this follow semantic versioning conventions?
   3. Are there breaking changes that warrant a minor version bump?
   
   Please document the versioning strategy for consistency.



##########
README.md:
##########
@@ -75,42 +75,6 @@ python -m hugegraph_llm.demo.rag_demo.app
 > [!NOTE]
 > Examples assume you've activated the virtual environment with `source .venv/bin/activate`
 
-#### GraphRAG - Question Answering
-
-```python
-from hugegraph_llm.operators.graph_rag_task import RAGPipeline
-
-# Initialize RAG pipeline
-graph_rag = RAGPipeline()
-
-# Ask questions about your graph
-result = (graph_rag
-    .extract_keywords(text="Tell me about Al Pacino.")
-    .keywords_to_vid()
-    .query_graphdb(max_deep=2, max_graph_items=30)
-    .merge_dedup_rerank()
-    .synthesize_answer()
-    .run())
-```
-
-#### Knowledge Graph Construction
-
-```python
-from hugegraph_llm.models.llms.init_llm import LLMs
-from hugegraph_llm.operators.kg_construction_task import KgBuilder
-
-# Build KG from text
-TEXT = "Your text content here..."
-builder = KgBuilder(LLMs().get_chat_llm())
-
-(builder
-    .import_schema(from_hugegraph="hugegraph")
-    .chunk_split(TEXT)
-    .extract_info(extract_type="property_graph")
-    .commit_to_hugegraph()
-    .run())
-```
-
 #### Graph Machine Learning

Review Comment:
   ⚠️ **Important: Documentation Removal Without Replacement**
   
   This PR removes important user-facing examples:
   - GraphRAG Question Answering example (lines 78-112)
   - Knowledge Graph Construction example
   
   These are valuable quick-start guides for users. If they're being removed due to the new workflow engine architecture:
   
   1. Please provide updated examples that work with the new architecture
   2. Or document why these patterns are deprecated
   3. Consider adding migration guide for existing users
   
   Simply removing documentation without replacement creates a poor user experience.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org


Re: [PR] [Refactor] Add Fixed Workflow Execution Engine: Flow, Node, and Scheduler Architecture [incubator-hugegraph-ai]

Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin merged PR #302:
URL: https://github.com/apache/incubator-hugegraph-ai/pull/302


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org


Re: [PR] [Refactor] Add Fixed Workflow Execution Engine: Flow, Node, and Scheduler Architecture [incubator-hugegraph-ai]

Posted by "weijinglin (via GitHub)" <gi...@apache.org>.
weijinglin commented on code in PR #302:
URL: https://github.com/apache/incubator-hugegraph-ai/pull/302#discussion_r2454079766


##########
hugegraph-llm/src/hugegraph_llm/flows/scheduler.py:
##########
@@ -0,0 +1,189 @@
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+
+import threading
+from typing import Dict, Any
+from PyCGraph import GPipeline, GPipelineManager
+from hugegraph_llm.flows import FlowName
+from hugegraph_llm.flows.build_vector_index import BuildVectorIndexFlow
+from hugegraph_llm.flows.common import BaseFlow
+from hugegraph_llm.flows.build_example_index import BuildExampleIndexFlow
+from hugegraph_llm.flows.graph_extract import GraphExtractFlow
+from hugegraph_llm.flows.import_graph_data import ImportGraphDataFlow
+from hugegraph_llm.flows.update_vid_embeddings import UpdateVidEmbeddingsFlow
+from hugegraph_llm.flows.get_graph_index_info import GetGraphIndexInfoFlow
+from hugegraph_llm.flows.build_schema import BuildSchemaFlow
+from hugegraph_llm.flows.prompt_generate import PromptGenerateFlow
+from hugegraph_llm.flows.rag_flow_raw import RAGRawFlow
+from hugegraph_llm.flows.rag_flow_vector_only import RAGVectorOnlyFlow
+from hugegraph_llm.flows.rag_flow_graph_only import RAGGraphOnlyFlow
+from hugegraph_llm.flows.rag_flow_graph_vector import RAGGraphVectorFlow
+from hugegraph_llm.state.ai_state import WkFlowInput
+from hugegraph_llm.utils.log import log
+from hugegraph_llm.flows.text2gremlin import Text2GremlinFlow
+
+
+class Scheduler:
+    pipeline_pool: Dict[str, Any]
+    max_pipeline: int
+
+    def __init__(self, max_pipeline: int = 10):
+        self.pipeline_pool = {}
+        # pipeline_pool act as a manager of GPipelineManager which used for pipeline management
+        self.pipeline_pool[FlowName.BUILD_VECTOR_INDEX] = {
+            "manager": GPipelineManager(),
+            "flow": BuildVectorIndexFlow(),
+        }
+        self.pipeline_pool[FlowName.GRAPH_EXTRACT] = {
+            "manager": GPipelineManager(),
+            "flow": GraphExtractFlow(),
+        }
+        self.pipeline_pool[FlowName.IMPORT_GRAPH_DATA] = {
+            "manager": GPipelineManager(),
+            "flow": ImportGraphDataFlow(),
+        }
+        self.pipeline_pool[FlowName.UPDATE_VID_EMBEDDINGS] = {
+            "manager": GPipelineManager(),
+            "flow": UpdateVidEmbeddingsFlow(),
+        }
+        self.pipeline_pool[FlowName.GET_GRAPH_INDEX_INFO] = {
+            "manager": GPipelineManager(),
+            "flow": GetGraphIndexInfoFlow(),
+        }
+        self.pipeline_pool[FlowName.BUILD_SCHEMA] = {
+            "manager": GPipelineManager(),
+            "flow": BuildSchemaFlow(),
+        }
+        self.pipeline_pool[FlowName.PROMPT_GENERATE] = {
+            "manager": GPipelineManager(),
+            "flow": PromptGenerateFlow(),
+        }
+        self.pipeline_pool[FlowName.TEXT2GREMLIN] = {
+            "manager": GPipelineManager(),
+            "flow": Text2GremlinFlow(),
+        }
+        # New split rag pipelines
+        self.pipeline_pool[FlowName.RAG_RAW] = {
+            "manager": GPipelineManager(),
+            "flow": RAGRawFlow(),
+        }
+        self.pipeline_pool[FlowName.RAG_VECTOR_ONLY] = {
+            "manager": GPipelineManager(),
+            "flow": RAGVectorOnlyFlow(),
+        }
+        self.pipeline_pool[FlowName.RAG_GRAPH_ONLY] = {
+            "manager": GPipelineManager(),
+            "flow": RAGGraphOnlyFlow(),
+        }
+        self.pipeline_pool[FlowName.RAG_GRAPH_VECTOR] = {
+            "manager": GPipelineManager(),
+            "flow": RAGGraphVectorFlow(),
+        }
+        self.pipeline_pool[FlowName.BUILD_EXAMPLES_INDEX] = {
+            "manager": GPipelineManager(),
+            "flow": BuildExampleIndexFlow(),
+        }
+        self.max_pipeline = max_pipeline
+
+    # TODO: Implement Agentic Workflow
+    def agentic_flow(self):
+        pass
+
+    def schedule_flow(self, flow_name: str, *args, **kwargs):
+        if flow_name not in self.pipeline_pool:
+            raise ValueError(f"Unsupported workflow {flow_name}")
+        manager: GPipelineManager = self.pipeline_pool[flow_name]["manager"]
+        flow: BaseFlow = self.pipeline_pool[flow_name]["flow"]
+        pipeline: GPipeline = manager.fetch()
+        if pipeline is None:
+            # call coresponding flow_func to create new workflow
+            pipeline = flow.build_flow(*args, **kwargs)
+            status = pipeline.init()
+            if status.isErr():
+                error_msg = f"Error in flow init: {status.getInfo()}"
+                log.error(error_msg)
+                raise RuntimeError(error_msg)
+            status = pipeline.run()
+            if status.isErr():
+                manager.add(pipeline)

Review Comment:
   pipeline包含两种状态:WkInput,WkState,这两种状态一个在pipeline.run()启动的时候重置,一个在pipeline.run()结束时重置,即使Pipeline.run返回错误码,这两个状态也会重置,这点确认过了



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org


Re: [PR] [Refactor] Add Fixed Workflow Execution Engine: Flow, Node, and Scheduler Architecture [incubator-hugegraph-ai]

Posted by "weijinglin (via GitHub)" <gi...@apache.org>.
weijinglin commented on code in PR #302:
URL: https://github.com/apache/incubator-hugegraph-ai/pull/302#discussion_r2454092008


##########
hugegraph-llm/src/hugegraph_llm/nodes/base_node.py:
##########
@@ -0,0 +1,83 @@
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+
+from typing import Any, Dict, Optional
+from PyCGraph import GNode, CStatus
+from hugegraph_llm.nodes.util import init_context
+from hugegraph_llm.state.ai_state import WkFlowInput, WkFlowState
+from hugegraph_llm.utils.log import log
+
+
+class BaseNode(GNode):
+    context: Optional[WkFlowState] = None
+    wk_input: Optional[WkFlowInput] = None
+
+    def init(self):
+        return init_context(self)
+
+    def node_init(self):
+        """
+        Node initialization method, can be overridden by subclasses.
+        Returns a CStatus object indicating whether initialization succeeded.
+        """
+        if self.wk_input is None or self.context is None:
+            return CStatus(-1, "wk_input or context not initialized")
+        if self.wk_input.data_json is not None:
+            self.context.assign_from_json(self.wk_input.data_json)
+            self.wk_input.data_json = None
+        return CStatus()
+
+    def run(self):
+        """
+        Main logic for node execution, can be overridden by subclasses.
+        Returns a CStatus object indicating whether execution succeeded.
+        """
+        sts = self.node_init()
+        if sts.isErr():
+            return sts
+        if self.context is None:

Review Comment:
   执行operator_schedule的时候并不会持锁,不会出现这个问题。只要处理好异常情况,不会出现死锁情况



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org


Re: [PR] [Refactor] Add Fixed Workflow Execution Engine: Flow, Node, and Scheduler Architecture [incubator-hugegraph-ai]

Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on code in PR #302:
URL: https://github.com/apache/incubator-hugegraph-ai/pull/302#discussion_r2444558314


##########
hugegraph-llm/pyproject.toml:
##########
@@ -85,3 +86,4 @@ allow-direct-references = true
 
 [tool.uv.sources]
 hugegraph-python-client = { workspace = true }
+pycgraph = { git = "https://github.com/ChunelFeng/CGraph.git", subdirectory = "python", rev = "main", marker = "sys_platform == 'linux'"  }

Review Comment:
   ‼️ **Critical: Platform-Specific Dependency Issue**
   
   The `pycgraph` dependency is restricted to Linux only:
   ```python
   pycgraph = { git = "https://github.com/ChunelFeng/CGraph.git", subdirectory = "python", rev = "main", marker = "sys_platform == 'linux'"  }
   ```
   
   This will break on macOS and Windows platforms. Consider:
   1. Making it cross-platform compatible
   2. Providing fallback implementation for other platforms
   3. Clearly documenting platform limitations in README
   
   This is a blocking issue for cross-platform users.



##########
hugegraph-llm/src/hugegraph_llm/api/rag_api.py:
##########
@@ -181,6 +204,13 @@ def text2gremlin_api(req: GremlinGenerateRequest):
         try:
             set_graph_config(req)
 
+            # Basic parameter validation: empty query => 400

Review Comment:
   ‼️ **Critical: Inconsistent Query Validation**
   
   Empty query validation is only added to `text2gremlin_api`:
   ```python
   if not req.query or not str(req.query).strip():
       raise HTTPException(
           status_code=status.HTTP_400_BAD_REQUEST,
           detail="Query must not be empty.",
       )
   ```
   
   However, other endpoints like `rag_answer_api` and `graph_rag_recall_api` also accept queries but lack this validation. 
   
   **Recommendation**: Apply consistent validation across all query-accepting endpoints to prevent potential errors downstream.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org


Re: [PR] [Refactor] Add Fixed Workflow Execution Engine: Flow, Node, and Scheduler Architecture [incubator-hugegraph-ai]

Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on code in PR #302:
URL: https://github.com/apache/incubator-hugegraph-ai/pull/302#discussion_r2447066591


##########
hugegraph-llm/README.md:
##########
@@ -248,6 +170,79 @@ The system supports both English and Chinese prompts. To switch languages:
 
 **LLM Provider Support**: This project uses [LiteLLM](https://docs.litellm.ai/docs/providers) for multi-provider LLM support.
 
+### Programmatic Examples (new workflow engine)
+
+If you previously used high-level classes like `RAGPipeline` or `KgBuilder`, the project now exposes stable flows through the `Scheduler` API. Use `SchedulerSingleton.get_instance().schedule_flow(...)` to invoke workflows programmatically. Below are concise, working examples that match the new architecture.
+
+1) RAG (graph-only) query example
+
+```python
+from hugegraph_llm.flows.scheduler import SchedulerSingleton
+
+scheduler = SchedulerSingleton.get_instance()
+res = scheduler.schedule_flow(
+  "rag_graph_only",
+  query="Tell me about Al Pacino.",
+  graph_only_answer=True,
+  vector_only_answer=False,
+  raw_answer=False,
+  gremlin_tmpl_num=-1,
+  gremlin_prompt=None,
+)
+
+print(res.get("graph_only_answer"))

Review Comment:
   🧹 **Minor: Example lacks error handling**
   
   The programmatic examples don't show how to handle errors from `schedule_flow`. Users might encounter runtime issues without guidance on exception handling.
   
   **Recommendation**: Add error handling example:
   
   ```python
   from hugegraph_llm.flows.scheduler import SchedulerSingleton
   
   scheduler = SchedulerSingleton.get_instance()
   try:
       res = scheduler.schedule_flow(
           "rag_graph_only",
           query="Tell me about Al Pacino.",
           graph_only_answer=True,
       )
       print(res.get("graph_only_answer"))
   except KeyError as e:
       print(f"Flow not found: {e}")
   except Exception as e:
       print(f"Execution error: {e}")
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org


Re: [PR] [Refactor] Add Fixed Workflow Execution Engine: Flow, Node, and Scheduler Architecture [incubator-hugegraph-ai]

Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on code in PR #302:
URL: https://github.com/apache/incubator-hugegraph-ai/pull/302#discussion_r2453912731


##########
hugegraph-llm/src/hugegraph_llm/flows/scheduler.py:
##########
@@ -0,0 +1,189 @@
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+
+import threading
+from typing import Dict, Any
+from PyCGraph import GPipeline, GPipelineManager
+from hugegraph_llm.flows import FlowName
+from hugegraph_llm.flows.build_vector_index import BuildVectorIndexFlow
+from hugegraph_llm.flows.common import BaseFlow
+from hugegraph_llm.flows.build_example_index import BuildExampleIndexFlow
+from hugegraph_llm.flows.graph_extract import GraphExtractFlow
+from hugegraph_llm.flows.import_graph_data import ImportGraphDataFlow
+from hugegraph_llm.flows.update_vid_embeddings import UpdateVidEmbeddingsFlow
+from hugegraph_llm.flows.get_graph_index_info import GetGraphIndexInfoFlow
+from hugegraph_llm.flows.build_schema import BuildSchemaFlow
+from hugegraph_llm.flows.prompt_generate import PromptGenerateFlow
+from hugegraph_llm.flows.rag_flow_raw import RAGRawFlow
+from hugegraph_llm.flows.rag_flow_vector_only import RAGVectorOnlyFlow
+from hugegraph_llm.flows.rag_flow_graph_only import RAGGraphOnlyFlow
+from hugegraph_llm.flows.rag_flow_graph_vector import RAGGraphVectorFlow
+from hugegraph_llm.state.ai_state import WkFlowInput
+from hugegraph_llm.utils.log import log
+from hugegraph_llm.flows.text2gremlin import Text2GremlinFlow
+
+
+class Scheduler:
+    pipeline_pool: Dict[str, Any]
+    max_pipeline: int
+
+    def __init__(self, max_pipeline: int = 10):
+        self.pipeline_pool = {}
+        # pipeline_pool act as a manager of GPipelineManager which used for pipeline management
+        self.pipeline_pool[FlowName.BUILD_VECTOR_INDEX] = {
+            "manager": GPipelineManager(),
+            "flow": BuildVectorIndexFlow(),
+        }
+        self.pipeline_pool[FlowName.GRAPH_EXTRACT] = {
+            "manager": GPipelineManager(),
+            "flow": GraphExtractFlow(),
+        }
+        self.pipeline_pool[FlowName.IMPORT_GRAPH_DATA] = {
+            "manager": GPipelineManager(),
+            "flow": ImportGraphDataFlow(),
+        }
+        self.pipeline_pool[FlowName.UPDATE_VID_EMBEDDINGS] = {
+            "manager": GPipelineManager(),
+            "flow": UpdateVidEmbeddingsFlow(),
+        }
+        self.pipeline_pool[FlowName.GET_GRAPH_INDEX_INFO] = {
+            "manager": GPipelineManager(),
+            "flow": GetGraphIndexInfoFlow(),
+        }
+        self.pipeline_pool[FlowName.BUILD_SCHEMA] = {
+            "manager": GPipelineManager(),
+            "flow": BuildSchemaFlow(),
+        }
+        self.pipeline_pool[FlowName.PROMPT_GENERATE] = {
+            "manager": GPipelineManager(),
+            "flow": PromptGenerateFlow(),
+        }
+        self.pipeline_pool[FlowName.TEXT2GREMLIN] = {
+            "manager": GPipelineManager(),
+            "flow": Text2GremlinFlow(),
+        }
+        # New split rag pipelines
+        self.pipeline_pool[FlowName.RAG_RAW] = {
+            "manager": GPipelineManager(),
+            "flow": RAGRawFlow(),
+        }
+        self.pipeline_pool[FlowName.RAG_VECTOR_ONLY] = {
+            "manager": GPipelineManager(),
+            "flow": RAGVectorOnlyFlow(),
+        }
+        self.pipeline_pool[FlowName.RAG_GRAPH_ONLY] = {
+            "manager": GPipelineManager(),
+            "flow": RAGGraphOnlyFlow(),
+        }
+        self.pipeline_pool[FlowName.RAG_GRAPH_VECTOR] = {
+            "manager": GPipelineManager(),
+            "flow": RAGGraphVectorFlow(),
+        }
+        self.pipeline_pool[FlowName.BUILD_EXAMPLES_INDEX] = {
+            "manager": GPipelineManager(),
+            "flow": BuildExampleIndexFlow(),
+        }
+        self.max_pipeline = max_pipeline
+
+    # TODO: Implement Agentic Workflow
+    def agentic_flow(self):
+        pass
+
+    def schedule_flow(self, flow_name: str, *args, **kwargs):
+        if flow_name not in self.pipeline_pool:
+            raise ValueError(f"Unsupported workflow {flow_name}")
+        manager: GPipelineManager = self.pipeline_pool[flow_name]["manager"]
+        flow: BaseFlow = self.pipeline_pool[flow_name]["flow"]
+        pipeline: GPipeline = manager.fetch()
+        if pipeline is None:
+            # call coresponding flow_func to create new workflow
+            pipeline = flow.build_flow(*args, **kwargs)
+            status = pipeline.init()
+            if status.isErr():
+                error_msg = f"Error in flow init: {status.getInfo()}"
+                log.error(error_msg)
+                raise RuntimeError(error_msg)
+            status = pipeline.run()
+            if status.isErr():
+                manager.add(pipeline)

Review Comment:
   ‼️ **Critical: Pipeline错误处理逻辑有误**
   
   ```python
   status = pipeline.run()
   if status.isErr():
       manager.add(pipeline)
       error_msg = f"Error in flow run: {status.getInfo()}"
       log.error(error_msg)
       raise RuntimeError(error_msg)
   ```
   
   **严重问题**: 失败的pipeline被添加回pool中重用!
   
   这会导致:
   1. 失败的pipeline状态未重置就被复用
   2. 下次获取到的可能是脏状态的pipeline
   3. 可能引发连锁失败或数据污染
   
   建议修复:
   ```python
   status = pipeline.run()
   if status.isErr():
       # 不要将失败的pipeline加回池中
       error_msg = f"Error in flow run: {status.getInfo()}"
       log.error(error_msg)
       raise RuntimeError(error_msg)
   # 只有成功的pipeline才能复用
   manager.add(pipeline)
   return pipeline.get_output()
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org


Re: [PR] [Refactor] Add Fixed Workflow Execution Engine: Flow, Node, and Scheduler Architecture [incubator-hugegraph-ai]

Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on code in PR #302:
URL: https://github.com/apache/incubator-hugegraph-ai/pull/302#discussion_r2447061036


##########
hugegraph-llm/pyproject.toml:
##########
@@ -85,3 +86,4 @@ allow-direct-references = true
 
 [tool.uv.sources]
 hugegraph-python-client = { workspace = true }
+pycgraph = { git = "https://github.com/ChunelFeng/CGraph.git", subdirectory = "python", rev = "248bfcfeddfa2bc23a1d585a3925c71189dba6cc"}

Review Comment:
   ‼️ **Critical: Missing dependency declaration**
   
   The project adds `pycgraph` dependency via git reference but doesn't document:
   1. Why this specific commit hash (`248bfcf`) is required
   2. What happens if this remote repository becomes unavailable
   3. Whether this should eventually be replaced with a versioned release
   
   **Recommendation**: Add a comment in `pyproject.toml` explaining the pinned commit, or better yet, work with upstream to create a tagged release.



##########
hugegraph-llm/pyproject.toml:
##########
@@ -17,7 +17,7 @@
 
 [project]
 name = "hugegraph-llm"
-version = "1.5.0"
+version = "1.7.0"

Review Comment:
   ⚠️ **Important: Version bump rationale missing**
   
   Version jumped from `1.5.0` to `1.7.0`, skipping `1.6.0`. This suggests either:
   - Major architectural changes (which aligns with this PR's scope)
   - Potential versioning mistake
   
   **Recommendation**: 
   - Document the version bump rationale in the PR description or CHANGELOG
   - If this is a breaking change (old RAGPipeline/KgBuilder removed), consider bumping to `2.0.0` per semantic versioning



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org


Re: [PR] [Refactor] Add Fixed Workflow Execution Engine: Flow, Node, and Scheduler Architecture [incubator-hugegraph-ai]

Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on code in PR #302:
URL: https://github.com/apache/incubator-hugegraph-ai/pull/302#discussion_r2453914535


##########
hugegraph-llm/src/hugegraph_llm/flows/scheduler.py:
##########
@@ -0,0 +1,189 @@
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+
+import threading
+from typing import Dict, Any
+from PyCGraph import GPipeline, GPipelineManager
+from hugegraph_llm.flows import FlowName
+from hugegraph_llm.flows.build_vector_index import BuildVectorIndexFlow
+from hugegraph_llm.flows.common import BaseFlow
+from hugegraph_llm.flows.build_example_index import BuildExampleIndexFlow
+from hugegraph_llm.flows.graph_extract import GraphExtractFlow
+from hugegraph_llm.flows.import_graph_data import ImportGraphDataFlow
+from hugegraph_llm.flows.update_vid_embeddings import UpdateVidEmbeddingsFlow
+from hugegraph_llm.flows.get_graph_index_info import GetGraphIndexInfoFlow
+from hugegraph_llm.flows.build_schema import BuildSchemaFlow
+from hugegraph_llm.flows.prompt_generate import PromptGenerateFlow
+from hugegraph_llm.flows.rag_flow_raw import RAGRawFlow
+from hugegraph_llm.flows.rag_flow_vector_only import RAGVectorOnlyFlow
+from hugegraph_llm.flows.rag_flow_graph_only import RAGGraphOnlyFlow
+from hugegraph_llm.flows.rag_flow_graph_vector import RAGGraphVectorFlow
+from hugegraph_llm.state.ai_state import WkFlowInput
+from hugegraph_llm.utils.log import log
+from hugegraph_llm.flows.text2gremlin import Text2GremlinFlow
+
+
+class Scheduler:
+    pipeline_pool: Dict[str, Any]
+    max_pipeline: int
+
+    def __init__(self, max_pipeline: int = 10):

Review Comment:
   ⚠️ **Important: 新增大量flow文件缺少单元测试覆盖**
   
   本PR新增了以下核心flow实现:
   - `RAGRawFlow`, `RAGVectorOnlyFlow`, `RAGGraphOnlyFlow`, `RAGGraphVectorFlow`
   - `Text2GremlinFlow`, `BuildExampleIndexFlow`
   - 以及多个其他flow类
   
   但从diff来看,缺少对应的测试文件。
   
   建议:
   1. 为每个flow类添加单元测试
   2. 测试应覆盖:
      - 正常执行路径
      - 错误处理路径
      - 边界条件(空输入、超大输入等)
      - Pipeline状态转换
   3. 集成测试验证flow之间的协作
   4. 考虑添加性能基准测试
   
   这对于如此大规模的架构重构至关重要!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org


Re: [PR] [Refactor] Add Fixed Workflow Execution Engine: Flow, Node, and Scheduler Architecture [incubator-hugegraph-ai]

Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on code in PR #302:
URL: https://github.com/apache/incubator-hugegraph-ai/pull/302#discussion_r2453915197


##########
hugegraph-llm/src/hugegraph_llm/nodes/base_node.py:
##########
@@ -0,0 +1,83 @@
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+
+from typing import Any, Dict, Optional
+from PyCGraph import GNode, CStatus
+from hugegraph_llm.nodes.util import init_context
+from hugegraph_llm.state.ai_state import WkFlowInput, WkFlowState
+from hugegraph_llm.utils.log import log
+
+
+class BaseNode(GNode):

Review Comment:
   🧹 **Minor: BaseNode类缺少类型注解和文档字符串**
   
   ```python
   class BaseNode(GNode):
       context: WkFlowState = None
       wk_input: WkFlowInput = None
   ```
   
   建议改进:
   ```python
   from typing import Optional
   
   class BaseNode(GNode):
       """
       工作流节点基类,提供上下文管理和操作调度功能。
       
       所有自定义节点应继承此类并实现 operator_schedule 方法。
       
       Attributes:
           context: 工作流共享状态
           wk_input: 工作流输入参数
       """
       context: Optional[WkFlowState] = None
       wk_input: Optional[WkFlowInput] = None
       
       def operator_schedule(self, data_json: dict) -> Optional[dict]:
           """
           子类必须实现的操作调度方法。
           
           Args:
               data_json: 上下文序列化的JSON数据
               
           Returns:
               处理结果的字典,或None表示无更新
               
           Raises:
               NotImplementedError: 子类未实现此方法
           """
           raise NotImplementedError("Subclasses must implement operator_schedule")
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org


Re: [PR] [Refactor] Add Fixed Workflow Execution Engine: Flow, Node, and Scheduler Architecture [incubator-hugegraph-ai]

Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on code in PR #302:
URL: https://github.com/apache/incubator-hugegraph-ai/pull/302#discussion_r2447061965


##########
hugegraph-llm/src/hugegraph_llm/api/rag_api.py:
##########
@@ -48,6 +50,13 @@ def rag_http_api(
     def rag_answer_api(req: RAGRequest):
         set_graph_config(req)
 
+        # Basic parameter validation: empty query => 400

Review Comment:
   ⚠️ **Important: Input validation added but inconsistent**
   
   Good addition of empty query validation, but the validation logic is duplicated across multiple endpoints (`rag_answer_api`, `graph_rag_recall_api`, `text2gremlin_api`).
   
   **Recommendation**: Extract to a shared validator function or Pydantic validator:
   
   ```python
   # In rag_requests.py
   from pydantic import field_validator
   
   class BaseQueryRequest(BaseModel):
       query: str
       
       @field_validator('query')
       @classmethod
       def validate_non_empty(cls, v):
           if not v or not v.strip():
               raise ValueError('Query must not be empty')
           return v.strip()
   ```
   
   This ensures consistent validation across all endpoints and reduces code duplication.



##########
README.md:
##########
@@ -75,42 +75,6 @@ python -m hugegraph_llm.demo.rag_demo.app
 > [!NOTE]
 > Examples assume you've activated the virtual environment with `source .venv/bin/activate`
 

Review Comment:
   ‼️ **Critical: Breaking change without deprecation period**
   
   The README removes code examples for `RAGPipeline` and `KgBuilder` without:
   1. Marking them as deprecated in the actual code
   2. Providing a deprecation timeline
   3. Ensuring backward compatibility
   
   **Impact**: Existing users will experience immediate breakage upon upgrading.
   
   **Recommendation**: 
   - Keep the old classes with `@deprecated` decorators that proxy to new scheduler
   - Add deprecation warnings
   - Document migration path in a MIGRATION.md file
   - Consider keeping old examples in a "Legacy" section with deprecation notice



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org


Re: [PR] [Refactor] Add Fixed Workflow Execution Engine: Flow, Node, and Scheduler Architecture [incubator-hugegraph-ai]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #302:
URL: https://github.com/apache/incubator-hugegraph-ai/pull/302#discussion_r2444546395


##########
hugegraph-llm/src/hugegraph_llm/nodes/base_node.py:
##########
@@ -0,0 +1,79 @@
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+
+from PyCGraph import GNode, CStatus
+from hugegraph_llm.nodes.util import init_context
+from hugegraph_llm.state.ai_state import WkFlowInput, WkFlowState
+from hugegraph_llm.utils.log import log
+
+
+class BaseNode(GNode):
+    context: WkFlowState = None
+    wk_input: WkFlowInput = None
+
+    def init(self):
+        return init_context(self)
+
+    def node_init(self):
+        """
+        Node initialization method, can be overridden by subclasses.
+        Returns a CStatus object indicating whether initialization succeeded.
+        """
+        if self.wk_input.data_json is not None:
+            self.context.assign_from_json(self.wk_input.data_json)
+            self.wk_input.data_json = None
+        return CStatus()
+
+    def run(self):
+        """
+        Main logic for node execution, can be overridden by subclasses.
+        Returns a CStatus object indicating whether execution succeeded.
+        """
+        sts = self.node_init()
+        if sts.isErr():
+            return sts
+        if self.context is None:
+            return CStatus(-1, "Context not initialized")
+        self.context.lock()
+        try:
+            data_json = self.context.to_json()
+        finally:
+            self.context.unlock()
+
+        try:
+            res = self.operator_schedule(data_json)
+        except Exception as exc:
+            import traceback
+
+            node_info = f"Node type: {type(self).__name__}, Node object: {self}"
+            err_msg = f"Node failed: {exc}\n{node_info}\n{traceback.format_exc()}"
+            return CStatus(-1, err_msg)
+
+        self.context.lock()
+        try:
+            if res is not None and isinstance(res, dict):
+                self.context.assign_from_json(res)
+            elif res is not None:
+                log.warning(f"operator_schedule returned non-dict type: {type(res)}")

Review Comment:
   [nitpick] Using f-string in logging call. Consider using log.warning("operator_schedule returned non-dict type: %s", type(res)) for better performance.
   ```suggestion
                   log.warning("operator_schedule returned non-dict type: %s", type(res))
   ```



##########
hugegraph-llm/src/hugegraph_llm/flows/scheduler.py:
##########
@@ -0,0 +1,192 @@
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+
+import threading
+from typing import Dict, Any
+from PyCGraph import GPipeline, GPipelineManager
+from hugegraph_llm.flows.build_vector_index import BuildVectorIndexFlow
+from hugegraph_llm.flows.common import BaseFlow
+from hugegraph_llm.flows.build_example_index import BuildExampleIndexFlow
+from hugegraph_llm.flows.graph_extract import GraphExtractFlow
+from hugegraph_llm.flows.import_graph_data import ImportGraphDataFlow
+from hugegraph_llm.flows.update_vid_embeddings import UpdateVidEmbeddingsFlow
+from hugegraph_llm.flows.get_graph_index_info import GetGraphIndexInfoFlow
+from hugegraph_llm.flows.build_schema import BuildSchemaFlow
+from hugegraph_llm.flows.prompt_generate import PromptGenerateFlow
+from hugegraph_llm.flows.rag_flow_raw import RAGRawFlow
+from hugegraph_llm.flows.rag_flow_vector_only import RAGVectorOnlyFlow
+from hugegraph_llm.flows.rag_flow_graph_only import RAGGraphOnlyFlow
+from hugegraph_llm.flows.rag_flow_graph_vector import RAGGraphVectorFlow
+from hugegraph_llm.state.ai_state import WkFlowInput
+from hugegraph_llm.utils.log import log
+from hugegraph_llm.flows.text2gremlin import Text2GremlinFlow
+
+
+class Scheduler:
+    pipeline_pool: Dict[str, Any]
+    max_pipeline: int
+
+    def __init__(self, max_pipeline: int = 10):
+        self.pipeline_pool = {}
+        # pipeline_pool act as a manager of GPipelineManager which used for pipeline management
+        self.pipeline_pool["build_vector_index"] = {
+            "manager": GPipelineManager(),
+            "flow": BuildVectorIndexFlow(),
+        }
+        self.pipeline_pool["graph_extract"] = {
+            "manager": GPipelineManager(),
+            "flow": GraphExtractFlow(),
+        }
+        self.pipeline_pool["import_graph_data"] = {
+            "manager": GPipelineManager(),
+            "flow": ImportGraphDataFlow(),
+        }
+        self.pipeline_pool["update_vid_embeddings"] = {
+            "manager": GPipelineManager(),
+            "flow": UpdateVidEmbeddingsFlow(),
+        }
+        self.pipeline_pool["get_graph_index_info"] = {
+            "manager": GPipelineManager(),
+            "flow": GetGraphIndexInfoFlow(),
+        }
+        self.pipeline_pool["build_schema"] = {
+            "manager": GPipelineManager(),
+            "flow": BuildSchemaFlow(),
+        }
+        self.pipeline_pool["prompt_generate"] = {
+            "manager": GPipelineManager(),
+            "flow": PromptGenerateFlow(),
+        }
+        self.pipeline_pool["text2gremlin"] = {
+            "manager": GPipelineManager(),
+            "flow": Text2GremlinFlow(),
+        }
+        # New split rag pipelines
+        self.pipeline_pool["rag_raw"] = {
+            "manager": GPipelineManager(),
+            "flow": RAGRawFlow(),
+        }
+        self.pipeline_pool["rag_vector_only"] = {
+            "manager": GPipelineManager(),
+            "flow": RAGVectorOnlyFlow(),
+        }
+        self.pipeline_pool["rag_graph_only"] = {
+            "manager": GPipelineManager(),
+            "flow": RAGGraphOnlyFlow(),
+        }
+        self.pipeline_pool["rag_graph_vector"] = {
+            "manager": GPipelineManager(),
+            "flow": RAGGraphVectorFlow(),
+        }
+        self.pipeline_pool["build_examples_index"] = {
+            "manager": GPipelineManager(),
+            "flow": BuildExampleIndexFlow(),
+        }
+        self.max_pipeline = max_pipeline
+
+    # TODO: Implement Agentic Workflow
+    def agentic_flow(self):
+        pass
+
+    def schedule_flow(self, flow_name: str, *args, **kwargs):
+        if flow_name not in self.pipeline_pool:
+            raise ValueError(f"Unsupported workflow {flow_name}")
+        manager: GPipelineManager = self.pipeline_pool[flow_name]["manager"]
+        flow: BaseFlow = self.pipeline_pool[flow_name]["flow"]
+        pipeline: GPipeline = manager.fetch()
+        if pipeline is None:
+            # call coresponding flow_func to create new workflow
+            pipeline = flow.build_flow(*args, **kwargs)
+            status = pipeline.init()
+            if status.isErr():
+                error_msg = f"Error in flow init: {status.getInfo()}"
+                log.error(error_msg)
+                raise RuntimeError(error_msg)
+            status = pipeline.run()
+            if status.isErr():
+                manager.add(pipeline)

Review Comment:
   Pipeline is added to manager before raising error, but this might be incorrect. Consider if failed pipelines should be added back to the pool.
   ```suggestion
   
   ```



##########
hugegraph-llm/src/hugegraph_llm/nodes/base_node.py:
##########
@@ -0,0 +1,79 @@
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+
+from PyCGraph import GNode, CStatus
+from hugegraph_llm.nodes.util import init_context
+from hugegraph_llm.state.ai_state import WkFlowInput, WkFlowState
+from hugegraph_llm.utils.log import log
+
+
+class BaseNode(GNode):
+    context: WkFlowState = None
+    wk_input: WkFlowInput = None
+
+    def init(self):
+        return init_context(self)
+
+    def node_init(self):
+        """
+        Node initialization method, can be overridden by subclasses.
+        Returns a CStatus object indicating whether initialization succeeded.
+        """
+        if self.wk_input.data_json is not None:
+            self.context.assign_from_json(self.wk_input.data_json)
+            self.wk_input.data_json = None
+        return CStatus()
+
+    def run(self):
+        """
+        Main logic for node execution, can be overridden by subclasses.
+        Returns a CStatus object indicating whether execution succeeded.
+        """
+        sts = self.node_init()
+        if sts.isErr():
+            return sts
+        if self.context is None:
+            return CStatus(-1, "Context not initialized")
+        self.context.lock()
+        try:
+            data_json = self.context.to_json()
+        finally:
+            self.context.unlock()
+
+        try:
+            res = self.operator_schedule(data_json)

Review Comment:
   The broad exception catch should be more specific. Consider catching specific exceptions that might occur during operator scheduling.
   ```suggestion
               res = self.operator_schedule(data_json)
           # Catch only Exception, not BaseException (e.g., KeyboardInterrupt, SystemExit)
           # This is to ensure that operator errors are handled gracefully, but critical signals propagate.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org


Re: [PR] [Refactor] Add Fixed Workflow Execution Engine: Flow, Node, and Scheduler Architecture [incubator-hugegraph-ai]

Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on code in PR #302:
URL: https://github.com/apache/incubator-hugegraph-ai/pull/302#discussion_r2453913954


##########
hugegraph-llm/src/hugegraph_llm/nodes/base_node.py:
##########
@@ -0,0 +1,83 @@
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+
+from typing import Any, Dict, Optional
+from PyCGraph import GNode, CStatus
+from hugegraph_llm.nodes.util import init_context
+from hugegraph_llm.state.ai_state import WkFlowInput, WkFlowState
+from hugegraph_llm.utils.log import log
+
+
+class BaseNode(GNode):
+    context: Optional[WkFlowState] = None
+    wk_input: Optional[WkFlowInput] = None
+
+    def init(self):
+        return init_context(self)
+
+    def node_init(self):
+        """
+        Node initialization method, can be overridden by subclasses.
+        Returns a CStatus object indicating whether initialization succeeded.
+        """
+        if self.wk_input is None or self.context is None:
+            return CStatus(-1, "wk_input or context not initialized")
+        if self.wk_input.data_json is not None:
+            self.context.assign_from_json(self.wk_input.data_json)
+            self.wk_input.data_json = None
+        return CStatus()
+
+    def run(self):
+        """
+        Main logic for node execution, can be overridden by subclasses.
+        Returns a CStatus object indicating whether execution succeeded.
+        """
+        sts = self.node_init()
+        if sts.isErr():
+            return sts
+        if self.context is None:

Review Comment:
   ⚠️ **Important: Context锁的异常安全性问题**
   
   ```python
   self.context.lock()
   try:
       data_json = self.context.to_json()
   finally:
       self.context.unlock()
   
   try:
       res = self.operator_schedule(data_json)
   except Exception as exc:
       # ...
       
   self.context.lock()
   try:
       if res is not None and isinstance(res, dict):
           self.context.assign_from_json(res)
       # ...
   finally:
       self.context.unlock()
   ```
   
   潜在问题:
   1. 如果 `operator_schedule` 抛出异常,第二个lock块不会执行,这是否符合预期?
   2. 多次lock/unlock增加了死锁风险
   3. 没有超时机制,lock可能永久阻塞
   
   建议:
   1. 使用 `contextlib.contextmanager` 封装lock逻辑
   2. 添加lock超时机制
   3. 考虑使用RLock如果需要可重入锁
   4. 明确文档说明异常情况下的锁释放策略



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org


Re: [PR] [Refactor] Add Fixed Workflow Execution Engine: Flow, Node, and Scheduler Architecture [incubator-hugegraph-ai]

Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on code in PR #302:
URL: https://github.com/apache/incubator-hugegraph-ai/pull/302#discussion_r2453911392


##########
hugegraph-llm/pyproject.toml:
##########
@@ -88,3 +89,6 @@ allow-direct-references = true
 
 [tool.uv.sources]
 hugegraph-python-client = { workspace = true }
+# We encountered a bug in PyCGraph's latest release version, so we're using a specific commit from the main branch (without the bug) as the project dependency.
+# TODO: Replace this command in the future when a new PyCGraph release version (after 3.1.2) is available.
+pycgraph = { git = "https://github.com/ChunelFeng/CGraph.git", subdirectory = "python", rev = "248bfcfeddfa2bc23a1d585a3925c71189dba6cc"}

Review Comment:
   ‼️ **Critical: pycgraph依赖锁定到特定commit存在风险**
   
   ```python
   pycgraph = { git = "https://github.com/ChunelFeng/CGraph.git", subdirectory = "python", rev = "248bfcfeddfa2bc23a1d585a3925c71189dba6cc"}
   ```
   
   问题:
   1. **可重现性问题**: 依赖特定commit而非版本标签,使得依赖管理不透明
   2. **安全风险**: 如果上游仓库历史被重写,这个commit可能消失
   3. **维护成本**: 未来升级困难,需要手动跟踪上游修复
   
   建议:
   1. 与PyCGraph团队沟通,尽快发布包含修复的正式版本(>3.1.2)
   2. 添加issue追踪链接,说明使用特定commit的原因
   3. 在代码注释中详细说明遇到的bug及其影响范围
   4. 设置定期检查机制,确保新版本发布后及时切换



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org