You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hugegraph.apache.org by "vichayturen (via GitHub)" <gi...@apache.org> on 2024/04/18 03:34:05 UTC
[PR] Added local LLM API, changed the config displayed in the initial gradio demo, and added openai's apibase configuration. [incubator-hugegraph-ai]
vichayturen opened a new pull request, #41:
URL: https://github.com/apache/incubator-hugegraph-ai/pull/41
Added local LLM API, changed the config displayed in the initial gradio demo, and added openai's apibase configuration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] Added local LLM API, changed the config displayed in the initial gradio demo, and added openai's apibase configuration. [incubator-hugegraph-ai]
Posted by "liuxiaocs7 (via GitHub)" <gi...@apache.org>.
liuxiaocs7 commented on PR #41:
URL: https://github.com/apache/incubator-hugegraph-ai/pull/41#issuecomment-2073873700
Hi, @simon824, @imbajin, any other comments?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] Added local LLM API, changed the config displayed in the initial gradio demo, and added openai's apibase configuration. [incubator-hugegraph-ai]
Posted by "vichayturen (via GitHub)" <gi...@apache.org>.
vichayturen commented on code in PR #41:
URL: https://github.com/apache/incubator-hugegraph-ai/pull/41#discussion_r1570630154
##########
hugegraph-llm/llm_api/requirements.txt:
##########
@@ -0,0 +1,6 @@
+torch==2.1.2
+transformers==4.39.3
+fastapi==0.110.1
+accelerate==0.29.2
+chardet==5.2.0
+uvicorn==0.29.0
Review Comment:
Blank lines have been added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] Added local LLM API, changed the config displayed in the initial gradio demo, and added openai's apibase configuration. [incubator-hugegraph-ai]
Posted by "vichayturen (via GitHub)" <gi...@apache.org>.
vichayturen commented on code in PR #41:
URL: https://github.com/apache/incubator-hugegraph-ai/pull/41#discussion_r1571577635
##########
hugegraph-llm/llm_api/main.py:
##########
@@ -0,0 +1,170 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import gc
+import json
+from contextlib import asynccontextmanager
+from enum import Enum
+from typing import Any, Literal, List, Dict
+
+import torch
+import uvicorn
+from fastapi import FastAPI, HTTPException, status
+from fastapi.middleware.cors import CORSMiddleware
+from pydantic import BaseModel
+from transformers import AutoModelForCausalLM, AutoTokenizer, Qwen2ForCausalLM, \
+ Qwen2Tokenizer, GenerationConfig
+from typing_extensions import TypedDict
+
+
+class Message(TypedDict):
+ role: Literal["system", "user", "assistant", "tool", "function"]
+ content: str
+
+
+class ChatRequest(BaseModel):
+ messages: List[Message]
+
+
+class ChatResponse(BaseModel):
+ content: str
+
+
+class Role(Enum):
+ SYSTEM = "system"
+ USER = "user"
+ ASSISTANT = "assistant"
+ TOOL = "tool"
+ FUNCTION = "function"
+
+
+class QwenChatModel:
+ def __init__(self, model_name_or_path: str, device: str = "cuda",
+ generation_config: GenerationConfig = None):
+ self.model: Qwen2ForCausalLM = AutoModelForCausalLM.from_pretrained(
+ model_name_or_path,
+ torch_dtype="auto",
+ device_map=device
+ )
+ self.tokenizer: Qwen2Tokenizer = AutoTokenizer.from_pretrained(model_name_or_path)
+ self.device = torch.device(device)
+ self.generation_config = generation_config
+
+ @torch.inference_mode()
+ async def achat(self, messages: List[Message]):
+ text = self.tokenizer.apply_chat_template(
+ messages,
+ tokenize=False,
+ add_generation_prompt=True
+ )
+ model_inputs = self.tokenizer([text], return_tensors="pt").to(self.device)
+ generated_ids = self.model.generate(
+ model_inputs.input_ids,
+ max_new_tokens=512,
Review Comment:
It has been added to the command line arguments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] Added local LLM API, changed the config displayed in the initial gradio demo, and added openai's apibase configuration. [incubator-hugegraph-ai]
Posted by "vichayturen (via GitHub)" <gi...@apache.org>.
vichayturen commented on code in PR #41:
URL: https://github.com/apache/incubator-hugegraph-ai/pull/41#discussion_r1570635132
##########
hugegraph-llm/llm_api/README.md:
##########
@@ -0,0 +1,4 @@
+# local LLM Api
+
Review Comment:
Some usage examples have been given.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] Added local LLM API, changed the config displayed in the initial gradio demo, and added openai's apibase configuration. [incubator-hugegraph-ai]
Posted by "simon824 (via GitHub)" <gi...@apache.org>.
simon824 commented on code in PR #41:
URL: https://github.com/apache/incubator-hugegraph-ai/pull/41#discussion_r1570514060
##########
hugegraph-llm/llm_api/README.md:
##########
@@ -0,0 +1,4 @@
+# local LLM Api
+
Review Comment:
maybe we can provide some usage examples here
##########
hugegraph-llm/llm_api/requirements.txt:
##########
@@ -0,0 +1,6 @@
+torch==2.1.2
+transformers==4.39.3
+fastapi==0.110.1
+accelerate==0.29.2
+chardet==5.2.0
+uvicorn==0.29.0
Review Comment:
expect a blank line here
##########
hugegraph-llm/llm_api/requirements.txt:
##########
@@ -0,0 +1,6 @@
+torch==2.1.2
+transformers==4.39.3
+fastapi==0.110.1
+accelerate==0.29.2
+chardet==5.2.0
Review Comment:
can we remove [chardet](https://github.com/chardet/chardet), apache project cannot introduce dependencies under the LGPL license.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] Added local LLM API, changed the config displayed in the initial gradio demo, and added openai's apibase configuration. [incubator-hugegraph-ai]
Posted by "liuxiaocs7 (via GitHub)" <gi...@apache.org>.
liuxiaocs7 commented on code in PR #41:
URL: https://github.com/apache/incubator-hugegraph-ai/pull/41#discussion_r1570663139
##########
hugegraph-llm/llm_api/main.py:
##########
@@ -0,0 +1,170 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import gc
+import json
+from contextlib import asynccontextmanager
+from enum import Enum
+from typing import Any, Literal, List, Dict
+
+import torch
+import uvicorn
+from fastapi import FastAPI, HTTPException, status
+from fastapi.middleware.cors import CORSMiddleware
+from pydantic import BaseModel
+from transformers import AutoModelForCausalLM, AutoTokenizer, Qwen2ForCausalLM, \
+ Qwen2Tokenizer, GenerationConfig
+from typing_extensions import TypedDict
+
+
+class Message(TypedDict):
+ role: Literal["system", "user", "assistant", "tool", "function"]
+ content: str
+
+
+class ChatRequest(BaseModel):
+ messages: List[Message]
+
+
+class ChatResponse(BaseModel):
+ content: str
+
+
+class Role(Enum):
+ SYSTEM = "system"
+ USER = "user"
+ ASSISTANT = "assistant"
+ TOOL = "tool"
+ FUNCTION = "function"
+
+
+class QwenChatModel:
+ def __init__(self, model_name_or_path: str, device: str = "cuda",
+ generation_config: GenerationConfig = None):
+ self.model: Qwen2ForCausalLM = AutoModelForCausalLM.from_pretrained(
+ model_name_or_path,
+ torch_dtype="auto",
+ device_map=device
+ )
+ self.tokenizer: Qwen2Tokenizer = AutoTokenizer.from_pretrained(model_name_or_path)
+ self.device = torch.device(device)
+ self.generation_config = generation_config
+
+ @torch.inference_mode()
+ async def achat(self, messages: List[Message]):
+ text = self.tokenizer.apply_chat_template(
+ messages,
+ tokenize=False,
+ add_generation_prompt=True
+ )
+ model_inputs = self.tokenizer([text], return_tensors="pt").to(self.device)
+ generated_ids = self.model.generate(
+ model_inputs.input_ids,
+ max_new_tokens=512,
+ )
+ generated_ids = [
+ output_ids[len(input_ids):]
+ for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
+ ]
+ response = self.tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
+ return response
+
+
+def torch_gc() -> None:
+ r"""
+ Collects GPU memory.
+ """
+ gc.collect()
+ if torch.cuda.is_available():
+ torch.cuda.empty_cache()
+ torch.cuda.ipc_collect()
+
+
+@asynccontextmanager
+async def lifespan(app: "FastAPI"): # collects GPU memory
+ yield
+ torch_gc()
+
+
+def dictify(data: "BaseModel") -> Dict[str, Any]:
+ try: # pydantic v2
+ return data.model_dump(exclude_unset=True)
+ except AttributeError: # pydantic v1
+ return data.dict(exclude_unset=True)
+
+
+def jsonify(data: "BaseModel") -> str:
+ try: # pydantic v2
+ return json.dumps(data.model_dump(exclude_unset=True), ensure_ascii=False)
+ except AttributeError: # pydantic v1
+ return data.json(exclude_unset=True, ensure_ascii=False)
Review Comment:
What do these two methods do, they don't seem to be used
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] Added local LLM API, changed the config displayed in the initial gradio demo, and added openai's apibase configuration. [incubator-hugegraph-ai]
Posted by "liuxiaocs7 (via GitHub)" <gi...@apache.org>.
liuxiaocs7 commented on code in PR #41:
URL: https://github.com/apache/incubator-hugegraph-ai/pull/41#discussion_r1570641808
##########
.github/workflows/pylint.yml:
##########
@@ -24,6 +24,7 @@ jobs:
pip install pylint pytest
pip install -r ./hugegraph-llm/requirements.txt
pip install -r ./hugegraph-python-client/requirements.txt
+ pip install -r ./hugegraph-llm/llm_api/requirements.txt
Review Comment:
move line 27 under line 25
##########
hugegraph-llm/src/hugegraph_llm/utils/gradio_demo.py:
##########
@@ -146,110 +146,113 @@ def init_config(
return content
-with gr.Blocks() as hugegraph_llm:
- gr.Markdown(
- """# HugeGraph LLM Demo
- 1. Set up the HugeGraph server."""
- )
- with gr.Row():
- inp = [
- gr.Textbox(value="127.0.0.1", label="ip"),
- gr.Textbox(value="8080", label="port"),
- gr.Textbox(value="admin", label="user"),
- gr.Textbox(value="admin", label="pwd"),
- gr.Textbox(value="hugegraph", label="graph"),
- ]
- gr.Markdown("2. Set up the LLM.")
- with gr.Row():
- inp2 = [
- gr.Textbox(value="ernie", label="type"),
- gr.Textbox(value="", label="api_key"),
- gr.Textbox(value="", label="secret_key"),
- gr.Textbox(
- value="https://aip.baidubce.com/rpc/2.0/ai_custom/v1/wenxinworkshop/"
- "chat/completions_pro?access_token=",
- label="llm_url",
- ),
- gr.Textbox(value="wenxin", label="model_name"),
- gr.Textbox(value="4000", label="max_token"),
- ]
- with gr.Row():
- out = gr.Textbox(label="Output")
- btn = gr.Button("Initialize configs")
- btn.click(fn=init_config, inputs=inp + inp2, outputs=out) # pylint: disable=no-member
-
- gr.Markdown(
- """## 1. build knowledge graph
- - Text: The input text.
- - Schema: Accepts two types of text as below:
- - User-defined JSON format Schema.
- - Specify the name of the HugeGraph graph instance, and it will
- automatically extract the schema of the graph.
- - Disambiguate word sense: Whether to perform word sense disambiguation.
- - Commit to hugegraph: Whether to commit the constructed knowledge graph to the
- HugeGraph server.
- """
- )
- TEXT = (
- "Meet Sarah, a 30-year-old attorney, and her roommate, James, whom she's shared a home with"
- " since 2010. James, in his professional life, works as a journalist. Additionally, Sarah"
- " is the proud owner of the website www.sarahsplace.com, while James manages his own"
- " webpage, though the specific URL is not mentioned here. These two individuals, Sarah and"
- " James, have not only forged a strong personal bond as roommates but have also carved out"
- " their distinctive digital presence through their respective webpages, showcasing their"
- " varied interests and experiences."
- )
-
- SCHEMA = """{
- "vertices": [
- {"vertex_label": "person", "properties": ["name", "age", "occupation"]},
- {"vertex_label": "webpage", "properties": ["name", "url"]}
- ],
- "edges": [
- {
- "edge_label": "roommate",
- "source_vertex_label": "person",
- "target_vertex_label": "person",
- "properties": {}
- }
- ]
- }
- """
-
- with gr.Row():
- inp = [
- gr.Textbox(value=TEXT, label="Text"),
- gr.Textbox(value=SCHEMA, label="Schema"),
- gr.Textbox(value="false", label="Disambiguate word sense"),
- gr.Textbox(value="false", label="Commit to hugegraph"),
- ]
- with gr.Row():
- out = gr.Textbox(label="Output")
- btn = gr.Button("Build knowledge graph")
- btn.click(fn=build_kg, inputs=inp, outputs=out) # pylint: disable=no-member
-
- gr.Markdown("""## 2. Retrieval augmented generation by hugegraph""")
- with gr.Row():
- inp = gr.Textbox(value="Tell me about Al Pacino.", label="Question")
- with gr.Row():
- out = gr.Textbox(label="Answer")
- btn = gr.Button("Retrieval augmented generation")
- btn.click(fn=graph_rag, inputs=inp, outputs=out) # pylint: disable=no-member
-
- gr.Markdown("""## 3. Others """)
- with gr.Row():
- inp = []
- out = gr.Textbox(label="Output")
- btn = gr.Button("Initialize HugeGraph test data")
- btn.click(fn=init_hg_test_data, inputs=inp, outputs=out) # pylint: disable=no-member
-
- with gr.Row():
- inp = gr.Textbox(value="g.V().limit(10)", label="Gremlin query")
- out = gr.Textbox(label="Output")
- btn = gr.Button("Run gremlin query on HugeGraph")
- btn.click(fn=run_gremlin_query, inputs=inp, outputs=out) # pylint: disable=no-member
-
if __name__ == "__main__":
app = FastAPI()
+ initial_config = Config(section=Constants.HUGEGRAPH_CONFIG)
+ with gr.Blocks() as hugegraph_llm:
+ gr.Markdown(
+ """# HugeGraph LLM Demo
+ 1. Set up the HugeGraph server."""
Review Comment:
better to paste to show what changes made.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] Added local LLM API, changed the config displayed in the initial gradio demo, and added openai's apibase configuration. [incubator-hugegraph-ai]
Posted by "vichayturen (via GitHub)" <gi...@apache.org>.
vichayturen commented on code in PR #41:
URL: https://github.com/apache/incubator-hugegraph-ai/pull/41#discussion_r1570632371
##########
hugegraph-llm/llm_api/requirements.txt:
##########
@@ -0,0 +1,6 @@
+torch==2.1.2
+transformers==4.39.3
+fastapi==0.110.1
+accelerate==0.29.2
+chardet==5.2.0
Review Comment:
The chardet dependency has been replaced by the charset-normalizer of the MIT license.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] Added local LLM API, changed the config displayed in the initial gradio demo, and added openai's apibase configuration. [incubator-hugegraph-ai]
Posted by "vichayturen (via GitHub)" <gi...@apache.org>.
vichayturen commented on code in PR #41:
URL: https://github.com/apache/incubator-hugegraph-ai/pull/41#discussion_r1571563853
##########
hugegraph-llm/llm_api/main.py:
##########
@@ -0,0 +1,170 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import gc
+import json
+from contextlib import asynccontextmanager
+from enum import Enum
+from typing import Any, Literal, List, Dict
+
+import torch
+import uvicorn
+from fastapi import FastAPI, HTTPException, status
+from fastapi.middleware.cors import CORSMiddleware
+from pydantic import BaseModel
+from transformers import AutoModelForCausalLM, AutoTokenizer, Qwen2ForCausalLM, \
+ Qwen2Tokenizer, GenerationConfig
+from typing_extensions import TypedDict
+
+
+class Message(TypedDict):
+ role: Literal["system", "user", "assistant", "tool", "function"]
+ content: str
+
+
+class ChatRequest(BaseModel):
+ messages: List[Message]
+
+
+class ChatResponse(BaseModel):
+ content: str
+
+
+class Role(Enum):
+ SYSTEM = "system"
+ USER = "user"
+ ASSISTANT = "assistant"
+ TOOL = "tool"
+ FUNCTION = "function"
+
+
+class QwenChatModel:
+ def __init__(self, model_name_or_path: str, device: str = "cuda",
+ generation_config: GenerationConfig = None):
+ self.model: Qwen2ForCausalLM = AutoModelForCausalLM.from_pretrained(
+ model_name_or_path,
+ torch_dtype="auto",
+ device_map=device
+ )
+ self.tokenizer: Qwen2Tokenizer = AutoTokenizer.from_pretrained(model_name_or_path)
+ self.device = torch.device(device)
+ self.generation_config = generation_config
+
+ @torch.inference_mode()
+ async def achat(self, messages: List[Message]):
+ text = self.tokenizer.apply_chat_template(
+ messages,
+ tokenize=False,
+ add_generation_prompt=True
+ )
+ model_inputs = self.tokenizer([text], return_tensors="pt").to(self.device)
+ generated_ids = self.model.generate(
+ model_inputs.input_ids,
+ max_new_tokens=512,
+ )
+ generated_ids = [
+ output_ids[len(input_ids):]
+ for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
+ ]
+ response = self.tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
+ return response
+
+
+def torch_gc() -> None:
+ r"""
+ Collects GPU memory.
+ """
+ gc.collect()
+ if torch.cuda.is_available():
+ torch.cuda.empty_cache()
+ torch.cuda.ipc_collect()
+
+
+@asynccontextmanager
+async def lifespan(app: "FastAPI"): # collects GPU memory
+ yield
+ torch_gc()
+
+
+def dictify(data: "BaseModel") -> Dict[str, Any]:
+ try: # pydantic v2
+ return data.model_dump(exclude_unset=True)
+ except AttributeError: # pydantic v1
+ return data.dict(exclude_unset=True)
+
+
+def jsonify(data: "BaseModel") -> str:
+ try: # pydantic v2
+ return json.dumps(data.model_dump(exclude_unset=True), ensure_ascii=False)
+ except AttributeError: # pydantic v1
+ return data.json(exclude_unset=True, ensure_ascii=False)
Review Comment:
ok, they have been removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] feat: added local LLM API, changed the config displayed in the initial gradio demo, and added openai's apibase configuration. [incubator-hugegraph-ai]
Posted by "simon824 (via GitHub)" <gi...@apache.org>.
simon824 merged PR #41:
URL: https://github.com/apache/incubator-hugegraph-ai/pull/41
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] Added local LLM API, changed the config displayed in the initial gradio demo, and added openai's apibase configuration. [incubator-hugegraph-ai]
Posted by "liuxiaocs7 (via GitHub)" <gi...@apache.org>.
liuxiaocs7 commented on code in PR #41:
URL: https://github.com/apache/incubator-hugegraph-ai/pull/41#discussion_r1570649077
##########
hugegraph-llm/llm_api/main.py:
##########
@@ -0,0 +1,170 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import gc
+import json
+from contextlib import asynccontextmanager
+from enum import Enum
+from typing import Any, Literal, List, Dict
+
+import torch
+import uvicorn
+from fastapi import FastAPI, HTTPException, status
+from fastapi.middleware.cors import CORSMiddleware
+from pydantic import BaseModel
+from transformers import AutoModelForCausalLM, AutoTokenizer, Qwen2ForCausalLM, \
+ Qwen2Tokenizer, GenerationConfig
+from typing_extensions import TypedDict
+
+
+class Message(TypedDict):
+ role: Literal["system", "user", "assistant", "tool", "function"]
+ content: str
+
+
+class ChatRequest(BaseModel):
+ messages: List[Message]
+
+
+class ChatResponse(BaseModel):
+ content: str
+
+
+class Role(Enum):
+ SYSTEM = "system"
+ USER = "user"
+ ASSISTANT = "assistant"
+ TOOL = "tool"
+ FUNCTION = "function"
+
+
+class QwenChatModel:
+ def __init__(self, model_name_or_path: str, device: str = "cuda",
+ generation_config: GenerationConfig = None):
+ self.model: Qwen2ForCausalLM = AutoModelForCausalLM.from_pretrained(
+ model_name_or_path,
+ torch_dtype="auto",
+ device_map=device
+ )
+ self.tokenizer: Qwen2Tokenizer = AutoTokenizer.from_pretrained(model_name_or_path)
+ self.device = torch.device(device)
+ self.generation_config = generation_config
+
+ @torch.inference_mode()
+ async def achat(self, messages: List[Message]):
+ text = self.tokenizer.apply_chat_template(
+ messages,
+ tokenize=False,
+ add_generation_prompt=True
+ )
+ model_inputs = self.tokenizer([text], return_tensors="pt").to(self.device)
+ generated_ids = self.model.generate(
+ model_inputs.input_ids,
+ max_new_tokens=512,
Review Comment:
make this param configurable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] Added local LLM API, changed the config displayed in the initial gradio demo, and added openai's apibase configuration. [incubator-hugegraph-ai]
Posted by "vichayturen (via GitHub)" <gi...@apache.org>.
vichayturen commented on code in PR #41:
URL: https://github.com/apache/incubator-hugegraph-ai/pull/41#discussion_r1571563681
##########
hugegraph-llm/src/hugegraph_llm/utils/gradio_demo.py:
##########
@@ -146,110 +146,113 @@ def init_config(
return content
-with gr.Blocks() as hugegraph_llm:
- gr.Markdown(
- """# HugeGraph LLM Demo
- 1. Set up the HugeGraph server."""
- )
- with gr.Row():
- inp = [
- gr.Textbox(value="127.0.0.1", label="ip"),
- gr.Textbox(value="8080", label="port"),
- gr.Textbox(value="admin", label="user"),
- gr.Textbox(value="admin", label="pwd"),
- gr.Textbox(value="hugegraph", label="graph"),
- ]
- gr.Markdown("2. Set up the LLM.")
- with gr.Row():
- inp2 = [
- gr.Textbox(value="ernie", label="type"),
- gr.Textbox(value="", label="api_key"),
- gr.Textbox(value="", label="secret_key"),
- gr.Textbox(
- value="https://aip.baidubce.com/rpc/2.0/ai_custom/v1/wenxinworkshop/"
- "chat/completions_pro?access_token=",
- label="llm_url",
- ),
- gr.Textbox(value="wenxin", label="model_name"),
- gr.Textbox(value="4000", label="max_token"),
- ]
- with gr.Row():
- out = gr.Textbox(label="Output")
- btn = gr.Button("Initialize configs")
- btn.click(fn=init_config, inputs=inp + inp2, outputs=out) # pylint: disable=no-member
-
- gr.Markdown(
- """## 1. build knowledge graph
- - Text: The input text.
- - Schema: Accepts two types of text as below:
- - User-defined JSON format Schema.
- - Specify the name of the HugeGraph graph instance, and it will
- automatically extract the schema of the graph.
- - Disambiguate word sense: Whether to perform word sense disambiguation.
- - Commit to hugegraph: Whether to commit the constructed knowledge graph to the
- HugeGraph server.
- """
- )
- TEXT = (
- "Meet Sarah, a 30-year-old attorney, and her roommate, James, whom she's shared a home with"
- " since 2010. James, in his professional life, works as a journalist. Additionally, Sarah"
- " is the proud owner of the website www.sarahsplace.com, while James manages his own"
- " webpage, though the specific URL is not mentioned here. These two individuals, Sarah and"
- " James, have not only forged a strong personal bond as roommates but have also carved out"
- " their distinctive digital presence through their respective webpages, showcasing their"
- " varied interests and experiences."
- )
-
- SCHEMA = """{
- "vertices": [
- {"vertex_label": "person", "properties": ["name", "age", "occupation"]},
- {"vertex_label": "webpage", "properties": ["name", "url"]}
- ],
- "edges": [
- {
- "edge_label": "roommate",
- "source_vertex_label": "person",
- "target_vertex_label": "person",
- "properties": {}
- }
- ]
- }
- """
-
- with gr.Row():
- inp = [
- gr.Textbox(value=TEXT, label="Text"),
- gr.Textbox(value=SCHEMA, label="Schema"),
- gr.Textbox(value="false", label="Disambiguate word sense"),
- gr.Textbox(value="false", label="Commit to hugegraph"),
- ]
- with gr.Row():
- out = gr.Textbox(label="Output")
- btn = gr.Button("Build knowledge graph")
- btn.click(fn=build_kg, inputs=inp, outputs=out) # pylint: disable=no-member
-
- gr.Markdown("""## 2. Retrieval augmented generation by hugegraph""")
- with gr.Row():
- inp = gr.Textbox(value="Tell me about Al Pacino.", label="Question")
- with gr.Row():
- out = gr.Textbox(label="Answer")
- btn = gr.Button("Retrieval augmented generation")
- btn.click(fn=graph_rag, inputs=inp, outputs=out) # pylint: disable=no-member
-
- gr.Markdown("""## 3. Others """)
- with gr.Row():
- inp = []
- out = gr.Textbox(label="Output")
- btn = gr.Button("Initialize HugeGraph test data")
- btn.click(fn=init_hg_test_data, inputs=inp, outputs=out) # pylint: disable=no-member
-
- with gr.Row():
- inp = gr.Textbox(value="g.V().limit(10)", label="Gremlin query")
- out = gr.Textbox(label="Output")
- btn = gr.Button("Run gremlin query on HugeGraph")
- btn.click(fn=run_gremlin_query, inputs=inp, outputs=out) # pylint: disable=no-member
-
if __name__ == "__main__":
app = FastAPI()
+ initial_config = Config(section=Constants.HUGEGRAPH_CONFIG)
+ with gr.Blocks() as hugegraph_llm:
+ gr.Markdown(
+ """# HugeGraph LLM Demo
+ 1. Set up the HugeGraph server."""
Review Comment:
Here I replace the constant value in gr.Textbox with the value of reading config.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] Added local LLM API, changed the config displayed in the initial gradio demo, and added openai's apibase configuration. [incubator-hugegraph-ai]
Posted by "vichayturen (via GitHub)" <gi...@apache.org>.
vichayturen commented on code in PR #41:
URL: https://github.com/apache/incubator-hugegraph-ai/pull/41#discussion_r1571563136
##########
.github/workflows/pylint.yml:
##########
@@ -24,6 +24,7 @@ jobs:
pip install pylint pytest
pip install -r ./hugegraph-llm/requirements.txt
pip install -r ./hugegraph-python-client/requirements.txt
+ pip install -r ./hugegraph-llm/llm_api/requirements.txt
Review Comment:
ok，it has been done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] Added local LLM API, changed the config displayed in the initial gradio demo, and added openai's apibase configuration. [incubator-hugegraph-ai]
Posted by "simon824 (via GitHub)" <gi...@apache.org>.
simon824 commented on code in PR #41:
URL: https://github.com/apache/incubator-hugegraph-ai/pull/41#discussion_r1570511141
##########
hugegraph-llm/llm_api/requirements.txt:
##########
@@ -0,0 +1,6 @@
+torch==2.1.2
+transformers==4.39.3
+fastapi==0.110.1
+accelerate==0.29.2
+chardet==5.2.0
Review Comment:
need remove [chardet](https://github.com/chardet/chardet), cannot introduce dependencies under the `LGPL` license in apache projects.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org