You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hugegraph.apache.org by "mkj-git (via GitHub)" <gi...@apache.org> on 2025/02/26 16:11:53 UTC
[D] Secondary index and full tet query in Hugegraph [incubator-hugegraph]
GitHub user mkj-git created a discussion: Secondary index and full tet query in Hugegraph
In Janusgraph secondary index used to be e.g elastic search, same way in hugegraph also has that concept?
```
graph.schema().indexLabel("personByAge")
.onV("person") // Apply to vertices of type "person"
.by("age") // Index the "age" property
.secondary() // Use a secondary index
.ifNotExist()
.create();
```
What is the role of `secondary()` here and how it works?
Also, full text search isnt supported yet, so what is the future plan for this?
```
// For full-text query (not supported now)
SEARCH(3, "search"),
```
GitHub link: https://github.com/apache/incubator-hugegraph/discussions/2739
----
This is an automatically sent email for dev@hugegraph.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hugegraph.apache.org
Re: [D] Secondary index and full tet query in Hugegraph [incubator-hugegraph]
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
GitHub user imbajin added a comment to the discussion: Secondary index and full tet query in Hugegraph
> In Janusgraph secondary index used to be e.g elastic search, same way in hugegraph also has that concept?
>
> ```
> graph.schema().indexLabel("personByAge")
> .onV("person") // Apply to vertices of type "person"
> .by("age") // Index the "age" property
> .secondary() // Use a secondary index
> .ifNotExist()
> .create();
> ```
>
> What is the role of `secondary()` here and how it works?
>
> Also, full text search isnt supported yet, so what is the future plan for this?
>
> ```
> // For full-text query (not supported now)
> SEARCH(3, "search"),
> ```
@dosu, you should refer https://github.com/apache/incubator-hugegraph/issues/884#issuecomment-595670139 to answer the question in English
> The hugegraph database index support is already relatively comprehensive. Graph database indexes generally include the following aspects:
- Graph Index / Edge Index: Mainly used to accelerate the retrieval of a vertex's associated edges, generally using adjacency lists or cross-linked lists, and can also use hash indexes. hugegraph uses adjacency lists.
- Supernode Index: Mainly used to accelerate the retrieval of some associated edges of supernodes, such as querying the call edges of the 12315 number in the most recent week. hugegraph is implemented through sortkey, which is equivalent to an enhanced version of the adjacency list.
- Secondary Index (ordinary attribute index): Mainly used to accelerate the retrieval of vertices based on attributes, or the retrieval of edges based on attributes. Databases generally use btree or hash implementations.
- Range Index (comparable attribute index): Mainly used to accelerate the retrieval of vertices or edges based on attribute ranges, such as people with `age>18`. Databases generally use btree implementations.
- Shard Index (attribute index that can be retrieved by prefix + comparison): Mainly used to accelerate the retrieval of vertices or edges based on attribute ranges, such as people with `city=Beijing && age>18`.
- Search Index (full-text searchable attribute index): Mainly used to accelerate the retrieval of vertices or edges based on any text segment or word in the attribute value, such as people with `city contains Bay`.
The implementation of attribute indexes in hugegraph depends on the storage and mainly relies on lsm-tree; the implementation principle of full-text indexes is to perform word segmentation on the attribute text and then store the index.
Reference document: [2.5 IndexLabel](https://hugegraph.apache.org/docs/clients/hugegraph-client/#25-indexlabel)
Perhaps we can use some automatic translation plugins or LLM to automatically summarize and translate some common historical questions
GitHub link: https://github.com/apache/incubator-hugegraph/discussions/2739#discussioncomment-12334934
----
This is an automatically sent email for dev@hugegraph.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hugegraph.apache.org
Re: [D] Secondary index and full tet query in Hugegraph [incubator-hugegraph]
Posted by "mkj-git (via GitHub)" <gi...@apache.org>.
GitHub user mkj-git deleted a comment on the discussion: Secondary index and full tet query in Hugegraph
@dosu is secondary index dependent on some external storage or search server? How it is stored?
GitHub link: https://github.com/apache/incubator-hugegraph/discussions/2739#discussioncomment-12328105
----
This is an automatically sent email for dev@hugegraph.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hugegraph.apache.org
Re: [D] Secondary index and full tet query in Hugegraph [incubator-hugegraph]
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
GitHub user imbajin edited a comment on the discussion: Secondary index and full tet query in Hugegraph
1. The reason why we changed from JG's cheat mode to a built-in tokenizer to achieve simple `full-text` indexing is because in production environment, ES will encounter many problems in a large number of point and edge situations(And it has increased a significant amount of write/query process burden and maintenance costs, also have the issue of data consistency)
- In addition, based on actual user feedback, the demand for complete full-text indexing is not very strong, and graph databases are not attempting to create a universal DB collection similar to `PostgreSQL`
- This requires a significant amount of manpower to maintain different backend/storage systems. Initially, this may seem like a good idea, but later iterations may encounter a lot of trouble. This is why JG's data structure has not changed much over the years, as it provides 2 heavy backend abstractions, making it difficult to optimize and adjust performance separately (otherwise the cost would be too high)
2. We do consider implementing a built-in vector index engine(like faiss-java or `lucene`) , but please note that it is not an external implementation of a separate **vector database** (Too heavy)
GitHub link: https://github.com/apache/incubator-hugegraph/discussions/2739#discussioncomment-12346626
----
This is an automatically sent email for dev@hugegraph.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hugegraph.apache.org
Re: [D] Secondary index and full tet query in Hugegraph [hugegraph]
Posted by "tomasciudad-indra (via GitHub)" <gi...@apache.org>.
GitHub user tomasciudad-indra added a comment to the discussion: Secondary index and full tet query in Hugegraph
Thanks @imbajin for your quick answer, I will review the PR of course. Thanks
GitHub link: https://github.com/apache/hugegraph/discussions/2739#discussioncomment-17236986
----
This is an automatically sent email for dev@hugegraph.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hugegraph.apache.org
Re: [D] Secondary index and full tet query in Hugegraph [incubator-hugegraph]
Posted by "mkj-git (via GitHub)" <gi...@apache.org>.
GitHub user mkj-git added a comment to the discussion: Secondary index and full tet query in Hugegraph
So what I think is that Hugegraph right now has very limited capability in terms of search. Since external search engine support isn't supported right now therefore that pain will be there till the time its not implemented.
I feel supporting external search engine like Elasticsearch, and vector DB will make this even more powerful.
GitHub link: https://github.com/apache/incubator-hugegraph/discussions/2739#discussioncomment-12340176
----
This is an automatically sent email for dev@hugegraph.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hugegraph.apache.org
Re: [D] Secondary index and full tet query in Hugegraph [incubator-hugegraph]
Posted by "mkj-git (via GitHub)" <gi...@apache.org>.
GitHub user mkj-git edited a discussion: Secondary index and full tet query in Hugegraph
In Janusgraph secondary index used to be e.g elastic search, same way in hugegraph also has that concept?
```
graph.schema().indexLabel("personByAge")
.onV("person") // Apply to vertices of type "person"
.by("age") // Index the "age" property
.secondary() // Use a secondary index
.ifNotExist()
.create();
```
What is the role of `secondary()` here and how it works?
Also, full text search isnt supported yet, so what is the future plan for this?
```
// For full-text query (not supported now)
SEARCH(3, "search"),
```
GitHub link: https://github.com/apache/incubator-hugegraph/discussions/2739
----
This is an automatically sent email for dev@hugegraph.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hugegraph.apache.org
Re: [D] Secondary index and full tet query in Hugegraph [hugegraph]
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
GitHub user imbajin added a comment to the discussion: Secondary index and full tet query in Hugegraph
@tomasciudad-indra Thanks for the follow-up. Short answer: we do not have a short-term plan to add a separate full-text indexing engine in HugeGraph; vector indexing is the current priority, but we are open to well-argued proposals from the community.
A practical direction would be an external module that writes full-text indexes asynchronously. That keeps the core HugeGraph codebase minimally invasive, allows the module to be maintained independently, and keeps the impact on OLTP latency low while still preserving eventual consistency.
Also, PR #2922 already contains the initial vector index integration design, so comments and ideas there are very welcome.
GitHub link: https://github.com/apache/hugegraph/discussions/2739#discussioncomment-17235855
----
This is an automatically sent email for dev@hugegraph.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hugegraph.apache.org
Re: [D] Secondary index and full tet query in Hugegraph [incubator-hugegraph]
Posted by "mkj-git (via GitHub)" <gi...@apache.org>.
GitHub user mkj-git added a comment to the discussion: Secondary index and full tet query in Hugegraph
@dosu is secondary index dependent on some external storage or search server? How it is stored?
GitHub link: https://github.com/apache/incubator-hugegraph/discussions/2739#discussioncomment-12328105
----
This is an automatically sent email for dev@hugegraph.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hugegraph.apache.org
Re: [D] Secondary index and full tet query in Hugegraph [incubator-hugegraph]
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
GitHub user imbajin added a comment to the discussion: Secondary index and full tet query in Hugegraph
1. The reason why we changed from JG's cheat mode to a built-in tokenizer to achieve simple `full-text` indexing is because in practical applications, ES will encounter many problems in a large number of point and edge situations(And it has increased a significant amount of process burden and maintenance costs)
- In addition, based on actual user feedback, the demand for complete full-text indexing is not very strong, and graph databases are not attempting to create a universal DB collection similar to `PostgreSQL`
- This requires a significant amount of manpower to maintain different backend/storage systems. Initially, this may seem like a good idea, but later iterations may encounter a lot of trouble. This is why JG's data structure has not changed much over the years, as it provides 2 heavy backend abstractions, making it difficult to optimize and adjust performance separately (otherwise the cost would be too high)
2. We do consider implementing a built-in vector index engine(like faiss-java or `lucene`) , but please note that it is not an external implementation of a separate **vector database** (Too heavy)
GitHub link: https://github.com/apache/incubator-hugegraph/discussions/2739#discussioncomment-12346626
----
This is an automatically sent email for dev@hugegraph.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hugegraph.apache.org
Re: [D] Secondary index and full tet query in Hugegraph [hugegraph]
Posted by "tomasciudad-indra (via GitHub)" <gi...@apache.org>.
GitHub user tomasciudad-indra added a comment to the discussion: Secondary index and full tet query in Hugegraph
Thanks for this discussion, we are in the same position now, we are looking to integrate hugegraph with elastic/opensearch for full text search, do you have in mind or in the roadmap?
GitHub link: https://github.com/apache/hugegraph/discussions/2739#discussioncomment-17235556
----
This is an automatically sent email for dev@hugegraph.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hugegraph.apache.org
Re: [D] Secondary index and full tet query in Hugegraph [incubator-hugegraph]
Posted by "dosubot[bot] (via GitHub)" <gi...@apache.org>.
GitHub user dosubot[bot] added a comment to the discussion: Secondary index and full tet query in Hugegraph
<!-- Answer -->
The `secondary()` method in HugeGraph is used to create a secondary index on a vertex or edge label. This type of index improves the performance of queries that filter vertices or edges based on property values. In your example, the `secondary()` method is used to create an index on the "age" property of the "person" vertex label, allowing for efficient querying by this property <sup>[[1]](https://github.com/apache/incubator-hugegraph/blob/master/hugegraph-server/hugegraph-example/src/main/java/org/apache/hugegraph/example/Example2.java)</sup>.
Regarding full-text search, it is currently not supported in HugeGraph, and I couldn't find specific information about future plans or a roadmap for supporting full-text search in the repository <sup>[[2]](https://github.com/apache/incubator-hugegraph/blob/master/hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/backend/tx/GraphIndexTransaction.java)</sup>.
<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
---
<sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/b626c470-c9d9-45d7-8ef7-66d5c3329633?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/b626c470-c9d9-45d7-8ef7-66d5c3329633?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/b626c470-c9d9-45d7-8ef7-66d5c3329633?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/b626c470-c9d9-45d7-8ef7-66d5c3329633?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/b626c470-c9d9-45d7-8ef7-66d5c3329633?feedback_type=hallucination) | [Bug Report](https://app.dosu.dev/response-feedback/b626c470-c9d9-45d7-8ef7-66d5c3329633?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/b626c470-c9d9-45d7-8ef7-66d5c3329633?feedback_type=other)</sup>
GitHub link: https://github.com/apache/incubator-hugegraph/discussions/2739#discussioncomment-12328059
----
This is an automatically sent email for dev@hugegraph.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hugegraph.apache.org
Re: [D] Secondary index and full tet query in Hugegraph [incubator-hugegraph]
Posted by "dosubot[bot] (via GitHub)" <gi...@apache.org>.
GitHub user dosubot[bot] added a comment to the discussion: Secondary index and full tet query in Hugegraph
<!-- Answer -->
The `secondary()` method in HugeGraph is used to create a secondary index on a vertex or edge label, which helps improve query performance by allowing efficient retrieval based on specific attributes. This type of index is typically implemented using B-tree or hash structures <sup>[[1]](https://github.com/apache/incubator-hugegraph-doc/blob/master/content/en/docs/clients/hugegraph-client.md)</sup>.
Regarding full-text search, HugeGraph already supports it through the `search()` method, which allows for full-text search capabilities <sup>[[1]](https://github.com/apache/incubator-hugegraph-doc/blob/master/content/en/docs/clients/hugegraph-client.md)</sup>.
<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
---
<sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/24af0f18-4f0c-416c-9352-6b939e2add9e?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/24af0f18-4f0c-416c-9352-6b939e2add9e?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/24af0f18-4f0c-416c-9352-6b939e2add9e?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/24af0f18-4f0c-416c-9352-6b939e2add9e?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/24af0f18-4f0c-416c-9352-6b939e2add9e?feedback_type=hallucination) | [Bug Report](https://app.dosu.dev/response-feedback/24af0f18-4f0c-416c-9352-6b939e2add9e?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/24af0f18-4f0c-416c-9352-6b939e2add9e?feedback_type=other)</sup>
GitHub link: https://github.com/apache/incubator-hugegraph/discussions/2739#discussioncomment-12334952
----
This is an automatically sent email for dev@hugegraph.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hugegraph.apache.org
Re: [D] Secondary index and full tet query in Hugegraph [incubator-hugegraph]
Posted by "dosubot[bot] (via GitHub)" <gi...@apache.org>.
GitHub user dosubot[bot] added a comment to the discussion: Secondary index and full tet query in Hugegraph
<!-- Answer -->
I couldn't find specific information on whether HugeGraph's secondary index is dependent on external storage or search servers. However, HugeGraph supports various storage backends for storing secondary indexes, including Cassandra, ScyllaDB, HBase, RocksDB, MySQL, Palo, PostgreSQL, and Hstore <sup>[[1]](https://github.com/apache/incubator-hugegraph/blob/master/hugegraph-server/hugegraph-dist/src/main/java/org/apache/hugegraph/dist/RegisterUtil.java)</sup>. Unfortunately, details on how secondary indexes are stored internally across these different storage backends are not available.
<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
---
<sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/32d87fb8-dacc-47e2-90e1-c55294b25aa6?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/32d87fb8-dacc-47e2-90e1-c55294b25aa6?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/32d87fb8-dacc-47e2-90e1-c55294b25aa6?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/32d87fb8-dacc-47e2-90e1-c55294b25aa6?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/32d87fb8-dacc-47e2-90e1-c55294b25aa6?feedback_type=hallucination) | [Bug Report](https://app.dosu.dev/response-feedback/32d87fb8-dacc-47e2-90e1-c55294b25aa6?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/32d87fb8-dacc-47e2-90e1-c55294b25aa6?feedback_type=other)</sup>
GitHub link: https://github.com/apache/incubator-hugegraph/discussions/2739#discussioncomment-12328122
----
This is an automatically sent email for dev@hugegraph.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hugegraph.apache.org
Re: [D] Secondary index and full tet query in Hugegraph [incubator-hugegraph]
Posted by "mkj-git (via GitHub)" <gi...@apache.org>.
GitHub user mkj-git added a comment to the discussion: Secondary index and full tet query in Hugegraph
I agree that having a inbuilt full-text search is better than external one, but is it scalable and does it support same level of query support as ES or any other search engine server?
Does hugegraph support any vector indexing right now?
GitHub link: https://github.com/apache/incubator-hugegraph/discussions/2739#discussioncomment-12358656
----
This is an automatically sent email for dev@hugegraph.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hugegraph.apache.org