You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gsoc@community.apache.org by "Imba Jin (Jira)" <ji...@apache.org> on 2024/03/16 16:10:00 UTC
[jira] [Updated] (GSOC-270) [GSoC][HugeGraph] Support Memory Management Module
[ https://issues.apache.org/jira/browse/GSOC-270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Imba Jin updated GSOC-270:
--------------------------
Description:
h2. Description
When the *JVM* *GC* performs a large amount of garbage collection, the latency of the request is often high, and the *response time* becomes uncontrollable. To reduce request latency and response time jitter, the *hugegraph-server* *graph* *query* *engine* has already used off-heap memory in most *OLTP* {*}algorithms{*}.
However, at present, hugegraph cannot control memory based on a single request Query, so a Query may exhaust the memory of the entire process and cause {*}OOM{*}, or even cause the service to be unable to respond to other requests. To solve this problem, we can implement a memory management module based on a single Query. Applicants will work with community developers to complete this task, and the specific implementation plan and division of labor/priority can be adjusted as needed.
Overall, it can be divided into 3 modules:
# {*}Memory management implementation module{*}. Implement the life cycle management of memory objects, memory capacity restrictions and other functions, and provide the Allocator interface (including allocation, release interface). This is a relatively independent module.
# Integrate the *Allocator module* into the HugeGraph context and provide a unified interface for memory transformation.
# Transform the places where a large amount of memory is occupied, and adapt to use Allocator for object allocation and release.
h2. Recommended Skills
# {*}Java/{*}{*}JVM{*} {*}Basics{*}: Deep understanding of Java's memory model, including the management and operation of heap memory and off-heap memory.
# {*}Java NIO{*}: Java NIO library provides an interface for operating off-heap memory, which needs to be mastered. (Familiarity with Netty or other memory management basic libraries is preferred)
# {*}Concurrent Programming{*}: Since memory management involves multi-thread concurrent operations, it is necessary to have knowledge of concurrent programming and multi-thread safety.
# {*}Data Structures{*}: Understand and apply appropriate data structures to manage memory, such as using queues, stacks, etc., to manage memory blocks.
# {*}Operating System{*}: Understand the memory management mechanism of the operating system in order to better understand and optimize Java's off-heap memory management.
h2. Task List
* Implement a *unified* {*}memory pool{*}, independently manage JVM off-heap memory, and adapt the memory allocation methods of various native collections, so that the memory mainly used by the algorithm comes from the unified memory pool, and it is returned to the memory pool after release.
* Each request corresponds to a unified memory pool, and the memory usage of a request can be controlled by counting the memory usage of a request.
* Complete related *unit tests UT* and basic documentation (better with the perf diff).
*PS:* More tech details could refer: [https://github.com/apache/incubator-hugegraph/wiki/MemoryManagement]
*- Difficulty:* Hard
*- Project size:* ~350 hour (full-time/large)
h2. *Potential Mentor*
* Jermy Li: jermy@apache.org (Apache HugeGraph PPMC)
* Imba Jin: jin@apache.org (Apache HugeGraph PPMC)
was:
h2. Description
When the *JVM* ** *GC* performs a large amount of garbage collection, the latency of the request is often high, and the *response time* becomes uncontrollable. To reduce request latency and response time jitter, the *hugegraph-server* *graph* ** *query* *engine* has already used off-heap memory in most *OLTP* \{*}algorithms{*}.
However, at present, hugegraph cannot control memory based on a single request Query, so a Query may exhaust the memory of the entire process and cause {*}OOM{*}, or even cause the service to be unable to respond to other requests. To solve this problem, we can implement a memory management module based on a single Query. Applicants will work with community developers to complete this task, and the specific implementation plan and division of labor/priority can be adjusted as needed.
Overall, it can be divided into 3 modules: #
{*}Memory management implementation module{*}. Implement the life cycle management of memory objects, memory capacity restrictions and other functions, and provide the Allocator interface (including allocation, release interface). This is a relatively independent module.
#
Integrate the *Allocator module* into the HugeGraph context and provide a unified interface for memory transformation.
#
Transform the places where a large amount of memory is occupied, and adapt to use Allocator for object allocation and release.
h2. Recommended Skills
#
{*}Java/{*}{*}JVM{*} \{*}Basics{*}: Deep understanding of Java's memory model, including the management and operation of heap memory and off-heap memory.
#
{*}Java NIO{*}: Java NIO library provides an interface for operating off-heap memory, which needs to be mastered. (Familiarity with Netty or other memory management basic libraries is preferred)
#
{*}Concurrent Programming{*}: Since memory management involves multi-thread concurrent operations, it is necessary to have knowledge of concurrent programming and multi-thread safety.
#
{*}Data Structures{*}: Understand and apply appropriate data structures to manage memory, such as using queues, stacks, etc., to manage memory blocks.
#
{*}Operating System{*}: Understand the memory management mechanism of the operating system in order to better understand and optimize Java's off-heap memory management.
h2. Task List
*
Implement a *unified* {*}memory pool{*}, independently manage JVM off-heap memory, and adapt the memory allocation methods of various native collections, so that the memory mainly used by the algorithm comes from the unified memory pool, and it is returned to the memory pool after release.
*
Each request corresponds to a unified memory pool, and the memory usage of a request can be controlled by counting the memory usage of a request.
*
Complete related *unit tests UT* and basic documentation (better with the perf diff).
*PS:* More tech details could refer: https://github.com/apache/incubator-hugegraph/wiki/MemoryManagement *
*Difficulty:* Hard
*
*Project size:* ~350 hour (full-time/large)
h2. *Potential Mentor*
Jermy Li: jermy@apache.org (Apache HugeGraph PPMC)
Imba Jin: jin@apache.org (Apache HugeGraph PPMC)
> [GSoC][HugeGraph] Support Memory Management Module
> --------------------------------------------------
>
> Key: GSOC-270
> URL: https://issues.apache.org/jira/browse/GSOC-270
> Project: Comdev GSOC
> Issue Type: New Feature
> Reporter: Imba Jin
> Priority: Major
> Labels: full-time, gsoc2024, hugegraph, mentor
>
> h2. Description
> When the *JVM* *GC* performs a large amount of garbage collection, the latency of the request is often high, and the *response time* becomes uncontrollable. To reduce request latency and response time jitter, the *hugegraph-server* *graph* *query* *engine* has already used off-heap memory in most *OLTP* {*}algorithms{*}.
>
> However, at present, hugegraph cannot control memory based on a single request Query, so a Query may exhaust the memory of the entire process and cause {*}OOM{*}, or even cause the service to be unable to respond to other requests. To solve this problem, we can implement a memory management module based on a single Query. Applicants will work with community developers to complete this task, and the specific implementation plan and division of labor/priority can be adjusted as needed.
>
> Overall, it can be divided into 3 modules:
> # {*}Memory management implementation module{*}. Implement the life cycle management of memory objects, memory capacity restrictions and other functions, and provide the Allocator interface (including allocation, release interface). This is a relatively independent module.
> # Integrate the *Allocator module* into the HugeGraph context and provide a unified interface for memory transformation.
> # Transform the places where a large amount of memory is occupied, and adapt to use Allocator for object allocation and release.
> h2. Recommended Skills
> # {*}Java/{*}{*}JVM{*} {*}Basics{*}: Deep understanding of Java's memory model, including the management and operation of heap memory and off-heap memory.
> # {*}Java NIO{*}: Java NIO library provides an interface for operating off-heap memory, which needs to be mastered. (Familiarity with Netty or other memory management basic libraries is preferred)
> # {*}Concurrent Programming{*}: Since memory management involves multi-thread concurrent operations, it is necessary to have knowledge of concurrent programming and multi-thread safety.
> # {*}Data Structures{*}: Understand and apply appropriate data structures to manage memory, such as using queues, stacks, etc., to manage memory blocks.
> # {*}Operating System{*}: Understand the memory management mechanism of the operating system in order to better understand and optimize Java's off-heap memory management.
> h2. Task List
> * Implement a *unified* {*}memory pool{*}, independently manage JVM off-heap memory, and adapt the memory allocation methods of various native collections, so that the memory mainly used by the algorithm comes from the unified memory pool, and it is returned to the memory pool after release.
> * Each request corresponds to a unified memory pool, and the memory usage of a request can be controlled by counting the memory usage of a request.
> * Complete related *unit tests UT* and basic documentation (better with the perf diff).
>
> *PS:* More tech details could refer: [https://github.com/apache/incubator-hugegraph/wiki/MemoryManagement]
>
> *- Difficulty:* Hard
> *- Project size:* ~350 hour (full-time/large)
> h2. *Potential Mentor*
> * Jermy Li: jermy@apache.org (Apache HugeGraph PPMC)
> * Imba Jin: jin@apache.org (Apache HugeGraph PPMC)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: gsoc-unsubscribe@community.apache.org
For additional commands, e-mail: gsoc-help@community.apache.org