You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudberry.apache.org by "fanfuxiaoran (via GitHub)" <gi...@apache.org> on 2025/05/06 06:56:21 UTC

[PR] [WIP] Add perfmon [cloudberry]

fanfuxiaoran opened a new pull request, #1085:
URL: https://github.com/apache/cloudberry/pull/1085

   <!-- Thank you for your contribution to Apache Cloudberry (Incubating)! -->
   
   Fixes #ISSUE_Number
   
   ### What does this PR do?
   <!-- Brief overview of the changes, including any major features or fixes -->
   
   This PR introduces the ​​perfmon​​ extension, designed to monitor the Cloudberry Database.
   
   #### Key Features:
   - ​**Cluster-wide Metrics Collection​​**: Gathers performance metrics from each machine in the database cluster.
   - **Real-time Query Monitoring​​:**
      - Displays active queries with detailed execution plans.
      - Tracks progress of each query plan node (similar to EXPLAIN ANALYZE for running queries).
      - Captures performance metrics (CPU, memory, spill files) across multiple segments.
   - ​​**Query History​​**: Provides a historical record of executed queries.
   
   #### Components:
   - **gpmmon**​​ (Shared Library)
   Added to shared_preload_libraries for background operation.
   Runs as a backgroud worker process to collect and store metrics in the gpperfmon database.
   - **gpmon**​​ (Shared Library)
   Added to shared_preload_libraries for query monitoring.
   Implements hooks to capture runtime query data.
   Provides the pg_query_state() function to inspect live query execution status.
   - **gpsmon**​​ (Binary)
   Deployed per-instance to collect system-level metrics (CPU, memory, etc.).
   ### Type of Change
   - [ ] Bug fix (non-breaking change)
   - [x] New feature (non-breaking change)
   - [ ] Breaking change (fix or feature with breaking changes)
   - [ ] Documentation update
   
   ### Breaking Changes
   <!-- Remove if not applicable. If yes, explain impact and migration path -->
   
   ### Test Plan
   <!-- How did you test these changes? -->
   - [ ] Unit tests added/updated
   - [x] Integration tests added/updated
   - [x] Passed `make installcheck`
   - [ ] Passed `make -C src/test installcheck-cbdb-parallel`
   
   ### Impact
   <!-- Remove sections that don't apply -->
   **Performance:**
   <!-- Any performance implications? -->
   
   **User-facing changes:**
   <!-- Any changes visible to users? -->
   
   **Dependencies:**
   <!-- New dependencies or version changes? -->
   
   ### Checklist
   - [ ] Followed [contribution guide](https://cloudberry.apache.org/contribute/code)
   - [ ] Added/updated documentation
   - [ ] Reviewed code for security implications
   - [ ] Requested review from [cloudberry committers](https://github.com/orgs/apache/teams/cloudberry-committers)
   
   ### Additional Context
   <!-- Any other information that would help reviewers? Remove if none -->
   
   ### CI Skip Instructions
   <!--
   To skip CI builds, add the appropriate CI skip identifier to your PR title.
   The identifier must:
   - Be in square brackets []
   - Include the word "ci" and either "skip" or "no"
   - Only use for documentation-only changes or when absolutely necessary
   -->
   
   ---
   <!-- Join our community:
   - Mailing list: [dev@cloudberry.apache.org](https://lists.apache.org/list.html?dev@cloudberry.apache.org) (subscribe: dev-subscribe@cloudberry.apache.org)
   - Discussions: https://github.com/apache/cloudberry/discussions -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


Re: [PR] Introduce extension perfmon to monitor database cluster [cloudberry]

Posted by "tuhaihe (via GitHub)" <gi...@apache.org>.
tuhaihe commented on PR #1085:
URL: https://github.com/apache/cloudberry/pull/1085#issuecomment-4631223986

   > However there is one thing I really look forward in this PR: pg_query_state extension support. I believe if it's working fine for cbdb, we can borrow it from this PR and merge separately.
   > We would love to have query progress tracking in cloudberry. If you're short of hands on this project, I guess we can try to take it over.
   
   @my-ship-it Would love to hear your thoughts!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


Re: [PR] Introduce extension perfmon to monitor database cluster [cloudberry]

Posted by "my-ship-it (via GitHub)" <gi...@apache.org>.
my-ship-it commented on PR #1085:
URL: https://github.com/apache/cloudberry/pull/1085#issuecomment-4631340652

   > > However there is one thing I really look forward in this PR: pg_query_state extension support. I believe if it's working fine for cbdb, we can borrow it from this PR and merge separately.
   > > We would love to have query progress tracking in cloudberry. If you're short of hands on this project, I guess we can try to take it over.
   > 
   > @my-ship-it Would love to hear your thoughts!
   
   Thanks @Smyatkin-Maxim for the input. I agree with this direction:
   
   - The full gpperfmon stack overlaps with gpcc and is probably not worth maintaining separately,
   so we can keep this PR closed as @tuhaihe suggested.
   - The pg_query_state part is indeed the most valuable piece here — live query progress tracking
   is something Cloudberry users have been asking for, and it makes sense as a standalone extension
   independent of the gpperfmon/gpcc discussion.
   
   +1 on extracting it into a separate PR. @Smyatkin-Maxim if you'd like to take it over, that
   would be very welcome — feel free to borrow the relevant code from this PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


Re: [PR] Introduce extension perfmon to monitor database cluster [cloudberry]

Posted by "tuhaihe (via GitHub)" <gi...@apache.org>.
tuhaihe commented on PR #1085:
URL: https://github.com/apache/cloudberry/pull/1085#issuecomment-6183802050

   I believe #1934 took over some of the work from this project. I will be closing this project. If anything needs to be done, we can reopen it or create a new project for it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


Re: [PR] Introduce extension perfmon to monitor database cluster [cloudberry]

Posted by "Smyatkin-Maxim (via GitHub)" <gi...@apache.org>.
Smyatkin-Maxim commented on PR #1085:
URL: https://github.com/apache/cloudberry/pull/1085#issuecomment-4630680589

   @tuhaine @fanfuxiaoran I'm more or less sure that gpperfmon is inferior to gpcc with one exception: simplicity. As soon as we provide installation and usage guidlines for gpcc, gpperfmon probably gonna be unnecessary.
   
   However there is one thing I really look forward in this PR: pg_query_state extension support. I believe if it's working fine for cbdb, we can introduce borrow it from current PR and merge separately.
   We would love to have query progress tracking in cloudberry. If you're short of hands on this project, I guess we can try to  take it over.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


Re: [PR] Introduce extension perfmon to monitor database cluster [cloudberry]

Posted by "Smyatkin-Maxim (via GitHub)" <gi...@apache.org>.
Smyatkin-Maxim commented on PR #1085:
URL: https://github.com/apache/cloudberry/pull/1085#issuecomment-4682805697

   Cool, we will be looking into it shortly. Probably @roaldm153 will be taking it over


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


Re: [PR] Introduce extension perfmon to monitor database cluster [cloudberry]

Posted by "tuhaihe (via GitHub)" <gi...@apache.org>.
tuhaihe commented on PR #1085:
URL: https://github.com/apache/cloudberry/pull/1085#issuecomment-4175815957

   As #1629 is merged, could we close this PR? How about it? @my-ship-it 
   If we want to reactivate it, we can reopen this PR. FYI.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


Re: [PR] Introduce extension perfmon to monitor database cluster [cloudberry]

Posted by "tuhaihe (via GitHub)" <gi...@apache.org>.
tuhaihe closed pull request #1085: Introduce extension perfmon to monitor database cluster
URL: https://github.com/apache/cloudberry/pull/1085


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org