You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "thiagotnunes (via GitHub)" <gi...@apache.org> on 2024/08/16 05:46:53 UTC

[PR] feat: optimize Spanner changestream metadata table [beam]

thiagotnunes opened a new pull request, #32213:
URL: https://github.com/apache/beam/pull/32213

   Adds indexes to the SpannerIO change streams metadata table. This serves to optimize two specific queries:
   
   1. `getUnfinishedMinWatermark`
   2. `getAllPartitionsCreatedAfter`
   
   (micro-)Benchmarking results with a metadata table containing 11 million rows:
   
   | Method | No Indexes (current) | Index (this PR) | Improvement |
   | --------- | -------- | ------- |
   | getUnfinishedMinWatermark | 3.190s (+/- 0.91s) | 0.020s (+/- 0.002s) |
   | getAllPartitionsCreatedAfter | 1.250s (+/- 0.230s) | 0.012s (+/- 0.001s) |
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat: optimize Spanner changestream metadata table [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn merged PR #32213:
URL: https://github.com/apache/beam/pull/32213


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat: optimize Spanner changestream metadata table [beam]

Posted by "thiagotnunes (via GitHub)" <gi...@apache.org>.
thiagotnunes commented on PR #32213:
URL: https://github.com/apache/beam/pull/32213#issuecomment-2295523054

   Run Java_GCP_IO_Direct PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat: optimize Spanner changestream metadata table [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32213:
URL: https://github.com/apache/beam/pull/32213#issuecomment-2295586549

   Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`:
   
   R: @robertwb for label java.
   R: @Abacn for label io.
   R: @nielm for label spanner.
   
   Available commands:
   - `stop reviewer notifications` - opt out of the automated review tooling
   - `remind me after tests pass` - tag the comment author after tests pass
   - `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
   
   The PR bot will only process comments in the main thread (not review comments).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat: optimize Spanner changestream metadata table [beam]

Posted by "velamkao (via GitHub)" <gi...@apache.org>.
velamkao commented on PR #32213:
URL: https://github.com/apache/beam/pull/32213#issuecomment-2316619097

   Hi @thiagotnunes would you know if this change to add the index is going to be part of Beam 2.59.0 release?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat: optimize Spanner changestream metadata table [beam]

Posted by "thiagotnunes (via GitHub)" <gi...@apache.org>.
thiagotnunes commented on PR #32213:
URL: https://github.com/apache/beam/pull/32213#issuecomment-2295580975

   assign set of reviewers


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat: optimize Spanner changestream metadata table [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32213:
URL: https://github.com/apache/beam/pull/32213#issuecomment-2295587102

   Reviewers are already assigned to this PR: @robertwb @Abacn @nielm


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat: optimize Spanner changestream metadata table [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32213:
URL: https://github.com/apache/beam/pull/32213#issuecomment-2292989364

   Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat: optimize Spanner changestream metadata table [beam]

Posted by "velamkao (via GitHub)" <gi...@apache.org>.
velamkao commented on PR #32213:
URL: https://github.com/apache/beam/pull/32213#issuecomment-2316631352

   > feat: index creation toga4/spream#106
   
   Never mind, I can see its there https://github.com/apache/beam/blob/v2.59.0-RC1/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/dao/PartitionMetadataAdminDao.java


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org