You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@devlake.apache.org by "sstojak1 (via GitHub)" <gi...@apache.org> on 2024/05/11 06:08:06 UTC
[I] [Refactor][Bitbucket_Server] Refactor title [incubator-devlake]
sstojak1 opened a new issue, #7457:
URL: https://github.com/apache/incubator-devlake/issues/7457
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
## What and why to refactor
What are you trying to refactor? Why should it be refactored now?
The [pr_collector](https://github.com/apache/incubator-devlake/blob/5aac71130d12868f89074414c441e54cef51ca64/backend/plugins/bitbucket_server/tasks/pr_collector.go#L30) for Bitbucket Server consistently adds the same data to the RAW_PULL_REQUEST_TABLE after each run.
Consequently, the extractApiPullRequests process slows down because it has to sift through all the records in the raw table, including duplicates.
For instance, if a repository has 1000 pull requests, after 10 job runs, the raw table will contain 10,000 rows, and extractApiPullRequests will have to process each of these records.
## Describe the solution you'd like
How to refactor?
1. Perhaps we could go with deleting all records from the raw table and importing PR again with each job run. This would prevent duplicates and avoid slowing down extractApiPullRequests task.
2. Check how Bitbucket plugin is doing it and maybe reuse the logic if it's better?
## Related issues
Please link any other
## Additional context
Add any other context or screenshots about the feature request here.
How to recreate:
Run Collect Data for Bitbucket Server more than once and observe the size of _raw_bitbucket_server_api_pull_requests table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Refactor][Bitbucket_Server] Speed up PR collector/extractor [incubator-devlake]
Posted by "Startrekzky (via GitHub)" <gi...@apache.org>.
Startrekzky commented on issue #7457:
URL: https://github.com/apache/incubator-devlake/issues/7457#issuecomment-2111482679
Hi @sstojak1 , it's a valid refactore, but may I know why the same data keeps been adding to the RAW_PULL_REQUEST_TABLE after each run?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Refactor][Bitbucket_Server] Speed up PR collector/extractor [incubator-devlake]
Posted by "sstojak1 (via GitHub)" <gi...@apache.org>.
sstojak1 commented on issue #7457:
URL: https://github.com/apache/incubator-devlake/issues/7457#issuecomment-2116865353
@klesh That might work! Is there a way to run Devlake locally in debug mode? I'd like to go through the ApiCollector impl to understand its impact on the rest of the steps for importing BB server data...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Refactor][Bitbucket_Server] Speed up PR collector/extractor [incubator-devlake]
Posted by "sstojak1 (via GitHub)" <gi...@apache.org>.
sstojak1 commented on issue #7457:
URL: https://github.com/apache/incubator-devlake/issues/7457#issuecomment-2115965969
Hi @Startrekzky I would say that this is because BB Server API doesn't support date query parameter. So you cannot fetch PRs that were updated/created after last import job.
Because of that Devlake is importing all PRs during each job run ([check](https://github.com/apache/incubator-devlake/blob/e24102f2964c2035fff948a736a97108d000ba33/backend/plugins/bitbucket/tasks/pr_collector.go#L4) how BB Cloud is not doing that since date query parameter is supported for that tool).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Refactor][Bitbucket_Server] Speed up PR collector/extractor [incubator-devlake]
Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on issue #7457:
URL: https://github.com/apache/incubator-devlake/issues/7457#issuecomment-2153854946
@sstojak1 Thanks for the reminder. Done. 🤝
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Refactor][Bitbucket_Server] Speed up PR collector/extractor [incubator-devlake]
Posted by "d4x1 (via GitHub)" <gi...@apache.org>.
d4x1 closed issue #7457: [Refactor][Bitbucket_Server] Speed up PR collector/extractor
URL: https://github.com/apache/incubator-devlake/issues/7457
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Refactor][Bitbucket_Server] Speed up PR collector/extractor [incubator-devlake]
Posted by "sstojak1 (via GitHub)" <gi...@apache.org>.
sstojak1 commented on issue #7457:
URL: https://github.com/apache/incubator-devlake/issues/7457#issuecomment-2160980798
@klesh ready to be reviewed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Refactor][Bitbucket_Server] Speed up PR collector/extractor [incubator-devlake]
Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on issue #7457:
URL: https://github.com/apache/incubator-devlake/issues/7457#issuecomment-2117106405
Yes, sure. You may follow [this guide](https://devlake.apache.org/docs/DeveloperManuals/DeveloperSetup).
In case you wanna execute specific subtasks, you may go to the `backend` folder and run sth like the following:
```
go run plugins/jira/jira.go -c 2 -b 8 -t "extractWorklogs"
```
remember to change the plugin name and arguments accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Refactor][Bitbucket_Server] Speed up PR collector/extractor [incubator-devlake]
Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on issue #7457:
URL: https://github.com/apache/incubator-devlake/issues/7457#issuecomment-2116546794
Based on the information you provided, it seems the collector would benefit from using the simpler ApiCollector since it would purge related records from the raw data table before saving new PR information. This might be more efficient compared to the StatefulApiCollector in this context.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Refactor][Bitbucket_Server] Speed up PR collector/extractor [incubator-devlake]
Posted by "sstojak1 (via GitHub)" <gi...@apache.org>.
sstojak1 commented on issue #7457:
URL: https://github.com/apache/incubator-devlake/issues/7457#issuecomment-2152352273
@klesh can you please assign this task to me? I have started to work on it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org