You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@devlake.apache.org by "Feggah (via GitHub)" <gi...@apache.org> on 2023/07/18 11:22:37 UTC
[GitHub] [incubator-devlake] Feggah opened a new issue, #5692: [Feature][AzureDevOps/GitHub] Support extracting Azure Pipelines metrics from GitHub Repositories
Feggah opened a new issue, #5692:
URL: https://github.com/apache/incubator-devlake/issues/5692
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-devlake/issues?q=is%3Aissue) and found no similar feature requirement.
### Use case
I have GitHub Repositories (Data Scopes of GitHub Plugin) and I use Azure Pipelines as a CI/CD tool. Azure Pipelines is part of AzureDevOps.
I want to be able to select GitHub Repositories and configure DevLake to extract DORA metrics from Azure Pipelines + GitHub Repo. But, today, I can only use GitHub Actions with GitHub Repositories.
### Description
_No response_
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] Startrekzky commented on issue #5692: [Feature][AzureDevOps/GitHub] Support extracting Azure Pipelines metrics from GitHub Repositories
Posted by "Startrekzky (via GitHub)" <gi...@apache.org>.
Startrekzky commented on issue #5692:
URL: https://github.com/apache/incubator-devlake/issues/5692#issuecomment-1642068631
Thank you, @Feggah . This is pretty informative. I just found out we seem to have supported this feature in https://github.com/apache/incubator-devlake/issues/5249. However, I didn't find any docs about how to configure it.
@CamilleTeruel Do you have any idea how can we use this feature?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] keon94 commented on issue #5692: [Feature][AzureDevOps/GitHub] Support extracting Azure Pipelines metrics from GitHub Repositories
Posted by "keon94 (via GitHub)" <gi...@apache.org>.
keon94 commented on issue #5692:
URL: https://github.com/apache/incubator-devlake/issues/5692#issuecomment-1649929313
@Feggah There is no special config. Is this a private repo?
Try this for troubleshooting (which is what the backend code does). Use POSTMAN or curl using your Azure devops token to:
1. Call `GET https://dev.azure.com/${YOUR_ORG_NAME}/${YOUR_PROJECT_NAME}/_apis/serviceendpoint/endpoints`. Look in the returned JSON for the field `type`, and make sure you see `GitHub`, and the field `id` for it.
2. Call `GET https://dev.azure.com/${YOUR_ORG_NAME}/${YOUR_PROJECT_NAME}/_apis/sourceProviders/GitHub/repositories?serviceEndpointId=${ID_FROM_ABOVE}` and see if your github repo is in there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] Feggah commented on issue #5692: [Feature][AzureDevOps/GitHub] Support extracting Azure Pipelines metrics from GitHub Repositories
Posted by "Feggah (via GitHub)" <gi...@apache.org>.
Feggah commented on issue #5692:
URL: https://github.com/apache/incubator-devlake/issues/5692#issuecomment-1642053249
We (as a company) decided to use GitHub as the tool to version our repositories. As we grew bigger, each team/tribe decided to use a different CI/CD tool to manage their applications.
This means that we have all possible combinations of tools, for example:
- Github Repos + GitHub Actions (supported by DevLake)
- GitHub Repos + Azure Pipelines (not supported, that's my feature request)
- GitHub Repos + CircleCI
- etc...
I don't know if this is a common use case in another companies (I imagine it is), but here we have 2k+ applications with this usecase.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] Feggah commented on issue #5692: [Feature][AzureDevOps/GitHub] Support extracting Azure Pipelines metrics from GitHub Repositories
Posted by "Feggah (via GitHub)" <gi...@apache.org>.
Feggah commented on issue #5692:
URL: https://github.com/apache/incubator-devlake/issues/5692#issuecomment-1654232984
@Startrekzky I upgraded to v0.18.0-beta3 and tried to add this external repository. The problem now is that the repositories are loading forever:
By looking at the logs of the DevLake API:
I think the problem is that my AzureDevOps project has a lot of Service Connections with different types. The fix probably would be skipping Service Connections which are not type == GitHub? Then it will only use valid Service Connections to find repositories
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] keon94 commented on issue #5692: [Feature][AzureDevOps/GitHub] Support extracting Azure Pipelines metrics from GitHub Repositories
Posted by "keon94 (via GitHub)" <gi...@apache.org>.
keon94 commented on issue #5692:
URL: https://github.com/apache/incubator-devlake/issues/5692#issuecomment-1645020633
@Feggah We do support Github-exclusive repos. They are prefixed with `GitHub` in the data-scopes seletion window. Example:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] Startrekzky commented on issue #5692: [Feature][AzureDevOps/GitHub] Support extracting Azure Pipelines metrics from GitHub Repositories
Posted by "Startrekzky (via GitHub)" <gi...@apache.org>.
Startrekzky commented on issue #5692:
URL: https://github.com/apache/incubator-devlake/issues/5692#issuecomment-1685741595
Hi @Feggah, sorry that I missed the notifications. I'll forward it to @klesh to take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] Startrekzky commented on issue #5692: [Feature][AzureDevOps/GitHub] Support extracting Azure Pipelines metrics from GitHub Repositories
Posted by "Startrekzky (via GitHub)" <gi...@apache.org>.
Startrekzky commented on issue #5692:
URL: https://github.com/apache/incubator-devlake/issues/5692#issuecomment-1697555238
Hi @Feggah , this issue is fixed in https://github.com/apache/incubator-devlake/releases/tag/v0.18.0-beta8 . Please let me know if you still have this problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] keon94 commented on issue #5692: [Feature][AzureDevOps/GitHub] Support extracting Azure Pipelines metrics from GitHub Repositories
Posted by "keon94 (via GitHub)" <gi...@apache.org>.
keon94 commented on issue #5692:
URL: https://github.com/apache/incubator-devlake/issues/5692#issuecomment-1651915634
@Startrekzky Invalid scope ID is likely happening because UI is sending an empty scope to the backend. As far as the list of scopes being empty, I think this has been resolved in beta2, by https://github.com/apache/incubator-devlake/pull/5708.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] Feggah commented on issue #5692: [Feature][AzureDevOps/GitHub] Support extracting Azure Pipelines metrics from GitHub Repositories
Posted by "Feggah (via GitHub)" <gi...@apache.org>.
Feggah commented on issue #5692:
URL: https://github.com/apache/incubator-devlake/issues/5692#issuecomment-1652191382
@Startrekzky could you also check if you can see this GitHub repository when creating a Blueprint instead of adding directly to the Connection?
Because I can't see any GitHub repos from an AzureDevOps Connection when creating a blueprint. Maybe there is a bug in that flow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] CamilleTeruel commented on issue #5692: [Feature][AzureDevOps/GitHub] Support extracting Azure Pipelines metrics from GitHub Repositories
Posted by "CamilleTeruel (via GitHub)" <gi...@apache.org>.
CamilleTeruel commented on issue #5692:
URL: https://github.com/apache/incubator-devlake/issues/5692#issuecomment-1693523054
fixed via #5950
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] Startrekzky commented on issue #5692: [Feature][AzureDevOps/GitHub] Support extracting Azure Pipelines metrics from GitHub Repositories
Posted by "Startrekzky (via GitHub)" <gi...@apache.org>.
Startrekzky commented on issue #5692:
URL: https://github.com/apache/incubator-devlake/issues/5692#issuecomment-1641280398
Got it. May I know why you use this combination of GitHub repo and Azure Pipeline? Are you maintaining an open-source project? I want to know if it is a common use case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] Feggah commented on issue #5692: [Feature][AzureDevOps/GitHub] Support extracting Azure Pipelines metrics from GitHub Repositories
Posted by "Feggah (via GitHub)" <gi...@apache.org>.
Feggah commented on issue #5692:
URL: https://github.com/apache/incubator-devlake/issues/5692#issuecomment-1646652819
I cannot reproduce your test @keon94 . Is there any special configuration?
I have a AzureDevOps connection, but I can only see Azure Repositories. Do I need to select something special?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] Startrekzky commented on issue #5692: [Feature][AzureDevOps/GitHub] Support extracting Azure Pipelines metrics from GitHub Repositories
Posted by "Startrekzky (via GitHub)" <gi...@apache.org>.
Startrekzky commented on issue #5692:
URL: https://github.com/apache/incubator-devlake/issues/5692#issuecomment-1651977064
@keon94 Got it. I'll re-test it in beta3 and let you know the results.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] Feggah commented on issue #5692: [Feature][AzureDevOps/GitHub] Support extracting Azure Pipelines metrics from GitHub Repositories
Posted by "Feggah (via GitHub)" <gi...@apache.org>.
Feggah commented on issue #5692:
URL: https://github.com/apache/incubator-devlake/issues/5692#issuecomment-1650394134
> Is this a private repo?
Yes, they are private
---
Using the POSTMAN it worked. I got a lot of repositories from AzureDevOps API. See below:
GET https://dev.azure.com/REDACTED/REDACTED/_apis/sourceProviders/GitHub/repositories?serviceEndpointId=REDACTED
But I also have a lot of ServiceConnections (or ServiceEndpoints) in my AzureDevOps Project. There are multiple GitHub connections. Maybe that's why it is not displaying in DevLake?
https://dev.azure.com/REDACTED/REDACTED/_apis/serviceendpoint/endpoints
Some thoughts:
- Maybe DevLake is selecting a ServiceConnection without the necessary permissions?
- The GET request that returned 2000 repositories took 40.2 seconds. Maybe DevLake is timeouting the request?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] klesh commented on issue #5692: [Feature][AzureDevOps/GitHub] Support extracting Azure Pipelines metrics from GitHub Repositories
Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on issue #5692:
URL: https://github.com/apache/incubator-devlake/issues/5692#issuecomment-1687761465
@CamilleTeruel Can you take a look at this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] CamilleTeruel closed issue #5692: [Feature][AzureDevOps/GitHub] Support extracting Azure Pipelines metrics from GitHub Repositories
Posted by "CamilleTeruel (via GitHub)" <gi...@apache.org>.
CamilleTeruel closed issue #5692: [Feature][AzureDevOps/GitHub] Support extracting Azure Pipelines metrics from GitHub Repositories
URL: https://github.com/apache/incubator-devlake/issues/5692
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] Startrekzky commented on issue #5692: [Feature][AzureDevOps/GitHub] Support extracting Azure Pipelines metrics from GitHub Repositories
Posted by "Startrekzky (via GitHub)" <gi...@apache.org>.
Startrekzky commented on issue #5692:
URL: https://github.com/apache/incubator-devlake/issues/5692#issuecomment-1651680806
Hi @keon94 , I tested it in v0.18.0-alpha17 (around beta1 I guess?) and the results were:
1. When adding a data scope to an Azure connection, I can fetch the external Github repo
2. However, after I added it to the connection, it didn't show the repo's name.
3. When I tried to delete the scope, it was not deletable.
4. When I tried to add the connection to a project and collect the data of the external repo, it threw an error:
```
error patching the blueprint (400) Wraps: (2) Wraps: (3) make plan for blueprint failed (400) Wraps: (4) Invalid scope id (400)"
```
which seemed to have something to do with the scope id.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] Startrekzky commented on issue #5692: [Feature][AzureDevOps/GitHub] Support extracting Azure Pipelines metrics from GitHub Repositories
Posted by "Startrekzky (via GitHub)" <gi...@apache.org>.
Startrekzky commented on issue #5692:
URL: https://github.com/apache/incubator-devlake/issues/5692#issuecomment-1653263828
Hi @Feggah , the v0.18.0's flow of adding repos has changed. I can only add data scope (external repos in this case) in the Data Connection page now.
And I just verified that in v0.18.0-beta3, when I tried to add repos to a Azure DevOps connection, it can list out the external GitHub repos(see the pic). cc. @keon94
@Feggah Can you show me the screenshot of adding the external data scope in your v0.17.0-beta13? Perhaps it's a bug in that version. I'm not sure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org