You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@devlake.apache.org by "narrowizard (via GitHub)" <gi...@apache.org> on 2025/04/23 07:23:43 UTC

[I] [Feature][customize] Enhance customize plugin: Automatically create accounts when importing issues.csv [incubator-devlake]

narrowizard opened a new issue, #8400:
URL: https://github.com/apache/incubator-devlake/issues/8400

   ### Search before asking
   
   - [x] I had searched in the [issues](https://github.com/apache/incubator-devlake/issues?q=is%3Aissue) and found no similar feature requirement.
   
   
   ### Use case
   
   When importing issue data from third-party platforms that aren't covered by existing DevLake plugins, users often leverage the `customize` plugin's ability to ingest data via `issues.csv` through the `/plugins/customize/csvfiles/issues.csv` API endpoint.
   
   While this process successfully creates issues domain layer entities, it currently only focuses on the issue data itself. It does not automatically create or update corresponding entries in the `accounts` domain layer table based on fields like `creator_name` or `assignee_name` present in the `issues.csv`.
   
   This limitation means that even if issue data is successfully imported, the associated user data required for analyses related to individuals (like work distribution, assignee performance, etc.) is missing or requires a separate, potentially manual, process to populate the accounts table. This creates a significant gap in the imported data's completeness for common engineering metrics and requires extra effort from users.
   
   ### Description
   
   This feature request proposes enhancing the customize plugin's handling of the /plugins/customize/csvfiles/issues.csv import process.
   
   Specifically, when data is ingested via this endpoint, the plugin should, in addition to creating/updating issues domain entities, also process the creator_name and assignee_name fields within the CSV data.
   
   For each unique name found in either the creator_name or assignee_name columns across the entire imported CSV, the plugin should perform the following action:
   
   - Check if an account with a matching identifier (derived from the name) already exists in the accounts domain layer table.
   - If an account does not exist for that name, a new account entry should be automatically created in the accounts table, using the name from the CSV. (The exact mechanism for generating the account_id and mapping the name would need to be determined during implementation, potentially hashing the name or using the name itself if suitable for the schema).
   
   Implementing this feature would greatly streamline the process of importing issue data from external sources via the customize plugin, ensuring that essential related account information is automatically populated in the domain layer, making the data ready for richer analysis without manual intervention.
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [x] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][customize] Enhance customize plugin: Automatically create accounts when importing issues.csv [incubator-devlake]

Posted by "narrowizard (via GitHub)" <gi...@apache.org>.
narrowizard closed issue #8400: [Feature][customize] Enhance customize plugin: Automatically create accounts when importing issues.csv
URL: https://github.com/apache/incubator-devlake/issues/8400


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][customize] Enhance customize plugin: Automatically create accounts when importing issues.csv [incubator-devlake]

Posted by "narrowizard (via GitHub)" <gi...@apache.org>.
narrowizard commented on issue #8400:
URL: https://github.com/apache/incubator-devlake/issues/8400#issuecomment-2826469600

   closed by #8401 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][customize] Enhance customize plugin: Automatically create accounts when importing issues.csv [incubator-devlake]

Posted by "narrowizard (via GitHub)" <gi...@apache.org>.
narrowizard commented on issue #8400:
URL: https://github.com/apache/incubator-devlake/issues/8400#issuecomment-2823327115

   ## Solution Proposal: Extend customize plugin to sync account creation during issues.csv import
   
   This proposal outlines a solution to extend the DevLake `customize` plugin's functionality. The goal is to process and create relevant account information simultaneously with the import of issues data via the `/plugins/customize/csvfiles/issues.csv` interface, and then link these created accounts to the imported issue data.
   
   The implementation steps and data processing involved are as follows:
   
   1.  **Account Data Insertion:**
       * When issues.csv data is received via the `/plugins/customize/csvfiles/issues.csv` interface, the plugin will first process the account-related information.
       * Extract all unique account names from the `creator_name` and `assignee_name` fields within the issues.csv (e.g., Alice Smith, Bob Johnson).
       * For each extracted account name, perform a data insertion operation into DevLake's `accounts` table (if the account does not already exist).
       * The key fields to be populated in the `accounts` table and their data sources/mapping rules are as follows:
           * `id`: Generate a unique ID based on the account name using the rule `csv:CsvAccount:0:${accountName}`.
           * `_raw_data_params`: Use the rule `csv:CsvBoard:0:${projectName}`. This might require obtaining project name information from the import context.
           * `full_name`: Map to the `accountName` extracted from the CSV (i.e., the value of creator_name or assignee_name).
           * `user_name`: Also map to the `accountName` extracted from the CSV.
           * `created_date`: Record the data import time.
           * `status`: Set to `0`.
           * Other fields (such as `created_at`, `updated_at`, `raw_data_id`, `raw_data_remark`, `email`, `avatar_url`, `organization`, `status`) should be populated as `null` or default values since corresponding information is not available in the CSV or specified in the plan.
   
   2.  **Issue Data Update/Association:**
       * After creating the relevant account entries in the `accounts` table, the plugin will proceed to process the main issues.csv data.
       * When mapping the issues.csv data to DevLake's `issues` domain layer entities, leverage the account information created in the first step.
       * Use the `creator_name` from issues.csv to look up the corresponding account `id` created in the `accounts` table, and populate the `creator_id` field in the `issues` table with this ID.
       * Use the `assignee_name` from issues.csv to look up the corresponding account `id` created in the `accounts` table, and populate the `assignee_id` field in the `issues` table with this ID.
   
   By following these steps, we can ensure that when importing issues.csv data via the customize plugin, not only is the issues data itself correctly imported, but the related creator and assignee account information is also automatically created and associated, providing a more complete view of the DevOps data.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org