You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@devlake.apache.org by yu...@apache.org on 2022/10/19 09:44:38 UTC

[incubator-devlake-website] branch main updated (b33c8f435 -> f79d801d7)

This is an automated email from the ASF dual-hosted git repository.

yumeng pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-devlake-website.git


    from b33c8f435 doc: froze v0.14 docs
     new fdfa88f0d docs: migration script code example for v0.15
     new f79d801d7 Update DBMigration.md

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 docs/DeveloperManuals/DBMigration.md | 146 +++++++++--------------------------
 1 file changed, 35 insertions(+), 111 deletions(-)


[incubator-devlake-website] 02/02: Update DBMigration.md

Posted by yu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

yumeng pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-devlake-website.git

commit f79d801d74c28482a4c84a86b5edebf56016ae69
Author: Yumeng Wang <yu...@merico.dev>
AuthorDate: Wed Oct 19 17:39:31 2022 +0800

    Update DBMigration.md
---
 docs/DeveloperManuals/DBMigration.md | 48 +++++++++++++++++-------------------
 1 file changed, 22 insertions(+), 26 deletions(-)

diff --git a/docs/DeveloperManuals/DBMigration.md b/docs/DeveloperManuals/DBMigration.md
index 269391fcb..865589992 100644
--- a/docs/DeveloperManuals/DBMigration.md
+++ b/docs/DeveloperManuals/DBMigration.md
@@ -7,14 +7,13 @@ sidebar_position: 3
 
 ## Summary
 Starting in v0.10.0, DevLake provides a lightweight migration tool for executing migration scripts.
-Both framework itself and plugins define their migration scripts in their own migration folder.
+Both the framework and the plugins can define their migration scripts in their own migration folder.
 The migration scripts are written with gorm in Golang to support different SQL dialects.
 
 
-## Migration Script
-Migration script describes how to do database migration.
-They implement the `Script` interface.
-When DevLake starts, scripts register themselves to the framework by invoking the `Register` function.
+## Migration Scripts
+Migration scripts describes how to do database migration and implements the `Script` interface.
+When DevLake starts, the scripts register themselves to the framework by invoking the `Register` function.
 The method `Up` contains the steps of migration.
 
 ```go
@@ -28,35 +27,34 @@ type Script interface {
 }
 ```
 
-## Migration Model
+## The Migration Model
 
-For each migration we define a "snapshot" datamodel of the model that we wish to perform the migration on.
-The fields on this model shall be identical to the actual model, but unlike the actual one, this one will
+For each migration, we define a "snapshot" datamodel of the model that we wish to perform the migration on.
+The fields on this model shall be identical to the actual model; but unlike the actual one, this one will
 never change in the future. The naming convention of these models is `<ModelName>YYYYMMDD` and they must implement
 the `func TableName() string` method, and consumed by the `Script::Up` method.
 
 ## Table `migration_history`
 
-The table tracks migration scripts execution and schemas changes.
-From which, DevLake could figure out the current state of database schemas.
+The table tracks migration scripts execution and schemas changes, and from which, DevLake can figure out the current state of database schemas.
 
 ## Execution
 
 Each plugin has a `migrationscripts` subpackage that lists all the migrations to be executed for that plugin. You
-will need to add your migration to that list for the framework to pick it up. Similarly, there is such a package
+will need to add your migration to that list for the framework to pick it up. Similarly, there is a package
 for the framework-only migrations defined under the `models` package.
 
 
 ## How It Works
 1. Check `migration_history` table, calculate all the migration scripts need to be executed.
-2. Sort scripts by `Version` and `Name` in ascending order. You should NOT change these 2 values for the script after release for whatever reasons, or user may fail to upgrade due to duplicate execution.
-3. Execute scripts.
-4. Save results in the `migration_history` table.
+2. Sort scripts by `Version` and `Name` in ascending order. Please do NOT change these two values for the script after release for any reasons; otherwise, users may fail to upgrade due to the duplicated execution.
+3. Execute the scripts.
+4. Save the results in the `migration_history` table.
 
 
 ## Best Practices
 
-When you write a new migration script, please pay attention to the fault tolerance and the side effect. It would be better if the failed script could be safely retry, in case of something goes wrong during the migration. For this purpose, the migration scripts should be well-designed. For example, if you created a temporary table in the Up method, it should be dropped before exiting, regardless of success or failure. 
+When you write a new migration script, please pay attention to the fault tolerance and the side effect. It would be better if the failed script could be safely retried, in case if something goes wrong during the migration. For this purpose, the migration scripts should be well-designed. For example, if you have created a temporary table in the Up method, it should be dropped before exiting, regardless of success or failure. 
 
 Suppose we want to change the type of the Primary Key `name` of table `users` from `int` to `varchar(255)`
 
@@ -67,7 +65,7 @@ Suppose we want to change the type of the Primary Key `name` of table `users` fr
 
 With these steps, the `defer` functions would be executed in reverse order if any error occurred during the migration process so the database would roll back to the original state in most cases.
 
-However, you don't neccessary deal with all these mess. We had summarized some of the most useful code examples for you to follow:
+However, you don't neccessary deal with all the mess. We had summarized some of the most useful code examples for you to follow:
 
 - [Create new tables](https://github.com/apache/incubator-devlake/blob/main/models/migrationscripts/20220406_add_frame_tables.go)
 [Rename column](https://github.com/apache/incubator-devlake/blob/main/models/migrationscripts/20220505_rename_pipeline_step_to_stage.go)
@@ -75,20 +73,18 @@ However, you don't neccessary deal with all these mess. We had summarized some o
 - [Change the values(or type) of Primary Key](https://github.com/apache/incubator-devlake/blob/main/models/migrationscripts/20220913_fix_commitfile_id_toolong.go)
 - [Change the values(or type) of Column](https://github.com/apache/incubator-devlake/blob/main/models/migrationscripts/20220929_modify_lead_time_minutes.go)
 
-The above examples should cover most of the scenarios you may run into. Feel free to post issue on our github repo otherwise.
+The above examples should cover most of the scenarios you may encounter. If you come across other scenarios, feel free to create issues in our GitHub Issue Tracker for discussions. 
 
 
-In order to help others understand the script you wrote, there are a couple of rules you have to follow:
+In order to help others understand the script you have written, there are a couple of rules we suggest to follow:
 
-In order to help others understand the script you wrote, there are a couple of rules you have to follow:
-
-- Name your script in a meaningful way. For instance `renamePipelineStepToStage` is far more better than `modifyPipelines`
-- The script should keep only the targeted `fields` you are attempting to operate except when using `migrationhelper.Transform` which is a full table tranformation that requires full table definition, if this is the case, add comment to the end of the fields to indicate which ones are the target.
-- Add comment to the script when the operation are too complicated to express in plain code.
+- Name your script in a meaningful way. For instance, `renamePipelineStepToStage` is more descriptive than `modifyPipelines`.
+- The script should keep only the targeted `fields` you are attempting to operate except when using `migrationhelper.Transform`, which is a full table tranformation that requires full table definition. If this is the case, add comment to the end of the fields to indicate which ones are the targets.
+- Add comments to the script when the operation is too complicated to be expressed in plain code.
 
 Other rules to follow when writing a migration script:
 
-- The migration script should use only the interfaces and packages offerred by the framework like `core`, `errors` and `migrationhelper`, do NOT import `gorm` or package from `plugin` directly.
-- The name of `model struct` defined in your script should be suffixed with the `Version` of the script to distinguish from other scripts in the same package to keep it self-contained, i.e. `tasks20221018`. do NOT refer `struct` defined in other scripts.
-- All scripts and models names should be `camelCase` to avoid accidentally reference from other packages
+- The migration script should only use the interfaces and packages offerred by the framework like `core`, `errors` and `migrationhelper`. Do NOT import `gorm` or package from `plugin` directly.
+- The name of `model struct` defined in your script should be suffixed with the `Version` of the script to distinguish from other scripts in the same package to keep it self-contained, i.e. `tasks20221018`. Do NOT refer `struct` defined in other scripts.
+- All scripts and models names should be `camelCase` to avoid accidental reference from other packages.
 


[incubator-devlake-website] 01/02: docs: migration script code example for v0.15

Posted by yu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

yumeng pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-devlake-website.git

commit fdfa88f0d14e0e93c33fc040afb0250466c75b8a
Author: Klesh Wong <zh...@merico.dev>
AuthorDate: Tue Oct 18 15:13:21 2022 +0800

    docs: migration script code example for v0.15
---
 docs/DeveloperManuals/DBMigration.md | 124 ++++++++---------------------------
 1 file changed, 26 insertions(+), 98 deletions(-)

diff --git a/docs/DeveloperManuals/DBMigration.md b/docs/DeveloperManuals/DBMigration.md
index b4394d33c..269391fcb 100644
--- a/docs/DeveloperManuals/DBMigration.md
+++ b/docs/DeveloperManuals/DBMigration.md
@@ -49,118 +49,46 @@ for the framework-only migrations defined under the `models` package.
 
 ## How It Works
 1. Check `migration_history` table, calculate all the migration scripts need to be executed.
-2. Sort scripts by Version in ascending order.
+2. Sort scripts by `Version` and `Name` in ascending order. You should NOT change these 2 values for the script after release for whatever reasons, or user may fail to upgrade due to duplicate execution.
 3. Execute scripts.
 4. Save results in the `migration_history` table.
 
 
 ## Best Practices
-When you write a new migration script, please pay attention to the fault tolerance and the side effect. It would be better if the failed script could be safely retry, in case of something goes wrong during the migration. For this purpose, the migration scripts should be well-designed. For example, if you created a temporary table in the Up method, it should be dropped before exiting, regardless of success or failure. Using the defer statement to do some cleanup is a good idea. Let's demo [...]
 
-Suppose we want to recalculate the column `name` of the table `user`
+When you write a new migration script, please pay attention to the fault tolerance and the side effect. It would be better if the failed script could be safely retry, in case of something goes wrong during the migration. For this purpose, the migration scripts should be well-designed. For example, if you created a temporary table in the Up method, it should be dropped before exiting, regardless of success or failure. 
 
-1. rename `user` to `user_bak` (stop if error, define `defer` to rename back on error)
-2. create new `user` (stop if error, define `defer` to drop TABLE on error)
-3. convert data from `user_bak` to `user` (stop if error)
-4. drop `user_bak`
+Suppose we want to change the type of the Primary Key `name` of table `users` from `int` to `varchar(255)`
 
-```golang
+1. Rename `users` to `users_20221018` (stop if error, otherwise define a `defer` to rename back on error)
+2. Create new `users` (stop if error, otherwise define a `defer` to drop the table on error)
+3. Convert data from `users_20221018` to `users` (stop if error)
+4. Drop table `users_20221018`
 
-type User struct {
-	name string `gorm:"type:varchar(255)"`
-}
+With these steps, the `defer` functions would be executed in reverse order if any error occurred during the migration process so the database would roll back to the original state in most cases.
 
-func (User) TableName() string {
-	return "user"
-}
+However, you don't neccessary deal with all these mess. We had summarized some of the most useful code examples for you to follow:
 
-type NewUser struct {
-	name string `gorm:"type:text"`
-}
+- [Create new tables](https://github.com/apache/incubator-devlake/blob/main/models/migrationscripts/20220406_add_frame_tables.go)
+[Rename column](https://github.com/apache/incubator-devlake/blob/main/models/migrationscripts/20220505_rename_pipeline_step_to_stage.go)
+- [Add columns with default value](https://github.com/apache/incubator-devlake/blob/main/models/migrationscripts/20220616_add_blueprint_mode.go)
+- [Change the values(or type) of Primary Key](https://github.com/apache/incubator-devlake/blob/main/models/migrationscripts/20220913_fix_commitfile_id_toolong.go)
+- [Change the values(or type) of Column](https://github.com/apache/incubator-devlake/blob/main/models/migrationscripts/20220929_modify_lead_time_minutes.go)
 
-func (NewUser) TableName() string {
-	return "user"
-}
+The above examples should cover most of the scenarios you may run into. Feel free to post issue on our github repo otherwise.
 
-type UserBak struct {
-	name string `gorm:"type:varchar(255)"`
-}
 
-func (UserBak) TableName() string {
-	return "user_bak"
-}
+In order to help others understand the script you wrote, there are a couple of rules you have to follow:
 
-func (*exampleScript) Up(ctx context.Context, db *gorm.DB) (errs errors.Error) {
-	var err error
-
-	// rename the user_bak to cache old table
-	err = db.Migrator().RenameTable(&User{}, &UserBak{})
-	if err != nil {
-		return errors.Default.Wrap(err, "error no rename user to user_bak")
-	}
-
-	// rollback for rename back
-	defer func() {
-		if errs != nil {
-			err = db.Migrator().RenameTable(&UserBak{}, &User{})
-			if err != nil {
-				errs = errors.Default.Wrap(err, fmt.Sprintf("fail to rollback table user_bak , you must to rollback by yourself. %s", err.Error()))
-			}
-		}
-	}()
-
-	// create new user table
-	err = db.Migrator().AutoMigrate(&NewUser{})
-
-	if err != nil {
-		return errors.Default.Wrap(err, "error on auto migrate user")
-	}
-
-	// rollback for create new table
-	defer func() {
-		if errs != nil {
-			err = db.Migrator().DropTable(&User{})
-			if err != nil {
-				errs = errors.Default.Wrap(err, fmt.Sprintf("fail to rollback table OldTable , you must to rollback by yourself. %s", err.Error()))
-			}
-		}
-	}()
-
-	// update old id to new id and write to the new table
-	cursor, err := db.Model(&UserBak{}).Rows()
-	if err != nil {
-		return errors.Default.Wrap(err, "error on select NewTable")
-	}
-	defer cursor.Close()
-
-	// caculate and save the data to new table
-	batch, err := helper.NewBatchSave(api.BasicRes, reflect.TypeOf(&NewUser{}), 200)
-	if err != nil {
-		return errors.Default.Wrap(err, "error getting batch from table user")
-	}
-	defer batch.Close()
-	for cursor.Next() {
-		ot := UserBak{}
-		err = db.ScanRows(cursor, &ot)
-		if err != nil {
-			return errors.Default.Wrap(err, "error scan rows from table user_bak")
-		}
-		nt := NewUser(ot)
-
-		nt.name = nt.name + "new"
-
-		err = batch.Add(&nt)
-		if err != nil {
-			return errors.Default.Wrap(err, "error on user batch add")
-		}
-	}
-
-	// drop the old table
-	err = db.Migrator().DropTable(&UserBak{})
-	if err != nil {
-		return errors.Default.Wrap(err, "error no drop user_bak")
-	}
-}
+In order to help others understand the script you wrote, there are a couple of rules you have to follow:
 
-```
+- Name your script in a meaningful way. For instance `renamePipelineStepToStage` is far more better than `modifyPipelines`
+- The script should keep only the targeted `fields` you are attempting to operate except when using `migrationhelper.Transform` which is a full table tranformation that requires full table definition, if this is the case, add comment to the end of the fields to indicate which ones are the target.
+- Add comment to the script when the operation are too complicated to express in plain code.
+
+Other rules to follow when writing a migration script:
+
+- The migration script should use only the interfaces and packages offerred by the framework like `core`, `errors` and `migrationhelper`, do NOT import `gorm` or package from `plugin` directly.
+- The name of `model struct` defined in your script should be suffixed with the `Version` of the script to distinguish from other scripts in the same package to keep it self-contained, i.e. `tasks20221018`. do NOT refer `struct` defined in other scripts.
+- All scripts and models names should be `camelCase` to avoid accidentally reference from other packages