You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "xiaochen-zhou (via GitHub)" <gi...@apache.org> on 2025/08/16 03:02:17 UTC

[PR] [Feature][Transform-V2] Support jsonpath extracting multiple fields in a configuration block [seatunnel]

xiaochen-zhou opened a new pull request, #9712:
URL: https://github.com/apache/seatunnel/pull/9712

   
   ### Purpose of this pull request
   
   Support jsonpath extracting multiple fields in a configuration block
   
   ```bash
   
     JsonPath {
       plugin_input = "fake"
       plugin_output = "fake1"
       columns = [
        {
           "src_field" = "mysql_fields"
           "path" = "$.mysql_fields.id.v,$.mysql_fields.code.v,$.mysql_fields.group_code.v,$.mysql_fields.user_id.v,$.mysql_fields.patient_id.v,$.mysql_fields.doctor_id.v,$.mysql_fields.price.v,$.mysql_fields.status.v,$.mysql_fields.create_time.v,$.mysql_fields.update_time.v,$.nested_data.user.profile.name,$.nested_data.user.profile.age,$.nested_data.user.settings.theme,$.nested_data.orders[0].id,$.nested_data.orders[0].amount,$.nested_data.orders[1].id,$.array_fields[0].type,$.array_fields[0].value,$.array_fields[1].type,$.array_fields[1].value"
           "dest_field" = "id,code,group_code,user_id,patient_id,doctor_id,price,status,create_time,update_time,user_name,user_age,user_theme,first_order_id,first_order_amount,second_order_id,first_type,first_value,second_type,second_value"
           "dest_type" = "bigint,string,string,bigint,bigint,bigint,double,int,string,string,string,int,string,int,double,int,string,int,string,int"
        }
       ]
     }
   
   ```
   
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes.
   
   
   ### How was this patch tested?
   
   Add new tests
   
   
   ### Check list
   
   * [ ] If any new Jar binary package adding in your PR, please add License Notice according
     [New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/contribution/new-license.md)
   * [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
   * [ ] If you are contributing the connector code, please check that the following files are updated:
     1. Update [plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it
     2. Update the pom file of [seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
     3. Add ci label in [label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
     4. Add e2e testcase in [seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/)
     5. Update connector [plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Transform-V2] Support jsonpath extracting multiple fields in a configuration block [seatunnel]

Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on code in PR #9712:
URL: https://github.com/apache/seatunnel/pull/9712#discussion_r2284462499


##########
seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/jsonpath/JsonPathTransformConfig.java:
##########
@@ -114,24 +114,42 @@ public static JsonPathTransformConfig of(ReadonlyConfig config, CatalogTable tab
                             .map(ErrorHandleWay::valueOf)
                             .orElse(null);
 
-            SeaTunnelDataType<?> srcFieldDataType =
-                    SeaTunnelDataTypeConvertorUtil.deserializeSeaTunnelDataType(srcField, type);
+            String[] pathArray = paths.split(";");

Review Comment:
   why not use `,`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Transform-V2] Support jsonpath extracting multiple fields in a configuration block [seatunnel]

Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on PR #9712:
URL: https://github.com/apache/seatunnel/pull/9712#issuecomment-3218896477

   waiting test case passes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Transform-V2] Support jsonpath extracting multiple fields in a configuration block [seatunnel]

Posted by "xiaochen-zhou (via GitHub)" <gi...@apache.org>.
xiaochen-zhou commented on code in PR #9712:
URL: https://github.com/apache/seatunnel/pull/9712#discussion_r2286720377


##########
seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/jsonpath/JsonPathTransformConfig.java:
##########
@@ -114,24 +114,42 @@ public static JsonPathTransformConfig of(ReadonlyConfig config, CatalogTable tab
                             .map(ErrorHandleWay::valueOf)
                             .orElse(null);
 
-            SeaTunnelDataType<?> srcFieldDataType =
-                    SeaTunnelDataTypeConvertorUtil.deserializeSeaTunnelDataType(srcField, type);
+            String[] pathArray = paths.split(";");

Review Comment:
   > why not use `,`?
   
   When dest_type contains some complex data types like decimal(4,2), use `,` may cause issues.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Transform-V2] Support jsonpath extracting multiple fields in a configuration block [seatunnel]

Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on code in PR #9712:
URL: https://github.com/apache/seatunnel/pull/9712#discussion_r2287346148


##########
docs/en/transform-v2/jsonpath.md:
##########
@@ -161,6 +161,26 @@ transform {
 }
 ```
 
+The same result can be achieved with much simpler configuration using batch field extraction:
+
+```hocon
+transform {
+  JsonPath {
+    plugin_input = "fake"
+    plugin_output = "fake1"
+    columns = [
+     {
+        "src_field" = "data"
+        "path" = "$.data.c_string;$.data.c_boolean;$.data.c_integer;$.data.c_float;$.data.c_double;$.data.c_decimal;$.data.c_date;$.data.c_datetime;$.data.c_array;$.data.c_map_array"

Review Comment:
   I'm thinking can we use array type value?
   
   ```
   "path" = ["$.data.c_string", $.data.c_boolean]
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Transform-V2] Support jsonpath extracting multiple fields in a configuration block [seatunnel]

Posted by "corgy-w (via GitHub)" <gi...@apache.org>.
corgy-w merged PR #9712:
URL: https://github.com/apache/seatunnel/pull/9712


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Transform-V2] Support jsonpath extracting multiple fields in a configuration block [seatunnel]

Posted by "xiaochen-zhou (via GitHub)" <gi...@apache.org>.
xiaochen-zhou commented on PR #9712:
URL: https://github.com/apache/seatunnel/pull/9712#issuecomment-3222447445

   > waiting test case passes.
   
   done.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org