You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "devbiu (via GitHub)" <gi...@apache.org> on 2023/06/27 03:26:35 UTC

[GitHub] [seatunnel] devbiu opened a new issue, #4980: [Bug] Split transform plugin not available in Flink

devbiu opened a new issue, #4980:
URL: https://github.com/apache/seatunnel/issues/4980

   ### Search before asking
   
   - [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
   
   
   ### What happened
   
   Synchronize Kafka to MySQL using Flink 
   the same data as the topics of coffee
   
   [Id":"10", "name":"test_admin", "sex":"man", "age":"26", "raw_message":"sun#wuk"]
   
   [Id":"11", "name":"test_admin", "sex":"man", "age":"28", "raw_message":"tang#sz"]
   
   {id":"12":"test_admin","sex":"man":"age":"28", "raw_message":"zhu#bj"}
   
   [Id":"14", "test_admin","sex":"man","age":"28", "raw_message":"bai,gj"]
   
   Because there is no good split source_field, so we try raw_message with the nickname of spark
   
   The separator used is # humour, also includes, split
   
   The sinkWe use is mysql, its structure has id, name, sex, age, raw_message and breaking parameters first_name, last_name
   
   and console for printing
   
   However, the result does not contain first_name to split raw_message, last_name has always been aired, the output of console does not contain first_name, last_name
   
   The other paragraphs were also accompanied in the mysql table.
   
   ### SeaTunnel Version
   
   seatunnel 2.3.0
   
   ### SeaTunnel Config
   
   ```conf
   hocon like 
   env {
     job.name = "seatunnel_split_8dlUQM7J"
     
     job.mode = "STREAMING"
   }
   
   source {
     Kafka {
       bootstrap.servers = ""
       
       commit_on_checkpoint = "true"
       
       format = "json"
       
       pattern = "false"
       
       schema = {
         fields {
           id = "int"
           
           name = "string"
           
           sex = "string"
           
           age = "string"
           
           raw_message = "string"
         }
       }
       
       start_mode = "earliest"
       
       topic = "demo02"
     }
   }
   
   transform {
     split {
       fields = ["first_name", "last_name"]
       
       separator = "#"
     }
   }
   
   sink {
     Jdbc {
       batch_interval_ms = 1000
       
       batch_size = 300
       
       connection_check_timeout_sec = 30
       
       driver = "com.mysql.cj.jdbc.Driver"
       
       is_exactly_once = "false"
       
       max_commit_attempts = 3
       
       max_retries = 3
       
       password = ""
       
       support_upsert_by_query_primary_key_exist = "false"
       
       table = "xxxx"
       
       transaction_timeout_sec = -1
       
       url = ""
       
       user = ""
     }
     
     Console {
       
     }
   }
   ```
   
   
   ### Running Command
   
   ```shell
   /opt/seatunnel/seatunnel/bin/start-seatunnel-flink-connector-v2.sh  -D env.java.opts="-Dfile.encoding=UTF-8 -Dsun.jnu.encoding=UTF-8" --config xxx.conf
   ```
   
   
   ### Error Exception
   
   ```log
   not exception
   But the field value output by the data split is empty
   ```
   
   
   ### Flink or Spark Version
   
   flink 1.13.6 
   
   ### Java or Scala Version
   
   java 11
   
   ### Screenshots
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [seatunnel] github-actions[bot] commented on issue #4980: [Bug] Split transform plugin not available in Flink

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #4980:
URL: https://github.com/apache/seatunnel/issues/4980#issuecomment-1674080435

   This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [seatunnel] github-actions[bot] closed issue #4980: [Bug] Split transform plugin not available in Flink

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #4980: [Bug] Split transform plugin not available in Flink
URL: https://github.com/apache/seatunnel/issues/4980


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [seatunnel] github-actions[bot] commented on issue #4980: [Bug] Split transform plugin not available in Flink

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #4980:
URL: https://github.com/apache/seatunnel/issues/4980#issuecomment-1656979070

   This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [seatunnel] devbiu commented on issue #4980: [Bug] Split transform plugin not available in Flink

Posted by "devbiu (via GitHub)" <gi...@apache.org>.
devbiu commented on issue #4980:
URL: https://github.com/apache/seatunnel/issues/4980#issuecomment-1608718677

   The results of synchronizing to MySQL are as follows
   <img width="589" alt="image" src="https://github.com/apache/seatunnel/assets/99388822/608fafda-da73-4226-a6aa-e03288838094">
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [seatunnel] devbiu commented on issue #4980: [Bug] Split transform plugin not available in Flink

Posted by "devbiu (via GitHub)" <gi...@apache.org>.
devbiu commented on issue #4980:
URL: https://github.com/apache/seatunnel/issues/4980#issuecomment-1612311447

   
   
   
   > The config should follow the doc of https://seatunnel.apache.org/docs/2.3.2/transform-v2/split ; add split_field = "raw_message"
   Thank you for your answer. Version 2.3.2 does require the addition of split_ Field="raw_message"
   But I am using seatunel version 2.3.0 and did not see any split_ field
   
   doc of https://seatunnel.apache.org/docs/2.3.0/transform/split
   <img width="839" alt="image" src="https://github.com/apache/seatunnel/assets/99388822/85d329f7-26ab-494f-ac67-3dcb4cb679dc">
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [seatunnel] john8628 commented on issue #4980: [Bug] Split transform plugin not available in Flink

Posted by "john8628 (via GitHub)" <gi...@apache.org>.
john8628 commented on issue #4980:
URL: https://github.com/apache/seatunnel/issues/4980#issuecomment-1612306134

   The config should follow the doc of  https://seatunnel.apache.org/docs/2.3.2/transform-v2/split ;  add   split_field = "raw_message"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org