You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "jeanleen (via GitHub)" <gi...@apache.org> on 2024/01/20 07:38:58 UTC

[I] [Bug] [jdbc write] the job did not end, running every minutes [seatunnel]

jeanleen opened a new issue, #6260:
URL: https://github.com/apache/seatunnel/issues/6260

   ### Search before asking
   
   - [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
   
   
   ### What happened
   
   first  i delete the data from mysql ,then write data ,so i set two jdbc configuration in sink module. when write data to mysql, the job did not end ,run every minute.
   
   ### SeaTunnel Version
   
   2.3.3
   
   ### SeaTunnel Config
   
   ```conf
   # Defining the runtime environment
   env {
     parallelism = 2
   #   spark.master = "yarn"
   #   spark.deploy-mode= "cluster"
     job.name = ""
   #   spark.executor.instances = 2
   #   spark.executor.cores = 2
   #   spark.executor.memory = "4g"
     job.mode = "BATCH"
   }
   source{
   # Jdbc {
   #        
   #         
   #         query = """
   #                  select * from test.users
   #
   #         """
   # #         query = """SELECT  PRO_ID,PRO_BOUGHT,CATE_1,TO_CHAR(SYNC_DATETIME, 'YYYY-MM-DD')  as SYNC_DATETIME FROM ( SELECT PRO_ID,PRO_BOUGHT,CATE_1,SYNC_DATETIME FROM SMT_GET_PRODUCT ORDER BY UPDATE_TIME ) WHERE ROWNUM <= 100 """
   # #           query ="SELECT  PRO_ID,PRO_BOUGHT,CATE_1,TO_CHAR(SYNC_DATETIME, 'YYYY-MM-DD') as SYNC_DATETIME FROM SMT_GET_PRODUCT where TO_CHAR(SYNC_DATETIME, 'YYYY-MM-DD')='2024-01-16'"
   #     }
   
   
     HdfsFile {
   #   schema {
   #     fields {
   #       name = string
   #       address = string
   #     }
   #   }
     path = "/user/hive/warehouse/test.db/users/part_ymd=jian"
     file_format_type = "orc"
     fs.defaultFS = "hdfs://hadoop121"
     parse_partition_from_path = false
     read_columns =[name,address]
   
     result_table_name = "users"
   
     # If you would like to get more information about how to configure seatunnel and see full list of source plugins,
     # please go to https://seatunnel.apache.org/docs/category/source-v2
   }
   
   
   # #
   #   Hive {
   #     table_name = "test.users"
   #     metastore_uri = "thrift://hadoop121:9083"
   #   }
   
   #     Jdbc {
   #         
   #         query = """
   #                    SELECT ID, CATEGORY_ID, PRODUCT_ID, RANK, AVG_RANK_7, AVG_RANK_7_UP,
   #                            AVG_RANK_7_INCREASE, CREATE_USER_NAME,TO_CHAR(CREATE_DATETIME, 'YYYY-MM-DD')as CREATE_DATETIME,
   #                            UPDATE_USER_NAME, UPDATE_DATETIME
   #                    FROM product_selection_aliexpress_rank
   #
   #         """
   # #         query = """SELECT  PRO_ID,PRO_BOUGHT,CATE_1,TO_CHAR(SYNC_DATETIME, 'YYYY-MM-DD')  as SYNC_DATETIME FROM ( SELECT PRO_ID,PRO_BOUGHT,CATE_1,SYNC_DATETIME FROM SMT_GET_PRODUCT ORDER BY UPDATE_TIME ) WHERE ROWNUM <= 100 """
   # #           query ="SELECT  PRO_ID,PRO_BOUGHT,CATE_1,TO_CHAR(SYNC_DATETIME, 'YYYY-MM-DD') as SYNC_DATETIME FROM SMT_GET_PRODUCT where TO_CHAR(SYNC_DATETIME, 'YYYY-MM-DD')='2024-01-16'"
   #     }
   }
   
   transform{
   
       Filter {
           source_table_name = "users"
           fields = [name]
           result_table_name = "fake_name"
       }
       Filter {
           source_table_name = "users"
           fields = [name,address]
           result_table_name = "users_all"
       }
   
   
   
   }
   
   
   
   
   sink {
   
   # 测试是否支持事务,初步测试不支持 20240120
   #    jdbc {
   #       
   #         auto_commit= false
   #         query = """
   #           start transaction ;
   #           delete from test.users where name = 'jian' ;
   #           insert into test.users(name,address) values(?,?) ;
   #           COMMIT;
   #            """
   #
   #         }
   
   # 分两步执行,第一步删除,第二步插入
       jdbc {
              
               
               query = """
                 delete from test.users where name = 'jian'
                  """
               }
   #       console {
   #         source_table_name = "users_all"
   #       }
   #
   #         console {
   #         source_table_name = "fake_name"
   #       }
       jdbc {
             
   
               query = """
                 insert into test.users(name,address) values(?,?)
                  """
               }
   
   
   
       }
   
   
   
   # create table test.sink_user
   # (  name string,age int)
   # comment '测试seatunnel hive'
   # ROW FORMAT DELIMITED FIELDS TERMINATED BY '#'
   # STORED AS orc ;
   ```
   
   
   ### Running Command
   
   ```shell
   /usr/local/apache-seatunnel-2.3.3/bin/seatunnel.sh --config ods_pdc_product_selection_aliexpress_rank.conf
   ```
   
   
   ### Error Exception
   
   ```log
   2024-01-20 15:25:23,458 INFO  org.apache.seatunnel.engine.client.job.ClientJobProxy - Submit job finished, job id: 800989361205149698, job name: SeaTunnel
   2024-01-20 15:25:23,466 WARN  org.apache.seatunnel.engine.client.job.JobMetricsRunner - Failed to get job metrics summary, it maybe first-run
   2024-01-20 15:26:23,624 INFO  org.apache.seatunnel.engine.client.job.JobMetricsRunner - 
   ***********************************************
              Job Progress Information
   ***********************************************
   Job Id                    :  800989361205149698
   Read Count So Far         :                   1
   Write Count So Far        :                   1
   Average Read Count        :                 0/s
   Average Write Count       :                 0/s
   Last Statistic Time       : 2024-01-20 15:25:23
   Current Statistic Time    : 2024-01-20 15:26:23
   ***********************************************
   
   2024-01-20 15:27:23,593 INFO  org.apache.seatunnel.engine.client.job.JobMetricsRunner - 
   ***********************************************
              Job Progress Information
   ***********************************************
   Job Id                    :  800989361205149698
   Read Count So Far         :                   2
   Write Count So Far        :                   2
   Average Read Count        :                 0/s
   Average Write Count       :                 0/s
   Last Statistic Time       : 2024-01-20 15:26:23
   Current Statistic Time    : 2024-01-20 15:27:23
   ***********************************************
   
   2024-01-20 15:28:23,560 INFO  org.apache.seatunnel.engine.client.job.JobMetricsRunner - 
   ***********************************************
              Job Progress Information
   ***********************************************
   Job Id                    :  800989361205149698
   Read Count So Far         :                   2
   Write Count So Far        :                   2
   Average Read Count        :                 0/s
   Average Write Count       :                 0/s
   Last Statistic Time       : 2024-01-20 15:27:23
   Current Statistic Time    : 2024-01-20 15:28:23
   ***********************************************
   ```
   
   
   ### Zeta or Flink or Spark Version
   
   _No response_
   
   ### Java or Scala Version
   
   _No response_
   
   ### Screenshots
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [jdbc write] the job did not end, running every minutes [seatunnel]

Posted by "liugddx (via GitHub)" <gi...@apache.org>.
liugddx commented on issue #6260:
URL: https://github.com/apache/seatunnel/issues/6260#issuecomment-1903241729

   set `parallelism = 1` and try again.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [jdbc write] the job did not end, running every minutes [seatunnel]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6260:
URL: https://github.com/apache/seatunnel/issues/6260#issuecomment-1958439994

   This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [jdbc write] the job did not end, running every minutes [seatunnel]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6260:
URL: https://github.com/apache/seatunnel/issues/6260#issuecomment-1972206092

   This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [jdbc write] the job did not end, running every minutes [seatunnel]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #6260: [Bug] [jdbc write] the job did not end, running every minutes
URL: https://github.com/apache/seatunnel/issues/6260


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org