You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2021/11/24 00:39:19 UTC
[GitHub] [dolphinscheduler] mchcz opened a new issue #6973: [Bug] [json split] json split issue
mchcz opened a new issue #6973:
URL: https://github.com/apache/dolphinscheduler/issues/6973
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues.
### What happened
**BACKGROUND:** The current version is upgraded from 1.0.2 to 1.3.5, and now the test is upgraded from 1.3.5 to 2.0.0
**Q1:** Some older versions of the generated jason do not have **tenantId** fields
Code: processDefinition.setTenantId (jsonObject.get ("tenantId"));asInt();
**Q2:** Some older versions of the generated json do not have **description** fields
Code: taskDefinitionLog.setDescription ("description").asText());
**Q3:** Some older versions of the generated json do not have **workerGroup** field
Code: taskDefinitionLog.setWorkerGroup ("workergroup").asText();
**Q4:** Dependencies in a workflow that do not exist in the source workflow (dependency disabled, deleted in the source workflow) are abnormal.
Code: long taskCode , "ALL".equals (depTasks)? 0L : processCodeTaskNameCodeEntry.getValue().get(depTasks);
**Q5:** When the json split is complete, look at the workflow dependency information, the code is shown on the page and the workflow cannot be selected, and the log is reported as duplicate key
Error information: Duplicate key (through reference chain: org.apache.dolphinscheduler.dao.entity.TaskDefinitionLog."taskParamMap")
### What you expected to happen
It can be upgraded to version 2.0.0 normally
### How to reproduce
Described in "What happened"
### Anything else
_No response_
### Version
2.0.0-alpha
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] CalvinKirs closed issue #6973: [Bug] [json split] json split issue
Posted by GitBox <gi...@apache.org>.
CalvinKirs closed issue #6973:
URL: https://github.com/apache/dolphinscheduler/issues/6973
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] Jiezhi edited a comment on issue #6973: [Bug] [json split] json split issue
Posted by GitBox <gi...@apache.org>.
Jiezhi edited a comment on issue #6973:
URL: https://github.com/apache/dolphinscheduler/issues/6973#issuecomment-979779210
When upgrade db `sh ./script/create-dolphinscheduler.sh` from 1.3.7 to 2.0.0-alpha
```java
15:29:55.114 [main] INFO org.apache.dolphinscheduler.common.utils.ScriptRunner - sql: alter table t_ds_process_definition modify `description` text COMMENT 'description' after `version`
15:29:55.167 [main] INFO org.apache.dolphinscheduler.common.utils.ScriptRunner - sql: alter table t_ds_process_definition modify `release_state` tinyint(4) DEFAULT NULL COMMENT 'process definition release state???0:offline,1:online' after `project_code`
15:29:55.208 [main] INFO org.apache.dolphinscheduler.common.utils.ScriptRunner - sql: alter table t_ds_process_definition modify `create_time` datetime DEFAULT NULL COMMENT 'create time' after `tenant_id`
15:29:55.256 [main] INFO org.apache.dolphinscheduler.dao.upgrade.UpgradeDao - sqlSQLFilePath/opt/dolphinscheduler/apache-dolphinscheduler-2.0.0-bin/sql/upgrade/2.0.0_schema/mysql/dolphinscheduler_dml.sql
15:29:55.276 [main] DEBUG com.alibaba.druid.pool.PreparedStatementPool - stmt enter cache
15:29:55.278 [main] DEBUG com.alibaba.druid.pool.PreparedStatementPool - stmt enter cache
15:29:55.302 [main] DEBUG com.alibaba.druid.pool.PreparedStatementPool - stmt enter cache
15:29:55.303 [main] DEBUG com.alibaba.druid.pool.PreparedStatementPool - stmt enter cache
15:29:55.435 [main] DEBUG com.alibaba.druid.pool.PreparedStatementPool - stmt enter cache
15:29:55.454 [main] DEBUG com.alibaba.druid.pool.PreparedStatementPool - stmt enter cache
15:29:55.530 [main] DEBUG com.alibaba.druid.pool.PreparedStatementPool - stmt enter cache
15:29:55.793 [main] ERROR org.apache.dolphinscheduler.dao.upgrade.UpgradeDao - json split error
java.lang.NullPointerException: null
at org.apache.dolphinscheduler.dao.upgrade.UpgradeDao.convertDependence(UpgradeDao.java:792)
at org.apache.dolphinscheduler.dao.upgrade.UpgradeDao.processDefinitionJsonSplit(UpgradeDao.java:608)
at org.apache.dolphinscheduler.dao.upgrade.UpgradeDao.upgradeDolphinSchedulerTo200(UpgradeDao.java:322)
at org.apache.dolphinscheduler.dao.upgrade.DolphinSchedulerManager.upgradeDolphinScheduler(DolphinSchedulerManager.java:122)
at org.apache.dolphinscheduler.dao.upgrade.shell.CreateDolphinScheduler.main(CreateDolphinScheduler.java:41)
15:29:55.794 [main] INFO org.apache.dolphinscheduler.dao.upgrade.UpgradeDao - dolphinscheduler
15:29:55.794 [main] INFO org.apache.dolphinscheduler.common.utils.ScriptRunner - sql: /* * Licensed toxxxx license. */
alter table t_ds_process_definition drop primary key, ADD PRIMARY KEY (`id`,`code`)
15:29:55.827 [main] INFO org.apache.dolphinscheduler.common.utils.ScriptRunner - sql: ALTER TABLE t_ds_process_definition drop KEY `process_definition_unique`
15:29:55.837 [main] INFO org.apache.dolphinscheduler.common.utils.ScriptRunner - sql: ALTER TABLE t_ds_process_definition drop KEY `process_definition_index`
15:29:55.845 [main] INFO org.apache.dolphinscheduler.common.utils.ScriptRunner - sql: alter table t_ds_process_definition drop process_definition_json
15:29:55.866 [main] INFO org.apache.dolphinscheduler.common.utils.ScriptRunner - sql: alter table t_ds_process_definition drop connects
15:29:55.887 [main] INFO org.apache.dolphinscheduler.common.utils.ScriptRunner - sql: alter table t_ds_process_definition drop receivers
15:29:55.907 [main] INFO org.apache.dolphinscheduler.common.utils.ScriptRunner - sql: alter table t_ds_process_definition drop receivers_cc
15:29:55.927 [main] INFO org.apache.dolphinscheduler.common.utils.ScriptRunner - sql: alter table t_ds_process_definition drop modify_by
15:29:55.945 [main] INFO org.apache.dolphinscheduler.common.utils.ScriptRunner - sql: alter table t_ds_process_definition drop resource_ids
15:29:55.974 [main] INFO org.apache.dolphinscheduler.dao.upgrade.shell.CreateDolphinScheduler - upgrade DolphinScheduler finished
15:29:55.974 [main] INFO org.apache.dolphinscheduler.dao.upgrade.shell.CreateDolphinScheduler - create DolphinScheduler success
```
It seems upgrade table t_ds_process_definition failed, and the api of `process-definition` failed to work too!
```java
[INFO] 2021-11-26 16:13:54.023 org.apache.dolphinscheduler.api.aspect.AccessLogAspect:[76] - REQUEST TRANCE_ID:bca3655b-922c-4f82-a424-f383dede1098, LOGIN_USER:admin, URI:/dolphinscheduler/projects/3645618595296/process-definition, METHOD:GET, HANDLER:org.apache.dolphinscheduler.api.controller.ProcessDefinitionController.queryProcessDefinitionListPaging, ARGS:{searchVal=, projectCode=3645618595296, pageNo=1, pageSize=10, userId=0}
[ERROR] 2021-11-26 16:13:54.042 org.apache.dolphinscheduler.api.exceptions.ApiExceptionHandler:[46] - 分页查询工作流定义列表错误
java.lang.NullPointerException: null
at org.apache.dolphinscheduler.api.service.impl.ProcessDefinitionServiceImpl.queryProcessDefinitionListPaging(ProcessDefinitionServiceImpl.java:411)
at org.apache.dolphinscheduler.api.service.impl.ProcessDefinitionServiceImpl$$FastClassBySpringCGLIB$$e8e34ed9.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:687)
at org.apache.dolphinscheduler.api.service.impl.ProcessDefinitionServiceImpl$$EnhancerBySpringCGLIB$$d9e8ca3d.queryProcessDefinitionListPaging(<generated>)
at org.apache.dolphinscheduler.api.controller.ProcessDefinitionController.queryProcessDefinitionListPaging(ProcessDefinitionController.java:493)
at org.apache.dolphinscheduler.api.controller.ProcessDefinitionController$$FastClassBySpringCGLIB$$dc9bf5db.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:752)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:88)
at org.apache.dolphinscheduler.api.aspect.AccessLogAspect.doAround(AccessLogAspect.java:87)
at sun.reflect.GeneratedMethodAccessor156.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:644)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:633)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] JinyLeeChina commented on issue #6973: [Bug] [json split] json split issue
Posted by GitBox <gi...@apache.org>.
JinyLeeChina commented on issue #6973:
URL: https://github.com/apache/dolphinscheduler/issues/6973#issuecomment-978797786
OK, let me check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] Jiezhi commented on issue #6973: [Bug] [json split] json split issue
Posted by GitBox <gi...@apache.org>.
Jiezhi commented on issue #6973:
URL: https://github.com/apache/dolphinscheduler/issues/6973#issuecomment-979779210
When upgrade db `sh ./script/create-dolphinscheduler.sh` from 1.3.7 to 2.0.0-alpha
```java
15:29:55.114 [main] INFO org.apache.dolphinscheduler.common.utils.ScriptRunner - sql: alter table t_ds_process_definition modify `description` text COMMENT 'description' after `version`
15:29:55.167 [main] INFO org.apache.dolphinscheduler.common.utils.ScriptRunner - sql: alter table t_ds_process_definition modify `release_state` tinyint(4) DEFAULT NULL COMMENT 'process definition release state???0:offline,1:online' after `project_code`
15:29:55.208 [main] INFO org.apache.dolphinscheduler.common.utils.ScriptRunner - sql: alter table t_ds_process_definition modify `create_time` datetime DEFAULT NULL COMMENT 'create time' after `tenant_id`
15:29:55.256 [main] INFO org.apache.dolphinscheduler.dao.upgrade.UpgradeDao - sqlSQLFilePath/opt/dolphinscheduler/apache-dolphinscheduler-2.0.0-bin/sql/upgrade/2.0.0_schema/mysql/dolphinscheduler_dml.sql
15:29:55.276 [main] DEBUG com.alibaba.druid.pool.PreparedStatementPool - stmt enter cache
15:29:55.278 [main] DEBUG com.alibaba.druid.pool.PreparedStatementPool - stmt enter cache
15:29:55.302 [main] DEBUG com.alibaba.druid.pool.PreparedStatementPool - stmt enter cache
15:29:55.303 [main] DEBUG com.alibaba.druid.pool.PreparedStatementPool - stmt enter cache
15:29:55.435 [main] DEBUG com.alibaba.druid.pool.PreparedStatementPool - stmt enter cache
15:29:55.454 [main] DEBUG com.alibaba.druid.pool.PreparedStatementPool - stmt enter cache
15:29:55.530 [main] DEBUG com.alibaba.druid.pool.PreparedStatementPool - stmt enter cache
15:29:55.793 [main] ERROR org.apache.dolphinscheduler.dao.upgrade.UpgradeDao - json split error
java.lang.NullPointerException: null
at org.apache.dolphinscheduler.dao.upgrade.UpgradeDao.convertDependence(UpgradeDao.java:792)
at org.apache.dolphinscheduler.dao.upgrade.UpgradeDao.processDefinitionJsonSplit(UpgradeDao.java:608)
at org.apache.dolphinscheduler.dao.upgrade.UpgradeDao.upgradeDolphinSchedulerTo200(UpgradeDao.java:322)
at org.apache.dolphinscheduler.dao.upgrade.DolphinSchedulerManager.upgradeDolphinScheduler(DolphinSchedulerManager.java:122)
at org.apache.dolphinscheduler.dao.upgrade.shell.CreateDolphinScheduler.main(CreateDolphinScheduler.java:41)
15:29:55.794 [main] INFO org.apache.dolphinscheduler.dao.upgrade.UpgradeDao - dolphinscheduler
15:29:55.794 [main] INFO org.apache.dolphinscheduler.common.utils.ScriptRunner - sql: /* * Licensed to the Apache Software Foundation (ASF) under one or more * contributor license agreements. See the NOTICE file distributed with * this work for additional information regarding copyright ownership. * The ASF licenses this file to You under the Apache License, Version 2.0 * (the "License"); you may not use this file except in compliance with * the License. You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ alter table t_ds_process_definition drop primary key, ADD PRIMARY KEY (`id`,`code`)
15:29:55.827 [main] INFO org.apache.dolphinscheduler.common.utils.ScriptRunner - sql: ALTER TABLE t_ds_process_definition drop KEY `process_definition_unique`
15:29:55.837 [main] INFO org.apache.dolphinscheduler.common.utils.ScriptRunner - sql: ALTER TABLE t_ds_process_definition drop KEY `process_definition_index`
15:29:55.845 [main] INFO org.apache.dolphinscheduler.common.utils.ScriptRunner - sql: alter table t_ds_process_definition drop process_definition_json
15:29:55.866 [main] INFO org.apache.dolphinscheduler.common.utils.ScriptRunner - sql: alter table t_ds_process_definition drop connects
15:29:55.887 [main] INFO org.apache.dolphinscheduler.common.utils.ScriptRunner - sql: alter table t_ds_process_definition drop receivers
15:29:55.907 [main] INFO org.apache.dolphinscheduler.common.utils.ScriptRunner - sql: alter table t_ds_process_definition drop receivers_cc
15:29:55.927 [main] INFO org.apache.dolphinscheduler.common.utils.ScriptRunner - sql: alter table t_ds_process_definition drop modify_by
15:29:55.945 [main] INFO org.apache.dolphinscheduler.common.utils.ScriptRunner - sql: alter table t_ds_process_definition drop resource_ids
15:29:55.974 [main] INFO org.apache.dolphinscheduler.dao.upgrade.shell.CreateDolphinScheduler - upgrade DolphinScheduler finished
15:29:55.974 [main] INFO org.apache.dolphinscheduler.dao.upgrade.shell.CreateDolphinScheduler - create DolphinScheduler success
```
It seems upgrade table t_ds_process_definition failed, and the api of `process-definition` failed to work too!
```java
[INFO] 2021-11-26 16:13:54.023 org.apache.dolphinscheduler.api.aspect.AccessLogAspect:[76] - REQUEST TRANCE_ID:bca3655b-922c-4f82-a424-f383dede1098, LOGIN_USER:admin, URI:/dolphinscheduler/projects/3645618595296/process-definition, METHOD:GET, HANDLER:org.apache.dolphinscheduler.api.controller.ProcessDefinitionController.queryProcessDefinitionListPaging, ARGS:{searchVal=, projectCode=3645618595296, pageNo=1, pageSize=10, userId=0}
[ERROR] 2021-11-26 16:13:54.042 org.apache.dolphinscheduler.api.exceptions.ApiExceptionHandler:[46] - 分页查询工作流定义列表错误
java.lang.NullPointerException: null
at org.apache.dolphinscheduler.api.service.impl.ProcessDefinitionServiceImpl.queryProcessDefinitionListPaging(ProcessDefinitionServiceImpl.java:411)
at org.apache.dolphinscheduler.api.service.impl.ProcessDefinitionServiceImpl$$FastClassBySpringCGLIB$$e8e34ed9.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:687)
at org.apache.dolphinscheduler.api.service.impl.ProcessDefinitionServiceImpl$$EnhancerBySpringCGLIB$$d9e8ca3d.queryProcessDefinitionListPaging(<generated>)
at org.apache.dolphinscheduler.api.controller.ProcessDefinitionController.queryProcessDefinitionListPaging(ProcessDefinitionController.java:493)
at org.apache.dolphinscheduler.api.controller.ProcessDefinitionController$$FastClassBySpringCGLIB$$dc9bf5db.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:752)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:88)
at org.apache.dolphinscheduler.api.aspect.AccessLogAspect.doAround(AccessLogAspect.java:87)
at sun.reflect.GeneratedMethodAccessor156.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:644)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:633)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] CalvinKirs commented on issue #6973: [Bug] [json split] json split issue
Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on issue #6973:
URL: https://github.com/apache/dolphinscheduler/issues/6973#issuecomment-982366882
fixd #7030
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #6973: [Bug] [json split] json split issue
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #6973:
URL: https://github.com/apache/dolphinscheduler/issues/6973#issuecomment-977320382
Hi:
* Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
* If you haven't received a reply for a long time, you can subscribe to the developer's email，Mail subscription steps reference https://dolphinscheduler.apache.org/en-us/community/development/subscribe.html ,Then write the issue URL in the email content and send question to dev@dolphinscheduler.apache.org.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org