You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@servicecomb.apache.org by Willem Jiang <wi...@gmail.com> on 2019/05/01 00:38:32 UTC
Re: [saga-actuator]consider about adding StreamBasedSaga engine to
integrate with shardingsphere
Willem Jiang
Twitter: willemjiang
Weibo: 姜宁willem
On Mon, Apr 29, 2019 at 7:58 PM zhaojun <zh...@126.com> wrote:
>
> Hi, Willem
>
> Saga actuator could not compatible with ShardingSphere currently, there are 2 main problem exist.
> 1. If sql execute in saga transport instead of ShardingSphere, we can not see the result of logic-sql even in one transaction, it is like this:
> update t_order set status=’start’ where usr_id=’tom’;
> select status from t_order where usr_id=’tom’; —> we can’t query ’start’ record as actuator haven’t executed.
> Insert into t_order values(?,?,?) ;
I think we can break the whole SQL interactions into smaller steps,
and let saga-actuator do the job one by one.
In this case, you can break the whole calling graph into several sub
calling graphs.
Any thought?
> 2. If logic-sql execute in ShardingSphere, we cannot handle recovery before submit graph result, as event log only wrote at saga engine.
>
> so we should integrate saga-transaction like omega send event log step by step.
> It is better to make every instance do recovery Independently, instead of providing another coordinator center service.
> I feel that embed saga should have following capability.
> 1).It can provide service in jar package independent
> 2).each embed saga only recovery their own transaction-data of this instance.
> If one instance crashed, we can introduce external service to do failover.
>
> so we consider about exending saga-acutator, if it can support submit task step by step in one transaction, it is a good choice.
> of course, there have many things we should do.
>
> ------------------
> Zhao Jun
> Apache Sharding-Sphere & ServiceComb
>
> > On Apr 29, 2019, at 5:17 PM, Willem Jiang <wi...@gmail.com> wrote:
> >
> > First Saga actuator need to build up the calling grapha before sending
> > out the request. I don't think you can do the step by step SQL
> > invocation with Saga actuator.
> > If you want to call the SQL execution step by step , you may need to
> > switch to ServiceComb Pack project which has a coordinator to take
> > care of the distributed transaction. But that introduce another
> > endpoint(Alpha) to shardingsphere.
> >
> > From my understanding, Saga actuator is most efficient way to execute
> > the SQL across different data nodes.
> >
> >
> > Willem Jiang
> >
> > Twitter: willemjiang
> > Weibo: 姜宁willem
> >
> > On Fri, Apr 26, 2019 at 6:41 PM zhaojun <zh...@126.com> wrote:
> >>
> >> Hi, all
> >>
> >> currently, we have integrated with saga using graph based engine in shardingsphere[1]
> >> it need us to collect all participated actual SQL, then submit to saga actuator in commit/rollback phase.
> >> if application crashed before invoking saga actuator, undo log of branch transaction SQL will not be saved,
> >> so recovery thread will not be executed correctly.
> >>
> >> it's better that encapsulating every actual SQL as a saga task in shardingsphere side,
> >> then submit to saga actuator realtime instead of batch processing all the SQLs at commit/rollback phase.
> >> this architecture will make the boundary more clear between shardingsphre and saga, currently we have done some additional works for integrating saga.
> >>
> >> any thought?
> >>
> >> [1]:https://github.com/sharding-sphere/shardingsphere-spi-impl/tree/master/sharding-transaction-spi-impl/sharding-transaction-base-spi-impl/sharding-transaction-base-saga <https://github.com/sharding-sphere/shardingsphere-spi-impl/tree/master/sharding-transaction-spi-impl/sharding-transaction-base-spi-impl/sharding-transaction-base-saga>
> >>
> >>
> >>
> >> ------------------
> >> Zhao Jun
> >> Apache Sharding-Sphere & ServiceComb
> >>
>
Re: [saga-actuator]consider about adding StreamBasedSaga engine to
integrate with shardingsphere
Posted by Sheng Wu <wu...@gmail.com>.
Feedback from our Seata and SkyWalking integration.
We will provide plugins for there TC and TM.
Also, Seata team has the plan to distribute their binary including
SkyWalking binary tar in TC, for user convenience.
Then users could use a single command in TC to open SkyWalking agent
working with the globally distributed tracing system, and benefits from
SkyWalking other monitoring/analysis capabilities.
Sheng Wu 吴晟
Apache SkyWalking, ShardingSphere, Zipkin
Twitter, wusheng1108
Willem Jiang <wi...@gmail.com> 于2019年5月1日周三 上午8:44写道：
> Willem Jiang
>
> Twitter: willemjiang
> Weibo: 姜宁willem
>
> On Mon, Apr 29, 2019 at 7:58 PM zhaojun <zh...@126.com> wrote:
> >
> > Hi, Willem
> >
> > Saga actuator could not compatible with ShardingSphere currently, there
> are 2 main problem exist.
> > 1. If sql execute in saga transport instead of ShardingSphere, we can
> not see the result of logic-sql even in one transaction, it is like this:
> > update t_order set status=’start’ where usr_id=’tom’;
> > select status from t_order where usr_id=’tom’;
> —> we can’t query ’start’ record as actuator haven’t executed.
> > Insert into t_order values(?,?,?) ;
>
> I think we can break the whole SQL interactions into smaller steps,
> and let saga-actuator do the job one by one.
> In this case, you can break the whole calling graph into several sub
> calling graphs.
> Any thought?
>
> > 2. If logic-sql execute in ShardingSphere, we cannot handle recovery
> before submit graph result, as event log only wrote at saga engine.
> >
> > so we should integrate saga-transaction like omega send event log step
> by step.
> > It is better to make every instance do recovery Independently, instead
> of providing another coordinator center service.
> > I feel that embed saga should have following capability.
> > 1).It can provide service in jar package independent
> > 2).each embed saga only recovery their own transaction-data of this
> instance.
> > If one instance crashed, we can introduce external service to do
> failover.
> >
> > so we consider about exending saga-acutator, if it can support submit
> task step by step in one transaction, it is a good choice.
> > of course, there have many things we should do.
> >
> > ------------------
> > Zhao Jun
> > Apache Sharding-Sphere & ServiceComb
> >
> > > On Apr 29, 2019, at 5:17 PM, Willem Jiang <wi...@gmail.com>
> wrote:
> > >
> > > First Saga actuator need to build up the calling grapha before sending
> > > out the request. I don't think you can do the step by step SQL
> > > invocation with Saga actuator.
> > > If you want to call the SQL execution step by step , you may need to
> > > switch to ServiceComb Pack project which has a coordinator to take
> > > care of the distributed transaction. But that introduce another
> > > endpoint(Alpha) to shardingsphere.
> > >
> > > From my understanding, Saga actuator is most efficient way to execute
> > > the SQL across different data nodes.
> > >
> > >
> > > Willem Jiang
> > >
> > > Twitter: willemjiang
> > > Weibo: 姜宁willem
> > >
> > > On Fri, Apr 26, 2019 at 6:41 PM zhaojun <zh...@126.com> wrote:
> > >>
> > >> Hi, all
> > >>
> > >> currently, we have integrated with saga using graph based engine in
> shardingsphere[1]
> > >> it need us to collect all participated actual SQL, then submit to
> saga actuator in commit/rollback phase.
> > >> if application crashed before invoking saga actuator, undo log of
> branch transaction SQL will not be saved,
> > >> so recovery thread will not be executed correctly.
> > >>
> > >> it's better that encapsulating every actual SQL as a saga task in
> shardingsphere side,
> > >> then submit to saga actuator realtime instead of batch processing all
> the SQLs at commit/rollback phase.
> > >> this architecture will make the boundary more clear between
> shardingsphre and saga, currently we have done some additional works for
> integrating saga.
> > >>
> > >> any thought?
> > >>
> > >> [1]:
> https://github.com/sharding-sphere/shardingsphere-spi-impl/tree/master/sharding-transaction-spi-impl/sharding-transaction-base-spi-impl/sharding-transaction-base-saga
> <
> https://github.com/sharding-sphere/shardingsphere-spi-impl/tree/master/sharding-transaction-spi-impl/sharding-transaction-base-spi-impl/sharding-transaction-base-saga
> >
> > >>
> > >>
> > >>
> > >> ------------------
> > >> Zhao Jun
> > >> Apache Sharding-Sphere & ServiceComb
> > >>
> >
>
Re: [saga-actuator]consider about adding StreamBasedSaga engine to
integrate with shardingsphere
Posted by Sheng Wu <wu...@gmail.com>.
Feedback from our Seata and SkyWalking integration.
We will provide plugins for there TC and TM.
Also, Seata team has the plan to distribute their binary including
SkyWalking binary tar in TC, for user convenience.
Then users could use a single command in TC to open SkyWalking agent
working with the globally distributed tracing system, and benefits from
SkyWalking other monitoring/analysis capabilities.
Sheng Wu 吴晟
Apache SkyWalking, ShardingSphere, Zipkin
Twitter, wusheng1108
Willem Jiang <wi...@gmail.com> 于2019年5月1日周三 上午8:44写道：
> Willem Jiang
>
> Twitter: willemjiang
> Weibo: 姜宁willem
>
> On Mon, Apr 29, 2019 at 7:58 PM zhaojun <zh...@126.com> wrote:
> >
> > Hi, Willem
> >
> > Saga actuator could not compatible with ShardingSphere currently, there
> are 2 main problem exist.
> > 1. If sql execute in saga transport instead of ShardingSphere, we can
> not see the result of logic-sql even in one transaction, it is like this:
> > update t_order set status=’start’ where usr_id=’tom’;
> > select status from t_order where usr_id=’tom’;
> —> we can’t query ’start’ record as actuator haven’t executed.
> > Insert into t_order values(?,?,?) ;
>
> I think we can break the whole SQL interactions into smaller steps,
> and let saga-actuator do the job one by one.
> In this case, you can break the whole calling graph into several sub
> calling graphs.
> Any thought?
>
> > 2. If logic-sql execute in ShardingSphere, we cannot handle recovery
> before submit graph result, as event log only wrote at saga engine.
> >
> > so we should integrate saga-transaction like omega send event log step
> by step.
> > It is better to make every instance do recovery Independently, instead
> of providing another coordinator center service.
> > I feel that embed saga should have following capability.
> > 1).It can provide service in jar package independent
> > 2).each embed saga only recovery their own transaction-data of this
> instance.
> > If one instance crashed, we can introduce external service to do
> failover.
> >
> > so we consider about exending saga-acutator, if it can support submit
> task step by step in one transaction, it is a good choice.
> > of course, there have many things we should do.
> >
> > ------------------
> > Zhao Jun
> > Apache Sharding-Sphere & ServiceComb
> >
> > > On Apr 29, 2019, at 5:17 PM, Willem Jiang <wi...@gmail.com>
> wrote:
> > >
> > > First Saga actuator need to build up the calling grapha before sending
> > > out the request. I don't think you can do the step by step SQL
> > > invocation with Saga actuator.
> > > If you want to call the SQL execution step by step , you may need to
> > > switch to ServiceComb Pack project which has a coordinator to take
> > > care of the distributed transaction. But that introduce another
> > > endpoint(Alpha) to shardingsphere.
> > >
> > > From my understanding, Saga actuator is most efficient way to execute
> > > the SQL across different data nodes.
> > >
> > >
> > > Willem Jiang
> > >
> > > Twitter: willemjiang
> > > Weibo: 姜宁willem
> > >
> > > On Fri, Apr 26, 2019 at 6:41 PM zhaojun <zh...@126.com> wrote:
> > >>
> > >> Hi, all
> > >>
> > >> currently, we have integrated with saga using graph based engine in
> shardingsphere[1]
> > >> it need us to collect all participated actual SQL, then submit to
> saga actuator in commit/rollback phase.
> > >> if application crashed before invoking saga actuator, undo log of
> branch transaction SQL will not be saved,
> > >> so recovery thread will not be executed correctly.
> > >>
> > >> it's better that encapsulating every actual SQL as a saga task in
> shardingsphere side,
> > >> then submit to saga actuator realtime instead of batch processing all
> the SQLs at commit/rollback phase.
> > >> this architecture will make the boundary more clear between
> shardingsphre and saga, currently we have done some additional works for
> integrating saga.
> > >>
> > >> any thought?
> > >>
> > >> [1]:
> https://github.com/sharding-sphere/shardingsphere-spi-impl/tree/master/sharding-transaction-spi-impl/sharding-transaction-base-spi-impl/sharding-transaction-base-saga
> <
> https://github.com/sharding-sphere/shardingsphere-spi-impl/tree/master/sharding-transaction-spi-impl/sharding-transaction-base-spi-impl/sharding-transaction-base-saga
> >
> > >>
> > >>
> > >>
> > >> ------------------
> > >> Zhao Jun
> > >> Apache Sharding-Sphere & ServiceComb
> > >>
> >
>
Re: [saga-actuator]consider about adding StreamBasedSaga engine to
integrate with shardingsphere
Posted by zhaojun <zh...@126.com>.
I have just created a issue[1] in JIRA
[1]: https://issues.apache.org/jira/browse/SCB-1276 <https://issues.apache.org/jira/browse/SCB-1276>
------------------
Zhao Jun
Apache Sharding-Sphere & ServiceComb
> On May 5, 2019, at 8:46 AM, zhaojun <zh...@126.com> wrote:
>
> OK, I’ll try to write a JIRA today.
>
> ------------------
> Zhao Jun
> Apache Sharding-Sphere & ServiceComb
>
>
>> On May 2, 2019, at 3:18 PM, Willem Jiang <wi...@gmail.com> wrote:
>>
>> It sounds good to me.
>>
>> Willem Jiang
>>
>> Twitter: willemjiang
>> Weibo: 姜宁willem
>>
>> On Thu, May 2, 2019 at 11:14 AM 赵俊 <zh...@jd.com> wrote:
>>>
>>> Let SQL executor running command one by one is good to me.
>>>
>>> One logic-sql will be routed to multi-actual SQLs, then executed in sharding-sphere execute-engine.
>>> A actual SQL will be wrapped as a SQL execute-callback in sharding-sphere.
>>> We could provide a SPI for this SQL execute-callback.
>>> For integrated with saga, we could implement a saga execute-callback, it will wrap event sourcing logic like saga-actuator task have done.
>>>
>>>
>>>> On May 1, 2019, at 5:06 PM, Willem Jiang <wi...@gmail.com> wrote:
>>>>
>>>> We could do some enhancement on it.
>>>> From my understanding, We could create a calling graph which has three
>>>> sub graph of it, or we could let the SQL executor running the command
>>>> one by one.
>>>> Any thoughts?
>>>>
>>>> Willem Jiang
>>>>
>>>> Twitter: willemjiang
>>>> Weibo: 姜宁willem
>>>>
>>>> On Wed, May 1, 2019 at 4:29 PM zhaojun <zh...@126.com> wrote:
>>>>>
>>>>> But saga-actuator don’t support 3 logic SQL is a global transaction while you submit 3 times separately.
>>>>>
>>>>> ------------------
>>>>> Zhao Jun
>>>>> Apache Sharding-Sphere & ServiceComb
>>>>>
>>>>>> On May 1, 2019, at 8:38 AM, Willem Jiang <wi...@gmail.com> wrote:
>>>>>>
>>>>>> Willem Jiang
>>>>>>
>>>>>> Twitter: willemjiang
>>>>>> Weibo: 姜宁willem
>>>>>>
>>>>>> On Mon, Apr 29, 2019 at 7:58 PM zhaojun <zh...@126.com> wrote:
>>>>>>>
>>>>>>> Hi, Willem
>>>>>>>
>>>>>>> Saga actuator could not compatible with ShardingSphere currently, there are 2 main problem exist.
>>>>>>> 1. If sql execute in saga transport instead of ShardingSphere, we can not see the result of logic-sql even in one transaction, it is like this:
>>>>>>> update t_order set status=’start’ where usr_id=’tom’;
>>>>>>> select status from t_order where usr_id=’tom’; —> we can’t query ’start’ record as actuator haven’t executed.
>>>>>>> Insert into t_order values(?,?,?) ;
>>>>>>
>>>>>> I think we can break the whole SQL interactions into smaller steps,
>>>>>> and let saga-actuator do the job one by one.
>>>>>> In this case, you can break the whole calling graph into several sub
>>>>>> calling graphs.
>>>>>> Any thought?
>>>>>>
>>>>>>> 2. If logic-sql execute in ShardingSphere, we cannot handle recovery before submit graph result, as event log only wrote at saga engine.
>>>>>>>
>>>>>>> so we should integrate saga-transaction like omega send event log step by step.
>>>>>>> It is better to make every instance do recovery Independently, instead of providing another coordinator center service.
>>>>>>> I feel that embed saga should have following capability.
>>>>>>> 1).It can provide service in jar package independent
>>>>>>> 2).each embed saga only recovery their own transaction-data of this instance.
>>>>>>> If one instance crashed, we can introduce external service to do failover.
>>>>>>>
>>>>>>> so we consider about exending saga-acutator, if it can support submit task step by step in one transaction, it is a good choice.
>>>>>>> of course, there have many things we should do.
>>>>>>>
>>>>>>> ------------------
>>>>>>> Zhao Jun
>>>>>>> Apache Sharding-Sphere & ServiceComb
>>>>>>>
>>>>>>>> On Apr 29, 2019, at 5:17 PM, Willem Jiang <wi...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> First Saga actuator need to build up the calling grapha before sending
>>>>>>>> out the request. I don't think you can do the step by step SQL
>>>>>>>> invocation with Saga actuator.
>>>>>>>> If you want to call the SQL execution step by step , you may need to
>>>>>>>> switch to ServiceComb Pack project which has a coordinator to take
>>>>>>>> care of the distributed transaction. But that introduce another
>>>>>>>> endpoint(Alpha) to shardingsphere.
>>>>>>>>
>>>>>>>> From my understanding, Saga actuator is most efficient way to execute
>>>>>>>> the SQL across different data nodes.
>>>>>>>>
>>>>>>>>
>>>>>>>> Willem Jiang
>>>>>>>>
>>>>>>>> Twitter: willemjiang
>>>>>>>> Weibo: 姜宁willem
>>>>>>>>
>>>>>>>> On Fri, Apr 26, 2019 at 6:41 PM zhaojun <zh...@126.com> wrote:
>>>>>>>>>
>>>>>>>>> Hi, all
>>>>>>>>>
>>>>>>>>> currently, we have integrated with saga using graph based engine in shardingsphere[1]
>>>>>>>>> it need us to collect all participated actual SQL, then submit to saga actuator in commit/rollback phase.
>>>>>>>>> if application crashed before invoking saga actuator, undo log of branch transaction SQL will not be saved,
>>>>>>>>> so recovery thread will not be executed correctly.
>>>>>>>>>
>>>>>>>>> it's better that encapsulating every actual SQL as a saga task in shardingsphere side,
>>>>>>>>> then submit to saga actuator realtime instead of batch processing all the SQLs at commit/rollback phase.
>>>>>>>>> this architecture will make the boundary more clear between shardingsphre and saga, currently we have done some additional works for integrating saga.
>>>>>>>>>
>>>>>>>>> any thought?
>>>>>>>>>
>>>>>>>>> [1]:https://github.com/sharding-sphere/shardingsphere-spi-impl/tree/master/sharding-transaction-spi-impl/sharding-transaction-base-spi-impl/sharding-transaction-base-saga <https://github.com/sharding-sphere/shardingsphere-spi-impl/tree/master/sharding-transaction-spi-impl/sharding-transaction-base-spi-impl/sharding-transaction-base-saga>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ------------------
>>>>>>>>> Zhao Jun
>>>>>>>>> Apache Sharding-Sphere & ServiceComb
>>>>>>>>>
>>>>>>>
>>>>>
>>>
>
Re: [saga-actuator]consider about adding StreamBasedSaga engine to
integrate with shardingsphere
Posted by zhaojun <zh...@126.com>.
I have just created a issue[1] in JIRA
[1]: https://issues.apache.org/jira/browse/SCB-1276 <https://issues.apache.org/jira/browse/SCB-1276>
------------------
Zhao Jun
Apache Sharding-Sphere & ServiceComb
> On May 5, 2019, at 8:46 AM, zhaojun <zh...@126.com> wrote:
>
> OK, I’ll try to write a JIRA today.
>
> ------------------
> Zhao Jun
> Apache Sharding-Sphere & ServiceComb
>
>
>> On May 2, 2019, at 3:18 PM, Willem Jiang <wi...@gmail.com> wrote:
>>
>> It sounds good to me.
>>
>> Willem Jiang
>>
>> Twitter: willemjiang
>> Weibo: 姜宁willem
>>
>> On Thu, May 2, 2019 at 11:14 AM 赵俊 <zh...@jd.com> wrote:
>>>
>>> Let SQL executor running command one by one is good to me.
>>>
>>> One logic-sql will be routed to multi-actual SQLs, then executed in sharding-sphere execute-engine.
>>> A actual SQL will be wrapped as a SQL execute-callback in sharding-sphere.
>>> We could provide a SPI for this SQL execute-callback.
>>> For integrated with saga, we could implement a saga execute-callback, it will wrap event sourcing logic like saga-actuator task have done.
>>>
>>>
>>>> On May 1, 2019, at 5:06 PM, Willem Jiang <wi...@gmail.com> wrote:
>>>>
>>>> We could do some enhancement on it.
>>>> From my understanding, We could create a calling graph which has three
>>>> sub graph of it, or we could let the SQL executor running the command
>>>> one by one.
>>>> Any thoughts?
>>>>
>>>> Willem Jiang
>>>>
>>>> Twitter: willemjiang
>>>> Weibo: 姜宁willem
>>>>
>>>> On Wed, May 1, 2019 at 4:29 PM zhaojun <zh...@126.com> wrote:
>>>>>
>>>>> But saga-actuator don’t support 3 logic SQL is a global transaction while you submit 3 times separately.
>>>>>
>>>>> ------------------
>>>>> Zhao Jun
>>>>> Apache Sharding-Sphere & ServiceComb
>>>>>
>>>>>> On May 1, 2019, at 8:38 AM, Willem Jiang <wi...@gmail.com> wrote:
>>>>>>
>>>>>> Willem Jiang
>>>>>>
>>>>>> Twitter: willemjiang
>>>>>> Weibo: 姜宁willem
>>>>>>
>>>>>> On Mon, Apr 29, 2019 at 7:58 PM zhaojun <zh...@126.com> wrote:
>>>>>>>
>>>>>>> Hi, Willem
>>>>>>>
>>>>>>> Saga actuator could not compatible with ShardingSphere currently, there are 2 main problem exist.
>>>>>>> 1. If sql execute in saga transport instead of ShardingSphere, we can not see the result of logic-sql even in one transaction, it is like this:
>>>>>>> update t_order set status=’start’ where usr_id=’tom’;
>>>>>>> select status from t_order where usr_id=’tom’; —> we can’t query ’start’ record as actuator haven’t executed.
>>>>>>> Insert into t_order values(?,?,?) ;
>>>>>>
>>>>>> I think we can break the whole SQL interactions into smaller steps,
>>>>>> and let saga-actuator do the job one by one.
>>>>>> In this case, you can break the whole calling graph into several sub
>>>>>> calling graphs.
>>>>>> Any thought?
>>>>>>
>>>>>>> 2. If logic-sql execute in ShardingSphere, we cannot handle recovery before submit graph result, as event log only wrote at saga engine.
>>>>>>>
>>>>>>> so we should integrate saga-transaction like omega send event log step by step.
>>>>>>> It is better to make every instance do recovery Independently, instead of providing another coordinator center service.
>>>>>>> I feel that embed saga should have following capability.
>>>>>>> 1).It can provide service in jar package independent
>>>>>>> 2).each embed saga only recovery their own transaction-data of this instance.
>>>>>>> If one instance crashed, we can introduce external service to do failover.
>>>>>>>
>>>>>>> so we consider about exending saga-acutator, if it can support submit task step by step in one transaction, it is a good choice.
>>>>>>> of course, there have many things we should do.
>>>>>>>
>>>>>>> ------------------
>>>>>>> Zhao Jun
>>>>>>> Apache Sharding-Sphere & ServiceComb
>>>>>>>
>>>>>>>> On Apr 29, 2019, at 5:17 PM, Willem Jiang <wi...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> First Saga actuator need to build up the calling grapha before sending
>>>>>>>> out the request. I don't think you can do the step by step SQL
>>>>>>>> invocation with Saga actuator.
>>>>>>>> If you want to call the SQL execution step by step , you may need to
>>>>>>>> switch to ServiceComb Pack project which has a coordinator to take
>>>>>>>> care of the distributed transaction. But that introduce another
>>>>>>>> endpoint(Alpha) to shardingsphere.
>>>>>>>>
>>>>>>>> From my understanding, Saga actuator is most efficient way to execute
>>>>>>>> the SQL across different data nodes.
>>>>>>>>
>>>>>>>>
>>>>>>>> Willem Jiang
>>>>>>>>
>>>>>>>> Twitter: willemjiang
>>>>>>>> Weibo: 姜宁willem
>>>>>>>>
>>>>>>>> On Fri, Apr 26, 2019 at 6:41 PM zhaojun <zh...@126.com> wrote:
>>>>>>>>>
>>>>>>>>> Hi, all
>>>>>>>>>
>>>>>>>>> currently, we have integrated with saga using graph based engine in shardingsphere[1]
>>>>>>>>> it need us to collect all participated actual SQL, then submit to saga actuator in commit/rollback phase.
>>>>>>>>> if application crashed before invoking saga actuator, undo log of branch transaction SQL will not be saved,
>>>>>>>>> so recovery thread will not be executed correctly.
>>>>>>>>>
>>>>>>>>> it's better that encapsulating every actual SQL as a saga task in shardingsphere side,
>>>>>>>>> then submit to saga actuator realtime instead of batch processing all the SQLs at commit/rollback phase.
>>>>>>>>> this architecture will make the boundary more clear between shardingsphre and saga, currently we have done some additional works for integrating saga.
>>>>>>>>>
>>>>>>>>> any thought?
>>>>>>>>>
>>>>>>>>> [1]:https://github.com/sharding-sphere/shardingsphere-spi-impl/tree/master/sharding-transaction-spi-impl/sharding-transaction-base-spi-impl/sharding-transaction-base-saga <https://github.com/sharding-sphere/shardingsphere-spi-impl/tree/master/sharding-transaction-spi-impl/sharding-transaction-base-spi-impl/sharding-transaction-base-saga>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ------------------
>>>>>>>>> Zhao Jun
>>>>>>>>> Apache Sharding-Sphere & ServiceComb
>>>>>>>>>
>>>>>>>
>>>>>
>>>
>
Re: [saga-actuator]consider about adding StreamBasedSaga engine to
integrate with shardingsphere
Posted by zhaojun <zh...@126.com>.
OK, I’ll try to write a JIRA today.
------------------
Zhao Jun
Apache Sharding-Sphere & ServiceComb
> On May 2, 2019, at 3:18 PM, Willem Jiang <wi...@gmail.com> wrote:
>
> It sounds good to me.
>
> Willem Jiang
>
> Twitter: willemjiang
> Weibo: 姜宁willem
>
> On Thu, May 2, 2019 at 11:14 AM 赵俊 <zh...@jd.com> wrote:
>>
>> Let SQL executor running command one by one is good to me.
>>
>> One logic-sql will be routed to multi-actual SQLs, then executed in sharding-sphere execute-engine.
>> A actual SQL will be wrapped as a SQL execute-callback in sharding-sphere.
>> We could provide a SPI for this SQL execute-callback.
>> For integrated with saga, we could implement a saga execute-callback, it will wrap event sourcing logic like saga-actuator task have done.
>>
>>
>>> On May 1, 2019, at 5:06 PM, Willem Jiang <wi...@gmail.com> wrote:
>>>
>>> We could do some enhancement on it.
>>> From my understanding, We could create a calling graph which has three
>>> sub graph of it, or we could let the SQL executor running the command
>>> one by one.
>>> Any thoughts?
>>>
>>> Willem Jiang
>>>
>>> Twitter: willemjiang
>>> Weibo: 姜宁willem
>>>
>>> On Wed, May 1, 2019 at 4:29 PM zhaojun <zh...@126.com> wrote:
>>>>
>>>> But saga-actuator don’t support 3 logic SQL is a global transaction while you submit 3 times separately.
>>>>
>>>> ------------------
>>>> Zhao Jun
>>>> Apache Sharding-Sphere & ServiceComb
>>>>
>>>>> On May 1, 2019, at 8:38 AM, Willem Jiang <wi...@gmail.com> wrote:
>>>>>
>>>>> Willem Jiang
>>>>>
>>>>> Twitter: willemjiang
>>>>> Weibo: 姜宁willem
>>>>>
>>>>> On Mon, Apr 29, 2019 at 7:58 PM zhaojun <zh...@126.com> wrote:
>>>>>>
>>>>>> Hi, Willem
>>>>>>
>>>>>> Saga actuator could not compatible with ShardingSphere currently, there are 2 main problem exist.
>>>>>> 1. If sql execute in saga transport instead of ShardingSphere, we can not see the result of logic-sql even in one transaction, it is like this:
>>>>>> update t_order set status=’start’ where usr_id=’tom’;
>>>>>> select status from t_order where usr_id=’tom’; —> we can’t query ’start’ record as actuator haven’t executed.
>>>>>> Insert into t_order values(?,?,?) ;
>>>>>
>>>>> I think we can break the whole SQL interactions into smaller steps,
>>>>> and let saga-actuator do the job one by one.
>>>>> In this case, you can break the whole calling graph into several sub
>>>>> calling graphs.
>>>>> Any thought?
>>>>>
>>>>>> 2. If logic-sql execute in ShardingSphere, we cannot handle recovery before submit graph result, as event log only wrote at saga engine.
>>>>>>
>>>>>> so we should integrate saga-transaction like omega send event log step by step.
>>>>>> It is better to make every instance do recovery Independently, instead of providing another coordinator center service.
>>>>>> I feel that embed saga should have following capability.
>>>>>> 1).It can provide service in jar package independent
>>>>>> 2).each embed saga only recovery their own transaction-data of this instance.
>>>>>> If one instance crashed, we can introduce external service to do failover.
>>>>>>
>>>>>> so we consider about exending saga-acutator, if it can support submit task step by step in one transaction, it is a good choice.
>>>>>> of course, there have many things we should do.
>>>>>>
>>>>>> ------------------
>>>>>> Zhao Jun
>>>>>> Apache Sharding-Sphere & ServiceComb
>>>>>>
>>>>>>> On Apr 29, 2019, at 5:17 PM, Willem Jiang <wi...@gmail.com> wrote:
>>>>>>>
>>>>>>> First Saga actuator need to build up the calling grapha before sending
>>>>>>> out the request. I don't think you can do the step by step SQL
>>>>>>> invocation with Saga actuator.
>>>>>>> If you want to call the SQL execution step by step , you may need to
>>>>>>> switch to ServiceComb Pack project which has a coordinator to take
>>>>>>> care of the distributed transaction. But that introduce another
>>>>>>> endpoint(Alpha) to shardingsphere.
>>>>>>>
>>>>>>> From my understanding, Saga actuator is most efficient way to execute
>>>>>>> the SQL across different data nodes.
>>>>>>>
>>>>>>>
>>>>>>> Willem Jiang
>>>>>>>
>>>>>>> Twitter: willemjiang
>>>>>>> Weibo: 姜宁willem
>>>>>>>
>>>>>>> On Fri, Apr 26, 2019 at 6:41 PM zhaojun <zh...@126.com> wrote:
>>>>>>>>
>>>>>>>> Hi, all
>>>>>>>>
>>>>>>>> currently, we have integrated with saga using graph based engine in shardingsphere[1]
>>>>>>>> it need us to collect all participated actual SQL, then submit to saga actuator in commit/rollback phase.
>>>>>>>> if application crashed before invoking saga actuator, undo log of branch transaction SQL will not be saved,
>>>>>>>> so recovery thread will not be executed correctly.
>>>>>>>>
>>>>>>>> it's better that encapsulating every actual SQL as a saga task in shardingsphere side,
>>>>>>>> then submit to saga actuator realtime instead of batch processing all the SQLs at commit/rollback phase.
>>>>>>>> this architecture will make the boundary more clear between shardingsphre and saga, currently we have done some additional works for integrating saga.
>>>>>>>>
>>>>>>>> any thought?
>>>>>>>>
>>>>>>>> [1]:https://github.com/sharding-sphere/shardingsphere-spi-impl/tree/master/sharding-transaction-spi-impl/sharding-transaction-base-spi-impl/sharding-transaction-base-saga <https://github.com/sharding-sphere/shardingsphere-spi-impl/tree/master/sharding-transaction-spi-impl/sharding-transaction-base-spi-impl/sharding-transaction-base-saga>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ------------------
>>>>>>>> Zhao Jun
>>>>>>>> Apache Sharding-Sphere & ServiceComb
>>>>>>>>
>>>>>>
>>>>
>>
Re: [saga-actuator]consider about adding StreamBasedSaga engine to
integrate with shardingsphere
Posted by zhaojun <zh...@126.com>.
OK, I’ll try to write a JIRA today.
------------------
Zhao Jun
Apache Sharding-Sphere & ServiceComb
> On May 2, 2019, at 3:18 PM, Willem Jiang <wi...@gmail.com> wrote:
>
> It sounds good to me.
>
> Willem Jiang
>
> Twitter: willemjiang
> Weibo: 姜宁willem
>
> On Thu, May 2, 2019 at 11:14 AM 赵俊 <zh...@jd.com> wrote:
>>
>> Let SQL executor running command one by one is good to me.
>>
>> One logic-sql will be routed to multi-actual SQLs, then executed in sharding-sphere execute-engine.
>> A actual SQL will be wrapped as a SQL execute-callback in sharding-sphere.
>> We could provide a SPI for this SQL execute-callback.
>> For integrated with saga, we could implement a saga execute-callback, it will wrap event sourcing logic like saga-actuator task have done.
>>
>>
>>> On May 1, 2019, at 5:06 PM, Willem Jiang <wi...@gmail.com> wrote:
>>>
>>> We could do some enhancement on it.
>>> From my understanding, We could create a calling graph which has three
>>> sub graph of it, or we could let the SQL executor running the command
>>> one by one.
>>> Any thoughts?
>>>
>>> Willem Jiang
>>>
>>> Twitter: willemjiang
>>> Weibo: 姜宁willem
>>>
>>> On Wed, May 1, 2019 at 4:29 PM zhaojun <zh...@126.com> wrote:
>>>>
>>>> But saga-actuator don’t support 3 logic SQL is a global transaction while you submit 3 times separately.
>>>>
>>>> ------------------
>>>> Zhao Jun
>>>> Apache Sharding-Sphere & ServiceComb
>>>>
>>>>> On May 1, 2019, at 8:38 AM, Willem Jiang <wi...@gmail.com> wrote:
>>>>>
>>>>> Willem Jiang
>>>>>
>>>>> Twitter: willemjiang
>>>>> Weibo: 姜宁willem
>>>>>
>>>>> On Mon, Apr 29, 2019 at 7:58 PM zhaojun <zh...@126.com> wrote:
>>>>>>
>>>>>> Hi, Willem
>>>>>>
>>>>>> Saga actuator could not compatible with ShardingSphere currently, there are 2 main problem exist.
>>>>>> 1. If sql execute in saga transport instead of ShardingSphere, we can not see the result of logic-sql even in one transaction, it is like this:
>>>>>> update t_order set status=’start’ where usr_id=’tom’;
>>>>>> select status from t_order where usr_id=’tom’; —> we can’t query ’start’ record as actuator haven’t executed.
>>>>>> Insert into t_order values(?,?,?) ;
>>>>>
>>>>> I think we can break the whole SQL interactions into smaller steps,
>>>>> and let saga-actuator do the job one by one.
>>>>> In this case, you can break the whole calling graph into several sub
>>>>> calling graphs.
>>>>> Any thought?
>>>>>
>>>>>> 2. If logic-sql execute in ShardingSphere, we cannot handle recovery before submit graph result, as event log only wrote at saga engine.
>>>>>>
>>>>>> so we should integrate saga-transaction like omega send event log step by step.
>>>>>> It is better to make every instance do recovery Independently, instead of providing another coordinator center service.
>>>>>> I feel that embed saga should have following capability.
>>>>>> 1).It can provide service in jar package independent
>>>>>> 2).each embed saga only recovery their own transaction-data of this instance.
>>>>>> If one instance crashed, we can introduce external service to do failover.
>>>>>>
>>>>>> so we consider about exending saga-acutator, if it can support submit task step by step in one transaction, it is a good choice.
>>>>>> of course, there have many things we should do.
>>>>>>
>>>>>> ------------------
>>>>>> Zhao Jun
>>>>>> Apache Sharding-Sphere & ServiceComb
>>>>>>
>>>>>>> On Apr 29, 2019, at 5:17 PM, Willem Jiang <wi...@gmail.com> wrote:
>>>>>>>
>>>>>>> First Saga actuator need to build up the calling grapha before sending
>>>>>>> out the request. I don't think you can do the step by step SQL
>>>>>>> invocation with Saga actuator.
>>>>>>> If you want to call the SQL execution step by step , you may need to
>>>>>>> switch to ServiceComb Pack project which has a coordinator to take
>>>>>>> care of the distributed transaction. But that introduce another
>>>>>>> endpoint(Alpha) to shardingsphere.
>>>>>>>
>>>>>>> From my understanding, Saga actuator is most efficient way to execute
>>>>>>> the SQL across different data nodes.
>>>>>>>
>>>>>>>
>>>>>>> Willem Jiang
>>>>>>>
>>>>>>> Twitter: willemjiang
>>>>>>> Weibo: 姜宁willem
>>>>>>>
>>>>>>> On Fri, Apr 26, 2019 at 6:41 PM zhaojun <zh...@126.com> wrote:
>>>>>>>>
>>>>>>>> Hi, all
>>>>>>>>
>>>>>>>> currently, we have integrated with saga using graph based engine in shardingsphere[1]
>>>>>>>> it need us to collect all participated actual SQL, then submit to saga actuator in commit/rollback phase.
>>>>>>>> if application crashed before invoking saga actuator, undo log of branch transaction SQL will not be saved,
>>>>>>>> so recovery thread will not be executed correctly.
>>>>>>>>
>>>>>>>> it's better that encapsulating every actual SQL as a saga task in shardingsphere side,
>>>>>>>> then submit to saga actuator realtime instead of batch processing all the SQLs at commit/rollback phase.
>>>>>>>> this architecture will make the boundary more clear between shardingsphre and saga, currently we have done some additional works for integrating saga.
>>>>>>>>
>>>>>>>> any thought?
>>>>>>>>
>>>>>>>> [1]:https://github.com/sharding-sphere/shardingsphere-spi-impl/tree/master/sharding-transaction-spi-impl/sharding-transaction-base-spi-impl/sharding-transaction-base-saga <https://github.com/sharding-sphere/shardingsphere-spi-impl/tree/master/sharding-transaction-spi-impl/sharding-transaction-base-spi-impl/sharding-transaction-base-saga>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ------------------
>>>>>>>> Zhao Jun
>>>>>>>> Apache Sharding-Sphere & ServiceComb
>>>>>>>>
>>>>>>
>>>>
>>
Re: [saga-actuator]consider about adding StreamBasedSaga engine to
integrate with shardingsphere
Posted by Willem Jiang <wi...@gmail.com>.
It sounds good to me.
Willem Jiang
Twitter: willemjiang
Weibo: 姜宁willem
On Thu, May 2, 2019 at 11:14 AM 赵俊 <zh...@jd.com> wrote:
>
> Let SQL executor running command one by one is good to me.
>
> One logic-sql will be routed to multi-actual SQLs, then executed in sharding-sphere execute-engine.
> A actual SQL will be wrapped as a SQL execute-callback in sharding-sphere.
> We could provide a SPI for this SQL execute-callback.
> For integrated with saga, we could implement a saga execute-callback, it will wrap event sourcing logic like saga-actuator task have done.
>
>
> > On May 1, 2019, at 5:06 PM, Willem Jiang <wi...@gmail.com> wrote:
> >
> > We could do some enhancement on it.
> > From my understanding, We could create a calling graph which has three
> > sub graph of it, or we could let the SQL executor running the command
> > one by one.
> > Any thoughts?
> >
> > Willem Jiang
> >
> > Twitter: willemjiang
> > Weibo: 姜宁willem
> >
> > On Wed, May 1, 2019 at 4:29 PM zhaojun <zh...@126.com> wrote:
> >>
> >> But saga-actuator don’t support 3 logic SQL is a global transaction while you submit 3 times separately.
> >>
> >> ------------------
> >> Zhao Jun
> >> Apache Sharding-Sphere & ServiceComb
> >>
> >>> On May 1, 2019, at 8:38 AM, Willem Jiang <wi...@gmail.com> wrote:
> >>>
> >>> Willem Jiang
> >>>
> >>> Twitter: willemjiang
> >>> Weibo: 姜宁willem
> >>>
> >>> On Mon, Apr 29, 2019 at 7:58 PM zhaojun <zh...@126.com> wrote:
> >>>>
> >>>> Hi, Willem
> >>>>
> >>>> Saga actuator could not compatible with ShardingSphere currently, there are 2 main problem exist.
> >>>> 1. If sql execute in saga transport instead of ShardingSphere, we can not see the result of logic-sql even in one transaction, it is like this:
> >>>> update t_order set status=’start’ where usr_id=’tom’;
> >>>> select status from t_order where usr_id=’tom’; —> we can’t query ’start’ record as actuator haven’t executed.
> >>>> Insert into t_order values(?,?,?) ;
> >>>
> >>> I think we can break the whole SQL interactions into smaller steps,
> >>> and let saga-actuator do the job one by one.
> >>> In this case, you can break the whole calling graph into several sub
> >>> calling graphs.
> >>> Any thought?
> >>>
> >>>> 2. If logic-sql execute in ShardingSphere, we cannot handle recovery before submit graph result, as event log only wrote at saga engine.
> >>>>
> >>>> so we should integrate saga-transaction like omega send event log step by step.
> >>>> It is better to make every instance do recovery Independently, instead of providing another coordinator center service.
> >>>> I feel that embed saga should have following capability.
> >>>> 1).It can provide service in jar package independent
> >>>> 2).each embed saga only recovery their own transaction-data of this instance.
> >>>> If one instance crashed, we can introduce external service to do failover.
> >>>>
> >>>> so we consider about exending saga-acutator, if it can support submit task step by step in one transaction, it is a good choice.
> >>>> of course, there have many things we should do.
> >>>>
> >>>> ------------------
> >>>> Zhao Jun
> >>>> Apache Sharding-Sphere & ServiceComb
> >>>>
> >>>>> On Apr 29, 2019, at 5:17 PM, Willem Jiang <wi...@gmail.com> wrote:
> >>>>>
> >>>>> First Saga actuator need to build up the calling grapha before sending
> >>>>> out the request. I don't think you can do the step by step SQL
> >>>>> invocation with Saga actuator.
> >>>>> If you want to call the SQL execution step by step , you may need to
> >>>>> switch to ServiceComb Pack project which has a coordinator to take
> >>>>> care of the distributed transaction. But that introduce another
> >>>>> endpoint(Alpha) to shardingsphere.
> >>>>>
> >>>>> From my understanding, Saga actuator is most efficient way to execute
> >>>>> the SQL across different data nodes.
> >>>>>
> >>>>>
> >>>>> Willem Jiang
> >>>>>
> >>>>> Twitter: willemjiang
> >>>>> Weibo: 姜宁willem
> >>>>>
> >>>>> On Fri, Apr 26, 2019 at 6:41 PM zhaojun <zh...@126.com> wrote:
> >>>>>>
> >>>>>> Hi, all
> >>>>>>
> >>>>>> currently, we have integrated with saga using graph based engine in shardingsphere[1]
> >>>>>> it need us to collect all participated actual SQL, then submit to saga actuator in commit/rollback phase.
> >>>>>> if application crashed before invoking saga actuator, undo log of branch transaction SQL will not be saved,
> >>>>>> so recovery thread will not be executed correctly.
> >>>>>>
> >>>>>> it's better that encapsulating every actual SQL as a saga task in shardingsphere side,
> >>>>>> then submit to saga actuator realtime instead of batch processing all the SQLs at commit/rollback phase.
> >>>>>> this architecture will make the boundary more clear between shardingsphre and saga, currently we have done some additional works for integrating saga.
> >>>>>>
> >>>>>> any thought?
> >>>>>>
> >>>>>> [1]:https://github.com/sharding-sphere/shardingsphere-spi-impl/tree/master/sharding-transaction-spi-impl/sharding-transaction-base-spi-impl/sharding-transaction-base-saga <https://github.com/sharding-sphere/shardingsphere-spi-impl/tree/master/sharding-transaction-spi-impl/sharding-transaction-base-spi-impl/sharding-transaction-base-saga>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> ------------------
> >>>>>> Zhao Jun
> >>>>>> Apache Sharding-Sphere & ServiceComb
> >>>>>>
> >>>>
> >>
>
Re: [saga-actuator]consider about adding StreamBasedSaga engine to
integrate with shardingsphere
Posted by Willem Jiang <wi...@gmail.com>.
It sounds good to me.
Willem Jiang
Twitter: willemjiang
Weibo: 姜宁willem
On Thu, May 2, 2019 at 11:14 AM 赵俊 <zh...@jd.com> wrote:
>
> Let SQL executor running command one by one is good to me.
>
> One logic-sql will be routed to multi-actual SQLs, then executed in sharding-sphere execute-engine.
> A actual SQL will be wrapped as a SQL execute-callback in sharding-sphere.
> We could provide a SPI for this SQL execute-callback.
> For integrated with saga, we could implement a saga execute-callback, it will wrap event sourcing logic like saga-actuator task have done.
>
>
> > On May 1, 2019, at 5:06 PM, Willem Jiang <wi...@gmail.com> wrote:
> >
> > We could do some enhancement on it.
> > From my understanding, We could create a calling graph which has three
> > sub graph of it, or we could let the SQL executor running the command
> > one by one.
> > Any thoughts?
> >
> > Willem Jiang
> >
> > Twitter: willemjiang
> > Weibo: 姜宁willem
> >
> > On Wed, May 1, 2019 at 4:29 PM zhaojun <zh...@126.com> wrote:
> >>
> >> But saga-actuator don’t support 3 logic SQL is a global transaction while you submit 3 times separately.
> >>
> >> ------------------
> >> Zhao Jun
> >> Apache Sharding-Sphere & ServiceComb
> >>
> >>> On May 1, 2019, at 8:38 AM, Willem Jiang <wi...@gmail.com> wrote:
> >>>
> >>> Willem Jiang
> >>>
> >>> Twitter: willemjiang
> >>> Weibo: 姜宁willem
> >>>
> >>> On Mon, Apr 29, 2019 at 7:58 PM zhaojun <zh...@126.com> wrote:
> >>>>
> >>>> Hi, Willem
> >>>>
> >>>> Saga actuator could not compatible with ShardingSphere currently, there are 2 main problem exist.
> >>>> 1. If sql execute in saga transport instead of ShardingSphere, we can not see the result of logic-sql even in one transaction, it is like this:
> >>>> update t_order set status=’start’ where usr_id=’tom’;
> >>>> select status from t_order where usr_id=’tom’; —> we can’t query ’start’ record as actuator haven’t executed.
> >>>> Insert into t_order values(?,?,?) ;
> >>>
> >>> I think we can break the whole SQL interactions into smaller steps,
> >>> and let saga-actuator do the job one by one.
> >>> In this case, you can break the whole calling graph into several sub
> >>> calling graphs.
> >>> Any thought?
> >>>
> >>>> 2. If logic-sql execute in ShardingSphere, we cannot handle recovery before submit graph result, as event log only wrote at saga engine.
> >>>>
> >>>> so we should integrate saga-transaction like omega send event log step by step.
> >>>> It is better to make every instance do recovery Independently, instead of providing another coordinator center service.
> >>>> I feel that embed saga should have following capability.
> >>>> 1).It can provide service in jar package independent
> >>>> 2).each embed saga only recovery their own transaction-data of this instance.
> >>>> If one instance crashed, we can introduce external service to do failover.
> >>>>
> >>>> so we consider about exending saga-acutator, if it can support submit task step by step in one transaction, it is a good choice.
> >>>> of course, there have many things we should do.
> >>>>
> >>>> ------------------
> >>>> Zhao Jun
> >>>> Apache Sharding-Sphere & ServiceComb
> >>>>
> >>>>> On Apr 29, 2019, at 5:17 PM, Willem Jiang <wi...@gmail.com> wrote:
> >>>>>
> >>>>> First Saga actuator need to build up the calling grapha before sending
> >>>>> out the request. I don't think you can do the step by step SQL
> >>>>> invocation with Saga actuator.
> >>>>> If you want to call the SQL execution step by step , you may need to
> >>>>> switch to ServiceComb Pack project which has a coordinator to take
> >>>>> care of the distributed transaction. But that introduce another
> >>>>> endpoint(Alpha) to shardingsphere.
> >>>>>
> >>>>> From my understanding, Saga actuator is most efficient way to execute
> >>>>> the SQL across different data nodes.
> >>>>>
> >>>>>
> >>>>> Willem Jiang
> >>>>>
> >>>>> Twitter: willemjiang
> >>>>> Weibo: 姜宁willem
> >>>>>
> >>>>> On Fri, Apr 26, 2019 at 6:41 PM zhaojun <zh...@126.com> wrote:
> >>>>>>
> >>>>>> Hi, all
> >>>>>>
> >>>>>> currently, we have integrated with saga using graph based engine in shardingsphere[1]
> >>>>>> it need us to collect all participated actual SQL, then submit to saga actuator in commit/rollback phase.
> >>>>>> if application crashed before invoking saga actuator, undo log of branch transaction SQL will not be saved,
> >>>>>> so recovery thread will not be executed correctly.
> >>>>>>
> >>>>>> it's better that encapsulating every actual SQL as a saga task in shardingsphere side,
> >>>>>> then submit to saga actuator realtime instead of batch processing all the SQLs at commit/rollback phase.
> >>>>>> this architecture will make the boundary more clear between shardingsphre and saga, currently we have done some additional works for integrating saga.
> >>>>>>
> >>>>>> any thought?
> >>>>>>
> >>>>>> [1]:https://github.com/sharding-sphere/shardingsphere-spi-impl/tree/master/sharding-transaction-spi-impl/sharding-transaction-base-spi-impl/sharding-transaction-base-saga <https://github.com/sharding-sphere/shardingsphere-spi-impl/tree/master/sharding-transaction-spi-impl/sharding-transaction-base-spi-impl/sharding-transaction-base-saga>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> ------------------
> >>>>>> Zhao Jun
> >>>>>> Apache Sharding-Sphere & ServiceComb
> >>>>>>
> >>>>
> >>
>
Re: [saga-actuator]consider about adding StreamBasedSaga engine to
integrate with shardingsphere
Posted by 赵俊 <zh...@jd.com>.
Let SQL executor running command one by one is good to me.
One logic-sql will be routed to multi-actual SQLs, then executed in sharding-sphere execute-engine.
A actual SQL will be wrapped as a SQL execute-callback in sharding-sphere.
We could provide a SPI for this SQL execute-callback.
For integrated with saga, we could implement a saga execute-callback, it will wrap event sourcing logic like saga-actuator task have done.
> On May 1, 2019, at 5:06 PM, Willem Jiang <wi...@gmail.com> wrote:
>
> We could do some enhancement on it.
> From my understanding, We could create a calling graph which has three
> sub graph of it, or we could let the SQL executor running the command
> one by one.
> Any thoughts?
>
> Willem Jiang
>
> Twitter: willemjiang
> Weibo: 姜宁willem
>
> On Wed, May 1, 2019 at 4:29 PM zhaojun <zh...@126.com> wrote:
>>
>> But saga-actuator don’t support 3 logic SQL is a global transaction while you submit 3 times separately.
>>
>> ------------------
>> Zhao Jun
>> Apache Sharding-Sphere & ServiceComb
>>
>>> On May 1, 2019, at 8:38 AM, Willem Jiang <wi...@gmail.com> wrote:
>>>
>>> Willem Jiang
>>>
>>> Twitter: willemjiang
>>> Weibo: 姜宁willem
>>>
>>> On Mon, Apr 29, 2019 at 7:58 PM zhaojun <zh...@126.com> wrote:
>>>>
>>>> Hi, Willem
>>>>
>>>> Saga actuator could not compatible with ShardingSphere currently, there are 2 main problem exist.
>>>> 1. If sql execute in saga transport instead of ShardingSphere, we can not see the result of logic-sql even in one transaction, it is like this:
>>>> update t_order set status=’start’ where usr_id=’tom’;
>>>> select status from t_order where usr_id=’tom’; —> we can’t query ’start’ record as actuator haven’t executed.
>>>> Insert into t_order values(?,?,?) ;
>>>
>>> I think we can break the whole SQL interactions into smaller steps,
>>> and let saga-actuator do the job one by one.
>>> In this case, you can break the whole calling graph into several sub
>>> calling graphs.
>>> Any thought?
>>>
>>>> 2. If logic-sql execute in ShardingSphere, we cannot handle recovery before submit graph result, as event log only wrote at saga engine.
>>>>
>>>> so we should integrate saga-transaction like omega send event log step by step.
>>>> It is better to make every instance do recovery Independently, instead of providing another coordinator center service.
>>>> I feel that embed saga should have following capability.
>>>> 1).It can provide service in jar package independent
>>>> 2).each embed saga only recovery their own transaction-data of this instance.
>>>> If one instance crashed, we can introduce external service to do failover.
>>>>
>>>> so we consider about exending saga-acutator, if it can support submit task step by step in one transaction, it is a good choice.
>>>> of course, there have many things we should do.
>>>>
>>>> ------------------
>>>> Zhao Jun
>>>> Apache Sharding-Sphere & ServiceComb
>>>>
>>>>> On Apr 29, 2019, at 5:17 PM, Willem Jiang <wi...@gmail.com> wrote:
>>>>>
>>>>> First Saga actuator need to build up the calling grapha before sending
>>>>> out the request. I don't think you can do the step by step SQL
>>>>> invocation with Saga actuator.
>>>>> If you want to call the SQL execution step by step , you may need to
>>>>> switch to ServiceComb Pack project which has a coordinator to take
>>>>> care of the distributed transaction. But that introduce another
>>>>> endpoint(Alpha) to shardingsphere.
>>>>>
>>>>> From my understanding, Saga actuator is most efficient way to execute
>>>>> the SQL across different data nodes.
>>>>>
>>>>>
>>>>> Willem Jiang
>>>>>
>>>>> Twitter: willemjiang
>>>>> Weibo: 姜宁willem
>>>>>
>>>>> On Fri, Apr 26, 2019 at 6:41 PM zhaojun <zh...@126.com> wrote:
>>>>>>
>>>>>> Hi, all
>>>>>>
>>>>>> currently, we have integrated with saga using graph based engine in shardingsphere[1]
>>>>>> it need us to collect all participated actual SQL, then submit to saga actuator in commit/rollback phase.
>>>>>> if application crashed before invoking saga actuator, undo log of branch transaction SQL will not be saved,
>>>>>> so recovery thread will not be executed correctly.
>>>>>>
>>>>>> it's better that encapsulating every actual SQL as a saga task in shardingsphere side,
>>>>>> then submit to saga actuator realtime instead of batch processing all the SQLs at commit/rollback phase.
>>>>>> this architecture will make the boundary more clear between shardingsphre and saga, currently we have done some additional works for integrating saga.
>>>>>>
>>>>>> any thought?
>>>>>>
>>>>>> [1]:https://github.com/sharding-sphere/shardingsphere-spi-impl/tree/master/sharding-transaction-spi-impl/sharding-transaction-base-spi-impl/sharding-transaction-base-saga <https://github.com/sharding-sphere/shardingsphere-spi-impl/tree/master/sharding-transaction-spi-impl/sharding-transaction-base-spi-impl/sharding-transaction-base-saga>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ------------------
>>>>>> Zhao Jun
>>>>>> Apache Sharding-Sphere & ServiceComb
>>>>>>
>>>>
>>
Re: [saga-actuator]consider about adding StreamBasedSaga engine to
integrate with shardingsphere
Posted by Willem Jiang <wi...@gmail.com>.
We could do some enhancement on it.
From my understanding, We could create a calling graph which has three
sub graph of it, or we could let the SQL executor running the command
one by one.
Any thoughts?
Willem Jiang
Twitter: willemjiang
Weibo: 姜宁willem
On Wed, May 1, 2019 at 4:29 PM zhaojun <zh...@126.com> wrote:
>
> But saga-actuator don’t support 3 logic SQL is a global transaction while you submit 3 times separately.
>
> ------------------
> Zhao Jun
> Apache Sharding-Sphere & ServiceComb
>
> > On May 1, 2019, at 8:38 AM, Willem Jiang <wi...@gmail.com> wrote:
> >
> > Willem Jiang
> >
> > Twitter: willemjiang
> > Weibo: 姜宁willem
> >
> > On Mon, Apr 29, 2019 at 7:58 PM zhaojun <zh...@126.com> wrote:
> >>
> >> Hi, Willem
> >>
> >> Saga actuator could not compatible with ShardingSphere currently, there are 2 main problem exist.
> >> 1. If sql execute in saga transport instead of ShardingSphere, we can not see the result of logic-sql even in one transaction, it is like this:
> >> update t_order set status=’start’ where usr_id=’tom’;
> >> select status from t_order where usr_id=’tom’; —> we can’t query ’start’ record as actuator haven’t executed.
> >> Insert into t_order values(?,?,?) ;
> >
> > I think we can break the whole SQL interactions into smaller steps,
> > and let saga-actuator do the job one by one.
> > In this case, you can break the whole calling graph into several sub
> > calling graphs.
> > Any thought?
> >
> >> 2. If logic-sql execute in ShardingSphere, we cannot handle recovery before submit graph result, as event log only wrote at saga engine.
> >>
> >> so we should integrate saga-transaction like omega send event log step by step.
> >> It is better to make every instance do recovery Independently, instead of providing another coordinator center service.
> >> I feel that embed saga should have following capability.
> >> 1).It can provide service in jar package independent
> >> 2).each embed saga only recovery their own transaction-data of this instance.
> >> If one instance crashed, we can introduce external service to do failover.
> >>
> >> so we consider about exending saga-acutator, if it can support submit task step by step in one transaction, it is a good choice.
> >> of course, there have many things we should do.
> >>
> >> ------------------
> >> Zhao Jun
> >> Apache Sharding-Sphere & ServiceComb
> >>
> >>> On Apr 29, 2019, at 5:17 PM, Willem Jiang <wi...@gmail.com> wrote:
> >>>
> >>> First Saga actuator need to build up the calling grapha before sending
> >>> out the request. I don't think you can do the step by step SQL
> >>> invocation with Saga actuator.
> >>> If you want to call the SQL execution step by step , you may need to
> >>> switch to ServiceComb Pack project which has a coordinator to take
> >>> care of the distributed transaction. But that introduce another
> >>> endpoint(Alpha) to shardingsphere.
> >>>
> >>> From my understanding, Saga actuator is most efficient way to execute
> >>> the SQL across different data nodes.
> >>>
> >>>
> >>> Willem Jiang
> >>>
> >>> Twitter: willemjiang
> >>> Weibo: 姜宁willem
> >>>
> >>> On Fri, Apr 26, 2019 at 6:41 PM zhaojun <zh...@126.com> wrote:
> >>>>
> >>>> Hi, all
> >>>>
> >>>> currently, we have integrated with saga using graph based engine in shardingsphere[1]
> >>>> it need us to collect all participated actual SQL, then submit to saga actuator in commit/rollback phase.
> >>>> if application crashed before invoking saga actuator, undo log of branch transaction SQL will not be saved,
> >>>> so recovery thread will not be executed correctly.
> >>>>
> >>>> it's better that encapsulating every actual SQL as a saga task in shardingsphere side,
> >>>> then submit to saga actuator realtime instead of batch processing all the SQLs at commit/rollback phase.
> >>>> this architecture will make the boundary more clear between shardingsphre and saga, currently we have done some additional works for integrating saga.
> >>>>
> >>>> any thought?
> >>>>
> >>>> [1]:https://github.com/sharding-sphere/shardingsphere-spi-impl/tree/master/sharding-transaction-spi-impl/sharding-transaction-base-spi-impl/sharding-transaction-base-saga <https://github.com/sharding-sphere/shardingsphere-spi-impl/tree/master/sharding-transaction-spi-impl/sharding-transaction-base-spi-impl/sharding-transaction-base-saga>
> >>>>
> >>>>
> >>>>
> >>>> ------------------
> >>>> Zhao Jun
> >>>> Apache Sharding-Sphere & ServiceComb
> >>>>
> >>
>
Re: [saga-actuator]consider about adding StreamBasedSaga engine to
integrate with shardingsphere
Posted by zhaojun <zh...@126.com>.
But saga-actuator don’t support 3 logic SQL is a global transaction while you submit 3 times separately.
------------------
Zhao Jun
Apache Sharding-Sphere & ServiceComb
> On May 1, 2019, at 8:38 AM, Willem Jiang <wi...@gmail.com> wrote:
>
> Willem Jiang
>
> Twitter: willemjiang
> Weibo: 姜宁willem
>
> On Mon, Apr 29, 2019 at 7:58 PM zhaojun <zh...@126.com> wrote:
>>
>> Hi, Willem
>>
>> Saga actuator could not compatible with ShardingSphere currently, there are 2 main problem exist.
>> 1. If sql execute in saga transport instead of ShardingSphere, we can not see the result of logic-sql even in one transaction, it is like this:
>> update t_order set status=’start’ where usr_id=’tom’;
>> select status from t_order where usr_id=’tom’; —> we can’t query ’start’ record as actuator haven’t executed.
>> Insert into t_order values(?,?,?) ;
>
> I think we can break the whole SQL interactions into smaller steps,
> and let saga-actuator do the job one by one.
> In this case, you can break the whole calling graph into several sub
> calling graphs.
> Any thought?
>
>> 2. If logic-sql execute in ShardingSphere, we cannot handle recovery before submit graph result, as event log only wrote at saga engine.
>>
>> so we should integrate saga-transaction like omega send event log step by step.
>> It is better to make every instance do recovery Independently, instead of providing another coordinator center service.
>> I feel that embed saga should have following capability.
>> 1).It can provide service in jar package independent
>> 2).each embed saga only recovery their own transaction-data of this instance.
>> If one instance crashed, we can introduce external service to do failover.
>>
>> so we consider about exending saga-acutator, if it can support submit task step by step in one transaction, it is a good choice.
>> of course, there have many things we should do.
>>
>> ------------------
>> Zhao Jun
>> Apache Sharding-Sphere & ServiceComb
>>
>>> On Apr 29, 2019, at 5:17 PM, Willem Jiang <wi...@gmail.com> wrote:
>>>
>>> First Saga actuator need to build up the calling grapha before sending
>>> out the request. I don't think you can do the step by step SQL
>>> invocation with Saga actuator.
>>> If you want to call the SQL execution step by step , you may need to
>>> switch to ServiceComb Pack project which has a coordinator to take
>>> care of the distributed transaction. But that introduce another
>>> endpoint(Alpha) to shardingsphere.
>>>
>>> From my understanding, Saga actuator is most efficient way to execute
>>> the SQL across different data nodes.
>>>
>>>
>>> Willem Jiang
>>>
>>> Twitter: willemjiang
>>> Weibo: 姜宁willem
>>>
>>> On Fri, Apr 26, 2019 at 6:41 PM zhaojun <zh...@126.com> wrote:
>>>>
>>>> Hi, all
>>>>
>>>> currently, we have integrated with saga using graph based engine in shardingsphere[1]
>>>> it need us to collect all participated actual SQL, then submit to saga actuator in commit/rollback phase.
>>>> if application crashed before invoking saga actuator, undo log of branch transaction SQL will not be saved,
>>>> so recovery thread will not be executed correctly.
>>>>
>>>> it's better that encapsulating every actual SQL as a saga task in shardingsphere side,
>>>> then submit to saga actuator realtime instead of batch processing all the SQLs at commit/rollback phase.
>>>> this architecture will make the boundary more clear between shardingsphre and saga, currently we have done some additional works for integrating saga.
>>>>
>>>> any thought?
>>>>
>>>> [1]:https://github.com/sharding-sphere/shardingsphere-spi-impl/tree/master/sharding-transaction-spi-impl/sharding-transaction-base-spi-impl/sharding-transaction-base-saga <https://github.com/sharding-sphere/shardingsphere-spi-impl/tree/master/sharding-transaction-spi-impl/sharding-transaction-base-spi-impl/sharding-transaction-base-saga>
>>>>
>>>>
>>>>
>>>> ------------------
>>>> Zhao Jun
>>>> Apache Sharding-Sphere & ServiceComb
>>>>
>>
Re: [saga-actuator]consider about adding StreamBasedSaga engine to
integrate with shardingsphere
Posted by zhaojun <zh...@126.com>.
But saga-actuator don’t support 3 logic SQL is a global transaction while you submit 3 times separately.
------------------
Zhao Jun
Apache Sharding-Sphere & ServiceComb
> On May 1, 2019, at 8:38 AM, Willem Jiang <wi...@gmail.com> wrote:
>
> Willem Jiang
>
> Twitter: willemjiang
> Weibo: 姜宁willem
>
> On Mon, Apr 29, 2019 at 7:58 PM zhaojun <zh...@126.com> wrote:
>>
>> Hi, Willem
>>
>> Saga actuator could not compatible with ShardingSphere currently, there are 2 main problem exist.
>> 1. If sql execute in saga transport instead of ShardingSphere, we can not see the result of logic-sql even in one transaction, it is like this:
>> update t_order set status=’start’ where usr_id=’tom’;
>> select status from t_order where usr_id=’tom’; —> we can’t query ’start’ record as actuator haven’t executed.
>> Insert into t_order values(?,?,?) ;
>
> I think we can break the whole SQL interactions into smaller steps,
> and let saga-actuator do the job one by one.
> In this case, you can break the whole calling graph into several sub
> calling graphs.
> Any thought?
>
>> 2. If logic-sql execute in ShardingSphere, we cannot handle recovery before submit graph result, as event log only wrote at saga engine.
>>
>> so we should integrate saga-transaction like omega send event log step by step.
>> It is better to make every instance do recovery Independently, instead of providing another coordinator center service.
>> I feel that embed saga should have following capability.
>> 1).It can provide service in jar package independent
>> 2).each embed saga only recovery their own transaction-data of this instance.
>> If one instance crashed, we can introduce external service to do failover.
>>
>> so we consider about exending saga-acutator, if it can support submit task step by step in one transaction, it is a good choice.
>> of course, there have many things we should do.
>>
>> ------------------
>> Zhao Jun
>> Apache Sharding-Sphere & ServiceComb
>>
>>> On Apr 29, 2019, at 5:17 PM, Willem Jiang <wi...@gmail.com> wrote:
>>>
>>> First Saga actuator need to build up the calling grapha before sending
>>> out the request. I don't think you can do the step by step SQL
>>> invocation with Saga actuator.
>>> If you want to call the SQL execution step by step , you may need to
>>> switch to ServiceComb Pack project which has a coordinator to take
>>> care of the distributed transaction. But that introduce another
>>> endpoint(Alpha) to shardingsphere.
>>>
>>> From my understanding, Saga actuator is most efficient way to execute
>>> the SQL across different data nodes.
>>>
>>>
>>> Willem Jiang
>>>
>>> Twitter: willemjiang
>>> Weibo: 姜宁willem
>>>
>>> On Fri, Apr 26, 2019 at 6:41 PM zhaojun <zh...@126.com> wrote:
>>>>
>>>> Hi, all
>>>>
>>>> currently, we have integrated with saga using graph based engine in shardingsphere[1]
>>>> it need us to collect all participated actual SQL, then submit to saga actuator in commit/rollback phase.
>>>> if application crashed before invoking saga actuator, undo log of branch transaction SQL will not be saved,
>>>> so recovery thread will not be executed correctly.
>>>>
>>>> it's better that encapsulating every actual SQL as a saga task in shardingsphere side,
>>>> then submit to saga actuator realtime instead of batch processing all the SQLs at commit/rollback phase.
>>>> this architecture will make the boundary more clear between shardingsphre and saga, currently we have done some additional works for integrating saga.
>>>>
>>>> any thought?
>>>>
>>>> [1]:https://github.com/sharding-sphere/shardingsphere-spi-impl/tree/master/sharding-transaction-spi-impl/sharding-transaction-base-spi-impl/sharding-transaction-base-saga <https://github.com/sharding-sphere/shardingsphere-spi-impl/tree/master/sharding-transaction-spi-impl/sharding-transaction-base-spi-impl/sharding-transaction-base-saga>
>>>>
>>>>
>>>>
>>>> ------------------
>>>> Zhao Jun
>>>> Apache Sharding-Sphere & ServiceComb
>>>>
>>