You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by Andrey Gura <ag...@apache.org> on 2023/05/15 11:52:31 UTC
[DISCUSSION] IEP-100: Compute API: Phase 1 - Simple remote job execution
Hi, Igniters!
Please take a look at the first phase of proposal for Apache ignite 3
Compute API: Simple remote job execution [1].
While most issues are already resolved due to simplicity of the proposal
any feedback and ideas will be really useful for the next phases.
Thanks!
[1]
https://cwiki.apache.org/confluence/display/IGNITE/IEP-100%3A+Compute+API%3A+Phase+1+-+Simple+remote+job+execution
Re: [DISCUSSION] IEP-100: Compute API: Phase 1 - Simple remote job execution
Posted by Andrey Gura <ag...@apache.org>.
> Creating a compute job for each record would quickly get unmanageable and broadcast isn’t equivalent as it doesn’t take into account the possibility of partitions migrating during calculations.
Of course creating a compute job for each record is a waste of
resources. Even in current really simple API a user can execute bulk
operation, it is the responsibility of the job's code. But we also
should provide API which simplifies some WAs which user should use in
current design.
Thanks for highlighting these important aspects of Compute API.
On Mon, May 15, 2023 at 5:39 PM Stephen Darlington
<st...@gridgain.com> wrote:
>
> Right, it doesn’t need to be the same — exposing partitions in this manner is weird — but I think we need to support bulk operations in some capacity. Creating a compute job for each record would quickly get unmanageable and broadcast isn’t equivalent as it doesn’t take into account the possibility of partitions migrating during calculations.
>
> > On 15 May 2023, at 14:30, Andrey Gura <ag...@apache.org> wrote:
> >
> >> In Ignite 2, for bulk operations there’s an affinityRun method that takes a partition as a parameter. I don’t see an equivalent here. Is one in the plan?
> >
> > Ideally users don't need to know a partition for execution of
> > computation and the right tool for it is colocation. Method with
> > partitions was introduced in order to execute scan queries on specific
> > partitions and it is some kind of hack. I believe this approach should
> > be revised if possible.
> >
> >> Are other APIs like apply, async methods, map-reduce being considered in later phases?
> >
> > All methods are async and return CompletableFuture. Yes, Map/reduce,
> > etc will be designed in later phases.
> >
> > On Mon, May 15, 2023 at 4:08 PM Stephen Darlington
> > <st...@gridgain.com> wrote:
> >>
> >> In Ignite 2, for bulk operations there’s an affinityRun method that takes a partition as a parameter. I don’t see an equivalent here. Is one in the plan?
> >>
> >> Are other APIs like apply, async methods, map-reduce being considered in later phases?
> >>
> >>> On 15 May 2023, at 12:52, Andrey Gura <ag...@apache.org> wrote:
> >>>
> >>> Hi, Igniters!
> >>>
> >>> Please take a look at the first phase of proposal for Apache ignite 3
> >>> Compute API: Simple remote job execution [1].
> >>>
> >>> While most issues are already resolved due to simplicity of the proposal
> >>> any feedback and ideas will be really useful for the next phases.
> >>>
> >>> Thanks!
> >>>
> >>> [1]
> >>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-100%3A+Compute+API%3A+Phase+1+-+Simple+remote+job+execution
> >>
>
Re: [DISCUSSION] IEP-100: Compute API: Phase 1 - Simple remote job execution
Posted by Stephen Darlington <st...@gridgain.com>.
Right, it doesn’t need to be the same — exposing partitions in this manner is weird — but I think we need to support bulk operations in some capacity. Creating a compute job for each record would quickly get unmanageable and broadcast isn’t equivalent as it doesn’t take into account the possibility of partitions migrating during calculations.
> On 15 May 2023, at 14:30, Andrey Gura <ag...@apache.org> wrote:
>
>> In Ignite 2, for bulk operations there’s an affinityRun method that takes a partition as a parameter. I don’t see an equivalent here. Is one in the plan?
>
> Ideally users don't need to know a partition for execution of
> computation and the right tool for it is colocation. Method with
> partitions was introduced in order to execute scan queries on specific
> partitions and it is some kind of hack. I believe this approach should
> be revised if possible.
>
>> Are other APIs like apply, async methods, map-reduce being considered in later phases?
>
> All methods are async and return CompletableFuture. Yes, Map/reduce,
> etc will be designed in later phases.
>
> On Mon, May 15, 2023 at 4:08 PM Stephen Darlington
> <st...@gridgain.com> wrote:
>>
>> In Ignite 2, for bulk operations there’s an affinityRun method that takes a partition as a parameter. I don’t see an equivalent here. Is one in the plan?
>>
>> Are other APIs like apply, async methods, map-reduce being considered in later phases?
>>
>>> On 15 May 2023, at 12:52, Andrey Gura <ag...@apache.org> wrote:
>>>
>>> Hi, Igniters!
>>>
>>> Please take a look at the first phase of proposal for Apache ignite 3
>>> Compute API: Simple remote job execution [1].
>>>
>>> While most issues are already resolved due to simplicity of the proposal
>>> any feedback and ideas will be really useful for the next phases.
>>>
>>> Thanks!
>>>
>>> [1]
>>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-100%3A+Compute+API%3A+Phase+1+-+Simple+remote+job+execution
>>
Re: [DISCUSSION] IEP-100: Compute API: Phase 1 - Simple remote job execution
Posted by Andrey Gura <ag...@apache.org>.
> In Ignite 2, for bulk operations there’s an affinityRun method that takes a partition as a parameter. I don’t see an equivalent here. Is one in the plan?
Ideally users don't need to know a partition for execution of
computation and the right tool for it is colocation. Method with
partitions was introduced in order to execute scan queries on specific
partitions and it is some kind of hack. I believe this approach should
be revised if possible.
> Are other APIs like apply, async methods, map-reduce being considered in later phases?
All methods are async and return CompletableFuture. Yes, Map/reduce,
etc will be designed in later phases.
On Mon, May 15, 2023 at 4:08 PM Stephen Darlington
<st...@gridgain.com> wrote:
>
> In Ignite 2, for bulk operations there’s an affinityRun method that takes a partition as a parameter. I don’t see an equivalent here. Is one in the plan?
>
> Are other APIs like apply, async methods, map-reduce being considered in later phases?
>
> > On 15 May 2023, at 12:52, Andrey Gura <ag...@apache.org> wrote:
> >
> > Hi, Igniters!
> >
> > Please take a look at the first phase of proposal for Apache ignite 3
> > Compute API: Simple remote job execution [1].
> >
> > While most issues are already resolved due to simplicity of the proposal
> > any feedback and ideas will be really useful for the next phases.
> >
> > Thanks!
> >
> > [1]
> > https://cwiki.apache.org/confluence/display/IGNITE/IEP-100%3A+Compute+API%3A+Phase+1+-+Simple+remote+job+execution
>
Re: [DISCUSSION] IEP-100: Compute API: Phase 1 - Simple remote job execution
Posted by Stephen Darlington <st...@gridgain.com>.
In Ignite 2, for bulk operations there’s an affinityRun method that takes a partition as a parameter. I don’t see an equivalent here. Is one in the plan?
Are other APIs like apply, async methods, map-reduce being considered in later phases?
> On 15 May 2023, at 12:52, Andrey Gura <ag...@apache.org> wrote:
>
> Hi, Igniters!
>
> Please take a look at the first phase of proposal for Apache ignite 3
> Compute API: Simple remote job execution [1].
>
> While most issues are already resolved due to simplicity of the proposal
> any feedback and ideas will be really useful for the next phases.
>
> Thanks!
>
> [1]
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-100%3A+Compute+API%3A+Phase+1+-+Simple+remote+job+execution