You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by Mikhail Pochatkin <m....@gmail.com> on 2024/05/16 07:42:40 UTC

[DISCUSSION] IEP-120: MapReduce API Ignite3

Igniters,

Please review the proposal [1] and let me know what you think

[1] IEP-120: MapReduce API - Apache Ignite - Apache Software Foundation
<https://cwiki.apache.org/confluence/display/IGNITE/IEP-120%3A+MapReduce+API>

Re: [DISCUSSION] IEP-120: MapReduce API Ignite3

Posted by Stephen Darlington <sd...@apache.org>.
Couple of quick comments:

   - In AI2, co-located tasks “pin” the partition in place. Looks like
   that's not the plan for AI3. I can imagine the technical reasons behind
   that, but it does push some complexity back to the user with some
   non-obvious behaviour (i.e., despite using the co-location API, you don’t
   get co-located behaviour). Maybe if the partition mapping changes,
   task-to-node mappings could be recalculated? At the very least, it feels
   like there should be a warning
   - This comes up quite a lot with AI2: the reducer runs on a single node,
   which can be a significant bottleneck for some use-cases. It would be good
   to be able to parallelise it. Hadoop, for example, adds a “combine” phases
   to its map-reduce implementation (
   https://cwiki.apache.org/confluence/display/HADOOP2/HadoopMapReduce)



On Thu, 16 May 2024 at 08:42, Mikhail Pochatkin <m....@gmail.com>
wrote:

> Igniters,
>
> Please review the proposal [1] and let me know what you think
>
> [1] IEP-120: MapReduce API - Apache Ignite - Apache Software Foundation
> <
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-120%3A+MapReduce+API
> >
>

Re: [DISCUSSION] IEP-120: MapReduce API Ignite3

Posted by Pavel Tupitsyn <pt...@apache.org>.
Mikhail, could you please use JIRA labels to link tickets to IEPs?
- Add iep-xxx label to tickets
- Use "JIRA" macro in Confluence with "labels=iep-xxx" filter

On Thu, May 16, 2024 at 10:43 AM Mikhail Pochatkin <m....@gmail.com>
wrote:

> Igniters,
>
> Please review the proposal [1] and let me know what you think
>
> [1] IEP-120: MapReduce API - Apache Ignite - Apache Software Foundation
> <
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-120%3A+MapReduce+API
> >
>