Prevent job/operator from spilling to disk

[Maximilian Bode <ma...@tngtech.com>]

Hi everyone,

is there a way to prevent operators from spilling to disk? If not, would it be conceivable to make this configurable either per job or operator?

The use case is a batch application with the formal requirement not to persist in-flight data to disk (even temporarily) so it would be preferable to see the job fail and then be able to grant sufficient memory and run it again.

Cheers,
Max

— 
Maximilian Bode * Software Consultant * maximilian.bode@tngtech.com
TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
Sitz: Unterföhring * Amtsgericht München * HRB 135082

Re: Prevent job/operator from spilling to disk

[Fabian Hueske <fh...@gmail.com>]

Hi Max,

it is not possible to deactivate spilling to disk at the moment.
It might be possible to implement, but this would require a few more
changes to make it feasible.
For instance, we would need to add more fine-grained control about how
memory is distributed among operators.
This is currently statically decided by the optimizer and there is no
parameter to influence it.

Cheers, Fabian

2016-05-04 13:46 GMT+02:00 Maximilian Bode <ma...@tngtech.com>:

> Hi everyone,
>
> is there a way to prevent operators from spilling to disk? If not, would
> it be conceivable to make this configurable either per job or operator?
>
> The use case is a batch application with the formal requirement not to
> persist in-flight data to disk (even temporarily) so it would be preferable
> to see the job fail and then be able to grant sufficient memory and run it
> again.
>
> Cheers,
> Max
>
> —
> Maximilian Bode * Software Consultant * maximilian.bode@tngtech.com
> TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
> Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
> Sitz: Unterföhring * Amtsgericht München * HRB 135082
>
>