Re: Why is Async checkpointing made default?

[Chandni Singh <ch...@datatorrent.com>]

Hi,

There hasn't been a close bracket here (borrowing Ram's expression :-) ).
>From what I see the majority agrees with making the fix.

Do we need to start a vote for this?

On Fri, Nov 27, 2015 at 3:00 PM, Thomas Weise <th...@datatorrent.com>
wrote:

> Chetan,
>
> Would like to understand how the checkpointed callback helps you with what
> you indicated. This may require a specific example. Let's take it offline.
>
> Thomas
>
> On Wed, Nov 25, 2015 at 4:07 PM, Chetan Narsude (cnarsude) <
> cnarsude@cisco.com> wrote:
>
> > Yes - a few but cannot share the details - protected under NDA - ping me
> > in private and I can probably be able to give you more generic details on
> > similar cooked up examples.
> >
> >  The part that follows “e.g.” below is an example that probably is
> > sufficient to infer the use case logically, I think. I shared that to
> > exemplify how changing the semantics will break semver.
> >
> > —
> > Chetan
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On 11/25/15, 3:51 PM, "Thomas Weise" <th...@datatorrent.com> wrote:
> >
> > >Do you have a specific example?
> > >
> > >I see this happening in committed(), but not in checkpointed() where the
> > >checkpoint remains intermediate, whether it was copied to HDFS or not.
> > >
> > >
> > >On Wed, Nov 25, 2015 at 3:42 PM, Chetan Narsude (cnarsude) <
> > >cnarsude@cisco.com> wrote:
> > >
> > >> >
> > >> >Until we have this, how about we restore the previous behavior
> > >> >temporarily?
> > >> >Calling checkpointed() immediately does not seem to pose any
> practical
> > >> >issue but ensures that the code that was written under this
> assumption
> > >>is
> > >> >not broken.
> > >>
> > >> We can¹t do it. It would be incorrect. It breaks all the other code
> that
> > >> (unassumingly) correctly complied to the semantics. e.g. an operator
> > >>which
> > >> informs interesting parties that the checkpointed data is available
> for
> > >> immediate consumption from storage.
> > >>
> > >> ‹
> > >> Chetan
> > >>
> > >>
> >
> >
>

Re: Why is Async checkpointing made default?

[Pramod Immaneni <pr...@datatorrent.com>]

There is a pull request for that :)

> On Dec 11, 2015, at 9:43 AM, Thomas Weise <th...@datatorrent.com> wrote:
>
> I think we should focus on getting APEXCORE-78 done instead. It will
> address a number of existing use cases, including those where we tried to
> utilize checkpointed(...).
>
>
> On Fri, Dec 11, 2015 at 9:36 AM, Chandni Singh <ch...@datatorrent.com>
> wrote:
>
>> Hi,
>>
>> There hasn't been a close bracket here (borrowing Ram's expression :-) ).
>> From what I see the majority agrees with making the fix.
>>
>> Do we need to start a vote for this?
>>
>> On Fri, Nov 27, 2015 at 3:00 PM, Thomas Weise <th...@datatorrent.com>
>> wrote:
>>
>>> Chetan,
>>>
>>> Would like to understand how the checkpointed callback helps you with
>> what
>>> you indicated. This may require a specific example. Let's take it
>> offline.
>>>
>>> Thomas
>>>
>>> On Wed, Nov 25, 2015 at 4:07 PM, Chetan Narsude (cnarsude) <
>>> cnarsude@cisco.com> wrote:
>>>
>>>> Yes - a few but cannot share the details - protected under NDA - ping
>> me
>>>> in private and I can probably be able to give you more generic details
>> on
>>>> similar cooked up examples.
>>>>
>>>> The part that follows “e.g.” below is an example that probably is
>>>> sufficient to infer the use case logically, I think. I shared that to
>>>> exemplify how changing the semantics will break semver.
>>>>
>>>> —
>>>> Chetan
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> On 11/25/15, 3:51 PM, "Thomas Weise" <th...@datatorrent.com> wrote:
>>>>>
>>>>> Do you have a specific example?
>>>>>
>>>>> I see this happening in committed(), but not in checkpointed() where
>> the
>>>>> checkpoint remains intermediate, whether it was copied to HDFS or not.
>>>>>
>>>>>
>>>>> On Wed, Nov 25, 2015 at 3:42 PM, Chetan Narsude (cnarsude) <
>>>>> cnarsude@cisco.com> wrote:
>>>>>
>>>>>>>
>>>>>>> Until we have this, how about we restore the previous behavior
>>>>>>> temporarily?
>>>>>>> Calling checkpointed() immediately does not seem to pose any
>>> practical
>>>>>>> issue but ensures that the code that was written under this
>>> assumption
>>>>>> is
>>>>>>> not broken.
>>>>>>
>>>>>> We can¹t do it. It would be incorrect. It breaks all the other code
>>> that
>>>>>> (unassumingly) correctly complied to the semantics. e.g. an operator
>>>>>> which
>>>>>> informs interesting parties that the checkpointed data is available
>>> for
>>>>>> immediate consumption from storage.
>>>>>>
>>>>>> ‹
>>>>>> Chetan
>>>>>>
>>>>>>
>>>>
>>>>
>>>
>>

Re: Why is Async checkpointing made default?

[Thomas Weise <th...@datatorrent.com>]

I think we should focus on getting APEXCORE-78 done instead. It will
address a number of existing use cases, including those where we tried to
utilize checkpointed(...).


On Fri, Dec 11, 2015 at 9:36 AM, Chandni Singh <ch...@datatorrent.com>
wrote:

> Hi,
>
> There hasn't been a close bracket here (borrowing Ram's expression :-) ).
> From what I see the majority agrees with making the fix.
>
> Do we need to start a vote for this?
>
> On Fri, Nov 27, 2015 at 3:00 PM, Thomas Weise <th...@datatorrent.com>
> wrote:
>
> > Chetan,
> >
> > Would like to understand how the checkpointed callback helps you with
> what
> > you indicated. This may require a specific example. Let's take it
> offline.
> >
> > Thomas
> >
> > On Wed, Nov 25, 2015 at 4:07 PM, Chetan Narsude (cnarsude) <
> > cnarsude@cisco.com> wrote:
> >
> > > Yes - a few but cannot share the details - protected under NDA - ping
> me
> > > in private and I can probably be able to give you more generic details
> on
> > > similar cooked up examples.
> > >
> > >  The part that follows “e.g.” below is an example that probably is
> > > sufficient to infer the use case logically, I think. I shared that to
> > > exemplify how changing the semantics will break semver.
> > >
> > > —
> > > Chetan
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On 11/25/15, 3:51 PM, "Thomas Weise" <th...@datatorrent.com> wrote:
> > >
> > > >Do you have a specific example?
> > > >
> > > >I see this happening in committed(), but not in checkpointed() where
> the
> > > >checkpoint remains intermediate, whether it was copied to HDFS or not.
> > > >
> > > >
> > > >On Wed, Nov 25, 2015 at 3:42 PM, Chetan Narsude (cnarsude) <
> > > >cnarsude@cisco.com> wrote:
> > > >
> > > >> >
> > > >> >Until we have this, how about we restore the previous behavior
> > > >> >temporarily?
> > > >> >Calling checkpointed() immediately does not seem to pose any
> > practical
> > > >> >issue but ensures that the code that was written under this
> > assumption
> > > >>is
> > > >> >not broken.
> > > >>
> > > >> We can¹t do it. It would be incorrect. It breaks all the other code
> > that
> > > >> (unassumingly) correctly complied to the semantics. e.g. an operator
> > > >>which
> > > >> informs interesting parties that the checkpointed data is available
> > for
> > > >> immediate consumption from storage.
> > > >>
> > > >> ‹
> > > >> Chetan
> > > >>
> > > >>
> > >
> > >
> >
>