You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by Alexey Goncharuk <al...@gmail.com> on 2018/02/07 07:29:03 UTC
Re: Apache Ignite 2.4 release
Guys,
Thanks to Dmitriy Pavlov we found the ticket [1] which causes a major
slowdown when page replacement starts. Even though it's not a regression, I
suggest we consider it a blocker for 2.4 because this is a huge performance
issue which can make it virtually impossible to use native persistence when
data size is significantly larger than memory size.
Any objections?
[1] https://issues.apache.org/jira/browse/IGNITE-7638
2018-01-30 17:10 GMT+03:00 Pavel Tupitsyn <pt...@apache.org>:
> Igniters, I will handle 2.4 release if there are no objections.
> Let's take a bit more time for testing and start vote by the end of this
> week.
>
> Pavel
>
> On Sat, Jan 27, 2018 at 3:32 AM, Denis Magda <dm...@apache.org> wrote:
>
> > Hi Vyacheslav,
> >
> > According to the previous review notes the impact of the changes might be
> > significant, thus, I would recommend us to move the changes to the next
> > release.
> >
> > BTW, don’t hesitate to ping reviewers more frequently if there is a
> > pending/abandon review. We are all the people who are tend to forget or
> > miss notifications ;)
> >
> > > On Jan 26, 2018, at 2:04 AM, Vyacheslav Daradur <da...@gmail.com>
> > wrote:
> > >
> > > Hi, Vladimir, it's good news. I'm looking forward to new Ignite
> release!
> > >
> > > Could you please share a release schedule for 'varint' optimizations?
> > >
> > > The task [1] is waiting for review for 5 months.
> > >
> > > [1] https://issues.apache.org/jira/browse/IGNITE-5097
> > >
> > >
> > > On Fri, Jan 26, 2018 at 12:51 PM, Vladimir Ozerov <
> vozerov@gridgain.com>
> > wrote:
> > >> Hi Igniters,
> > >>
> > >> As far as I can see all required tasks and fixes were merged. I
> propose
> > to
> > >> take several days of silence to test what we've done and start vote at
> > the
> > >> beginning of the next week.
> > >>
> > >> Makes sense?
> > >>
> > >> On Mon, Jan 22, 2018 at 8:39 PM, Denis Magda <dm...@apache.org>
> wrote:
> > >>
> > >>> Ok, let’s target Wednesday as a code freeze date.
> > >>>
> > >>> Community members who are involved in 2.4 release please merge you
> > fixes
> > >>> and optimizations by that time.
> > >>>
> > >>> —
> > >>> Denis
> > >>>
> > >>>> On Jan 22, 2018, at 8:24 AM, Anton Vinogradov <av...@apache.org>
> wrote:
> > >>>>
> > >>>> Issues
> > >>>> https://issues.apache.org/jira/browse/IGNITE-6711
> > >>>> https://issues.apache.org/jira/browse/IGNITE-6902
> > >>>>
> > >>>> are in master and ignite-2.4
> > >>>>
> > >>>> On Mon, Jan 22, 2018 at 6:43 PM, Denis Magda <dm...@apache.org>
> > wrote:
> > >>>>
> > >>>>> Igniters,
> > >>>>>
> > >>>>> What’s left and what prevents us from passing 2.4 through the QA
> > phase?
> > >>>>>
> > >>>>> Petr, Anton, Vladimir, Alex G., others please chime in.
> > >>>>>
> > >>>>> —
> > >>>>> Denis
> > >>>>>
> > >>>>>> On Jan 17, 2018, at 7:05 PM, Dmitriy Setrakyan <
> > dsetrakyan@apache.org>
> > >>>>> wrote:
> > >>>>>>
> > >>>>>> Great news, indeed! Looking forward to 2.4 release.
> > >>>>>>
> > >>>>>> On Wed, Jan 17, 2018 at 2:55 AM, Vladimir Ozerov <
> > vozerov@gridgain.com
> > >>>>
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>> Igniters,
> > >>>>>>>
> > >>>>>>> Great news - two very important tickets - baseline topology and
> > Java 8
> > >>>>>>> support - were merged to master. At this point it makes sense to
> > >>> create
> > >>>>> a
> > >>>>>>> branch to stabilize the release and finalize outstanding
> tickets. I
> > >>>>> created
> > >>>>>>> the branch *ignite-2.4*. Please follow the rules below when
> merging
> > >>> new
> > >>>>>>> commits to master:
> > >>>>>>> 1) If ticket is targeted for 2.4 release, please merge to master,
> > then
> > >>>>>>> cherry-pick to ignite-2.4
> > >>>>>>> 2) Otherwise just merge to master.
> > >>>>>>>
> > >>>>>>> Vladimir.
> > >>>>>>>
> > >>>>>>> On Tue, Jan 16, 2018 at 12:32 PM, Anton Vinogradov <
> > >>>>>>> avinogradov@gridgain.com
> > >>>>>>>> wrote:
> > >>>>>>>
> > >>>>>>>> Denis,
> > >>>>>>>>
> > >>>>>>>> I'll review https://issues.apache.org/jira/browse/IGNITE-6902
> > >>>>>>>>
> > >>>>>>>> On Fri, Jan 12, 2018 at 11:45 PM, Denis Magda <
> dmagda@apache.org>
> > >>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> Here is the page create before:
> > >>>>>>>>> https://cwiki.apache.org/confluence/display/IGNITE/
> > >>> Apache+Ignite+2.4
> > >>>>> <
> > >>>>>>>>> https://cwiki.apache.org/confluence/display/IGNITE/
> > >>> Apache+Ignite+2.4>
> > >>>>>>>>>
> > >>>>>>>>> This is good news. It will be perfect if we also release as
> many
> > >>>>>>>> "required
> > >>>>>>>>> tickets" as we can:
> > >>>>>>>>> https://cwiki.apache.org/confluence/display/IGNITE/
> > >>> Apache+Ignite+2.4
> > >>>>> <
> > >>>>>>>>> https://cwiki.apache.org/confluence/display/IGNITE/
> > >>> Apache+Ignite+2.4>
> > >>>>>>>>>
> > >>>>>>>>> My favorite is memory metrics in bytes:
> > https://issues.apache.org/
> > >>>>>>>>> jira/browse/IGNITE-6902 <https://issues.apache.org/
> > >>>>>>>> jira/browse/IGNITE-6902
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> The discussion lists are flooded with the demands for this
> > essential
> > >>>>>>>>> capability. Who is going to review it?
> > >>>>>>>>>
> > >>>>>>>>> -
> > >>>>>>>>> Denis
> > >>>>>>>>>
> > >>>>>>>>>> On Jan 12, 2018, at 11:22 AM, Dmitriy Setrakyan <
> > >>>>>>> dsetrakyan@apache.org
> > >>>>>>>>>
> > >>>>>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>> +1
> > >>>>>>>>>>
> > >>>>>>>>>> Do we have a list of features for 2.4 documented anywhere?
> > >>>>>>>>>>
> > >>>>>>>>>> D.
> > >>>>>>>>>>
> > >>>>>>>>>> On Fri, Jan 12, 2018 at 7:43 AM, Pavel Tupitsyn <
> > >>>>>>> ptupitsyn@apache.org>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> +1
> > >>>>>>>>>>>
> > >>>>>>>>>>> But some stuff is not yet in master (like baseline topology),
> > so
> > >>> it
> > >>>>>>> is
> > >>>>>>>>> too
> > >>>>>>>>>>> soon to create a branch, I think.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Pavel
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Fri, Jan 12, 2018 at 5:31 PM, Vladimir Ozerov <
> > >>>>>>>> vozerov@gridgain.com>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Igniters,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> A number of major improvements have been made (or being
> > finalized
> > >>>>>>> at
> > >>>>>>>>> the
> > >>>>>>>>>>>> moment) to Ignite since recent 2.3 release. This includes
> > >>> baseline
> > >>>>>>>>>>>> topology, new DDL commands, migration to Java 8 and Java 9
> > >>> support,
> > >>>>>>>>>>>> critical performance improvements to WAL, etc..
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I think it makes sense to make a new release. What do you
> > think
> > >>> if
> > >>>>>>> we
> > >>>>>>>>>>>> release AI 2.4 with all this great stuff by the end of Jan?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> If there are no objections, I'll create a branch to start
> > >>>>>>>> stabilization
> > >>>>>>>>>>>> process.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Vladimir.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>
> > >>>>>
> > >>>
> > >>>
> > >
> > >
> > >
> > > --
> > > Best Regards, Vyacheslav D.
> >
> >
>
Re: Apache Ignite 2.4 release
Posted by Raymond Wilson <ra...@trimble.com>.
Thanks for the clarification Dmitry.
Sent from my iPhone
> On 7/02/2018, at 11:13 PM, Dmitry Pavlov <dp...@gmail.com> wrote:
>
> Hi Raymond,
>
> Minor releases were discussed already at dev list. Community does not
> support minor releases. In case of emergency fix is required, full release
> process has to be followed for next release, e.g. for 2.5. Exactly that
> type of release was 2.2, it was emergency release for fix persistence
> usability issues.
>
> Early access build may being build using DEVNOTES.txt instruction from
> Ignite source code.
>
> IMO this fix is mandatory because problem occurs every time when Durable
> Memory pages replacement is started (rotation of pages with RAM and disk).
> It is general case for most of databases, when there is plenty of HDD space
> but not so much RAM.
>
> Full fix with hashing algoritm change will require around a week, but in
> the same time I will try to find out workaround to provide fast fix for 2.4.
>
> Sincerely,
> Dmitriy Pavlov
>
> ср, 7 февр. 2018 г. в 12:07, Raymond Wilson <ra...@trimble.com>:
>
>> I guess this depends on how long it will take to implement.
>>
>> Is it acceptable to release 2.4 without this fix (which allows integrators
>> earlier access to elements such as thin client protocol etc), but follow it
>> up with a 2.4.1 release containing the performance fix?
>>
>> What is the community policy for minor point releases? Does the
>> build/release process allow for a more agile release cadence?
>>
>> Thanks,
>> Raymond.
>>
>> Sent from my iPhone
>>
>>> On 7/02/2018, at 8:29 PM, Alexey Goncharuk <al...@gmail.com>
>> wrote:
>>>
>>> Guys,
>>>
>>> Thanks to Dmitriy Pavlov we found the ticket [1] which causes a major
>>> slowdown when page replacement starts. Even though it's not a
>> regression, I
>>> suggest we consider it a blocker for 2.4 because this is a huge
>> performance
>>> issue which can make it virtually impossible to use native persistence
>> when
>>> data size is significantly larger than memory size.
>>>
>>> Any objections?
>>>
>>> [1] https://issues.apache.org/jira/browse/IGNITE-7638
>>>
>>> 2018-01-30 17:10 GMT+03:00 Pavel Tupitsyn <pt...@apache.org>:
>>>
>>>> Igniters, I will handle 2.4 release if there are no objections.
>>>> Let's take a bit more time for testing and start vote by the end of this
>>>> week.
>>>>
>>>> Pavel
>>>>
>>>>> On Sat, Jan 27, 2018 at 3:32 AM, Denis Magda <dm...@apache.org>
>> wrote:
>>>>>
>>>>> Hi Vyacheslav,
>>>>>
>>>>> According to the previous review notes the impact of the changes might
>> be
>>>>> significant, thus, I would recommend us to move the changes to the next
>>>>> release.
>>>>>
>>>>> BTW, don’t hesitate to ping reviewers more frequently if there is a
>>>>> pending/abandon review. We are all the people who are tend to forget or
>>>>> miss notifications ;)
>>>>>
>>>>>> On Jan 26, 2018, at 2:04 AM, Vyacheslav Daradur <da...@gmail.com>
>>>>> wrote:
>>>>>>
>>>>>> Hi, Vladimir, it's good news. I'm looking forward to new Ignite
>>>> release!
>>>>>>
>>>>>> Could you please share a release schedule for 'varint' optimizations?
>>>>>>
>>>>>> The task [1] is waiting for review for 5 months.
>>>>>>
>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-5097
>>>>>>
>>>>>>
>>>>>> On Fri, Jan 26, 2018 at 12:51 PM, Vladimir Ozerov <
>>>> vozerov@gridgain.com>
>>>>> wrote:
>>>>>>> Hi Igniters,
>>>>>>>
>>>>>>> As far as I can see all required tasks and fixes were merged. I
>>>> propose
>>>>> to
>>>>>>> take several days of silence to test what we've done and start vote
>> at
>>>>> the
>>>>>>> beginning of the next week.
>>>>>>>
>>>>>>> Makes sense?
>>>>>>>
>>>>>>> On Mon, Jan 22, 2018 at 8:39 PM, Denis Magda <dm...@apache.org>
>>>> wrote:
>>>>>>>
>>>>>>>> Ok, let’s target Wednesday as a code freeze date.
>>>>>>>>
>>>>>>>> Community members who are involved in 2.4 release please merge you
>>>>> fixes
>>>>>>>> and optimizations by that time.
>>>>>>>>
>>>>>>>> —
>>>>>>>> Denis
>>>>>>>>
>>>>>>>>> On Jan 22, 2018, at 8:24 AM, Anton Vinogradov <av...@apache.org>
>>>> wrote:
>>>>>>>>>
>>>>>>>>> Issues
>>>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6711
>>>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6902
>>>>>>>>>
>>>>>>>>> are in master and ignite-2.4
>>>>>>>>>
>>>>>>>>> On Mon, Jan 22, 2018 at 6:43 PM, Denis Magda <dm...@apache.org>
>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Igniters,
>>>>>>>>>>
>>>>>>>>>> What’s left and what prevents us from passing 2.4 through the QA
>>>>> phase?
>>>>>>>>>>
>>>>>>>>>> Petr, Anton, Vladimir, Alex G., others please chime in.
>>>>>>>>>>
>>>>>>>>>> —
>>>>>>>>>> Denis
>>>>>>>>>>
>>>>>>>>>>> On Jan 17, 2018, at 7:05 PM, Dmitriy Setrakyan <
>>>>> dsetrakyan@apache.org>
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Great news, indeed! Looking forward to 2.4 release.
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Jan 17, 2018 at 2:55 AM, Vladimir Ozerov <
>>>>> vozerov@gridgain.com
>>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Igniters,
>>>>>>>>>>>>
>>>>>>>>>>>> Great news - two very important tickets - baseline topology and
>>>>> Java 8
>>>>>>>>>>>> support - were merged to master. At this point it makes sense to
>>>>>>>> create
>>>>>>>>>> a
>>>>>>>>>>>> branch to stabilize the release and finalize outstanding
>>>> tickets. I
>>>>>>>>>> created
>>>>>>>>>>>> the branch *ignite-2.4*. Please follow the rules below when
>>>> merging
>>>>>>>> new
>>>>>>>>>>>> commits to master:
>>>>>>>>>>>> 1) If ticket is targeted for 2.4 release, please merge to
>> master,
>>>>> then
>>>>>>>>>>>> cherry-pick to ignite-2.4
>>>>>>>>>>>> 2) Otherwise just merge to master.
>>>>>>>>>>>>
>>>>>>>>>>>> Vladimir.
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Jan 16, 2018 at 12:32 PM, Anton Vinogradov <
>>>>>>>>>>>> avinogradov@gridgain.com
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Denis,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'll review https://issues.apache.org/jira/browse/IGNITE-6902
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Jan 12, 2018 at 11:45 PM, Denis Magda <
>>>> dmagda@apache.org>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Here is the page create before:
>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/IGNITE/
>>>>>>>> Apache+Ignite+2.4
>>>>>>>>>> <
>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/IGNITE/
>>>>>>>> Apache+Ignite+2.4>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This is good news. It will be perfect if we also release as
>>>> many
>>>>>>>>>>>>> "required
>>>>>>>>>>>>>> tickets" as we can:
>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/IGNITE/
>>>>>>>> Apache+Ignite+2.4
>>>>>>>>>> <
>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/IGNITE/
>>>>>>>> Apache+Ignite+2.4>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> My favorite is memory metrics in bytes:
>>>>> https://issues.apache.org/
>>>>>>>>>>>>>> jira/browse/IGNITE-6902 <https://issues.apache.org/
>>>>>>>>>>>>> jira/browse/IGNITE-6902
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The discussion lists are flooded with the demands for this
>>>>> essential
>>>>>>>>>>>>>> capability. Who is going to review it?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -
>>>>>>>>>>>>>> Denis
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Jan 12, 2018, at 11:22 AM, Dmitriy Setrakyan <
>>>>>>>>>>>> dsetrakyan@apache.org
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> +1
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Do we have a list of features for 2.4 documented anywhere?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> D.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Jan 12, 2018 at 7:43 AM, Pavel Tupitsyn <
>>>>>>>>>>>> ptupitsyn@apache.org>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> +1
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> But some stuff is not yet in master (like baseline
>> topology),
>>>>> so
>>>>>>>> it
>>>>>>>>>>>> is
>>>>>>>>>>>>>> too
>>>>>>>>>>>>>>>> soon to create a branch, I think.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Pavel
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, Jan 12, 2018 at 5:31 PM, Vladimir Ozerov <
>>>>>>>>>>>>> vozerov@gridgain.com>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Igniters,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> A number of major improvements have been made (or being
>>>>> finalized
>>>>>>>>>>>> at
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> moment) to Ignite since recent 2.3 release. This includes
>>>>>>>> baseline
>>>>>>>>>>>>>>>>> topology, new DDL commands, migration to Java 8 and Java 9
>>>>>>>> support,
>>>>>>>>>>>>>>>>> critical performance improvements to WAL, etc..
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I think it makes sense to make a new release. What do you
>>>>> think
>>>>>>>> if
>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>> release AI 2.4 with all this great stuff by the end of Jan?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> If there are no objections, I'll create a branch to start
>>>>>>>>>>>>> stabilization
>>>>>>>>>>>>>>>>> process.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Vladimir.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best Regards, Vyacheslav D.
>>>>>
>>>>>
>>>>
>>
Re: Apache Ignite 2.4 release
Posted by Dmitry Pavlov <dp...@gmail.com>.
Hi Raymond,
Minor releases were discussed already at dev list. Community does not
support minor releases. In case of emergency fix is required, full release
process has to be followed for next release, e.g. for 2.5. Exactly that
type of release was 2.2, it was emergency release for fix persistence
usability issues.
Early access build may being build using DEVNOTES.txt instruction from
Ignite source code.
IMO this fix is mandatory because problem occurs every time when Durable
Memory pages replacement is started (rotation of pages with RAM and disk).
It is general case for most of databases, when there is plenty of HDD space
but not so much RAM.
Full fix with hashing algoritm change will require around a week, but in
the same time I will try to find out workaround to provide fast fix for 2.4.
Sincerely,
Dmitriy Pavlov
ср, 7 февр. 2018 г. в 12:07, Raymond Wilson <ra...@trimble.com>:
> I guess this depends on how long it will take to implement.
>
> Is it acceptable to release 2.4 without this fix (which allows integrators
> earlier access to elements such as thin client protocol etc), but follow it
> up with a 2.4.1 release containing the performance fix?
>
> What is the community policy for minor point releases? Does the
> build/release process allow for a more agile release cadence?
>
> Thanks,
> Raymond.
>
> Sent from my iPhone
>
> > On 7/02/2018, at 8:29 PM, Alexey Goncharuk <al...@gmail.com>
> wrote:
> >
> > Guys,
> >
> > Thanks to Dmitriy Pavlov we found the ticket [1] which causes a major
> > slowdown when page replacement starts. Even though it's not a
> regression, I
> > suggest we consider it a blocker for 2.4 because this is a huge
> performance
> > issue which can make it virtually impossible to use native persistence
> when
> > data size is significantly larger than memory size.
> >
> > Any objections?
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-7638
> >
> > 2018-01-30 17:10 GMT+03:00 Pavel Tupitsyn <pt...@apache.org>:
> >
> >> Igniters, I will handle 2.4 release if there are no objections.
> >> Let's take a bit more time for testing and start vote by the end of this
> >> week.
> >>
> >> Pavel
> >>
> >>> On Sat, Jan 27, 2018 at 3:32 AM, Denis Magda <dm...@apache.org>
> wrote:
> >>>
> >>> Hi Vyacheslav,
> >>>
> >>> According to the previous review notes the impact of the changes might
> be
> >>> significant, thus, I would recommend us to move the changes to the next
> >>> release.
> >>>
> >>> BTW, don’t hesitate to ping reviewers more frequently if there is a
> >>> pending/abandon review. We are all the people who are tend to forget or
> >>> miss notifications ;)
> >>>
> >>>> On Jan 26, 2018, at 2:04 AM, Vyacheslav Daradur <da...@gmail.com>
> >>> wrote:
> >>>>
> >>>> Hi, Vladimir, it's good news. I'm looking forward to new Ignite
> >> release!
> >>>>
> >>>> Could you please share a release schedule for 'varint' optimizations?
> >>>>
> >>>> The task [1] is waiting for review for 5 months.
> >>>>
> >>>> [1] https://issues.apache.org/jira/browse/IGNITE-5097
> >>>>
> >>>>
> >>>> On Fri, Jan 26, 2018 at 12:51 PM, Vladimir Ozerov <
> >> vozerov@gridgain.com>
> >>> wrote:
> >>>>> Hi Igniters,
> >>>>>
> >>>>> As far as I can see all required tasks and fixes were merged. I
> >> propose
> >>> to
> >>>>> take several days of silence to test what we've done and start vote
> at
> >>> the
> >>>>> beginning of the next week.
> >>>>>
> >>>>> Makes sense?
> >>>>>
> >>>>> On Mon, Jan 22, 2018 at 8:39 PM, Denis Magda <dm...@apache.org>
> >> wrote:
> >>>>>
> >>>>>> Ok, let’s target Wednesday as a code freeze date.
> >>>>>>
> >>>>>> Community members who are involved in 2.4 release please merge you
> >>> fixes
> >>>>>> and optimizations by that time.
> >>>>>>
> >>>>>> —
> >>>>>> Denis
> >>>>>>
> >>>>>>> On Jan 22, 2018, at 8:24 AM, Anton Vinogradov <av...@apache.org>
> >> wrote:
> >>>>>>>
> >>>>>>> Issues
> >>>>>>> https://issues.apache.org/jira/browse/IGNITE-6711
> >>>>>>> https://issues.apache.org/jira/browse/IGNITE-6902
> >>>>>>>
> >>>>>>> are in master and ignite-2.4
> >>>>>>>
> >>>>>>> On Mon, Jan 22, 2018 at 6:43 PM, Denis Magda <dm...@apache.org>
> >>> wrote:
> >>>>>>>
> >>>>>>>> Igniters,
> >>>>>>>>
> >>>>>>>> What’s left and what prevents us from passing 2.4 through the QA
> >>> phase?
> >>>>>>>>
> >>>>>>>> Petr, Anton, Vladimir, Alex G., others please chime in.
> >>>>>>>>
> >>>>>>>> —
> >>>>>>>> Denis
> >>>>>>>>
> >>>>>>>>> On Jan 17, 2018, at 7:05 PM, Dmitriy Setrakyan <
> >>> dsetrakyan@apache.org>
> >>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>> Great news, indeed! Looking forward to 2.4 release.
> >>>>>>>>>
> >>>>>>>>> On Wed, Jan 17, 2018 at 2:55 AM, Vladimir Ozerov <
> >>> vozerov@gridgain.com
> >>>>>>>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Igniters,
> >>>>>>>>>>
> >>>>>>>>>> Great news - two very important tickets - baseline topology and
> >>> Java 8
> >>>>>>>>>> support - were merged to master. At this point it makes sense to
> >>>>>> create
> >>>>>>>> a
> >>>>>>>>>> branch to stabilize the release and finalize outstanding
> >> tickets. I
> >>>>>>>> created
> >>>>>>>>>> the branch *ignite-2.4*. Please follow the rules below when
> >> merging
> >>>>>> new
> >>>>>>>>>> commits to master:
> >>>>>>>>>> 1) If ticket is targeted for 2.4 release, please merge to
> master,
> >>> then
> >>>>>>>>>> cherry-pick to ignite-2.4
> >>>>>>>>>> 2) Otherwise just merge to master.
> >>>>>>>>>>
> >>>>>>>>>> Vladimir.
> >>>>>>>>>>
> >>>>>>>>>> On Tue, Jan 16, 2018 at 12:32 PM, Anton Vinogradov <
> >>>>>>>>>> avinogradov@gridgain.com
> >>>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Denis,
> >>>>>>>>>>>
> >>>>>>>>>>> I'll review https://issues.apache.org/jira/browse/IGNITE-6902
> >>>>>>>>>>>
> >>>>>>>>>>> On Fri, Jan 12, 2018 at 11:45 PM, Denis Magda <
> >> dmagda@apache.org>
> >>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Here is the page create before:
> >>>>>>>>>>>> https://cwiki.apache.org/confluence/display/IGNITE/
> >>>>>> Apache+Ignite+2.4
> >>>>>>>> <
> >>>>>>>>>>>> https://cwiki.apache.org/confluence/display/IGNITE/
> >>>>>> Apache+Ignite+2.4>
> >>>>>>>>>>>>
> >>>>>>>>>>>> This is good news. It will be perfect if we also release as
> >> many
> >>>>>>>>>>> "required
> >>>>>>>>>>>> tickets" as we can:
> >>>>>>>>>>>> https://cwiki.apache.org/confluence/display/IGNITE/
> >>>>>> Apache+Ignite+2.4
> >>>>>>>> <
> >>>>>>>>>>>> https://cwiki.apache.org/confluence/display/IGNITE/
> >>>>>> Apache+Ignite+2.4>
> >>>>>>>>>>>>
> >>>>>>>>>>>> My favorite is memory metrics in bytes:
> >>> https://issues.apache.org/
> >>>>>>>>>>>> jira/browse/IGNITE-6902 <https://issues.apache.org/
> >>>>>>>>>>> jira/browse/IGNITE-6902
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> The discussion lists are flooded with the demands for this
> >>> essential
> >>>>>>>>>>>> capability. Who is going to review it?
> >>>>>>>>>>>>
> >>>>>>>>>>>> -
> >>>>>>>>>>>> Denis
> >>>>>>>>>>>>
> >>>>>>>>>>>>> On Jan 12, 2018, at 11:22 AM, Dmitriy Setrakyan <
> >>>>>>>>>> dsetrakyan@apache.org
> >>>>>>>>>>>>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> +1
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Do we have a list of features for 2.4 documented anywhere?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> D.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Fri, Jan 12, 2018 at 7:43 AM, Pavel Tupitsyn <
> >>>>>>>>>> ptupitsyn@apache.org>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> +1
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> But some stuff is not yet in master (like baseline
> topology),
> >>> so
> >>>>>> it
> >>>>>>>>>> is
> >>>>>>>>>>>> too
> >>>>>>>>>>>>>> soon to create a branch, I think.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Pavel
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Fri, Jan 12, 2018 at 5:31 PM, Vladimir Ozerov <
> >>>>>>>>>>> vozerov@gridgain.com>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Igniters,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> A number of major improvements have been made (or being
> >>> finalized
> >>>>>>>>>> at
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>> moment) to Ignite since recent 2.3 release. This includes
> >>>>>> baseline
> >>>>>>>>>>>>>>> topology, new DDL commands, migration to Java 8 and Java 9
> >>>>>> support,
> >>>>>>>>>>>>>>> critical performance improvements to WAL, etc..
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I think it makes sense to make a new release. What do you
> >>> think
> >>>>>> if
> >>>>>>>>>> we
> >>>>>>>>>>>>>>> release AI 2.4 with all this great stuff by the end of Jan?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> If there are no objections, I'll create a branch to start
> >>>>>>>>>>> stabilization
> >>>>>>>>>>>>>>> process.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Vladimir.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Best Regards, Vyacheslav D.
> >>>
> >>>
> >>
>
Re: Apache Ignite 2.4 release
Posted by Raymond Wilson <ra...@trimble.com>.
I guess this depends on how long it will take to implement.
Is it acceptable to release 2.4 without this fix (which allows integrators earlier access to elements such as thin client protocol etc), but follow it up with a 2.4.1 release containing the performance fix?
What is the community policy for minor point releases? Does the build/release process allow for a more agile release cadence?
Thanks,
Raymond.
Sent from my iPhone
> On 7/02/2018, at 8:29 PM, Alexey Goncharuk <al...@gmail.com> wrote:
>
> Guys,
>
> Thanks to Dmitriy Pavlov we found the ticket [1] which causes a major
> slowdown when page replacement starts. Even though it's not a regression, I
> suggest we consider it a blocker for 2.4 because this is a huge performance
> issue which can make it virtually impossible to use native persistence when
> data size is significantly larger than memory size.
>
> Any objections?
>
> [1] https://issues.apache.org/jira/browse/IGNITE-7638
>
> 2018-01-30 17:10 GMT+03:00 Pavel Tupitsyn <pt...@apache.org>:
>
>> Igniters, I will handle 2.4 release if there are no objections.
>> Let's take a bit more time for testing and start vote by the end of this
>> week.
>>
>> Pavel
>>
>>> On Sat, Jan 27, 2018 at 3:32 AM, Denis Magda <dm...@apache.org> wrote:
>>>
>>> Hi Vyacheslav,
>>>
>>> According to the previous review notes the impact of the changes might be
>>> significant, thus, I would recommend us to move the changes to the next
>>> release.
>>>
>>> BTW, don’t hesitate to ping reviewers more frequently if there is a
>>> pending/abandon review. We are all the people who are tend to forget or
>>> miss notifications ;)
>>>
>>>> On Jan 26, 2018, at 2:04 AM, Vyacheslav Daradur <da...@gmail.com>
>>> wrote:
>>>>
>>>> Hi, Vladimir, it's good news. I'm looking forward to new Ignite
>> release!
>>>>
>>>> Could you please share a release schedule for 'varint' optimizations?
>>>>
>>>> The task [1] is waiting for review for 5 months.
>>>>
>>>> [1] https://issues.apache.org/jira/browse/IGNITE-5097
>>>>
>>>>
>>>> On Fri, Jan 26, 2018 at 12:51 PM, Vladimir Ozerov <
>> vozerov@gridgain.com>
>>> wrote:
>>>>> Hi Igniters,
>>>>>
>>>>> As far as I can see all required tasks and fixes were merged. I
>> propose
>>> to
>>>>> take several days of silence to test what we've done and start vote at
>>> the
>>>>> beginning of the next week.
>>>>>
>>>>> Makes sense?
>>>>>
>>>>> On Mon, Jan 22, 2018 at 8:39 PM, Denis Magda <dm...@apache.org>
>> wrote:
>>>>>
>>>>>> Ok, let’s target Wednesday as a code freeze date.
>>>>>>
>>>>>> Community members who are involved in 2.4 release please merge you
>>> fixes
>>>>>> and optimizations by that time.
>>>>>>
>>>>>> —
>>>>>> Denis
>>>>>>
>>>>>>> On Jan 22, 2018, at 8:24 AM, Anton Vinogradov <av...@apache.org>
>> wrote:
>>>>>>>
>>>>>>> Issues
>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6711
>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6902
>>>>>>>
>>>>>>> are in master and ignite-2.4
>>>>>>>
>>>>>>> On Mon, Jan 22, 2018 at 6:43 PM, Denis Magda <dm...@apache.org>
>>> wrote:
>>>>>>>
>>>>>>>> Igniters,
>>>>>>>>
>>>>>>>> What’s left and what prevents us from passing 2.4 through the QA
>>> phase?
>>>>>>>>
>>>>>>>> Petr, Anton, Vladimir, Alex G., others please chime in.
>>>>>>>>
>>>>>>>> —
>>>>>>>> Denis
>>>>>>>>
>>>>>>>>> On Jan 17, 2018, at 7:05 PM, Dmitriy Setrakyan <
>>> dsetrakyan@apache.org>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Great news, indeed! Looking forward to 2.4 release.
>>>>>>>>>
>>>>>>>>> On Wed, Jan 17, 2018 at 2:55 AM, Vladimir Ozerov <
>>> vozerov@gridgain.com
>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Igniters,
>>>>>>>>>>
>>>>>>>>>> Great news - two very important tickets - baseline topology and
>>> Java 8
>>>>>>>>>> support - were merged to master. At this point it makes sense to
>>>>>> create
>>>>>>>> a
>>>>>>>>>> branch to stabilize the release and finalize outstanding
>> tickets. I
>>>>>>>> created
>>>>>>>>>> the branch *ignite-2.4*. Please follow the rules below when
>> merging
>>>>>> new
>>>>>>>>>> commits to master:
>>>>>>>>>> 1) If ticket is targeted for 2.4 release, please merge to master,
>>> then
>>>>>>>>>> cherry-pick to ignite-2.4
>>>>>>>>>> 2) Otherwise just merge to master.
>>>>>>>>>>
>>>>>>>>>> Vladimir.
>>>>>>>>>>
>>>>>>>>>> On Tue, Jan 16, 2018 at 12:32 PM, Anton Vinogradov <
>>>>>>>>>> avinogradov@gridgain.com
>>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Denis,
>>>>>>>>>>>
>>>>>>>>>>> I'll review https://issues.apache.org/jira/browse/IGNITE-6902
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Jan 12, 2018 at 11:45 PM, Denis Magda <
>> dmagda@apache.org>
>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Here is the page create before:
>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/IGNITE/
>>>>>> Apache+Ignite+2.4
>>>>>>>> <
>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/IGNITE/
>>>>>> Apache+Ignite+2.4>
>>>>>>>>>>>>
>>>>>>>>>>>> This is good news. It will be perfect if we also release as
>> many
>>>>>>>>>>> "required
>>>>>>>>>>>> tickets" as we can:
>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/IGNITE/
>>>>>> Apache+Ignite+2.4
>>>>>>>> <
>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/IGNITE/
>>>>>> Apache+Ignite+2.4>
>>>>>>>>>>>>
>>>>>>>>>>>> My favorite is memory metrics in bytes:
>>> https://issues.apache.org/
>>>>>>>>>>>> jira/browse/IGNITE-6902 <https://issues.apache.org/
>>>>>>>>>>> jira/browse/IGNITE-6902
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> The discussion lists are flooded with the demands for this
>>> essential
>>>>>>>>>>>> capability. Who is going to review it?
>>>>>>>>>>>>
>>>>>>>>>>>> -
>>>>>>>>>>>> Denis
>>>>>>>>>>>>
>>>>>>>>>>>>> On Jan 12, 2018, at 11:22 AM, Dmitriy Setrakyan <
>>>>>>>>>> dsetrakyan@apache.org
>>>>>>>>>>>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> +1
>>>>>>>>>>>>>
>>>>>>>>>>>>> Do we have a list of features for 2.4 documented anywhere?
>>>>>>>>>>>>>
>>>>>>>>>>>>> D.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Jan 12, 2018 at 7:43 AM, Pavel Tupitsyn <
>>>>>>>>>> ptupitsyn@apache.org>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> +1
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> But some stuff is not yet in master (like baseline topology),
>>> so
>>>>>> it
>>>>>>>>>> is
>>>>>>>>>>>> too
>>>>>>>>>>>>>> soon to create a branch, I think.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Pavel
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Jan 12, 2018 at 5:31 PM, Vladimir Ozerov <
>>>>>>>>>>> vozerov@gridgain.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Igniters,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> A number of major improvements have been made (or being
>>> finalized
>>>>>>>>>> at
>>>>>>>>>>>> the
>>>>>>>>>>>>>>> moment) to Ignite since recent 2.3 release. This includes
>>>>>> baseline
>>>>>>>>>>>>>>> topology, new DDL commands, migration to Java 8 and Java 9
>>>>>> support,
>>>>>>>>>>>>>>> critical performance improvements to WAL, etc..
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I think it makes sense to make a new release. What do you
>>> think
>>>>>> if
>>>>>>>>>> we
>>>>>>>>>>>>>>> release AI 2.4 with all this great stuff by the end of Jan?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> If there are no objections, I'll create a branch to start
>>>>>>>>>>> stabilization
>>>>>>>>>>>>>>> process.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Vladimir.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Best Regards, Vyacheslav D.
>>>
>>>
>>
Re: Apache Ignite 2.4 release
Posted by Denis Magda <dm...@apache.org>.
>
> Neither Cassandra, nor Mongo or any others claim themselves
> to be ACID, so it is not valid to refer to their defaults.
This is why I referred to VoltDB as to an example. It supports ACID
transactions and persists changes to disk in a mode similar to LOG_ONLY.
--
Denis
On Sun, Feb 18, 2018 at 11:37 PM, Vladimir Ozerov <vo...@gridgain.com>
wrote:
> Alex,
>
> You get me right. DEFAULT -> LOG_ONLY doesn't introduce any dramatic
> changes when comparing 2.3 to 2.4 - Ignite was unsafe out of the box in
> 2.3, and it is unsafe in 2.4 as well.
>
> The very problem is that we claim ourselves to be ACID, while in reality we
> are only "AI" out of the box, because durability is not guaranteed due to
> zero backups and LOG_ONLY and consistency is not guaranteed due to
> PRIMARY_SYNC. Neither Cassandra, nor Mongo or any others claim themselves
> to be ACID, so it is not valid to refer to their defaults.
>
> On Mon, Feb 19, 2018 at 10:06 AM, Alexey Goncharuk <
> alexey.goncharuk@gmail.com> wrote:
>
> > In terms of 'safety', Ignite default settings are far beyond optimal. For
> > in-memory mode, we have 0 backups by default, which means partition loss
> in
> > a case of node failure, we have readFromBackup=true and PRIMARY_SYNC by
> > default which effectively cancels linearizability property for cache
> > updates, so setting the default WAL mode to LOG_ONLY does not seem to be
> a
> > bigger evil than it currently is. If we are to move to safer defaults, we
> > should change all of the affected sides.
> >
> > I also want to clarify the difference between guarantees in
> > non-fsync modes. We should distinguish the loss of durability (the loss
> of
> > the last update) because the update did not make it to disk and data loss
> > because the disk content was shuffled due to an incomplete page write. In
> > my understanding, the current situation is:
> > FSYNC: loss of durability: not possible, data loss: not possible
> > LOG_ONLY: loss of durability: possible only if OS/power fails, data loss:
> > possible only if OS/power fails
> > BACKGROUND: loss of durability: possible if Ignite process fails, data
> > loss: possible only if OS/power fails
> >
> > The data loss situation can be mitigated in the cluster using a large
> > enough replication factor (this is what Dmitriy was describing in the
> case
> > of LOG_ONLY and 3 backups configuration).
> >
> > Denis,
> > I do not think it is fair to compare Ignite defaults to Cassandra's
> > defaults because Cassandra is _not_ transactional _eventually consistent_
> > datastore, they claim much weaker guarantees than Ignite.
> >
> > All in all, I'm ok to change the WAL default right now, but I would
> revisit
> > all those settings in 3.0 and made Ignite safe-first.
> >
> > 2018-02-17 3:24 GMT+03:00 Denis Magda <dm...@apache.org>:
> >
> > > Classic relational databases have no choice rather than to use FSYNC by
> > > default. RDBMS is all about consistency.
> > >
> > > Distributed databases try to balance consistency and performance. For
> > > instance, why to fsync every update if there is usually 1 backup copy?
> > > This is probably why VoltDB [1] and Cassandra use the modes comparable
> to
> > > Ignite's LOG_ONLY.
> > >
> > > Ignite as a distributed database should care of both consistency and
> > > performance.
> > >
> > > My vote goes to FSYNC, LOG_ONLY (default), BACKGROUND, NONE.
> > >
> > >
> > > [1] https://docs.voltdb.com/UsingVoltDB/CmdLogConfig.php
> > >
> > > --
> > > Denis
> > >
> > >
> > > On Fri, Feb 16, 2018 at 2:14 PM, Dmitriy Setrakyan <
> > dsetrakyan@apache.org>
> > > wrote:
> > >
> > > > Vova,
> > > >
> > > > I hear your concerns, but at the same time I know that one of the
> > largest
> > > > banks in eastern Europe is using Ignite in LOG_ONLY mode with 3
> backups
> > > to
> > > > move money. The rational is that the probability of failure of 4
> > servers
> > > at
> > > > hardware level at the same time is very low. However, if the JVM
> > process
> > > > fails on any server, then it can be safely restarted without loosing
> > > data.
> > > > In my view, this is why LOG_ONLY mode makes sense as a default.
> > > >
> > > > I still vote to change the default to LOG_ONLY, deprecate the DEFAULT
> > > name
> > > > altogether and add FSYNC mode instead.
> > > >
> > > > D.
> > > >
> > > > On Fri, Feb 16, 2018 at 4:05 PM, Vladimir Ozerov <
> vozerov@gridgain.com
> > >
> > > > wrote:
> > > >
> > > > > Sergey,
> > > > >
> > > > > We do not have backups by default either, so essentially we are
> > loosing
> > > > > data by default. Moreover, backups are less reliable option than
> > fsync
> > > > > because a lot of users cannot afford putting servers into separate
> > > power
> > > > > circuits, so a single power failure may easily lead to poweroff of
> > the
> > > > > whole cluster at once, so data is lost still. This is normal
> practice
> > > > even
> > > > > for enterprise deployments (e.g. asynchronous replication).
> > > > >
> > > > > To make things even worse, we employ PRIMARY_SYNC mode by default!
> So
> > > > even
> > > > > if you configured backups, you still may loose data due to a single
> > > node
> > > > > failure - just shutdown the PRIMARY after commit is confirmed to
> the
> > > > client
> > > > > and your recent update will disappers.
> > > > >
> > > > > So this is what user should do to make himself safe:
> > > > > 1) Learn about WAL modes
> > > > > 2) Learn about backups
> > > > > 3) Learn about synchronization modes
> > > > > 4) Cross his fingers that he understood everything correctly and
> that
> > > > there
> > > > > are no other hidden surprises in Ignite which could lead to data
> > loss.
> > > > >
> > > > > Way to much for a product, claiming to be A*C*ID and persistent,
> > don't
> > > > you
> > > > > think so?
> > > > >
> > > > > Leaving deafult WAL mode with fsync resolves all these issues.
> > > > >
> > > > > Vladimir.
> > > > >
> > > > >
> > > > > On Fri, Feb 16, 2018 at 11:43 PM, Sergey Kozlov <
> > skozlov@gridgain.com>
> > > > > wrote:
> > > > >
> > > > > > I suppose some approaches used by classic databases makes no
> sense
> > > for
> > > > > > Ignite. FSYNC requirement for databases has the nature of single
> > host
> > > > > > solution. If you have corrupted db files you have corrupted
> (lost)
> > > > data.
> > > > > >
> > > > > > For Ignite the enough number of backups and the failure detecting
> > > logic
> > > > > can
> > > > > > provide the data consistency in term "cluster data consistency".
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Fri, Feb 16, 2018 at 8:57 PM, Dmitry Pavlov <
> > > dpavlov.spb@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi, all WAL modes except NONE protects from data consistency
> > > problem
> > > > > > > (B+Tree, pages, etc), which is why I suggest to avoid saying
> > > > > 'corrupted'
> > > > > > > about 'unapplied updates'.
> > > > > > >
> > > > > > > Log Only and Background may cause unapplied updates in case of
> > > > > OS/process
> > > > > > > failures.
> > > > > > >
> > > > > > > None mode IMO is not an option in case data consistency is
> > needed.
> > > > > > >
> > > > > > > пт, 16 февр. 2018 г. в 20:49, Valentin Kulichenko <
> > > > > > > valentin.kulichenko@gmail.com>:
> > > > > > >
> > > > > > > > Guys,
> > > > > > > >
> > > > > > > > While we're on this topic, what is the difference between
> > > > BACKGROUND
> > > > > > and
> > > > > > > > NONE in terms of semantics and provided guarantees? To me it
> > > looks
> > > > > like
> > > > > > > > both guarantee to recover the state since last checkpoint and
> > > > > anything
> > > > > > > else
> > > > > > > > can potentially be lost, so from user perspective they are
> the
> > > > same.
> > > > > > Am I
> > > > > > > > missing something here?
> > > > > > > >
> > > > > > > > Also there is the following in Javadoc for NONE: "If an
> Ignite
> > > node
> > > > > is
> > > > > > > > terminated in NONE mode abruptly, it is likely that the data
> > > stored
> > > > > on
> > > > > > > disk
> > > > > > > > is corrupted and work directory will need to be cleared for a
> > > node
> > > > > > > > restart.". If this is really the case, I'm not sure NONE
> makes
> > > > sense
> > > > > at
> > > > > > > > all. Why would I enable persistence if I'm likely to clear
> the
> > > > > storage
> > > > > > on
> > > > > > > > restart?
> > > > > > > >
> > > > > > > > -Val
> > > > > > > >
> > > > > > > > On Fri, Feb 16, 2018 at 8:39 AM, Vladimir Ozerov <
> > > > > vozerov@gridgain.com
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > What is the reason to have DEFAULT mode at all if you claim
> > > > > LOG_ONLY
> > > > > > to
> > > > > > > > be
> > > > > > > > > completely safe? :)
> > > > > > > > >
> > > > > > > > > And how it could be safe provided that without fsync we
> loose
> > > > part
> > > > > of
> > > > > > > WAL
> > > > > > > > > itself in case of crash?
> > > > > > > > >
> > > > > > > > > пт, 16 февр. 2018 г. в 19:32, Dmitry Pavlov <
> > > > dpavlov.spb@gmail.com
> > > > > >:
> > > > > > > > >
> > > > > > > > > > Thank you. Data can't be corrupted in case crash because
> of
> > > WAL
> > > > > > > replay
> > > > > > > > > > (since completed checkpoint). Physical records are used
> to
> > > > > restore
> > > > > > > > > probably
> > > > > > > > > > corrupted pages in persistent store (we overwrite so
> called
> > > > 'grey
> > > > > > > > zone' -
> > > > > > > > > > pages we don't know for sure if they have been written).
> > > > > > > > > >
> > > > > > > > > > Only one effect is unwritten one or several last
> > > transactions.
> > > > It
> > > > > > is
> > > > > > > > not
> > > > > > > > > > the same with corrupted data.
> > > > > > > > > >
> > > > > > > > > > пт, 16 февр. 2018 г. в 19:19, Vladimir Ozerov <
> > > > > > vozerov@gridgain.com
> > > > > > > >:
> > > > > > > > > >
> > > > > > > > > > > Log only mode is not safe - data might be corrupted in
> > case
> > > > of
> > > > > > > system
> > > > > > > > > > > crash. Oracle - fsync, Postgres - fsync, SQL Server -
> > > fsync,
> > > > > > > > Cassandra
> > > > > > > > > -
> > > > > > > > > > > similar to our “background”.
> > > > > > > > > > >
> > > > > > > > > > > пт, 16 февр. 2018 г. в 19:11, Dmitry Pavlov <
> > > > > > dpavlov.spb@gmail.com
> > > > > > > >:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Vladimir,
> > > > > > > > > > > >
> > > > > > > > > > > > What you saying is defenetely make sence.
> > > > > > > > > > > >
> > > > > > > > > > > > In the same time LOG_ONLY is also safe mode, user
> will
> > be
> > > > > able
> > > > > > to
> > > > > > > > > > restore
> > > > > > > > > > > > system after crash. If it is not true, we should
> create
> > > > > > critical
> > > > > > > > > ticket
> > > > > > > > > > > and
> > > > > > > > > > > > fix it.
> > > > > > > > > > > >
> > > > > > > > > > > > Do you know other databases defaults, such as
> > Cassandra,
> > > > > > Oracle,
> > > > > > > > > > Postgre?
> > > > > > > > > > > >
> > > > > > > > > > > > Sincerely,
> > > > > > > > > > > > Dmitriy Pavlov
> > > > > > > > > > > >
> > > > > > > > > > > > пт, 16 февр. 2018 г. в 18:41, Vladimir Ozerov <
> > > > > > > > vozerov@gridgain.com
> > > > > > > > > >:
> > > > > > > > > > > >
> > > > > > > > > > > > > Igniters,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Sorry for pouring oil on the flames, but from
> > database
> > > > > > > > perspective
> > > > > > > > > > > moving
> > > > > > > > > > > > > from FSYNC to non-FSYNC mode appears to be a
> mistake.
> > > > When
> > > > > > you
> > > > > > > > work
> > > > > > > > > > > with
> > > > > > > > > > > > > database, your main expectation is that it will
> save
> > > your
> > > > > > data.
> > > > > > > > All
> > > > > > > > > > > > > production database vendor make sure that you are
> > safe,
> > > > not
> > > > > > > that
> > > > > > > > > you
> > > > > > > > > > > are
> > > > > > > > > > > > > fast. Moreover, some vendors even prevent you from
> > > being
> > > > in
> > > > > > > > unsafe
> > > > > > > > > > mode
> > > > > > > > > > > > > (e.g. you cannot disable fsync in SQL Server at
> all).
> > > > > > > > > > > > >
> > > > > > > > > > > > > If we continue going in this direction, we will end
> > up
> > > > > with a
> > > > > > > > > > product,
> > > > > > > > > > > > > which is unsafe out of the box and require tons of
> > > > > > > documentation
> > > > > > > > to
> > > > > > > > > > > > > understand how to make it safe. Definitely not the
> > > right
> > > > > > > message
> > > > > > > > to
> > > > > > > > > > the
> > > > > > > > > > > > > market. This is like a car without brakes - would
> you
> > > > like
> > > > > to
> > > > > > > > drive
> > > > > > > > > > it?
> > > > > > > > > > > > If
> > > > > > > > > > > > > this is Need For Speed game and you have unlimited
> > > lives
> > > > > > > > (in-memory
> > > > > > > > > > > cache
> > > > > > > > > > > > > with backing store), then yes. If this is a real
> life
> > > > with
> > > > > > > > > > > (persistence)
> > > > > > > > > > > > -
> > > > > > > > > > > > > then no.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Feb 16, 2018 at 5:20 PM, Dmitriy Setrakyan
> <
> > > > > > > > > > > > dsetrakyan@apache.org>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Well, I cannot say that I like the name LOG_ONLY,
> > > but I
> > > > > > would
> > > > > > > > > vote
> > > > > > > > > > to
> > > > > > > > > > > > > keep
> > > > > > > > > > > > > > it for now, given that it is already documented
> in
> > > many
> > > > > > > places,
> > > > > > > > > > > blogs,
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > examples.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > D.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Feb 16, 2018 at 8:13 AM, Ivan Rakov <
> > > > > > > > > ivan.glukos@gmail.com
> > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Looks like it's an Ignite term - I've never
> heard
> > > of
> > > > it
> > > > > > > > outside
> > > > > > > > > > > > Ignite
> > > > > > > > > > > > > > > scope.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Though, renaming existing enum value requires
> > > keeping
> > > > > old
> > > > > > > as
> > > > > > > > > > > > > deprecated.
> > > > > > > > > > > > > > > DEFAULT is confusing enough to pay this price.
> > > > > > > > > > > > > > > As for LOG_ONLY, I think we can keep it as long
> > as
> > > it
> > > > > has
> > > > > > > > good
> > > > > > > > > > and
> > > > > > > > > > > > > > > definitive javadoc.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Best Regards,
> > > > > > > > > > > > > > > Ivan Rakov
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On 16.02.2018 17:07, Dmitriy Setrakyan wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >> Igniters, just to clarify, does the term
> > LOG_ONLY
> > > > mean
> > > > > > > > > anything
> > > > > > > > > > in
> > > > > > > > > > > > the
> > > > > > > > > > > > > > >> industry or is this just an Ignite term?
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> D.
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> On Fri, Feb 16, 2018 at 8:03 AM, Anton
> > Vinogradov
> > > <
> > > > > > > > > > > > > > >> avinogradov@gridgain.com>
> > > > > > > > > > > > > > >> wrote:
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Log only mode: flushes application buffers.
> > > > > > > > > > > > > > >>> So, in synced mode without fsync guarantee.
> > > That's
> > > > > why
> > > > > > I
> > > > > > > > > > propose
> > > > > > > > > > > to
> > > > > > > > > > > > > > >>> rename
> > > > > > > > > > > > > > >>> it as SYNC.
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>> On Fri, Feb 16, 2018 at 4:49 PM, Ilya
> Lantukh <
> > > > > > > > > > > > ilantukh@gridgain.com
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > >>> wrote:
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>> I am OK with either FSYNC or STRICT variant.
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>> LOG_ONLY name means "log without fsync".
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>> On Fri, Feb 16, 2018 at 4:05 PM, Dmitriy
> > > > Setrakyan <
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>> dsetrakyan@apache.org>
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>>> wrote:
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>> On Fri, Feb 16, 2018 at 7:02 AM, Ivan Rakov
> <
> > > > > > > > > > > > ivan.glukos@gmail.com>
> > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > >>>> wrote:
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>>> Why create a new term to define something
> > that
> > > > has
> > > > > > > > already
> > > > > > > > > > been
> > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > >>>>> defined?
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>>> That makes sense. I'm ok with FSYNC.
> > > > > > > > > > > > > > >>>>>> Anton, I don't understand why we should
> > rename
> > > > > > > LOG_ONLY
> > > > > > > > to
> > > > > > > > > > > SYNC.
> > > > > > > > > > > > > We
> > > > > > > > > > > > > > >>>>>> started this discussion with bad naming of
> > > > > DEFAULT,
> > > > > > > but
> > > > > > > > > this
> > > > > > > > > > > has
> > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > >>>>> nothing
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>>> to
> > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > >>>>>> do with LOG_ONLY (even though it may be
> > > > > scientific -
> > > > > > > but
> > > > > > > > > > SYNC
> > > > > > > > > > > > > sounds
> > > > > > > > > > > > > > >>>>>> scientific as well).
> > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > >>>>>> I agree with Ivan, we should not go wild
> > with
> > > > > > > renaming.
> > > > > > > > > > > > However, I
> > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > >>>> would
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>>> like to find out what is the meaning behind
> > the
> > > > > > LOG_ONLY
> > > > > > > > > name.
> > > > > > > > > > > Can
> > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > >>>> someone
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>>> explain?
> > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > >>>>> D.
> > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>> --
> > > > > > > > > > > > > > >>>> Best regards,
> > > > > > > > > > > > > > >>>> Ilya
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Sergey Kozlov
> > > > > > GridGain Systems
> > > > > > www.gridgain.com
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Apache Ignite 2.4 release
Posted by Vladimir Ozerov <vo...@gridgain.com>.
Hi,
I merged performance fix for TCP metrics.
Nikolay,
I think it is OK to merge your fix today. Let's agree on a final code
freeze on Monday, 5 Mar.
On Thu, Mar 1, 2018 at 8:24 PM, Denis Magda <dm...@apache.org> wrote:
> Hi Vladimir,
>
> The regression, as well as the performance, is definitely bad. Please work
> with Sergey to see how much time he needs to rerun the whole test set if
> you merge the changes.
>
> I would target the end of the next week as a release date if the vote goes
> well. Probably, we can run the test suites over the weekend if your changes
> get to the branch.
>
> Nikolay Izhikov, follow up with Vladimir and Sergey tomorrow. If they
> decide to merge the fixes then you are free to merge your fixed for Spark
> examples.
>
> --
> Denis
>
> On Thu, Mar 1, 2018 at 7:46 AM, Vladimir Ozerov <vo...@gridgain.com>
> wrote:
>
> > Hi,
> >
> > We found one nasty regression in DROP COLUMN feature and fixed it today.
> > Another problem is performance - we found that recent changes to TCP
> > communication metrics caused considerable slowdown in in-memory mode. I
> > almost fixed it and will merge the fix in the nearest day if benchmarks
> are
> > ok.
> >
> > That said I think we will be able to start vote in the beginning of the
> > next week.
> >
> > Any objections?
> >
> > Vladimir.
> >
> > On Mon, Feb 26, 2018 at 11:22 PM, Denis Magda <dm...@apache.org> wrote:
> >
> > > Good question!
> > >
> > > We've been waiting for a sign-off from Alexey Goncharuk and Sergey
> Kozlov
> > > who are benchmarking and trying to address performance issues in
> > > coopeartion.
> > >
> > > Guys, please share your results and forecast.
> > >
> > > --
> > > Denis
> > >
> > > On Sun, Feb 25, 2018 at 12:10 AM, Pavel Tupitsyn <ptupitsyn@apache.org
> >
> > > wrote:
> > >
> > > > Igniters,
> > > >
> > > > Any update on the 2.4 release status? Anything else to merge there?
> > > >
> > > > Pavel
> > > >
> > > > On Mon, Feb 19, 2018 at 9:30 PM, Dmitriy Setrakyan <
> > > dsetrakyan@apache.org>
> > > > wrote:
> > > >
> > > > > On Mon, Feb 19, 2018 at 1:37 AM, Vladimir Ozerov <
> > vozerov@gridgain.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Alex,
> > > > > >
> > > > > > You get me right. DEFAULT -> LOG_ONLY doesn't introduce any
> > dramatic
> > > > > > changes when comparing 2.3 to 2.4 - Ignite was unsafe out of the
> > box
> > > in
> > > > > > 2.3, and it is unsafe in 2.4 as well.
> > > > > >
> > > > > > The very problem is that we claim ourselves to be ACID, while in
> > > > reality
> > > > > we
> > > > > > are only "AI" out of the box, because durability is not
> guaranteed
> > > due
> > > > to
> > > > > > zero backups and LOG_ONLY and consistency is not guaranteed due
> to
> > > > > > PRIMARY_SYNC. Neither Cassandra, nor Mongo or any others claim
> > > > themselves
> > > > > > to be ACID, so it is not valid to refer to their defaults.
> > > > > >
> > > > >
> > > > > Vladimir,
> > > > > Ignite can be fully ACID, but at the same time have non-ACID
> > defaults,
> > > as
> > > > > long as we clearly document how to get ACID behavior. I do not see
> an
> > > > issue
> > > > > with it.
> > > > >
> > > > > D.
> > > > >
> > > >
> > >
> >
>
Re: Apache Ignite 2.4 release
Posted by Denis Magda <dm...@apache.org>.
Hi Vladimir,
The regression, as well as the performance, is definitely bad. Please work
with Sergey to see how much time he needs to rerun the whole test set if
you merge the changes.
I would target the end of the next week as a release date if the vote goes
well. Probably, we can run the test suites over the weekend if your changes
get to the branch.
Nikolay Izhikov, follow up with Vladimir and Sergey tomorrow. If they
decide to merge the fixes then you are free to merge your fixed for Spark
examples.
--
Denis
On Thu, Mar 1, 2018 at 7:46 AM, Vladimir Ozerov <vo...@gridgain.com>
wrote:
> Hi,
>
> We found one nasty regression in DROP COLUMN feature and fixed it today.
> Another problem is performance - we found that recent changes to TCP
> communication metrics caused considerable slowdown in in-memory mode. I
> almost fixed it and will merge the fix in the nearest day if benchmarks are
> ok.
>
> That said I think we will be able to start vote in the beginning of the
> next week.
>
> Any objections?
>
> Vladimir.
>
> On Mon, Feb 26, 2018 at 11:22 PM, Denis Magda <dm...@apache.org> wrote:
>
> > Good question!
> >
> > We've been waiting for a sign-off from Alexey Goncharuk and Sergey Kozlov
> > who are benchmarking and trying to address performance issues in
> > coopeartion.
> >
> > Guys, please share your results and forecast.
> >
> > --
> > Denis
> >
> > On Sun, Feb 25, 2018 at 12:10 AM, Pavel Tupitsyn <pt...@apache.org>
> > wrote:
> >
> > > Igniters,
> > >
> > > Any update on the 2.4 release status? Anything else to merge there?
> > >
> > > Pavel
> > >
> > > On Mon, Feb 19, 2018 at 9:30 PM, Dmitriy Setrakyan <
> > dsetrakyan@apache.org>
> > > wrote:
> > >
> > > > On Mon, Feb 19, 2018 at 1:37 AM, Vladimir Ozerov <
> vozerov@gridgain.com
> > >
> > > > wrote:
> > > >
> > > > > Alex,
> > > > >
> > > > > You get me right. DEFAULT -> LOG_ONLY doesn't introduce any
> dramatic
> > > > > changes when comparing 2.3 to 2.4 - Ignite was unsafe out of the
> box
> > in
> > > > > 2.3, and it is unsafe in 2.4 as well.
> > > > >
> > > > > The very problem is that we claim ourselves to be ACID, while in
> > > reality
> > > > we
> > > > > are only "AI" out of the box, because durability is not guaranteed
> > due
> > > to
> > > > > zero backups and LOG_ONLY and consistency is not guaranteed due to
> > > > > PRIMARY_SYNC. Neither Cassandra, nor Mongo or any others claim
> > > themselves
> > > > > to be ACID, so it is not valid to refer to their defaults.
> > > > >
> > > >
> > > > Vladimir,
> > > > Ignite can be fully ACID, but at the same time have non-ACID
> defaults,
> > as
> > > > long as we clearly document how to get ACID behavior. I do not see an
> > > issue
> > > > with it.
> > > >
> > > > D.
> > > >
> > >
> >
>
Re: Apache Ignite 2.4 release
Posted by Vladimir Ozerov <vo...@gridgain.com>.
Hi,
We found one nasty regression in DROP COLUMN feature and fixed it today.
Another problem is performance - we found that recent changes to TCP
communication metrics caused considerable slowdown in in-memory mode. I
almost fixed it and will merge the fix in the nearest day if benchmarks are
ok.
That said I think we will be able to start vote in the beginning of the
next week.
Any objections?
Vladimir.
On Mon, Feb 26, 2018 at 11:22 PM, Denis Magda <dm...@apache.org> wrote:
> Good question!
>
> We've been waiting for a sign-off from Alexey Goncharuk and Sergey Kozlov
> who are benchmarking and trying to address performance issues in
> coopeartion.
>
> Guys, please share your results and forecast.
>
> --
> Denis
>
> On Sun, Feb 25, 2018 at 12:10 AM, Pavel Tupitsyn <pt...@apache.org>
> wrote:
>
> > Igniters,
> >
> > Any update on the 2.4 release status? Anything else to merge there?
> >
> > Pavel
> >
> > On Mon, Feb 19, 2018 at 9:30 PM, Dmitriy Setrakyan <
> dsetrakyan@apache.org>
> > wrote:
> >
> > > On Mon, Feb 19, 2018 at 1:37 AM, Vladimir Ozerov <vozerov@gridgain.com
> >
> > > wrote:
> > >
> > > > Alex,
> > > >
> > > > You get me right. DEFAULT -> LOG_ONLY doesn't introduce any dramatic
> > > > changes when comparing 2.3 to 2.4 - Ignite was unsafe out of the box
> in
> > > > 2.3, and it is unsafe in 2.4 as well.
> > > >
> > > > The very problem is that we claim ourselves to be ACID, while in
> > reality
> > > we
> > > > are only "AI" out of the box, because durability is not guaranteed
> due
> > to
> > > > zero backups and LOG_ONLY and consistency is not guaranteed due to
> > > > PRIMARY_SYNC. Neither Cassandra, nor Mongo or any others claim
> > themselves
> > > > to be ACID, so it is not valid to refer to their defaults.
> > > >
> > >
> > > Vladimir,
> > > Ignite can be fully ACID, but at the same time have non-ACID defaults,
> as
> > > long as we clearly document how to get ACID behavior. I do not see an
> > issue
> > > with it.
> > >
> > > D.
> > >
> >
>
Re: Apache Ignite 2.4 release
Posted by Denis Magda <dm...@apache.org>.
Good question!
We've been waiting for a sign-off from Alexey Goncharuk and Sergey Kozlov
who are benchmarking and trying to address performance issues in
coopeartion.
Guys, please share your results and forecast.
--
Denis
On Sun, Feb 25, 2018 at 12:10 AM, Pavel Tupitsyn <pt...@apache.org>
wrote:
> Igniters,
>
> Any update on the 2.4 release status? Anything else to merge there?
>
> Pavel
>
> On Mon, Feb 19, 2018 at 9:30 PM, Dmitriy Setrakyan <ds...@apache.org>
> wrote:
>
> > On Mon, Feb 19, 2018 at 1:37 AM, Vladimir Ozerov <vo...@gridgain.com>
> > wrote:
> >
> > > Alex,
> > >
> > > You get me right. DEFAULT -> LOG_ONLY doesn't introduce any dramatic
> > > changes when comparing 2.3 to 2.4 - Ignite was unsafe out of the box in
> > > 2.3, and it is unsafe in 2.4 as well.
> > >
> > > The very problem is that we claim ourselves to be ACID, while in
> reality
> > we
> > > are only "AI" out of the box, because durability is not guaranteed due
> to
> > > zero backups and LOG_ONLY and consistency is not guaranteed due to
> > > PRIMARY_SYNC. Neither Cassandra, nor Mongo or any others claim
> themselves
> > > to be ACID, so it is not valid to refer to their defaults.
> > >
> >
> > Vladimir,
> > Ignite can be fully ACID, but at the same time have non-ACID defaults, as
> > long as we clearly document how to get ACID behavior. I do not see an
> issue
> > with it.
> >
> > D.
> >
>
Re: Apache Ignite 2.4 release
Posted by Pavel Tupitsyn <pt...@apache.org>.
Igniters,
Any update on the 2.4 release status? Anything else to merge there?
Pavel
On Mon, Feb 19, 2018 at 9:30 PM, Dmitriy Setrakyan <ds...@apache.org>
wrote:
> On Mon, Feb 19, 2018 at 1:37 AM, Vladimir Ozerov <vo...@gridgain.com>
> wrote:
>
> > Alex,
> >
> > You get me right. DEFAULT -> LOG_ONLY doesn't introduce any dramatic
> > changes when comparing 2.3 to 2.4 - Ignite was unsafe out of the box in
> > 2.3, and it is unsafe in 2.4 as well.
> >
> > The very problem is that we claim ourselves to be ACID, while in reality
> we
> > are only "AI" out of the box, because durability is not guaranteed due to
> > zero backups and LOG_ONLY and consistency is not guaranteed due to
> > PRIMARY_SYNC. Neither Cassandra, nor Mongo or any others claim themselves
> > to be ACID, so it is not valid to refer to their defaults.
> >
>
> Vladimir,
> Ignite can be fully ACID, but at the same time have non-ACID defaults, as
> long as we clearly document how to get ACID behavior. I do not see an issue
> with it.
>
> D.
>
Re: Apache Ignite 2.4 release
Posted by Dmitriy Setrakyan <ds...@apache.org>.
On Mon, Feb 19, 2018 at 1:37 AM, Vladimir Ozerov <vo...@gridgain.com>
wrote:
> Alex,
>
> You get me right. DEFAULT -> LOG_ONLY doesn't introduce any dramatic
> changes when comparing 2.3 to 2.4 - Ignite was unsafe out of the box in
> 2.3, and it is unsafe in 2.4 as well.
>
> The very problem is that we claim ourselves to be ACID, while in reality we
> are only "AI" out of the box, because durability is not guaranteed due to
> zero backups and LOG_ONLY and consistency is not guaranteed due to
> PRIMARY_SYNC. Neither Cassandra, nor Mongo or any others claim themselves
> to be ACID, so it is not valid to refer to their defaults.
>
Vladimir,
Ignite can be fully ACID, but at the same time have non-ACID defaults, as
long as we clearly document how to get ACID behavior. I do not see an issue
with it.
D.
Re: Apache Ignite 2.4 release
Posted by Vladimir Ozerov <vo...@gridgain.com>.
Alex,
You get me right. DEFAULT -> LOG_ONLY doesn't introduce any dramatic
changes when comparing 2.3 to 2.4 - Ignite was unsafe out of the box in
2.3, and it is unsafe in 2.4 as well.
The very problem is that we claim ourselves to be ACID, while in reality we
are only "AI" out of the box, because durability is not guaranteed due to
zero backups and LOG_ONLY and consistency is not guaranteed due to
PRIMARY_SYNC. Neither Cassandra, nor Mongo or any others claim themselves
to be ACID, so it is not valid to refer to their defaults.
On Mon, Feb 19, 2018 at 10:06 AM, Alexey Goncharuk <
alexey.goncharuk@gmail.com> wrote:
> In terms of 'safety', Ignite default settings are far beyond optimal. For
> in-memory mode, we have 0 backups by default, which means partition loss in
> a case of node failure, we have readFromBackup=true and PRIMARY_SYNC by
> default which effectively cancels linearizability property for cache
> updates, so setting the default WAL mode to LOG_ONLY does not seem to be a
> bigger evil than it currently is. If we are to move to safer defaults, we
> should change all of the affected sides.
>
> I also want to clarify the difference between guarantees in
> non-fsync modes. We should distinguish the loss of durability (the loss of
> the last update) because the update did not make it to disk and data loss
> because the disk content was shuffled due to an incomplete page write. In
> my understanding, the current situation is:
> FSYNC: loss of durability: not possible, data loss: not possible
> LOG_ONLY: loss of durability: possible only if OS/power fails, data loss:
> possible only if OS/power fails
> BACKGROUND: loss of durability: possible if Ignite process fails, data
> loss: possible only if OS/power fails
>
> The data loss situation can be mitigated in the cluster using a large
> enough replication factor (this is what Dmitriy was describing in the case
> of LOG_ONLY and 3 backups configuration).
>
> Denis,
> I do not think it is fair to compare Ignite defaults to Cassandra's
> defaults because Cassandra is _not_ transactional _eventually consistent_
> datastore, they claim much weaker guarantees than Ignite.
>
> All in all, I'm ok to change the WAL default right now, but I would revisit
> all those settings in 3.0 and made Ignite safe-first.
>
> 2018-02-17 3:24 GMT+03:00 Denis Magda <dm...@apache.org>:
>
> > Classic relational databases have no choice rather than to use FSYNC by
> > default. RDBMS is all about consistency.
> >
> > Distributed databases try to balance consistency and performance. For
> > instance, why to fsync every update if there is usually 1 backup copy?
> > This is probably why VoltDB [1] and Cassandra use the modes comparable to
> > Ignite's LOG_ONLY.
> >
> > Ignite as a distributed database should care of both consistency and
> > performance.
> >
> > My vote goes to FSYNC, LOG_ONLY (default), BACKGROUND, NONE.
> >
> >
> > [1] https://docs.voltdb.com/UsingVoltDB/CmdLogConfig.php
> >
> > --
> > Denis
> >
> >
> > On Fri, Feb 16, 2018 at 2:14 PM, Dmitriy Setrakyan <
> dsetrakyan@apache.org>
> > wrote:
> >
> > > Vova,
> > >
> > > I hear your concerns, but at the same time I know that one of the
> largest
> > > banks in eastern Europe is using Ignite in LOG_ONLY mode with 3 backups
> > to
> > > move money. The rational is that the probability of failure of 4
> servers
> > at
> > > hardware level at the same time is very low. However, if the JVM
> process
> > > fails on any server, then it can be safely restarted without loosing
> > data.
> > > In my view, this is why LOG_ONLY mode makes sense as a default.
> > >
> > > I still vote to change the default to LOG_ONLY, deprecate the DEFAULT
> > name
> > > altogether and add FSYNC mode instead.
> > >
> > > D.
> > >
> > > On Fri, Feb 16, 2018 at 4:05 PM, Vladimir Ozerov <vozerov@gridgain.com
> >
> > > wrote:
> > >
> > > > Sergey,
> > > >
> > > > We do not have backups by default either, so essentially we are
> loosing
> > > > data by default. Moreover, backups are less reliable option than
> fsync
> > > > because a lot of users cannot afford putting servers into separate
> > power
> > > > circuits, so a single power failure may easily lead to poweroff of
> the
> > > > whole cluster at once, so data is lost still. This is normal practice
> > > even
> > > > for enterprise deployments (e.g. asynchronous replication).
> > > >
> > > > To make things even worse, we employ PRIMARY_SYNC mode by default! So
> > > even
> > > > if you configured backups, you still may loose data due to a single
> > node
> > > > failure - just shutdown the PRIMARY after commit is confirmed to the
> > > client
> > > > and your recent update will disappers.
> > > >
> > > > So this is what user should do to make himself safe:
> > > > 1) Learn about WAL modes
> > > > 2) Learn about backups
> > > > 3) Learn about synchronization modes
> > > > 4) Cross his fingers that he understood everything correctly and that
> > > there
> > > > are no other hidden surprises in Ignite which could lead to data
> loss.
> > > >
> > > > Way to much for a product, claiming to be A*C*ID and persistent,
> don't
> > > you
> > > > think so?
> > > >
> > > > Leaving deafult WAL mode with fsync resolves all these issues.
> > > >
> > > > Vladimir.
> > > >
> > > >
> > > > On Fri, Feb 16, 2018 at 11:43 PM, Sergey Kozlov <
> skozlov@gridgain.com>
> > > > wrote:
> > > >
> > > > > I suppose some approaches used by classic databases makes no sense
> > for
> > > > > Ignite. FSYNC requirement for databases has the nature of single
> host
> > > > > solution. If you have corrupted db files you have corrupted (lost)
> > > data.
> > > > >
> > > > > For Ignite the enough number of backups and the failure detecting
> > logic
> > > > can
> > > > > provide the data consistency in term "cluster data consistency".
> > > > >
> > > > >
> > > > >
> > > > > On Fri, Feb 16, 2018 at 8:57 PM, Dmitry Pavlov <
> > dpavlov.spb@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi, all WAL modes except NONE protects from data consistency
> > problem
> > > > > > (B+Tree, pages, etc), which is why I suggest to avoid saying
> > > > 'corrupted'
> > > > > > about 'unapplied updates'.
> > > > > >
> > > > > > Log Only and Background may cause unapplied updates in case of
> > > > OS/process
> > > > > > failures.
> > > > > >
> > > > > > None mode IMO is not an option in case data consistency is
> needed.
> > > > > >
> > > > > > пт, 16 февр. 2018 г. в 20:49, Valentin Kulichenko <
> > > > > > valentin.kulichenko@gmail.com>:
> > > > > >
> > > > > > > Guys,
> > > > > > >
> > > > > > > While we're on this topic, what is the difference between
> > > BACKGROUND
> > > > > and
> > > > > > > NONE in terms of semantics and provided guarantees? To me it
> > looks
> > > > like
> > > > > > > both guarantee to recover the state since last checkpoint and
> > > > anything
> > > > > > else
> > > > > > > can potentially be lost, so from user perspective they are the
> > > same.
> > > > > Am I
> > > > > > > missing something here?
> > > > > > >
> > > > > > > Also there is the following in Javadoc for NONE: "If an Ignite
> > node
> > > > is
> > > > > > > terminated in NONE mode abruptly, it is likely that the data
> > stored
> > > > on
> > > > > > disk
> > > > > > > is corrupted and work directory will need to be cleared for a
> > node
> > > > > > > restart.". If this is really the case, I'm not sure NONE makes
> > > sense
> > > > at
> > > > > > > all. Why would I enable persistence if I'm likely to clear the
> > > > storage
> > > > > on
> > > > > > > restart?
> > > > > > >
> > > > > > > -Val
> > > > > > >
> > > > > > > On Fri, Feb 16, 2018 at 8:39 AM, Vladimir Ozerov <
> > > > vozerov@gridgain.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > What is the reason to have DEFAULT mode at all if you claim
> > > > LOG_ONLY
> > > > > to
> > > > > > > be
> > > > > > > > completely safe? :)
> > > > > > > >
> > > > > > > > And how it could be safe provided that without fsync we loose
> > > part
> > > > of
> > > > > > WAL
> > > > > > > > itself in case of crash?
> > > > > > > >
> > > > > > > > пт, 16 февр. 2018 г. в 19:32, Dmitry Pavlov <
> > > dpavlov.spb@gmail.com
> > > > >:
> > > > > > > >
> > > > > > > > > Thank you. Data can't be corrupted in case crash because of
> > WAL
> > > > > > replay
> > > > > > > > > (since completed checkpoint). Physical records are used to
> > > > restore
> > > > > > > > probably
> > > > > > > > > corrupted pages in persistent store (we overwrite so called
> > > 'grey
> > > > > > > zone' -
> > > > > > > > > pages we don't know for sure if they have been written).
> > > > > > > > >
> > > > > > > > > Only one effect is unwritten one or several last
> > transactions.
> > > It
> > > > > is
> > > > > > > not
> > > > > > > > > the same with corrupted data.
> > > > > > > > >
> > > > > > > > > пт, 16 февр. 2018 г. в 19:19, Vladimir Ozerov <
> > > > > vozerov@gridgain.com
> > > > > > >:
> > > > > > > > >
> > > > > > > > > > Log only mode is not safe - data might be corrupted in
> case
> > > of
> > > > > > system
> > > > > > > > > > crash. Oracle - fsync, Postgres - fsync, SQL Server -
> > fsync,
> > > > > > > Cassandra
> > > > > > > > -
> > > > > > > > > > similar to our “background”.
> > > > > > > > > >
> > > > > > > > > > пт, 16 февр. 2018 г. в 19:11, Dmitry Pavlov <
> > > > > dpavlov.spb@gmail.com
> > > > > > >:
> > > > > > > > > >
> > > > > > > > > > > Hi Vladimir,
> > > > > > > > > > >
> > > > > > > > > > > What you saying is defenetely make sence.
> > > > > > > > > > >
> > > > > > > > > > > In the same time LOG_ONLY is also safe mode, user will
> be
> > > > able
> > > > > to
> > > > > > > > > restore
> > > > > > > > > > > system after crash. If it is not true, we should create
> > > > > critical
> > > > > > > > ticket
> > > > > > > > > > and
> > > > > > > > > > > fix it.
> > > > > > > > > > >
> > > > > > > > > > > Do you know other databases defaults, such as
> Cassandra,
> > > > > Oracle,
> > > > > > > > > Postgre?
> > > > > > > > > > >
> > > > > > > > > > > Sincerely,
> > > > > > > > > > > Dmitriy Pavlov
> > > > > > > > > > >
> > > > > > > > > > > пт, 16 февр. 2018 г. в 18:41, Vladimir Ozerov <
> > > > > > > vozerov@gridgain.com
> > > > > > > > >:
> > > > > > > > > > >
> > > > > > > > > > > > Igniters,
> > > > > > > > > > > >
> > > > > > > > > > > > Sorry for pouring oil on the flames, but from
> database
> > > > > > > perspective
> > > > > > > > > > moving
> > > > > > > > > > > > from FSYNC to non-FSYNC mode appears to be a mistake.
> > > When
> > > > > you
> > > > > > > work
> > > > > > > > > > with
> > > > > > > > > > > > database, your main expectation is that it will save
> > your
> > > > > data.
> > > > > > > All
> > > > > > > > > > > > production database vendor make sure that you are
> safe,
> > > not
> > > > > > that
> > > > > > > > you
> > > > > > > > > > are
> > > > > > > > > > > > fast. Moreover, some vendors even prevent you from
> > being
> > > in
> > > > > > > unsafe
> > > > > > > > > mode
> > > > > > > > > > > > (e.g. you cannot disable fsync in SQL Server at all).
> > > > > > > > > > > >
> > > > > > > > > > > > If we continue going in this direction, we will end
> up
> > > > with a
> > > > > > > > > product,
> > > > > > > > > > > > which is unsafe out of the box and require tons of
> > > > > > documentation
> > > > > > > to
> > > > > > > > > > > > understand how to make it safe. Definitely not the
> > right
> > > > > > message
> > > > > > > to
> > > > > > > > > the
> > > > > > > > > > > > market. This is like a car without brakes - would you
> > > like
> > > > to
> > > > > > > drive
> > > > > > > > > it?
> > > > > > > > > > > If
> > > > > > > > > > > > this is Need For Speed game and you have unlimited
> > lives
> > > > > > > (in-memory
> > > > > > > > > > cache
> > > > > > > > > > > > with backing store), then yes. If this is a real life
> > > with
> > > > > > > > > > (persistence)
> > > > > > > > > > > -
> > > > > > > > > > > > then no.
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Feb 16, 2018 at 5:20 PM, Dmitriy Setrakyan <
> > > > > > > > > > > dsetrakyan@apache.org>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Well, I cannot say that I like the name LOG_ONLY,
> > but I
> > > > > would
> > > > > > > > vote
> > > > > > > > > to
> > > > > > > > > > > > keep
> > > > > > > > > > > > > it for now, given that it is already documented in
> > many
> > > > > > places,
> > > > > > > > > > blogs,
> > > > > > > > > > > > and
> > > > > > > > > > > > > examples.
> > > > > > > > > > > > >
> > > > > > > > > > > > > D.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Feb 16, 2018 at 8:13 AM, Ivan Rakov <
> > > > > > > > ivan.glukos@gmail.com
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Looks like it's an Ignite term - I've never heard
> > of
> > > it
> > > > > > > outside
> > > > > > > > > > > Ignite
> > > > > > > > > > > > > > scope.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Though, renaming existing enum value requires
> > keeping
> > > > old
> > > > > > as
> > > > > > > > > > > > deprecated.
> > > > > > > > > > > > > > DEFAULT is confusing enough to pay this price.
> > > > > > > > > > > > > > As for LOG_ONLY, I think we can keep it as long
> as
> > it
> > > > has
> > > > > > > good
> > > > > > > > > and
> > > > > > > > > > > > > > definitive javadoc.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best Regards,
> > > > > > > > > > > > > > Ivan Rakov
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On 16.02.2018 17:07, Dmitriy Setrakyan wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >> Igniters, just to clarify, does the term
> LOG_ONLY
> > > mean
> > > > > > > > anything
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > >> industry or is this just an Ignite term?
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> D.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> On Fri, Feb 16, 2018 at 8:03 AM, Anton
> Vinogradov
> > <
> > > > > > > > > > > > > >> avinogradov@gridgain.com>
> > > > > > > > > > > > > >> wrote:
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Log only mode: flushes application buffers.
> > > > > > > > > > > > > >>> So, in synced mode without fsync guarantee.
> > That's
> > > > why
> > > > > I
> > > > > > > > > propose
> > > > > > > > > > to
> > > > > > > > > > > > > >>> rename
> > > > > > > > > > > > > >>> it as SYNC.
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>> On Fri, Feb 16, 2018 at 4:49 PM, Ilya Lantukh <
> > > > > > > > > > > ilantukh@gridgain.com
> > > > > > > > > > > > >
> > > > > > > > > > > > > >>> wrote:
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>> I am OK with either FSYNC or STRICT variant.
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>> LOG_ONLY name means "log without fsync".
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>> On Fri, Feb 16, 2018 at 4:05 PM, Dmitriy
> > > Setrakyan <
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>> dsetrakyan@apache.org>
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>>> wrote:
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>> On Fri, Feb 16, 2018 at 7:02 AM, Ivan Rakov <
> > > > > > > > > > > ivan.glukos@gmail.com>
> > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > >>>> wrote:
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>>> Why create a new term to define something
> that
> > > has
> > > > > > > already
> > > > > > > > > been
> > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > >>>>> defined?
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>>> That makes sense. I'm ok with FSYNC.
> > > > > > > > > > > > > >>>>>> Anton, I don't understand why we should
> rename
> > > > > > LOG_ONLY
> > > > > > > to
> > > > > > > > > > SYNC.
> > > > > > > > > > > > We
> > > > > > > > > > > > > >>>>>> started this discussion with bad naming of
> > > > DEFAULT,
> > > > > > but
> > > > > > > > this
> > > > > > > > > > has
> > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > >>>>> nothing
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>>> to
> > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > >>>>>> do with LOG_ONLY (even though it may be
> > > > scientific -
> > > > > > but
> > > > > > > > > SYNC
> > > > > > > > > > > > sounds
> > > > > > > > > > > > > >>>>>> scientific as well).
> > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > >>>>>> I agree with Ivan, we should not go wild
> with
> > > > > > renaming.
> > > > > > > > > > > However, I
> > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > >>>> would
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>>> like to find out what is the meaning behind
> the
> > > > > LOG_ONLY
> > > > > > > > name.
> > > > > > > > > > Can
> > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > >>>> someone
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>>> explain?
> > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > >>>>> D.
> > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>> --
> > > > > > > > > > > > > >>>> Best regards,
> > > > > > > > > > > > > >>>> Ilya
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Sergey Kozlov
> > > > > GridGain Systems
> > > > > www.gridgain.com
> > > > >
> > > >
> > >
> >
>
Re: Apache Ignite 2.4 release
Posted by Dmitry Pavlov <dp...@gmail.com>.
Hi Folks,
We have performance suggestions logged during Ignite start. Can we create
data safety suggestions and list all these options during startup?
Arguments about Ignite is distributed and may have backup for protection
make sence.
Who can provide reference to test of such or similar scenario:
- distributed recovery when LOG_ONLY mode is enabled on 2 nodes, 1 backup.
- and 1 node is crashed during WAL logging of transaction with huge number
of records. But node 2 was managed to log all records in TX. (Node1 WAL:
RR crash, Node2 WAL: RRRRRR).
- node 1 returns to cluster
- it should be possible to see consistent state with all TX data after
delta rebalancing.
If such test exist, runs in TC and is not flaky I'm also ok with new
relaxed default.
Sincerely,
Dmitriy Pavlov
пн, 19 февр. 2018 г. в 10:06, Alexey Goncharuk <al...@gmail.com>:
> In terms of 'safety', Ignite default settings are far beyond optimal. For
> in-memory mode, we have 0 backups by default, which means partition loss in
> a case of node failure, we have readFromBackup=true and PRIMARY_SYNC by
> default which effectively cancels linearizability property for cache
> updates, so setting the default WAL mode to LOG_ONLY does not seem to be a
> bigger evil than it currently is. If we are to move to safer defaults, we
> should change all of the affected sides.
>
> I also want to clarify the difference between guarantees in
> non-fsync modes. We should distinguish the loss of durability (the loss of
> the last update) because the update did not make it to disk and data loss
> because the disk content was shuffled due to an incomplete page write. In
> my understanding, the current situation is:
> FSYNC: loss of durability: not possible, data loss: not possible
> LOG_ONLY: loss of durability: possible only if OS/power fails, data loss:
> possible only if OS/power fails
> BACKGROUND: loss of durability: possible if Ignite process fails, data
> loss: possible only if OS/power fails
>
> The data loss situation can be mitigated in the cluster using a large
> enough replication factor (this is what Dmitriy was describing in the case
> of LOG_ONLY and 3 backups configuration).
>
> Denis,
> I do not think it is fair to compare Ignite defaults to Cassandra's
> defaults because Cassandra is _not_ transactional _eventually consistent_
> datastore, they claim much weaker guarantees than Ignite.
>
> All in all, I'm ok to change the WAL default right now, but I would revisit
> all those settings in 3.0 and made Ignite safe-first.
>
> 2018-02-17 3:24 GMT+03:00 Denis Magda <dm...@apache.org>:
>
> > Classic relational databases have no choice rather than to use FSYNC by
> > default. RDBMS is all about consistency.
> >
> > Distributed databases try to balance consistency and performance. For
> > instance, why to fsync every update if there is usually 1 backup copy?
> > This is probably why VoltDB [1] and Cassandra use the modes comparable to
> > Ignite's LOG_ONLY.
> >
> > Ignite as a distributed database should care of both consistency and
> > performance.
> >
> > My vote goes to FSYNC, LOG_ONLY (default), BACKGROUND, NONE.
> >
> >
> > [1] https://docs.voltdb.com/UsingVoltDB/CmdLogConfig.php
> >
> > --
> > Denis
> >
> >
> > On Fri, Feb 16, 2018 at 2:14 PM, Dmitriy Setrakyan <
> dsetrakyan@apache.org>
> > wrote:
> >
> > > Vova,
> > >
> > > I hear your concerns, but at the same time I know that one of the
> largest
> > > banks in eastern Europe is using Ignite in LOG_ONLY mode with 3 backups
> > to
> > > move money. The rational is that the probability of failure of 4
> servers
> > at
> > > hardware level at the same time is very low. However, if the JVM
> process
> > > fails on any server, then it can be safely restarted without loosing
> > data.
> > > In my view, this is why LOG_ONLY mode makes sense as a default.
> > >
> > > I still vote to change the default to LOG_ONLY, deprecate the DEFAULT
> > name
> > > altogether and add FSYNC mode instead.
> > >
> > > D.
> > >
> > > On Fri, Feb 16, 2018 at 4:05 PM, Vladimir Ozerov <vozerov@gridgain.com
> >
> > > wrote:
> > >
> > > > Sergey,
> > > >
> > > > We do not have backups by default either, so essentially we are
> loosing
> > > > data by default. Moreover, backups are less reliable option than
> fsync
> > > > because a lot of users cannot afford putting servers into separate
> > power
> > > > circuits, so a single power failure may easily lead to poweroff of
> the
> > > > whole cluster at once, so data is lost still. This is normal practice
> > > even
> > > > for enterprise deployments (e.g. asynchronous replication).
> > > >
> > > > To make things even worse, we employ PRIMARY_SYNC mode by default! So
> > > even
> > > > if you configured backups, you still may loose data due to a single
> > node
> > > > failure - just shutdown the PRIMARY after commit is confirmed to the
> > > client
> > > > and your recent update will disappers.
> > > >
> > > > So this is what user should do to make himself safe:
> > > > 1) Learn about WAL modes
> > > > 2) Learn about backups
> > > > 3) Learn about synchronization modes
> > > > 4) Cross his fingers that he understood everything correctly and that
> > > there
> > > > are no other hidden surprises in Ignite which could lead to data
> loss.
> > > >
> > > > Way to much for a product, claiming to be A*C*ID and persistent,
> don't
> > > you
> > > > think so?
> > > >
> > > > Leaving deafult WAL mode with fsync resolves all these issues.
> > > >
> > > > Vladimir.
> > > >
> > > >
> > > > On Fri, Feb 16, 2018 at 11:43 PM, Sergey Kozlov <
> skozlov@gridgain.com>
> > > > wrote:
> > > >
> > > > > I suppose some approaches used by classic databases makes no sense
> > for
> > > > > Ignite. FSYNC requirement for databases has the nature of single
> host
> > > > > solution. If you have corrupted db files you have corrupted (lost)
> > > data.
> > > > >
> > > > > For Ignite the enough number of backups and the failure detecting
> > logic
> > > > can
> > > > > provide the data consistency in term "cluster data consistency".
> > > > >
> > > > >
> > > > >
> > > > > On Fri, Feb 16, 2018 at 8:57 PM, Dmitry Pavlov <
> > dpavlov.spb@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi, all WAL modes except NONE protects from data consistency
> > problem
> > > > > > (B+Tree, pages, etc), which is why I suggest to avoid saying
> > > > 'corrupted'
> > > > > > about 'unapplied updates'.
> > > > > >
> > > > > > Log Only and Background may cause unapplied updates in case of
> > > > OS/process
> > > > > > failures.
> > > > > >
> > > > > > None mode IMO is not an option in case data consistency is
> needed.
> > > > > >
> > > > > > пт, 16 февр. 2018 г. в 20:49, Valentin Kulichenko <
> > > > > > valentin.kulichenko@gmail.com>:
> > > > > >
> > > > > > > Guys,
> > > > > > >
> > > > > > > While we're on this topic, what is the difference between
> > > BACKGROUND
> > > > > and
> > > > > > > NONE in terms of semantics and provided guarantees? To me it
> > looks
> > > > like
> > > > > > > both guarantee to recover the state since last checkpoint and
> > > > anything
> > > > > > else
> > > > > > > can potentially be lost, so from user perspective they are the
> > > same.
> > > > > Am I
> > > > > > > missing something here?
> > > > > > >
> > > > > > > Also there is the following in Javadoc for NONE: "If an Ignite
> > node
> > > > is
> > > > > > > terminated in NONE mode abruptly, it is likely that the data
> > stored
> > > > on
> > > > > > disk
> > > > > > > is corrupted and work directory will need to be cleared for a
> > node
> > > > > > > restart.". If this is really the case, I'm not sure NONE makes
> > > sense
> > > > at
> > > > > > > all. Why would I enable persistence if I'm likely to clear the
> > > > storage
> > > > > on
> > > > > > > restart?
> > > > > > >
> > > > > > > -Val
> > > > > > >
> > > > > > > On Fri, Feb 16, 2018 at 8:39 AM, Vladimir Ozerov <
> > > > vozerov@gridgain.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > What is the reason to have DEFAULT mode at all if you claim
> > > > LOG_ONLY
> > > > > to
> > > > > > > be
> > > > > > > > completely safe? :)
> > > > > > > >
> > > > > > > > And how it could be safe provided that without fsync we loose
> > > part
> > > > of
> > > > > > WAL
> > > > > > > > itself in case of crash?
> > > > > > > >
> > > > > > > > пт, 16 февр. 2018 г. в 19:32, Dmitry Pavlov <
> > > dpavlov.spb@gmail.com
> > > > >:
> > > > > > > >
> > > > > > > > > Thank you. Data can't be corrupted in case crash because of
> > WAL
> > > > > > replay
> > > > > > > > > (since completed checkpoint). Physical records are used to
> > > > restore
> > > > > > > > probably
> > > > > > > > > corrupted pages in persistent store (we overwrite so called
> > > 'grey
> > > > > > > zone' -
> > > > > > > > > pages we don't know for sure if they have been written).
> > > > > > > > >
> > > > > > > > > Only one effect is unwritten one or several last
> > transactions.
> > > It
> > > > > is
> > > > > > > not
> > > > > > > > > the same with corrupted data.
> > > > > > > > >
> > > > > > > > > пт, 16 февр. 2018 г. в 19:19, Vladimir Ozerov <
> > > > > vozerov@gridgain.com
> > > > > > >:
> > > > > > > > >
> > > > > > > > > > Log only mode is not safe - data might be corrupted in
> case
> > > of
> > > > > > system
> > > > > > > > > > crash. Oracle - fsync, Postgres - fsync, SQL Server -
> > fsync,
> > > > > > > Cassandra
> > > > > > > > -
> > > > > > > > > > similar to our “background”.
> > > > > > > > > >
> > > > > > > > > > пт, 16 февр. 2018 г. в 19:11, Dmitry Pavlov <
> > > > > dpavlov.spb@gmail.com
> > > > > > >:
> > > > > > > > > >
> > > > > > > > > > > Hi Vladimir,
> > > > > > > > > > >
> > > > > > > > > > > What you saying is defenetely make sence.
> > > > > > > > > > >
> > > > > > > > > > > In the same time LOG_ONLY is also safe mode, user will
> be
> > > > able
> > > > > to
> > > > > > > > > restore
> > > > > > > > > > > system after crash. If it is not true, we should create
> > > > > critical
> > > > > > > > ticket
> > > > > > > > > > and
> > > > > > > > > > > fix it.
> > > > > > > > > > >
> > > > > > > > > > > Do you know other databases defaults, such as
> Cassandra,
> > > > > Oracle,
> > > > > > > > > Postgre?
> > > > > > > > > > >
> > > > > > > > > > > Sincerely,
> > > > > > > > > > > Dmitriy Pavlov
> > > > > > > > > > >
> > > > > > > > > > > пт, 16 февр. 2018 г. в 18:41, Vladimir Ozerov <
> > > > > > > vozerov@gridgain.com
> > > > > > > > >:
> > > > > > > > > > >
> > > > > > > > > > > > Igniters,
> > > > > > > > > > > >
> > > > > > > > > > > > Sorry for pouring oil on the flames, but from
> database
> > > > > > > perspective
> > > > > > > > > > moving
> > > > > > > > > > > > from FSYNC to non-FSYNC mode appears to be a mistake.
> > > When
> > > > > you
> > > > > > > work
> > > > > > > > > > with
> > > > > > > > > > > > database, your main expectation is that it will save
> > your
> > > > > data.
> > > > > > > All
> > > > > > > > > > > > production database vendor make sure that you are
> safe,
> > > not
> > > > > > that
> > > > > > > > you
> > > > > > > > > > are
> > > > > > > > > > > > fast. Moreover, some vendors even prevent you from
> > being
> > > in
> > > > > > > unsafe
> > > > > > > > > mode
> > > > > > > > > > > > (e.g. you cannot disable fsync in SQL Server at all).
> > > > > > > > > > > >
> > > > > > > > > > > > If we continue going in this direction, we will end
> up
> > > > with a
> > > > > > > > > product,
> > > > > > > > > > > > which is unsafe out of the box and require tons of
> > > > > > documentation
> > > > > > > to
> > > > > > > > > > > > understand how to make it safe. Definitely not the
> > right
> > > > > > message
> > > > > > > to
> > > > > > > > > the
> > > > > > > > > > > > market. This is like a car without brakes - would you
> > > like
> > > > to
> > > > > > > drive
> > > > > > > > > it?
> > > > > > > > > > > If
> > > > > > > > > > > > this is Need For Speed game and you have unlimited
> > lives
> > > > > > > (in-memory
> > > > > > > > > > cache
> > > > > > > > > > > > with backing store), then yes. If this is a real life
> > > with
> > > > > > > > > > (persistence)
> > > > > > > > > > > -
> > > > > > > > > > > > then no.
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Feb 16, 2018 at 5:20 PM, Dmitriy Setrakyan <
> > > > > > > > > > > dsetrakyan@apache.org>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Well, I cannot say that I like the name LOG_ONLY,
> > but I
> > > > > would
> > > > > > > > vote
> > > > > > > > > to
> > > > > > > > > > > > keep
> > > > > > > > > > > > > it for now, given that it is already documented in
> > many
> > > > > > places,
> > > > > > > > > > blogs,
> > > > > > > > > > > > and
> > > > > > > > > > > > > examples.
> > > > > > > > > > > > >
> > > > > > > > > > > > > D.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Feb 16, 2018 at 8:13 AM, Ivan Rakov <
> > > > > > > > ivan.glukos@gmail.com
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Looks like it's an Ignite term - I've never heard
> > of
> > > it
> > > > > > > outside
> > > > > > > > > > > Ignite
> > > > > > > > > > > > > > scope.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Though, renaming existing enum value requires
> > keeping
> > > > old
> > > > > > as
> > > > > > > > > > > > deprecated.
> > > > > > > > > > > > > > DEFAULT is confusing enough to pay this price.
> > > > > > > > > > > > > > As for LOG_ONLY, I think we can keep it as long
> as
> > it
> > > > has
> > > > > > > good
> > > > > > > > > and
> > > > > > > > > > > > > > definitive javadoc.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best Regards,
> > > > > > > > > > > > > > Ivan Rakov
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On 16.02.2018 17:07, Dmitriy Setrakyan wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >> Igniters, just to clarify, does the term
> LOG_ONLY
> > > mean
> > > > > > > > anything
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > >> industry or is this just an Ignite term?
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> D.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> On Fri, Feb 16, 2018 at 8:03 AM, Anton
> Vinogradov
> > <
> > > > > > > > > > > > > >> avinogradov@gridgain.com>
> > > > > > > > > > > > > >> wrote:
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Log only mode: flushes application buffers.
> > > > > > > > > > > > > >>> So, in synced mode without fsync guarantee.
> > That's
> > > > why
> > > > > I
> > > > > > > > > propose
> > > > > > > > > > to
> > > > > > > > > > > > > >>> rename
> > > > > > > > > > > > > >>> it as SYNC.
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>> On Fri, Feb 16, 2018 at 4:49 PM, Ilya Lantukh <
> > > > > > > > > > > ilantukh@gridgain.com
> > > > > > > > > > > > >
> > > > > > > > > > > > > >>> wrote:
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>> I am OK with either FSYNC or STRICT variant.
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>> LOG_ONLY name means "log without fsync".
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>> On Fri, Feb 16, 2018 at 4:05 PM, Dmitriy
> > > Setrakyan <
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>> dsetrakyan@apache.org>
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>>> wrote:
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>> On Fri, Feb 16, 2018 at 7:02 AM, Ivan Rakov <
> > > > > > > > > > > ivan.glukos@gmail.com>
> > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > >>>> wrote:
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>>> Why create a new term to define something
> that
> > > has
> > > > > > > already
> > > > > > > > > been
> > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > >>>>> defined?
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>>> That makes sense. I'm ok with FSYNC.
> > > > > > > > > > > > > >>>>>> Anton, I don't understand why we should
> rename
> > > > > > LOG_ONLY
> > > > > > > to
> > > > > > > > > > SYNC.
> > > > > > > > > > > > We
> > > > > > > > > > > > > >>>>>> started this discussion with bad naming of
> > > > DEFAULT,
> > > > > > but
> > > > > > > > this
> > > > > > > > > > has
> > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > >>>>> nothing
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>>> to
> > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > >>>>>> do with LOG_ONLY (even though it may be
> > > > scientific -
> > > > > > but
> > > > > > > > > SYNC
> > > > > > > > > > > > sounds
> > > > > > > > > > > > > >>>>>> scientific as well).
> > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > >>>>>> I agree with Ivan, we should not go wild
> with
> > > > > > renaming.
> > > > > > > > > > > However, I
> > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > >>>> would
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>>> like to find out what is the meaning behind
> the
> > > > > LOG_ONLY
> > > > > > > > name.
> > > > > > > > > > Can
> > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > >>>> someone
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>>> explain?
> > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > >>>>> D.
> > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>> --
> > > > > > > > > > > > > >>>> Best regards,
> > > > > > > > > > > > > >>>> Ilya
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Sergey Kozlov
> > > > > GridGain Systems
> > > > > www.gridgain.com
> > > > >
> > > >
> > >
> >
>
Re: Apache Ignite 2.4 release
Posted by Denis Magda <dm...@apache.org>.
Alex, please clarify some of the points
data loss because the disk content was shuffled due to an incomplete page
> write
are you talking about some phase of the checkpointing process? Do you mean
that the process just passed over changes to the file but didn't call
fsync, thus, offloading this to OS to decide?
BACKGROUND: loss of durability: possible if Ignite process fails, data
> loss: possible only if OS/power fails
This should be true if mmap files are not used for BACKGROUND, otherwise,
it provides the same guarantees as LOG_ONLY, right?
--
Denis
On Sun, Feb 18, 2018 at 11:06 PM, Alexey Goncharuk <
alexey.goncharuk@gmail.com> wrote:
> In terms of 'safety', Ignite default settings are far beyond optimal. For
> in-memory mode, we have 0 backups by default, which means partition loss in
> a case of node failure, we have readFromBackup=true and PRIMARY_SYNC by
> default which effectively cancels linearizability property for cache
> updates, so setting the default WAL mode to LOG_ONLY does not seem to be a
> bigger evil than it currently is. If we are to move to safer defaults, we
> should change all of the affected sides.
>
> I also want to clarify the difference between guarantees in
> non-fsync modes. We should distinguish the loss of durability (the loss of
> the last update) because the update did not make it to disk and data loss
> because the disk content was shuffled due to an incomplete page write. In
> my understanding, the current situation is:
> FSYNC: loss of durability: not possible, data loss: not possible
> LOG_ONLY: loss of durability: possible only if OS/power fails, data loss:
> possible only if OS/power fails
> BACKGROUND: loss of durability: possible if Ignite process fails, data
> loss: possible only if OS/power fails
>
> The data loss situation can be mitigated in the cluster using a large
> enough replication factor (this is what Dmitriy was describing in the case
> of LOG_ONLY and 3 backups configuration).
>
> Denis,
> I do not think it is fair to compare Ignite defaults to Cassandra's
> defaults because Cassandra is _not_ transactional _eventually consistent_
> datastore, they claim much weaker guarantees than Ignite.
>
> All in all, I'm ok to change the WAL default right now, but I would revisit
> all those settings in 3.0 and made Ignite safe-first.
>
> 2018-02-17 3:24 GMT+03:00 Denis Magda <dm...@apache.org>:
>
> > Classic relational databases have no choice rather than to use FSYNC by
> > default. RDBMS is all about consistency.
> >
> > Distributed databases try to balance consistency and performance. For
> > instance, why to fsync every update if there is usually 1 backup copy?
> > This is probably why VoltDB [1] and Cassandra use the modes comparable to
> > Ignite's LOG_ONLY.
> >
> > Ignite as a distributed database should care of both consistency and
> > performance.
> >
> > My vote goes to FSYNC, LOG_ONLY (default), BACKGROUND, NONE.
> >
> >
> > [1] https://docs.voltdb.com/UsingVoltDB/CmdLogConfig.php
> >
> > --
> > Denis
> >
> >
> > On Fri, Feb 16, 2018 at 2:14 PM, Dmitriy Setrakyan <
> dsetrakyan@apache.org>
> > wrote:
> >
> > > Vova,
> > >
> > > I hear your concerns, but at the same time I know that one of the
> largest
> > > banks in eastern Europe is using Ignite in LOG_ONLY mode with 3 backups
> > to
> > > move money. The rational is that the probability of failure of 4
> servers
> > at
> > > hardware level at the same time is very low. However, if the JVM
> process
> > > fails on any server, then it can be safely restarted without loosing
> > data.
> > > In my view, this is why LOG_ONLY mode makes sense as a default.
> > >
> > > I still vote to change the default to LOG_ONLY, deprecate the DEFAULT
> > name
> > > altogether and add FSYNC mode instead.
> > >
> > > D.
> > >
> > > On Fri, Feb 16, 2018 at 4:05 PM, Vladimir Ozerov <vozerov@gridgain.com
> >
> > > wrote:
> > >
> > > > Sergey,
> > > >
> > > > We do not have backups by default either, so essentially we are
> loosing
> > > > data by default. Moreover, backups are less reliable option than
> fsync
> > > > because a lot of users cannot afford putting servers into separate
> > power
> > > > circuits, so a single power failure may easily lead to poweroff of
> the
> > > > whole cluster at once, so data is lost still. This is normal practice
> > > even
> > > > for enterprise deployments (e.g. asynchronous replication).
> > > >
> > > > To make things even worse, we employ PRIMARY_SYNC mode by default! So
> > > even
> > > > if you configured backups, you still may loose data due to a single
> > node
> > > > failure - just shutdown the PRIMARY after commit is confirmed to the
> > > client
> > > > and your recent update will disappers.
> > > >
> > > > So this is what user should do to make himself safe:
> > > > 1) Learn about WAL modes
> > > > 2) Learn about backups
> > > > 3) Learn about synchronization modes
> > > > 4) Cross his fingers that he understood everything correctly and that
> > > there
> > > > are no other hidden surprises in Ignite which could lead to data
> loss.
> > > >
> > > > Way to much for a product, claiming to be A*C*ID and persistent,
> don't
> > > you
> > > > think so?
> > > >
> > > > Leaving deafult WAL mode with fsync resolves all these issues.
> > > >
> > > > Vladimir.
> > > >
> > > >
> > > > On Fri, Feb 16, 2018 at 11:43 PM, Sergey Kozlov <
> skozlov@gridgain.com>
> > > > wrote:
> > > >
> > > > > I suppose some approaches used by classic databases makes no sense
> > for
> > > > > Ignite. FSYNC requirement for databases has the nature of single
> host
> > > > > solution. If you have corrupted db files you have corrupted (lost)
> > > data.
> > > > >
> > > > > For Ignite the enough number of backups and the failure detecting
> > logic
> > > > can
> > > > > provide the data consistency in term "cluster data consistency".
> > > > >
> > > > >
> > > > >
> > > > > On Fri, Feb 16, 2018 at 8:57 PM, Dmitry Pavlov <
> > dpavlov.spb@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi, all WAL modes except NONE protects from data consistency
> > problem
> > > > > > (B+Tree, pages, etc), which is why I suggest to avoid saying
> > > > 'corrupted'
> > > > > > about 'unapplied updates'.
> > > > > >
> > > > > > Log Only and Background may cause unapplied updates in case of
> > > > OS/process
> > > > > > failures.
> > > > > >
> > > > > > None mode IMO is not an option in case data consistency is
> needed.
> > > > > >
> > > > > > пт, 16 февр. 2018 г. в 20:49, Valentin Kulichenko <
> > > > > > valentin.kulichenko@gmail.com>:
> > > > > >
> > > > > > > Guys,
> > > > > > >
> > > > > > > While we're on this topic, what is the difference between
> > > BACKGROUND
> > > > > and
> > > > > > > NONE in terms of semantics and provided guarantees? To me it
> > looks
> > > > like
> > > > > > > both guarantee to recover the state since last checkpoint and
> > > > anything
> > > > > > else
> > > > > > > can potentially be lost, so from user perspective they are the
> > > same.
> > > > > Am I
> > > > > > > missing something here?
> > > > > > >
> > > > > > > Also there is the following in Javadoc for NONE: "If an Ignite
> > node
> > > > is
> > > > > > > terminated in NONE mode abruptly, it is likely that the data
> > stored
> > > > on
> > > > > > disk
> > > > > > > is corrupted and work directory will need to be cleared for a
> > node
> > > > > > > restart.". If this is really the case, I'm not sure NONE makes
> > > sense
> > > > at
> > > > > > > all. Why would I enable persistence if I'm likely to clear the
> > > > storage
> > > > > on
> > > > > > > restart?
> > > > > > >
> > > > > > > -Val
> > > > > > >
> > > > > > > On Fri, Feb 16, 2018 at 8:39 AM, Vladimir Ozerov <
> > > > vozerov@gridgain.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > What is the reason to have DEFAULT mode at all if you claim
> > > > LOG_ONLY
> > > > > to
> > > > > > > be
> > > > > > > > completely safe? :)
> > > > > > > >
> > > > > > > > And how it could be safe provided that without fsync we loose
> > > part
> > > > of
> > > > > > WAL
> > > > > > > > itself in case of crash?
> > > > > > > >
> > > > > > > > пт, 16 февр. 2018 г. в 19:32, Dmitry Pavlov <
> > > dpavlov.spb@gmail.com
> > > > >:
> > > > > > > >
> > > > > > > > > Thank you. Data can't be corrupted in case crash because of
> > WAL
> > > > > > replay
> > > > > > > > > (since completed checkpoint). Physical records are used to
> > > > restore
> > > > > > > > probably
> > > > > > > > > corrupted pages in persistent store (we overwrite so called
> > > 'grey
> > > > > > > zone' -
> > > > > > > > > pages we don't know for sure if they have been written).
> > > > > > > > >
> > > > > > > > > Only one effect is unwritten one or several last
> > transactions.
> > > It
> > > > > is
> > > > > > > not
> > > > > > > > > the same with corrupted data.
> > > > > > > > >
> > > > > > > > > пт, 16 февр. 2018 г. в 19:19, Vladimir Ozerov <
> > > > > vozerov@gridgain.com
> > > > > > >:
> > > > > > > > >
> > > > > > > > > > Log only mode is not safe - data might be corrupted in
> case
> > > of
> > > > > > system
> > > > > > > > > > crash. Oracle - fsync, Postgres - fsync, SQL Server -
> > fsync,
> > > > > > > Cassandra
> > > > > > > > -
> > > > > > > > > > similar to our “background”.
> > > > > > > > > >
> > > > > > > > > > пт, 16 февр. 2018 г. в 19:11, Dmitry Pavlov <
> > > > > dpavlov.spb@gmail.com
> > > > > > >:
> > > > > > > > > >
> > > > > > > > > > > Hi Vladimir,
> > > > > > > > > > >
> > > > > > > > > > > What you saying is defenetely make sence.
> > > > > > > > > > >
> > > > > > > > > > > In the same time LOG_ONLY is also safe mode, user will
> be
> > > > able
> > > > > to
> > > > > > > > > restore
> > > > > > > > > > > system after crash. If it is not true, we should create
> > > > > critical
> > > > > > > > ticket
> > > > > > > > > > and
> > > > > > > > > > > fix it.
> > > > > > > > > > >
> > > > > > > > > > > Do you know other databases defaults, such as
> Cassandra,
> > > > > Oracle,
> > > > > > > > > Postgre?
> > > > > > > > > > >
> > > > > > > > > > > Sincerely,
> > > > > > > > > > > Dmitriy Pavlov
> > > > > > > > > > >
> > > > > > > > > > > пт, 16 февр. 2018 г. в 18:41, Vladimir Ozerov <
> > > > > > > vozerov@gridgain.com
> > > > > > > > >:
> > > > > > > > > > >
> > > > > > > > > > > > Igniters,
> > > > > > > > > > > >
> > > > > > > > > > > > Sorry for pouring oil on the flames, but from
> database
> > > > > > > perspective
> > > > > > > > > > moving
> > > > > > > > > > > > from FSYNC to non-FSYNC mode appears to be a mistake.
> > > When
> > > > > you
> > > > > > > work
> > > > > > > > > > with
> > > > > > > > > > > > database, your main expectation is that it will save
> > your
> > > > > data.
> > > > > > > All
> > > > > > > > > > > > production database vendor make sure that you are
> safe,
> > > not
> > > > > > that
> > > > > > > > you
> > > > > > > > > > are
> > > > > > > > > > > > fast. Moreover, some vendors even prevent you from
> > being
> > > in
> > > > > > > unsafe
> > > > > > > > > mode
> > > > > > > > > > > > (e.g. you cannot disable fsync in SQL Server at all).
> > > > > > > > > > > >
> > > > > > > > > > > > If we continue going in this direction, we will end
> up
> > > > with a
> > > > > > > > > product,
> > > > > > > > > > > > which is unsafe out of the box and require tons of
> > > > > > documentation
> > > > > > > to
> > > > > > > > > > > > understand how to make it safe. Definitely not the
> > right
> > > > > > message
> > > > > > > to
> > > > > > > > > the
> > > > > > > > > > > > market. This is like a car without brakes - would you
> > > like
> > > > to
> > > > > > > drive
> > > > > > > > > it?
> > > > > > > > > > > If
> > > > > > > > > > > > this is Need For Speed game and you have unlimited
> > lives
> > > > > > > (in-memory
> > > > > > > > > > cache
> > > > > > > > > > > > with backing store), then yes. If this is a real life
> > > with
> > > > > > > > > > (persistence)
> > > > > > > > > > > -
> > > > > > > > > > > > then no.
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Feb 16, 2018 at 5:20 PM, Dmitriy Setrakyan <
> > > > > > > > > > > dsetrakyan@apache.org>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Well, I cannot say that I like the name LOG_ONLY,
> > but I
> > > > > would
> > > > > > > > vote
> > > > > > > > > to
> > > > > > > > > > > > keep
> > > > > > > > > > > > > it for now, given that it is already documented in
> > many
> > > > > > places,
> > > > > > > > > > blogs,
> > > > > > > > > > > > and
> > > > > > > > > > > > > examples.
> > > > > > > > > > > > >
> > > > > > > > > > > > > D.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Feb 16, 2018 at 8:13 AM, Ivan Rakov <
> > > > > > > > ivan.glukos@gmail.com
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Looks like it's an Ignite term - I've never heard
> > of
> > > it
> > > > > > > outside
> > > > > > > > > > > Ignite
> > > > > > > > > > > > > > scope.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Though, renaming existing enum value requires
> > keeping
> > > > old
> > > > > > as
> > > > > > > > > > > > deprecated.
> > > > > > > > > > > > > > DEFAULT is confusing enough to pay this price.
> > > > > > > > > > > > > > As for LOG_ONLY, I think we can keep it as long
> as
> > it
> > > > has
> > > > > > > good
> > > > > > > > > and
> > > > > > > > > > > > > > definitive javadoc.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best Regards,
> > > > > > > > > > > > > > Ivan Rakov
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On 16.02.2018 17:07, Dmitriy Setrakyan wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >> Igniters, just to clarify, does the term
> LOG_ONLY
> > > mean
> > > > > > > > anything
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > >> industry or is this just an Ignite term?
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> D.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> On Fri, Feb 16, 2018 at 8:03 AM, Anton
> Vinogradov
> > <
> > > > > > > > > > > > > >> avinogradov@gridgain.com>
> > > > > > > > > > > > > >> wrote:
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Log only mode: flushes application buffers.
> > > > > > > > > > > > > >>> So, in synced mode without fsync guarantee.
> > That's
> > > > why
> > > > > I
> > > > > > > > > propose
> > > > > > > > > > to
> > > > > > > > > > > > > >>> rename
> > > > > > > > > > > > > >>> it as SYNC.
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>> On Fri, Feb 16, 2018 at 4:49 PM, Ilya Lantukh <
> > > > > > > > > > > ilantukh@gridgain.com
> > > > > > > > > > > > >
> > > > > > > > > > > > > >>> wrote:
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>> I am OK with either FSYNC or STRICT variant.
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>> LOG_ONLY name means "log without fsync".
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>> On Fri, Feb 16, 2018 at 4:05 PM, Dmitriy
> > > Setrakyan <
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>> dsetrakyan@apache.org>
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>>> wrote:
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>> On Fri, Feb 16, 2018 at 7:02 AM, Ivan Rakov <
> > > > > > > > > > > ivan.glukos@gmail.com>
> > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > >>>> wrote:
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>>> Why create a new term to define something
> that
> > > has
> > > > > > > already
> > > > > > > > > been
> > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > >>>>> defined?
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>>> That makes sense. I'm ok with FSYNC.
> > > > > > > > > > > > > >>>>>> Anton, I don't understand why we should
> rename
> > > > > > LOG_ONLY
> > > > > > > to
> > > > > > > > > > SYNC.
> > > > > > > > > > > > We
> > > > > > > > > > > > > >>>>>> started this discussion with bad naming of
> > > > DEFAULT,
> > > > > > but
> > > > > > > > this
> > > > > > > > > > has
> > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > >>>>> nothing
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>>> to
> > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > >>>>>> do with LOG_ONLY (even though it may be
> > > > scientific -
> > > > > > but
> > > > > > > > > SYNC
> > > > > > > > > > > > sounds
> > > > > > > > > > > > > >>>>>> scientific as well).
> > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > >>>>>> I agree with Ivan, we should not go wild
> with
> > > > > > renaming.
> > > > > > > > > > > However, I
> > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > >>>> would
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>>> like to find out what is the meaning behind
> the
> > > > > LOG_ONLY
> > > > > > > > name.
> > > > > > > > > > Can
> > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > >>>> someone
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>>> explain?
> > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > >>>>> D.
> > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>> --
> > > > > > > > > > > > > >>>> Best regards,
> > > > > > > > > > > > > >>>> Ilya
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Sergey Kozlov
> > > > > GridGain Systems
> > > > > www.gridgain.com
> > > > >
> > > >
> > >
> >
>
Re: Apache Ignite 2.4 release
Posted by Alexey Goncharuk <al...@gmail.com>.
In terms of 'safety', Ignite default settings are far beyond optimal. For
in-memory mode, we have 0 backups by default, which means partition loss in
a case of node failure, we have readFromBackup=true and PRIMARY_SYNC by
default which effectively cancels linearizability property for cache
updates, so setting the default WAL mode to LOG_ONLY does not seem to be a
bigger evil than it currently is. If we are to move to safer defaults, we
should change all of the affected sides.
I also want to clarify the difference between guarantees in
non-fsync modes. We should distinguish the loss of durability (the loss of
the last update) because the update did not make it to disk and data loss
because the disk content was shuffled due to an incomplete page write. In
my understanding, the current situation is:
FSYNC: loss of durability: not possible, data loss: not possible
LOG_ONLY: loss of durability: possible only if OS/power fails, data loss:
possible only if OS/power fails
BACKGROUND: loss of durability: possible if Ignite process fails, data
loss: possible only if OS/power fails
The data loss situation can be mitigated in the cluster using a large
enough replication factor (this is what Dmitriy was describing in the case
of LOG_ONLY and 3 backups configuration).
Denis,
I do not think it is fair to compare Ignite defaults to Cassandra's
defaults because Cassandra is _not_ transactional _eventually consistent_
datastore, they claim much weaker guarantees than Ignite.
All in all, I'm ok to change the WAL default right now, but I would revisit
all those settings in 3.0 and made Ignite safe-first.
2018-02-17 3:24 GMT+03:00 Denis Magda <dm...@apache.org>:
> Classic relational databases have no choice rather than to use FSYNC by
> default. RDBMS is all about consistency.
>
> Distributed databases try to balance consistency and performance. For
> instance, why to fsync every update if there is usually 1 backup copy?
> This is probably why VoltDB [1] and Cassandra use the modes comparable to
> Ignite's LOG_ONLY.
>
> Ignite as a distributed database should care of both consistency and
> performance.
>
> My vote goes to FSYNC, LOG_ONLY (default), BACKGROUND, NONE.
>
>
> [1] https://docs.voltdb.com/UsingVoltDB/CmdLogConfig.php
>
> --
> Denis
>
>
> On Fri, Feb 16, 2018 at 2:14 PM, Dmitriy Setrakyan <ds...@apache.org>
> wrote:
>
> > Vova,
> >
> > I hear your concerns, but at the same time I know that one of the largest
> > banks in eastern Europe is using Ignite in LOG_ONLY mode with 3 backups
> to
> > move money. The rational is that the probability of failure of 4 servers
> at
> > hardware level at the same time is very low. However, if the JVM process
> > fails on any server, then it can be safely restarted without loosing
> data.
> > In my view, this is why LOG_ONLY mode makes sense as a default.
> >
> > I still vote to change the default to LOG_ONLY, deprecate the DEFAULT
> name
> > altogether and add FSYNC mode instead.
> >
> > D.
> >
> > On Fri, Feb 16, 2018 at 4:05 PM, Vladimir Ozerov <vo...@gridgain.com>
> > wrote:
> >
> > > Sergey,
> > >
> > > We do not have backups by default either, so essentially we are loosing
> > > data by default. Moreover, backups are less reliable option than fsync
> > > because a lot of users cannot afford putting servers into separate
> power
> > > circuits, so a single power failure may easily lead to poweroff of the
> > > whole cluster at once, so data is lost still. This is normal practice
> > even
> > > for enterprise deployments (e.g. asynchronous replication).
> > >
> > > To make things even worse, we employ PRIMARY_SYNC mode by default! So
> > even
> > > if you configured backups, you still may loose data due to a single
> node
> > > failure - just shutdown the PRIMARY after commit is confirmed to the
> > client
> > > and your recent update will disappers.
> > >
> > > So this is what user should do to make himself safe:
> > > 1) Learn about WAL modes
> > > 2) Learn about backups
> > > 3) Learn about synchronization modes
> > > 4) Cross his fingers that he understood everything correctly and that
> > there
> > > are no other hidden surprises in Ignite which could lead to data loss.
> > >
> > > Way to much for a product, claiming to be A*C*ID and persistent, don't
> > you
> > > think so?
> > >
> > > Leaving deafult WAL mode with fsync resolves all these issues.
> > >
> > > Vladimir.
> > >
> > >
> > > On Fri, Feb 16, 2018 at 11:43 PM, Sergey Kozlov <sk...@gridgain.com>
> > > wrote:
> > >
> > > > I suppose some approaches used by classic databases makes no sense
> for
> > > > Ignite. FSYNC requirement for databases has the nature of single host
> > > > solution. If you have corrupted db files you have corrupted (lost)
> > data.
> > > >
> > > > For Ignite the enough number of backups and the failure detecting
> logic
> > > can
> > > > provide the data consistency in term "cluster data consistency".
> > > >
> > > >
> > > >
> > > > On Fri, Feb 16, 2018 at 8:57 PM, Dmitry Pavlov <
> dpavlov.spb@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi, all WAL modes except NONE protects from data consistency
> problem
> > > > > (B+Tree, pages, etc), which is why I suggest to avoid saying
> > > 'corrupted'
> > > > > about 'unapplied updates'.
> > > > >
> > > > > Log Only and Background may cause unapplied updates in case of
> > > OS/process
> > > > > failures.
> > > > >
> > > > > None mode IMO is not an option in case data consistency is needed.
> > > > >
> > > > > пт, 16 февр. 2018 г. в 20:49, Valentin Kulichenko <
> > > > > valentin.kulichenko@gmail.com>:
> > > > >
> > > > > > Guys,
> > > > > >
> > > > > > While we're on this topic, what is the difference between
> > BACKGROUND
> > > > and
> > > > > > NONE in terms of semantics and provided guarantees? To me it
> looks
> > > like
> > > > > > both guarantee to recover the state since last checkpoint and
> > > anything
> > > > > else
> > > > > > can potentially be lost, so from user perspective they are the
> > same.
> > > > Am I
> > > > > > missing something here?
> > > > > >
> > > > > > Also there is the following in Javadoc for NONE: "If an Ignite
> node
> > > is
> > > > > > terminated in NONE mode abruptly, it is likely that the data
> stored
> > > on
> > > > > disk
> > > > > > is corrupted and work directory will need to be cleared for a
> node
> > > > > > restart.". If this is really the case, I'm not sure NONE makes
> > sense
> > > at
> > > > > > all. Why would I enable persistence if I'm likely to clear the
> > > storage
> > > > on
> > > > > > restart?
> > > > > >
> > > > > > -Val
> > > > > >
> > > > > > On Fri, Feb 16, 2018 at 8:39 AM, Vladimir Ozerov <
> > > vozerov@gridgain.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > What is the reason to have DEFAULT mode at all if you claim
> > > LOG_ONLY
> > > > to
> > > > > > be
> > > > > > > completely safe? :)
> > > > > > >
> > > > > > > And how it could be safe provided that without fsync we loose
> > part
> > > of
> > > > > WAL
> > > > > > > itself in case of crash?
> > > > > > >
> > > > > > > пт, 16 февр. 2018 г. в 19:32, Dmitry Pavlov <
> > dpavlov.spb@gmail.com
> > > >:
> > > > > > >
> > > > > > > > Thank you. Data can't be corrupted in case crash because of
> WAL
> > > > > replay
> > > > > > > > (since completed checkpoint). Physical records are used to
> > > restore
> > > > > > > probably
> > > > > > > > corrupted pages in persistent store (we overwrite so called
> > 'grey
> > > > > > zone' -
> > > > > > > > pages we don't know for sure if they have been written).
> > > > > > > >
> > > > > > > > Only one effect is unwritten one or several last
> transactions.
> > It
> > > > is
> > > > > > not
> > > > > > > > the same with corrupted data.
> > > > > > > >
> > > > > > > > пт, 16 февр. 2018 г. в 19:19, Vladimir Ozerov <
> > > > vozerov@gridgain.com
> > > > > >:
> > > > > > > >
> > > > > > > > > Log only mode is not safe - data might be corrupted in case
> > of
> > > > > system
> > > > > > > > > crash. Oracle - fsync, Postgres - fsync, SQL Server -
> fsync,
> > > > > > Cassandra
> > > > > > > -
> > > > > > > > > similar to our “background”.
> > > > > > > > >
> > > > > > > > > пт, 16 февр. 2018 г. в 19:11, Dmitry Pavlov <
> > > > dpavlov.spb@gmail.com
> > > > > >:
> > > > > > > > >
> > > > > > > > > > Hi Vladimir,
> > > > > > > > > >
> > > > > > > > > > What you saying is defenetely make sence.
> > > > > > > > > >
> > > > > > > > > > In the same time LOG_ONLY is also safe mode, user will be
> > > able
> > > > to
> > > > > > > > restore
> > > > > > > > > > system after crash. If it is not true, we should create
> > > > critical
> > > > > > > ticket
> > > > > > > > > and
> > > > > > > > > > fix it.
> > > > > > > > > >
> > > > > > > > > > Do you know other databases defaults, such as Cassandra,
> > > > Oracle,
> > > > > > > > Postgre?
> > > > > > > > > >
> > > > > > > > > > Sincerely,
> > > > > > > > > > Dmitriy Pavlov
> > > > > > > > > >
> > > > > > > > > > пт, 16 февр. 2018 г. в 18:41, Vladimir Ozerov <
> > > > > > vozerov@gridgain.com
> > > > > > > >:
> > > > > > > > > >
> > > > > > > > > > > Igniters,
> > > > > > > > > > >
> > > > > > > > > > > Sorry for pouring oil on the flames, but from database
> > > > > > perspective
> > > > > > > > > moving
> > > > > > > > > > > from FSYNC to non-FSYNC mode appears to be a mistake.
> > When
> > > > you
> > > > > > work
> > > > > > > > > with
> > > > > > > > > > > database, your main expectation is that it will save
> your
> > > > data.
> > > > > > All
> > > > > > > > > > > production database vendor make sure that you are safe,
> > not
> > > > > that
> > > > > > > you
> > > > > > > > > are
> > > > > > > > > > > fast. Moreover, some vendors even prevent you from
> being
> > in
> > > > > > unsafe
> > > > > > > > mode
> > > > > > > > > > > (e.g. you cannot disable fsync in SQL Server at all).
> > > > > > > > > > >
> > > > > > > > > > > If we continue going in this direction, we will end up
> > > with a
> > > > > > > > product,
> > > > > > > > > > > which is unsafe out of the box and require tons of
> > > > > documentation
> > > > > > to
> > > > > > > > > > > understand how to make it safe. Definitely not the
> right
> > > > > message
> > > > > > to
> > > > > > > > the
> > > > > > > > > > > market. This is like a car without brakes - would you
> > like
> > > to
> > > > > > drive
> > > > > > > > it?
> > > > > > > > > > If
> > > > > > > > > > > this is Need For Speed game and you have unlimited
> lives
> > > > > > (in-memory
> > > > > > > > > cache
> > > > > > > > > > > with backing store), then yes. If this is a real life
> > with
> > > > > > > > > (persistence)
> > > > > > > > > > -
> > > > > > > > > > > then no.
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Feb 16, 2018 at 5:20 PM, Dmitriy Setrakyan <
> > > > > > > > > > dsetrakyan@apache.org>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Well, I cannot say that I like the name LOG_ONLY,
> but I
> > > > would
> > > > > > > vote
> > > > > > > > to
> > > > > > > > > > > keep
> > > > > > > > > > > > it for now, given that it is already documented in
> many
> > > > > places,
> > > > > > > > > blogs,
> > > > > > > > > > > and
> > > > > > > > > > > > examples.
> > > > > > > > > > > >
> > > > > > > > > > > > D.
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Feb 16, 2018 at 8:13 AM, Ivan Rakov <
> > > > > > > ivan.glukos@gmail.com
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Looks like it's an Ignite term - I've never heard
> of
> > it
> > > > > > outside
> > > > > > > > > > Ignite
> > > > > > > > > > > > > scope.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Though, renaming existing enum value requires
> keeping
> > > old
> > > > > as
> > > > > > > > > > > deprecated.
> > > > > > > > > > > > > DEFAULT is confusing enough to pay this price.
> > > > > > > > > > > > > As for LOG_ONLY, I think we can keep it as long as
> it
> > > has
> > > > > > good
> > > > > > > > and
> > > > > > > > > > > > > definitive javadoc.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best Regards,
> > > > > > > > > > > > > Ivan Rakov
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On 16.02.2018 17:07, Dmitriy Setrakyan wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > >> Igniters, just to clarify, does the term LOG_ONLY
> > mean
> > > > > > > anything
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > >> industry or is this just an Ignite term?
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> D.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> On Fri, Feb 16, 2018 at 8:03 AM, Anton Vinogradov
> <
> > > > > > > > > > > > >> avinogradov@gridgain.com>
> > > > > > > > > > > > >> wrote:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Log only mode: flushes application buffers.
> > > > > > > > > > > > >>> So, in synced mode without fsync guarantee.
> That's
> > > why
> > > > I
> > > > > > > > propose
> > > > > > > > > to
> > > > > > > > > > > > >>> rename
> > > > > > > > > > > > >>> it as SYNC.
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>> On Fri, Feb 16, 2018 at 4:49 PM, Ilya Lantukh <
> > > > > > > > > > ilantukh@gridgain.com
> > > > > > > > > > > >
> > > > > > > > > > > > >>> wrote:
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>> I am OK with either FSYNC or STRICT variant.
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>> LOG_ONLY name means "log without fsync".
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>> On Fri, Feb 16, 2018 at 4:05 PM, Dmitriy
> > Setrakyan <
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>> dsetrakyan@apache.org>
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>>> wrote:
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>> On Fri, Feb 16, 2018 at 7:02 AM, Ivan Rakov <
> > > > > > > > > > ivan.glukos@gmail.com>
> > > > > > > > > > > > >>>>>
> > > > > > > > > > > > >>>> wrote:
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>>> Why create a new term to define something that
> > has
> > > > > > already
> > > > > > > > been
> > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > >>>>> defined?
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>>> That makes sense. I'm ok with FSYNC.
> > > > > > > > > > > > >>>>>> Anton, I don't understand why we should rename
> > > > > LOG_ONLY
> > > > > > to
> > > > > > > > > SYNC.
> > > > > > > > > > > We
> > > > > > > > > > > > >>>>>> started this discussion with bad naming of
> > > DEFAULT,
> > > > > but
> > > > > > > this
> > > > > > > > > has
> > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > >>>>> nothing
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>>> to
> > > > > > > > > > > > >>>>>
> > > > > > > > > > > > >>>>>> do with LOG_ONLY (even though it may be
> > > scientific -
> > > > > but
> > > > > > > > SYNC
> > > > > > > > > > > sounds
> > > > > > > > > > > > >>>>>> scientific as well).
> > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > >>>>>> I agree with Ivan, we should not go wild with
> > > > > renaming.
> > > > > > > > > > However, I
> > > > > > > > > > > > >>>>>
> > > > > > > > > > > > >>>> would
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>>> like to find out what is the meaning behind the
> > > > LOG_ONLY
> > > > > > > name.
> > > > > > > > > Can
> > > > > > > > > > > > >>>>>
> > > > > > > > > > > > >>>> someone
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>>> explain?
> > > > > > > > > > > > >>>>>
> > > > > > > > > > > > >>>>> D.
> > > > > > > > > > > > >>>>>
> > > > > > > > > > > > >>>>>
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>> --
> > > > > > > > > > > > >>>> Best regards,
> > > > > > > > > > > > >>>> Ilya
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Sergey Kozlov
> > > > GridGain Systems
> > > > www.gridgain.com
> > > >
> > >
> >
>
Re: Apache Ignite 2.4 release
Posted by Denis Magda <dm...@apache.org>.
Classic relational databases have no choice rather than to use FSYNC by
default. RDBMS is all about consistency.
Distributed databases try to balance consistency and performance. For
instance, why to fsync every update if there is usually 1 backup copy?
This is probably why VoltDB [1] and Cassandra use the modes comparable to
Ignite's LOG_ONLY.
Ignite as a distributed database should care of both consistency and
performance.
My vote goes to FSYNC, LOG_ONLY (default), BACKGROUND, NONE.
[1] https://docs.voltdb.com/UsingVoltDB/CmdLogConfig.php
--
Denis
On Fri, Feb 16, 2018 at 2:14 PM, Dmitriy Setrakyan <ds...@apache.org>
wrote:
> Vova,
>
> I hear your concerns, but at the same time I know that one of the largest
> banks in eastern Europe is using Ignite in LOG_ONLY mode with 3 backups to
> move money. The rational is that the probability of failure of 4 servers at
> hardware level at the same time is very low. However, if the JVM process
> fails on any server, then it can be safely restarted without loosing data.
> In my view, this is why LOG_ONLY mode makes sense as a default.
>
> I still vote to change the default to LOG_ONLY, deprecate the DEFAULT name
> altogether and add FSYNC mode instead.
>
> D.
>
> On Fri, Feb 16, 2018 at 4:05 PM, Vladimir Ozerov <vo...@gridgain.com>
> wrote:
>
> > Sergey,
> >
> > We do not have backups by default either, so essentially we are loosing
> > data by default. Moreover, backups are less reliable option than fsync
> > because a lot of users cannot afford putting servers into separate power
> > circuits, so a single power failure may easily lead to poweroff of the
> > whole cluster at once, so data is lost still. This is normal practice
> even
> > for enterprise deployments (e.g. asynchronous replication).
> >
> > To make things even worse, we employ PRIMARY_SYNC mode by default! So
> even
> > if you configured backups, you still may loose data due to a single node
> > failure - just shutdown the PRIMARY after commit is confirmed to the
> client
> > and your recent update will disappers.
> >
> > So this is what user should do to make himself safe:
> > 1) Learn about WAL modes
> > 2) Learn about backups
> > 3) Learn about synchronization modes
> > 4) Cross his fingers that he understood everything correctly and that
> there
> > are no other hidden surprises in Ignite which could lead to data loss.
> >
> > Way to much for a product, claiming to be A*C*ID and persistent, don't
> you
> > think so?
> >
> > Leaving deafult WAL mode with fsync resolves all these issues.
> >
> > Vladimir.
> >
> >
> > On Fri, Feb 16, 2018 at 11:43 PM, Sergey Kozlov <sk...@gridgain.com>
> > wrote:
> >
> > > I suppose some approaches used by classic databases makes no sense for
> > > Ignite. FSYNC requirement for databases has the nature of single host
> > > solution. If you have corrupted db files you have corrupted (lost)
> data.
> > >
> > > For Ignite the enough number of backups and the failure detecting logic
> > can
> > > provide the data consistency in term "cluster data consistency".
> > >
> > >
> > >
> > > On Fri, Feb 16, 2018 at 8:57 PM, Dmitry Pavlov <dp...@gmail.com>
> > > wrote:
> > >
> > > > Hi, all WAL modes except NONE protects from data consistency problem
> > > > (B+Tree, pages, etc), which is why I suggest to avoid saying
> > 'corrupted'
> > > > about 'unapplied updates'.
> > > >
> > > > Log Only and Background may cause unapplied updates in case of
> > OS/process
> > > > failures.
> > > >
> > > > None mode IMO is not an option in case data consistency is needed.
> > > >
> > > > пт, 16 февр. 2018 г. в 20:49, Valentin Kulichenko <
> > > > valentin.kulichenko@gmail.com>:
> > > >
> > > > > Guys,
> > > > >
> > > > > While we're on this topic, what is the difference between
> BACKGROUND
> > > and
> > > > > NONE in terms of semantics and provided guarantees? To me it looks
> > like
> > > > > both guarantee to recover the state since last checkpoint and
> > anything
> > > > else
> > > > > can potentially be lost, so from user perspective they are the
> same.
> > > Am I
> > > > > missing something here?
> > > > >
> > > > > Also there is the following in Javadoc for NONE: "If an Ignite node
> > is
> > > > > terminated in NONE mode abruptly, it is likely that the data stored
> > on
> > > > disk
> > > > > is corrupted and work directory will need to be cleared for a node
> > > > > restart.". If this is really the case, I'm not sure NONE makes
> sense
> > at
> > > > > all. Why would I enable persistence if I'm likely to clear the
> > storage
> > > on
> > > > > restart?
> > > > >
> > > > > -Val
> > > > >
> > > > > On Fri, Feb 16, 2018 at 8:39 AM, Vladimir Ozerov <
> > vozerov@gridgain.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > What is the reason to have DEFAULT mode at all if you claim
> > LOG_ONLY
> > > to
> > > > > be
> > > > > > completely safe? :)
> > > > > >
> > > > > > And how it could be safe provided that without fsync we loose
> part
> > of
> > > > WAL
> > > > > > itself in case of crash?
> > > > > >
> > > > > > пт, 16 февр. 2018 г. в 19:32, Dmitry Pavlov <
> dpavlov.spb@gmail.com
> > >:
> > > > > >
> > > > > > > Thank you. Data can't be corrupted in case crash because of WAL
> > > > replay
> > > > > > > (since completed checkpoint). Physical records are used to
> > restore
> > > > > > probably
> > > > > > > corrupted pages in persistent store (we overwrite so called
> 'grey
> > > > > zone' -
> > > > > > > pages we don't know for sure if they have been written).
> > > > > > >
> > > > > > > Only one effect is unwritten one or several last transactions.
> It
> > > is
> > > > > not
> > > > > > > the same with corrupted data.
> > > > > > >
> > > > > > > пт, 16 февр. 2018 г. в 19:19, Vladimir Ozerov <
> > > vozerov@gridgain.com
> > > > >:
> > > > > > >
> > > > > > > > Log only mode is not safe - data might be corrupted in case
> of
> > > > system
> > > > > > > > crash. Oracle - fsync, Postgres - fsync, SQL Server - fsync,
> > > > > Cassandra
> > > > > > -
> > > > > > > > similar to our “background”.
> > > > > > > >
> > > > > > > > пт, 16 февр. 2018 г. в 19:11, Dmitry Pavlov <
> > > dpavlov.spb@gmail.com
> > > > >:
> > > > > > > >
> > > > > > > > > Hi Vladimir,
> > > > > > > > >
> > > > > > > > > What you saying is defenetely make sence.
> > > > > > > > >
> > > > > > > > > In the same time LOG_ONLY is also safe mode, user will be
> > able
> > > to
> > > > > > > restore
> > > > > > > > > system after crash. If it is not true, we should create
> > > critical
> > > > > > ticket
> > > > > > > > and
> > > > > > > > > fix it.
> > > > > > > > >
> > > > > > > > > Do you know other databases defaults, such as Cassandra,
> > > Oracle,
> > > > > > > Postgre?
> > > > > > > > >
> > > > > > > > > Sincerely,
> > > > > > > > > Dmitriy Pavlov
> > > > > > > > >
> > > > > > > > > пт, 16 февр. 2018 г. в 18:41, Vladimir Ozerov <
> > > > > vozerov@gridgain.com
> > > > > > >:
> > > > > > > > >
> > > > > > > > > > Igniters,
> > > > > > > > > >
> > > > > > > > > > Sorry for pouring oil on the flames, but from database
> > > > > perspective
> > > > > > > > moving
> > > > > > > > > > from FSYNC to non-FSYNC mode appears to be a mistake.
> When
> > > you
> > > > > work
> > > > > > > > with
> > > > > > > > > > database, your main expectation is that it will save your
> > > data.
> > > > > All
> > > > > > > > > > production database vendor make sure that you are safe,
> not
> > > > that
> > > > > > you
> > > > > > > > are
> > > > > > > > > > fast. Moreover, some vendors even prevent you from being
> in
> > > > > unsafe
> > > > > > > mode
> > > > > > > > > > (e.g. you cannot disable fsync in SQL Server at all).
> > > > > > > > > >
> > > > > > > > > > If we continue going in this direction, we will end up
> > with a
> > > > > > > product,
> > > > > > > > > > which is unsafe out of the box and require tons of
> > > > documentation
> > > > > to
> > > > > > > > > > understand how to make it safe. Definitely not the right
> > > > message
> > > > > to
> > > > > > > the
> > > > > > > > > > market. This is like a car without brakes - would you
> like
> > to
> > > > > drive
> > > > > > > it?
> > > > > > > > > If
> > > > > > > > > > this is Need For Speed game and you have unlimited lives
> > > > > (in-memory
> > > > > > > > cache
> > > > > > > > > > with backing store), then yes. If this is a real life
> with
> > > > > > > > (persistence)
> > > > > > > > > -
> > > > > > > > > > then no.
> > > > > > > > > >
> > > > > > > > > > On Fri, Feb 16, 2018 at 5:20 PM, Dmitriy Setrakyan <
> > > > > > > > > dsetrakyan@apache.org>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Well, I cannot say that I like the name LOG_ONLY, but I
> > > would
> > > > > > vote
> > > > > > > to
> > > > > > > > > > keep
> > > > > > > > > > > it for now, given that it is already documented in many
> > > > places,
> > > > > > > > blogs,
> > > > > > > > > > and
> > > > > > > > > > > examples.
> > > > > > > > > > >
> > > > > > > > > > > D.
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Feb 16, 2018 at 8:13 AM, Ivan Rakov <
> > > > > > ivan.glukos@gmail.com
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Looks like it's an Ignite term - I've never heard of
> it
> > > > > outside
> > > > > > > > > Ignite
> > > > > > > > > > > > scope.
> > > > > > > > > > > >
> > > > > > > > > > > > Though, renaming existing enum value requires keeping
> > old
> > > > as
> > > > > > > > > > deprecated.
> > > > > > > > > > > > DEFAULT is confusing enough to pay this price.
> > > > > > > > > > > > As for LOG_ONLY, I think we can keep it as long as it
> > has
> > > > > good
> > > > > > > and
> > > > > > > > > > > > definitive javadoc.
> > > > > > > > > > > >
> > > > > > > > > > > > Best Regards,
> > > > > > > > > > > > Ivan Rakov
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On 16.02.2018 17:07, Dmitriy Setrakyan wrote:
> > > > > > > > > > > >
> > > > > > > > > > > >> Igniters, just to clarify, does the term LOG_ONLY
> mean
> > > > > > anything
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > >> industry or is this just an Ignite term?
> > > > > > > > > > > >>
> > > > > > > > > > > >> D.
> > > > > > > > > > > >>
> > > > > > > > > > > >> On Fri, Feb 16, 2018 at 8:03 AM, Anton Vinogradov <
> > > > > > > > > > > >> avinogradov@gridgain.com>
> > > > > > > > > > > >> wrote:
> > > > > > > > > > > >>
> > > > > > > > > > > >> Log only mode: flushes application buffers.
> > > > > > > > > > > >>> So, in synced mode without fsync guarantee. That's
> > why
> > > I
> > > > > > > propose
> > > > > > > > to
> > > > > > > > > > > >>> rename
> > > > > > > > > > > >>> it as SYNC.
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> On Fri, Feb 16, 2018 at 4:49 PM, Ilya Lantukh <
> > > > > > > > > ilantukh@gridgain.com
> > > > > > > > > > >
> > > > > > > > > > > >>> wrote:
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> I am OK with either FSYNC or STRICT variant.
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>> LOG_ONLY name means "log without fsync".
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>> On Fri, Feb 16, 2018 at 4:05 PM, Dmitriy
> Setrakyan <
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>> dsetrakyan@apache.org>
> > > > > > > > > > > >>>
> > > > > > > > > > > >>>> wrote:
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>> On Fri, Feb 16, 2018 at 7:02 AM, Ivan Rakov <
> > > > > > > > > ivan.glukos@gmail.com>
> > > > > > > > > > > >>>>>
> > > > > > > > > > > >>>> wrote:
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>>> Why create a new term to define something that
> has
> > > > > already
> > > > > > > been
> > > > > > > > > > > >>>>>>
> > > > > > > > > > > >>>>> defined?
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>>> That makes sense. I'm ok with FSYNC.
> > > > > > > > > > > >>>>>> Anton, I don't understand why we should rename
> > > > LOG_ONLY
> > > > > to
> > > > > > > > SYNC.
> > > > > > > > > > We
> > > > > > > > > > > >>>>>> started this discussion with bad naming of
> > DEFAULT,
> > > > but
> > > > > > this
> > > > > > > > has
> > > > > > > > > > > >>>>>>
> > > > > > > > > > > >>>>> nothing
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>>> to
> > > > > > > > > > > >>>>>
> > > > > > > > > > > >>>>>> do with LOG_ONLY (even though it may be
> > scientific -
> > > > but
> > > > > > > SYNC
> > > > > > > > > > sounds
> > > > > > > > > > > >>>>>> scientific as well).
> > > > > > > > > > > >>>>>>
> > > > > > > > > > > >>>>>> I agree with Ivan, we should not go wild with
> > > > renaming.
> > > > > > > > > However, I
> > > > > > > > > > > >>>>>
> > > > > > > > > > > >>>> would
> > > > > > > > > > > >>>
> > > > > > > > > > > >>>> like to find out what is the meaning behind the
> > > LOG_ONLY
> > > > > > name.
> > > > > > > > Can
> > > > > > > > > > > >>>>>
> > > > > > > > > > > >>>> someone
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>>> explain?
> > > > > > > > > > > >>>>>
> > > > > > > > > > > >>>>> D.
> > > > > > > > > > > >>>>>
> > > > > > > > > > > >>>>>
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>> --
> > > > > > > > > > > >>>> Best regards,
> > > > > > > > > > > >>>> Ilya
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Sergey Kozlov
> > > GridGain Systems
> > > www.gridgain.com
> > >
> >
>
Re: Apache Ignite 2.4 release
Posted by Dmitriy Setrakyan <ds...@apache.org>.
Vova,
I hear your concerns, but at the same time I know that one of the largest
banks in eastern Europe is using Ignite in LOG_ONLY mode with 3 backups to
move money. The rational is that the probability of failure of 4 servers at
hardware level at the same time is very low. However, if the JVM process
fails on any server, then it can be safely restarted without loosing data.
In my view, this is why LOG_ONLY mode makes sense as a default.
I still vote to change the default to LOG_ONLY, deprecate the DEFAULT name
altogether and add FSYNC mode instead.
D.
On Fri, Feb 16, 2018 at 4:05 PM, Vladimir Ozerov <vo...@gridgain.com>
wrote:
> Sergey,
>
> We do not have backups by default either, so essentially we are loosing
> data by default. Moreover, backups are less reliable option than fsync
> because a lot of users cannot afford putting servers into separate power
> circuits, so a single power failure may easily lead to poweroff of the
> whole cluster at once, so data is lost still. This is normal practice even
> for enterprise deployments (e.g. asynchronous replication).
>
> To make things even worse, we employ PRIMARY_SYNC mode by default! So even
> if you configured backups, you still may loose data due to a single node
> failure - just shutdown the PRIMARY after commit is confirmed to the client
> and your recent update will disappers.
>
> So this is what user should do to make himself safe:
> 1) Learn about WAL modes
> 2) Learn about backups
> 3) Learn about synchronization modes
> 4) Cross his fingers that he understood everything correctly and that there
> are no other hidden surprises in Ignite which could lead to data loss.
>
> Way to much for a product, claiming to be A*C*ID and persistent, don't you
> think so?
>
> Leaving deafult WAL mode with fsync resolves all these issues.
>
> Vladimir.
>
>
> On Fri, Feb 16, 2018 at 11:43 PM, Sergey Kozlov <sk...@gridgain.com>
> wrote:
>
> > I suppose some approaches used by classic databases makes no sense for
> > Ignite. FSYNC requirement for databases has the nature of single host
> > solution. If you have corrupted db files you have corrupted (lost) data.
> >
> > For Ignite the enough number of backups and the failure detecting logic
> can
> > provide the data consistency in term "cluster data consistency".
> >
> >
> >
> > On Fri, Feb 16, 2018 at 8:57 PM, Dmitry Pavlov <dp...@gmail.com>
> > wrote:
> >
> > > Hi, all WAL modes except NONE protects from data consistency problem
> > > (B+Tree, pages, etc), which is why I suggest to avoid saying
> 'corrupted'
> > > about 'unapplied updates'.
> > >
> > > Log Only and Background may cause unapplied updates in case of
> OS/process
> > > failures.
> > >
> > > None mode IMO is not an option in case data consistency is needed.
> > >
> > > пт, 16 февр. 2018 г. в 20:49, Valentin Kulichenko <
> > > valentin.kulichenko@gmail.com>:
> > >
> > > > Guys,
> > > >
> > > > While we're on this topic, what is the difference between BACKGROUND
> > and
> > > > NONE in terms of semantics and provided guarantees? To me it looks
> like
> > > > both guarantee to recover the state since last checkpoint and
> anything
> > > else
> > > > can potentially be lost, so from user perspective they are the same.
> > Am I
> > > > missing something here?
> > > >
> > > > Also there is the following in Javadoc for NONE: "If an Ignite node
> is
> > > > terminated in NONE mode abruptly, it is likely that the data stored
> on
> > > disk
> > > > is corrupted and work directory will need to be cleared for a node
> > > > restart.". If this is really the case, I'm not sure NONE makes sense
> at
> > > > all. Why would I enable persistence if I'm likely to clear the
> storage
> > on
> > > > restart?
> > > >
> > > > -Val
> > > >
> > > > On Fri, Feb 16, 2018 at 8:39 AM, Vladimir Ozerov <
> vozerov@gridgain.com
> > >
> > > > wrote:
> > > >
> > > > > What is the reason to have DEFAULT mode at all if you claim
> LOG_ONLY
> > to
> > > > be
> > > > > completely safe? :)
> > > > >
> > > > > And how it could be safe provided that without fsync we loose part
> of
> > > WAL
> > > > > itself in case of crash?
> > > > >
> > > > > пт, 16 февр. 2018 г. в 19:32, Dmitry Pavlov <dpavlov.spb@gmail.com
> >:
> > > > >
> > > > > > Thank you. Data can't be corrupted in case crash because of WAL
> > > replay
> > > > > > (since completed checkpoint). Physical records are used to
> restore
> > > > > probably
> > > > > > corrupted pages in persistent store (we overwrite so called 'grey
> > > > zone' -
> > > > > > pages we don't know for sure if they have been written).
> > > > > >
> > > > > > Only one effect is unwritten one or several last transactions. It
> > is
> > > > not
> > > > > > the same with corrupted data.
> > > > > >
> > > > > > пт, 16 февр. 2018 г. в 19:19, Vladimir Ozerov <
> > vozerov@gridgain.com
> > > >:
> > > > > >
> > > > > > > Log only mode is not safe - data might be corrupted in case of
> > > system
> > > > > > > crash. Oracle - fsync, Postgres - fsync, SQL Server - fsync,
> > > > Cassandra
> > > > > -
> > > > > > > similar to our “background”.
> > > > > > >
> > > > > > > пт, 16 февр. 2018 г. в 19:11, Dmitry Pavlov <
> > dpavlov.spb@gmail.com
> > > >:
> > > > > > >
> > > > > > > > Hi Vladimir,
> > > > > > > >
> > > > > > > > What you saying is defenetely make sence.
> > > > > > > >
> > > > > > > > In the same time LOG_ONLY is also safe mode, user will be
> able
> > to
> > > > > > restore
> > > > > > > > system after crash. If it is not true, we should create
> > critical
> > > > > ticket
> > > > > > > and
> > > > > > > > fix it.
> > > > > > > >
> > > > > > > > Do you know other databases defaults, such as Cassandra,
> > Oracle,
> > > > > > Postgre?
> > > > > > > >
> > > > > > > > Sincerely,
> > > > > > > > Dmitriy Pavlov
> > > > > > > >
> > > > > > > > пт, 16 февр. 2018 г. в 18:41, Vladimir Ozerov <
> > > > vozerov@gridgain.com
> > > > > >:
> > > > > > > >
> > > > > > > > > Igniters,
> > > > > > > > >
> > > > > > > > > Sorry for pouring oil on the flames, but from database
> > > > perspective
> > > > > > > moving
> > > > > > > > > from FSYNC to non-FSYNC mode appears to be a mistake. When
> > you
> > > > work
> > > > > > > with
> > > > > > > > > database, your main expectation is that it will save your
> > data.
> > > > All
> > > > > > > > > production database vendor make sure that you are safe, not
> > > that
> > > > > you
> > > > > > > are
> > > > > > > > > fast. Moreover, some vendors even prevent you from being in
> > > > unsafe
> > > > > > mode
> > > > > > > > > (e.g. you cannot disable fsync in SQL Server at all).
> > > > > > > > >
> > > > > > > > > If we continue going in this direction, we will end up
> with a
> > > > > > product,
> > > > > > > > > which is unsafe out of the box and require tons of
> > > documentation
> > > > to
> > > > > > > > > understand how to make it safe. Definitely not the right
> > > message
> > > > to
> > > > > > the
> > > > > > > > > market. This is like a car without brakes - would you like
> to
> > > > drive
> > > > > > it?
> > > > > > > > If
> > > > > > > > > this is Need For Speed game and you have unlimited lives
> > > > (in-memory
> > > > > > > cache
> > > > > > > > > with backing store), then yes. If this is a real life with
> > > > > > > (persistence)
> > > > > > > > -
> > > > > > > > > then no.
> > > > > > > > >
> > > > > > > > > On Fri, Feb 16, 2018 at 5:20 PM, Dmitriy Setrakyan <
> > > > > > > > dsetrakyan@apache.org>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Well, I cannot say that I like the name LOG_ONLY, but I
> > would
> > > > > vote
> > > > > > to
> > > > > > > > > keep
> > > > > > > > > > it for now, given that it is already documented in many
> > > places,
> > > > > > > blogs,
> > > > > > > > > and
> > > > > > > > > > examples.
> > > > > > > > > >
> > > > > > > > > > D.
> > > > > > > > > >
> > > > > > > > > > On Fri, Feb 16, 2018 at 8:13 AM, Ivan Rakov <
> > > > > ivan.glukos@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Looks like it's an Ignite term - I've never heard of it
> > > > outside
> > > > > > > > Ignite
> > > > > > > > > > > scope.
> > > > > > > > > > >
> > > > > > > > > > > Though, renaming existing enum value requires keeping
> old
> > > as
> > > > > > > > > deprecated.
> > > > > > > > > > > DEFAULT is confusing enough to pay this price.
> > > > > > > > > > > As for LOG_ONLY, I think we can keep it as long as it
> has
> > > > good
> > > > > > and
> > > > > > > > > > > definitive javadoc.
> > > > > > > > > > >
> > > > > > > > > > > Best Regards,
> > > > > > > > > > > Ivan Rakov
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On 16.02.2018 17:07, Dmitriy Setrakyan wrote:
> > > > > > > > > > >
> > > > > > > > > > >> Igniters, just to clarify, does the term LOG_ONLY mean
> > > > > anything
> > > > > > in
> > > > > > > > the
> > > > > > > > > > >> industry or is this just an Ignite term?
> > > > > > > > > > >>
> > > > > > > > > > >> D.
> > > > > > > > > > >>
> > > > > > > > > > >> On Fri, Feb 16, 2018 at 8:03 AM, Anton Vinogradov <
> > > > > > > > > > >> avinogradov@gridgain.com>
> > > > > > > > > > >> wrote:
> > > > > > > > > > >>
> > > > > > > > > > >> Log only mode: flushes application buffers.
> > > > > > > > > > >>> So, in synced mode without fsync guarantee. That's
> why
> > I
> > > > > > propose
> > > > > > > to
> > > > > > > > > > >>> rename
> > > > > > > > > > >>> it as SYNC.
> > > > > > > > > > >>>
> > > > > > > > > > >>> On Fri, Feb 16, 2018 at 4:49 PM, Ilya Lantukh <
> > > > > > > > ilantukh@gridgain.com
> > > > > > > > > >
> > > > > > > > > > >>> wrote:
> > > > > > > > > > >>>
> > > > > > > > > > >>> I am OK with either FSYNC or STRICT variant.
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> LOG_ONLY name means "log without fsync".
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> On Fri, Feb 16, 2018 at 4:05 PM, Dmitriy Setrakyan <
> > > > > > > > > > >>>>
> > > > > > > > > > >>> dsetrakyan@apache.org>
> > > > > > > > > > >>>
> > > > > > > > > > >>>> wrote:
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> On Fri, Feb 16, 2018 at 7:02 AM, Ivan Rakov <
> > > > > > > > ivan.glukos@gmail.com>
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>> wrote:
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>> Why create a new term to define something that has
> > > > already
> > > > > > been
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>> defined?
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>> That makes sense. I'm ok with FSYNC.
> > > > > > > > > > >>>>>> Anton, I don't understand why we should rename
> > > LOG_ONLY
> > > > to
> > > > > > > SYNC.
> > > > > > > > > We
> > > > > > > > > > >>>>>> started this discussion with bad naming of
> DEFAULT,
> > > but
> > > > > this
> > > > > > > has
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>> nothing
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>> to
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>>> do with LOG_ONLY (even though it may be
> scientific -
> > > but
> > > > > > SYNC
> > > > > > > > > sounds
> > > > > > > > > > >>>>>> scientific as well).
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>> I agree with Ivan, we should not go wild with
> > > renaming.
> > > > > > > > However, I
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>> would
> > > > > > > > > > >>>
> > > > > > > > > > >>>> like to find out what is the meaning behind the
> > LOG_ONLY
> > > > > name.
> > > > > > > Can
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>> someone
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>> explain?
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> D.
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> --
> > > > > > > > > > >>>> Best regards,
> > > > > > > > > > >>>> Ilya
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Sergey Kozlov
> > GridGain Systems
> > www.gridgain.com
> >
>
Re: Apache Ignite 2.4 release
Posted by Vladimir Ozerov <vo...@gridgain.com>.
Sergey,
We do not have backups by default either, so essentially we are loosing
data by default. Moreover, backups are less reliable option than fsync
because a lot of users cannot afford putting servers into separate power
circuits, so a single power failure may easily lead to poweroff of the
whole cluster at once, so data is lost still. This is normal practice even
for enterprise deployments (e.g. asynchronous replication).
To make things even worse, we employ PRIMARY_SYNC mode by default! So even
if you configured backups, you still may loose data due to a single node
failure - just shutdown the PRIMARY after commit is confirmed to the client
and your recent update will disappers.
So this is what user should do to make himself safe:
1) Learn about WAL modes
2) Learn about backups
3) Learn about synchronization modes
4) Cross his fingers that he understood everything correctly and that there
are no other hidden surprises in Ignite which could lead to data loss.
Way to much for a product, claiming to be A*C*ID and persistent, don't you
think so?
Leaving deafult WAL mode with fsync resolves all these issues.
Vladimir.
On Fri, Feb 16, 2018 at 11:43 PM, Sergey Kozlov <sk...@gridgain.com>
wrote:
> I suppose some approaches used by classic databases makes no sense for
> Ignite. FSYNC requirement for databases has the nature of single host
> solution. If you have corrupted db files you have corrupted (lost) data.
>
> For Ignite the enough number of backups and the failure detecting logic can
> provide the data consistency in term "cluster data consistency".
>
>
>
> On Fri, Feb 16, 2018 at 8:57 PM, Dmitry Pavlov <dp...@gmail.com>
> wrote:
>
> > Hi, all WAL modes except NONE protects from data consistency problem
> > (B+Tree, pages, etc), which is why I suggest to avoid saying 'corrupted'
> > about 'unapplied updates'.
> >
> > Log Only and Background may cause unapplied updates in case of OS/process
> > failures.
> >
> > None mode IMO is not an option in case data consistency is needed.
> >
> > пт, 16 февр. 2018 г. в 20:49, Valentin Kulichenko <
> > valentin.kulichenko@gmail.com>:
> >
> > > Guys,
> > >
> > > While we're on this topic, what is the difference between BACKGROUND
> and
> > > NONE in terms of semantics and provided guarantees? To me it looks like
> > > both guarantee to recover the state since last checkpoint and anything
> > else
> > > can potentially be lost, so from user perspective they are the same.
> Am I
> > > missing something here?
> > >
> > > Also there is the following in Javadoc for NONE: "If an Ignite node is
> > > terminated in NONE mode abruptly, it is likely that the data stored on
> > disk
> > > is corrupted and work directory will need to be cleared for a node
> > > restart.". If this is really the case, I'm not sure NONE makes sense at
> > > all. Why would I enable persistence if I'm likely to clear the storage
> on
> > > restart?
> > >
> > > -Val
> > >
> > > On Fri, Feb 16, 2018 at 8:39 AM, Vladimir Ozerov <vozerov@gridgain.com
> >
> > > wrote:
> > >
> > > > What is the reason to have DEFAULT mode at all if you claim LOG_ONLY
> to
> > > be
> > > > completely safe? :)
> > > >
> > > > And how it could be safe provided that without fsync we loose part of
> > WAL
> > > > itself in case of crash?
> > > >
> > > > пт, 16 февр. 2018 г. в 19:32, Dmitry Pavlov <dp...@gmail.com>:
> > > >
> > > > > Thank you. Data can't be corrupted in case crash because of WAL
> > replay
> > > > > (since completed checkpoint). Physical records are used to restore
> > > > probably
> > > > > corrupted pages in persistent store (we overwrite so called 'grey
> > > zone' -
> > > > > pages we don't know for sure if they have been written).
> > > > >
> > > > > Only one effect is unwritten one or several last transactions. It
> is
> > > not
> > > > > the same with corrupted data.
> > > > >
> > > > > пт, 16 февр. 2018 г. в 19:19, Vladimir Ozerov <
> vozerov@gridgain.com
> > >:
> > > > >
> > > > > > Log only mode is not safe - data might be corrupted in case of
> > system
> > > > > > crash. Oracle - fsync, Postgres - fsync, SQL Server - fsync,
> > > Cassandra
> > > > -
> > > > > > similar to our “background”.
> > > > > >
> > > > > > пт, 16 февр. 2018 г. в 19:11, Dmitry Pavlov <
> dpavlov.spb@gmail.com
> > >:
> > > > > >
> > > > > > > Hi Vladimir,
> > > > > > >
> > > > > > > What you saying is defenetely make sence.
> > > > > > >
> > > > > > > In the same time LOG_ONLY is also safe mode, user will be able
> to
> > > > > restore
> > > > > > > system after crash. If it is not true, we should create
> critical
> > > > ticket
> > > > > > and
> > > > > > > fix it.
> > > > > > >
> > > > > > > Do you know other databases defaults, such as Cassandra,
> Oracle,
> > > > > Postgre?
> > > > > > >
> > > > > > > Sincerely,
> > > > > > > Dmitriy Pavlov
> > > > > > >
> > > > > > > пт, 16 февр. 2018 г. в 18:41, Vladimir Ozerov <
> > > vozerov@gridgain.com
> > > > >:
> > > > > > >
> > > > > > > > Igniters,
> > > > > > > >
> > > > > > > > Sorry for pouring oil on the flames, but from database
> > > perspective
> > > > > > moving
> > > > > > > > from FSYNC to non-FSYNC mode appears to be a mistake. When
> you
> > > work
> > > > > > with
> > > > > > > > database, your main expectation is that it will save your
> data.
> > > All
> > > > > > > > production database vendor make sure that you are safe, not
> > that
> > > > you
> > > > > > are
> > > > > > > > fast. Moreover, some vendors even prevent you from being in
> > > unsafe
> > > > > mode
> > > > > > > > (e.g. you cannot disable fsync in SQL Server at all).
> > > > > > > >
> > > > > > > > If we continue going in this direction, we will end up with a
> > > > > product,
> > > > > > > > which is unsafe out of the box and require tons of
> > documentation
> > > to
> > > > > > > > understand how to make it safe. Definitely not the right
> > message
> > > to
> > > > > the
> > > > > > > > market. This is like a car without brakes - would you like to
> > > drive
> > > > > it?
> > > > > > > If
> > > > > > > > this is Need For Speed game and you have unlimited lives
> > > (in-memory
> > > > > > cache
> > > > > > > > with backing store), then yes. If this is a real life with
> > > > > > (persistence)
> > > > > > > -
> > > > > > > > then no.
> > > > > > > >
> > > > > > > > On Fri, Feb 16, 2018 at 5:20 PM, Dmitriy Setrakyan <
> > > > > > > dsetrakyan@apache.org>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Well, I cannot say that I like the name LOG_ONLY, but I
> would
> > > > vote
> > > > > to
> > > > > > > > keep
> > > > > > > > > it for now, given that it is already documented in many
> > places,
> > > > > > blogs,
> > > > > > > > and
> > > > > > > > > examples.
> > > > > > > > >
> > > > > > > > > D.
> > > > > > > > >
> > > > > > > > > On Fri, Feb 16, 2018 at 8:13 AM, Ivan Rakov <
> > > > ivan.glukos@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Looks like it's an Ignite term - I've never heard of it
> > > outside
> > > > > > > Ignite
> > > > > > > > > > scope.
> > > > > > > > > >
> > > > > > > > > > Though, renaming existing enum value requires keeping old
> > as
> > > > > > > > deprecated.
> > > > > > > > > > DEFAULT is confusing enough to pay this price.
> > > > > > > > > > As for LOG_ONLY, I think we can keep it as long as it has
> > > good
> > > > > and
> > > > > > > > > > definitive javadoc.
> > > > > > > > > >
> > > > > > > > > > Best Regards,
> > > > > > > > > > Ivan Rakov
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On 16.02.2018 17:07, Dmitriy Setrakyan wrote:
> > > > > > > > > >
> > > > > > > > > >> Igniters, just to clarify, does the term LOG_ONLY mean
> > > > anything
> > > > > in
> > > > > > > the
> > > > > > > > > >> industry or is this just an Ignite term?
> > > > > > > > > >>
> > > > > > > > > >> D.
> > > > > > > > > >>
> > > > > > > > > >> On Fri, Feb 16, 2018 at 8:03 AM, Anton Vinogradov <
> > > > > > > > > >> avinogradov@gridgain.com>
> > > > > > > > > >> wrote:
> > > > > > > > > >>
> > > > > > > > > >> Log only mode: flushes application buffers.
> > > > > > > > > >>> So, in synced mode without fsync guarantee. That's why
> I
> > > > > propose
> > > > > > to
> > > > > > > > > >>> rename
> > > > > > > > > >>> it as SYNC.
> > > > > > > > > >>>
> > > > > > > > > >>> On Fri, Feb 16, 2018 at 4:49 PM, Ilya Lantukh <
> > > > > > > ilantukh@gridgain.com
> > > > > > > > >
> > > > > > > > > >>> wrote:
> > > > > > > > > >>>
> > > > > > > > > >>> I am OK with either FSYNC or STRICT variant.
> > > > > > > > > >>>>
> > > > > > > > > >>>> LOG_ONLY name means "log without fsync".
> > > > > > > > > >>>>
> > > > > > > > > >>>> On Fri, Feb 16, 2018 at 4:05 PM, Dmitriy Setrakyan <
> > > > > > > > > >>>>
> > > > > > > > > >>> dsetrakyan@apache.org>
> > > > > > > > > >>>
> > > > > > > > > >>>> wrote:
> > > > > > > > > >>>>
> > > > > > > > > >>>> On Fri, Feb 16, 2018 at 7:02 AM, Ivan Rakov <
> > > > > > > ivan.glukos@gmail.com>
> > > > > > > > > >>>>>
> > > > > > > > > >>>> wrote:
> > > > > > > > > >>>>
> > > > > > > > > >>>>> Why create a new term to define something that has
> > > already
> > > > > been
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>> defined?
> > > > > > > > > >>>>
> > > > > > > > > >>>>> That makes sense. I'm ok with FSYNC.
> > > > > > > > > >>>>>> Anton, I don't understand why we should rename
> > LOG_ONLY
> > > to
> > > > > > SYNC.
> > > > > > > > We
> > > > > > > > > >>>>>> started this discussion with bad naming of DEFAULT,
> > but
> > > > this
> > > > > > has
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>> nothing
> > > > > > > > > >>>>
> > > > > > > > > >>>>> to
> > > > > > > > > >>>>>
> > > > > > > > > >>>>>> do with LOG_ONLY (even though it may be scientific -
> > but
> > > > > SYNC
> > > > > > > > sounds
> > > > > > > > > >>>>>> scientific as well).
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> I agree with Ivan, we should not go wild with
> > renaming.
> > > > > > > However, I
> > > > > > > > > >>>>>
> > > > > > > > > >>>> would
> > > > > > > > > >>>
> > > > > > > > > >>>> like to find out what is the meaning behind the
> LOG_ONLY
> > > > name.
> > > > > > Can
> > > > > > > > > >>>>>
> > > > > > > > > >>>> someone
> > > > > > > > > >>>>
> > > > > > > > > >>>>> explain?
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> D.
> > > > > > > > > >>>>>
> > > > > > > > > >>>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>> --
> > > > > > > > > >>>> Best regards,
> > > > > > > > > >>>> Ilya
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
>
> --
> Sergey Kozlov
> GridGain Systems
> www.gridgain.com
>
Re: Apache Ignite 2.4 release
Posted by Sergey Kozlov <sk...@gridgain.com>.
I suppose some approaches used by classic databases makes no sense for
Ignite. FSYNC requirement for databases has the nature of single host
solution. If you have corrupted db files you have corrupted (lost) data.
For Ignite the enough number of backups and the failure detecting logic can
provide the data consistency in term "cluster data consistency".
On Fri, Feb 16, 2018 at 8:57 PM, Dmitry Pavlov <dp...@gmail.com>
wrote:
> Hi, all WAL modes except NONE protects from data consistency problem
> (B+Tree, pages, etc), which is why I suggest to avoid saying 'corrupted'
> about 'unapplied updates'.
>
> Log Only and Background may cause unapplied updates in case of OS/process
> failures.
>
> None mode IMO is not an option in case data consistency is needed.
>
> пт, 16 февр. 2018 г. в 20:49, Valentin Kulichenko <
> valentin.kulichenko@gmail.com>:
>
> > Guys,
> >
> > While we're on this topic, what is the difference between BACKGROUND and
> > NONE in terms of semantics and provided guarantees? To me it looks like
> > both guarantee to recover the state since last checkpoint and anything
> else
> > can potentially be lost, so from user perspective they are the same. Am I
> > missing something here?
> >
> > Also there is the following in Javadoc for NONE: "If an Ignite node is
> > terminated in NONE mode abruptly, it is likely that the data stored on
> disk
> > is corrupted and work directory will need to be cleared for a node
> > restart.". If this is really the case, I'm not sure NONE makes sense at
> > all. Why would I enable persistence if I'm likely to clear the storage on
> > restart?
> >
> > -Val
> >
> > On Fri, Feb 16, 2018 at 8:39 AM, Vladimir Ozerov <vo...@gridgain.com>
> > wrote:
> >
> > > What is the reason to have DEFAULT mode at all if you claim LOG_ONLY to
> > be
> > > completely safe? :)
> > >
> > > And how it could be safe provided that without fsync we loose part of
> WAL
> > > itself in case of crash?
> > >
> > > пт, 16 февр. 2018 г. в 19:32, Dmitry Pavlov <dp...@gmail.com>:
> > >
> > > > Thank you. Data can't be corrupted in case crash because of WAL
> replay
> > > > (since completed checkpoint). Physical records are used to restore
> > > probably
> > > > corrupted pages in persistent store (we overwrite so called 'grey
> > zone' -
> > > > pages we don't know for sure if they have been written).
> > > >
> > > > Only one effect is unwritten one or several last transactions. It is
> > not
> > > > the same with corrupted data.
> > > >
> > > > пт, 16 февр. 2018 г. в 19:19, Vladimir Ozerov <vozerov@gridgain.com
> >:
> > > >
> > > > > Log only mode is not safe - data might be corrupted in case of
> system
> > > > > crash. Oracle - fsync, Postgres - fsync, SQL Server - fsync,
> > Cassandra
> > > -
> > > > > similar to our “background”.
> > > > >
> > > > > пт, 16 февр. 2018 г. в 19:11, Dmitry Pavlov <dpavlov.spb@gmail.com
> >:
> > > > >
> > > > > > Hi Vladimir,
> > > > > >
> > > > > > What you saying is defenetely make sence.
> > > > > >
> > > > > > In the same time LOG_ONLY is also safe mode, user will be able to
> > > > restore
> > > > > > system after crash. If it is not true, we should create critical
> > > ticket
> > > > > and
> > > > > > fix it.
> > > > > >
> > > > > > Do you know other databases defaults, such as Cassandra, Oracle,
> > > > Postgre?
> > > > > >
> > > > > > Sincerely,
> > > > > > Dmitriy Pavlov
> > > > > >
> > > > > > пт, 16 февр. 2018 г. в 18:41, Vladimir Ozerov <
> > vozerov@gridgain.com
> > > >:
> > > > > >
> > > > > > > Igniters,
> > > > > > >
> > > > > > > Sorry for pouring oil on the flames, but from database
> > perspective
> > > > > moving
> > > > > > > from FSYNC to non-FSYNC mode appears to be a mistake. When you
> > work
> > > > > with
> > > > > > > database, your main expectation is that it will save your data.
> > All
> > > > > > > production database vendor make sure that you are safe, not
> that
> > > you
> > > > > are
> > > > > > > fast. Moreover, some vendors even prevent you from being in
> > unsafe
> > > > mode
> > > > > > > (e.g. you cannot disable fsync in SQL Server at all).
> > > > > > >
> > > > > > > If we continue going in this direction, we will end up with a
> > > > product,
> > > > > > > which is unsafe out of the box and require tons of
> documentation
> > to
> > > > > > > understand how to make it safe. Definitely not the right
> message
> > to
> > > > the
> > > > > > > market. This is like a car without brakes - would you like to
> > drive
> > > > it?
> > > > > > If
> > > > > > > this is Need For Speed game and you have unlimited lives
> > (in-memory
> > > > > cache
> > > > > > > with backing store), then yes. If this is a real life with
> > > > > (persistence)
> > > > > > -
> > > > > > > then no.
> > > > > > >
> > > > > > > On Fri, Feb 16, 2018 at 5:20 PM, Dmitriy Setrakyan <
> > > > > > dsetrakyan@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Well, I cannot say that I like the name LOG_ONLY, but I would
> > > vote
> > > > to
> > > > > > > keep
> > > > > > > > it for now, given that it is already documented in many
> places,
> > > > > blogs,
> > > > > > > and
> > > > > > > > examples.
> > > > > > > >
> > > > > > > > D.
> > > > > > > >
> > > > > > > > On Fri, Feb 16, 2018 at 8:13 AM, Ivan Rakov <
> > > ivan.glukos@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Looks like it's an Ignite term - I've never heard of it
> > outside
> > > > > > Ignite
> > > > > > > > > scope.
> > > > > > > > >
> > > > > > > > > Though, renaming existing enum value requires keeping old
> as
> > > > > > > deprecated.
> > > > > > > > > DEFAULT is confusing enough to pay this price.
> > > > > > > > > As for LOG_ONLY, I think we can keep it as long as it has
> > good
> > > > and
> > > > > > > > > definitive javadoc.
> > > > > > > > >
> > > > > > > > > Best Regards,
> > > > > > > > > Ivan Rakov
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On 16.02.2018 17:07, Dmitriy Setrakyan wrote:
> > > > > > > > >
> > > > > > > > >> Igniters, just to clarify, does the term LOG_ONLY mean
> > > anything
> > > > in
> > > > > > the
> > > > > > > > >> industry or is this just an Ignite term?
> > > > > > > > >>
> > > > > > > > >> D.
> > > > > > > > >>
> > > > > > > > >> On Fri, Feb 16, 2018 at 8:03 AM, Anton Vinogradov <
> > > > > > > > >> avinogradov@gridgain.com>
> > > > > > > > >> wrote:
> > > > > > > > >>
> > > > > > > > >> Log only mode: flushes application buffers.
> > > > > > > > >>> So, in synced mode without fsync guarantee. That's why I
> > > > propose
> > > > > to
> > > > > > > > >>> rename
> > > > > > > > >>> it as SYNC.
> > > > > > > > >>>
> > > > > > > > >>> On Fri, Feb 16, 2018 at 4:49 PM, Ilya Lantukh <
> > > > > > ilantukh@gridgain.com
> > > > > > > >
> > > > > > > > >>> wrote:
> > > > > > > > >>>
> > > > > > > > >>> I am OK with either FSYNC or STRICT variant.
> > > > > > > > >>>>
> > > > > > > > >>>> LOG_ONLY name means "log without fsync".
> > > > > > > > >>>>
> > > > > > > > >>>> On Fri, Feb 16, 2018 at 4:05 PM, Dmitriy Setrakyan <
> > > > > > > > >>>>
> > > > > > > > >>> dsetrakyan@apache.org>
> > > > > > > > >>>
> > > > > > > > >>>> wrote:
> > > > > > > > >>>>
> > > > > > > > >>>> On Fri, Feb 16, 2018 at 7:02 AM, Ivan Rakov <
> > > > > > ivan.glukos@gmail.com>
> > > > > > > > >>>>>
> > > > > > > > >>>> wrote:
> > > > > > > > >>>>
> > > > > > > > >>>>> Why create a new term to define something that has
> > already
> > > > been
> > > > > > > > >>>>>>
> > > > > > > > >>>>> defined?
> > > > > > > > >>>>
> > > > > > > > >>>>> That makes sense. I'm ok with FSYNC.
> > > > > > > > >>>>>> Anton, I don't understand why we should rename
> LOG_ONLY
> > to
> > > > > SYNC.
> > > > > > > We
> > > > > > > > >>>>>> started this discussion with bad naming of DEFAULT,
> but
> > > this
> > > > > has
> > > > > > > > >>>>>>
> > > > > > > > >>>>> nothing
> > > > > > > > >>>>
> > > > > > > > >>>>> to
> > > > > > > > >>>>>
> > > > > > > > >>>>>> do with LOG_ONLY (even though it may be scientific -
> but
> > > > SYNC
> > > > > > > sounds
> > > > > > > > >>>>>> scientific as well).
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> I agree with Ivan, we should not go wild with
> renaming.
> > > > > > However, I
> > > > > > > > >>>>>
> > > > > > > > >>>> would
> > > > > > > > >>>
> > > > > > > > >>>> like to find out what is the meaning behind the LOG_ONLY
> > > name.
> > > > > Can
> > > > > > > > >>>>>
> > > > > > > > >>>> someone
> > > > > > > > >>>>
> > > > > > > > >>>>> explain?
> > > > > > > > >>>>>
> > > > > > > > >>>>> D.
> > > > > > > > >>>>>
> > > > > > > > >>>>>
> > > > > > > > >>>>
> > > > > > > > >>>> --
> > > > > > > > >>>> Best regards,
> > > > > > > > >>>> Ilya
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
--
Sergey Kozlov
GridGain Systems
www.gridgain.com
Re: Apache Ignite 2.4 release
Posted by Dmitry Pavlov <dp...@gmail.com>.
Hi, all WAL modes except NONE protects from data consistency problem
(B+Tree, pages, etc), which is why I suggest to avoid saying 'corrupted'
about 'unapplied updates'.
Log Only and Background may cause unapplied updates in case of OS/process
failures.
None mode IMO is not an option in case data consistency is needed.
пт, 16 февр. 2018 г. в 20:49, Valentin Kulichenko <
valentin.kulichenko@gmail.com>:
> Guys,
>
> While we're on this topic, what is the difference between BACKGROUND and
> NONE in terms of semantics and provided guarantees? To me it looks like
> both guarantee to recover the state since last checkpoint and anything else
> can potentially be lost, so from user perspective they are the same. Am I
> missing something here?
>
> Also there is the following in Javadoc for NONE: "If an Ignite node is
> terminated in NONE mode abruptly, it is likely that the data stored on disk
> is corrupted and work directory will need to be cleared for a node
> restart.". If this is really the case, I'm not sure NONE makes sense at
> all. Why would I enable persistence if I'm likely to clear the storage on
> restart?
>
> -Val
>
> On Fri, Feb 16, 2018 at 8:39 AM, Vladimir Ozerov <vo...@gridgain.com>
> wrote:
>
> > What is the reason to have DEFAULT mode at all if you claim LOG_ONLY to
> be
> > completely safe? :)
> >
> > And how it could be safe provided that without fsync we loose part of WAL
> > itself in case of crash?
> >
> > пт, 16 февр. 2018 г. в 19:32, Dmitry Pavlov <dp...@gmail.com>:
> >
> > > Thank you. Data can't be corrupted in case crash because of WAL replay
> > > (since completed checkpoint). Physical records are used to restore
> > probably
> > > corrupted pages in persistent store (we overwrite so called 'grey
> zone' -
> > > pages we don't know for sure if they have been written).
> > >
> > > Only one effect is unwritten one or several last transactions. It is
> not
> > > the same with corrupted data.
> > >
> > > пт, 16 февр. 2018 г. в 19:19, Vladimir Ozerov <vo...@gridgain.com>:
> > >
> > > > Log only mode is not safe - data might be corrupted in case of system
> > > > crash. Oracle - fsync, Postgres - fsync, SQL Server - fsync,
> Cassandra
> > -
> > > > similar to our “background”.
> > > >
> > > > пт, 16 февр. 2018 г. в 19:11, Dmitry Pavlov <dp...@gmail.com>:
> > > >
> > > > > Hi Vladimir,
> > > > >
> > > > > What you saying is defenetely make sence.
> > > > >
> > > > > In the same time LOG_ONLY is also safe mode, user will be able to
> > > restore
> > > > > system after crash. If it is not true, we should create critical
> > ticket
> > > > and
> > > > > fix it.
> > > > >
> > > > > Do you know other databases defaults, such as Cassandra, Oracle,
> > > Postgre?
> > > > >
> > > > > Sincerely,
> > > > > Dmitriy Pavlov
> > > > >
> > > > > пт, 16 февр. 2018 г. в 18:41, Vladimir Ozerov <
> vozerov@gridgain.com
> > >:
> > > > >
> > > > > > Igniters,
> > > > > >
> > > > > > Sorry for pouring oil on the flames, but from database
> perspective
> > > > moving
> > > > > > from FSYNC to non-FSYNC mode appears to be a mistake. When you
> work
> > > > with
> > > > > > database, your main expectation is that it will save your data.
> All
> > > > > > production database vendor make sure that you are safe, not that
> > you
> > > > are
> > > > > > fast. Moreover, some vendors even prevent you from being in
> unsafe
> > > mode
> > > > > > (e.g. you cannot disable fsync in SQL Server at all).
> > > > > >
> > > > > > If we continue going in this direction, we will end up with a
> > > product,
> > > > > > which is unsafe out of the box and require tons of documentation
> to
> > > > > > understand how to make it safe. Definitely not the right message
> to
> > > the
> > > > > > market. This is like a car without brakes - would you like to
> drive
> > > it?
> > > > > If
> > > > > > this is Need For Speed game and you have unlimited lives
> (in-memory
> > > > cache
> > > > > > with backing store), then yes. If this is a real life with
> > > > (persistence)
> > > > > -
> > > > > > then no.
> > > > > >
> > > > > > On Fri, Feb 16, 2018 at 5:20 PM, Dmitriy Setrakyan <
> > > > > dsetrakyan@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > Well, I cannot say that I like the name LOG_ONLY, but I would
> > vote
> > > to
> > > > > > keep
> > > > > > > it for now, given that it is already documented in many places,
> > > > blogs,
> > > > > > and
> > > > > > > examples.
> > > > > > >
> > > > > > > D.
> > > > > > >
> > > > > > > On Fri, Feb 16, 2018 at 8:13 AM, Ivan Rakov <
> > ivan.glukos@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > Looks like it's an Ignite term - I've never heard of it
> outside
> > > > > Ignite
> > > > > > > > scope.
> > > > > > > >
> > > > > > > > Though, renaming existing enum value requires keeping old as
> > > > > > deprecated.
> > > > > > > > DEFAULT is confusing enough to pay this price.
> > > > > > > > As for LOG_ONLY, I think we can keep it as long as it has
> good
> > > and
> > > > > > > > definitive javadoc.
> > > > > > > >
> > > > > > > > Best Regards,
> > > > > > > > Ivan Rakov
> > > > > > > >
> > > > > > > >
> > > > > > > > On 16.02.2018 17:07, Dmitriy Setrakyan wrote:
> > > > > > > >
> > > > > > > >> Igniters, just to clarify, does the term LOG_ONLY mean
> > anything
> > > in
> > > > > the
> > > > > > > >> industry or is this just an Ignite term?
> > > > > > > >>
> > > > > > > >> D.
> > > > > > > >>
> > > > > > > >> On Fri, Feb 16, 2018 at 8:03 AM, Anton Vinogradov <
> > > > > > > >> avinogradov@gridgain.com>
> > > > > > > >> wrote:
> > > > > > > >>
> > > > > > > >> Log only mode: flushes application buffers.
> > > > > > > >>> So, in synced mode without fsync guarantee. That's why I
> > > propose
> > > > to
> > > > > > > >>> rename
> > > > > > > >>> it as SYNC.
> > > > > > > >>>
> > > > > > > >>> On Fri, Feb 16, 2018 at 4:49 PM, Ilya Lantukh <
> > > > > ilantukh@gridgain.com
> > > > > > >
> > > > > > > >>> wrote:
> > > > > > > >>>
> > > > > > > >>> I am OK with either FSYNC or STRICT variant.
> > > > > > > >>>>
> > > > > > > >>>> LOG_ONLY name means "log without fsync".
> > > > > > > >>>>
> > > > > > > >>>> On Fri, Feb 16, 2018 at 4:05 PM, Dmitriy Setrakyan <
> > > > > > > >>>>
> > > > > > > >>> dsetrakyan@apache.org>
> > > > > > > >>>
> > > > > > > >>>> wrote:
> > > > > > > >>>>
> > > > > > > >>>> On Fri, Feb 16, 2018 at 7:02 AM, Ivan Rakov <
> > > > > ivan.glukos@gmail.com>
> > > > > > > >>>>>
> > > > > > > >>>> wrote:
> > > > > > > >>>>
> > > > > > > >>>>> Why create a new term to define something that has
> already
> > > been
> > > > > > > >>>>>>
> > > > > > > >>>>> defined?
> > > > > > > >>>>
> > > > > > > >>>>> That makes sense. I'm ok with FSYNC.
> > > > > > > >>>>>> Anton, I don't understand why we should rename LOG_ONLY
> to
> > > > SYNC.
> > > > > > We
> > > > > > > >>>>>> started this discussion with bad naming of DEFAULT, but
> > this
> > > > has
> > > > > > > >>>>>>
> > > > > > > >>>>> nothing
> > > > > > > >>>>
> > > > > > > >>>>> to
> > > > > > > >>>>>
> > > > > > > >>>>>> do with LOG_ONLY (even though it may be scientific - but
> > > SYNC
> > > > > > sounds
> > > > > > > >>>>>> scientific as well).
> > > > > > > >>>>>>
> > > > > > > >>>>>> I agree with Ivan, we should not go wild with renaming.
> > > > > However, I
> > > > > > > >>>>>
> > > > > > > >>>> would
> > > > > > > >>>
> > > > > > > >>>> like to find out what is the meaning behind the LOG_ONLY
> > name.
> > > > Can
> > > > > > > >>>>>
> > > > > > > >>>> someone
> > > > > > > >>>>
> > > > > > > >>>>> explain?
> > > > > > > >>>>>
> > > > > > > >>>>> D.
> > > > > > > >>>>>
> > > > > > > >>>>>
> > > > > > > >>>>
> > > > > > > >>>> --
> > > > > > > >>>> Best regards,
> > > > > > > >>>> Ilya
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Apache Ignite 2.4 release
Posted by Valentin Kulichenko <va...@gmail.com>.
Guys,
While we're on this topic, what is the difference between BACKGROUND and
NONE in terms of semantics and provided guarantees? To me it looks like
both guarantee to recover the state since last checkpoint and anything else
can potentially be lost, so from user perspective they are the same. Am I
missing something here?
Also there is the following in Javadoc for NONE: "If an Ignite node is
terminated in NONE mode abruptly, it is likely that the data stored on disk
is corrupted and work directory will need to be cleared for a node
restart.". If this is really the case, I'm not sure NONE makes sense at
all. Why would I enable persistence if I'm likely to clear the storage on
restart?
-Val
On Fri, Feb 16, 2018 at 8:39 AM, Vladimir Ozerov <vo...@gridgain.com>
wrote:
> What is the reason to have DEFAULT mode at all if you claim LOG_ONLY to be
> completely safe? :)
>
> And how it could be safe provided that without fsync we loose part of WAL
> itself in case of crash?
>
> пт, 16 февр. 2018 г. в 19:32, Dmitry Pavlov <dp...@gmail.com>:
>
> > Thank you. Data can't be corrupted in case crash because of WAL replay
> > (since completed checkpoint). Physical records are used to restore
> probably
> > corrupted pages in persistent store (we overwrite so called 'grey zone' -
> > pages we don't know for sure if they have been written).
> >
> > Only one effect is unwritten one or several last transactions. It is not
> > the same with corrupted data.
> >
> > пт, 16 февр. 2018 г. в 19:19, Vladimir Ozerov <vo...@gridgain.com>:
> >
> > > Log only mode is not safe - data might be corrupted in case of system
> > > crash. Oracle - fsync, Postgres - fsync, SQL Server - fsync, Cassandra
> -
> > > similar to our “background”.
> > >
> > > пт, 16 февр. 2018 г. в 19:11, Dmitry Pavlov <dp...@gmail.com>:
> > >
> > > > Hi Vladimir,
> > > >
> > > > What you saying is defenetely make sence.
> > > >
> > > > In the same time LOG_ONLY is also safe mode, user will be able to
> > restore
> > > > system after crash. If it is not true, we should create critical
> ticket
> > > and
> > > > fix it.
> > > >
> > > > Do you know other databases defaults, such as Cassandra, Oracle,
> > Postgre?
> > > >
> > > > Sincerely,
> > > > Dmitriy Pavlov
> > > >
> > > > пт, 16 февр. 2018 г. в 18:41, Vladimir Ozerov <vozerov@gridgain.com
> >:
> > > >
> > > > > Igniters,
> > > > >
> > > > > Sorry for pouring oil on the flames, but from database perspective
> > > moving
> > > > > from FSYNC to non-FSYNC mode appears to be a mistake. When you work
> > > with
> > > > > database, your main expectation is that it will save your data. All
> > > > > production database vendor make sure that you are safe, not that
> you
> > > are
> > > > > fast. Moreover, some vendors even prevent you from being in unsafe
> > mode
> > > > > (e.g. you cannot disable fsync in SQL Server at all).
> > > > >
> > > > > If we continue going in this direction, we will end up with a
> > product,
> > > > > which is unsafe out of the box and require tons of documentation to
> > > > > understand how to make it safe. Definitely not the right message to
> > the
> > > > > market. This is like a car without brakes - would you like to drive
> > it?
> > > > If
> > > > > this is Need For Speed game and you have unlimited lives (in-memory
> > > cache
> > > > > with backing store), then yes. If this is a real life with
> > > (persistence)
> > > > -
> > > > > then no.
> > > > >
> > > > > On Fri, Feb 16, 2018 at 5:20 PM, Dmitriy Setrakyan <
> > > > dsetrakyan@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Well, I cannot say that I like the name LOG_ONLY, but I would
> vote
> > to
> > > > > keep
> > > > > > it for now, given that it is already documented in many places,
> > > blogs,
> > > > > and
> > > > > > examples.
> > > > > >
> > > > > > D.
> > > > > >
> > > > > > On Fri, Feb 16, 2018 at 8:13 AM, Ivan Rakov <
> ivan.glukos@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Looks like it's an Ignite term - I've never heard of it outside
> > > > Ignite
> > > > > > > scope.
> > > > > > >
> > > > > > > Though, renaming existing enum value requires keeping old as
> > > > > deprecated.
> > > > > > > DEFAULT is confusing enough to pay this price.
> > > > > > > As for LOG_ONLY, I think we can keep it as long as it has good
> > and
> > > > > > > definitive javadoc.
> > > > > > >
> > > > > > > Best Regards,
> > > > > > > Ivan Rakov
> > > > > > >
> > > > > > >
> > > > > > > On 16.02.2018 17:07, Dmitriy Setrakyan wrote:
> > > > > > >
> > > > > > >> Igniters, just to clarify, does the term LOG_ONLY mean
> anything
> > in
> > > > the
> > > > > > >> industry or is this just an Ignite term?
> > > > > > >>
> > > > > > >> D.
> > > > > > >>
> > > > > > >> On Fri, Feb 16, 2018 at 8:03 AM, Anton Vinogradov <
> > > > > > >> avinogradov@gridgain.com>
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >> Log only mode: flushes application buffers.
> > > > > > >>> So, in synced mode without fsync guarantee. That's why I
> > propose
> > > to
> > > > > > >>> rename
> > > > > > >>> it as SYNC.
> > > > > > >>>
> > > > > > >>> On Fri, Feb 16, 2018 at 4:49 PM, Ilya Lantukh <
> > > > ilantukh@gridgain.com
> > > > > >
> > > > > > >>> wrote:
> > > > > > >>>
> > > > > > >>> I am OK with either FSYNC or STRICT variant.
> > > > > > >>>>
> > > > > > >>>> LOG_ONLY name means "log without fsync".
> > > > > > >>>>
> > > > > > >>>> On Fri, Feb 16, 2018 at 4:05 PM, Dmitriy Setrakyan <
> > > > > > >>>>
> > > > > > >>> dsetrakyan@apache.org>
> > > > > > >>>
> > > > > > >>>> wrote:
> > > > > > >>>>
> > > > > > >>>> On Fri, Feb 16, 2018 at 7:02 AM, Ivan Rakov <
> > > > ivan.glukos@gmail.com>
> > > > > > >>>>>
> > > > > > >>>> wrote:
> > > > > > >>>>
> > > > > > >>>>> Why create a new term to define something that has already
> > been
> > > > > > >>>>>>
> > > > > > >>>>> defined?
> > > > > > >>>>
> > > > > > >>>>> That makes sense. I'm ok with FSYNC.
> > > > > > >>>>>> Anton, I don't understand why we should rename LOG_ONLY to
> > > SYNC.
> > > > > We
> > > > > > >>>>>> started this discussion with bad naming of DEFAULT, but
> this
> > > has
> > > > > > >>>>>>
> > > > > > >>>>> nothing
> > > > > > >>>>
> > > > > > >>>>> to
> > > > > > >>>>>
> > > > > > >>>>>> do with LOG_ONLY (even though it may be scientific - but
> > SYNC
> > > > > sounds
> > > > > > >>>>>> scientific as well).
> > > > > > >>>>>>
> > > > > > >>>>>> I agree with Ivan, we should not go wild with renaming.
> > > > However, I
> > > > > > >>>>>
> > > > > > >>>> would
> > > > > > >>>
> > > > > > >>>> like to find out what is the meaning behind the LOG_ONLY
> name.
> > > Can
> > > > > > >>>>>
> > > > > > >>>> someone
> > > > > > >>>>
> > > > > > >>>>> explain?
> > > > > > >>>>>
> > > > > > >>>>> D.
> > > > > > >>>>>
> > > > > > >>>>>
> > > > > > >>>>
> > > > > > >>>> --
> > > > > > >>>> Best regards,
> > > > > > >>>> Ilya
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Apache Ignite 2.4 release
Posted by Vladimir Ozerov <vo...@gridgain.com>.
What is the reason to have DEFAULT mode at all if you claim LOG_ONLY to be
completely safe? :)
And how it could be safe provided that without fsync we loose part of WAL
itself in case of crash?
пт, 16 февр. 2018 г. в 19:32, Dmitry Pavlov <dp...@gmail.com>:
> Thank you. Data can't be corrupted in case crash because of WAL replay
> (since completed checkpoint). Physical records are used to restore probably
> corrupted pages in persistent store (we overwrite so called 'grey zone' -
> pages we don't know for sure if they have been written).
>
> Only one effect is unwritten one or several last transactions. It is not
> the same with corrupted data.
>
> пт, 16 февр. 2018 г. в 19:19, Vladimir Ozerov <vo...@gridgain.com>:
>
> > Log only mode is not safe - data might be corrupted in case of system
> > crash. Oracle - fsync, Postgres - fsync, SQL Server - fsync, Cassandra -
> > similar to our “background”.
> >
> > пт, 16 февр. 2018 г. в 19:11, Dmitry Pavlov <dp...@gmail.com>:
> >
> > > Hi Vladimir,
> > >
> > > What you saying is defenetely make sence.
> > >
> > > In the same time LOG_ONLY is also safe mode, user will be able to
> restore
> > > system after crash. If it is not true, we should create critical ticket
> > and
> > > fix it.
> > >
> > > Do you know other databases defaults, such as Cassandra, Oracle,
> Postgre?
> > >
> > > Sincerely,
> > > Dmitriy Pavlov
> > >
> > > пт, 16 февр. 2018 г. в 18:41, Vladimir Ozerov <vo...@gridgain.com>:
> > >
> > > > Igniters,
> > > >
> > > > Sorry for pouring oil on the flames, but from database perspective
> > moving
> > > > from FSYNC to non-FSYNC mode appears to be a mistake. When you work
> > with
> > > > database, your main expectation is that it will save your data. All
> > > > production database vendor make sure that you are safe, not that you
> > are
> > > > fast. Moreover, some vendors even prevent you from being in unsafe
> mode
> > > > (e.g. you cannot disable fsync in SQL Server at all).
> > > >
> > > > If we continue going in this direction, we will end up with a
> product,
> > > > which is unsafe out of the box and require tons of documentation to
> > > > understand how to make it safe. Definitely not the right message to
> the
> > > > market. This is like a car without brakes - would you like to drive
> it?
> > > If
> > > > this is Need For Speed game and you have unlimited lives (in-memory
> > cache
> > > > with backing store), then yes. If this is a real life with
> > (persistence)
> > > -
> > > > then no.
> > > >
> > > > On Fri, Feb 16, 2018 at 5:20 PM, Dmitriy Setrakyan <
> > > dsetrakyan@apache.org>
> > > > wrote:
> > > >
> > > > > Well, I cannot say that I like the name LOG_ONLY, but I would vote
> to
> > > > keep
> > > > > it for now, given that it is already documented in many places,
> > blogs,
> > > > and
> > > > > examples.
> > > > >
> > > > > D.
> > > > >
> > > > > On Fri, Feb 16, 2018 at 8:13 AM, Ivan Rakov <ivan.glukos@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > > Looks like it's an Ignite term - I've never heard of it outside
> > > Ignite
> > > > > > scope.
> > > > > >
> > > > > > Though, renaming existing enum value requires keeping old as
> > > > deprecated.
> > > > > > DEFAULT is confusing enough to pay this price.
> > > > > > As for LOG_ONLY, I think we can keep it as long as it has good
> and
> > > > > > definitive javadoc.
> > > > > >
> > > > > > Best Regards,
> > > > > > Ivan Rakov
> > > > > >
> > > > > >
> > > > > > On 16.02.2018 17:07, Dmitriy Setrakyan wrote:
> > > > > >
> > > > > >> Igniters, just to clarify, does the term LOG_ONLY mean anything
> in
> > > the
> > > > > >> industry or is this just an Ignite term?
> > > > > >>
> > > > > >> D.
> > > > > >>
> > > > > >> On Fri, Feb 16, 2018 at 8:03 AM, Anton Vinogradov <
> > > > > >> avinogradov@gridgain.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >> Log only mode: flushes application buffers.
> > > > > >>> So, in synced mode without fsync guarantee. That's why I
> propose
> > to
> > > > > >>> rename
> > > > > >>> it as SYNC.
> > > > > >>>
> > > > > >>> On Fri, Feb 16, 2018 at 4:49 PM, Ilya Lantukh <
> > > ilantukh@gridgain.com
> > > > >
> > > > > >>> wrote:
> > > > > >>>
> > > > > >>> I am OK with either FSYNC or STRICT variant.
> > > > > >>>>
> > > > > >>>> LOG_ONLY name means "log without fsync".
> > > > > >>>>
> > > > > >>>> On Fri, Feb 16, 2018 at 4:05 PM, Dmitriy Setrakyan <
> > > > > >>>>
> > > > > >>> dsetrakyan@apache.org>
> > > > > >>>
> > > > > >>>> wrote:
> > > > > >>>>
> > > > > >>>> On Fri, Feb 16, 2018 at 7:02 AM, Ivan Rakov <
> > > ivan.glukos@gmail.com>
> > > > > >>>>>
> > > > > >>>> wrote:
> > > > > >>>>
> > > > > >>>>> Why create a new term to define something that has already
> been
> > > > > >>>>>>
> > > > > >>>>> defined?
> > > > > >>>>
> > > > > >>>>> That makes sense. I'm ok with FSYNC.
> > > > > >>>>>> Anton, I don't understand why we should rename LOG_ONLY to
> > SYNC.
> > > > We
> > > > > >>>>>> started this discussion with bad naming of DEFAULT, but this
> > has
> > > > > >>>>>>
> > > > > >>>>> nothing
> > > > > >>>>
> > > > > >>>>> to
> > > > > >>>>>
> > > > > >>>>>> do with LOG_ONLY (even though it may be scientific - but
> SYNC
> > > > sounds
> > > > > >>>>>> scientific as well).
> > > > > >>>>>>
> > > > > >>>>>> I agree with Ivan, we should not go wild with renaming.
> > > However, I
> > > > > >>>>>
> > > > > >>>> would
> > > > > >>>
> > > > > >>>> like to find out what is the meaning behind the LOG_ONLY name.
> > Can
> > > > > >>>>>
> > > > > >>>> someone
> > > > > >>>>
> > > > > >>>>> explain?
> > > > > >>>>>
> > > > > >>>>> D.
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>
> > > > > >>>> --
> > > > > >>>> Best regards,
> > > > > >>>> Ilya
> > > > > >>>>
> > > > > >>>>
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Apache Ignite 2.4 release
Posted by Dmitry Pavlov <dp...@gmail.com>.
Thank you. Data can't be corrupted in case crash because of WAL replay
(since completed checkpoint). Physical records are used to restore probably
corrupted pages in persistent store (we overwrite so called 'grey zone' -
pages we don't know for sure if they have been written).
Only one effect is unwritten one or several last transactions. It is not
the same with corrupted data.
пт, 16 февр. 2018 г. в 19:19, Vladimir Ozerov <vo...@gridgain.com>:
> Log only mode is not safe - data might be corrupted in case of system
> crash. Oracle - fsync, Postgres - fsync, SQL Server - fsync, Cassandra -
> similar to our “background”.
>
> пт, 16 февр. 2018 г. в 19:11, Dmitry Pavlov <dp...@gmail.com>:
>
> > Hi Vladimir,
> >
> > What you saying is defenetely make sence.
> >
> > In the same time LOG_ONLY is also safe mode, user will be able to restore
> > system after crash. If it is not true, we should create critical ticket
> and
> > fix it.
> >
> > Do you know other databases defaults, such as Cassandra, Oracle, Postgre?
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > пт, 16 февр. 2018 г. в 18:41, Vladimir Ozerov <vo...@gridgain.com>:
> >
> > > Igniters,
> > >
> > > Sorry for pouring oil on the flames, but from database perspective
> moving
> > > from FSYNC to non-FSYNC mode appears to be a mistake. When you work
> with
> > > database, your main expectation is that it will save your data. All
> > > production database vendor make sure that you are safe, not that you
> are
> > > fast. Moreover, some vendors even prevent you from being in unsafe mode
> > > (e.g. you cannot disable fsync in SQL Server at all).
> > >
> > > If we continue going in this direction, we will end up with a product,
> > > which is unsafe out of the box and require tons of documentation to
> > > understand how to make it safe. Definitely not the right message to the
> > > market. This is like a car without brakes - would you like to drive it?
> > If
> > > this is Need For Speed game and you have unlimited lives (in-memory
> cache
> > > with backing store), then yes. If this is a real life with
> (persistence)
> > -
> > > then no.
> > >
> > > On Fri, Feb 16, 2018 at 5:20 PM, Dmitriy Setrakyan <
> > dsetrakyan@apache.org>
> > > wrote:
> > >
> > > > Well, I cannot say that I like the name LOG_ONLY, but I would vote to
> > > keep
> > > > it for now, given that it is already documented in many places,
> blogs,
> > > and
> > > > examples.
> > > >
> > > > D.
> > > >
> > > > On Fri, Feb 16, 2018 at 8:13 AM, Ivan Rakov <iv...@gmail.com>
> > > wrote:
> > > >
> > > > > Looks like it's an Ignite term - I've never heard of it outside
> > Ignite
> > > > > scope.
> > > > >
> > > > > Though, renaming existing enum value requires keeping old as
> > > deprecated.
> > > > > DEFAULT is confusing enough to pay this price.
> > > > > As for LOG_ONLY, I think we can keep it as long as it has good and
> > > > > definitive javadoc.
> > > > >
> > > > > Best Regards,
> > > > > Ivan Rakov
> > > > >
> > > > >
> > > > > On 16.02.2018 17:07, Dmitriy Setrakyan wrote:
> > > > >
> > > > >> Igniters, just to clarify, does the term LOG_ONLY mean anything in
> > the
> > > > >> industry or is this just an Ignite term?
> > > > >>
> > > > >> D.
> > > > >>
> > > > >> On Fri, Feb 16, 2018 at 8:03 AM, Anton Vinogradov <
> > > > >> avinogradov@gridgain.com>
> > > > >> wrote:
> > > > >>
> > > > >> Log only mode: flushes application buffers.
> > > > >>> So, in synced mode without fsync guarantee. That's why I propose
> to
> > > > >>> rename
> > > > >>> it as SYNC.
> > > > >>>
> > > > >>> On Fri, Feb 16, 2018 at 4:49 PM, Ilya Lantukh <
> > ilantukh@gridgain.com
> > > >
> > > > >>> wrote:
> > > > >>>
> > > > >>> I am OK with either FSYNC or STRICT variant.
> > > > >>>>
> > > > >>>> LOG_ONLY name means "log without fsync".
> > > > >>>>
> > > > >>>> On Fri, Feb 16, 2018 at 4:05 PM, Dmitriy Setrakyan <
> > > > >>>>
> > > > >>> dsetrakyan@apache.org>
> > > > >>>
> > > > >>>> wrote:
> > > > >>>>
> > > > >>>> On Fri, Feb 16, 2018 at 7:02 AM, Ivan Rakov <
> > ivan.glukos@gmail.com>
> > > > >>>>>
> > > > >>>> wrote:
> > > > >>>>
> > > > >>>>> Why create a new term to define something that has already been
> > > > >>>>>>
> > > > >>>>> defined?
> > > > >>>>
> > > > >>>>> That makes sense. I'm ok with FSYNC.
> > > > >>>>>> Anton, I don't understand why we should rename LOG_ONLY to
> SYNC.
> > > We
> > > > >>>>>> started this discussion with bad naming of DEFAULT, but this
> has
> > > > >>>>>>
> > > > >>>>> nothing
> > > > >>>>
> > > > >>>>> to
> > > > >>>>>
> > > > >>>>>> do with LOG_ONLY (even though it may be scientific - but SYNC
> > > sounds
> > > > >>>>>> scientific as well).
> > > > >>>>>>
> > > > >>>>>> I agree with Ivan, we should not go wild with renaming.
> > However, I
> > > > >>>>>
> > > > >>>> would
> > > > >>>
> > > > >>>> like to find out what is the meaning behind the LOG_ONLY name.
> Can
> > > > >>>>>
> > > > >>>> someone
> > > > >>>>
> > > > >>>>> explain?
> > > > >>>>>
> > > > >>>>> D.
> > > > >>>>>
> > > > >>>>>
> > > > >>>>
> > > > >>>> --
> > > > >>>> Best regards,
> > > > >>>> Ilya
> > > > >>>>
> > > > >>>>
> > > > >
> > > >
> > >
> >
>
Re: Apache Ignite 2.4 release
Posted by Vladimir Ozerov <vo...@gridgain.com>.
Log only mode is not safe - data might be corrupted in case of system
crash. Oracle - fsync, Postgres - fsync, SQL Server - fsync, Cassandra -
similar to our “background”.
пт, 16 февр. 2018 г. в 19:11, Dmitry Pavlov <dp...@gmail.com>:
> Hi Vladimir,
>
> What you saying is defenetely make sence.
>
> In the same time LOG_ONLY is also safe mode, user will be able to restore
> system after crash. If it is not true, we should create critical ticket and
> fix it.
>
> Do you know other databases defaults, such as Cassandra, Oracle, Postgre?
>
> Sincerely,
> Dmitriy Pavlov
>
> пт, 16 февр. 2018 г. в 18:41, Vladimir Ozerov <vo...@gridgain.com>:
>
> > Igniters,
> >
> > Sorry for pouring oil on the flames, but from database perspective moving
> > from FSYNC to non-FSYNC mode appears to be a mistake. When you work with
> > database, your main expectation is that it will save your data. All
> > production database vendor make sure that you are safe, not that you are
> > fast. Moreover, some vendors even prevent you from being in unsafe mode
> > (e.g. you cannot disable fsync in SQL Server at all).
> >
> > If we continue going in this direction, we will end up with a product,
> > which is unsafe out of the box and require tons of documentation to
> > understand how to make it safe. Definitely not the right message to the
> > market. This is like a car without brakes - would you like to drive it?
> If
> > this is Need For Speed game and you have unlimited lives (in-memory cache
> > with backing store), then yes. If this is a real life with (persistence)
> -
> > then no.
> >
> > On Fri, Feb 16, 2018 at 5:20 PM, Dmitriy Setrakyan <
> dsetrakyan@apache.org>
> > wrote:
> >
> > > Well, I cannot say that I like the name LOG_ONLY, but I would vote to
> > keep
> > > it for now, given that it is already documented in many places, blogs,
> > and
> > > examples.
> > >
> > > D.
> > >
> > > On Fri, Feb 16, 2018 at 8:13 AM, Ivan Rakov <iv...@gmail.com>
> > wrote:
> > >
> > > > Looks like it's an Ignite term - I've never heard of it outside
> Ignite
> > > > scope.
> > > >
> > > > Though, renaming existing enum value requires keeping old as
> > deprecated.
> > > > DEFAULT is confusing enough to pay this price.
> > > > As for LOG_ONLY, I think we can keep it as long as it has good and
> > > > definitive javadoc.
> > > >
> > > > Best Regards,
> > > > Ivan Rakov
> > > >
> > > >
> > > > On 16.02.2018 17:07, Dmitriy Setrakyan wrote:
> > > >
> > > >> Igniters, just to clarify, does the term LOG_ONLY mean anything in
> the
> > > >> industry or is this just an Ignite term?
> > > >>
> > > >> D.
> > > >>
> > > >> On Fri, Feb 16, 2018 at 8:03 AM, Anton Vinogradov <
> > > >> avinogradov@gridgain.com>
> > > >> wrote:
> > > >>
> > > >> Log only mode: flushes application buffers.
> > > >>> So, in synced mode without fsync guarantee. That's why I propose to
> > > >>> rename
> > > >>> it as SYNC.
> > > >>>
> > > >>> On Fri, Feb 16, 2018 at 4:49 PM, Ilya Lantukh <
> ilantukh@gridgain.com
> > >
> > > >>> wrote:
> > > >>>
> > > >>> I am OK with either FSYNC or STRICT variant.
> > > >>>>
> > > >>>> LOG_ONLY name means "log without fsync".
> > > >>>>
> > > >>>> On Fri, Feb 16, 2018 at 4:05 PM, Dmitriy Setrakyan <
> > > >>>>
> > > >>> dsetrakyan@apache.org>
> > > >>>
> > > >>>> wrote:
> > > >>>>
> > > >>>> On Fri, Feb 16, 2018 at 7:02 AM, Ivan Rakov <
> ivan.glukos@gmail.com>
> > > >>>>>
> > > >>>> wrote:
> > > >>>>
> > > >>>>> Why create a new term to define something that has already been
> > > >>>>>>
> > > >>>>> defined?
> > > >>>>
> > > >>>>> That makes sense. I'm ok with FSYNC.
> > > >>>>>> Anton, I don't understand why we should rename LOG_ONLY to SYNC.
> > We
> > > >>>>>> started this discussion with bad naming of DEFAULT, but this has
> > > >>>>>>
> > > >>>>> nothing
> > > >>>>
> > > >>>>> to
> > > >>>>>
> > > >>>>>> do with LOG_ONLY (even though it may be scientific - but SYNC
> > sounds
> > > >>>>>> scientific as well).
> > > >>>>>>
> > > >>>>>> I agree with Ivan, we should not go wild with renaming.
> However, I
> > > >>>>>
> > > >>>> would
> > > >>>
> > > >>>> like to find out what is the meaning behind the LOG_ONLY name. Can
> > > >>>>>
> > > >>>> someone
> > > >>>>
> > > >>>>> explain?
> > > >>>>>
> > > >>>>> D.
> > > >>>>>
> > > >>>>>
> > > >>>>
> > > >>>> --
> > > >>>> Best regards,
> > > >>>> Ilya
> > > >>>>
> > > >>>>
> > > >
> > >
> >
>
Re: Apache Ignite 2.4 release
Posted by Dmitry Pavlov <dp...@gmail.com>.
Hi Vladimir,
What you saying is defenetely make sence.
In the same time LOG_ONLY is also safe mode, user will be able to restore
system after crash. If it is not true, we should create critical ticket and
fix it.
Do you know other databases defaults, such as Cassandra, Oracle, Postgre?
Sincerely,
Dmitriy Pavlov
пт, 16 февр. 2018 г. в 18:41, Vladimir Ozerov <vo...@gridgain.com>:
> Igniters,
>
> Sorry for pouring oil on the flames, but from database perspective moving
> from FSYNC to non-FSYNC mode appears to be a mistake. When you work with
> database, your main expectation is that it will save your data. All
> production database vendor make sure that you are safe, not that you are
> fast. Moreover, some vendors even prevent you from being in unsafe mode
> (e.g. you cannot disable fsync in SQL Server at all).
>
> If we continue going in this direction, we will end up with a product,
> which is unsafe out of the box and require tons of documentation to
> understand how to make it safe. Definitely not the right message to the
> market. This is like a car without brakes - would you like to drive it? If
> this is Need For Speed game and you have unlimited lives (in-memory cache
> with backing store), then yes. If this is a real life with (persistence) -
> then no.
>
> On Fri, Feb 16, 2018 at 5:20 PM, Dmitriy Setrakyan <ds...@apache.org>
> wrote:
>
> > Well, I cannot say that I like the name LOG_ONLY, but I would vote to
> keep
> > it for now, given that it is already documented in many places, blogs,
> and
> > examples.
> >
> > D.
> >
> > On Fri, Feb 16, 2018 at 8:13 AM, Ivan Rakov <iv...@gmail.com>
> wrote:
> >
> > > Looks like it's an Ignite term - I've never heard of it outside Ignite
> > > scope.
> > >
> > > Though, renaming existing enum value requires keeping old as
> deprecated.
> > > DEFAULT is confusing enough to pay this price.
> > > As for LOG_ONLY, I think we can keep it as long as it has good and
> > > definitive javadoc.
> > >
> > > Best Regards,
> > > Ivan Rakov
> > >
> > >
> > > On 16.02.2018 17:07, Dmitriy Setrakyan wrote:
> > >
> > >> Igniters, just to clarify, does the term LOG_ONLY mean anything in the
> > >> industry or is this just an Ignite term?
> > >>
> > >> D.
> > >>
> > >> On Fri, Feb 16, 2018 at 8:03 AM, Anton Vinogradov <
> > >> avinogradov@gridgain.com>
> > >> wrote:
> > >>
> > >> Log only mode: flushes application buffers.
> > >>> So, in synced mode without fsync guarantee. That's why I propose to
> > >>> rename
> > >>> it as SYNC.
> > >>>
> > >>> On Fri, Feb 16, 2018 at 4:49 PM, Ilya Lantukh <ilantukh@gridgain.com
> >
> > >>> wrote:
> > >>>
> > >>> I am OK with either FSYNC or STRICT variant.
> > >>>>
> > >>>> LOG_ONLY name means "log without fsync".
> > >>>>
> > >>>> On Fri, Feb 16, 2018 at 4:05 PM, Dmitriy Setrakyan <
> > >>>>
> > >>> dsetrakyan@apache.org>
> > >>>
> > >>>> wrote:
> > >>>>
> > >>>> On Fri, Feb 16, 2018 at 7:02 AM, Ivan Rakov <iv...@gmail.com>
> > >>>>>
> > >>>> wrote:
> > >>>>
> > >>>>> Why create a new term to define something that has already been
> > >>>>>>
> > >>>>> defined?
> > >>>>
> > >>>>> That makes sense. I'm ok with FSYNC.
> > >>>>>> Anton, I don't understand why we should rename LOG_ONLY to SYNC.
> We
> > >>>>>> started this discussion with bad naming of DEFAULT, but this has
> > >>>>>>
> > >>>>> nothing
> > >>>>
> > >>>>> to
> > >>>>>
> > >>>>>> do with LOG_ONLY (even though it may be scientific - but SYNC
> sounds
> > >>>>>> scientific as well).
> > >>>>>>
> > >>>>>> I agree with Ivan, we should not go wild with renaming. However, I
> > >>>>>
> > >>>> would
> > >>>
> > >>>> like to find out what is the meaning behind the LOG_ONLY name. Can
> > >>>>>
> > >>>> someone
> > >>>>
> > >>>>> explain?
> > >>>>>
> > >>>>> D.
> > >>>>>
> > >>>>>
> > >>>>
> > >>>> --
> > >>>> Best regards,
> > >>>> Ilya
> > >>>>
> > >>>>
> > >
> >
>
Re: Apache Ignite 2.4 release
Posted by Vladimir Ozerov <vo...@gridgain.com>.
Igniters,
Sorry for pouring oil on the flames, but from database perspective moving
from FSYNC to non-FSYNC mode appears to be a mistake. When you work with
database, your main expectation is that it will save your data. All
production database vendor make sure that you are safe, not that you are
fast. Moreover, some vendors even prevent you from being in unsafe mode
(e.g. you cannot disable fsync in SQL Server at all).
If we continue going in this direction, we will end up with a product,
which is unsafe out of the box and require tons of documentation to
understand how to make it safe. Definitely not the right message to the
market. This is like a car without brakes - would you like to drive it? If
this is Need For Speed game and you have unlimited lives (in-memory cache
with backing store), then yes. If this is a real life with (persistence) -
then no.
On Fri, Feb 16, 2018 at 5:20 PM, Dmitriy Setrakyan <ds...@apache.org>
wrote:
> Well, I cannot say that I like the name LOG_ONLY, but I would vote to keep
> it for now, given that it is already documented in many places, blogs, and
> examples.
>
> D.
>
> On Fri, Feb 16, 2018 at 8:13 AM, Ivan Rakov <iv...@gmail.com> wrote:
>
> > Looks like it's an Ignite term - I've never heard of it outside Ignite
> > scope.
> >
> > Though, renaming existing enum value requires keeping old as deprecated.
> > DEFAULT is confusing enough to pay this price.
> > As for LOG_ONLY, I think we can keep it as long as it has good and
> > definitive javadoc.
> >
> > Best Regards,
> > Ivan Rakov
> >
> >
> > On 16.02.2018 17:07, Dmitriy Setrakyan wrote:
> >
> >> Igniters, just to clarify, does the term LOG_ONLY mean anything in the
> >> industry or is this just an Ignite term?
> >>
> >> D.
> >>
> >> On Fri, Feb 16, 2018 at 8:03 AM, Anton Vinogradov <
> >> avinogradov@gridgain.com>
> >> wrote:
> >>
> >> Log only mode: flushes application buffers.
> >>> So, in synced mode without fsync guarantee. That's why I propose to
> >>> rename
> >>> it as SYNC.
> >>>
> >>> On Fri, Feb 16, 2018 at 4:49 PM, Ilya Lantukh <il...@gridgain.com>
> >>> wrote:
> >>>
> >>> I am OK with either FSYNC or STRICT variant.
> >>>>
> >>>> LOG_ONLY name means "log without fsync".
> >>>>
> >>>> On Fri, Feb 16, 2018 at 4:05 PM, Dmitriy Setrakyan <
> >>>>
> >>> dsetrakyan@apache.org>
> >>>
> >>>> wrote:
> >>>>
> >>>> On Fri, Feb 16, 2018 at 7:02 AM, Ivan Rakov <iv...@gmail.com>
> >>>>>
> >>>> wrote:
> >>>>
> >>>>> Why create a new term to define something that has already been
> >>>>>>
> >>>>> defined?
> >>>>
> >>>>> That makes sense. I'm ok with FSYNC.
> >>>>>> Anton, I don't understand why we should rename LOG_ONLY to SYNC. We
> >>>>>> started this discussion with bad naming of DEFAULT, but this has
> >>>>>>
> >>>>> nothing
> >>>>
> >>>>> to
> >>>>>
> >>>>>> do with LOG_ONLY (even though it may be scientific - but SYNC sounds
> >>>>>> scientific as well).
> >>>>>>
> >>>>>> I agree with Ivan, we should not go wild with renaming. However, I
> >>>>>
> >>>> would
> >>>
> >>>> like to find out what is the meaning behind the LOG_ONLY name. Can
> >>>>>
> >>>> someone
> >>>>
> >>>>> explain?
> >>>>>
> >>>>> D.
> >>>>>
> >>>>>
> >>>>
> >>>> --
> >>>> Best regards,
> >>>> Ilya
> >>>>
> >>>>
> >
>
Re: Apache Ignite 2.4 release
Posted by Dmitriy Setrakyan <ds...@apache.org>.
Well, I cannot say that I like the name LOG_ONLY, but I would vote to keep
it for now, given that it is already documented in many places, blogs, and
examples.
D.
On Fri, Feb 16, 2018 at 8:13 AM, Ivan Rakov <iv...@gmail.com> wrote:
> Looks like it's an Ignite term - I've never heard of it outside Ignite
> scope.
>
> Though, renaming existing enum value requires keeping old as deprecated.
> DEFAULT is confusing enough to pay this price.
> As for LOG_ONLY, I think we can keep it as long as it has good and
> definitive javadoc.
>
> Best Regards,
> Ivan Rakov
>
>
> On 16.02.2018 17:07, Dmitriy Setrakyan wrote:
>
>> Igniters, just to clarify, does the term LOG_ONLY mean anything in the
>> industry or is this just an Ignite term?
>>
>> D.
>>
>> On Fri, Feb 16, 2018 at 8:03 AM, Anton Vinogradov <
>> avinogradov@gridgain.com>
>> wrote:
>>
>> Log only mode: flushes application buffers.
>>> So, in synced mode without fsync guarantee. That's why I propose to
>>> rename
>>> it as SYNC.
>>>
>>> On Fri, Feb 16, 2018 at 4:49 PM, Ilya Lantukh <il...@gridgain.com>
>>> wrote:
>>>
>>> I am OK with either FSYNC or STRICT variant.
>>>>
>>>> LOG_ONLY name means "log without fsync".
>>>>
>>>> On Fri, Feb 16, 2018 at 4:05 PM, Dmitriy Setrakyan <
>>>>
>>> dsetrakyan@apache.org>
>>>
>>>> wrote:
>>>>
>>>> On Fri, Feb 16, 2018 at 7:02 AM, Ivan Rakov <iv...@gmail.com>
>>>>>
>>>> wrote:
>>>>
>>>>> Why create a new term to define something that has already been
>>>>>>
>>>>> defined?
>>>>
>>>>> That makes sense. I'm ok with FSYNC.
>>>>>> Anton, I don't understand why we should rename LOG_ONLY to SYNC. We
>>>>>> started this discussion with bad naming of DEFAULT, but this has
>>>>>>
>>>>> nothing
>>>>
>>>>> to
>>>>>
>>>>>> do with LOG_ONLY (even though it may be scientific - but SYNC sounds
>>>>>> scientific as well).
>>>>>>
>>>>>> I agree with Ivan, we should not go wild with renaming. However, I
>>>>>
>>>> would
>>>
>>>> like to find out what is the meaning behind the LOG_ONLY name. Can
>>>>>
>>>> someone
>>>>
>>>>> explain?
>>>>>
>>>>> D.
>>>>>
>>>>>
>>>>
>>>> --
>>>> Best regards,
>>>> Ilya
>>>>
>>>>
>
Re: Apache Ignite 2.4 release
Posted by Ivan Rakov <iv...@gmail.com>.
Looks like it's an Ignite term - I've never heard of it outside Ignite
scope.
Though, renaming existing enum value requires keeping old as deprecated.
DEFAULT is confusing enough to pay this price.
As for LOG_ONLY, I think we can keep it as long as it has good and
definitive javadoc.
Best Regards,
Ivan Rakov
On 16.02.2018 17:07, Dmitriy Setrakyan wrote:
> Igniters, just to clarify, does the term LOG_ONLY mean anything in the
> industry or is this just an Ignite term?
>
> D.
>
> On Fri, Feb 16, 2018 at 8:03 AM, Anton Vinogradov <av...@gridgain.com>
> wrote:
>
>> Log only mode: flushes application buffers.
>> So, in synced mode without fsync guarantee. That's why I propose to rename
>> it as SYNC.
>>
>> On Fri, Feb 16, 2018 at 4:49 PM, Ilya Lantukh <il...@gridgain.com>
>> wrote:
>>
>>> I am OK with either FSYNC or STRICT variant.
>>>
>>> LOG_ONLY name means "log without fsync".
>>>
>>> On Fri, Feb 16, 2018 at 4:05 PM, Dmitriy Setrakyan <
>> dsetrakyan@apache.org>
>>> wrote:
>>>
>>>> On Fri, Feb 16, 2018 at 7:02 AM, Ivan Rakov <iv...@gmail.com>
>>> wrote:
>>>>> Why create a new term to define something that has already been
>>> defined?
>>>>> That makes sense. I'm ok with FSYNC.
>>>>> Anton, I don't understand why we should rename LOG_ONLY to SYNC. We
>>>>> started this discussion with bad naming of DEFAULT, but this has
>>> nothing
>>>> to
>>>>> do with LOG_ONLY (even though it may be scientific - but SYNC sounds
>>>>> scientific as well).
>>>>>
>>>> I agree with Ivan, we should not go wild with renaming. However, I
>> would
>>>> like to find out what is the meaning behind the LOG_ONLY name. Can
>>> someone
>>>> explain?
>>>>
>>>> D.
>>>>
>>>
>>>
>>> --
>>> Best regards,
>>> Ilya
>>>
Re: Apache Ignite 2.4 release
Posted by Dmitriy Setrakyan <ds...@apache.org>.
Igniters, just to clarify, does the term LOG_ONLY mean anything in the
industry or is this just an Ignite term?
D.
On Fri, Feb 16, 2018 at 8:03 AM, Anton Vinogradov <av...@gridgain.com>
wrote:
> Log only mode: flushes application buffers.
> So, in synced mode without fsync guarantee. That's why I propose to rename
> it as SYNC.
>
> On Fri, Feb 16, 2018 at 4:49 PM, Ilya Lantukh <il...@gridgain.com>
> wrote:
>
> > I am OK with either FSYNC or STRICT variant.
> >
> > LOG_ONLY name means "log without fsync".
> >
> > On Fri, Feb 16, 2018 at 4:05 PM, Dmitriy Setrakyan <
> dsetrakyan@apache.org>
> > wrote:
> >
> > > On Fri, Feb 16, 2018 at 7:02 AM, Ivan Rakov <iv...@gmail.com>
> > wrote:
> > >
> > > > Why create a new term to define something that has already been
> > defined?
> > > >>
> > > > That makes sense. I'm ok with FSYNC.
> > > > Anton, I don't understand why we should rename LOG_ONLY to SYNC. We
> > > > started this discussion with bad naming of DEFAULT, but this has
> > nothing
> > > to
> > > > do with LOG_ONLY (even though it may be scientific - but SYNC sounds
> > > > scientific as well).
> > > >
> > >
> > > I agree with Ivan, we should not go wild with renaming. However, I
> would
> > > like to find out what is the meaning behind the LOG_ONLY name. Can
> > someone
> > > explain?
> > >
> > > D.
> > >
> >
> >
> >
> > --
> > Best regards,
> > Ilya
> >
>
Re: Apache Ignite 2.4 release
Posted by Anton Vinogradov <av...@gridgain.com>.
Log only mode: flushes application buffers.
So, in synced mode without fsync guarantee. That's why I propose to rename
it as SYNC.
On Fri, Feb 16, 2018 at 4:49 PM, Ilya Lantukh <il...@gridgain.com> wrote:
> I am OK with either FSYNC or STRICT variant.
>
> LOG_ONLY name means "log without fsync".
>
> On Fri, Feb 16, 2018 at 4:05 PM, Dmitriy Setrakyan <ds...@apache.org>
> wrote:
>
> > On Fri, Feb 16, 2018 at 7:02 AM, Ivan Rakov <iv...@gmail.com>
> wrote:
> >
> > > Why create a new term to define something that has already been
> defined?
> > >>
> > > That makes sense. I'm ok with FSYNC.
> > > Anton, I don't understand why we should rename LOG_ONLY to SYNC. We
> > > started this discussion with bad naming of DEFAULT, but this has
> nothing
> > to
> > > do with LOG_ONLY (even though it may be scientific - but SYNC sounds
> > > scientific as well).
> > >
> >
> > I agree with Ivan, we should not go wild with renaming. However, I would
> > like to find out what is the meaning behind the LOG_ONLY name. Can
> someone
> > explain?
> >
> > D.
> >
>
>
>
> --
> Best regards,
> Ilya
>
Re: Apache Ignite 2.4 release
Posted by Ilya Lantukh <il...@gridgain.com>.
I am OK with either FSYNC or STRICT variant.
LOG_ONLY name means "log without fsync".
On Fri, Feb 16, 2018 at 4:05 PM, Dmitriy Setrakyan <ds...@apache.org>
wrote:
> On Fri, Feb 16, 2018 at 7:02 AM, Ivan Rakov <iv...@gmail.com> wrote:
>
> > Why create a new term to define something that has already been defined?
> >>
> > That makes sense. I'm ok with FSYNC.
> > Anton, I don't understand why we should rename LOG_ONLY to SYNC. We
> > started this discussion with bad naming of DEFAULT, but this has nothing
> to
> > do with LOG_ONLY (even though it may be scientific - but SYNC sounds
> > scientific as well).
> >
>
> I agree with Ivan, we should not go wild with renaming. However, I would
> like to find out what is the meaning behind the LOG_ONLY name. Can someone
> explain?
>
> D.
>
--
Best regards,
Ilya
Re: Apache Ignite 2.4 release
Posted by Dmitriy Setrakyan <ds...@apache.org>.
On Fri, Feb 16, 2018 at 7:02 AM, Ivan Rakov <iv...@gmail.com> wrote:
> Why create a new term to define something that has already been defined?
>>
> That makes sense. I'm ok with FSYNC.
> Anton, I don't understand why we should rename LOG_ONLY to SYNC. We
> started this discussion with bad naming of DEFAULT, but this has nothing to
> do with LOG_ONLY (even though it may be scientific - but SYNC sounds
> scientific as well).
>
I agree with Ivan, we should not go wild with renaming. However, I would
like to find out what is the meaning behind the LOG_ONLY name. Can someone
explain?
D.
Re: Apache Ignite 2.4 release
Posted by Ivan Rakov <iv...@gmail.com>.
> Why create a new term to define something that has already been defined?
That makes sense. I'm ok with FSYNC.
Anton, I don't understand why we should rename LOG_ONLY to SYNC. We started this discussion with bad naming of DEFAULT, but this has nothing to do with LOG_ONLY (even though it may be scientific - but SYNC sounds scientific as well).
Best Regards,
Ivan Rakov
On 16.02.2018 15:55, Anton Vinogradov wrote:
> >> I had idea to name old default as FSYNC, but it would be too scientific.
> So, then it can be FSYNC, SYNC, BACKGROUND and NONE!
>
> On Fri, Feb 16, 2018 at 3:49 PM, Dmitriy Setrakyan <ds...@apache.org>
> wrote:
>
>> On Fri, Feb 16, 2018 at 6:26 AM, Dmitry Pavlov <dp...@gmail.com>
>> wrote:
>>
>>> I had idea to name old default as FSYNC, but it would be too scientific.
>>>
>> I like FSYNC, I do not think it is too scientific. Definitely not more
>> scientific than LOG_ONLY.
>>
>> For old DEFAULT, STRICT or STRICT_SYNC - IMO are best options, so I agree
>>> with Ivan.
>>>
>> Not sure I like STRICT. In Linux world, fsync is a well known command which
>> does exactly what our FSYNC mode will do:
>> http://man7.org/linux/man-pages/man2/fsync.2.html . STRICT, on the other
>> hand, is not a commonly understood term. Why create a new term to define
>> something that has already been defined?
>>
>> Also, what if tomorrow we need to add an even stricter parameter? Then we
>> are back to the same problem we are trying to fix right now.
>>
>> D.
>>
Re: Apache Ignite 2.4 release
Posted by Anton Vinogradov <av...@gridgain.com>.
>> I had idea to name old default as FSYNC, but it would be too scientific.
So, then it can be FSYNC, SYNC, BACKGROUND and NONE!
On Fri, Feb 16, 2018 at 3:49 PM, Dmitriy Setrakyan <ds...@apache.org>
wrote:
> On Fri, Feb 16, 2018 at 6:26 AM, Dmitry Pavlov <dp...@gmail.com>
> wrote:
>
> > I had idea to name old default as FSYNC, but it would be too scientific.
> >
>
> I like FSYNC, I do not think it is too scientific. Definitely not more
> scientific than LOG_ONLY.
>
> For old DEFAULT, STRICT or STRICT_SYNC - IMO are best options, so I agree
> > with Ivan.
> >
>
> Not sure I like STRICT. In Linux world, fsync is a well known command which
> does exactly what our FSYNC mode will do:
> http://man7.org/linux/man-pages/man2/fsync.2.html . STRICT, on the other
> hand, is not a commonly understood term. Why create a new term to define
> something that has already been defined?
>
> Also, what if tomorrow we need to add an even stricter parameter? Then we
> are back to the same problem we are trying to fix right now.
>
> D.
>
Re: Apache Ignite 2.4 release
Posted by Dmitriy Setrakyan <ds...@apache.org>.
On Fri, Feb 16, 2018 at 6:26 AM, Dmitry Pavlov <dp...@gmail.com>
wrote:
> I had idea to name old default as FSYNC, but it would be too scientific.
>
I like FSYNC, I do not think it is too scientific. Definitely not more
scientific than LOG_ONLY.
For old DEFAULT, STRICT or STRICT_SYNC - IMO are best options, so I agree
> with Ivan.
>
Not sure I like STRICT. In Linux world, fsync is a well known command which
does exactly what our FSYNC mode will do:
http://man7.org/linux/man-pages/man2/fsync.2.html . STRICT, on the other
hand, is not a commonly understood term. Why create a new term to define
something that has already been defined?
Also, what if tomorrow we need to add an even stricter parameter? Then we
are back to the same problem we are trying to fix right now.
D.
Re: Apache Ignite 2.4 release
Posted by Dmitry Pavlov <dp...@gmail.com>.
I had idea to name old default as FSYNC, but it would be too scientific.
For old DEFAULT, STRICT or STRICT_SYNC - IMO are best options, so I agree
with Ivan.
пт, 16 февр. 2018 г. в 15:21, Anton Vinogradov <av...@gridgain.com>:
> typo
> NODE -> NONE
>
> On Fri, Feb 16, 2018 at 3:21 PM, Anton Vinogradov <
> avinogradov@gridgain.com>
> wrote:
>
> > What about
> > FULL_SYNC
> > SYNC -> default
> > BACKGROUND
> > NODE
> > ?
> >
> > On Fri, Feb 16, 2018 at 3:09 PM, Ivan Rakov <iv...@gmail.com>
> wrote:
> >
> >> From my point of view, STRICT is the best option. The name signalizes to
> >> user that this mode provides optional strict guarantees.
> >> FULL_SYNC can be messed with CacheWriteSynchronizationMode#FULL_SYNC. I
> >> don't like the idea of naming different things with same names.
> >>
> >> Best Regards,
> >> Ivan Rakov
> >>
> >>
> >> On 16.02.2018 15:01, Dmitriy Setrakyan wrote:
> >>
> >>> BTW, Ilya, why not name the enum value FULL_SYNC instead of STRICT?
> >>>
> >>> On Fri, Feb 16, 2018 at 5:43 AM, Dmitriy Setrakyan <
> >>> dsetrakyan@apache.org>
> >>> wrote:
> >>>
> >>> Naming one of the enum constants DEFAULT was a huge mistake. Not sure
> how
> >>>> it passed a code review, but let us all be more careful going forward.
> >>>>
> >>>> I agree with Ilya. The only remedy right now is to deprecate the
> DEFAULT
> >>>> constant.
> >>>>
> >>>> D.
> >>>>
> >>>> On Fri, Feb 16, 2018 at 5:37 AM, Ilya Lantukh <il...@gridgain.com>
> >>>> wrote:
> >>>>
> >>>> Hi all,
> >>>>>
> >>>>> I'd like to suggest to change default WALMode. Currently we have:
> >>>>> DEFAULT (write and fsync),
> >>>>> LOG_ONLY (write without fsync),
> >>>>> BACKGROUND,
> >>>>> NONE.
> >>>>>
> >>>>> It turns out that fsyncs in current DEFAULT mode significantly
> >>>>> restricts
> >>>>> Ignite performance. Compared to LOG_ONLY, it offers additional
> >>>>> guarantees
> >>>>> that data won't be lost in case of OS or hardware failure, but such
> >>>>> guarantees aren't needed very often, and tradeoff is too big.
> >>>>>
> >>>>> I suggest to rename current DEFAULT to STRICT and make LOG_ONLY new
> >>>>> default
> >>>>> mode. We can leave DEFAULT as @Deprecated and treat it as STRICT, so
> >>>>> that
> >>>>> users with old configs will have the same behaviour.
> >>>>>
> >>>>> What do you think?
> >>>>>
> >>>>> On Fri, Feb 16, 2018 at 12:35 AM, Denis Magda <dm...@apache.org>
> >>>>> wrote:
> >>>>>
> >>>>> Vladimir,
> >>>>>>
> >>>>>> I would suggest not to do this because we still need to spend time
> on
> >>>>>> testing, documentation, etc. If someone shows interest in this
> >>>>>> features
> >>>>>> they can assemble binaries from the master.
> >>>>>>
> >>>>>> --
> >>>>>> Denis
> >>>>>>
> >>>>>> On Thu, Feb 15, 2018 at 6:43 AM, Nikolay Izhikov <
> nizhikov@apache.org
> >>>>>> >
> >>>>>> wrote:
> >>>>>>
> >>>>>> +1
> >>>>>>>
> >>>>>>> В Чт, 15/02/2018 в 17:27 +0300, Vladimir Ozerov пишет:
> >>>>>>>
> >>>>>>>> Igniters,
> >>>>>>>>
> >>>>>>>> AI 2.4 release was shifted a bit and over this time we implemented
> >>>>>>>>
> >>>>>>> two
> >>>>>
> >>>>>> important SQL features:
> >>>>>>>> 1) COPY command for fast file upload to the cluster [1]
> >>>>>>>> 2) Streaming mode for thin driver [2]
> >>>>>>>>
> >>>>>>>> Both commands are very important for fast data ingestion into
> Ignite
> >>>>>>>> through SQL. I would like to ask community to consider to include
> >>>>>>>>
> >>>>>>> these
> >>>>>
> >>>>>> two
> >>>>>>>
> >>>>>>>> features into AI 2.4 in *experimental* state because both of them
> >>>>>>>>
> >>>>>>> will
> >>>>>
> >>>>>> be
> >>>>>>
> >>>>>>> improved in various ways in the nearest time. If we do so, we will
> >>>>>>>>
> >>>>>>> be
> >>>>>
> >>>>>> able
> >>>>>>>
> >>>>>>>> to collect some feedback from the users before AI 2.5 release.
> What
> >>>>>>>>
> >>>>>>> do
> >>>>>
> >>>>>> you
> >>>>>>>
> >>>>>>>> think?
> >>>>>>>>
> >>>>>>>> Vladimir.
> >>>>>>>>
> >>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-6917
> >>>>>>>> [2] https://issues.apache.org/jira/browse/IGNITE-7253
> >>>>>>>>
> >>>>>>>> On Tue, Feb 13, 2018 at 1:22 AM, Dmitriy Setrakyan <
> >>>>>>>>
> >>>>>>> dsetrakyan@apache.org>
> >>>>>>>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>> On Mon, Feb 12, 2018 at 9:22 AM, Dmitry Pavlov <
> >>>>>>>>>
> >>>>>>>> dpavlov.spb@gmail.com>
> >>>>>>
> >>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>> Hi,
> >>>>>>>>>>
> >>>>>>>>>> Unfortunately, a quick fix did not give us too much performance
> >>>>>>>>>>
> >>>>>>>>> boost.
> >>>>>>>
> >>>>>>>> I'm going to implement a complete algorithm change for storing
> >>>>>>>>>>
> >>>>>>>>> the
> >>>>>
> >>>>>> page
> >>>>>>>
> >>>>>>>> identifier. But this change is quite significant and will
> >>>>>>>>>>
> >>>>>>>>> require
> >>>>>
> >>>>>> re-testing. I suggest including
> >>>>>>>>>> https://issues.apache.org/jira/browse/IGNITE-7638 in the next
> >>>>>>>>>>
> >>>>>>>>> version,
> >>>>>>>
> >>>>>>>> for
> >>>>>>>>>
> >>>>>>>>>> example, to 2.5.
> >>>>>>>>>>
> >>>>>>>>>> Sincerely,
> >>>>>>>>>> Dmitriy Pavlov
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Dmitriy, thanks for the update! Are there other tickets that are
> >>>>>>>>>
> >>>>>>>> holding
> >>>>>>>
> >>>>>>>> the release at this point? I remember that there was a performance
> >>>>>>>>> degradation issue in FULL_SYNC mode, but I cannot find a ticket.
> >>>>>>>>>
> >>>>>>>>> D.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>
> >>>>> --
> >>>>> Best regards,
> >>>>> Ilya
> >>>>>
> >>>>>
> >>>>
> >>
> >
>
Re: Apache Ignite 2.4 release
Posted by Anton Vinogradov <av...@gridgain.com>.
typo
NODE -> NONE
On Fri, Feb 16, 2018 at 3:21 PM, Anton Vinogradov <av...@gridgain.com>
wrote:
> What about
> FULL_SYNC
> SYNC -> default
> BACKGROUND
> NODE
> ?
>
> On Fri, Feb 16, 2018 at 3:09 PM, Ivan Rakov <iv...@gmail.com> wrote:
>
>> From my point of view, STRICT is the best option. The name signalizes to
>> user that this mode provides optional strict guarantees.
>> FULL_SYNC can be messed with CacheWriteSynchronizationMode#FULL_SYNC. I
>> don't like the idea of naming different things with same names.
>>
>> Best Regards,
>> Ivan Rakov
>>
>>
>> On 16.02.2018 15:01, Dmitriy Setrakyan wrote:
>>
>>> BTW, Ilya, why not name the enum value FULL_SYNC instead of STRICT?
>>>
>>> On Fri, Feb 16, 2018 at 5:43 AM, Dmitriy Setrakyan <
>>> dsetrakyan@apache.org>
>>> wrote:
>>>
>>> Naming one of the enum constants DEFAULT was a huge mistake. Not sure how
>>>> it passed a code review, but let us all be more careful going forward.
>>>>
>>>> I agree with Ilya. The only remedy right now is to deprecate the DEFAULT
>>>> constant.
>>>>
>>>> D.
>>>>
>>>> On Fri, Feb 16, 2018 at 5:37 AM, Ilya Lantukh <il...@gridgain.com>
>>>> wrote:
>>>>
>>>> Hi all,
>>>>>
>>>>> I'd like to suggest to change default WALMode. Currently we have:
>>>>> DEFAULT (write and fsync),
>>>>> LOG_ONLY (write without fsync),
>>>>> BACKGROUND,
>>>>> NONE.
>>>>>
>>>>> It turns out that fsyncs in current DEFAULT mode significantly
>>>>> restricts
>>>>> Ignite performance. Compared to LOG_ONLY, it offers additional
>>>>> guarantees
>>>>> that data won't be lost in case of OS or hardware failure, but such
>>>>> guarantees aren't needed very often, and tradeoff is too big.
>>>>>
>>>>> I suggest to rename current DEFAULT to STRICT and make LOG_ONLY new
>>>>> default
>>>>> mode. We can leave DEFAULT as @Deprecated and treat it as STRICT, so
>>>>> that
>>>>> users with old configs will have the same behaviour.
>>>>>
>>>>> What do you think?
>>>>>
>>>>> On Fri, Feb 16, 2018 at 12:35 AM, Denis Magda <dm...@apache.org>
>>>>> wrote:
>>>>>
>>>>> Vladimir,
>>>>>>
>>>>>> I would suggest not to do this because we still need to spend time on
>>>>>> testing, documentation, etc. If someone shows interest in this
>>>>>> features
>>>>>> they can assemble binaries from the master.
>>>>>>
>>>>>> --
>>>>>> Denis
>>>>>>
>>>>>> On Thu, Feb 15, 2018 at 6:43 AM, Nikolay Izhikov <nizhikov@apache.org
>>>>>> >
>>>>>> wrote:
>>>>>>
>>>>>> +1
>>>>>>>
>>>>>>> В Чт, 15/02/2018 в 17:27 +0300, Vladimir Ozerov пишет:
>>>>>>>
>>>>>>>> Igniters,
>>>>>>>>
>>>>>>>> AI 2.4 release was shifted a bit and over this time we implemented
>>>>>>>>
>>>>>>> two
>>>>>
>>>>>> important SQL features:
>>>>>>>> 1) COPY command for fast file upload to the cluster [1]
>>>>>>>> 2) Streaming mode for thin driver [2]
>>>>>>>>
>>>>>>>> Both commands are very important for fast data ingestion into Ignite
>>>>>>>> through SQL. I would like to ask community to consider to include
>>>>>>>>
>>>>>>> these
>>>>>
>>>>>> two
>>>>>>>
>>>>>>>> features into AI 2.4 in *experimental* state because both of them
>>>>>>>>
>>>>>>> will
>>>>>
>>>>>> be
>>>>>>
>>>>>>> improved in various ways in the nearest time. If we do so, we will
>>>>>>>>
>>>>>>> be
>>>>>
>>>>>> able
>>>>>>>
>>>>>>>> to collect some feedback from the users before AI 2.5 release. What
>>>>>>>>
>>>>>>> do
>>>>>
>>>>>> you
>>>>>>>
>>>>>>>> think?
>>>>>>>>
>>>>>>>> Vladimir.
>>>>>>>>
>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-6917
>>>>>>>> [2] https://issues.apache.org/jira/browse/IGNITE-7253
>>>>>>>>
>>>>>>>> On Tue, Feb 13, 2018 at 1:22 AM, Dmitriy Setrakyan <
>>>>>>>>
>>>>>>> dsetrakyan@apache.org>
>>>>>>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> On Mon, Feb 12, 2018 at 9:22 AM, Dmitry Pavlov <
>>>>>>>>>
>>>>>>>> dpavlov.spb@gmail.com>
>>>>>>
>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> Unfortunately, a quick fix did not give us too much performance
>>>>>>>>>>
>>>>>>>>> boost.
>>>>>>>
>>>>>>>> I'm going to implement a complete algorithm change for storing
>>>>>>>>>>
>>>>>>>>> the
>>>>>
>>>>>> page
>>>>>>>
>>>>>>>> identifier. But this change is quite significant and will
>>>>>>>>>>
>>>>>>>>> require
>>>>>
>>>>>> re-testing. I suggest including
>>>>>>>>>> https://issues.apache.org/jira/browse/IGNITE-7638 in the next
>>>>>>>>>>
>>>>>>>>> version,
>>>>>>>
>>>>>>>> for
>>>>>>>>>
>>>>>>>>>> example, to 2.5.
>>>>>>>>>>
>>>>>>>>>> Sincerely,
>>>>>>>>>> Dmitriy Pavlov
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Dmitriy, thanks for the update! Are there other tickets that are
>>>>>>>>>
>>>>>>>> holding
>>>>>>>
>>>>>>>> the release at this point? I remember that there was a performance
>>>>>>>>> degradation issue in FULL_SYNC mode, but I cannot find a ticket.
>>>>>>>>>
>>>>>>>>> D.
>>>>>>>>>
>>>>>>>>>
>>>>>
>>>>> --
>>>>> Best regards,
>>>>> Ilya
>>>>>
>>>>>
>>>>
>>
>
Re: Apache Ignite 2.4 release
Posted by Anton Vinogradov <av...@gridgain.com>.
What about
FULL_SYNC
SYNC -> default
BACKGROUND
NODE
?
On Fri, Feb 16, 2018 at 3:09 PM, Ivan Rakov <iv...@gmail.com> wrote:
> From my point of view, STRICT is the best option. The name signalizes to
> user that this mode provides optional strict guarantees.
> FULL_SYNC can be messed with CacheWriteSynchronizationMode#FULL_SYNC. I
> don't like the idea of naming different things with same names.
>
> Best Regards,
> Ivan Rakov
>
>
> On 16.02.2018 15:01, Dmitriy Setrakyan wrote:
>
>> BTW, Ilya, why not name the enum value FULL_SYNC instead of STRICT?
>>
>> On Fri, Feb 16, 2018 at 5:43 AM, Dmitriy Setrakyan <dsetrakyan@apache.org
>> >
>> wrote:
>>
>> Naming one of the enum constants DEFAULT was a huge mistake. Not sure how
>>> it passed a code review, but let us all be more careful going forward.
>>>
>>> I agree with Ilya. The only remedy right now is to deprecate the DEFAULT
>>> constant.
>>>
>>> D.
>>>
>>> On Fri, Feb 16, 2018 at 5:37 AM, Ilya Lantukh <il...@gridgain.com>
>>> wrote:
>>>
>>> Hi all,
>>>>
>>>> I'd like to suggest to change default WALMode. Currently we have:
>>>> DEFAULT (write and fsync),
>>>> LOG_ONLY (write without fsync),
>>>> BACKGROUND,
>>>> NONE.
>>>>
>>>> It turns out that fsyncs in current DEFAULT mode significantly restricts
>>>> Ignite performance. Compared to LOG_ONLY, it offers additional
>>>> guarantees
>>>> that data won't be lost in case of OS or hardware failure, but such
>>>> guarantees aren't needed very often, and tradeoff is too big.
>>>>
>>>> I suggest to rename current DEFAULT to STRICT and make LOG_ONLY new
>>>> default
>>>> mode. We can leave DEFAULT as @Deprecated and treat it as STRICT, so
>>>> that
>>>> users with old configs will have the same behaviour.
>>>>
>>>> What do you think?
>>>>
>>>> On Fri, Feb 16, 2018 at 12:35 AM, Denis Magda <dm...@apache.org>
>>>> wrote:
>>>>
>>>> Vladimir,
>>>>>
>>>>> I would suggest not to do this because we still need to spend time on
>>>>> testing, documentation, etc. If someone shows interest in this features
>>>>> they can assemble binaries from the master.
>>>>>
>>>>> --
>>>>> Denis
>>>>>
>>>>> On Thu, Feb 15, 2018 at 6:43 AM, Nikolay Izhikov <ni...@apache.org>
>>>>> wrote:
>>>>>
>>>>> +1
>>>>>>
>>>>>> В Чт, 15/02/2018 в 17:27 +0300, Vladimir Ozerov пишет:
>>>>>>
>>>>>>> Igniters,
>>>>>>>
>>>>>>> AI 2.4 release was shifted a bit and over this time we implemented
>>>>>>>
>>>>>> two
>>>>
>>>>> important SQL features:
>>>>>>> 1) COPY command for fast file upload to the cluster [1]
>>>>>>> 2) Streaming mode for thin driver [2]
>>>>>>>
>>>>>>> Both commands are very important for fast data ingestion into Ignite
>>>>>>> through SQL. I would like to ask community to consider to include
>>>>>>>
>>>>>> these
>>>>
>>>>> two
>>>>>>
>>>>>>> features into AI 2.4 in *experimental* state because both of them
>>>>>>>
>>>>>> will
>>>>
>>>>> be
>>>>>
>>>>>> improved in various ways in the nearest time. If we do so, we will
>>>>>>>
>>>>>> be
>>>>
>>>>> able
>>>>>>
>>>>>>> to collect some feedback from the users before AI 2.5 release. What
>>>>>>>
>>>>>> do
>>>>
>>>>> you
>>>>>>
>>>>>>> think?
>>>>>>>
>>>>>>> Vladimir.
>>>>>>>
>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-6917
>>>>>>> [2] https://issues.apache.org/jira/browse/IGNITE-7253
>>>>>>>
>>>>>>> On Tue, Feb 13, 2018 at 1:22 AM, Dmitriy Setrakyan <
>>>>>>>
>>>>>> dsetrakyan@apache.org>
>>>>>>
>>>>>>> wrote:
>>>>>>>
>>>>>>> On Mon, Feb 12, 2018 at 9:22 AM, Dmitry Pavlov <
>>>>>>>>
>>>>>>> dpavlov.spb@gmail.com>
>>>>>
>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> Unfortunately, a quick fix did not give us too much performance
>>>>>>>>>
>>>>>>>> boost.
>>>>>>
>>>>>>> I'm going to implement a complete algorithm change for storing
>>>>>>>>>
>>>>>>>> the
>>>>
>>>>> page
>>>>>>
>>>>>>> identifier. But this change is quite significant and will
>>>>>>>>>
>>>>>>>> require
>>>>
>>>>> re-testing. I suggest including
>>>>>>>>> https://issues.apache.org/jira/browse/IGNITE-7638 in the next
>>>>>>>>>
>>>>>>>> version,
>>>>>>
>>>>>>> for
>>>>>>>>
>>>>>>>>> example, to 2.5.
>>>>>>>>>
>>>>>>>>> Sincerely,
>>>>>>>>> Dmitriy Pavlov
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Dmitriy, thanks for the update! Are there other tickets that are
>>>>>>>>
>>>>>>> holding
>>>>>>
>>>>>>> the release at this point? I remember that there was a performance
>>>>>>>> degradation issue in FULL_SYNC mode, but I cannot find a ticket.
>>>>>>>>
>>>>>>>> D.
>>>>>>>>
>>>>>>>>
>>>>
>>>> --
>>>> Best regards,
>>>> Ilya
>>>>
>>>>
>>>
>
Re: Apache Ignite 2.4 release
Posted by Ivan Rakov <iv...@gmail.com>.
From my point of view, STRICT is the best option. The name signalizes
to user that this mode provides optional strict guarantees.
FULL_SYNC can be messed with CacheWriteSynchronizationMode#FULL_SYNC. I
don't like the idea of naming different things with same names.
Best Regards,
Ivan Rakov
On 16.02.2018 15:01, Dmitriy Setrakyan wrote:
> BTW, Ilya, why not name the enum value FULL_SYNC instead of STRICT?
>
> On Fri, Feb 16, 2018 at 5:43 AM, Dmitriy Setrakyan <ds...@apache.org>
> wrote:
>
>> Naming one of the enum constants DEFAULT was a huge mistake. Not sure how
>> it passed a code review, but let us all be more careful going forward.
>>
>> I agree with Ilya. The only remedy right now is to deprecate the DEFAULT
>> constant.
>>
>> D.
>>
>> On Fri, Feb 16, 2018 at 5:37 AM, Ilya Lantukh <il...@gridgain.com>
>> wrote:
>>
>>> Hi all,
>>>
>>> I'd like to suggest to change default WALMode. Currently we have:
>>> DEFAULT (write and fsync),
>>> LOG_ONLY (write without fsync),
>>> BACKGROUND,
>>> NONE.
>>>
>>> It turns out that fsyncs in current DEFAULT mode significantly restricts
>>> Ignite performance. Compared to LOG_ONLY, it offers additional guarantees
>>> that data won't be lost in case of OS or hardware failure, but such
>>> guarantees aren't needed very often, and tradeoff is too big.
>>>
>>> I suggest to rename current DEFAULT to STRICT and make LOG_ONLY new
>>> default
>>> mode. We can leave DEFAULT as @Deprecated and treat it as STRICT, so that
>>> users with old configs will have the same behaviour.
>>>
>>> What do you think?
>>>
>>> On Fri, Feb 16, 2018 at 12:35 AM, Denis Magda <dm...@apache.org> wrote:
>>>
>>>> Vladimir,
>>>>
>>>> I would suggest not to do this because we still need to spend time on
>>>> testing, documentation, etc. If someone shows interest in this features
>>>> they can assemble binaries from the master.
>>>>
>>>> --
>>>> Denis
>>>>
>>>> On Thu, Feb 15, 2018 at 6:43 AM, Nikolay Izhikov <ni...@apache.org>
>>>> wrote:
>>>>
>>>>> +1
>>>>>
>>>>> В Чт, 15/02/2018 в 17:27 +0300, Vladimir Ozerov пишет:
>>>>>> Igniters,
>>>>>>
>>>>>> AI 2.4 release was shifted a bit and over this time we implemented
>>> two
>>>>>> important SQL features:
>>>>>> 1) COPY command for fast file upload to the cluster [1]
>>>>>> 2) Streaming mode for thin driver [2]
>>>>>>
>>>>>> Both commands are very important for fast data ingestion into Ignite
>>>>>> through SQL. I would like to ask community to consider to include
>>> these
>>>>> two
>>>>>> features into AI 2.4 in *experimental* state because both of them
>>> will
>>>> be
>>>>>> improved in various ways in the nearest time. If we do so, we will
>>> be
>>>>> able
>>>>>> to collect some feedback from the users before AI 2.5 release. What
>>> do
>>>>> you
>>>>>> think?
>>>>>>
>>>>>> Vladimir.
>>>>>>
>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-6917
>>>>>> [2] https://issues.apache.org/jira/browse/IGNITE-7253
>>>>>>
>>>>>> On Tue, Feb 13, 2018 at 1:22 AM, Dmitriy Setrakyan <
>>>>> dsetrakyan@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> On Mon, Feb 12, 2018 at 9:22 AM, Dmitry Pavlov <
>>>> dpavlov.spb@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> Unfortunately, a quick fix did not give us too much performance
>>>>> boost.
>>>>>>>> I'm going to implement a complete algorithm change for storing
>>> the
>>>>> page
>>>>>>>> identifier. But this change is quite significant and will
>>> require
>>>>>>>> re-testing. I suggest including
>>>>>>>> https://issues.apache.org/jira/browse/IGNITE-7638 in the next
>>>>> version,
>>>>>>> for
>>>>>>>> example, to 2.5.
>>>>>>>>
>>>>>>>> Sincerely,
>>>>>>>> Dmitriy Pavlov
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> Dmitriy, thanks for the update! Are there other tickets that are
>>>>> holding
>>>>>>> the release at this point? I remember that there was a performance
>>>>>>> degradation issue in FULL_SYNC mode, but I cannot find a ticket.
>>>>>>>
>>>>>>> D.
>>>>>>>
>>>
>>>
>>> --
>>> Best regards,
>>> Ilya
>>>
>>
Re: Apache Ignite 2.4 release
Posted by Dmitriy Setrakyan <ds...@apache.org>.
BTW, Ilya, why not name the enum value FULL_SYNC instead of STRICT?
On Fri, Feb 16, 2018 at 5:43 AM, Dmitriy Setrakyan <ds...@apache.org>
wrote:
> Naming one of the enum constants DEFAULT was a huge mistake. Not sure how
> it passed a code review, but let us all be more careful going forward.
>
> I agree with Ilya. The only remedy right now is to deprecate the DEFAULT
> constant.
>
> D.
>
> On Fri, Feb 16, 2018 at 5:37 AM, Ilya Lantukh <il...@gridgain.com>
> wrote:
>
>> Hi all,
>>
>> I'd like to suggest to change default WALMode. Currently we have:
>> DEFAULT (write and fsync),
>> LOG_ONLY (write without fsync),
>> BACKGROUND,
>> NONE.
>>
>> It turns out that fsyncs in current DEFAULT mode significantly restricts
>> Ignite performance. Compared to LOG_ONLY, it offers additional guarantees
>> that data won't be lost in case of OS or hardware failure, but such
>> guarantees aren't needed very often, and tradeoff is too big.
>>
>> I suggest to rename current DEFAULT to STRICT and make LOG_ONLY new
>> default
>> mode. We can leave DEFAULT as @Deprecated and treat it as STRICT, so that
>> users with old configs will have the same behaviour.
>>
>> What do you think?
>>
>> On Fri, Feb 16, 2018 at 12:35 AM, Denis Magda <dm...@apache.org> wrote:
>>
>> > Vladimir,
>> >
>> > I would suggest not to do this because we still need to spend time on
>> > testing, documentation, etc. If someone shows interest in this features
>> > they can assemble binaries from the master.
>> >
>> > --
>> > Denis
>> >
>> > On Thu, Feb 15, 2018 at 6:43 AM, Nikolay Izhikov <ni...@apache.org>
>> > wrote:
>> >
>> > > +1
>> > >
>> > > В Чт, 15/02/2018 в 17:27 +0300, Vladimir Ozerov пишет:
>> > > > Igniters,
>> > > >
>> > > > AI 2.4 release was shifted a bit and over this time we implemented
>> two
>> > > > important SQL features:
>> > > > 1) COPY command for fast file upload to the cluster [1]
>> > > > 2) Streaming mode for thin driver [2]
>> > > >
>> > > > Both commands are very important for fast data ingestion into Ignite
>> > > > through SQL. I would like to ask community to consider to include
>> these
>> > > two
>> > > > features into AI 2.4 in *experimental* state because both of them
>> will
>> > be
>> > > > improved in various ways in the nearest time. If we do so, we will
>> be
>> > > able
>> > > > to collect some feedback from the users before AI 2.5 release. What
>> do
>> > > you
>> > > > think?
>> > > >
>> > > > Vladimir.
>> > > >
>> > > > [1] https://issues.apache.org/jira/browse/IGNITE-6917
>> > > > [2] https://issues.apache.org/jira/browse/IGNITE-7253
>> > > >
>> > > > On Tue, Feb 13, 2018 at 1:22 AM, Dmitriy Setrakyan <
>> > > dsetrakyan@apache.org>
>> > > > wrote:
>> > > >
>> > > > > On Mon, Feb 12, 2018 at 9:22 AM, Dmitry Pavlov <
>> > dpavlov.spb@gmail.com>
>> > > > > wrote:
>> > > > >
>> > > > > > Hi,
>> > > > > >
>> > > > > > Unfortunately, a quick fix did not give us too much performance
>> > > boost.
>> > > > > >
>> > > > > > I'm going to implement a complete algorithm change for storing
>> the
>> > > page
>> > > > > > identifier. But this change is quite significant and will
>> require
>> > > > > > re-testing. I suggest including
>> > > > > > https://issues.apache.org/jira/browse/IGNITE-7638 in the next
>> > > version,
>> > > > >
>> > > > > for
>> > > > > > example, to 2.5.
>> > > > > >
>> > > > > > Sincerely,
>> > > > > > Dmitriy Pavlov
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > > > Dmitriy, thanks for the update! Are there other tickets that are
>> > > holding
>> > > > > the release at this point? I remember that there was a performance
>> > > > > degradation issue in FULL_SYNC mode, but I cannot find a ticket.
>> > > > >
>> > > > > D.
>> > > > >
>> > >
>> >
>>
>>
>>
>> --
>> Best regards,
>> Ilya
>>
>
>
Re: Apache Ignite 2.4 release
Posted by Dmitriy Setrakyan <ds...@apache.org>.
Naming one of the enum constants DEFAULT was a huge mistake. Not sure how
it passed a code review, but let us all be more careful going forward.
I agree with Ilya. The only remedy right now is to deprecate the DEFAULT
constant.
D.
On Fri, Feb 16, 2018 at 5:37 AM, Ilya Lantukh <il...@gridgain.com> wrote:
> Hi all,
>
> I'd like to suggest to change default WALMode. Currently we have:
> DEFAULT (write and fsync),
> LOG_ONLY (write without fsync),
> BACKGROUND,
> NONE.
>
> It turns out that fsyncs in current DEFAULT mode significantly restricts
> Ignite performance. Compared to LOG_ONLY, it offers additional guarantees
> that data won't be lost in case of OS or hardware failure, but such
> guarantees aren't needed very often, and tradeoff is too big.
>
> I suggest to rename current DEFAULT to STRICT and make LOG_ONLY new default
> mode. We can leave DEFAULT as @Deprecated and treat it as STRICT, so that
> users with old configs will have the same behaviour.
>
> What do you think?
>
> On Fri, Feb 16, 2018 at 12:35 AM, Denis Magda <dm...@apache.org> wrote:
>
> > Vladimir,
> >
> > I would suggest not to do this because we still need to spend time on
> > testing, documentation, etc. If someone shows interest in this features
> > they can assemble binaries from the master.
> >
> > --
> > Denis
> >
> > On Thu, Feb 15, 2018 at 6:43 AM, Nikolay Izhikov <ni...@apache.org>
> > wrote:
> >
> > > +1
> > >
> > > В Чт, 15/02/2018 в 17:27 +0300, Vladimir Ozerov пишет:
> > > > Igniters,
> > > >
> > > > AI 2.4 release was shifted a bit and over this time we implemented
> two
> > > > important SQL features:
> > > > 1) COPY command for fast file upload to the cluster [1]
> > > > 2) Streaming mode for thin driver [2]
> > > >
> > > > Both commands are very important for fast data ingestion into Ignite
> > > > through SQL. I would like to ask community to consider to include
> these
> > > two
> > > > features into AI 2.4 in *experimental* state because both of them
> will
> > be
> > > > improved in various ways in the nearest time. If we do so, we will be
> > > able
> > > > to collect some feedback from the users before AI 2.5 release. What
> do
> > > you
> > > > think?
> > > >
> > > > Vladimir.
> > > >
> > > > [1] https://issues.apache.org/jira/browse/IGNITE-6917
> > > > [2] https://issues.apache.org/jira/browse/IGNITE-7253
> > > >
> > > > On Tue, Feb 13, 2018 at 1:22 AM, Dmitriy Setrakyan <
> > > dsetrakyan@apache.org>
> > > > wrote:
> > > >
> > > > > On Mon, Feb 12, 2018 at 9:22 AM, Dmitry Pavlov <
> > dpavlov.spb@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > Unfortunately, a quick fix did not give us too much performance
> > > boost.
> > > > > >
> > > > > > I'm going to implement a complete algorithm change for storing
> the
> > > page
> > > > > > identifier. But this change is quite significant and will require
> > > > > > re-testing. I suggest including
> > > > > > https://issues.apache.org/jira/browse/IGNITE-7638 in the next
> > > version,
> > > > >
> > > > > for
> > > > > > example, to 2.5.
> > > > > >
> > > > > > Sincerely,
> > > > > > Dmitriy Pavlov
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > > Dmitriy, thanks for the update! Are there other tickets that are
> > > holding
> > > > > the release at this point? I remember that there was a performance
> > > > > degradation issue in FULL_SYNC mode, but I cannot find a ticket.
> > > > >
> > > > > D.
> > > > >
> > >
> >
>
>
>
> --
> Best regards,
> Ilya
>
Re: Apache Ignite 2.4 release
Posted by Ilya Lantukh <il...@gridgain.com>.
Hi all,
I'd like to suggest to change default WALMode. Currently we have:
DEFAULT (write and fsync),
LOG_ONLY (write without fsync),
BACKGROUND,
NONE.
It turns out that fsyncs in current DEFAULT mode significantly restricts
Ignite performance. Compared to LOG_ONLY, it offers additional guarantees
that data won't be lost in case of OS or hardware failure, but such
guarantees aren't needed very often, and tradeoff is too big.
I suggest to rename current DEFAULT to STRICT and make LOG_ONLY new default
mode. We can leave DEFAULT as @Deprecated and treat it as STRICT, so that
users with old configs will have the same behaviour.
What do you think?
On Fri, Feb 16, 2018 at 12:35 AM, Denis Magda <dm...@apache.org> wrote:
> Vladimir,
>
> I would suggest not to do this because we still need to spend time on
> testing, documentation, etc. If someone shows interest in this features
> they can assemble binaries from the master.
>
> --
> Denis
>
> On Thu, Feb 15, 2018 at 6:43 AM, Nikolay Izhikov <ni...@apache.org>
> wrote:
>
> > +1
> >
> > В Чт, 15/02/2018 в 17:27 +0300, Vladimir Ozerov пишет:
> > > Igniters,
> > >
> > > AI 2.4 release was shifted a bit and over this time we implemented two
> > > important SQL features:
> > > 1) COPY command for fast file upload to the cluster [1]
> > > 2) Streaming mode for thin driver [2]
> > >
> > > Both commands are very important for fast data ingestion into Ignite
> > > through SQL. I would like to ask community to consider to include these
> > two
> > > features into AI 2.4 in *experimental* state because both of them will
> be
> > > improved in various ways in the nearest time. If we do so, we will be
> > able
> > > to collect some feedback from the users before AI 2.5 release. What do
> > you
> > > think?
> > >
> > > Vladimir.
> > >
> > > [1] https://issues.apache.org/jira/browse/IGNITE-6917
> > > [2] https://issues.apache.org/jira/browse/IGNITE-7253
> > >
> > > On Tue, Feb 13, 2018 at 1:22 AM, Dmitriy Setrakyan <
> > dsetrakyan@apache.org>
> > > wrote:
> > >
> > > > On Mon, Feb 12, 2018 at 9:22 AM, Dmitry Pavlov <
> dpavlov.spb@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > Unfortunately, a quick fix did not give us too much performance
> > boost.
> > > > >
> > > > > I'm going to implement a complete algorithm change for storing the
> > page
> > > > > identifier. But this change is quite significant and will require
> > > > > re-testing. I suggest including
> > > > > https://issues.apache.org/jira/browse/IGNITE-7638 in the next
> > version,
> > > >
> > > > for
> > > > > example, to 2.5.
> > > > >
> > > > > Sincerely,
> > > > > Dmitriy Pavlov
> > > > >
> > > > >
> > > > >
> > > >
> > > > Dmitriy, thanks for the update! Are there other tickets that are
> > holding
> > > > the release at this point? I remember that there was a performance
> > > > degradation issue in FULL_SYNC mode, but I cannot find a ticket.
> > > >
> > > > D.
> > > >
> >
>
--
Best regards,
Ilya
Re: Apache Ignite 2.4 release
Posted by Denis Magda <dm...@apache.org>.
Vladimir,
I would suggest not to do this because we still need to spend time on
testing, documentation, etc. If someone shows interest in this features
they can assemble binaries from the master.
--
Denis
On Thu, Feb 15, 2018 at 6:43 AM, Nikolay Izhikov <ni...@apache.org>
wrote:
> +1
>
> В Чт, 15/02/2018 в 17:27 +0300, Vladimir Ozerov пишет:
> > Igniters,
> >
> > AI 2.4 release was shifted a bit and over this time we implemented two
> > important SQL features:
> > 1) COPY command for fast file upload to the cluster [1]
> > 2) Streaming mode for thin driver [2]
> >
> > Both commands are very important for fast data ingestion into Ignite
> > through SQL. I would like to ask community to consider to include these
> two
> > features into AI 2.4 in *experimental* state because both of them will be
> > improved in various ways in the nearest time. If we do so, we will be
> able
> > to collect some feedback from the users before AI 2.5 release. What do
> you
> > think?
> >
> > Vladimir.
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-6917
> > [2] https://issues.apache.org/jira/browse/IGNITE-7253
> >
> > On Tue, Feb 13, 2018 at 1:22 AM, Dmitriy Setrakyan <
> dsetrakyan@apache.org>
> > wrote:
> >
> > > On Mon, Feb 12, 2018 at 9:22 AM, Dmitry Pavlov <dp...@gmail.com>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > Unfortunately, a quick fix did not give us too much performance
> boost.
> > > >
> > > > I'm going to implement a complete algorithm change for storing the
> page
> > > > identifier. But this change is quite significant and will require
> > > > re-testing. I suggest including
> > > > https://issues.apache.org/jira/browse/IGNITE-7638 in the next
> version,
> > >
> > > for
> > > > example, to 2.5.
> > > >
> > > > Sincerely,
> > > > Dmitriy Pavlov
> > > >
> > > >
> > > >
> > >
> > > Dmitriy, thanks for the update! Are there other tickets that are
> holding
> > > the release at this point? I remember that there was a performance
> > > degradation issue in FULL_SYNC mode, but I cannot find a ticket.
> > >
> > > D.
> > >
>
Re: Apache Ignite 2.4 release
Posted by Nikolay Izhikov <ni...@apache.org>.
+1
В Чт, 15/02/2018 в 17:27 +0300, Vladimir Ozerov пишет:
> Igniters,
>
> AI 2.4 release was shifted a bit and over this time we implemented two
> important SQL features:
> 1) COPY command for fast file upload to the cluster [1]
> 2) Streaming mode for thin driver [2]
>
> Both commands are very important for fast data ingestion into Ignite
> through SQL. I would like to ask community to consider to include these two
> features into AI 2.4 in *experimental* state because both of them will be
> improved in various ways in the nearest time. If we do so, we will be able
> to collect some feedback from the users before AI 2.5 release. What do you
> think?
>
> Vladimir.
>
> [1] https://issues.apache.org/jira/browse/IGNITE-6917
> [2] https://issues.apache.org/jira/browse/IGNITE-7253
>
> On Tue, Feb 13, 2018 at 1:22 AM, Dmitriy Setrakyan <ds...@apache.org>
> wrote:
>
> > On Mon, Feb 12, 2018 at 9:22 AM, Dmitry Pavlov <dp...@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > > Unfortunately, a quick fix did not give us too much performance boost.
> > >
> > > I'm going to implement a complete algorithm change for storing the page
> > > identifier. But this change is quite significant and will require
> > > re-testing. I suggest including
> > > https://issues.apache.org/jira/browse/IGNITE-7638 in the next version,
> >
> > for
> > > example, to 2.5.
> > >
> > > Sincerely,
> > > Dmitriy Pavlov
> > >
> > >
> > >
> >
> > Dmitriy, thanks for the update! Are there other tickets that are holding
> > the release at this point? I remember that there was a performance
> > degradation issue in FULL_SYNC mode, but I cannot find a ticket.
> >
> > D.
> >
Re: Apache Ignite 2.4 release
Posted by Vladimir Ozerov <vo...@gridgain.com>.
Igniters,
AI 2.4 release was shifted a bit and over this time we implemented two
important SQL features:
1) COPY command for fast file upload to the cluster [1]
2) Streaming mode for thin driver [2]
Both commands are very important for fast data ingestion into Ignite
through SQL. I would like to ask community to consider to include these two
features into AI 2.4 in *experimental* state because both of them will be
improved in various ways in the nearest time. If we do so, we will be able
to collect some feedback from the users before AI 2.5 release. What do you
think?
Vladimir.
[1] https://issues.apache.org/jira/browse/IGNITE-6917
[2] https://issues.apache.org/jira/browse/IGNITE-7253
On Tue, Feb 13, 2018 at 1:22 AM, Dmitriy Setrakyan <ds...@apache.org>
wrote:
> On Mon, Feb 12, 2018 at 9:22 AM, Dmitry Pavlov <dp...@gmail.com>
> wrote:
>
> > Hi,
> >
> > Unfortunately, a quick fix did not give us too much performance boost.
> >
> > I'm going to implement a complete algorithm change for storing the page
> > identifier. But this change is quite significant and will require
> > re-testing. I suggest including
> > https://issues.apache.org/jira/browse/IGNITE-7638 in the next version,
> for
> > example, to 2.5.
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> >
> >
> Dmitriy, thanks for the update! Are there other tickets that are holding
> the release at this point? I remember that there was a performance
> degradation issue in FULL_SYNC mode, but I cannot find a ticket.
>
> D.
>
Re: Apache Ignite 2.4 release
Posted by Dmitriy Setrakyan <ds...@apache.org>.
On Mon, Feb 12, 2018 at 9:22 AM, Dmitry Pavlov <dp...@gmail.com>
wrote:
> Hi,
>
> Unfortunately, a quick fix did not give us too much performance boost.
>
> I'm going to implement a complete algorithm change for storing the page
> identifier. But this change is quite significant and will require
> re-testing. I suggest including
> https://issues.apache.org/jira/browse/IGNITE-7638 in the next version, for
> example, to 2.5.
>
> Sincerely,
> Dmitriy Pavlov
>
>
>
Dmitriy, thanks for the update! Are there other tickets that are holding
the release at this point? I remember that there was a performance
degradation issue in FULL_SYNC mode, but I cannot find a ticket.
D.
Re: Apache Ignite 2.4 release
Posted by Dmitry Pavlov <dp...@gmail.com>.
Hi,
Unfortunately, a quick fix did not give us too much performance boost.
I'm going to implement a complete algorithm change for storing the page
identifier. But this change is quite significant and will require
re-testing. I suggest including
https://issues.apache.org/jira/browse/IGNITE-7638 in the next version, for
example, to 2.5.
Sincerely,
Dmitriy Pavlov
ср, 7 февр. 2018 г. в 22:27, Dmitriy Setrakyan <ds...@apache.org>:
> Agree on both, the performance fix and the spark data frames. Let's get
> them into the release.
>
> However, Raymond is right. We should know how long the performance fix will
> take. If it adds another month to the development, we should include it
> into the next release. I am hoping that it can be done faster though.
>
>
> Alexey Goncharuk, Dmitriy Pavlov, any ideas?
>
> D.
>
> On Wed, Feb 7, 2018 at 9:07 AM, Nikolay Izhikov <ni...@apache.org>
> wrote:
>
> > Hello, Igniters.
> >
> > Please, consider including IGNITE-7337 - Spark Data Frames: support
> saving
> > a data frame in Ignite [1] in the 2.4 release.
> > It seems we can merge it into the master in a day or few.
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-7337
> >
> >
> > В Ср, 07/02/2018 в 08:35 -0800, Denis Magda пишет:
> > > I’m voting for the blocker addition into the release. Sergey K. how
> will
> > it affect your testing cycles? Do you need to re-run everything from
> > scratch and how many days you need?
> > >
> > > —
> > > Denis
> > >
> > > > On Feb 6, 2018, at 11:29 PM, Alexey Goncharuk <
> > alexey.goncharuk@gmail.com> wrote:
> > > >
> > > > Guys,
> > > >
> > > > Thanks to Dmitriy Pavlov we found the ticket [1] which causes a major
> > > > slowdown when page replacement starts. Even though it's not a
> > regression, I
> > > > suggest we consider it a blocker for 2.4 because this is a huge
> > performance
> > > > issue which can make it virtually impossible to use native
> persistence
> > when
> > > > data size is significantly larger than memory size.
> > > >
> > > > Any objections?
> > > >
> > > > [1] https://issues.apache.org/jira/browse/IGNITE-7638
> > > >
> > > > 2018-01-30 17:10 GMT+03:00 Pavel Tupitsyn <pt...@apache.org>:
> > > >
> > > > > Igniters, I will handle 2.4 release if there are no objections.
> > > > > Let's take a bit more time for testing and start vote by the end of
> > this
> > > > > week.
> > > > >
> > > > > Pavel
> > > > >
> > > > > On Sat, Jan 27, 2018 at 3:32 AM, Denis Magda <dm...@apache.org>
> > wrote:
> > > > >
> > > > > > Hi Vyacheslav,
> > > > > >
> > > > > > According to the previous review notes the impact of the changes
> > might be
> > > > > > significant, thus, I would recommend us to move the changes to
> the
> > next
> > > > > > release.
> > > > > >
> > > > > > BTW, don’t hesitate to ping reviewers more frequently if there
> is a
> > > > > > pending/abandon review. We are all the people who are tend to
> > forget or
> > > > > > miss notifications ;)
> > > > > >
> > > > > > > On Jan 26, 2018, at 2:04 AM, Vyacheslav Daradur <
> > daradurvs@gmail.com>
> > > > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > Hi, Vladimir, it's good news. I'm looking forward to new Ignite
> > > > >
> > > > > release!
> > > > > > >
> > > > > > > Could you please share a release schedule for 'varint'
> > optimizations?
> > > > > > >
> > > > > > > The task [1] is waiting for review for 5 months.
> > > > > > >
> > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-5097
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Jan 26, 2018 at 12:51 PM, Vladimir Ozerov <
> > > > >
> > > > > vozerov@gridgain.com>
> > > > > > wrote:
> > > > > > > > Hi Igniters,
> > > > > > > >
> > > > > > > > As far as I can see all required tasks and fixes were
> merged. I
> > > > >
> > > > > propose
> > > > > > to
> > > > > > > > take several days of silence to test what we've done and
> start
> > vote at
> > > > > >
> > > > > > the
> > > > > > > > beginning of the next week.
> > > > > > > >
> > > > > > > > Makes sense?
> > > > > > > >
> > > > > > > > On Mon, Jan 22, 2018 at 8:39 PM, Denis Magda <
> > dmagda@apache.org>
> > > > >
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Ok, let’s target Wednesday as a code freeze date.
> > > > > > > > >
> > > > > > > > > Community members who are involved in 2.4 release please
> > merge you
> > > > > >
> > > > > > fixes
> > > > > > > > > and optimizations by that time.
> > > > > > > > >
> > > > > > > > > —
> > > > > > > > > Denis
> > > > > > > > >
> > > > > > > > > > On Jan 22, 2018, at 8:24 AM, Anton Vinogradov <
> > av@apache.org>
> > > > >
> > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Issues
> > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-6711
> > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-6902
> > > > > > > > > >
> > > > > > > > > > are in master and ignite-2.4
> > > > > > > > > >
> > > > > > > > > > On Mon, Jan 22, 2018 at 6:43 PM, Denis Magda <
> > dmagda@apache.org>
> > > > > >
> > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Igniters,
> > > > > > > > > > >
> > > > > > > > > > > What’s left and what prevents us from passing 2.4
> > through the QA
> > > > > >
> > > > > > phase?
> > > > > > > > > > >
> > > > > > > > > > > Petr, Anton, Vladimir, Alex G., others please chime in.
> > > > > > > > > > >
> > > > > > > > > > > —
> > > > > > > > > > > Denis
> > > > > > > > > > >
> > > > > > > > > > > > On Jan 17, 2018, at 7:05 PM, Dmitriy Setrakyan <
> > > > > >
> > > > > > dsetrakyan@apache.org>
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Great news, indeed! Looking forward to 2.4 release.
> > > > > > > > > > > >
> > > > > > > > > > > > On Wed, Jan 17, 2018 at 2:55 AM, Vladimir Ozerov <
> > > > > >
> > > > > > vozerov@gridgain.com
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Igniters,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Great news - two very important tickets - baseline
> > topology and
> > > > > >
> > > > > > Java 8
> > > > > > > > > > > > > support - were merged to master. At this point it
> > makes sense to
> > > > > > > > >
> > > > > > > > > create
> > > > > > > > > > > a
> > > > > > > > > > > > > branch to stabilize the release and finalize
> > outstanding
> > > > >
> > > > > tickets. I
> > > > > > > > > > > created
> > > > > > > > > > > > > the branch *ignite-2.4*. Please follow the rules
> > below when
> > > > >
> > > > > merging
> > > > > > > > > new
> > > > > > > > > > > > > commits to master:
> > > > > > > > > > > > > 1) If ticket is targeted for 2.4 release, please
> > merge to master,
> > > > > >
> > > > > > then
> > > > > > > > > > > > > cherry-pick to ignite-2.4
> > > > > > > > > > > > > 2) Otherwise just merge to master.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Vladimir.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Jan 16, 2018 at 12:32 PM, Anton Vinogradov
> <
> > > > > > > > > > > > > avinogradov@gridgain.com
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > Denis,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I'll review https://issues.apache.org/
> > jira/browse/IGNITE-6902
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Jan 12, 2018 at 11:45 PM, Denis Magda <
> > > > >
> > > > > dmagda@apache.org>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Here is the page create before:
> > > > > > > > > > > > > > > https://cwiki.apache.org/
> > confluence/display/IGNITE/
> > > > > > > > >
> > > > > > > > > Apache+Ignite+2.4
> > > > > > > > > > > <
> > > > > > > > > > > > > > > https://cwiki.apache.org/
> > confluence/display/IGNITE/
> > > > > > > > >
> > > > > > > > > Apache+Ignite+2.4>
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > This is good news. It will be perfect if we
> also
> > release as
> > > > >
> > > > > many
> > > > > > > > > > > > > > "required
> > > > > > > > > > > > > > > tickets" as we can:
> > > > > > > > > > > > > > > https://cwiki.apache.org/
> > confluence/display/IGNITE/
> > > > > > > > >
> > > > > > > > > Apache+Ignite+2.4
> > > > > > > > > > > <
> > > > > > > > > > > > > > > https://cwiki.apache.org/
> > confluence/display/IGNITE/
> > > > > > > > >
> > > > > > > > > Apache+Ignite+2.4>
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > My favorite is memory metrics in bytes:
> > > > > >
> > > > > > https://issues.apache.org/
> > > > > > > > > > > > > > > jira/browse/IGNITE-6902 <
> > https://issues.apache.org/
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > jira/browse/IGNITE-6902
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > The discussion lists are flooded with the
> > demands for this
> > > > > >
> > > > > > essential
> > > > > > > > > > > > > > > capability. Who is going to review it?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > -
> > > > > > > > > > > > > > > Denis
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Jan 12, 2018, at 11:22 AM, Dmitriy
> > Setrakyan <
> > > > > > > > > > > > >
> > > > > > > > > > > > > dsetrakyan@apache.org
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > +1
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Do we have a list of features for 2.4
> > documented anywhere?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > D.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Fri, Jan 12, 2018 at 7:43 AM, Pavel
> > Tupitsyn <
> > > > > > > > > > > > >
> > > > > > > > > > > > > ptupitsyn@apache.org>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > +1
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > But some stuff is not yet in master (like
> > baseline topology),
> > > > > >
> > > > > > so
> > > > > > > > > it
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > too
> > > > > > > > > > > > > > > > > soon to create a branch, I think.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Pavel
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Fri, Jan 12, 2018 at 5:31 PM, Vladimir
> > Ozerov <
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > vozerov@gridgain.com>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Igniters,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > A number of major improvements have been
> > made (or being
> > > > > >
> > > > > > finalized
> > > > > > > > > > > > > at
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > moment) to Ignite since recent 2.3
> > release. This includes
> > > > > > > > >
> > > > > > > > > baseline
> > > > > > > > > > > > > > > > > > topology, new DDL commands, migration to
> > Java 8 and Java 9
> > > > > > > > >
> > > > > > > > > support,
> > > > > > > > > > > > > > > > > > critical performance improvements to WAL,
> > etc..
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I think it makes sense to make a new
> > release. What do you
> > > > > >
> > > > > > think
> > > > > > > > > if
> > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > release AI 2.4 with all this great stuff
> > by the end of Jan?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > If there are no objections, I'll create a
> > branch to start
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > stabilization
> > > > > > > > > > > > > > > > > > process.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Vladimir.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Best Regards, Vyacheslav D.
> > > > > >
> > > > > >
> > >
> > >
> >
>
Re: Apache Ignite 2.4 release
Posted by Nikolay Izhikov <ni...@apache.org>.
Hello, Dmitriy.
Data Frame Read feature is documented in https://issues.apache.org/jira/browse/IGNITE-7345
AFAIK page isn't published yet.
I will document Data Frame Save feature in a few days.
There is a ticket for it - https://issues.apache.org/jira/browse/IGNITE-7655
В Вс, 11/02/2018 в 15:09 -0800, Dmitriy Setrakyan пишет:
> On Sun, Feb 11, 2018 at 2:23 PM, Valentin Kulichenko <
> valentin.kulichenko@gmail.com> wrote:
>
> > IGNITE-7337 (Data frame save functionality) is merged into 2.4.
> >
>
> Thanks Val, this is great news! Also, thanks for Nikolai Izhikov who
> implemented most of the code.
>
> Has this been documented already. Where can we point the community to read
> about this new functionality. What is possible now that was not possible
> before?
Re: Apache Ignite 2.4 release
Posted by Dmitriy Setrakyan <ds...@apache.org>.
On Sun, Feb 11, 2018 at 2:23 PM, Valentin Kulichenko <
valentin.kulichenko@gmail.com> wrote:
> IGNITE-7337 (Data frame save functionality) is merged into 2.4.
>
Thanks Val, this is great news! Also, thanks for Nikolai Izhikov who
implemented most of the code.
Has this been documented already. Where can we point the community to read
about this new functionality. What is possible now that was not possible
before?
Re: Apache Ignite 2.4 release
Posted by Valentin Kulichenko <va...@gmail.com>.
IGNITE-7337 (Data frame save functionality) is merged into 2.4.
-Val
On Fri, Feb 9, 2018 at 11:47 AM, Valentin Kulichenko <
valentin.kulichenko@gmail.com> wrote:
> Nikolay,
>
> To merge it to 2.4, you need to merge the change to ignite-2.4 release.
> Let's do this if we come to an agreement in the neighbor thread.
>
> -Val
>
> On Thu, Feb 8, 2018 at 8:21 PM, Nikolay Izhikov <ni...@apache.org>
> wrote:
>
>> Hello, Dmitriy.
>>
>> IGNITE-7337 are merged to master [1]
>>
>> Do I need to do something more to include this feature into 2.4. release?
>>
>> [1] https://github.com/apache/ignite/commit/7c01452990ad0de0fb84
>> ab4c0424a6d71e5bccba
>>
>> В Ср, 07/02/2018 в 11:26 -0800, Dmitriy Setrakyan пишет:
>> > Agree on both, the performance fix and the spark data frames. Let's get
>> > them into the release.
>> >
>> > However, Raymond is right. We should know how long the performance fix
>> will
>> > take. If it adds another month to the development, we should include it
>> > into the next release. I am hoping that it can be done faster though.
>> >
>> >
>> > Alexey Goncharuk, Dmitriy Pavlov, any ideas?
>> >
>> > D.
>> >
>> > On Wed, Feb 7, 2018 at 9:07 AM, Nikolay Izhikov <ni...@apache.org>
>> wrote:
>> >
>> > > Hello, Igniters.
>> > >
>> > > Please, consider including IGNITE-7337 - Spark Data Frames: support
>> saving
>> > > a data frame in Ignite [1] in the 2.4 release.
>> > > It seems we can merge it into the master in a day or few.
>> > >
>> > > [1] https://issues.apache.org/jira/browse/IGNITE-7337
>> > >
>> > >
>> > > В Ср, 07/02/2018 в 08:35 -0800, Denis Magda пишет:
>> > > > I’m voting for the blocker addition into the release. Sergey K. how
>> will
>> > >
>> > > it affect your testing cycles? Do you need to re-run everything from
>> > > scratch and how many days you need?
>> > > >
>> > > > —
>> > > > Denis
>> > > >
>> > > > > On Feb 6, 2018, at 11:29 PM, Alexey Goncharuk <
>> > >
>> > > alexey.goncharuk@gmail.com> wrote:
>> > > > >
>> > > > > Guys,
>> > > > >
>> > > > > Thanks to Dmitriy Pavlov we found the ticket [1] which causes a
>> major
>> > > > > slowdown when page replacement starts. Even though it's not a
>> > >
>> > > regression, I
>> > > > > suggest we consider it a blocker for 2.4 because this is a huge
>> > >
>> > > performance
>> > > > > issue which can make it virtually impossible to use native
>> persistence
>> > >
>> > > when
>> > > > > data size is significantly larger than memory size.
>> > > > >
>> > > > > Any objections?
>> > > > >
>> > > > > [1] https://issues.apache.org/jira/browse/IGNITE-7638
>> > > > >
>> > > > > 2018-01-30 17:10 GMT+03:00 Pavel Tupitsyn <pt...@apache.org>:
>> > > > >
>> > > > > > Igniters, I will handle 2.4 release if there are no objections.
>> > > > > > Let's take a bit more time for testing and start vote by the
>> end of
>> > >
>> > > this
>> > > > > > week.
>> > > > > >
>> > > > > > Pavel
>> > > > > >
>> > > > > > On Sat, Jan 27, 2018 at 3:32 AM, Denis Magda <dmagda@apache.org
>> >
>> > >
>> > > wrote:
>> > > > > >
>> > > > > > > Hi Vyacheslav,
>> > > > > > >
>> > > > > > > According to the previous review notes the impact of the
>> changes
>> > >
>> > > might be
>> > > > > > > significant, thus, I would recommend us to move the changes
>> to the
>> > >
>> > > next
>> > > > > > > release.
>> > > > > > >
>> > > > > > > BTW, don’t hesitate to ping reviewers more frequently if
>> there is a
>> > > > > > > pending/abandon review. We are all the people who are tend to
>> > >
>> > > forget or
>> > > > > > > miss notifications ;)
>> > > > > > >
>> > > > > > > > On Jan 26, 2018, at 2:04 AM, Vyacheslav Daradur <
>> > >
>> > > daradurvs@gmail.com>
>> > > > > > >
>> > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > Hi, Vladimir, it's good news. I'm looking forward to new
>> Ignite
>> > > > > >
>> > > > > > release!
>> > > > > > > >
>> > > > > > > > Could you please share a release schedule for 'varint'
>> > >
>> > > optimizations?
>> > > > > > > >
>> > > > > > > > The task [1] is waiting for review for 5 months.
>> > > > > > > >
>> > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-5097
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Fri, Jan 26, 2018 at 12:51 PM, Vladimir Ozerov <
>> > > > > >
>> > > > > > vozerov@gridgain.com>
>> > > > > > > wrote:
>> > > > > > > > > Hi Igniters,
>> > > > > > > > >
>> > > > > > > > > As far as I can see all required tasks and fixes were
>> merged. I
>> > > > > >
>> > > > > > propose
>> > > > > > > to
>> > > > > > > > > take several days of silence to test what we've done and
>> start
>> > >
>> > > vote at
>> > > > > > >
>> > > > > > > the
>> > > > > > > > > beginning of the next week.
>> > > > > > > > >
>> > > > > > > > > Makes sense?
>> > > > > > > > >
>> > > > > > > > > On Mon, Jan 22, 2018 at 8:39 PM, Denis Magda <
>> > >
>> > > dmagda@apache.org>
>> > > > > >
>> > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > Ok, let’s target Wednesday as a code freeze date.
>> > > > > > > > > >
>> > > > > > > > > > Community members who are involved in 2.4 release please
>> > >
>> > > merge you
>> > > > > > >
>> > > > > > > fixes
>> > > > > > > > > > and optimizations by that time.
>> > > > > > > > > >
>> > > > > > > > > > —
>> > > > > > > > > > Denis
>> > > > > > > > > >
>> > > > > > > > > > > On Jan 22, 2018, at 8:24 AM, Anton Vinogradov <
>> > >
>> > > av@apache.org>
>> > > > > >
>> > > > > > wrote:
>> > > > > > > > > > >
>> > > > > > > > > > > Issues
>> > > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-6711
>> > > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-6902
>> > > > > > > > > > >
>> > > > > > > > > > > are in master and ignite-2.4
>> > > > > > > > > > >
>> > > > > > > > > > > On Mon, Jan 22, 2018 at 6:43 PM, Denis Magda <
>> > >
>> > > dmagda@apache.org>
>> > > > > > >
>> > > > > > > wrote:
>> > > > > > > > > > >
>> > > > > > > > > > > > Igniters,
>> > > > > > > > > > > >
>> > > > > > > > > > > > What’s left and what prevents us from passing 2.4
>> > >
>> > > through the QA
>> > > > > > >
>> > > > > > > phase?
>> > > > > > > > > > > >
>> > > > > > > > > > > > Petr, Anton, Vladimir, Alex G., others please chime
>> in.
>> > > > > > > > > > > >
>> > > > > > > > > > > > —
>> > > > > > > > > > > > Denis
>> > > > > > > > > > > >
>> > > > > > > > > > > > > On Jan 17, 2018, at 7:05 PM, Dmitriy Setrakyan <
>> > > > > > >
>> > > > > > > dsetrakyan@apache.org>
>> > > > > > > > > > > > wrote:
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Great news, indeed! Looking forward to 2.4
>> release.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > On Wed, Jan 17, 2018 at 2:55 AM, Vladimir Ozerov <
>> > > > > > >
>> > > > > > > vozerov@gridgain.com
>> > > > > > > > > > >
>> > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > > Igniters,
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Great news - two very important tickets -
>> baseline
>> > >
>> > > topology and
>> > > > > > >
>> > > > > > > Java 8
>> > > > > > > > > > > > > > support - were merged to master. At this point
>> it
>> > >
>> > > makes sense to
>> > > > > > > > > >
>> > > > > > > > > > create
>> > > > > > > > > > > > a
>> > > > > > > > > > > > > > branch to stabilize the release and finalize
>> > >
>> > > outstanding
>> > > > > >
>> > > > > > tickets. I
>> > > > > > > > > > > > created
>> > > > > > > > > > > > > > the branch *ignite-2.4*. Please follow the rules
>> > >
>> > > below when
>> > > > > >
>> > > > > > merging
>> > > > > > > > > > new
>> > > > > > > > > > > > > > commits to master:
>> > > > > > > > > > > > > > 1) If ticket is targeted for 2.4 release, please
>> > >
>> > > merge to master,
>> > > > > > >
>> > > > > > > then
>> > > > > > > > > > > > > > cherry-pick to ignite-2.4
>> > > > > > > > > > > > > > 2) Otherwise just merge to master.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Vladimir.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > On Tue, Jan 16, 2018 at 12:32 PM, Anton
>> Vinogradov <
>> > > > > > > > > > > > > > avinogradov@gridgain.com
>> > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > Denis,
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > I'll review https://issues.apache.org/
>> > >
>> > > jira/browse/IGNITE-6902
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > On Fri, Jan 12, 2018 at 11:45 PM, Denis Magda
>> <
>> > > > > >
>> > > > > > dmagda@apache.org>
>> > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > Here is the page create before:
>> > > > > > > > > > > > > > > > https://cwiki.apache.org/
>> > >
>> > > confluence/display/IGNITE/
>> > > > > > > > > >
>> > > > > > > > > > Apache+Ignite+2.4
>> > > > > > > > > > > > <
>> > > > > > > > > > > > > > > > https://cwiki.apache.org/
>> > >
>> > > confluence/display/IGNITE/
>> > > > > > > > > >
>> > > > > > > > > > Apache+Ignite+2.4>
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > This is good news. It will be perfect if we
>> also
>> > >
>> > > release as
>> > > > > >
>> > > > > > many
>> > > > > > > > > > > > > > > "required
>> > > > > > > > > > > > > > > > tickets" as we can:
>> > > > > > > > > > > > > > > > https://cwiki.apache.org/
>> > >
>> > > confluence/display/IGNITE/
>> > > > > > > > > >
>> > > > > > > > > > Apache+Ignite+2.4
>> > > > > > > > > > > > <
>> > > > > > > > > > > > > > > > https://cwiki.apache.org/
>> > >
>> > > confluence/display/IGNITE/
>> > > > > > > > > >
>> > > > > > > > > > Apache+Ignite+2.4>
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > My favorite is memory metrics in bytes:
>> > > > > > >
>> > > > > > > https://issues.apache.org/
>> > > > > > > > > > > > > > > > jira/browse/IGNITE-6902 <
>> > >
>> > > https://issues.apache.org/
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > jira/browse/IGNITE-6902
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > The discussion lists are flooded with the
>> > >
>> > > demands for this
>> > > > > > >
>> > > > > > > essential
>> > > > > > > > > > > > > > > > capability. Who is going to review it?
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > -
>> > > > > > > > > > > > > > > > Denis
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > On Jan 12, 2018, at 11:22 AM, Dmitriy
>> > >
>> > > Setrakyan <
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > dsetrakyan@apache.org
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > +1
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > Do we have a list of features for 2.4
>> > >
>> > > documented anywhere?
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > D.
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > On Fri, Jan 12, 2018 at 7:43 AM, Pavel
>> > >
>> > > Tupitsyn <
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > ptupitsyn@apache.org>
>> > > > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > +1
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > But some stuff is not yet in master
>> (like
>> > >
>> > > baseline topology),
>> > > > > > >
>> > > > > > > so
>> > > > > > > > > > it
>> > > > > > > > > > > > > > is
>> > > > > > > > > > > > > > > > too
>> > > > > > > > > > > > > > > > > > soon to create a branch, I think.
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > Pavel
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > On Fri, Jan 12, 2018 at 5:31 PM,
>> Vladimir
>> > >
>> > > Ozerov <
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > vozerov@gridgain.com>
>> > > > > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > Igniters,
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > A number of major improvements have
>> been
>> > >
>> > > made (or being
>> > > > > > >
>> > > > > > > finalized
>> > > > > > > > > > > > > > at
>> > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > moment) to Ignite since recent 2.3
>> > >
>> > > release. This includes
>> > > > > > > > > >
>> > > > > > > > > > baseline
>> > > > > > > > > > > > > > > > > > > topology, new DDL commands, migration
>> to
>> > >
>> > > Java 8 and Java 9
>> > > > > > > > > >
>> > > > > > > > > > support,
>> > > > > > > > > > > > > > > > > > > critical performance improvements to
>> WAL,
>> > >
>> > > etc..
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > I think it makes sense to make a new
>> > >
>> > > release. What do you
>> > > > > > >
>> > > > > > > think
>> > > > > > > > > > if
>> > > > > > > > > > > > > > we
>> > > > > > > > > > > > > > > > > > > release AI 2.4 with all this great
>> stuff
>> > >
>> > > by the end of Jan?
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > If there are no objections, I'll
>> create a
>> > >
>> > > branch to start
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > stabilization
>> > > > > > > > > > > > > > > > > > > process.
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > Vladimir.
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > --
>> > > > > > > > Best Regards, Vyacheslav D.
>> > > > > > >
>> > > > > > >
>> > > >
>> > > >
>>
>
>
Re: Apache Ignite 2.4 release
Posted by Valentin Kulichenko <va...@gmail.com>.
Nikolay,
To merge it to 2.4, you need to merge the change to ignite-2.4 release.
Let's do this if we come to an agreement in the neighbor thread.
-Val
On Thu, Feb 8, 2018 at 8:21 PM, Nikolay Izhikov <ni...@apache.org> wrote:
> Hello, Dmitriy.
>
> IGNITE-7337 are merged to master [1]
>
> Do I need to do something more to include this feature into 2.4. release?
>
> [1] https://github.com/apache/ignite/commit/7c01452990ad0de0fb84ab4c0424a6
> d71e5bccba
>
> В Ср, 07/02/2018 в 11:26 -0800, Dmitriy Setrakyan пишет:
> > Agree on both, the performance fix and the spark data frames. Let's get
> > them into the release.
> >
> > However, Raymond is right. We should know how long the performance fix
> will
> > take. If it adds another month to the development, we should include it
> > into the next release. I am hoping that it can be done faster though.
> >
> >
> > Alexey Goncharuk, Dmitriy Pavlov, any ideas?
> >
> > D.
> >
> > On Wed, Feb 7, 2018 at 9:07 AM, Nikolay Izhikov <ni...@apache.org>
> wrote:
> >
> > > Hello, Igniters.
> > >
> > > Please, consider including IGNITE-7337 - Spark Data Frames: support
> saving
> > > a data frame in Ignite [1] in the 2.4 release.
> > > It seems we can merge it into the master in a day or few.
> > >
> > > [1] https://issues.apache.org/jira/browse/IGNITE-7337
> > >
> > >
> > > В Ср, 07/02/2018 в 08:35 -0800, Denis Magda пишет:
> > > > I’m voting for the blocker addition into the release. Sergey K. how
> will
> > >
> > > it affect your testing cycles? Do you need to re-run everything from
> > > scratch and how many days you need?
> > > >
> > > > —
> > > > Denis
> > > >
> > > > > On Feb 6, 2018, at 11:29 PM, Alexey Goncharuk <
> > >
> > > alexey.goncharuk@gmail.com> wrote:
> > > > >
> > > > > Guys,
> > > > >
> > > > > Thanks to Dmitriy Pavlov we found the ticket [1] which causes a
> major
> > > > > slowdown when page replacement starts. Even though it's not a
> > >
> > > regression, I
> > > > > suggest we consider it a blocker for 2.4 because this is a huge
> > >
> > > performance
> > > > > issue which can make it virtually impossible to use native
> persistence
> > >
> > > when
> > > > > data size is significantly larger than memory size.
> > > > >
> > > > > Any objections?
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/IGNITE-7638
> > > > >
> > > > > 2018-01-30 17:10 GMT+03:00 Pavel Tupitsyn <pt...@apache.org>:
> > > > >
> > > > > > Igniters, I will handle 2.4 release if there are no objections.
> > > > > > Let's take a bit more time for testing and start vote by the end
> of
> > >
> > > this
> > > > > > week.
> > > > > >
> > > > > > Pavel
> > > > > >
> > > > > > On Sat, Jan 27, 2018 at 3:32 AM, Denis Magda <dm...@apache.org>
> > >
> > > wrote:
> > > > > >
> > > > > > > Hi Vyacheslav,
> > > > > > >
> > > > > > > According to the previous review notes the impact of the
> changes
> > >
> > > might be
> > > > > > > significant, thus, I would recommend us to move the changes to
> the
> > >
> > > next
> > > > > > > release.
> > > > > > >
> > > > > > > BTW, don’t hesitate to ping reviewers more frequently if there
> is a
> > > > > > > pending/abandon review. We are all the people who are tend to
> > >
> > > forget or
> > > > > > > miss notifications ;)
> > > > > > >
> > > > > > > > On Jan 26, 2018, at 2:04 AM, Vyacheslav Daradur <
> > >
> > > daradurvs@gmail.com>
> > > > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > Hi, Vladimir, it's good news. I'm looking forward to new
> Ignite
> > > > > >
> > > > > > release!
> > > > > > > >
> > > > > > > > Could you please share a release schedule for 'varint'
> > >
> > > optimizations?
> > > > > > > >
> > > > > > > > The task [1] is waiting for review for 5 months.
> > > > > > > >
> > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-5097
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Jan 26, 2018 at 12:51 PM, Vladimir Ozerov <
> > > > > >
> > > > > > vozerov@gridgain.com>
> > > > > > > wrote:
> > > > > > > > > Hi Igniters,
> > > > > > > > >
> > > > > > > > > As far as I can see all required tasks and fixes were
> merged. I
> > > > > >
> > > > > > propose
> > > > > > > to
> > > > > > > > > take several days of silence to test what we've done and
> start
> > >
> > > vote at
> > > > > > >
> > > > > > > the
> > > > > > > > > beginning of the next week.
> > > > > > > > >
> > > > > > > > > Makes sense?
> > > > > > > > >
> > > > > > > > > On Mon, Jan 22, 2018 at 8:39 PM, Denis Magda <
> > >
> > > dmagda@apache.org>
> > > > > >
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Ok, let’s target Wednesday as a code freeze date.
> > > > > > > > > >
> > > > > > > > > > Community members who are involved in 2.4 release please
> > >
> > > merge you
> > > > > > >
> > > > > > > fixes
> > > > > > > > > > and optimizations by that time.
> > > > > > > > > >
> > > > > > > > > > —
> > > > > > > > > > Denis
> > > > > > > > > >
> > > > > > > > > > > On Jan 22, 2018, at 8:24 AM, Anton Vinogradov <
> > >
> > > av@apache.org>
> > > > > >
> > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > Issues
> > > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-6711
> > > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-6902
> > > > > > > > > > >
> > > > > > > > > > > are in master and ignite-2.4
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Jan 22, 2018 at 6:43 PM, Denis Magda <
> > >
> > > dmagda@apache.org>
> > > > > > >
> > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Igniters,
> > > > > > > > > > > >
> > > > > > > > > > > > What’s left and what prevents us from passing 2.4
> > >
> > > through the QA
> > > > > > >
> > > > > > > phase?
> > > > > > > > > > > >
> > > > > > > > > > > > Petr, Anton, Vladimir, Alex G., others please chime
> in.
> > > > > > > > > > > >
> > > > > > > > > > > > —
> > > > > > > > > > > > Denis
> > > > > > > > > > > >
> > > > > > > > > > > > > On Jan 17, 2018, at 7:05 PM, Dmitriy Setrakyan <
> > > > > > >
> > > > > > > dsetrakyan@apache.org>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Great news, indeed! Looking forward to 2.4 release.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Wed, Jan 17, 2018 at 2:55 AM, Vladimir Ozerov <
> > > > > > >
> > > > > > > vozerov@gridgain.com
> > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Igniters,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Great news - two very important tickets -
> baseline
> > >
> > > topology and
> > > > > > >
> > > > > > > Java 8
> > > > > > > > > > > > > > support - were merged to master. At this point it
> > >
> > > makes sense to
> > > > > > > > > >
> > > > > > > > > > create
> > > > > > > > > > > > a
> > > > > > > > > > > > > > branch to stabilize the release and finalize
> > >
> > > outstanding
> > > > > >
> > > > > > tickets. I
> > > > > > > > > > > > created
> > > > > > > > > > > > > > the branch *ignite-2.4*. Please follow the rules
> > >
> > > below when
> > > > > >
> > > > > > merging
> > > > > > > > > > new
> > > > > > > > > > > > > > commits to master:
> > > > > > > > > > > > > > 1) If ticket is targeted for 2.4 release, please
> > >
> > > merge to master,
> > > > > > >
> > > > > > > then
> > > > > > > > > > > > > > cherry-pick to ignite-2.4
> > > > > > > > > > > > > > 2) Otherwise just merge to master.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Vladimir.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Jan 16, 2018 at 12:32 PM, Anton
> Vinogradov <
> > > > > > > > > > > > > > avinogradov@gridgain.com
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > Denis,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I'll review https://issues.apache.org/
> > >
> > > jira/browse/IGNITE-6902
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Fri, Jan 12, 2018 at 11:45 PM, Denis Magda <
> > > > > >
> > > > > > dmagda@apache.org>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Here is the page create before:
> > > > > > > > > > > > > > > > https://cwiki.apache.org/
> > >
> > > confluence/display/IGNITE/
> > > > > > > > > >
> > > > > > > > > > Apache+Ignite+2.4
> > > > > > > > > > > > <
> > > > > > > > > > > > > > > > https://cwiki.apache.org/
> > >
> > > confluence/display/IGNITE/
> > > > > > > > > >
> > > > > > > > > > Apache+Ignite+2.4>
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > This is good news. It will be perfect if we
> also
> > >
> > > release as
> > > > > >
> > > > > > many
> > > > > > > > > > > > > > > "required
> > > > > > > > > > > > > > > > tickets" as we can:
> > > > > > > > > > > > > > > > https://cwiki.apache.org/
> > >
> > > confluence/display/IGNITE/
> > > > > > > > > >
> > > > > > > > > > Apache+Ignite+2.4
> > > > > > > > > > > > <
> > > > > > > > > > > > > > > > https://cwiki.apache.org/
> > >
> > > confluence/display/IGNITE/
> > > > > > > > > >
> > > > > > > > > > Apache+Ignite+2.4>
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > My favorite is memory metrics in bytes:
> > > > > > >
> > > > > > > https://issues.apache.org/
> > > > > > > > > > > > > > > > jira/browse/IGNITE-6902 <
> > >
> > > https://issues.apache.org/
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > jira/browse/IGNITE-6902
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > The discussion lists are flooded with the
> > >
> > > demands for this
> > > > > > >
> > > > > > > essential
> > > > > > > > > > > > > > > > capability. Who is going to review it?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > -
> > > > > > > > > > > > > > > > Denis
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Jan 12, 2018, at 11:22 AM, Dmitriy
> > >
> > > Setrakyan <
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > dsetrakyan@apache.org
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > +1
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Do we have a list of features for 2.4
> > >
> > > documented anywhere?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > D.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Fri, Jan 12, 2018 at 7:43 AM, Pavel
> > >
> > > Tupitsyn <
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > ptupitsyn@apache.org>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > +1
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > But some stuff is not yet in master (like
> > >
> > > baseline topology),
> > > > > > >
> > > > > > > so
> > > > > > > > > > it
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > too
> > > > > > > > > > > > > > > > > > soon to create a branch, I think.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Pavel
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Fri, Jan 12, 2018 at 5:31 PM, Vladimir
> > >
> > > Ozerov <
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > vozerov@gridgain.com>
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Igniters,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > A number of major improvements have
> been
> > >
> > > made (or being
> > > > > > >
> > > > > > > finalized
> > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > moment) to Ignite since recent 2.3
> > >
> > > release. This includes
> > > > > > > > > >
> > > > > > > > > > baseline
> > > > > > > > > > > > > > > > > > > topology, new DDL commands, migration
> to
> > >
> > > Java 8 and Java 9
> > > > > > > > > >
> > > > > > > > > > support,
> > > > > > > > > > > > > > > > > > > critical performance improvements to
> WAL,
> > >
> > > etc..
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I think it makes sense to make a new
> > >
> > > release. What do you
> > > > > > >
> > > > > > > think
> > > > > > > > > > if
> > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > release AI 2.4 with all this great
> stuff
> > >
> > > by the end of Jan?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > If there are no objections, I'll
> create a
> > >
> > > branch to start
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > stabilization
> > > > > > > > > > > > > > > > > > > process.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Vladimir.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Best Regards, Vyacheslav D.
> > > > > > >
> > > > > > >
> > > >
> > > >
>
Re: Apache Ignite 2.4 release
Posted by Nikolay Izhikov <ni...@apache.org>.
Hello, Dmitriy.
IGNITE-7337 are merged to master [1]
Do I need to do something more to include this feature into 2.4. release?
[1] https://github.com/apache/ignite/commit/7c01452990ad0de0fb84ab4c0424a6d71e5bccba
В Ср, 07/02/2018 в 11:26 -0800, Dmitriy Setrakyan пишет:
> Agree on both, the performance fix and the spark data frames. Let's get
> them into the release.
>
> However, Raymond is right. We should know how long the performance fix will
> take. If it adds another month to the development, we should include it
> into the next release. I am hoping that it can be done faster though.
>
>
> Alexey Goncharuk, Dmitriy Pavlov, any ideas?
>
> D.
>
> On Wed, Feb 7, 2018 at 9:07 AM, Nikolay Izhikov <ni...@apache.org> wrote:
>
> > Hello, Igniters.
> >
> > Please, consider including IGNITE-7337 - Spark Data Frames: support saving
> > a data frame in Ignite [1] in the 2.4 release.
> > It seems we can merge it into the master in a day or few.
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-7337
> >
> >
> > В Ср, 07/02/2018 в 08:35 -0800, Denis Magda пишет:
> > > I’m voting for the blocker addition into the release. Sergey K. how will
> >
> > it affect your testing cycles? Do you need to re-run everything from
> > scratch and how many days you need?
> > >
> > > —
> > > Denis
> > >
> > > > On Feb 6, 2018, at 11:29 PM, Alexey Goncharuk <
> >
> > alexey.goncharuk@gmail.com> wrote:
> > > >
> > > > Guys,
> > > >
> > > > Thanks to Dmitriy Pavlov we found the ticket [1] which causes a major
> > > > slowdown when page replacement starts. Even though it's not a
> >
> > regression, I
> > > > suggest we consider it a blocker for 2.4 because this is a huge
> >
> > performance
> > > > issue which can make it virtually impossible to use native persistence
> >
> > when
> > > > data size is significantly larger than memory size.
> > > >
> > > > Any objections?
> > > >
> > > > [1] https://issues.apache.org/jira/browse/IGNITE-7638
> > > >
> > > > 2018-01-30 17:10 GMT+03:00 Pavel Tupitsyn <pt...@apache.org>:
> > > >
> > > > > Igniters, I will handle 2.4 release if there are no objections.
> > > > > Let's take a bit more time for testing and start vote by the end of
> >
> > this
> > > > > week.
> > > > >
> > > > > Pavel
> > > > >
> > > > > On Sat, Jan 27, 2018 at 3:32 AM, Denis Magda <dm...@apache.org>
> >
> > wrote:
> > > > >
> > > > > > Hi Vyacheslav,
> > > > > >
> > > > > > According to the previous review notes the impact of the changes
> >
> > might be
> > > > > > significant, thus, I would recommend us to move the changes to the
> >
> > next
> > > > > > release.
> > > > > >
> > > > > > BTW, don’t hesitate to ping reviewers more frequently if there is a
> > > > > > pending/abandon review. We are all the people who are tend to
> >
> > forget or
> > > > > > miss notifications ;)
> > > > > >
> > > > > > > On Jan 26, 2018, at 2:04 AM, Vyacheslav Daradur <
> >
> > daradurvs@gmail.com>
> > > > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > Hi, Vladimir, it's good news. I'm looking forward to new Ignite
> > > > >
> > > > > release!
> > > > > > >
> > > > > > > Could you please share a release schedule for 'varint'
> >
> > optimizations?
> > > > > > >
> > > > > > > The task [1] is waiting for review for 5 months.
> > > > > > >
> > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-5097
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Jan 26, 2018 at 12:51 PM, Vladimir Ozerov <
> > > > >
> > > > > vozerov@gridgain.com>
> > > > > > wrote:
> > > > > > > > Hi Igniters,
> > > > > > > >
> > > > > > > > As far as I can see all required tasks and fixes were merged. I
> > > > >
> > > > > propose
> > > > > > to
> > > > > > > > take several days of silence to test what we've done and start
> >
> > vote at
> > > > > >
> > > > > > the
> > > > > > > > beginning of the next week.
> > > > > > > >
> > > > > > > > Makes sense?
> > > > > > > >
> > > > > > > > On Mon, Jan 22, 2018 at 8:39 PM, Denis Magda <
> >
> > dmagda@apache.org>
> > > > >
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Ok, let’s target Wednesday as a code freeze date.
> > > > > > > > >
> > > > > > > > > Community members who are involved in 2.4 release please
> >
> > merge you
> > > > > >
> > > > > > fixes
> > > > > > > > > and optimizations by that time.
> > > > > > > > >
> > > > > > > > > —
> > > > > > > > > Denis
> > > > > > > > >
> > > > > > > > > > On Jan 22, 2018, at 8:24 AM, Anton Vinogradov <
> >
> > av@apache.org>
> > > > >
> > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Issues
> > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-6711
> > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-6902
> > > > > > > > > >
> > > > > > > > > > are in master and ignite-2.4
> > > > > > > > > >
> > > > > > > > > > On Mon, Jan 22, 2018 at 6:43 PM, Denis Magda <
> >
> > dmagda@apache.org>
> > > > > >
> > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Igniters,
> > > > > > > > > > >
> > > > > > > > > > > What’s left and what prevents us from passing 2.4
> >
> > through the QA
> > > > > >
> > > > > > phase?
> > > > > > > > > > >
> > > > > > > > > > > Petr, Anton, Vladimir, Alex G., others please chime in.
> > > > > > > > > > >
> > > > > > > > > > > —
> > > > > > > > > > > Denis
> > > > > > > > > > >
> > > > > > > > > > > > On Jan 17, 2018, at 7:05 PM, Dmitriy Setrakyan <
> > > > > >
> > > > > > dsetrakyan@apache.org>
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Great news, indeed! Looking forward to 2.4 release.
> > > > > > > > > > > >
> > > > > > > > > > > > On Wed, Jan 17, 2018 at 2:55 AM, Vladimir Ozerov <
> > > > > >
> > > > > > vozerov@gridgain.com
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Igniters,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Great news - two very important tickets - baseline
> >
> > topology and
> > > > > >
> > > > > > Java 8
> > > > > > > > > > > > > support - were merged to master. At this point it
> >
> > makes sense to
> > > > > > > > >
> > > > > > > > > create
> > > > > > > > > > > a
> > > > > > > > > > > > > branch to stabilize the release and finalize
> >
> > outstanding
> > > > >
> > > > > tickets. I
> > > > > > > > > > > created
> > > > > > > > > > > > > the branch *ignite-2.4*. Please follow the rules
> >
> > below when
> > > > >
> > > > > merging
> > > > > > > > > new
> > > > > > > > > > > > > commits to master:
> > > > > > > > > > > > > 1) If ticket is targeted for 2.4 release, please
> >
> > merge to master,
> > > > > >
> > > > > > then
> > > > > > > > > > > > > cherry-pick to ignite-2.4
> > > > > > > > > > > > > 2) Otherwise just merge to master.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Vladimir.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Jan 16, 2018 at 12:32 PM, Anton Vinogradov <
> > > > > > > > > > > > > avinogradov@gridgain.com
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > Denis,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I'll review https://issues.apache.org/
> >
> > jira/browse/IGNITE-6902
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Jan 12, 2018 at 11:45 PM, Denis Magda <
> > > > >
> > > > > dmagda@apache.org>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Here is the page create before:
> > > > > > > > > > > > > > > https://cwiki.apache.org/
> >
> > confluence/display/IGNITE/
> > > > > > > > >
> > > > > > > > > Apache+Ignite+2.4
> > > > > > > > > > > <
> > > > > > > > > > > > > > > https://cwiki.apache.org/
> >
> > confluence/display/IGNITE/
> > > > > > > > >
> > > > > > > > > Apache+Ignite+2.4>
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > This is good news. It will be perfect if we also
> >
> > release as
> > > > >
> > > > > many
> > > > > > > > > > > > > > "required
> > > > > > > > > > > > > > > tickets" as we can:
> > > > > > > > > > > > > > > https://cwiki.apache.org/
> >
> > confluence/display/IGNITE/
> > > > > > > > >
> > > > > > > > > Apache+Ignite+2.4
> > > > > > > > > > > <
> > > > > > > > > > > > > > > https://cwiki.apache.org/
> >
> > confluence/display/IGNITE/
> > > > > > > > >
> > > > > > > > > Apache+Ignite+2.4>
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > My favorite is memory metrics in bytes:
> > > > > >
> > > > > > https://issues.apache.org/
> > > > > > > > > > > > > > > jira/browse/IGNITE-6902 <
> >
> > https://issues.apache.org/
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > jira/browse/IGNITE-6902
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > The discussion lists are flooded with the
> >
> > demands for this
> > > > > >
> > > > > > essential
> > > > > > > > > > > > > > > capability. Who is going to review it?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > -
> > > > > > > > > > > > > > > Denis
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Jan 12, 2018, at 11:22 AM, Dmitriy
> >
> > Setrakyan <
> > > > > > > > > > > > >
> > > > > > > > > > > > > dsetrakyan@apache.org
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > +1
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Do we have a list of features for 2.4
> >
> > documented anywhere?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > D.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Fri, Jan 12, 2018 at 7:43 AM, Pavel
> >
> > Tupitsyn <
> > > > > > > > > > > > >
> > > > > > > > > > > > > ptupitsyn@apache.org>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > +1
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > But some stuff is not yet in master (like
> >
> > baseline topology),
> > > > > >
> > > > > > so
> > > > > > > > > it
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > too
> > > > > > > > > > > > > > > > > soon to create a branch, I think.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Pavel
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Fri, Jan 12, 2018 at 5:31 PM, Vladimir
> >
> > Ozerov <
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > vozerov@gridgain.com>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Igniters,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > A number of major improvements have been
> >
> > made (or being
> > > > > >
> > > > > > finalized
> > > > > > > > > > > > > at
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > moment) to Ignite since recent 2.3
> >
> > release. This includes
> > > > > > > > >
> > > > > > > > > baseline
> > > > > > > > > > > > > > > > > > topology, new DDL commands, migration to
> >
> > Java 8 and Java 9
> > > > > > > > >
> > > > > > > > > support,
> > > > > > > > > > > > > > > > > > critical performance improvements to WAL,
> >
> > etc..
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I think it makes sense to make a new
> >
> > release. What do you
> > > > > >
> > > > > > think
> > > > > > > > > if
> > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > release AI 2.4 with all this great stuff
> >
> > by the end of Jan?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > If there are no objections, I'll create a
> >
> > branch to start
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > stabilization
> > > > > > > > > > > > > > > > > > process.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Vladimir.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Best Regards, Vyacheslav D.
> > > > > >
> > > > > >
> > >
> > >
Re: Apache Ignite 2.4 release
Posted by Dmitriy Setrakyan <ds...@apache.org>.
Agree on both, the performance fix and the spark data frames. Let's get
them into the release.
However, Raymond is right. We should know how long the performance fix will
take. If it adds another month to the development, we should include it
into the next release. I am hoping that it can be done faster though.
Alexey Goncharuk, Dmitriy Pavlov, any ideas?
D.
On Wed, Feb 7, 2018 at 9:07 AM, Nikolay Izhikov <ni...@apache.org> wrote:
> Hello, Igniters.
>
> Please, consider including IGNITE-7337 - Spark Data Frames: support saving
> a data frame in Ignite [1] in the 2.4 release.
> It seems we can merge it into the master in a day or few.
>
> [1] https://issues.apache.org/jira/browse/IGNITE-7337
>
>
> В Ср, 07/02/2018 в 08:35 -0800, Denis Magda пишет:
> > I’m voting for the blocker addition into the release. Sergey K. how will
> it affect your testing cycles? Do you need to re-run everything from
> scratch and how many days you need?
> >
> > —
> > Denis
> >
> > > On Feb 6, 2018, at 11:29 PM, Alexey Goncharuk <
> alexey.goncharuk@gmail.com> wrote:
> > >
> > > Guys,
> > >
> > > Thanks to Dmitriy Pavlov we found the ticket [1] which causes a major
> > > slowdown when page replacement starts. Even though it's not a
> regression, I
> > > suggest we consider it a blocker for 2.4 because this is a huge
> performance
> > > issue which can make it virtually impossible to use native persistence
> when
> > > data size is significantly larger than memory size.
> > >
> > > Any objections?
> > >
> > > [1] https://issues.apache.org/jira/browse/IGNITE-7638
> > >
> > > 2018-01-30 17:10 GMT+03:00 Pavel Tupitsyn <pt...@apache.org>:
> > >
> > > > Igniters, I will handle 2.4 release if there are no objections.
> > > > Let's take a bit more time for testing and start vote by the end of
> this
> > > > week.
> > > >
> > > > Pavel
> > > >
> > > > On Sat, Jan 27, 2018 at 3:32 AM, Denis Magda <dm...@apache.org>
> wrote:
> > > >
> > > > > Hi Vyacheslav,
> > > > >
> > > > > According to the previous review notes the impact of the changes
> might be
> > > > > significant, thus, I would recommend us to move the changes to the
> next
> > > > > release.
> > > > >
> > > > > BTW, don’t hesitate to ping reviewers more frequently if there is a
> > > > > pending/abandon review. We are all the people who are tend to
> forget or
> > > > > miss notifications ;)
> > > > >
> > > > > > On Jan 26, 2018, at 2:04 AM, Vyacheslav Daradur <
> daradurvs@gmail.com>
> > > > >
> > > > > wrote:
> > > > > >
> > > > > > Hi, Vladimir, it's good news. I'm looking forward to new Ignite
> > > >
> > > > release!
> > > > > >
> > > > > > Could you please share a release schedule for 'varint'
> optimizations?
> > > > > >
> > > > > > The task [1] is waiting for review for 5 months.
> > > > > >
> > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-5097
> > > > > >
> > > > > >
> > > > > > On Fri, Jan 26, 2018 at 12:51 PM, Vladimir Ozerov <
> > > >
> > > > vozerov@gridgain.com>
> > > > > wrote:
> > > > > > > Hi Igniters,
> > > > > > >
> > > > > > > As far as I can see all required tasks and fixes were merged. I
> > > >
> > > > propose
> > > > > to
> > > > > > > take several days of silence to test what we've done and start
> vote at
> > > > >
> > > > > the
> > > > > > > beginning of the next week.
> > > > > > >
> > > > > > > Makes sense?
> > > > > > >
> > > > > > > On Mon, Jan 22, 2018 at 8:39 PM, Denis Magda <
> dmagda@apache.org>
> > > >
> > > > wrote:
> > > > > > >
> > > > > > > > Ok, let’s target Wednesday as a code freeze date.
> > > > > > > >
> > > > > > > > Community members who are involved in 2.4 release please
> merge you
> > > > >
> > > > > fixes
> > > > > > > > and optimizations by that time.
> > > > > > > >
> > > > > > > > —
> > > > > > > > Denis
> > > > > > > >
> > > > > > > > > On Jan 22, 2018, at 8:24 AM, Anton Vinogradov <
> av@apache.org>
> > > >
> > > > wrote:
> > > > > > > > >
> > > > > > > > > Issues
> > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-6711
> > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-6902
> > > > > > > > >
> > > > > > > > > are in master and ignite-2.4
> > > > > > > > >
> > > > > > > > > On Mon, Jan 22, 2018 at 6:43 PM, Denis Magda <
> dmagda@apache.org>
> > > > >
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Igniters,
> > > > > > > > > >
> > > > > > > > > > What’s left and what prevents us from passing 2.4
> through the QA
> > > > >
> > > > > phase?
> > > > > > > > > >
> > > > > > > > > > Petr, Anton, Vladimir, Alex G., others please chime in.
> > > > > > > > > >
> > > > > > > > > > —
> > > > > > > > > > Denis
> > > > > > > > > >
> > > > > > > > > > > On Jan 17, 2018, at 7:05 PM, Dmitriy Setrakyan <
> > > > >
> > > > > dsetrakyan@apache.org>
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > Great news, indeed! Looking forward to 2.4 release.
> > > > > > > > > > >
> > > > > > > > > > > On Wed, Jan 17, 2018 at 2:55 AM, Vladimir Ozerov <
> > > > >
> > > > > vozerov@gridgain.com
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Igniters,
> > > > > > > > > > > >
> > > > > > > > > > > > Great news - two very important tickets - baseline
> topology and
> > > > >
> > > > > Java 8
> > > > > > > > > > > > support - were merged to master. At this point it
> makes sense to
> > > > > > > >
> > > > > > > > create
> > > > > > > > > > a
> > > > > > > > > > > > branch to stabilize the release and finalize
> outstanding
> > > >
> > > > tickets. I
> > > > > > > > > > created
> > > > > > > > > > > > the branch *ignite-2.4*. Please follow the rules
> below when
> > > >
> > > > merging
> > > > > > > > new
> > > > > > > > > > > > commits to master:
> > > > > > > > > > > > 1) If ticket is targeted for 2.4 release, please
> merge to master,
> > > > >
> > > > > then
> > > > > > > > > > > > cherry-pick to ignite-2.4
> > > > > > > > > > > > 2) Otherwise just merge to master.
> > > > > > > > > > > >
> > > > > > > > > > > > Vladimir.
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Jan 16, 2018 at 12:32 PM, Anton Vinogradov <
> > > > > > > > > > > > avinogradov@gridgain.com
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > Denis,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I'll review https://issues.apache.org/
> jira/browse/IGNITE-6902
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Jan 12, 2018 at 11:45 PM, Denis Magda <
> > > >
> > > > dmagda@apache.org>
> > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Here is the page create before:
> > > > > > > > > > > > > > https://cwiki.apache.org/
> confluence/display/IGNITE/
> > > > > > > >
> > > > > > > > Apache+Ignite+2.4
> > > > > > > > > > <
> > > > > > > > > > > > > > https://cwiki.apache.org/
> confluence/display/IGNITE/
> > > > > > > >
> > > > > > > > Apache+Ignite+2.4>
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > This is good news. It will be perfect if we also
> release as
> > > >
> > > > many
> > > > > > > > > > > > > "required
> > > > > > > > > > > > > > tickets" as we can:
> > > > > > > > > > > > > > https://cwiki.apache.org/
> confluence/display/IGNITE/
> > > > > > > >
> > > > > > > > Apache+Ignite+2.4
> > > > > > > > > > <
> > > > > > > > > > > > > > https://cwiki.apache.org/
> confluence/display/IGNITE/
> > > > > > > >
> > > > > > > > Apache+Ignite+2.4>
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > My favorite is memory metrics in bytes:
> > > > >
> > > > > https://issues.apache.org/
> > > > > > > > > > > > > > jira/browse/IGNITE-6902 <
> https://issues.apache.org/
> > > > > > > > > > > > >
> > > > > > > > > > > > > jira/browse/IGNITE-6902
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The discussion lists are flooded with the
> demands for this
> > > > >
> > > > > essential
> > > > > > > > > > > > > > capability. Who is going to review it?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > -
> > > > > > > > > > > > > > Denis
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Jan 12, 2018, at 11:22 AM, Dmitriy
> Setrakyan <
> > > > > > > > > > > >
> > > > > > > > > > > > dsetrakyan@apache.org
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > +1
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Do we have a list of features for 2.4
> documented anywhere?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > D.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Fri, Jan 12, 2018 at 7:43 AM, Pavel
> Tupitsyn <
> > > > > > > > > > > >
> > > > > > > > > > > > ptupitsyn@apache.org>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > +1
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > But some stuff is not yet in master (like
> baseline topology),
> > > > >
> > > > > so
> > > > > > > > it
> > > > > > > > > > > > is
> > > > > > > > > > > > > > too
> > > > > > > > > > > > > > > > soon to create a branch, I think.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Pavel
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Fri, Jan 12, 2018 at 5:31 PM, Vladimir
> Ozerov <
> > > > > > > > > > > > >
> > > > > > > > > > > > > vozerov@gridgain.com>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Igniters,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > A number of major improvements have been
> made (or being
> > > > >
> > > > > finalized
> > > > > > > > > > > > at
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > moment) to Ignite since recent 2.3
> release. This includes
> > > > > > > >
> > > > > > > > baseline
> > > > > > > > > > > > > > > > > topology, new DDL commands, migration to
> Java 8 and Java 9
> > > > > > > >
> > > > > > > > support,
> > > > > > > > > > > > > > > > > critical performance improvements to WAL,
> etc..
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I think it makes sense to make a new
> release. What do you
> > > > >
> > > > > think
> > > > > > > > if
> > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > release AI 2.4 with all this great stuff
> by the end of Jan?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > If there are no objections, I'll create a
> branch to start
> > > > > > > > > > > > >
> > > > > > > > > > > > > stabilization
> > > > > > > > > > > > > > > > > process.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Vladimir.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Best Regards, Vyacheslav D.
> > > > >
> > > > >
> >
> >
>
Re: Apache Ignite 2.4 release
Posted by Nikolay Izhikov <ni...@apache.org>.
Hello, Igniters.
Please, consider including IGNITE-7337 - Spark Data Frames: support saving a data frame in Ignite [1] in the 2.4 release.
It seems we can merge it into the master in a day or few.
[1] https://issues.apache.org/jira/browse/IGNITE-7337
В Ср, 07/02/2018 в 08:35 -0800, Denis Magda пишет:
> I’m voting for the blocker addition into the release. Sergey K. how will it affect your testing cycles? Do you need to re-run everything from scratch and how many days you need?
>
> —
> Denis
>
> > On Feb 6, 2018, at 11:29 PM, Alexey Goncharuk <al...@gmail.com> wrote:
> >
> > Guys,
> >
> > Thanks to Dmitriy Pavlov we found the ticket [1] which causes a major
> > slowdown when page replacement starts. Even though it's not a regression, I
> > suggest we consider it a blocker for 2.4 because this is a huge performance
> > issue which can make it virtually impossible to use native persistence when
> > data size is significantly larger than memory size.
> >
> > Any objections?
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-7638
> >
> > 2018-01-30 17:10 GMT+03:00 Pavel Tupitsyn <pt...@apache.org>:
> >
> > > Igniters, I will handle 2.4 release if there are no objections.
> > > Let's take a bit more time for testing and start vote by the end of this
> > > week.
> > >
> > > Pavel
> > >
> > > On Sat, Jan 27, 2018 at 3:32 AM, Denis Magda <dm...@apache.org> wrote:
> > >
> > > > Hi Vyacheslav,
> > > >
> > > > According to the previous review notes the impact of the changes might be
> > > > significant, thus, I would recommend us to move the changes to the next
> > > > release.
> > > >
> > > > BTW, don’t hesitate to ping reviewers more frequently if there is a
> > > > pending/abandon review. We are all the people who are tend to forget or
> > > > miss notifications ;)
> > > >
> > > > > On Jan 26, 2018, at 2:04 AM, Vyacheslav Daradur <da...@gmail.com>
> > > >
> > > > wrote:
> > > > >
> > > > > Hi, Vladimir, it's good news. I'm looking forward to new Ignite
> > >
> > > release!
> > > > >
> > > > > Could you please share a release schedule for 'varint' optimizations?
> > > > >
> > > > > The task [1] is waiting for review for 5 months.
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/IGNITE-5097
> > > > >
> > > > >
> > > > > On Fri, Jan 26, 2018 at 12:51 PM, Vladimir Ozerov <
> > >
> > > vozerov@gridgain.com>
> > > > wrote:
> > > > > > Hi Igniters,
> > > > > >
> > > > > > As far as I can see all required tasks and fixes were merged. I
> > >
> > > propose
> > > > to
> > > > > > take several days of silence to test what we've done and start vote at
> > > >
> > > > the
> > > > > > beginning of the next week.
> > > > > >
> > > > > > Makes sense?
> > > > > >
> > > > > > On Mon, Jan 22, 2018 at 8:39 PM, Denis Magda <dm...@apache.org>
> > >
> > > wrote:
> > > > > >
> > > > > > > Ok, let’s target Wednesday as a code freeze date.
> > > > > > >
> > > > > > > Community members who are involved in 2.4 release please merge you
> > > >
> > > > fixes
> > > > > > > and optimizations by that time.
> > > > > > >
> > > > > > > —
> > > > > > > Denis
> > > > > > >
> > > > > > > > On Jan 22, 2018, at 8:24 AM, Anton Vinogradov <av...@apache.org>
> > >
> > > wrote:
> > > > > > > >
> > > > > > > > Issues
> > > > > > > > https://issues.apache.org/jira/browse/IGNITE-6711
> > > > > > > > https://issues.apache.org/jira/browse/IGNITE-6902
> > > > > > > >
> > > > > > > > are in master and ignite-2.4
> > > > > > > >
> > > > > > > > On Mon, Jan 22, 2018 at 6:43 PM, Denis Magda <dm...@apache.org>
> > > >
> > > > wrote:
> > > > > > > >
> > > > > > > > > Igniters,
> > > > > > > > >
> > > > > > > > > What’s left and what prevents us from passing 2.4 through the QA
> > > >
> > > > phase?
> > > > > > > > >
> > > > > > > > > Petr, Anton, Vladimir, Alex G., others please chime in.
> > > > > > > > >
> > > > > > > > > —
> > > > > > > > > Denis
> > > > > > > > >
> > > > > > > > > > On Jan 17, 2018, at 7:05 PM, Dmitriy Setrakyan <
> > > >
> > > > dsetrakyan@apache.org>
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Great news, indeed! Looking forward to 2.4 release.
> > > > > > > > > >
> > > > > > > > > > On Wed, Jan 17, 2018 at 2:55 AM, Vladimir Ozerov <
> > > >
> > > > vozerov@gridgain.com
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Igniters,
> > > > > > > > > > >
> > > > > > > > > > > Great news - two very important tickets - baseline topology and
> > > >
> > > > Java 8
> > > > > > > > > > > support - were merged to master. At this point it makes sense to
> > > > > > >
> > > > > > > create
> > > > > > > > > a
> > > > > > > > > > > branch to stabilize the release and finalize outstanding
> > >
> > > tickets. I
> > > > > > > > > created
> > > > > > > > > > > the branch *ignite-2.4*. Please follow the rules below when
> > >
> > > merging
> > > > > > > new
> > > > > > > > > > > commits to master:
> > > > > > > > > > > 1) If ticket is targeted for 2.4 release, please merge to master,
> > > >
> > > > then
> > > > > > > > > > > cherry-pick to ignite-2.4
> > > > > > > > > > > 2) Otherwise just merge to master.
> > > > > > > > > > >
> > > > > > > > > > > Vladimir.
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Jan 16, 2018 at 12:32 PM, Anton Vinogradov <
> > > > > > > > > > > avinogradov@gridgain.com
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > Denis,
> > > > > > > > > > > >
> > > > > > > > > > > > I'll review https://issues.apache.org/jira/browse/IGNITE-6902
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Jan 12, 2018 at 11:45 PM, Denis Magda <
> > >
> > > dmagda@apache.org>
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Here is the page create before:
> > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/IGNITE/
> > > > > > >
> > > > > > > Apache+Ignite+2.4
> > > > > > > > > <
> > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/IGNITE/
> > > > > > >
> > > > > > > Apache+Ignite+2.4>
> > > > > > > > > > > > >
> > > > > > > > > > > > > This is good news. It will be perfect if we also release as
> > >
> > > many
> > > > > > > > > > > > "required
> > > > > > > > > > > > > tickets" as we can:
> > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/IGNITE/
> > > > > > >
> > > > > > > Apache+Ignite+2.4
> > > > > > > > > <
> > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/IGNITE/
> > > > > > >
> > > > > > > Apache+Ignite+2.4>
> > > > > > > > > > > > >
> > > > > > > > > > > > > My favorite is memory metrics in bytes:
> > > >
> > > > https://issues.apache.org/
> > > > > > > > > > > > > jira/browse/IGNITE-6902 <https://issues.apache.org/
> > > > > > > > > > > >
> > > > > > > > > > > > jira/browse/IGNITE-6902
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > The discussion lists are flooded with the demands for this
> > > >
> > > > essential
> > > > > > > > > > > > > capability. Who is going to review it?
> > > > > > > > > > > > >
> > > > > > > > > > > > > -
> > > > > > > > > > > > > Denis
> > > > > > > > > > > > >
> > > > > > > > > > > > > > On Jan 12, 2018, at 11:22 AM, Dmitriy Setrakyan <
> > > > > > > > > > >
> > > > > > > > > > > dsetrakyan@apache.org
> > > > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > +1
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Do we have a list of features for 2.4 documented anywhere?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > D.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Jan 12, 2018 at 7:43 AM, Pavel Tupitsyn <
> > > > > > > > > > >
> > > > > > > > > > > ptupitsyn@apache.org>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > +1
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > But some stuff is not yet in master (like baseline topology),
> > > >
> > > > so
> > > > > > > it
> > > > > > > > > > > is
> > > > > > > > > > > > > too
> > > > > > > > > > > > > > > soon to create a branch, I think.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Pavel
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Fri, Jan 12, 2018 at 5:31 PM, Vladimir Ozerov <
> > > > > > > > > > > >
> > > > > > > > > > > > vozerov@gridgain.com>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Igniters,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > A number of major improvements have been made (or being
> > > >
> > > > finalized
> > > > > > > > > > > at
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > moment) to Ignite since recent 2.3 release. This includes
> > > > > > >
> > > > > > > baseline
> > > > > > > > > > > > > > > > topology, new DDL commands, migration to Java 8 and Java 9
> > > > > > >
> > > > > > > support,
> > > > > > > > > > > > > > > > critical performance improvements to WAL, etc..
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I think it makes sense to make a new release. What do you
> > > >
> > > > think
> > > > > > > if
> > > > > > > > > > > we
> > > > > > > > > > > > > > > > release AI 2.4 with all this great stuff by the end of Jan?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > If there are no objections, I'll create a branch to start
> > > > > > > > > > > >
> > > > > > > > > > > > stabilization
> > > > > > > > > > > > > > > > process.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Vladimir.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Best Regards, Vyacheslav D.
> > > >
> > > >
>
>
Re: Apache Ignite 2.4 release
Posted by Denis Magda <dm...@apache.org>.
I’m voting for the blocker addition into the release. Sergey K. how will it affect your testing cycles? Do you need to re-run everything from scratch and how many days you need?
—
Denis
> On Feb 6, 2018, at 11:29 PM, Alexey Goncharuk <al...@gmail.com> wrote:
>
> Guys,
>
> Thanks to Dmitriy Pavlov we found the ticket [1] which causes a major
> slowdown when page replacement starts. Even though it's not a regression, I
> suggest we consider it a blocker for 2.4 because this is a huge performance
> issue which can make it virtually impossible to use native persistence when
> data size is significantly larger than memory size.
>
> Any objections?
>
> [1] https://issues.apache.org/jira/browse/IGNITE-7638
>
> 2018-01-30 17:10 GMT+03:00 Pavel Tupitsyn <pt...@apache.org>:
>
>> Igniters, I will handle 2.4 release if there are no objections.
>> Let's take a bit more time for testing and start vote by the end of this
>> week.
>>
>> Pavel
>>
>> On Sat, Jan 27, 2018 at 3:32 AM, Denis Magda <dm...@apache.org> wrote:
>>
>>> Hi Vyacheslav,
>>>
>>> According to the previous review notes the impact of the changes might be
>>> significant, thus, I would recommend us to move the changes to the next
>>> release.
>>>
>>> BTW, don’t hesitate to ping reviewers more frequently if there is a
>>> pending/abandon review. We are all the people who are tend to forget or
>>> miss notifications ;)
>>>
>>>> On Jan 26, 2018, at 2:04 AM, Vyacheslav Daradur <da...@gmail.com>
>>> wrote:
>>>>
>>>> Hi, Vladimir, it's good news. I'm looking forward to new Ignite
>> release!
>>>>
>>>> Could you please share a release schedule for 'varint' optimizations?
>>>>
>>>> The task [1] is waiting for review for 5 months.
>>>>
>>>> [1] https://issues.apache.org/jira/browse/IGNITE-5097
>>>>
>>>>
>>>> On Fri, Jan 26, 2018 at 12:51 PM, Vladimir Ozerov <
>> vozerov@gridgain.com>
>>> wrote:
>>>>> Hi Igniters,
>>>>>
>>>>> As far as I can see all required tasks and fixes were merged. I
>> propose
>>> to
>>>>> take several days of silence to test what we've done and start vote at
>>> the
>>>>> beginning of the next week.
>>>>>
>>>>> Makes sense?
>>>>>
>>>>> On Mon, Jan 22, 2018 at 8:39 PM, Denis Magda <dm...@apache.org>
>> wrote:
>>>>>
>>>>>> Ok, let’s target Wednesday as a code freeze date.
>>>>>>
>>>>>> Community members who are involved in 2.4 release please merge you
>>> fixes
>>>>>> and optimizations by that time.
>>>>>>
>>>>>> —
>>>>>> Denis
>>>>>>
>>>>>>> On Jan 22, 2018, at 8:24 AM, Anton Vinogradov <av...@apache.org>
>> wrote:
>>>>>>>
>>>>>>> Issues
>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6711
>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6902
>>>>>>>
>>>>>>> are in master and ignite-2.4
>>>>>>>
>>>>>>> On Mon, Jan 22, 2018 at 6:43 PM, Denis Magda <dm...@apache.org>
>>> wrote:
>>>>>>>
>>>>>>>> Igniters,
>>>>>>>>
>>>>>>>> What’s left and what prevents us from passing 2.4 through the QA
>>> phase?
>>>>>>>>
>>>>>>>> Petr, Anton, Vladimir, Alex G., others please chime in.
>>>>>>>>
>>>>>>>> —
>>>>>>>> Denis
>>>>>>>>
>>>>>>>>> On Jan 17, 2018, at 7:05 PM, Dmitriy Setrakyan <
>>> dsetrakyan@apache.org>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Great news, indeed! Looking forward to 2.4 release.
>>>>>>>>>
>>>>>>>>> On Wed, Jan 17, 2018 at 2:55 AM, Vladimir Ozerov <
>>> vozerov@gridgain.com
>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Igniters,
>>>>>>>>>>
>>>>>>>>>> Great news - two very important tickets - baseline topology and
>>> Java 8
>>>>>>>>>> support - were merged to master. At this point it makes sense to
>>>>>> create
>>>>>>>> a
>>>>>>>>>> branch to stabilize the release and finalize outstanding
>> tickets. I
>>>>>>>> created
>>>>>>>>>> the branch *ignite-2.4*. Please follow the rules below when
>> merging
>>>>>> new
>>>>>>>>>> commits to master:
>>>>>>>>>> 1) If ticket is targeted for 2.4 release, please merge to master,
>>> then
>>>>>>>>>> cherry-pick to ignite-2.4
>>>>>>>>>> 2) Otherwise just merge to master.
>>>>>>>>>>
>>>>>>>>>> Vladimir.
>>>>>>>>>>
>>>>>>>>>> On Tue, Jan 16, 2018 at 12:32 PM, Anton Vinogradov <
>>>>>>>>>> avinogradov@gridgain.com
>>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Denis,
>>>>>>>>>>>
>>>>>>>>>>> I'll review https://issues.apache.org/jira/browse/IGNITE-6902
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Jan 12, 2018 at 11:45 PM, Denis Magda <
>> dmagda@apache.org>
>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Here is the page create before:
>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/IGNITE/
>>>>>> Apache+Ignite+2.4
>>>>>>>> <
>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/IGNITE/
>>>>>> Apache+Ignite+2.4>
>>>>>>>>>>>>
>>>>>>>>>>>> This is good news. It will be perfect if we also release as
>> many
>>>>>>>>>>> "required
>>>>>>>>>>>> tickets" as we can:
>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/IGNITE/
>>>>>> Apache+Ignite+2.4
>>>>>>>> <
>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/IGNITE/
>>>>>> Apache+Ignite+2.4>
>>>>>>>>>>>>
>>>>>>>>>>>> My favorite is memory metrics in bytes:
>>> https://issues.apache.org/
>>>>>>>>>>>> jira/browse/IGNITE-6902 <https://issues.apache.org/
>>>>>>>>>>> jira/browse/IGNITE-6902
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> The discussion lists are flooded with the demands for this
>>> essential
>>>>>>>>>>>> capability. Who is going to review it?
>>>>>>>>>>>>
>>>>>>>>>>>> -
>>>>>>>>>>>> Denis
>>>>>>>>>>>>
>>>>>>>>>>>>> On Jan 12, 2018, at 11:22 AM, Dmitriy Setrakyan <
>>>>>>>>>> dsetrakyan@apache.org
>>>>>>>>>>>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> +1
>>>>>>>>>>>>>
>>>>>>>>>>>>> Do we have a list of features for 2.4 documented anywhere?
>>>>>>>>>>>>>
>>>>>>>>>>>>> D.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Jan 12, 2018 at 7:43 AM, Pavel Tupitsyn <
>>>>>>>>>> ptupitsyn@apache.org>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> +1
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> But some stuff is not yet in master (like baseline topology),
>>> so
>>>>>> it
>>>>>>>>>> is
>>>>>>>>>>>> too
>>>>>>>>>>>>>> soon to create a branch, I think.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Pavel
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Jan 12, 2018 at 5:31 PM, Vladimir Ozerov <
>>>>>>>>>>> vozerov@gridgain.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Igniters,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> A number of major improvements have been made (or being
>>> finalized
>>>>>>>>>> at
>>>>>>>>>>>> the
>>>>>>>>>>>>>>> moment) to Ignite since recent 2.3 release. This includes
>>>>>> baseline
>>>>>>>>>>>>>>> topology, new DDL commands, migration to Java 8 and Java 9
>>>>>> support,
>>>>>>>>>>>>>>> critical performance improvements to WAL, etc..
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I think it makes sense to make a new release. What do you
>>> think
>>>>>> if
>>>>>>>>>> we
>>>>>>>>>>>>>>> release AI 2.4 with all this great stuff by the end of Jan?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> If there are no objections, I'll create a branch to start
>>>>>>>>>>> stabilization
>>>>>>>>>>>>>>> process.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Vladimir.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Best Regards, Vyacheslav D.
>>>
>>>
>>