You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by Alexey Goncharuk <al...@gmail.com> on 2020/11/02 10:14:08 UTC
[DISCUSS] Ignite 3.0 development approach
Igniters,
I wanted to pitch a rather radical idea regarding the Ignite 3.0
development which has occurred to me some time ago.
We already have several IEPs targeted to Ignite 3.0 which imply major
changes to the codebase (the change in replication protocol and thus
transactions, change in binary format, updated metastorage, etc). We also
planned significant changes in public APIs: configuration format change,
improvements in cache APIs, SQL APIs, transaction mode rework. The wishlist
of changes for Ignite 3.0 is huge.
So, I was wondering whether it makes sense to try to change the old
codebase, or start with a new baseline and move old pieces of code that do
not require significant rework. Personally, I would go with the second
option for the following reasons:
- We have a chance to shift the development paradigm in the project and
introduce the practice of true unit-tests. In the new baseline at the
beginning there will be no ability to run an end-to-end scenario, thus we
will be forced to write unit-tests. So far, such practice was hard to
implement because of tight coupling between Ignite components and inability
to instantiate components without an instance of KernalContext. For
example, we should be able to thoroughly test internal primitives, such as
replication protocol (without actual communication), distributed
metastorage contracts, persistence layer, etc.
- We will significantly reduce the development cycle in the beginning
(right now the RunAll takes two hours of astronomical time with empty TC;
in the new approach developer will be able to run ALL tests locally in a
matter of minutes)
- We can get rid of TC bot and enforce green TC by integrating TC build
results with GitHub PRs (the same way Travis is currently integrated to PR
check). We should restrict PR merge without a TC check
- We will still have to re-write all tests, but only once. If we try to
modify the old codebase, we would need to modify all the tests for every
major change (public API change, configuration change)
- We will have fewer conflicts when working together. For example, I
cannot imagine how one would merge two changes of getting rid of
IgniteFuture and changes in replication protocol, for example
Technically, I would suggest creating a new repository for Ignite 3.0
(perhaps, a new clean branch, but a new repo looks nicer to me) and a new
Ignite 3.0 TeamCity project.
While it may seem quite radical, I do believe that this approach will give
us more benefits than trying to make such major changes in the existing
codebase. If needed, let's schedule a discord chat like before to discuss
this.
WDYT?
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
In my opinion, what you are actually proposing is writing a new
IMDG/distributed database.
I'm not sure why one would assume that this new product will be
particularly successful with users. We have some very good developers out
there now, but some of the people who actually wrote Ignite 2.x are no
longer with us. Maybe it will be a new and good IMDG, maybe not. Time has
passed, the bar is now higher than it was before. People might as well
consider moving to a different IMDG not called Ignite, than moving to a
different, experimental IMDG named Ignite 3.0.
Maybe there are some bad parts that we have (such as
persistence format/separation, split-brain protocol, etc), but we have 95
out of 100 parts which are very good and battle tested, and I would prefer
them to be kept intact instead of cherry-picking them one by one into the
new grid.
We have very good distributed compute/peer class loading support, people
don't realize how good it is, but they will notice if it is gone/rewritten
from scratch to "just ok" state instead of "very good".
I think that we should just do a deep refactoring/API change in 3.0, and
leave the rest of stuff to 3.x or 4.0. I also happen to think that the code
we already have is very valuable, and it would be a pity to see it replaced
with cleaner, worse offering.
Regards,
--
Ilya Kasnacheev
пн, 2 нояб. 2020 г. в 18:18, Alexey Goncharuk <al...@gmail.com>:
> Nikolay, Pavel,
>
> Thanks for the feedback! First of all, I wanted to stress that I do not
> intend to rewrite everything from scratch (I never used this phrase). There
> are significant parts of code that would be moved with minimal
> modifications. Second, I never said that we will get rid of the old tests
> codebase. Every single test from Ignite 2.x should be moved to Ignite 3
> regardless of how we choose to proceed.
>
> My point is that for some parts of the code a clean bottom-up
> implementation will be cheaper in many ways. Let me give you a few concrete
> examples:
>
> - I think no one can object that we need a cleanly separated persistence
> layer for Ignite. There is a very raw draft IEP for this already. On the
> other hand, I think we also can agree that we need a split-brain
> resistant
> replication protocol for caches. There is also an IEP for this. Neither
> of
> the changes is a good fit for 2.x because they are likely to introduce
> breaking changes in the persistence layer, configuration and behavior.
> Additionally, these components are now tightly coupled, so there is no
> way
> these two changes can be implemented in parallel and then merged
> together
> easily. So what we will end up with is having to implement these changes
> sequentially, fixing all existing tests twice, and essentially throwing
> away half of the work done because the other part of the change is
> re-implemented
> - Similar example goes with getting rid of IgniteInternalFuture and
> replacing it with CompletableFuture, and any other change that touches
> the
> asynchronous part of the code.
>
> Third, I do not see how this choice affects the UX of Ignite. The end user
> experience must be fixed in the IEP regardless of the development process
> and the fact that we have gaps in this area in Ignite 2.x just confirms
> that.
>
> Pavel, agree that a repo/branch is a technicality, I guess if reformulate,
> my point is that we might agree to have a single development master branch
> with 'disassembled' end-to-end functionality for some period of time to
> speed up development, and re-assemble the core features after having
> submodules tested independently.
>
> Nikolay,
> >We have many features that have to evolve.
> >Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
> This is not very specific. In the end, resources are limited and we will
> not be able to drive both tracks simultaneously, especially after a couple
> of features having been implemented for Ignite 3.0. If there are indeed
> some major changes that we want to do in Ignite 2.x instead of putting
> effort into 3.0 - let's discuss them. I am just not aware of any, that's
> why I am eager to move forward with Ignite 3.0.
>
> >We have bugs and issues that can be fixed in 2.x without breaking backward
> compatibility.
> >We have many users who are happy with the 2.x with all it’s issues.
> These changes will be covered by end-to-end tests and migrated to Ignite
> 3.0, so I see no issues here.
>
> Finally, Anton & Nikolay
> I do not have an estimate for this simply because the activity is
> community-driven and it depends on the number of people willing to
> contribute. With the current pace, I would hope to have an RC of Ignite 3.0
> to be ready by the end of 2021. My gut feeling is that by moving with
> incremental changes, we will not be able to implement even half of the
> wishlist by that time.
> I doubt that releasing several major releases with breaking changes will
> make Ignite users happy either because each upgrade will cost Ignite users
> money, so the fewer major versions we release, the better. Thus my wish to
> include all breaking changes in one release.
>
> I'll be now quiet for a while, let's see what other community members
> think.
>
> пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <pt...@apache.org>:
>
> > 1. Rewriting from scratch is never a good idea.
> > We don't want to follow the path of Netscape and lose all our users
> > by the time we have a working 3.0 [1]
> >
> > 2. Not sure about new repo - seems like some pain and no gain, what's the
> > problem with a branch?
> >
> > 3. We should keep existing integration tests when possible.
> > We have accumulated a lot of edge case knowledge over the years,
> > it is not a good idea to send all of that down the drain.
> > Yes, integration tests are slow, but they are the most valuable.
> > I think we can move more stuff into nightly runs and have a fast and
> modern
> > basic suite.
> >
> >
> > Alexey, you are much more familiar with the Ignite core codebase than
> most
> > of us,
> > can you please explain in more detail which particular feature, in your
> > opinion,
> > mandates this "start from scratch" approach?
> > Is it really not possible at all to follow a less radical way?
> >
> >
> > [1]
> >
> >
> https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
> >
> > On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <ni...@apache.org>
> > wrote:
> >
> > > Hello, Alexey.
> > >
> > > I think that «rewriting from scratch» approach has a high risk to make
> > new
> > > features unusable.
> > > At the time Ignite2 was started no-one wants to do bad UX or bad
> > features.
> > > Nevertheless, it happen.
> > >
> > > I think we can avoid it with the Ignite3 and successors if we will move
> > > step by step without keeping backward compatibility
> > > With the step by step approach, we can focus on each component
> > separately.
> > >
> > > > What new features are we planning to implement for Ignite 2.x?
> > >
> > > We have many features that have to evolve.
> > > Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
> > > We have bugs and issues that can be fixed in 2.x without breaking
> > backward
> > > compatibility.
> > > We have many users who are happy with the 2.x with all it’s issues.
> > >
> > > > 2 нояб. 2020 г., в 14:09, Anton Vinogradov <av...@apache.org>
> написал(а):
> > > >
> > > > Alexey,
> > > >
> > > > Do we have any estimates of how fast we'll be able to gain
> > > production-ready
> > > > AI 3.0 in case of a "new repo" choice?
> > > >
> > > > On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk <
> > > alexey.goncharuk@gmail.com>
> > > > wrote:
> > > >
> > > >> Nikolay,
> > > >>
> > > >> What new features are we planning to implement for Ignite 2.x? I
> think
> > > once
> > > >> we commence working on Ignite 3.0, we should gradually cease the
> > > activity
> > > >> on Ignite 2.x to mere bugfixes because such parallel development
> will
> > be
> > > >> overwhelming regardless of how we choose to proceed.
> > > >>
> > > >> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <ni...@apache.org>:
> > > >>
> > > >>> To be clear:
> > > >>>
> > > >>>> I would suggest creating a new repository for Ignite 3.0
> (perhaps, a
> > > >> new
> > > >>> clean branch, but a new repo looks nicer to me) and a new Ignite
> 3.0
> > > >>> TeamCity project.
> > > >>>
> > > >>> +1 for new Team City project.
> > > >>> +1 for new branch for Ignite3.
> > > >>> -1 for new repo.
> > > >>>
> > > >>>
> > > >>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <NIzhikov.dev@gmail.com
> >
> > > >>> написал(а):
> > > >>>>
> > > >>>> Hello, Alexey.
> > > >>>>
> > > >>>> I think it will hurt our project more than help.
> > > >>>> Developing new features for 2 separate branches with the different
> > > APIs
> > > >>> and internal structure is overwhelming
> > > >>>>
> > > >>>> Maybe we should relax a bit requirements for Ignite3?
> > > >>>> Maybe we should move step by step and make Ignite3 with new
> > > >>> configuration than Ignite4 with new transactions, etc?
> > > >>>>
> > > >>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <
> > > >> alexey.goncharuk@gmail.com>
> > > >>> написал(а):
> > > >>>>>
> > > >>>>> Igniters,
> > > >>>>>
> > > >>>>> I wanted to pitch a rather radical idea regarding the Ignite 3.0
> > > >>>>> development which has occurred to me some time ago.
> > > >>>>>
> > > >>>>> We already have several IEPs targeted to Ignite 3.0 which imply
> > major
> > > >>>>> changes to the codebase (the change in replication protocol and
> > thus
> > > >>>>> transactions, change in binary format, updated metastorage, etc).
> > We
> > > >>> also
> > > >>>>> planned significant changes in public APIs: configuration format
> > > >> change,
> > > >>>>> improvements in cache APIs, SQL APIs, transaction mode rework.
> The
> > > >>> wishlist
> > > >>>>> of changes for Ignite 3.0 is huge.
> > > >>>>>
> > > >>>>> So, I was wondering whether it makes sense to try to change the
> old
> > > >>>>> codebase, or start with a new baseline and move old pieces of
> code
> > > >> that
> > > >>> do
> > > >>>>> not require significant rework. Personally, I would go with the
> > > second
> > > >>>>> option for the following reasons:
> > > >>>>>
> > > >>>>> - We have a chance to shift the development paradigm in the
> project
> > > >> and
> > > >>>>> introduce the practice of true unit-tests. In the new baseline at
> > the
> > > >>>>> beginning there will be no ability to run an end-to-end scenario,
> > > >> thus
> > > >>> we
> > > >>>>> will be forced to write unit-tests. So far, such practice was
> hard
> > to
> > > >>>>> implement because of tight coupling between Ignite components and
> > > >>> inability
> > > >>>>> to instantiate components without an instance of KernalContext.
> For
> > > >>>>> example, we should be able to thoroughly test internal
> primitives,
> > > >>> such as
> > > >>>>> replication protocol (without actual communication), distributed
> > > >>>>> metastorage contracts, persistence layer, etc.
> > > >>>>> - We will significantly reduce the development cycle in the
> > beginning
> > > >>>>> (right now the RunAll takes two hours of astronomical time with
> > empty
> > > >>> TC;
> > > >>>>> in the new approach developer will be able to run ALL tests
> locally
> > > >> in
> > > >>> a
> > > >>>>> matter of minutes)
> > > >>>>> - We can get rid of TC bot and enforce green TC by integrating TC
> > > >> build
> > > >>>>> results with GitHub PRs (the same way Travis is currently
> > integrated
> > > >>> to PR
> > > >>>>> check). We should restrict PR merge without a TC check
> > > >>>>> - We will still have to re-write all tests, but only once. If we
> > try
> > > >> to
> > > >>>>> modify the old codebase, we would need to modify all the tests
> for
> > > >>> every
> > > >>>>> major change (public API change, configuration change)
> > > >>>>> - We will have fewer conflicts when working together. For
> example,
> > I
> > > >>>>> cannot imagine how one would merge two changes of getting rid of
> > > >>>>> IgniteFuture and changes in replication protocol, for example
> > > >>>>>
> > > >>>>> Technically, I would suggest creating a new repository for Ignite
> > 3.0
> > > >>>>> (perhaps, a new clean branch, but a new repo looks nicer to me)
> > and a
> > > >>> new
> > > >>>>> Ignite 3.0 TeamCity project.
> > > >>>>>
> > > >>>>> While it may seem quite radical, I do believe that this approach
> > will
> > > >>> give
> > > >>>>> us more benefits than trying to make such major changes in the
> > > >> existing
> > > >>>>> codebase. If needed, let's schedule a discord chat like before to
> > > >>> discuss
> > > >>>>> this.
> > > >>>>>
> > > >>>>> WDYT?
> > > >>>>
> > > >>>
> > > >>>
> > > >>
> > >
> > >
> >
>
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Nikita Ivanov <ni...@gmail.com>.
+1 on having a separate repo. Make the work cleaner and more effective.
--
Nikita Ivanov
On Mon, Nov 2, 2020 at 11:09 PM Nikolay Izhikov <ni...@apache.org> wrote:
> Igniters, should we have a call for this topic?
>
> > 2 нояб. 2020 г., в 18:53, Pavel Tupitsyn <pt...@apache.org>
> написал(а):
> >
> >> not intend to rewrite everything from scratch
> >
> >> Every single test from Ignite 2.x should be moved to Ignite 3
> >> regardless of how we choose to proceed.
> >
> > Alexey, thank you for the explanation, this addresses all of my concerns.
> >
> >
> >
> >
> >
> > On Mon, Nov 2, 2020 at 6:43 PM Andrey Mashenkov <
> andrey.mashenkov@gmail.com>
> > wrote:
> >
> >> Hi, Igniters.
> >>
> >> * AFAIU, we need a new repo if we want to apply different restrictions
> to
> >> pull requests,
> >> otherwise I see no difference for myself.
> >> E.g. make static analysis (do we have?), compile, styles, and javadoc
> >> checks mandatory.
> >>
> >> I think that relaxed requirements here will lead to bad product quality.
> >>
> >> * Agree with Pavel, we should 'keep' integrations tests somehow.
> >> During active development tests will be broken most of time, so,
> >> I'd port them e.g. suite-by-suite once we will have a stable and
> featured
> >> environment to run them and of course make test's code clear and avoid
> >> bad/non-relevant ones.
> >>
> >> * I like bottom-up approach.
> >> With it we could make a better framework. I mean clear component
> lifecycle,
> >> component wiring mechanics, general methods to approach core components
> >> such as exchange/communication
> >> to avoid code mess like we have in ExchangeFuture with all these custom
> >> callbacks for each component, interfaces like
> >> PartitionsExchangeAware, IgniteChangeGlobalStateSupport and
> >> a pack of
> start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected
> >> and so on in various unexpected places.
> >> Hope, we will be able to port most of the good code to the new framework
> >> version.
> >>
> >>
> >>
> >> On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk <
> >> alexey.goncharuk@gmail.com>
> >> wrote:
> >>
> >>> Nikolay, Pavel,
> >>>
> >>> Thanks for the feedback! First of all, I wanted to stress that I do not
> >>> intend to rewrite everything from scratch (I never used this phrase).
> >> There
> >>> are significant parts of code that would be moved with minimal
> >>> modifications. Second, I never said that we will get rid of the old
> tests
> >>> codebase. Every single test from Ignite 2.x should be moved to Ignite 3
> >>> regardless of how we choose to proceed.
> >>>
> >>> My point is that for some parts of the code a clean bottom-up
> >>> implementation will be cheaper in many ways. Let me give you a few
> >> concrete
> >>> examples:
> >>>
> >>> - I think no one can object that we need a cleanly separated
> >> persistence
> >>> layer for Ignite. There is a very raw draft IEP for this already. On
> >> the
> >>> other hand, I think we also can agree that we need a split-brain
> >>> resistant
> >>> replication protocol for caches. There is also an IEP for this.
> >> Neither
> >>> of
> >>> the changes is a good fit for 2.x because they are likely to
> introduce
> >>> breaking changes in the persistence layer, configuration and
> behavior.
> >>> Additionally, these components are now tightly coupled, so there is
> no
> >>> way
> >>> these two changes can be implemented in parallel and then merged
> >>> together
> >>> easily. So what we will end up with is having to implement these
> >> changes
> >>> sequentially, fixing all existing tests twice, and essentially
> >> throwing
> >>> away half of the work done because the other part of the change is
> >>> re-implemented
> >>> - Similar example goes with getting rid of IgniteInternalFuture and
> >>> replacing it with CompletableFuture, and any other change that
> touches
> >>> the
> >>> asynchronous part of the code.
> >>>
> >>> Third, I do not see how this choice affects the UX of Ignite. The end
> >> user
> >>> experience must be fixed in the IEP regardless of the development
> process
> >>> and the fact that we have gaps in this area in Ignite 2.x just confirms
> >>> that.
> >>>
> >>> Pavel, agree that a repo/branch is a technicality, I guess if
> >> reformulate,
> >>> my point is that we might agree to have a single development master
> >> branch
> >>> with 'disassembled' end-to-end functionality for some period of time to
> >>> speed up development, and re-assemble the core features after having
> >>> submodules tested independently.
> >>>
> >>> Nikolay,
> >>>> We have many features that have to evolve.
> >>>> Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
> >>> This is not very specific. In the end, resources are limited and we
> will
> >>> not be able to drive both tracks simultaneously, especially after a
> >> couple
> >>> of features having been implemented for Ignite 3.0. If there are indeed
> >>> some major changes that we want to do in Ignite 2.x instead of putting
> >>> effort into 3.0 - let's discuss them. I am just not aware of any,
> that's
> >>> why I am eager to move forward with Ignite 3.0.
> >>>
> >>>> We have bugs and issues that can be fixed in 2.x without breaking
> >> backward
> >>> compatibility.
> >>>> We have many users who are happy with the 2.x with all it’s issues.
> >>> These changes will be covered by end-to-end tests and migrated to
> Ignite
> >>> 3.0, so I see no issues here.
> >>>
> >>> Finally, Anton & Nikolay
> >>> I do not have an estimate for this simply because the activity is
> >>> community-driven and it depends on the number of people willing to
> >>> contribute. With the current pace, I would hope to have an RC of Ignite
> >> 3.0
> >>> to be ready by the end of 2021. My gut feeling is that by moving with
> >>> incremental changes, we will not be able to implement even half of the
> >>> wishlist by that time.
> >>> I doubt that releasing several major releases with breaking changes
> will
> >>> make Ignite users happy either because each upgrade will cost Ignite
> >> users
> >>> money, so the fewer major versions we release, the better. Thus my wish
> >> to
> >>> include all breaking changes in one release.
> >>>
> >>> I'll be now quiet for a while, let's see what other community members
> >>> think.
> >>>
> >>> пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <pt...@apache.org>:
> >>>
> >>>> 1. Rewriting from scratch is never a good idea.
> >>>> We don't want to follow the path of Netscape and lose all our users
> >>>> by the time we have a working 3.0 [1]
> >>>>
> >>>> 2. Not sure about new repo - seems like some pain and no gain, what's
> >> the
> >>>> problem with a branch?
> >>>>
> >>>> 3. We should keep existing integration tests when possible.
> >>>> We have accumulated a lot of edge case knowledge over the years,
> >>>> it is not a good idea to send all of that down the drain.
> >>>> Yes, integration tests are slow, but they are the most valuable.
> >>>> I think we can move more stuff into nightly runs and have a fast and
> >>> modern
> >>>> basic suite.
> >>>>
> >>>>
> >>>> Alexey, you are much more familiar with the Ignite core codebase than
> >>> most
> >>>> of us,
> >>>> can you please explain in more detail which particular feature, in
> your
> >>>> opinion,
> >>>> mandates this "start from scratch" approach?
> >>>> Is it really not possible at all to follow a less radical way?
> >>>>
> >>>>
> >>>> [1]
> >>>>
> >>>>
> >>>
> >>
> https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
> >>>>
> >>>> On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <ni...@apache.org>
> >>>> wrote:
> >>>>
> >>>>> Hello, Alexey.
> >>>>>
> >>>>> I think that «rewriting from scratch» approach has a high risk to
> >> make
> >>>> new
> >>>>> features unusable.
> >>>>> At the time Ignite2 was started no-one wants to do bad UX or bad
> >>>> features.
> >>>>> Nevertheless, it happen.
> >>>>>
> >>>>> I think we can avoid it with the Ignite3 and successors if we will
> >> move
> >>>>> step by step without keeping backward compatibility
> >>>>> With the step by step approach, we can focus on each component
> >>>> separately.
> >>>>>
> >>>>>> What new features are we planning to implement for Ignite 2.x?
> >>>>>
> >>>>> We have many features that have to evolve.
> >>>>> Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
> >>>>> We have bugs and issues that can be fixed in 2.x without breaking
> >>>> backward
> >>>>> compatibility.
> >>>>> We have many users who are happy with the 2.x with all it’s issues.
> >>>>>
> >>>>>> 2 нояб. 2020 г., в 14:09, Anton Vinogradov <av...@apache.org>
> >>> написал(а):
> >>>>>>
> >>>>>> Alexey,
> >>>>>>
> >>>>>> Do we have any estimates of how fast we'll be able to gain
> >>>>> production-ready
> >>>>>> AI 3.0 in case of a "new repo" choice?
> >>>>>>
> >>>>>> On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk <
> >>>>> alexey.goncharuk@gmail.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Nikolay,
> >>>>>>>
> >>>>>>> What new features are we planning to implement for Ignite 2.x? I
> >>> think
> >>>>> once
> >>>>>>> we commence working on Ignite 3.0, we should gradually cease the
> >>>>> activity
> >>>>>>> on Ignite 2.x to mere bugfixes because such parallel development
> >>> will
> >>>> be
> >>>>>>> overwhelming regardless of how we choose to proceed.
> >>>>>>>
> >>>>>>> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <nizhikov@apache.org
> >>> :
> >>>>>>>
> >>>>>>>> To be clear:
> >>>>>>>>
> >>>>>>>>> I would suggest creating a new repository for Ignite 3.0
> >>> (perhaps, a
> >>>>>>> new
> >>>>>>>> clean branch, but a new repo looks nicer to me) and a new Ignite
> >>> 3.0
> >>>>>>>> TeamCity project.
> >>>>>>>>
> >>>>>>>> +1 for new Team City project.
> >>>>>>>> +1 for new branch for Ignite3.
> >>>>>>>> -1 for new repo.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <
> >> NIzhikov.dev@gmail.com
> >>>>
> >>>>>>>> написал(а):
> >>>>>>>>>
> >>>>>>>>> Hello, Alexey.
> >>>>>>>>>
> >>>>>>>>> I think it will hurt our project more than help.
> >>>>>>>>> Developing new features for 2 separate branches with the
> >> different
> >>>>> APIs
> >>>>>>>> and internal structure is overwhelming
> >>>>>>>>>
> >>>>>>>>> Maybe we should relax a bit requirements for Ignite3?
> >>>>>>>>> Maybe we should move step by step and make Ignite3 with new
> >>>>>>>> configuration than Ignite4 with new transactions, etc?
> >>>>>>>>>
> >>>>>>>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <
> >>>>>>> alexey.goncharuk@gmail.com>
> >>>>>>>> написал(а):
> >>>>>>>>>>
> >>>>>>>>>> Igniters,
> >>>>>>>>>>
> >>>>>>>>>> I wanted to pitch a rather radical idea regarding the Ignite
> >> 3.0
> >>>>>>>>>> development which has occurred to me some time ago.
> >>>>>>>>>>
> >>>>>>>>>> We already have several IEPs targeted to Ignite 3.0 which imply
> >>>> major
> >>>>>>>>>> changes to the codebase (the change in replication protocol and
> >>>> thus
> >>>>>>>>>> transactions, change in binary format, updated metastorage,
> >> etc).
> >>>> We
> >>>>>>>> also
> >>>>>>>>>> planned significant changes in public APIs: configuration
> >> format
> >>>>>>> change,
> >>>>>>>>>> improvements in cache APIs, SQL APIs, transaction mode rework.
> >>> The
> >>>>>>>> wishlist
> >>>>>>>>>> of changes for Ignite 3.0 is huge.
> >>>>>>>>>>
> >>>>>>>>>> So, I was wondering whether it makes sense to try to change the
> >>> old
> >>>>>>>>>> codebase, or start with a new baseline and move old pieces of
> >>> code
> >>>>>>> that
> >>>>>>>> do
> >>>>>>>>>> not require significant rework. Personally, I would go with the
> >>>>> second
> >>>>>>>>>> option for the following reasons:
> >>>>>>>>>>
> >>>>>>>>>> - We have a chance to shift the development paradigm in the
> >>> project
> >>>>>>> and
> >>>>>>>>>> introduce the practice of true unit-tests. In the new baseline
> >> at
> >>>> the
> >>>>>>>>>> beginning there will be no ability to run an end-to-end
> >> scenario,
> >>>>>>> thus
> >>>>>>>> we
> >>>>>>>>>> will be forced to write unit-tests. So far, such practice was
> >>> hard
> >>>> to
> >>>>>>>>>> implement because of tight coupling between Ignite components
> >> and
> >>>>>>>> inability
> >>>>>>>>>> to instantiate components without an instance of KernalContext.
> >>> For
> >>>>>>>>>> example, we should be able to thoroughly test internal
> >>> primitives,
> >>>>>>>> such as
> >>>>>>>>>> replication protocol (without actual communication),
> >> distributed
> >>>>>>>>>> metastorage contracts, persistence layer, etc.
> >>>>>>>>>> - We will significantly reduce the development cycle in the
> >>>> beginning
> >>>>>>>>>> (right now the RunAll takes two hours of astronomical time with
> >>>> empty
> >>>>>>>> TC;
> >>>>>>>>>> in the new approach developer will be able to run ALL tests
> >>> locally
> >>>>>>> in
> >>>>>>>> a
> >>>>>>>>>> matter of minutes)
> >>>>>>>>>> - We can get rid of TC bot and enforce green TC by integrating
> >> TC
> >>>>>>> build
> >>>>>>>>>> results with GitHub PRs (the same way Travis is currently
> >>>> integrated
> >>>>>>>> to PR
> >>>>>>>>>> check). We should restrict PR merge without a TC check
> >>>>>>>>>> - We will still have to re-write all tests, but only once. If
> >> we
> >>>> try
> >>>>>>> to
> >>>>>>>>>> modify the old codebase, we would need to modify all the tests
> >>> for
> >>>>>>>> every
> >>>>>>>>>> major change (public API change, configuration change)
> >>>>>>>>>> - We will have fewer conflicts when working together. For
> >>> example,
> >>>> I
> >>>>>>>>>> cannot imagine how one would merge two changes of getting rid
> >> of
> >>>>>>>>>> IgniteFuture and changes in replication protocol, for example
> >>>>>>>>>>
> >>>>>>>>>> Technically, I would suggest creating a new repository for
> >> Ignite
> >>>> 3.0
> >>>>>>>>>> (perhaps, a new clean branch, but a new repo looks nicer to me)
> >>>> and a
> >>>>>>>> new
> >>>>>>>>>> Ignite 3.0 TeamCity project.
> >>>>>>>>>>
> >>>>>>>>>> While it may seem quite radical, I do believe that this
> >> approach
> >>>> will
> >>>>>>>> give
> >>>>>>>>>> us more benefits than trying to make such major changes in the
> >>>>>>> existing
> >>>>>>>>>> codebase. If needed, let's schedule a discord chat like before
> >> to
> >>>>>>>> discuss
> >>>>>>>>>> this.
> >>>>>>>>>>
> >>>>>>>>>> WDYT?
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>
> >>
> >> --
> >> Best regards,
> >> Andrey V. Mashenkov
> >>
>
>
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Petr Ivanov <mr...@gmail.com>.
Rather strange, cause the repository on GitHub is only a mirror of Apache's GitBox.
Although — I guess Apache applied the same policy to its repositories, if not was it's author...
> On 22 Dec 2020, at 13:34, Pavel Tupitsyn <pt...@apache.org> wrote:
>
> Ivan, it is the new GitHub default
>
> "On Oct. 1, 2020, any new repositories you create will use main as the
> default branch, instead of master" [1]
>
> [1]
> https://www.zdnet.com/article/github-to-replace-master-with-main-starting-next-month/
>
> On Tue, Dec 22, 2020 at 1:12 PM Ivan Pavlukhin <vo...@gmail.com> wrote:
>
>> Also I noticed that ignite-3 repository has "main" but not "master"
>> branch. Who can shed light on this? Did not find an explanation in
>> this thread.
>>
>> 2020-12-22 13:09 GMT+03:00, Ivan Pavlukhin <vo...@gmail.com>:
>>> I noticed some free-from commit messages in ignite-3 repository. I
>>> think we should use ticket-based workflow and commit messages as
>>> usual.
>>>
>>> [1] https://github.com/apache/ignite-3/commits/main
>>>
>>> 2020-12-21 10:55 GMT+03:00, Petr Ivanov <mr...@gmail.com>:
>>>> There is no problem to have both in new repository, if skilled
>> enthusiast
>>>> will take over that job.
>>>>
>>>> I guess we will stick to Maven for time being but development of
>>>> Gradle-based building system can be done in parallel.
>>>> I can even add corresponding development build configurations for
>>>> TeamCity,
>>>> or even introduce some kind of switch — so that we can test old and new
>>>> build approaches and provide seamless transition if we agree on that.
>>>>
>>>>> On 19 Dec 2020, at 01:00, Valentin Kulichenko
>>>>> <va...@gmail.com> wrote:
>>>>>
>>>>> Hi Ivan,
>>>>>
>>>>> There was a very brief discussion around this. Basically, it looks like
>>>>> switching from Maven to something else is not going to bring much
>> value,
>>>>> but at the same time will be quite demanding because the community has
>>>>> much
>>>>> more experience with Maven. However, I would say that it is still
>>>>> debatable at this point -- please feel free to share your thoughts on
>>>>> this.
>>>>>
>>>>> -Val
>>>>>
>>>>> On Thu, Dec 17, 2020 at 10:53 PM Ivan Pavlukhin <vo...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Igniters,
>>>>>>
>>>>>> Forgive me that I am not reading dev list carefully these days. Was it
>>>>>> explicitly decided that Maven should be used as a build system for
>>>>>> Ignite 3? As there is a new repository we possibly can update our
>>>>>> build tools as well. What do you think?
>>>>>>
>>>>>> 2020-12-17 22:45 GMT+03:00, Valentin Kulichenko <
>>>>>> valentin.kulichenko@gmail.com>:
>>>>>>> Hi Dmitriy,
>>>>>>>
>>>>>>> I don't think there is any reason for concern at this point. The
>>>>>> community
>>>>>>> agreed on the scope of the changes for 3.0 - it is described on Wiki.
>>>>>>> The
>>>>>>> scope is quite big, so it is clear that 2.x and 3.x will have to
>> exist
>>>>>>> in
>>>>>>> parallel for a significant amount of time, so we need a place where
>> we
>>>>>> can
>>>>>>> merge the code for 3.x. Thus, I've created this new repo. We already
>>>>>>> have
>>>>>>> multiple IEPs, as well as several contributors who are actively
>>>>>>> involved
>>>>>> in
>>>>>>> development. Some of the first PRs were merged today.
>>>>>>>
>>>>>>> I didn't hear any objections since the repo was created.
>>>>>>>
>>>>>>> -Val
>>>>>>>
>>>>>>> On Wed, Dec 16, 2020 at 7:28 AM Dmitriy Pavlov <dp...@apache.org>
>>>>>> wrote:
>>>>>>>
>>>>>>>> Folks, I'm a little bit concerned about the simultaneous
>>>>>>>> - existence of the repo https://github.com/apache/ignite-3 and PRs
>>>>>>>> for
>>>>>>>> that
>>>>>>>> repo
>>>>>>>> - and a couple of downvotes from PMC members.
>>>>>>>>
>>>>>>>> Is it all fine here? Was there any vote /discussion where it was
>>>>>>>> discussed
>>>>>>>> and consensus approved? What is the status of the ignite-3 repo?
>>>>>>>>
>>>>>>>> вт, 15 дек. 2020 г. в 17:30, Carbone, Adam
>>>>>>>> <Adam.Carbone@bottomline.com
>>>>>>> :
>>>>>>>>
>>>>>>>>> I don't believe Java 7 was LTS, and I hope that others will have
>>>>>>>>> upgraded
>>>>>>>>> long before that. If that is the release timeframe for 3.0, then I
>>>>>>>> suppose
>>>>>>>>> that would makes sense, I would still doubt that people would be
>>>>>>>>> going
>>>>>>>>> newer than java 11, just my opinion of what I'm seeing.
>>>>>>>>>
>>>>>>>>> Regards
>>>>>>>>> ~adam
>>>>>>>>>
>>>>>>>>> Adam Carbone | Director of Innovation – Intelligent Platform Team |
>>>>>>>>> Bottomline Technologies
>>>>>>>>> Office: 603-501-6446 | Mobile: 603-570-8418
>>>>>>>>> www.bottomline.com
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 12/15/20, 4:25 AM, "Ilya Kasnacheev" <
>> ilya.kasnacheev@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Hello!
>>>>>>>>>
>>>>>>>>> I guess Ignite 3.0 will be ready for production use somewhere in
>>>>>>>> 2022,
>>>>>>>>> realistically. By that time, Java 8 will be long enough out of
>>>>>>>> support
>>>>>>>>> so
>>>>>>>>> that most companies will actually forbid its use, WRT
>>>>>>>>> vulnerabilities
>>>>>>>>> et
>>>>>>>>> all.
>>>>>>>>>
>>>>>>>>> After all we have managed to upgrade from Java 7 to Java 8 and
>>>>>> only
>>>>>>>>> got a
>>>>>>>>> minor amount of complaints.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> --
>>>>>>>>> Ilya Kasnacheev
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> пн, 14 дек. 2020 г. в 19:06, Carbone, Adam <
>>>>>>>>> Adam.Carbone@bottomline.com>:
>>>>>>>>>
>>>>>>>>>> So just one bit to consider... Are there features that you need
>>>>>>>>> to
>>>>>>>>> use in
>>>>>>>>>> these newer versions of java? Or are we just updating to stay
>>>>>>>>> current? The
>>>>>>>>>> reason I ask is that there are still lots of people in an
>>>>>>>> enterprise
>>>>>>>>> space
>>>>>>>>>> that are beholden to having to support legacy JAVAEE supported
>>>>>>>>> applications
>>>>>>>>>> on Websphere, Weblogic, Redhat, etc... and the organizations
>>>>>> that
>>>>>>>>> use those
>>>>>>>>>> platforms are slow to move... Most of them are still on Java8
>>>>>>>>>>
>>>>>>>>>> So as a platform I think a strong consideration needs to be
>>>>>>>>> towards
>>>>>>>>> having
>>>>>>>>>> the broadest possible support profile until it becomes an
>>>>>>>> impediment
>>>>>>>>> to
>>>>>>>>>> doing things that the platform needs. So far I haven't seen huge
>>>>>>>>> things in
>>>>>>>>>> the newer versions of java that are must haves ( a few
>>>>>> exceptions
>>>>>>>> are
>>>>>>>>>> things that would be really nice to take advantage of ).
>>>>>>>>>>
>>>>>>>>>> I think that apache commons has taken the right approach by
>>>>>>>>> staying
>>>>>>>>> on
>>>>>>>>>> java 8 giving the largest possible user base.
>>>>>>>>>>
>>>>>>>>>> Even standardizing on java 11 would have to make us reconsider
>>>>>>>>> Ignite as
>>>>>>>>>> the platform we are using, we are not so invested in it as of
>>>>>>>>> now,
>>>>>>>>> even
>>>>>>>>>> though we have big plans to leverage it. Just because we aren't
>>>>>>>> sure
>>>>>>>>> when
>>>>>>>>>> we are going to be able to upgrade from java8. It has support
>>>>>>>>> through 2022,
>>>>>>>>>> so I imagine that is when we will be discussing that.
>>>>>>>>>>
>>>>>>>>>> Regards
>>>>>>>>>>
>>>>>>>>>> ~Adam
>>>>>>>>>>
>>>>>>>>>> Adam Carbone | Director of Innovation – Intelligent Platform
>>>>>> Team
>>>>>>>>> |
>>>>>>>>>> Bottomline Technologies
>>>>>>>>>> Office: 603-501-6446 | Mobile: 603-570-8418
>>>>>>>>>> www.bottomline.com
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 11/24/20, 7:38 AM, "Alexey Zinoviev" <zaleslaw.sin@gmail.com
>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Java 15 is better, VarHandles, ForeignMemory access and so
>>>>>>>>> on.
>>>>>>>>>>
>>>>>>>>>> In both cases, I support the Java version 11 and higher for
>>>>>>>>> the
>>>>>>>>>> development!
>>>>>>>>>>
>>>>>>>>>> вт, 24 нояб. 2020 г. в 15:21, Andrey Mashenkov <
>>>>>>>>>> andrey.mashenkov@gmail.com>:
>>>>>>>>>>
>>>>>>>>>>> Let's add maven plugins for sanity checks at the early
>>>>>>>> stage.
>>>>>>>>>>> I've created a ticket for this [1].
>>>>>>>>>>>
>>>>>>>>>>> Also, I've found initial pom.xml has a target version Java
>>>>>>>>> 8.
>>>>>>>>>>> Do we intend to move to Java 11 (or may be next LTS) and
>>>>>>>>> drop
>>>>>>>>> Java 8
>>>>>>>>>> in
>>>>>>>>>>> Ignite 3.0?
>>>>>>>>>>>
>>>>>>>>>>> [1]
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>> https://urldefense.com/v3/__https://issues.apache.org/jira/browse/IGNITE-13751__;!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKAO2Ejs8$
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Nov 24, 2020 at 5:40 AM Valentin Kulichenko <
>>>>>>>>>>> valentin.kulichenko@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Folks,
>>>>>>>>>>>>
>>>>>>>>>>>> I went ahead and created the repository [1]. I also
>>>>>>>>> configured a
>>>>>>>>>> TeamCity
>>>>>>>>>>>> project [2] that runs all available JUnit tests on every
>>>>>>>>> PR
>>>>>>>>>> creation or
>>>>>>>>>>>> update. It also sends the status update to GitHub so
>>>>>> that
>>>>>>>>> it's
>>>>>>>>>> reflected
>>>>>>>>>>> in
>>>>>>>>>>>> the PR itself so that we can do merges directly from
>>>>>>>> GitHub.
>>>>>>>>> Basic
>>>>>>>>>> steps
>>>>>>>>>>> to
>>>>>>>>>>>> make a change are described on the Wiki page [3].
>>>>>>>>>>>>
>>>>>>>>>>>> Let me know if you have any questions.
>>>>>>>>>>>>
>>>>>>>>>>>> [1]
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>> https://urldefense.com/v3/__https://github.com/apache/ignite-3__;!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKIq24lxF$
>>>>>>>>>>>> [2]
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>> https://urldefense.com/v3/__https://ci.ignite.apache.org/project/ignite3__;!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKFGL_oJx$
>>>>>>>>>>>> [3]
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/IGNITE/Apache*Ignite*3.0*ApacheIgnite3.0-DevelopmentProcess__;Kysj!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKNhWzQ0s$
>>>>>>>>>>>>
>>>>>>>>>>>> -Val
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Nov 18, 2020 at 4:24 PM Valentin Kulichenko <
>>>>>>>>>>>> valentin.kulichenko@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks, guys. It looks like we are much closer to the
>>>>>>>>> consensus
>>>>>>>>>> now. I
>>>>>>>>>>>>> totally on board with the plan, but I would also like
>>>>>>>>> to
>>>>>>>>> address
>>>>>>>>>> the
>>>>>>>>>>>>> short-term needs. As I've already mentioned earlier,
>>>>>>>> there
>>>>>>>>> are
>>>>>>>>>> several
>>>>>>>>>>>>> active IEPs, but we still don't have even a
>>>>>> preliminary
>>>>>>>>> technical
>>>>>>>>>>> process
>>>>>>>>>>>>> for working on these IEPs. I believe this might be
>>>>>>>>> frustrating
>>>>>>>>>> for the
>>>>>>>>>>>>> folks who would like to commit code.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The scope we agreed on is quite big, and it will
>>>>>> surely
>>>>>>>>> take
>>>>>>>>>>> significant
>>>>>>>>>>>>> time to implement all the changes and stabilize them.
>>>>>>>>> Therefore,
>>>>>>>>>> it's
>>>>>>>>>>>> clear
>>>>>>>>>>>>> to me that we will have to maintain 2.x and 3.x in
>>>>>>>>> parallel for
>>>>>>>>>> quite
>>>>>>>>>>>> some
>>>>>>>>>>>>> time - this needs to be addressed somehow. I'm
>>>>>>>>> convinced
>>>>>>>>> that
>>>>>>>>>> having a
>>>>>>>>>>>>> separate repo is the ONLY way to do that, and so far,
>>>>>> I
>>>>>>>>> haven't
>>>>>>>>>> heard
>>>>>>>>>>> any
>>>>>>>>>>>>> clear alternatives or reasons why we shouldn't do
>>>>>> this.
>>>>>>>>>>>>>
>>>>>>>>>>>>> That said, I'm inclined to proceed with this in the
>>>>>>>>> next
>>>>>>>>> few
>>>>>>>>>> days - I
>>>>>>>>>>>> will
>>>>>>>>>>>>> create a repo and describe the process (which we, of
>>>>>>>>> course, can
>>>>>>>>>>> discuss
>>>>>>>>>>>>> and modify going forward). Let's, at the very least,
>>>>>>>>> try
>>>>>>>>> and see
>>>>>>>>>> where
>>>>>>>>>>> it
>>>>>>>>>>>>> leads us.
>>>>>>>>>>>>>
>>>>>>>>>>>>> If someone has any concrete alternative options on how
>>>>>>>>> to
>>>>>>>>> we can
>>>>>>>>>>> maintain
>>>>>>>>>>>>> two major versions in parallel, let's have another
>>>>>>>>> voice
>>>>>>>>>> discussion
>>>>>>>>>>> this
>>>>>>>>>>>>> Friday. If we do the meeting, we should set it up with
>>>>>>>>> a
>>>>>>>>> clear
>>>>>>>>>> goal to
>>>>>>>>>>>> make
>>>>>>>>>>>>> a decision. Please let me know if there is interest in
>>>>>>>>> this.
>>>>>>>>>>>>>
>>>>>>>>>>>>> -Val
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Nov 16, 2020 at 6:31 AM Alexey Goncharuk <
>>>>>>>>>>>>> alexey.goncharuk@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Good,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I think we have an intermediate agreement on the
>>>>>> scope
>>>>>>>> and
>>>>>>>>>>> significance
>>>>>>>>>>>> of
>>>>>>>>>>>>>> the changes we want to make. I suggest creating
>>>>>>>>> separate
>>>>>>>>>> discussion
>>>>>>>>>>>>>> streams
>>>>>>>>>>>>>> and calls for each of the suggested topics so that:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> - It is clear for the community what is the
>>>>>>>> motivation
>>>>>>>>> of the
>>>>>>>>>>> stream
>>>>>>>>>>>>>> (this includes both functional targets and
>>>>>>>>> technical
>>>>>>>>> debt
>>>>>>>>>> issues
>>>>>>>>>>>>>> pointed
>>>>>>>>>>>>>> out by Sergey)
>>>>>>>>>>>>>> - Who is planning to take an active part in each
>>>>>> of
>>>>>>>> the
>>>>>>>>>> streams
>>>>>>>>>>> (i.e.
>>>>>>>>>>>>>> the 'design committee', as Sergey suggested)
>>>>>>>>>>>>>> - What are the intermediate and final goals for
>>>>>>>>> each
>>>>>>>>> of the
>>>>>>>>>> streams
>>>>>>>>>>>>>> - What are the cross-stream interactions and how
>>>>>> we
>>>>>>>>>> integrate them
>>>>>>>>>>>>>> - How each of the streams will be integrated with
>>>>>>>>> the
>>>>>>>>> current
>>>>>>>>>>>> codebase
>>>>>>>>>>>>>> based on the above (here is where we will see
>>>>>>>>> whether
>>>>>>>>>> drop-in or
>>>>>>>>>>>>>> incremental approaches make more sense)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Best regards,
>>>>>>>>>>> Andrey V. Mashenkov
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Best regards,
>>>>>> Ivan Pavlukhin
>>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> Best regards,
>>> Ivan Pavlukhin
>>>
>>
>>
>> --
>>
>> Best regards,
>> Ivan Pavlukhin
>>
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Ivan Pavlukhin <vo...@gmail.com>.
Pavel, thanks for explanation!
2020-12-22 13:34 GMT+03:00, Pavel Tupitsyn <pt...@apache.org>:
> Ivan, it is the new GitHub default
>
> "On Oct. 1, 2020, any new repositories you create will use main as the
> default branch, instead of master" [1]
>
> [1]
> https://www.zdnet.com/article/github-to-replace-master-with-main-starting-next-month/
>
> On Tue, Dec 22, 2020 at 1:12 PM Ivan Pavlukhin <vo...@gmail.com> wrote:
>
>> Also I noticed that ignite-3 repository has "main" but not "master"
>> branch. Who can shed light on this? Did not find an explanation in
>> this thread.
>>
>> 2020-12-22 13:09 GMT+03:00, Ivan Pavlukhin <vo...@gmail.com>:
>> > I noticed some free-from commit messages in ignite-3 repository. I
>> > think we should use ticket-based workflow and commit messages as
>> > usual.
>> >
>> > [1] https://github.com/apache/ignite-3/commits/main
>> >
>> > 2020-12-21 10:55 GMT+03:00, Petr Ivanov <mr...@gmail.com>:
>> >> There is no problem to have both in new repository, if skilled
>> enthusiast
>> >> will take over that job.
>> >>
>> >> I guess we will stick to Maven for time being but development of
>> >> Gradle-based building system can be done in parallel.
>> >> I can even add corresponding development build configurations for
>> >> TeamCity,
>> >> or even introduce some kind of switch — so that we can test old and
>> >> new
>> >> build approaches and provide seamless transition if we agree on that.
>> >>
>> >>> On 19 Dec 2020, at 01:00, Valentin Kulichenko
>> >>> <va...@gmail.com> wrote:
>> >>>
>> >>> Hi Ivan,
>> >>>
>> >>> There was a very brief discussion around this. Basically, it looks
>> >>> like
>> >>> switching from Maven to something else is not going to bring much
>> value,
>> >>> but at the same time will be quite demanding because the community
>> >>> has
>> >>> much
>> >>> more experience with Maven. However, I would say that it is still
>> >>> debatable at this point -- please feel free to share your thoughts on
>> >>> this.
>> >>>
>> >>> -Val
>> >>>
>> >>> On Thu, Dec 17, 2020 at 10:53 PM Ivan Pavlukhin <vo...@gmail.com>
>> >>> wrote:
>> >>>
>> >>>> Hi Igniters,
>> >>>>
>> >>>> Forgive me that I am not reading dev list carefully these days. Was
>> >>>> it
>> >>>> explicitly decided that Maven should be used as a build system for
>> >>>> Ignite 3? As there is a new repository we possibly can update our
>> >>>> build tools as well. What do you think?
>> >>>>
>> >>>> 2020-12-17 22:45 GMT+03:00, Valentin Kulichenko <
>> >>>> valentin.kulichenko@gmail.com>:
>> >>>>> Hi Dmitriy,
>> >>>>>
>> >>>>> I don't think there is any reason for concern at this point. The
>> >>>> community
>> >>>>> agreed on the scope of the changes for 3.0 - it is described on
>> >>>>> Wiki.
>> >>>>> The
>> >>>>> scope is quite big, so it is clear that 2.x and 3.x will have to
>> exist
>> >>>>> in
>> >>>>> parallel for a significant amount of time, so we need a place where
>> we
>> >>>> can
>> >>>>> merge the code for 3.x. Thus, I've created this new repo. We
>> >>>>> already
>> >>>>> have
>> >>>>> multiple IEPs, as well as several contributors who are actively
>> >>>>> involved
>> >>>> in
>> >>>>> development. Some of the first PRs were merged today.
>> >>>>>
>> >>>>> I didn't hear any objections since the repo was created.
>> >>>>>
>> >>>>> -Val
>> >>>>>
>> >>>>> On Wed, Dec 16, 2020 at 7:28 AM Dmitriy Pavlov <dp...@apache.org>
>> >>>> wrote:
>> >>>>>
>> >>>>>> Folks, I'm a little bit concerned about the simultaneous
>> >>>>>> - existence of the repo https://github.com/apache/ignite-3 and PRs
>> >>>>>> for
>> >>>>>> that
>> >>>>>> repo
>> >>>>>> - and a couple of downvotes from PMC members.
>> >>>>>>
>> >>>>>> Is it all fine here? Was there any vote /discussion where it was
>> >>>>>> discussed
>> >>>>>> and consensus approved? What is the status of the ignite-3 repo?
>> >>>>>>
>> >>>>>> вт, 15 дек. 2020 г. в 17:30, Carbone, Adam
>> >>>>>> <Adam.Carbone@bottomline.com
>> >>>>> :
>> >>>>>>
>> >>>>>>> I don't believe Java 7 was LTS, and I hope that others will have
>> >>>>>>> upgraded
>> >>>>>>> long before that. If that is the release timeframe for 3.0, then
>> >>>>>>> I
>> >>>>>> suppose
>> >>>>>>> that would makes sense, I would still doubt that people would be
>> >>>>>>> going
>> >>>>>>> newer than java 11, just my opinion of what I'm seeing.
>> >>>>>>>
>> >>>>>>> Regards
>> >>>>>>> ~adam
>> >>>>>>>
>> >>>>>>> Adam Carbone | Director of Innovation – Intelligent Platform Team
>> >>>>>>> |
>> >>>>>>> Bottomline Technologies
>> >>>>>>> Office: 603-501-6446 | Mobile: 603-570-8418
>> >>>>>>> www.bottomline.com
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> On 12/15/20, 4:25 AM, "Ilya Kasnacheev" <
>> ilya.kasnacheev@gmail.com>
>> >>>>>>> wrote:
>> >>>>>>>
>> >>>>>>> Hello!
>> >>>>>>>
>> >>>>>>> I guess Ignite 3.0 will be ready for production use somewhere
>> >>>>>>> in
>> >>>>>> 2022,
>> >>>>>>> realistically. By that time, Java 8 will be long enough out of
>> >>>>>> support
>> >>>>>>> so
>> >>>>>>> that most companies will actually forbid its use, WRT
>> >>>>>>> vulnerabilities
>> >>>>>>> et
>> >>>>>>> all.
>> >>>>>>>
>> >>>>>>> After all we have managed to upgrade from Java 7 to Java 8 and
>> >>>> only
>> >>>>>>> got a
>> >>>>>>> minor amount of complaints.
>> >>>>>>>
>> >>>>>>> Regards,
>> >>>>>>> --
>> >>>>>>> Ilya Kasnacheev
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> пн, 14 дек. 2020 г. в 19:06, Carbone, Adam <
>> >>>>>>> Adam.Carbone@bottomline.com>:
>> >>>>>>>
>> >>>>>>>> So just one bit to consider... Are there features that you need
>> >>>>>>> to
>> >>>>>>> use in
>> >>>>>>>> these newer versions of java? Or are we just updating to stay
>> >>>>>>> current? The
>> >>>>>>>> reason I ask is that there are still lots of people in an
>> >>>>>> enterprise
>> >>>>>>> space
>> >>>>>>>> that are beholden to having to support legacy JAVAEE supported
>> >>>>>>> applications
>> >>>>>>>> on Websphere, Weblogic, Redhat, etc... and the organizations
>> >>>> that
>> >>>>>>> use those
>> >>>>>>>> platforms are slow to move... Most of them are still on Java8
>> >>>>>>>>
>> >>>>>>>> So as a platform I think a strong consideration needs to be
>> >>>>>>> towards
>> >>>>>>> having
>> >>>>>>>> the broadest possible support profile until it becomes an
>> >>>>>> impediment
>> >>>>>>> to
>> >>>>>>>> doing things that the platform needs. So far I haven't seen huge
>> >>>>>>> things in
>> >>>>>>>> the newer versions of java that are must haves ( a few
>> >>>> exceptions
>> >>>>>> are
>> >>>>>>>> things that would be really nice to take advantage of ).
>> >>>>>>>>
>> >>>>>>>> I think that apache commons has taken the right approach by
>> >>>>>>> staying
>> >>>>>>> on
>> >>>>>>>> java 8 giving the largest possible user base.
>> >>>>>>>>
>> >>>>>>>> Even standardizing on java 11 would have to make us reconsider
>> >>>>>>> Ignite as
>> >>>>>>>> the platform we are using, we are not so invested in it as of
>> >>>>>>> now,
>> >>>>>>> even
>> >>>>>>>> though we have big plans to leverage it. Just because we aren't
>> >>>>>> sure
>> >>>>>>> when
>> >>>>>>>> we are going to be able to upgrade from java8. It has support
>> >>>>>>> through 2022,
>> >>>>>>>> so I imagine that is when we will be discussing that.
>> >>>>>>>>
>> >>>>>>>> Regards
>> >>>>>>>>
>> >>>>>>>> ~Adam
>> >>>>>>>>
>> >>>>>>>> Adam Carbone | Director of Innovation – Intelligent Platform
>> >>>> Team
>> >>>>>>> |
>> >>>>>>>> Bottomline Technologies
>> >>>>>>>> Office: 603-501-6446 | Mobile: 603-570-8418
>> >>>>>>>> www.bottomline.com
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> On 11/24/20, 7:38 AM, "Alexey Zinoviev" <zaleslaw.sin@gmail.com
>> >>>>>
>> >>>>>>> wrote:
>> >>>>>>>>
>> >>>>>>>> Java 15 is better, VarHandles, ForeignMemory access and so
>> >>>>>>> on.
>> >>>>>>>>
>> >>>>>>>> In both cases, I support the Java version 11 and higher for
>> >>>>>>> the
>> >>>>>>>> development!
>> >>>>>>>>
>> >>>>>>>> вт, 24 нояб. 2020 г. в 15:21, Andrey Mashenkov <
>> >>>>>>>> andrey.mashenkov@gmail.com>:
>> >>>>>>>>
>> >>>>>>>>> Let's add maven plugins for sanity checks at the early
>> >>>>>> stage.
>> >>>>>>>>> I've created a ticket for this [1].
>> >>>>>>>>>
>> >>>>>>>>> Also, I've found initial pom.xml has a target version Java
>> >>>>>>> 8.
>> >>>>>>>>> Do we intend to move to Java 11 (or may be next LTS) and
>> >>>>>>> drop
>> >>>>>>> Java 8
>> >>>>>>>> in
>> >>>>>>>>> Ignite 3.0?
>> >>>>>>>>>
>> >>>>>>>>> [1]
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>
>> https://urldefense.com/v3/__https://issues.apache.org/jira/browse/IGNITE-13751__;!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKAO2Ejs8$
>> >>>>>>>>>
>> >>>>>>>>> On Tue, Nov 24, 2020 at 5:40 AM Valentin Kulichenko <
>> >>>>>>>>> valentin.kulichenko@gmail.com> wrote:
>> >>>>>>>>>
>> >>>>>>>>>> Folks,
>> >>>>>>>>>>
>> >>>>>>>>>> I went ahead and created the repository [1]. I also
>> >>>>>>> configured a
>> >>>>>>>> TeamCity
>> >>>>>>>>>> project [2] that runs all available JUnit tests on every
>> >>>>>>> PR
>> >>>>>>>> creation or
>> >>>>>>>>>> update. It also sends the status update to GitHub so
>> >>>> that
>> >>>>>>> it's
>> >>>>>>>> reflected
>> >>>>>>>>> in
>> >>>>>>>>>> the PR itself so that we can do merges directly from
>> >>>>>> GitHub.
>> >>>>>>> Basic
>> >>>>>>>> steps
>> >>>>>>>>> to
>> >>>>>>>>>> make a change are described on the Wiki page [3].
>> >>>>>>>>>>
>> >>>>>>>>>> Let me know if you have any questions.
>> >>>>>>>>>>
>> >>>>>>>>>> [1]
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>
>> https://urldefense.com/v3/__https://github.com/apache/ignite-3__;!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKIq24lxF$
>> >>>>>>>>>> [2]
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>
>> https://urldefense.com/v3/__https://ci.ignite.apache.org/project/ignite3__;!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKFGL_oJx$
>> >>>>>>>>>> [3]
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>
>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/IGNITE/Apache*Ignite*3.0*ApacheIgnite3.0-DevelopmentProcess__;Kysj!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKNhWzQ0s$
>> >>>>>>>>>>
>> >>>>>>>>>> -Val
>> >>>>>>>>>>
>> >>>>>>>>>> On Wed, Nov 18, 2020 at 4:24 PM Valentin Kulichenko <
>> >>>>>>>>>> valentin.kulichenko@gmail.com> wrote:
>> >>>>>>>>>>
>> >>>>>>>>>>> Thanks, guys. It looks like we are much closer to the
>> >>>>>>> consensus
>> >>>>>>>> now. I
>> >>>>>>>>>>> totally on board with the plan, but I would also like
>> >>>>>>> to
>> >>>>>>> address
>> >>>>>>>> the
>> >>>>>>>>>>> short-term needs. As I've already mentioned earlier,
>> >>>>>> there
>> >>>>>>> are
>> >>>>>>>> several
>> >>>>>>>>>>> active IEPs, but we still don't have even a
>> >>>> preliminary
>> >>>>>>> technical
>> >>>>>>>>> process
>> >>>>>>>>>>> for working on these IEPs. I believe this might be
>> >>>>>>> frustrating
>> >>>>>>>> for the
>> >>>>>>>>>>> folks who would like to commit code.
>> >>>>>>>>>>>
>> >>>>>>>>>>> The scope we agreed on is quite big, and it will
>> >>>> surely
>> >>>>>>> take
>> >>>>>>>>> significant
>> >>>>>>>>>>> time to implement all the changes and stabilize them.
>> >>>>>>> Therefore,
>> >>>>>>>> it's
>> >>>>>>>>>> clear
>> >>>>>>>>>>> to me that we will have to maintain 2.x and 3.x in
>> >>>>>>> parallel for
>> >>>>>>>> quite
>> >>>>>>>>>> some
>> >>>>>>>>>>> time - this needs to be addressed somehow. I'm
>> >>>>>>> convinced
>> >>>>>>> that
>> >>>>>>>> having a
>> >>>>>>>>>>> separate repo is the ONLY way to do that, and so far,
>> >>>> I
>> >>>>>>> haven't
>> >>>>>>>> heard
>> >>>>>>>>> any
>> >>>>>>>>>>> clear alternatives or reasons why we shouldn't do
>> >>>> this.
>> >>>>>>>>>>>
>> >>>>>>>>>>> That said, I'm inclined to proceed with this in the
>> >>>>>>> next
>> >>>>>>> few
>> >>>>>>>> days - I
>> >>>>>>>>>> will
>> >>>>>>>>>>> create a repo and describe the process (which we, of
>> >>>>>>> course, can
>> >>>>>>>>> discuss
>> >>>>>>>>>>> and modify going forward). Let's, at the very least,
>> >>>>>>> try
>> >>>>>>> and see
>> >>>>>>>> where
>> >>>>>>>>> it
>> >>>>>>>>>>> leads us.
>> >>>>>>>>>>>
>> >>>>>>>>>>> If someone has any concrete alternative options on how
>> >>>>>>> to
>> >>>>>>> we can
>> >>>>>>>>> maintain
>> >>>>>>>>>>> two major versions in parallel, let's have another
>> >>>>>>> voice
>> >>>>>>>> discussion
>> >>>>>>>>> this
>> >>>>>>>>>>> Friday. If we do the meeting, we should set it up with
>> >>>>>>> a
>> >>>>>>> clear
>> >>>>>>>> goal to
>> >>>>>>>>>> make
>> >>>>>>>>>>> a decision. Please let me know if there is interest in
>> >>>>>>> this.
>> >>>>>>>>>>>
>> >>>>>>>>>>> -Val
>> >>>>>>>>>>>
>> >>>>>>>>>>> On Mon, Nov 16, 2020 at 6:31 AM Alexey Goncharuk <
>> >>>>>>>>>>> alexey.goncharuk@gmail.com> wrote:
>> >>>>>>>>>>>
>> >>>>>>>>>>>> Good,
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> I think we have an intermediate agreement on the
>> >>>> scope
>> >>>>>> and
>> >>>>>>>>> significance
>> >>>>>>>>>> of
>> >>>>>>>>>>>> the changes we want to make. I suggest creating
>> >>>>>>> separate
>> >>>>>>>> discussion
>> >>>>>>>>>>>> streams
>> >>>>>>>>>>>> and calls for each of the suggested topics so that:
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> - It is clear for the community what is the
>> >>>>>> motivation
>> >>>>>>> of the
>> >>>>>>>>> stream
>> >>>>>>>>>>>> (this includes both functional targets and
>> >>>>>>> technical
>> >>>>>>> debt
>> >>>>>>>> issues
>> >>>>>>>>>>>> pointed
>> >>>>>>>>>>>> out by Sergey)
>> >>>>>>>>>>>> - Who is planning to take an active part in each
>> >>>> of
>> >>>>>> the
>> >>>>>>>> streams
>> >>>>>>>>> (i.e.
>> >>>>>>>>>>>> the 'design committee', as Sergey suggested)
>> >>>>>>>>>>>> - What are the intermediate and final goals for
>> >>>>>>> each
>> >>>>>>> of the
>> >>>>>>>> streams
>> >>>>>>>>>>>> - What are the cross-stream interactions and how
>> >>>> we
>> >>>>>>>> integrate them
>> >>>>>>>>>>>> - How each of the streams will be integrated with
>> >>>>>>> the
>> >>>>>>> current
>> >>>>>>>>>> codebase
>> >>>>>>>>>>>> based on the above (here is where we will see
>> >>>>>>> whether
>> >>>>>>>> drop-in or
>> >>>>>>>>>>>> incremental approaches make more sense)
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> --
>> >>>>>>>>> Best regards,
>> >>>>>>>>> Andrey V. Mashenkov
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>>
>> >>>> --
>> >>>>
>> >>>> Best regards,
>> >>>> Ivan Pavlukhin
>> >>>>
>> >>
>> >>
>> >
>> >
>> > --
>> >
>> > Best regards,
>> > Ivan Pavlukhin
>> >
>>
>>
>> --
>>
>> Best regards,
>> Ivan Pavlukhin
>>
>
--
Best regards,
Ivan Pavlukhin
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Pavel Tupitsyn <pt...@apache.org>.
Ivan, it is the new GitHub default
"On Oct. 1, 2020, any new repositories you create will use main as the
default branch, instead of master" [1]
[1]
https://www.zdnet.com/article/github-to-replace-master-with-main-starting-next-month/
On Tue, Dec 22, 2020 at 1:12 PM Ivan Pavlukhin <vo...@gmail.com> wrote:
> Also I noticed that ignite-3 repository has "main" but not "master"
> branch. Who can shed light on this? Did not find an explanation in
> this thread.
>
> 2020-12-22 13:09 GMT+03:00, Ivan Pavlukhin <vo...@gmail.com>:
> > I noticed some free-from commit messages in ignite-3 repository. I
> > think we should use ticket-based workflow and commit messages as
> > usual.
> >
> > [1] https://github.com/apache/ignite-3/commits/main
> >
> > 2020-12-21 10:55 GMT+03:00, Petr Ivanov <mr...@gmail.com>:
> >> There is no problem to have both in new repository, if skilled
> enthusiast
> >> will take over that job.
> >>
> >> I guess we will stick to Maven for time being but development of
> >> Gradle-based building system can be done in parallel.
> >> I can even add corresponding development build configurations for
> >> TeamCity,
> >> or even introduce some kind of switch — so that we can test old and new
> >> build approaches and provide seamless transition if we agree on that.
> >>
> >>> On 19 Dec 2020, at 01:00, Valentin Kulichenko
> >>> <va...@gmail.com> wrote:
> >>>
> >>> Hi Ivan,
> >>>
> >>> There was a very brief discussion around this. Basically, it looks like
> >>> switching from Maven to something else is not going to bring much
> value,
> >>> but at the same time will be quite demanding because the community has
> >>> much
> >>> more experience with Maven. However, I would say that it is still
> >>> debatable at this point -- please feel free to share your thoughts on
> >>> this.
> >>>
> >>> -Val
> >>>
> >>> On Thu, Dec 17, 2020 at 10:53 PM Ivan Pavlukhin <vo...@gmail.com>
> >>> wrote:
> >>>
> >>>> Hi Igniters,
> >>>>
> >>>> Forgive me that I am not reading dev list carefully these days. Was it
> >>>> explicitly decided that Maven should be used as a build system for
> >>>> Ignite 3? As there is a new repository we possibly can update our
> >>>> build tools as well. What do you think?
> >>>>
> >>>> 2020-12-17 22:45 GMT+03:00, Valentin Kulichenko <
> >>>> valentin.kulichenko@gmail.com>:
> >>>>> Hi Dmitriy,
> >>>>>
> >>>>> I don't think there is any reason for concern at this point. The
> >>>> community
> >>>>> agreed on the scope of the changes for 3.0 - it is described on Wiki.
> >>>>> The
> >>>>> scope is quite big, so it is clear that 2.x and 3.x will have to
> exist
> >>>>> in
> >>>>> parallel for a significant amount of time, so we need a place where
> we
> >>>> can
> >>>>> merge the code for 3.x. Thus, I've created this new repo. We already
> >>>>> have
> >>>>> multiple IEPs, as well as several contributors who are actively
> >>>>> involved
> >>>> in
> >>>>> development. Some of the first PRs were merged today.
> >>>>>
> >>>>> I didn't hear any objections since the repo was created.
> >>>>>
> >>>>> -Val
> >>>>>
> >>>>> On Wed, Dec 16, 2020 at 7:28 AM Dmitriy Pavlov <dp...@apache.org>
> >>>> wrote:
> >>>>>
> >>>>>> Folks, I'm a little bit concerned about the simultaneous
> >>>>>> - existence of the repo https://github.com/apache/ignite-3 and PRs
> >>>>>> for
> >>>>>> that
> >>>>>> repo
> >>>>>> - and a couple of downvotes from PMC members.
> >>>>>>
> >>>>>> Is it all fine here? Was there any vote /discussion where it was
> >>>>>> discussed
> >>>>>> and consensus approved? What is the status of the ignite-3 repo?
> >>>>>>
> >>>>>> вт, 15 дек. 2020 г. в 17:30, Carbone, Adam
> >>>>>> <Adam.Carbone@bottomline.com
> >>>>> :
> >>>>>>
> >>>>>>> I don't believe Java 7 was LTS, and I hope that others will have
> >>>>>>> upgraded
> >>>>>>> long before that. If that is the release timeframe for 3.0, then I
> >>>>>> suppose
> >>>>>>> that would makes sense, I would still doubt that people would be
> >>>>>>> going
> >>>>>>> newer than java 11, just my opinion of what I'm seeing.
> >>>>>>>
> >>>>>>> Regards
> >>>>>>> ~adam
> >>>>>>>
> >>>>>>> Adam Carbone | Director of Innovation – Intelligent Platform Team |
> >>>>>>> Bottomline Technologies
> >>>>>>> Office: 603-501-6446 | Mobile: 603-570-8418
> >>>>>>> www.bottomline.com
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On 12/15/20, 4:25 AM, "Ilya Kasnacheev" <
> ilya.kasnacheev@gmail.com>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>> Hello!
> >>>>>>>
> >>>>>>> I guess Ignite 3.0 will be ready for production use somewhere in
> >>>>>> 2022,
> >>>>>>> realistically. By that time, Java 8 will be long enough out of
> >>>>>> support
> >>>>>>> so
> >>>>>>> that most companies will actually forbid its use, WRT
> >>>>>>> vulnerabilities
> >>>>>>> et
> >>>>>>> all.
> >>>>>>>
> >>>>>>> After all we have managed to upgrade from Java 7 to Java 8 and
> >>>> only
> >>>>>>> got a
> >>>>>>> minor amount of complaints.
> >>>>>>>
> >>>>>>> Regards,
> >>>>>>> --
> >>>>>>> Ilya Kasnacheev
> >>>>>>>
> >>>>>>>
> >>>>>>> пн, 14 дек. 2020 г. в 19:06, Carbone, Adam <
> >>>>>>> Adam.Carbone@bottomline.com>:
> >>>>>>>
> >>>>>>>> So just one bit to consider... Are there features that you need
> >>>>>>> to
> >>>>>>> use in
> >>>>>>>> these newer versions of java? Or are we just updating to stay
> >>>>>>> current? The
> >>>>>>>> reason I ask is that there are still lots of people in an
> >>>>>> enterprise
> >>>>>>> space
> >>>>>>>> that are beholden to having to support legacy JAVAEE supported
> >>>>>>> applications
> >>>>>>>> on Websphere, Weblogic, Redhat, etc... and the organizations
> >>>> that
> >>>>>>> use those
> >>>>>>>> platforms are slow to move... Most of them are still on Java8
> >>>>>>>>
> >>>>>>>> So as a platform I think a strong consideration needs to be
> >>>>>>> towards
> >>>>>>> having
> >>>>>>>> the broadest possible support profile until it becomes an
> >>>>>> impediment
> >>>>>>> to
> >>>>>>>> doing things that the platform needs. So far I haven't seen huge
> >>>>>>> things in
> >>>>>>>> the newer versions of java that are must haves ( a few
> >>>> exceptions
> >>>>>> are
> >>>>>>>> things that would be really nice to take advantage of ).
> >>>>>>>>
> >>>>>>>> I think that apache commons has taken the right approach by
> >>>>>>> staying
> >>>>>>> on
> >>>>>>>> java 8 giving the largest possible user base.
> >>>>>>>>
> >>>>>>>> Even standardizing on java 11 would have to make us reconsider
> >>>>>>> Ignite as
> >>>>>>>> the platform we are using, we are not so invested in it as of
> >>>>>>> now,
> >>>>>>> even
> >>>>>>>> though we have big plans to leverage it. Just because we aren't
> >>>>>> sure
> >>>>>>> when
> >>>>>>>> we are going to be able to upgrade from java8. It has support
> >>>>>>> through 2022,
> >>>>>>>> so I imagine that is when we will be discussing that.
> >>>>>>>>
> >>>>>>>> Regards
> >>>>>>>>
> >>>>>>>> ~Adam
> >>>>>>>>
> >>>>>>>> Adam Carbone | Director of Innovation – Intelligent Platform
> >>>> Team
> >>>>>>> |
> >>>>>>>> Bottomline Technologies
> >>>>>>>> Office: 603-501-6446 | Mobile: 603-570-8418
> >>>>>>>> www.bottomline.com
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On 11/24/20, 7:38 AM, "Alexey Zinoviev" <zaleslaw.sin@gmail.com
> >>>>>
> >>>>>>> wrote:
> >>>>>>>>
> >>>>>>>> Java 15 is better, VarHandles, ForeignMemory access and so
> >>>>>>> on.
> >>>>>>>>
> >>>>>>>> In both cases, I support the Java version 11 and higher for
> >>>>>>> the
> >>>>>>>> development!
> >>>>>>>>
> >>>>>>>> вт, 24 нояб. 2020 г. в 15:21, Andrey Mashenkov <
> >>>>>>>> andrey.mashenkov@gmail.com>:
> >>>>>>>>
> >>>>>>>>> Let's add maven plugins for sanity checks at the early
> >>>>>> stage.
> >>>>>>>>> I've created a ticket for this [1].
> >>>>>>>>>
> >>>>>>>>> Also, I've found initial pom.xml has a target version Java
> >>>>>>> 8.
> >>>>>>>>> Do we intend to move to Java 11 (or may be next LTS) and
> >>>>>>> drop
> >>>>>>> Java 8
> >>>>>>>> in
> >>>>>>>>> Ignite 3.0?
> >>>>>>>>>
> >>>>>>>>> [1]
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> https://urldefense.com/v3/__https://issues.apache.org/jira/browse/IGNITE-13751__;!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKAO2Ejs8$
> >>>>>>>>>
> >>>>>>>>> On Tue, Nov 24, 2020 at 5:40 AM Valentin Kulichenko <
> >>>>>>>>> valentin.kulichenko@gmail.com> wrote:
> >>>>>>>>>
> >>>>>>>>>> Folks,
> >>>>>>>>>>
> >>>>>>>>>> I went ahead and created the repository [1]. I also
> >>>>>>> configured a
> >>>>>>>> TeamCity
> >>>>>>>>>> project [2] that runs all available JUnit tests on every
> >>>>>>> PR
> >>>>>>>> creation or
> >>>>>>>>>> update. It also sends the status update to GitHub so
> >>>> that
> >>>>>>> it's
> >>>>>>>> reflected
> >>>>>>>>> in
> >>>>>>>>>> the PR itself so that we can do merges directly from
> >>>>>> GitHub.
> >>>>>>> Basic
> >>>>>>>> steps
> >>>>>>>>> to
> >>>>>>>>>> make a change are described on the Wiki page [3].
> >>>>>>>>>>
> >>>>>>>>>> Let me know if you have any questions.
> >>>>>>>>>>
> >>>>>>>>>> [1]
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> https://urldefense.com/v3/__https://github.com/apache/ignite-3__;!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKIq24lxF$
> >>>>>>>>>> [2]
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> https://urldefense.com/v3/__https://ci.ignite.apache.org/project/ignite3__;!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKFGL_oJx$
> >>>>>>>>>> [3]
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/IGNITE/Apache*Ignite*3.0*ApacheIgnite3.0-DevelopmentProcess__;Kysj!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKNhWzQ0s$
> >>>>>>>>>>
> >>>>>>>>>> -Val
> >>>>>>>>>>
> >>>>>>>>>> On Wed, Nov 18, 2020 at 4:24 PM Valentin Kulichenko <
> >>>>>>>>>> valentin.kulichenko@gmail.com> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Thanks, guys. It looks like we are much closer to the
> >>>>>>> consensus
> >>>>>>>> now. I
> >>>>>>>>>>> totally on board with the plan, but I would also like
> >>>>>>> to
> >>>>>>> address
> >>>>>>>> the
> >>>>>>>>>>> short-term needs. As I've already mentioned earlier,
> >>>>>> there
> >>>>>>> are
> >>>>>>>> several
> >>>>>>>>>>> active IEPs, but we still don't have even a
> >>>> preliminary
> >>>>>>> technical
> >>>>>>>>> process
> >>>>>>>>>>> for working on these IEPs. I believe this might be
> >>>>>>> frustrating
> >>>>>>>> for the
> >>>>>>>>>>> folks who would like to commit code.
> >>>>>>>>>>>
> >>>>>>>>>>> The scope we agreed on is quite big, and it will
> >>>> surely
> >>>>>>> take
> >>>>>>>>> significant
> >>>>>>>>>>> time to implement all the changes and stabilize them.
> >>>>>>> Therefore,
> >>>>>>>> it's
> >>>>>>>>>> clear
> >>>>>>>>>>> to me that we will have to maintain 2.x and 3.x in
> >>>>>>> parallel for
> >>>>>>>> quite
> >>>>>>>>>> some
> >>>>>>>>>>> time - this needs to be addressed somehow. I'm
> >>>>>>> convinced
> >>>>>>> that
> >>>>>>>> having a
> >>>>>>>>>>> separate repo is the ONLY way to do that, and so far,
> >>>> I
> >>>>>>> haven't
> >>>>>>>> heard
> >>>>>>>>> any
> >>>>>>>>>>> clear alternatives or reasons why we shouldn't do
> >>>> this.
> >>>>>>>>>>>
> >>>>>>>>>>> That said, I'm inclined to proceed with this in the
> >>>>>>> next
> >>>>>>> few
> >>>>>>>> days - I
> >>>>>>>>>> will
> >>>>>>>>>>> create a repo and describe the process (which we, of
> >>>>>>> course, can
> >>>>>>>>> discuss
> >>>>>>>>>>> and modify going forward). Let's, at the very least,
> >>>>>>> try
> >>>>>>> and see
> >>>>>>>> where
> >>>>>>>>> it
> >>>>>>>>>>> leads us.
> >>>>>>>>>>>
> >>>>>>>>>>> If someone has any concrete alternative options on how
> >>>>>>> to
> >>>>>>> we can
> >>>>>>>>> maintain
> >>>>>>>>>>> two major versions in parallel, let's have another
> >>>>>>> voice
> >>>>>>>> discussion
> >>>>>>>>> this
> >>>>>>>>>>> Friday. If we do the meeting, we should set it up with
> >>>>>>> a
> >>>>>>> clear
> >>>>>>>> goal to
> >>>>>>>>>> make
> >>>>>>>>>>> a decision. Please let me know if there is interest in
> >>>>>>> this.
> >>>>>>>>>>>
> >>>>>>>>>>> -Val
> >>>>>>>>>>>
> >>>>>>>>>>> On Mon, Nov 16, 2020 at 6:31 AM Alexey Goncharuk <
> >>>>>>>>>>> alexey.goncharuk@gmail.com> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Good,
> >>>>>>>>>>>>
> >>>>>>>>>>>> I think we have an intermediate agreement on the
> >>>> scope
> >>>>>> and
> >>>>>>>>> significance
> >>>>>>>>>> of
> >>>>>>>>>>>> the changes we want to make. I suggest creating
> >>>>>>> separate
> >>>>>>>> discussion
> >>>>>>>>>>>> streams
> >>>>>>>>>>>> and calls for each of the suggested topics so that:
> >>>>>>>>>>>>
> >>>>>>>>>>>> - It is clear for the community what is the
> >>>>>> motivation
> >>>>>>> of the
> >>>>>>>>> stream
> >>>>>>>>>>>> (this includes both functional targets and
> >>>>>>> technical
> >>>>>>> debt
> >>>>>>>> issues
> >>>>>>>>>>>> pointed
> >>>>>>>>>>>> out by Sergey)
> >>>>>>>>>>>> - Who is planning to take an active part in each
> >>>> of
> >>>>>> the
> >>>>>>>> streams
> >>>>>>>>> (i.e.
> >>>>>>>>>>>> the 'design committee', as Sergey suggested)
> >>>>>>>>>>>> - What are the intermediate and final goals for
> >>>>>>> each
> >>>>>>> of the
> >>>>>>>> streams
> >>>>>>>>>>>> - What are the cross-stream interactions and how
> >>>> we
> >>>>>>>> integrate them
> >>>>>>>>>>>> - How each of the streams will be integrated with
> >>>>>>> the
> >>>>>>> current
> >>>>>>>>>> codebase
> >>>>>>>>>>>> based on the above (here is where we will see
> >>>>>>> whether
> >>>>>>>> drop-in or
> >>>>>>>>>>>> incremental approaches make more sense)
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> Best regards,
> >>>>>>>>> Andrey V. Mashenkov
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>>
> >>>> Best regards,
> >>>> Ivan Pavlukhin
> >>>>
> >>
> >>
> >
> >
> > --
> >
> > Best regards,
> > Ivan Pavlukhin
> >
>
>
> --
>
> Best regards,
> Ivan Pavlukhin
>
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Ivan Pavlukhin <vo...@gmail.com>.
Also I noticed that ignite-3 repository has "main" but not "master"
branch. Who can shed light on this? Did not find an explanation in
this thread.
2020-12-22 13:09 GMT+03:00, Ivan Pavlukhin <vo...@gmail.com>:
> I noticed some free-from commit messages in ignite-3 repository. I
> think we should use ticket-based workflow and commit messages as
> usual.
>
> [1] https://github.com/apache/ignite-3/commits/main
>
> 2020-12-21 10:55 GMT+03:00, Petr Ivanov <mr...@gmail.com>:
>> There is no problem to have both in new repository, if skilled enthusiast
>> will take over that job.
>>
>> I guess we will stick to Maven for time being but development of
>> Gradle-based building system can be done in parallel.
>> I can even add corresponding development build configurations for
>> TeamCity,
>> or even introduce some kind of switch — so that we can test old and new
>> build approaches and provide seamless transition if we agree on that.
>>
>>> On 19 Dec 2020, at 01:00, Valentin Kulichenko
>>> <va...@gmail.com> wrote:
>>>
>>> Hi Ivan,
>>>
>>> There was a very brief discussion around this. Basically, it looks like
>>> switching from Maven to something else is not going to bring much value,
>>> but at the same time will be quite demanding because the community has
>>> much
>>> more experience with Maven. However, I would say that it is still
>>> debatable at this point -- please feel free to share your thoughts on
>>> this.
>>>
>>> -Val
>>>
>>> On Thu, Dec 17, 2020 at 10:53 PM Ivan Pavlukhin <vo...@gmail.com>
>>> wrote:
>>>
>>>> Hi Igniters,
>>>>
>>>> Forgive me that I am not reading dev list carefully these days. Was it
>>>> explicitly decided that Maven should be used as a build system for
>>>> Ignite 3? As there is a new repository we possibly can update our
>>>> build tools as well. What do you think?
>>>>
>>>> 2020-12-17 22:45 GMT+03:00, Valentin Kulichenko <
>>>> valentin.kulichenko@gmail.com>:
>>>>> Hi Dmitriy,
>>>>>
>>>>> I don't think there is any reason for concern at this point. The
>>>> community
>>>>> agreed on the scope of the changes for 3.0 - it is described on Wiki.
>>>>> The
>>>>> scope is quite big, so it is clear that 2.x and 3.x will have to exist
>>>>> in
>>>>> parallel for a significant amount of time, so we need a place where we
>>>> can
>>>>> merge the code for 3.x. Thus, I've created this new repo. We already
>>>>> have
>>>>> multiple IEPs, as well as several contributors who are actively
>>>>> involved
>>>> in
>>>>> development. Some of the first PRs were merged today.
>>>>>
>>>>> I didn't hear any objections since the repo was created.
>>>>>
>>>>> -Val
>>>>>
>>>>> On Wed, Dec 16, 2020 at 7:28 AM Dmitriy Pavlov <dp...@apache.org>
>>>> wrote:
>>>>>
>>>>>> Folks, I'm a little bit concerned about the simultaneous
>>>>>> - existence of the repo https://github.com/apache/ignite-3 and PRs
>>>>>> for
>>>>>> that
>>>>>> repo
>>>>>> - and a couple of downvotes from PMC members.
>>>>>>
>>>>>> Is it all fine here? Was there any vote /discussion where it was
>>>>>> discussed
>>>>>> and consensus approved? What is the status of the ignite-3 repo?
>>>>>>
>>>>>> вт, 15 дек. 2020 г. в 17:30, Carbone, Adam
>>>>>> <Adam.Carbone@bottomline.com
>>>>> :
>>>>>>
>>>>>>> I don't believe Java 7 was LTS, and I hope that others will have
>>>>>>> upgraded
>>>>>>> long before that. If that is the release timeframe for 3.0, then I
>>>>>> suppose
>>>>>>> that would makes sense, I would still doubt that people would be
>>>>>>> going
>>>>>>> newer than java 11, just my opinion of what I'm seeing.
>>>>>>>
>>>>>>> Regards
>>>>>>> ~adam
>>>>>>>
>>>>>>> Adam Carbone | Director of Innovation – Intelligent Platform Team |
>>>>>>> Bottomline Technologies
>>>>>>> Office: 603-501-6446 | Mobile: 603-570-8418
>>>>>>> www.bottomline.com
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 12/15/20, 4:25 AM, "Ilya Kasnacheev" <il...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Hello!
>>>>>>>
>>>>>>> I guess Ignite 3.0 will be ready for production use somewhere in
>>>>>> 2022,
>>>>>>> realistically. By that time, Java 8 will be long enough out of
>>>>>> support
>>>>>>> so
>>>>>>> that most companies will actually forbid its use, WRT
>>>>>>> vulnerabilities
>>>>>>> et
>>>>>>> all.
>>>>>>>
>>>>>>> After all we have managed to upgrade from Java 7 to Java 8 and
>>>> only
>>>>>>> got a
>>>>>>> minor amount of complaints.
>>>>>>>
>>>>>>> Regards,
>>>>>>> --
>>>>>>> Ilya Kasnacheev
>>>>>>>
>>>>>>>
>>>>>>> пн, 14 дек. 2020 г. в 19:06, Carbone, Adam <
>>>>>>> Adam.Carbone@bottomline.com>:
>>>>>>>
>>>>>>>> So just one bit to consider... Are there features that you need
>>>>>>> to
>>>>>>> use in
>>>>>>>> these newer versions of java? Or are we just updating to stay
>>>>>>> current? The
>>>>>>>> reason I ask is that there are still lots of people in an
>>>>>> enterprise
>>>>>>> space
>>>>>>>> that are beholden to having to support legacy JAVAEE supported
>>>>>>> applications
>>>>>>>> on Websphere, Weblogic, Redhat, etc... and the organizations
>>>> that
>>>>>>> use those
>>>>>>>> platforms are slow to move... Most of them are still on Java8
>>>>>>>>
>>>>>>>> So as a platform I think a strong consideration needs to be
>>>>>>> towards
>>>>>>> having
>>>>>>>> the broadest possible support profile until it becomes an
>>>>>> impediment
>>>>>>> to
>>>>>>>> doing things that the platform needs. So far I haven't seen huge
>>>>>>> things in
>>>>>>>> the newer versions of java that are must haves ( a few
>>>> exceptions
>>>>>> are
>>>>>>>> things that would be really nice to take advantage of ).
>>>>>>>>
>>>>>>>> I think that apache commons has taken the right approach by
>>>>>>> staying
>>>>>>> on
>>>>>>>> java 8 giving the largest possible user base.
>>>>>>>>
>>>>>>>> Even standardizing on java 11 would have to make us reconsider
>>>>>>> Ignite as
>>>>>>>> the platform we are using, we are not so invested in it as of
>>>>>>> now,
>>>>>>> even
>>>>>>>> though we have big plans to leverage it. Just because we aren't
>>>>>> sure
>>>>>>> when
>>>>>>>> we are going to be able to upgrade from java8. It has support
>>>>>>> through 2022,
>>>>>>>> so I imagine that is when we will be discussing that.
>>>>>>>>
>>>>>>>> Regards
>>>>>>>>
>>>>>>>> ~Adam
>>>>>>>>
>>>>>>>> Adam Carbone | Director of Innovation – Intelligent Platform
>>>> Team
>>>>>>> |
>>>>>>>> Bottomline Technologies
>>>>>>>> Office: 603-501-6446 | Mobile: 603-570-8418
>>>>>>>> www.bottomline.com
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 11/24/20, 7:38 AM, "Alexey Zinoviev" <zaleslaw.sin@gmail.com
>>>>>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Java 15 is better, VarHandles, ForeignMemory access and so
>>>>>>> on.
>>>>>>>>
>>>>>>>> In both cases, I support the Java version 11 and higher for
>>>>>>> the
>>>>>>>> development!
>>>>>>>>
>>>>>>>> вт, 24 нояб. 2020 г. в 15:21, Andrey Mashenkov <
>>>>>>>> andrey.mashenkov@gmail.com>:
>>>>>>>>
>>>>>>>>> Let's add maven plugins for sanity checks at the early
>>>>>> stage.
>>>>>>>>> I've created a ticket for this [1].
>>>>>>>>>
>>>>>>>>> Also, I've found initial pom.xml has a target version Java
>>>>>>> 8.
>>>>>>>>> Do we intend to move to Java 11 (or may be next LTS) and
>>>>>>> drop
>>>>>>> Java 8
>>>>>>>> in
>>>>>>>>> Ignite 3.0?
>>>>>>>>>
>>>>>>>>> [1]
>>>>>>>>
>>>>>>>
>>>>>>
>>>> https://urldefense.com/v3/__https://issues.apache.org/jira/browse/IGNITE-13751__;!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKAO2Ejs8$
>>>>>>>>>
>>>>>>>>> On Tue, Nov 24, 2020 at 5:40 AM Valentin Kulichenko <
>>>>>>>>> valentin.kulichenko@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Folks,
>>>>>>>>>>
>>>>>>>>>> I went ahead and created the repository [1]. I also
>>>>>>> configured a
>>>>>>>> TeamCity
>>>>>>>>>> project [2] that runs all available JUnit tests on every
>>>>>>> PR
>>>>>>>> creation or
>>>>>>>>>> update. It also sends the status update to GitHub so
>>>> that
>>>>>>> it's
>>>>>>>> reflected
>>>>>>>>> in
>>>>>>>>>> the PR itself so that we can do merges directly from
>>>>>> GitHub.
>>>>>>> Basic
>>>>>>>> steps
>>>>>>>>> to
>>>>>>>>>> make a change are described on the Wiki page [3].
>>>>>>>>>>
>>>>>>>>>> Let me know if you have any questions.
>>>>>>>>>>
>>>>>>>>>> [1]
>>>>>>>>
>>>>>>>
>>>>>>
>>>> https://urldefense.com/v3/__https://github.com/apache/ignite-3__;!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKIq24lxF$
>>>>>>>>>> [2]
>>>>>>>>
>>>>>>>
>>>>>>
>>>> https://urldefense.com/v3/__https://ci.ignite.apache.org/project/ignite3__;!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKFGL_oJx$
>>>>>>>>>> [3]
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/IGNITE/Apache*Ignite*3.0*ApacheIgnite3.0-DevelopmentProcess__;Kysj!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKNhWzQ0s$
>>>>>>>>>>
>>>>>>>>>> -Val
>>>>>>>>>>
>>>>>>>>>> On Wed, Nov 18, 2020 at 4:24 PM Valentin Kulichenko <
>>>>>>>>>> valentin.kulichenko@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Thanks, guys. It looks like we are much closer to the
>>>>>>> consensus
>>>>>>>> now. I
>>>>>>>>>>> totally on board with the plan, but I would also like
>>>>>>> to
>>>>>>> address
>>>>>>>> the
>>>>>>>>>>> short-term needs. As I've already mentioned earlier,
>>>>>> there
>>>>>>> are
>>>>>>>> several
>>>>>>>>>>> active IEPs, but we still don't have even a
>>>> preliminary
>>>>>>> technical
>>>>>>>>> process
>>>>>>>>>>> for working on these IEPs. I believe this might be
>>>>>>> frustrating
>>>>>>>> for the
>>>>>>>>>>> folks who would like to commit code.
>>>>>>>>>>>
>>>>>>>>>>> The scope we agreed on is quite big, and it will
>>>> surely
>>>>>>> take
>>>>>>>>> significant
>>>>>>>>>>> time to implement all the changes and stabilize them.
>>>>>>> Therefore,
>>>>>>>> it's
>>>>>>>>>> clear
>>>>>>>>>>> to me that we will have to maintain 2.x and 3.x in
>>>>>>> parallel for
>>>>>>>> quite
>>>>>>>>>> some
>>>>>>>>>>> time - this needs to be addressed somehow. I'm
>>>>>>> convinced
>>>>>>> that
>>>>>>>> having a
>>>>>>>>>>> separate repo is the ONLY way to do that, and so far,
>>>> I
>>>>>>> haven't
>>>>>>>> heard
>>>>>>>>> any
>>>>>>>>>>> clear alternatives or reasons why we shouldn't do
>>>> this.
>>>>>>>>>>>
>>>>>>>>>>> That said, I'm inclined to proceed with this in the
>>>>>>> next
>>>>>>> few
>>>>>>>> days - I
>>>>>>>>>> will
>>>>>>>>>>> create a repo and describe the process (which we, of
>>>>>>> course, can
>>>>>>>>> discuss
>>>>>>>>>>> and modify going forward). Let's, at the very least,
>>>>>>> try
>>>>>>> and see
>>>>>>>> where
>>>>>>>>> it
>>>>>>>>>>> leads us.
>>>>>>>>>>>
>>>>>>>>>>> If someone has any concrete alternative options on how
>>>>>>> to
>>>>>>> we can
>>>>>>>>> maintain
>>>>>>>>>>> two major versions in parallel, let's have another
>>>>>>> voice
>>>>>>>> discussion
>>>>>>>>> this
>>>>>>>>>>> Friday. If we do the meeting, we should set it up with
>>>>>>> a
>>>>>>> clear
>>>>>>>> goal to
>>>>>>>>>> make
>>>>>>>>>>> a decision. Please let me know if there is interest in
>>>>>>> this.
>>>>>>>>>>>
>>>>>>>>>>> -Val
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Nov 16, 2020 at 6:31 AM Alexey Goncharuk <
>>>>>>>>>>> alexey.goncharuk@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Good,
>>>>>>>>>>>>
>>>>>>>>>>>> I think we have an intermediate agreement on the
>>>> scope
>>>>>> and
>>>>>>>>> significance
>>>>>>>>>> of
>>>>>>>>>>>> the changes we want to make. I suggest creating
>>>>>>> separate
>>>>>>>> discussion
>>>>>>>>>>>> streams
>>>>>>>>>>>> and calls for each of the suggested topics so that:
>>>>>>>>>>>>
>>>>>>>>>>>> - It is clear for the community what is the
>>>>>> motivation
>>>>>>> of the
>>>>>>>>> stream
>>>>>>>>>>>> (this includes both functional targets and
>>>>>>> technical
>>>>>>> debt
>>>>>>>> issues
>>>>>>>>>>>> pointed
>>>>>>>>>>>> out by Sergey)
>>>>>>>>>>>> - Who is planning to take an active part in each
>>>> of
>>>>>> the
>>>>>>>> streams
>>>>>>>>> (i.e.
>>>>>>>>>>>> the 'design committee', as Sergey suggested)
>>>>>>>>>>>> - What are the intermediate and final goals for
>>>>>>> each
>>>>>>> of the
>>>>>>>> streams
>>>>>>>>>>>> - What are the cross-stream interactions and how
>>>> we
>>>>>>>> integrate them
>>>>>>>>>>>> - How each of the streams will be integrated with
>>>>>>> the
>>>>>>> current
>>>>>>>>>> codebase
>>>>>>>>>>>> based on the above (here is where we will see
>>>>>>> whether
>>>>>>>> drop-in or
>>>>>>>>>>>> incremental approaches make more sense)
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Best regards,
>>>>>>>>> Andrey V. Mashenkov
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Best regards,
>>>> Ivan Pavlukhin
>>>>
>>
>>
>
>
> --
>
> Best regards,
> Ivan Pavlukhin
>
--
Best regards,
Ivan Pavlukhin
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Ivan Pavlukhin <vo...@gmail.com>.
I noticed some free-from commit messages in ignite-3 repository. I
think we should use ticket-based workflow and commit messages as
usual.
[1] https://github.com/apache/ignite-3/commits/main
2020-12-21 10:55 GMT+03:00, Petr Ivanov <mr...@gmail.com>:
> There is no problem to have both in new repository, if skilled enthusiast
> will take over that job.
>
> I guess we will stick to Maven for time being but development of
> Gradle-based building system can be done in parallel.
> I can even add corresponding development build configurations for TeamCity,
> or even introduce some kind of switch — so that we can test old and new
> build approaches and provide seamless transition if we agree on that.
>
>> On 19 Dec 2020, at 01:00, Valentin Kulichenko
>> <va...@gmail.com> wrote:
>>
>> Hi Ivan,
>>
>> There was a very brief discussion around this. Basically, it looks like
>> switching from Maven to something else is not going to bring much value,
>> but at the same time will be quite demanding because the community has
>> much
>> more experience with Maven. However, I would say that it is still
>> debatable at this point -- please feel free to share your thoughts on
>> this.
>>
>> -Val
>>
>> On Thu, Dec 17, 2020 at 10:53 PM Ivan Pavlukhin <vo...@gmail.com>
>> wrote:
>>
>>> Hi Igniters,
>>>
>>> Forgive me that I am not reading dev list carefully these days. Was it
>>> explicitly decided that Maven should be used as a build system for
>>> Ignite 3? As there is a new repository we possibly can update our
>>> build tools as well. What do you think?
>>>
>>> 2020-12-17 22:45 GMT+03:00, Valentin Kulichenko <
>>> valentin.kulichenko@gmail.com>:
>>>> Hi Dmitriy,
>>>>
>>>> I don't think there is any reason for concern at this point. The
>>> community
>>>> agreed on the scope of the changes for 3.0 - it is described on Wiki.
>>>> The
>>>> scope is quite big, so it is clear that 2.x and 3.x will have to exist
>>>> in
>>>> parallel for a significant amount of time, so we need a place where we
>>> can
>>>> merge the code for 3.x. Thus, I've created this new repo. We already
>>>> have
>>>> multiple IEPs, as well as several contributors who are actively
>>>> involved
>>> in
>>>> development. Some of the first PRs were merged today.
>>>>
>>>> I didn't hear any objections since the repo was created.
>>>>
>>>> -Val
>>>>
>>>> On Wed, Dec 16, 2020 at 7:28 AM Dmitriy Pavlov <dp...@apache.org>
>>> wrote:
>>>>
>>>>> Folks, I'm a little bit concerned about the simultaneous
>>>>> - existence of the repo https://github.com/apache/ignite-3 and PRs for
>>>>> that
>>>>> repo
>>>>> - and a couple of downvotes from PMC members.
>>>>>
>>>>> Is it all fine here? Was there any vote /discussion where it was
>>>>> discussed
>>>>> and consensus approved? What is the status of the ignite-3 repo?
>>>>>
>>>>> вт, 15 дек. 2020 г. в 17:30, Carbone, Adam
>>>>> <Adam.Carbone@bottomline.com
>>>> :
>>>>>
>>>>>> I don't believe Java 7 was LTS, and I hope that others will have
>>>>>> upgraded
>>>>>> long before that. If that is the release timeframe for 3.0, then I
>>>>> suppose
>>>>>> that would makes sense, I would still doubt that people would be
>>>>>> going
>>>>>> newer than java 11, just my opinion of what I'm seeing.
>>>>>>
>>>>>> Regards
>>>>>> ~adam
>>>>>>
>>>>>> Adam Carbone | Director of Innovation – Intelligent Platform Team |
>>>>>> Bottomline Technologies
>>>>>> Office: 603-501-6446 | Mobile: 603-570-8418
>>>>>> www.bottomline.com
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 12/15/20, 4:25 AM, "Ilya Kasnacheev" <il...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> Hello!
>>>>>>
>>>>>> I guess Ignite 3.0 will be ready for production use somewhere in
>>>>> 2022,
>>>>>> realistically. By that time, Java 8 will be long enough out of
>>>>> support
>>>>>> so
>>>>>> that most companies will actually forbid its use, WRT
>>>>>> vulnerabilities
>>>>>> et
>>>>>> all.
>>>>>>
>>>>>> After all we have managed to upgrade from Java 7 to Java 8 and
>>> only
>>>>>> got a
>>>>>> minor amount of complaints.
>>>>>>
>>>>>> Regards,
>>>>>> --
>>>>>> Ilya Kasnacheev
>>>>>>
>>>>>>
>>>>>> пн, 14 дек. 2020 г. в 19:06, Carbone, Adam <
>>>>>> Adam.Carbone@bottomline.com>:
>>>>>>
>>>>>>> So just one bit to consider... Are there features that you need
>>>>>> to
>>>>>> use in
>>>>>>> these newer versions of java? Or are we just updating to stay
>>>>>> current? The
>>>>>>> reason I ask is that there are still lots of people in an
>>>>> enterprise
>>>>>> space
>>>>>>> that are beholden to having to support legacy JAVAEE supported
>>>>>> applications
>>>>>>> on Websphere, Weblogic, Redhat, etc... and the organizations
>>> that
>>>>>> use those
>>>>>>> platforms are slow to move... Most of them are still on Java8
>>>>>>>
>>>>>>> So as a platform I think a strong consideration needs to be
>>>>>> towards
>>>>>> having
>>>>>>> the broadest possible support profile until it becomes an
>>>>> impediment
>>>>>> to
>>>>>>> doing things that the platform needs. So far I haven't seen huge
>>>>>> things in
>>>>>>> the newer versions of java that are must haves ( a few
>>> exceptions
>>>>> are
>>>>>>> things that would be really nice to take advantage of ).
>>>>>>>
>>>>>>> I think that apache commons has taken the right approach by
>>>>>> staying
>>>>>> on
>>>>>>> java 8 giving the largest possible user base.
>>>>>>>
>>>>>>> Even standardizing on java 11 would have to make us reconsider
>>>>>> Ignite as
>>>>>>> the platform we are using, we are not so invested in it as of
>>>>>> now,
>>>>>> even
>>>>>>> though we have big plans to leverage it. Just because we aren't
>>>>> sure
>>>>>> when
>>>>>>> we are going to be able to upgrade from java8. It has support
>>>>>> through 2022,
>>>>>>> so I imagine that is when we will be discussing that.
>>>>>>>
>>>>>>> Regards
>>>>>>>
>>>>>>> ~Adam
>>>>>>>
>>>>>>> Adam Carbone | Director of Innovation – Intelligent Platform
>>> Team
>>>>>> |
>>>>>>> Bottomline Technologies
>>>>>>> Office: 603-501-6446 | Mobile: 603-570-8418
>>>>>>> www.bottomline.com
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 11/24/20, 7:38 AM, "Alexey Zinoviev" <zaleslaw.sin@gmail.com
>>>>
>>>>>> wrote:
>>>>>>>
>>>>>>> Java 15 is better, VarHandles, ForeignMemory access and so
>>>>>> on.
>>>>>>>
>>>>>>> In both cases, I support the Java version 11 and higher for
>>>>>> the
>>>>>>> development!
>>>>>>>
>>>>>>> вт, 24 нояб. 2020 г. в 15:21, Andrey Mashenkov <
>>>>>>> andrey.mashenkov@gmail.com>:
>>>>>>>
>>>>>>>> Let's add maven plugins for sanity checks at the early
>>>>> stage.
>>>>>>>> I've created a ticket for this [1].
>>>>>>>>
>>>>>>>> Also, I've found initial pom.xml has a target version Java
>>>>>> 8.
>>>>>>>> Do we intend to move to Java 11 (or may be next LTS) and
>>>>>> drop
>>>>>> Java 8
>>>>>>> in
>>>>>>>> Ignite 3.0?
>>>>>>>>
>>>>>>>> [1]
>>>>>>>
>>>>>>
>>>>>
>>> https://urldefense.com/v3/__https://issues.apache.org/jira/browse/IGNITE-13751__;!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKAO2Ejs8$
>>>>>>>>
>>>>>>>> On Tue, Nov 24, 2020 at 5:40 AM Valentin Kulichenko <
>>>>>>>> valentin.kulichenko@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Folks,
>>>>>>>>>
>>>>>>>>> I went ahead and created the repository [1]. I also
>>>>>> configured a
>>>>>>> TeamCity
>>>>>>>>> project [2] that runs all available JUnit tests on every
>>>>>> PR
>>>>>>> creation or
>>>>>>>>> update. It also sends the status update to GitHub so
>>> that
>>>>>> it's
>>>>>>> reflected
>>>>>>>> in
>>>>>>>>> the PR itself so that we can do merges directly from
>>>>> GitHub.
>>>>>> Basic
>>>>>>> steps
>>>>>>>> to
>>>>>>>>> make a change are described on the Wiki page [3].
>>>>>>>>>
>>>>>>>>> Let me know if you have any questions.
>>>>>>>>>
>>>>>>>>> [1]
>>>>>>>
>>>>>>
>>>>>
>>> https://urldefense.com/v3/__https://github.com/apache/ignite-3__;!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKIq24lxF$
>>>>>>>>> [2]
>>>>>>>
>>>>>>
>>>>>
>>> https://urldefense.com/v3/__https://ci.ignite.apache.org/project/ignite3__;!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKFGL_oJx$
>>>>>>>>> [3]
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/IGNITE/Apache*Ignite*3.0*ApacheIgnite3.0-DevelopmentProcess__;Kysj!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKNhWzQ0s$
>>>>>>>>>
>>>>>>>>> -Val
>>>>>>>>>
>>>>>>>>> On Wed, Nov 18, 2020 at 4:24 PM Valentin Kulichenko <
>>>>>>>>> valentin.kulichenko@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Thanks, guys. It looks like we are much closer to the
>>>>>> consensus
>>>>>>> now. I
>>>>>>>>>> totally on board with the plan, but I would also like
>>>>>> to
>>>>>> address
>>>>>>> the
>>>>>>>>>> short-term needs. As I've already mentioned earlier,
>>>>> there
>>>>>> are
>>>>>>> several
>>>>>>>>>> active IEPs, but we still don't have even a
>>> preliminary
>>>>>> technical
>>>>>>>> process
>>>>>>>>>> for working on these IEPs. I believe this might be
>>>>>> frustrating
>>>>>>> for the
>>>>>>>>>> folks who would like to commit code.
>>>>>>>>>>
>>>>>>>>>> The scope we agreed on is quite big, and it will
>>> surely
>>>>>> take
>>>>>>>> significant
>>>>>>>>>> time to implement all the changes and stabilize them.
>>>>>> Therefore,
>>>>>>> it's
>>>>>>>>> clear
>>>>>>>>>> to me that we will have to maintain 2.x and 3.x in
>>>>>> parallel for
>>>>>>> quite
>>>>>>>>> some
>>>>>>>>>> time - this needs to be addressed somehow. I'm
>>>>>> convinced
>>>>>> that
>>>>>>> having a
>>>>>>>>>> separate repo is the ONLY way to do that, and so far,
>>> I
>>>>>> haven't
>>>>>>> heard
>>>>>>>> any
>>>>>>>>>> clear alternatives or reasons why we shouldn't do
>>> this.
>>>>>>>>>>
>>>>>>>>>> That said, I'm inclined to proceed with this in the
>>>>>> next
>>>>>> few
>>>>>>> days - I
>>>>>>>>> will
>>>>>>>>>> create a repo and describe the process (which we, of
>>>>>> course, can
>>>>>>>> discuss
>>>>>>>>>> and modify going forward). Let's, at the very least,
>>>>>> try
>>>>>> and see
>>>>>>> where
>>>>>>>> it
>>>>>>>>>> leads us.
>>>>>>>>>>
>>>>>>>>>> If someone has any concrete alternative options on how
>>>>>> to
>>>>>> we can
>>>>>>>> maintain
>>>>>>>>>> two major versions in parallel, let's have another
>>>>>> voice
>>>>>>> discussion
>>>>>>>> this
>>>>>>>>>> Friday. If we do the meeting, we should set it up with
>>>>>> a
>>>>>> clear
>>>>>>> goal to
>>>>>>>>> make
>>>>>>>>>> a decision. Please let me know if there is interest in
>>>>>> this.
>>>>>>>>>>
>>>>>>>>>> -Val
>>>>>>>>>>
>>>>>>>>>> On Mon, Nov 16, 2020 at 6:31 AM Alexey Goncharuk <
>>>>>>>>>> alexey.goncharuk@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Good,
>>>>>>>>>>>
>>>>>>>>>>> I think we have an intermediate agreement on the
>>> scope
>>>>> and
>>>>>>>> significance
>>>>>>>>> of
>>>>>>>>>>> the changes we want to make. I suggest creating
>>>>>> separate
>>>>>>> discussion
>>>>>>>>>>> streams
>>>>>>>>>>> and calls for each of the suggested topics so that:
>>>>>>>>>>>
>>>>>>>>>>> - It is clear for the community what is the
>>>>> motivation
>>>>>> of the
>>>>>>>> stream
>>>>>>>>>>> (this includes both functional targets and
>>>>>> technical
>>>>>> debt
>>>>>>> issues
>>>>>>>>>>> pointed
>>>>>>>>>>> out by Sergey)
>>>>>>>>>>> - Who is planning to take an active part in each
>>> of
>>>>> the
>>>>>>> streams
>>>>>>>> (i.e.
>>>>>>>>>>> the 'design committee', as Sergey suggested)
>>>>>>>>>>> - What are the intermediate and final goals for
>>>>>> each
>>>>>> of the
>>>>>>> streams
>>>>>>>>>>> - What are the cross-stream interactions and how
>>> we
>>>>>>> integrate them
>>>>>>>>>>> - How each of the streams will be integrated with
>>>>>> the
>>>>>> current
>>>>>>>>> codebase
>>>>>>>>>>> based on the above (here is where we will see
>>>>>> whether
>>>>>>> drop-in or
>>>>>>>>>>> incremental approaches make more sense)
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Best regards,
>>>>>>>> Andrey V. Mashenkov
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> Best regards,
>>> Ivan Pavlukhin
>>>
>
>
--
Best regards,
Ivan Pavlukhin
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Petr Ivanov <mr...@gmail.com>.
There is no problem to have both in new repository, if skilled enthusiast will take over that job.
I guess we will stick to Maven for time being but development of Gradle-based building system can be done in parallel.
I can even add corresponding development build configurations for TeamCity, or even introduce some kind of switch — so that we can test old and new build approaches and provide seamless transition if we agree on that.
> On 19 Dec 2020, at 01:00, Valentin Kulichenko <va...@gmail.com> wrote:
>
> Hi Ivan,
>
> There was a very brief discussion around this. Basically, it looks like
> switching from Maven to something else is not going to bring much value,
> but at the same time will be quite demanding because the community has much
> more experience with Maven. However, I would say that it is still
> debatable at this point -- please feel free to share your thoughts on this.
>
> -Val
>
> On Thu, Dec 17, 2020 at 10:53 PM Ivan Pavlukhin <vo...@gmail.com> wrote:
>
>> Hi Igniters,
>>
>> Forgive me that I am not reading dev list carefully these days. Was it
>> explicitly decided that Maven should be used as a build system for
>> Ignite 3? As there is a new repository we possibly can update our
>> build tools as well. What do you think?
>>
>> 2020-12-17 22:45 GMT+03:00, Valentin Kulichenko <
>> valentin.kulichenko@gmail.com>:
>>> Hi Dmitriy,
>>>
>>> I don't think there is any reason for concern at this point. The
>> community
>>> agreed on the scope of the changes for 3.0 - it is described on Wiki. The
>>> scope is quite big, so it is clear that 2.x and 3.x will have to exist in
>>> parallel for a significant amount of time, so we need a place where we
>> can
>>> merge the code for 3.x. Thus, I've created this new repo. We already have
>>> multiple IEPs, as well as several contributors who are actively involved
>> in
>>> development. Some of the first PRs were merged today.
>>>
>>> I didn't hear any objections since the repo was created.
>>>
>>> -Val
>>>
>>> On Wed, Dec 16, 2020 at 7:28 AM Dmitriy Pavlov <dp...@apache.org>
>> wrote:
>>>
>>>> Folks, I'm a little bit concerned about the simultaneous
>>>> - existence of the repo https://github.com/apache/ignite-3 and PRs for
>>>> that
>>>> repo
>>>> - and a couple of downvotes from PMC members.
>>>>
>>>> Is it all fine here? Was there any vote /discussion where it was
>>>> discussed
>>>> and consensus approved? What is the status of the ignite-3 repo?
>>>>
>>>> вт, 15 дек. 2020 г. в 17:30, Carbone, Adam <Adam.Carbone@bottomline.com
>>> :
>>>>
>>>>> I don't believe Java 7 was LTS, and I hope that others will have
>>>>> upgraded
>>>>> long before that. If that is the release timeframe for 3.0, then I
>>>> suppose
>>>>> that would makes sense, I would still doubt that people would be going
>>>>> newer than java 11, just my opinion of what I'm seeing.
>>>>>
>>>>> Regards
>>>>> ~adam
>>>>>
>>>>> Adam Carbone | Director of Innovation – Intelligent Platform Team |
>>>>> Bottomline Technologies
>>>>> Office: 603-501-6446 | Mobile: 603-570-8418
>>>>> www.bottomline.com
>>>>>
>>>>>
>>>>>
>>>>> On 12/15/20, 4:25 AM, "Ilya Kasnacheev" <il...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Hello!
>>>>>
>>>>> I guess Ignite 3.0 will be ready for production use somewhere in
>>>> 2022,
>>>>> realistically. By that time, Java 8 will be long enough out of
>>>> support
>>>>> so
>>>>> that most companies will actually forbid its use, WRT
>>>>> vulnerabilities
>>>>> et
>>>>> all.
>>>>>
>>>>> After all we have managed to upgrade from Java 7 to Java 8 and
>> only
>>>>> got a
>>>>> minor amount of complaints.
>>>>>
>>>>> Regards,
>>>>> --
>>>>> Ilya Kasnacheev
>>>>>
>>>>>
>>>>> пн, 14 дек. 2020 г. в 19:06, Carbone, Adam <
>>>>> Adam.Carbone@bottomline.com>:
>>>>>
>>>>>> So just one bit to consider... Are there features that you need
>>>>> to
>>>>> use in
>>>>>> these newer versions of java? Or are we just updating to stay
>>>>> current? The
>>>>>> reason I ask is that there are still lots of people in an
>>>> enterprise
>>>>> space
>>>>>> that are beholden to having to support legacy JAVAEE supported
>>>>> applications
>>>>>> on Websphere, Weblogic, Redhat, etc... and the organizations
>> that
>>>>> use those
>>>>>> platforms are slow to move... Most of them are still on Java8
>>>>>>
>>>>>> So as a platform I think a strong consideration needs to be
>>>>> towards
>>>>> having
>>>>>> the broadest possible support profile until it becomes an
>>>> impediment
>>>>> to
>>>>>> doing things that the platform needs. So far I haven't seen huge
>>>>> things in
>>>>>> the newer versions of java that are must haves ( a few
>> exceptions
>>>> are
>>>>>> things that would be really nice to take advantage of ).
>>>>>>
>>>>>> I think that apache commons has taken the right approach by
>>>>> staying
>>>>> on
>>>>>> java 8 giving the largest possible user base.
>>>>>>
>>>>>> Even standardizing on java 11 would have to make us reconsider
>>>>> Ignite as
>>>>>> the platform we are using, we are not so invested in it as of
>>>>> now,
>>>>> even
>>>>>> though we have big plans to leverage it. Just because we aren't
>>>> sure
>>>>> when
>>>>>> we are going to be able to upgrade from java8. It has support
>>>>> through 2022,
>>>>>> so I imagine that is when we will be discussing that.
>>>>>>
>>>>>> Regards
>>>>>>
>>>>>> ~Adam
>>>>>>
>>>>>> Adam Carbone | Director of Innovation – Intelligent Platform
>> Team
>>>>> |
>>>>>> Bottomline Technologies
>>>>>> Office: 603-501-6446 | Mobile: 603-570-8418
>>>>>> www.bottomline.com
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 11/24/20, 7:38 AM, "Alexey Zinoviev" <zaleslaw.sin@gmail.com
>>>
>>>>> wrote:
>>>>>>
>>>>>> Java 15 is better, VarHandles, ForeignMemory access and so
>>>>> on.
>>>>>>
>>>>>> In both cases, I support the Java version 11 and higher for
>>>>> the
>>>>>> development!
>>>>>>
>>>>>> вт, 24 нояб. 2020 г. в 15:21, Andrey Mashenkov <
>>>>>> andrey.mashenkov@gmail.com>:
>>>>>>
>>>>>>> Let's add maven plugins for sanity checks at the early
>>>> stage.
>>>>>>> I've created a ticket for this [1].
>>>>>>>
>>>>>>> Also, I've found initial pom.xml has a target version Java
>>>>> 8.
>>>>>>> Do we intend to move to Java 11 (or may be next LTS) and
>>>>> drop
>>>>> Java 8
>>>>>> in
>>>>>>> Ignite 3.0?
>>>>>>>
>>>>>>> [1]
>>>>>>
>>>>>
>>>>
>> https://urldefense.com/v3/__https://issues.apache.org/jira/browse/IGNITE-13751__;!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKAO2Ejs8$
>>>>>>>
>>>>>>> On Tue, Nov 24, 2020 at 5:40 AM Valentin Kulichenko <
>>>>>>> valentin.kulichenko@gmail.com> wrote:
>>>>>>>
>>>>>>>> Folks,
>>>>>>>>
>>>>>>>> I went ahead and created the repository [1]. I also
>>>>> configured a
>>>>>> TeamCity
>>>>>>>> project [2] that runs all available JUnit tests on every
>>>>> PR
>>>>>> creation or
>>>>>>>> update. It also sends the status update to GitHub so
>> that
>>>>> it's
>>>>>> reflected
>>>>>>> in
>>>>>>>> the PR itself so that we can do merges directly from
>>>> GitHub.
>>>>> Basic
>>>>>> steps
>>>>>>> to
>>>>>>>> make a change are described on the Wiki page [3].
>>>>>>>>
>>>>>>>> Let me know if you have any questions.
>>>>>>>>
>>>>>>>> [1]
>>>>>>
>>>>>
>>>>
>> https://urldefense.com/v3/__https://github.com/apache/ignite-3__;!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKIq24lxF$
>>>>>>>> [2]
>>>>>>
>>>>>
>>>>
>> https://urldefense.com/v3/__https://ci.ignite.apache.org/project/ignite3__;!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKFGL_oJx$
>>>>>>>> [3]
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/IGNITE/Apache*Ignite*3.0*ApacheIgnite3.0-DevelopmentProcess__;Kysj!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKNhWzQ0s$
>>>>>>>>
>>>>>>>> -Val
>>>>>>>>
>>>>>>>> On Wed, Nov 18, 2020 at 4:24 PM Valentin Kulichenko <
>>>>>>>> valentin.kulichenko@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Thanks, guys. It looks like we are much closer to the
>>>>> consensus
>>>>>> now. I
>>>>>>>>> totally on board with the plan, but I would also like
>>>>> to
>>>>> address
>>>>>> the
>>>>>>>>> short-term needs. As I've already mentioned earlier,
>>>> there
>>>>> are
>>>>>> several
>>>>>>>>> active IEPs, but we still don't have even a
>> preliminary
>>>>> technical
>>>>>>> process
>>>>>>>>> for working on these IEPs. I believe this might be
>>>>> frustrating
>>>>>> for the
>>>>>>>>> folks who would like to commit code.
>>>>>>>>>
>>>>>>>>> The scope we agreed on is quite big, and it will
>> surely
>>>>> take
>>>>>>> significant
>>>>>>>>> time to implement all the changes and stabilize them.
>>>>> Therefore,
>>>>>> it's
>>>>>>>> clear
>>>>>>>>> to me that we will have to maintain 2.x and 3.x in
>>>>> parallel for
>>>>>> quite
>>>>>>>> some
>>>>>>>>> time - this needs to be addressed somehow. I'm
>>>>> convinced
>>>>> that
>>>>>> having a
>>>>>>>>> separate repo is the ONLY way to do that, and so far,
>> I
>>>>> haven't
>>>>>> heard
>>>>>>> any
>>>>>>>>> clear alternatives or reasons why we shouldn't do
>> this.
>>>>>>>>>
>>>>>>>>> That said, I'm inclined to proceed with this in the
>>>>> next
>>>>> few
>>>>>> days - I
>>>>>>>> will
>>>>>>>>> create a repo and describe the process (which we, of
>>>>> course, can
>>>>>>> discuss
>>>>>>>>> and modify going forward). Let's, at the very least,
>>>>> try
>>>>> and see
>>>>>> where
>>>>>>> it
>>>>>>>>> leads us.
>>>>>>>>>
>>>>>>>>> If someone has any concrete alternative options on how
>>>>> to
>>>>> we can
>>>>>>> maintain
>>>>>>>>> two major versions in parallel, let's have another
>>>>> voice
>>>>>> discussion
>>>>>>> this
>>>>>>>>> Friday. If we do the meeting, we should set it up with
>>>>> a
>>>>> clear
>>>>>> goal to
>>>>>>>> make
>>>>>>>>> a decision. Please let me know if there is interest in
>>>>> this.
>>>>>>>>>
>>>>>>>>> -Val
>>>>>>>>>
>>>>>>>>> On Mon, Nov 16, 2020 at 6:31 AM Alexey Goncharuk <
>>>>>>>>> alexey.goncharuk@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Good,
>>>>>>>>>>
>>>>>>>>>> I think we have an intermediate agreement on the
>> scope
>>>> and
>>>>>>> significance
>>>>>>>> of
>>>>>>>>>> the changes we want to make. I suggest creating
>>>>> separate
>>>>>> discussion
>>>>>>>>>> streams
>>>>>>>>>> and calls for each of the suggested topics so that:
>>>>>>>>>>
>>>>>>>>>> - It is clear for the community what is the
>>>> motivation
>>>>> of the
>>>>>>> stream
>>>>>>>>>> (this includes both functional targets and
>>>>> technical
>>>>> debt
>>>>>> issues
>>>>>>>>>> pointed
>>>>>>>>>> out by Sergey)
>>>>>>>>>> - Who is planning to take an active part in each
>> of
>>>> the
>>>>>> streams
>>>>>>> (i.e.
>>>>>>>>>> the 'design committee', as Sergey suggested)
>>>>>>>>>> - What are the intermediate and final goals for
>>>>> each
>>>>> of the
>>>>>> streams
>>>>>>>>>> - What are the cross-stream interactions and how
>> we
>>>>>> integrate them
>>>>>>>>>> - How each of the streams will be integrated with
>>>>> the
>>>>> current
>>>>>>>> codebase
>>>>>>>>>> based on the above (here is where we will see
>>>>> whether
>>>>>> drop-in or
>>>>>>>>>> incremental approaches make more sense)
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Best regards,
>>>>>>> Andrey V. Mashenkov
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>>
>> --
>>
>> Best regards,
>> Ivan Pavlukhin
>>
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Valentin Kulichenko <va...@gmail.com>.
Hi Ivan,
There was a very brief discussion around this. Basically, it looks like
switching from Maven to something else is not going to bring much value,
but at the same time will be quite demanding because the community has much
more experience with Maven. However, I would say that it is still
debatable at this point -- please feel free to share your thoughts on this.
-Val
On Thu, Dec 17, 2020 at 10:53 PM Ivan Pavlukhin <vo...@gmail.com> wrote:
> Hi Igniters,
>
> Forgive me that I am not reading dev list carefully these days. Was it
> explicitly decided that Maven should be used as a build system for
> Ignite 3? As there is a new repository we possibly can update our
> build tools as well. What do you think?
>
> 2020-12-17 22:45 GMT+03:00, Valentin Kulichenko <
> valentin.kulichenko@gmail.com>:
> > Hi Dmitriy,
> >
> > I don't think there is any reason for concern at this point. The
> community
> > agreed on the scope of the changes for 3.0 - it is described on Wiki. The
> > scope is quite big, so it is clear that 2.x and 3.x will have to exist in
> > parallel for a significant amount of time, so we need a place where we
> can
> > merge the code for 3.x. Thus, I've created this new repo. We already have
> > multiple IEPs, as well as several contributors who are actively involved
> in
> > development. Some of the first PRs were merged today.
> >
> > I didn't hear any objections since the repo was created.
> >
> > -Val
> >
> > On Wed, Dec 16, 2020 at 7:28 AM Dmitriy Pavlov <dp...@apache.org>
> wrote:
> >
> >> Folks, I'm a little bit concerned about the simultaneous
> >> - existence of the repo https://github.com/apache/ignite-3 and PRs for
> >> that
> >> repo
> >> - and a couple of downvotes from PMC members.
> >>
> >> Is it all fine here? Was there any vote /discussion where it was
> >> discussed
> >> and consensus approved? What is the status of the ignite-3 repo?
> >>
> >> вт, 15 дек. 2020 г. в 17:30, Carbone, Adam <Adam.Carbone@bottomline.com
> >:
> >>
> >> > I don't believe Java 7 was LTS, and I hope that others will have
> >> > upgraded
> >> > long before that. If that is the release timeframe for 3.0, then I
> >> suppose
> >> > that would makes sense, I would still doubt that people would be going
> >> > newer than java 11, just my opinion of what I'm seeing.
> >> >
> >> > Regards
> >> > ~adam
> >> >
> >> > Adam Carbone | Director of Innovation – Intelligent Platform Team |
> >> > Bottomline Technologies
> >> > Office: 603-501-6446 | Mobile: 603-570-8418
> >> > www.bottomline.com
> >> >
> >> >
> >> >
> >> > On 12/15/20, 4:25 AM, "Ilya Kasnacheev" <il...@gmail.com>
> >> > wrote:
> >> >
> >> > Hello!
> >> >
> >> > I guess Ignite 3.0 will be ready for production use somewhere in
> >> 2022,
> >> > realistically. By that time, Java 8 will be long enough out of
> >> support
> >> > so
> >> > that most companies will actually forbid its use, WRT
> >> > vulnerabilities
> >> > et
> >> > all.
> >> >
> >> > After all we have managed to upgrade from Java 7 to Java 8 and
> only
> >> > got a
> >> > minor amount of complaints.
> >> >
> >> > Regards,
> >> > --
> >> > Ilya Kasnacheev
> >> >
> >> >
> >> > пн, 14 дек. 2020 г. в 19:06, Carbone, Adam <
> >> > Adam.Carbone@bottomline.com>:
> >> >
> >> > > So just one bit to consider... Are there features that you need
> >> > to
> >> > use in
> >> > > these newer versions of java? Or are we just updating to stay
> >> > current? The
> >> > > reason I ask is that there are still lots of people in an
> >> enterprise
> >> > space
> >> > > that are beholden to having to support legacy JAVAEE supported
> >> > applications
> >> > > on Websphere, Weblogic, Redhat, etc... and the organizations
> that
> >> > use those
> >> > > platforms are slow to move... Most of them are still on Java8
> >> > >
> >> > > So as a platform I think a strong consideration needs to be
> >> > towards
> >> > having
> >> > > the broadest possible support profile until it becomes an
> >> impediment
> >> > to
> >> > > doing things that the platform needs. So far I haven't seen huge
> >> > things in
> >> > > the newer versions of java that are must haves ( a few
> exceptions
> >> are
> >> > > things that would be really nice to take advantage of ).
> >> > >
> >> > > I think that apache commons has taken the right approach by
> >> > staying
> >> > on
> >> > > java 8 giving the largest possible user base.
> >> > >
> >> > > Even standardizing on java 11 would have to make us reconsider
> >> > Ignite as
> >> > > the platform we are using, we are not so invested in it as of
> >> > now,
> >> > even
> >> > > though we have big plans to leverage it. Just because we aren't
> >> sure
> >> > when
> >> > > we are going to be able to upgrade from java8. It has support
> >> > through 2022,
> >> > > so I imagine that is when we will be discussing that.
> >> > >
> >> > > Regards
> >> > >
> >> > > ~Adam
> >> > >
> >> > > Adam Carbone | Director of Innovation – Intelligent Platform
> Team
> >> > |
> >> > > Bottomline Technologies
> >> > > Office: 603-501-6446 | Mobile: 603-570-8418
> >> > > www.bottomline.com
> >> > >
> >> > >
> >> > >
> >> > > On 11/24/20, 7:38 AM, "Alexey Zinoviev" <zaleslaw.sin@gmail.com
> >
> >> > wrote:
> >> > >
> >> > > Java 15 is better, VarHandles, ForeignMemory access and so
> >> > on.
> >> > >
> >> > > In both cases, I support the Java version 11 and higher for
> >> > the
> >> > > development!
> >> > >
> >> > > вт, 24 нояб. 2020 г. в 15:21, Andrey Mashenkov <
> >> > > andrey.mashenkov@gmail.com>:
> >> > >
> >> > > > Let's add maven plugins for sanity checks at the early
> >> stage.
> >> > > > I've created a ticket for this [1].
> >> > > >
> >> > > > Also, I've found initial pom.xml has a target version Java
> >> > 8.
> >> > > > Do we intend to move to Java 11 (or may be next LTS) and
> >> > drop
> >> > Java 8
> >> > > in
> >> > > > Ignite 3.0?
> >> > > >
> >> > > > [1]
> >> > >
> >> >
> >>
> https://urldefense.com/v3/__https://issues.apache.org/jira/browse/IGNITE-13751__;!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKAO2Ejs8$
> >> > > >
> >> > > > On Tue, Nov 24, 2020 at 5:40 AM Valentin Kulichenko <
> >> > > > valentin.kulichenko@gmail.com> wrote:
> >> > > >
> >> > > > > Folks,
> >> > > > >
> >> > > > > I went ahead and created the repository [1]. I also
> >> > configured a
> >> > > TeamCity
> >> > > > > project [2] that runs all available JUnit tests on every
> >> > PR
> >> > > creation or
> >> > > > > update. It also sends the status update to GitHub so
> that
> >> > it's
> >> > > reflected
> >> > > > in
> >> > > > > the PR itself so that we can do merges directly from
> >> GitHub.
> >> > Basic
> >> > > steps
> >> > > > to
> >> > > > > make a change are described on the Wiki page [3].
> >> > > > >
> >> > > > > Let me know if you have any questions.
> >> > > > >
> >> > > > > [1]
> >> > >
> >> >
> >>
> https://urldefense.com/v3/__https://github.com/apache/ignite-3__;!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKIq24lxF$
> >> > > > > [2]
> >> > >
> >> >
> >>
> https://urldefense.com/v3/__https://ci.ignite.apache.org/project/ignite3__;!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKFGL_oJx$
> >> > > > > [3]
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/IGNITE/Apache*Ignite*3.0*ApacheIgnite3.0-DevelopmentProcess__;Kysj!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKNhWzQ0s$
> >> > > > >
> >> > > > > -Val
> >> > > > >
> >> > > > > On Wed, Nov 18, 2020 at 4:24 PM Valentin Kulichenko <
> >> > > > > valentin.kulichenko@gmail.com> wrote:
> >> > > > >
> >> > > > > > Thanks, guys. It looks like we are much closer to the
> >> > consensus
> >> > > now. I
> >> > > > > > totally on board with the plan, but I would also like
> >> > to
> >> > address
> >> > > the
> >> > > > > > short-term needs. As I've already mentioned earlier,
> >> there
> >> > are
> >> > > several
> >> > > > > > active IEPs, but we still don't have even a
> preliminary
> >> > technical
> >> > > > process
> >> > > > > > for working on these IEPs. I believe this might be
> >> > frustrating
> >> > > for the
> >> > > > > > folks who would like to commit code.
> >> > > > > >
> >> > > > > > The scope we agreed on is quite big, and it will
> surely
> >> > take
> >> > > > significant
> >> > > > > > time to implement all the changes and stabilize them.
> >> > Therefore,
> >> > > it's
> >> > > > > clear
> >> > > > > > to me that we will have to maintain 2.x and 3.x in
> >> > parallel for
> >> > > quite
> >> > > > > some
> >> > > > > > time - this needs to be addressed somehow. I'm
> >> > convinced
> >> > that
> >> > > having a
> >> > > > > > separate repo is the ONLY way to do that, and so far,
> I
> >> > haven't
> >> > > heard
> >> > > > any
> >> > > > > > clear alternatives or reasons why we shouldn't do
> this.
> >> > > > > >
> >> > > > > > That said, I'm inclined to proceed with this in the
> >> > next
> >> > few
> >> > > days - I
> >> > > > > will
> >> > > > > > create a repo and describe the process (which we, of
> >> > course, can
> >> > > > discuss
> >> > > > > > and modify going forward). Let's, at the very least,
> >> > try
> >> > and see
> >> > > where
> >> > > > it
> >> > > > > > leads us.
> >> > > > > >
> >> > > > > > If someone has any concrete alternative options on how
> >> > to
> >> > we can
> >> > > > maintain
> >> > > > > > two major versions in parallel, let's have another
> >> > voice
> >> > > discussion
> >> > > > this
> >> > > > > > Friday. If we do the meeting, we should set it up with
> >> > a
> >> > clear
> >> > > goal to
> >> > > > > make
> >> > > > > > a decision. Please let me know if there is interest in
> >> > this.
> >> > > > > >
> >> > > > > > -Val
> >> > > > > >
> >> > > > > > On Mon, Nov 16, 2020 at 6:31 AM Alexey Goncharuk <
> >> > > > > > alexey.goncharuk@gmail.com> wrote:
> >> > > > > >
> >> > > > > >> Good,
> >> > > > > >>
> >> > > > > >> I think we have an intermediate agreement on the
> scope
> >> and
> >> > > > significance
> >> > > > > of
> >> > > > > >> the changes we want to make. I suggest creating
> >> > separate
> >> > > discussion
> >> > > > > >> streams
> >> > > > > >> and calls for each of the suggested topics so that:
> >> > > > > >>
> >> > > > > >> - It is clear for the community what is the
> >> motivation
> >> > of the
> >> > > > stream
> >> > > > > >> (this includes both functional targets and
> >> > technical
> >> > debt
> >> > > issues
> >> > > > > >> pointed
> >> > > > > >> out by Sergey)
> >> > > > > >> - Who is planning to take an active part in each
> of
> >> the
> >> > > streams
> >> > > > (i.e.
> >> > > > > >> the 'design committee', as Sergey suggested)
> >> > > > > >> - What are the intermediate and final goals for
> >> > each
> >> > of the
> >> > > streams
> >> > > > > >> - What are the cross-stream interactions and how
> we
> >> > > integrate them
> >> > > > > >> - How each of the streams will be integrated with
> >> > the
> >> > current
> >> > > > > codebase
> >> > > > > >> based on the above (here is where we will see
> >> > whether
> >> > > drop-in or
> >> > > > > >> incremental approaches make more sense)
> >> > > > > >>
> >> > > > > >
> >> > > > >
> >> > > >
> >> > > >
> >> > > > --
> >> > > > Best regards,
> >> > > > Andrey V. Mashenkov
> >> > > >
> >> > >
> >> > >
> >> >
> >> >
> >>
> >
>
>
> --
>
> Best regards,
> Ivan Pavlukhin
>
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Ivan Pavlukhin <vo...@gmail.com>.
Hi Igniters,
Forgive me that I am not reading dev list carefully these days. Was it
explicitly decided that Maven should be used as a build system for
Ignite 3? As there is a new repository we possibly can update our
build tools as well. What do you think?
2020-12-17 22:45 GMT+03:00, Valentin Kulichenko <va...@gmail.com>:
> Hi Dmitriy,
>
> I don't think there is any reason for concern at this point. The community
> agreed on the scope of the changes for 3.0 - it is described on Wiki. The
> scope is quite big, so it is clear that 2.x and 3.x will have to exist in
> parallel for a significant amount of time, so we need a place where we can
> merge the code for 3.x. Thus, I've created this new repo. We already have
> multiple IEPs, as well as several contributors who are actively involved in
> development. Some of the first PRs were merged today.
>
> I didn't hear any objections since the repo was created.
>
> -Val
>
> On Wed, Dec 16, 2020 at 7:28 AM Dmitriy Pavlov <dp...@apache.org> wrote:
>
>> Folks, I'm a little bit concerned about the simultaneous
>> - existence of the repo https://github.com/apache/ignite-3 and PRs for
>> that
>> repo
>> - and a couple of downvotes from PMC members.
>>
>> Is it all fine here? Was there any vote /discussion where it was
>> discussed
>> and consensus approved? What is the status of the ignite-3 repo?
>>
>> вт, 15 дек. 2020 г. в 17:30, Carbone, Adam <Ad...@bottomline.com>:
>>
>> > I don't believe Java 7 was LTS, and I hope that others will have
>> > upgraded
>> > long before that. If that is the release timeframe for 3.0, then I
>> suppose
>> > that would makes sense, I would still doubt that people would be going
>> > newer than java 11, just my opinion of what I'm seeing.
>> >
>> > Regards
>> > ~adam
>> >
>> > Adam Carbone | Director of Innovation – Intelligent Platform Team |
>> > Bottomline Technologies
>> > Office: 603-501-6446 | Mobile: 603-570-8418
>> > www.bottomline.com
>> >
>> >
>> >
>> > On 12/15/20, 4:25 AM, "Ilya Kasnacheev" <il...@gmail.com>
>> > wrote:
>> >
>> > Hello!
>> >
>> > I guess Ignite 3.0 will be ready for production use somewhere in
>> 2022,
>> > realistically. By that time, Java 8 will be long enough out of
>> support
>> > so
>> > that most companies will actually forbid its use, WRT
>> > vulnerabilities
>> > et
>> > all.
>> >
>> > After all we have managed to upgrade from Java 7 to Java 8 and only
>> > got a
>> > minor amount of complaints.
>> >
>> > Regards,
>> > --
>> > Ilya Kasnacheev
>> >
>> >
>> > пн, 14 дек. 2020 г. в 19:06, Carbone, Adam <
>> > Adam.Carbone@bottomline.com>:
>> >
>> > > So just one bit to consider... Are there features that you need
>> > to
>> > use in
>> > > these newer versions of java? Or are we just updating to stay
>> > current? The
>> > > reason I ask is that there are still lots of people in an
>> enterprise
>> > space
>> > > that are beholden to having to support legacy JAVAEE supported
>> > applications
>> > > on Websphere, Weblogic, Redhat, etc... and the organizations that
>> > use those
>> > > platforms are slow to move... Most of them are still on Java8
>> > >
>> > > So as a platform I think a strong consideration needs to be
>> > towards
>> > having
>> > > the broadest possible support profile until it becomes an
>> impediment
>> > to
>> > > doing things that the platform needs. So far I haven't seen huge
>> > things in
>> > > the newer versions of java that are must haves ( a few exceptions
>> are
>> > > things that would be really nice to take advantage of ).
>> > >
>> > > I think that apache commons has taken the right approach by
>> > staying
>> > on
>> > > java 8 giving the largest possible user base.
>> > >
>> > > Even standardizing on java 11 would have to make us reconsider
>> > Ignite as
>> > > the platform we are using, we are not so invested in it as of
>> > now,
>> > even
>> > > though we have big plans to leverage it. Just because we aren't
>> sure
>> > when
>> > > we are going to be able to upgrade from java8. It has support
>> > through 2022,
>> > > so I imagine that is when we will be discussing that.
>> > >
>> > > Regards
>> > >
>> > > ~Adam
>> > >
>> > > Adam Carbone | Director of Innovation – Intelligent Platform Team
>> > |
>> > > Bottomline Technologies
>> > > Office: 603-501-6446 | Mobile: 603-570-8418
>> > > www.bottomline.com
>> > >
>> > >
>> > >
>> > > On 11/24/20, 7:38 AM, "Alexey Zinoviev" <za...@gmail.com>
>> > wrote:
>> > >
>> > > Java 15 is better, VarHandles, ForeignMemory access and so
>> > on.
>> > >
>> > > In both cases, I support the Java version 11 and higher for
>> > the
>> > > development!
>> > >
>> > > вт, 24 нояб. 2020 г. в 15:21, Andrey Mashenkov <
>> > > andrey.mashenkov@gmail.com>:
>> > >
>> > > > Let's add maven plugins for sanity checks at the early
>> stage.
>> > > > I've created a ticket for this [1].
>> > > >
>> > > > Also, I've found initial pom.xml has a target version Java
>> > 8.
>> > > > Do we intend to move to Java 11 (or may be next LTS) and
>> > drop
>> > Java 8
>> > > in
>> > > > Ignite 3.0?
>> > > >
>> > > > [1]
>> > >
>> >
>> https://urldefense.com/v3/__https://issues.apache.org/jira/browse/IGNITE-13751__;!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKAO2Ejs8$
>> > > >
>> > > > On Tue, Nov 24, 2020 at 5:40 AM Valentin Kulichenko <
>> > > > valentin.kulichenko@gmail.com> wrote:
>> > > >
>> > > > > Folks,
>> > > > >
>> > > > > I went ahead and created the repository [1]. I also
>> > configured a
>> > > TeamCity
>> > > > > project [2] that runs all available JUnit tests on every
>> > PR
>> > > creation or
>> > > > > update. It also sends the status update to GitHub so that
>> > it's
>> > > reflected
>> > > > in
>> > > > > the PR itself so that we can do merges directly from
>> GitHub.
>> > Basic
>> > > steps
>> > > > to
>> > > > > make a change are described on the Wiki page [3].
>> > > > >
>> > > > > Let me know if you have any questions.
>> > > > >
>> > > > > [1]
>> > >
>> >
>> https://urldefense.com/v3/__https://github.com/apache/ignite-3__;!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKIq24lxF$
>> > > > > [2]
>> > >
>> >
>> https://urldefense.com/v3/__https://ci.ignite.apache.org/project/ignite3__;!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKFGL_oJx$
>> > > > > [3]
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/IGNITE/Apache*Ignite*3.0*ApacheIgnite3.0-DevelopmentProcess__;Kysj!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKNhWzQ0s$
>> > > > >
>> > > > > -Val
>> > > > >
>> > > > > On Wed, Nov 18, 2020 at 4:24 PM Valentin Kulichenko <
>> > > > > valentin.kulichenko@gmail.com> wrote:
>> > > > >
>> > > > > > Thanks, guys. It looks like we are much closer to the
>> > consensus
>> > > now. I
>> > > > > > totally on board with the plan, but I would also like
>> > to
>> > address
>> > > the
>> > > > > > short-term needs. As I've already mentioned earlier,
>> there
>> > are
>> > > several
>> > > > > > active IEPs, but we still don't have even a preliminary
>> > technical
>> > > > process
>> > > > > > for working on these IEPs. I believe this might be
>> > frustrating
>> > > for the
>> > > > > > folks who would like to commit code.
>> > > > > >
>> > > > > > The scope we agreed on is quite big, and it will surely
>> > take
>> > > > significant
>> > > > > > time to implement all the changes and stabilize them.
>> > Therefore,
>> > > it's
>> > > > > clear
>> > > > > > to me that we will have to maintain 2.x and 3.x in
>> > parallel for
>> > > quite
>> > > > > some
>> > > > > > time - this needs to be addressed somehow. I'm
>> > convinced
>> > that
>> > > having a
>> > > > > > separate repo is the ONLY way to do that, and so far, I
>> > haven't
>> > > heard
>> > > > any
>> > > > > > clear alternatives or reasons why we shouldn't do this.
>> > > > > >
>> > > > > > That said, I'm inclined to proceed with this in the
>> > next
>> > few
>> > > days - I
>> > > > > will
>> > > > > > create a repo and describe the process (which we, of
>> > course, can
>> > > > discuss
>> > > > > > and modify going forward). Let's, at the very least,
>> > try
>> > and see
>> > > where
>> > > > it
>> > > > > > leads us.
>> > > > > >
>> > > > > > If someone has any concrete alternative options on how
>> > to
>> > we can
>> > > > maintain
>> > > > > > two major versions in parallel, let's have another
>> > voice
>> > > discussion
>> > > > this
>> > > > > > Friday. If we do the meeting, we should set it up with
>> > a
>> > clear
>> > > goal to
>> > > > > make
>> > > > > > a decision. Please let me know if there is interest in
>> > this.
>> > > > > >
>> > > > > > -Val
>> > > > > >
>> > > > > > On Mon, Nov 16, 2020 at 6:31 AM Alexey Goncharuk <
>> > > > > > alexey.goncharuk@gmail.com> wrote:
>> > > > > >
>> > > > > >> Good,
>> > > > > >>
>> > > > > >> I think we have an intermediate agreement on the scope
>> and
>> > > > significance
>> > > > > of
>> > > > > >> the changes we want to make. I suggest creating
>> > separate
>> > > discussion
>> > > > > >> streams
>> > > > > >> and calls for each of the suggested topics so that:
>> > > > > >>
>> > > > > >> - It is clear for the community what is the
>> motivation
>> > of the
>> > > > stream
>> > > > > >> (this includes both functional targets and
>> > technical
>> > debt
>> > > issues
>> > > > > >> pointed
>> > > > > >> out by Sergey)
>> > > > > >> - Who is planning to take an active part in each of
>> the
>> > > streams
>> > > > (i.e.
>> > > > > >> the 'design committee', as Sergey suggested)
>> > > > > >> - What are the intermediate and final goals for
>> > each
>> > of the
>> > > streams
>> > > > > >> - What are the cross-stream interactions and how we
>> > > integrate them
>> > > > > >> - How each of the streams will be integrated with
>> > the
>> > current
>> > > > > codebase
>> > > > > >> based on the above (here is where we will see
>> > whether
>> > > drop-in or
>> > > > > >> incremental approaches make more sense)
>> > > > > >>
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > > > --
>> > > > Best regards,
>> > > > Andrey V. Mashenkov
>> > > >
>> > >
>> > >
>> >
>> >
>>
>
--
Best regards,
Ivan Pavlukhin
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Valentin Kulichenko <va...@gmail.com>.
Hi Dmitriy,
I don't think there is any reason for concern at this point. The community
agreed on the scope of the changes for 3.0 - it is described on Wiki. The
scope is quite big, so it is clear that 2.x and 3.x will have to exist in
parallel for a significant amount of time, so we need a place where we can
merge the code for 3.x. Thus, I've created this new repo. We already have
multiple IEPs, as well as several contributors who are actively involved in
development. Some of the first PRs were merged today.
I didn't hear any objections since the repo was created.
-Val
On Wed, Dec 16, 2020 at 7:28 AM Dmitriy Pavlov <dp...@apache.org> wrote:
> Folks, I'm a little bit concerned about the simultaneous
> - existence of the repo https://github.com/apache/ignite-3 and PRs for
> that
> repo
> - and a couple of downvotes from PMC members.
>
> Is it all fine here? Was there any vote /discussion where it was discussed
> and consensus approved? What is the status of the ignite-3 repo?
>
> вт, 15 дек. 2020 г. в 17:30, Carbone, Adam <Ad...@bottomline.com>:
>
> > I don't believe Java 7 was LTS, and I hope that others will have upgraded
> > long before that. If that is the release timeframe for 3.0, then I
> suppose
> > that would makes sense, I would still doubt that people would be going
> > newer than java 11, just my opinion of what I'm seeing.
> >
> > Regards
> > ~adam
> >
> > Adam Carbone | Director of Innovation – Intelligent Platform Team |
> > Bottomline Technologies
> > Office: 603-501-6446 | Mobile: 603-570-8418
> > www.bottomline.com
> >
> >
> >
> > On 12/15/20, 4:25 AM, "Ilya Kasnacheev" <il...@gmail.com>
> > wrote:
> >
> > Hello!
> >
> > I guess Ignite 3.0 will be ready for production use somewhere in
> 2022,
> > realistically. By that time, Java 8 will be long enough out of
> support
> > so
> > that most companies will actually forbid its use, WRT vulnerabilities
> > et
> > all.
> >
> > After all we have managed to upgrade from Java 7 to Java 8 and only
> > got a
> > minor amount of complaints.
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > пн, 14 дек. 2020 г. в 19:06, Carbone, Adam <
> > Adam.Carbone@bottomline.com>:
> >
> > > So just one bit to consider... Are there features that you need to
> > use in
> > > these newer versions of java? Or are we just updating to stay
> > current? The
> > > reason I ask is that there are still lots of people in an
> enterprise
> > space
> > > that are beholden to having to support legacy JAVAEE supported
> > applications
> > > on Websphere, Weblogic, Redhat, etc... and the organizations that
> > use those
> > > platforms are slow to move... Most of them are still on Java8
> > >
> > > So as a platform I think a strong consideration needs to be towards
> > having
> > > the broadest possible support profile until it becomes an
> impediment
> > to
> > > doing things that the platform needs. So far I haven't seen huge
> > things in
> > > the newer versions of java that are must haves ( a few exceptions
> are
> > > things that would be really nice to take advantage of ).
> > >
> > > I think that apache commons has taken the right approach by staying
> > on
> > > java 8 giving the largest possible user base.
> > >
> > > Even standardizing on java 11 would have to make us reconsider
> > Ignite as
> > > the platform we are using, we are not so invested in it as of now,
> > even
> > > though we have big plans to leverage it. Just because we aren't
> sure
> > when
> > > we are going to be able to upgrade from java8. It has support
> > through 2022,
> > > so I imagine that is when we will be discussing that.
> > >
> > > Regards
> > >
> > > ~Adam
> > >
> > > Adam Carbone | Director of Innovation – Intelligent Platform Team |
> > > Bottomline Technologies
> > > Office: 603-501-6446 | Mobile: 603-570-8418
> > > www.bottomline.com
> > >
> > >
> > >
> > > On 11/24/20, 7:38 AM, "Alexey Zinoviev" <za...@gmail.com>
> > wrote:
> > >
> > > Java 15 is better, VarHandles, ForeignMemory access and so on.
> > >
> > > In both cases, I support the Java version 11 and higher for the
> > > development!
> > >
> > > вт, 24 нояб. 2020 г. в 15:21, Andrey Mashenkov <
> > > andrey.mashenkov@gmail.com>:
> > >
> > > > Let's add maven plugins for sanity checks at the early
> stage.
> > > > I've created a ticket for this [1].
> > > >
> > > > Also, I've found initial pom.xml has a target version Java 8.
> > > > Do we intend to move to Java 11 (or may be next LTS) and drop
> > Java 8
> > > in
> > > > Ignite 3.0?
> > > >
> > > > [1]
> > >
> >
> https://urldefense.com/v3/__https://issues.apache.org/jira/browse/IGNITE-13751__;!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKAO2Ejs8$
> > > >
> > > > On Tue, Nov 24, 2020 at 5:40 AM Valentin Kulichenko <
> > > > valentin.kulichenko@gmail.com> wrote:
> > > >
> > > > > Folks,
> > > > >
> > > > > I went ahead and created the repository [1]. I also
> > configured a
> > > TeamCity
> > > > > project [2] that runs all available JUnit tests on every PR
> > > creation or
> > > > > update. It also sends the status update to GitHub so that
> > it's
> > > reflected
> > > > in
> > > > > the PR itself so that we can do merges directly from
> GitHub.
> > Basic
> > > steps
> > > > to
> > > > > make a change are described on the Wiki page [3].
> > > > >
> > > > > Let me know if you have any questions.
> > > > >
> > > > > [1]
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/ignite-3__;!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKIq24lxF$
> > > > > [2]
> > >
> >
> https://urldefense.com/v3/__https://ci.ignite.apache.org/project/ignite3__;!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKFGL_oJx$
> > > > > [3]
> > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/IGNITE/Apache*Ignite*3.0*ApacheIgnite3.0-DevelopmentProcess__;Kysj!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKNhWzQ0s$
> > > > >
> > > > > -Val
> > > > >
> > > > > On Wed, Nov 18, 2020 at 4:24 PM Valentin Kulichenko <
> > > > > valentin.kulichenko@gmail.com> wrote:
> > > > >
> > > > > > Thanks, guys. It looks like we are much closer to the
> > consensus
> > > now. I
> > > > > > totally on board with the plan, but I would also like to
> > address
> > > the
> > > > > > short-term needs. As I've already mentioned earlier,
> there
> > are
> > > several
> > > > > > active IEPs, but we still don't have even a preliminary
> > technical
> > > > process
> > > > > > for working on these IEPs. I believe this might be
> > frustrating
> > > for the
> > > > > > folks who would like to commit code.
> > > > > >
> > > > > > The scope we agreed on is quite big, and it will surely
> > take
> > > > significant
> > > > > > time to implement all the changes and stabilize them.
> > Therefore,
> > > it's
> > > > > clear
> > > > > > to me that we will have to maintain 2.x and 3.x in
> > parallel for
> > > quite
> > > > > some
> > > > > > time - this needs to be addressed somehow. I'm convinced
> > that
> > > having a
> > > > > > separate repo is the ONLY way to do that, and so far, I
> > haven't
> > > heard
> > > > any
> > > > > > clear alternatives or reasons why we shouldn't do this.
> > > > > >
> > > > > > That said, I'm inclined to proceed with this in the next
> > few
> > > days - I
> > > > > will
> > > > > > create a repo and describe the process (which we, of
> > course, can
> > > > discuss
> > > > > > and modify going forward). Let's, at the very least, try
> > and see
> > > where
> > > > it
> > > > > > leads us.
> > > > > >
> > > > > > If someone has any concrete alternative options on how to
> > we can
> > > > maintain
> > > > > > two major versions in parallel, let's have another voice
> > > discussion
> > > > this
> > > > > > Friday. If we do the meeting, we should set it up with a
> > clear
> > > goal to
> > > > > make
> > > > > > a decision. Please let me know if there is interest in
> > this.
> > > > > >
> > > > > > -Val
> > > > > >
> > > > > > On Mon, Nov 16, 2020 at 6:31 AM Alexey Goncharuk <
> > > > > > alexey.goncharuk@gmail.com> wrote:
> > > > > >
> > > > > >> Good,
> > > > > >>
> > > > > >> I think we have an intermediate agreement on the scope
> and
> > > > significance
> > > > > of
> > > > > >> the changes we want to make. I suggest creating separate
> > > discussion
> > > > > >> streams
> > > > > >> and calls for each of the suggested topics so that:
> > > > > >>
> > > > > >> - It is clear for the community what is the
> motivation
> > of the
> > > > stream
> > > > > >> (this includes both functional targets and technical
> > debt
> > > issues
> > > > > >> pointed
> > > > > >> out by Sergey)
> > > > > >> - Who is planning to take an active part in each of
> the
> > > streams
> > > > (i.e.
> > > > > >> the 'design committee', as Sergey suggested)
> > > > > >> - What are the intermediate and final goals for each
> > of the
> > > streams
> > > > > >> - What are the cross-stream interactions and how we
> > > integrate them
> > > > > >> - How each of the streams will be integrated with the
> > current
> > > > > codebase
> > > > > >> based on the above (here is where we will see whether
> > > drop-in or
> > > > > >> incremental approaches make more sense)
> > > > > >>
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > > Andrey V. Mashenkov
> > > >
> > >
> > >
> >
> >
>
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Dmitriy Pavlov <dp...@apache.org>.
Folks, I'm a little bit concerned about the simultaneous
- existence of the repo https://github.com/apache/ignite-3 and PRs for that
repo
- and a couple of downvotes from PMC members.
Is it all fine here? Was there any vote /discussion where it was discussed
and consensus approved? What is the status of the ignite-3 repo?
вт, 15 дек. 2020 г. в 17:30, Carbone, Adam <Ad...@bottomline.com>:
> I don't believe Java 7 was LTS, and I hope that others will have upgraded
> long before that. If that is the release timeframe for 3.0, then I suppose
> that would makes sense, I would still doubt that people would be going
> newer than java 11, just my opinion of what I'm seeing.
>
> Regards
> ~adam
>
> Adam Carbone | Director of Innovation – Intelligent Platform Team |
> Bottomline Technologies
> Office: 603-501-6446 | Mobile: 603-570-8418
> www.bottomline.com
>
>
>
> On 12/15/20, 4:25 AM, "Ilya Kasnacheev" <il...@gmail.com>
> wrote:
>
> Hello!
>
> I guess Ignite 3.0 will be ready for production use somewhere in 2022,
> realistically. By that time, Java 8 will be long enough out of support
> so
> that most companies will actually forbid its use, WRT vulnerabilities
> et
> all.
>
> After all we have managed to upgrade from Java 7 to Java 8 and only
> got a
> minor amount of complaints.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пн, 14 дек. 2020 г. в 19:06, Carbone, Adam <
> Adam.Carbone@bottomline.com>:
>
> > So just one bit to consider... Are there features that you need to
> use in
> > these newer versions of java? Or are we just updating to stay
> current? The
> > reason I ask is that there are still lots of people in an enterprise
> space
> > that are beholden to having to support legacy JAVAEE supported
> applications
> > on Websphere, Weblogic, Redhat, etc... and the organizations that
> use those
> > platforms are slow to move... Most of them are still on Java8
> >
> > So as a platform I think a strong consideration needs to be towards
> having
> > the broadest possible support profile until it becomes an impediment
> to
> > doing things that the platform needs. So far I haven't seen huge
> things in
> > the newer versions of java that are must haves ( a few exceptions are
> > things that would be really nice to take advantage of ).
> >
> > I think that apache commons has taken the right approach by staying
> on
> > java 8 giving the largest possible user base.
> >
> > Even standardizing on java 11 would have to make us reconsider
> Ignite as
> > the platform we are using, we are not so invested in it as of now,
> even
> > though we have big plans to leverage it. Just because we aren't sure
> when
> > we are going to be able to upgrade from java8. It has support
> through 2022,
> > so I imagine that is when we will be discussing that.
> >
> > Regards
> >
> > ~Adam
> >
> > Adam Carbone | Director of Innovation – Intelligent Platform Team |
> > Bottomline Technologies
> > Office: 603-501-6446 | Mobile: 603-570-8418
> > www.bottomline.com
> >
> >
> >
> > On 11/24/20, 7:38 AM, "Alexey Zinoviev" <za...@gmail.com>
> wrote:
> >
> > Java 15 is better, VarHandles, ForeignMemory access and so on.
> >
> > In both cases, I support the Java version 11 and higher for the
> > development!
> >
> > вт, 24 нояб. 2020 г. в 15:21, Andrey Mashenkov <
> > andrey.mashenkov@gmail.com>:
> >
> > > Let's add maven plugins for sanity checks at the early stage.
> > > I've created a ticket for this [1].
> > >
> > > Also, I've found initial pom.xml has a target version Java 8.
> > > Do we intend to move to Java 11 (or may be next LTS) and drop
> Java 8
> > in
> > > Ignite 3.0?
> > >
> > > [1]
> >
> https://urldefense.com/v3/__https://issues.apache.org/jira/browse/IGNITE-13751__;!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKAO2Ejs8$
> > >
> > > On Tue, Nov 24, 2020 at 5:40 AM Valentin Kulichenko <
> > > valentin.kulichenko@gmail.com> wrote:
> > >
> > > > Folks,
> > > >
> > > > I went ahead and created the repository [1]. I also
> configured a
> > TeamCity
> > > > project [2] that runs all available JUnit tests on every PR
> > creation or
> > > > update. It also sends the status update to GitHub so that
> it's
> > reflected
> > > in
> > > > the PR itself so that we can do merges directly from GitHub.
> Basic
> > steps
> > > to
> > > > make a change are described on the Wiki page [3].
> > > >
> > > > Let me know if you have any questions.
> > > >
> > > > [1]
> >
> https://urldefense.com/v3/__https://github.com/apache/ignite-3__;!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKIq24lxF$
> > > > [2]
> >
> https://urldefense.com/v3/__https://ci.ignite.apache.org/project/ignite3__;!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKFGL_oJx$
> > > > [3]
> > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/IGNITE/Apache*Ignite*3.0*ApacheIgnite3.0-DevelopmentProcess__;Kysj!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKNhWzQ0s$
> > > >
> > > > -Val
> > > >
> > > > On Wed, Nov 18, 2020 at 4:24 PM Valentin Kulichenko <
> > > > valentin.kulichenko@gmail.com> wrote:
> > > >
> > > > > Thanks, guys. It looks like we are much closer to the
> consensus
> > now. I
> > > > > totally on board with the plan, but I would also like to
> address
> > the
> > > > > short-term needs. As I've already mentioned earlier, there
> are
> > several
> > > > > active IEPs, but we still don't have even a preliminary
> technical
> > > process
> > > > > for working on these IEPs. I believe this might be
> frustrating
> > for the
> > > > > folks who would like to commit code.
> > > > >
> > > > > The scope we agreed on is quite big, and it will surely
> take
> > > significant
> > > > > time to implement all the changes and stabilize them.
> Therefore,
> > it's
> > > > clear
> > > > > to me that we will have to maintain 2.x and 3.x in
> parallel for
> > quite
> > > > some
> > > > > time - this needs to be addressed somehow. I'm convinced
> that
> > having a
> > > > > separate repo is the ONLY way to do that, and so far, I
> haven't
> > heard
> > > any
> > > > > clear alternatives or reasons why we shouldn't do this.
> > > > >
> > > > > That said, I'm inclined to proceed with this in the next
> few
> > days - I
> > > > will
> > > > > create a repo and describe the process (which we, of
> course, can
> > > discuss
> > > > > and modify going forward). Let's, at the very least, try
> and see
> > where
> > > it
> > > > > leads us.
> > > > >
> > > > > If someone has any concrete alternative options on how to
> we can
> > > maintain
> > > > > two major versions in parallel, let's have another voice
> > discussion
> > > this
> > > > > Friday. If we do the meeting, we should set it up with a
> clear
> > goal to
> > > > make
> > > > > a decision. Please let me know if there is interest in
> this.
> > > > >
> > > > > -Val
> > > > >
> > > > > On Mon, Nov 16, 2020 at 6:31 AM Alexey Goncharuk <
> > > > > alexey.goncharuk@gmail.com> wrote:
> > > > >
> > > > >> Good,
> > > > >>
> > > > >> I think we have an intermediate agreement on the scope and
> > > significance
> > > > of
> > > > >> the changes we want to make. I suggest creating separate
> > discussion
> > > > >> streams
> > > > >> and calls for each of the suggested topics so that:
> > > > >>
> > > > >> - It is clear for the community what is the motivation
> of the
> > > stream
> > > > >> (this includes both functional targets and technical
> debt
> > issues
> > > > >> pointed
> > > > >> out by Sergey)
> > > > >> - Who is planning to take an active part in each of the
> > streams
> > > (i.e.
> > > > >> the 'design committee', as Sergey suggested)
> > > > >> - What are the intermediate and final goals for each
> of the
> > streams
> > > > >> - What are the cross-stream interactions and how we
> > integrate them
> > > > >> - How each of the streams will be integrated with the
> current
> > > > codebase
> > > > >> based on the above (here is where we will see whether
> > drop-in or
> > > > >> incremental approaches make more sense)
> > > > >>
> > > > >
> > > >
> > >
> > >
> > > --
> > > Best regards,
> > > Andrey V. Mashenkov
> > >
> >
> >
>
>
Re: [DISCUSS] Ignite 3.0 development approach
Posted by "Carbone, Adam" <Ad...@bottomline.com>.
I don't believe Java 7 was LTS, and I hope that others will have upgraded long before that. If that is the release timeframe for 3.0, then I suppose that would makes sense, I would still doubt that people would be going newer than java 11, just my opinion of what I'm seeing.
Regards
~adam
Adam Carbone | Director of Innovation – Intelligent Platform Team | Bottomline Technologies
Office: 603-501-6446 | Mobile: 603-570-8418
www.bottomline.com
On 12/15/20, 4:25 AM, "Ilya Kasnacheev" <il...@gmail.com> wrote:
Hello!
I guess Ignite 3.0 will be ready for production use somewhere in 2022,
realistically. By that time, Java 8 will be long enough out of support so
that most companies will actually forbid its use, WRT vulnerabilities et
all.
After all we have managed to upgrade from Java 7 to Java 8 and only got a
minor amount of complaints.
Regards,
--
Ilya Kasnacheev
пн, 14 дек. 2020 г. в 19:06, Carbone, Adam <Ad...@bottomline.com>:
> So just one bit to consider... Are there features that you need to use in
> these newer versions of java? Or are we just updating to stay current? The
> reason I ask is that there are still lots of people in an enterprise space
> that are beholden to having to support legacy JAVAEE supported applications
> on Websphere, Weblogic, Redhat, etc... and the organizations that use those
> platforms are slow to move... Most of them are still on Java8
>
> So as a platform I think a strong consideration needs to be towards having
> the broadest possible support profile until it becomes an impediment to
> doing things that the platform needs. So far I haven't seen huge things in
> the newer versions of java that are must haves ( a few exceptions are
> things that would be really nice to take advantage of ).
>
> I think that apache commons has taken the right approach by staying on
> java 8 giving the largest possible user base.
>
> Even standardizing on java 11 would have to make us reconsider Ignite as
> the platform we are using, we are not so invested in it as of now, even
> though we have big plans to leverage it. Just because we aren't sure when
> we are going to be able to upgrade from java8. It has support through 2022,
> so I imagine that is when we will be discussing that.
>
> Regards
>
> ~Adam
>
> Adam Carbone | Director of Innovation – Intelligent Platform Team |
> Bottomline Technologies
> Office: 603-501-6446 | Mobile: 603-570-8418
> www.bottomline.com
>
>
>
> On 11/24/20, 7:38 AM, "Alexey Zinoviev" <za...@gmail.com> wrote:
>
> Java 15 is better, VarHandles, ForeignMemory access and so on.
>
> In both cases, I support the Java version 11 and higher for the
> development!
>
> вт, 24 нояб. 2020 г. в 15:21, Andrey Mashenkov <
> andrey.mashenkov@gmail.com>:
>
> > Let's add maven plugins for sanity checks at the early stage.
> > I've created a ticket for this [1].
> >
> > Also, I've found initial pom.xml has a target version Java 8.
> > Do we intend to move to Java 11 (or may be next LTS) and drop Java 8
> in
> > Ignite 3.0?
> >
> > [1]
> https://urldefense.com/v3/__https://issues.apache.org/jira/browse/IGNITE-13751__;!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKAO2Ejs8$
> >
> > On Tue, Nov 24, 2020 at 5:40 AM Valentin Kulichenko <
> > valentin.kulichenko@gmail.com> wrote:
> >
> > > Folks,
> > >
> > > I went ahead and created the repository [1]. I also configured a
> TeamCity
> > > project [2] that runs all available JUnit tests on every PR
> creation or
> > > update. It also sends the status update to GitHub so that it's
> reflected
> > in
> > > the PR itself so that we can do merges directly from GitHub. Basic
> steps
> > to
> > > make a change are described on the Wiki page [3].
> > >
> > > Let me know if you have any questions.
> > >
> > > [1]
> https://urldefense.com/v3/__https://github.com/apache/ignite-3__;!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKIq24lxF$
> > > [2]
> https://urldefense.com/v3/__https://ci.ignite.apache.org/project/ignite3__;!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKFGL_oJx$
> > > [3]
> > >
> > >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/IGNITE/Apache*Ignite*3.0*ApacheIgnite3.0-DevelopmentProcess__;Kysj!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKNhWzQ0s$
> > >
> > > -Val
> > >
> > > On Wed, Nov 18, 2020 at 4:24 PM Valentin Kulichenko <
> > > valentin.kulichenko@gmail.com> wrote:
> > >
> > > > Thanks, guys. It looks like we are much closer to the consensus
> now. I
> > > > totally on board with the plan, but I would also like to address
> the
> > > > short-term needs. As I've already mentioned earlier, there are
> several
> > > > active IEPs, but we still don't have even a preliminary technical
> > process
> > > > for working on these IEPs. I believe this might be frustrating
> for the
> > > > folks who would like to commit code.
> > > >
> > > > The scope we agreed on is quite big, and it will surely take
> > significant
> > > > time to implement all the changes and stabilize them. Therefore,
> it's
> > > clear
> > > > to me that we will have to maintain 2.x and 3.x in parallel for
> quite
> > > some
> > > > time - this needs to be addressed somehow. I'm convinced that
> having a
> > > > separate repo is the ONLY way to do that, and so far, I haven't
> heard
> > any
> > > > clear alternatives or reasons why we shouldn't do this.
> > > >
> > > > That said, I'm inclined to proceed with this in the next few
> days - I
> > > will
> > > > create a repo and describe the process (which we, of course, can
> > discuss
> > > > and modify going forward). Let's, at the very least, try and see
> where
> > it
> > > > leads us.
> > > >
> > > > If someone has any concrete alternative options on how to we can
> > maintain
> > > > two major versions in parallel, let's have another voice
> discussion
> > this
> > > > Friday. If we do the meeting, we should set it up with a clear
> goal to
> > > make
> > > > a decision. Please let me know if there is interest in this.
> > > >
> > > > -Val
> > > >
> > > > On Mon, Nov 16, 2020 at 6:31 AM Alexey Goncharuk <
> > > > alexey.goncharuk@gmail.com> wrote:
> > > >
> > > >> Good,
> > > >>
> > > >> I think we have an intermediate agreement on the scope and
> > significance
> > > of
> > > >> the changes we want to make. I suggest creating separate
> discussion
> > > >> streams
> > > >> and calls for each of the suggested topics so that:
> > > >>
> > > >> - It is clear for the community what is the motivation of the
> > stream
> > > >> (this includes both functional targets and technical debt
> issues
> > > >> pointed
> > > >> out by Sergey)
> > > >> - Who is planning to take an active part in each of the
> streams
> > (i.e.
> > > >> the 'design committee', as Sergey suggested)
> > > >> - What are the intermediate and final goals for each of the
> streams
> > > >> - What are the cross-stream interactions and how we
> integrate them
> > > >> - How each of the streams will be integrated with the current
> > > codebase
> > > >> based on the above (here is where we will see whether
> drop-in or
> > > >> incremental approaches make more sense)
> > > >>
> > > >
> > >
> >
> >
> > --
> > Best regards,
> > Andrey V. Mashenkov
> >
>
>
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
I guess Ignite 3.0 will be ready for production use somewhere in 2022,
realistically. By that time, Java 8 will be long enough out of support so
that most companies will actually forbid its use, WRT vulnerabilities et
all.
After all we have managed to upgrade from Java 7 to Java 8 and only got a
minor amount of complaints.
Regards,
--
Ilya Kasnacheev
пн, 14 дек. 2020 г. в 19:06, Carbone, Adam <Ad...@bottomline.com>:
> So just one bit to consider... Are there features that you need to use in
> these newer versions of java? Or are we just updating to stay current? The
> reason I ask is that there are still lots of people in an enterprise space
> that are beholden to having to support legacy JAVAEE supported applications
> on Websphere, Weblogic, Redhat, etc... and the organizations that use those
> platforms are slow to move... Most of them are still on Java8
>
> So as a platform I think a strong consideration needs to be towards having
> the broadest possible support profile until it becomes an impediment to
> doing things that the platform needs. So far I haven't seen huge things in
> the newer versions of java that are must haves ( a few exceptions are
> things that would be really nice to take advantage of ).
>
> I think that apache commons has taken the right approach by staying on
> java 8 giving the largest possible user base.
>
> Even standardizing on java 11 would have to make us reconsider Ignite as
> the platform we are using, we are not so invested in it as of now, even
> though we have big plans to leverage it. Just because we aren't sure when
> we are going to be able to upgrade from java8. It has support through 2022,
> so I imagine that is when we will be discussing that.
>
> Regards
>
> ~Adam
>
> Adam Carbone | Director of Innovation – Intelligent Platform Team |
> Bottomline Technologies
> Office: 603-501-6446 | Mobile: 603-570-8418
> www.bottomline.com
>
>
>
> On 11/24/20, 7:38 AM, "Alexey Zinoviev" <za...@gmail.com> wrote:
>
> Java 15 is better, VarHandles, ForeignMemory access and so on.
>
> In both cases, I support the Java version 11 and higher for the
> development!
>
> вт, 24 нояб. 2020 г. в 15:21, Andrey Mashenkov <
> andrey.mashenkov@gmail.com>:
>
> > Let's add maven plugins for sanity checks at the early stage.
> > I've created a ticket for this [1].
> >
> > Also, I've found initial pom.xml has a target version Java 8.
> > Do we intend to move to Java 11 (or may be next LTS) and drop Java 8
> in
> > Ignite 3.0?
> >
> > [1]
> https://urldefense.com/v3/__https://issues.apache.org/jira/browse/IGNITE-13751__;!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKAO2Ejs8$
> >
> > On Tue, Nov 24, 2020 at 5:40 AM Valentin Kulichenko <
> > valentin.kulichenko@gmail.com> wrote:
> >
> > > Folks,
> > >
> > > I went ahead and created the repository [1]. I also configured a
> TeamCity
> > > project [2] that runs all available JUnit tests on every PR
> creation or
> > > update. It also sends the status update to GitHub so that it's
> reflected
> > in
> > > the PR itself so that we can do merges directly from GitHub. Basic
> steps
> > to
> > > make a change are described on the Wiki page [3].
> > >
> > > Let me know if you have any questions.
> > >
> > > [1]
> https://urldefense.com/v3/__https://github.com/apache/ignite-3__;!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKIq24lxF$
> > > [2]
> https://urldefense.com/v3/__https://ci.ignite.apache.org/project/ignite3__;!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKFGL_oJx$
> > > [3]
> > >
> > >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/IGNITE/Apache*Ignite*3.0*ApacheIgnite3.0-DevelopmentProcess__;Kysj!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKNhWzQ0s$
> > >
> > > -Val
> > >
> > > On Wed, Nov 18, 2020 at 4:24 PM Valentin Kulichenko <
> > > valentin.kulichenko@gmail.com> wrote:
> > >
> > > > Thanks, guys. It looks like we are much closer to the consensus
> now. I
> > > > totally on board with the plan, but I would also like to address
> the
> > > > short-term needs. As I've already mentioned earlier, there are
> several
> > > > active IEPs, but we still don't have even a preliminary technical
> > process
> > > > for working on these IEPs. I believe this might be frustrating
> for the
> > > > folks who would like to commit code.
> > > >
> > > > The scope we agreed on is quite big, and it will surely take
> > significant
> > > > time to implement all the changes and stabilize them. Therefore,
> it's
> > > clear
> > > > to me that we will have to maintain 2.x and 3.x in parallel for
> quite
> > > some
> > > > time - this needs to be addressed somehow. I'm convinced that
> having a
> > > > separate repo is the ONLY way to do that, and so far, I haven't
> heard
> > any
> > > > clear alternatives or reasons why we shouldn't do this.
> > > >
> > > > That said, I'm inclined to proceed with this in the next few
> days - I
> > > will
> > > > create a repo and describe the process (which we, of course, can
> > discuss
> > > > and modify going forward). Let's, at the very least, try and see
> where
> > it
> > > > leads us.
> > > >
> > > > If someone has any concrete alternative options on how to we can
> > maintain
> > > > two major versions in parallel, let's have another voice
> discussion
> > this
> > > > Friday. If we do the meeting, we should set it up with a clear
> goal to
> > > make
> > > > a decision. Please let me know if there is interest in this.
> > > >
> > > > -Val
> > > >
> > > > On Mon, Nov 16, 2020 at 6:31 AM Alexey Goncharuk <
> > > > alexey.goncharuk@gmail.com> wrote:
> > > >
> > > >> Good,
> > > >>
> > > >> I think we have an intermediate agreement on the scope and
> > significance
> > > of
> > > >> the changes we want to make. I suggest creating separate
> discussion
> > > >> streams
> > > >> and calls for each of the suggested topics so that:
> > > >>
> > > >> - It is clear for the community what is the motivation of the
> > stream
> > > >> (this includes both functional targets and technical debt
> issues
> > > >> pointed
> > > >> out by Sergey)
> > > >> - Who is planning to take an active part in each of the
> streams
> > (i.e.
> > > >> the 'design committee', as Sergey suggested)
> > > >> - What are the intermediate and final goals for each of the
> streams
> > > >> - What are the cross-stream interactions and how we
> integrate them
> > > >> - How each of the streams will be integrated with the current
> > > codebase
> > > >> based on the above (here is where we will see whether
> drop-in or
> > > >> incremental approaches make more sense)
> > > >>
> > > >
> > >
> >
> >
> > --
> > Best regards,
> > Andrey V. Mashenkov
> >
>
>
Re: [DISCUSS] Ignite 3.0 development approach
Posted by "Carbone, Adam" <Ad...@bottomline.com>.
So just one bit to consider... Are there features that you need to use in these newer versions of java? Or are we just updating to stay current? The reason I ask is that there are still lots of people in an enterprise space that are beholden to having to support legacy JAVAEE supported applications on Websphere, Weblogic, Redhat, etc... and the organizations that use those platforms are slow to move... Most of them are still on Java8
So as a platform I think a strong consideration needs to be towards having the broadest possible support profile until it becomes an impediment to doing things that the platform needs. So far I haven't seen huge things in the newer versions of java that are must haves ( a few exceptions are things that would be really nice to take advantage of ).
I think that apache commons has taken the right approach by staying on java 8 giving the largest possible user base.
Even standardizing on java 11 would have to make us reconsider Ignite as the platform we are using, we are not so invested in it as of now, even though we have big plans to leverage it. Just because we aren't sure when we are going to be able to upgrade from java8. It has support through 2022, so I imagine that is when we will be discussing that.
Regards
~Adam
Adam Carbone | Director of Innovation – Intelligent Platform Team | Bottomline Technologies
Office: 603-501-6446 | Mobile: 603-570-8418
www.bottomline.com
On 11/24/20, 7:38 AM, "Alexey Zinoviev" <za...@gmail.com> wrote:
Java 15 is better, VarHandles, ForeignMemory access and so on.
In both cases, I support the Java version 11 and higher for the development!
вт, 24 нояб. 2020 г. в 15:21, Andrey Mashenkov <an...@gmail.com>:
> Let's add maven plugins for sanity checks at the early stage.
> I've created a ticket for this [1].
>
> Also, I've found initial pom.xml has a target version Java 8.
> Do we intend to move to Java 11 (or may be next LTS) and drop Java 8 in
> Ignite 3.0?
>
> [1] https://urldefense.com/v3/__https://issues.apache.org/jira/browse/IGNITE-13751__;!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKAO2Ejs8$
>
> On Tue, Nov 24, 2020 at 5:40 AM Valentin Kulichenko <
> valentin.kulichenko@gmail.com> wrote:
>
> > Folks,
> >
> > I went ahead and created the repository [1]. I also configured a TeamCity
> > project [2] that runs all available JUnit tests on every PR creation or
> > update. It also sends the status update to GitHub so that it's reflected
> in
> > the PR itself so that we can do merges directly from GitHub. Basic steps
> to
> > make a change are described on the Wiki page [3].
> >
> > Let me know if you have any questions.
> >
> > [1] https://urldefense.com/v3/__https://github.com/apache/ignite-3__;!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKIq24lxF$
> > [2] https://urldefense.com/v3/__https://ci.ignite.apache.org/project/ignite3__;!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKFGL_oJx$
> > [3]
> >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/IGNITE/Apache*Ignite*3.0*ApacheIgnite3.0-DevelopmentProcess__;Kysj!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKNhWzQ0s$
> >
> > -Val
> >
> > On Wed, Nov 18, 2020 at 4:24 PM Valentin Kulichenko <
> > valentin.kulichenko@gmail.com> wrote:
> >
> > > Thanks, guys. It looks like we are much closer to the consensus now. I
> > > totally on board with the plan, but I would also like to address the
> > > short-term needs. As I've already mentioned earlier, there are several
> > > active IEPs, but we still don't have even a preliminary technical
> process
> > > for working on these IEPs. I believe this might be frustrating for the
> > > folks who would like to commit code.
> > >
> > > The scope we agreed on is quite big, and it will surely take
> significant
> > > time to implement all the changes and stabilize them. Therefore, it's
> > clear
> > > to me that we will have to maintain 2.x and 3.x in parallel for quite
> > some
> > > time - this needs to be addressed somehow. I'm convinced that having a
> > > separate repo is the ONLY way to do that, and so far, I haven't heard
> any
> > > clear alternatives or reasons why we shouldn't do this.
> > >
> > > That said, I'm inclined to proceed with this in the next few days - I
> > will
> > > create a repo and describe the process (which we, of course, can
> discuss
> > > and modify going forward). Let's, at the very least, try and see where
> it
> > > leads us.
> > >
> > > If someone has any concrete alternative options on how to we can
> maintain
> > > two major versions in parallel, let's have another voice discussion
> this
> > > Friday. If we do the meeting, we should set it up with a clear goal to
> > make
> > > a decision. Please let me know if there is interest in this.
> > >
> > > -Val
> > >
> > > On Mon, Nov 16, 2020 at 6:31 AM Alexey Goncharuk <
> > > alexey.goncharuk@gmail.com> wrote:
> > >
> > >> Good,
> > >>
> > >> I think we have an intermediate agreement on the scope and
> significance
> > of
> > >> the changes we want to make. I suggest creating separate discussion
> > >> streams
> > >> and calls for each of the suggested topics so that:
> > >>
> > >> - It is clear for the community what is the motivation of the
> stream
> > >> (this includes both functional targets and technical debt issues
> > >> pointed
> > >> out by Sergey)
> > >> - Who is planning to take an active part in each of the streams
> (i.e.
> > >> the 'design committee', as Sergey suggested)
> > >> - What are the intermediate and final goals for each of the streams
> > >> - What are the cross-stream interactions and how we integrate them
> > >> - How each of the streams will be integrated with the current
> > codebase
> > >> based on the above (here is where we will see whether drop-in or
> > >> incremental approaches make more sense)
> > >>
> > >
> >
>
>
> --
> Best regards,
> Andrey V. Mashenkov
>
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Petr Ivanov <mr...@gmail.com>.
Hi, Val.
Thanks for comments.
Let me explain some ambiguous points.
> [Val] What do we use Ant tasks for? I'm sure we do use them a lot for
> release packaging, but it will apparently be significantly simplified. Are
> there any other cases?
We have ant tasks for C++ / .NET version changing (including calling Javascript via ant), assembling, javadoc modification and so on.
There are at least 11 modules with maven-antrun-plugin, plus core and parent pom.
I suggest some revision is required and a) removing of obsolete ones and b) trying to replace ant call with more native maven plugin (or at least redesign approach to be more clear what it does and why).
> [Val] Could you elaborate on this? What should be parallelized in your
> view, and how exactly it will speed up the process?
I was talking about stable parallel build (current it is not stable, producing different results from build to build and having race condition issues) and parallel tests (we have no unit tests for now, but when we will have, forkCount=1 should be the default behaviour).
For both — correct set of plugins and their version should be introduced and tests.
That is not in high priority, more "nice to have" feature.
> [Val] I'm not sure we need this. Gradle does seem to be nicer than Maven in
> many aspects, but this will be a big transition for the community. The
> value of such a transition is not clear. We also seem to have much more
> experience in Maven than in Gradle.
Agree. I was thinking of Gradle only as a possible alternative if we will face impossible to solve with maven issues.
> On 28 Nov 2020, at 07:31, Valentin Kulichenko <va...@gmail.com> wrote:
>
> Petr,
>
> You have some great points! My comments are below.
>
> -Val
>
> On Fri, Nov 27, 2020 at 4:28 AM Petr Ivanov <mr...@gmail.com> wrote:
>
>> More or less, unless we specifically forbid that, I guess
>>
>> However there is bigger concern: JDK 15 is STS, so after half of a year it
>> will be out of support and no major production team will use that JDK in
>> their environment.
>> I would stick to JDK 11 as it is LTS at least until JDK17, plus — a lot of
>> efforts should be made to enforce Apache Ignite be built on JDK 11 alone,
>> not to mention 15th version...
>>
>
> [Val] I think we will have to stick with Java 11, simply because it's the
> current LTS. If we go with 15, almost no one will be able to use Ignite in
> production :) We can switch to 17 in the future in case there is any value.
>
>
>> Also, If we are going to introduce such major changes, I'd like to purpose
>> maven project refactoring as well:
>> 1. Full revision of all ant-calling tasks with javascript functions and
>> alike — the complexity of those are overwhelming, something new should be
>> at least researched.
>>
>
> [Val] What do we use Ant tasks for? I'm sure we do use them a lot for
> release packaging, but it will apparently be significantly simplified. Are
> there any other cases?
>
>
>> 2. Full revisions of profiles (for both root and modules) as there are
>> some obsolete ones, and some that do ambiguous or, even worse, totally
>> unknown tasks.
>>
>
> [Val] Agree - the number of profiles should be at least minimized. In the
> best case, we should not have any profiles at all. They are non-intuitive
> for developers, and also often confuse IDEs.
>
>
>> 3. Introduce plugin and dependency management sections to control over and
>> concrete versions of software we are relying in our project. Additionally —
>> add BOM with all Ignite modules and their dependencies, which should help
>> our users to better embed Ignite to their projects.
>> 4. Up all versions of plugins and dependencies where possible to there
>> current production versions (for plugins — it should be a must if we are
>> ever going to build project under latest JDK versions).
>>
>
> [Val] Agree with both.
>
>
>> 5. Prepare project for parallel building, testing and assembling where
>> possible for faster development process, with possible additional
>> enhancement like incremental rebuild.
>>
>
> [Val] Could you elaborate on this? What should be parallelized in your
> view, and how exactly it will speed up the process?
>
>
>> 6. Possibly — research alternate builders, like Gradle (thought there are
>> a lot of questions to its race condition issues during multimodule builds).
>>
>
> [Val] I'm not sure we need this. Gradle does seem to be nicer than Maven in
> many aspects, but this will be a big transition for the community. The
> value of such a transition is not clear. We also seem to have much more
> experience in Maven than in Gradle.
>
>
>> And last, but not least — think of migrating to 'CI/CD as a Code' approach
>> for our main instrument TeamCity.
>> Whole project (both test and release build configurations) can be
>> described using DSL (Kotlin in case of TC) and stored in VCS, forcing
>> infrastructure changes to go through the same development processes
>> including discussions, review, change history and etc.
>>
>
> [Val] Huge +1.
>
>
>>
>> Only I am not sure for now about where should the code be stored — in
>> separate repository (secure, but disables testing of code with TC settings
>> both in single PR), or alongside project's code (can be possible security
>> hole).
>> That would require additional dev thread I think.
>>
>>
>>
>> WDYT?
>>
>>> On 24 Nov 2020, at 20:04, Pavel Tupitsyn <pt...@apache.org> wrote:
>>>
>>> If we use Java15 for development, can the resulting package be used from
>> a
>>> Java11 app (the latest LTS)?
>>>
>>> On Tue, Nov 24, 2020 at 7:51 PM Andrey Mashenkov <
>> andrey.mashenkov@gmail.com>
>>> wrote:
>>>
>>>> Jave15 looks awesome.
>>>>
>>>> * Hidden classes [1] can be used by codegenerators.
>>>> * Records [2] can replace boilerplate code like IgniteBiTuple,
>> GridTupleX.
>>>>
>>>> [1] https://openjdk.java.net/jeps/371
>>>> [2] https://openjdk.java.net/jeps/384
>>>>
>>>> On Tue, Nov 24, 2020 at 3:38 PM Alexey Zinoviev <zaleslaw.sin@gmail.com
>>>
>>>> wrote:
>>>>
>>>>> Java 15 is better, VarHandles, ForeignMemory access and so on.
>>>>>
>>>>> In both cases, I support the Java version 11 and higher for the
>>>>> development!
>>>>>
>>>>> вт, 24 нояб. 2020 г. в 15:21, Andrey Mashenkov <
>>>> andrey.mashenkov@gmail.com
>>>>>> :
>>>>>
>>>>>> Let's add maven plugins for sanity checks at the early stage.
>>>>>> I've created a ticket for this [1].
>>>>>>
>>>>>> Also, I've found initial pom.xml has a target version Java 8.
>>>>>> Do we intend to move to Java 11 (or may be next LTS) and drop Java 8
>> in
>>>>>> Ignite 3.0?
>>>>>>
>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13751
>>>>>>
>>>>>> On Tue, Nov 24, 2020 at 5:40 AM Valentin Kulichenko <
>>>>>> valentin.kulichenko@gmail.com> wrote:
>>>>>>
>>>>>>> Folks,
>>>>>>>
>>>>>>> I went ahead and created the repository [1]. I also configured a
>>>>> TeamCity
>>>>>>> project [2] that runs all available JUnit tests on every PR creation
>>>> or
>>>>>>> update. It also sends the status update to GitHub so that it's
>>>>> reflected
>>>>>> in
>>>>>>> the PR itself so that we can do merges directly from GitHub. Basic
>>>>> steps
>>>>>> to
>>>>>>> make a change are described on the Wiki page [3].
>>>>>>>
>>>>>>> Let me know if you have any questions.
>>>>>>>
>>>>>>> [1] https://github.com/apache/ignite-3
>>>>>>> [2] https://ci.ignite.apache.org/project/ignite3
>>>>>>> [3]
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0#ApacheIgnite3.0-DevelopmentProcess
>>>>>>>
>>>>>>> -Val
>>>>>>>
>>>>>>> On Wed, Nov 18, 2020 at 4:24 PM Valentin Kulichenko <
>>>>>>> valentin.kulichenko@gmail.com> wrote:
>>>>>>>
>>>>>>>> Thanks, guys. It looks like we are much closer to the consensus
>>>> now.
>>>>> I
>>>>>>>> totally on board with the plan, but I would also like to address
>>>> the
>>>>>>>> short-term needs. As I've already mentioned earlier, there are
>>>>> several
>>>>>>>> active IEPs, but we still don't have even a preliminary technical
>>>>>> process
>>>>>>>> for working on these IEPs. I believe this might be frustrating for
>>>>> the
>>>>>>>> folks who would like to commit code.
>>>>>>>>
>>>>>>>> The scope we agreed on is quite big, and it will surely take
>>>>>> significant
>>>>>>>> time to implement all the changes and stabilize them. Therefore,
>>>> it's
>>>>>>> clear
>>>>>>>> to me that we will have to maintain 2.x and 3.x in parallel for
>>>> quite
>>>>>>> some
>>>>>>>> time - this needs to be addressed somehow. I'm convinced that
>>>> having
>>>>> a
>>>>>>>> separate repo is the ONLY way to do that, and so far, I haven't
>>>> heard
>>>>>> any
>>>>>>>> clear alternatives or reasons why we shouldn't do this.
>>>>>>>>
>>>>>>>> That said, I'm inclined to proceed with this in the next few days
>>>> - I
>>>>>>> will
>>>>>>>> create a repo and describe the process (which we, of course, can
>>>>>> discuss
>>>>>>>> and modify going forward). Let's, at the very least, try and see
>>>>> where
>>>>>> it
>>>>>>>> leads us.
>>>>>>>>
>>>>>>>> If someone has any concrete alternative options on how to we can
>>>>>> maintain
>>>>>>>> two major versions in parallel, let's have another voice discussion
>>>>>> this
>>>>>>>> Friday. If we do the meeting, we should set it up with a clear goal
>>>>> to
>>>>>>> make
>>>>>>>> a decision. Please let me know if there is interest in this.
>>>>>>>>
>>>>>>>> -Val
>>>>>>>>
>>>>>>>> On Mon, Nov 16, 2020 at 6:31 AM Alexey Goncharuk <
>>>>>>>> alexey.goncharuk@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Good,
>>>>>>>>>
>>>>>>>>> I think we have an intermediate agreement on the scope and
>>>>>> significance
>>>>>>> of
>>>>>>>>> the changes we want to make. I suggest creating separate
>>>> discussion
>>>>>>>>> streams
>>>>>>>>> and calls for each of the suggested topics so that:
>>>>>>>>>
>>>>>>>>> - It is clear for the community what is the motivation of the
>>>>>> stream
>>>>>>>>> (this includes both functional targets and technical debt
>>>> issues
>>>>>>>>> pointed
>>>>>>>>> out by Sergey)
>>>>>>>>> - Who is planning to take an active part in each of the streams
>>>>>> (i.e.
>>>>>>>>> the 'design committee', as Sergey suggested)
>>>>>>>>> - What are the intermediate and final goals for each of the
>>>>> streams
>>>>>>>>> - What are the cross-stream interactions and how we integrate
>>>>> them
>>>>>>>>> - How each of the streams will be integrated with the current
>>>>>>> codebase
>>>>>>>>> based on the above (here is where we will see whether drop-in
>>>> or
>>>>>>>>> incremental approaches make more sense)
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best regards,
>>>>>> Andrey V. Mashenkov
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Best regards,
>>>> Andrey V. Mashenkov
>>>>
>>
>>
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Valentin Kulichenko <va...@gmail.com>.
Petr,
You have some great points! My comments are below.
-Val
On Fri, Nov 27, 2020 at 4:28 AM Petr Ivanov <mr...@gmail.com> wrote:
> More or less, unless we specifically forbid that, I guess
>
> However there is bigger concern: JDK 15 is STS, so after half of a year it
> will be out of support and no major production team will use that JDK in
> their environment.
> I would stick to JDK 11 as it is LTS at least until JDK17, plus — a lot of
> efforts should be made to enforce Apache Ignite be built on JDK 11 alone,
> not to mention 15th version...
>
[Val] I think we will have to stick with Java 11, simply because it's the
current LTS. If we go with 15, almost no one will be able to use Ignite in
production :) We can switch to 17 in the future in case there is any value.
> Also, If we are going to introduce such major changes, I'd like to purpose
> maven project refactoring as well:
> 1. Full revision of all ant-calling tasks with javascript functions and
> alike — the complexity of those are overwhelming, something new should be
> at least researched.
>
[Val] What do we use Ant tasks for? I'm sure we do use them a lot for
release packaging, but it will apparently be significantly simplified. Are
there any other cases?
> 2. Full revisions of profiles (for both root and modules) as there are
> some obsolete ones, and some that do ambiguous or, even worse, totally
> unknown tasks.
>
[Val] Agree - the number of profiles should be at least minimized. In the
best case, we should not have any profiles at all. They are non-intuitive
for developers, and also often confuse IDEs.
> 3. Introduce plugin and dependency management sections to control over and
> concrete versions of software we are relying in our project. Additionally —
> add BOM with all Ignite modules and their dependencies, which should help
> our users to better embed Ignite to their projects.
> 4. Up all versions of plugins and dependencies where possible to there
> current production versions (for plugins — it should be a must if we are
> ever going to build project under latest JDK versions).
>
[Val] Agree with both.
> 5. Prepare project for parallel building, testing and assembling where
> possible for faster development process, with possible additional
> enhancement like incremental rebuild.
>
[Val] Could you elaborate on this? What should be parallelized in your
view, and how exactly it will speed up the process?
> 6. Possibly — research alternate builders, like Gradle (thought there are
> a lot of questions to its race condition issues during multimodule builds).
>
[Val] I'm not sure we need this. Gradle does seem to be nicer than Maven in
many aspects, but this will be a big transition for the community. The
value of such a transition is not clear. We also seem to have much more
experience in Maven than in Gradle.
> And last, but not least — think of migrating to 'CI/CD as a Code' approach
> for our main instrument TeamCity.
> Whole project (both test and release build configurations) can be
> described using DSL (Kotlin in case of TC) and stored in VCS, forcing
> infrastructure changes to go through the same development processes
> including discussions, review, change history and etc.
>
[Val] Huge +1.
>
> Only I am not sure for now about where should the code be stored — in
> separate repository (secure, but disables testing of code with TC settings
> both in single PR), or alongside project's code (can be possible security
> hole).
> That would require additional dev thread I think.
>
>
>
> WDYT?
>
> > On 24 Nov 2020, at 20:04, Pavel Tupitsyn <pt...@apache.org> wrote:
> >
> > If we use Java15 for development, can the resulting package be used from
> a
> > Java11 app (the latest LTS)?
> >
> > On Tue, Nov 24, 2020 at 7:51 PM Andrey Mashenkov <
> andrey.mashenkov@gmail.com>
> > wrote:
> >
> >> Jave15 looks awesome.
> >>
> >> * Hidden classes [1] can be used by codegenerators.
> >> * Records [2] can replace boilerplate code like IgniteBiTuple,
> GridTupleX.
> >>
> >> [1] https://openjdk.java.net/jeps/371
> >> [2] https://openjdk.java.net/jeps/384
> >>
> >> On Tue, Nov 24, 2020 at 3:38 PM Alexey Zinoviev <zaleslaw.sin@gmail.com
> >
> >> wrote:
> >>
> >>> Java 15 is better, VarHandles, ForeignMemory access and so on.
> >>>
> >>> In both cases, I support the Java version 11 and higher for the
> >>> development!
> >>>
> >>> вт, 24 нояб. 2020 г. в 15:21, Andrey Mashenkov <
> >> andrey.mashenkov@gmail.com
> >>>> :
> >>>
> >>>> Let's add maven plugins for sanity checks at the early stage.
> >>>> I've created a ticket for this [1].
> >>>>
> >>>> Also, I've found initial pom.xml has a target version Java 8.
> >>>> Do we intend to move to Java 11 (or may be next LTS) and drop Java 8
> in
> >>>> Ignite 3.0?
> >>>>
> >>>> [1] https://issues.apache.org/jira/browse/IGNITE-13751
> >>>>
> >>>> On Tue, Nov 24, 2020 at 5:40 AM Valentin Kulichenko <
> >>>> valentin.kulichenko@gmail.com> wrote:
> >>>>
> >>>>> Folks,
> >>>>>
> >>>>> I went ahead and created the repository [1]. I also configured a
> >>> TeamCity
> >>>>> project [2] that runs all available JUnit tests on every PR creation
> >> or
> >>>>> update. It also sends the status update to GitHub so that it's
> >>> reflected
> >>>> in
> >>>>> the PR itself so that we can do merges directly from GitHub. Basic
> >>> steps
> >>>> to
> >>>>> make a change are described on the Wiki page [3].
> >>>>>
> >>>>> Let me know if you have any questions.
> >>>>>
> >>>>> [1] https://github.com/apache/ignite-3
> >>>>> [2] https://ci.ignite.apache.org/project/ignite3
> >>>>> [3]
> >>>>>
> >>>>>
> >>>>
> >>>
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0#ApacheIgnite3.0-DevelopmentProcess
> >>>>>
> >>>>> -Val
> >>>>>
> >>>>> On Wed, Nov 18, 2020 at 4:24 PM Valentin Kulichenko <
> >>>>> valentin.kulichenko@gmail.com> wrote:
> >>>>>
> >>>>>> Thanks, guys. It looks like we are much closer to the consensus
> >> now.
> >>> I
> >>>>>> totally on board with the plan, but I would also like to address
> >> the
> >>>>>> short-term needs. As I've already mentioned earlier, there are
> >>> several
> >>>>>> active IEPs, but we still don't have even a preliminary technical
> >>>> process
> >>>>>> for working on these IEPs. I believe this might be frustrating for
> >>> the
> >>>>>> folks who would like to commit code.
> >>>>>>
> >>>>>> The scope we agreed on is quite big, and it will surely take
> >>>> significant
> >>>>>> time to implement all the changes and stabilize them. Therefore,
> >> it's
> >>>>> clear
> >>>>>> to me that we will have to maintain 2.x and 3.x in parallel for
> >> quite
> >>>>> some
> >>>>>> time - this needs to be addressed somehow. I'm convinced that
> >> having
> >>> a
> >>>>>> separate repo is the ONLY way to do that, and so far, I haven't
> >> heard
> >>>> any
> >>>>>> clear alternatives or reasons why we shouldn't do this.
> >>>>>>
> >>>>>> That said, I'm inclined to proceed with this in the next few days
> >> - I
> >>>>> will
> >>>>>> create a repo and describe the process (which we, of course, can
> >>>> discuss
> >>>>>> and modify going forward). Let's, at the very least, try and see
> >>> where
> >>>> it
> >>>>>> leads us.
> >>>>>>
> >>>>>> If someone has any concrete alternative options on how to we can
> >>>> maintain
> >>>>>> two major versions in parallel, let's have another voice discussion
> >>>> this
> >>>>>> Friday. If we do the meeting, we should set it up with a clear goal
> >>> to
> >>>>> make
> >>>>>> a decision. Please let me know if there is interest in this.
> >>>>>>
> >>>>>> -Val
> >>>>>>
> >>>>>> On Mon, Nov 16, 2020 at 6:31 AM Alexey Goncharuk <
> >>>>>> alexey.goncharuk@gmail.com> wrote:
> >>>>>>
> >>>>>>> Good,
> >>>>>>>
> >>>>>>> I think we have an intermediate agreement on the scope and
> >>>> significance
> >>>>> of
> >>>>>>> the changes we want to make. I suggest creating separate
> >> discussion
> >>>>>>> streams
> >>>>>>> and calls for each of the suggested topics so that:
> >>>>>>>
> >>>>>>> - It is clear for the community what is the motivation of the
> >>>> stream
> >>>>>>> (this includes both functional targets and technical debt
> >> issues
> >>>>>>> pointed
> >>>>>>> out by Sergey)
> >>>>>>> - Who is planning to take an active part in each of the streams
> >>>> (i.e.
> >>>>>>> the 'design committee', as Sergey suggested)
> >>>>>>> - What are the intermediate and final goals for each of the
> >>> streams
> >>>>>>> - What are the cross-stream interactions and how we integrate
> >>> them
> >>>>>>> - How each of the streams will be integrated with the current
> >>>>> codebase
> >>>>>>> based on the above (here is where we will see whether drop-in
> >> or
> >>>>>>> incremental approaches make more sense)
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> Best regards,
> >>>> Andrey V. Mashenkov
> >>>>
> >>>
> >>
> >>
> >> --
> >> Best regards,
> >> Andrey V. Mashenkov
> >>
>
>
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
One can still submit a PR which mines bitcoin and run tests on it. CI/CD
configuration alongside code does not solve the issue that any user may run
code on TC.
Regards,
--
Ilya Kasnacheev
пт, 27 нояб. 2020 г. в 16:48, Petr Ivanov <mr...@gmail.com>:
> > Storing CI/CD code (yaml definitions for Travis/Azure/GH Actions, Jenkins
> > pipelines, etc) in the same repo is very common.
> > Secrets are stored separately (e.g. GitHub secrets), TeamCity probably
> has
> > a similar feature.
>
>
> My main security concern — anyone, including third-party person without
> even committer permissions can modify build configuration in such a way
> that it will be doing something not intend to do (bitcoin mining for
> instance) or even something harmful (like trying to attack underlying TC
> infrastructure). If we are to store CI/CD configuration alongside code,
> some restrictions are required.
>
>
>
> > On 27 Nov 2020, at 15:59, Pavel Tupitsyn <pt...@apache.org> wrote:
> >
> >> migrating to 'CI/CD as a Code'
> >
> > Huge +1 for this.
> >
> >
> >> where should the code be stored ..
> >> alongside project's code (can be possible security hole)
> >
> > Storing CI/CD code (yaml definitions for Travis/Azure/GH Actions, Jenkins
> > pipelines, etc) in the same repo is very common.
> > Secrets are stored separately (e.g. GitHub secrets), TeamCity probably
> has
> > a similar feature.
> >
> >
> > On Fri, Nov 27, 2020 at 3:28 PM Petr Ivanov <mr...@gmail.com> wrote:
> >
> >> More or less, unless we specifically forbid that, I guess
> >>
> >> However there is bigger concern: JDK 15 is STS, so after half of a year
> it
> >> will be out of support and no major production team will use that JDK in
> >> their environment.
> >> I would stick to JDK 11 as it is LTS at least until JDK17, plus — a lot
> of
> >> efforts should be made to enforce Apache Ignite be built on JDK 11
> alone,
> >> not to mention 15th version...
> >>
> >>
> >>
> >> Also, If we are going to introduce such major changes, I'd like to
> purpose
> >> maven project refactoring as well:
> >> 1. Full revision of all ant-calling tasks with javascript functions and
> >> alike — the complexity of those are overwhelming, something new should
> be
> >> at least researched.
> >> 2. Full revisions of profiles (for both root and modules) as there are
> >> some obsolete ones, and some that do ambiguous or, even worse, totally
> >> unknown tasks.
> >> 3. Introduce plugin and dependency management sections to control over
> and
> >> concrete versions of software we are relying in our project.
> Additionally —
> >> add BOM with all Ignite modules and their dependencies, which should
> help
> >> our users to better embed Ignite to their projects.
> >> 4. Up all versions of plugins and dependencies where possible to there
> >> current production versions (for plugins — it should be a must if we are
> >> ever going to build project under latest JDK versions).
> >> 5. Prepare project for parallel building, testing and assembling where
> >> possible for faster development process, with possible additional
> >> enhancement like incremental rebuild.
> >> 6. Possibly — research alternate builders, like Gradle (thought there
> are
> >> a lot of questions to its race condition issues during multimodule
> builds).
> >>
> >>
> >>
> >> And last, but not least — think of migrating to 'CI/CD as a Code'
> approach
> >> for our main instrument TeamCity.
> >> Whole project (both test and release build configurations) can be
> >> described using DSL (Kotlin in case of TC) and stored in VCS, forcing
> >> infrastructure changes to go through the same development processes
> >> including discussions, review, change history and etc.
> >>
> >> Only I am not sure for now about where should the code be stored — in
> >> separate repository (secure, but disables testing of code with TC
> settings
> >> both in single PR), or alongside project's code (can be possible
> security
> >> hole).
> >> That would require additional dev thread I think.
> >>
> >>
> >>
> >> WDYT?
> >>
> >>> On 24 Nov 2020, at 20:04, Pavel Tupitsyn <pt...@apache.org> wrote:
> >>>
> >>> If we use Java15 for development, can the resulting package be used
> from
> >> a
> >>> Java11 app (the latest LTS)?
> >>>
> >>> On Tue, Nov 24, 2020 at 7:51 PM Andrey Mashenkov <
> >> andrey.mashenkov@gmail.com>
> >>> wrote:
> >>>
> >>>> Jave15 looks awesome.
> >>>>
> >>>> * Hidden classes [1] can be used by codegenerators.
> >>>> * Records [2] can replace boilerplate code like IgniteBiTuple,
> >> GridTupleX.
> >>>>
> >>>> [1] https://openjdk.java.net/jeps/371
> >>>> [2] https://openjdk.java.net/jeps/384
> >>>>
> >>>> On Tue, Nov 24, 2020 at 3:38 PM Alexey Zinoviev <
> zaleslaw.sin@gmail.com
> >>>
> >>>> wrote:
> >>>>
> >>>>> Java 15 is better, VarHandles, ForeignMemory access and so on.
> >>>>>
> >>>>> In both cases, I support the Java version 11 and higher for the
> >>>>> development!
> >>>>>
> >>>>> вт, 24 нояб. 2020 г. в 15:21, Andrey Mashenkov <
> >>>> andrey.mashenkov@gmail.com
> >>>>>> :
> >>>>>
> >>>>>> Let's add maven plugins for sanity checks at the early stage.
> >>>>>> I've created a ticket for this [1].
> >>>>>>
> >>>>>> Also, I've found initial pom.xml has a target version Java 8.
> >>>>>> Do we intend to move to Java 11 (or may be next LTS) and drop Java 8
> >> in
> >>>>>> Ignite 3.0?
> >>>>>>
> >>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13751
> >>>>>>
> >>>>>> On Tue, Nov 24, 2020 at 5:40 AM Valentin Kulichenko <
> >>>>>> valentin.kulichenko@gmail.com> wrote:
> >>>>>>
> >>>>>>> Folks,
> >>>>>>>
> >>>>>>> I went ahead and created the repository [1]. I also configured a
> >>>>> TeamCity
> >>>>>>> project [2] that runs all available JUnit tests on every PR
> creation
> >>>> or
> >>>>>>> update. It also sends the status update to GitHub so that it's
> >>>>> reflected
> >>>>>> in
> >>>>>>> the PR itself so that we can do merges directly from GitHub. Basic
> >>>>> steps
> >>>>>> to
> >>>>>>> make a change are described on the Wiki page [3].
> >>>>>>>
> >>>>>>> Let me know if you have any questions.
> >>>>>>>
> >>>>>>> [1] https://github.com/apache/ignite-3
> >>>>>>> [2] https://ci.ignite.apache.org/project/ignite3
> >>>>>>> [3]
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0#ApacheIgnite3.0-DevelopmentProcess
> >>>>>>>
> >>>>>>> -Val
> >>>>>>>
> >>>>>>> On Wed, Nov 18, 2020 at 4:24 PM Valentin Kulichenko <
> >>>>>>> valentin.kulichenko@gmail.com> wrote:
> >>>>>>>
> >>>>>>>> Thanks, guys. It looks like we are much closer to the consensus
> >>>> now.
> >>>>> I
> >>>>>>>> totally on board with the plan, but I would also like to address
> >>>> the
> >>>>>>>> short-term needs. As I've already mentioned earlier, there are
> >>>>> several
> >>>>>>>> active IEPs, but we still don't have even a preliminary technical
> >>>>>> process
> >>>>>>>> for working on these IEPs. I believe this might be frustrating for
> >>>>> the
> >>>>>>>> folks who would like to commit code.
> >>>>>>>>
> >>>>>>>> The scope we agreed on is quite big, and it will surely take
> >>>>>> significant
> >>>>>>>> time to implement all the changes and stabilize them. Therefore,
> >>>> it's
> >>>>>>> clear
> >>>>>>>> to me that we will have to maintain 2.x and 3.x in parallel for
> >>>> quite
> >>>>>>> some
> >>>>>>>> time - this needs to be addressed somehow. I'm convinced that
> >>>> having
> >>>>> a
> >>>>>>>> separate repo is the ONLY way to do that, and so far, I haven't
> >>>> heard
> >>>>>> any
> >>>>>>>> clear alternatives or reasons why we shouldn't do this.
> >>>>>>>>
> >>>>>>>> That said, I'm inclined to proceed with this in the next few days
> >>>> - I
> >>>>>>> will
> >>>>>>>> create a repo and describe the process (which we, of course, can
> >>>>>> discuss
> >>>>>>>> and modify going forward). Let's, at the very least, try and see
> >>>>> where
> >>>>>> it
> >>>>>>>> leads us.
> >>>>>>>>
> >>>>>>>> If someone has any concrete alternative options on how to we can
> >>>>>> maintain
> >>>>>>>> two major versions in parallel, let's have another voice
> discussion
> >>>>>> this
> >>>>>>>> Friday. If we do the meeting, we should set it up with a clear
> goal
> >>>>> to
> >>>>>>> make
> >>>>>>>> a decision. Please let me know if there is interest in this.
> >>>>>>>>
> >>>>>>>> -Val
> >>>>>>>>
> >>>>>>>> On Mon, Nov 16, 2020 at 6:31 AM Alexey Goncharuk <
> >>>>>>>> alexey.goncharuk@gmail.com> wrote:
> >>>>>>>>
> >>>>>>>>> Good,
> >>>>>>>>>
> >>>>>>>>> I think we have an intermediate agreement on the scope and
> >>>>>> significance
> >>>>>>> of
> >>>>>>>>> the changes we want to make. I suggest creating separate
> >>>> discussion
> >>>>>>>>> streams
> >>>>>>>>> and calls for each of the suggested topics so that:
> >>>>>>>>>
> >>>>>>>>> - It is clear for the community what is the motivation of the
> >>>>>> stream
> >>>>>>>>> (this includes both functional targets and technical debt
> >>>> issues
> >>>>>>>>> pointed
> >>>>>>>>> out by Sergey)
> >>>>>>>>> - Who is planning to take an active part in each of the streams
> >>>>>> (i.e.
> >>>>>>>>> the 'design committee', as Sergey suggested)
> >>>>>>>>> - What are the intermediate and final goals for each of the
> >>>>> streams
> >>>>>>>>> - What are the cross-stream interactions and how we integrate
> >>>>> them
> >>>>>>>>> - How each of the streams will be integrated with the current
> >>>>>>> codebase
> >>>>>>>>> based on the above (here is where we will see whether drop-in
> >>>> or
> >>>>>>>>> incremental approaches make more sense)
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Best regards,
> >>>>>> Andrey V. Mashenkov
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> Best regards,
> >>>> Andrey V. Mashenkov
> >>>>
> >>
> >>
>
>
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Petr Ivanov <mr...@gmail.com>.
> Storing CI/CD code (yaml definitions for Travis/Azure/GH Actions, Jenkins
> pipelines, etc) in the same repo is very common.
> Secrets are stored separately (e.g. GitHub secrets), TeamCity probably has
> a similar feature.
My main security concern — anyone, including third-party person without even committer permissions can modify build configuration in such a way that it will be doing something not intend to do (bitcoin mining for instance) or even something harmful (like trying to attack underlying TC infrastructure). If we are to store CI/CD configuration alongside code, some restrictions are required.
> On 27 Nov 2020, at 15:59, Pavel Tupitsyn <pt...@apache.org> wrote:
>
>> migrating to 'CI/CD as a Code'
>
> Huge +1 for this.
>
>
>> where should the code be stored ..
>> alongside project's code (can be possible security hole)
>
> Storing CI/CD code (yaml definitions for Travis/Azure/GH Actions, Jenkins
> pipelines, etc) in the same repo is very common.
> Secrets are stored separately (e.g. GitHub secrets), TeamCity probably has
> a similar feature.
>
>
> On Fri, Nov 27, 2020 at 3:28 PM Petr Ivanov <mr...@gmail.com> wrote:
>
>> More or less, unless we specifically forbid that, I guess
>>
>> However there is bigger concern: JDK 15 is STS, so after half of a year it
>> will be out of support and no major production team will use that JDK in
>> their environment.
>> I would stick to JDK 11 as it is LTS at least until JDK17, plus — a lot of
>> efforts should be made to enforce Apache Ignite be built on JDK 11 alone,
>> not to mention 15th version...
>>
>>
>>
>> Also, If we are going to introduce such major changes, I'd like to purpose
>> maven project refactoring as well:
>> 1. Full revision of all ant-calling tasks with javascript functions and
>> alike — the complexity of those are overwhelming, something new should be
>> at least researched.
>> 2. Full revisions of profiles (for both root and modules) as there are
>> some obsolete ones, and some that do ambiguous or, even worse, totally
>> unknown tasks.
>> 3. Introduce plugin and dependency management sections to control over and
>> concrete versions of software we are relying in our project. Additionally —
>> add BOM with all Ignite modules and their dependencies, which should help
>> our users to better embed Ignite to their projects.
>> 4. Up all versions of plugins and dependencies where possible to there
>> current production versions (for plugins — it should be a must if we are
>> ever going to build project under latest JDK versions).
>> 5. Prepare project for parallel building, testing and assembling where
>> possible for faster development process, with possible additional
>> enhancement like incremental rebuild.
>> 6. Possibly — research alternate builders, like Gradle (thought there are
>> a lot of questions to its race condition issues during multimodule builds).
>>
>>
>>
>> And last, but not least — think of migrating to 'CI/CD as a Code' approach
>> for our main instrument TeamCity.
>> Whole project (both test and release build configurations) can be
>> described using DSL (Kotlin in case of TC) and stored in VCS, forcing
>> infrastructure changes to go through the same development processes
>> including discussions, review, change history and etc.
>>
>> Only I am not sure for now about where should the code be stored — in
>> separate repository (secure, but disables testing of code with TC settings
>> both in single PR), or alongside project's code (can be possible security
>> hole).
>> That would require additional dev thread I think.
>>
>>
>>
>> WDYT?
>>
>>> On 24 Nov 2020, at 20:04, Pavel Tupitsyn <pt...@apache.org> wrote:
>>>
>>> If we use Java15 for development, can the resulting package be used from
>> a
>>> Java11 app (the latest LTS)?
>>>
>>> On Tue, Nov 24, 2020 at 7:51 PM Andrey Mashenkov <
>> andrey.mashenkov@gmail.com>
>>> wrote:
>>>
>>>> Jave15 looks awesome.
>>>>
>>>> * Hidden classes [1] can be used by codegenerators.
>>>> * Records [2] can replace boilerplate code like IgniteBiTuple,
>> GridTupleX.
>>>>
>>>> [1] https://openjdk.java.net/jeps/371
>>>> [2] https://openjdk.java.net/jeps/384
>>>>
>>>> On Tue, Nov 24, 2020 at 3:38 PM Alexey Zinoviev <zaleslaw.sin@gmail.com
>>>
>>>> wrote:
>>>>
>>>>> Java 15 is better, VarHandles, ForeignMemory access and so on.
>>>>>
>>>>> In both cases, I support the Java version 11 and higher for the
>>>>> development!
>>>>>
>>>>> вт, 24 нояб. 2020 г. в 15:21, Andrey Mashenkov <
>>>> andrey.mashenkov@gmail.com
>>>>>> :
>>>>>
>>>>>> Let's add maven plugins for sanity checks at the early stage.
>>>>>> I've created a ticket for this [1].
>>>>>>
>>>>>> Also, I've found initial pom.xml has a target version Java 8.
>>>>>> Do we intend to move to Java 11 (or may be next LTS) and drop Java 8
>> in
>>>>>> Ignite 3.0?
>>>>>>
>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13751
>>>>>>
>>>>>> On Tue, Nov 24, 2020 at 5:40 AM Valentin Kulichenko <
>>>>>> valentin.kulichenko@gmail.com> wrote:
>>>>>>
>>>>>>> Folks,
>>>>>>>
>>>>>>> I went ahead and created the repository [1]. I also configured a
>>>>> TeamCity
>>>>>>> project [2] that runs all available JUnit tests on every PR creation
>>>> or
>>>>>>> update. It also sends the status update to GitHub so that it's
>>>>> reflected
>>>>>> in
>>>>>>> the PR itself so that we can do merges directly from GitHub. Basic
>>>>> steps
>>>>>> to
>>>>>>> make a change are described on the Wiki page [3].
>>>>>>>
>>>>>>> Let me know if you have any questions.
>>>>>>>
>>>>>>> [1] https://github.com/apache/ignite-3
>>>>>>> [2] https://ci.ignite.apache.org/project/ignite3
>>>>>>> [3]
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0#ApacheIgnite3.0-DevelopmentProcess
>>>>>>>
>>>>>>> -Val
>>>>>>>
>>>>>>> On Wed, Nov 18, 2020 at 4:24 PM Valentin Kulichenko <
>>>>>>> valentin.kulichenko@gmail.com> wrote:
>>>>>>>
>>>>>>>> Thanks, guys. It looks like we are much closer to the consensus
>>>> now.
>>>>> I
>>>>>>>> totally on board with the plan, but I would also like to address
>>>> the
>>>>>>>> short-term needs. As I've already mentioned earlier, there are
>>>>> several
>>>>>>>> active IEPs, but we still don't have even a preliminary technical
>>>>>> process
>>>>>>>> for working on these IEPs. I believe this might be frustrating for
>>>>> the
>>>>>>>> folks who would like to commit code.
>>>>>>>>
>>>>>>>> The scope we agreed on is quite big, and it will surely take
>>>>>> significant
>>>>>>>> time to implement all the changes and stabilize them. Therefore,
>>>> it's
>>>>>>> clear
>>>>>>>> to me that we will have to maintain 2.x and 3.x in parallel for
>>>> quite
>>>>>>> some
>>>>>>>> time - this needs to be addressed somehow. I'm convinced that
>>>> having
>>>>> a
>>>>>>>> separate repo is the ONLY way to do that, and so far, I haven't
>>>> heard
>>>>>> any
>>>>>>>> clear alternatives or reasons why we shouldn't do this.
>>>>>>>>
>>>>>>>> That said, I'm inclined to proceed with this in the next few days
>>>> - I
>>>>>>> will
>>>>>>>> create a repo and describe the process (which we, of course, can
>>>>>> discuss
>>>>>>>> and modify going forward). Let's, at the very least, try and see
>>>>> where
>>>>>> it
>>>>>>>> leads us.
>>>>>>>>
>>>>>>>> If someone has any concrete alternative options on how to we can
>>>>>> maintain
>>>>>>>> two major versions in parallel, let's have another voice discussion
>>>>>> this
>>>>>>>> Friday. If we do the meeting, we should set it up with a clear goal
>>>>> to
>>>>>>> make
>>>>>>>> a decision. Please let me know if there is interest in this.
>>>>>>>>
>>>>>>>> -Val
>>>>>>>>
>>>>>>>> On Mon, Nov 16, 2020 at 6:31 AM Alexey Goncharuk <
>>>>>>>> alexey.goncharuk@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Good,
>>>>>>>>>
>>>>>>>>> I think we have an intermediate agreement on the scope and
>>>>>> significance
>>>>>>> of
>>>>>>>>> the changes we want to make. I suggest creating separate
>>>> discussion
>>>>>>>>> streams
>>>>>>>>> and calls for each of the suggested topics so that:
>>>>>>>>>
>>>>>>>>> - It is clear for the community what is the motivation of the
>>>>>> stream
>>>>>>>>> (this includes both functional targets and technical debt
>>>> issues
>>>>>>>>> pointed
>>>>>>>>> out by Sergey)
>>>>>>>>> - Who is planning to take an active part in each of the streams
>>>>>> (i.e.
>>>>>>>>> the 'design committee', as Sergey suggested)
>>>>>>>>> - What are the intermediate and final goals for each of the
>>>>> streams
>>>>>>>>> - What are the cross-stream interactions and how we integrate
>>>>> them
>>>>>>>>> - How each of the streams will be integrated with the current
>>>>>>> codebase
>>>>>>>>> based on the above (here is where we will see whether drop-in
>>>> or
>>>>>>>>> incremental approaches make more sense)
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best regards,
>>>>>> Andrey V. Mashenkov
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Best regards,
>>>> Andrey V. Mashenkov
>>>>
>>
>>
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Pavel Tupitsyn <pt...@apache.org>.
> migrating to 'CI/CD as a Code'
Huge +1 for this.
> where should the code be stored ..
> alongside project's code (can be possible security hole)
Storing CI/CD code (yaml definitions for Travis/Azure/GH Actions, Jenkins
pipelines, etc) in the same repo is very common.
Secrets are stored separately (e.g. GitHub secrets), TeamCity probably has
a similar feature.
On Fri, Nov 27, 2020 at 3:28 PM Petr Ivanov <mr...@gmail.com> wrote:
> More or less, unless we specifically forbid that, I guess
>
> However there is bigger concern: JDK 15 is STS, so after half of a year it
> will be out of support and no major production team will use that JDK in
> their environment.
> I would stick to JDK 11 as it is LTS at least until JDK17, plus — a lot of
> efforts should be made to enforce Apache Ignite be built on JDK 11 alone,
> not to mention 15th version...
>
>
>
> Also, If we are going to introduce such major changes, I'd like to purpose
> maven project refactoring as well:
> 1. Full revision of all ant-calling tasks with javascript functions and
> alike — the complexity of those are overwhelming, something new should be
> at least researched.
> 2. Full revisions of profiles (for both root and modules) as there are
> some obsolete ones, and some that do ambiguous or, even worse, totally
> unknown tasks.
> 3. Introduce plugin and dependency management sections to control over and
> concrete versions of software we are relying in our project. Additionally —
> add BOM with all Ignite modules and their dependencies, which should help
> our users to better embed Ignite to their projects.
> 4. Up all versions of plugins and dependencies where possible to there
> current production versions (for plugins — it should be a must if we are
> ever going to build project under latest JDK versions).
> 5. Prepare project for parallel building, testing and assembling where
> possible for faster development process, with possible additional
> enhancement like incremental rebuild.
> 6. Possibly — research alternate builders, like Gradle (thought there are
> a lot of questions to its race condition issues during multimodule builds).
>
>
>
> And last, but not least — think of migrating to 'CI/CD as a Code' approach
> for our main instrument TeamCity.
> Whole project (both test and release build configurations) can be
> described using DSL (Kotlin in case of TC) and stored in VCS, forcing
> infrastructure changes to go through the same development processes
> including discussions, review, change history and etc.
>
> Only I am not sure for now about where should the code be stored — in
> separate repository (secure, but disables testing of code with TC settings
> both in single PR), or alongside project's code (can be possible security
> hole).
> That would require additional dev thread I think.
>
>
>
> WDYT?
>
> > On 24 Nov 2020, at 20:04, Pavel Tupitsyn <pt...@apache.org> wrote:
> >
> > If we use Java15 for development, can the resulting package be used from
> a
> > Java11 app (the latest LTS)?
> >
> > On Tue, Nov 24, 2020 at 7:51 PM Andrey Mashenkov <
> andrey.mashenkov@gmail.com>
> > wrote:
> >
> >> Jave15 looks awesome.
> >>
> >> * Hidden classes [1] can be used by codegenerators.
> >> * Records [2] can replace boilerplate code like IgniteBiTuple,
> GridTupleX.
> >>
> >> [1] https://openjdk.java.net/jeps/371
> >> [2] https://openjdk.java.net/jeps/384
> >>
> >> On Tue, Nov 24, 2020 at 3:38 PM Alexey Zinoviev <zaleslaw.sin@gmail.com
> >
> >> wrote:
> >>
> >>> Java 15 is better, VarHandles, ForeignMemory access and so on.
> >>>
> >>> In both cases, I support the Java version 11 and higher for the
> >>> development!
> >>>
> >>> вт, 24 нояб. 2020 г. в 15:21, Andrey Mashenkov <
> >> andrey.mashenkov@gmail.com
> >>>> :
> >>>
> >>>> Let's add maven plugins for sanity checks at the early stage.
> >>>> I've created a ticket for this [1].
> >>>>
> >>>> Also, I've found initial pom.xml has a target version Java 8.
> >>>> Do we intend to move to Java 11 (or may be next LTS) and drop Java 8
> in
> >>>> Ignite 3.0?
> >>>>
> >>>> [1] https://issues.apache.org/jira/browse/IGNITE-13751
> >>>>
> >>>> On Tue, Nov 24, 2020 at 5:40 AM Valentin Kulichenko <
> >>>> valentin.kulichenko@gmail.com> wrote:
> >>>>
> >>>>> Folks,
> >>>>>
> >>>>> I went ahead and created the repository [1]. I also configured a
> >>> TeamCity
> >>>>> project [2] that runs all available JUnit tests on every PR creation
> >> or
> >>>>> update. It also sends the status update to GitHub so that it's
> >>> reflected
> >>>> in
> >>>>> the PR itself so that we can do merges directly from GitHub. Basic
> >>> steps
> >>>> to
> >>>>> make a change are described on the Wiki page [3].
> >>>>>
> >>>>> Let me know if you have any questions.
> >>>>>
> >>>>> [1] https://github.com/apache/ignite-3
> >>>>> [2] https://ci.ignite.apache.org/project/ignite3
> >>>>> [3]
> >>>>>
> >>>>>
> >>>>
> >>>
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0#ApacheIgnite3.0-DevelopmentProcess
> >>>>>
> >>>>> -Val
> >>>>>
> >>>>> On Wed, Nov 18, 2020 at 4:24 PM Valentin Kulichenko <
> >>>>> valentin.kulichenko@gmail.com> wrote:
> >>>>>
> >>>>>> Thanks, guys. It looks like we are much closer to the consensus
> >> now.
> >>> I
> >>>>>> totally on board with the plan, but I would also like to address
> >> the
> >>>>>> short-term needs. As I've already mentioned earlier, there are
> >>> several
> >>>>>> active IEPs, but we still don't have even a preliminary technical
> >>>> process
> >>>>>> for working on these IEPs. I believe this might be frustrating for
> >>> the
> >>>>>> folks who would like to commit code.
> >>>>>>
> >>>>>> The scope we agreed on is quite big, and it will surely take
> >>>> significant
> >>>>>> time to implement all the changes and stabilize them. Therefore,
> >> it's
> >>>>> clear
> >>>>>> to me that we will have to maintain 2.x and 3.x in parallel for
> >> quite
> >>>>> some
> >>>>>> time - this needs to be addressed somehow. I'm convinced that
> >> having
> >>> a
> >>>>>> separate repo is the ONLY way to do that, and so far, I haven't
> >> heard
> >>>> any
> >>>>>> clear alternatives or reasons why we shouldn't do this.
> >>>>>>
> >>>>>> That said, I'm inclined to proceed with this in the next few days
> >> - I
> >>>>> will
> >>>>>> create a repo and describe the process (which we, of course, can
> >>>> discuss
> >>>>>> and modify going forward). Let's, at the very least, try and see
> >>> where
> >>>> it
> >>>>>> leads us.
> >>>>>>
> >>>>>> If someone has any concrete alternative options on how to we can
> >>>> maintain
> >>>>>> two major versions in parallel, let's have another voice discussion
> >>>> this
> >>>>>> Friday. If we do the meeting, we should set it up with a clear goal
> >>> to
> >>>>> make
> >>>>>> a decision. Please let me know if there is interest in this.
> >>>>>>
> >>>>>> -Val
> >>>>>>
> >>>>>> On Mon, Nov 16, 2020 at 6:31 AM Alexey Goncharuk <
> >>>>>> alexey.goncharuk@gmail.com> wrote:
> >>>>>>
> >>>>>>> Good,
> >>>>>>>
> >>>>>>> I think we have an intermediate agreement on the scope and
> >>>> significance
> >>>>> of
> >>>>>>> the changes we want to make. I suggest creating separate
> >> discussion
> >>>>>>> streams
> >>>>>>> and calls for each of the suggested topics so that:
> >>>>>>>
> >>>>>>> - It is clear for the community what is the motivation of the
> >>>> stream
> >>>>>>> (this includes both functional targets and technical debt
> >> issues
> >>>>>>> pointed
> >>>>>>> out by Sergey)
> >>>>>>> - Who is planning to take an active part in each of the streams
> >>>> (i.e.
> >>>>>>> the 'design committee', as Sergey suggested)
> >>>>>>> - What are the intermediate and final goals for each of the
> >>> streams
> >>>>>>> - What are the cross-stream interactions and how we integrate
> >>> them
> >>>>>>> - How each of the streams will be integrated with the current
> >>>>> codebase
> >>>>>>> based on the above (here is where we will see whether drop-in
> >> or
> >>>>>>> incremental approaches make more sense)
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> Best regards,
> >>>> Andrey V. Mashenkov
> >>>>
> >>>
> >>
> >>
> >> --
> >> Best regards,
> >> Andrey V. Mashenkov
> >>
>
>
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Petr Ivanov <mr...@gmail.com>.
More or less, unless we specifically forbid that, I guess
However there is bigger concern: JDK 15 is STS, so after half of a year it will be out of support and no major production team will use that JDK in their environment.
I would stick to JDK 11 as it is LTS at least until JDK17, plus — a lot of efforts should be made to enforce Apache Ignite be built on JDK 11 alone, not to mention 15th version...
Also, If we are going to introduce such major changes, I'd like to purpose maven project refactoring as well:
1. Full revision of all ant-calling tasks with javascript functions and alike — the complexity of those are overwhelming, something new should be at least researched.
2. Full revisions of profiles (for both root and modules) as there are some obsolete ones, and some that do ambiguous or, even worse, totally unknown tasks.
3. Introduce plugin and dependency management sections to control over and concrete versions of software we are relying in our project. Additionally — add BOM with all Ignite modules and their dependencies, which should help our users to better embed Ignite to their projects.
4. Up all versions of plugins and dependencies where possible to there current production versions (for plugins — it should be a must if we are ever going to build project under latest JDK versions).
5. Prepare project for parallel building, testing and assembling where possible for faster development process, with possible additional enhancement like incremental rebuild.
6. Possibly — research alternate builders, like Gradle (thought there are a lot of questions to its race condition issues during multimodule builds).
And last, but not least — think of migrating to 'CI/CD as a Code' approach for our main instrument TeamCity.
Whole project (both test and release build configurations) can be described using DSL (Kotlin in case of TC) and stored in VCS, forcing infrastructure changes to go through the same development processes including discussions, review, change history and etc.
Only I am not sure for now about where should the code be stored — in separate repository (secure, but disables testing of code with TC settings both in single PR), or alongside project's code (can be possible security hole).
That would require additional dev thread I think.
WDYT?
> On 24 Nov 2020, at 20:04, Pavel Tupitsyn <pt...@apache.org> wrote:
>
> If we use Java15 for development, can the resulting package be used from a
> Java11 app (the latest LTS)?
>
> On Tue, Nov 24, 2020 at 7:51 PM Andrey Mashenkov <an...@gmail.com>
> wrote:
>
>> Jave15 looks awesome.
>>
>> * Hidden classes [1] can be used by codegenerators.
>> * Records [2] can replace boilerplate code like IgniteBiTuple, GridTupleX.
>>
>> [1] https://openjdk.java.net/jeps/371
>> [2] https://openjdk.java.net/jeps/384
>>
>> On Tue, Nov 24, 2020 at 3:38 PM Alexey Zinoviev <za...@gmail.com>
>> wrote:
>>
>>> Java 15 is better, VarHandles, ForeignMemory access and so on.
>>>
>>> In both cases, I support the Java version 11 and higher for the
>>> development!
>>>
>>> вт, 24 нояб. 2020 г. в 15:21, Andrey Mashenkov <
>> andrey.mashenkov@gmail.com
>>>> :
>>>
>>>> Let's add maven plugins for sanity checks at the early stage.
>>>> I've created a ticket for this [1].
>>>>
>>>> Also, I've found initial pom.xml has a target version Java 8.
>>>> Do we intend to move to Java 11 (or may be next LTS) and drop Java 8 in
>>>> Ignite 3.0?
>>>>
>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13751
>>>>
>>>> On Tue, Nov 24, 2020 at 5:40 AM Valentin Kulichenko <
>>>> valentin.kulichenko@gmail.com> wrote:
>>>>
>>>>> Folks,
>>>>>
>>>>> I went ahead and created the repository [1]. I also configured a
>>> TeamCity
>>>>> project [2] that runs all available JUnit tests on every PR creation
>> or
>>>>> update. It also sends the status update to GitHub so that it's
>>> reflected
>>>> in
>>>>> the PR itself so that we can do merges directly from GitHub. Basic
>>> steps
>>>> to
>>>>> make a change are described on the Wiki page [3].
>>>>>
>>>>> Let me know if you have any questions.
>>>>>
>>>>> [1] https://github.com/apache/ignite-3
>>>>> [2] https://ci.ignite.apache.org/project/ignite3
>>>>> [3]
>>>>>
>>>>>
>>>>
>>>
>> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0#ApacheIgnite3.0-DevelopmentProcess
>>>>>
>>>>> -Val
>>>>>
>>>>> On Wed, Nov 18, 2020 at 4:24 PM Valentin Kulichenko <
>>>>> valentin.kulichenko@gmail.com> wrote:
>>>>>
>>>>>> Thanks, guys. It looks like we are much closer to the consensus
>> now.
>>> I
>>>>>> totally on board with the plan, but I would also like to address
>> the
>>>>>> short-term needs. As I've already mentioned earlier, there are
>>> several
>>>>>> active IEPs, but we still don't have even a preliminary technical
>>>> process
>>>>>> for working on these IEPs. I believe this might be frustrating for
>>> the
>>>>>> folks who would like to commit code.
>>>>>>
>>>>>> The scope we agreed on is quite big, and it will surely take
>>>> significant
>>>>>> time to implement all the changes and stabilize them. Therefore,
>> it's
>>>>> clear
>>>>>> to me that we will have to maintain 2.x and 3.x in parallel for
>> quite
>>>>> some
>>>>>> time - this needs to be addressed somehow. I'm convinced that
>> having
>>> a
>>>>>> separate repo is the ONLY way to do that, and so far, I haven't
>> heard
>>>> any
>>>>>> clear alternatives or reasons why we shouldn't do this.
>>>>>>
>>>>>> That said, I'm inclined to proceed with this in the next few days
>> - I
>>>>> will
>>>>>> create a repo and describe the process (which we, of course, can
>>>> discuss
>>>>>> and modify going forward). Let's, at the very least, try and see
>>> where
>>>> it
>>>>>> leads us.
>>>>>>
>>>>>> If someone has any concrete alternative options on how to we can
>>>> maintain
>>>>>> two major versions in parallel, let's have another voice discussion
>>>> this
>>>>>> Friday. If we do the meeting, we should set it up with a clear goal
>>> to
>>>>> make
>>>>>> a decision. Please let me know if there is interest in this.
>>>>>>
>>>>>> -Val
>>>>>>
>>>>>> On Mon, Nov 16, 2020 at 6:31 AM Alexey Goncharuk <
>>>>>> alexey.goncharuk@gmail.com> wrote:
>>>>>>
>>>>>>> Good,
>>>>>>>
>>>>>>> I think we have an intermediate agreement on the scope and
>>>> significance
>>>>> of
>>>>>>> the changes we want to make. I suggest creating separate
>> discussion
>>>>>>> streams
>>>>>>> and calls for each of the suggested topics so that:
>>>>>>>
>>>>>>> - It is clear for the community what is the motivation of the
>>>> stream
>>>>>>> (this includes both functional targets and technical debt
>> issues
>>>>>>> pointed
>>>>>>> out by Sergey)
>>>>>>> - Who is planning to take an active part in each of the streams
>>>> (i.e.
>>>>>>> the 'design committee', as Sergey suggested)
>>>>>>> - What are the intermediate and final goals for each of the
>>> streams
>>>>>>> - What are the cross-stream interactions and how we integrate
>>> them
>>>>>>> - How each of the streams will be integrated with the current
>>>>> codebase
>>>>>>> based on the above (here is where we will see whether drop-in
>> or
>>>>>>> incremental approaches make more sense)
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Best regards,
>>>> Andrey V. Mashenkov
>>>>
>>>
>>
>>
>> --
>> Best regards,
>> Andrey V. Mashenkov
>>
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Pavel Tupitsyn <pt...@apache.org>.
If we use Java15 for development, can the resulting package be used from a
Java11 app (the latest LTS)?
On Tue, Nov 24, 2020 at 7:51 PM Andrey Mashenkov <an...@gmail.com>
wrote:
> Jave15 looks awesome.
>
> * Hidden classes [1] can be used by codegenerators.
> * Records [2] can replace boilerplate code like IgniteBiTuple, GridTupleX.
>
> [1] https://openjdk.java.net/jeps/371
> [2] https://openjdk.java.net/jeps/384
>
> On Tue, Nov 24, 2020 at 3:38 PM Alexey Zinoviev <za...@gmail.com>
> wrote:
>
> > Java 15 is better, VarHandles, ForeignMemory access and so on.
> >
> > In both cases, I support the Java version 11 and higher for the
> > development!
> >
> > вт, 24 нояб. 2020 г. в 15:21, Andrey Mashenkov <
> andrey.mashenkov@gmail.com
> > >:
> >
> > > Let's add maven plugins for sanity checks at the early stage.
> > > I've created a ticket for this [1].
> > >
> > > Also, I've found initial pom.xml has a target version Java 8.
> > > Do we intend to move to Java 11 (or may be next LTS) and drop Java 8 in
> > > Ignite 3.0?
> > >
> > > [1] https://issues.apache.org/jira/browse/IGNITE-13751
> > >
> > > On Tue, Nov 24, 2020 at 5:40 AM Valentin Kulichenko <
> > > valentin.kulichenko@gmail.com> wrote:
> > >
> > > > Folks,
> > > >
> > > > I went ahead and created the repository [1]. I also configured a
> > TeamCity
> > > > project [2] that runs all available JUnit tests on every PR creation
> or
> > > > update. It also sends the status update to GitHub so that it's
> > reflected
> > > in
> > > > the PR itself so that we can do merges directly from GitHub. Basic
> > steps
> > > to
> > > > make a change are described on the Wiki page [3].
> > > >
> > > > Let me know if you have any questions.
> > > >
> > > > [1] https://github.com/apache/ignite-3
> > > > [2] https://ci.ignite.apache.org/project/ignite3
> > > > [3]
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0#ApacheIgnite3.0-DevelopmentProcess
> > > >
> > > > -Val
> > > >
> > > > On Wed, Nov 18, 2020 at 4:24 PM Valentin Kulichenko <
> > > > valentin.kulichenko@gmail.com> wrote:
> > > >
> > > > > Thanks, guys. It looks like we are much closer to the consensus
> now.
> > I
> > > > > totally on board with the plan, but I would also like to address
> the
> > > > > short-term needs. As I've already mentioned earlier, there are
> > several
> > > > > active IEPs, but we still don't have even a preliminary technical
> > > process
> > > > > for working on these IEPs. I believe this might be frustrating for
> > the
> > > > > folks who would like to commit code.
> > > > >
> > > > > The scope we agreed on is quite big, and it will surely take
> > > significant
> > > > > time to implement all the changes and stabilize them. Therefore,
> it's
> > > > clear
> > > > > to me that we will have to maintain 2.x and 3.x in parallel for
> quite
> > > > some
> > > > > time - this needs to be addressed somehow. I'm convinced that
> having
> > a
> > > > > separate repo is the ONLY way to do that, and so far, I haven't
> heard
> > > any
> > > > > clear alternatives or reasons why we shouldn't do this.
> > > > >
> > > > > That said, I'm inclined to proceed with this in the next few days
> - I
> > > > will
> > > > > create a repo and describe the process (which we, of course, can
> > > discuss
> > > > > and modify going forward). Let's, at the very least, try and see
> > where
> > > it
> > > > > leads us.
> > > > >
> > > > > If someone has any concrete alternative options on how to we can
> > > maintain
> > > > > two major versions in parallel, let's have another voice discussion
> > > this
> > > > > Friday. If we do the meeting, we should set it up with a clear goal
> > to
> > > > make
> > > > > a decision. Please let me know if there is interest in this.
> > > > >
> > > > > -Val
> > > > >
> > > > > On Mon, Nov 16, 2020 at 6:31 AM Alexey Goncharuk <
> > > > > alexey.goncharuk@gmail.com> wrote:
> > > > >
> > > > >> Good,
> > > > >>
> > > > >> I think we have an intermediate agreement on the scope and
> > > significance
> > > > of
> > > > >> the changes we want to make. I suggest creating separate
> discussion
> > > > >> streams
> > > > >> and calls for each of the suggested topics so that:
> > > > >>
> > > > >> - It is clear for the community what is the motivation of the
> > > stream
> > > > >> (this includes both functional targets and technical debt
> issues
> > > > >> pointed
> > > > >> out by Sergey)
> > > > >> - Who is planning to take an active part in each of the streams
> > > (i.e.
> > > > >> the 'design committee', as Sergey suggested)
> > > > >> - What are the intermediate and final goals for each of the
> > streams
> > > > >> - What are the cross-stream interactions and how we integrate
> > them
> > > > >> - How each of the streams will be integrated with the current
> > > > codebase
> > > > >> based on the above (here is where we will see whether drop-in
> or
> > > > >> incremental approaches make more sense)
> > > > >>
> > > > >
> > > >
> > >
> > >
> > > --
> > > Best regards,
> > > Andrey V. Mashenkov
> > >
> >
>
>
> --
> Best regards,
> Andrey V. Mashenkov
>
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Andrey Mashenkov <an...@gmail.com>.
Jave15 looks awesome.
* Hidden classes [1] can be used by codegenerators.
* Records [2] can replace boilerplate code like IgniteBiTuple, GridTupleX.
[1] https://openjdk.java.net/jeps/371
[2] https://openjdk.java.net/jeps/384
On Tue, Nov 24, 2020 at 3:38 PM Alexey Zinoviev <za...@gmail.com>
wrote:
> Java 15 is better, VarHandles, ForeignMemory access and so on.
>
> In both cases, I support the Java version 11 and higher for the
> development!
>
> вт, 24 нояб. 2020 г. в 15:21, Andrey Mashenkov <andrey.mashenkov@gmail.com
> >:
>
> > Let's add maven plugins for sanity checks at the early stage.
> > I've created a ticket for this [1].
> >
> > Also, I've found initial pom.xml has a target version Java 8.
> > Do we intend to move to Java 11 (or may be next LTS) and drop Java 8 in
> > Ignite 3.0?
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-13751
> >
> > On Tue, Nov 24, 2020 at 5:40 AM Valentin Kulichenko <
> > valentin.kulichenko@gmail.com> wrote:
> >
> > > Folks,
> > >
> > > I went ahead and created the repository [1]. I also configured a
> TeamCity
> > > project [2] that runs all available JUnit tests on every PR creation or
> > > update. It also sends the status update to GitHub so that it's
> reflected
> > in
> > > the PR itself so that we can do merges directly from GitHub. Basic
> steps
> > to
> > > make a change are described on the Wiki page [3].
> > >
> > > Let me know if you have any questions.
> > >
> > > [1] https://github.com/apache/ignite-3
> > > [2] https://ci.ignite.apache.org/project/ignite3
> > > [3]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0#ApacheIgnite3.0-DevelopmentProcess
> > >
> > > -Val
> > >
> > > On Wed, Nov 18, 2020 at 4:24 PM Valentin Kulichenko <
> > > valentin.kulichenko@gmail.com> wrote:
> > >
> > > > Thanks, guys. It looks like we are much closer to the consensus now.
> I
> > > > totally on board with the plan, but I would also like to address the
> > > > short-term needs. As I've already mentioned earlier, there are
> several
> > > > active IEPs, but we still don't have even a preliminary technical
> > process
> > > > for working on these IEPs. I believe this might be frustrating for
> the
> > > > folks who would like to commit code.
> > > >
> > > > The scope we agreed on is quite big, and it will surely take
> > significant
> > > > time to implement all the changes and stabilize them. Therefore, it's
> > > clear
> > > > to me that we will have to maintain 2.x and 3.x in parallel for quite
> > > some
> > > > time - this needs to be addressed somehow. I'm convinced that having
> a
> > > > separate repo is the ONLY way to do that, and so far, I haven't heard
> > any
> > > > clear alternatives or reasons why we shouldn't do this.
> > > >
> > > > That said, I'm inclined to proceed with this in the next few days - I
> > > will
> > > > create a repo and describe the process (which we, of course, can
> > discuss
> > > > and modify going forward). Let's, at the very least, try and see
> where
> > it
> > > > leads us.
> > > >
> > > > If someone has any concrete alternative options on how to we can
> > maintain
> > > > two major versions in parallel, let's have another voice discussion
> > this
> > > > Friday. If we do the meeting, we should set it up with a clear goal
> to
> > > make
> > > > a decision. Please let me know if there is interest in this.
> > > >
> > > > -Val
> > > >
> > > > On Mon, Nov 16, 2020 at 6:31 AM Alexey Goncharuk <
> > > > alexey.goncharuk@gmail.com> wrote:
> > > >
> > > >> Good,
> > > >>
> > > >> I think we have an intermediate agreement on the scope and
> > significance
> > > of
> > > >> the changes we want to make. I suggest creating separate discussion
> > > >> streams
> > > >> and calls for each of the suggested topics so that:
> > > >>
> > > >> - It is clear for the community what is the motivation of the
> > stream
> > > >> (this includes both functional targets and technical debt issues
> > > >> pointed
> > > >> out by Sergey)
> > > >> - Who is planning to take an active part in each of the streams
> > (i.e.
> > > >> the 'design committee', as Sergey suggested)
> > > >> - What are the intermediate and final goals for each of the
> streams
> > > >> - What are the cross-stream interactions and how we integrate
> them
> > > >> - How each of the streams will be integrated with the current
> > > codebase
> > > >> based on the above (here is where we will see whether drop-in or
> > > >> incremental approaches make more sense)
> > > >>
> > > >
> > >
> >
> >
> > --
> > Best regards,
> > Andrey V. Mashenkov
> >
>
--
Best regards,
Andrey V. Mashenkov
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Alexey Zinoviev <za...@gmail.com>.
Java 15 is better, VarHandles, ForeignMemory access and so on.
In both cases, I support the Java version 11 and higher for the development!
вт, 24 нояб. 2020 г. в 15:21, Andrey Mashenkov <an...@gmail.com>:
> Let's add maven plugins for sanity checks at the early stage.
> I've created a ticket for this [1].
>
> Also, I've found initial pom.xml has a target version Java 8.
> Do we intend to move to Java 11 (or may be next LTS) and drop Java 8 in
> Ignite 3.0?
>
> [1] https://issues.apache.org/jira/browse/IGNITE-13751
>
> On Tue, Nov 24, 2020 at 5:40 AM Valentin Kulichenko <
> valentin.kulichenko@gmail.com> wrote:
>
> > Folks,
> >
> > I went ahead and created the repository [1]. I also configured a TeamCity
> > project [2] that runs all available JUnit tests on every PR creation or
> > update. It also sends the status update to GitHub so that it's reflected
> in
> > the PR itself so that we can do merges directly from GitHub. Basic steps
> to
> > make a change are described on the Wiki page [3].
> >
> > Let me know if you have any questions.
> >
> > [1] https://github.com/apache/ignite-3
> > [2] https://ci.ignite.apache.org/project/ignite3
> > [3]
> >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0#ApacheIgnite3.0-DevelopmentProcess
> >
> > -Val
> >
> > On Wed, Nov 18, 2020 at 4:24 PM Valentin Kulichenko <
> > valentin.kulichenko@gmail.com> wrote:
> >
> > > Thanks, guys. It looks like we are much closer to the consensus now. I
> > > totally on board with the plan, but I would also like to address the
> > > short-term needs. As I've already mentioned earlier, there are several
> > > active IEPs, but we still don't have even a preliminary technical
> process
> > > for working on these IEPs. I believe this might be frustrating for the
> > > folks who would like to commit code.
> > >
> > > The scope we agreed on is quite big, and it will surely take
> significant
> > > time to implement all the changes and stabilize them. Therefore, it's
> > clear
> > > to me that we will have to maintain 2.x and 3.x in parallel for quite
> > some
> > > time - this needs to be addressed somehow. I'm convinced that having a
> > > separate repo is the ONLY way to do that, and so far, I haven't heard
> any
> > > clear alternatives or reasons why we shouldn't do this.
> > >
> > > That said, I'm inclined to proceed with this in the next few days - I
> > will
> > > create a repo and describe the process (which we, of course, can
> discuss
> > > and modify going forward). Let's, at the very least, try and see where
> it
> > > leads us.
> > >
> > > If someone has any concrete alternative options on how to we can
> maintain
> > > two major versions in parallel, let's have another voice discussion
> this
> > > Friday. If we do the meeting, we should set it up with a clear goal to
> > make
> > > a decision. Please let me know if there is interest in this.
> > >
> > > -Val
> > >
> > > On Mon, Nov 16, 2020 at 6:31 AM Alexey Goncharuk <
> > > alexey.goncharuk@gmail.com> wrote:
> > >
> > >> Good,
> > >>
> > >> I think we have an intermediate agreement on the scope and
> significance
> > of
> > >> the changes we want to make. I suggest creating separate discussion
> > >> streams
> > >> and calls for each of the suggested topics so that:
> > >>
> > >> - It is clear for the community what is the motivation of the
> stream
> > >> (this includes both functional targets and technical debt issues
> > >> pointed
> > >> out by Sergey)
> > >> - Who is planning to take an active part in each of the streams
> (i.e.
> > >> the 'design committee', as Sergey suggested)
> > >> - What are the intermediate and final goals for each of the streams
> > >> - What are the cross-stream interactions and how we integrate them
> > >> - How each of the streams will be integrated with the current
> > codebase
> > >> based on the above (here is where we will see whether drop-in or
> > >> incremental approaches make more sense)
> > >>
> > >
> >
>
>
> --
> Best regards,
> Andrey V. Mashenkov
>
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Andrey Mashenkov <an...@gmail.com>.
Let's add maven plugins for sanity checks at the early stage.
I've created a ticket for this [1].
Also, I've found initial pom.xml has a target version Java 8.
Do we intend to move to Java 11 (or may be next LTS) and drop Java 8 in
Ignite 3.0?
[1] https://issues.apache.org/jira/browse/IGNITE-13751
On Tue, Nov 24, 2020 at 5:40 AM Valentin Kulichenko <
valentin.kulichenko@gmail.com> wrote:
> Folks,
>
> I went ahead and created the repository [1]. I also configured a TeamCity
> project [2] that runs all available JUnit tests on every PR creation or
> update. It also sends the status update to GitHub so that it's reflected in
> the PR itself so that we can do merges directly from GitHub. Basic steps to
> make a change are described on the Wiki page [3].
>
> Let me know if you have any questions.
>
> [1] https://github.com/apache/ignite-3
> [2] https://ci.ignite.apache.org/project/ignite3
> [3]
>
> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0#ApacheIgnite3.0-DevelopmentProcess
>
> -Val
>
> On Wed, Nov 18, 2020 at 4:24 PM Valentin Kulichenko <
> valentin.kulichenko@gmail.com> wrote:
>
> > Thanks, guys. It looks like we are much closer to the consensus now. I
> > totally on board with the plan, but I would also like to address the
> > short-term needs. As I've already mentioned earlier, there are several
> > active IEPs, but we still don't have even a preliminary technical process
> > for working on these IEPs. I believe this might be frustrating for the
> > folks who would like to commit code.
> >
> > The scope we agreed on is quite big, and it will surely take significant
> > time to implement all the changes and stabilize them. Therefore, it's
> clear
> > to me that we will have to maintain 2.x and 3.x in parallel for quite
> some
> > time - this needs to be addressed somehow. I'm convinced that having a
> > separate repo is the ONLY way to do that, and so far, I haven't heard any
> > clear alternatives or reasons why we shouldn't do this.
> >
> > That said, I'm inclined to proceed with this in the next few days - I
> will
> > create a repo and describe the process (which we, of course, can discuss
> > and modify going forward). Let's, at the very least, try and see where it
> > leads us.
> >
> > If someone has any concrete alternative options on how to we can maintain
> > two major versions in parallel, let's have another voice discussion this
> > Friday. If we do the meeting, we should set it up with a clear goal to
> make
> > a decision. Please let me know if there is interest in this.
> >
> > -Val
> >
> > On Mon, Nov 16, 2020 at 6:31 AM Alexey Goncharuk <
> > alexey.goncharuk@gmail.com> wrote:
> >
> >> Good,
> >>
> >> I think we have an intermediate agreement on the scope and significance
> of
> >> the changes we want to make. I suggest creating separate discussion
> >> streams
> >> and calls for each of the suggested topics so that:
> >>
> >> - It is clear for the community what is the motivation of the stream
> >> (this includes both functional targets and technical debt issues
> >> pointed
> >> out by Sergey)
> >> - Who is planning to take an active part in each of the streams (i.e.
> >> the 'design committee', as Sergey suggested)
> >> - What are the intermediate and final goals for each of the streams
> >> - What are the cross-stream interactions and how we integrate them
> >> - How each of the streams will be integrated with the current
> codebase
> >> based on the above (here is where we will see whether drop-in or
> >> incremental approaches make more sense)
> >>
> >
>
--
Best regards,
Andrey V. Mashenkov
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Valentin Kulichenko <va...@gmail.com>.
Folks,
I went ahead and created the repository [1]. I also configured a TeamCity
project [2] that runs all available JUnit tests on every PR creation or
update. It also sends the status update to GitHub so that it's reflected in
the PR itself so that we can do merges directly from GitHub. Basic steps to
make a change are described on the Wiki page [3].
Let me know if you have any questions.
[1] https://github.com/apache/ignite-3
[2] https://ci.ignite.apache.org/project/ignite3
[3]
https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0#ApacheIgnite3.0-DevelopmentProcess
-Val
On Wed, Nov 18, 2020 at 4:24 PM Valentin Kulichenko <
valentin.kulichenko@gmail.com> wrote:
> Thanks, guys. It looks like we are much closer to the consensus now. I
> totally on board with the plan, but I would also like to address the
> short-term needs. As I've already mentioned earlier, there are several
> active IEPs, but we still don't have even a preliminary technical process
> for working on these IEPs. I believe this might be frustrating for the
> folks who would like to commit code.
>
> The scope we agreed on is quite big, and it will surely take significant
> time to implement all the changes and stabilize them. Therefore, it's clear
> to me that we will have to maintain 2.x and 3.x in parallel for quite some
> time - this needs to be addressed somehow. I'm convinced that having a
> separate repo is the ONLY way to do that, and so far, I haven't heard any
> clear alternatives or reasons why we shouldn't do this.
>
> That said, I'm inclined to proceed with this in the next few days - I will
> create a repo and describe the process (which we, of course, can discuss
> and modify going forward). Let's, at the very least, try and see where it
> leads us.
>
> If someone has any concrete alternative options on how to we can maintain
> two major versions in parallel, let's have another voice discussion this
> Friday. If we do the meeting, we should set it up with a clear goal to make
> a decision. Please let me know if there is interest in this.
>
> -Val
>
> On Mon, Nov 16, 2020 at 6:31 AM Alexey Goncharuk <
> alexey.goncharuk@gmail.com> wrote:
>
>> Good,
>>
>> I think we have an intermediate agreement on the scope and significance of
>> the changes we want to make. I suggest creating separate discussion
>> streams
>> and calls for each of the suggested topics so that:
>>
>> - It is clear for the community what is the motivation of the stream
>> (this includes both functional targets and technical debt issues
>> pointed
>> out by Sergey)
>> - Who is planning to take an active part in each of the streams (i.e.
>> the 'design committee', as Sergey suggested)
>> - What are the intermediate and final goals for each of the streams
>> - What are the cross-stream interactions and how we integrate them
>> - How each of the streams will be integrated with the current codebase
>> based on the above (here is where we will see whether drop-in or
>> incremental approaches make more sense)
>>
>
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Valentin Kulichenko <va...@gmail.com>.
Thanks, guys. It looks like we are much closer to the consensus now. I
totally on board with the plan, but I would also like to address the
short-term needs. As I've already mentioned earlier, there are several
active IEPs, but we still don't have even a preliminary technical process
for working on these IEPs. I believe this might be frustrating for the
folks who would like to commit code.
The scope we agreed on is quite big, and it will surely take significant
time to implement all the changes and stabilize them. Therefore, it's clear
to me that we will have to maintain 2.x and 3.x in parallel for quite some
time - this needs to be addressed somehow. I'm convinced that having a
separate repo is the ONLY way to do that, and so far, I haven't heard any
clear alternatives or reasons why we shouldn't do this.
That said, I'm inclined to proceed with this in the next few days - I will
create a repo and describe the process (which we, of course, can discuss
and modify going forward). Let's, at the very least, try and see where it
leads us.
If someone has any concrete alternative options on how to we can maintain
two major versions in parallel, let's have another voice discussion this
Friday. If we do the meeting, we should set it up with a clear goal to make
a decision. Please let me know if there is interest in this.
-Val
On Mon, Nov 16, 2020 at 6:31 AM Alexey Goncharuk <al...@gmail.com>
wrote:
> Good,
>
> I think we have an intermediate agreement on the scope and significance of
> the changes we want to make. I suggest creating separate discussion streams
> and calls for each of the suggested topics so that:
>
> - It is clear for the community what is the motivation of the stream
> (this includes both functional targets and technical debt issues pointed
> out by Sergey)
> - Who is planning to take an active part in each of the streams (i.e.
> the 'design committee', as Sergey suggested)
> - What are the intermediate and final goals for each of the streams
> - What are the cross-stream interactions and how we integrate them
> - How each of the streams will be integrated with the current codebase
> based on the above (here is where we will see whether drop-in or
> incremental approaches make more sense)
>
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Alexey Goncharuk <al...@gmail.com>.
Good,
I think we have an intermediate agreement on the scope and significance of
the changes we want to make. I suggest creating separate discussion streams
and calls for each of the suggested topics so that:
- It is clear for the community what is the motivation of the stream
(this includes both functional targets and technical debt issues pointed
out by Sergey)
- Who is planning to take an active part in each of the streams (i.e.
the 'design committee', as Sergey suggested)
- What are the intermediate and final goals for each of the streams
- What are the cross-stream interactions and how we integrate them
- How each of the streams will be integrated with the current codebase
based on the above (here is where we will see whether drop-in or
incremental approaches make more sense)
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Nikolay Izhikov <ni...@apache.org>.
Sergey.
> pay our (already huge) technical debt,
Can you, please, make your statement more specific?
What specific points of technical debt do we have?
I think we should write it down and solve the issues step by step.
> 16 нояб. 2020 г., в 14:28, Sergey Chugunov <se...@gmail.com> написал(а):
>
> Igniters,
>
> I agree that create or not create is not a question, rephrasing
> Shakespeare.
>
> My main point is that developing new features on top of old 2.x-style
> architecture is a bad idea. We will write the code and spend some time
> stabilizing it (which is expected and fine). But then, when we finally
> decide to fix our architecture and pay our (already huge) technical debt,
> we will have to rewrite this code again and spend time stabilizing it again.
>
> Creating new components on top of 2.x (which is actually 1.x, nothing
> fundamentally new was introduced in terms of architecture) is equal to
> wasting time now and creating more worthless work for the future.
>
> Earlier I suggested to rank all new features according to their criticality
> and amount of breaking changes and shape 3.0 scope based on this analysis.
> Let's get back to this idea and prepare a scope based on publicly shared
> arguments.
>
> One more thing I would add here. Our users are smart people and make
> decisions about upgrading or not upgrading to a new version based on
> cost/value balance. Incremental approach keeps cost (public API breaking
> changes) high but brings questionable amounts of value with each iteration.
> If we add more valuable features to 3.0 and force users to pay the cost
> only once they will be happier than if we split really needed changes to
> several major releases and send our users to hell of endless rewriting
> their codebases. In the latter case we'll see users to be much more
> reluctant to upgrade to newer versions.
>
> Hope this makes sense.
>
> On Mon, Nov 16, 2020 at 2:24 PM Nikolay Izhikov <ni...@apache.org> wrote:
>
>>> Let's indeed focus on Sergey's suggestions on the design->development
>> approach.
>>
>> +1
>>
>>> - API & configuration cleanup
>>> - New management tool
>>> - Schema-first approach
>>> - New replication infrastructure
>>
>> +1.
>>
>>> 16 нояб. 2020 г., в 13:40, Alexey Goncharuk <al...@gmail.com>
>> написал(а):
>>>
>>> Folks,
>>>
>>> I think we are overly driven away by the phrase 'new repo' rather than
>> the
>>> essence of my suggestion. We can keep developing in the same repo, we can
>>> even keep developing in the master branch. My point is that Ignite 3.0
>> is a
>>> chance to move on with the architecture, so if we really want to make
>>> architectural improvements, we should not strive for incremental changes
>>> for *some parts of the code*.
>>>
>>> Maxim,
>>>
>>> To comment on your examples: I think that the huge effort that is
>> currently
>>> required to make any significant change in Ignite is the perfect example
>> of
>>> how we lack structure in the codebase. Yes, theoretically we can
>> introduce
>>> incremental changes in the code that will improve the structure, but my
>>> question is: we did not do it before, what will enforce us to make these
>>> changes now? With the current approach, adding a new feature increases
>> the
>>> test time non-linearly because without proper decoupling you have to test
>>> all possible combinations of features together. We can move faster than
>>> that.
>>>
>>> I also do not agree that we should reduce the scope of Ignite 3.0 that
>>> much. I do not see how the schema-first approach can be properly and
>>> reliably implemented without a reliable HA metastorage, which in turn
>>> requires a reliable replication protocol to be implemented. Besides, if a
>>> number of people want to work on some Ignite feature, why should they
>> wait
>>> because not all community members have time to review the changes?
>>>
>>> Let's indeed focus on Sergey's suggestions on the design->development
>>> approach. I back both Nikolay's and Maxim's scope, but I think we should
>>> unite them, not intersect, and the minimal list of changes to be included
>>> to Ignite 3.0 is:
>>>
>>> - API & configuration cleanup
>>> - New management tool
>>> - Schema-first approach
>>> - New replication infrastructure
>>>
>>> Any smaller subset of changes will leave Ignite 3.0 in a transient state
>>> with people being too afraid to move to it because there are more major
>>> breaking changes scheduled.
>>>
>>> пт, 13 нояб. 2020 г. в 18:28, Alexey Zinoviev <za...@gmail.com>:
>>>
>>>> I'm -1 for creating a new repo.
>>>> Also I support Maxim's plan for 3.0
>>>>
>>>> пт, 13 нояб. 2020 г. в 15:50, Maxim Muzafarov <mm...@apache.org>:
>>>>
>>>>> Val,
>>>>>
>>>>>
>>>>> Why *creating a new repo* is the main point we faced with? Would it be
>>>>> better to discuss the components design approach and scope management
>>>>> first suggested by Sergey Chugunov? I doubt that new repo will solve
>>>>> move us forward.
>>>>>
>>>>> Currently, I'm -1 to create a new repo with the inputs above.
>>>>>
>>>>> In addition to Nikolay's answer I see the following drawbacks of
>>>>> creating new repo:
>>>>> - we have very few positive examples of finalizing really huge
>>>>> improvements to *production-ready* states the others remains
>>>>> incomplete (MVCC, Calcite, Zookeeper, Tracing, Thread per Partition,
>>>>> etc)
>>>>> - AFAIK, the Native Persistence took a very long period of
>>>>> stabilization even after it has been developed (we must take it into
>>>>> account for developing new features like IEP-61)
>>>>> - feature development for a long period of time (like 3.0) without any
>>>>> releases will lead to all these changes became obsolete at the moment
>>>>> of release (AFAIK the 2.8 which released a year ago still has no big
>>>>> deployments)
>>>>> - human resources -- some of the Igniters may lose their interest for
>>>>> 3.0 during development, some of them may switch to different projects,
>>>>> etc.
>>>>> - do we all estimating the scope of 3.0 correct? The 2.8 release took
>> 1.5
>>>>> years.
>>>>>
>>>>> Have I missed something?
>>>>>
>>>>>
>>>>> I suggest the following plan:
>>>>>
>>>>> - initiate 3.0 development in the master branch (after 2.10 release
>>>>> change version to 3.0-SNAPSHOT instead of 2.11-SNAPSHOT)
>>>>> - cleanup and collapse all the current APIs (see To Be Removed List
>>>>> For Discussion on Apache Ignite 3.0 Wishlist)
>>>>> - reduce the scope for 3.0 even more. I suggest focusing on two
>>>>> things: Calcite + Schema-first approach
>>>>> - create feature branches for proposed IEPs (for 3.0 only)
>>>>> - create the release road map (allocate e.g. IEP-61 to 4.0 etc.)
>>>>>
>>>>> On Fri, 13 Nov 2020 at 14:03, Ivan Daschinsky <iv...@gmail.com>
>>>> wrote:
>>>>>>
>>>>>>>> b. Implement IEP-61 - Common Replication Infrastructure
>>>>>> I suppose, that this is the main cause of the current discussion.
>>>>>> I hardly believe that this activity can be done without at least
>>>>> creating a
>>>>>> completely new branch.
>>>>>>
>>>>>> пт, 13 нояб. 2020 г. в 11:12, Nikolay Izhikov <ni...@apache.org>:
>>>>>>
>>>>>>> My suggestion:
>>>>>>>
>>>>>>> 1. Reduce Ignite3 scope to the following:
>>>>>>> a. Delete all deprecated API and support of obsolete internal
>>>>>>> protocols.
>>>>>>> b. Implement IEP-61 - Common Replication Infrastructure
>>>>>>> c. Implement new Ignite management tool ignitectl as
>>>> suggested
>>>>>>> during Ignite3 discussion.
>>>>>>>
>>>>>>> 2. Implement and release following improvements like transactions,
>>>>> Calcite
>>>>>>> based SQL, etc in the ongoing releases - Ignite 4, 5, 6
>>>>>>>
>>>>>>> My concern against separate Ignite 3 repo is the following:
>>>>>>>
>>>>>>> 1. We spread community to the two very separated part - Ignite3
>>>>> developers
>>>>>>> and Ignite2 maintainers. believe it’s bad for our community.
>>>>>>> That can lead to the situation when we don’t fix critical or
>>>>>>> blocker issueds «because they will not exists in Ignite3»
>>>>>>> That will lead to the solutions never reviewed or reviewed
>>>>> poorly.
>>>>>>>
>>>>>>> 2. It seems for me that current scope of Ignite3 is too big to be
>>>>>>> implemented in any reasonable time.
>>>>>>>
>>>>>>>
>>>>>>>> 13 нояб. 2020 г., в 10:57, Nikolay Izhikov <NIzhikov.dev@gmail.com
>>>>>
>>>>>>> написал(а):
>>>>>>>>
>>>>>>>> Hello, Valentin.
>>>>>>>>
>>>>>>>>> Nikolay, Maxim, are you OK with this route?
>>>>>>>>
>>>>>>>> -1 to have another repo for Ignite3 development.
>>>>>>>>
>>>>>>>>> 13 нояб. 2020 г., в 03:04, Valentin Kulichenko <
>>>>>>> valentin.kulichenko@gmail.com> написал(а):
>>>>>>>>>
>>>>>>>>> Folks,
>>>>>>>>>
>>>>>>>>> We already have multiple IEPs for Ignite 3.0, and as far as I
>>>> know,
>>>>>>> there are contributors that would like to work on them (or probably
>>>>> already
>>>>>>> started). That said, we should make a decision as soon as possible.
>>>>>>>>>
>>>>>>>>> At this point, it doesn't seem that there are any strong
>>>> objections
>>>>> to
>>>>>>> the technical side of things. So I would suggest the following:
>>>>>>>>>
>>>>>>>>> 1. Proceed with Alexey's approach to the development process, as
>>>> it
>>>>>>> seems to be the best (in my opinion - the only) way to address all
>>>> the
>>>>>>> technical concerns and issues expressed in the thread. We'll start by
>>>>>>> creating a new repo and a new TC project.
>>>>>>>>> 2. Start a separate discussion around transparency. If there are
>>>> any
>>>>>>> changes we need to make to our contributor guidelines, I am happy to
>>>>> talk
>>>>>>> them through, but I don't think it's reasonable to delay feature
>>>>>>> development because of this. In the short term, I will make sure that
>>>>>>> everything that happens within the new repo is as open to the
>>>>> community as
>>>>>>> possible.
>>>>>>>>>
>>>>>>>>> Nikolay, Maxim, are you OK with this route?
>>>>>>>>>
>>>>>>>>> -Val
>>>>>>>>>
>>>>>>>>> On Tue, Nov 10, 2020 at 4:55 PM Valentin Kulichenko <
>>>>>>> valentin.kulichenko@gmail.com> wrote:
>>>>>>>>> Maxim,
>>>>>>>>>
>>>>>>>>> 2.x and 3.x will have to coexist for some time - I don't see how
>>>> we
>>>>> can
>>>>>>> avoid this considering the set of proposed changes. That said, we
>>>>>>> effectively will need to have two "masters" - one for each major
>>>>> version.
>>>>>>> Master for 3.x can technically be a branch in the existing repo, but
>>>>> having
>>>>>>> a separate repo seems cleaner, simply because it will not be a
>>>>> "branch" in
>>>>>>> the traditional sense.
>>>>>>>>>
>>>>>>>>> Note that the new repo will still be under the Apache org, with
>>>> the
>>>>>>> same set of committers, managed by the community, etc. All the
>>>>> development
>>>>>>> happening for 3.0 must follow the rules that we currently have (if
>>>>>>> anything, it's an opportunity to improve those rules).
>>>>>>>>>
>>>>>>>>> As I said during the call on Friday, I strongly believe that if
>>>>> there
>>>>>>> is a transparency issue, it will exist regardless of the approach we
>>>>> choose
>>>>>>> for 3.0. If community members develop without IEPs or public
>>>>> discussions,
>>>>>>> this will happen for both 2.x and 3.x unless we address this
>>>>> separately. I
>>>>>>> don't see how this is related to Alexey's suggestion, which targets
>>>>>>> *technical* issues with the product more than anything else. This a
>>>>> way to
>>>>>>> achieve better modularity, introduce better coverage with unit tests,
>>>>>>> reduce conflicts during development, etc.
>>>>>>>>>
>>>>>>>>> Coming back to transparency, let's identify the issues and fix
>>>>> them. It
>>>>>>> probably makes sense to have a separate discussion on this topic.
>>>>>>>>>
>>>>>>>>> -Val
>>>>>>>>>
>>>>>>>>> On Tue, Nov 10, 2020 at 1:05 PM Maxim Muzafarov <
>>>> mmuzaf@apache.org>
>>>>>>> wrote:
>>>>>>>>> Sergey,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Your summary makes sense to me.
>>>>>>>>>
>>>>>>>>> However, how we come up from *Development transparency* to
>>>> *create a
>>>>>>>>> separate public repository dedicated for 3.0*? For me *development
>>>>>>>>> transparency* is about making changes in the master branch. These
>>>>>>>>> changes will definitely be seen by all the Ignite developers.
>>>>>>>>>
>>>>>>>>> A dedicated public repository is technically public and visible
>>>> for
>>>>>>>>> everyone, but it allows development without IEPs, without public
>>>>>>>>> discussion (since all the code changes are not related to the
>>>> master
>>>>>>>>> branch) it also allows a large number of assumptions and
>>>> deviations
>>>>>>>>> (like code-style violations). It also not about *development
>>>>>>>>> transparency* since developers which are working on 3.0 is only a
>>>>>>>>> subset of all Ignite developers which may continue working on 2.x.
>>>>> For
>>>>>>>>> me, this would be a huge step backwards.
>>>>>>>>>
>>>>>>>>> Ignite veterans should remember how long the branch stabilization
>>>>> took
>>>>>>>>> for the 2.x version with the PDS.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I think each breaking change should be passed through the master
>>>>> branch.
>>>>>>>>>
>>>>>>>>> On Tue, 10 Nov 2020 at 22:18, Alexei Scherbakov
>>>>>>>>> <al...@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>> Makes sense to me.
>>>>>>>>>>
>>>>>>>>>> вт, 10 нояб. 2020 г. в 18:47, Sergey Chugunov <
>>>>>>> sergey.chugunov@gmail.com>:
>>>>>>>>>>
>>>>>>>>>>> Igniters,
>>>>>>>>>>>
>>>>>>>>>>> I thought over Friday meeting ideas and concerns and summarized
>>>>> them
>>>>>>> in
>>>>>>>>>>> these three points:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 1. *Components design unification approach.* New proposed
>>>>> components
>>>>>>>>>>> will be developed by different contributors, but they need to
>>>> be
>>>>>>> unified
>>>>>>>>>>> and should integrate with each other easily. To ensure that I
>>>>>>> suggest
>>>>>>>>>>> calling an architecture group that will create design
>>>> guidelines
>>>>>>> for all
>>>>>>>>>>> components and high-level overview of overall architecture.
>>>> How
>>>>>>> code is
>>>>>>>>>>> split into components, what are component boundaries, how
>>>>> component
>>>>>>>>>>> lifecycle works and what are its interfaces - all these and
>>>>> other
>>>>>>>>>>> questions
>>>>>>>>>>> should be covered.
>>>>>>>>>>>
>>>>>>>>>>> 2. *Scope management.* Apache 3.0 should be implemented
>>>> within a
>>>>>>>>>>> reasonable time, so we need some procedure to decide whether a
>>>>>>>>>>> particular
>>>>>>>>>>> feature should be dropped from the scope of 3.0 and postponed
>>>>> to 3.1
>>>>>>>>>>> release. To do so I suggest to range all features by two
>>>>> parameters:
>>>>>>>>>>> criticality for 3.0 and amount of breaking changes. 3.0 scope
>>>>> should
>>>>>>>>>>> include features of high criticality AND features with a big
>>>>> amount
>>>>>>> of
>>>>>>>>>>> breaking changes. All other features can be made optional.
>>>>>>>>>>>
>>>>>>>>>>> 3. *Development transparency.* Development of all components
>>>>> should
>>>>>>> be
>>>>>>>>>>> made as transparent for everyone as possible. Any contributor
>>>>>>> should be
>>>>>>>>>>> able to look over any component at any stage of development.
>>>> To
>>>>>>> achieve
>>>>>>>>>>> this I suggest to create a separate public repository
>>>> dedicated
>>>>> for
>>>>>>> 3.0
>>>>>>>>>>> development. It will make the code available for everyone but
>>>>> when
>>>>>>>>>>> development of 3.0 is done we won't loose any stars of our
>>>>> current
>>>>>>>>>>> repository as we merge dev repo into main one and drop dev.
>>>>>>>>>>>
>>>>>>>>>>> Do these ideas make sense to you? Are there any concerns not
>>>>> covered
>>>>>>> by
>>>>>>>>>>> these suggestions?
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Nov 6, 2020 at 7:36 PM Kseniya Romanova <
>>>>>>> romanova.ks.spb@gmail.com
>>>>>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Here are the slides from Alexey Goncharuk. Let's think this
>>>> over
>>>>> and
>>>>>>>>>>>> continue on Monday:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>
>>>>>
>>>>
>> https://go.gridgain.com/rs/491-TWR-806/images/Ignite_3_Plans_and_development_process.pdf
>>>>>>>>>>>>
>>>>>>>>>>>> чт, 5 нояб. 2020 г. в 11:13, Anton Vinogradov <av...@apache.org>:
>>>>>>>>>>>>
>>>>>>>>>>>>> Folks,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Should we perform cleanup work before (r)evolutional changes?
>>>>>>>>>>>>> My huge proposal is to get rid of things which we don't need
>>>>> anyway
>>>>>>>>>>>>> - local caches,
>>>>>>>>>>>>> - strange tx modes,
>>>>>>>>>>>>> - code overcomplexity because of RollingUpgrade feature never
>>>>>>> attended
>>>>>>>>>>> at
>>>>>>>>>>>>> AI,
>>>>>>>>>>>>> - etc,
>>>>>>>>>>>>> before choosing the way.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Nov 3, 2020 at 3:31 PM Valentin Kulichenko <
>>>>>>>>>>>>> valentin.kulichenko@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Ksenia, thanks for scheduling this on such short notice!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> As for the original topic, I do support Alexey's idea. We're
>>>>> not
>>>>>>>>>>> going
>>>>>>>>>>>> to
>>>>>>>>>>>>>> rewrite anything from scratch, as most of the components are
>>>>> going
>>>>>>> to
>>>>>>>>>>>> be
>>>>>>>>>>>>>> moved as-is or with minimal modifications. However, the
>>>> changes
>>>>>>> that
>>>>>>>>>>>> are
>>>>>>>>>>>>>> proposed imply serious rework of the core parts of the code,
>>>>> which
>>>>>>>>>>> are
>>>>>>>>>>>>> not
>>>>>>>>>>>>>> properly decoupled from each other and from other parts. This
>>>>> makes
>>>>>>>>>>> the
>>>>>>>>>>>>>> incremental approach borderline impossible. Developing in a
>>>> new
>>>>>>> repo,
>>>>>>>>>>>>>> however, addresses this concern. As a bonus, we can also
>>>>> refactor
>>>>>>> the
>>>>>>>>>>>>> code,
>>>>>>>>>>>>>> introduce better decoupling, get rid of kernel context, and
>>>>> develop
>>>>>>>>>>>> unit
>>>>>>>>>>>>>> tests (finally!).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Basically, this proposal only affects the *process*, not the
>>>>> set of
>>>>>>>>>>>>> changes
>>>>>>>>>>>>>> we had discussed before. Ignite 3.0 is our unique chance to
>>>>> make
>>>>>>>>>>> things
>>>>>>>>>>>>>> right.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -Val
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Nov 3, 2020 at 3:06 AM Kseniya Romanova <
>>>>>>>>>>>>> romanova.ks.spb@gmail.com
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Pavel, all the interesting points will be anyway published
>>>>> here in
>>>>>>>>>>>>>> English
>>>>>>>>>>>>>>> (as the principal "if it's not on devlist it doesn't
>>>>> happened" is
>>>>>>>>>>>> still
>>>>>>>>>>>>>>> relevant). This is just a quick call for a group of
>>>>> developers.
>>>>>>>>>>> Later
>>>>>>>>>>>>> we
>>>>>>>>>>>>>>> can do a separate presentation of idea and discussion in
>>>>> English
>>>>>>> as
>>>>>>>>>>>> we
>>>>>>>>>>>>>> did
>>>>>>>>>>>>>>> for the Ignite 3.0 draft of changes.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> вт, 3 нояб. 2020 г. в 13:52, Pavel Tupitsyn <
>>>>> ptupitsyn@apache.org
>>>>>>>>>>>> :
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Kseniya,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks for scheduling this call.
>>>>>>>>>>>>>>>> Do you think we can switch to English if non-Russian
>>>> speaking
>>>>>>>>>>>>> community
>>>>>>>>>>>>>>>> members decide to join?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Tue, Nov 3, 2020 at 1:32 PM Kseniya Romanova <
>>>>>>>>>>>>>>> romanova.ks.spb@gmail.com
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Let's do this community discussion open. Here's the link
>>>> on
>>>>>>>>>>> zoom
>>>>>>>>>>>>> call
>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>> Russian for Friday 6 PM:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>> https://www.meetup.com/Moscow-Apache-Ignite-Meetup/events/274360378/
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> вт, 3 нояб. 2020 г. в 12:49, Nikolay Izhikov <
>>>>>>>>>>>> nizhikov@apache.org
>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Time works for me.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 3 нояб. 2020 г., в 12:40, Alexey Goncharuk <
>>>>>>>>>>>>>>>> alexey.goncharuk@gmail.com
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> написал(а):
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Nikolay,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I am up for the call. I will try to explain my reasoning
>>>>> in
>>>>>>>>>>>>>> greater
>>>>>>>>>>>>>>>>>> detail
>>>>>>>>>>>>>>>>>>> and will be glad to hear the concerns. Will this Friday,
>>>>>>>>>>> Nov
>>>>>>>>>>>>> 6th,
>>>>>>>>>>>>>>>> work?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> вт, 3 нояб. 2020 г. в 10:09, Nikolay Izhikov <
>>>>>>>>>>>>>> nizhikov@apache.org
>>>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Igniters, should we have a call for this topic?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> 2 нояб. 2020 г., в 18:53, Pavel Tupitsyn <
>>>>>>>>>>>>> ptupitsyn@apache.org
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> написал(а):
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> not intend to rewrite everything from scratch
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Every single test from Ignite 2.x should be moved to
>>>>>>>>>>>> Ignite
>>>>>>>>>>>>> 3
>>>>>>>>>>>>>>>>>>>>>> regardless of how we choose to proceed.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Alexey, thank you for the explanation, this addresses
>>>>> all
>>>>>>>>>>>> of
>>>>>>>>>>>>> my
>>>>>>>>>>>>>>>>>> concerns.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Mon, Nov 2, 2020 at 6:43 PM Andrey Mashenkov <
>>>>>>>>>>>>>>>>>>>> andrey.mashenkov@gmail.com>
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hi, Igniters.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> * AFAIU, we need a new repo if we want to apply
>>>>>>>>>>> different
>>>>>>>>>>>>>>>>> restrictions
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>> pull requests,
>>>>>>>>>>>>>>>>>>>>>> otherwise I see no difference for myself.
>>>>>>>>>>>>>>>>>>>>>> E.g. make static analysis (do we have?), compile,
>>>>>>>>>>> styles,
>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> javadoc
>>>>>>>>>>>>>>>>>>>>>> checks mandatory.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I think that relaxed requirements here will lead to
>>>> bad
>>>>>>>>>>>>>> product
>>>>>>>>>>>>>>>>>> quality.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> * Agree with Pavel, we should 'keep' integrations
>>>> tests
>>>>>>>>>>>>>> somehow.
>>>>>>>>>>>>>>>>>>>>>> During active development tests will be broken most
>>>> of
>>>>>>>>>>>> time,
>>>>>>>>>>>>>> so,
>>>>>>>>>>>>>>>>>>>>>> I'd port them e.g. suite-by-suite once we will have a
>>>>>>>>>>>> stable
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>> featured
>>>>>>>>>>>>>>>>>>>>>> environment to run them and of course make test's
>>>> code
>>>>>>>>>>>> clear
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> avoid
>>>>>>>>>>>>>>>>>>>>>> bad/non-relevant ones.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> * I like bottom-up approach.
>>>>>>>>>>>>>>>>>>>>>> With it we could make a better framework. I mean
>>>> clear
>>>>>>>>>>>>>> component
>>>>>>>>>>>>>>>>>>>> lifecycle,
>>>>>>>>>>>>>>>>>>>>>> component wiring mechanics, general methods to
>>>> approach
>>>>>>>>>>>> core
>>>>>>>>>>>>>>>>>> components
>>>>>>>>>>>>>>>>>>>>>> such as exchange/communication
>>>>>>>>>>>>>>>>>>>>>> to avoid code mess like we have in ExchangeFuture
>>>> with
>>>>>>>>>>> all
>>>>>>>>>>>>>> these
>>>>>>>>>>>>>>>>>> custom
>>>>>>>>>>>>>>>>>>>>>> callbacks for each component, interfaces like
>>>>>>>>>>>>>>>>>>>>>> PartitionsExchangeAware,
>>>> IgniteChangeGlobalStateSupport
>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>> a pack of
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>> start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected
>>>>>>>>>>>>>>>>>>>>>> and so on in various unexpected places.
>>>>>>>>>>>>>>>>>>>>>> Hope, we will be able to port most of the good code
>>>> to
>>>>>>>>>>> the
>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>> framework
>>>>>>>>>>>>>>>>>>>>>> version.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk <
>>>>>>>>>>>>>>>>>>>>>> alexey.goncharuk@gmail.com>
>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Nikolay, Pavel,
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback! First of all, I wanted to
>>>>>>>>>>> stress
>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>> do
>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>> intend to rewrite everything from scratch (I never
>>>>> used
>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>> phrase).
>>>>>>>>>>>>>>>>>>>>>> There
>>>>>>>>>>>>>>>>>>>>>>> are significant parts of code that would be moved
>>>> with
>>>>>>>>>>>>>> minimal
>>>>>>>>>>>>>>>>>>>>>>> modifications. Second, I never said that we will get
>>>>>>>>>>> rid
>>>>>>>>>>>> of
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> old
>>>>>>>>>>>>>>>>>>>> tests
>>>>>>>>>>>>>>>>>>>>>>> codebase. Every single test from Ignite 2.x should
>>>> be
>>>>>>>>>>>> moved
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> Ignite 3
>>>>>>>>>>>>>>>>>>>>>>> regardless of how we choose to proceed.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> My point is that for some parts of the code a clean
>>>>>>>>>>>>> bottom-up
>>>>>>>>>>>>>>>>>>>>>>> implementation will be cheaper in many ways. Let me
>>>>>>>>>>> give
>>>>>>>>>>>>> you
>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>> few
>>>>>>>>>>>>>>>>>>>>>> concrete
>>>>>>>>>>>>>>>>>>>>>>> examples:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> - I think no one can object that we need a cleanly
>>>>>>>>>>>>> separated
>>>>>>>>>>>>>>>>>>>>>> persistence
>>>>>>>>>>>>>>>>>>>>>>> layer for Ignite. There is a very raw draft IEP for
>>>>>>>>>>> this
>>>>>>>>>>>>>>>> already.
>>>>>>>>>>>>>>>>> On
>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> other hand, I think we also can agree that we need a
>>>>>>>>>>>>>>> split-brain
>>>>>>>>>>>>>>>>>>>>>>> resistant
>>>>>>>>>>>>>>>>>>>>>>> replication protocol for caches. There is also an
>>>> IEP
>>>>>>>>>>>> for
>>>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>>>>>>>>> Neither
>>>>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>> the changes is a good fit for 2.x because they are
>>>>>>>>>>>> likely
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> introduce
>>>>>>>>>>>>>>>>>>>>>>> breaking changes in the persistence layer,
>>>>>>>>>>> configuration
>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>> behavior.
>>>>>>>>>>>>>>>>>>>>>>> Additionally, these components are now tightly
>>>>>>>>>>> coupled,
>>>>>>>>>>>> so
>>>>>>>>>>>>>>> there
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>> no
>>>>>>>>>>>>>>>>>>>>>>> way
>>>>>>>>>>>>>>>>>>>>>>> these two changes can be implemented in parallel and
>>>>>>>>>>>> then
>>>>>>>>>>>>>>> merged
>>>>>>>>>>>>>>>>>>>>>>> together
>>>>>>>>>>>>>>>>>>>>>>> easily. So what we will end up with is having to
>>>>>>>>>>>> implement
>>>>>>>>>>>>>>> these
>>>>>>>>>>>>>>>>>>>>>> changes
>>>>>>>>>>>>>>>>>>>>>>> sequentially, fixing all existing tests twice, and
>>>>>>>>>>>>>> essentially
>>>>>>>>>>>>>>>>>>>>>> throwing
>>>>>>>>>>>>>>>>>>>>>>> away half of the work done because the other part of
>>>>>>>>>>> the
>>>>>>>>>>>>>>> change
>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>> re-implemented
>>>>>>>>>>>>>>>>>>>>>>> - Similar example goes with getting rid of
>>>>>>>>>>>>>>> IgniteInternalFuture
>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>> replacing it with CompletableFuture, and any other
>>>>>>>>>>>> change
>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>> touches
>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> asynchronous part of the code.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Third, I do not see how this choice affects the UX
>>>> of
>>>>>>>>>>>>> Ignite.
>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>> end
>>>>>>>>>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>>>>>>>> experience must be fixed in the IEP regardless of
>>>> the
>>>>>>>>>>>>>>> development
>>>>>>>>>>>>>>>>>>>> process
>>>>>>>>>>>>>>>>>>>>>>> and the fact that we have gaps in this area in
>>>> Ignite
>>>>>>>>>>> 2.x
>>>>>>>>>>>>>> just
>>>>>>>>>>>>>>>>>> confirms
>>>>>>>>>>>>>>>>>>>>>>> that.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Pavel, agree that a repo/branch is a technicality, I
>>>>>>>>>>>> guess
>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>>>>> reformulate,
>>>>>>>>>>>>>>>>>>>>>>> my point is that we might agree to have a single
>>>>>>>>>>>>> development
>>>>>>>>>>>>>>>> master
>>>>>>>>>>>>>>>>>>>>>> branch
>>>>>>>>>>>>>>>>>>>>>>> with 'disassembled' end-to-end functionality for
>>>> some
>>>>>>>>>>>>> period
>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>> time
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>> speed up development, and re-assemble the core
>>>>> features
>>>>>>>>>>>>> after
>>>>>>>>>>>>>>>>> having
>>>>>>>>>>>>>>>>>>>>>>> submodules tested independently.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Nikolay,
>>>>>>>>>>>>>>>>>>>>>>>> We have many features that have to evolve.
>>>>>>>>>>>>>>>>>>>>>>>> Snapshots, rebalance, tooling, tracing, zookeeper
>>>>>>>>>>>> support,
>>>>>>>>>>>>>>> etc.
>>>>>>>>>>>>>>>>>>>>>>> This is not very specific. In the end, resources are
>>>>>>>>>>>>> limited
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>> not be able to drive both tracks simultaneously,
>>>>>>>>>>>> especially
>>>>>>>>>>>>>>>> after a
>>>>>>>>>>>>>>>>>>>>>> couple
>>>>>>>>>>>>>>>>>>>>>>> of features having been implemented for Ignite 3.0.
>>>> If
>>>>>>>>>>>>> there
>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>> indeed
>>>>>>>>>>>>>>>>>>>>>>> some major changes that we want to do in Ignite 2.x
>>>>>>>>>>>> instead
>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>> putting
>>>>>>>>>>>>>>>>>>>>>>> effort into 3.0 - let's discuss them. I am just not
>>>>>>>>>>> aware
>>>>>>>>>>>>> of
>>>>>>>>>>>>>>> any,
>>>>>>>>>>>>>>>>>>>> that's
>>>>>>>>>>>>>>>>>>>>>>> why I am eager to move forward with Ignite 3.0.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> We have bugs and issues that can be fixed in 2.x
>>>>>>>>>>> without
>>>>>>>>>>>>>>>> breaking
>>>>>>>>>>>>>>>>>>>>>> backward
>>>>>>>>>>>>>>>>>>>>>>> compatibility.
>>>>>>>>>>>>>>>>>>>>>>>> We have many users who are happy with the 2.x with
>>>>> all
>>>>>>>>>>>>> it’s
>>>>>>>>>>>>>>>>> issues.
>>>>>>>>>>>>>>>>>>>>>>> These changes will be covered by end-to-end tests
>>>> and
>>>>>>>>>>>>>> migrated
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> Ignite
>>>>>>>>>>>>>>>>>>>>>>> 3.0, so I see no issues here.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Finally, Anton & Nikolay
>>>>>>>>>>>>>>>>>>>>>>> I do not have an estimate for this simply because
>>>> the
>>>>>>>>>>>>>> activity
>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>> community-driven and it depends on the number of
>>>>> people
>>>>>>>>>>>>>> willing
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>> contribute. With the current pace, I would hope to
>>>>> have
>>>>>>>>>>>> an
>>>>>>>>>>>>> RC
>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>> Ignite
>>>>>>>>>>>>>>>>>>>>>> 3.0
>>>>>>>>>>>>>>>>>>>>>>> to be ready by the end of 2021. My gut feeling is
>>>> that
>>>>>>>>>>> by
>>>>>>>>>>>>>>> moving
>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>>>> incremental changes, we will not be able to
>>>> implement
>>>>>>>>>>>> even
>>>>>>>>>>>>>> half
>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> wishlist by that time.
>>>>>>>>>>>>>>>>>>>>>>> I doubt that releasing several major releases with
>>>>>>>>>>>> breaking
>>>>>>>>>>>>>>>> changes
>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>> make Ignite users happy either because each upgrade
>>>>>>>>>>> will
>>>>>>>>>>>>> cost
>>>>>>>>>>>>>>>>> Ignite
>>>>>>>>>>>>>>>>>>>>>> users
>>>>>>>>>>>>>>>>>>>>>>> money, so the fewer major versions we release, the
>>>>>>>>>>>> better.
>>>>>>>>>>>>>> Thus
>>>>>>>>>>>>>>>> my
>>>>>>>>>>>>>>>>>> wish
>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>> include all breaking changes in one release.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I'll be now quiet for a while, let's see what other
>>>>>>>>>>>>> community
>>>>>>>>>>>>>>>>> members
>>>>>>>>>>>>>>>>>>>>>>> think.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <
>>>>>>>>>>>>>>>> ptupitsyn@apache.org
>>>>>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> 1. Rewriting from scratch is never a good idea.
>>>>>>>>>>>>>>>>>>>>>>>> We don't want to follow the path of Netscape and
>>>> lose
>>>>>>>>>>>> all
>>>>>>>>>>>>>> our
>>>>>>>>>>>>>>>>> users
>>>>>>>>>>>>>>>>>>>>>>>> by the time we have a working 3.0 [1]
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> 2. Not sure about new repo - seems like some pain
>>>> and
>>>>>>>>>>> no
>>>>>>>>>>>>>> gain,
>>>>>>>>>>>>>>>>>> what's
>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> problem with a branch?
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> 3. We should keep existing integration tests when
>>>>>>>>>>>>> possible.
>>>>>>>>>>>>>>>>>>>>>>>> We have accumulated a lot of edge case knowledge
>>>> over
>>>>>>>>>>>> the
>>>>>>>>>>>>>>> years,
>>>>>>>>>>>>>>>>>>>>>>>> it is not a good idea to send all of that down the
>>>>>>>>>>>> drain.
>>>>>>>>>>>>>>>>>>>>>>>> Yes, integration tests are slow, but they are the
>>>>> most
>>>>>>>>>>>>>>> valuable.
>>>>>>>>>>>>>>>>>>>>>>>> I think we can move more stuff into nightly runs
>>>> and
>>>>>>>>>>>> have
>>>>>>>>>>>>> a
>>>>>>>>>>>>>>> fast
>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>> modern
>>>>>>>>>>>>>>>>>>>>>>>> basic suite.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Alexey, you are much more familiar with the Ignite
>>>>>>>>>>> core
>>>>>>>>>>>>>>> codebase
>>>>>>>>>>>>>>>>>> than
>>>>>>>>>>>>>>>>>>>>>>> most
>>>>>>>>>>>>>>>>>>>>>>>> of us,
>>>>>>>>>>>>>>>>>>>>>>>> can you please explain in more detail which
>>>>> particular
>>>>>>>>>>>>>>> feature,
>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>> your
>>>>>>>>>>>>>>>>>>>>>>>> opinion,
>>>>>>>>>>>>>>>>>>>>>>>> mandates this "start from scratch" approach?
>>>>>>>>>>>>>>>>>>>>>>>> Is it really not possible at all to follow a less
>>>>>>>>>>>> radical
>>>>>>>>>>>>>> way?
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>
>>>>>
>>>>
>> https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <
>>>>>>>>>>>>>>>>> nizhikov@apache.org
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Hello, Alexey.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> I think that «rewriting from scratch» approach
>>>> has a
>>>>>>>>>>>> high
>>>>>>>>>>>>>>> risk
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>> make
>>>>>>>>>>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>>>>>>>> features unusable.
>>>>>>>>>>>>>>>>>>>>>>>>> At the time Ignite2 was started no-one wants to do
>>>>>>>>>>> bad
>>>>>>>>>>>> UX
>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>> bad
>>>>>>>>>>>>>>>>>>>>>>>> features.
>>>>>>>>>>>>>>>>>>>>>>>>> Nevertheless, it happen.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> I think we can avoid it with the Ignite3 and
>>>>>>>>>>> successors
>>>>>>>>>>>>> if
>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>> move
>>>>>>>>>>>>>>>>>>>>>>>>> step by step without keeping backward
>>>> compatibility
>>>>>>>>>>>>>>>>>>>>>>>>> With the step by step approach, we can focus on
>>>> each
>>>>>>>>>>>>>>> component
>>>>>>>>>>>>>>>>>>>>>>>> separately.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> What new features are we planning to implement
>>>> for
>>>>>>>>>>>>> Ignite
>>>>>>>>>>>>>>> 2.x?
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> We have many features that have to evolve.
>>>>>>>>>>>>>>>>>>>>>>>>> Snapshots, rebalance, tooling, tracing, zookeeper
>>>>>>>>>>>>> support,
>>>>>>>>>>>>>>> etc.
>>>>>>>>>>>>>>>>>>>>>>>>> We have bugs and issues that can be fixed in 2.x
>>>>>>>>>>>> without
>>>>>>>>>>>>>>>> breaking
>>>>>>>>>>>>>>>>>>>>>>>> backward
>>>>>>>>>>>>>>>>>>>>>>>>> compatibility.
>>>>>>>>>>>>>>>>>>>>>>>>> We have many users who are happy with the 2.x with
>>>>>>>>>>> all
>>>>>>>>>>>>> it’s
>>>>>>>>>>>>>>>>> issues.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> 2 нояб. 2020 г., в 14:09, Anton Vinogradov <
>>>>>>>>>>>>> av@apache.org
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> написал(а):
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Alexey,
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Do we have any estimates of how fast we'll be
>>>> able
>>>>>>>>>>> to
>>>>>>>>>>>>> gain
>>>>>>>>>>>>>>>>>>>>>>>>> production-ready
>>>>>>>>>>>>>>>>>>>>>>>>>> AI 3.0 in case of a "new repo" choice?
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk <
>>>>>>>>>>>>>>>>>>>>>>>>> alexey.goncharuk@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Nikolay,
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> What new features are we planning to implement
>>>> for
>>>>>>>>>>>>> Ignite
>>>>>>>>>>>>>>>> 2.x?
>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>>> think
>>>>>>>>>>>>>>>>>>>>>>>>> once
>>>>>>>>>>>>>>>>>>>>>>>>>>> we commence working on Ignite 3.0, we should
>>>>>>>>>>>> gradually
>>>>>>>>>>>>>>> cease
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>> activity
>>>>>>>>>>>>>>>>>>>>>>>>>>> on Ignite 2.x to mere bugfixes because such
>>>>>>>>>>> parallel
>>>>>>>>>>>>>>>>> development
>>>>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>>> overwhelming regardless of how we choose to
>>>>>>>>>>> proceed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <
>>>>>>>>>>>>>>>>>> nizhikov@apache.org
>>>>>>>>>>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> To be clear:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would suggest creating a new repository for
>>>>>>>>>>>> Ignite
>>>>>>>>>>>>>> 3.0
>>>>>>>>>>>>>>>>>>>>>>> (perhaps, a
>>>>>>>>>>>>>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>>>>>>>>>>> clean branch, but a new repo looks nicer to me)
>>>>>>>>>>> and
>>>>>>>>>>>> a
>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>> Ignite
>>>>>>>>>>>>>>>>>>>>>>> 3.0
>>>>>>>>>>>>>>>>>>>>>>>>>>>> TeamCity project.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> +1 for new Team City project.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> +1 for new branch for Ignite3.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> -1 for new repo.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <
>>>>>>>>>>>>>>>>>>>>>> NIzhikov.dev@gmail.com
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> написал(а):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello, Alexey.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think it will hurt our project more than
>>>> help.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Developing new features for 2 separate
>>>> branches
>>>>>>>>>>>> with
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> different
>>>>>>>>>>>>>>>>>>>>>>>>> APIs
>>>>>>>>>>>>>>>>>>>>>>>>>>>> and internal structure is overwhelming
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Maybe we should relax a bit requirements for
>>>>>>>>>>>> Ignite3?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Maybe we should move step by step and make
>>>>>>>>>>> Ignite3
>>>>>>>>>>>>> with
>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>>>>>>>>>>> configuration than Ignite4 with new
>>>> transactions,
>>>>>>>>>>>> etc?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <
>>>>>>>>>>>>>>>>>>>>>>>>>>> alexey.goncharuk@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> написал(а):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Igniters,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I wanted to pitch a rather radical idea
>>>>>>>>>>> regarding
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> Ignite
>>>>>>>>>>>>>>>>>>>>>> 3.0
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> development which has occurred to me some
>>>> time
>>>>>>>>>>>> ago.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We already have several IEPs targeted to
>>>> Ignite
>>>>>>>>>>>> 3.0
>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>>> imply
>>>>>>>>>>>>>>>>>>>>>>>> major
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> changes to the codebase (the change in
>>>>>>>>>>> replication
>>>>>>>>>>>>>>>> protocol
>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>> thus
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> transactions, change in binary format,
>>>> updated
>>>>>>>>>>>>>>>> metastorage,
>>>>>>>>>>>>>>>>>>>>>> etc).
>>>>>>>>>>>>>>>>>>>>>>>> We
>>>>>>>>>>>>>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> planned significant changes in public APIs:
>>>>>>>>>>>>>>> configuration
>>>>>>>>>>>>>>>>>>>>>> format
>>>>>>>>>>>>>>>>>>>>>>>>>>> change,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> improvements in cache APIs, SQL APIs,
>>>>>>>>>>> transaction
>>>>>>>>>>>>> mode
>>>>>>>>>>>>>>>>> rework.
>>>>>>>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>>>>>>>>>> wishlist
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of changes for Ignite 3.0 is huge.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> So, I was wondering whether it makes sense to
>>>>>>>>>>> try
>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> change
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> old
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> codebase, or start with a new baseline and
>>>> move
>>>>>>>>>>>> old
>>>>>>>>>>>>>>> pieces
>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>> code
>>>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>> do
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not require significant rework. Personally, I
>>>>>>>>>>>> would
>>>>>>>>>>>>> go
>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>> second
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> option for the following reasons:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - We have a chance to shift the development
>>>>>>>>>>>> paradigm
>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> project
>>>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> introduce the practice of true unit-tests. In
>>>>>>>>>>> the
>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>> baseline
>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> beginning there will be no ability to run an
>>>>>>>>>>>>>> end-to-end
>>>>>>>>>>>>>>>>>>>>>> scenario,
>>>>>>>>>>>>>>>>>>>>>>>>>>> thus
>>>>>>>>>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be forced to write unit-tests. So far,
>>>>> such
>>>>>>>>>>>>>>> practice
>>>>>>>>>>>>>>>>> was
>>>>>>>>>>>>>>>>>>>>>>> hard
>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implement because of tight coupling between
>>>>>>>>>>> Ignite
>>>>>>>>>>>>>>>>> components
>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>> inability
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to instantiate components without an instance
>>>>> of
>>>>>>>>>>>>>>>>>> KernalContext.
>>>>>>>>>>>>>>>>>>>>>>> For
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> example, we should be able to thoroughly test
>>>>>>>>>>>>> internal
>>>>>>>>>>>>>>>>>>>>>>> primitives,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> such as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> replication protocol (without actual
>>>>>>>>>>>> communication),
>>>>>>>>>>>>>>>>>>>>>> distributed
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> metastorage contracts, persistence layer,
>>>> etc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - We will significantly reduce the
>>>> development
>>>>>>>>>>>> cycle
>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> beginning
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (right now the RunAll takes two hours of
>>>>>>>>>>>>> astronomical
>>>>>>>>>>>>>>> time
>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>>>>>>>>>>>> TC;
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the new approach developer will be able to
>>>>>>>>>>> run
>>>>>>>>>>>>> ALL
>>>>>>>>>>>>>>>> tests
>>>>>>>>>>>>>>>>>>>>>>> locally
>>>>>>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> matter of minutes)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - We can get rid of TC bot and enforce green
>>>> TC
>>>>>>>>>>> by
>>>>>>>>>>>>>>>>> integrating
>>>>>>>>>>>>>>>>>>>>>> TC
>>>>>>>>>>>>>>>>>>>>>>>>>>> build
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> results with GitHub PRs (the same way Travis
>>>> is
>>>>>>>>>>>>>>> currently
>>>>>>>>>>>>>>>>>>>>>>>> integrated
>>>>>>>>>>>>>>>>>>>>>>>>>>>> to PR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> check). We should restrict PR merge without a
>>>>> TC
>>>>>>>>>>>>> check
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - We will still have to re-write all tests,
>>>> but
>>>>>>>>>>>> only
>>>>>>>>>>>>>>> once.
>>>>>>>>>>>>>>>>> If
>>>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>>> try
>>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> modify the old codebase, we would need to
>>>>> modify
>>>>>>>>>>>> all
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> tests
>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>>> every
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> major change (public API change,
>>>> configuration
>>>>>>>>>>>>> change)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - We will have fewer conflicts when working
>>>>>>>>>>>>> together.
>>>>>>>>>>>>>>> For
>>>>>>>>>>>>>>>>>>>>>>> example,
>>>>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cannot imagine how one would merge two
>>>> changes
>>>>>>>>>>> of
>>>>>>>>>>>>>>> getting
>>>>>>>>>>>>>>>>> rid
>>>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> IgniteFuture and changes in replication
>>>>>>>>>>> protocol,
>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>> example
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Technically, I would suggest creating a new
>>>>>>>>>>>>> repository
>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>> Ignite
>>>>>>>>>>>>>>>>>>>>>>>> 3.0
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (perhaps, a new clean branch, but a new repo
>>>>>>>>>>> looks
>>>>>>>>>>>>>> nicer
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> me)
>>>>>>>>>>>>>>>>>>>>>>>> and a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ignite 3.0 TeamCity project.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> While it may seem quite radical, I do believe
>>>>>>>>>>> that
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>> approach
>>>>>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>>>>>>> give
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> us more benefits than trying to make such
>>>> major
>>>>>>>>>>>>>> changes
>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>> existing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> codebase. If needed, let's schedule a discord
>>>>>>>>>>> chat
>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>> before
>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> discuss
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> WDYT?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>>>>>>> Andrey V. Mashenkov
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>>
>>>>>>>>>> Best regards,
>>>>>>>>>> Alexei Scherbakov
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Sincerely yours, Ivan Daschinskiy
>>>>>
>>>>
>>
>>
>>
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Sergey Chugunov <se...@gmail.com>.
Igniters,
I agree that create or not create is not a question, rephrasing
Shakespeare.
My main point is that developing new features on top of old 2.x-style
architecture is a bad idea. We will write the code and spend some time
stabilizing it (which is expected and fine). But then, when we finally
decide to fix our architecture and pay our (already huge) technical debt,
we will have to rewrite this code again and spend time stabilizing it again.
Creating new components on top of 2.x (which is actually 1.x, nothing
fundamentally new was introduced in terms of architecture) is equal to
wasting time now and creating more worthless work for the future.
Earlier I suggested to rank all new features according to their criticality
and amount of breaking changes and shape 3.0 scope based on this analysis.
Let's get back to this idea and prepare a scope based on publicly shared
arguments.
One more thing I would add here. Our users are smart people and make
decisions about upgrading or not upgrading to a new version based on
cost/value balance. Incremental approach keeps cost (public API breaking
changes) high but brings questionable amounts of value with each iteration.
If we add more valuable features to 3.0 and force users to pay the cost
only once they will be happier than if we split really needed changes to
several major releases and send our users to hell of endless rewriting
their codebases. In the latter case we'll see users to be much more
reluctant to upgrade to newer versions.
Hope this makes sense.
On Mon, Nov 16, 2020 at 2:24 PM Nikolay Izhikov <ni...@apache.org> wrote:
> > Let's indeed focus on Sergey's suggestions on the design->development
> approach.
>
> +1
>
> > - API & configuration cleanup
> > - New management tool
> > - Schema-first approach
> > - New replication infrastructure
>
> +1.
>
> > 16 нояб. 2020 г., в 13:40, Alexey Goncharuk <al...@gmail.com>
> написал(а):
> >
> > Folks,
> >
> > I think we are overly driven away by the phrase 'new repo' rather than
> the
> > essence of my suggestion. We can keep developing in the same repo, we can
> > even keep developing in the master branch. My point is that Ignite 3.0
> is a
> > chance to move on with the architecture, so if we really want to make
> > architectural improvements, we should not strive for incremental changes
> > for *some parts of the code*.
> >
> > Maxim,
> >
> > To comment on your examples: I think that the huge effort that is
> currently
> > required to make any significant change in Ignite is the perfect example
> of
> > how we lack structure in the codebase. Yes, theoretically we can
> introduce
> > incremental changes in the code that will improve the structure, but my
> > question is: we did not do it before, what will enforce us to make these
> > changes now? With the current approach, adding a new feature increases
> the
> > test time non-linearly because without proper decoupling you have to test
> > all possible combinations of features together. We can move faster than
> > that.
> >
> > I also do not agree that we should reduce the scope of Ignite 3.0 that
> > much. I do not see how the schema-first approach can be properly and
> > reliably implemented without a reliable HA metastorage, which in turn
> > requires a reliable replication protocol to be implemented. Besides, if a
> > number of people want to work on some Ignite feature, why should they
> wait
> > because not all community members have time to review the changes?
> >
> > Let's indeed focus on Sergey's suggestions on the design->development
> > approach. I back both Nikolay's and Maxim's scope, but I think we should
> > unite them, not intersect, and the minimal list of changes to be included
> > to Ignite 3.0 is:
> >
> > - API & configuration cleanup
> > - New management tool
> > - Schema-first approach
> > - New replication infrastructure
> >
> > Any smaller subset of changes will leave Ignite 3.0 in a transient state
> > with people being too afraid to move to it because there are more major
> > breaking changes scheduled.
> >
> > пт, 13 нояб. 2020 г. в 18:28, Alexey Zinoviev <za...@gmail.com>:
> >
> >> I'm -1 for creating a new repo.
> >> Also I support Maxim's plan for 3.0
> >>
> >> пт, 13 нояб. 2020 г. в 15:50, Maxim Muzafarov <mm...@apache.org>:
> >>
> >>> Val,
> >>>
> >>>
> >>> Why *creating a new repo* is the main point we faced with? Would it be
> >>> better to discuss the components design approach and scope management
> >>> first suggested by Sergey Chugunov? I doubt that new repo will solve
> >>> move us forward.
> >>>
> >>> Currently, I'm -1 to create a new repo with the inputs above.
> >>>
> >>> In addition to Nikolay's answer I see the following drawbacks of
> >>> creating new repo:
> >>> - we have very few positive examples of finalizing really huge
> >>> improvements to *production-ready* states the others remains
> >>> incomplete (MVCC, Calcite, Zookeeper, Tracing, Thread per Partition,
> >>> etc)
> >>> - AFAIK, the Native Persistence took a very long period of
> >>> stabilization even after it has been developed (we must take it into
> >>> account for developing new features like IEP-61)
> >>> - feature development for a long period of time (like 3.0) without any
> >>> releases will lead to all these changes became obsolete at the moment
> >>> of release (AFAIK the 2.8 which released a year ago still has no big
> >>> deployments)
> >>> - human resources -- some of the Igniters may lose their interest for
> >>> 3.0 during development, some of them may switch to different projects,
> >>> etc.
> >>> - do we all estimating the scope of 3.0 correct? The 2.8 release took
> 1.5
> >>> years.
> >>>
> >>> Have I missed something?
> >>>
> >>>
> >>> I suggest the following plan:
> >>>
> >>> - initiate 3.0 development in the master branch (after 2.10 release
> >>> change version to 3.0-SNAPSHOT instead of 2.11-SNAPSHOT)
> >>> - cleanup and collapse all the current APIs (see To Be Removed List
> >>> For Discussion on Apache Ignite 3.0 Wishlist)
> >>> - reduce the scope for 3.0 even more. I suggest focusing on two
> >>> things: Calcite + Schema-first approach
> >>> - create feature branches for proposed IEPs (for 3.0 only)
> >>> - create the release road map (allocate e.g. IEP-61 to 4.0 etc.)
> >>>
> >>> On Fri, 13 Nov 2020 at 14:03, Ivan Daschinsky <iv...@gmail.com>
> >> wrote:
> >>>>
> >>>>>> b. Implement IEP-61 - Common Replication Infrastructure
> >>>> I suppose, that this is the main cause of the current discussion.
> >>>> I hardly believe that this activity can be done without at least
> >>> creating a
> >>>> completely new branch.
> >>>>
> >>>> пт, 13 нояб. 2020 г. в 11:12, Nikolay Izhikov <ni...@apache.org>:
> >>>>
> >>>>> My suggestion:
> >>>>>
> >>>>> 1. Reduce Ignite3 scope to the following:
> >>>>> a. Delete all deprecated API and support of obsolete internal
> >>>>> protocols.
> >>>>> b. Implement IEP-61 - Common Replication Infrastructure
> >>>>> c. Implement new Ignite management tool ignitectl as
> >> suggested
> >>>>> during Ignite3 discussion.
> >>>>>
> >>>>> 2. Implement and release following improvements like transactions,
> >>> Calcite
> >>>>> based SQL, etc in the ongoing releases - Ignite 4, 5, 6
> >>>>>
> >>>>> My concern against separate Ignite 3 repo is the following:
> >>>>>
> >>>>> 1. We spread community to the two very separated part - Ignite3
> >>> developers
> >>>>> and Ignite2 maintainers. believe it’s bad for our community.
> >>>>> That can lead to the situation when we don’t fix critical or
> >>>>> blocker issueds «because they will not exists in Ignite3»
> >>>>> That will lead to the solutions never reviewed or reviewed
> >>> poorly.
> >>>>>
> >>>>> 2. It seems for me that current scope of Ignite3 is too big to be
> >>>>> implemented in any reasonable time.
> >>>>>
> >>>>>
> >>>>>> 13 нояб. 2020 г., в 10:57, Nikolay Izhikov <NIzhikov.dev@gmail.com
> >>>
> >>>>> написал(а):
> >>>>>>
> >>>>>> Hello, Valentin.
> >>>>>>
> >>>>>>> Nikolay, Maxim, are you OK with this route?
> >>>>>>
> >>>>>> -1 to have another repo for Ignite3 development.
> >>>>>>
> >>>>>>> 13 нояб. 2020 г., в 03:04, Valentin Kulichenko <
> >>>>> valentin.kulichenko@gmail.com> написал(а):
> >>>>>>>
> >>>>>>> Folks,
> >>>>>>>
> >>>>>>> We already have multiple IEPs for Ignite 3.0, and as far as I
> >> know,
> >>>>> there are contributors that would like to work on them (or probably
> >>> already
> >>>>> started). That said, we should make a decision as soon as possible.
> >>>>>>>
> >>>>>>> At this point, it doesn't seem that there are any strong
> >> objections
> >>> to
> >>>>> the technical side of things. So I would suggest the following:
> >>>>>>>
> >>>>>>> 1. Proceed with Alexey's approach to the development process, as
> >> it
> >>>>> seems to be the best (in my opinion - the only) way to address all
> >> the
> >>>>> technical concerns and issues expressed in the thread. We'll start by
> >>>>> creating a new repo and a new TC project.
> >>>>>>> 2. Start a separate discussion around transparency. If there are
> >> any
> >>>>> changes we need to make to our contributor guidelines, I am happy to
> >>> talk
> >>>>> them through, but I don't think it's reasonable to delay feature
> >>>>> development because of this. In the short term, I will make sure that
> >>>>> everything that happens within the new repo is as open to the
> >>> community as
> >>>>> possible.
> >>>>>>>
> >>>>>>> Nikolay, Maxim, are you OK with this route?
> >>>>>>>
> >>>>>>> -Val
> >>>>>>>
> >>>>>>> On Tue, Nov 10, 2020 at 4:55 PM Valentin Kulichenko <
> >>>>> valentin.kulichenko@gmail.com> wrote:
> >>>>>>> Maxim,
> >>>>>>>
> >>>>>>> 2.x and 3.x will have to coexist for some time - I don't see how
> >> we
> >>> can
> >>>>> avoid this considering the set of proposed changes. That said, we
> >>>>> effectively will need to have two "masters" - one for each major
> >>> version.
> >>>>> Master for 3.x can technically be a branch in the existing repo, but
> >>> having
> >>>>> a separate repo seems cleaner, simply because it will not be a
> >>> "branch" in
> >>>>> the traditional sense.
> >>>>>>>
> >>>>>>> Note that the new repo will still be under the Apache org, with
> >> the
> >>>>> same set of committers, managed by the community, etc. All the
> >>> development
> >>>>> happening for 3.0 must follow the rules that we currently have (if
> >>>>> anything, it's an opportunity to improve those rules).
> >>>>>>>
> >>>>>>> As I said during the call on Friday, I strongly believe that if
> >>> there
> >>>>> is a transparency issue, it will exist regardless of the approach we
> >>> choose
> >>>>> for 3.0. If community members develop without IEPs or public
> >>> discussions,
> >>>>> this will happen for both 2.x and 3.x unless we address this
> >>> separately. I
> >>>>> don't see how this is related to Alexey's suggestion, which targets
> >>>>> *technical* issues with the product more than anything else. This a
> >>> way to
> >>>>> achieve better modularity, introduce better coverage with unit tests,
> >>>>> reduce conflicts during development, etc.
> >>>>>>>
> >>>>>>> Coming back to transparency, let's identify the issues and fix
> >>> them. It
> >>>>> probably makes sense to have a separate discussion on this topic.
> >>>>>>>
> >>>>>>> -Val
> >>>>>>>
> >>>>>>> On Tue, Nov 10, 2020 at 1:05 PM Maxim Muzafarov <
> >> mmuzaf@apache.org>
> >>>>> wrote:
> >>>>>>> Sergey,
> >>>>>>>
> >>>>>>>
> >>>>>>> Your summary makes sense to me.
> >>>>>>>
> >>>>>>> However, how we come up from *Development transparency* to
> >> *create a
> >>>>>>> separate public repository dedicated for 3.0*? For me *development
> >>>>>>> transparency* is about making changes in the master branch. These
> >>>>>>> changes will definitely be seen by all the Ignite developers.
> >>>>>>>
> >>>>>>> A dedicated public repository is technically public and visible
> >> for
> >>>>>>> everyone, but it allows development without IEPs, without public
> >>>>>>> discussion (since all the code changes are not related to the
> >> master
> >>>>>>> branch) it also allows a large number of assumptions and
> >> deviations
> >>>>>>> (like code-style violations). It also not about *development
> >>>>>>> transparency* since developers which are working on 3.0 is only a
> >>>>>>> subset of all Ignite developers which may continue working on 2.x.
> >>> For
> >>>>>>> me, this would be a huge step backwards.
> >>>>>>>
> >>>>>>> Ignite veterans should remember how long the branch stabilization
> >>> took
> >>>>>>> for the 2.x version with the PDS.
> >>>>>>>
> >>>>>>>
> >>>>>>> I think each breaking change should be passed through the master
> >>> branch.
> >>>>>>>
> >>>>>>> On Tue, 10 Nov 2020 at 22:18, Alexei Scherbakov
> >>>>>>> <al...@gmail.com> wrote:
> >>>>>>>>
> >>>>>>>> Makes sense to me.
> >>>>>>>>
> >>>>>>>> вт, 10 нояб. 2020 г. в 18:47, Sergey Chugunov <
> >>>>> sergey.chugunov@gmail.com>:
> >>>>>>>>
> >>>>>>>>> Igniters,
> >>>>>>>>>
> >>>>>>>>> I thought over Friday meeting ideas and concerns and summarized
> >>> them
> >>>>> in
> >>>>>>>>> these three points:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> 1. *Components design unification approach.* New proposed
> >>> components
> >>>>>>>>> will be developed by different contributors, but they need to
> >> be
> >>>>> unified
> >>>>>>>>> and should integrate with each other easily. To ensure that I
> >>>>> suggest
> >>>>>>>>> calling an architecture group that will create design
> >> guidelines
> >>>>> for all
> >>>>>>>>> components and high-level overview of overall architecture.
> >> How
> >>>>> code is
> >>>>>>>>> split into components, what are component boundaries, how
> >>> component
> >>>>>>>>> lifecycle works and what are its interfaces - all these and
> >>> other
> >>>>>>>>> questions
> >>>>>>>>> should be covered.
> >>>>>>>>>
> >>>>>>>>> 2. *Scope management.* Apache 3.0 should be implemented
> >> within a
> >>>>>>>>> reasonable time, so we need some procedure to decide whether a
> >>>>>>>>> particular
> >>>>>>>>> feature should be dropped from the scope of 3.0 and postponed
> >>> to 3.1
> >>>>>>>>> release. To do so I suggest to range all features by two
> >>> parameters:
> >>>>>>>>> criticality for 3.0 and amount of breaking changes. 3.0 scope
> >>> should
> >>>>>>>>> include features of high criticality AND features with a big
> >>> amount
> >>>>> of
> >>>>>>>>> breaking changes. All other features can be made optional.
> >>>>>>>>>
> >>>>>>>>> 3. *Development transparency.* Development of all components
> >>> should
> >>>>> be
> >>>>>>>>> made as transparent for everyone as possible. Any contributor
> >>>>> should be
> >>>>>>>>> able to look over any component at any stage of development.
> >> To
> >>>>> achieve
> >>>>>>>>> this I suggest to create a separate public repository
> >> dedicated
> >>> for
> >>>>> 3.0
> >>>>>>>>> development. It will make the code available for everyone but
> >>> when
> >>>>>>>>> development of 3.0 is done we won't loose any stars of our
> >>> current
> >>>>>>>>> repository as we merge dev repo into main one and drop dev.
> >>>>>>>>>
> >>>>>>>>> Do these ideas make sense to you? Are there any concerns not
> >>> covered
> >>>>> by
> >>>>>>>>> these suggestions?
> >>>>>>>>>
> >>>>>>>>> On Fri, Nov 6, 2020 at 7:36 PM Kseniya Romanova <
> >>>>> romanova.ks.spb@gmail.com
> >>>>>>>>>>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Here are the slides from Alexey Goncharuk. Let's think this
> >> over
> >>> and
> >>>>>>>>>> continue on Monday:
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>
> >>>
> >>
> https://go.gridgain.com/rs/491-TWR-806/images/Ignite_3_Plans_and_development_process.pdf
> >>>>>>>>>>
> >>>>>>>>>> чт, 5 нояб. 2020 г. в 11:13, Anton Vinogradov <av...@apache.org>:
> >>>>>>>>>>
> >>>>>>>>>>> Folks,
> >>>>>>>>>>>
> >>>>>>>>>>> Should we perform cleanup work before (r)evolutional changes?
> >>>>>>>>>>> My huge proposal is to get rid of things which we don't need
> >>> anyway
> >>>>>>>>>>> - local caches,
> >>>>>>>>>>> - strange tx modes,
> >>>>>>>>>>> - code overcomplexity because of RollingUpgrade feature never
> >>>>> attended
> >>>>>>>>> at
> >>>>>>>>>>> AI,
> >>>>>>>>>>> - etc,
> >>>>>>>>>>> before choosing the way.
> >>>>>>>>>>>
> >>>>>>>>>>> On Tue, Nov 3, 2020 at 3:31 PM Valentin Kulichenko <
> >>>>>>>>>>> valentin.kulichenko@gmail.com> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Ksenia, thanks for scheduling this on such short notice!
> >>>>>>>>>>>>
> >>>>>>>>>>>> As for the original topic, I do support Alexey's idea. We're
> >>> not
> >>>>>>>>> going
> >>>>>>>>>> to
> >>>>>>>>>>>> rewrite anything from scratch, as most of the components are
> >>> going
> >>>>> to
> >>>>>>>>>> be
> >>>>>>>>>>>> moved as-is or with minimal modifications. However, the
> >> changes
> >>>>> that
> >>>>>>>>>> are
> >>>>>>>>>>>> proposed imply serious rework of the core parts of the code,
> >>> which
> >>>>>>>>> are
> >>>>>>>>>>> not
> >>>>>>>>>>>> properly decoupled from each other and from other parts. This
> >>> makes
> >>>>>>>>> the
> >>>>>>>>>>>> incremental approach borderline impossible. Developing in a
> >> new
> >>>>> repo,
> >>>>>>>>>>>> however, addresses this concern. As a bonus, we can also
> >>> refactor
> >>>>> the
> >>>>>>>>>>> code,
> >>>>>>>>>>>> introduce better decoupling, get rid of kernel context, and
> >>> develop
> >>>>>>>>>> unit
> >>>>>>>>>>>> tests (finally!).
> >>>>>>>>>>>>
> >>>>>>>>>>>> Basically, this proposal only affects the *process*, not the
> >>> set of
> >>>>>>>>>>> changes
> >>>>>>>>>>>> we had discussed before. Ignite 3.0 is our unique chance to
> >>> make
> >>>>>>>>> things
> >>>>>>>>>>>> right.
> >>>>>>>>>>>>
> >>>>>>>>>>>> -Val
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Tue, Nov 3, 2020 at 3:06 AM Kseniya Romanova <
> >>>>>>>>>>> romanova.ks.spb@gmail.com
> >>>>>>>>>>>>>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Pavel, all the interesting points will be anyway published
> >>> here in
> >>>>>>>>>>>> English
> >>>>>>>>>>>>> (as the principal "if it's not on devlist it doesn't
> >>> happened" is
> >>>>>>>>>> still
> >>>>>>>>>>>>> relevant). This is just a quick call for a group of
> >>> developers.
> >>>>>>>>> Later
> >>>>>>>>>>> we
> >>>>>>>>>>>>> can do a separate presentation of idea and discussion in
> >>> English
> >>>>> as
> >>>>>>>>>> we
> >>>>>>>>>>>> did
> >>>>>>>>>>>>> for the Ignite 3.0 draft of changes.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> вт, 3 нояб. 2020 г. в 13:52, Pavel Tupitsyn <
> >>> ptupitsyn@apache.org
> >>>>>>>>>> :
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Kseniya,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks for scheduling this call.
> >>>>>>>>>>>>>> Do you think we can switch to English if non-Russian
> >> speaking
> >>>>>>>>>>> community
> >>>>>>>>>>>>>> members decide to join?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Tue, Nov 3, 2020 at 1:32 PM Kseniya Romanova <
> >>>>>>>>>>>>> romanova.ks.spb@gmail.com
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Let's do this community discussion open. Here's the link
> >> on
> >>>>>>>>> zoom
> >>>>>>>>>>> call
> >>>>>>>>>>>>> in
> >>>>>>>>>>>>>>> Russian for Friday 6 PM:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>> https://www.meetup.com/Moscow-Apache-Ignite-Meetup/events/274360378/
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> вт, 3 нояб. 2020 г. в 12:49, Nikolay Izhikov <
> >>>>>>>>>> nizhikov@apache.org
> >>>>>>>>>>>> :
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Time works for me.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> 3 нояб. 2020 г., в 12:40, Alexey Goncharuk <
> >>>>>>>>>>>>>> alexey.goncharuk@gmail.com
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> написал(а):
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Nikolay,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I am up for the call. I will try to explain my reasoning
> >>> in
> >>>>>>>>>>>> greater
> >>>>>>>>>>>>>>>> detail
> >>>>>>>>>>>>>>>>> and will be glad to hear the concerns. Will this Friday,
> >>>>>>>>> Nov
> >>>>>>>>>>> 6th,
> >>>>>>>>>>>>>> work?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> вт, 3 нояб. 2020 г. в 10:09, Nikolay Izhikov <
> >>>>>>>>>>>> nizhikov@apache.org
> >>>>>>>>>>>>>> :
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Igniters, should we have a call for this topic?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> 2 нояб. 2020 г., в 18:53, Pavel Tupitsyn <
> >>>>>>>>>>> ptupitsyn@apache.org
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> написал(а):
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> not intend to rewrite everything from scratch
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Every single test from Ignite 2.x should be moved to
> >>>>>>>>>> Ignite
> >>>>>>>>>>> 3
> >>>>>>>>>>>>>>>>>>>> regardless of how we choose to proceed.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Alexey, thank you for the explanation, this addresses
> >>> all
> >>>>>>>>>> of
> >>>>>>>>>>> my
> >>>>>>>>>>>>>>>> concerns.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On Mon, Nov 2, 2020 at 6:43 PM Andrey Mashenkov <
> >>>>>>>>>>>>>>>>>> andrey.mashenkov@gmail.com>
> >>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Hi, Igniters.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> * AFAIU, we need a new repo if we want to apply
> >>>>>>>>> different
> >>>>>>>>>>>>>>> restrictions
> >>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>> pull requests,
> >>>>>>>>>>>>>>>>>>>> otherwise I see no difference for myself.
> >>>>>>>>>>>>>>>>>>>> E.g. make static analysis (do we have?), compile,
> >>>>>>>>> styles,
> >>>>>>>>>>> and
> >>>>>>>>>>>>>>> javadoc
> >>>>>>>>>>>>>>>>>>>> checks mandatory.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> I think that relaxed requirements here will lead to
> >> bad
> >>>>>>>>>>>> product
> >>>>>>>>>>>>>>>> quality.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> * Agree with Pavel, we should 'keep' integrations
> >> tests
> >>>>>>>>>>>> somehow.
> >>>>>>>>>>>>>>>>>>>> During active development tests will be broken most
> >> of
> >>>>>>>>>> time,
> >>>>>>>>>>>> so,
> >>>>>>>>>>>>>>>>>>>> I'd port them e.g. suite-by-suite once we will have a
> >>>>>>>>>> stable
> >>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>> featured
> >>>>>>>>>>>>>>>>>>>> environment to run them and of course make test's
> >> code
> >>>>>>>>>> clear
> >>>>>>>>>>>> and
> >>>>>>>>>>>>>>> avoid
> >>>>>>>>>>>>>>>>>>>> bad/non-relevant ones.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> * I like bottom-up approach.
> >>>>>>>>>>>>>>>>>>>> With it we could make a better framework. I mean
> >> clear
> >>>>>>>>>>>> component
> >>>>>>>>>>>>>>>>>> lifecycle,
> >>>>>>>>>>>>>>>>>>>> component wiring mechanics, general methods to
> >> approach
> >>>>>>>>>> core
> >>>>>>>>>>>>>>>> components
> >>>>>>>>>>>>>>>>>>>> such as exchange/communication
> >>>>>>>>>>>>>>>>>>>> to avoid code mess like we have in ExchangeFuture
> >> with
> >>>>>>>>> all
> >>>>>>>>>>>> these
> >>>>>>>>>>>>>>>> custom
> >>>>>>>>>>>>>>>>>>>> callbacks for each component, interfaces like
> >>>>>>>>>>>>>>>>>>>> PartitionsExchangeAware,
> >> IgniteChangeGlobalStateSupport
> >>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>> a pack of
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>
> >> start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected
> >>>>>>>>>>>>>>>>>>>> and so on in various unexpected places.
> >>>>>>>>>>>>>>>>>>>> Hope, we will be able to port most of the good code
> >> to
> >>>>>>>>> the
> >>>>>>>>>>> new
> >>>>>>>>>>>>>>>> framework
> >>>>>>>>>>>>>>>>>>>> version.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk <
> >>>>>>>>>>>>>>>>>>>> alexey.goncharuk@gmail.com>
> >>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Nikolay, Pavel,
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Thanks for the feedback! First of all, I wanted to
> >>>>>>>>> stress
> >>>>>>>>>>>> that
> >>>>>>>>>>>>> I
> >>>>>>>>>>>>>> do
> >>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>>> intend to rewrite everything from scratch (I never
> >>> used
> >>>>>>>>>>> this
> >>>>>>>>>>>>>>> phrase).
> >>>>>>>>>>>>>>>>>>>> There
> >>>>>>>>>>>>>>>>>>>>> are significant parts of code that would be moved
> >> with
> >>>>>>>>>>>> minimal
> >>>>>>>>>>>>>>>>>>>>> modifications. Second, I never said that we will get
> >>>>>>>>> rid
> >>>>>>>>>> of
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>> old
> >>>>>>>>>>>>>>>>>> tests
> >>>>>>>>>>>>>>>>>>>>> codebase. Every single test from Ignite 2.x should
> >> be
> >>>>>>>>>> moved
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>>>> Ignite 3
> >>>>>>>>>>>>>>>>>>>>> regardless of how we choose to proceed.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> My point is that for some parts of the code a clean
> >>>>>>>>>>> bottom-up
> >>>>>>>>>>>>>>>>>>>>> implementation will be cheaper in many ways. Let me
> >>>>>>>>> give
> >>>>>>>>>>> you
> >>>>>>>>>>>> a
> >>>>>>>>>>>>>> few
> >>>>>>>>>>>>>>>>>>>> concrete
> >>>>>>>>>>>>>>>>>>>>> examples:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> - I think no one can object that we need a cleanly
> >>>>>>>>>>> separated
> >>>>>>>>>>>>>>>>>>>> persistence
> >>>>>>>>>>>>>>>>>>>>> layer for Ignite. There is a very raw draft IEP for
> >>>>>>>>> this
> >>>>>>>>>>>>>> already.
> >>>>>>>>>>>>>>> On
> >>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> other hand, I think we also can agree that we need a
> >>>>>>>>>>>>> split-brain
> >>>>>>>>>>>>>>>>>>>>> resistant
> >>>>>>>>>>>>>>>>>>>>> replication protocol for caches. There is also an
> >> IEP
> >>>>>>>>>> for
> >>>>>>>>>>>>> this.
> >>>>>>>>>>>>>>>>>>>> Neither
> >>>>>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>> the changes is a good fit for 2.x because they are
> >>>>>>>>>> likely
> >>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>> introduce
> >>>>>>>>>>>>>>>>>>>>> breaking changes in the persistence layer,
> >>>>>>>>> configuration
> >>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>> behavior.
> >>>>>>>>>>>>>>>>>>>>> Additionally, these components are now tightly
> >>>>>>>>> coupled,
> >>>>>>>>>> so
> >>>>>>>>>>>>> there
> >>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>> no
> >>>>>>>>>>>>>>>>>>>>> way
> >>>>>>>>>>>>>>>>>>>>> these two changes can be implemented in parallel and
> >>>>>>>>>> then
> >>>>>>>>>>>>> merged
> >>>>>>>>>>>>>>>>>>>>> together
> >>>>>>>>>>>>>>>>>>>>> easily. So what we will end up with is having to
> >>>>>>>>>> implement
> >>>>>>>>>>>>> these
> >>>>>>>>>>>>>>>>>>>> changes
> >>>>>>>>>>>>>>>>>>>>> sequentially, fixing all existing tests twice, and
> >>>>>>>>>>>> essentially
> >>>>>>>>>>>>>>>>>>>> throwing
> >>>>>>>>>>>>>>>>>>>>> away half of the work done because the other part of
> >>>>>>>>> the
> >>>>>>>>>>>>> change
> >>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>> re-implemented
> >>>>>>>>>>>>>>>>>>>>> - Similar example goes with getting rid of
> >>>>>>>>>>>>> IgniteInternalFuture
> >>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>> replacing it with CompletableFuture, and any other
> >>>>>>>>>> change
> >>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>> touches
> >>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> asynchronous part of the code.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Third, I do not see how this choice affects the UX
> >> of
> >>>>>>>>>>> Ignite.
> >>>>>>>>>>>>> The
> >>>>>>>>>>>>>>> end
> >>>>>>>>>>>>>>>>>>>> user
> >>>>>>>>>>>>>>>>>>>>> experience must be fixed in the IEP regardless of
> >> the
> >>>>>>>>>>>>> development
> >>>>>>>>>>>>>>>>>> process
> >>>>>>>>>>>>>>>>>>>>> and the fact that we have gaps in this area in
> >> Ignite
> >>>>>>>>> 2.x
> >>>>>>>>>>>> just
> >>>>>>>>>>>>>>>> confirms
> >>>>>>>>>>>>>>>>>>>>> that.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Pavel, agree that a repo/branch is a technicality, I
> >>>>>>>>>> guess
> >>>>>>>>>>> if
> >>>>>>>>>>>>>>>>>>>> reformulate,
> >>>>>>>>>>>>>>>>>>>>> my point is that we might agree to have a single
> >>>>>>>>>>> development
> >>>>>>>>>>>>>> master
> >>>>>>>>>>>>>>>>>>>> branch
> >>>>>>>>>>>>>>>>>>>>> with 'disassembled' end-to-end functionality for
> >> some
> >>>>>>>>>>> period
> >>>>>>>>>>>> of
> >>>>>>>>>>>>>>> time
> >>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>> speed up development, and re-assemble the core
> >>> features
> >>>>>>>>>>> after
> >>>>>>>>>>>>>>> having
> >>>>>>>>>>>>>>>>>>>>> submodules tested independently.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Nikolay,
> >>>>>>>>>>>>>>>>>>>>>> We have many features that have to evolve.
> >>>>>>>>>>>>>>>>>>>>>> Snapshots, rebalance, tooling, tracing, zookeeper
> >>>>>>>>>> support,
> >>>>>>>>>>>>> etc.
> >>>>>>>>>>>>>>>>>>>>> This is not very specific. In the end, resources are
> >>>>>>>>>>> limited
> >>>>>>>>>>>>> and
> >>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>>> not be able to drive both tracks simultaneously,
> >>>>>>>>>> especially
> >>>>>>>>>>>>>> after a
> >>>>>>>>>>>>>>>>>>>> couple
> >>>>>>>>>>>>>>>>>>>>> of features having been implemented for Ignite 3.0.
> >> If
> >>>>>>>>>>> there
> >>>>>>>>>>>>> are
> >>>>>>>>>>>>>>>> indeed
> >>>>>>>>>>>>>>>>>>>>> some major changes that we want to do in Ignite 2.x
> >>>>>>>>>> instead
> >>>>>>>>>>>> of
> >>>>>>>>>>>>>>>> putting
> >>>>>>>>>>>>>>>>>>>>> effort into 3.0 - let's discuss them. I am just not
> >>>>>>>>> aware
> >>>>>>>>>>> of
> >>>>>>>>>>>>> any,
> >>>>>>>>>>>>>>>>>> that's
> >>>>>>>>>>>>>>>>>>>>> why I am eager to move forward with Ignite 3.0.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> We have bugs and issues that can be fixed in 2.x
> >>>>>>>>> without
> >>>>>>>>>>>>>> breaking
> >>>>>>>>>>>>>>>>>>>> backward
> >>>>>>>>>>>>>>>>>>>>> compatibility.
> >>>>>>>>>>>>>>>>>>>>>> We have many users who are happy with the 2.x with
> >>> all
> >>>>>>>>>>> it’s
> >>>>>>>>>>>>>>> issues.
> >>>>>>>>>>>>>>>>>>>>> These changes will be covered by end-to-end tests
> >> and
> >>>>>>>>>>>> migrated
> >>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>> Ignite
> >>>>>>>>>>>>>>>>>>>>> 3.0, so I see no issues here.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Finally, Anton & Nikolay
> >>>>>>>>>>>>>>>>>>>>> I do not have an estimate for this simply because
> >> the
> >>>>>>>>>>>> activity
> >>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>> community-driven and it depends on the number of
> >>> people
> >>>>>>>>>>>> willing
> >>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>> contribute. With the current pace, I would hope to
> >>> have
> >>>>>>>>>> an
> >>>>>>>>>>> RC
> >>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>> Ignite
> >>>>>>>>>>>>>>>>>>>> 3.0
> >>>>>>>>>>>>>>>>>>>>> to be ready by the end of 2021. My gut feeling is
> >> that
> >>>>>>>>> by
> >>>>>>>>>>>>> moving
> >>>>>>>>>>>>>>> with
> >>>>>>>>>>>>>>>>>>>>> incremental changes, we will not be able to
> >> implement
> >>>>>>>>>> even
> >>>>>>>>>>>> half
> >>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> wishlist by that time.
> >>>>>>>>>>>>>>>>>>>>> I doubt that releasing several major releases with
> >>>>>>>>>> breaking
> >>>>>>>>>>>>>> changes
> >>>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>>> make Ignite users happy either because each upgrade
> >>>>>>>>> will
> >>>>>>>>>>> cost
> >>>>>>>>>>>>>>> Ignite
> >>>>>>>>>>>>>>>>>>>> users
> >>>>>>>>>>>>>>>>>>>>> money, so the fewer major versions we release, the
> >>>>>>>>>> better.
> >>>>>>>>>>>> Thus
> >>>>>>>>>>>>>> my
> >>>>>>>>>>>>>>>> wish
> >>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>> include all breaking changes in one release.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> I'll be now quiet for a while, let's see what other
> >>>>>>>>>>> community
> >>>>>>>>>>>>>>> members
> >>>>>>>>>>>>>>>>>>>>> think.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <
> >>>>>>>>>>>>>> ptupitsyn@apache.org
> >>>>>>>>>>>>>>>> :
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> 1. Rewriting from scratch is never a good idea.
> >>>>>>>>>>>>>>>>>>>>>> We don't want to follow the path of Netscape and
> >> lose
> >>>>>>>>>> all
> >>>>>>>>>>>> our
> >>>>>>>>>>>>>>> users
> >>>>>>>>>>>>>>>>>>>>>> by the time we have a working 3.0 [1]
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> 2. Not sure about new repo - seems like some pain
> >> and
> >>>>>>>>> no
> >>>>>>>>>>>> gain,
> >>>>>>>>>>>>>>>> what's
> >>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>> problem with a branch?
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> 3. We should keep existing integration tests when
> >>>>>>>>>>> possible.
> >>>>>>>>>>>>>>>>>>>>>> We have accumulated a lot of edge case knowledge
> >> over
> >>>>>>>>>> the
> >>>>>>>>>>>>> years,
> >>>>>>>>>>>>>>>>>>>>>> it is not a good idea to send all of that down the
> >>>>>>>>>> drain.
> >>>>>>>>>>>>>>>>>>>>>> Yes, integration tests are slow, but they are the
> >>> most
> >>>>>>>>>>>>> valuable.
> >>>>>>>>>>>>>>>>>>>>>> I think we can move more stuff into nightly runs
> >> and
> >>>>>>>>>> have
> >>>>>>>>>>> a
> >>>>>>>>>>>>> fast
> >>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>> modern
> >>>>>>>>>>>>>>>>>>>>>> basic suite.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Alexey, you are much more familiar with the Ignite
> >>>>>>>>> core
> >>>>>>>>>>>>> codebase
> >>>>>>>>>>>>>>>> than
> >>>>>>>>>>>>>>>>>>>>> most
> >>>>>>>>>>>>>>>>>>>>>> of us,
> >>>>>>>>>>>>>>>>>>>>>> can you please explain in more detail which
> >>> particular
> >>>>>>>>>>>>> feature,
> >>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>> your
> >>>>>>>>>>>>>>>>>>>>>> opinion,
> >>>>>>>>>>>>>>>>>>>>>> mandates this "start from scratch" approach?
> >>>>>>>>>>>>>>>>>>>>>> Is it really not possible at all to follow a less
> >>>>>>>>>> radical
> >>>>>>>>>>>> way?
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> [1]
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>
> >>>
> >>
> https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <
> >>>>>>>>>>>>>>> nizhikov@apache.org
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Hello, Alexey.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> I think that «rewriting from scratch» approach
> >> has a
> >>>>>>>>>> high
> >>>>>>>>>>>>> risk
> >>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>> make
> >>>>>>>>>>>>>>>>>>>>>> new
> >>>>>>>>>>>>>>>>>>>>>>> features unusable.
> >>>>>>>>>>>>>>>>>>>>>>> At the time Ignite2 was started no-one wants to do
> >>>>>>>>> bad
> >>>>>>>>>> UX
> >>>>>>>>>>>> or
> >>>>>>>>>>>>>> bad
> >>>>>>>>>>>>>>>>>>>>>> features.
> >>>>>>>>>>>>>>>>>>>>>>> Nevertheless, it happen.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> I think we can avoid it with the Ignite3 and
> >>>>>>>>> successors
> >>>>>>>>>>> if
> >>>>>>>>>>>> we
> >>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>> move
> >>>>>>>>>>>>>>>>>>>>>>> step by step without keeping backward
> >> compatibility
> >>>>>>>>>>>>>>>>>>>>>>> With the step by step approach, we can focus on
> >> each
> >>>>>>>>>>>>> component
> >>>>>>>>>>>>>>>>>>>>>> separately.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> What new features are we planning to implement
> >> for
> >>>>>>>>>>> Ignite
> >>>>>>>>>>>>> 2.x?
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> We have many features that have to evolve.
> >>>>>>>>>>>>>>>>>>>>>>> Snapshots, rebalance, tooling, tracing, zookeeper
> >>>>>>>>>>> support,
> >>>>>>>>>>>>> etc.
> >>>>>>>>>>>>>>>>>>>>>>> We have bugs and issues that can be fixed in 2.x
> >>>>>>>>>> without
> >>>>>>>>>>>>>> breaking
> >>>>>>>>>>>>>>>>>>>>>> backward
> >>>>>>>>>>>>>>>>>>>>>>> compatibility.
> >>>>>>>>>>>>>>>>>>>>>>> We have many users who are happy with the 2.x with
> >>>>>>>>> all
> >>>>>>>>>>> it’s
> >>>>>>>>>>>>>>> issues.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> 2 нояб. 2020 г., в 14:09, Anton Vinogradov <
> >>>>>>>>>>> av@apache.org
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> написал(а):
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Alexey,
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Do we have any estimates of how fast we'll be
> >> able
> >>>>>>>>> to
> >>>>>>>>>>> gain
> >>>>>>>>>>>>>>>>>>>>>>> production-ready
> >>>>>>>>>>>>>>>>>>>>>>>> AI 3.0 in case of a "new repo" choice?
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk <
> >>>>>>>>>>>>>>>>>>>>>>> alexey.goncharuk@gmail.com>
> >>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Nikolay,
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> What new features are we planning to implement
> >> for
> >>>>>>>>>>> Ignite
> >>>>>>>>>>>>>> 2.x?
> >>>>>>>>>>>>>>> I
> >>>>>>>>>>>>>>>>>>>>> think
> >>>>>>>>>>>>>>>>>>>>>>> once
> >>>>>>>>>>>>>>>>>>>>>>>>> we commence working on Ignite 3.0, we should
> >>>>>>>>>> gradually
> >>>>>>>>>>>>> cease
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> activity
> >>>>>>>>>>>>>>>>>>>>>>>>> on Ignite 2.x to mere bugfixes because such
> >>>>>>>>> parallel
> >>>>>>>>>>>>>>> development
> >>>>>>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>> overwhelming regardless of how we choose to
> >>>>>>>>> proceed.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <
> >>>>>>>>>>>>>>>> nizhikov@apache.org
> >>>>>>>>>>>>>>>>>>>>> :
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> To be clear:
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> I would suggest creating a new repository for
> >>>>>>>>>> Ignite
> >>>>>>>>>>>> 3.0
> >>>>>>>>>>>>>>>>>>>>> (perhaps, a
> >>>>>>>>>>>>>>>>>>>>>>>>> new
> >>>>>>>>>>>>>>>>>>>>>>>>>> clean branch, but a new repo looks nicer to me)
> >>>>>>>>> and
> >>>>>>>>>> a
> >>>>>>>>>>>> new
> >>>>>>>>>>>>>>> Ignite
> >>>>>>>>>>>>>>>>>>>>> 3.0
> >>>>>>>>>>>>>>>>>>>>>>>>>> TeamCity project.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> +1 for new Team City project.
> >>>>>>>>>>>>>>>>>>>>>>>>>> +1 for new branch for Ignite3.
> >>>>>>>>>>>>>>>>>>>>>>>>>> -1 for new repo.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <
> >>>>>>>>>>>>>>>>>>>> NIzhikov.dev@gmail.com
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> написал(а):
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Hello, Alexey.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> I think it will hurt our project more than
> >> help.
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Developing new features for 2 separate
> >> branches
> >>>>>>>>>> with
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>> different
> >>>>>>>>>>>>>>>>>>>>>>> APIs
> >>>>>>>>>>>>>>>>>>>>>>>>>> and internal structure is overwhelming
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Maybe we should relax a bit requirements for
> >>>>>>>>>> Ignite3?
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Maybe we should move step by step and make
> >>>>>>>>> Ignite3
> >>>>>>>>>>> with
> >>>>>>>>>>>>> new
> >>>>>>>>>>>>>>>>>>>>>>>>>> configuration than Ignite4 with new
> >> transactions,
> >>>>>>>>>> etc?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <
> >>>>>>>>>>>>>>>>>>>>>>>>> alexey.goncharuk@gmail.com>
> >>>>>>>>>>>>>>>>>>>>>>>>>> написал(а):
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Igniters,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I wanted to pitch a rather radical idea
> >>>>>>>>> regarding
> >>>>>>>>>>> the
> >>>>>>>>>>>>>> Ignite
> >>>>>>>>>>>>>>>>>>>> 3.0
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> development which has occurred to me some
> >> time
> >>>>>>>>>> ago.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> We already have several IEPs targeted to
> >> Ignite
> >>>>>>>>>> 3.0
> >>>>>>>>>>>>> which
> >>>>>>>>>>>>>>>> imply
> >>>>>>>>>>>>>>>>>>>>>> major
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> changes to the codebase (the change in
> >>>>>>>>> replication
> >>>>>>>>>>>>>> protocol
> >>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>> thus
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> transactions, change in binary format,
> >> updated
> >>>>>>>>>>>>>> metastorage,
> >>>>>>>>>>>>>>>>>>>> etc).
> >>>>>>>>>>>>>>>>>>>>>> We
> >>>>>>>>>>>>>>>>>>>>>>>>>> also
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> planned significant changes in public APIs:
> >>>>>>>>>>>>> configuration
> >>>>>>>>>>>>>>>>>>>> format
> >>>>>>>>>>>>>>>>>>>>>>>>> change,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> improvements in cache APIs, SQL APIs,
> >>>>>>>>> transaction
> >>>>>>>>>>> mode
> >>>>>>>>>>>>>>> rework.
> >>>>>>>>>>>>>>>>>>>>> The
> >>>>>>>>>>>>>>>>>>>>>>>>>> wishlist
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> of changes for Ignite 3.0 is huge.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> So, I was wondering whether it makes sense to
> >>>>>>>>> try
> >>>>>>>>>> to
> >>>>>>>>>>>>>> change
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> old
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> codebase, or start with a new baseline and
> >> move
> >>>>>>>>>> old
> >>>>>>>>>>>>> pieces
> >>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>> code
> >>>>>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>>> do
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> not require significant rework. Personally, I
> >>>>>>>>>> would
> >>>>>>>>>>> go
> >>>>>>>>>>>>>> with
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> second
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> option for the following reasons:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> - We have a chance to shift the development
> >>>>>>>>>> paradigm
> >>>>>>>>>>>> in
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> project
> >>>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> introduce the practice of true unit-tests. In
> >>>>>>>>> the
> >>>>>>>>>>> new
> >>>>>>>>>>>>>>> baseline
> >>>>>>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> beginning there will be no ability to run an
> >>>>>>>>>>>> end-to-end
> >>>>>>>>>>>>>>>>>>>> scenario,
> >>>>>>>>>>>>>>>>>>>>>>>>> thus
> >>>>>>>>>>>>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> will be forced to write unit-tests. So far,
> >>> such
> >>>>>>>>>>>>> practice
> >>>>>>>>>>>>>>> was
> >>>>>>>>>>>>>>>>>>>>> hard
> >>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> implement because of tight coupling between
> >>>>>>>>> Ignite
> >>>>>>>>>>>>>>> components
> >>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>> inability
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> to instantiate components without an instance
> >>> of
> >>>>>>>>>>>>>>>> KernalContext.
> >>>>>>>>>>>>>>>>>>>>> For
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> example, we should be able to thoroughly test
> >>>>>>>>>>> internal
> >>>>>>>>>>>>>>>>>>>>> primitives,
> >>>>>>>>>>>>>>>>>>>>>>>>>> such as
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> replication protocol (without actual
> >>>>>>>>>> communication),
> >>>>>>>>>>>>>>>>>>>> distributed
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> metastorage contracts, persistence layer,
> >> etc.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> - We will significantly reduce the
> >> development
> >>>>>>>>>> cycle
> >>>>>>>>>>>> in
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>> beginning
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> (right now the RunAll takes two hours of
> >>>>>>>>>>> astronomical
> >>>>>>>>>>>>> time
> >>>>>>>>>>>>>>>> with
> >>>>>>>>>>>>>>>>>>>>>> empty
> >>>>>>>>>>>>>>>>>>>>>>>>>> TC;
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> in the new approach developer will be able to
> >>>>>>>>> run
> >>>>>>>>>>> ALL
> >>>>>>>>>>>>>> tests
> >>>>>>>>>>>>>>>>>>>>> locally
> >>>>>>>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> matter of minutes)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> - We can get rid of TC bot and enforce green
> >> TC
> >>>>>>>>> by
> >>>>>>>>>>>>>>> integrating
> >>>>>>>>>>>>>>>>>>>> TC
> >>>>>>>>>>>>>>>>>>>>>>>>> build
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> results with GitHub PRs (the same way Travis
> >> is
> >>>>>>>>>>>>> currently
> >>>>>>>>>>>>>>>>>>>>>> integrated
> >>>>>>>>>>>>>>>>>>>>>>>>>> to PR
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> check). We should restrict PR merge without a
> >>> TC
> >>>>>>>>>>> check
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> - We will still have to re-write all tests,
> >> but
> >>>>>>>>>> only
> >>>>>>>>>>>>> once.
> >>>>>>>>>>>>>>> If
> >>>>>>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>>>> try
> >>>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> modify the old codebase, we would need to
> >>> modify
> >>>>>>>>>> all
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>> tests
> >>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>>> every
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> major change (public API change,
> >> configuration
> >>>>>>>>>>> change)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> - We will have fewer conflicts when working
> >>>>>>>>>>> together.
> >>>>>>>>>>>>> For
> >>>>>>>>>>>>>>>>>>>>> example,
> >>>>>>>>>>>>>>>>>>>>>> I
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> cannot imagine how one would merge two
> >> changes
> >>>>>>>>> of
> >>>>>>>>>>>>> getting
> >>>>>>>>>>>>>>> rid
> >>>>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> IgniteFuture and changes in replication
> >>>>>>>>> protocol,
> >>>>>>>>>>> for
> >>>>>>>>>>>>>>> example
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Technically, I would suggest creating a new
> >>>>>>>>>>> repository
> >>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>> Ignite
> >>>>>>>>>>>>>>>>>>>>>> 3.0
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> (perhaps, a new clean branch, but a new repo
> >>>>>>>>> looks
> >>>>>>>>>>>> nicer
> >>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>> me)
> >>>>>>>>>>>>>>>>>>>>>> and a
> >>>>>>>>>>>>>>>>>>>>>>>>>> new
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Ignite 3.0 TeamCity project.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> While it may seem quite radical, I do believe
> >>>>>>>>> that
> >>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>> approach
> >>>>>>>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>>>>>>>> give
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> us more benefits than trying to make such
> >> major
> >>>>>>>>>>>> changes
> >>>>>>>>>>>>> in
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>> existing
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> codebase. If needed, let's schedule a discord
> >>>>>>>>> chat
> >>>>>>>>>>>> like
> >>>>>>>>>>>>>>> before
> >>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>> discuss
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> this.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> WDYT?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>>>>>>> Andrey V. Mashenkov
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>>
> >>>>>>>> Best regards,
> >>>>>>>> Alexei Scherbakov
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>> --
> >>>> Sincerely yours, Ivan Daschinskiy
> >>>
> >>
>
>
>
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Nikolay Izhikov <ni...@apache.org>.
> Let's indeed focus on Sergey's suggestions on the design->development approach.
+1
> - API & configuration cleanup
> - New management tool
> - Schema-first approach
> - New replication infrastructure
+1.
> 16 нояб. 2020 г., в 13:40, Alexey Goncharuk <al...@gmail.com> написал(а):
>
> Folks,
>
> I think we are overly driven away by the phrase 'new repo' rather than the
> essence of my suggestion. We can keep developing in the same repo, we can
> even keep developing in the master branch. My point is that Ignite 3.0 is a
> chance to move on with the architecture, so if we really want to make
> architectural improvements, we should not strive for incremental changes
> for *some parts of the code*.
>
> Maxim,
>
> To comment on your examples: I think that the huge effort that is currently
> required to make any significant change in Ignite is the perfect example of
> how we lack structure in the codebase. Yes, theoretically we can introduce
> incremental changes in the code that will improve the structure, but my
> question is: we did not do it before, what will enforce us to make these
> changes now? With the current approach, adding a new feature increases the
> test time non-linearly because without proper decoupling you have to test
> all possible combinations of features together. We can move faster than
> that.
>
> I also do not agree that we should reduce the scope of Ignite 3.0 that
> much. I do not see how the schema-first approach can be properly and
> reliably implemented without a reliable HA metastorage, which in turn
> requires a reliable replication protocol to be implemented. Besides, if a
> number of people want to work on some Ignite feature, why should they wait
> because not all community members have time to review the changes?
>
> Let's indeed focus on Sergey's suggestions on the design->development
> approach. I back both Nikolay's and Maxim's scope, but I think we should
> unite them, not intersect, and the minimal list of changes to be included
> to Ignite 3.0 is:
>
> - API & configuration cleanup
> - New management tool
> - Schema-first approach
> - New replication infrastructure
>
> Any smaller subset of changes will leave Ignite 3.0 in a transient state
> with people being too afraid to move to it because there are more major
> breaking changes scheduled.
>
> пт, 13 нояб. 2020 г. в 18:28, Alexey Zinoviev <za...@gmail.com>:
>
>> I'm -1 for creating a new repo.
>> Also I support Maxim's plan for 3.0
>>
>> пт, 13 нояб. 2020 г. в 15:50, Maxim Muzafarov <mm...@apache.org>:
>>
>>> Val,
>>>
>>>
>>> Why *creating a new repo* is the main point we faced with? Would it be
>>> better to discuss the components design approach and scope management
>>> first suggested by Sergey Chugunov? I doubt that new repo will solve
>>> move us forward.
>>>
>>> Currently, I'm -1 to create a new repo with the inputs above.
>>>
>>> In addition to Nikolay's answer I see the following drawbacks of
>>> creating new repo:
>>> - we have very few positive examples of finalizing really huge
>>> improvements to *production-ready* states the others remains
>>> incomplete (MVCC, Calcite, Zookeeper, Tracing, Thread per Partition,
>>> etc)
>>> - AFAIK, the Native Persistence took a very long period of
>>> stabilization even after it has been developed (we must take it into
>>> account for developing new features like IEP-61)
>>> - feature development for a long period of time (like 3.0) without any
>>> releases will lead to all these changes became obsolete at the moment
>>> of release (AFAIK the 2.8 which released a year ago still has no big
>>> deployments)
>>> - human resources -- some of the Igniters may lose their interest for
>>> 3.0 during development, some of them may switch to different projects,
>>> etc.
>>> - do we all estimating the scope of 3.0 correct? The 2.8 release took 1.5
>>> years.
>>>
>>> Have I missed something?
>>>
>>>
>>> I suggest the following plan:
>>>
>>> - initiate 3.0 development in the master branch (after 2.10 release
>>> change version to 3.0-SNAPSHOT instead of 2.11-SNAPSHOT)
>>> - cleanup and collapse all the current APIs (see To Be Removed List
>>> For Discussion on Apache Ignite 3.0 Wishlist)
>>> - reduce the scope for 3.0 even more. I suggest focusing on two
>>> things: Calcite + Schema-first approach
>>> - create feature branches for proposed IEPs (for 3.0 only)
>>> - create the release road map (allocate e.g. IEP-61 to 4.0 etc.)
>>>
>>> On Fri, 13 Nov 2020 at 14:03, Ivan Daschinsky <iv...@gmail.com>
>> wrote:
>>>>
>>>>>> b. Implement IEP-61 - Common Replication Infrastructure
>>>> I suppose, that this is the main cause of the current discussion.
>>>> I hardly believe that this activity can be done without at least
>>> creating a
>>>> completely new branch.
>>>>
>>>> пт, 13 нояб. 2020 г. в 11:12, Nikolay Izhikov <ni...@apache.org>:
>>>>
>>>>> My suggestion:
>>>>>
>>>>> 1. Reduce Ignite3 scope to the following:
>>>>> a. Delete all deprecated API and support of obsolete internal
>>>>> protocols.
>>>>> b. Implement IEP-61 - Common Replication Infrastructure
>>>>> c. Implement new Ignite management tool ignitectl as
>> suggested
>>>>> during Ignite3 discussion.
>>>>>
>>>>> 2. Implement and release following improvements like transactions,
>>> Calcite
>>>>> based SQL, etc in the ongoing releases - Ignite 4, 5, 6
>>>>>
>>>>> My concern against separate Ignite 3 repo is the following:
>>>>>
>>>>> 1. We spread community to the two very separated part - Ignite3
>>> developers
>>>>> and Ignite2 maintainers. believe it’s bad for our community.
>>>>> That can lead to the situation when we don’t fix critical or
>>>>> blocker issueds «because they will not exists in Ignite3»
>>>>> That will lead to the solutions never reviewed or reviewed
>>> poorly.
>>>>>
>>>>> 2. It seems for me that current scope of Ignite3 is too big to be
>>>>> implemented in any reasonable time.
>>>>>
>>>>>
>>>>>> 13 нояб. 2020 г., в 10:57, Nikolay Izhikov <NIzhikov.dev@gmail.com
>>>
>>>>> написал(а):
>>>>>>
>>>>>> Hello, Valentin.
>>>>>>
>>>>>>> Nikolay, Maxim, are you OK with this route?
>>>>>>
>>>>>> -1 to have another repo for Ignite3 development.
>>>>>>
>>>>>>> 13 нояб. 2020 г., в 03:04, Valentin Kulichenko <
>>>>> valentin.kulichenko@gmail.com> написал(а):
>>>>>>>
>>>>>>> Folks,
>>>>>>>
>>>>>>> We already have multiple IEPs for Ignite 3.0, and as far as I
>> know,
>>>>> there are contributors that would like to work on them (or probably
>>> already
>>>>> started). That said, we should make a decision as soon as possible.
>>>>>>>
>>>>>>> At this point, it doesn't seem that there are any strong
>> objections
>>> to
>>>>> the technical side of things. So I would suggest the following:
>>>>>>>
>>>>>>> 1. Proceed with Alexey's approach to the development process, as
>> it
>>>>> seems to be the best (in my opinion - the only) way to address all
>> the
>>>>> technical concerns and issues expressed in the thread. We'll start by
>>>>> creating a new repo and a new TC project.
>>>>>>> 2. Start a separate discussion around transparency. If there are
>> any
>>>>> changes we need to make to our contributor guidelines, I am happy to
>>> talk
>>>>> them through, but I don't think it's reasonable to delay feature
>>>>> development because of this. In the short term, I will make sure that
>>>>> everything that happens within the new repo is as open to the
>>> community as
>>>>> possible.
>>>>>>>
>>>>>>> Nikolay, Maxim, are you OK with this route?
>>>>>>>
>>>>>>> -Val
>>>>>>>
>>>>>>> On Tue, Nov 10, 2020 at 4:55 PM Valentin Kulichenko <
>>>>> valentin.kulichenko@gmail.com> wrote:
>>>>>>> Maxim,
>>>>>>>
>>>>>>> 2.x and 3.x will have to coexist for some time - I don't see how
>> we
>>> can
>>>>> avoid this considering the set of proposed changes. That said, we
>>>>> effectively will need to have two "masters" - one for each major
>>> version.
>>>>> Master for 3.x can technically be a branch in the existing repo, but
>>> having
>>>>> a separate repo seems cleaner, simply because it will not be a
>>> "branch" in
>>>>> the traditional sense.
>>>>>>>
>>>>>>> Note that the new repo will still be under the Apache org, with
>> the
>>>>> same set of committers, managed by the community, etc. All the
>>> development
>>>>> happening for 3.0 must follow the rules that we currently have (if
>>>>> anything, it's an opportunity to improve those rules).
>>>>>>>
>>>>>>> As I said during the call on Friday, I strongly believe that if
>>> there
>>>>> is a transparency issue, it will exist regardless of the approach we
>>> choose
>>>>> for 3.0. If community members develop without IEPs or public
>>> discussions,
>>>>> this will happen for both 2.x and 3.x unless we address this
>>> separately. I
>>>>> don't see how this is related to Alexey's suggestion, which targets
>>>>> *technical* issues with the product more than anything else. This a
>>> way to
>>>>> achieve better modularity, introduce better coverage with unit tests,
>>>>> reduce conflicts during development, etc.
>>>>>>>
>>>>>>> Coming back to transparency, let's identify the issues and fix
>>> them. It
>>>>> probably makes sense to have a separate discussion on this topic.
>>>>>>>
>>>>>>> -Val
>>>>>>>
>>>>>>> On Tue, Nov 10, 2020 at 1:05 PM Maxim Muzafarov <
>> mmuzaf@apache.org>
>>>>> wrote:
>>>>>>> Sergey,
>>>>>>>
>>>>>>>
>>>>>>> Your summary makes sense to me.
>>>>>>>
>>>>>>> However, how we come up from *Development transparency* to
>> *create a
>>>>>>> separate public repository dedicated for 3.0*? For me *development
>>>>>>> transparency* is about making changes in the master branch. These
>>>>>>> changes will definitely be seen by all the Ignite developers.
>>>>>>>
>>>>>>> A dedicated public repository is technically public and visible
>> for
>>>>>>> everyone, but it allows development without IEPs, without public
>>>>>>> discussion (since all the code changes are not related to the
>> master
>>>>>>> branch) it also allows a large number of assumptions and
>> deviations
>>>>>>> (like code-style violations). It also not about *development
>>>>>>> transparency* since developers which are working on 3.0 is only a
>>>>>>> subset of all Ignite developers which may continue working on 2.x.
>>> For
>>>>>>> me, this would be a huge step backwards.
>>>>>>>
>>>>>>> Ignite veterans should remember how long the branch stabilization
>>> took
>>>>>>> for the 2.x version with the PDS.
>>>>>>>
>>>>>>>
>>>>>>> I think each breaking change should be passed through the master
>>> branch.
>>>>>>>
>>>>>>> On Tue, 10 Nov 2020 at 22:18, Alexei Scherbakov
>>>>>>> <al...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> Makes sense to me.
>>>>>>>>
>>>>>>>> вт, 10 нояб. 2020 г. в 18:47, Sergey Chugunov <
>>>>> sergey.chugunov@gmail.com>:
>>>>>>>>
>>>>>>>>> Igniters,
>>>>>>>>>
>>>>>>>>> I thought over Friday meeting ideas and concerns and summarized
>>> them
>>>>> in
>>>>>>>>> these three points:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 1. *Components design unification approach.* New proposed
>>> components
>>>>>>>>> will be developed by different contributors, but they need to
>> be
>>>>> unified
>>>>>>>>> and should integrate with each other easily. To ensure that I
>>>>> suggest
>>>>>>>>> calling an architecture group that will create design
>> guidelines
>>>>> for all
>>>>>>>>> components and high-level overview of overall architecture.
>> How
>>>>> code is
>>>>>>>>> split into components, what are component boundaries, how
>>> component
>>>>>>>>> lifecycle works and what are its interfaces - all these and
>>> other
>>>>>>>>> questions
>>>>>>>>> should be covered.
>>>>>>>>>
>>>>>>>>> 2. *Scope management.* Apache 3.0 should be implemented
>> within a
>>>>>>>>> reasonable time, so we need some procedure to decide whether a
>>>>>>>>> particular
>>>>>>>>> feature should be dropped from the scope of 3.0 and postponed
>>> to 3.1
>>>>>>>>> release. To do so I suggest to range all features by two
>>> parameters:
>>>>>>>>> criticality for 3.0 and amount of breaking changes. 3.0 scope
>>> should
>>>>>>>>> include features of high criticality AND features with a big
>>> amount
>>>>> of
>>>>>>>>> breaking changes. All other features can be made optional.
>>>>>>>>>
>>>>>>>>> 3. *Development transparency.* Development of all components
>>> should
>>>>> be
>>>>>>>>> made as transparent for everyone as possible. Any contributor
>>>>> should be
>>>>>>>>> able to look over any component at any stage of development.
>> To
>>>>> achieve
>>>>>>>>> this I suggest to create a separate public repository
>> dedicated
>>> for
>>>>> 3.0
>>>>>>>>> development. It will make the code available for everyone but
>>> when
>>>>>>>>> development of 3.0 is done we won't loose any stars of our
>>> current
>>>>>>>>> repository as we merge dev repo into main one and drop dev.
>>>>>>>>>
>>>>>>>>> Do these ideas make sense to you? Are there any concerns not
>>> covered
>>>>> by
>>>>>>>>> these suggestions?
>>>>>>>>>
>>>>>>>>> On Fri, Nov 6, 2020 at 7:36 PM Kseniya Romanova <
>>>>> romanova.ks.spb@gmail.com
>>>>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Here are the slides from Alexey Goncharuk. Let's think this
>> over
>>> and
>>>>>>>>>> continue on Monday:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>
>>>
>> https://go.gridgain.com/rs/491-TWR-806/images/Ignite_3_Plans_and_development_process.pdf
>>>>>>>>>>
>>>>>>>>>> чт, 5 нояб. 2020 г. в 11:13, Anton Vinogradov <av...@apache.org>:
>>>>>>>>>>
>>>>>>>>>>> Folks,
>>>>>>>>>>>
>>>>>>>>>>> Should we perform cleanup work before (r)evolutional changes?
>>>>>>>>>>> My huge proposal is to get rid of things which we don't need
>>> anyway
>>>>>>>>>>> - local caches,
>>>>>>>>>>> - strange tx modes,
>>>>>>>>>>> - code overcomplexity because of RollingUpgrade feature never
>>>>> attended
>>>>>>>>> at
>>>>>>>>>>> AI,
>>>>>>>>>>> - etc,
>>>>>>>>>>> before choosing the way.
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Nov 3, 2020 at 3:31 PM Valentin Kulichenko <
>>>>>>>>>>> valentin.kulichenko@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Ksenia, thanks for scheduling this on such short notice!
>>>>>>>>>>>>
>>>>>>>>>>>> As for the original topic, I do support Alexey's idea. We're
>>> not
>>>>>>>>> going
>>>>>>>>>> to
>>>>>>>>>>>> rewrite anything from scratch, as most of the components are
>>> going
>>>>> to
>>>>>>>>>> be
>>>>>>>>>>>> moved as-is or with minimal modifications. However, the
>> changes
>>>>> that
>>>>>>>>>> are
>>>>>>>>>>>> proposed imply serious rework of the core parts of the code,
>>> which
>>>>>>>>> are
>>>>>>>>>>> not
>>>>>>>>>>>> properly decoupled from each other and from other parts. This
>>> makes
>>>>>>>>> the
>>>>>>>>>>>> incremental approach borderline impossible. Developing in a
>> new
>>>>> repo,
>>>>>>>>>>>> however, addresses this concern. As a bonus, we can also
>>> refactor
>>>>> the
>>>>>>>>>>> code,
>>>>>>>>>>>> introduce better decoupling, get rid of kernel context, and
>>> develop
>>>>>>>>>> unit
>>>>>>>>>>>> tests (finally!).
>>>>>>>>>>>>
>>>>>>>>>>>> Basically, this proposal only affects the *process*, not the
>>> set of
>>>>>>>>>>> changes
>>>>>>>>>>>> we had discussed before. Ignite 3.0 is our unique chance to
>>> make
>>>>>>>>> things
>>>>>>>>>>>> right.
>>>>>>>>>>>>
>>>>>>>>>>>> -Val
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Nov 3, 2020 at 3:06 AM Kseniya Romanova <
>>>>>>>>>>> romanova.ks.spb@gmail.com
>>>>>>>>>>>>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Pavel, all the interesting points will be anyway published
>>> here in
>>>>>>>>>>>> English
>>>>>>>>>>>>> (as the principal "if it's not on devlist it doesn't
>>> happened" is
>>>>>>>>>> still
>>>>>>>>>>>>> relevant). This is just a quick call for a group of
>>> developers.
>>>>>>>>> Later
>>>>>>>>>>> we
>>>>>>>>>>>>> can do a separate presentation of idea and discussion in
>>> English
>>>>> as
>>>>>>>>>> we
>>>>>>>>>>>> did
>>>>>>>>>>>>> for the Ignite 3.0 draft of changes.
>>>>>>>>>>>>>
>>>>>>>>>>>>> вт, 3 нояб. 2020 г. в 13:52, Pavel Tupitsyn <
>>> ptupitsyn@apache.org
>>>>>>>>>> :
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Kseniya,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks for scheduling this call.
>>>>>>>>>>>>>> Do you think we can switch to English if non-Russian
>> speaking
>>>>>>>>>>> community
>>>>>>>>>>>>>> members decide to join?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Nov 3, 2020 at 1:32 PM Kseniya Romanova <
>>>>>>>>>>>>> romanova.ks.spb@gmail.com
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Let's do this community discussion open. Here's the link
>> on
>>>>>>>>> zoom
>>>>>>>>>>> call
>>>>>>>>>>>>> in
>>>>>>>>>>>>>>> Russian for Friday 6 PM:
>>>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>> https://www.meetup.com/Moscow-Apache-Ignite-Meetup/events/274360378/
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> вт, 3 нояб. 2020 г. в 12:49, Nikolay Izhikov <
>>>>>>>>>> nizhikov@apache.org
>>>>>>>>>>>> :
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Time works for me.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 3 нояб. 2020 г., в 12:40, Alexey Goncharuk <
>>>>>>>>>>>>>> alexey.goncharuk@gmail.com
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> написал(а):
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Nikolay,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I am up for the call. I will try to explain my reasoning
>>> in
>>>>>>>>>>>> greater
>>>>>>>>>>>>>>>> detail
>>>>>>>>>>>>>>>>> and will be glad to hear the concerns. Will this Friday,
>>>>>>>>> Nov
>>>>>>>>>>> 6th,
>>>>>>>>>>>>>> work?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> вт, 3 нояб. 2020 г. в 10:09, Nikolay Izhikov <
>>>>>>>>>>>> nizhikov@apache.org
>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Igniters, should we have a call for this topic?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 2 нояб. 2020 г., в 18:53, Pavel Tupitsyn <
>>>>>>>>>>> ptupitsyn@apache.org
>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> написал(а):
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> not intend to rewrite everything from scratch
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Every single test from Ignite 2.x should be moved to
>>>>>>>>>> Ignite
>>>>>>>>>>> 3
>>>>>>>>>>>>>>>>>>>> regardless of how we choose to proceed.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Alexey, thank you for the explanation, this addresses
>>> all
>>>>>>>>>> of
>>>>>>>>>>> my
>>>>>>>>>>>>>>>> concerns.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Mon, Nov 2, 2020 at 6:43 PM Andrey Mashenkov <
>>>>>>>>>>>>>>>>>> andrey.mashenkov@gmail.com>
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi, Igniters.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> * AFAIU, we need a new repo if we want to apply
>>>>>>>>> different
>>>>>>>>>>>>>>> restrictions
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> pull requests,
>>>>>>>>>>>>>>>>>>>> otherwise I see no difference for myself.
>>>>>>>>>>>>>>>>>>>> E.g. make static analysis (do we have?), compile,
>>>>>>>>> styles,
>>>>>>>>>>> and
>>>>>>>>>>>>>>> javadoc
>>>>>>>>>>>>>>>>>>>> checks mandatory.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I think that relaxed requirements here will lead to
>> bad
>>>>>>>>>>>> product
>>>>>>>>>>>>>>>> quality.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> * Agree with Pavel, we should 'keep' integrations
>> tests
>>>>>>>>>>>> somehow.
>>>>>>>>>>>>>>>>>>>> During active development tests will be broken most
>> of
>>>>>>>>>> time,
>>>>>>>>>>>> so,
>>>>>>>>>>>>>>>>>>>> I'd port them e.g. suite-by-suite once we will have a
>>>>>>>>>> stable
>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>> featured
>>>>>>>>>>>>>>>>>>>> environment to run them and of course make test's
>> code
>>>>>>>>>> clear
>>>>>>>>>>>> and
>>>>>>>>>>>>>>> avoid
>>>>>>>>>>>>>>>>>>>> bad/non-relevant ones.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> * I like bottom-up approach.
>>>>>>>>>>>>>>>>>>>> With it we could make a better framework. I mean
>> clear
>>>>>>>>>>>> component
>>>>>>>>>>>>>>>>>> lifecycle,
>>>>>>>>>>>>>>>>>>>> component wiring mechanics, general methods to
>> approach
>>>>>>>>>> core
>>>>>>>>>>>>>>>> components
>>>>>>>>>>>>>>>>>>>> such as exchange/communication
>>>>>>>>>>>>>>>>>>>> to avoid code mess like we have in ExchangeFuture
>> with
>>>>>>>>> all
>>>>>>>>>>>> these
>>>>>>>>>>>>>>>> custom
>>>>>>>>>>>>>>>>>>>> callbacks for each component, interfaces like
>>>>>>>>>>>>>>>>>>>> PartitionsExchangeAware,
>> IgniteChangeGlobalStateSupport
>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>> a pack of
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>
>> start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected
>>>>>>>>>>>>>>>>>>>> and so on in various unexpected places.
>>>>>>>>>>>>>>>>>>>> Hope, we will be able to port most of the good code
>> to
>>>>>>>>> the
>>>>>>>>>>> new
>>>>>>>>>>>>>>>> framework
>>>>>>>>>>>>>>>>>>>> version.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk <
>>>>>>>>>>>>>>>>>>>> alexey.goncharuk@gmail.com>
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Nikolay, Pavel,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback! First of all, I wanted to
>>>>>>>>> stress
>>>>>>>>>>>> that
>>>>>>>>>>>>> I
>>>>>>>>>>>>>> do
>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>> intend to rewrite everything from scratch (I never
>>> used
>>>>>>>>>>> this
>>>>>>>>>>>>>>> phrase).
>>>>>>>>>>>>>>>>>>>> There
>>>>>>>>>>>>>>>>>>>>> are significant parts of code that would be moved
>> with
>>>>>>>>>>>> minimal
>>>>>>>>>>>>>>>>>>>>> modifications. Second, I never said that we will get
>>>>>>>>> rid
>>>>>>>>>> of
>>>>>>>>>>>> the
>>>>>>>>>>>>>> old
>>>>>>>>>>>>>>>>>> tests
>>>>>>>>>>>>>>>>>>>>> codebase. Every single test from Ignite 2.x should
>> be
>>>>>>>>>> moved
>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> Ignite 3
>>>>>>>>>>>>>>>>>>>>> regardless of how we choose to proceed.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> My point is that for some parts of the code a clean
>>>>>>>>>>> bottom-up
>>>>>>>>>>>>>>>>>>>>> implementation will be cheaper in many ways. Let me
>>>>>>>>> give
>>>>>>>>>>> you
>>>>>>>>>>>> a
>>>>>>>>>>>>>> few
>>>>>>>>>>>>>>>>>>>> concrete
>>>>>>>>>>>>>>>>>>>>> examples:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> - I think no one can object that we need a cleanly
>>>>>>>>>>> separated
>>>>>>>>>>>>>>>>>>>> persistence
>>>>>>>>>>>>>>>>>>>>> layer for Ignite. There is a very raw draft IEP for
>>>>>>>>> this
>>>>>>>>>>>>>> already.
>>>>>>>>>>>>>>> On
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> other hand, I think we also can agree that we need a
>>>>>>>>>>>>> split-brain
>>>>>>>>>>>>>>>>>>>>> resistant
>>>>>>>>>>>>>>>>>>>>> replication protocol for caches. There is also an
>> IEP
>>>>>>>>>> for
>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>>>>>>> Neither
>>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>> the changes is a good fit for 2.x because they are
>>>>>>>>>> likely
>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> introduce
>>>>>>>>>>>>>>>>>>>>> breaking changes in the persistence layer,
>>>>>>>>> configuration
>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>> behavior.
>>>>>>>>>>>>>>>>>>>>> Additionally, these components are now tightly
>>>>>>>>> coupled,
>>>>>>>>>> so
>>>>>>>>>>>>> there
>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>> no
>>>>>>>>>>>>>>>>>>>>> way
>>>>>>>>>>>>>>>>>>>>> these two changes can be implemented in parallel and
>>>>>>>>>> then
>>>>>>>>>>>>> merged
>>>>>>>>>>>>>>>>>>>>> together
>>>>>>>>>>>>>>>>>>>>> easily. So what we will end up with is having to
>>>>>>>>>> implement
>>>>>>>>>>>>> these
>>>>>>>>>>>>>>>>>>>> changes
>>>>>>>>>>>>>>>>>>>>> sequentially, fixing all existing tests twice, and
>>>>>>>>>>>> essentially
>>>>>>>>>>>>>>>>>>>> throwing
>>>>>>>>>>>>>>>>>>>>> away half of the work done because the other part of
>>>>>>>>> the
>>>>>>>>>>>>> change
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>> re-implemented
>>>>>>>>>>>>>>>>>>>>> - Similar example goes with getting rid of
>>>>>>>>>>>>> IgniteInternalFuture
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>> replacing it with CompletableFuture, and any other
>>>>>>>>>> change
>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>> touches
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> asynchronous part of the code.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Third, I do not see how this choice affects the UX
>> of
>>>>>>>>>>> Ignite.
>>>>>>>>>>>>> The
>>>>>>>>>>>>>>> end
>>>>>>>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>>>>>> experience must be fixed in the IEP regardless of
>> the
>>>>>>>>>>>>> development
>>>>>>>>>>>>>>>>>> process
>>>>>>>>>>>>>>>>>>>>> and the fact that we have gaps in this area in
>> Ignite
>>>>>>>>> 2.x
>>>>>>>>>>>> just
>>>>>>>>>>>>>>>> confirms
>>>>>>>>>>>>>>>>>>>>> that.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Pavel, agree that a repo/branch is a technicality, I
>>>>>>>>>> guess
>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>>> reformulate,
>>>>>>>>>>>>>>>>>>>>> my point is that we might agree to have a single
>>>>>>>>>>> development
>>>>>>>>>>>>>> master
>>>>>>>>>>>>>>>>>>>> branch
>>>>>>>>>>>>>>>>>>>>> with 'disassembled' end-to-end functionality for
>> some
>>>>>>>>>>> period
>>>>>>>>>>>> of
>>>>>>>>>>>>>>> time
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> speed up development, and re-assemble the core
>>> features
>>>>>>>>>>> after
>>>>>>>>>>>>>>> having
>>>>>>>>>>>>>>>>>>>>> submodules tested independently.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Nikolay,
>>>>>>>>>>>>>>>>>>>>>> We have many features that have to evolve.
>>>>>>>>>>>>>>>>>>>>>> Snapshots, rebalance, tooling, tracing, zookeeper
>>>>>>>>>> support,
>>>>>>>>>>>>> etc.
>>>>>>>>>>>>>>>>>>>>> This is not very specific. In the end, resources are
>>>>>>>>>>> limited
>>>>>>>>>>>>> and
>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>> not be able to drive both tracks simultaneously,
>>>>>>>>>> especially
>>>>>>>>>>>>>> after a
>>>>>>>>>>>>>>>>>>>> couple
>>>>>>>>>>>>>>>>>>>>> of features having been implemented for Ignite 3.0.
>> If
>>>>>>>>>>> there
>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>> indeed
>>>>>>>>>>>>>>>>>>>>> some major changes that we want to do in Ignite 2.x
>>>>>>>>>> instead
>>>>>>>>>>>> of
>>>>>>>>>>>>>>>> putting
>>>>>>>>>>>>>>>>>>>>> effort into 3.0 - let's discuss them. I am just not
>>>>>>>>> aware
>>>>>>>>>>> of
>>>>>>>>>>>>> any,
>>>>>>>>>>>>>>>>>> that's
>>>>>>>>>>>>>>>>>>>>> why I am eager to move forward with Ignite 3.0.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> We have bugs and issues that can be fixed in 2.x
>>>>>>>>> without
>>>>>>>>>>>>>> breaking
>>>>>>>>>>>>>>>>>>>> backward
>>>>>>>>>>>>>>>>>>>>> compatibility.
>>>>>>>>>>>>>>>>>>>>>> We have many users who are happy with the 2.x with
>>> all
>>>>>>>>>>> it’s
>>>>>>>>>>>>>>> issues.
>>>>>>>>>>>>>>>>>>>>> These changes will be covered by end-to-end tests
>> and
>>>>>>>>>>>> migrated
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> Ignite
>>>>>>>>>>>>>>>>>>>>> 3.0, so I see no issues here.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Finally, Anton & Nikolay
>>>>>>>>>>>>>>>>>>>>> I do not have an estimate for this simply because
>> the
>>>>>>>>>>>> activity
>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>> community-driven and it depends on the number of
>>> people
>>>>>>>>>>>> willing
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> contribute. With the current pace, I would hope to
>>> have
>>>>>>>>>> an
>>>>>>>>>>> RC
>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>> Ignite
>>>>>>>>>>>>>>>>>>>> 3.0
>>>>>>>>>>>>>>>>>>>>> to be ready by the end of 2021. My gut feeling is
>> that
>>>>>>>>> by
>>>>>>>>>>>>> moving
>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>> incremental changes, we will not be able to
>> implement
>>>>>>>>>> even
>>>>>>>>>>>> half
>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> wishlist by that time.
>>>>>>>>>>>>>>>>>>>>> I doubt that releasing several major releases with
>>>>>>>>>> breaking
>>>>>>>>>>>>>> changes
>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>> make Ignite users happy either because each upgrade
>>>>>>>>> will
>>>>>>>>>>> cost
>>>>>>>>>>>>>>> Ignite
>>>>>>>>>>>>>>>>>>>> users
>>>>>>>>>>>>>>>>>>>>> money, so the fewer major versions we release, the
>>>>>>>>>> better.
>>>>>>>>>>>> Thus
>>>>>>>>>>>>>> my
>>>>>>>>>>>>>>>> wish
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> include all breaking changes in one release.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I'll be now quiet for a while, let's see what other
>>>>>>>>>>> community
>>>>>>>>>>>>>>> members
>>>>>>>>>>>>>>>>>>>>> think.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <
>>>>>>>>>>>>>> ptupitsyn@apache.org
>>>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> 1. Rewriting from scratch is never a good idea.
>>>>>>>>>>>>>>>>>>>>>> We don't want to follow the path of Netscape and
>> lose
>>>>>>>>>> all
>>>>>>>>>>>> our
>>>>>>>>>>>>>>> users
>>>>>>>>>>>>>>>>>>>>>> by the time we have a working 3.0 [1]
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> 2. Not sure about new repo - seems like some pain
>> and
>>>>>>>>> no
>>>>>>>>>>>> gain,
>>>>>>>>>>>>>>>> what's
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> problem with a branch?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> 3. We should keep existing integration tests when
>>>>>>>>>>> possible.
>>>>>>>>>>>>>>>>>>>>>> We have accumulated a lot of edge case knowledge
>> over
>>>>>>>>>> the
>>>>>>>>>>>>> years,
>>>>>>>>>>>>>>>>>>>>>> it is not a good idea to send all of that down the
>>>>>>>>>> drain.
>>>>>>>>>>>>>>>>>>>>>> Yes, integration tests are slow, but they are the
>>> most
>>>>>>>>>>>>> valuable.
>>>>>>>>>>>>>>>>>>>>>> I think we can move more stuff into nightly runs
>> and
>>>>>>>>>> have
>>>>>>>>>>> a
>>>>>>>>>>>>> fast
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>> modern
>>>>>>>>>>>>>>>>>>>>>> basic suite.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Alexey, you are much more familiar with the Ignite
>>>>>>>>> core
>>>>>>>>>>>>> codebase
>>>>>>>>>>>>>>>> than
>>>>>>>>>>>>>>>>>>>>> most
>>>>>>>>>>>>>>>>>>>>>> of us,
>>>>>>>>>>>>>>>>>>>>>> can you please explain in more detail which
>>> particular
>>>>>>>>>>>>> feature,
>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>> your
>>>>>>>>>>>>>>>>>>>>>> opinion,
>>>>>>>>>>>>>>>>>>>>>> mandates this "start from scratch" approach?
>>>>>>>>>>>>>>>>>>>>>> Is it really not possible at all to follow a less
>>>>>>>>>> radical
>>>>>>>>>>>> way?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>
>>>
>> https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <
>>>>>>>>>>>>>>> nizhikov@apache.org
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Hello, Alexey.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I think that «rewriting from scratch» approach
>> has a
>>>>>>>>>> high
>>>>>>>>>>>>> risk
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> make
>>>>>>>>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>>>>>> features unusable.
>>>>>>>>>>>>>>>>>>>>>>> At the time Ignite2 was started no-one wants to do
>>>>>>>>> bad
>>>>>>>>>> UX
>>>>>>>>>>>> or
>>>>>>>>>>>>>> bad
>>>>>>>>>>>>>>>>>>>>>> features.
>>>>>>>>>>>>>>>>>>>>>>> Nevertheless, it happen.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I think we can avoid it with the Ignite3 and
>>>>>>>>> successors
>>>>>>>>>>> if
>>>>>>>>>>>> we
>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>> move
>>>>>>>>>>>>>>>>>>>>>>> step by step without keeping backward
>> compatibility
>>>>>>>>>>>>>>>>>>>>>>> With the step by step approach, we can focus on
>> each
>>>>>>>>>>>>> component
>>>>>>>>>>>>>>>>>>>>>> separately.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> What new features are we planning to implement
>> for
>>>>>>>>>>> Ignite
>>>>>>>>>>>>> 2.x?
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> We have many features that have to evolve.
>>>>>>>>>>>>>>>>>>>>>>> Snapshots, rebalance, tooling, tracing, zookeeper
>>>>>>>>>>> support,
>>>>>>>>>>>>> etc.
>>>>>>>>>>>>>>>>>>>>>>> We have bugs and issues that can be fixed in 2.x
>>>>>>>>>> without
>>>>>>>>>>>>>> breaking
>>>>>>>>>>>>>>>>>>>>>> backward
>>>>>>>>>>>>>>>>>>>>>>> compatibility.
>>>>>>>>>>>>>>>>>>>>>>> We have many users who are happy with the 2.x with
>>>>>>>>> all
>>>>>>>>>>> it’s
>>>>>>>>>>>>>>> issues.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> 2 нояб. 2020 г., в 14:09, Anton Vinogradov <
>>>>>>>>>>> av@apache.org
>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> написал(а):
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Alexey,
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Do we have any estimates of how fast we'll be
>> able
>>>>>>>>> to
>>>>>>>>>>> gain
>>>>>>>>>>>>>>>>>>>>>>> production-ready
>>>>>>>>>>>>>>>>>>>>>>>> AI 3.0 in case of a "new repo" choice?
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk <
>>>>>>>>>>>>>>>>>>>>>>> alexey.goncharuk@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Nikolay,
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> What new features are we planning to implement
>> for
>>>>>>>>>>> Ignite
>>>>>>>>>>>>>> 2.x?
>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>> think
>>>>>>>>>>>>>>>>>>>>>>> once
>>>>>>>>>>>>>>>>>>>>>>>>> we commence working on Ignite 3.0, we should
>>>>>>>>>> gradually
>>>>>>>>>>>>> cease
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> activity
>>>>>>>>>>>>>>>>>>>>>>>>> on Ignite 2.x to mere bugfixes because such
>>>>>>>>> parallel
>>>>>>>>>>>>>>> development
>>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>> overwhelming regardless of how we choose to
>>>>>>>>> proceed.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <
>>>>>>>>>>>>>>>> nizhikov@apache.org
>>>>>>>>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> To be clear:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> I would suggest creating a new repository for
>>>>>>>>>> Ignite
>>>>>>>>>>>> 3.0
>>>>>>>>>>>>>>>>>>>>> (perhaps, a
>>>>>>>>>>>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>>>>>>>>> clean branch, but a new repo looks nicer to me)
>>>>>>>>> and
>>>>>>>>>> a
>>>>>>>>>>>> new
>>>>>>>>>>>>>>> Ignite
>>>>>>>>>>>>>>>>>>>>> 3.0
>>>>>>>>>>>>>>>>>>>>>>>>>> TeamCity project.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> +1 for new Team City project.
>>>>>>>>>>>>>>>>>>>>>>>>>> +1 for new branch for Ignite3.
>>>>>>>>>>>>>>>>>>>>>>>>>> -1 for new repo.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <
>>>>>>>>>>>>>>>>>>>> NIzhikov.dev@gmail.com
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> написал(а):
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello, Alexey.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> I think it will hurt our project more than
>> help.
>>>>>>>>>>>>>>>>>>>>>>>>>>> Developing new features for 2 separate
>> branches
>>>>>>>>>> with
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> different
>>>>>>>>>>>>>>>>>>>>>>> APIs
>>>>>>>>>>>>>>>>>>>>>>>>>> and internal structure is overwhelming
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Maybe we should relax a bit requirements for
>>>>>>>>>> Ignite3?
>>>>>>>>>>>>>>>>>>>>>>>>>>> Maybe we should move step by step and make
>>>>>>>>> Ignite3
>>>>>>>>>>> with
>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>>>>>>>>> configuration than Ignite4 with new
>> transactions,
>>>>>>>>>> etc?
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <
>>>>>>>>>>>>>>>>>>>>>>>>> alexey.goncharuk@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>> написал(а):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Igniters,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I wanted to pitch a rather radical idea
>>>>>>>>> regarding
>>>>>>>>>>> the
>>>>>>>>>>>>>> Ignite
>>>>>>>>>>>>>>>>>>>> 3.0
>>>>>>>>>>>>>>>>>>>>>>>>>>>> development which has occurred to me some
>> time
>>>>>>>>>> ago.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> We already have several IEPs targeted to
>> Ignite
>>>>>>>>>> 3.0
>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>> imply
>>>>>>>>>>>>>>>>>>>>>> major
>>>>>>>>>>>>>>>>>>>>>>>>>>>> changes to the codebase (the change in
>>>>>>>>> replication
>>>>>>>>>>>>>> protocol
>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>> thus
>>>>>>>>>>>>>>>>>>>>>>>>>>>> transactions, change in binary format,
>> updated
>>>>>>>>>>>>>> metastorage,
>>>>>>>>>>>>>>>>>>>> etc).
>>>>>>>>>>>>>>>>>>>>>> We
>>>>>>>>>>>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>>>>>>>>>>>> planned significant changes in public APIs:
>>>>>>>>>>>>> configuration
>>>>>>>>>>>>>>>>>>>> format
>>>>>>>>>>>>>>>>>>>>>>>>> change,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> improvements in cache APIs, SQL APIs,
>>>>>>>>> transaction
>>>>>>>>>>> mode
>>>>>>>>>>>>>>> rework.
>>>>>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>>>>>>>> wishlist
>>>>>>>>>>>>>>>>>>>>>>>>>>>> of changes for Ignite 3.0 is huge.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> So, I was wondering whether it makes sense to
>>>>>>>>> try
>>>>>>>>>> to
>>>>>>>>>>>>>> change
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> old
>>>>>>>>>>>>>>>>>>>>>>>>>>>> codebase, or start with a new baseline and
>> move
>>>>>>>>>> old
>>>>>>>>>>>>> pieces
>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>> code
>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>> do
>>>>>>>>>>>>>>>>>>>>>>>>>>>> not require significant rework. Personally, I
>>>>>>>>>> would
>>>>>>>>>>> go
>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> second
>>>>>>>>>>>>>>>>>>>>>>>>>>>> option for the following reasons:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> - We have a chance to shift the development
>>>>>>>>>> paradigm
>>>>>>>>>>>> in
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> project
>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>> introduce the practice of true unit-tests. In
>>>>>>>>> the
>>>>>>>>>>> new
>>>>>>>>>>>>>>> baseline
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> beginning there will be no ability to run an
>>>>>>>>>>>> end-to-end
>>>>>>>>>>>>>>>>>>>> scenario,
>>>>>>>>>>>>>>>>>>>>>>>>> thus
>>>>>>>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be forced to write unit-tests. So far,
>>> such
>>>>>>>>>>>>> practice
>>>>>>>>>>>>>>> was
>>>>>>>>>>>>>>>>>>>>> hard
>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> implement because of tight coupling between
>>>>>>>>> Ignite
>>>>>>>>>>>>>>> components
>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>> inability
>>>>>>>>>>>>>>>>>>>>>>>>>>>> to instantiate components without an instance
>>> of
>>>>>>>>>>>>>>>> KernalContext.
>>>>>>>>>>>>>>>>>>>>> For
>>>>>>>>>>>>>>>>>>>>>>>>>>>> example, we should be able to thoroughly test
>>>>>>>>>>> internal
>>>>>>>>>>>>>>>>>>>>> primitives,
>>>>>>>>>>>>>>>>>>>>>>>>>> such as
>>>>>>>>>>>>>>>>>>>>>>>>>>>> replication protocol (without actual
>>>>>>>>>> communication),
>>>>>>>>>>>>>>>>>>>> distributed
>>>>>>>>>>>>>>>>>>>>>>>>>>>> metastorage contracts, persistence layer,
>> etc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> - We will significantly reduce the
>> development
>>>>>>>>>> cycle
>>>>>>>>>>>> in
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> beginning
>>>>>>>>>>>>>>>>>>>>>>>>>>>> (right now the RunAll takes two hours of
>>>>>>>>>>> astronomical
>>>>>>>>>>>>> time
>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>>>>>>>>>> TC;
>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the new approach developer will be able to
>>>>>>>>> run
>>>>>>>>>>> ALL
>>>>>>>>>>>>>> tests
>>>>>>>>>>>>>>>>>>>>> locally
>>>>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> matter of minutes)
>>>>>>>>>>>>>>>>>>>>>>>>>>>> - We can get rid of TC bot and enforce green
>> TC
>>>>>>>>> by
>>>>>>>>>>>>>>> integrating
>>>>>>>>>>>>>>>>>>>> TC
>>>>>>>>>>>>>>>>>>>>>>>>> build
>>>>>>>>>>>>>>>>>>>>>>>>>>>> results with GitHub PRs (the same way Travis
>> is
>>>>>>>>>>>>> currently
>>>>>>>>>>>>>>>>>>>>>> integrated
>>>>>>>>>>>>>>>>>>>>>>>>>> to PR
>>>>>>>>>>>>>>>>>>>>>>>>>>>> check). We should restrict PR merge without a
>>> TC
>>>>>>>>>>> check
>>>>>>>>>>>>>>>>>>>>>>>>>>>> - We will still have to re-write all tests,
>> but
>>>>>>>>>> only
>>>>>>>>>>>>> once.
>>>>>>>>>>>>>>> If
>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>> try
>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> modify the old codebase, we would need to
>>> modify
>>>>>>>>>> all
>>>>>>>>>>>> the
>>>>>>>>>>>>>>> tests
>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>> every
>>>>>>>>>>>>>>>>>>>>>>>>>>>> major change (public API change,
>> configuration
>>>>>>>>>>> change)
>>>>>>>>>>>>>>>>>>>>>>>>>>>> - We will have fewer conflicts when working
>>>>>>>>>>> together.
>>>>>>>>>>>>> For
>>>>>>>>>>>>>>>>>>>>> example,
>>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>>>>>>>> cannot imagine how one would merge two
>> changes
>>>>>>>>> of
>>>>>>>>>>>>> getting
>>>>>>>>>>>>>>> rid
>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>>>>>>> IgniteFuture and changes in replication
>>>>>>>>> protocol,
>>>>>>>>>>> for
>>>>>>>>>>>>>>> example
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Technically, I would suggest creating a new
>>>>>>>>>>> repository
>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>> Ignite
>>>>>>>>>>>>>>>>>>>>>> 3.0
>>>>>>>>>>>>>>>>>>>>>>>>>>>> (perhaps, a new clean branch, but a new repo
>>>>>>>>> looks
>>>>>>>>>>>> nicer
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> me)
>>>>>>>>>>>>>>>>>>>>>> and a
>>>>>>>>>>>>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ignite 3.0 TeamCity project.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> While it may seem quite radical, I do believe
>>>>>>>>> that
>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>> approach
>>>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>>>>> give
>>>>>>>>>>>>>>>>>>>>>>>>>>>> us more benefits than trying to make such
>> major
>>>>>>>>>>>> changes
>>>>>>>>>>>>> in
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>> existing
>>>>>>>>>>>>>>>>>>>>>>>>>>>> codebase. If needed, let's schedule a discord
>>>>>>>>> chat
>>>>>>>>>>>> like
>>>>>>>>>>>>>>> before
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>> discuss
>>>>>>>>>>>>>>>>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> WDYT?
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>>>>> Andrey V. Mashenkov
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>> Best regards,
>>>>>>>> Alexei Scherbakov
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> Sincerely yours, Ivan Daschinskiy
>>>
>>
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Alexey Goncharuk <al...@gmail.com>.
Folks,
I think we are overly driven away by the phrase 'new repo' rather than the
essence of my suggestion. We can keep developing in the same repo, we can
even keep developing in the master branch. My point is that Ignite 3.0 is a
chance to move on with the architecture, so if we really want to make
architectural improvements, we should not strive for incremental changes
for *some parts of the code*.
Maxim,
To comment on your examples: I think that the huge effort that is currently
required to make any significant change in Ignite is the perfect example of
how we lack structure in the codebase. Yes, theoretically we can introduce
incremental changes in the code that will improve the structure, but my
question is: we did not do it before, what will enforce us to make these
changes now? With the current approach, adding a new feature increases the
test time non-linearly because without proper decoupling you have to test
all possible combinations of features together. We can move faster than
that.
I also do not agree that we should reduce the scope of Ignite 3.0 that
much. I do not see how the schema-first approach can be properly and
reliably implemented without a reliable HA metastorage, which in turn
requires a reliable replication protocol to be implemented. Besides, if a
number of people want to work on some Ignite feature, why should they wait
because not all community members have time to review the changes?
Let's indeed focus on Sergey's suggestions on the design->development
approach. I back both Nikolay's and Maxim's scope, but I think we should
unite them, not intersect, and the minimal list of changes to be included
to Ignite 3.0 is:
- API & configuration cleanup
- New management tool
- Schema-first approach
- New replication infrastructure
Any smaller subset of changes will leave Ignite 3.0 in a transient state
with people being too afraid to move to it because there are more major
breaking changes scheduled.
пт, 13 нояб. 2020 г. в 18:28, Alexey Zinoviev <za...@gmail.com>:
> I'm -1 for creating a new repo.
> Also I support Maxim's plan for 3.0
>
> пт, 13 нояб. 2020 г. в 15:50, Maxim Muzafarov <mm...@apache.org>:
>
> > Val,
> >
> >
> > Why *creating a new repo* is the main point we faced with? Would it be
> > better to discuss the components design approach and scope management
> > first suggested by Sergey Chugunov? I doubt that new repo will solve
> > move us forward.
> >
> > Currently, I'm -1 to create a new repo with the inputs above.
> >
> > In addition to Nikolay's answer I see the following drawbacks of
> > creating new repo:
> > - we have very few positive examples of finalizing really huge
> > improvements to *production-ready* states the others remains
> > incomplete (MVCC, Calcite, Zookeeper, Tracing, Thread per Partition,
> > etc)
> > - AFAIK, the Native Persistence took a very long period of
> > stabilization even after it has been developed (we must take it into
> > account for developing new features like IEP-61)
> > - feature development for a long period of time (like 3.0) without any
> > releases will lead to all these changes became obsolete at the moment
> > of release (AFAIK the 2.8 which released a year ago still has no big
> > deployments)
> > - human resources -- some of the Igniters may lose their interest for
> > 3.0 during development, some of them may switch to different projects,
> > etc.
> > - do we all estimating the scope of 3.0 correct? The 2.8 release took 1.5
> > years.
> >
> > Have I missed something?
> >
> >
> > I suggest the following plan:
> >
> > - initiate 3.0 development in the master branch (after 2.10 release
> > change version to 3.0-SNAPSHOT instead of 2.11-SNAPSHOT)
> > - cleanup and collapse all the current APIs (see To Be Removed List
> > For Discussion on Apache Ignite 3.0 Wishlist)
> > - reduce the scope for 3.0 even more. I suggest focusing on two
> > things: Calcite + Schema-first approach
> > - create feature branches for proposed IEPs (for 3.0 only)
> > - create the release road map (allocate e.g. IEP-61 to 4.0 etc.)
> >
> > On Fri, 13 Nov 2020 at 14:03, Ivan Daschinsky <iv...@gmail.com>
> wrote:
> > >
> > > >> b. Implement IEP-61 - Common Replication Infrastructure
> > > I suppose, that this is the main cause of the current discussion.
> > > I hardly believe that this activity can be done without at least
> > creating a
> > > completely new branch.
> > >
> > > пт, 13 нояб. 2020 г. в 11:12, Nikolay Izhikov <ni...@apache.org>:
> > >
> > > > My suggestion:
> > > >
> > > > 1. Reduce Ignite3 scope to the following:
> > > > a. Delete all deprecated API and support of obsolete internal
> > > > protocols.
> > > > b. Implement IEP-61 - Common Replication Infrastructure
> > > > c. Implement new Ignite management tool ignitectl as
> suggested
> > > > during Ignite3 discussion.
> > > >
> > > > 2. Implement and release following improvements like transactions,
> > Calcite
> > > > based SQL, etc in the ongoing releases - Ignite 4, 5, 6
> > > >
> > > > My concern against separate Ignite 3 repo is the following:
> > > >
> > > > 1. We spread community to the two very separated part - Ignite3
> > developers
> > > > and Ignite2 maintainers. believe it’s bad for our community.
> > > > That can lead to the situation when we don’t fix critical or
> > > > blocker issueds «because they will not exists in Ignite3»
> > > > That will lead to the solutions never reviewed or reviewed
> > poorly.
> > > >
> > > > 2. It seems for me that current scope of Ignite3 is too big to be
> > > > implemented in any reasonable time.
> > > >
> > > >
> > > > > 13 нояб. 2020 г., в 10:57, Nikolay Izhikov <NIzhikov.dev@gmail.com
> >
> > > > написал(а):
> > > > >
> > > > > Hello, Valentin.
> > > > >
> > > > >> Nikolay, Maxim, are you OK with this route?
> > > > >
> > > > > -1 to have another repo for Ignite3 development.
> > > > >
> > > > >> 13 нояб. 2020 г., в 03:04, Valentin Kulichenko <
> > > > valentin.kulichenko@gmail.com> написал(а):
> > > > >>
> > > > >> Folks,
> > > > >>
> > > > >> We already have multiple IEPs for Ignite 3.0, and as far as I
> know,
> > > > there are contributors that would like to work on them (or probably
> > already
> > > > started). That said, we should make a decision as soon as possible.
> > > > >>
> > > > >> At this point, it doesn't seem that there are any strong
> objections
> > to
> > > > the technical side of things. So I would suggest the following:
> > > > >>
> > > > >> 1. Proceed with Alexey's approach to the development process, as
> it
> > > > seems to be the best (in my opinion - the only) way to address all
> the
> > > > technical concerns and issues expressed in the thread. We'll start by
> > > > creating a new repo and a new TC project.
> > > > >> 2. Start a separate discussion around transparency. If there are
> any
> > > > changes we need to make to our contributor guidelines, I am happy to
> > talk
> > > > them through, but I don't think it's reasonable to delay feature
> > > > development because of this. In the short term, I will make sure that
> > > > everything that happens within the new repo is as open to the
> > community as
> > > > possible.
> > > > >>
> > > > >> Nikolay, Maxim, are you OK with this route?
> > > > >>
> > > > >> -Val
> > > > >>
> > > > >> On Tue, Nov 10, 2020 at 4:55 PM Valentin Kulichenko <
> > > > valentin.kulichenko@gmail.com> wrote:
> > > > >> Maxim,
> > > > >>
> > > > >> 2.x and 3.x will have to coexist for some time - I don't see how
> we
> > can
> > > > avoid this considering the set of proposed changes. That said, we
> > > > effectively will need to have two "masters" - one for each major
> > version.
> > > > Master for 3.x can technically be a branch in the existing repo, but
> > having
> > > > a separate repo seems cleaner, simply because it will not be a
> > "branch" in
> > > > the traditional sense.
> > > > >>
> > > > >> Note that the new repo will still be under the Apache org, with
> the
> > > > same set of committers, managed by the community, etc. All the
> > development
> > > > happening for 3.0 must follow the rules that we currently have (if
> > > > anything, it's an opportunity to improve those rules).
> > > > >>
> > > > >> As I said during the call on Friday, I strongly believe that if
> > there
> > > > is a transparency issue, it will exist regardless of the approach we
> > choose
> > > > for 3.0. If community members develop without IEPs or public
> > discussions,
> > > > this will happen for both 2.x and 3.x unless we address this
> > separately. I
> > > > don't see how this is related to Alexey's suggestion, which targets
> > > > *technical* issues with the product more than anything else. This a
> > way to
> > > > achieve better modularity, introduce better coverage with unit tests,
> > > > reduce conflicts during development, etc.
> > > > >>
> > > > >> Coming back to transparency, let's identify the issues and fix
> > them. It
> > > > probably makes sense to have a separate discussion on this topic.
> > > > >>
> > > > >> -Val
> > > > >>
> > > > >> On Tue, Nov 10, 2020 at 1:05 PM Maxim Muzafarov <
> mmuzaf@apache.org>
> > > > wrote:
> > > > >> Sergey,
> > > > >>
> > > > >>
> > > > >> Your summary makes sense to me.
> > > > >>
> > > > >> However, how we come up from *Development transparency* to
> *create a
> > > > >> separate public repository dedicated for 3.0*? For me *development
> > > > >> transparency* is about making changes in the master branch. These
> > > > >> changes will definitely be seen by all the Ignite developers.
> > > > >>
> > > > >> A dedicated public repository is technically public and visible
> for
> > > > >> everyone, but it allows development without IEPs, without public
> > > > >> discussion (since all the code changes are not related to the
> master
> > > > >> branch) it also allows a large number of assumptions and
> deviations
> > > > >> (like code-style violations). It also not about *development
> > > > >> transparency* since developers which are working on 3.0 is only a
> > > > >> subset of all Ignite developers which may continue working on 2.x.
> > For
> > > > >> me, this would be a huge step backwards.
> > > > >>
> > > > >> Ignite veterans should remember how long the branch stabilization
> > took
> > > > >> for the 2.x version with the PDS.
> > > > >>
> > > > >>
> > > > >> I think each breaking change should be passed through the master
> > branch.
> > > > >>
> > > > >> On Tue, 10 Nov 2020 at 22:18, Alexei Scherbakov
> > > > >> <al...@gmail.com> wrote:
> > > > >>>
> > > > >>> Makes sense to me.
> > > > >>>
> > > > >>> вт, 10 нояб. 2020 г. в 18:47, Sergey Chugunov <
> > > > sergey.chugunov@gmail.com>:
> > > > >>>
> > > > >>>> Igniters,
> > > > >>>>
> > > > >>>> I thought over Friday meeting ideas and concerns and summarized
> > them
> > > > in
> > > > >>>> these three points:
> > > > >>>>
> > > > >>>>
> > > > >>>> 1. *Components design unification approach.* New proposed
> > components
> > > > >>>> will be developed by different contributors, but they need to
> be
> > > > unified
> > > > >>>> and should integrate with each other easily. To ensure that I
> > > > suggest
> > > > >>>> calling an architecture group that will create design
> guidelines
> > > > for all
> > > > >>>> components and high-level overview of overall architecture.
> How
> > > > code is
> > > > >>>> split into components, what are component boundaries, how
> > component
> > > > >>>> lifecycle works and what are its interfaces - all these and
> > other
> > > > >>>> questions
> > > > >>>> should be covered.
> > > > >>>>
> > > > >>>> 2. *Scope management.* Apache 3.0 should be implemented
> within a
> > > > >>>> reasonable time, so we need some procedure to decide whether a
> > > > >>>> particular
> > > > >>>> feature should be dropped from the scope of 3.0 and postponed
> > to 3.1
> > > > >>>> release. To do so I suggest to range all features by two
> > parameters:
> > > > >>>> criticality for 3.0 and amount of breaking changes. 3.0 scope
> > should
> > > > >>>> include features of high criticality AND features with a big
> > amount
> > > > of
> > > > >>>> breaking changes. All other features can be made optional.
> > > > >>>>
> > > > >>>> 3. *Development transparency.* Development of all components
> > should
> > > > be
> > > > >>>> made as transparent for everyone as possible. Any contributor
> > > > should be
> > > > >>>> able to look over any component at any stage of development.
> To
> > > > achieve
> > > > >>>> this I suggest to create a separate public repository
> dedicated
> > for
> > > > 3.0
> > > > >>>> development. It will make the code available for everyone but
> > when
> > > > >>>> development of 3.0 is done we won't loose any stars of our
> > current
> > > > >>>> repository as we merge dev repo into main one and drop dev.
> > > > >>>>
> > > > >>>> Do these ideas make sense to you? Are there any concerns not
> > covered
> > > > by
> > > > >>>> these suggestions?
> > > > >>>>
> > > > >>>> On Fri, Nov 6, 2020 at 7:36 PM Kseniya Romanova <
> > > > romanova.ks.spb@gmail.com
> > > > >>>>>
> > > > >>>> wrote:
> > > > >>>>
> > > > >>>>> Here are the slides from Alexey Goncharuk. Let's think this
> over
> > and
> > > > >>>>> continue on Monday:
> > > > >>>>>
> > > > >>>>>
> > > > >>>>
> > > >
> >
> https://go.gridgain.com/rs/491-TWR-806/images/Ignite_3_Plans_and_development_process.pdf
> > > > >>>>>
> > > > >>>>> чт, 5 нояб. 2020 г. в 11:13, Anton Vinogradov <av...@apache.org>:
> > > > >>>>>
> > > > >>>>>> Folks,
> > > > >>>>>>
> > > > >>>>>> Should we perform cleanup work before (r)evolutional changes?
> > > > >>>>>> My huge proposal is to get rid of things which we don't need
> > anyway
> > > > >>>>>> - local caches,
> > > > >>>>>> - strange tx modes,
> > > > >>>>>> - code overcomplexity because of RollingUpgrade feature never
> > > > attended
> > > > >>>> at
> > > > >>>>>> AI,
> > > > >>>>>> - etc,
> > > > >>>>>> before choosing the way.
> > > > >>>>>>
> > > > >>>>>> On Tue, Nov 3, 2020 at 3:31 PM Valentin Kulichenko <
> > > > >>>>>> valentin.kulichenko@gmail.com> wrote:
> > > > >>>>>>
> > > > >>>>>>> Ksenia, thanks for scheduling this on such short notice!
> > > > >>>>>>>
> > > > >>>>>>> As for the original topic, I do support Alexey's idea. We're
> > not
> > > > >>>> going
> > > > >>>>> to
> > > > >>>>>>> rewrite anything from scratch, as most of the components are
> > going
> > > > to
> > > > >>>>> be
> > > > >>>>>>> moved as-is or with minimal modifications. However, the
> changes
> > > > that
> > > > >>>>> are
> > > > >>>>>>> proposed imply serious rework of the core parts of the code,
> > which
> > > > >>>> are
> > > > >>>>>> not
> > > > >>>>>>> properly decoupled from each other and from other parts. This
> > makes
> > > > >>>> the
> > > > >>>>>>> incremental approach borderline impossible. Developing in a
> new
> > > > repo,
> > > > >>>>>>> however, addresses this concern. As a bonus, we can also
> > refactor
> > > > the
> > > > >>>>>> code,
> > > > >>>>>>> introduce better decoupling, get rid of kernel context, and
> > develop
> > > > >>>>> unit
> > > > >>>>>>> tests (finally!).
> > > > >>>>>>>
> > > > >>>>>>> Basically, this proposal only affects the *process*, not the
> > set of
> > > > >>>>>> changes
> > > > >>>>>>> we had discussed before. Ignite 3.0 is our unique chance to
> > make
> > > > >>>> things
> > > > >>>>>>> right.
> > > > >>>>>>>
> > > > >>>>>>> -Val
> > > > >>>>>>>
> > > > >>>>>>> On Tue, Nov 3, 2020 at 3:06 AM Kseniya Romanova <
> > > > >>>>>> romanova.ks.spb@gmail.com
> > > > >>>>>>>>
> > > > >>>>>>> wrote:
> > > > >>>>>>>
> > > > >>>>>>>> Pavel, all the interesting points will be anyway published
> > here in
> > > > >>>>>>> English
> > > > >>>>>>>> (as the principal "if it's not on devlist it doesn't
> > happened" is
> > > > >>>>> still
> > > > >>>>>>>> relevant). This is just a quick call for a group of
> > developers.
> > > > >>>> Later
> > > > >>>>>> we
> > > > >>>>>>>> can do a separate presentation of idea and discussion in
> > English
> > > > as
> > > > >>>>> we
> > > > >>>>>>> did
> > > > >>>>>>>> for the Ignite 3.0 draft of changes.
> > > > >>>>>>>>
> > > > >>>>>>>> вт, 3 нояб. 2020 г. в 13:52, Pavel Tupitsyn <
> > ptupitsyn@apache.org
> > > > >>>>> :
> > > > >>>>>>>>
> > > > >>>>>>>>> Kseniya,
> > > > >>>>>>>>>
> > > > >>>>>>>>> Thanks for scheduling this call.
> > > > >>>>>>>>> Do you think we can switch to English if non-Russian
> speaking
> > > > >>>>>> community
> > > > >>>>>>>>> members decide to join?
> > > > >>>>>>>>>
> > > > >>>>>>>>> On Tue, Nov 3, 2020 at 1:32 PM Kseniya Romanova <
> > > > >>>>>>>> romanova.ks.spb@gmail.com
> > > > >>>>>>>>>>
> > > > >>>>>>>>> wrote:
> > > > >>>>>>>>>
> > > > >>>>>>>>>> Let's do this community discussion open. Here's the link
> on
> > > > >>>> zoom
> > > > >>>>>> call
> > > > >>>>>>>> in
> > > > >>>>>>>>>> Russian for Friday 6 PM:
> > > > >>>>>>>>>>
> > > > >>>>>>
> > > > https://www.meetup.com/Moscow-Apache-Ignite-Meetup/events/274360378/
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> вт, 3 нояб. 2020 г. в 12:49, Nikolay Izhikov <
> > > > >>>>> nizhikov@apache.org
> > > > >>>>>>> :
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>> Time works for me.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>> 3 нояб. 2020 г., в 12:40, Alexey Goncharuk <
> > > > >>>>>>>>> alexey.goncharuk@gmail.com
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> написал(а):
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Nikolay,
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> I am up for the call. I will try to explain my reasoning
> > in
> > > > >>>>>>> greater
> > > > >>>>>>>>>>> detail
> > > > >>>>>>>>>>>> and will be glad to hear the concerns. Will this Friday,
> > > > >>>> Nov
> > > > >>>>>> 6th,
> > > > >>>>>>>>> work?
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> вт, 3 нояб. 2020 г. в 10:09, Nikolay Izhikov <
> > > > >>>>>>> nizhikov@apache.org
> > > > >>>>>>>>> :
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Igniters, should we have a call for this topic?
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> 2 нояб. 2020 г., в 18:53, Pavel Tupitsyn <
> > > > >>>>>> ptupitsyn@apache.org
> > > > >>>>>>>>
> > > > >>>>>>>>>>>>> написал(а):
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> not intend to rewrite everything from scratch
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Every single test from Ignite 2.x should be moved to
> > > > >>>>> Ignite
> > > > >>>>>> 3
> > > > >>>>>>>>>>>>>>> regardless of how we choose to proceed.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Alexey, thank you for the explanation, this addresses
> > all
> > > > >>>>> of
> > > > >>>>>> my
> > > > >>>>>>>>>>> concerns.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> On Mon, Nov 2, 2020 at 6:43 PM Andrey Mashenkov <
> > > > >>>>>>>>>>>>> andrey.mashenkov@gmail.com>
> > > > >>>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Hi, Igniters.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> * AFAIU, we need a new repo if we want to apply
> > > > >>>> different
> > > > >>>>>>>>>> restrictions
> > > > >>>>>>>>>>>>> to
> > > > >>>>>>>>>>>>>>> pull requests,
> > > > >>>>>>>>>>>>>>> otherwise I see no difference for myself.
> > > > >>>>>>>>>>>>>>> E.g. make static analysis (do we have?), compile,
> > > > >>>> styles,
> > > > >>>>>> and
> > > > >>>>>>>>>> javadoc
> > > > >>>>>>>>>>>>>>> checks mandatory.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> I think that relaxed requirements here will lead to
> bad
> > > > >>>>>>> product
> > > > >>>>>>>>>>> quality.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> * Agree with Pavel, we should 'keep' integrations
> tests
> > > > >>>>>>> somehow.
> > > > >>>>>>>>>>>>>>> During active development tests will be broken most
> of
> > > > >>>>> time,
> > > > >>>>>>> so,
> > > > >>>>>>>>>>>>>>> I'd port them e.g. suite-by-suite once we will have a
> > > > >>>>> stable
> > > > >>>>>>> and
> > > > >>>>>>>>>>>>> featured
> > > > >>>>>>>>>>>>>>> environment to run them and of course make test's
> code
> > > > >>>>> clear
> > > > >>>>>>> and
> > > > >>>>>>>>>> avoid
> > > > >>>>>>>>>>>>>>> bad/non-relevant ones.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> * I like bottom-up approach.
> > > > >>>>>>>>>>>>>>> With it we could make a better framework. I mean
> clear
> > > > >>>>>>> component
> > > > >>>>>>>>>>>>> lifecycle,
> > > > >>>>>>>>>>>>>>> component wiring mechanics, general methods to
> approach
> > > > >>>>> core
> > > > >>>>>>>>>>> components
> > > > >>>>>>>>>>>>>>> such as exchange/communication
> > > > >>>>>>>>>>>>>>> to avoid code mess like we have in ExchangeFuture
> with
> > > > >>>> all
> > > > >>>>>>> these
> > > > >>>>>>>>>>> custom
> > > > >>>>>>>>>>>>>>> callbacks for each component, interfaces like
> > > > >>>>>>>>>>>>>>> PartitionsExchangeAware,
> IgniteChangeGlobalStateSupport
> > > > >>>>> and
> > > > >>>>>>>>>>>>>>> a pack of
> > > > >>>>>>>>>>>>>
> > > > >>>>>>
> start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected
> > > > >>>>>>>>>>>>>>> and so on in various unexpected places.
> > > > >>>>>>>>>>>>>>> Hope, we will be able to port most of the good code
> to
> > > > >>>> the
> > > > >>>>>> new
> > > > >>>>>>>>>>> framework
> > > > >>>>>>>>>>>>>>> version.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk <
> > > > >>>>>>>>>>>>>>> alexey.goncharuk@gmail.com>
> > > > >>>>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> Nikolay, Pavel,
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> Thanks for the feedback! First of all, I wanted to
> > > > >>>> stress
> > > > >>>>>>> that
> > > > >>>>>>>> I
> > > > >>>>>>>>> do
> > > > >>>>>>>>>>> not
> > > > >>>>>>>>>>>>>>>> intend to rewrite everything from scratch (I never
> > used
> > > > >>>>>> this
> > > > >>>>>>>>>> phrase).
> > > > >>>>>>>>>>>>>>> There
> > > > >>>>>>>>>>>>>>>> are significant parts of code that would be moved
> with
> > > > >>>>>>> minimal
> > > > >>>>>>>>>>>>>>>> modifications. Second, I never said that we will get
> > > > >>>> rid
> > > > >>>>> of
> > > > >>>>>>> the
> > > > >>>>>>>>> old
> > > > >>>>>>>>>>>>> tests
> > > > >>>>>>>>>>>>>>>> codebase. Every single test from Ignite 2.x should
> be
> > > > >>>>> moved
> > > > >>>>>>> to
> > > > >>>>>>>>>>> Ignite 3
> > > > >>>>>>>>>>>>>>>> regardless of how we choose to proceed.
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> My point is that for some parts of the code a clean
> > > > >>>>>> bottom-up
> > > > >>>>>>>>>>>>>>>> implementation will be cheaper in many ways. Let me
> > > > >>>> give
> > > > >>>>>> you
> > > > >>>>>>> a
> > > > >>>>>>>>> few
> > > > >>>>>>>>>>>>>>> concrete
> > > > >>>>>>>>>>>>>>>> examples:
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> - I think no one can object that we need a cleanly
> > > > >>>>>> separated
> > > > >>>>>>>>>>>>>>> persistence
> > > > >>>>>>>>>>>>>>>> layer for Ignite. There is a very raw draft IEP for
> > > > >>>> this
> > > > >>>>>>>>> already.
> > > > >>>>>>>>>> On
> > > > >>>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>> other hand, I think we also can agree that we need a
> > > > >>>>>>>> split-brain
> > > > >>>>>>>>>>>>>>>> resistant
> > > > >>>>>>>>>>>>>>>> replication protocol for caches. There is also an
> IEP
> > > > >>>>> for
> > > > >>>>>>>> this.
> > > > >>>>>>>>>>>>>>> Neither
> > > > >>>>>>>>>>>>>>>> of
> > > > >>>>>>>>>>>>>>>> the changes is a good fit for 2.x because they are
> > > > >>>>> likely
> > > > >>>>>> to
> > > > >>>>>>>>>>>>> introduce
> > > > >>>>>>>>>>>>>>>> breaking changes in the persistence layer,
> > > > >>>> configuration
> > > > >>>>>> and
> > > > >>>>>>>>>>>>> behavior.
> > > > >>>>>>>>>>>>>>>> Additionally, these components are now tightly
> > > > >>>> coupled,
> > > > >>>>> so
> > > > >>>>>>>> there
> > > > >>>>>>>>>> is
> > > > >>>>>>>>>>>>> no
> > > > >>>>>>>>>>>>>>>> way
> > > > >>>>>>>>>>>>>>>> these two changes can be implemented in parallel and
> > > > >>>>> then
> > > > >>>>>>>> merged
> > > > >>>>>>>>>>>>>>>> together
> > > > >>>>>>>>>>>>>>>> easily. So what we will end up with is having to
> > > > >>>>> implement
> > > > >>>>>>>> these
> > > > >>>>>>>>>>>>>>> changes
> > > > >>>>>>>>>>>>>>>> sequentially, fixing all existing tests twice, and
> > > > >>>>>>> essentially
> > > > >>>>>>>>>>>>>>> throwing
> > > > >>>>>>>>>>>>>>>> away half of the work done because the other part of
> > > > >>>> the
> > > > >>>>>>>> change
> > > > >>>>>>>>> is
> > > > >>>>>>>>>>>>>>>> re-implemented
> > > > >>>>>>>>>>>>>>>> - Similar example goes with getting rid of
> > > > >>>>>>>> IgniteInternalFuture
> > > > >>>>>>>>>> and
> > > > >>>>>>>>>>>>>>>> replacing it with CompletableFuture, and any other
> > > > >>>>> change
> > > > >>>>>>> that
> > > > >>>>>>>>>>>>> touches
> > > > >>>>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>> asynchronous part of the code.
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> Third, I do not see how this choice affects the UX
> of
> > > > >>>>>> Ignite.
> > > > >>>>>>>> The
> > > > >>>>>>>>>> end
> > > > >>>>>>>>>>>>>>> user
> > > > >>>>>>>>>>>>>>>> experience must be fixed in the IEP regardless of
> the
> > > > >>>>>>>> development
> > > > >>>>>>>>>>>>> process
> > > > >>>>>>>>>>>>>>>> and the fact that we have gaps in this area in
> Ignite
> > > > >>>> 2.x
> > > > >>>>>>> just
> > > > >>>>>>>>>>> confirms
> > > > >>>>>>>>>>>>>>>> that.
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> Pavel, agree that a repo/branch is a technicality, I
> > > > >>>>> guess
> > > > >>>>>> if
> > > > >>>>>>>>>>>>>>> reformulate,
> > > > >>>>>>>>>>>>>>>> my point is that we might agree to have a single
> > > > >>>>>> development
> > > > >>>>>>>>> master
> > > > >>>>>>>>>>>>>>> branch
> > > > >>>>>>>>>>>>>>>> with 'disassembled' end-to-end functionality for
> some
> > > > >>>>>> period
> > > > >>>>>>> of
> > > > >>>>>>>>>> time
> > > > >>>>>>>>>>> to
> > > > >>>>>>>>>>>>>>>> speed up development, and re-assemble the core
> > features
> > > > >>>>>> after
> > > > >>>>>>>>>> having
> > > > >>>>>>>>>>>>>>>> submodules tested independently.
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> Nikolay,
> > > > >>>>>>>>>>>>>>>>> We have many features that have to evolve.
> > > > >>>>>>>>>>>>>>>>> Snapshots, rebalance, tooling, tracing, zookeeper
> > > > >>>>> support,
> > > > >>>>>>>> etc.
> > > > >>>>>>>>>>>>>>>> This is not very specific. In the end, resources are
> > > > >>>>>> limited
> > > > >>>>>>>> and
> > > > >>>>>>>>> we
> > > > >>>>>>>>>>>>> will
> > > > >>>>>>>>>>>>>>>> not be able to drive both tracks simultaneously,
> > > > >>>>> especially
> > > > >>>>>>>>> after a
> > > > >>>>>>>>>>>>>>> couple
> > > > >>>>>>>>>>>>>>>> of features having been implemented for Ignite 3.0.
> If
> > > > >>>>>> there
> > > > >>>>>>>> are
> > > > >>>>>>>>>>> indeed
> > > > >>>>>>>>>>>>>>>> some major changes that we want to do in Ignite 2.x
> > > > >>>>> instead
> > > > >>>>>>> of
> > > > >>>>>>>>>>> putting
> > > > >>>>>>>>>>>>>>>> effort into 3.0 - let's discuss them. I am just not
> > > > >>>> aware
> > > > >>>>>> of
> > > > >>>>>>>> any,
> > > > >>>>>>>>>>>>> that's
> > > > >>>>>>>>>>>>>>>> why I am eager to move forward with Ignite 3.0.
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> We have bugs and issues that can be fixed in 2.x
> > > > >>>> without
> > > > >>>>>>>>> breaking
> > > > >>>>>>>>>>>>>>> backward
> > > > >>>>>>>>>>>>>>>> compatibility.
> > > > >>>>>>>>>>>>>>>>> We have many users who are happy with the 2.x with
> > all
> > > > >>>>>> it’s
> > > > >>>>>>>>>> issues.
> > > > >>>>>>>>>>>>>>>> These changes will be covered by end-to-end tests
> and
> > > > >>>>>>> migrated
> > > > >>>>>>>> to
> > > > >>>>>>>>>>>>> Ignite
> > > > >>>>>>>>>>>>>>>> 3.0, so I see no issues here.
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> Finally, Anton & Nikolay
> > > > >>>>>>>>>>>>>>>> I do not have an estimate for this simply because
> the
> > > > >>>>>>> activity
> > > > >>>>>>>> is
> > > > >>>>>>>>>>>>>>>> community-driven and it depends on the number of
> > people
> > > > >>>>>>> willing
> > > > >>>>>>>>> to
> > > > >>>>>>>>>>>>>>>> contribute. With the current pace, I would hope to
> > have
> > > > >>>>> an
> > > > >>>>>> RC
> > > > >>>>>>>> of
> > > > >>>>>>>>>>> Ignite
> > > > >>>>>>>>>>>>>>> 3.0
> > > > >>>>>>>>>>>>>>>> to be ready by the end of 2021. My gut feeling is
> that
> > > > >>>> by
> > > > >>>>>>>> moving
> > > > >>>>>>>>>> with
> > > > >>>>>>>>>>>>>>>> incremental changes, we will not be able to
> implement
> > > > >>>>> even
> > > > >>>>>>> half
> > > > >>>>>>>>> of
> > > > >>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>> wishlist by that time.
> > > > >>>>>>>>>>>>>>>> I doubt that releasing several major releases with
> > > > >>>>> breaking
> > > > >>>>>>>>> changes
> > > > >>>>>>>>>>>>> will
> > > > >>>>>>>>>>>>>>>> make Ignite users happy either because each upgrade
> > > > >>>> will
> > > > >>>>>> cost
> > > > >>>>>>>>>> Ignite
> > > > >>>>>>>>>>>>>>> users
> > > > >>>>>>>>>>>>>>>> money, so the fewer major versions we release, the
> > > > >>>>> better.
> > > > >>>>>>> Thus
> > > > >>>>>>>>> my
> > > > >>>>>>>>>>> wish
> > > > >>>>>>>>>>>>>>> to
> > > > >>>>>>>>>>>>>>>> include all breaking changes in one release.
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> I'll be now quiet for a while, let's see what other
> > > > >>>>>> community
> > > > >>>>>>>>>> members
> > > > >>>>>>>>>>>>>>>> think.
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <
> > > > >>>>>>>>> ptupitsyn@apache.org
> > > > >>>>>>>>>>> :
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> 1. Rewriting from scratch is never a good idea.
> > > > >>>>>>>>>>>>>>>>> We don't want to follow the path of Netscape and
> lose
> > > > >>>>> all
> > > > >>>>>>> our
> > > > >>>>>>>>>> users
> > > > >>>>>>>>>>>>>>>>> by the time we have a working 3.0 [1]
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> 2. Not sure about new repo - seems like some pain
> and
> > > > >>>> no
> > > > >>>>>>> gain,
> > > > >>>>>>>>>>> what's
> > > > >>>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>> problem with a branch?
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> 3. We should keep existing integration tests when
> > > > >>>>>> possible.
> > > > >>>>>>>>>>>>>>>>> We have accumulated a lot of edge case knowledge
> over
> > > > >>>>> the
> > > > >>>>>>>> years,
> > > > >>>>>>>>>>>>>>>>> it is not a good idea to send all of that down the
> > > > >>>>> drain.
> > > > >>>>>>>>>>>>>>>>> Yes, integration tests are slow, but they are the
> > most
> > > > >>>>>>>> valuable.
> > > > >>>>>>>>>>>>>>>>> I think we can move more stuff into nightly runs
> and
> > > > >>>>> have
> > > > >>>>>> a
> > > > >>>>>>>> fast
> > > > >>>>>>>>>> and
> > > > >>>>>>>>>>>>>>>> modern
> > > > >>>>>>>>>>>>>>>>> basic suite.
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> Alexey, you are much more familiar with the Ignite
> > > > >>>> core
> > > > >>>>>>>> codebase
> > > > >>>>>>>>>>> than
> > > > >>>>>>>>>>>>>>>> most
> > > > >>>>>>>>>>>>>>>>> of us,
> > > > >>>>>>>>>>>>>>>>> can you please explain in more detail which
> > particular
> > > > >>>>>>>> feature,
> > > > >>>>>>>>> in
> > > > >>>>>>>>>>>>> your
> > > > >>>>>>>>>>>>>>>>> opinion,
> > > > >>>>>>>>>>>>>>>>> mandates this "start from scratch" approach?
> > > > >>>>>>>>>>>>>>>>> Is it really not possible at all to follow a less
> > > > >>>>> radical
> > > > >>>>>>> way?
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> [1]
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>>
> > > > >>>>
> > > >
> >
> https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <
> > > > >>>>>>>>>> nizhikov@apache.org
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> Hello, Alexey.
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> I think that «rewriting from scratch» approach
> has a
> > > > >>>>> high
> > > > >>>>>>>> risk
> > > > >>>>>>>>> to
> > > > >>>>>>>>>>>>>>> make
> > > > >>>>>>>>>>>>>>>>> new
> > > > >>>>>>>>>>>>>>>>>> features unusable.
> > > > >>>>>>>>>>>>>>>>>> At the time Ignite2 was started no-one wants to do
> > > > >>>> bad
> > > > >>>>> UX
> > > > >>>>>>> or
> > > > >>>>>>>>> bad
> > > > >>>>>>>>>>>>>>>>> features.
> > > > >>>>>>>>>>>>>>>>>> Nevertheless, it happen.
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> I think we can avoid it with the Ignite3 and
> > > > >>>> successors
> > > > >>>>>> if
> > > > >>>>>>> we
> > > > >>>>>>>>>> will
> > > > >>>>>>>>>>>>>>> move
> > > > >>>>>>>>>>>>>>>>>> step by step without keeping backward
> compatibility
> > > > >>>>>>>>>>>>>>>>>> With the step by step approach, we can focus on
> each
> > > > >>>>>>>> component
> > > > >>>>>>>>>>>>>>>>> separately.
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> What new features are we planning to implement
> for
> > > > >>>>>> Ignite
> > > > >>>>>>>> 2.x?
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> We have many features that have to evolve.
> > > > >>>>>>>>>>>>>>>>>> Snapshots, rebalance, tooling, tracing, zookeeper
> > > > >>>>>> support,
> > > > >>>>>>>> etc.
> > > > >>>>>>>>>>>>>>>>>> We have bugs and issues that can be fixed in 2.x
> > > > >>>>> without
> > > > >>>>>>>>> breaking
> > > > >>>>>>>>>>>>>>>>> backward
> > > > >>>>>>>>>>>>>>>>>> compatibility.
> > > > >>>>>>>>>>>>>>>>>> We have many users who are happy with the 2.x with
> > > > >>>> all
> > > > >>>>>> it’s
> > > > >>>>>>>>>> issues.
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> 2 нояб. 2020 г., в 14:09, Anton Vinogradov <
> > > > >>>>>> av@apache.org
> > > > >>>>>>>>
> > > > >>>>>>>>>>>>>>>> написал(а):
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> Alexey,
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> Do we have any estimates of how fast we'll be
> able
> > > > >>>> to
> > > > >>>>>> gain
> > > > >>>>>>>>>>>>>>>>>> production-ready
> > > > >>>>>>>>>>>>>>>>>>> AI 3.0 in case of a "new repo" choice?
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk <
> > > > >>>>>>>>>>>>>>>>>> alexey.goncharuk@gmail.com>
> > > > >>>>>>>>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> Nikolay,
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> What new features are we planning to implement
> for
> > > > >>>>>> Ignite
> > > > >>>>>>>>> 2.x?
> > > > >>>>>>>>>> I
> > > > >>>>>>>>>>>>>>>> think
> > > > >>>>>>>>>>>>>>>>>> once
> > > > >>>>>>>>>>>>>>>>>>>> we commence working on Ignite 3.0, we should
> > > > >>>>> gradually
> > > > >>>>>>>> cease
> > > > >>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>> activity
> > > > >>>>>>>>>>>>>>>>>>>> on Ignite 2.x to mere bugfixes because such
> > > > >>>> parallel
> > > > >>>>>>>>>> development
> > > > >>>>>>>>>>>>>>>> will
> > > > >>>>>>>>>>>>>>>>> be
> > > > >>>>>>>>>>>>>>>>>>>> overwhelming regardless of how we choose to
> > > > >>>> proceed.
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <
> > > > >>>>>>>>>>> nizhikov@apache.org
> > > > >>>>>>>>>>>>>>>> :
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> To be clear:
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>> I would suggest creating a new repository for
> > > > >>>>> Ignite
> > > > >>>>>>> 3.0
> > > > >>>>>>>>>>>>>>>> (perhaps, a
> > > > >>>>>>>>>>>>>>>>>>>> new
> > > > >>>>>>>>>>>>>>>>>>>>> clean branch, but a new repo looks nicer to me)
> > > > >>>> and
> > > > >>>>> a
> > > > >>>>>>> new
> > > > >>>>>>>>>> Ignite
> > > > >>>>>>>>>>>>>>>> 3.0
> > > > >>>>>>>>>>>>>>>>>>>>> TeamCity project.
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> +1 for new Team City project.
> > > > >>>>>>>>>>>>>>>>>>>>> +1 for new branch for Ignite3.
> > > > >>>>>>>>>>>>>>>>>>>>> -1 for new repo.
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <
> > > > >>>>>>>>>>>>>>> NIzhikov.dev@gmail.com
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> написал(а):
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>> Hello, Alexey.
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>> I think it will hurt our project more than
> help.
> > > > >>>>>>>>>>>>>>>>>>>>>> Developing new features for 2 separate
> branches
> > > > >>>>> with
> > > > >>>>>>> the
> > > > >>>>>>>>>>>>>>> different
> > > > >>>>>>>>>>>>>>>>>> APIs
> > > > >>>>>>>>>>>>>>>>>>>>> and internal structure is overwhelming
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>> Maybe we should relax a bit requirements for
> > > > >>>>> Ignite3?
> > > > >>>>>>>>>>>>>>>>>>>>>> Maybe we should move step by step and make
> > > > >>>> Ignite3
> > > > >>>>>> with
> > > > >>>>>>>> new
> > > > >>>>>>>>>>>>>>>>>>>>> configuration than Ignite4 with new
> transactions,
> > > > >>>>> etc?
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <
> > > > >>>>>>>>>>>>>>>>>>>> alexey.goncharuk@gmail.com>
> > > > >>>>>>>>>>>>>>>>>>>>> написал(а):
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>> Igniters,
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>> I wanted to pitch a rather radical idea
> > > > >>>> regarding
> > > > >>>>>> the
> > > > >>>>>>>>> Ignite
> > > > >>>>>>>>>>>>>>> 3.0
> > > > >>>>>>>>>>>>>>>>>>>>>>> development which has occurred to me some
> time
> > > > >>>>> ago.
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>> We already have several IEPs targeted to
> Ignite
> > > > >>>>> 3.0
> > > > >>>>>>>> which
> > > > >>>>>>>>>>> imply
> > > > >>>>>>>>>>>>>>>>> major
> > > > >>>>>>>>>>>>>>>>>>>>>>> changes to the codebase (the change in
> > > > >>>> replication
> > > > >>>>>>>>> protocol
> > > > >>>>>>>>>>> and
> > > > >>>>>>>>>>>>>>>>> thus
> > > > >>>>>>>>>>>>>>>>>>>>>>> transactions, change in binary format,
> updated
> > > > >>>>>>>>> metastorage,
> > > > >>>>>>>>>>>>>>> etc).
> > > > >>>>>>>>>>>>>>>>> We
> > > > >>>>>>>>>>>>>>>>>>>>> also
> > > > >>>>>>>>>>>>>>>>>>>>>>> planned significant changes in public APIs:
> > > > >>>>>>>> configuration
> > > > >>>>>>>>>>>>>>> format
> > > > >>>>>>>>>>>>>>>>>>>> change,
> > > > >>>>>>>>>>>>>>>>>>>>>>> improvements in cache APIs, SQL APIs,
> > > > >>>> transaction
> > > > >>>>>> mode
> > > > >>>>>>>>>> rework.
> > > > >>>>>>>>>>>>>>>> The
> > > > >>>>>>>>>>>>>>>>>>>>> wishlist
> > > > >>>>>>>>>>>>>>>>>>>>>>> of changes for Ignite 3.0 is huge.
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>> So, I was wondering whether it makes sense to
> > > > >>>> try
> > > > >>>>> to
> > > > >>>>>>>>> change
> > > > >>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>> old
> > > > >>>>>>>>>>>>>>>>>>>>>>> codebase, or start with a new baseline and
> move
> > > > >>>>> old
> > > > >>>>>>>> pieces
> > > > >>>>>>>>>> of
> > > > >>>>>>>>>>>>>>>> code
> > > > >>>>>>>>>>>>>>>>>>>> that
> > > > >>>>>>>>>>>>>>>>>>>>> do
> > > > >>>>>>>>>>>>>>>>>>>>>>> not require significant rework. Personally, I
> > > > >>>>> would
> > > > >>>>>> go
> > > > >>>>>>>>> with
> > > > >>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>> second
> > > > >>>>>>>>>>>>>>>>>>>>>>> option for the following reasons:
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>> - We have a chance to shift the development
> > > > >>>>> paradigm
> > > > >>>>>>> in
> > > > >>>>>>>>> the
> > > > >>>>>>>>>>>>>>>> project
> > > > >>>>>>>>>>>>>>>>>>>> and
> > > > >>>>>>>>>>>>>>>>>>>>>>> introduce the practice of true unit-tests. In
> > > > >>>> the
> > > > >>>>>> new
> > > > >>>>>>>>>> baseline
> > > > >>>>>>>>>>>>>>> at
> > > > >>>>>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>>>>>>> beginning there will be no ability to run an
> > > > >>>>>>> end-to-end
> > > > >>>>>>>>>>>>>>> scenario,
> > > > >>>>>>>>>>>>>>>>>>>> thus
> > > > >>>>>>>>>>>>>>>>>>>>> we
> > > > >>>>>>>>>>>>>>>>>>>>>>> will be forced to write unit-tests. So far,
> > such
> > > > >>>>>>>> practice
> > > > >>>>>>>>>> was
> > > > >>>>>>>>>>>>>>>> hard
> > > > >>>>>>>>>>>>>>>>> to
> > > > >>>>>>>>>>>>>>>>>>>>>>> implement because of tight coupling between
> > > > >>>> Ignite
> > > > >>>>>>>>>> components
> > > > >>>>>>>>>>>>>>> and
> > > > >>>>>>>>>>>>>>>>>>>>> inability
> > > > >>>>>>>>>>>>>>>>>>>>>>> to instantiate components without an instance
> > of
> > > > >>>>>>>>>>> KernalContext.
> > > > >>>>>>>>>>>>>>>> For
> > > > >>>>>>>>>>>>>>>>>>>>>>> example, we should be able to thoroughly test
> > > > >>>>>> internal
> > > > >>>>>>>>>>>>>>>> primitives,
> > > > >>>>>>>>>>>>>>>>>>>>> such as
> > > > >>>>>>>>>>>>>>>>>>>>>>> replication protocol (without actual
> > > > >>>>> communication),
> > > > >>>>>>>>>>>>>>> distributed
> > > > >>>>>>>>>>>>>>>>>>>>>>> metastorage contracts, persistence layer,
> etc.
> > > > >>>>>>>>>>>>>>>>>>>>>>> - We will significantly reduce the
> development
> > > > >>>>> cycle
> > > > >>>>>>> in
> > > > >>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>> beginning
> > > > >>>>>>>>>>>>>>>>>>>>>>> (right now the RunAll takes two hours of
> > > > >>>>>> astronomical
> > > > >>>>>>>> time
> > > > >>>>>>>>>>> with
> > > > >>>>>>>>>>>>>>>>> empty
> > > > >>>>>>>>>>>>>>>>>>>>> TC;
> > > > >>>>>>>>>>>>>>>>>>>>>>> in the new approach developer will be able to
> > > > >>>> run
> > > > >>>>>> ALL
> > > > >>>>>>>>> tests
> > > > >>>>>>>>>>>>>>>> locally
> > > > >>>>>>>>>>>>>>>>>>>> in
> > > > >>>>>>>>>>>>>>>>>>>>> a
> > > > >>>>>>>>>>>>>>>>>>>>>>> matter of minutes)
> > > > >>>>>>>>>>>>>>>>>>>>>>> - We can get rid of TC bot and enforce green
> TC
> > > > >>>> by
> > > > >>>>>>>>>> integrating
> > > > >>>>>>>>>>>>>>> TC
> > > > >>>>>>>>>>>>>>>>>>>> build
> > > > >>>>>>>>>>>>>>>>>>>>>>> results with GitHub PRs (the same way Travis
> is
> > > > >>>>>>>> currently
> > > > >>>>>>>>>>>>>>>>> integrated
> > > > >>>>>>>>>>>>>>>>>>>>> to PR
> > > > >>>>>>>>>>>>>>>>>>>>>>> check). We should restrict PR merge without a
> > TC
> > > > >>>>>> check
> > > > >>>>>>>>>>>>>>>>>>>>>>> - We will still have to re-write all tests,
> but
> > > > >>>>> only
> > > > >>>>>>>> once.
> > > > >>>>>>>>>> If
> > > > >>>>>>>>>>>>>>> we
> > > > >>>>>>>>>>>>>>>>> try
> > > > >>>>>>>>>>>>>>>>>>>> to
> > > > >>>>>>>>>>>>>>>>>>>>>>> modify the old codebase, we would need to
> > modify
> > > > >>>>> all
> > > > >>>>>>> the
> > > > >>>>>>>>>> tests
> > > > >>>>>>>>>>>>>>>> for
> > > > >>>>>>>>>>>>>>>>>>>>> every
> > > > >>>>>>>>>>>>>>>>>>>>>>> major change (public API change,
> configuration
> > > > >>>>>> change)
> > > > >>>>>>>>>>>>>>>>>>>>>>> - We will have fewer conflicts when working
> > > > >>>>>> together.
> > > > >>>>>>>> For
> > > > >>>>>>>>>>>>>>>> example,
> > > > >>>>>>>>>>>>>>>>> I
> > > > >>>>>>>>>>>>>>>>>>>>>>> cannot imagine how one would merge two
> changes
> > > > >>>> of
> > > > >>>>>>>> getting
> > > > >>>>>>>>>> rid
> > > > >>>>>>>>>>>>>>> of
> > > > >>>>>>>>>>>>>>>>>>>>>>> IgniteFuture and changes in replication
> > > > >>>> protocol,
> > > > >>>>>> for
> > > > >>>>>>>>>> example
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>> Technically, I would suggest creating a new
> > > > >>>>>> repository
> > > > >>>>>>>> for
> > > > >>>>>>>>>>>>>>> Ignite
> > > > >>>>>>>>>>>>>>>>> 3.0
> > > > >>>>>>>>>>>>>>>>>>>>>>> (perhaps, a new clean branch, but a new repo
> > > > >>>> looks
> > > > >>>>>>> nicer
> > > > >>>>>>>>> to
> > > > >>>>>>>>>>> me)
> > > > >>>>>>>>>>>>>>>>> and a
> > > > >>>>>>>>>>>>>>>>>>>>> new
> > > > >>>>>>>>>>>>>>>>>>>>>>> Ignite 3.0 TeamCity project.
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>> While it may seem quite radical, I do believe
> > > > >>>> that
> > > > >>>>>>> this
> > > > >>>>>>>>>>>>>>> approach
> > > > >>>>>>>>>>>>>>>>> will
> > > > >>>>>>>>>>>>>>>>>>>>> give
> > > > >>>>>>>>>>>>>>>>>>>>>>> us more benefits than trying to make such
> major
> > > > >>>>>>> changes
> > > > >>>>>>>> in
> > > > >>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>>>> existing
> > > > >>>>>>>>>>>>>>>>>>>>>>> codebase. If needed, let's schedule a discord
> > > > >>>> chat
> > > > >>>>>>> like
> > > > >>>>>>>>>> before
> > > > >>>>>>>>>>>>>>> to
> > > > >>>>>>>>>>>>>>>>>>>>> discuss
> > > > >>>>>>>>>>>>>>>>>>>>>>> this.
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>> WDYT?
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> --
> > > > >>>>>>>>>>>>>>> Best regards,
> > > > >>>>>>>>>>>>>>> Andrey V. Mashenkov
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>>
> > > > >>>>
> > > > >>>
> > > > >>>
> > > > >>> --
> > > > >>>
> > > > >>> Best regards,
> > > > >>> Alexei Scherbakov
> > > > >
> > > >
> > > >
> > > >
> > >
> > > --
> > > Sincerely yours, Ivan Daschinskiy
> >
>
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Alexey Zinoviev <za...@gmail.com>.
I'm -1 for creating a new repo.
Also I support Maxim's plan for 3.0
пт, 13 нояб. 2020 г. в 15:50, Maxim Muzafarov <mm...@apache.org>:
> Val,
>
>
> Why *creating a new repo* is the main point we faced with? Would it be
> better to discuss the components design approach and scope management
> first suggested by Sergey Chugunov? I doubt that new repo will solve
> move us forward.
>
> Currently, I'm -1 to create a new repo with the inputs above.
>
> In addition to Nikolay's answer I see the following drawbacks of
> creating new repo:
> - we have very few positive examples of finalizing really huge
> improvements to *production-ready* states the others remains
> incomplete (MVCC, Calcite, Zookeeper, Tracing, Thread per Partition,
> etc)
> - AFAIK, the Native Persistence took a very long period of
> stabilization even after it has been developed (we must take it into
> account for developing new features like IEP-61)
> - feature development for a long period of time (like 3.0) without any
> releases will lead to all these changes became obsolete at the moment
> of release (AFAIK the 2.8 which released a year ago still has no big
> deployments)
> - human resources -- some of the Igniters may lose their interest for
> 3.0 during development, some of them may switch to different projects,
> etc.
> - do we all estimating the scope of 3.0 correct? The 2.8 release took 1.5
> years.
>
> Have I missed something?
>
>
> I suggest the following plan:
>
> - initiate 3.0 development in the master branch (after 2.10 release
> change version to 3.0-SNAPSHOT instead of 2.11-SNAPSHOT)
> - cleanup and collapse all the current APIs (see To Be Removed List
> For Discussion on Apache Ignite 3.0 Wishlist)
> - reduce the scope for 3.0 even more. I suggest focusing on two
> things: Calcite + Schema-first approach
> - create feature branches for proposed IEPs (for 3.0 only)
> - create the release road map (allocate e.g. IEP-61 to 4.0 etc.)
>
> On Fri, 13 Nov 2020 at 14:03, Ivan Daschinsky <iv...@gmail.com> wrote:
> >
> > >> b. Implement IEP-61 - Common Replication Infrastructure
> > I suppose, that this is the main cause of the current discussion.
> > I hardly believe that this activity can be done without at least
> creating a
> > completely new branch.
> >
> > пт, 13 нояб. 2020 г. в 11:12, Nikolay Izhikov <ni...@apache.org>:
> >
> > > My suggestion:
> > >
> > > 1. Reduce Ignite3 scope to the following:
> > > a. Delete all deprecated API and support of obsolete internal
> > > protocols.
> > > b. Implement IEP-61 - Common Replication Infrastructure
> > > c. Implement new Ignite management tool ignitectl as suggested
> > > during Ignite3 discussion.
> > >
> > > 2. Implement and release following improvements like transactions,
> Calcite
> > > based SQL, etc in the ongoing releases - Ignite 4, 5, 6
> > >
> > > My concern against separate Ignite 3 repo is the following:
> > >
> > > 1. We spread community to the two very separated part - Ignite3
> developers
> > > and Ignite2 maintainers. believe it’s bad for our community.
> > > That can lead to the situation when we don’t fix critical or
> > > blocker issueds «because they will not exists in Ignite3»
> > > That will lead to the solutions never reviewed or reviewed
> poorly.
> > >
> > > 2. It seems for me that current scope of Ignite3 is too big to be
> > > implemented in any reasonable time.
> > >
> > >
> > > > 13 нояб. 2020 г., в 10:57, Nikolay Izhikov <NI...@gmail.com>
> > > написал(а):
> > > >
> > > > Hello, Valentin.
> > > >
> > > >> Nikolay, Maxim, are you OK with this route?
> > > >
> > > > -1 to have another repo for Ignite3 development.
> > > >
> > > >> 13 нояб. 2020 г., в 03:04, Valentin Kulichenko <
> > > valentin.kulichenko@gmail.com> написал(а):
> > > >>
> > > >> Folks,
> > > >>
> > > >> We already have multiple IEPs for Ignite 3.0, and as far as I know,
> > > there are contributors that would like to work on them (or probably
> already
> > > started). That said, we should make a decision as soon as possible.
> > > >>
> > > >> At this point, it doesn't seem that there are any strong objections
> to
> > > the technical side of things. So I would suggest the following:
> > > >>
> > > >> 1. Proceed with Alexey's approach to the development process, as it
> > > seems to be the best (in my opinion - the only) way to address all the
> > > technical concerns and issues expressed in the thread. We'll start by
> > > creating a new repo and a new TC project.
> > > >> 2. Start a separate discussion around transparency. If there are any
> > > changes we need to make to our contributor guidelines, I am happy to
> talk
> > > them through, but I don't think it's reasonable to delay feature
> > > development because of this. In the short term, I will make sure that
> > > everything that happens within the new repo is as open to the
> community as
> > > possible.
> > > >>
> > > >> Nikolay, Maxim, are you OK with this route?
> > > >>
> > > >> -Val
> > > >>
> > > >> On Tue, Nov 10, 2020 at 4:55 PM Valentin Kulichenko <
> > > valentin.kulichenko@gmail.com> wrote:
> > > >> Maxim,
> > > >>
> > > >> 2.x and 3.x will have to coexist for some time - I don't see how we
> can
> > > avoid this considering the set of proposed changes. That said, we
> > > effectively will need to have two "masters" - one for each major
> version.
> > > Master for 3.x can technically be a branch in the existing repo, but
> having
> > > a separate repo seems cleaner, simply because it will not be a
> "branch" in
> > > the traditional sense.
> > > >>
> > > >> Note that the new repo will still be under the Apache org, with the
> > > same set of committers, managed by the community, etc. All the
> development
> > > happening for 3.0 must follow the rules that we currently have (if
> > > anything, it's an opportunity to improve those rules).
> > > >>
> > > >> As I said during the call on Friday, I strongly believe that if
> there
> > > is a transparency issue, it will exist regardless of the approach we
> choose
> > > for 3.0. If community members develop without IEPs or public
> discussions,
> > > this will happen for both 2.x and 3.x unless we address this
> separately. I
> > > don't see how this is related to Alexey's suggestion, which targets
> > > *technical* issues with the product more than anything else. This a
> way to
> > > achieve better modularity, introduce better coverage with unit tests,
> > > reduce conflicts during development, etc.
> > > >>
> > > >> Coming back to transparency, let's identify the issues and fix
> them. It
> > > probably makes sense to have a separate discussion on this topic.
> > > >>
> > > >> -Val
> > > >>
> > > >> On Tue, Nov 10, 2020 at 1:05 PM Maxim Muzafarov <mm...@apache.org>
> > > wrote:
> > > >> Sergey,
> > > >>
> > > >>
> > > >> Your summary makes sense to me.
> > > >>
> > > >> However, how we come up from *Development transparency* to *create a
> > > >> separate public repository dedicated for 3.0*? For me *development
> > > >> transparency* is about making changes in the master branch. These
> > > >> changes will definitely be seen by all the Ignite developers.
> > > >>
> > > >> A dedicated public repository is technically public and visible for
> > > >> everyone, but it allows development without IEPs, without public
> > > >> discussion (since all the code changes are not related to the master
> > > >> branch) it also allows a large number of assumptions and deviations
> > > >> (like code-style violations). It also not about *development
> > > >> transparency* since developers which are working on 3.0 is only a
> > > >> subset of all Ignite developers which may continue working on 2.x.
> For
> > > >> me, this would be a huge step backwards.
> > > >>
> > > >> Ignite veterans should remember how long the branch stabilization
> took
> > > >> for the 2.x version with the PDS.
> > > >>
> > > >>
> > > >> I think each breaking change should be passed through the master
> branch.
> > > >>
> > > >> On Tue, 10 Nov 2020 at 22:18, Alexei Scherbakov
> > > >> <al...@gmail.com> wrote:
> > > >>>
> > > >>> Makes sense to me.
> > > >>>
> > > >>> вт, 10 нояб. 2020 г. в 18:47, Sergey Chugunov <
> > > sergey.chugunov@gmail.com>:
> > > >>>
> > > >>>> Igniters,
> > > >>>>
> > > >>>> I thought over Friday meeting ideas and concerns and summarized
> them
> > > in
> > > >>>> these three points:
> > > >>>>
> > > >>>>
> > > >>>> 1. *Components design unification approach.* New proposed
> components
> > > >>>> will be developed by different contributors, but they need to be
> > > unified
> > > >>>> and should integrate with each other easily. To ensure that I
> > > suggest
> > > >>>> calling an architecture group that will create design guidelines
> > > for all
> > > >>>> components and high-level overview of overall architecture. How
> > > code is
> > > >>>> split into components, what are component boundaries, how
> component
> > > >>>> lifecycle works and what are its interfaces - all these and
> other
> > > >>>> questions
> > > >>>> should be covered.
> > > >>>>
> > > >>>> 2. *Scope management.* Apache 3.0 should be implemented within a
> > > >>>> reasonable time, so we need some procedure to decide whether a
> > > >>>> particular
> > > >>>> feature should be dropped from the scope of 3.0 and postponed
> to 3.1
> > > >>>> release. To do so I suggest to range all features by two
> parameters:
> > > >>>> criticality for 3.0 and amount of breaking changes. 3.0 scope
> should
> > > >>>> include features of high criticality AND features with a big
> amount
> > > of
> > > >>>> breaking changes. All other features can be made optional.
> > > >>>>
> > > >>>> 3. *Development transparency.* Development of all components
> should
> > > be
> > > >>>> made as transparent for everyone as possible. Any contributor
> > > should be
> > > >>>> able to look over any component at any stage of development. To
> > > achieve
> > > >>>> this I suggest to create a separate public repository dedicated
> for
> > > 3.0
> > > >>>> development. It will make the code available for everyone but
> when
> > > >>>> development of 3.0 is done we won't loose any stars of our
> current
> > > >>>> repository as we merge dev repo into main one and drop dev.
> > > >>>>
> > > >>>> Do these ideas make sense to you? Are there any concerns not
> covered
> > > by
> > > >>>> these suggestions?
> > > >>>>
> > > >>>> On Fri, Nov 6, 2020 at 7:36 PM Kseniya Romanova <
> > > romanova.ks.spb@gmail.com
> > > >>>>>
> > > >>>> wrote:
> > > >>>>
> > > >>>>> Here are the slides from Alexey Goncharuk. Let's think this over
> and
> > > >>>>> continue on Monday:
> > > >>>>>
> > > >>>>>
> > > >>>>
> > >
> https://go.gridgain.com/rs/491-TWR-806/images/Ignite_3_Plans_and_development_process.pdf
> > > >>>>>
> > > >>>>> чт, 5 нояб. 2020 г. в 11:13, Anton Vinogradov <av...@apache.org>:
> > > >>>>>
> > > >>>>>> Folks,
> > > >>>>>>
> > > >>>>>> Should we perform cleanup work before (r)evolutional changes?
> > > >>>>>> My huge proposal is to get rid of things which we don't need
> anyway
> > > >>>>>> - local caches,
> > > >>>>>> - strange tx modes,
> > > >>>>>> - code overcomplexity because of RollingUpgrade feature never
> > > attended
> > > >>>> at
> > > >>>>>> AI,
> > > >>>>>> - etc,
> > > >>>>>> before choosing the way.
> > > >>>>>>
> > > >>>>>> On Tue, Nov 3, 2020 at 3:31 PM Valentin Kulichenko <
> > > >>>>>> valentin.kulichenko@gmail.com> wrote:
> > > >>>>>>
> > > >>>>>>> Ksenia, thanks for scheduling this on such short notice!
> > > >>>>>>>
> > > >>>>>>> As for the original topic, I do support Alexey's idea. We're
> not
> > > >>>> going
> > > >>>>> to
> > > >>>>>>> rewrite anything from scratch, as most of the components are
> going
> > > to
> > > >>>>> be
> > > >>>>>>> moved as-is or with minimal modifications. However, the changes
> > > that
> > > >>>>> are
> > > >>>>>>> proposed imply serious rework of the core parts of the code,
> which
> > > >>>> are
> > > >>>>>> not
> > > >>>>>>> properly decoupled from each other and from other parts. This
> makes
> > > >>>> the
> > > >>>>>>> incremental approach borderline impossible. Developing in a new
> > > repo,
> > > >>>>>>> however, addresses this concern. As a bonus, we can also
> refactor
> > > the
> > > >>>>>> code,
> > > >>>>>>> introduce better decoupling, get rid of kernel context, and
> develop
> > > >>>>> unit
> > > >>>>>>> tests (finally!).
> > > >>>>>>>
> > > >>>>>>> Basically, this proposal only affects the *process*, not the
> set of
> > > >>>>>> changes
> > > >>>>>>> we had discussed before. Ignite 3.0 is our unique chance to
> make
> > > >>>> things
> > > >>>>>>> right.
> > > >>>>>>>
> > > >>>>>>> -Val
> > > >>>>>>>
> > > >>>>>>> On Tue, Nov 3, 2020 at 3:06 AM Kseniya Romanova <
> > > >>>>>> romanova.ks.spb@gmail.com
> > > >>>>>>>>
> > > >>>>>>> wrote:
> > > >>>>>>>
> > > >>>>>>>> Pavel, all the interesting points will be anyway published
> here in
> > > >>>>>>> English
> > > >>>>>>>> (as the principal "if it's not on devlist it doesn't
> happened" is
> > > >>>>> still
> > > >>>>>>>> relevant). This is just a quick call for a group of
> developers.
> > > >>>> Later
> > > >>>>>> we
> > > >>>>>>>> can do a separate presentation of idea and discussion in
> English
> > > as
> > > >>>>> we
> > > >>>>>>> did
> > > >>>>>>>> for the Ignite 3.0 draft of changes.
> > > >>>>>>>>
> > > >>>>>>>> вт, 3 нояб. 2020 г. в 13:52, Pavel Tupitsyn <
> ptupitsyn@apache.org
> > > >>>>> :
> > > >>>>>>>>
> > > >>>>>>>>> Kseniya,
> > > >>>>>>>>>
> > > >>>>>>>>> Thanks for scheduling this call.
> > > >>>>>>>>> Do you think we can switch to English if non-Russian speaking
> > > >>>>>> community
> > > >>>>>>>>> members decide to join?
> > > >>>>>>>>>
> > > >>>>>>>>> On Tue, Nov 3, 2020 at 1:32 PM Kseniya Romanova <
> > > >>>>>>>> romanova.ks.spb@gmail.com
> > > >>>>>>>>>>
> > > >>>>>>>>> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>>> Let's do this community discussion open. Here's the link on
> > > >>>> zoom
> > > >>>>>> call
> > > >>>>>>>> in
> > > >>>>>>>>>> Russian for Friday 6 PM:
> > > >>>>>>>>>>
> > > >>>>>>
> > > https://www.meetup.com/Moscow-Apache-Ignite-Meetup/events/274360378/
> > > >>>>>>>>>>
> > > >>>>>>>>>> вт, 3 нояб. 2020 г. в 12:49, Nikolay Izhikov <
> > > >>>>> nizhikov@apache.org
> > > >>>>>>> :
> > > >>>>>>>>>>
> > > >>>>>>>>>>> Time works for me.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> 3 нояб. 2020 г., в 12:40, Alexey Goncharuk <
> > > >>>>>>>>> alexey.goncharuk@gmail.com
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> написал(а):
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Nikolay,
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> I am up for the call. I will try to explain my reasoning
> in
> > > >>>>>>> greater
> > > >>>>>>>>>>> detail
> > > >>>>>>>>>>>> and will be glad to hear the concerns. Will this Friday,
> > > >>>> Nov
> > > >>>>>> 6th,
> > > >>>>>>>>> work?
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> вт, 3 нояб. 2020 г. в 10:09, Nikolay Izhikov <
> > > >>>>>>> nizhikov@apache.org
> > > >>>>>>>>> :
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>> Igniters, should we have a call for this topic?
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> 2 нояб. 2020 г., в 18:53, Pavel Tupitsyn <
> > > >>>>>> ptupitsyn@apache.org
> > > >>>>>>>>
> > > >>>>>>>>>>>>> написал(а):
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> not intend to rewrite everything from scratch
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Every single test from Ignite 2.x should be moved to
> > > >>>>> Ignite
> > > >>>>>> 3
> > > >>>>>>>>>>>>>>> regardless of how we choose to proceed.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Alexey, thank you for the explanation, this addresses
> all
> > > >>>>> of
> > > >>>>>> my
> > > >>>>>>>>>>> concerns.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> On Mon, Nov 2, 2020 at 6:43 PM Andrey Mashenkov <
> > > >>>>>>>>>>>>> andrey.mashenkov@gmail.com>
> > > >>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Hi, Igniters.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> * AFAIU, we need a new repo if we want to apply
> > > >>>> different
> > > >>>>>>>>>> restrictions
> > > >>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>> pull requests,
> > > >>>>>>>>>>>>>>> otherwise I see no difference for myself.
> > > >>>>>>>>>>>>>>> E.g. make static analysis (do we have?), compile,
> > > >>>> styles,
> > > >>>>>> and
> > > >>>>>>>>>> javadoc
> > > >>>>>>>>>>>>>>> checks mandatory.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> I think that relaxed requirements here will lead to bad
> > > >>>>>>> product
> > > >>>>>>>>>>> quality.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> * Agree with Pavel, we should 'keep' integrations tests
> > > >>>>>>> somehow.
> > > >>>>>>>>>>>>>>> During active development tests will be broken most of
> > > >>>>> time,
> > > >>>>>>> so,
> > > >>>>>>>>>>>>>>> I'd port them e.g. suite-by-suite once we will have a
> > > >>>>> stable
> > > >>>>>>> and
> > > >>>>>>>>>>>>> featured
> > > >>>>>>>>>>>>>>> environment to run them and of course make test's code
> > > >>>>> clear
> > > >>>>>>> and
> > > >>>>>>>>>> avoid
> > > >>>>>>>>>>>>>>> bad/non-relevant ones.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> * I like bottom-up approach.
> > > >>>>>>>>>>>>>>> With it we could make a better framework. I mean clear
> > > >>>>>>> component
> > > >>>>>>>>>>>>> lifecycle,
> > > >>>>>>>>>>>>>>> component wiring mechanics, general methods to approach
> > > >>>>> core
> > > >>>>>>>>>>> components
> > > >>>>>>>>>>>>>>> such as exchange/communication
> > > >>>>>>>>>>>>>>> to avoid code mess like we have in ExchangeFuture with
> > > >>>> all
> > > >>>>>>> these
> > > >>>>>>>>>>> custom
> > > >>>>>>>>>>>>>>> callbacks for each component, interfaces like
> > > >>>>>>>>>>>>>>> PartitionsExchangeAware, IgniteChangeGlobalStateSupport
> > > >>>>> and
> > > >>>>>>>>>>>>>>> a pack of
> > > >>>>>>>>>>>>>
> > > >>>>>> start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected
> > > >>>>>>>>>>>>>>> and so on in various unexpected places.
> > > >>>>>>>>>>>>>>> Hope, we will be able to port most of the good code to
> > > >>>> the
> > > >>>>>> new
> > > >>>>>>>>>>> framework
> > > >>>>>>>>>>>>>>> version.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk <
> > > >>>>>>>>>>>>>>> alexey.goncharuk@gmail.com>
> > > >>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Nikolay, Pavel,
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Thanks for the feedback! First of all, I wanted to
> > > >>>> stress
> > > >>>>>>> that
> > > >>>>>>>> I
> > > >>>>>>>>> do
> > > >>>>>>>>>>> not
> > > >>>>>>>>>>>>>>>> intend to rewrite everything from scratch (I never
> used
> > > >>>>>> this
> > > >>>>>>>>>> phrase).
> > > >>>>>>>>>>>>>>> There
> > > >>>>>>>>>>>>>>>> are significant parts of code that would be moved with
> > > >>>>>>> minimal
> > > >>>>>>>>>>>>>>>> modifications. Second, I never said that we will get
> > > >>>> rid
> > > >>>>> of
> > > >>>>>>> the
> > > >>>>>>>>> old
> > > >>>>>>>>>>>>> tests
> > > >>>>>>>>>>>>>>>> codebase. Every single test from Ignite 2.x should be
> > > >>>>> moved
> > > >>>>>>> to
> > > >>>>>>>>>>> Ignite 3
> > > >>>>>>>>>>>>>>>> regardless of how we choose to proceed.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> My point is that for some parts of the code a clean
> > > >>>>>> bottom-up
> > > >>>>>>>>>>>>>>>> implementation will be cheaper in many ways. Let me
> > > >>>> give
> > > >>>>>> you
> > > >>>>>>> a
> > > >>>>>>>>> few
> > > >>>>>>>>>>>>>>> concrete
> > > >>>>>>>>>>>>>>>> examples:
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> - I think no one can object that we need a cleanly
> > > >>>>>> separated
> > > >>>>>>>>>>>>>>> persistence
> > > >>>>>>>>>>>>>>>> layer for Ignite. There is a very raw draft IEP for
> > > >>>> this
> > > >>>>>>>>> already.
> > > >>>>>>>>>> On
> > > >>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>> other hand, I think we also can agree that we need a
> > > >>>>>>>> split-brain
> > > >>>>>>>>>>>>>>>> resistant
> > > >>>>>>>>>>>>>>>> replication protocol for caches. There is also an IEP
> > > >>>>> for
> > > >>>>>>>> this.
> > > >>>>>>>>>>>>>>> Neither
> > > >>>>>>>>>>>>>>>> of
> > > >>>>>>>>>>>>>>>> the changes is a good fit for 2.x because they are
> > > >>>>> likely
> > > >>>>>> to
> > > >>>>>>>>>>>>> introduce
> > > >>>>>>>>>>>>>>>> breaking changes in the persistence layer,
> > > >>>> configuration
> > > >>>>>> and
> > > >>>>>>>>>>>>> behavior.
> > > >>>>>>>>>>>>>>>> Additionally, these components are now tightly
> > > >>>> coupled,
> > > >>>>> so
> > > >>>>>>>> there
> > > >>>>>>>>>> is
> > > >>>>>>>>>>>>> no
> > > >>>>>>>>>>>>>>>> way
> > > >>>>>>>>>>>>>>>> these two changes can be implemented in parallel and
> > > >>>>> then
> > > >>>>>>>> merged
> > > >>>>>>>>>>>>>>>> together
> > > >>>>>>>>>>>>>>>> easily. So what we will end up with is having to
> > > >>>>> implement
> > > >>>>>>>> these
> > > >>>>>>>>>>>>>>> changes
> > > >>>>>>>>>>>>>>>> sequentially, fixing all existing tests twice, and
> > > >>>>>>> essentially
> > > >>>>>>>>>>>>>>> throwing
> > > >>>>>>>>>>>>>>>> away half of the work done because the other part of
> > > >>>> the
> > > >>>>>>>> change
> > > >>>>>>>>> is
> > > >>>>>>>>>>>>>>>> re-implemented
> > > >>>>>>>>>>>>>>>> - Similar example goes with getting rid of
> > > >>>>>>>> IgniteInternalFuture
> > > >>>>>>>>>> and
> > > >>>>>>>>>>>>>>>> replacing it with CompletableFuture, and any other
> > > >>>>> change
> > > >>>>>>> that
> > > >>>>>>>>>>>>> touches
> > > >>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>> asynchronous part of the code.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Third, I do not see how this choice affects the UX of
> > > >>>>>> Ignite.
> > > >>>>>>>> The
> > > >>>>>>>>>> end
> > > >>>>>>>>>>>>>>> user
> > > >>>>>>>>>>>>>>>> experience must be fixed in the IEP regardless of the
> > > >>>>>>>> development
> > > >>>>>>>>>>>>> process
> > > >>>>>>>>>>>>>>>> and the fact that we have gaps in this area in Ignite
> > > >>>> 2.x
> > > >>>>>>> just
> > > >>>>>>>>>>> confirms
> > > >>>>>>>>>>>>>>>> that.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Pavel, agree that a repo/branch is a technicality, I
> > > >>>>> guess
> > > >>>>>> if
> > > >>>>>>>>>>>>>>> reformulate,
> > > >>>>>>>>>>>>>>>> my point is that we might agree to have a single
> > > >>>>>> development
> > > >>>>>>>>> master
> > > >>>>>>>>>>>>>>> branch
> > > >>>>>>>>>>>>>>>> with 'disassembled' end-to-end functionality for some
> > > >>>>>> period
> > > >>>>>>> of
> > > >>>>>>>>>> time
> > > >>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>> speed up development, and re-assemble the core
> features
> > > >>>>>> after
> > > >>>>>>>>>> having
> > > >>>>>>>>>>>>>>>> submodules tested independently.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Nikolay,
> > > >>>>>>>>>>>>>>>>> We have many features that have to evolve.
> > > >>>>>>>>>>>>>>>>> Snapshots, rebalance, tooling, tracing, zookeeper
> > > >>>>> support,
> > > >>>>>>>> etc.
> > > >>>>>>>>>>>>>>>> This is not very specific. In the end, resources are
> > > >>>>>> limited
> > > >>>>>>>> and
> > > >>>>>>>>> we
> > > >>>>>>>>>>>>> will
> > > >>>>>>>>>>>>>>>> not be able to drive both tracks simultaneously,
> > > >>>>> especially
> > > >>>>>>>>> after a
> > > >>>>>>>>>>>>>>> couple
> > > >>>>>>>>>>>>>>>> of features having been implemented for Ignite 3.0. If
> > > >>>>>> there
> > > >>>>>>>> are
> > > >>>>>>>>>>> indeed
> > > >>>>>>>>>>>>>>>> some major changes that we want to do in Ignite 2.x
> > > >>>>> instead
> > > >>>>>>> of
> > > >>>>>>>>>>> putting
> > > >>>>>>>>>>>>>>>> effort into 3.0 - let's discuss them. I am just not
> > > >>>> aware
> > > >>>>>> of
> > > >>>>>>>> any,
> > > >>>>>>>>>>>>> that's
> > > >>>>>>>>>>>>>>>> why I am eager to move forward with Ignite 3.0.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> We have bugs and issues that can be fixed in 2.x
> > > >>>> without
> > > >>>>>>>>> breaking
> > > >>>>>>>>>>>>>>> backward
> > > >>>>>>>>>>>>>>>> compatibility.
> > > >>>>>>>>>>>>>>>>> We have many users who are happy with the 2.x with
> all
> > > >>>>>> it’s
> > > >>>>>>>>>> issues.
> > > >>>>>>>>>>>>>>>> These changes will be covered by end-to-end tests and
> > > >>>>>>> migrated
> > > >>>>>>>> to
> > > >>>>>>>>>>>>> Ignite
> > > >>>>>>>>>>>>>>>> 3.0, so I see no issues here.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Finally, Anton & Nikolay
> > > >>>>>>>>>>>>>>>> I do not have an estimate for this simply because the
> > > >>>>>>> activity
> > > >>>>>>>> is
> > > >>>>>>>>>>>>>>>> community-driven and it depends on the number of
> people
> > > >>>>>>> willing
> > > >>>>>>>>> to
> > > >>>>>>>>>>>>>>>> contribute. With the current pace, I would hope to
> have
> > > >>>>> an
> > > >>>>>> RC
> > > >>>>>>>> of
> > > >>>>>>>>>>> Ignite
> > > >>>>>>>>>>>>>>> 3.0
> > > >>>>>>>>>>>>>>>> to be ready by the end of 2021. My gut feeling is that
> > > >>>> by
> > > >>>>>>>> moving
> > > >>>>>>>>>> with
> > > >>>>>>>>>>>>>>>> incremental changes, we will not be able to implement
> > > >>>>> even
> > > >>>>>>> half
> > > >>>>>>>>> of
> > > >>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>> wishlist by that time.
> > > >>>>>>>>>>>>>>>> I doubt that releasing several major releases with
> > > >>>>> breaking
> > > >>>>>>>>> changes
> > > >>>>>>>>>>>>> will
> > > >>>>>>>>>>>>>>>> make Ignite users happy either because each upgrade
> > > >>>> will
> > > >>>>>> cost
> > > >>>>>>>>>> Ignite
> > > >>>>>>>>>>>>>>> users
> > > >>>>>>>>>>>>>>>> money, so the fewer major versions we release, the
> > > >>>>> better.
> > > >>>>>>> Thus
> > > >>>>>>>>> my
> > > >>>>>>>>>>> wish
> > > >>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>> include all breaking changes in one release.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> I'll be now quiet for a while, let's see what other
> > > >>>>>> community
> > > >>>>>>>>>> members
> > > >>>>>>>>>>>>>>>> think.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <
> > > >>>>>>>>> ptupitsyn@apache.org
> > > >>>>>>>>>>> :
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> 1. Rewriting from scratch is never a good idea.
> > > >>>>>>>>>>>>>>>>> We don't want to follow the path of Netscape and lose
> > > >>>>> all
> > > >>>>>>> our
> > > >>>>>>>>>> users
> > > >>>>>>>>>>>>>>>>> by the time we have a working 3.0 [1]
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> 2. Not sure about new repo - seems like some pain and
> > > >>>> no
> > > >>>>>>> gain,
> > > >>>>>>>>>>> what's
> > > >>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>> problem with a branch?
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> 3. We should keep existing integration tests when
> > > >>>>>> possible.
> > > >>>>>>>>>>>>>>>>> We have accumulated a lot of edge case knowledge over
> > > >>>>> the
> > > >>>>>>>> years,
> > > >>>>>>>>>>>>>>>>> it is not a good idea to send all of that down the
> > > >>>>> drain.
> > > >>>>>>>>>>>>>>>>> Yes, integration tests are slow, but they are the
> most
> > > >>>>>>>> valuable.
> > > >>>>>>>>>>>>>>>>> I think we can move more stuff into nightly runs and
> > > >>>>> have
> > > >>>>>> a
> > > >>>>>>>> fast
> > > >>>>>>>>>> and
> > > >>>>>>>>>>>>>>>> modern
> > > >>>>>>>>>>>>>>>>> basic suite.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Alexey, you are much more familiar with the Ignite
> > > >>>> core
> > > >>>>>>>> codebase
> > > >>>>>>>>>>> than
> > > >>>>>>>>>>>>>>>> most
> > > >>>>>>>>>>>>>>>>> of us,
> > > >>>>>>>>>>>>>>>>> can you please explain in more detail which
> particular
> > > >>>>>>>> feature,
> > > >>>>>>>>> in
> > > >>>>>>>>>>>>> your
> > > >>>>>>>>>>>>>>>>> opinion,
> > > >>>>>>>>>>>>>>>>> mandates this "start from scratch" approach?
> > > >>>>>>>>>>>>>>>>> Is it really not possible at all to follow a less
> > > >>>>> radical
> > > >>>>>>> way?
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> [1]
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > >
> https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <
> > > >>>>>>>>>> nizhikov@apache.org
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Hello, Alexey.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> I think that «rewriting from scratch» approach has a
> > > >>>>> high
> > > >>>>>>>> risk
> > > >>>>>>>>> to
> > > >>>>>>>>>>>>>>> make
> > > >>>>>>>>>>>>>>>>> new
> > > >>>>>>>>>>>>>>>>>> features unusable.
> > > >>>>>>>>>>>>>>>>>> At the time Ignite2 was started no-one wants to do
> > > >>>> bad
> > > >>>>> UX
> > > >>>>>>> or
> > > >>>>>>>>> bad
> > > >>>>>>>>>>>>>>>>> features.
> > > >>>>>>>>>>>>>>>>>> Nevertheless, it happen.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> I think we can avoid it with the Ignite3 and
> > > >>>> successors
> > > >>>>>> if
> > > >>>>>>> we
> > > >>>>>>>>>> will
> > > >>>>>>>>>>>>>>> move
> > > >>>>>>>>>>>>>>>>>> step by step without keeping backward compatibility
> > > >>>>>>>>>>>>>>>>>> With the step by step approach, we can focus on each
> > > >>>>>>>> component
> > > >>>>>>>>>>>>>>>>> separately.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> What new features are we planning to implement for
> > > >>>>>> Ignite
> > > >>>>>>>> 2.x?
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> We have many features that have to evolve.
> > > >>>>>>>>>>>>>>>>>> Snapshots, rebalance, tooling, tracing, zookeeper
> > > >>>>>> support,
> > > >>>>>>>> etc.
> > > >>>>>>>>>>>>>>>>>> We have bugs and issues that can be fixed in 2.x
> > > >>>>> without
> > > >>>>>>>>> breaking
> > > >>>>>>>>>>>>>>>>> backward
> > > >>>>>>>>>>>>>>>>>> compatibility.
> > > >>>>>>>>>>>>>>>>>> We have many users who are happy with the 2.x with
> > > >>>> all
> > > >>>>>> it’s
> > > >>>>>>>>>> issues.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> 2 нояб. 2020 г., в 14:09, Anton Vinogradov <
> > > >>>>>> av@apache.org
> > > >>>>>>>>
> > > >>>>>>>>>>>>>>>> написал(а):
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Alexey,
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Do we have any estimates of how fast we'll be able
> > > >>>> to
> > > >>>>>> gain
> > > >>>>>>>>>>>>>>>>>> production-ready
> > > >>>>>>>>>>>>>>>>>>> AI 3.0 in case of a "new repo" choice?
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk <
> > > >>>>>>>>>>>>>>>>>> alexey.goncharuk@gmail.com>
> > > >>>>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> Nikolay,
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> What new features are we planning to implement for
> > > >>>>>> Ignite
> > > >>>>>>>>> 2.x?
> > > >>>>>>>>>> I
> > > >>>>>>>>>>>>>>>> think
> > > >>>>>>>>>>>>>>>>>> once
> > > >>>>>>>>>>>>>>>>>>>> we commence working on Ignite 3.0, we should
> > > >>>>> gradually
> > > >>>>>>>> cease
> > > >>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>> activity
> > > >>>>>>>>>>>>>>>>>>>> on Ignite 2.x to mere bugfixes because such
> > > >>>> parallel
> > > >>>>>>>>>> development
> > > >>>>>>>>>>>>>>>> will
> > > >>>>>>>>>>>>>>>>> be
> > > >>>>>>>>>>>>>>>>>>>> overwhelming regardless of how we choose to
> > > >>>> proceed.
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <
> > > >>>>>>>>>>> nizhikov@apache.org
> > > >>>>>>>>>>>>>>>> :
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> To be clear:
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> I would suggest creating a new repository for
> > > >>>>> Ignite
> > > >>>>>>> 3.0
> > > >>>>>>>>>>>>>>>> (perhaps, a
> > > >>>>>>>>>>>>>>>>>>>> new
> > > >>>>>>>>>>>>>>>>>>>>> clean branch, but a new repo looks nicer to me)
> > > >>>> and
> > > >>>>> a
> > > >>>>>>> new
> > > >>>>>>>>>> Ignite
> > > >>>>>>>>>>>>>>>> 3.0
> > > >>>>>>>>>>>>>>>>>>>>> TeamCity project.
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> +1 for new Team City project.
> > > >>>>>>>>>>>>>>>>>>>>> +1 for new branch for Ignite3.
> > > >>>>>>>>>>>>>>>>>>>>> -1 for new repo.
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <
> > > >>>>>>>>>>>>>>> NIzhikov.dev@gmail.com
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> написал(а):
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> Hello, Alexey.
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> I think it will hurt our project more than help.
> > > >>>>>>>>>>>>>>>>>>>>>> Developing new features for 2 separate branches
> > > >>>>> with
> > > >>>>>>> the
> > > >>>>>>>>>>>>>>> different
> > > >>>>>>>>>>>>>>>>>> APIs
> > > >>>>>>>>>>>>>>>>>>>>> and internal structure is overwhelming
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> Maybe we should relax a bit requirements for
> > > >>>>> Ignite3?
> > > >>>>>>>>>>>>>>>>>>>>>> Maybe we should move step by step and make
> > > >>>> Ignite3
> > > >>>>>> with
> > > >>>>>>>> new
> > > >>>>>>>>>>>>>>>>>>>>> configuration than Ignite4 with new transactions,
> > > >>>>> etc?
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <
> > > >>>>>>>>>>>>>>>>>>>> alexey.goncharuk@gmail.com>
> > > >>>>>>>>>>>>>>>>>>>>> написал(а):
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> Igniters,
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> I wanted to pitch a rather radical idea
> > > >>>> regarding
> > > >>>>>> the
> > > >>>>>>>>> Ignite
> > > >>>>>>>>>>>>>>> 3.0
> > > >>>>>>>>>>>>>>>>>>>>>>> development which has occurred to me some time
> > > >>>>> ago.
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> We already have several IEPs targeted to Ignite
> > > >>>>> 3.0
> > > >>>>>>>> which
> > > >>>>>>>>>>> imply
> > > >>>>>>>>>>>>>>>>> major
> > > >>>>>>>>>>>>>>>>>>>>>>> changes to the codebase (the change in
> > > >>>> replication
> > > >>>>>>>>> protocol
> > > >>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>> thus
> > > >>>>>>>>>>>>>>>>>>>>>>> transactions, change in binary format, updated
> > > >>>>>>>>> metastorage,
> > > >>>>>>>>>>>>>>> etc).
> > > >>>>>>>>>>>>>>>>> We
> > > >>>>>>>>>>>>>>>>>>>>> also
> > > >>>>>>>>>>>>>>>>>>>>>>> planned significant changes in public APIs:
> > > >>>>>>>> configuration
> > > >>>>>>>>>>>>>>> format
> > > >>>>>>>>>>>>>>>>>>>> change,
> > > >>>>>>>>>>>>>>>>>>>>>>> improvements in cache APIs, SQL APIs,
> > > >>>> transaction
> > > >>>>>> mode
> > > >>>>>>>>>> rework.
> > > >>>>>>>>>>>>>>>> The
> > > >>>>>>>>>>>>>>>>>>>>> wishlist
> > > >>>>>>>>>>>>>>>>>>>>>>> of changes for Ignite 3.0 is huge.
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> So, I was wondering whether it makes sense to
> > > >>>> try
> > > >>>>> to
> > > >>>>>>>>> change
> > > >>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>> old
> > > >>>>>>>>>>>>>>>>>>>>>>> codebase, or start with a new baseline and move
> > > >>>>> old
> > > >>>>>>>> pieces
> > > >>>>>>>>>> of
> > > >>>>>>>>>>>>>>>> code
> > > >>>>>>>>>>>>>>>>>>>> that
> > > >>>>>>>>>>>>>>>>>>>>> do
> > > >>>>>>>>>>>>>>>>>>>>>>> not require significant rework. Personally, I
> > > >>>>> would
> > > >>>>>> go
> > > >>>>>>>>> with
> > > >>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>> second
> > > >>>>>>>>>>>>>>>>>>>>>>> option for the following reasons:
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> - We have a chance to shift the development
> > > >>>>> paradigm
> > > >>>>>>> in
> > > >>>>>>>>> the
> > > >>>>>>>>>>>>>>>> project
> > > >>>>>>>>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>>>>>>> introduce the practice of true unit-tests. In
> > > >>>> the
> > > >>>>>> new
> > > >>>>>>>>>> baseline
> > > >>>>>>>>>>>>>>> at
> > > >>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>> beginning there will be no ability to run an
> > > >>>>>>> end-to-end
> > > >>>>>>>>>>>>>>> scenario,
> > > >>>>>>>>>>>>>>>>>>>> thus
> > > >>>>>>>>>>>>>>>>>>>>> we
> > > >>>>>>>>>>>>>>>>>>>>>>> will be forced to write unit-tests. So far,
> such
> > > >>>>>>>> practice
> > > >>>>>>>>>> was
> > > >>>>>>>>>>>>>>>> hard
> > > >>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>>>> implement because of tight coupling between
> > > >>>> Ignite
> > > >>>>>>>>>> components
> > > >>>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>>>>> inability
> > > >>>>>>>>>>>>>>>>>>>>>>> to instantiate components without an instance
> of
> > > >>>>>>>>>>> KernalContext.
> > > >>>>>>>>>>>>>>>> For
> > > >>>>>>>>>>>>>>>>>>>>>>> example, we should be able to thoroughly test
> > > >>>>>> internal
> > > >>>>>>>>>>>>>>>> primitives,
> > > >>>>>>>>>>>>>>>>>>>>> such as
> > > >>>>>>>>>>>>>>>>>>>>>>> replication protocol (without actual
> > > >>>>> communication),
> > > >>>>>>>>>>>>>>> distributed
> > > >>>>>>>>>>>>>>>>>>>>>>> metastorage contracts, persistence layer, etc.
> > > >>>>>>>>>>>>>>>>>>>>>>> - We will significantly reduce the development
> > > >>>>> cycle
> > > >>>>>>> in
> > > >>>>>>>>> the
> > > >>>>>>>>>>>>>>>>> beginning
> > > >>>>>>>>>>>>>>>>>>>>>>> (right now the RunAll takes two hours of
> > > >>>>>> astronomical
> > > >>>>>>>> time
> > > >>>>>>>>>>> with
> > > >>>>>>>>>>>>>>>>> empty
> > > >>>>>>>>>>>>>>>>>>>>> TC;
> > > >>>>>>>>>>>>>>>>>>>>>>> in the new approach developer will be able to
> > > >>>> run
> > > >>>>>> ALL
> > > >>>>>>>>> tests
> > > >>>>>>>>>>>>>>>> locally
> > > >>>>>>>>>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>>>>>>>> a
> > > >>>>>>>>>>>>>>>>>>>>>>> matter of minutes)
> > > >>>>>>>>>>>>>>>>>>>>>>> - We can get rid of TC bot and enforce green TC
> > > >>>> by
> > > >>>>>>>>>> integrating
> > > >>>>>>>>>>>>>>> TC
> > > >>>>>>>>>>>>>>>>>>>> build
> > > >>>>>>>>>>>>>>>>>>>>>>> results with GitHub PRs (the same way Travis is
> > > >>>>>>>> currently
> > > >>>>>>>>>>>>>>>>> integrated
> > > >>>>>>>>>>>>>>>>>>>>> to PR
> > > >>>>>>>>>>>>>>>>>>>>>>> check). We should restrict PR merge without a
> TC
> > > >>>>>> check
> > > >>>>>>>>>>>>>>>>>>>>>>> - We will still have to re-write all tests, but
> > > >>>>> only
> > > >>>>>>>> once.
> > > >>>>>>>>>> If
> > > >>>>>>>>>>>>>>> we
> > > >>>>>>>>>>>>>>>>> try
> > > >>>>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>>>> modify the old codebase, we would need to
> modify
> > > >>>>> all
> > > >>>>>>> the
> > > >>>>>>>>>> tests
> > > >>>>>>>>>>>>>>>> for
> > > >>>>>>>>>>>>>>>>>>>>> every
> > > >>>>>>>>>>>>>>>>>>>>>>> major change (public API change, configuration
> > > >>>>>> change)
> > > >>>>>>>>>>>>>>>>>>>>>>> - We will have fewer conflicts when working
> > > >>>>>> together.
> > > >>>>>>>> For
> > > >>>>>>>>>>>>>>>> example,
> > > >>>>>>>>>>>>>>>>> I
> > > >>>>>>>>>>>>>>>>>>>>>>> cannot imagine how one would merge two changes
> > > >>>> of
> > > >>>>>>>> getting
> > > >>>>>>>>>> rid
> > > >>>>>>>>>>>>>>> of
> > > >>>>>>>>>>>>>>>>>>>>>>> IgniteFuture and changes in replication
> > > >>>> protocol,
> > > >>>>>> for
> > > >>>>>>>>>> example
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> Technically, I would suggest creating a new
> > > >>>>>> repository
> > > >>>>>>>> for
> > > >>>>>>>>>>>>>>> Ignite
> > > >>>>>>>>>>>>>>>>> 3.0
> > > >>>>>>>>>>>>>>>>>>>>>>> (perhaps, a new clean branch, but a new repo
> > > >>>> looks
> > > >>>>>>> nicer
> > > >>>>>>>>> to
> > > >>>>>>>>>>> me)
> > > >>>>>>>>>>>>>>>>> and a
> > > >>>>>>>>>>>>>>>>>>>>> new
> > > >>>>>>>>>>>>>>>>>>>>>>> Ignite 3.0 TeamCity project.
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> While it may seem quite radical, I do believe
> > > >>>> that
> > > >>>>>>> this
> > > >>>>>>>>>>>>>>> approach
> > > >>>>>>>>>>>>>>>>> will
> > > >>>>>>>>>>>>>>>>>>>>> give
> > > >>>>>>>>>>>>>>>>>>>>>>> us more benefits than trying to make such major
> > > >>>>>>> changes
> > > >>>>>>>> in
> > > >>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>> existing
> > > >>>>>>>>>>>>>>>>>>>>>>> codebase. If needed, let's schedule a discord
> > > >>>> chat
> > > >>>>>>> like
> > > >>>>>>>>>> before
> > > >>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>> discuss
> > > >>>>>>>>>>>>>>>>>>>>>>> this.
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> WDYT?
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> --
> > > >>>>>>>>>>>>>>> Best regards,
> > > >>>>>>>>>>>>>>> Andrey V. Mashenkov
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>>
> > > >>> --
> > > >>>
> > > >>> Best regards,
> > > >>> Alexei Scherbakov
> > > >
> > >
> > >
> > >
> >
> > --
> > Sincerely yours, Ivan Daschinskiy
>
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Maxim Muzafarov <mm...@apache.org>.
Val,
Why *creating a new repo* is the main point we faced with? Would it be
better to discuss the components design approach and scope management
first suggested by Sergey Chugunov? I doubt that new repo will solve
move us forward.
Currently, I'm -1 to create a new repo with the inputs above.
In addition to Nikolay's answer I see the following drawbacks of
creating new repo:
- we have very few positive examples of finalizing really huge
improvements to *production-ready* states the others remains
incomplete (MVCC, Calcite, Zookeeper, Tracing, Thread per Partition,
etc)
- AFAIK, the Native Persistence took a very long period of
stabilization even after it has been developed (we must take it into
account for developing new features like IEP-61)
- feature development for a long period of time (like 3.0) without any
releases will lead to all these changes became obsolete at the moment
of release (AFAIK the 2.8 which released a year ago still has no big
deployments)
- human resources -- some of the Igniters may lose their interest for
3.0 during development, some of them may switch to different projects,
etc.
- do we all estimating the scope of 3.0 correct? The 2.8 release took 1.5 years.
Have I missed something?
I suggest the following plan:
- initiate 3.0 development in the master branch (after 2.10 release
change version to 3.0-SNAPSHOT instead of 2.11-SNAPSHOT)
- cleanup and collapse all the current APIs (see To Be Removed List
For Discussion on Apache Ignite 3.0 Wishlist)
- reduce the scope for 3.0 even more. I suggest focusing on two
things: Calcite + Schema-first approach
- create feature branches for proposed IEPs (for 3.0 only)
- create the release road map (allocate e.g. IEP-61 to 4.0 etc.)
On Fri, 13 Nov 2020 at 14:03, Ivan Daschinsky <iv...@gmail.com> wrote:
>
> >> b. Implement IEP-61 - Common Replication Infrastructure
> I suppose, that this is the main cause of the current discussion.
> I hardly believe that this activity can be done without at least creating a
> completely new branch.
>
> пт, 13 нояб. 2020 г. в 11:12, Nikolay Izhikov <ni...@apache.org>:
>
> > My suggestion:
> >
> > 1. Reduce Ignite3 scope to the following:
> > a. Delete all deprecated API and support of obsolete internal
> > protocols.
> > b. Implement IEP-61 - Common Replication Infrastructure
> > c. Implement new Ignite management tool ignitectl as suggested
> > during Ignite3 discussion.
> >
> > 2. Implement and release following improvements like transactions, Calcite
> > based SQL, etc in the ongoing releases - Ignite 4, 5, 6
> >
> > My concern against separate Ignite 3 repo is the following:
> >
> > 1. We spread community to the two very separated part - Ignite3 developers
> > and Ignite2 maintainers. believe it’s bad for our community.
> > That can lead to the situation when we don’t fix critical or
> > blocker issueds «because they will not exists in Ignite3»
> > That will lead to the solutions never reviewed or reviewed poorly.
> >
> > 2. It seems for me that current scope of Ignite3 is too big to be
> > implemented in any reasonable time.
> >
> >
> > > 13 нояб. 2020 г., в 10:57, Nikolay Izhikov <NI...@gmail.com>
> > написал(а):
> > >
> > > Hello, Valentin.
> > >
> > >> Nikolay, Maxim, are you OK with this route?
> > >
> > > -1 to have another repo for Ignite3 development.
> > >
> > >> 13 нояб. 2020 г., в 03:04, Valentin Kulichenko <
> > valentin.kulichenko@gmail.com> написал(а):
> > >>
> > >> Folks,
> > >>
> > >> We already have multiple IEPs for Ignite 3.0, and as far as I know,
> > there are contributors that would like to work on them (or probably already
> > started). That said, we should make a decision as soon as possible.
> > >>
> > >> At this point, it doesn't seem that there are any strong objections to
> > the technical side of things. So I would suggest the following:
> > >>
> > >> 1. Proceed with Alexey's approach to the development process, as it
> > seems to be the best (in my opinion - the only) way to address all the
> > technical concerns and issues expressed in the thread. We'll start by
> > creating a new repo and a new TC project.
> > >> 2. Start a separate discussion around transparency. If there are any
> > changes we need to make to our contributor guidelines, I am happy to talk
> > them through, but I don't think it's reasonable to delay feature
> > development because of this. In the short term, I will make sure that
> > everything that happens within the new repo is as open to the community as
> > possible.
> > >>
> > >> Nikolay, Maxim, are you OK with this route?
> > >>
> > >> -Val
> > >>
> > >> On Tue, Nov 10, 2020 at 4:55 PM Valentin Kulichenko <
> > valentin.kulichenko@gmail.com> wrote:
> > >> Maxim,
> > >>
> > >> 2.x and 3.x will have to coexist for some time - I don't see how we can
> > avoid this considering the set of proposed changes. That said, we
> > effectively will need to have two "masters" - one for each major version.
> > Master for 3.x can technically be a branch in the existing repo, but having
> > a separate repo seems cleaner, simply because it will not be a "branch" in
> > the traditional sense.
> > >>
> > >> Note that the new repo will still be under the Apache org, with the
> > same set of committers, managed by the community, etc. All the development
> > happening for 3.0 must follow the rules that we currently have (if
> > anything, it's an opportunity to improve those rules).
> > >>
> > >> As I said during the call on Friday, I strongly believe that if there
> > is a transparency issue, it will exist regardless of the approach we choose
> > for 3.0. If community members develop without IEPs or public discussions,
> > this will happen for both 2.x and 3.x unless we address this separately. I
> > don't see how this is related to Alexey's suggestion, which targets
> > *technical* issues with the product more than anything else. This a way to
> > achieve better modularity, introduce better coverage with unit tests,
> > reduce conflicts during development, etc.
> > >>
> > >> Coming back to transparency, let's identify the issues and fix them. It
> > probably makes sense to have a separate discussion on this topic.
> > >>
> > >> -Val
> > >>
> > >> On Tue, Nov 10, 2020 at 1:05 PM Maxim Muzafarov <mm...@apache.org>
> > wrote:
> > >> Sergey,
> > >>
> > >>
> > >> Your summary makes sense to me.
> > >>
> > >> However, how we come up from *Development transparency* to *create a
> > >> separate public repository dedicated for 3.0*? For me *development
> > >> transparency* is about making changes in the master branch. These
> > >> changes will definitely be seen by all the Ignite developers.
> > >>
> > >> A dedicated public repository is technically public and visible for
> > >> everyone, but it allows development without IEPs, without public
> > >> discussion (since all the code changes are not related to the master
> > >> branch) it also allows a large number of assumptions and deviations
> > >> (like code-style violations). It also not about *development
> > >> transparency* since developers which are working on 3.0 is only a
> > >> subset of all Ignite developers which may continue working on 2.x. For
> > >> me, this would be a huge step backwards.
> > >>
> > >> Ignite veterans should remember how long the branch stabilization took
> > >> for the 2.x version with the PDS.
> > >>
> > >>
> > >> I think each breaking change should be passed through the master branch.
> > >>
> > >> On Tue, 10 Nov 2020 at 22:18, Alexei Scherbakov
> > >> <al...@gmail.com> wrote:
> > >>>
> > >>> Makes sense to me.
> > >>>
> > >>> вт, 10 нояб. 2020 г. в 18:47, Sergey Chugunov <
> > sergey.chugunov@gmail.com>:
> > >>>
> > >>>> Igniters,
> > >>>>
> > >>>> I thought over Friday meeting ideas and concerns and summarized them
> > in
> > >>>> these three points:
> > >>>>
> > >>>>
> > >>>> 1. *Components design unification approach.* New proposed components
> > >>>> will be developed by different contributors, but they need to be
> > unified
> > >>>> and should integrate with each other easily. To ensure that I
> > suggest
> > >>>> calling an architecture group that will create design guidelines
> > for all
> > >>>> components and high-level overview of overall architecture. How
> > code is
> > >>>> split into components, what are component boundaries, how component
> > >>>> lifecycle works and what are its interfaces - all these and other
> > >>>> questions
> > >>>> should be covered.
> > >>>>
> > >>>> 2. *Scope management.* Apache 3.0 should be implemented within a
> > >>>> reasonable time, so we need some procedure to decide whether a
> > >>>> particular
> > >>>> feature should be dropped from the scope of 3.0 and postponed to 3.1
> > >>>> release. To do so I suggest to range all features by two parameters:
> > >>>> criticality for 3.0 and amount of breaking changes. 3.0 scope should
> > >>>> include features of high criticality AND features with a big amount
> > of
> > >>>> breaking changes. All other features can be made optional.
> > >>>>
> > >>>> 3. *Development transparency.* Development of all components should
> > be
> > >>>> made as transparent for everyone as possible. Any contributor
> > should be
> > >>>> able to look over any component at any stage of development. To
> > achieve
> > >>>> this I suggest to create a separate public repository dedicated for
> > 3.0
> > >>>> development. It will make the code available for everyone but when
> > >>>> development of 3.0 is done we won't loose any stars of our current
> > >>>> repository as we merge dev repo into main one and drop dev.
> > >>>>
> > >>>> Do these ideas make sense to you? Are there any concerns not covered
> > by
> > >>>> these suggestions?
> > >>>>
> > >>>> On Fri, Nov 6, 2020 at 7:36 PM Kseniya Romanova <
> > romanova.ks.spb@gmail.com
> > >>>>>
> > >>>> wrote:
> > >>>>
> > >>>>> Here are the slides from Alexey Goncharuk. Let's think this over and
> > >>>>> continue on Monday:
> > >>>>>
> > >>>>>
> > >>>>
> > https://go.gridgain.com/rs/491-TWR-806/images/Ignite_3_Plans_and_development_process.pdf
> > >>>>>
> > >>>>> чт, 5 нояб. 2020 г. в 11:13, Anton Vinogradov <av...@apache.org>:
> > >>>>>
> > >>>>>> Folks,
> > >>>>>>
> > >>>>>> Should we perform cleanup work before (r)evolutional changes?
> > >>>>>> My huge proposal is to get rid of things which we don't need anyway
> > >>>>>> - local caches,
> > >>>>>> - strange tx modes,
> > >>>>>> - code overcomplexity because of RollingUpgrade feature never
> > attended
> > >>>> at
> > >>>>>> AI,
> > >>>>>> - etc,
> > >>>>>> before choosing the way.
> > >>>>>>
> > >>>>>> On Tue, Nov 3, 2020 at 3:31 PM Valentin Kulichenko <
> > >>>>>> valentin.kulichenko@gmail.com> wrote:
> > >>>>>>
> > >>>>>>> Ksenia, thanks for scheduling this on such short notice!
> > >>>>>>>
> > >>>>>>> As for the original topic, I do support Alexey's idea. We're not
> > >>>> going
> > >>>>> to
> > >>>>>>> rewrite anything from scratch, as most of the components are going
> > to
> > >>>>> be
> > >>>>>>> moved as-is or with minimal modifications. However, the changes
> > that
> > >>>>> are
> > >>>>>>> proposed imply serious rework of the core parts of the code, which
> > >>>> are
> > >>>>>> not
> > >>>>>>> properly decoupled from each other and from other parts. This makes
> > >>>> the
> > >>>>>>> incremental approach borderline impossible. Developing in a new
> > repo,
> > >>>>>>> however, addresses this concern. As a bonus, we can also refactor
> > the
> > >>>>>> code,
> > >>>>>>> introduce better decoupling, get rid of kernel context, and develop
> > >>>>> unit
> > >>>>>>> tests (finally!).
> > >>>>>>>
> > >>>>>>> Basically, this proposal only affects the *process*, not the set of
> > >>>>>> changes
> > >>>>>>> we had discussed before. Ignite 3.0 is our unique chance to make
> > >>>> things
> > >>>>>>> right.
> > >>>>>>>
> > >>>>>>> -Val
> > >>>>>>>
> > >>>>>>> On Tue, Nov 3, 2020 at 3:06 AM Kseniya Romanova <
> > >>>>>> romanova.ks.spb@gmail.com
> > >>>>>>>>
> > >>>>>>> wrote:
> > >>>>>>>
> > >>>>>>>> Pavel, all the interesting points will be anyway published here in
> > >>>>>>> English
> > >>>>>>>> (as the principal "if it's not on devlist it doesn't happened" is
> > >>>>> still
> > >>>>>>>> relevant). This is just a quick call for a group of developers.
> > >>>> Later
> > >>>>>> we
> > >>>>>>>> can do a separate presentation of idea and discussion in English
> > as
> > >>>>> we
> > >>>>>>> did
> > >>>>>>>> for the Ignite 3.0 draft of changes.
> > >>>>>>>>
> > >>>>>>>> вт, 3 нояб. 2020 г. в 13:52, Pavel Tupitsyn <ptupitsyn@apache.org
> > >>>>> :
> > >>>>>>>>
> > >>>>>>>>> Kseniya,
> > >>>>>>>>>
> > >>>>>>>>> Thanks for scheduling this call.
> > >>>>>>>>> Do you think we can switch to English if non-Russian speaking
> > >>>>>> community
> > >>>>>>>>> members decide to join?
> > >>>>>>>>>
> > >>>>>>>>> On Tue, Nov 3, 2020 at 1:32 PM Kseniya Romanova <
> > >>>>>>>> romanova.ks.spb@gmail.com
> > >>>>>>>>>>
> > >>>>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> Let's do this community discussion open. Here's the link on
> > >>>> zoom
> > >>>>>> call
> > >>>>>>>> in
> > >>>>>>>>>> Russian for Friday 6 PM:
> > >>>>>>>>>>
> > >>>>>>
> > https://www.meetup.com/Moscow-Apache-Ignite-Meetup/events/274360378/
> > >>>>>>>>>>
> > >>>>>>>>>> вт, 3 нояб. 2020 г. в 12:49, Nikolay Izhikov <
> > >>>>> nizhikov@apache.org
> > >>>>>>> :
> > >>>>>>>>>>
> > >>>>>>>>>>> Time works for me.
> > >>>>>>>>>>>
> > >>>>>>>>>>>> 3 нояб. 2020 г., в 12:40, Alexey Goncharuk <
> > >>>>>>>>> alexey.goncharuk@gmail.com
> > >>>>>>>>>>>
> > >>>>>>>>>>> написал(а):
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Nikolay,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I am up for the call. I will try to explain my reasoning in
> > >>>>>>> greater
> > >>>>>>>>>>> detail
> > >>>>>>>>>>>> and will be glad to hear the concerns. Will this Friday,
> > >>>> Nov
> > >>>>>> 6th,
> > >>>>>>>>> work?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> вт, 3 нояб. 2020 г. в 10:09, Nikolay Izhikov <
> > >>>>>>> nizhikov@apache.org
> > >>>>>>>>> :
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> Igniters, should we have a call for this topic?
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> 2 нояб. 2020 г., в 18:53, Pavel Tupitsyn <
> > >>>>>> ptupitsyn@apache.org
> > >>>>>>>>
> > >>>>>>>>>>>>> написал(а):
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> not intend to rewrite everything from scratch
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Every single test from Ignite 2.x should be moved to
> > >>>>> Ignite
> > >>>>>> 3
> > >>>>>>>>>>>>>>> regardless of how we choose to proceed.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Alexey, thank you for the explanation, this addresses all
> > >>>>> of
> > >>>>>> my
> > >>>>>>>>>>> concerns.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On Mon, Nov 2, 2020 at 6:43 PM Andrey Mashenkov <
> > >>>>>>>>>>>>> andrey.mashenkov@gmail.com>
> > >>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Hi, Igniters.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> * AFAIU, we need a new repo if we want to apply
> > >>>> different
> > >>>>>>>>>> restrictions
> > >>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>> pull requests,
> > >>>>>>>>>>>>>>> otherwise I see no difference for myself.
> > >>>>>>>>>>>>>>> E.g. make static analysis (do we have?), compile,
> > >>>> styles,
> > >>>>>> and
> > >>>>>>>>>> javadoc
> > >>>>>>>>>>>>>>> checks mandatory.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> I think that relaxed requirements here will lead to bad
> > >>>>>>> product
> > >>>>>>>>>>> quality.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> * Agree with Pavel, we should 'keep' integrations tests
> > >>>>>>> somehow.
> > >>>>>>>>>>>>>>> During active development tests will be broken most of
> > >>>>> time,
> > >>>>>>> so,
> > >>>>>>>>>>>>>>> I'd port them e.g. suite-by-suite once we will have a
> > >>>>> stable
> > >>>>>>> and
> > >>>>>>>>>>>>> featured
> > >>>>>>>>>>>>>>> environment to run them and of course make test's code
> > >>>>> clear
> > >>>>>>> and
> > >>>>>>>>>> avoid
> > >>>>>>>>>>>>>>> bad/non-relevant ones.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> * I like bottom-up approach.
> > >>>>>>>>>>>>>>> With it we could make a better framework. I mean clear
> > >>>>>>> component
> > >>>>>>>>>>>>> lifecycle,
> > >>>>>>>>>>>>>>> component wiring mechanics, general methods to approach
> > >>>>> core
> > >>>>>>>>>>> components
> > >>>>>>>>>>>>>>> such as exchange/communication
> > >>>>>>>>>>>>>>> to avoid code mess like we have in ExchangeFuture with
> > >>>> all
> > >>>>>>> these
> > >>>>>>>>>>> custom
> > >>>>>>>>>>>>>>> callbacks for each component, interfaces like
> > >>>>>>>>>>>>>>> PartitionsExchangeAware, IgniteChangeGlobalStateSupport
> > >>>>> and
> > >>>>>>>>>>>>>>> a pack of
> > >>>>>>>>>>>>>
> > >>>>>> start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected
> > >>>>>>>>>>>>>>> and so on in various unexpected places.
> > >>>>>>>>>>>>>>> Hope, we will be able to port most of the good code to
> > >>>> the
> > >>>>>> new
> > >>>>>>>>>>> framework
> > >>>>>>>>>>>>>>> version.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk <
> > >>>>>>>>>>>>>>> alexey.goncharuk@gmail.com>
> > >>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Nikolay, Pavel,
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Thanks for the feedback! First of all, I wanted to
> > >>>> stress
> > >>>>>>> that
> > >>>>>>>> I
> > >>>>>>>>> do
> > >>>>>>>>>>> not
> > >>>>>>>>>>>>>>>> intend to rewrite everything from scratch (I never used
> > >>>>>> this
> > >>>>>>>>>> phrase).
> > >>>>>>>>>>>>>>> There
> > >>>>>>>>>>>>>>>> are significant parts of code that would be moved with
> > >>>>>>> minimal
> > >>>>>>>>>>>>>>>> modifications. Second, I never said that we will get
> > >>>> rid
> > >>>>> of
> > >>>>>>> the
> > >>>>>>>>> old
> > >>>>>>>>>>>>> tests
> > >>>>>>>>>>>>>>>> codebase. Every single test from Ignite 2.x should be
> > >>>>> moved
> > >>>>>>> to
> > >>>>>>>>>>> Ignite 3
> > >>>>>>>>>>>>>>>> regardless of how we choose to proceed.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> My point is that for some parts of the code a clean
> > >>>>>> bottom-up
> > >>>>>>>>>>>>>>>> implementation will be cheaper in many ways. Let me
> > >>>> give
> > >>>>>> you
> > >>>>>>> a
> > >>>>>>>>> few
> > >>>>>>>>>>>>>>> concrete
> > >>>>>>>>>>>>>>>> examples:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> - I think no one can object that we need a cleanly
> > >>>>>> separated
> > >>>>>>>>>>>>>>> persistence
> > >>>>>>>>>>>>>>>> layer for Ignite. There is a very raw draft IEP for
> > >>>> this
> > >>>>>>>>> already.
> > >>>>>>>>>> On
> > >>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>> other hand, I think we also can agree that we need a
> > >>>>>>>> split-brain
> > >>>>>>>>>>>>>>>> resistant
> > >>>>>>>>>>>>>>>> replication protocol for caches. There is also an IEP
> > >>>>> for
> > >>>>>>>> this.
> > >>>>>>>>>>>>>>> Neither
> > >>>>>>>>>>>>>>>> of
> > >>>>>>>>>>>>>>>> the changes is a good fit for 2.x because they are
> > >>>>> likely
> > >>>>>> to
> > >>>>>>>>>>>>> introduce
> > >>>>>>>>>>>>>>>> breaking changes in the persistence layer,
> > >>>> configuration
> > >>>>>> and
> > >>>>>>>>>>>>> behavior.
> > >>>>>>>>>>>>>>>> Additionally, these components are now tightly
> > >>>> coupled,
> > >>>>> so
> > >>>>>>>> there
> > >>>>>>>>>> is
> > >>>>>>>>>>>>> no
> > >>>>>>>>>>>>>>>> way
> > >>>>>>>>>>>>>>>> these two changes can be implemented in parallel and
> > >>>>> then
> > >>>>>>>> merged
> > >>>>>>>>>>>>>>>> together
> > >>>>>>>>>>>>>>>> easily. So what we will end up with is having to
> > >>>>> implement
> > >>>>>>>> these
> > >>>>>>>>>>>>>>> changes
> > >>>>>>>>>>>>>>>> sequentially, fixing all existing tests twice, and
> > >>>>>>> essentially
> > >>>>>>>>>>>>>>> throwing
> > >>>>>>>>>>>>>>>> away half of the work done because the other part of
> > >>>> the
> > >>>>>>>> change
> > >>>>>>>>> is
> > >>>>>>>>>>>>>>>> re-implemented
> > >>>>>>>>>>>>>>>> - Similar example goes with getting rid of
> > >>>>>>>> IgniteInternalFuture
> > >>>>>>>>>> and
> > >>>>>>>>>>>>>>>> replacing it with CompletableFuture, and any other
> > >>>>> change
> > >>>>>>> that
> > >>>>>>>>>>>>> touches
> > >>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>> asynchronous part of the code.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Third, I do not see how this choice affects the UX of
> > >>>>>> Ignite.
> > >>>>>>>> The
> > >>>>>>>>>> end
> > >>>>>>>>>>>>>>> user
> > >>>>>>>>>>>>>>>> experience must be fixed in the IEP regardless of the
> > >>>>>>>> development
> > >>>>>>>>>>>>> process
> > >>>>>>>>>>>>>>>> and the fact that we have gaps in this area in Ignite
> > >>>> 2.x
> > >>>>>>> just
> > >>>>>>>>>>> confirms
> > >>>>>>>>>>>>>>>> that.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Pavel, agree that a repo/branch is a technicality, I
> > >>>>> guess
> > >>>>>> if
> > >>>>>>>>>>>>>>> reformulate,
> > >>>>>>>>>>>>>>>> my point is that we might agree to have a single
> > >>>>>> development
> > >>>>>>>>> master
> > >>>>>>>>>>>>>>> branch
> > >>>>>>>>>>>>>>>> with 'disassembled' end-to-end functionality for some
> > >>>>>> period
> > >>>>>>> of
> > >>>>>>>>>> time
> > >>>>>>>>>>> to
> > >>>>>>>>>>>>>>>> speed up development, and re-assemble the core features
> > >>>>>> after
> > >>>>>>>>>> having
> > >>>>>>>>>>>>>>>> submodules tested independently.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Nikolay,
> > >>>>>>>>>>>>>>>>> We have many features that have to evolve.
> > >>>>>>>>>>>>>>>>> Snapshots, rebalance, tooling, tracing, zookeeper
> > >>>>> support,
> > >>>>>>>> etc.
> > >>>>>>>>>>>>>>>> This is not very specific. In the end, resources are
> > >>>>>> limited
> > >>>>>>>> and
> > >>>>>>>>> we
> > >>>>>>>>>>>>> will
> > >>>>>>>>>>>>>>>> not be able to drive both tracks simultaneously,
> > >>>>> especially
> > >>>>>>>>> after a
> > >>>>>>>>>>>>>>> couple
> > >>>>>>>>>>>>>>>> of features having been implemented for Ignite 3.0. If
> > >>>>>> there
> > >>>>>>>> are
> > >>>>>>>>>>> indeed
> > >>>>>>>>>>>>>>>> some major changes that we want to do in Ignite 2.x
> > >>>>> instead
> > >>>>>>> of
> > >>>>>>>>>>> putting
> > >>>>>>>>>>>>>>>> effort into 3.0 - let's discuss them. I am just not
> > >>>> aware
> > >>>>>> of
> > >>>>>>>> any,
> > >>>>>>>>>>>>> that's
> > >>>>>>>>>>>>>>>> why I am eager to move forward with Ignite 3.0.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> We have bugs and issues that can be fixed in 2.x
> > >>>> without
> > >>>>>>>>> breaking
> > >>>>>>>>>>>>>>> backward
> > >>>>>>>>>>>>>>>> compatibility.
> > >>>>>>>>>>>>>>>>> We have many users who are happy with the 2.x with all
> > >>>>>> it’s
> > >>>>>>>>>> issues.
> > >>>>>>>>>>>>>>>> These changes will be covered by end-to-end tests and
> > >>>>>>> migrated
> > >>>>>>>> to
> > >>>>>>>>>>>>> Ignite
> > >>>>>>>>>>>>>>>> 3.0, so I see no issues here.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Finally, Anton & Nikolay
> > >>>>>>>>>>>>>>>> I do not have an estimate for this simply because the
> > >>>>>>> activity
> > >>>>>>>> is
> > >>>>>>>>>>>>>>>> community-driven and it depends on the number of people
> > >>>>>>> willing
> > >>>>>>>>> to
> > >>>>>>>>>>>>>>>> contribute. With the current pace, I would hope to have
> > >>>>> an
> > >>>>>> RC
> > >>>>>>>> of
> > >>>>>>>>>>> Ignite
> > >>>>>>>>>>>>>>> 3.0
> > >>>>>>>>>>>>>>>> to be ready by the end of 2021. My gut feeling is that
> > >>>> by
> > >>>>>>>> moving
> > >>>>>>>>>> with
> > >>>>>>>>>>>>>>>> incremental changes, we will not be able to implement
> > >>>>> even
> > >>>>>>> half
> > >>>>>>>>> of
> > >>>>>>>>>>> the
> > >>>>>>>>>>>>>>>> wishlist by that time.
> > >>>>>>>>>>>>>>>> I doubt that releasing several major releases with
> > >>>>> breaking
> > >>>>>>>>> changes
> > >>>>>>>>>>>>> will
> > >>>>>>>>>>>>>>>> make Ignite users happy either because each upgrade
> > >>>> will
> > >>>>>> cost
> > >>>>>>>>>> Ignite
> > >>>>>>>>>>>>>>> users
> > >>>>>>>>>>>>>>>> money, so the fewer major versions we release, the
> > >>>>> better.
> > >>>>>>> Thus
> > >>>>>>>>> my
> > >>>>>>>>>>> wish
> > >>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>> include all breaking changes in one release.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> I'll be now quiet for a while, let's see what other
> > >>>>>> community
> > >>>>>>>>>> members
> > >>>>>>>>>>>>>>>> think.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <
> > >>>>>>>>> ptupitsyn@apache.org
> > >>>>>>>>>>> :
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> 1. Rewriting from scratch is never a good idea.
> > >>>>>>>>>>>>>>>>> We don't want to follow the path of Netscape and lose
> > >>>>> all
> > >>>>>>> our
> > >>>>>>>>>> users
> > >>>>>>>>>>>>>>>>> by the time we have a working 3.0 [1]
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> 2. Not sure about new repo - seems like some pain and
> > >>>> no
> > >>>>>>> gain,
> > >>>>>>>>>>> what's
> > >>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>> problem with a branch?
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> 3. We should keep existing integration tests when
> > >>>>>> possible.
> > >>>>>>>>>>>>>>>>> We have accumulated a lot of edge case knowledge over
> > >>>>> the
> > >>>>>>>> years,
> > >>>>>>>>>>>>>>>>> it is not a good idea to send all of that down the
> > >>>>> drain.
> > >>>>>>>>>>>>>>>>> Yes, integration tests are slow, but they are the most
> > >>>>>>>> valuable.
> > >>>>>>>>>>>>>>>>> I think we can move more stuff into nightly runs and
> > >>>>> have
> > >>>>>> a
> > >>>>>>>> fast
> > >>>>>>>>>> and
> > >>>>>>>>>>>>>>>> modern
> > >>>>>>>>>>>>>>>>> basic suite.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Alexey, you are much more familiar with the Ignite
> > >>>> core
> > >>>>>>>> codebase
> > >>>>>>>>>>> than
> > >>>>>>>>>>>>>>>> most
> > >>>>>>>>>>>>>>>>> of us,
> > >>>>>>>>>>>>>>>>> can you please explain in more detail which particular
> > >>>>>>>> feature,
> > >>>>>>>>> in
> > >>>>>>>>>>>>> your
> > >>>>>>>>>>>>>>>>> opinion,
> > >>>>>>>>>>>>>>>>> mandates this "start from scratch" approach?
> > >>>>>>>>>>>>>>>>> Is it really not possible at all to follow a less
> > >>>>> radical
> > >>>>>>> way?
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> [1]
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <
> > >>>>>>>>>> nizhikov@apache.org
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Hello, Alexey.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> I think that «rewriting from scratch» approach has a
> > >>>>> high
> > >>>>>>>> risk
> > >>>>>>>>> to
> > >>>>>>>>>>>>>>> make
> > >>>>>>>>>>>>>>>>> new
> > >>>>>>>>>>>>>>>>>> features unusable.
> > >>>>>>>>>>>>>>>>>> At the time Ignite2 was started no-one wants to do
> > >>>> bad
> > >>>>> UX
> > >>>>>>> or
> > >>>>>>>>> bad
> > >>>>>>>>>>>>>>>>> features.
> > >>>>>>>>>>>>>>>>>> Nevertheless, it happen.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> I think we can avoid it with the Ignite3 and
> > >>>> successors
> > >>>>>> if
> > >>>>>>> we
> > >>>>>>>>>> will
> > >>>>>>>>>>>>>>> move
> > >>>>>>>>>>>>>>>>>> step by step without keeping backward compatibility
> > >>>>>>>>>>>>>>>>>> With the step by step approach, we can focus on each
> > >>>>>>>> component
> > >>>>>>>>>>>>>>>>> separately.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> What new features are we planning to implement for
> > >>>>>> Ignite
> > >>>>>>>> 2.x?
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> We have many features that have to evolve.
> > >>>>>>>>>>>>>>>>>> Snapshots, rebalance, tooling, tracing, zookeeper
> > >>>>>> support,
> > >>>>>>>> etc.
> > >>>>>>>>>>>>>>>>>> We have bugs and issues that can be fixed in 2.x
> > >>>>> without
> > >>>>>>>>> breaking
> > >>>>>>>>>>>>>>>>> backward
> > >>>>>>>>>>>>>>>>>> compatibility.
> > >>>>>>>>>>>>>>>>>> We have many users who are happy with the 2.x with
> > >>>> all
> > >>>>>> it’s
> > >>>>>>>>>> issues.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> 2 нояб. 2020 г., в 14:09, Anton Vinogradov <
> > >>>>>> av@apache.org
> > >>>>>>>>
> > >>>>>>>>>>>>>>>> написал(а):
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Alexey,
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Do we have any estimates of how fast we'll be able
> > >>>> to
> > >>>>>> gain
> > >>>>>>>>>>>>>>>>>> production-ready
> > >>>>>>>>>>>>>>>>>>> AI 3.0 in case of a "new repo" choice?
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk <
> > >>>>>>>>>>>>>>>>>> alexey.goncharuk@gmail.com>
> > >>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Nikolay,
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> What new features are we planning to implement for
> > >>>>>> Ignite
> > >>>>>>>>> 2.x?
> > >>>>>>>>>> I
> > >>>>>>>>>>>>>>>> think
> > >>>>>>>>>>>>>>>>>> once
> > >>>>>>>>>>>>>>>>>>>> we commence working on Ignite 3.0, we should
> > >>>>> gradually
> > >>>>>>>> cease
> > >>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>> activity
> > >>>>>>>>>>>>>>>>>>>> on Ignite 2.x to mere bugfixes because such
> > >>>> parallel
> > >>>>>>>>>> development
> > >>>>>>>>>>>>>>>> will
> > >>>>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>> overwhelming regardless of how we choose to
> > >>>> proceed.
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <
> > >>>>>>>>>>> nizhikov@apache.org
> > >>>>>>>>>>>>>>>> :
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> To be clear:
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> I would suggest creating a new repository for
> > >>>>> Ignite
> > >>>>>>> 3.0
> > >>>>>>>>>>>>>>>> (perhaps, a
> > >>>>>>>>>>>>>>>>>>>> new
> > >>>>>>>>>>>>>>>>>>>>> clean branch, but a new repo looks nicer to me)
> > >>>> and
> > >>>>> a
> > >>>>>>> new
> > >>>>>>>>>> Ignite
> > >>>>>>>>>>>>>>>> 3.0
> > >>>>>>>>>>>>>>>>>>>>> TeamCity project.
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> +1 for new Team City project.
> > >>>>>>>>>>>>>>>>>>>>> +1 for new branch for Ignite3.
> > >>>>>>>>>>>>>>>>>>>>> -1 for new repo.
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <
> > >>>>>>>>>>>>>>> NIzhikov.dev@gmail.com
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> написал(а):
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Hello, Alexey.
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> I think it will hurt our project more than help.
> > >>>>>>>>>>>>>>>>>>>>>> Developing new features for 2 separate branches
> > >>>>> with
> > >>>>>>> the
> > >>>>>>>>>>>>>>> different
> > >>>>>>>>>>>>>>>>>> APIs
> > >>>>>>>>>>>>>>>>>>>>> and internal structure is overwhelming
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Maybe we should relax a bit requirements for
> > >>>>> Ignite3?
> > >>>>>>>>>>>>>>>>>>>>>> Maybe we should move step by step and make
> > >>>> Ignite3
> > >>>>>> with
> > >>>>>>>> new
> > >>>>>>>>>>>>>>>>>>>>> configuration than Ignite4 with new transactions,
> > >>>>> etc?
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <
> > >>>>>>>>>>>>>>>>>>>> alexey.goncharuk@gmail.com>
> > >>>>>>>>>>>>>>>>>>>>> написал(а):
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Igniters,
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> I wanted to pitch a rather radical idea
> > >>>> regarding
> > >>>>>> the
> > >>>>>>>>> Ignite
> > >>>>>>>>>>>>>>> 3.0
> > >>>>>>>>>>>>>>>>>>>>>>> development which has occurred to me some time
> > >>>>> ago.
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> We already have several IEPs targeted to Ignite
> > >>>>> 3.0
> > >>>>>>>> which
> > >>>>>>>>>>> imply
> > >>>>>>>>>>>>>>>>> major
> > >>>>>>>>>>>>>>>>>>>>>>> changes to the codebase (the change in
> > >>>> replication
> > >>>>>>>>> protocol
> > >>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>> thus
> > >>>>>>>>>>>>>>>>>>>>>>> transactions, change in binary format, updated
> > >>>>>>>>> metastorage,
> > >>>>>>>>>>>>>>> etc).
> > >>>>>>>>>>>>>>>>> We
> > >>>>>>>>>>>>>>>>>>>>> also
> > >>>>>>>>>>>>>>>>>>>>>>> planned significant changes in public APIs:
> > >>>>>>>> configuration
> > >>>>>>>>>>>>>>> format
> > >>>>>>>>>>>>>>>>>>>> change,
> > >>>>>>>>>>>>>>>>>>>>>>> improvements in cache APIs, SQL APIs,
> > >>>> transaction
> > >>>>>> mode
> > >>>>>>>>>> rework.
> > >>>>>>>>>>>>>>>> The
> > >>>>>>>>>>>>>>>>>>>>> wishlist
> > >>>>>>>>>>>>>>>>>>>>>>> of changes for Ignite 3.0 is huge.
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> So, I was wondering whether it makes sense to
> > >>>> try
> > >>>>> to
> > >>>>>>>>> change
> > >>>>>>>>>>> the
> > >>>>>>>>>>>>>>>> old
> > >>>>>>>>>>>>>>>>>>>>>>> codebase, or start with a new baseline and move
> > >>>>> old
> > >>>>>>>> pieces
> > >>>>>>>>>> of
> > >>>>>>>>>>>>>>>> code
> > >>>>>>>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>>>> do
> > >>>>>>>>>>>>>>>>>>>>>>> not require significant rework. Personally, I
> > >>>>> would
> > >>>>>> go
> > >>>>>>>>> with
> > >>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>> second
> > >>>>>>>>>>>>>>>>>>>>>>> option for the following reasons:
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> - We have a chance to shift the development
> > >>>>> paradigm
> > >>>>>>> in
> > >>>>>>>>> the
> > >>>>>>>>>>>>>>>> project
> > >>>>>>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>>>> introduce the practice of true unit-tests. In
> > >>>> the
> > >>>>>> new
> > >>>>>>>>>> baseline
> > >>>>>>>>>>>>>>> at
> > >>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>> beginning there will be no ability to run an
> > >>>>>>> end-to-end
> > >>>>>>>>>>>>>>> scenario,
> > >>>>>>>>>>>>>>>>>>>> thus
> > >>>>>>>>>>>>>>>>>>>>> we
> > >>>>>>>>>>>>>>>>>>>>>>> will be forced to write unit-tests. So far, such
> > >>>>>>>> practice
> > >>>>>>>>>> was
> > >>>>>>>>>>>>>>>> hard
> > >>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>> implement because of tight coupling between
> > >>>> Ignite
> > >>>>>>>>>> components
> > >>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>> inability
> > >>>>>>>>>>>>>>>>>>>>>>> to instantiate components without an instance of
> > >>>>>>>>>>> KernalContext.
> > >>>>>>>>>>>>>>>> For
> > >>>>>>>>>>>>>>>>>>>>>>> example, we should be able to thoroughly test
> > >>>>>> internal
> > >>>>>>>>>>>>>>>> primitives,
> > >>>>>>>>>>>>>>>>>>>>> such as
> > >>>>>>>>>>>>>>>>>>>>>>> replication protocol (without actual
> > >>>>> communication),
> > >>>>>>>>>>>>>>> distributed
> > >>>>>>>>>>>>>>>>>>>>>>> metastorage contracts, persistence layer, etc.
> > >>>>>>>>>>>>>>>>>>>>>>> - We will significantly reduce the development
> > >>>>> cycle
> > >>>>>>> in
> > >>>>>>>>> the
> > >>>>>>>>>>>>>>>>> beginning
> > >>>>>>>>>>>>>>>>>>>>>>> (right now the RunAll takes two hours of
> > >>>>>> astronomical
> > >>>>>>>> time
> > >>>>>>>>>>> with
> > >>>>>>>>>>>>>>>>> empty
> > >>>>>>>>>>>>>>>>>>>>> TC;
> > >>>>>>>>>>>>>>>>>>>>>>> in the new approach developer will be able to
> > >>>> run
> > >>>>>> ALL
> > >>>>>>>>> tests
> > >>>>>>>>>>>>>>>> locally
> > >>>>>>>>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>>>>>>> matter of minutes)
> > >>>>>>>>>>>>>>>>>>>>>>> - We can get rid of TC bot and enforce green TC
> > >>>> by
> > >>>>>>>>>> integrating
> > >>>>>>>>>>>>>>> TC
> > >>>>>>>>>>>>>>>>>>>> build
> > >>>>>>>>>>>>>>>>>>>>>>> results with GitHub PRs (the same way Travis is
> > >>>>>>>> currently
> > >>>>>>>>>>>>>>>>> integrated
> > >>>>>>>>>>>>>>>>>>>>> to PR
> > >>>>>>>>>>>>>>>>>>>>>>> check). We should restrict PR merge without a TC
> > >>>>>> check
> > >>>>>>>>>>>>>>>>>>>>>>> - We will still have to re-write all tests, but
> > >>>>> only
> > >>>>>>>> once.
> > >>>>>>>>>> If
> > >>>>>>>>>>>>>>> we
> > >>>>>>>>>>>>>>>>> try
> > >>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>> modify the old codebase, we would need to modify
> > >>>>> all
> > >>>>>>> the
> > >>>>>>>>>> tests
> > >>>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>>>> every
> > >>>>>>>>>>>>>>>>>>>>>>> major change (public API change, configuration
> > >>>>>> change)
> > >>>>>>>>>>>>>>>>>>>>>>> - We will have fewer conflicts when working
> > >>>>>> together.
> > >>>>>>>> For
> > >>>>>>>>>>>>>>>> example,
> > >>>>>>>>>>>>>>>>> I
> > >>>>>>>>>>>>>>>>>>>>>>> cannot imagine how one would merge two changes
> > >>>> of
> > >>>>>>>> getting
> > >>>>>>>>>> rid
> > >>>>>>>>>>>>>>> of
> > >>>>>>>>>>>>>>>>>>>>>>> IgniteFuture and changes in replication
> > >>>> protocol,
> > >>>>>> for
> > >>>>>>>>>> example
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Technically, I would suggest creating a new
> > >>>>>> repository
> > >>>>>>>> for
> > >>>>>>>>>>>>>>> Ignite
> > >>>>>>>>>>>>>>>>> 3.0
> > >>>>>>>>>>>>>>>>>>>>>>> (perhaps, a new clean branch, but a new repo
> > >>>> looks
> > >>>>>>> nicer
> > >>>>>>>>> to
> > >>>>>>>>>>> me)
> > >>>>>>>>>>>>>>>>> and a
> > >>>>>>>>>>>>>>>>>>>>> new
> > >>>>>>>>>>>>>>>>>>>>>>> Ignite 3.0 TeamCity project.
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> While it may seem quite radical, I do believe
> > >>>> that
> > >>>>>>> this
> > >>>>>>>>>>>>>>> approach
> > >>>>>>>>>>>>>>>>> will
> > >>>>>>>>>>>>>>>>>>>>> give
> > >>>>>>>>>>>>>>>>>>>>>>> us more benefits than trying to make such major
> > >>>>>>> changes
> > >>>>>>>> in
> > >>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>> existing
> > >>>>>>>>>>>>>>>>>>>>>>> codebase. If needed, let's schedule a discord
> > >>>> chat
> > >>>>>>> like
> > >>>>>>>>>> before
> > >>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>> discuss
> > >>>>>>>>>>>>>>>>>>>>>>> this.
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> WDYT?
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> --
> > >>>>>>>>>>>>>>> Best regards,
> > >>>>>>>>>>>>>>> Andrey V. Mashenkov
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>>
> > >>> --
> > >>>
> > >>> Best regards,
> > >>> Alexei Scherbakov
> > >
> >
> >
> >
>
> --
> Sincerely yours, Ivan Daschinskiy
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Ivan Daschinsky <iv...@gmail.com>.
>> b. Implement IEP-61 - Common Replication Infrastructure
I suppose, that this is the main cause of the current discussion.
I hardly believe that this activity can be done without at least creating a
completely new branch.
пт, 13 нояб. 2020 г. в 11:12, Nikolay Izhikov <ni...@apache.org>:
> My suggestion:
>
> 1. Reduce Ignite3 scope to the following:
> a. Delete all deprecated API and support of obsolete internal
> protocols.
> b. Implement IEP-61 - Common Replication Infrastructure
> c. Implement new Ignite management tool ignitectl as suggested
> during Ignite3 discussion.
>
> 2. Implement and release following improvements like transactions, Calcite
> based SQL, etc in the ongoing releases - Ignite 4, 5, 6
>
> My concern against separate Ignite 3 repo is the following:
>
> 1. We spread community to the two very separated part - Ignite3 developers
> and Ignite2 maintainers. believe it’s bad for our community.
> That can lead to the situation when we don’t fix critical or
> blocker issueds «because they will not exists in Ignite3»
> That will lead to the solutions never reviewed or reviewed poorly.
>
> 2. It seems for me that current scope of Ignite3 is too big to be
> implemented in any reasonable time.
>
>
> > 13 нояб. 2020 г., в 10:57, Nikolay Izhikov <NI...@gmail.com>
> написал(а):
> >
> > Hello, Valentin.
> >
> >> Nikolay, Maxim, are you OK with this route?
> >
> > -1 to have another repo for Ignite3 development.
> >
> >> 13 нояб. 2020 г., в 03:04, Valentin Kulichenko <
> valentin.kulichenko@gmail.com> написал(а):
> >>
> >> Folks,
> >>
> >> We already have multiple IEPs for Ignite 3.0, and as far as I know,
> there are contributors that would like to work on them (or probably already
> started). That said, we should make a decision as soon as possible.
> >>
> >> At this point, it doesn't seem that there are any strong objections to
> the technical side of things. So I would suggest the following:
> >>
> >> 1. Proceed with Alexey's approach to the development process, as it
> seems to be the best (in my opinion - the only) way to address all the
> technical concerns and issues expressed in the thread. We'll start by
> creating a new repo and a new TC project.
> >> 2. Start a separate discussion around transparency. If there are any
> changes we need to make to our contributor guidelines, I am happy to talk
> them through, but I don't think it's reasonable to delay feature
> development because of this. In the short term, I will make sure that
> everything that happens within the new repo is as open to the community as
> possible.
> >>
> >> Nikolay, Maxim, are you OK with this route?
> >>
> >> -Val
> >>
> >> On Tue, Nov 10, 2020 at 4:55 PM Valentin Kulichenko <
> valentin.kulichenko@gmail.com> wrote:
> >> Maxim,
> >>
> >> 2.x and 3.x will have to coexist for some time - I don't see how we can
> avoid this considering the set of proposed changes. That said, we
> effectively will need to have two "masters" - one for each major version.
> Master for 3.x can technically be a branch in the existing repo, but having
> a separate repo seems cleaner, simply because it will not be a "branch" in
> the traditional sense.
> >>
> >> Note that the new repo will still be under the Apache org, with the
> same set of committers, managed by the community, etc. All the development
> happening for 3.0 must follow the rules that we currently have (if
> anything, it's an opportunity to improve those rules).
> >>
> >> As I said during the call on Friday, I strongly believe that if there
> is a transparency issue, it will exist regardless of the approach we choose
> for 3.0. If community members develop without IEPs or public discussions,
> this will happen for both 2.x and 3.x unless we address this separately. I
> don't see how this is related to Alexey's suggestion, which targets
> *technical* issues with the product more than anything else. This a way to
> achieve better modularity, introduce better coverage with unit tests,
> reduce conflicts during development, etc.
> >>
> >> Coming back to transparency, let's identify the issues and fix them. It
> probably makes sense to have a separate discussion on this topic.
> >>
> >> -Val
> >>
> >> On Tue, Nov 10, 2020 at 1:05 PM Maxim Muzafarov <mm...@apache.org>
> wrote:
> >> Sergey,
> >>
> >>
> >> Your summary makes sense to me.
> >>
> >> However, how we come up from *Development transparency* to *create a
> >> separate public repository dedicated for 3.0*? For me *development
> >> transparency* is about making changes in the master branch. These
> >> changes will definitely be seen by all the Ignite developers.
> >>
> >> A dedicated public repository is technically public and visible for
> >> everyone, but it allows development without IEPs, without public
> >> discussion (since all the code changes are not related to the master
> >> branch) it also allows a large number of assumptions and deviations
> >> (like code-style violations). It also not about *development
> >> transparency* since developers which are working on 3.0 is only a
> >> subset of all Ignite developers which may continue working on 2.x. For
> >> me, this would be a huge step backwards.
> >>
> >> Ignite veterans should remember how long the branch stabilization took
> >> for the 2.x version with the PDS.
> >>
> >>
> >> I think each breaking change should be passed through the master branch.
> >>
> >> On Tue, 10 Nov 2020 at 22:18, Alexei Scherbakov
> >> <al...@gmail.com> wrote:
> >>>
> >>> Makes sense to me.
> >>>
> >>> вт, 10 нояб. 2020 г. в 18:47, Sergey Chugunov <
> sergey.chugunov@gmail.com>:
> >>>
> >>>> Igniters,
> >>>>
> >>>> I thought over Friday meeting ideas and concerns and summarized them
> in
> >>>> these three points:
> >>>>
> >>>>
> >>>> 1. *Components design unification approach.* New proposed components
> >>>> will be developed by different contributors, but they need to be
> unified
> >>>> and should integrate with each other easily. To ensure that I
> suggest
> >>>> calling an architecture group that will create design guidelines
> for all
> >>>> components and high-level overview of overall architecture. How
> code is
> >>>> split into components, what are component boundaries, how component
> >>>> lifecycle works and what are its interfaces - all these and other
> >>>> questions
> >>>> should be covered.
> >>>>
> >>>> 2. *Scope management.* Apache 3.0 should be implemented within a
> >>>> reasonable time, so we need some procedure to decide whether a
> >>>> particular
> >>>> feature should be dropped from the scope of 3.0 and postponed to 3.1
> >>>> release. To do so I suggest to range all features by two parameters:
> >>>> criticality for 3.0 and amount of breaking changes. 3.0 scope should
> >>>> include features of high criticality AND features with a big amount
> of
> >>>> breaking changes. All other features can be made optional.
> >>>>
> >>>> 3. *Development transparency.* Development of all components should
> be
> >>>> made as transparent for everyone as possible. Any contributor
> should be
> >>>> able to look over any component at any stage of development. To
> achieve
> >>>> this I suggest to create a separate public repository dedicated for
> 3.0
> >>>> development. It will make the code available for everyone but when
> >>>> development of 3.0 is done we won't loose any stars of our current
> >>>> repository as we merge dev repo into main one and drop dev.
> >>>>
> >>>> Do these ideas make sense to you? Are there any concerns not covered
> by
> >>>> these suggestions?
> >>>>
> >>>> On Fri, Nov 6, 2020 at 7:36 PM Kseniya Romanova <
> romanova.ks.spb@gmail.com
> >>>>>
> >>>> wrote:
> >>>>
> >>>>> Here are the slides from Alexey Goncharuk. Let's think this over and
> >>>>> continue on Monday:
> >>>>>
> >>>>>
> >>>>
> https://go.gridgain.com/rs/491-TWR-806/images/Ignite_3_Plans_and_development_process.pdf
> >>>>>
> >>>>> чт, 5 нояб. 2020 г. в 11:13, Anton Vinogradov <av...@apache.org>:
> >>>>>
> >>>>>> Folks,
> >>>>>>
> >>>>>> Should we perform cleanup work before (r)evolutional changes?
> >>>>>> My huge proposal is to get rid of things which we don't need anyway
> >>>>>> - local caches,
> >>>>>> - strange tx modes,
> >>>>>> - code overcomplexity because of RollingUpgrade feature never
> attended
> >>>> at
> >>>>>> AI,
> >>>>>> - etc,
> >>>>>> before choosing the way.
> >>>>>>
> >>>>>> On Tue, Nov 3, 2020 at 3:31 PM Valentin Kulichenko <
> >>>>>> valentin.kulichenko@gmail.com> wrote:
> >>>>>>
> >>>>>>> Ksenia, thanks for scheduling this on such short notice!
> >>>>>>>
> >>>>>>> As for the original topic, I do support Alexey's idea. We're not
> >>>> going
> >>>>> to
> >>>>>>> rewrite anything from scratch, as most of the components are going
> to
> >>>>> be
> >>>>>>> moved as-is or with minimal modifications. However, the changes
> that
> >>>>> are
> >>>>>>> proposed imply serious rework of the core parts of the code, which
> >>>> are
> >>>>>> not
> >>>>>>> properly decoupled from each other and from other parts. This makes
> >>>> the
> >>>>>>> incremental approach borderline impossible. Developing in a new
> repo,
> >>>>>>> however, addresses this concern. As a bonus, we can also refactor
> the
> >>>>>> code,
> >>>>>>> introduce better decoupling, get rid of kernel context, and develop
> >>>>> unit
> >>>>>>> tests (finally!).
> >>>>>>>
> >>>>>>> Basically, this proposal only affects the *process*, not the set of
> >>>>>> changes
> >>>>>>> we had discussed before. Ignite 3.0 is our unique chance to make
> >>>> things
> >>>>>>> right.
> >>>>>>>
> >>>>>>> -Val
> >>>>>>>
> >>>>>>> On Tue, Nov 3, 2020 at 3:06 AM Kseniya Romanova <
> >>>>>> romanova.ks.spb@gmail.com
> >>>>>>>>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Pavel, all the interesting points will be anyway published here in
> >>>>>>> English
> >>>>>>>> (as the principal "if it's not on devlist it doesn't happened" is
> >>>>> still
> >>>>>>>> relevant). This is just a quick call for a group of developers.
> >>>> Later
> >>>>>> we
> >>>>>>>> can do a separate presentation of idea and discussion in English
> as
> >>>>> we
> >>>>>>> did
> >>>>>>>> for the Ignite 3.0 draft of changes.
> >>>>>>>>
> >>>>>>>> вт, 3 нояб. 2020 г. в 13:52, Pavel Tupitsyn <ptupitsyn@apache.org
> >>>>> :
> >>>>>>>>
> >>>>>>>>> Kseniya,
> >>>>>>>>>
> >>>>>>>>> Thanks for scheduling this call.
> >>>>>>>>> Do you think we can switch to English if non-Russian speaking
> >>>>>> community
> >>>>>>>>> members decide to join?
> >>>>>>>>>
> >>>>>>>>> On Tue, Nov 3, 2020 at 1:32 PM Kseniya Romanova <
> >>>>>>>> romanova.ks.spb@gmail.com
> >>>>>>>>>>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Let's do this community discussion open. Here's the link on
> >>>> zoom
> >>>>>> call
> >>>>>>>> in
> >>>>>>>>>> Russian for Friday 6 PM:
> >>>>>>>>>>
> >>>>>>
> https://www.meetup.com/Moscow-Apache-Ignite-Meetup/events/274360378/
> >>>>>>>>>>
> >>>>>>>>>> вт, 3 нояб. 2020 г. в 12:49, Nikolay Izhikov <
> >>>>> nizhikov@apache.org
> >>>>>>> :
> >>>>>>>>>>
> >>>>>>>>>>> Time works for me.
> >>>>>>>>>>>
> >>>>>>>>>>>> 3 нояб. 2020 г., в 12:40, Alexey Goncharuk <
> >>>>>>>>> alexey.goncharuk@gmail.com
> >>>>>>>>>>>
> >>>>>>>>>>> написал(а):
> >>>>>>>>>>>>
> >>>>>>>>>>>> Nikolay,
> >>>>>>>>>>>>
> >>>>>>>>>>>> I am up for the call. I will try to explain my reasoning in
> >>>>>>> greater
> >>>>>>>>>>> detail
> >>>>>>>>>>>> and will be glad to hear the concerns. Will this Friday,
> >>>> Nov
> >>>>>> 6th,
> >>>>>>>>> work?
> >>>>>>>>>>>>
> >>>>>>>>>>>> вт, 3 нояб. 2020 г. в 10:09, Nikolay Izhikov <
> >>>>>>> nizhikov@apache.org
> >>>>>>>>> :
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Igniters, should we have a call for this topic?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> 2 нояб. 2020 г., в 18:53, Pavel Tupitsyn <
> >>>>>> ptupitsyn@apache.org
> >>>>>>>>
> >>>>>>>>>>>>> написал(а):
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> not intend to rewrite everything from scratch
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Every single test from Ignite 2.x should be moved to
> >>>>> Ignite
> >>>>>> 3
> >>>>>>>>>>>>>>> regardless of how we choose to proceed.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Alexey, thank you for the explanation, this addresses all
> >>>>> of
> >>>>>> my
> >>>>>>>>>>> concerns.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Mon, Nov 2, 2020 at 6:43 PM Andrey Mashenkov <
> >>>>>>>>>>>>> andrey.mashenkov@gmail.com>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi, Igniters.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> * AFAIU, we need a new repo if we want to apply
> >>>> different
> >>>>>>>>>> restrictions
> >>>>>>>>>>>>> to
> >>>>>>>>>>>>>>> pull requests,
> >>>>>>>>>>>>>>> otherwise I see no difference for myself.
> >>>>>>>>>>>>>>> E.g. make static analysis (do we have?), compile,
> >>>> styles,
> >>>>>> and
> >>>>>>>>>> javadoc
> >>>>>>>>>>>>>>> checks mandatory.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I think that relaxed requirements here will lead to bad
> >>>>>>> product
> >>>>>>>>>>> quality.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> * Agree with Pavel, we should 'keep' integrations tests
> >>>>>>> somehow.
> >>>>>>>>>>>>>>> During active development tests will be broken most of
> >>>>> time,
> >>>>>>> so,
> >>>>>>>>>>>>>>> I'd port them e.g. suite-by-suite once we will have a
> >>>>> stable
> >>>>>>> and
> >>>>>>>>>>>>> featured
> >>>>>>>>>>>>>>> environment to run them and of course make test's code
> >>>>> clear
> >>>>>>> and
> >>>>>>>>>> avoid
> >>>>>>>>>>>>>>> bad/non-relevant ones.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> * I like bottom-up approach.
> >>>>>>>>>>>>>>> With it we could make a better framework. I mean clear
> >>>>>>> component
> >>>>>>>>>>>>> lifecycle,
> >>>>>>>>>>>>>>> component wiring mechanics, general methods to approach
> >>>>> core
> >>>>>>>>>>> components
> >>>>>>>>>>>>>>> such as exchange/communication
> >>>>>>>>>>>>>>> to avoid code mess like we have in ExchangeFuture with
> >>>> all
> >>>>>>> these
> >>>>>>>>>>> custom
> >>>>>>>>>>>>>>> callbacks for each component, interfaces like
> >>>>>>>>>>>>>>> PartitionsExchangeAware, IgniteChangeGlobalStateSupport
> >>>>> and
> >>>>>>>>>>>>>>> a pack of
> >>>>>>>>>>>>>
> >>>>>> start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected
> >>>>>>>>>>>>>>> and so on in various unexpected places.
> >>>>>>>>>>>>>>> Hope, we will be able to port most of the good code to
> >>>> the
> >>>>>> new
> >>>>>>>>>>> framework
> >>>>>>>>>>>>>>> version.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk <
> >>>>>>>>>>>>>>> alexey.goncharuk@gmail.com>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Nikolay, Pavel,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thanks for the feedback! First of all, I wanted to
> >>>> stress
> >>>>>>> that
> >>>>>>>> I
> >>>>>>>>> do
> >>>>>>>>>>> not
> >>>>>>>>>>>>>>>> intend to rewrite everything from scratch (I never used
> >>>>>> this
> >>>>>>>>>> phrase).
> >>>>>>>>>>>>>>> There
> >>>>>>>>>>>>>>>> are significant parts of code that would be moved with
> >>>>>>> minimal
> >>>>>>>>>>>>>>>> modifications. Second, I never said that we will get
> >>>> rid
> >>>>> of
> >>>>>>> the
> >>>>>>>>> old
> >>>>>>>>>>>>> tests
> >>>>>>>>>>>>>>>> codebase. Every single test from Ignite 2.x should be
> >>>>> moved
> >>>>>>> to
> >>>>>>>>>>> Ignite 3
> >>>>>>>>>>>>>>>> regardless of how we choose to proceed.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> My point is that for some parts of the code a clean
> >>>>>> bottom-up
> >>>>>>>>>>>>>>>> implementation will be cheaper in many ways. Let me
> >>>> give
> >>>>>> you
> >>>>>>> a
> >>>>>>>>> few
> >>>>>>>>>>>>>>> concrete
> >>>>>>>>>>>>>>>> examples:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> - I think no one can object that we need a cleanly
> >>>>>> separated
> >>>>>>>>>>>>>>> persistence
> >>>>>>>>>>>>>>>> layer for Ignite. There is a very raw draft IEP for
> >>>> this
> >>>>>>>>> already.
> >>>>>>>>>> On
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> other hand, I think we also can agree that we need a
> >>>>>>>> split-brain
> >>>>>>>>>>>>>>>> resistant
> >>>>>>>>>>>>>>>> replication protocol for caches. There is also an IEP
> >>>>> for
> >>>>>>>> this.
> >>>>>>>>>>>>>>> Neither
> >>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>> the changes is a good fit for 2.x because they are
> >>>>> likely
> >>>>>> to
> >>>>>>>>>>>>> introduce
> >>>>>>>>>>>>>>>> breaking changes in the persistence layer,
> >>>> configuration
> >>>>>> and
> >>>>>>>>>>>>> behavior.
> >>>>>>>>>>>>>>>> Additionally, these components are now tightly
> >>>> coupled,
> >>>>> so
> >>>>>>>> there
> >>>>>>>>>> is
> >>>>>>>>>>>>> no
> >>>>>>>>>>>>>>>> way
> >>>>>>>>>>>>>>>> these two changes can be implemented in parallel and
> >>>>> then
> >>>>>>>> merged
> >>>>>>>>>>>>>>>> together
> >>>>>>>>>>>>>>>> easily. So what we will end up with is having to
> >>>>> implement
> >>>>>>>> these
> >>>>>>>>>>>>>>> changes
> >>>>>>>>>>>>>>>> sequentially, fixing all existing tests twice, and
> >>>>>>> essentially
> >>>>>>>>>>>>>>> throwing
> >>>>>>>>>>>>>>>> away half of the work done because the other part of
> >>>> the
> >>>>>>>> change
> >>>>>>>>> is
> >>>>>>>>>>>>>>>> re-implemented
> >>>>>>>>>>>>>>>> - Similar example goes with getting rid of
> >>>>>>>> IgniteInternalFuture
> >>>>>>>>>> and
> >>>>>>>>>>>>>>>> replacing it with CompletableFuture, and any other
> >>>>> change
> >>>>>>> that
> >>>>>>>>>>>>> touches
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> asynchronous part of the code.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Third, I do not see how this choice affects the UX of
> >>>>>> Ignite.
> >>>>>>>> The
> >>>>>>>>>> end
> >>>>>>>>>>>>>>> user
> >>>>>>>>>>>>>>>> experience must be fixed in the IEP regardless of the
> >>>>>>>> development
> >>>>>>>>>>>>> process
> >>>>>>>>>>>>>>>> and the fact that we have gaps in this area in Ignite
> >>>> 2.x
> >>>>>>> just
> >>>>>>>>>>> confirms
> >>>>>>>>>>>>>>>> that.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Pavel, agree that a repo/branch is a technicality, I
> >>>>> guess
> >>>>>> if
> >>>>>>>>>>>>>>> reformulate,
> >>>>>>>>>>>>>>>> my point is that we might agree to have a single
> >>>>>> development
> >>>>>>>>> master
> >>>>>>>>>>>>>>> branch
> >>>>>>>>>>>>>>>> with 'disassembled' end-to-end functionality for some
> >>>>>> period
> >>>>>>> of
> >>>>>>>>>> time
> >>>>>>>>>>> to
> >>>>>>>>>>>>>>>> speed up development, and re-assemble the core features
> >>>>>> after
> >>>>>>>>>> having
> >>>>>>>>>>>>>>>> submodules tested independently.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Nikolay,
> >>>>>>>>>>>>>>>>> We have many features that have to evolve.
> >>>>>>>>>>>>>>>>> Snapshots, rebalance, tooling, tracing, zookeeper
> >>>>> support,
> >>>>>>>> etc.
> >>>>>>>>>>>>>>>> This is not very specific. In the end, resources are
> >>>>>> limited
> >>>>>>>> and
> >>>>>>>>> we
> >>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>> not be able to drive both tracks simultaneously,
> >>>>> especially
> >>>>>>>>> after a
> >>>>>>>>>>>>>>> couple
> >>>>>>>>>>>>>>>> of features having been implemented for Ignite 3.0. If
> >>>>>> there
> >>>>>>>> are
> >>>>>>>>>>> indeed
> >>>>>>>>>>>>>>>> some major changes that we want to do in Ignite 2.x
> >>>>> instead
> >>>>>>> of
> >>>>>>>>>>> putting
> >>>>>>>>>>>>>>>> effort into 3.0 - let's discuss them. I am just not
> >>>> aware
> >>>>>> of
> >>>>>>>> any,
> >>>>>>>>>>>>> that's
> >>>>>>>>>>>>>>>> why I am eager to move forward with Ignite 3.0.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> We have bugs and issues that can be fixed in 2.x
> >>>> without
> >>>>>>>>> breaking
> >>>>>>>>>>>>>>> backward
> >>>>>>>>>>>>>>>> compatibility.
> >>>>>>>>>>>>>>>>> We have many users who are happy with the 2.x with all
> >>>>>> it’s
> >>>>>>>>>> issues.
> >>>>>>>>>>>>>>>> These changes will be covered by end-to-end tests and
> >>>>>>> migrated
> >>>>>>>> to
> >>>>>>>>>>>>> Ignite
> >>>>>>>>>>>>>>>> 3.0, so I see no issues here.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Finally, Anton & Nikolay
> >>>>>>>>>>>>>>>> I do not have an estimate for this simply because the
> >>>>>>> activity
> >>>>>>>> is
> >>>>>>>>>>>>>>>> community-driven and it depends on the number of people
> >>>>>>> willing
> >>>>>>>>> to
> >>>>>>>>>>>>>>>> contribute. With the current pace, I would hope to have
> >>>>> an
> >>>>>> RC
> >>>>>>>> of
> >>>>>>>>>>> Ignite
> >>>>>>>>>>>>>>> 3.0
> >>>>>>>>>>>>>>>> to be ready by the end of 2021. My gut feeling is that
> >>>> by
> >>>>>>>> moving
> >>>>>>>>>> with
> >>>>>>>>>>>>>>>> incremental changes, we will not be able to implement
> >>>>> even
> >>>>>>> half
> >>>>>>>>> of
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>> wishlist by that time.
> >>>>>>>>>>>>>>>> I doubt that releasing several major releases with
> >>>>> breaking
> >>>>>>>>> changes
> >>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>> make Ignite users happy either because each upgrade
> >>>> will
> >>>>>> cost
> >>>>>>>>>> Ignite
> >>>>>>>>>>>>>>> users
> >>>>>>>>>>>>>>>> money, so the fewer major versions we release, the
> >>>>> better.
> >>>>>>> Thus
> >>>>>>>>> my
> >>>>>>>>>>> wish
> >>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>> include all breaking changes in one release.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I'll be now quiet for a while, let's see what other
> >>>>>> community
> >>>>>>>>>> members
> >>>>>>>>>>>>>>>> think.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <
> >>>>>>>>> ptupitsyn@apache.org
> >>>>>>>>>>> :
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> 1. Rewriting from scratch is never a good idea.
> >>>>>>>>>>>>>>>>> We don't want to follow the path of Netscape and lose
> >>>>> all
> >>>>>>> our
> >>>>>>>>>> users
> >>>>>>>>>>>>>>>>> by the time we have a working 3.0 [1]
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> 2. Not sure about new repo - seems like some pain and
> >>>> no
> >>>>>>> gain,
> >>>>>>>>>>> what's
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> problem with a branch?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> 3. We should keep existing integration tests when
> >>>>>> possible.
> >>>>>>>>>>>>>>>>> We have accumulated a lot of edge case knowledge over
> >>>>> the
> >>>>>>>> years,
> >>>>>>>>>>>>>>>>> it is not a good idea to send all of that down the
> >>>>> drain.
> >>>>>>>>>>>>>>>>> Yes, integration tests are slow, but they are the most
> >>>>>>>> valuable.
> >>>>>>>>>>>>>>>>> I think we can move more stuff into nightly runs and
> >>>>> have
> >>>>>> a
> >>>>>>>> fast
> >>>>>>>>>> and
> >>>>>>>>>>>>>>>> modern
> >>>>>>>>>>>>>>>>> basic suite.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Alexey, you are much more familiar with the Ignite
> >>>> core
> >>>>>>>> codebase
> >>>>>>>>>>> than
> >>>>>>>>>>>>>>>> most
> >>>>>>>>>>>>>>>>> of us,
> >>>>>>>>>>>>>>>>> can you please explain in more detail which particular
> >>>>>>>> feature,
> >>>>>>>>> in
> >>>>>>>>>>>>> your
> >>>>>>>>>>>>>>>>> opinion,
> >>>>>>>>>>>>>>>>> mandates this "start from scratch" approach?
> >>>>>>>>>>>>>>>>> Is it really not possible at all to follow a less
> >>>>> radical
> >>>>>>> way?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> [1]
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <
> >>>>>>>>>> nizhikov@apache.org
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Hello, Alexey.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I think that «rewriting from scratch» approach has a
> >>>>> high
> >>>>>>>> risk
> >>>>>>>>> to
> >>>>>>>>>>>>>>> make
> >>>>>>>>>>>>>>>>> new
> >>>>>>>>>>>>>>>>>> features unusable.
> >>>>>>>>>>>>>>>>>> At the time Ignite2 was started no-one wants to do
> >>>> bad
> >>>>> UX
> >>>>>>> or
> >>>>>>>>> bad
> >>>>>>>>>>>>>>>>> features.
> >>>>>>>>>>>>>>>>>> Nevertheless, it happen.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I think we can avoid it with the Ignite3 and
> >>>> successors
> >>>>>> if
> >>>>>>> we
> >>>>>>>>>> will
> >>>>>>>>>>>>>>> move
> >>>>>>>>>>>>>>>>>> step by step without keeping backward compatibility
> >>>>>>>>>>>>>>>>>> With the step by step approach, we can focus on each
> >>>>>>>> component
> >>>>>>>>>>>>>>>>> separately.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> What new features are we planning to implement for
> >>>>>> Ignite
> >>>>>>>> 2.x?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> We have many features that have to evolve.
> >>>>>>>>>>>>>>>>>> Snapshots, rebalance, tooling, tracing, zookeeper
> >>>>>> support,
> >>>>>>>> etc.
> >>>>>>>>>>>>>>>>>> We have bugs and issues that can be fixed in 2.x
> >>>>> without
> >>>>>>>>> breaking
> >>>>>>>>>>>>>>>>> backward
> >>>>>>>>>>>>>>>>>> compatibility.
> >>>>>>>>>>>>>>>>>> We have many users who are happy with the 2.x with
> >>>> all
> >>>>>> it’s
> >>>>>>>>>> issues.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> 2 нояб. 2020 г., в 14:09, Anton Vinogradov <
> >>>>>> av@apache.org
> >>>>>>>>
> >>>>>>>>>>>>>>>> написал(а):
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Alexey,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Do we have any estimates of how fast we'll be able
> >>>> to
> >>>>>> gain
> >>>>>>>>>>>>>>>>>> production-ready
> >>>>>>>>>>>>>>>>>>> AI 3.0 in case of a "new repo" choice?
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk <
> >>>>>>>>>>>>>>>>>> alexey.goncharuk@gmail.com>
> >>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Nikolay,
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> What new features are we planning to implement for
> >>>>>> Ignite
> >>>>>>>>> 2.x?
> >>>>>>>>>> I
> >>>>>>>>>>>>>>>> think
> >>>>>>>>>>>>>>>>>> once
> >>>>>>>>>>>>>>>>>>>> we commence working on Ignite 3.0, we should
> >>>>> gradually
> >>>>>>>> cease
> >>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> activity
> >>>>>>>>>>>>>>>>>>>> on Ignite 2.x to mere bugfixes because such
> >>>> parallel
> >>>>>>>>>> development
> >>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>> overwhelming regardless of how we choose to
> >>>> proceed.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <
> >>>>>>>>>>> nizhikov@apache.org
> >>>>>>>>>>>>>>>> :
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> To be clear:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> I would suggest creating a new repository for
> >>>>> Ignite
> >>>>>>> 3.0
> >>>>>>>>>>>>>>>> (perhaps, a
> >>>>>>>>>>>>>>>>>>>> new
> >>>>>>>>>>>>>>>>>>>>> clean branch, but a new repo looks nicer to me)
> >>>> and
> >>>>> a
> >>>>>>> new
> >>>>>>>>>> Ignite
> >>>>>>>>>>>>>>>> 3.0
> >>>>>>>>>>>>>>>>>>>>> TeamCity project.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> +1 for new Team City project.
> >>>>>>>>>>>>>>>>>>>>> +1 for new branch for Ignite3.
> >>>>>>>>>>>>>>>>>>>>> -1 for new repo.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <
> >>>>>>>>>>>>>>> NIzhikov.dev@gmail.com
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> написал(а):
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Hello, Alexey.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> I think it will hurt our project more than help.
> >>>>>>>>>>>>>>>>>>>>>> Developing new features for 2 separate branches
> >>>>> with
> >>>>>>> the
> >>>>>>>>>>>>>>> different
> >>>>>>>>>>>>>>>>>> APIs
> >>>>>>>>>>>>>>>>>>>>> and internal structure is overwhelming
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Maybe we should relax a bit requirements for
> >>>>> Ignite3?
> >>>>>>>>>>>>>>>>>>>>>> Maybe we should move step by step and make
> >>>> Ignite3
> >>>>>> with
> >>>>>>>> new
> >>>>>>>>>>>>>>>>>>>>> configuration than Ignite4 with new transactions,
> >>>>> etc?
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <
> >>>>>>>>>>>>>>>>>>>> alexey.goncharuk@gmail.com>
> >>>>>>>>>>>>>>>>>>>>> написал(а):
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Igniters,
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> I wanted to pitch a rather radical idea
> >>>> regarding
> >>>>>> the
> >>>>>>>>> Ignite
> >>>>>>>>>>>>>>> 3.0
> >>>>>>>>>>>>>>>>>>>>>>> development which has occurred to me some time
> >>>>> ago.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> We already have several IEPs targeted to Ignite
> >>>>> 3.0
> >>>>>>>> which
> >>>>>>>>>>> imply
> >>>>>>>>>>>>>>>>> major
> >>>>>>>>>>>>>>>>>>>>>>> changes to the codebase (the change in
> >>>> replication
> >>>>>>>>> protocol
> >>>>>>>>>>> and
> >>>>>>>>>>>>>>>>> thus
> >>>>>>>>>>>>>>>>>>>>>>> transactions, change in binary format, updated
> >>>>>>>>> metastorage,
> >>>>>>>>>>>>>>> etc).
> >>>>>>>>>>>>>>>>> We
> >>>>>>>>>>>>>>>>>>>>> also
> >>>>>>>>>>>>>>>>>>>>>>> planned significant changes in public APIs:
> >>>>>>>> configuration
> >>>>>>>>>>>>>>> format
> >>>>>>>>>>>>>>>>>>>> change,
> >>>>>>>>>>>>>>>>>>>>>>> improvements in cache APIs, SQL APIs,
> >>>> transaction
> >>>>>> mode
> >>>>>>>>>> rework.
> >>>>>>>>>>>>>>>> The
> >>>>>>>>>>>>>>>>>>>>> wishlist
> >>>>>>>>>>>>>>>>>>>>>>> of changes for Ignite 3.0 is huge.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> So, I was wondering whether it makes sense to
> >>>> try
> >>>>> to
> >>>>>>>>> change
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>> old
> >>>>>>>>>>>>>>>>>>>>>>> codebase, or start with a new baseline and move
> >>>>> old
> >>>>>>>> pieces
> >>>>>>>>>> of
> >>>>>>>>>>>>>>>> code
> >>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>> do
> >>>>>>>>>>>>>>>>>>>>>>> not require significant rework. Personally, I
> >>>>> would
> >>>>>> go
> >>>>>>>>> with
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> second
> >>>>>>>>>>>>>>>>>>>>>>> option for the following reasons:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> - We have a chance to shift the development
> >>>>> paradigm
> >>>>>>> in
> >>>>>>>>> the
> >>>>>>>>>>>>>>>> project
> >>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>> introduce the practice of true unit-tests. In
> >>>> the
> >>>>>> new
> >>>>>>>>>> baseline
> >>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> beginning there will be no ability to run an
> >>>>>>> end-to-end
> >>>>>>>>>>>>>>> scenario,
> >>>>>>>>>>>>>>>>>>>> thus
> >>>>>>>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>>>>> will be forced to write unit-tests. So far, such
> >>>>>>>> practice
> >>>>>>>>>> was
> >>>>>>>>>>>>>>>> hard
> >>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>> implement because of tight coupling between
> >>>> Ignite
> >>>>>>>>>> components
> >>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>> inability
> >>>>>>>>>>>>>>>>>>>>>>> to instantiate components without an instance of
> >>>>>>>>>>> KernalContext.
> >>>>>>>>>>>>>>>> For
> >>>>>>>>>>>>>>>>>>>>>>> example, we should be able to thoroughly test
> >>>>>> internal
> >>>>>>>>>>>>>>>> primitives,
> >>>>>>>>>>>>>>>>>>>>> such as
> >>>>>>>>>>>>>>>>>>>>>>> replication protocol (without actual
> >>>>> communication),
> >>>>>>>>>>>>>>> distributed
> >>>>>>>>>>>>>>>>>>>>>>> metastorage contracts, persistence layer, etc.
> >>>>>>>>>>>>>>>>>>>>>>> - We will significantly reduce the development
> >>>>> cycle
> >>>>>>> in
> >>>>>>>>> the
> >>>>>>>>>>>>>>>>> beginning
> >>>>>>>>>>>>>>>>>>>>>>> (right now the RunAll takes two hours of
> >>>>>> astronomical
> >>>>>>>> time
> >>>>>>>>>>> with
> >>>>>>>>>>>>>>>>> empty
> >>>>>>>>>>>>>>>>>>>>> TC;
> >>>>>>>>>>>>>>>>>>>>>>> in the new approach developer will be able to
> >>>> run
> >>>>>> ALL
> >>>>>>>>> tests
> >>>>>>>>>>>>>>>> locally
> >>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>> matter of minutes)
> >>>>>>>>>>>>>>>>>>>>>>> - We can get rid of TC bot and enforce green TC
> >>>> by
> >>>>>>>>>> integrating
> >>>>>>>>>>>>>>> TC
> >>>>>>>>>>>>>>>>>>>> build
> >>>>>>>>>>>>>>>>>>>>>>> results with GitHub PRs (the same way Travis is
> >>>>>>>> currently
> >>>>>>>>>>>>>>>>> integrated
> >>>>>>>>>>>>>>>>>>>>> to PR
> >>>>>>>>>>>>>>>>>>>>>>> check). We should restrict PR merge without a TC
> >>>>>> check
> >>>>>>>>>>>>>>>>>>>>>>> - We will still have to re-write all tests, but
> >>>>> only
> >>>>>>>> once.
> >>>>>>>>>> If
> >>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>> try
> >>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>> modify the old codebase, we would need to modify
> >>>>> all
> >>>>>>> the
> >>>>>>>>>> tests
> >>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>> every
> >>>>>>>>>>>>>>>>>>>>>>> major change (public API change, configuration
> >>>>>> change)
> >>>>>>>>>>>>>>>>>>>>>>> - We will have fewer conflicts when working
> >>>>>> together.
> >>>>>>>> For
> >>>>>>>>>>>>>>>> example,
> >>>>>>>>>>>>>>>>> I
> >>>>>>>>>>>>>>>>>>>>>>> cannot imagine how one would merge two changes
> >>>> of
> >>>>>>>> getting
> >>>>>>>>>> rid
> >>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>> IgniteFuture and changes in replication
> >>>> protocol,
> >>>>>> for
> >>>>>>>>>> example
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Technically, I would suggest creating a new
> >>>>>> repository
> >>>>>>>> for
> >>>>>>>>>>>>>>> Ignite
> >>>>>>>>>>>>>>>>> 3.0
> >>>>>>>>>>>>>>>>>>>>>>> (perhaps, a new clean branch, but a new repo
> >>>> looks
> >>>>>>> nicer
> >>>>>>>>> to
> >>>>>>>>>>> me)
> >>>>>>>>>>>>>>>>> and a
> >>>>>>>>>>>>>>>>>>>>> new
> >>>>>>>>>>>>>>>>>>>>>>> Ignite 3.0 TeamCity project.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> While it may seem quite radical, I do believe
> >>>> that
> >>>>>>> this
> >>>>>>>>>>>>>>> approach
> >>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>>> give
> >>>>>>>>>>>>>>>>>>>>>>> us more benefits than trying to make such major
> >>>>>>> changes
> >>>>>>>> in
> >>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>> existing
> >>>>>>>>>>>>>>>>>>>>>>> codebase. If needed, let's schedule a discord
> >>>> chat
> >>>>>>> like
> >>>>>>>>>> before
> >>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>> discuss
> >>>>>>>>>>>>>>>>>>>>>>> this.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> WDYT?
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>> Andrey V. Mashenkov
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>> --
> >>>
> >>> Best regards,
> >>> Alexei Scherbakov
> >
>
>
>
--
Sincerely yours, Ivan Daschinskiy
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Nikolay Izhikov <ni...@apache.org>.
My suggestion:
1. Reduce Ignite3 scope to the following:
a. Delete all deprecated API and support of obsolete internal protocols.
b. Implement IEP-61 - Common Replication Infrastructure
c. Implement new Ignite management tool ignitectl as suggested during Ignite3 discussion.
2. Implement and release following improvements like transactions, Calcite based SQL, etc in the ongoing releases - Ignite 4, 5, 6
My concern against separate Ignite 3 repo is the following:
1. We spread community to the two very separated part - Ignite3 developers and Ignite2 maintainers. believe it’s bad for our community.
That can lead to the situation when we don’t fix critical or blocker issueds «because they will not exists in Ignite3»
That will lead to the solutions never reviewed or reviewed poorly.
2. It seems for me that current scope of Ignite3 is too big to be implemented in any reasonable time.
> 13 нояб. 2020 г., в 10:57, Nikolay Izhikov <NI...@gmail.com> написал(а):
>
> Hello, Valentin.
>
>> Nikolay, Maxim, are you OK with this route?
>
> -1 to have another repo for Ignite3 development.
>
>> 13 нояб. 2020 г., в 03:04, Valentin Kulichenko <va...@gmail.com> написал(а):
>>
>> Folks,
>>
>> We already have multiple IEPs for Ignite 3.0, and as far as I know, there are contributors that would like to work on them (or probably already started). That said, we should make a decision as soon as possible.
>>
>> At this point, it doesn't seem that there are any strong objections to the technical side of things. So I would suggest the following:
>>
>> 1. Proceed with Alexey's approach to the development process, as it seems to be the best (in my opinion - the only) way to address all the technical concerns and issues expressed in the thread. We'll start by creating a new repo and a new TC project.
>> 2. Start a separate discussion around transparency. If there are any changes we need to make to our contributor guidelines, I am happy to talk them through, but I don't think it's reasonable to delay feature development because of this. In the short term, I will make sure that everything that happens within the new repo is as open to the community as possible.
>>
>> Nikolay, Maxim, are you OK with this route?
>>
>> -Val
>>
>> On Tue, Nov 10, 2020 at 4:55 PM Valentin Kulichenko <va...@gmail.com> wrote:
>> Maxim,
>>
>> 2.x and 3.x will have to coexist for some time - I don't see how we can avoid this considering the set of proposed changes. That said, we effectively will need to have two "masters" - one for each major version. Master for 3.x can technically be a branch in the existing repo, but having a separate repo seems cleaner, simply because it will not be a "branch" in the traditional sense.
>>
>> Note that the new repo will still be under the Apache org, with the same set of committers, managed by the community, etc. All the development happening for 3.0 must follow the rules that we currently have (if anything, it's an opportunity to improve those rules).
>>
>> As I said during the call on Friday, I strongly believe that if there is a transparency issue, it will exist regardless of the approach we choose for 3.0. If community members develop without IEPs or public discussions, this will happen for both 2.x and 3.x unless we address this separately. I don't see how this is related to Alexey's suggestion, which targets *technical* issues with the product more than anything else. This a way to achieve better modularity, introduce better coverage with unit tests, reduce conflicts during development, etc.
>>
>> Coming back to transparency, let's identify the issues and fix them. It probably makes sense to have a separate discussion on this topic.
>>
>> -Val
>>
>> On Tue, Nov 10, 2020 at 1:05 PM Maxim Muzafarov <mm...@apache.org> wrote:
>> Sergey,
>>
>>
>> Your summary makes sense to me.
>>
>> However, how we come up from *Development transparency* to *create a
>> separate public repository dedicated for 3.0*? For me *development
>> transparency* is about making changes in the master branch. These
>> changes will definitely be seen by all the Ignite developers.
>>
>> A dedicated public repository is technically public and visible for
>> everyone, but it allows development without IEPs, without public
>> discussion (since all the code changes are not related to the master
>> branch) it also allows a large number of assumptions and deviations
>> (like code-style violations). It also not about *development
>> transparency* since developers which are working on 3.0 is only a
>> subset of all Ignite developers which may continue working on 2.x. For
>> me, this would be a huge step backwards.
>>
>> Ignite veterans should remember how long the branch stabilization took
>> for the 2.x version with the PDS.
>>
>>
>> I think each breaking change should be passed through the master branch.
>>
>> On Tue, 10 Nov 2020 at 22:18, Alexei Scherbakov
>> <al...@gmail.com> wrote:
>>>
>>> Makes sense to me.
>>>
>>> вт, 10 нояб. 2020 г. в 18:47, Sergey Chugunov <se...@gmail.com>:
>>>
>>>> Igniters,
>>>>
>>>> I thought over Friday meeting ideas and concerns and summarized them in
>>>> these three points:
>>>>
>>>>
>>>> 1. *Components design unification approach.* New proposed components
>>>> will be developed by different contributors, but they need to be unified
>>>> and should integrate with each other easily. To ensure that I suggest
>>>> calling an architecture group that will create design guidelines for all
>>>> components and high-level overview of overall architecture. How code is
>>>> split into components, what are component boundaries, how component
>>>> lifecycle works and what are its interfaces - all these and other
>>>> questions
>>>> should be covered.
>>>>
>>>> 2. *Scope management.* Apache 3.0 should be implemented within a
>>>> reasonable time, so we need some procedure to decide whether a
>>>> particular
>>>> feature should be dropped from the scope of 3.0 and postponed to 3.1
>>>> release. To do so I suggest to range all features by two parameters:
>>>> criticality for 3.0 and amount of breaking changes. 3.0 scope should
>>>> include features of high criticality AND features with a big amount of
>>>> breaking changes. All other features can be made optional.
>>>>
>>>> 3. *Development transparency.* Development of all components should be
>>>> made as transparent for everyone as possible. Any contributor should be
>>>> able to look over any component at any stage of development. To achieve
>>>> this I suggest to create a separate public repository dedicated for 3.0
>>>> development. It will make the code available for everyone but when
>>>> development of 3.0 is done we won't loose any stars of our current
>>>> repository as we merge dev repo into main one and drop dev.
>>>>
>>>> Do these ideas make sense to you? Are there any concerns not covered by
>>>> these suggestions?
>>>>
>>>> On Fri, Nov 6, 2020 at 7:36 PM Kseniya Romanova <romanova.ks.spb@gmail.com
>>>>>
>>>> wrote:
>>>>
>>>>> Here are the slides from Alexey Goncharuk. Let's think this over and
>>>>> continue on Monday:
>>>>>
>>>>>
>>>> https://go.gridgain.com/rs/491-TWR-806/images/Ignite_3_Plans_and_development_process.pdf
>>>>>
>>>>> чт, 5 нояб. 2020 г. в 11:13, Anton Vinogradov <av...@apache.org>:
>>>>>
>>>>>> Folks,
>>>>>>
>>>>>> Should we perform cleanup work before (r)evolutional changes?
>>>>>> My huge proposal is to get rid of things which we don't need anyway
>>>>>> - local caches,
>>>>>> - strange tx modes,
>>>>>> - code overcomplexity because of RollingUpgrade feature never attended
>>>> at
>>>>>> AI,
>>>>>> - etc,
>>>>>> before choosing the way.
>>>>>>
>>>>>> On Tue, Nov 3, 2020 at 3:31 PM Valentin Kulichenko <
>>>>>> valentin.kulichenko@gmail.com> wrote:
>>>>>>
>>>>>>> Ksenia, thanks for scheduling this on such short notice!
>>>>>>>
>>>>>>> As for the original topic, I do support Alexey's idea. We're not
>>>> going
>>>>> to
>>>>>>> rewrite anything from scratch, as most of the components are going to
>>>>> be
>>>>>>> moved as-is or with minimal modifications. However, the changes that
>>>>> are
>>>>>>> proposed imply serious rework of the core parts of the code, which
>>>> are
>>>>>> not
>>>>>>> properly decoupled from each other and from other parts. This makes
>>>> the
>>>>>>> incremental approach borderline impossible. Developing in a new repo,
>>>>>>> however, addresses this concern. As a bonus, we can also refactor the
>>>>>> code,
>>>>>>> introduce better decoupling, get rid of kernel context, and develop
>>>>> unit
>>>>>>> tests (finally!).
>>>>>>>
>>>>>>> Basically, this proposal only affects the *process*, not the set of
>>>>>> changes
>>>>>>> we had discussed before. Ignite 3.0 is our unique chance to make
>>>> things
>>>>>>> right.
>>>>>>>
>>>>>>> -Val
>>>>>>>
>>>>>>> On Tue, Nov 3, 2020 at 3:06 AM Kseniya Romanova <
>>>>>> romanova.ks.spb@gmail.com
>>>>>>>>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Pavel, all the interesting points will be anyway published here in
>>>>>>> English
>>>>>>>> (as the principal "if it's not on devlist it doesn't happened" is
>>>>> still
>>>>>>>> relevant). This is just a quick call for a group of developers.
>>>> Later
>>>>>> we
>>>>>>>> can do a separate presentation of idea and discussion in English as
>>>>> we
>>>>>>> did
>>>>>>>> for the Ignite 3.0 draft of changes.
>>>>>>>>
>>>>>>>> вт, 3 нояб. 2020 г. в 13:52, Pavel Tupitsyn <ptupitsyn@apache.org
>>>>> :
>>>>>>>>
>>>>>>>>> Kseniya,
>>>>>>>>>
>>>>>>>>> Thanks for scheduling this call.
>>>>>>>>> Do you think we can switch to English if non-Russian speaking
>>>>>> community
>>>>>>>>> members decide to join?
>>>>>>>>>
>>>>>>>>> On Tue, Nov 3, 2020 at 1:32 PM Kseniya Romanova <
>>>>>>>> romanova.ks.spb@gmail.com
>>>>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Let's do this community discussion open. Here's the link on
>>>> zoom
>>>>>> call
>>>>>>>> in
>>>>>>>>>> Russian for Friday 6 PM:
>>>>>>>>>>
>>>>>> https://www.meetup.com/Moscow-Apache-Ignite-Meetup/events/274360378/
>>>>>>>>>>
>>>>>>>>>> вт, 3 нояб. 2020 г. в 12:49, Nikolay Izhikov <
>>>>> nizhikov@apache.org
>>>>>>> :
>>>>>>>>>>
>>>>>>>>>>> Time works for me.
>>>>>>>>>>>
>>>>>>>>>>>> 3 нояб. 2020 г., в 12:40, Alexey Goncharuk <
>>>>>>>>> alexey.goncharuk@gmail.com
>>>>>>>>>>>
>>>>>>>>>>> написал(а):
>>>>>>>>>>>>
>>>>>>>>>>>> Nikolay,
>>>>>>>>>>>>
>>>>>>>>>>>> I am up for the call. I will try to explain my reasoning in
>>>>>>> greater
>>>>>>>>>>> detail
>>>>>>>>>>>> and will be glad to hear the concerns. Will this Friday,
>>>> Nov
>>>>>> 6th,
>>>>>>>>> work?
>>>>>>>>>>>>
>>>>>>>>>>>> вт, 3 нояб. 2020 г. в 10:09, Nikolay Izhikov <
>>>>>>> nizhikov@apache.org
>>>>>>>>> :
>>>>>>>>>>>>
>>>>>>>>>>>>> Igniters, should we have a call for this topic?
>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2 нояб. 2020 г., в 18:53, Pavel Tupitsyn <
>>>>>> ptupitsyn@apache.org
>>>>>>>>
>>>>>>>>>>>>> написал(а):
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> not intend to rewrite everything from scratch
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Every single test from Ignite 2.x should be moved to
>>>>> Ignite
>>>>>> 3
>>>>>>>>>>>>>>> regardless of how we choose to proceed.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Alexey, thank you for the explanation, this addresses all
>>>>> of
>>>>>> my
>>>>>>>>>>> concerns.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Nov 2, 2020 at 6:43 PM Andrey Mashenkov <
>>>>>>>>>>>>> andrey.mashenkov@gmail.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi, Igniters.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> * AFAIU, we need a new repo if we want to apply
>>>> different
>>>>>>>>>> restrictions
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> pull requests,
>>>>>>>>>>>>>>> otherwise I see no difference for myself.
>>>>>>>>>>>>>>> E.g. make static analysis (do we have?), compile,
>>>> styles,
>>>>>> and
>>>>>>>>>> javadoc
>>>>>>>>>>>>>>> checks mandatory.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I think that relaxed requirements here will lead to bad
>>>>>>> product
>>>>>>>>>>> quality.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> * Agree with Pavel, we should 'keep' integrations tests
>>>>>>> somehow.
>>>>>>>>>>>>>>> During active development tests will be broken most of
>>>>> time,
>>>>>>> so,
>>>>>>>>>>>>>>> I'd port them e.g. suite-by-suite once we will have a
>>>>> stable
>>>>>>> and
>>>>>>>>>>>>> featured
>>>>>>>>>>>>>>> environment to run them and of course make test's code
>>>>> clear
>>>>>>> and
>>>>>>>>>> avoid
>>>>>>>>>>>>>>> bad/non-relevant ones.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> * I like bottom-up approach.
>>>>>>>>>>>>>>> With it we could make a better framework. I mean clear
>>>>>>> component
>>>>>>>>>>>>> lifecycle,
>>>>>>>>>>>>>>> component wiring mechanics, general methods to approach
>>>>> core
>>>>>>>>>>> components
>>>>>>>>>>>>>>> such as exchange/communication
>>>>>>>>>>>>>>> to avoid code mess like we have in ExchangeFuture with
>>>> all
>>>>>>> these
>>>>>>>>>>> custom
>>>>>>>>>>>>>>> callbacks for each component, interfaces like
>>>>>>>>>>>>>>> PartitionsExchangeAware, IgniteChangeGlobalStateSupport
>>>>> and
>>>>>>>>>>>>>>> a pack of
>>>>>>>>>>>>>
>>>>>> start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected
>>>>>>>>>>>>>>> and so on in various unexpected places.
>>>>>>>>>>>>>>> Hope, we will be able to port most of the good code to
>>>> the
>>>>>> new
>>>>>>>>>>> framework
>>>>>>>>>>>>>>> version.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk <
>>>>>>>>>>>>>>> alexey.goncharuk@gmail.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Nikolay, Pavel,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks for the feedback! First of all, I wanted to
>>>> stress
>>>>>>> that
>>>>>>>> I
>>>>>>>>> do
>>>>>>>>>>> not
>>>>>>>>>>>>>>>> intend to rewrite everything from scratch (I never used
>>>>>> this
>>>>>>>>>> phrase).
>>>>>>>>>>>>>>> There
>>>>>>>>>>>>>>>> are significant parts of code that would be moved with
>>>>>>> minimal
>>>>>>>>>>>>>>>> modifications. Second, I never said that we will get
>>>> rid
>>>>> of
>>>>>>> the
>>>>>>>>> old
>>>>>>>>>>>>> tests
>>>>>>>>>>>>>>>> codebase. Every single test from Ignite 2.x should be
>>>>> moved
>>>>>>> to
>>>>>>>>>>> Ignite 3
>>>>>>>>>>>>>>>> regardless of how we choose to proceed.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> My point is that for some parts of the code a clean
>>>>>> bottom-up
>>>>>>>>>>>>>>>> implementation will be cheaper in many ways. Let me
>>>> give
>>>>>> you
>>>>>>> a
>>>>>>>>> few
>>>>>>>>>>>>>>> concrete
>>>>>>>>>>>>>>>> examples:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> - I think no one can object that we need a cleanly
>>>>>> separated
>>>>>>>>>>>>>>> persistence
>>>>>>>>>>>>>>>> layer for Ignite. There is a very raw draft IEP for
>>>> this
>>>>>>>>> already.
>>>>>>>>>> On
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> other hand, I think we also can agree that we need a
>>>>>>>> split-brain
>>>>>>>>>>>>>>>> resistant
>>>>>>>>>>>>>>>> replication protocol for caches. There is also an IEP
>>>>> for
>>>>>>>> this.
>>>>>>>>>>>>>>> Neither
>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>> the changes is a good fit for 2.x because they are
>>>>> likely
>>>>>> to
>>>>>>>>>>>>> introduce
>>>>>>>>>>>>>>>> breaking changes in the persistence layer,
>>>> configuration
>>>>>> and
>>>>>>>>>>>>> behavior.
>>>>>>>>>>>>>>>> Additionally, these components are now tightly
>>>> coupled,
>>>>> so
>>>>>>>> there
>>>>>>>>>> is
>>>>>>>>>>>>> no
>>>>>>>>>>>>>>>> way
>>>>>>>>>>>>>>>> these two changes can be implemented in parallel and
>>>>> then
>>>>>>>> merged
>>>>>>>>>>>>>>>> together
>>>>>>>>>>>>>>>> easily. So what we will end up with is having to
>>>>> implement
>>>>>>>> these
>>>>>>>>>>>>>>> changes
>>>>>>>>>>>>>>>> sequentially, fixing all existing tests twice, and
>>>>>>> essentially
>>>>>>>>>>>>>>> throwing
>>>>>>>>>>>>>>>> away half of the work done because the other part of
>>>> the
>>>>>>>> change
>>>>>>>>> is
>>>>>>>>>>>>>>>> re-implemented
>>>>>>>>>>>>>>>> - Similar example goes with getting rid of
>>>>>>>> IgniteInternalFuture
>>>>>>>>>> and
>>>>>>>>>>>>>>>> replacing it with CompletableFuture, and any other
>>>>> change
>>>>>>> that
>>>>>>>>>>>>> touches
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> asynchronous part of the code.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Third, I do not see how this choice affects the UX of
>>>>>> Ignite.
>>>>>>>> The
>>>>>>>>>> end
>>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>> experience must be fixed in the IEP regardless of the
>>>>>>>> development
>>>>>>>>>>>>> process
>>>>>>>>>>>>>>>> and the fact that we have gaps in this area in Ignite
>>>> 2.x
>>>>>>> just
>>>>>>>>>>> confirms
>>>>>>>>>>>>>>>> that.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Pavel, agree that a repo/branch is a technicality, I
>>>>> guess
>>>>>> if
>>>>>>>>>>>>>>> reformulate,
>>>>>>>>>>>>>>>> my point is that we might agree to have a single
>>>>>> development
>>>>>>>>> master
>>>>>>>>>>>>>>> branch
>>>>>>>>>>>>>>>> with 'disassembled' end-to-end functionality for some
>>>>>> period
>>>>>>> of
>>>>>>>>>> time
>>>>>>>>>>> to
>>>>>>>>>>>>>>>> speed up development, and re-assemble the core features
>>>>>> after
>>>>>>>>>> having
>>>>>>>>>>>>>>>> submodules tested independently.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Nikolay,
>>>>>>>>>>>>>>>>> We have many features that have to evolve.
>>>>>>>>>>>>>>>>> Snapshots, rebalance, tooling, tracing, zookeeper
>>>>> support,
>>>>>>>> etc.
>>>>>>>>>>>>>>>> This is not very specific. In the end, resources are
>>>>>> limited
>>>>>>>> and
>>>>>>>>> we
>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>> not be able to drive both tracks simultaneously,
>>>>> especially
>>>>>>>>> after a
>>>>>>>>>>>>>>> couple
>>>>>>>>>>>>>>>> of features having been implemented for Ignite 3.0. If
>>>>>> there
>>>>>>>> are
>>>>>>>>>>> indeed
>>>>>>>>>>>>>>>> some major changes that we want to do in Ignite 2.x
>>>>> instead
>>>>>>> of
>>>>>>>>>>> putting
>>>>>>>>>>>>>>>> effort into 3.0 - let's discuss them. I am just not
>>>> aware
>>>>>> of
>>>>>>>> any,
>>>>>>>>>>>>> that's
>>>>>>>>>>>>>>>> why I am eager to move forward with Ignite 3.0.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> We have bugs and issues that can be fixed in 2.x
>>>> without
>>>>>>>>> breaking
>>>>>>>>>>>>>>> backward
>>>>>>>>>>>>>>>> compatibility.
>>>>>>>>>>>>>>>>> We have many users who are happy with the 2.x with all
>>>>>> it’s
>>>>>>>>>> issues.
>>>>>>>>>>>>>>>> These changes will be covered by end-to-end tests and
>>>>>>> migrated
>>>>>>>> to
>>>>>>>>>>>>> Ignite
>>>>>>>>>>>>>>>> 3.0, so I see no issues here.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Finally, Anton & Nikolay
>>>>>>>>>>>>>>>> I do not have an estimate for this simply because the
>>>>>>> activity
>>>>>>>> is
>>>>>>>>>>>>>>>> community-driven and it depends on the number of people
>>>>>>> willing
>>>>>>>>> to
>>>>>>>>>>>>>>>> contribute. With the current pace, I would hope to have
>>>>> an
>>>>>> RC
>>>>>>>> of
>>>>>>>>>>> Ignite
>>>>>>>>>>>>>>> 3.0
>>>>>>>>>>>>>>>> to be ready by the end of 2021. My gut feeling is that
>>>> by
>>>>>>>> moving
>>>>>>>>>> with
>>>>>>>>>>>>>>>> incremental changes, we will not be able to implement
>>>>> even
>>>>>>> half
>>>>>>>>> of
>>>>>>>>>>> the
>>>>>>>>>>>>>>>> wishlist by that time.
>>>>>>>>>>>>>>>> I doubt that releasing several major releases with
>>>>> breaking
>>>>>>>>> changes
>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>> make Ignite users happy either because each upgrade
>>>> will
>>>>>> cost
>>>>>>>>>> Ignite
>>>>>>>>>>>>>>> users
>>>>>>>>>>>>>>>> money, so the fewer major versions we release, the
>>>>> better.
>>>>>>> Thus
>>>>>>>>> my
>>>>>>>>>>> wish
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> include all breaking changes in one release.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I'll be now quiet for a while, let's see what other
>>>>>> community
>>>>>>>>>> members
>>>>>>>>>>>>>>>> think.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <
>>>>>>>>> ptupitsyn@apache.org
>>>>>>>>>>> :
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 1. Rewriting from scratch is never a good idea.
>>>>>>>>>>>>>>>>> We don't want to follow the path of Netscape and lose
>>>>> all
>>>>>>> our
>>>>>>>>>> users
>>>>>>>>>>>>>>>>> by the time we have a working 3.0 [1]
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2. Not sure about new repo - seems like some pain and
>>>> no
>>>>>>> gain,
>>>>>>>>>>> what's
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> problem with a branch?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 3. We should keep existing integration tests when
>>>>>> possible.
>>>>>>>>>>>>>>>>> We have accumulated a lot of edge case knowledge over
>>>>> the
>>>>>>>> years,
>>>>>>>>>>>>>>>>> it is not a good idea to send all of that down the
>>>>> drain.
>>>>>>>>>>>>>>>>> Yes, integration tests are slow, but they are the most
>>>>>>>> valuable.
>>>>>>>>>>>>>>>>> I think we can move more stuff into nightly runs and
>>>>> have
>>>>>> a
>>>>>>>> fast
>>>>>>>>>> and
>>>>>>>>>>>>>>>> modern
>>>>>>>>>>>>>>>>> basic suite.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Alexey, you are much more familiar with the Ignite
>>>> core
>>>>>>>> codebase
>>>>>>>>>>> than
>>>>>>>>>>>>>>>> most
>>>>>>>>>>>>>>>>> of us,
>>>>>>>>>>>>>>>>> can you please explain in more detail which particular
>>>>>>>> feature,
>>>>>>>>> in
>>>>>>>>>>>>> your
>>>>>>>>>>>>>>>>> opinion,
>>>>>>>>>>>>>>>>> mandates this "start from scratch" approach?
>>>>>>>>>>>>>>>>> Is it really not possible at all to follow a less
>>>>> radical
>>>>>>> way?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>> https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <
>>>>>>>>>> nizhikov@apache.org
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hello, Alexey.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I think that «rewriting from scratch» approach has a
>>>>> high
>>>>>>>> risk
>>>>>>>>> to
>>>>>>>>>>>>>>> make
>>>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>> features unusable.
>>>>>>>>>>>>>>>>>> At the time Ignite2 was started no-one wants to do
>>>> bad
>>>>> UX
>>>>>>> or
>>>>>>>>> bad
>>>>>>>>>>>>>>>>> features.
>>>>>>>>>>>>>>>>>> Nevertheless, it happen.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I think we can avoid it with the Ignite3 and
>>>> successors
>>>>>> if
>>>>>>> we
>>>>>>>>>> will
>>>>>>>>>>>>>>> move
>>>>>>>>>>>>>>>>>> step by step without keeping backward compatibility
>>>>>>>>>>>>>>>>>> With the step by step approach, we can focus on each
>>>>>>>> component
>>>>>>>>>>>>>>>>> separately.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> What new features are we planning to implement for
>>>>>> Ignite
>>>>>>>> 2.x?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> We have many features that have to evolve.
>>>>>>>>>>>>>>>>>> Snapshots, rebalance, tooling, tracing, zookeeper
>>>>>> support,
>>>>>>>> etc.
>>>>>>>>>>>>>>>>>> We have bugs and issues that can be fixed in 2.x
>>>>> without
>>>>>>>>> breaking
>>>>>>>>>>>>>>>>> backward
>>>>>>>>>>>>>>>>>> compatibility.
>>>>>>>>>>>>>>>>>> We have many users who are happy with the 2.x with
>>>> all
>>>>>> it’s
>>>>>>>>>> issues.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 2 нояб. 2020 г., в 14:09, Anton Vinogradov <
>>>>>> av@apache.org
>>>>>>>>
>>>>>>>>>>>>>>>> написал(а):
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Alexey,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Do we have any estimates of how fast we'll be able
>>>> to
>>>>>> gain
>>>>>>>>>>>>>>>>>> production-ready
>>>>>>>>>>>>>>>>>>> AI 3.0 in case of a "new repo" choice?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk <
>>>>>>>>>>>>>>>>>> alexey.goncharuk@gmail.com>
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Nikolay,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> What new features are we planning to implement for
>>>>>> Ignite
>>>>>>>>> 2.x?
>>>>>>>>>> I
>>>>>>>>>>>>>>>> think
>>>>>>>>>>>>>>>>>> once
>>>>>>>>>>>>>>>>>>>> we commence working on Ignite 3.0, we should
>>>>> gradually
>>>>>>>> cease
>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> activity
>>>>>>>>>>>>>>>>>>>> on Ignite 2.x to mere bugfixes because such
>>>> parallel
>>>>>>>>>> development
>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>> overwhelming regardless of how we choose to
>>>> proceed.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <
>>>>>>>>>>> nizhikov@apache.org
>>>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> To be clear:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I would suggest creating a new repository for
>>>>> Ignite
>>>>>>> 3.0
>>>>>>>>>>>>>>>> (perhaps, a
>>>>>>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>>>> clean branch, but a new repo looks nicer to me)
>>>> and
>>>>> a
>>>>>>> new
>>>>>>>>>> Ignite
>>>>>>>>>>>>>>>> 3.0
>>>>>>>>>>>>>>>>>>>>> TeamCity project.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> +1 for new Team City project.
>>>>>>>>>>>>>>>>>>>>> +1 for new branch for Ignite3.
>>>>>>>>>>>>>>>>>>>>> -1 for new repo.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <
>>>>>>>>>>>>>>> NIzhikov.dev@gmail.com
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> написал(а):
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hello, Alexey.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I think it will hurt our project more than help.
>>>>>>>>>>>>>>>>>>>>>> Developing new features for 2 separate branches
>>>>> with
>>>>>>> the
>>>>>>>>>>>>>>> different
>>>>>>>>>>>>>>>>>> APIs
>>>>>>>>>>>>>>>>>>>>> and internal structure is overwhelming
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Maybe we should relax a bit requirements for
>>>>> Ignite3?
>>>>>>>>>>>>>>>>>>>>>> Maybe we should move step by step and make
>>>> Ignite3
>>>>>> with
>>>>>>>> new
>>>>>>>>>>>>>>>>>>>>> configuration than Ignite4 with new transactions,
>>>>> etc?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <
>>>>>>>>>>>>>>>>>>>> alexey.goncharuk@gmail.com>
>>>>>>>>>>>>>>>>>>>>> написал(а):
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Igniters,
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I wanted to pitch a rather radical idea
>>>> regarding
>>>>>> the
>>>>>>>>> Ignite
>>>>>>>>>>>>>>> 3.0
>>>>>>>>>>>>>>>>>>>>>>> development which has occurred to me some time
>>>>> ago.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> We already have several IEPs targeted to Ignite
>>>>> 3.0
>>>>>>>> which
>>>>>>>>>>> imply
>>>>>>>>>>>>>>>>> major
>>>>>>>>>>>>>>>>>>>>>>> changes to the codebase (the change in
>>>> replication
>>>>>>>>> protocol
>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> thus
>>>>>>>>>>>>>>>>>>>>>>> transactions, change in binary format, updated
>>>>>>>>> metastorage,
>>>>>>>>>>>>>>> etc).
>>>>>>>>>>>>>>>>> We
>>>>>>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>>>>>>> planned significant changes in public APIs:
>>>>>>>> configuration
>>>>>>>>>>>>>>> format
>>>>>>>>>>>>>>>>>>>> change,
>>>>>>>>>>>>>>>>>>>>>>> improvements in cache APIs, SQL APIs,
>>>> transaction
>>>>>> mode
>>>>>>>>>> rework.
>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>>> wishlist
>>>>>>>>>>>>>>>>>>>>>>> of changes for Ignite 3.0 is huge.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> So, I was wondering whether it makes sense to
>>>> try
>>>>> to
>>>>>>>>> change
>>>>>>>>>>> the
>>>>>>>>>>>>>>>> old
>>>>>>>>>>>>>>>>>>>>>>> codebase, or start with a new baseline and move
>>>>> old
>>>>>>>> pieces
>>>>>>>>>> of
>>>>>>>>>>>>>>>> code
>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>> do
>>>>>>>>>>>>>>>>>>>>>>> not require significant rework. Personally, I
>>>>> would
>>>>>> go
>>>>>>>>> with
>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> second
>>>>>>>>>>>>>>>>>>>>>>> option for the following reasons:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> - We have a chance to shift the development
>>>>> paradigm
>>>>>>> in
>>>>>>>>> the
>>>>>>>>>>>>>>>> project
>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>> introduce the practice of true unit-tests. In
>>>> the
>>>>>> new
>>>>>>>>>> baseline
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> beginning there will be no ability to run an
>>>>>>> end-to-end
>>>>>>>>>>>>>>> scenario,
>>>>>>>>>>>>>>>>>>>> thus
>>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>> will be forced to write unit-tests. So far, such
>>>>>>>> practice
>>>>>>>>>> was
>>>>>>>>>>>>>>>> hard
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>> implement because of tight coupling between
>>>> Ignite
>>>>>>>>>> components
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>> inability
>>>>>>>>>>>>>>>>>>>>>>> to instantiate components without an instance of
>>>>>>>>>>> KernalContext.
>>>>>>>>>>>>>>>> For
>>>>>>>>>>>>>>>>>>>>>>> example, we should be able to thoroughly test
>>>>>> internal
>>>>>>>>>>>>>>>> primitives,
>>>>>>>>>>>>>>>>>>>>> such as
>>>>>>>>>>>>>>>>>>>>>>> replication protocol (without actual
>>>>> communication),
>>>>>>>>>>>>>>> distributed
>>>>>>>>>>>>>>>>>>>>>>> metastorage contracts, persistence layer, etc.
>>>>>>>>>>>>>>>>>>>>>>> - We will significantly reduce the development
>>>>> cycle
>>>>>>> in
>>>>>>>>> the
>>>>>>>>>>>>>>>>> beginning
>>>>>>>>>>>>>>>>>>>>>>> (right now the RunAll takes two hours of
>>>>>> astronomical
>>>>>>>> time
>>>>>>>>>>> with
>>>>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>>>>> TC;
>>>>>>>>>>>>>>>>>>>>>>> in the new approach developer will be able to
>>>> run
>>>>>> ALL
>>>>>>>>> tests
>>>>>>>>>>>>>>>> locally
>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>> matter of minutes)
>>>>>>>>>>>>>>>>>>>>>>> - We can get rid of TC bot and enforce green TC
>>>> by
>>>>>>>>>> integrating
>>>>>>>>>>>>>>> TC
>>>>>>>>>>>>>>>>>>>> build
>>>>>>>>>>>>>>>>>>>>>>> results with GitHub PRs (the same way Travis is
>>>>>>>> currently
>>>>>>>>>>>>>>>>> integrated
>>>>>>>>>>>>>>>>>>>>> to PR
>>>>>>>>>>>>>>>>>>>>>>> check). We should restrict PR merge without a TC
>>>>>> check
>>>>>>>>>>>>>>>>>>>>>>> - We will still have to re-write all tests, but
>>>>> only
>>>>>>>> once.
>>>>>>>>>> If
>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>> try
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>> modify the old codebase, we would need to modify
>>>>> all
>>>>>>> the
>>>>>>>>>> tests
>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>> every
>>>>>>>>>>>>>>>>>>>>>>> major change (public API change, configuration
>>>>>> change)
>>>>>>>>>>>>>>>>>>>>>>> - We will have fewer conflicts when working
>>>>>> together.
>>>>>>>> For
>>>>>>>>>>>>>>>> example,
>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>>> cannot imagine how one would merge two changes
>>>> of
>>>>>>>> getting
>>>>>>>>>> rid
>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>> IgniteFuture and changes in replication
>>>> protocol,
>>>>>> for
>>>>>>>>>> example
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Technically, I would suggest creating a new
>>>>>> repository
>>>>>>>> for
>>>>>>>>>>>>>>> Ignite
>>>>>>>>>>>>>>>>> 3.0
>>>>>>>>>>>>>>>>>>>>>>> (perhaps, a new clean branch, but a new repo
>>>> looks
>>>>>>> nicer
>>>>>>>>> to
>>>>>>>>>>> me)
>>>>>>>>>>>>>>>>> and a
>>>>>>>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>>>>>> Ignite 3.0 TeamCity project.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> While it may seem quite radical, I do believe
>>>> that
>>>>>>> this
>>>>>>>>>>>>>>> approach
>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>> give
>>>>>>>>>>>>>>>>>>>>>>> us more benefits than trying to make such major
>>>>>>> changes
>>>>>>>> in
>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> existing
>>>>>>>>>>>>>>>>>>>>>>> codebase. If needed, let's schedule a discord
>>>> chat
>>>>>>> like
>>>>>>>>>> before
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> discuss
>>>>>>>>>>>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> WDYT?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>> Andrey V. Mashenkov
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> Best regards,
>>> Alexei Scherbakov
>
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Nikolay Izhikov <ni...@apache.org>.
Hello, Valentin.
> Nikolay, Maxim, are you OK with this route?
-1 to have another repo for Ignite3 development.
> 13 нояб. 2020 г., в 03:04, Valentin Kulichenko <va...@gmail.com> написал(а):
>
> Folks,
>
> We already have multiple IEPs for Ignite 3.0, and as far as I know, there are contributors that would like to work on them (or probably already started). That said, we should make a decision as soon as possible.
>
> At this point, it doesn't seem that there are any strong objections to the technical side of things. So I would suggest the following:
>
> 1. Proceed with Alexey's approach to the development process, as it seems to be the best (in my opinion - the only) way to address all the technical concerns and issues expressed in the thread. We'll start by creating a new repo and a new TC project.
> 2. Start a separate discussion around transparency. If there are any changes we need to make to our contributor guidelines, I am happy to talk them through, but I don't think it's reasonable to delay feature development because of this. In the short term, I will make sure that everything that happens within the new repo is as open to the community as possible.
>
> Nikolay, Maxim, are you OK with this route?
>
> -Val
>
> On Tue, Nov 10, 2020 at 4:55 PM Valentin Kulichenko <va...@gmail.com> wrote:
> Maxim,
>
> 2.x and 3.x will have to coexist for some time - I don't see how we can avoid this considering the set of proposed changes. That said, we effectively will need to have two "masters" - one for each major version. Master for 3.x can technically be a branch in the existing repo, but having a separate repo seems cleaner, simply because it will not be a "branch" in the traditional sense.
>
> Note that the new repo will still be under the Apache org, with the same set of committers, managed by the community, etc. All the development happening for 3.0 must follow the rules that we currently have (if anything, it's an opportunity to improve those rules).
>
> As I said during the call on Friday, I strongly believe that if there is a transparency issue, it will exist regardless of the approach we choose for 3.0. If community members develop without IEPs or public discussions, this will happen for both 2.x and 3.x unless we address this separately. I don't see how this is related to Alexey's suggestion, which targets *technical* issues with the product more than anything else. This a way to achieve better modularity, introduce better coverage with unit tests, reduce conflicts during development, etc.
>
> Coming back to transparency, let's identify the issues and fix them. It probably makes sense to have a separate discussion on this topic.
>
> -Val
>
> On Tue, Nov 10, 2020 at 1:05 PM Maxim Muzafarov <mm...@apache.org> wrote:
> Sergey,
>
>
> Your summary makes sense to me.
>
> However, how we come up from *Development transparency* to *create a
> separate public repository dedicated for 3.0*? For me *development
> transparency* is about making changes in the master branch. These
> changes will definitely be seen by all the Ignite developers.
>
> A dedicated public repository is technically public and visible for
> everyone, but it allows development without IEPs, without public
> discussion (since all the code changes are not related to the master
> branch) it also allows a large number of assumptions and deviations
> (like code-style violations). It also not about *development
> transparency* since developers which are working on 3.0 is only a
> subset of all Ignite developers which may continue working on 2.x. For
> me, this would be a huge step backwards.
>
> Ignite veterans should remember how long the branch stabilization took
> for the 2.x version with the PDS.
>
>
> I think each breaking change should be passed through the master branch.
>
> On Tue, 10 Nov 2020 at 22:18, Alexei Scherbakov
> <al...@gmail.com> wrote:
> >
> > Makes sense to me.
> >
> > вт, 10 нояб. 2020 г. в 18:47, Sergey Chugunov <se...@gmail.com>:
> >
> > > Igniters,
> > >
> > > I thought over Friday meeting ideas and concerns and summarized them in
> > > these three points:
> > >
> > >
> > > 1. *Components design unification approach.* New proposed components
> > > will be developed by different contributors, but they need to be unified
> > > and should integrate with each other easily. To ensure that I suggest
> > > calling an architecture group that will create design guidelines for all
> > > components and high-level overview of overall architecture. How code is
> > > split into components, what are component boundaries, how component
> > > lifecycle works and what are its interfaces - all these and other
> > > questions
> > > should be covered.
> > >
> > > 2. *Scope management.* Apache 3.0 should be implemented within a
> > > reasonable time, so we need some procedure to decide whether a
> > > particular
> > > feature should be dropped from the scope of 3.0 and postponed to 3.1
> > > release. To do so I suggest to range all features by two parameters:
> > > criticality for 3.0 and amount of breaking changes. 3.0 scope should
> > > include features of high criticality AND features with a big amount of
> > > breaking changes. All other features can be made optional.
> > >
> > > 3. *Development transparency.* Development of all components should be
> > > made as transparent for everyone as possible. Any contributor should be
> > > able to look over any component at any stage of development. To achieve
> > > this I suggest to create a separate public repository dedicated for 3.0
> > > development. It will make the code available for everyone but when
> > > development of 3.0 is done we won't loose any stars of our current
> > > repository as we merge dev repo into main one and drop dev.
> > >
> > > Do these ideas make sense to you? Are there any concerns not covered by
> > > these suggestions?
> > >
> > > On Fri, Nov 6, 2020 at 7:36 PM Kseniya Romanova <romanova.ks.spb@gmail.com
> > > >
> > > wrote:
> > >
> > > > Here are the slides from Alexey Goncharuk. Let's think this over and
> > > > continue on Monday:
> > > >
> > > >
> > > https://go.gridgain.com/rs/491-TWR-806/images/Ignite_3_Plans_and_development_process.pdf
> > > >
> > > > чт, 5 нояб. 2020 г. в 11:13, Anton Vinogradov <av...@apache.org>:
> > > >
> > > > > Folks,
> > > > >
> > > > > Should we perform cleanup work before (r)evolutional changes?
> > > > > My huge proposal is to get rid of things which we don't need anyway
> > > > > - local caches,
> > > > > - strange tx modes,
> > > > > - code overcomplexity because of RollingUpgrade feature never attended
> > > at
> > > > > AI,
> > > > > - etc,
> > > > > before choosing the way.
> > > > >
> > > > > On Tue, Nov 3, 2020 at 3:31 PM Valentin Kulichenko <
> > > > > valentin.kulichenko@gmail.com> wrote:
> > > > >
> > > > > > Ksenia, thanks for scheduling this on such short notice!
> > > > > >
> > > > > > As for the original topic, I do support Alexey's idea. We're not
> > > going
> > > > to
> > > > > > rewrite anything from scratch, as most of the components are going to
> > > > be
> > > > > > moved as-is or with minimal modifications. However, the changes that
> > > > are
> > > > > > proposed imply serious rework of the core parts of the code, which
> > > are
> > > > > not
> > > > > > properly decoupled from each other and from other parts. This makes
> > > the
> > > > > > incremental approach borderline impossible. Developing in a new repo,
> > > > > > however, addresses this concern. As a bonus, we can also refactor the
> > > > > code,
> > > > > > introduce better decoupling, get rid of kernel context, and develop
> > > > unit
> > > > > > tests (finally!).
> > > > > >
> > > > > > Basically, this proposal only affects the *process*, not the set of
> > > > > changes
> > > > > > we had discussed before. Ignite 3.0 is our unique chance to make
> > > things
> > > > > > right.
> > > > > >
> > > > > > -Val
> > > > > >
> > > > > > On Tue, Nov 3, 2020 at 3:06 AM Kseniya Romanova <
> > > > > romanova.ks.spb@gmail.com
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Pavel, all the interesting points will be anyway published here in
> > > > > > English
> > > > > > > (as the principal "if it's not on devlist it doesn't happened" is
> > > > still
> > > > > > > relevant). This is just a quick call for a group of developers.
> > > Later
> > > > > we
> > > > > > > can do a separate presentation of idea and discussion in English as
> > > > we
> > > > > > did
> > > > > > > for the Ignite 3.0 draft of changes.
> > > > > > >
> > > > > > > вт, 3 нояб. 2020 г. в 13:52, Pavel Tupitsyn <ptupitsyn@apache.org
> > > >:
> > > > > > >
> > > > > > > > Kseniya,
> > > > > > > >
> > > > > > > > Thanks for scheduling this call.
> > > > > > > > Do you think we can switch to English if non-Russian speaking
> > > > > community
> > > > > > > > members decide to join?
> > > > > > > >
> > > > > > > > On Tue, Nov 3, 2020 at 1:32 PM Kseniya Romanova <
> > > > > > > romanova.ks.spb@gmail.com
> > > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Let's do this community discussion open. Here's the link on
> > > zoom
> > > > > call
> > > > > > > in
> > > > > > > > > Russian for Friday 6 PM:
> > > > > > > > >
> > > > > https://www.meetup.com/Moscow-Apache-Ignite-Meetup/events/274360378/
> > > > > > > > >
> > > > > > > > > вт, 3 нояб. 2020 г. в 12:49, Nikolay Izhikov <
> > > > nizhikov@apache.org
> > > > > >:
> > > > > > > > >
> > > > > > > > > > Time works for me.
> > > > > > > > > >
> > > > > > > > > > > 3 нояб. 2020 г., в 12:40, Alexey Goncharuk <
> > > > > > > > alexey.goncharuk@gmail.com
> > > > > > > > > >
> > > > > > > > > > написал(а):
> > > > > > > > > > >
> > > > > > > > > > > Nikolay,
> > > > > > > > > > >
> > > > > > > > > > > I am up for the call. I will try to explain my reasoning in
> > > > > > greater
> > > > > > > > > > detail
> > > > > > > > > > > and will be glad to hear the concerns. Will this Friday,
> > > Nov
> > > > > 6th,
> > > > > > > > work?
> > > > > > > > > > >
> > > > > > > > > > > вт, 3 нояб. 2020 г. в 10:09, Nikolay Izhikov <
> > > > > > nizhikov@apache.org
> > > > > > > >:
> > > > > > > > > > >
> > > > > > > > > > >> Igniters, should we have a call for this topic?
> > > > > > > > > > >>
> > > > > > > > > > >>> 2 нояб. 2020 г., в 18:53, Pavel Tupitsyn <
> > > > > ptupitsyn@apache.org
> > > > > > >
> > > > > > > > > > >> написал(а):
> > > > > > > > > > >>>
> > > > > > > > > > >>>> not intend to rewrite everything from scratch
> > > > > > > > > > >>>
> > > > > > > > > > >>>> Every single test from Ignite 2.x should be moved to
> > > > Ignite
> > > > > 3
> > > > > > > > > > >>>> regardless of how we choose to proceed.
> > > > > > > > > > >>>
> > > > > > > > > > >>> Alexey, thank you for the explanation, this addresses all
> > > > of
> > > > > my
> > > > > > > > > > concerns.
> > > > > > > > > > >>>
> > > > > > > > > > >>>
> > > > > > > > > > >>>
> > > > > > > > > > >>>
> > > > > > > > > > >>>
> > > > > > > > > > >>> On Mon, Nov 2, 2020 at 6:43 PM Andrey Mashenkov <
> > > > > > > > > > >> andrey.mashenkov@gmail.com>
> > > > > > > > > > >>> wrote:
> > > > > > > > > > >>>
> > > > > > > > > > >>>> Hi, Igniters.
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> * AFAIU, we need a new repo if we want to apply
> > > different
> > > > > > > > > restrictions
> > > > > > > > > > >> to
> > > > > > > > > > >>>> pull requests,
> > > > > > > > > > >>>> otherwise I see no difference for myself.
> > > > > > > > > > >>>> E.g. make static analysis (do we have?), compile,
> > > styles,
> > > > > and
> > > > > > > > > javadoc
> > > > > > > > > > >>>> checks mandatory.
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> I think that relaxed requirements here will lead to bad
> > > > > > product
> > > > > > > > > > quality.
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> * Agree with Pavel, we should 'keep' integrations tests
> > > > > > somehow.
> > > > > > > > > > >>>> During active development tests will be broken most of
> > > > time,
> > > > > > so,
> > > > > > > > > > >>>> I'd port them e.g. suite-by-suite once we will have a
> > > > stable
> > > > > > and
> > > > > > > > > > >> featured
> > > > > > > > > > >>>> environment to run them and of course make test's code
> > > > clear
> > > > > > and
> > > > > > > > > avoid
> > > > > > > > > > >>>> bad/non-relevant ones.
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> * I like bottom-up approach.
> > > > > > > > > > >>>> With it we could make a better framework. I mean clear
> > > > > > component
> > > > > > > > > > >> lifecycle,
> > > > > > > > > > >>>> component wiring mechanics, general methods to approach
> > > > core
> > > > > > > > > > components
> > > > > > > > > > >>>> such as exchange/communication
> > > > > > > > > > >>>> to avoid code mess like we have in ExchangeFuture with
> > > all
> > > > > > these
> > > > > > > > > > custom
> > > > > > > > > > >>>> callbacks for each component, interfaces like
> > > > > > > > > > >>>> PartitionsExchangeAware, IgniteChangeGlobalStateSupport
> > > > and
> > > > > > > > > > >>>> a pack of
> > > > > > > > > > >>
> > > > > start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected
> > > > > > > > > > >>>> and so on in various unexpected places.
> > > > > > > > > > >>>> Hope, we will be able to port most of the good code to
> > > the
> > > > > new
> > > > > > > > > > framework
> > > > > > > > > > >>>> version.
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk <
> > > > > > > > > > >>>> alexey.goncharuk@gmail.com>
> > > > > > > > > > >>>> wrote:
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>> Nikolay, Pavel,
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> Thanks for the feedback! First of all, I wanted to
> > > stress
> > > > > > that
> > > > > > > I
> > > > > > > > do
> > > > > > > > > > not
> > > > > > > > > > >>>>> intend to rewrite everything from scratch (I never used
> > > > > this
> > > > > > > > > phrase).
> > > > > > > > > > >>>> There
> > > > > > > > > > >>>>> are significant parts of code that would be moved with
> > > > > > minimal
> > > > > > > > > > >>>>> modifications. Second, I never said that we will get
> > > rid
> > > > of
> > > > > > the
> > > > > > > > old
> > > > > > > > > > >> tests
> > > > > > > > > > >>>>> codebase. Every single test from Ignite 2.x should be
> > > > moved
> > > > > > to
> > > > > > > > > > Ignite 3
> > > > > > > > > > >>>>> regardless of how we choose to proceed.
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> My point is that for some parts of the code a clean
> > > > > bottom-up
> > > > > > > > > > >>>>> implementation will be cheaper in many ways. Let me
> > > give
> > > > > you
> > > > > > a
> > > > > > > > few
> > > > > > > > > > >>>> concrete
> > > > > > > > > > >>>>> examples:
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> - I think no one can object that we need a cleanly
> > > > > separated
> > > > > > > > > > >>>> persistence
> > > > > > > > > > >>>>> layer for Ignite. There is a very raw draft IEP for
> > > this
> > > > > > > > already.
> > > > > > > > > On
> > > > > > > > > > >>>> the
> > > > > > > > > > >>>>> other hand, I think we also can agree that we need a
> > > > > > > split-brain
> > > > > > > > > > >>>>> resistant
> > > > > > > > > > >>>>> replication protocol for caches. There is also an IEP
> > > > for
> > > > > > > this.
> > > > > > > > > > >>>> Neither
> > > > > > > > > > >>>>> of
> > > > > > > > > > >>>>> the changes is a good fit for 2.x because they are
> > > > likely
> > > > > to
> > > > > > > > > > >> introduce
> > > > > > > > > > >>>>> breaking changes in the persistence layer,
> > > configuration
> > > > > and
> > > > > > > > > > >> behavior.
> > > > > > > > > > >>>>> Additionally, these components are now tightly
> > > coupled,
> > > > so
> > > > > > > there
> > > > > > > > > is
> > > > > > > > > > >> no
> > > > > > > > > > >>>>> way
> > > > > > > > > > >>>>> these two changes can be implemented in parallel and
> > > > then
> > > > > > > merged
> > > > > > > > > > >>>>> together
> > > > > > > > > > >>>>> easily. So what we will end up with is having to
> > > > implement
> > > > > > > these
> > > > > > > > > > >>>> changes
> > > > > > > > > > >>>>> sequentially, fixing all existing tests twice, and
> > > > > > essentially
> > > > > > > > > > >>>> throwing
> > > > > > > > > > >>>>> away half of the work done because the other part of
> > > the
> > > > > > > change
> > > > > > > > is
> > > > > > > > > > >>>>> re-implemented
> > > > > > > > > > >>>>> - Similar example goes with getting rid of
> > > > > > > IgniteInternalFuture
> > > > > > > > > and
> > > > > > > > > > >>>>> replacing it with CompletableFuture, and any other
> > > > change
> > > > > > that
> > > > > > > > > > >> touches
> > > > > > > > > > >>>>> the
> > > > > > > > > > >>>>> asynchronous part of the code.
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> Third, I do not see how this choice affects the UX of
> > > > > Ignite.
> > > > > > > The
> > > > > > > > > end
> > > > > > > > > > >>>> user
> > > > > > > > > > >>>>> experience must be fixed in the IEP regardless of the
> > > > > > > development
> > > > > > > > > > >> process
> > > > > > > > > > >>>>> and the fact that we have gaps in this area in Ignite
> > > 2.x
> > > > > > just
> > > > > > > > > > confirms
> > > > > > > > > > >>>>> that.
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> Pavel, agree that a repo/branch is a technicality, I
> > > > guess
> > > > > if
> > > > > > > > > > >>>> reformulate,
> > > > > > > > > > >>>>> my point is that we might agree to have a single
> > > > > development
> > > > > > > > master
> > > > > > > > > > >>>> branch
> > > > > > > > > > >>>>> with 'disassembled' end-to-end functionality for some
> > > > > period
> > > > > > of
> > > > > > > > > time
> > > > > > > > > > to
> > > > > > > > > > >>>>> speed up development, and re-assemble the core features
> > > > > after
> > > > > > > > > having
> > > > > > > > > > >>>>> submodules tested independently.
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> Nikolay,
> > > > > > > > > > >>>>>> We have many features that have to evolve.
> > > > > > > > > > >>>>>> Snapshots, rebalance, tooling, tracing, zookeeper
> > > > support,
> > > > > > > etc.
> > > > > > > > > > >>>>> This is not very specific. In the end, resources are
> > > > > limited
> > > > > > > and
> > > > > > > > we
> > > > > > > > > > >> will
> > > > > > > > > > >>>>> not be able to drive both tracks simultaneously,
> > > > especially
> > > > > > > > after a
> > > > > > > > > > >>>> couple
> > > > > > > > > > >>>>> of features having been implemented for Ignite 3.0. If
> > > > > there
> > > > > > > are
> > > > > > > > > > indeed
> > > > > > > > > > >>>>> some major changes that we want to do in Ignite 2.x
> > > > instead
> > > > > > of
> > > > > > > > > > putting
> > > > > > > > > > >>>>> effort into 3.0 - let's discuss them. I am just not
> > > aware
> > > > > of
> > > > > > > any,
> > > > > > > > > > >> that's
> > > > > > > > > > >>>>> why I am eager to move forward with Ignite 3.0.
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>>> We have bugs and issues that can be fixed in 2.x
> > > without
> > > > > > > > breaking
> > > > > > > > > > >>>> backward
> > > > > > > > > > >>>>> compatibility.
> > > > > > > > > > >>>>>> We have many users who are happy with the 2.x with all
> > > > > it’s
> > > > > > > > > issues.
> > > > > > > > > > >>>>> These changes will be covered by end-to-end tests and
> > > > > > migrated
> > > > > > > to
> > > > > > > > > > >> Ignite
> > > > > > > > > > >>>>> 3.0, so I see no issues here.
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> Finally, Anton & Nikolay
> > > > > > > > > > >>>>> I do not have an estimate for this simply because the
> > > > > > activity
> > > > > > > is
> > > > > > > > > > >>>>> community-driven and it depends on the number of people
> > > > > > willing
> > > > > > > > to
> > > > > > > > > > >>>>> contribute. With the current pace, I would hope to have
> > > > an
> > > > > RC
> > > > > > > of
> > > > > > > > > > Ignite
> > > > > > > > > > >>>> 3.0
> > > > > > > > > > >>>>> to be ready by the end of 2021. My gut feeling is that
> > > by
> > > > > > > moving
> > > > > > > > > with
> > > > > > > > > > >>>>> incremental changes, we will not be able to implement
> > > > even
> > > > > > half
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > >>>>> wishlist by that time.
> > > > > > > > > > >>>>> I doubt that releasing several major releases with
> > > > breaking
> > > > > > > > changes
> > > > > > > > > > >> will
> > > > > > > > > > >>>>> make Ignite users happy either because each upgrade
> > > will
> > > > > cost
> > > > > > > > > Ignite
> > > > > > > > > > >>>> users
> > > > > > > > > > >>>>> money, so the fewer major versions we release, the
> > > > better.
> > > > > > Thus
> > > > > > > > my
> > > > > > > > > > wish
> > > > > > > > > > >>>> to
> > > > > > > > > > >>>>> include all breaking changes in one release.
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> I'll be now quiet for a while, let's see what other
> > > > > community
> > > > > > > > > members
> > > > > > > > > > >>>>> think.
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <
> > > > > > > > ptupitsyn@apache.org
> > > > > > > > > >:
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>>> 1. Rewriting from scratch is never a good idea.
> > > > > > > > > > >>>>>> We don't want to follow the path of Netscape and lose
> > > > all
> > > > > > our
> > > > > > > > > users
> > > > > > > > > > >>>>>> by the time we have a working 3.0 [1]
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>> 2. Not sure about new repo - seems like some pain and
> > > no
> > > > > > gain,
> > > > > > > > > > what's
> > > > > > > > > > >>>> the
> > > > > > > > > > >>>>>> problem with a branch?
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>> 3. We should keep existing integration tests when
> > > > > possible.
> > > > > > > > > > >>>>>> We have accumulated a lot of edge case knowledge over
> > > > the
> > > > > > > years,
> > > > > > > > > > >>>>>> it is not a good idea to send all of that down the
> > > > drain.
> > > > > > > > > > >>>>>> Yes, integration tests are slow, but they are the most
> > > > > > > valuable.
> > > > > > > > > > >>>>>> I think we can move more stuff into nightly runs and
> > > > have
> > > > > a
> > > > > > > fast
> > > > > > > > > and
> > > > > > > > > > >>>>> modern
> > > > > > > > > > >>>>>> basic suite.
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>> Alexey, you are much more familiar with the Ignite
> > > core
> > > > > > > codebase
> > > > > > > > > > than
> > > > > > > > > > >>>>> most
> > > > > > > > > > >>>>>> of us,
> > > > > > > > > > >>>>>> can you please explain in more detail which particular
> > > > > > > feature,
> > > > > > > > in
> > > > > > > > > > >> your
> > > > > > > > > > >>>>>> opinion,
> > > > > > > > > > >>>>>> mandates this "start from scratch" approach?
> > > > > > > > > > >>>>>> Is it really not possible at all to follow a less
> > > > radical
> > > > > > way?
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>> [1]
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>> On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <
> > > > > > > > > nizhikov@apache.org
> > > > > > > > > > >
> > > > > > > > > > >>>>>> wrote:
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>>> Hello, Alexey.
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>> I think that «rewriting from scratch» approach has a
> > > > high
> > > > > > > risk
> > > > > > > > to
> > > > > > > > > > >>>> make
> > > > > > > > > > >>>>>> new
> > > > > > > > > > >>>>>>> features unusable.
> > > > > > > > > > >>>>>>> At the time Ignite2 was started no-one wants to do
> > > bad
> > > > UX
> > > > > > or
> > > > > > > > bad
> > > > > > > > > > >>>>>> features.
> > > > > > > > > > >>>>>>> Nevertheless, it happen.
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>> I think we can avoid it with the Ignite3 and
> > > successors
> > > > > if
> > > > > > we
> > > > > > > > > will
> > > > > > > > > > >>>> move
> > > > > > > > > > >>>>>>> step by step without keeping backward compatibility
> > > > > > > > > > >>>>>>> With the step by step approach, we can focus on each
> > > > > > > component
> > > > > > > > > > >>>>>> separately.
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>>> What new features are we planning to implement for
> > > > > Ignite
> > > > > > > 2.x?
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>> We have many features that have to evolve.
> > > > > > > > > > >>>>>>> Snapshots, rebalance, tooling, tracing, zookeeper
> > > > > support,
> > > > > > > etc.
> > > > > > > > > > >>>>>>> We have bugs and issues that can be fixed in 2.x
> > > > without
> > > > > > > > breaking
> > > > > > > > > > >>>>>> backward
> > > > > > > > > > >>>>>>> compatibility.
> > > > > > > > > > >>>>>>> We have many users who are happy with the 2.x with
> > > all
> > > > > it’s
> > > > > > > > > issues.
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>>> 2 нояб. 2020 г., в 14:09, Anton Vinogradov <
> > > > > av@apache.org
> > > > > > >
> > > > > > > > > > >>>>> написал(а):
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>> Alexey,
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>> Do we have any estimates of how fast we'll be able
> > > to
> > > > > gain
> > > > > > > > > > >>>>>>> production-ready
> > > > > > > > > > >>>>>>>> AI 3.0 in case of a "new repo" choice?
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>> On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk <
> > > > > > > > > > >>>>>>> alexey.goncharuk@gmail.com>
> > > > > > > > > > >>>>>>>> wrote:
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>>> Nikolay,
> > > > > > > > > > >>>>>>>>>
> > > > > > > > > > >>>>>>>>> What new features are we planning to implement for
> > > > > Ignite
> > > > > > > > 2.x?
> > > > > > > > > I
> > > > > > > > > > >>>>> think
> > > > > > > > > > >>>>>>> once
> > > > > > > > > > >>>>>>>>> we commence working on Ignite 3.0, we should
> > > > gradually
> > > > > > > cease
> > > > > > > > > the
> > > > > > > > > > >>>>>>> activity
> > > > > > > > > > >>>>>>>>> on Ignite 2.x to mere bugfixes because such
> > > parallel
> > > > > > > > > development
> > > > > > > > > > >>>>> will
> > > > > > > > > > >>>>>> be
> > > > > > > > > > >>>>>>>>> overwhelming regardless of how we choose to
> > > proceed.
> > > > > > > > > > >>>>>>>>>
> > > > > > > > > > >>>>>>>>> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <
> > > > > > > > > > nizhikov@apache.org
> > > > > > > > > > >>>>> :
> > > > > > > > > > >>>>>>>>>
> > > > > > > > > > >>>>>>>>>> To be clear:
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>> I would suggest creating a new repository for
> > > > Ignite
> > > > > > 3.0
> > > > > > > > > > >>>>> (perhaps, a
> > > > > > > > > > >>>>>>>>> new
> > > > > > > > > > >>>>>>>>>> clean branch, but a new repo looks nicer to me)
> > > and
> > > > a
> > > > > > new
> > > > > > > > > Ignite
> > > > > > > > > > >>>>> 3.0
> > > > > > > > > > >>>>>>>>>> TeamCity project.
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>> +1 for new Team City project.
> > > > > > > > > > >>>>>>>>>> +1 for new branch for Ignite3.
> > > > > > > > > > >>>>>>>>>> -1 for new repo.
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <
> > > > > > > > > > >>>> NIzhikov.dev@gmail.com
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>>>>>> написал(а):
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>> Hello, Alexey.
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>> I think it will hurt our project more than help.
> > > > > > > > > > >>>>>>>>>>> Developing new features for 2 separate branches
> > > > with
> > > > > > the
> > > > > > > > > > >>>> different
> > > > > > > > > > >>>>>>> APIs
> > > > > > > > > > >>>>>>>>>> and internal structure is overwhelming
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>> Maybe we should relax a bit requirements for
> > > > Ignite3?
> > > > > > > > > > >>>>>>>>>>> Maybe we should move step by step and make
> > > Ignite3
> > > > > with
> > > > > > > new
> > > > > > > > > > >>>>>>>>>> configuration than Ignite4 with new transactions,
> > > > etc?
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <
> > > > > > > > > > >>>>>>>>> alexey.goncharuk@gmail.com>
> > > > > > > > > > >>>>>>>>>> написал(а):
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> Igniters,
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> I wanted to pitch a rather radical idea
> > > regarding
> > > > > the
> > > > > > > > Ignite
> > > > > > > > > > >>>> 3.0
> > > > > > > > > > >>>>>>>>>>>> development which has occurred to me some time
> > > > ago.
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> We already have several IEPs targeted to Ignite
> > > > 3.0
> > > > > > > which
> > > > > > > > > > imply
> > > > > > > > > > >>>>>> major
> > > > > > > > > > >>>>>>>>>>>> changes to the codebase (the change in
> > > replication
> > > > > > > > protocol
> > > > > > > > > > and
> > > > > > > > > > >>>>>> thus
> > > > > > > > > > >>>>>>>>>>>> transactions, change in binary format, updated
> > > > > > > > metastorage,
> > > > > > > > > > >>>> etc).
> > > > > > > > > > >>>>>> We
> > > > > > > > > > >>>>>>>>>> also
> > > > > > > > > > >>>>>>>>>>>> planned significant changes in public APIs:
> > > > > > > configuration
> > > > > > > > > > >>>> format
> > > > > > > > > > >>>>>>>>> change,
> > > > > > > > > > >>>>>>>>>>>> improvements in cache APIs, SQL APIs,
> > > transaction
> > > > > mode
> > > > > > > > > rework.
> > > > > > > > > > >>>>> The
> > > > > > > > > > >>>>>>>>>> wishlist
> > > > > > > > > > >>>>>>>>>>>> of changes for Ignite 3.0 is huge.
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> So, I was wondering whether it makes sense to
> > > try
> > > > to
> > > > > > > > change
> > > > > > > > > > the
> > > > > > > > > > >>>>> old
> > > > > > > > > > >>>>>>>>>>>> codebase, or start with a new baseline and move
> > > > old
> > > > > > > pieces
> > > > > > > > > of
> > > > > > > > > > >>>>> code
> > > > > > > > > > >>>>>>>>> that
> > > > > > > > > > >>>>>>>>>> do
> > > > > > > > > > >>>>>>>>>>>> not require significant rework. Personally, I
> > > > would
> > > > > go
> > > > > > > > with
> > > > > > > > > > the
> > > > > > > > > > >>>>>>> second
> > > > > > > > > > >>>>>>>>>>>> option for the following reasons:
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> - We have a chance to shift the development
> > > > paradigm
> > > > > > in
> > > > > > > > the
> > > > > > > > > > >>>>> project
> > > > > > > > > > >>>>>>>>> and
> > > > > > > > > > >>>>>>>>>>>> introduce the practice of true unit-tests. In
> > > the
> > > > > new
> > > > > > > > > baseline
> > > > > > > > > > >>>> at
> > > > > > > > > > >>>>>> the
> > > > > > > > > > >>>>>>>>>>>> beginning there will be no ability to run an
> > > > > > end-to-end
> > > > > > > > > > >>>> scenario,
> > > > > > > > > > >>>>>>>>> thus
> > > > > > > > > > >>>>>>>>>> we
> > > > > > > > > > >>>>>>>>>>>> will be forced to write unit-tests. So far, such
> > > > > > > practice
> > > > > > > > > was
> > > > > > > > > > >>>>> hard
> > > > > > > > > > >>>>>> to
> > > > > > > > > > >>>>>>>>>>>> implement because of tight coupling between
> > > Ignite
> > > > > > > > > components
> > > > > > > > > > >>>> and
> > > > > > > > > > >>>>>>>>>> inability
> > > > > > > > > > >>>>>>>>>>>> to instantiate components without an instance of
> > > > > > > > > > KernalContext.
> > > > > > > > > > >>>>> For
> > > > > > > > > > >>>>>>>>>>>> example, we should be able to thoroughly test
> > > > > internal
> > > > > > > > > > >>>>> primitives,
> > > > > > > > > > >>>>>>>>>> such as
> > > > > > > > > > >>>>>>>>>>>> replication protocol (without actual
> > > > communication),
> > > > > > > > > > >>>> distributed
> > > > > > > > > > >>>>>>>>>>>> metastorage contracts, persistence layer, etc.
> > > > > > > > > > >>>>>>>>>>>> - We will significantly reduce the development
> > > > cycle
> > > > > > in
> > > > > > > > the
> > > > > > > > > > >>>>>> beginning
> > > > > > > > > > >>>>>>>>>>>> (right now the RunAll takes two hours of
> > > > > astronomical
> > > > > > > time
> > > > > > > > > > with
> > > > > > > > > > >>>>>> empty
> > > > > > > > > > >>>>>>>>>> TC;
> > > > > > > > > > >>>>>>>>>>>> in the new approach developer will be able to
> > > run
> > > > > ALL
> > > > > > > > tests
> > > > > > > > > > >>>>> locally
> > > > > > > > > > >>>>>>>>> in
> > > > > > > > > > >>>>>>>>>> a
> > > > > > > > > > >>>>>>>>>>>> matter of minutes)
> > > > > > > > > > >>>>>>>>>>>> - We can get rid of TC bot and enforce green TC
> > > by
> > > > > > > > > integrating
> > > > > > > > > > >>>> TC
> > > > > > > > > > >>>>>>>>> build
> > > > > > > > > > >>>>>>>>>>>> results with GitHub PRs (the same way Travis is
> > > > > > > currently
> > > > > > > > > > >>>>>> integrated
> > > > > > > > > > >>>>>>>>>> to PR
> > > > > > > > > > >>>>>>>>>>>> check). We should restrict PR merge without a TC
> > > > > check
> > > > > > > > > > >>>>>>>>>>>> - We will still have to re-write all tests, but
> > > > only
> > > > > > > once.
> > > > > > > > > If
> > > > > > > > > > >>>> we
> > > > > > > > > > >>>>>> try
> > > > > > > > > > >>>>>>>>> to
> > > > > > > > > > >>>>>>>>>>>> modify the old codebase, we would need to modify
> > > > all
> > > > > > the
> > > > > > > > > tests
> > > > > > > > > > >>>>> for
> > > > > > > > > > >>>>>>>>>> every
> > > > > > > > > > >>>>>>>>>>>> major change (public API change, configuration
> > > > > change)
> > > > > > > > > > >>>>>>>>>>>> - We will have fewer conflicts when working
> > > > > together.
> > > > > > > For
> > > > > > > > > > >>>>> example,
> > > > > > > > > > >>>>>> I
> > > > > > > > > > >>>>>>>>>>>> cannot imagine how one would merge two changes
> > > of
> > > > > > > getting
> > > > > > > > > rid
> > > > > > > > > > >>>> of
> > > > > > > > > > >>>>>>>>>>>> IgniteFuture and changes in replication
> > > protocol,
> > > > > for
> > > > > > > > > example
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> Technically, I would suggest creating a new
> > > > > repository
> > > > > > > for
> > > > > > > > > > >>>> Ignite
> > > > > > > > > > >>>>>> 3.0
> > > > > > > > > > >>>>>>>>>>>> (perhaps, a new clean branch, but a new repo
> > > looks
> > > > > > nicer
> > > > > > > > to
> > > > > > > > > > me)
> > > > > > > > > > >>>>>> and a
> > > > > > > > > > >>>>>>>>>> new
> > > > > > > > > > >>>>>>>>>>>> Ignite 3.0 TeamCity project.
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> While it may seem quite radical, I do believe
> > > that
> > > > > > this
> > > > > > > > > > >>>> approach
> > > > > > > > > > >>>>>> will
> > > > > > > > > > >>>>>>>>>> give
> > > > > > > > > > >>>>>>>>>>>> us more benefits than trying to make such major
> > > > > > changes
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > >>>>>>>>> existing
> > > > > > > > > > >>>>>>>>>>>> codebase. If needed, let's schedule a discord
> > > chat
> > > > > > like
> > > > > > > > > before
> > > > > > > > > > >>>> to
> > > > > > > > > > >>>>>>>>>> discuss
> > > > > > > > > > >>>>>>>>>>>> this.
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> WDYT?
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> --
> > > > > > > > > > >>>> Best regards,
> > > > > > > > > > >>>> Andrey V. Mashenkov
> > > > > > > > > > >>>>
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> >
> > Best regards,
> > Alexei Scherbakov
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Valentin Kulichenko <va...@gmail.com>.
Folks,
We already have multiple IEPs for Ignite 3.0, and as far as I know, there
are contributors that would like to work on them (or probably already
started). That said, we should make a decision as soon as possible.
At this point, it doesn't seem that there are any strong objections to the
technical side of things. So I would suggest the following:
1. Proceed with Alexey's approach to the development process, as it seems
to be the best (in my opinion - the only) way to address all the technical
concerns and issues expressed in the thread. We'll start by creating a new
repo and a new TC project.
2. Start a separate discussion around transparency. If there are any
changes we need to make to our contributor guidelines, I am happy to talk
them through, but I don't think it's reasonable to delay feature
development because of this. In the short term, I will make sure that
everything that happens within the new repo is as open to the community
as possible.
Nikolay, Maxim, are you OK with this route?
-Val
On Tue, Nov 10, 2020 at 4:55 PM Valentin Kulichenko <
valentin.kulichenko@gmail.com> wrote:
> Maxim,
>
> 2.x and 3.x will have to coexist for some time - I don't see how we can
> avoid this considering the set of proposed changes. That said, we
> effectively will need to have two "masters" - one for each major version.
> Master for 3.x can technically be a branch in the existing repo, but having
> a separate repo seems cleaner, simply because it will not be a "branch" in
> the traditional sense.
>
> Note that the new repo will still be under the Apache org, with the same
> set of committers, managed by the community, etc. All the development
> happening for 3.0 must follow the rules that we currently have (if
> anything, it's an opportunity to improve those rules).
>
> As I said during the call on Friday, I strongly believe that if there is a
> transparency issue, it will exist regardless of the approach we choose for
> 3.0. If community members develop without IEPs or public discussions, this
> will happen for both 2.x and 3.x unless we address this separately. I don't
> see how this is related to Alexey's suggestion, which targets *technical*
> issues with the product more than anything else. This a way to achieve
> better modularity, introduce better coverage with unit tests, reduce
> conflicts during development, etc.
>
> Coming back to transparency, let's identify the issues and fix them. It
> probably makes sense to have a separate discussion on this topic.
>
> -Val
>
> On Tue, Nov 10, 2020 at 1:05 PM Maxim Muzafarov <mm...@apache.org> wrote:
>
>> Sergey,
>>
>>
>> Your summary makes sense to me.
>>
>> However, how we come up from *Development transparency* to *create a
>> separate public repository dedicated for 3.0*? For me *development
>> transparency* is about making changes in the master branch. These
>> changes will definitely be seen by all the Ignite developers.
>>
>> A dedicated public repository is technically public and visible for
>> everyone, but it allows development without IEPs, without public
>> discussion (since all the code changes are not related to the master
>> branch) it also allows a large number of assumptions and deviations
>> (like code-style violations). It also not about *development
>> transparency* since developers which are working on 3.0 is only a
>> subset of all Ignite developers which may continue working on 2.x. For
>> me, this would be a huge step backwards.
>>
>> Ignite veterans should remember how long the branch stabilization took
>> for the 2.x version with the PDS.
>>
>>
>> I think each breaking change should be passed through the master branch.
>>
>> On Tue, 10 Nov 2020 at 22:18, Alexei Scherbakov
>> <al...@gmail.com> wrote:
>> >
>> > Makes sense to me.
>> >
>> > вт, 10 нояб. 2020 г. в 18:47, Sergey Chugunov <
>> sergey.chugunov@gmail.com>:
>> >
>> > > Igniters,
>> > >
>> > > I thought over Friday meeting ideas and concerns and summarized them
>> in
>> > > these three points:
>> > >
>> > >
>> > > 1. *Components design unification approach.* New proposed
>> components
>> > > will be developed by different contributors, but they need to be
>> unified
>> > > and should integrate with each other easily. To ensure that I
>> suggest
>> > > calling an architecture group that will create design guidelines
>> for all
>> > > components and high-level overview of overall architecture. How
>> code is
>> > > split into components, what are component boundaries, how component
>> > > lifecycle works and what are its interfaces - all these and other
>> > > questions
>> > > should be covered.
>> > >
>> > > 2. *Scope management.* Apache 3.0 should be implemented within a
>> > > reasonable time, so we need some procedure to decide whether a
>> > > particular
>> > > feature should be dropped from the scope of 3.0 and postponed to
>> 3.1
>> > > release. To do so I suggest to range all features by two
>> parameters:
>> > > criticality for 3.0 and amount of breaking changes. 3.0 scope
>> should
>> > > include features of high criticality AND features with a big
>> amount of
>> > > breaking changes. All other features can be made optional.
>> > >
>> > > 3. *Development transparency.* Development of all components
>> should be
>> > > made as transparent for everyone as possible. Any contributor
>> should be
>> > > able to look over any component at any stage of development. To
>> achieve
>> > > this I suggest to create a separate public repository dedicated
>> for 3.0
>> > > development. It will make the code available for everyone but when
>> > > development of 3.0 is done we won't loose any stars of our current
>> > > repository as we merge dev repo into main one and drop dev.
>> > >
>> > > Do these ideas make sense to you? Are there any concerns not covered
>> by
>> > > these suggestions?
>> > >
>> > > On Fri, Nov 6, 2020 at 7:36 PM Kseniya Romanova <
>> romanova.ks.spb@gmail.com
>> > > >
>> > > wrote:
>> > >
>> > > > Here are the slides from Alexey Goncharuk. Let's think this over and
>> > > > continue on Monday:
>> > > >
>> > > >
>> > >
>> https://go.gridgain.com/rs/491-TWR-806/images/Ignite_3_Plans_and_development_process.pdf
>> > > >
>> > > > чт, 5 нояб. 2020 г. в 11:13, Anton Vinogradov <av...@apache.org>:
>> > > >
>> > > > > Folks,
>> > > > >
>> > > > > Should we perform cleanup work before (r)evolutional changes?
>> > > > > My huge proposal is to get rid of things which we don't need
>> anyway
>> > > > > - local caches,
>> > > > > - strange tx modes,
>> > > > > - code overcomplexity because of RollingUpgrade feature never
>> attended
>> > > at
>> > > > > AI,
>> > > > > - etc,
>> > > > > before choosing the way.
>> > > > >
>> > > > > On Tue, Nov 3, 2020 at 3:31 PM Valentin Kulichenko <
>> > > > > valentin.kulichenko@gmail.com> wrote:
>> > > > >
>> > > > > > Ksenia, thanks for scheduling this on such short notice!
>> > > > > >
>> > > > > > As for the original topic, I do support Alexey's idea. We're not
>> > > going
>> > > > to
>> > > > > > rewrite anything from scratch, as most of the components are
>> going to
>> > > > be
>> > > > > > moved as-is or with minimal modifications. However, the changes
>> that
>> > > > are
>> > > > > > proposed imply serious rework of the core parts of the code,
>> which
>> > > are
>> > > > > not
>> > > > > > properly decoupled from each other and from other parts. This
>> makes
>> > > the
>> > > > > > incremental approach borderline impossible. Developing in a new
>> repo,
>> > > > > > however, addresses this concern. As a bonus, we can also
>> refactor the
>> > > > > code,
>> > > > > > introduce better decoupling, get rid of kernel context, and
>> develop
>> > > > unit
>> > > > > > tests (finally!).
>> > > > > >
>> > > > > > Basically, this proposal only affects the *process*, not the
>> set of
>> > > > > changes
>> > > > > > we had discussed before. Ignite 3.0 is our unique chance to make
>> > > things
>> > > > > > right.
>> > > > > >
>> > > > > > -Val
>> > > > > >
>> > > > > > On Tue, Nov 3, 2020 at 3:06 AM Kseniya Romanova <
>> > > > > romanova.ks.spb@gmail.com
>> > > > > > >
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Pavel, all the interesting points will be anyway published
>> here in
>> > > > > > English
>> > > > > > > (as the principal "if it's not on devlist it doesn't
>> happened" is
>> > > > still
>> > > > > > > relevant). This is just a quick call for a group of
>> developers.
>> > > Later
>> > > > > we
>> > > > > > > can do a separate presentation of idea and discussion in
>> English as
>> > > > we
>> > > > > > did
>> > > > > > > for the Ignite 3.0 draft of changes.
>> > > > > > >
>> > > > > > > вт, 3 нояб. 2020 г. в 13:52, Pavel Tupitsyn <
>> ptupitsyn@apache.org
>> > > >:
>> > > > > > >
>> > > > > > > > Kseniya,
>> > > > > > > >
>> > > > > > > > Thanks for scheduling this call.
>> > > > > > > > Do you think we can switch to English if non-Russian
>> speaking
>> > > > > community
>> > > > > > > > members decide to join?
>> > > > > > > >
>> > > > > > > > On Tue, Nov 3, 2020 at 1:32 PM Kseniya Romanova <
>> > > > > > > romanova.ks.spb@gmail.com
>> > > > > > > > >
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Let's do this community discussion open. Here's the link
>> on
>> > > zoom
>> > > > > call
>> > > > > > > in
>> > > > > > > > > Russian for Friday 6 PM:
>> > > > > > > > >
>> > > > >
>> https://www.meetup.com/Moscow-Apache-Ignite-Meetup/events/274360378/
>> > > > > > > > >
>> > > > > > > > > вт, 3 нояб. 2020 г. в 12:49, Nikolay Izhikov <
>> > > > nizhikov@apache.org
>> > > > > >:
>> > > > > > > > >
>> > > > > > > > > > Time works for me.
>> > > > > > > > > >
>> > > > > > > > > > > 3 нояб. 2020 г., в 12:40, Alexey Goncharuk <
>> > > > > > > > alexey.goncharuk@gmail.com
>> > > > > > > > > >
>> > > > > > > > > > написал(а):
>> > > > > > > > > > >
>> > > > > > > > > > > Nikolay,
>> > > > > > > > > > >
>> > > > > > > > > > > I am up for the call. I will try to explain my
>> reasoning in
>> > > > > > greater
>> > > > > > > > > > detail
>> > > > > > > > > > > and will be glad to hear the concerns. Will this
>> Friday,
>> > > Nov
>> > > > > 6th,
>> > > > > > > > work?
>> > > > > > > > > > >
>> > > > > > > > > > > вт, 3 нояб. 2020 г. в 10:09, Nikolay Izhikov <
>> > > > > > nizhikov@apache.org
>> > > > > > > >:
>> > > > > > > > > > >
>> > > > > > > > > > >> Igniters, should we have a call for this topic?
>> > > > > > > > > > >>
>> > > > > > > > > > >>> 2 нояб. 2020 г., в 18:53, Pavel Tupitsyn <
>> > > > > ptupitsyn@apache.org
>> > > > > > >
>> > > > > > > > > > >> написал(а):
>> > > > > > > > > > >>>
>> > > > > > > > > > >>>> not intend to rewrite everything from scratch
>> > > > > > > > > > >>>
>> > > > > > > > > > >>>> Every single test from Ignite 2.x should be moved
>> to
>> > > > Ignite
>> > > > > 3
>> > > > > > > > > > >>>> regardless of how we choose to proceed.
>> > > > > > > > > > >>>
>> > > > > > > > > > >>> Alexey, thank you for the explanation, this
>> addresses all
>> > > > of
>> > > > > my
>> > > > > > > > > > concerns.
>> > > > > > > > > > >>>
>> > > > > > > > > > >>>
>> > > > > > > > > > >>>
>> > > > > > > > > > >>>
>> > > > > > > > > > >>>
>> > > > > > > > > > >>> On Mon, Nov 2, 2020 at 6:43 PM Andrey Mashenkov <
>> > > > > > > > > > >> andrey.mashenkov@gmail.com>
>> > > > > > > > > > >>> wrote:
>> > > > > > > > > > >>>
>> > > > > > > > > > >>>> Hi, Igniters.
>> > > > > > > > > > >>>>
>> > > > > > > > > > >>>> * AFAIU, we need a new repo if we want to apply
>> > > different
>> > > > > > > > > restrictions
>> > > > > > > > > > >> to
>> > > > > > > > > > >>>> pull requests,
>> > > > > > > > > > >>>> otherwise I see no difference for myself.
>> > > > > > > > > > >>>> E.g. make static analysis (do we have?), compile,
>> > > styles,
>> > > > > and
>> > > > > > > > > javadoc
>> > > > > > > > > > >>>> checks mandatory.
>> > > > > > > > > > >>>>
>> > > > > > > > > > >>>> I think that relaxed requirements here will lead
>> to bad
>> > > > > > product
>> > > > > > > > > > quality.
>> > > > > > > > > > >>>>
>> > > > > > > > > > >>>> * Agree with Pavel, we should 'keep' integrations
>> tests
>> > > > > > somehow.
>> > > > > > > > > > >>>> During active development tests will be broken
>> most of
>> > > > time,
>> > > > > > so,
>> > > > > > > > > > >>>> I'd port them e.g. suite-by-suite once we will
>> have a
>> > > > stable
>> > > > > > and
>> > > > > > > > > > >> featured
>> > > > > > > > > > >>>> environment to run them and of course make test's
>> code
>> > > > clear
>> > > > > > and
>> > > > > > > > > avoid
>> > > > > > > > > > >>>> bad/non-relevant ones.
>> > > > > > > > > > >>>>
>> > > > > > > > > > >>>> * I like bottom-up approach.
>> > > > > > > > > > >>>> With it we could make a better framework. I mean
>> clear
>> > > > > > component
>> > > > > > > > > > >> lifecycle,
>> > > > > > > > > > >>>> component wiring mechanics, general methods to
>> approach
>> > > > core
>> > > > > > > > > > components
>> > > > > > > > > > >>>> such as exchange/communication
>> > > > > > > > > > >>>> to avoid code mess like we have in ExchangeFuture
>> with
>> > > all
>> > > > > > these
>> > > > > > > > > > custom
>> > > > > > > > > > >>>> callbacks for each component, interfaces like
>> > > > > > > > > > >>>> PartitionsExchangeAware,
>> IgniteChangeGlobalStateSupport
>> > > > and
>> > > > > > > > > > >>>> a pack of
>> > > > > > > > > > >>
>> > > > > start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected
>> > > > > > > > > > >>>> and so on in various unexpected places.
>> > > > > > > > > > >>>> Hope, we will be able to port most of the good
>> code to
>> > > the
>> > > > > new
>> > > > > > > > > > framework
>> > > > > > > > > > >>>> version.
>> > > > > > > > > > >>>>
>> > > > > > > > > > >>>>
>> > > > > > > > > > >>>>
>> > > > > > > > > > >>>> On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk <
>> > > > > > > > > > >>>> alexey.goncharuk@gmail.com>
>> > > > > > > > > > >>>> wrote:
>> > > > > > > > > > >>>>
>> > > > > > > > > > >>>>> Nikolay, Pavel,
>> > > > > > > > > > >>>>>
>> > > > > > > > > > >>>>> Thanks for the feedback! First of all, I wanted to
>> > > stress
>> > > > > > that
>> > > > > > > I
>> > > > > > > > do
>> > > > > > > > > > not
>> > > > > > > > > > >>>>> intend to rewrite everything from scratch (I
>> never used
>> > > > > this
>> > > > > > > > > phrase).
>> > > > > > > > > > >>>> There
>> > > > > > > > > > >>>>> are significant parts of code that would be moved
>> with
>> > > > > > minimal
>> > > > > > > > > > >>>>> modifications. Second, I never said that we will
>> get
>> > > rid
>> > > > of
>> > > > > > the
>> > > > > > > > old
>> > > > > > > > > > >> tests
>> > > > > > > > > > >>>>> codebase. Every single test from Ignite 2.x
>> should be
>> > > > moved
>> > > > > > to
>> > > > > > > > > > Ignite 3
>> > > > > > > > > > >>>>> regardless of how we choose to proceed.
>> > > > > > > > > > >>>>>
>> > > > > > > > > > >>>>> My point is that for some parts of the code a
>> clean
>> > > > > bottom-up
>> > > > > > > > > > >>>>> implementation will be cheaper in many ways. Let
>> me
>> > > give
>> > > > > you
>> > > > > > a
>> > > > > > > > few
>> > > > > > > > > > >>>> concrete
>> > > > > > > > > > >>>>> examples:
>> > > > > > > > > > >>>>>
>> > > > > > > > > > >>>>> - I think no one can object that we need a
>> cleanly
>> > > > > separated
>> > > > > > > > > > >>>> persistence
>> > > > > > > > > > >>>>> layer for Ignite. There is a very raw draft IEP
>> for
>> > > this
>> > > > > > > > already.
>> > > > > > > > > On
>> > > > > > > > > > >>>> the
>> > > > > > > > > > >>>>> other hand, I think we also can agree that we
>> need a
>> > > > > > > split-brain
>> > > > > > > > > > >>>>> resistant
>> > > > > > > > > > >>>>> replication protocol for caches. There is also
>> an IEP
>> > > > for
>> > > > > > > this.
>> > > > > > > > > > >>>> Neither
>> > > > > > > > > > >>>>> of
>> > > > > > > > > > >>>>> the changes is a good fit for 2.x because they
>> are
>> > > > likely
>> > > > > to
>> > > > > > > > > > >> introduce
>> > > > > > > > > > >>>>> breaking changes in the persistence layer,
>> > > configuration
>> > > > > and
>> > > > > > > > > > >> behavior.
>> > > > > > > > > > >>>>> Additionally, these components are now tightly
>> > > coupled,
>> > > > so
>> > > > > > > there
>> > > > > > > > > is
>> > > > > > > > > > >> no
>> > > > > > > > > > >>>>> way
>> > > > > > > > > > >>>>> these two changes can be implemented in parallel
>> and
>> > > > then
>> > > > > > > merged
>> > > > > > > > > > >>>>> together
>> > > > > > > > > > >>>>> easily. So what we will end up with is having to
>> > > > implement
>> > > > > > > these
>> > > > > > > > > > >>>> changes
>> > > > > > > > > > >>>>> sequentially, fixing all existing tests twice,
>> and
>> > > > > > essentially
>> > > > > > > > > > >>>> throwing
>> > > > > > > > > > >>>>> away half of the work done because the other
>> part of
>> > > the
>> > > > > > > change
>> > > > > > > > is
>> > > > > > > > > > >>>>> re-implemented
>> > > > > > > > > > >>>>> - Similar example goes with getting rid of
>> > > > > > > IgniteInternalFuture
>> > > > > > > > > and
>> > > > > > > > > > >>>>> replacing it with CompletableFuture, and any
>> other
>> > > > change
>> > > > > > that
>> > > > > > > > > > >> touches
>> > > > > > > > > > >>>>> the
>> > > > > > > > > > >>>>> asynchronous part of the code.
>> > > > > > > > > > >>>>>
>> > > > > > > > > > >>>>> Third, I do not see how this choice affects the
>> UX of
>> > > > > Ignite.
>> > > > > > > The
>> > > > > > > > > end
>> > > > > > > > > > >>>> user
>> > > > > > > > > > >>>>> experience must be fixed in the IEP regardless of
>> the
>> > > > > > > development
>> > > > > > > > > > >> process
>> > > > > > > > > > >>>>> and the fact that we have gaps in this area in
>> Ignite
>> > > 2.x
>> > > > > > just
>> > > > > > > > > > confirms
>> > > > > > > > > > >>>>> that.
>> > > > > > > > > > >>>>>
>> > > > > > > > > > >>>>> Pavel, agree that a repo/branch is a
>> technicality, I
>> > > > guess
>> > > > > if
>> > > > > > > > > > >>>> reformulate,
>> > > > > > > > > > >>>>> my point is that we might agree to have a single
>> > > > > development
>> > > > > > > > master
>> > > > > > > > > > >>>> branch
>> > > > > > > > > > >>>>> with 'disassembled' end-to-end functionality for
>> some
>> > > > > period
>> > > > > > of
>> > > > > > > > > time
>> > > > > > > > > > to
>> > > > > > > > > > >>>>> speed up development, and re-assemble the core
>> features
>> > > > > after
>> > > > > > > > > having
>> > > > > > > > > > >>>>> submodules tested independently.
>> > > > > > > > > > >>>>>
>> > > > > > > > > > >>>>> Nikolay,
>> > > > > > > > > > >>>>>> We have many features that have to evolve.
>> > > > > > > > > > >>>>>> Snapshots, rebalance, tooling, tracing, zookeeper
>> > > > support,
>> > > > > > > etc.
>> > > > > > > > > > >>>>> This is not very specific. In the end, resources
>> are
>> > > > > limited
>> > > > > > > and
>> > > > > > > > we
>> > > > > > > > > > >> will
>> > > > > > > > > > >>>>> not be able to drive both tracks simultaneously,
>> > > > especially
>> > > > > > > > after a
>> > > > > > > > > > >>>> couple
>> > > > > > > > > > >>>>> of features having been implemented for Ignite
>> 3.0. If
>> > > > > there
>> > > > > > > are
>> > > > > > > > > > indeed
>> > > > > > > > > > >>>>> some major changes that we want to do in Ignite
>> 2.x
>> > > > instead
>> > > > > > of
>> > > > > > > > > > putting
>> > > > > > > > > > >>>>> effort into 3.0 - let's discuss them. I am just
>> not
>> > > aware
>> > > > > of
>> > > > > > > any,
>> > > > > > > > > > >> that's
>> > > > > > > > > > >>>>> why I am eager to move forward with Ignite 3.0.
>> > > > > > > > > > >>>>>
>> > > > > > > > > > >>>>>> We have bugs and issues that can be fixed in 2.x
>> > > without
>> > > > > > > > breaking
>> > > > > > > > > > >>>> backward
>> > > > > > > > > > >>>>> compatibility.
>> > > > > > > > > > >>>>>> We have many users who are happy with the 2.x
>> with all
>> > > > > it’s
>> > > > > > > > > issues.
>> > > > > > > > > > >>>>> These changes will be covered by end-to-end tests
>> and
>> > > > > > migrated
>> > > > > > > to
>> > > > > > > > > > >> Ignite
>> > > > > > > > > > >>>>> 3.0, so I see no issues here.
>> > > > > > > > > > >>>>>
>> > > > > > > > > > >>>>> Finally, Anton & Nikolay
>> > > > > > > > > > >>>>> I do not have an estimate for this simply because
>> the
>> > > > > > activity
>> > > > > > > is
>> > > > > > > > > > >>>>> community-driven and it depends on the number of
>> people
>> > > > > > willing
>> > > > > > > > to
>> > > > > > > > > > >>>>> contribute. With the current pace, I would hope
>> to have
>> > > > an
>> > > > > RC
>> > > > > > > of
>> > > > > > > > > > Ignite
>> > > > > > > > > > >>>> 3.0
>> > > > > > > > > > >>>>> to be ready by the end of 2021. My gut feeling is
>> that
>> > > by
>> > > > > > > moving
>> > > > > > > > > with
>> > > > > > > > > > >>>>> incremental changes, we will not be able to
>> implement
>> > > > even
>> > > > > > half
>> > > > > > > > of
>> > > > > > > > > > the
>> > > > > > > > > > >>>>> wishlist by that time.
>> > > > > > > > > > >>>>> I doubt that releasing several major releases with
>> > > > breaking
>> > > > > > > > changes
>> > > > > > > > > > >> will
>> > > > > > > > > > >>>>> make Ignite users happy either because each
>> upgrade
>> > > will
>> > > > > cost
>> > > > > > > > > Ignite
>> > > > > > > > > > >>>> users
>> > > > > > > > > > >>>>> money, so the fewer major versions we release, the
>> > > > better.
>> > > > > > Thus
>> > > > > > > > my
>> > > > > > > > > > wish
>> > > > > > > > > > >>>> to
>> > > > > > > > > > >>>>> include all breaking changes in one release.
>> > > > > > > > > > >>>>>
>> > > > > > > > > > >>>>> I'll be now quiet for a while, let's see what
>> other
>> > > > > community
>> > > > > > > > > members
>> > > > > > > > > > >>>>> think.
>> > > > > > > > > > >>>>>
>> > > > > > > > > > >>>>> пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <
>> > > > > > > > ptupitsyn@apache.org
>> > > > > > > > > >:
>> > > > > > > > > > >>>>>
>> > > > > > > > > > >>>>>> 1. Rewriting from scratch is never a good idea.
>> > > > > > > > > > >>>>>> We don't want to follow the path of Netscape and
>> lose
>> > > > all
>> > > > > > our
>> > > > > > > > > users
>> > > > > > > > > > >>>>>> by the time we have a working 3.0 [1]
>> > > > > > > > > > >>>>>>
>> > > > > > > > > > >>>>>> 2. Not sure about new repo - seems like some
>> pain and
>> > > no
>> > > > > > gain,
>> > > > > > > > > > what's
>> > > > > > > > > > >>>> the
>> > > > > > > > > > >>>>>> problem with a branch?
>> > > > > > > > > > >>>>>>
>> > > > > > > > > > >>>>>> 3. We should keep existing integration tests when
>> > > > > possible.
>> > > > > > > > > > >>>>>> We have accumulated a lot of edge case knowledge
>> over
>> > > > the
>> > > > > > > years,
>> > > > > > > > > > >>>>>> it is not a good idea to send all of that down
>> the
>> > > > drain.
>> > > > > > > > > > >>>>>> Yes, integration tests are slow, but they are
>> the most
>> > > > > > > valuable.
>> > > > > > > > > > >>>>>> I think we can move more stuff into nightly runs
>> and
>> > > > have
>> > > > > a
>> > > > > > > fast
>> > > > > > > > > and
>> > > > > > > > > > >>>>> modern
>> > > > > > > > > > >>>>>> basic suite.
>> > > > > > > > > > >>>>>>
>> > > > > > > > > > >>>>>>
>> > > > > > > > > > >>>>>> Alexey, you are much more familiar with the
>> Ignite
>> > > core
>> > > > > > > codebase
>> > > > > > > > > > than
>> > > > > > > > > > >>>>> most
>> > > > > > > > > > >>>>>> of us,
>> > > > > > > > > > >>>>>> can you please explain in more detail which
>> particular
>> > > > > > > feature,
>> > > > > > > > in
>> > > > > > > > > > >> your
>> > > > > > > > > > >>>>>> opinion,
>> > > > > > > > > > >>>>>> mandates this "start from scratch" approach?
>> > > > > > > > > > >>>>>> Is it really not possible at all to follow a less
>> > > > radical
>> > > > > > way?
>> > > > > > > > > > >>>>>>
>> > > > > > > > > > >>>>>>
>> > > > > > > > > > >>>>>> [1]
>> > > > > > > > > > >>>>>>
>> > > > > > > > > > >>>>>>
>> > > > > > > > > > >>>>>
>> > > > > > > > > > >>>>
>> > > > > > > > > > >>
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
>> > > > > > > > > > >>>>>>
>> > > > > > > > > > >>>>>> On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <
>> > > > > > > > > nizhikov@apache.org
>> > > > > > > > > > >
>> > > > > > > > > > >>>>>> wrote:
>> > > > > > > > > > >>>>>>
>> > > > > > > > > > >>>>>>> Hello, Alexey.
>> > > > > > > > > > >>>>>>>
>> > > > > > > > > > >>>>>>> I think that «rewriting from scratch» approach
>> has a
>> > > > high
>> > > > > > > risk
>> > > > > > > > to
>> > > > > > > > > > >>>> make
>> > > > > > > > > > >>>>>> new
>> > > > > > > > > > >>>>>>> features unusable.
>> > > > > > > > > > >>>>>>> At the time Ignite2 was started no-one wants to
>> do
>> > > bad
>> > > > UX
>> > > > > > or
>> > > > > > > > bad
>> > > > > > > > > > >>>>>> features.
>> > > > > > > > > > >>>>>>> Nevertheless, it happen.
>> > > > > > > > > > >>>>>>>
>> > > > > > > > > > >>>>>>> I think we can avoid it with the Ignite3 and
>> > > successors
>> > > > > if
>> > > > > > we
>> > > > > > > > > will
>> > > > > > > > > > >>>> move
>> > > > > > > > > > >>>>>>> step by step without keeping backward
>> compatibility
>> > > > > > > > > > >>>>>>> With the step by step approach, we can focus on
>> each
>> > > > > > > component
>> > > > > > > > > > >>>>>> separately.
>> > > > > > > > > > >>>>>>>
>> > > > > > > > > > >>>>>>>> What new features are we planning to implement
>> for
>> > > > > Ignite
>> > > > > > > 2.x?
>> > > > > > > > > > >>>>>>>
>> > > > > > > > > > >>>>>>> We have many features that have to evolve.
>> > > > > > > > > > >>>>>>> Snapshots, rebalance, tooling, tracing,
>> zookeeper
>> > > > > support,
>> > > > > > > etc.
>> > > > > > > > > > >>>>>>> We have bugs and issues that can be fixed in 2.x
>> > > > without
>> > > > > > > > breaking
>> > > > > > > > > > >>>>>> backward
>> > > > > > > > > > >>>>>>> compatibility.
>> > > > > > > > > > >>>>>>> We have many users who are happy with the 2.x
>> with
>> > > all
>> > > > > it’s
>> > > > > > > > > issues.
>> > > > > > > > > > >>>>>>>
>> > > > > > > > > > >>>>>>>> 2 нояб. 2020 г., в 14:09, Anton Vinogradov <
>> > > > > av@apache.org
>> > > > > > >
>> > > > > > > > > > >>>>> написал(а):
>> > > > > > > > > > >>>>>>>>
>> > > > > > > > > > >>>>>>>> Alexey,
>> > > > > > > > > > >>>>>>>>
>> > > > > > > > > > >>>>>>>> Do we have any estimates of how fast we'll be
>> able
>> > > to
>> > > > > gain
>> > > > > > > > > > >>>>>>> production-ready
>> > > > > > > > > > >>>>>>>> AI 3.0 in case of a "new repo" choice?
>> > > > > > > > > > >>>>>>>>
>> > > > > > > > > > >>>>>>>> On Mon, Nov 2, 2020 at 2:01 PM Alexey
>> Goncharuk <
>> > > > > > > > > > >>>>>>> alexey.goncharuk@gmail.com>
>> > > > > > > > > > >>>>>>>> wrote:
>> > > > > > > > > > >>>>>>>>
>> > > > > > > > > > >>>>>>>>> Nikolay,
>> > > > > > > > > > >>>>>>>>>
>> > > > > > > > > > >>>>>>>>> What new features are we planning to
>> implement for
>> > > > > Ignite
>> > > > > > > > 2.x?
>> > > > > > > > > I
>> > > > > > > > > > >>>>> think
>> > > > > > > > > > >>>>>>> once
>> > > > > > > > > > >>>>>>>>> we commence working on Ignite 3.0, we should
>> > > > gradually
>> > > > > > > cease
>> > > > > > > > > the
>> > > > > > > > > > >>>>>>> activity
>> > > > > > > > > > >>>>>>>>> on Ignite 2.x to mere bugfixes because such
>> > > parallel
>> > > > > > > > > development
>> > > > > > > > > > >>>>> will
>> > > > > > > > > > >>>>>> be
>> > > > > > > > > > >>>>>>>>> overwhelming regardless of how we choose to
>> > > proceed.
>> > > > > > > > > > >>>>>>>>>
>> > > > > > > > > > >>>>>>>>> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <
>> > > > > > > > > > nizhikov@apache.org
>> > > > > > > > > > >>>>> :
>> > > > > > > > > > >>>>>>>>>
>> > > > > > > > > > >>>>>>>>>> To be clear:
>> > > > > > > > > > >>>>>>>>>>
>> > > > > > > > > > >>>>>>>>>>> I would suggest creating a new repository
>> for
>> > > > Ignite
>> > > > > > 3.0
>> > > > > > > > > > >>>>> (perhaps, a
>> > > > > > > > > > >>>>>>>>> new
>> > > > > > > > > > >>>>>>>>>> clean branch, but a new repo looks nicer to
>> me)
>> > > and
>> > > > a
>> > > > > > new
>> > > > > > > > > Ignite
>> > > > > > > > > > >>>>> 3.0
>> > > > > > > > > > >>>>>>>>>> TeamCity project.
>> > > > > > > > > > >>>>>>>>>>
>> > > > > > > > > > >>>>>>>>>> +1 for new Team City project.
>> > > > > > > > > > >>>>>>>>>> +1 for new branch for Ignite3.
>> > > > > > > > > > >>>>>>>>>> -1 for new repo.
>> > > > > > > > > > >>>>>>>>>>
>> > > > > > > > > > >>>>>>>>>>
>> > > > > > > > > > >>>>>>>>>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <
>> > > > > > > > > > >>>> NIzhikov.dev@gmail.com
>> > > > > > > > > > >>>>>>
>> > > > > > > > > > >>>>>>>>>> написал(а):
>> > > > > > > > > > >>>>>>>>>>>
>> > > > > > > > > > >>>>>>>>>>> Hello, Alexey.
>> > > > > > > > > > >>>>>>>>>>>
>> > > > > > > > > > >>>>>>>>>>> I think it will hurt our project more than
>> help.
>> > > > > > > > > > >>>>>>>>>>> Developing new features for 2 separate
>> branches
>> > > > with
>> > > > > > the
>> > > > > > > > > > >>>> different
>> > > > > > > > > > >>>>>>> APIs
>> > > > > > > > > > >>>>>>>>>> and internal structure is overwhelming
>> > > > > > > > > > >>>>>>>>>>>
>> > > > > > > > > > >>>>>>>>>>> Maybe we should relax a bit requirements for
>> > > > Ignite3?
>> > > > > > > > > > >>>>>>>>>>> Maybe we should move step by step and make
>> > > Ignite3
>> > > > > with
>> > > > > > > new
>> > > > > > > > > > >>>>>>>>>> configuration than Ignite4 with new
>> transactions,
>> > > > etc?
>> > > > > > > > > > >>>>>>>>>>>
>> > > > > > > > > > >>>>>>>>>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk
>> <
>> > > > > > > > > > >>>>>>>>> alexey.goncharuk@gmail.com>
>> > > > > > > > > > >>>>>>>>>> написал(а):
>> > > > > > > > > > >>>>>>>>>>>>
>> > > > > > > > > > >>>>>>>>>>>> Igniters,
>> > > > > > > > > > >>>>>>>>>>>>
>> > > > > > > > > > >>>>>>>>>>>> I wanted to pitch a rather radical idea
>> > > regarding
>> > > > > the
>> > > > > > > > Ignite
>> > > > > > > > > > >>>> 3.0
>> > > > > > > > > > >>>>>>>>>>>> development which has occurred to me some
>> time
>> > > > ago.
>> > > > > > > > > > >>>>>>>>>>>>
>> > > > > > > > > > >>>>>>>>>>>> We already have several IEPs targeted to
>> Ignite
>> > > > 3.0
>> > > > > > > which
>> > > > > > > > > > imply
>> > > > > > > > > > >>>>>> major
>> > > > > > > > > > >>>>>>>>>>>> changes to the codebase (the change in
>> > > replication
>> > > > > > > > protocol
>> > > > > > > > > > and
>> > > > > > > > > > >>>>>> thus
>> > > > > > > > > > >>>>>>>>>>>> transactions, change in binary format,
>> updated
>> > > > > > > > metastorage,
>> > > > > > > > > > >>>> etc).
>> > > > > > > > > > >>>>>> We
>> > > > > > > > > > >>>>>>>>>> also
>> > > > > > > > > > >>>>>>>>>>>> planned significant changes in public APIs:
>> > > > > > > configuration
>> > > > > > > > > > >>>> format
>> > > > > > > > > > >>>>>>>>> change,
>> > > > > > > > > > >>>>>>>>>>>> improvements in cache APIs, SQL APIs,
>> > > transaction
>> > > > > mode
>> > > > > > > > > rework.
>> > > > > > > > > > >>>>> The
>> > > > > > > > > > >>>>>>>>>> wishlist
>> > > > > > > > > > >>>>>>>>>>>> of changes for Ignite 3.0 is huge.
>> > > > > > > > > > >>>>>>>>>>>>
>> > > > > > > > > > >>>>>>>>>>>> So, I was wondering whether it makes sense
>> to
>> > > try
>> > > > to
>> > > > > > > > change
>> > > > > > > > > > the
>> > > > > > > > > > >>>>> old
>> > > > > > > > > > >>>>>>>>>>>> codebase, or start with a new baseline and
>> move
>> > > > old
>> > > > > > > pieces
>> > > > > > > > > of
>> > > > > > > > > > >>>>> code
>> > > > > > > > > > >>>>>>>>> that
>> > > > > > > > > > >>>>>>>>>> do
>> > > > > > > > > > >>>>>>>>>>>> not require significant rework.
>> Personally, I
>> > > > would
>> > > > > go
>> > > > > > > > with
>> > > > > > > > > > the
>> > > > > > > > > > >>>>>>> second
>> > > > > > > > > > >>>>>>>>>>>> option for the following reasons:
>> > > > > > > > > > >>>>>>>>>>>>
>> > > > > > > > > > >>>>>>>>>>>> - We have a chance to shift the development
>> > > > paradigm
>> > > > > > in
>> > > > > > > > the
>> > > > > > > > > > >>>>> project
>> > > > > > > > > > >>>>>>>>> and
>> > > > > > > > > > >>>>>>>>>>>> introduce the practice of true unit-tests.
>> In
>> > > the
>> > > > > new
>> > > > > > > > > baseline
>> > > > > > > > > > >>>> at
>> > > > > > > > > > >>>>>> the
>> > > > > > > > > > >>>>>>>>>>>> beginning there will be no ability to run
>> an
>> > > > > > end-to-end
>> > > > > > > > > > >>>> scenario,
>> > > > > > > > > > >>>>>>>>> thus
>> > > > > > > > > > >>>>>>>>>> we
>> > > > > > > > > > >>>>>>>>>>>> will be forced to write unit-tests. So
>> far, such
>> > > > > > > practice
>> > > > > > > > > was
>> > > > > > > > > > >>>>> hard
>> > > > > > > > > > >>>>>> to
>> > > > > > > > > > >>>>>>>>>>>> implement because of tight coupling between
>> > > Ignite
>> > > > > > > > > components
>> > > > > > > > > > >>>> and
>> > > > > > > > > > >>>>>>>>>> inability
>> > > > > > > > > > >>>>>>>>>>>> to instantiate components without an
>> instance of
>> > > > > > > > > > KernalContext.
>> > > > > > > > > > >>>>> For
>> > > > > > > > > > >>>>>>>>>>>> example, we should be able to thoroughly
>> test
>> > > > > internal
>> > > > > > > > > > >>>>> primitives,
>> > > > > > > > > > >>>>>>>>>> such as
>> > > > > > > > > > >>>>>>>>>>>> replication protocol (without actual
>> > > > communication),
>> > > > > > > > > > >>>> distributed
>> > > > > > > > > > >>>>>>>>>>>> metastorage contracts, persistence layer,
>> etc.
>> > > > > > > > > > >>>>>>>>>>>> - We will significantly reduce the
>> development
>> > > > cycle
>> > > > > > in
>> > > > > > > > the
>> > > > > > > > > > >>>>>> beginning
>> > > > > > > > > > >>>>>>>>>>>> (right now the RunAll takes two hours of
>> > > > > astronomical
>> > > > > > > time
>> > > > > > > > > > with
>> > > > > > > > > > >>>>>> empty
>> > > > > > > > > > >>>>>>>>>> TC;
>> > > > > > > > > > >>>>>>>>>>>> in the new approach developer will be able
>> to
>> > > run
>> > > > > ALL
>> > > > > > > > tests
>> > > > > > > > > > >>>>> locally
>> > > > > > > > > > >>>>>>>>> in
>> > > > > > > > > > >>>>>>>>>> a
>> > > > > > > > > > >>>>>>>>>>>> matter of minutes)
>> > > > > > > > > > >>>>>>>>>>>> - We can get rid of TC bot and enforce
>> green TC
>> > > by
>> > > > > > > > > integrating
>> > > > > > > > > > >>>> TC
>> > > > > > > > > > >>>>>>>>> build
>> > > > > > > > > > >>>>>>>>>>>> results with GitHub PRs (the same way
>> Travis is
>> > > > > > > currently
>> > > > > > > > > > >>>>>> integrated
>> > > > > > > > > > >>>>>>>>>> to PR
>> > > > > > > > > > >>>>>>>>>>>> check). We should restrict PR merge
>> without a TC
>> > > > > check
>> > > > > > > > > > >>>>>>>>>>>> - We will still have to re-write all
>> tests, but
>> > > > only
>> > > > > > > once.
>> > > > > > > > > If
>> > > > > > > > > > >>>> we
>> > > > > > > > > > >>>>>> try
>> > > > > > > > > > >>>>>>>>> to
>> > > > > > > > > > >>>>>>>>>>>> modify the old codebase, we would need to
>> modify
>> > > > all
>> > > > > > the
>> > > > > > > > > tests
>> > > > > > > > > > >>>>> for
>> > > > > > > > > > >>>>>>>>>> every
>> > > > > > > > > > >>>>>>>>>>>> major change (public API change,
>> configuration
>> > > > > change)
>> > > > > > > > > > >>>>>>>>>>>> - We will have fewer conflicts when working
>> > > > > together.
>> > > > > > > For
>> > > > > > > > > > >>>>> example,
>> > > > > > > > > > >>>>>> I
>> > > > > > > > > > >>>>>>>>>>>> cannot imagine how one would merge two
>> changes
>> > > of
>> > > > > > > getting
>> > > > > > > > > rid
>> > > > > > > > > > >>>> of
>> > > > > > > > > > >>>>>>>>>>>> IgniteFuture and changes in replication
>> > > protocol,
>> > > > > for
>> > > > > > > > > example
>> > > > > > > > > > >>>>>>>>>>>>
>> > > > > > > > > > >>>>>>>>>>>> Technically, I would suggest creating a new
>> > > > > repository
>> > > > > > > for
>> > > > > > > > > > >>>> Ignite
>> > > > > > > > > > >>>>>> 3.0
>> > > > > > > > > > >>>>>>>>>>>> (perhaps, a new clean branch, but a new
>> repo
>> > > looks
>> > > > > > nicer
>> > > > > > > > to
>> > > > > > > > > > me)
>> > > > > > > > > > >>>>>> and a
>> > > > > > > > > > >>>>>>>>>> new
>> > > > > > > > > > >>>>>>>>>>>> Ignite 3.0 TeamCity project.
>> > > > > > > > > > >>>>>>>>>>>>
>> > > > > > > > > > >>>>>>>>>>>> While it may seem quite radical, I do
>> believe
>> > > that
>> > > > > > this
>> > > > > > > > > > >>>> approach
>> > > > > > > > > > >>>>>> will
>> > > > > > > > > > >>>>>>>>>> give
>> > > > > > > > > > >>>>>>>>>>>> us more benefits than trying to make such
>> major
>> > > > > > changes
>> > > > > > > in
>> > > > > > > > > the
>> > > > > > > > > > >>>>>>>>> existing
>> > > > > > > > > > >>>>>>>>>>>> codebase. If needed, let's schedule a
>> discord
>> > > chat
>> > > > > > like
>> > > > > > > > > before
>> > > > > > > > > > >>>> to
>> > > > > > > > > > >>>>>>>>>> discuss
>> > > > > > > > > > >>>>>>>>>>>> this.
>> > > > > > > > > > >>>>>>>>>>>>
>> > > > > > > > > > >>>>>>>>>>>> WDYT?
>> > > > > > > > > > >>>>>>>>>>>
>> > > > > > > > > > >>>>>>>>>>
>> > > > > > > > > > >>>>>>>>>>
>> > > > > > > > > > >>>>>>>>>
>> > > > > > > > > > >>>>>>>
>> > > > > > > > > > >>>>>>>
>> > > > > > > > > > >>>>>>
>> > > > > > > > > > >>>>>
>> > > > > > > > > > >>>>
>> > > > > > > > > > >>>>
>> > > > > > > > > > >>>> --
>> > > > > > > > > > >>>> Best regards,
>> > > > > > > > > > >>>> Andrey V. Mashenkov
>> > > > > > > > > > >>>>
>> > > > > > > > > > >>
>> > > > > > > > > > >>
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> >
>> > --
>> >
>> > Best regards,
>> > Alexei Scherbakov
>>
>
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Valentin Kulichenko <va...@gmail.com>.
Maxim,
2.x and 3.x will have to coexist for some time - I don't see how we can
avoid this considering the set of proposed changes. That said, we
effectively will need to have two "masters" - one for each major version.
Master for 3.x can technically be a branch in the existing repo, but having
a separate repo seems cleaner, simply because it will not be a "branch" in
the traditional sense.
Note that the new repo will still be under the Apache org, with the same
set of committers, managed by the community, etc. All the development
happening for 3.0 must follow the rules that we currently have (if
anything, it's an opportunity to improve those rules).
As I said during the call on Friday, I strongly believe that if there is a
transparency issue, it will exist regardless of the approach we choose for
3.0. If community members develop without IEPs or public discussions, this
will happen for both 2.x and 3.x unless we address this separately. I don't
see how this is related to Alexey's suggestion, which targets *technical*
issues with the product more than anything else. This a way to achieve
better modularity, introduce better coverage with unit tests, reduce
conflicts during development, etc.
Coming back to transparency, let's identify the issues and fix them. It
probably makes sense to have a separate discussion on this topic.
-Val
On Tue, Nov 10, 2020 at 1:05 PM Maxim Muzafarov <mm...@apache.org> wrote:
> Sergey,
>
>
> Your summary makes sense to me.
>
> However, how we come up from *Development transparency* to *create a
> separate public repository dedicated for 3.0*? For me *development
> transparency* is about making changes in the master branch. These
> changes will definitely be seen by all the Ignite developers.
>
> A dedicated public repository is technically public and visible for
> everyone, but it allows development without IEPs, without public
> discussion (since all the code changes are not related to the master
> branch) it also allows a large number of assumptions and deviations
> (like code-style violations). It also not about *development
> transparency* since developers which are working on 3.0 is only a
> subset of all Ignite developers which may continue working on 2.x. For
> me, this would be a huge step backwards.
>
> Ignite veterans should remember how long the branch stabilization took
> for the 2.x version with the PDS.
>
>
> I think each breaking change should be passed through the master branch.
>
> On Tue, 10 Nov 2020 at 22:18, Alexei Scherbakov
> <al...@gmail.com> wrote:
> >
> > Makes sense to me.
> >
> > вт, 10 нояб. 2020 г. в 18:47, Sergey Chugunov <sergey.chugunov@gmail.com
> >:
> >
> > > Igniters,
> > >
> > > I thought over Friday meeting ideas and concerns and summarized them in
> > > these three points:
> > >
> > >
> > > 1. *Components design unification approach.* New proposed components
> > > will be developed by different contributors, but they need to be
> unified
> > > and should integrate with each other easily. To ensure that I
> suggest
> > > calling an architecture group that will create design guidelines
> for all
> > > components and high-level overview of overall architecture. How
> code is
> > > split into components, what are component boundaries, how component
> > > lifecycle works and what are its interfaces - all these and other
> > > questions
> > > should be covered.
> > >
> > > 2. *Scope management.* Apache 3.0 should be implemented within a
> > > reasonable time, so we need some procedure to decide whether a
> > > particular
> > > feature should be dropped from the scope of 3.0 and postponed to 3.1
> > > release. To do so I suggest to range all features by two parameters:
> > > criticality for 3.0 and amount of breaking changes. 3.0 scope should
> > > include features of high criticality AND features with a big amount
> of
> > > breaking changes. All other features can be made optional.
> > >
> > > 3. *Development transparency.* Development of all components should
> be
> > > made as transparent for everyone as possible. Any contributor
> should be
> > > able to look over any component at any stage of development. To
> achieve
> > > this I suggest to create a separate public repository dedicated for
> 3.0
> > > development. It will make the code available for everyone but when
> > > development of 3.0 is done we won't loose any stars of our current
> > > repository as we merge dev repo into main one and drop dev.
> > >
> > > Do these ideas make sense to you? Are there any concerns not covered by
> > > these suggestions?
> > >
> > > On Fri, Nov 6, 2020 at 7:36 PM Kseniya Romanova <
> romanova.ks.spb@gmail.com
> > > >
> > > wrote:
> > >
> > > > Here are the slides from Alexey Goncharuk. Let's think this over and
> > > > continue on Monday:
> > > >
> > > >
> > >
> https://go.gridgain.com/rs/491-TWR-806/images/Ignite_3_Plans_and_development_process.pdf
> > > >
> > > > чт, 5 нояб. 2020 г. в 11:13, Anton Vinogradov <av...@apache.org>:
> > > >
> > > > > Folks,
> > > > >
> > > > > Should we perform cleanup work before (r)evolutional changes?
> > > > > My huge proposal is to get rid of things which we don't need anyway
> > > > > - local caches,
> > > > > - strange tx modes,
> > > > > - code overcomplexity because of RollingUpgrade feature never
> attended
> > > at
> > > > > AI,
> > > > > - etc,
> > > > > before choosing the way.
> > > > >
> > > > > On Tue, Nov 3, 2020 at 3:31 PM Valentin Kulichenko <
> > > > > valentin.kulichenko@gmail.com> wrote:
> > > > >
> > > > > > Ksenia, thanks for scheduling this on such short notice!
> > > > > >
> > > > > > As for the original topic, I do support Alexey's idea. We're not
> > > going
> > > > to
> > > > > > rewrite anything from scratch, as most of the components are
> going to
> > > > be
> > > > > > moved as-is or with minimal modifications. However, the changes
> that
> > > > are
> > > > > > proposed imply serious rework of the core parts of the code,
> which
> > > are
> > > > > not
> > > > > > properly decoupled from each other and from other parts. This
> makes
> > > the
> > > > > > incremental approach borderline impossible. Developing in a new
> repo,
> > > > > > however, addresses this concern. As a bonus, we can also
> refactor the
> > > > > code,
> > > > > > introduce better decoupling, get rid of kernel context, and
> develop
> > > > unit
> > > > > > tests (finally!).
> > > > > >
> > > > > > Basically, this proposal only affects the *process*, not the set
> of
> > > > > changes
> > > > > > we had discussed before. Ignite 3.0 is our unique chance to make
> > > things
> > > > > > right.
> > > > > >
> > > > > > -Val
> > > > > >
> > > > > > On Tue, Nov 3, 2020 at 3:06 AM Kseniya Romanova <
> > > > > romanova.ks.spb@gmail.com
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Pavel, all the interesting points will be anyway published
> here in
> > > > > > English
> > > > > > > (as the principal "if it's not on devlist it doesn't happened"
> is
> > > > still
> > > > > > > relevant). This is just a quick call for a group of developers.
> > > Later
> > > > > we
> > > > > > > can do a separate presentation of idea and discussion in
> English as
> > > > we
> > > > > > did
> > > > > > > for the Ignite 3.0 draft of changes.
> > > > > > >
> > > > > > > вт, 3 нояб. 2020 г. в 13:52, Pavel Tupitsyn <
> ptupitsyn@apache.org
> > > >:
> > > > > > >
> > > > > > > > Kseniya,
> > > > > > > >
> > > > > > > > Thanks for scheduling this call.
> > > > > > > > Do you think we can switch to English if non-Russian speaking
> > > > > community
> > > > > > > > members decide to join?
> > > > > > > >
> > > > > > > > On Tue, Nov 3, 2020 at 1:32 PM Kseniya Romanova <
> > > > > > > romanova.ks.spb@gmail.com
> > > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Let's do this community discussion open. Here's the link on
> > > zoom
> > > > > call
> > > > > > > in
> > > > > > > > > Russian for Friday 6 PM:
> > > > > > > > >
> > > > >
> https://www.meetup.com/Moscow-Apache-Ignite-Meetup/events/274360378/
> > > > > > > > >
> > > > > > > > > вт, 3 нояб. 2020 г. в 12:49, Nikolay Izhikov <
> > > > nizhikov@apache.org
> > > > > >:
> > > > > > > > >
> > > > > > > > > > Time works for me.
> > > > > > > > > >
> > > > > > > > > > > 3 нояб. 2020 г., в 12:40, Alexey Goncharuk <
> > > > > > > > alexey.goncharuk@gmail.com
> > > > > > > > > >
> > > > > > > > > > написал(а):
> > > > > > > > > > >
> > > > > > > > > > > Nikolay,
> > > > > > > > > > >
> > > > > > > > > > > I am up for the call. I will try to explain my
> reasoning in
> > > > > > greater
> > > > > > > > > > detail
> > > > > > > > > > > and will be glad to hear the concerns. Will this
> Friday,
> > > Nov
> > > > > 6th,
> > > > > > > > work?
> > > > > > > > > > >
> > > > > > > > > > > вт, 3 нояб. 2020 г. в 10:09, Nikolay Izhikov <
> > > > > > nizhikov@apache.org
> > > > > > > >:
> > > > > > > > > > >
> > > > > > > > > > >> Igniters, should we have a call for this topic?
> > > > > > > > > > >>
> > > > > > > > > > >>> 2 нояб. 2020 г., в 18:53, Pavel Tupitsyn <
> > > > > ptupitsyn@apache.org
> > > > > > >
> > > > > > > > > > >> написал(а):
> > > > > > > > > > >>>
> > > > > > > > > > >>>> not intend to rewrite everything from scratch
> > > > > > > > > > >>>
> > > > > > > > > > >>>> Every single test from Ignite 2.x should be moved to
> > > > Ignite
> > > > > 3
> > > > > > > > > > >>>> regardless of how we choose to proceed.
> > > > > > > > > > >>>
> > > > > > > > > > >>> Alexey, thank you for the explanation, this
> addresses all
> > > > of
> > > > > my
> > > > > > > > > > concerns.
> > > > > > > > > > >>>
> > > > > > > > > > >>>
> > > > > > > > > > >>>
> > > > > > > > > > >>>
> > > > > > > > > > >>>
> > > > > > > > > > >>> On Mon, Nov 2, 2020 at 6:43 PM Andrey Mashenkov <
> > > > > > > > > > >> andrey.mashenkov@gmail.com>
> > > > > > > > > > >>> wrote:
> > > > > > > > > > >>>
> > > > > > > > > > >>>> Hi, Igniters.
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> * AFAIU, we need a new repo if we want to apply
> > > different
> > > > > > > > > restrictions
> > > > > > > > > > >> to
> > > > > > > > > > >>>> pull requests,
> > > > > > > > > > >>>> otherwise I see no difference for myself.
> > > > > > > > > > >>>> E.g. make static analysis (do we have?), compile,
> > > styles,
> > > > > and
> > > > > > > > > javadoc
> > > > > > > > > > >>>> checks mandatory.
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> I think that relaxed requirements here will lead to
> bad
> > > > > > product
> > > > > > > > > > quality.
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> * Agree with Pavel, we should 'keep' integrations
> tests
> > > > > > somehow.
> > > > > > > > > > >>>> During active development tests will be broken most
> of
> > > > time,
> > > > > > so,
> > > > > > > > > > >>>> I'd port them e.g. suite-by-suite once we will have
> a
> > > > stable
> > > > > > and
> > > > > > > > > > >> featured
> > > > > > > > > > >>>> environment to run them and of course make test's
> code
> > > > clear
> > > > > > and
> > > > > > > > > avoid
> > > > > > > > > > >>>> bad/non-relevant ones.
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> * I like bottom-up approach.
> > > > > > > > > > >>>> With it we could make a better framework. I mean
> clear
> > > > > > component
> > > > > > > > > > >> lifecycle,
> > > > > > > > > > >>>> component wiring mechanics, general methods to
> approach
> > > > core
> > > > > > > > > > components
> > > > > > > > > > >>>> such as exchange/communication
> > > > > > > > > > >>>> to avoid code mess like we have in ExchangeFuture
> with
> > > all
> > > > > > these
> > > > > > > > > > custom
> > > > > > > > > > >>>> callbacks for each component, interfaces like
> > > > > > > > > > >>>> PartitionsExchangeAware,
> IgniteChangeGlobalStateSupport
> > > > and
> > > > > > > > > > >>>> a pack of
> > > > > > > > > > >>
> > > > > start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected
> > > > > > > > > > >>>> and so on in various unexpected places.
> > > > > > > > > > >>>> Hope, we will be able to port most of the good code
> to
> > > the
> > > > > new
> > > > > > > > > > framework
> > > > > > > > > > >>>> version.
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk <
> > > > > > > > > > >>>> alexey.goncharuk@gmail.com>
> > > > > > > > > > >>>> wrote:
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>> Nikolay, Pavel,
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> Thanks for the feedback! First of all, I wanted to
> > > stress
> > > > > > that
> > > > > > > I
> > > > > > > > do
> > > > > > > > > > not
> > > > > > > > > > >>>>> intend to rewrite everything from scratch (I never
> used
> > > > > this
> > > > > > > > > phrase).
> > > > > > > > > > >>>> There
> > > > > > > > > > >>>>> are significant parts of code that would be moved
> with
> > > > > > minimal
> > > > > > > > > > >>>>> modifications. Second, I never said that we will
> get
> > > rid
> > > > of
> > > > > > the
> > > > > > > > old
> > > > > > > > > > >> tests
> > > > > > > > > > >>>>> codebase. Every single test from Ignite 2.x should
> be
> > > > moved
> > > > > > to
> > > > > > > > > > Ignite 3
> > > > > > > > > > >>>>> regardless of how we choose to proceed.
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> My point is that for some parts of the code a clean
> > > > > bottom-up
> > > > > > > > > > >>>>> implementation will be cheaper in many ways. Let me
> > > give
> > > > > you
> > > > > > a
> > > > > > > > few
> > > > > > > > > > >>>> concrete
> > > > > > > > > > >>>>> examples:
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> - I think no one can object that we need a cleanly
> > > > > separated
> > > > > > > > > > >>>> persistence
> > > > > > > > > > >>>>> layer for Ignite. There is a very raw draft IEP
> for
> > > this
> > > > > > > > already.
> > > > > > > > > On
> > > > > > > > > > >>>> the
> > > > > > > > > > >>>>> other hand, I think we also can agree that we
> need a
> > > > > > > split-brain
> > > > > > > > > > >>>>> resistant
> > > > > > > > > > >>>>> replication protocol for caches. There is also an
> IEP
> > > > for
> > > > > > > this.
> > > > > > > > > > >>>> Neither
> > > > > > > > > > >>>>> of
> > > > > > > > > > >>>>> the changes is a good fit for 2.x because they are
> > > > likely
> > > > > to
> > > > > > > > > > >> introduce
> > > > > > > > > > >>>>> breaking changes in the persistence layer,
> > > configuration
> > > > > and
> > > > > > > > > > >> behavior.
> > > > > > > > > > >>>>> Additionally, these components are now tightly
> > > coupled,
> > > > so
> > > > > > > there
> > > > > > > > > is
> > > > > > > > > > >> no
> > > > > > > > > > >>>>> way
> > > > > > > > > > >>>>> these two changes can be implemented in parallel
> and
> > > > then
> > > > > > > merged
> > > > > > > > > > >>>>> together
> > > > > > > > > > >>>>> easily. So what we will end up with is having to
> > > > implement
> > > > > > > these
> > > > > > > > > > >>>> changes
> > > > > > > > > > >>>>> sequentially, fixing all existing tests twice, and
> > > > > > essentially
> > > > > > > > > > >>>> throwing
> > > > > > > > > > >>>>> away half of the work done because the other part
> of
> > > the
> > > > > > > change
> > > > > > > > is
> > > > > > > > > > >>>>> re-implemented
> > > > > > > > > > >>>>> - Similar example goes with getting rid of
> > > > > > > IgniteInternalFuture
> > > > > > > > > and
> > > > > > > > > > >>>>> replacing it with CompletableFuture, and any other
> > > > change
> > > > > > that
> > > > > > > > > > >> touches
> > > > > > > > > > >>>>> the
> > > > > > > > > > >>>>> asynchronous part of the code.
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> Third, I do not see how this choice affects the UX
> of
> > > > > Ignite.
> > > > > > > The
> > > > > > > > > end
> > > > > > > > > > >>>> user
> > > > > > > > > > >>>>> experience must be fixed in the IEP regardless of
> the
> > > > > > > development
> > > > > > > > > > >> process
> > > > > > > > > > >>>>> and the fact that we have gaps in this area in
> Ignite
> > > 2.x
> > > > > > just
> > > > > > > > > > confirms
> > > > > > > > > > >>>>> that.
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> Pavel, agree that a repo/branch is a technicality,
> I
> > > > guess
> > > > > if
> > > > > > > > > > >>>> reformulate,
> > > > > > > > > > >>>>> my point is that we might agree to have a single
> > > > > development
> > > > > > > > master
> > > > > > > > > > >>>> branch
> > > > > > > > > > >>>>> with 'disassembled' end-to-end functionality for
> some
> > > > > period
> > > > > > of
> > > > > > > > > time
> > > > > > > > > > to
> > > > > > > > > > >>>>> speed up development, and re-assemble the core
> features
> > > > > after
> > > > > > > > > having
> > > > > > > > > > >>>>> submodules tested independently.
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> Nikolay,
> > > > > > > > > > >>>>>> We have many features that have to evolve.
> > > > > > > > > > >>>>>> Snapshots, rebalance, tooling, tracing, zookeeper
> > > > support,
> > > > > > > etc.
> > > > > > > > > > >>>>> This is not very specific. In the end, resources
> are
> > > > > limited
> > > > > > > and
> > > > > > > > we
> > > > > > > > > > >> will
> > > > > > > > > > >>>>> not be able to drive both tracks simultaneously,
> > > > especially
> > > > > > > > after a
> > > > > > > > > > >>>> couple
> > > > > > > > > > >>>>> of features having been implemented for Ignite
> 3.0. If
> > > > > there
> > > > > > > are
> > > > > > > > > > indeed
> > > > > > > > > > >>>>> some major changes that we want to do in Ignite 2.x
> > > > instead
> > > > > > of
> > > > > > > > > > putting
> > > > > > > > > > >>>>> effort into 3.0 - let's discuss them. I am just not
> > > aware
> > > > > of
> > > > > > > any,
> > > > > > > > > > >> that's
> > > > > > > > > > >>>>> why I am eager to move forward with Ignite 3.0.
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>>> We have bugs and issues that can be fixed in 2.x
> > > without
> > > > > > > > breaking
> > > > > > > > > > >>>> backward
> > > > > > > > > > >>>>> compatibility.
> > > > > > > > > > >>>>>> We have many users who are happy with the 2.x
> with all
> > > > > it’s
> > > > > > > > > issues.
> > > > > > > > > > >>>>> These changes will be covered by end-to-end tests
> and
> > > > > > migrated
> > > > > > > to
> > > > > > > > > > >> Ignite
> > > > > > > > > > >>>>> 3.0, so I see no issues here.
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> Finally, Anton & Nikolay
> > > > > > > > > > >>>>> I do not have an estimate for this simply because
> the
> > > > > > activity
> > > > > > > is
> > > > > > > > > > >>>>> community-driven and it depends on the number of
> people
> > > > > > willing
> > > > > > > > to
> > > > > > > > > > >>>>> contribute. With the current pace, I would hope to
> have
> > > > an
> > > > > RC
> > > > > > > of
> > > > > > > > > > Ignite
> > > > > > > > > > >>>> 3.0
> > > > > > > > > > >>>>> to be ready by the end of 2021. My gut feeling is
> that
> > > by
> > > > > > > moving
> > > > > > > > > with
> > > > > > > > > > >>>>> incremental changes, we will not be able to
> implement
> > > > even
> > > > > > half
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > >>>>> wishlist by that time.
> > > > > > > > > > >>>>> I doubt that releasing several major releases with
> > > > breaking
> > > > > > > > changes
> > > > > > > > > > >> will
> > > > > > > > > > >>>>> make Ignite users happy either because each upgrade
> > > will
> > > > > cost
> > > > > > > > > Ignite
> > > > > > > > > > >>>> users
> > > > > > > > > > >>>>> money, so the fewer major versions we release, the
> > > > better.
> > > > > > Thus
> > > > > > > > my
> > > > > > > > > > wish
> > > > > > > > > > >>>> to
> > > > > > > > > > >>>>> include all breaking changes in one release.
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> I'll be now quiet for a while, let's see what other
> > > > > community
> > > > > > > > > members
> > > > > > > > > > >>>>> think.
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <
> > > > > > > > ptupitsyn@apache.org
> > > > > > > > > >:
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>>> 1. Rewriting from scratch is never a good idea.
> > > > > > > > > > >>>>>> We don't want to follow the path of Netscape and
> lose
> > > > all
> > > > > > our
> > > > > > > > > users
> > > > > > > > > > >>>>>> by the time we have a working 3.0 [1]
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>> 2. Not sure about new repo - seems like some pain
> and
> > > no
> > > > > > gain,
> > > > > > > > > > what's
> > > > > > > > > > >>>> the
> > > > > > > > > > >>>>>> problem with a branch?
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>> 3. We should keep existing integration tests when
> > > > > possible.
> > > > > > > > > > >>>>>> We have accumulated a lot of edge case knowledge
> over
> > > > the
> > > > > > > years,
> > > > > > > > > > >>>>>> it is not a good idea to send all of that down the
> > > > drain.
> > > > > > > > > > >>>>>> Yes, integration tests are slow, but they are the
> most
> > > > > > > valuable.
> > > > > > > > > > >>>>>> I think we can move more stuff into nightly runs
> and
> > > > have
> > > > > a
> > > > > > > fast
> > > > > > > > > and
> > > > > > > > > > >>>>> modern
> > > > > > > > > > >>>>>> basic suite.
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>> Alexey, you are much more familiar with the Ignite
> > > core
> > > > > > > codebase
> > > > > > > > > > than
> > > > > > > > > > >>>>> most
> > > > > > > > > > >>>>>> of us,
> > > > > > > > > > >>>>>> can you please explain in more detail which
> particular
> > > > > > > feature,
> > > > > > > > in
> > > > > > > > > > >> your
> > > > > > > > > > >>>>>> opinion,
> > > > > > > > > > >>>>>> mandates this "start from scratch" approach?
> > > > > > > > > > >>>>>> Is it really not possible at all to follow a less
> > > > radical
> > > > > > way?
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>> [1]
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>> On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <
> > > > > > > > > nizhikov@apache.org
> > > > > > > > > > >
> > > > > > > > > > >>>>>> wrote:
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>>> Hello, Alexey.
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>> I think that «rewriting from scratch» approach
> has a
> > > > high
> > > > > > > risk
> > > > > > > > to
> > > > > > > > > > >>>> make
> > > > > > > > > > >>>>>> new
> > > > > > > > > > >>>>>>> features unusable.
> > > > > > > > > > >>>>>>> At the time Ignite2 was started no-one wants to
> do
> > > bad
> > > > UX
> > > > > > or
> > > > > > > > bad
> > > > > > > > > > >>>>>> features.
> > > > > > > > > > >>>>>>> Nevertheless, it happen.
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>> I think we can avoid it with the Ignite3 and
> > > successors
> > > > > if
> > > > > > we
> > > > > > > > > will
> > > > > > > > > > >>>> move
> > > > > > > > > > >>>>>>> step by step without keeping backward
> compatibility
> > > > > > > > > > >>>>>>> With the step by step approach, we can focus on
> each
> > > > > > > component
> > > > > > > > > > >>>>>> separately.
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>>> What new features are we planning to implement
> for
> > > > > Ignite
> > > > > > > 2.x?
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>> We have many features that have to evolve.
> > > > > > > > > > >>>>>>> Snapshots, rebalance, tooling, tracing, zookeeper
> > > > > support,
> > > > > > > etc.
> > > > > > > > > > >>>>>>> We have bugs and issues that can be fixed in 2.x
> > > > without
> > > > > > > > breaking
> > > > > > > > > > >>>>>> backward
> > > > > > > > > > >>>>>>> compatibility.
> > > > > > > > > > >>>>>>> We have many users who are happy with the 2.x
> with
> > > all
> > > > > it’s
> > > > > > > > > issues.
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>>> 2 нояб. 2020 г., в 14:09, Anton Vinogradov <
> > > > > av@apache.org
> > > > > > >
> > > > > > > > > > >>>>> написал(а):
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>> Alexey,
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>> Do we have any estimates of how fast we'll be
> able
> > > to
> > > > > gain
> > > > > > > > > > >>>>>>> production-ready
> > > > > > > > > > >>>>>>>> AI 3.0 in case of a "new repo" choice?
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>> On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk
> <
> > > > > > > > > > >>>>>>> alexey.goncharuk@gmail.com>
> > > > > > > > > > >>>>>>>> wrote:
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>>> Nikolay,
> > > > > > > > > > >>>>>>>>>
> > > > > > > > > > >>>>>>>>> What new features are we planning to implement
> for
> > > > > Ignite
> > > > > > > > 2.x?
> > > > > > > > > I
> > > > > > > > > > >>>>> think
> > > > > > > > > > >>>>>>> once
> > > > > > > > > > >>>>>>>>> we commence working on Ignite 3.0, we should
> > > > gradually
> > > > > > > cease
> > > > > > > > > the
> > > > > > > > > > >>>>>>> activity
> > > > > > > > > > >>>>>>>>> on Ignite 2.x to mere bugfixes because such
> > > parallel
> > > > > > > > > development
> > > > > > > > > > >>>>> will
> > > > > > > > > > >>>>>> be
> > > > > > > > > > >>>>>>>>> overwhelming regardless of how we choose to
> > > proceed.
> > > > > > > > > > >>>>>>>>>
> > > > > > > > > > >>>>>>>>> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <
> > > > > > > > > > nizhikov@apache.org
> > > > > > > > > > >>>>> :
> > > > > > > > > > >>>>>>>>>
> > > > > > > > > > >>>>>>>>>> To be clear:
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>> I would suggest creating a new repository for
> > > > Ignite
> > > > > > 3.0
> > > > > > > > > > >>>>> (perhaps, a
> > > > > > > > > > >>>>>>>>> new
> > > > > > > > > > >>>>>>>>>> clean branch, but a new repo looks nicer to
> me)
> > > and
> > > > a
> > > > > > new
> > > > > > > > > Ignite
> > > > > > > > > > >>>>> 3.0
> > > > > > > > > > >>>>>>>>>> TeamCity project.
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>> +1 for new Team City project.
> > > > > > > > > > >>>>>>>>>> +1 for new branch for Ignite3.
> > > > > > > > > > >>>>>>>>>> -1 for new repo.
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <
> > > > > > > > > > >>>> NIzhikov.dev@gmail.com
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>>>>>> написал(а):
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>> Hello, Alexey.
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>> I think it will hurt our project more than
> help.
> > > > > > > > > > >>>>>>>>>>> Developing new features for 2 separate
> branches
> > > > with
> > > > > > the
> > > > > > > > > > >>>> different
> > > > > > > > > > >>>>>>> APIs
> > > > > > > > > > >>>>>>>>>> and internal structure is overwhelming
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>> Maybe we should relax a bit requirements for
> > > > Ignite3?
> > > > > > > > > > >>>>>>>>>>> Maybe we should move step by step and make
> > > Ignite3
> > > > > with
> > > > > > > new
> > > > > > > > > > >>>>>>>>>> configuration than Ignite4 with new
> transactions,
> > > > etc?
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <
> > > > > > > > > > >>>>>>>>> alexey.goncharuk@gmail.com>
> > > > > > > > > > >>>>>>>>>> написал(а):
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> Igniters,
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> I wanted to pitch a rather radical idea
> > > regarding
> > > > > the
> > > > > > > > Ignite
> > > > > > > > > > >>>> 3.0
> > > > > > > > > > >>>>>>>>>>>> development which has occurred to me some
> time
> > > > ago.
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> We already have several IEPs targeted to
> Ignite
> > > > 3.0
> > > > > > > which
> > > > > > > > > > imply
> > > > > > > > > > >>>>>> major
> > > > > > > > > > >>>>>>>>>>>> changes to the codebase (the change in
> > > replication
> > > > > > > > protocol
> > > > > > > > > > and
> > > > > > > > > > >>>>>> thus
> > > > > > > > > > >>>>>>>>>>>> transactions, change in binary format,
> updated
> > > > > > > > metastorage,
> > > > > > > > > > >>>> etc).
> > > > > > > > > > >>>>>> We
> > > > > > > > > > >>>>>>>>>> also
> > > > > > > > > > >>>>>>>>>>>> planned significant changes in public APIs:
> > > > > > > configuration
> > > > > > > > > > >>>> format
> > > > > > > > > > >>>>>>>>> change,
> > > > > > > > > > >>>>>>>>>>>> improvements in cache APIs, SQL APIs,
> > > transaction
> > > > > mode
> > > > > > > > > rework.
> > > > > > > > > > >>>>> The
> > > > > > > > > > >>>>>>>>>> wishlist
> > > > > > > > > > >>>>>>>>>>>> of changes for Ignite 3.0 is huge.
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> So, I was wondering whether it makes sense
> to
> > > try
> > > > to
> > > > > > > > change
> > > > > > > > > > the
> > > > > > > > > > >>>>> old
> > > > > > > > > > >>>>>>>>>>>> codebase, or start with a new baseline and
> move
> > > > old
> > > > > > > pieces
> > > > > > > > > of
> > > > > > > > > > >>>>> code
> > > > > > > > > > >>>>>>>>> that
> > > > > > > > > > >>>>>>>>>> do
> > > > > > > > > > >>>>>>>>>>>> not require significant rework. Personally,
> I
> > > > would
> > > > > go
> > > > > > > > with
> > > > > > > > > > the
> > > > > > > > > > >>>>>>> second
> > > > > > > > > > >>>>>>>>>>>> option for the following reasons:
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> - We have a chance to shift the development
> > > > paradigm
> > > > > > in
> > > > > > > > the
> > > > > > > > > > >>>>> project
> > > > > > > > > > >>>>>>>>> and
> > > > > > > > > > >>>>>>>>>>>> introduce the practice of true unit-tests.
> In
> > > the
> > > > > new
> > > > > > > > > baseline
> > > > > > > > > > >>>> at
> > > > > > > > > > >>>>>> the
> > > > > > > > > > >>>>>>>>>>>> beginning there will be no ability to run an
> > > > > > end-to-end
> > > > > > > > > > >>>> scenario,
> > > > > > > > > > >>>>>>>>> thus
> > > > > > > > > > >>>>>>>>>> we
> > > > > > > > > > >>>>>>>>>>>> will be forced to write unit-tests. So far,
> such
> > > > > > > practice
> > > > > > > > > was
> > > > > > > > > > >>>>> hard
> > > > > > > > > > >>>>>> to
> > > > > > > > > > >>>>>>>>>>>> implement because of tight coupling between
> > > Ignite
> > > > > > > > > components
> > > > > > > > > > >>>> and
> > > > > > > > > > >>>>>>>>>> inability
> > > > > > > > > > >>>>>>>>>>>> to instantiate components without an
> instance of
> > > > > > > > > > KernalContext.
> > > > > > > > > > >>>>> For
> > > > > > > > > > >>>>>>>>>>>> example, we should be able to thoroughly
> test
> > > > > internal
> > > > > > > > > > >>>>> primitives,
> > > > > > > > > > >>>>>>>>>> such as
> > > > > > > > > > >>>>>>>>>>>> replication protocol (without actual
> > > > communication),
> > > > > > > > > > >>>> distributed
> > > > > > > > > > >>>>>>>>>>>> metastorage contracts, persistence layer,
> etc.
> > > > > > > > > > >>>>>>>>>>>> - We will significantly reduce the
> development
> > > > cycle
> > > > > > in
> > > > > > > > the
> > > > > > > > > > >>>>>> beginning
> > > > > > > > > > >>>>>>>>>>>> (right now the RunAll takes two hours of
> > > > > astronomical
> > > > > > > time
> > > > > > > > > > with
> > > > > > > > > > >>>>>> empty
> > > > > > > > > > >>>>>>>>>> TC;
> > > > > > > > > > >>>>>>>>>>>> in the new approach developer will be able
> to
> > > run
> > > > > ALL
> > > > > > > > tests
> > > > > > > > > > >>>>> locally
> > > > > > > > > > >>>>>>>>> in
> > > > > > > > > > >>>>>>>>>> a
> > > > > > > > > > >>>>>>>>>>>> matter of minutes)
> > > > > > > > > > >>>>>>>>>>>> - We can get rid of TC bot and enforce
> green TC
> > > by
> > > > > > > > > integrating
> > > > > > > > > > >>>> TC
> > > > > > > > > > >>>>>>>>> build
> > > > > > > > > > >>>>>>>>>>>> results with GitHub PRs (the same way
> Travis is
> > > > > > > currently
> > > > > > > > > > >>>>>> integrated
> > > > > > > > > > >>>>>>>>>> to PR
> > > > > > > > > > >>>>>>>>>>>> check). We should restrict PR merge without
> a TC
> > > > > check
> > > > > > > > > > >>>>>>>>>>>> - We will still have to re-write all tests,
> but
> > > > only
> > > > > > > once.
> > > > > > > > > If
> > > > > > > > > > >>>> we
> > > > > > > > > > >>>>>> try
> > > > > > > > > > >>>>>>>>> to
> > > > > > > > > > >>>>>>>>>>>> modify the old codebase, we would need to
> modify
> > > > all
> > > > > > the
> > > > > > > > > tests
> > > > > > > > > > >>>>> for
> > > > > > > > > > >>>>>>>>>> every
> > > > > > > > > > >>>>>>>>>>>> major change (public API change,
> configuration
> > > > > change)
> > > > > > > > > > >>>>>>>>>>>> - We will have fewer conflicts when working
> > > > > together.
> > > > > > > For
> > > > > > > > > > >>>>> example,
> > > > > > > > > > >>>>>> I
> > > > > > > > > > >>>>>>>>>>>> cannot imagine how one would merge two
> changes
> > > of
> > > > > > > getting
> > > > > > > > > rid
> > > > > > > > > > >>>> of
> > > > > > > > > > >>>>>>>>>>>> IgniteFuture and changes in replication
> > > protocol,
> > > > > for
> > > > > > > > > example
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> Technically, I would suggest creating a new
> > > > > repository
> > > > > > > for
> > > > > > > > > > >>>> Ignite
> > > > > > > > > > >>>>>> 3.0
> > > > > > > > > > >>>>>>>>>>>> (perhaps, a new clean branch, but a new repo
> > > looks
> > > > > > nicer
> > > > > > > > to
> > > > > > > > > > me)
> > > > > > > > > > >>>>>> and a
> > > > > > > > > > >>>>>>>>>> new
> > > > > > > > > > >>>>>>>>>>>> Ignite 3.0 TeamCity project.
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> While it may seem quite radical, I do
> believe
> > > that
> > > > > > this
> > > > > > > > > > >>>> approach
> > > > > > > > > > >>>>>> will
> > > > > > > > > > >>>>>>>>>> give
> > > > > > > > > > >>>>>>>>>>>> us more benefits than trying to make such
> major
> > > > > > changes
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > >>>>>>>>> existing
> > > > > > > > > > >>>>>>>>>>>> codebase. If needed, let's schedule a
> discord
> > > chat
> > > > > > like
> > > > > > > > > before
> > > > > > > > > > >>>> to
> > > > > > > > > > >>>>>>>>>> discuss
> > > > > > > > > > >>>>>>>>>>>> this.
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> WDYT?
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> --
> > > > > > > > > > >>>> Best regards,
> > > > > > > > > > >>>> Andrey V. Mashenkov
> > > > > > > > > > >>>>
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> >
> > Best regards,
> > Alexei Scherbakov
>
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Maxim Muzafarov <mm...@apache.org>.
Sergey,
Your summary makes sense to me.
However, how we come up from *Development transparency* to *create a
separate public repository dedicated for 3.0*? For me *development
transparency* is about making changes in the master branch. These
changes will definitely be seen by all the Ignite developers.
A dedicated public repository is technically public and visible for
everyone, but it allows development without IEPs, without public
discussion (since all the code changes are not related to the master
branch) it also allows a large number of assumptions and deviations
(like code-style violations). It also not about *development
transparency* since developers which are working on 3.0 is only a
subset of all Ignite developers which may continue working on 2.x. For
me, this would be a huge step backwards.
Ignite veterans should remember how long the branch stabilization took
for the 2.x version with the PDS.
I think each breaking change should be passed through the master branch.
On Tue, 10 Nov 2020 at 22:18, Alexei Scherbakov
<al...@gmail.com> wrote:
>
> Makes sense to me.
>
> вт, 10 нояб. 2020 г. в 18:47, Sergey Chugunov <se...@gmail.com>:
>
> > Igniters,
> >
> > I thought over Friday meeting ideas and concerns and summarized them in
> > these three points:
> >
> >
> > 1. *Components design unification approach.* New proposed components
> > will be developed by different contributors, but they need to be unified
> > and should integrate with each other easily. To ensure that I suggest
> > calling an architecture group that will create design guidelines for all
> > components and high-level overview of overall architecture. How code is
> > split into components, what are component boundaries, how component
> > lifecycle works and what are its interfaces - all these and other
> > questions
> > should be covered.
> >
> > 2. *Scope management.* Apache 3.0 should be implemented within a
> > reasonable time, so we need some procedure to decide whether a
> > particular
> > feature should be dropped from the scope of 3.0 and postponed to 3.1
> > release. To do so I suggest to range all features by two parameters:
> > criticality for 3.0 and amount of breaking changes. 3.0 scope should
> > include features of high criticality AND features with a big amount of
> > breaking changes. All other features can be made optional.
> >
> > 3. *Development transparency.* Development of all components should be
> > made as transparent for everyone as possible. Any contributor should be
> > able to look over any component at any stage of development. To achieve
> > this I suggest to create a separate public repository dedicated for 3.0
> > development. It will make the code available for everyone but when
> > development of 3.0 is done we won't loose any stars of our current
> > repository as we merge dev repo into main one and drop dev.
> >
> > Do these ideas make sense to you? Are there any concerns not covered by
> > these suggestions?
> >
> > On Fri, Nov 6, 2020 at 7:36 PM Kseniya Romanova <romanova.ks.spb@gmail.com
> > >
> > wrote:
> >
> > > Here are the slides from Alexey Goncharuk. Let's think this over and
> > > continue on Monday:
> > >
> > >
> > https://go.gridgain.com/rs/491-TWR-806/images/Ignite_3_Plans_and_development_process.pdf
> > >
> > > чт, 5 нояб. 2020 г. в 11:13, Anton Vinogradov <av...@apache.org>:
> > >
> > > > Folks,
> > > >
> > > > Should we perform cleanup work before (r)evolutional changes?
> > > > My huge proposal is to get rid of things which we don't need anyway
> > > > - local caches,
> > > > - strange tx modes,
> > > > - code overcomplexity because of RollingUpgrade feature never attended
> > at
> > > > AI,
> > > > - etc,
> > > > before choosing the way.
> > > >
> > > > On Tue, Nov 3, 2020 at 3:31 PM Valentin Kulichenko <
> > > > valentin.kulichenko@gmail.com> wrote:
> > > >
> > > > > Ksenia, thanks for scheduling this on such short notice!
> > > > >
> > > > > As for the original topic, I do support Alexey's idea. We're not
> > going
> > > to
> > > > > rewrite anything from scratch, as most of the components are going to
> > > be
> > > > > moved as-is or with minimal modifications. However, the changes that
> > > are
> > > > > proposed imply serious rework of the core parts of the code, which
> > are
> > > > not
> > > > > properly decoupled from each other and from other parts. This makes
> > the
> > > > > incremental approach borderline impossible. Developing in a new repo,
> > > > > however, addresses this concern. As a bonus, we can also refactor the
> > > > code,
> > > > > introduce better decoupling, get rid of kernel context, and develop
> > > unit
> > > > > tests (finally!).
> > > > >
> > > > > Basically, this proposal only affects the *process*, not the set of
> > > > changes
> > > > > we had discussed before. Ignite 3.0 is our unique chance to make
> > things
> > > > > right.
> > > > >
> > > > > -Val
> > > > >
> > > > > On Tue, Nov 3, 2020 at 3:06 AM Kseniya Romanova <
> > > > romanova.ks.spb@gmail.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Pavel, all the interesting points will be anyway published here in
> > > > > English
> > > > > > (as the principal "if it's not on devlist it doesn't happened" is
> > > still
> > > > > > relevant). This is just a quick call for a group of developers.
> > Later
> > > > we
> > > > > > can do a separate presentation of idea and discussion in English as
> > > we
> > > > > did
> > > > > > for the Ignite 3.0 draft of changes.
> > > > > >
> > > > > > вт, 3 нояб. 2020 г. в 13:52, Pavel Tupitsyn <ptupitsyn@apache.org
> > >:
> > > > > >
> > > > > > > Kseniya,
> > > > > > >
> > > > > > > Thanks for scheduling this call.
> > > > > > > Do you think we can switch to English if non-Russian speaking
> > > > community
> > > > > > > members decide to join?
> > > > > > >
> > > > > > > On Tue, Nov 3, 2020 at 1:32 PM Kseniya Romanova <
> > > > > > romanova.ks.spb@gmail.com
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Let's do this community discussion open. Here's the link on
> > zoom
> > > > call
> > > > > > in
> > > > > > > > Russian for Friday 6 PM:
> > > > > > > >
> > > > https://www.meetup.com/Moscow-Apache-Ignite-Meetup/events/274360378/
> > > > > > > >
> > > > > > > > вт, 3 нояб. 2020 г. в 12:49, Nikolay Izhikov <
> > > nizhikov@apache.org
> > > > >:
> > > > > > > >
> > > > > > > > > Time works for me.
> > > > > > > > >
> > > > > > > > > > 3 нояб. 2020 г., в 12:40, Alexey Goncharuk <
> > > > > > > alexey.goncharuk@gmail.com
> > > > > > > > >
> > > > > > > > > написал(а):
> > > > > > > > > >
> > > > > > > > > > Nikolay,
> > > > > > > > > >
> > > > > > > > > > I am up for the call. I will try to explain my reasoning in
> > > > > greater
> > > > > > > > > detail
> > > > > > > > > > and will be glad to hear the concerns. Will this Friday,
> > Nov
> > > > 6th,
> > > > > > > work?
> > > > > > > > > >
> > > > > > > > > > вт, 3 нояб. 2020 г. в 10:09, Nikolay Izhikov <
> > > > > nizhikov@apache.org
> > > > > > >:
> > > > > > > > > >
> > > > > > > > > >> Igniters, should we have a call for this topic?
> > > > > > > > > >>
> > > > > > > > > >>> 2 нояб. 2020 г., в 18:53, Pavel Tupitsyn <
> > > > ptupitsyn@apache.org
> > > > > >
> > > > > > > > > >> написал(а):
> > > > > > > > > >>>
> > > > > > > > > >>>> not intend to rewrite everything from scratch
> > > > > > > > > >>>
> > > > > > > > > >>>> Every single test from Ignite 2.x should be moved to
> > > Ignite
> > > > 3
> > > > > > > > > >>>> regardless of how we choose to proceed.
> > > > > > > > > >>>
> > > > > > > > > >>> Alexey, thank you for the explanation, this addresses all
> > > of
> > > > my
> > > > > > > > > concerns.
> > > > > > > > > >>>
> > > > > > > > > >>>
> > > > > > > > > >>>
> > > > > > > > > >>>
> > > > > > > > > >>>
> > > > > > > > > >>> On Mon, Nov 2, 2020 at 6:43 PM Andrey Mashenkov <
> > > > > > > > > >> andrey.mashenkov@gmail.com>
> > > > > > > > > >>> wrote:
> > > > > > > > > >>>
> > > > > > > > > >>>> Hi, Igniters.
> > > > > > > > > >>>>
> > > > > > > > > >>>> * AFAIU, we need a new repo if we want to apply
> > different
> > > > > > > > restrictions
> > > > > > > > > >> to
> > > > > > > > > >>>> pull requests,
> > > > > > > > > >>>> otherwise I see no difference for myself.
> > > > > > > > > >>>> E.g. make static analysis (do we have?), compile,
> > styles,
> > > > and
> > > > > > > > javadoc
> > > > > > > > > >>>> checks mandatory.
> > > > > > > > > >>>>
> > > > > > > > > >>>> I think that relaxed requirements here will lead to bad
> > > > > product
> > > > > > > > > quality.
> > > > > > > > > >>>>
> > > > > > > > > >>>> * Agree with Pavel, we should 'keep' integrations tests
> > > > > somehow.
> > > > > > > > > >>>> During active development tests will be broken most of
> > > time,
> > > > > so,
> > > > > > > > > >>>> I'd port them e.g. suite-by-suite once we will have a
> > > stable
> > > > > and
> > > > > > > > > >> featured
> > > > > > > > > >>>> environment to run them and of course make test's code
> > > clear
> > > > > and
> > > > > > > > avoid
> > > > > > > > > >>>> bad/non-relevant ones.
> > > > > > > > > >>>>
> > > > > > > > > >>>> * I like bottom-up approach.
> > > > > > > > > >>>> With it we could make a better framework. I mean clear
> > > > > component
> > > > > > > > > >> lifecycle,
> > > > > > > > > >>>> component wiring mechanics, general methods to approach
> > > core
> > > > > > > > > components
> > > > > > > > > >>>> such as exchange/communication
> > > > > > > > > >>>> to avoid code mess like we have in ExchangeFuture with
> > all
> > > > > these
> > > > > > > > > custom
> > > > > > > > > >>>> callbacks for each component, interfaces like
> > > > > > > > > >>>> PartitionsExchangeAware, IgniteChangeGlobalStateSupport
> > > and
> > > > > > > > > >>>> a pack of
> > > > > > > > > >>
> > > > start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected
> > > > > > > > > >>>> and so on in various unexpected places.
> > > > > > > > > >>>> Hope, we will be able to port most of the good code to
> > the
> > > > new
> > > > > > > > > framework
> > > > > > > > > >>>> version.
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>> On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk <
> > > > > > > > > >>>> alexey.goncharuk@gmail.com>
> > > > > > > > > >>>> wrote:
> > > > > > > > > >>>>
> > > > > > > > > >>>>> Nikolay, Pavel,
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> Thanks for the feedback! First of all, I wanted to
> > stress
> > > > > that
> > > > > > I
> > > > > > > do
> > > > > > > > > not
> > > > > > > > > >>>>> intend to rewrite everything from scratch (I never used
> > > > this
> > > > > > > > phrase).
> > > > > > > > > >>>> There
> > > > > > > > > >>>>> are significant parts of code that would be moved with
> > > > > minimal
> > > > > > > > > >>>>> modifications. Second, I never said that we will get
> > rid
> > > of
> > > > > the
> > > > > > > old
> > > > > > > > > >> tests
> > > > > > > > > >>>>> codebase. Every single test from Ignite 2.x should be
> > > moved
> > > > > to
> > > > > > > > > Ignite 3
> > > > > > > > > >>>>> regardless of how we choose to proceed.
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> My point is that for some parts of the code a clean
> > > > bottom-up
> > > > > > > > > >>>>> implementation will be cheaper in many ways. Let me
> > give
> > > > you
> > > > > a
> > > > > > > few
> > > > > > > > > >>>> concrete
> > > > > > > > > >>>>> examples:
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> - I think no one can object that we need a cleanly
> > > > separated
> > > > > > > > > >>>> persistence
> > > > > > > > > >>>>> layer for Ignite. There is a very raw draft IEP for
> > this
> > > > > > > already.
> > > > > > > > On
> > > > > > > > > >>>> the
> > > > > > > > > >>>>> other hand, I think we also can agree that we need a
> > > > > > split-brain
> > > > > > > > > >>>>> resistant
> > > > > > > > > >>>>> replication protocol for caches. There is also an IEP
> > > for
> > > > > > this.
> > > > > > > > > >>>> Neither
> > > > > > > > > >>>>> of
> > > > > > > > > >>>>> the changes is a good fit for 2.x because they are
> > > likely
> > > > to
> > > > > > > > > >> introduce
> > > > > > > > > >>>>> breaking changes in the persistence layer,
> > configuration
> > > > and
> > > > > > > > > >> behavior.
> > > > > > > > > >>>>> Additionally, these components are now tightly
> > coupled,
> > > so
> > > > > > there
> > > > > > > > is
> > > > > > > > > >> no
> > > > > > > > > >>>>> way
> > > > > > > > > >>>>> these two changes can be implemented in parallel and
> > > then
> > > > > > merged
> > > > > > > > > >>>>> together
> > > > > > > > > >>>>> easily. So what we will end up with is having to
> > > implement
> > > > > > these
> > > > > > > > > >>>> changes
> > > > > > > > > >>>>> sequentially, fixing all existing tests twice, and
> > > > > essentially
> > > > > > > > > >>>> throwing
> > > > > > > > > >>>>> away half of the work done because the other part of
> > the
> > > > > > change
> > > > > > > is
> > > > > > > > > >>>>> re-implemented
> > > > > > > > > >>>>> - Similar example goes with getting rid of
> > > > > > IgniteInternalFuture
> > > > > > > > and
> > > > > > > > > >>>>> replacing it with CompletableFuture, and any other
> > > change
> > > > > that
> > > > > > > > > >> touches
> > > > > > > > > >>>>> the
> > > > > > > > > >>>>> asynchronous part of the code.
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> Third, I do not see how this choice affects the UX of
> > > > Ignite.
> > > > > > The
> > > > > > > > end
> > > > > > > > > >>>> user
> > > > > > > > > >>>>> experience must be fixed in the IEP regardless of the
> > > > > > development
> > > > > > > > > >> process
> > > > > > > > > >>>>> and the fact that we have gaps in this area in Ignite
> > 2.x
> > > > > just
> > > > > > > > > confirms
> > > > > > > > > >>>>> that.
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> Pavel, agree that a repo/branch is a technicality, I
> > > guess
> > > > if
> > > > > > > > > >>>> reformulate,
> > > > > > > > > >>>>> my point is that we might agree to have a single
> > > > development
> > > > > > > master
> > > > > > > > > >>>> branch
> > > > > > > > > >>>>> with 'disassembled' end-to-end functionality for some
> > > > period
> > > > > of
> > > > > > > > time
> > > > > > > > > to
> > > > > > > > > >>>>> speed up development, and re-assemble the core features
> > > > after
> > > > > > > > having
> > > > > > > > > >>>>> submodules tested independently.
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> Nikolay,
> > > > > > > > > >>>>>> We have many features that have to evolve.
> > > > > > > > > >>>>>> Snapshots, rebalance, tooling, tracing, zookeeper
> > > support,
> > > > > > etc.
> > > > > > > > > >>>>> This is not very specific. In the end, resources are
> > > > limited
> > > > > > and
> > > > > > > we
> > > > > > > > > >> will
> > > > > > > > > >>>>> not be able to drive both tracks simultaneously,
> > > especially
> > > > > > > after a
> > > > > > > > > >>>> couple
> > > > > > > > > >>>>> of features having been implemented for Ignite 3.0. If
> > > > there
> > > > > > are
> > > > > > > > > indeed
> > > > > > > > > >>>>> some major changes that we want to do in Ignite 2.x
> > > instead
> > > > > of
> > > > > > > > > putting
> > > > > > > > > >>>>> effort into 3.0 - let's discuss them. I am just not
> > aware
> > > > of
> > > > > > any,
> > > > > > > > > >> that's
> > > > > > > > > >>>>> why I am eager to move forward with Ignite 3.0.
> > > > > > > > > >>>>>
> > > > > > > > > >>>>>> We have bugs and issues that can be fixed in 2.x
> > without
> > > > > > > breaking
> > > > > > > > > >>>> backward
> > > > > > > > > >>>>> compatibility.
> > > > > > > > > >>>>>> We have many users who are happy with the 2.x with all
> > > > it’s
> > > > > > > > issues.
> > > > > > > > > >>>>> These changes will be covered by end-to-end tests and
> > > > > migrated
> > > > > > to
> > > > > > > > > >> Ignite
> > > > > > > > > >>>>> 3.0, so I see no issues here.
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> Finally, Anton & Nikolay
> > > > > > > > > >>>>> I do not have an estimate for this simply because the
> > > > > activity
> > > > > > is
> > > > > > > > > >>>>> community-driven and it depends on the number of people
> > > > > willing
> > > > > > > to
> > > > > > > > > >>>>> contribute. With the current pace, I would hope to have
> > > an
> > > > RC
> > > > > > of
> > > > > > > > > Ignite
> > > > > > > > > >>>> 3.0
> > > > > > > > > >>>>> to be ready by the end of 2021. My gut feeling is that
> > by
> > > > > > moving
> > > > > > > > with
> > > > > > > > > >>>>> incremental changes, we will not be able to implement
> > > even
> > > > > half
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > >>>>> wishlist by that time.
> > > > > > > > > >>>>> I doubt that releasing several major releases with
> > > breaking
> > > > > > > changes
> > > > > > > > > >> will
> > > > > > > > > >>>>> make Ignite users happy either because each upgrade
> > will
> > > > cost
> > > > > > > > Ignite
> > > > > > > > > >>>> users
> > > > > > > > > >>>>> money, so the fewer major versions we release, the
> > > better.
> > > > > Thus
> > > > > > > my
> > > > > > > > > wish
> > > > > > > > > >>>> to
> > > > > > > > > >>>>> include all breaking changes in one release.
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> I'll be now quiet for a while, let's see what other
> > > > community
> > > > > > > > members
> > > > > > > > > >>>>> think.
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <
> > > > > > > ptupitsyn@apache.org
> > > > > > > > >:
> > > > > > > > > >>>>>
> > > > > > > > > >>>>>> 1. Rewriting from scratch is never a good idea.
> > > > > > > > > >>>>>> We don't want to follow the path of Netscape and lose
> > > all
> > > > > our
> > > > > > > > users
> > > > > > > > > >>>>>> by the time we have a working 3.0 [1]
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> 2. Not sure about new repo - seems like some pain and
> > no
> > > > > gain,
> > > > > > > > > what's
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>> problem with a branch?
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> 3. We should keep existing integration tests when
> > > > possible.
> > > > > > > > > >>>>>> We have accumulated a lot of edge case knowledge over
> > > the
> > > > > > years,
> > > > > > > > > >>>>>> it is not a good idea to send all of that down the
> > > drain.
> > > > > > > > > >>>>>> Yes, integration tests are slow, but they are the most
> > > > > > valuable.
> > > > > > > > > >>>>>> I think we can move more stuff into nightly runs and
> > > have
> > > > a
> > > > > > fast
> > > > > > > > and
> > > > > > > > > >>>>> modern
> > > > > > > > > >>>>>> basic suite.
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> Alexey, you are much more familiar with the Ignite
> > core
> > > > > > codebase
> > > > > > > > > than
> > > > > > > > > >>>>> most
> > > > > > > > > >>>>>> of us,
> > > > > > > > > >>>>>> can you please explain in more detail which particular
> > > > > > feature,
> > > > > > > in
> > > > > > > > > >> your
> > > > > > > > > >>>>>> opinion,
> > > > > > > > > >>>>>> mandates this "start from scratch" approach?
> > > > > > > > > >>>>>> Is it really not possible at all to follow a less
> > > radical
> > > > > way?
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> [1]
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>
> > > > > > > > > >>>>
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <
> > > > > > > > nizhikov@apache.org
> > > > > > > > > >
> > > > > > > > > >>>>>> wrote:
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>>> Hello, Alexey.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> I think that «rewriting from scratch» approach has a
> > > high
> > > > > > risk
> > > > > > > to
> > > > > > > > > >>>> make
> > > > > > > > > >>>>>> new
> > > > > > > > > >>>>>>> features unusable.
> > > > > > > > > >>>>>>> At the time Ignite2 was started no-one wants to do
> > bad
> > > UX
> > > > > or
> > > > > > > bad
> > > > > > > > > >>>>>> features.
> > > > > > > > > >>>>>>> Nevertheless, it happen.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> I think we can avoid it with the Ignite3 and
> > successors
> > > > if
> > > > > we
> > > > > > > > will
> > > > > > > > > >>>> move
> > > > > > > > > >>>>>>> step by step without keeping backward compatibility
> > > > > > > > > >>>>>>> With the step by step approach, we can focus on each
> > > > > > component
> > > > > > > > > >>>>>> separately.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>>> What new features are we planning to implement for
> > > > Ignite
> > > > > > 2.x?
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> We have many features that have to evolve.
> > > > > > > > > >>>>>>> Snapshots, rebalance, tooling, tracing, zookeeper
> > > > support,
> > > > > > etc.
> > > > > > > > > >>>>>>> We have bugs and issues that can be fixed in 2.x
> > > without
> > > > > > > breaking
> > > > > > > > > >>>>>> backward
> > > > > > > > > >>>>>>> compatibility.
> > > > > > > > > >>>>>>> We have many users who are happy with the 2.x with
> > all
> > > > it’s
> > > > > > > > issues.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>>> 2 нояб. 2020 г., в 14:09, Anton Vinogradov <
> > > > av@apache.org
> > > > > >
> > > > > > > > > >>>>> написал(а):
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> Alexey,
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> Do we have any estimates of how fast we'll be able
> > to
> > > > gain
> > > > > > > > > >>>>>>> production-ready
> > > > > > > > > >>>>>>>> AI 3.0 in case of a "new repo" choice?
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk <
> > > > > > > > > >>>>>>> alexey.goncharuk@gmail.com>
> > > > > > > > > >>>>>>>> wrote:
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>>> Nikolay,
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> What new features are we planning to implement for
> > > > Ignite
> > > > > > > 2.x?
> > > > > > > > I
> > > > > > > > > >>>>> think
> > > > > > > > > >>>>>>> once
> > > > > > > > > >>>>>>>>> we commence working on Ignite 3.0, we should
> > > gradually
> > > > > > cease
> > > > > > > > the
> > > > > > > > > >>>>>>> activity
> > > > > > > > > >>>>>>>>> on Ignite 2.x to mere bugfixes because such
> > parallel
> > > > > > > > development
> > > > > > > > > >>>>> will
> > > > > > > > > >>>>>> be
> > > > > > > > > >>>>>>>>> overwhelming regardless of how we choose to
> > proceed.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <
> > > > > > > > > nizhikov@apache.org
> > > > > > > > > >>>>> :
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>>> To be clear:
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> I would suggest creating a new repository for
> > > Ignite
> > > > > 3.0
> > > > > > > > > >>>>> (perhaps, a
> > > > > > > > > >>>>>>>>> new
> > > > > > > > > >>>>>>>>>> clean branch, but a new repo looks nicer to me)
> > and
> > > a
> > > > > new
> > > > > > > > Ignite
> > > > > > > > > >>>>> 3.0
> > > > > > > > > >>>>>>>>>> TeamCity project.
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>> +1 for new Team City project.
> > > > > > > > > >>>>>>>>>> +1 for new branch for Ignite3.
> > > > > > > > > >>>>>>>>>> -1 for new repo.
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <
> > > > > > > > > >>>> NIzhikov.dev@gmail.com
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>>>>>> написал(а):
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> Hello, Alexey.
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> I think it will hurt our project more than help.
> > > > > > > > > >>>>>>>>>>> Developing new features for 2 separate branches
> > > with
> > > > > the
> > > > > > > > > >>>> different
> > > > > > > > > >>>>>>> APIs
> > > > > > > > > >>>>>>>>>> and internal structure is overwhelming
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> Maybe we should relax a bit requirements for
> > > Ignite3?
> > > > > > > > > >>>>>>>>>>> Maybe we should move step by step and make
> > Ignite3
> > > > with
> > > > > > new
> > > > > > > > > >>>>>>>>>> configuration than Ignite4 with new transactions,
> > > etc?
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <
> > > > > > > > > >>>>>>>>> alexey.goncharuk@gmail.com>
> > > > > > > > > >>>>>>>>>> написал(а):
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> Igniters,
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> I wanted to pitch a rather radical idea
> > regarding
> > > > the
> > > > > > > Ignite
> > > > > > > > > >>>> 3.0
> > > > > > > > > >>>>>>>>>>>> development which has occurred to me some time
> > > ago.
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> We already have several IEPs targeted to Ignite
> > > 3.0
> > > > > > which
> > > > > > > > > imply
> > > > > > > > > >>>>>> major
> > > > > > > > > >>>>>>>>>>>> changes to the codebase (the change in
> > replication
> > > > > > > protocol
> > > > > > > > > and
> > > > > > > > > >>>>>> thus
> > > > > > > > > >>>>>>>>>>>> transactions, change in binary format, updated
> > > > > > > metastorage,
> > > > > > > > > >>>> etc).
> > > > > > > > > >>>>>> We
> > > > > > > > > >>>>>>>>>> also
> > > > > > > > > >>>>>>>>>>>> planned significant changes in public APIs:
> > > > > > configuration
> > > > > > > > > >>>> format
> > > > > > > > > >>>>>>>>> change,
> > > > > > > > > >>>>>>>>>>>> improvements in cache APIs, SQL APIs,
> > transaction
> > > > mode
> > > > > > > > rework.
> > > > > > > > > >>>>> The
> > > > > > > > > >>>>>>>>>> wishlist
> > > > > > > > > >>>>>>>>>>>> of changes for Ignite 3.0 is huge.
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> So, I was wondering whether it makes sense to
> > try
> > > to
> > > > > > > change
> > > > > > > > > the
> > > > > > > > > >>>>> old
> > > > > > > > > >>>>>>>>>>>> codebase, or start with a new baseline and move
> > > old
> > > > > > pieces
> > > > > > > > of
> > > > > > > > > >>>>> code
> > > > > > > > > >>>>>>>>> that
> > > > > > > > > >>>>>>>>>> do
> > > > > > > > > >>>>>>>>>>>> not require significant rework. Personally, I
> > > would
> > > > go
> > > > > > > with
> > > > > > > > > the
> > > > > > > > > >>>>>>> second
> > > > > > > > > >>>>>>>>>>>> option for the following reasons:
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> - We have a chance to shift the development
> > > paradigm
> > > > > in
> > > > > > > the
> > > > > > > > > >>>>> project
> > > > > > > > > >>>>>>>>> and
> > > > > > > > > >>>>>>>>>>>> introduce the practice of true unit-tests. In
> > the
> > > > new
> > > > > > > > baseline
> > > > > > > > > >>>> at
> > > > > > > > > >>>>>> the
> > > > > > > > > >>>>>>>>>>>> beginning there will be no ability to run an
> > > > > end-to-end
> > > > > > > > > >>>> scenario,
> > > > > > > > > >>>>>>>>> thus
> > > > > > > > > >>>>>>>>>> we
> > > > > > > > > >>>>>>>>>>>> will be forced to write unit-tests. So far, such
> > > > > > practice
> > > > > > > > was
> > > > > > > > > >>>>> hard
> > > > > > > > > >>>>>> to
> > > > > > > > > >>>>>>>>>>>> implement because of tight coupling between
> > Ignite
> > > > > > > > components
> > > > > > > > > >>>> and
> > > > > > > > > >>>>>>>>>> inability
> > > > > > > > > >>>>>>>>>>>> to instantiate components without an instance of
> > > > > > > > > KernalContext.
> > > > > > > > > >>>>> For
> > > > > > > > > >>>>>>>>>>>> example, we should be able to thoroughly test
> > > > internal
> > > > > > > > > >>>>> primitives,
> > > > > > > > > >>>>>>>>>> such as
> > > > > > > > > >>>>>>>>>>>> replication protocol (without actual
> > > communication),
> > > > > > > > > >>>> distributed
> > > > > > > > > >>>>>>>>>>>> metastorage contracts, persistence layer, etc.
> > > > > > > > > >>>>>>>>>>>> - We will significantly reduce the development
> > > cycle
> > > > > in
> > > > > > > the
> > > > > > > > > >>>>>> beginning
> > > > > > > > > >>>>>>>>>>>> (right now the RunAll takes two hours of
> > > > astronomical
> > > > > > time
> > > > > > > > > with
> > > > > > > > > >>>>>> empty
> > > > > > > > > >>>>>>>>>> TC;
> > > > > > > > > >>>>>>>>>>>> in the new approach developer will be able to
> > run
> > > > ALL
> > > > > > > tests
> > > > > > > > > >>>>> locally
> > > > > > > > > >>>>>>>>> in
> > > > > > > > > >>>>>>>>>> a
> > > > > > > > > >>>>>>>>>>>> matter of minutes)
> > > > > > > > > >>>>>>>>>>>> - We can get rid of TC bot and enforce green TC
> > by
> > > > > > > > integrating
> > > > > > > > > >>>> TC
> > > > > > > > > >>>>>>>>> build
> > > > > > > > > >>>>>>>>>>>> results with GitHub PRs (the same way Travis is
> > > > > > currently
> > > > > > > > > >>>>>> integrated
> > > > > > > > > >>>>>>>>>> to PR
> > > > > > > > > >>>>>>>>>>>> check). We should restrict PR merge without a TC
> > > > check
> > > > > > > > > >>>>>>>>>>>> - We will still have to re-write all tests, but
> > > only
> > > > > > once.
> > > > > > > > If
> > > > > > > > > >>>> we
> > > > > > > > > >>>>>> try
> > > > > > > > > >>>>>>>>> to
> > > > > > > > > >>>>>>>>>>>> modify the old codebase, we would need to modify
> > > all
> > > > > the
> > > > > > > > tests
> > > > > > > > > >>>>> for
> > > > > > > > > >>>>>>>>>> every
> > > > > > > > > >>>>>>>>>>>> major change (public API change, configuration
> > > > change)
> > > > > > > > > >>>>>>>>>>>> - We will have fewer conflicts when working
> > > > together.
> > > > > > For
> > > > > > > > > >>>>> example,
> > > > > > > > > >>>>>> I
> > > > > > > > > >>>>>>>>>>>> cannot imagine how one would merge two changes
> > of
> > > > > > getting
> > > > > > > > rid
> > > > > > > > > >>>> of
> > > > > > > > > >>>>>>>>>>>> IgniteFuture and changes in replication
> > protocol,
> > > > for
> > > > > > > > example
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> Technically, I would suggest creating a new
> > > > repository
> > > > > > for
> > > > > > > > > >>>> Ignite
> > > > > > > > > >>>>>> 3.0
> > > > > > > > > >>>>>>>>>>>> (perhaps, a new clean branch, but a new repo
> > looks
> > > > > nicer
> > > > > > > to
> > > > > > > > > me)
> > > > > > > > > >>>>>> and a
> > > > > > > > > >>>>>>>>>> new
> > > > > > > > > >>>>>>>>>>>> Ignite 3.0 TeamCity project.
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> While it may seem quite radical, I do believe
> > that
> > > > > this
> > > > > > > > > >>>> approach
> > > > > > > > > >>>>>> will
> > > > > > > > > >>>>>>>>>> give
> > > > > > > > > >>>>>>>>>>>> us more benefits than trying to make such major
> > > > > changes
> > > > > > in
> > > > > > > > the
> > > > > > > > > >>>>>>>>> existing
> > > > > > > > > >>>>>>>>>>>> codebase. If needed, let's schedule a discord
> > chat
> > > > > like
> > > > > > > > before
> > > > > > > > > >>>> to
> > > > > > > > > >>>>>>>>>> discuss
> > > > > > > > > >>>>>>>>>>>> this.
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> WDYT?
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>> --
> > > > > > > > > >>>> Best regards,
> > > > > > > > > >>>> Andrey V. Mashenkov
> > > > > > > > > >>>>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
> --
>
> Best regards,
> Alexei Scherbakov
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Alexei Scherbakov <al...@gmail.com>.
Makes sense to me.
вт, 10 нояб. 2020 г. в 18:47, Sergey Chugunov <se...@gmail.com>:
> Igniters,
>
> I thought over Friday meeting ideas and concerns and summarized them in
> these three points:
>
>
> 1. *Components design unification approach.* New proposed components
> will be developed by different contributors, but they need to be unified
> and should integrate with each other easily. To ensure that I suggest
> calling an architecture group that will create design guidelines for all
> components and high-level overview of overall architecture. How code is
> split into components, what are component boundaries, how component
> lifecycle works and what are its interfaces - all these and other
> questions
> should be covered.
>
> 2. *Scope management.* Apache 3.0 should be implemented within a
> reasonable time, so we need some procedure to decide whether a
> particular
> feature should be dropped from the scope of 3.0 and postponed to 3.1
> release. To do so I suggest to range all features by two parameters:
> criticality for 3.0 and amount of breaking changes. 3.0 scope should
> include features of high criticality AND features with a big amount of
> breaking changes. All other features can be made optional.
>
> 3. *Development transparency.* Development of all components should be
> made as transparent for everyone as possible. Any contributor should be
> able to look over any component at any stage of development. To achieve
> this I suggest to create a separate public repository dedicated for 3.0
> development. It will make the code available for everyone but when
> development of 3.0 is done we won't loose any stars of our current
> repository as we merge dev repo into main one and drop dev.
>
> Do these ideas make sense to you? Are there any concerns not covered by
> these suggestions?
>
> On Fri, Nov 6, 2020 at 7:36 PM Kseniya Romanova <romanova.ks.spb@gmail.com
> >
> wrote:
>
> > Here are the slides from Alexey Goncharuk. Let's think this over and
> > continue on Monday:
> >
> >
> https://go.gridgain.com/rs/491-TWR-806/images/Ignite_3_Plans_and_development_process.pdf
> >
> > чт, 5 нояб. 2020 г. в 11:13, Anton Vinogradov <av...@apache.org>:
> >
> > > Folks,
> > >
> > > Should we perform cleanup work before (r)evolutional changes?
> > > My huge proposal is to get rid of things which we don't need anyway
> > > - local caches,
> > > - strange tx modes,
> > > - code overcomplexity because of RollingUpgrade feature never attended
> at
> > > AI,
> > > - etc,
> > > before choosing the way.
> > >
> > > On Tue, Nov 3, 2020 at 3:31 PM Valentin Kulichenko <
> > > valentin.kulichenko@gmail.com> wrote:
> > >
> > > > Ksenia, thanks for scheduling this on such short notice!
> > > >
> > > > As for the original topic, I do support Alexey's idea. We're not
> going
> > to
> > > > rewrite anything from scratch, as most of the components are going to
> > be
> > > > moved as-is or with minimal modifications. However, the changes that
> > are
> > > > proposed imply serious rework of the core parts of the code, which
> are
> > > not
> > > > properly decoupled from each other and from other parts. This makes
> the
> > > > incremental approach borderline impossible. Developing in a new repo,
> > > > however, addresses this concern. As a bonus, we can also refactor the
> > > code,
> > > > introduce better decoupling, get rid of kernel context, and develop
> > unit
> > > > tests (finally!).
> > > >
> > > > Basically, this proposal only affects the *process*, not the set of
> > > changes
> > > > we had discussed before. Ignite 3.0 is our unique chance to make
> things
> > > > right.
> > > >
> > > > -Val
> > > >
> > > > On Tue, Nov 3, 2020 at 3:06 AM Kseniya Romanova <
> > > romanova.ks.spb@gmail.com
> > > > >
> > > > wrote:
> > > >
> > > > > Pavel, all the interesting points will be anyway published here in
> > > > English
> > > > > (as the principal "if it's not on devlist it doesn't happened" is
> > still
> > > > > relevant). This is just a quick call for a group of developers.
> Later
> > > we
> > > > > can do a separate presentation of idea and discussion in English as
> > we
> > > > did
> > > > > for the Ignite 3.0 draft of changes.
> > > > >
> > > > > вт, 3 нояб. 2020 г. в 13:52, Pavel Tupitsyn <ptupitsyn@apache.org
> >:
> > > > >
> > > > > > Kseniya,
> > > > > >
> > > > > > Thanks for scheduling this call.
> > > > > > Do you think we can switch to English if non-Russian speaking
> > > community
> > > > > > members decide to join?
> > > > > >
> > > > > > On Tue, Nov 3, 2020 at 1:32 PM Kseniya Romanova <
> > > > > romanova.ks.spb@gmail.com
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Let's do this community discussion open. Here's the link on
> zoom
> > > call
> > > > > in
> > > > > > > Russian for Friday 6 PM:
> > > > > > >
> > > https://www.meetup.com/Moscow-Apache-Ignite-Meetup/events/274360378/
> > > > > > >
> > > > > > > вт, 3 нояб. 2020 г. в 12:49, Nikolay Izhikov <
> > nizhikov@apache.org
> > > >:
> > > > > > >
> > > > > > > > Time works for me.
> > > > > > > >
> > > > > > > > > 3 нояб. 2020 г., в 12:40, Alexey Goncharuk <
> > > > > > alexey.goncharuk@gmail.com
> > > > > > > >
> > > > > > > > написал(а):
> > > > > > > > >
> > > > > > > > > Nikolay,
> > > > > > > > >
> > > > > > > > > I am up for the call. I will try to explain my reasoning in
> > > > greater
> > > > > > > > detail
> > > > > > > > > and will be glad to hear the concerns. Will this Friday,
> Nov
> > > 6th,
> > > > > > work?
> > > > > > > > >
> > > > > > > > > вт, 3 нояб. 2020 г. в 10:09, Nikolay Izhikov <
> > > > nizhikov@apache.org
> > > > > >:
> > > > > > > > >
> > > > > > > > >> Igniters, should we have a call for this topic?
> > > > > > > > >>
> > > > > > > > >>> 2 нояб. 2020 г., в 18:53, Pavel Tupitsyn <
> > > ptupitsyn@apache.org
> > > > >
> > > > > > > > >> написал(а):
> > > > > > > > >>>
> > > > > > > > >>>> not intend to rewrite everything from scratch
> > > > > > > > >>>
> > > > > > > > >>>> Every single test from Ignite 2.x should be moved to
> > Ignite
> > > 3
> > > > > > > > >>>> regardless of how we choose to proceed.
> > > > > > > > >>>
> > > > > > > > >>> Alexey, thank you for the explanation, this addresses all
> > of
> > > my
> > > > > > > > concerns.
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>> On Mon, Nov 2, 2020 at 6:43 PM Andrey Mashenkov <
> > > > > > > > >> andrey.mashenkov@gmail.com>
> > > > > > > > >>> wrote:
> > > > > > > > >>>
> > > > > > > > >>>> Hi, Igniters.
> > > > > > > > >>>>
> > > > > > > > >>>> * AFAIU, we need a new repo if we want to apply
> different
> > > > > > > restrictions
> > > > > > > > >> to
> > > > > > > > >>>> pull requests,
> > > > > > > > >>>> otherwise I see no difference for myself.
> > > > > > > > >>>> E.g. make static analysis (do we have?), compile,
> styles,
> > > and
> > > > > > > javadoc
> > > > > > > > >>>> checks mandatory.
> > > > > > > > >>>>
> > > > > > > > >>>> I think that relaxed requirements here will lead to bad
> > > > product
> > > > > > > > quality.
> > > > > > > > >>>>
> > > > > > > > >>>> * Agree with Pavel, we should 'keep' integrations tests
> > > > somehow.
> > > > > > > > >>>> During active development tests will be broken most of
> > time,
> > > > so,
> > > > > > > > >>>> I'd port them e.g. suite-by-suite once we will have a
> > stable
> > > > and
> > > > > > > > >> featured
> > > > > > > > >>>> environment to run them and of course make test's code
> > clear
> > > > and
> > > > > > > avoid
> > > > > > > > >>>> bad/non-relevant ones.
> > > > > > > > >>>>
> > > > > > > > >>>> * I like bottom-up approach.
> > > > > > > > >>>> With it we could make a better framework. I mean clear
> > > > component
> > > > > > > > >> lifecycle,
> > > > > > > > >>>> component wiring mechanics, general methods to approach
> > core
> > > > > > > > components
> > > > > > > > >>>> such as exchange/communication
> > > > > > > > >>>> to avoid code mess like we have in ExchangeFuture with
> all
> > > > these
> > > > > > > > custom
> > > > > > > > >>>> callbacks for each component, interfaces like
> > > > > > > > >>>> PartitionsExchangeAware, IgniteChangeGlobalStateSupport
> > and
> > > > > > > > >>>> a pack of
> > > > > > > > >>
> > > start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected
> > > > > > > > >>>> and so on in various unexpected places.
> > > > > > > > >>>> Hope, we will be able to port most of the good code to
> the
> > > new
> > > > > > > > framework
> > > > > > > > >>>> version.
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>> On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk <
> > > > > > > > >>>> alexey.goncharuk@gmail.com>
> > > > > > > > >>>> wrote:
> > > > > > > > >>>>
> > > > > > > > >>>>> Nikolay, Pavel,
> > > > > > > > >>>>>
> > > > > > > > >>>>> Thanks for the feedback! First of all, I wanted to
> stress
> > > > that
> > > > > I
> > > > > > do
> > > > > > > > not
> > > > > > > > >>>>> intend to rewrite everything from scratch (I never used
> > > this
> > > > > > > phrase).
> > > > > > > > >>>> There
> > > > > > > > >>>>> are significant parts of code that would be moved with
> > > > minimal
> > > > > > > > >>>>> modifications. Second, I never said that we will get
> rid
> > of
> > > > the
> > > > > > old
> > > > > > > > >> tests
> > > > > > > > >>>>> codebase. Every single test from Ignite 2.x should be
> > moved
> > > > to
> > > > > > > > Ignite 3
> > > > > > > > >>>>> regardless of how we choose to proceed.
> > > > > > > > >>>>>
> > > > > > > > >>>>> My point is that for some parts of the code a clean
> > > bottom-up
> > > > > > > > >>>>> implementation will be cheaper in many ways. Let me
> give
> > > you
> > > > a
> > > > > > few
> > > > > > > > >>>> concrete
> > > > > > > > >>>>> examples:
> > > > > > > > >>>>>
> > > > > > > > >>>>> - I think no one can object that we need a cleanly
> > > separated
> > > > > > > > >>>> persistence
> > > > > > > > >>>>> layer for Ignite. There is a very raw draft IEP for
> this
> > > > > > already.
> > > > > > > On
> > > > > > > > >>>> the
> > > > > > > > >>>>> other hand, I think we also can agree that we need a
> > > > > split-brain
> > > > > > > > >>>>> resistant
> > > > > > > > >>>>> replication protocol for caches. There is also an IEP
> > for
> > > > > this.
> > > > > > > > >>>> Neither
> > > > > > > > >>>>> of
> > > > > > > > >>>>> the changes is a good fit for 2.x because they are
> > likely
> > > to
> > > > > > > > >> introduce
> > > > > > > > >>>>> breaking changes in the persistence layer,
> configuration
> > > and
> > > > > > > > >> behavior.
> > > > > > > > >>>>> Additionally, these components are now tightly
> coupled,
> > so
> > > > > there
> > > > > > > is
> > > > > > > > >> no
> > > > > > > > >>>>> way
> > > > > > > > >>>>> these two changes can be implemented in parallel and
> > then
> > > > > merged
> > > > > > > > >>>>> together
> > > > > > > > >>>>> easily. So what we will end up with is having to
> > implement
> > > > > these
> > > > > > > > >>>> changes
> > > > > > > > >>>>> sequentially, fixing all existing tests twice, and
> > > > essentially
> > > > > > > > >>>> throwing
> > > > > > > > >>>>> away half of the work done because the other part of
> the
> > > > > change
> > > > > > is
> > > > > > > > >>>>> re-implemented
> > > > > > > > >>>>> - Similar example goes with getting rid of
> > > > > IgniteInternalFuture
> > > > > > > and
> > > > > > > > >>>>> replacing it with CompletableFuture, and any other
> > change
> > > > that
> > > > > > > > >> touches
> > > > > > > > >>>>> the
> > > > > > > > >>>>> asynchronous part of the code.
> > > > > > > > >>>>>
> > > > > > > > >>>>> Third, I do not see how this choice affects the UX of
> > > Ignite.
> > > > > The
> > > > > > > end
> > > > > > > > >>>> user
> > > > > > > > >>>>> experience must be fixed in the IEP regardless of the
> > > > > development
> > > > > > > > >> process
> > > > > > > > >>>>> and the fact that we have gaps in this area in Ignite
> 2.x
> > > > just
> > > > > > > > confirms
> > > > > > > > >>>>> that.
> > > > > > > > >>>>>
> > > > > > > > >>>>> Pavel, agree that a repo/branch is a technicality, I
> > guess
> > > if
> > > > > > > > >>>> reformulate,
> > > > > > > > >>>>> my point is that we might agree to have a single
> > > development
> > > > > > master
> > > > > > > > >>>> branch
> > > > > > > > >>>>> with 'disassembled' end-to-end functionality for some
> > > period
> > > > of
> > > > > > > time
> > > > > > > > to
> > > > > > > > >>>>> speed up development, and re-assemble the core features
> > > after
> > > > > > > having
> > > > > > > > >>>>> submodules tested independently.
> > > > > > > > >>>>>
> > > > > > > > >>>>> Nikolay,
> > > > > > > > >>>>>> We have many features that have to evolve.
> > > > > > > > >>>>>> Snapshots, rebalance, tooling, tracing, zookeeper
> > support,
> > > > > etc.
> > > > > > > > >>>>> This is not very specific. In the end, resources are
> > > limited
> > > > > and
> > > > > > we
> > > > > > > > >> will
> > > > > > > > >>>>> not be able to drive both tracks simultaneously,
> > especially
> > > > > > after a
> > > > > > > > >>>> couple
> > > > > > > > >>>>> of features having been implemented for Ignite 3.0. If
> > > there
> > > > > are
> > > > > > > > indeed
> > > > > > > > >>>>> some major changes that we want to do in Ignite 2.x
> > instead
> > > > of
> > > > > > > > putting
> > > > > > > > >>>>> effort into 3.0 - let's discuss them. I am just not
> aware
> > > of
> > > > > any,
> > > > > > > > >> that's
> > > > > > > > >>>>> why I am eager to move forward with Ignite 3.0.
> > > > > > > > >>>>>
> > > > > > > > >>>>>> We have bugs and issues that can be fixed in 2.x
> without
> > > > > > breaking
> > > > > > > > >>>> backward
> > > > > > > > >>>>> compatibility.
> > > > > > > > >>>>>> We have many users who are happy with the 2.x with all
> > > it’s
> > > > > > > issues.
> > > > > > > > >>>>> These changes will be covered by end-to-end tests and
> > > > migrated
> > > > > to
> > > > > > > > >> Ignite
> > > > > > > > >>>>> 3.0, so I see no issues here.
> > > > > > > > >>>>>
> > > > > > > > >>>>> Finally, Anton & Nikolay
> > > > > > > > >>>>> I do not have an estimate for this simply because the
> > > > activity
> > > > > is
> > > > > > > > >>>>> community-driven and it depends on the number of people
> > > > willing
> > > > > > to
> > > > > > > > >>>>> contribute. With the current pace, I would hope to have
> > an
> > > RC
> > > > > of
> > > > > > > > Ignite
> > > > > > > > >>>> 3.0
> > > > > > > > >>>>> to be ready by the end of 2021. My gut feeling is that
> by
> > > > > moving
> > > > > > > with
> > > > > > > > >>>>> incremental changes, we will not be able to implement
> > even
> > > > half
> > > > > > of
> > > > > > > > the
> > > > > > > > >>>>> wishlist by that time.
> > > > > > > > >>>>> I doubt that releasing several major releases with
> > breaking
> > > > > > changes
> > > > > > > > >> will
> > > > > > > > >>>>> make Ignite users happy either because each upgrade
> will
> > > cost
> > > > > > > Ignite
> > > > > > > > >>>> users
> > > > > > > > >>>>> money, so the fewer major versions we release, the
> > better.
> > > > Thus
> > > > > > my
> > > > > > > > wish
> > > > > > > > >>>> to
> > > > > > > > >>>>> include all breaking changes in one release.
> > > > > > > > >>>>>
> > > > > > > > >>>>> I'll be now quiet for a while, let's see what other
> > > community
> > > > > > > members
> > > > > > > > >>>>> think.
> > > > > > > > >>>>>
> > > > > > > > >>>>> пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <
> > > > > > ptupitsyn@apache.org
> > > > > > > >:
> > > > > > > > >>>>>
> > > > > > > > >>>>>> 1. Rewriting from scratch is never a good idea.
> > > > > > > > >>>>>> We don't want to follow the path of Netscape and lose
> > all
> > > > our
> > > > > > > users
> > > > > > > > >>>>>> by the time we have a working 3.0 [1]
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> 2. Not sure about new repo - seems like some pain and
> no
> > > > gain,
> > > > > > > > what's
> > > > > > > > >>>> the
> > > > > > > > >>>>>> problem with a branch?
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> 3. We should keep existing integration tests when
> > > possible.
> > > > > > > > >>>>>> We have accumulated a lot of edge case knowledge over
> > the
> > > > > years,
> > > > > > > > >>>>>> it is not a good idea to send all of that down the
> > drain.
> > > > > > > > >>>>>> Yes, integration tests are slow, but they are the most
> > > > > valuable.
> > > > > > > > >>>>>> I think we can move more stuff into nightly runs and
> > have
> > > a
> > > > > fast
> > > > > > > and
> > > > > > > > >>>>> modern
> > > > > > > > >>>>>> basic suite.
> > > > > > > > >>>>>>
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> Alexey, you are much more familiar with the Ignite
> core
> > > > > codebase
> > > > > > > > than
> > > > > > > > >>>>> most
> > > > > > > > >>>>>> of us,
> > > > > > > > >>>>>> can you please explain in more detail which particular
> > > > > feature,
> > > > > > in
> > > > > > > > >> your
> > > > > > > > >>>>>> opinion,
> > > > > > > > >>>>>> mandates this "start from scratch" approach?
> > > > > > > > >>>>>> Is it really not possible at all to follow a less
> > radical
> > > > way?
> > > > > > > > >>>>>>
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> [1]
> > > > > > > > >>>>>>
> > > > > > > > >>>>>>
> > > > > > > > >>>>>
> > > > > > > > >>>>
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <
> > > > > > > nizhikov@apache.org
> > > > > > > > >
> > > > > > > > >>>>>> wrote:
> > > > > > > > >>>>>>
> > > > > > > > >>>>>>> Hello, Alexey.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> I think that «rewriting from scratch» approach has a
> > high
> > > > > risk
> > > > > > to
> > > > > > > > >>>> make
> > > > > > > > >>>>>> new
> > > > > > > > >>>>>>> features unusable.
> > > > > > > > >>>>>>> At the time Ignite2 was started no-one wants to do
> bad
> > UX
> > > > or
> > > > > > bad
> > > > > > > > >>>>>> features.
> > > > > > > > >>>>>>> Nevertheless, it happen.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> I think we can avoid it with the Ignite3 and
> successors
> > > if
> > > > we
> > > > > > > will
> > > > > > > > >>>> move
> > > > > > > > >>>>>>> step by step without keeping backward compatibility
> > > > > > > > >>>>>>> With the step by step approach, we can focus on each
> > > > > component
> > > > > > > > >>>>>> separately.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>>> What new features are we planning to implement for
> > > Ignite
> > > > > 2.x?
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> We have many features that have to evolve.
> > > > > > > > >>>>>>> Snapshots, rebalance, tooling, tracing, zookeeper
> > > support,
> > > > > etc.
> > > > > > > > >>>>>>> We have bugs and issues that can be fixed in 2.x
> > without
> > > > > > breaking
> > > > > > > > >>>>>> backward
> > > > > > > > >>>>>>> compatibility.
> > > > > > > > >>>>>>> We have many users who are happy with the 2.x with
> all
> > > it’s
> > > > > > > issues.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>>> 2 нояб. 2020 г., в 14:09, Anton Vinogradov <
> > > av@apache.org
> > > > >
> > > > > > > > >>>>> написал(а):
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> Alexey,
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> Do we have any estimates of how fast we'll be able
> to
> > > gain
> > > > > > > > >>>>>>> production-ready
> > > > > > > > >>>>>>>> AI 3.0 in case of a "new repo" choice?
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk <
> > > > > > > > >>>>>>> alexey.goncharuk@gmail.com>
> > > > > > > > >>>>>>>> wrote:
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>>> Nikolay,
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> What new features are we planning to implement for
> > > Ignite
> > > > > > 2.x?
> > > > > > > I
> > > > > > > > >>>>> think
> > > > > > > > >>>>>>> once
> > > > > > > > >>>>>>>>> we commence working on Ignite 3.0, we should
> > gradually
> > > > > cease
> > > > > > > the
> > > > > > > > >>>>>>> activity
> > > > > > > > >>>>>>>>> on Ignite 2.x to mere bugfixes because such
> parallel
> > > > > > > development
> > > > > > > > >>>>> will
> > > > > > > > >>>>>> be
> > > > > > > > >>>>>>>>> overwhelming regardless of how we choose to
> proceed.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <
> > > > > > > > nizhikov@apache.org
> > > > > > > > >>>>> :
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>>> To be clear:
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>>> I would suggest creating a new repository for
> > Ignite
> > > > 3.0
> > > > > > > > >>>>> (perhaps, a
> > > > > > > > >>>>>>>>> new
> > > > > > > > >>>>>>>>>> clean branch, but a new repo looks nicer to me)
> and
> > a
> > > > new
> > > > > > > Ignite
> > > > > > > > >>>>> 3.0
> > > > > > > > >>>>>>>>>> TeamCity project.
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> +1 for new Team City project.
> > > > > > > > >>>>>>>>>> +1 for new branch for Ignite3.
> > > > > > > > >>>>>>>>>> -1 for new repo.
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <
> > > > > > > > >>>> NIzhikov.dev@gmail.com
> > > > > > > > >>>>>>
> > > > > > > > >>>>>>>>>> написал(а):
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> Hello, Alexey.
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> I think it will hurt our project more than help.
> > > > > > > > >>>>>>>>>>> Developing new features for 2 separate branches
> > with
> > > > the
> > > > > > > > >>>> different
> > > > > > > > >>>>>>> APIs
> > > > > > > > >>>>>>>>>> and internal structure is overwhelming
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> Maybe we should relax a bit requirements for
> > Ignite3?
> > > > > > > > >>>>>>>>>>> Maybe we should move step by step and make
> Ignite3
> > > with
> > > > > new
> > > > > > > > >>>>>>>>>> configuration than Ignite4 with new transactions,
> > etc?
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <
> > > > > > > > >>>>>>>>> alexey.goncharuk@gmail.com>
> > > > > > > > >>>>>>>>>> написал(а):
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> Igniters,
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> I wanted to pitch a rather radical idea
> regarding
> > > the
> > > > > > Ignite
> > > > > > > > >>>> 3.0
> > > > > > > > >>>>>>>>>>>> development which has occurred to me some time
> > ago.
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> We already have several IEPs targeted to Ignite
> > 3.0
> > > > > which
> > > > > > > > imply
> > > > > > > > >>>>>> major
> > > > > > > > >>>>>>>>>>>> changes to the codebase (the change in
> replication
> > > > > > protocol
> > > > > > > > and
> > > > > > > > >>>>>> thus
> > > > > > > > >>>>>>>>>>>> transactions, change in binary format, updated
> > > > > > metastorage,
> > > > > > > > >>>> etc).
> > > > > > > > >>>>>> We
> > > > > > > > >>>>>>>>>> also
> > > > > > > > >>>>>>>>>>>> planned significant changes in public APIs:
> > > > > configuration
> > > > > > > > >>>> format
> > > > > > > > >>>>>>>>> change,
> > > > > > > > >>>>>>>>>>>> improvements in cache APIs, SQL APIs,
> transaction
> > > mode
> > > > > > > rework.
> > > > > > > > >>>>> The
> > > > > > > > >>>>>>>>>> wishlist
> > > > > > > > >>>>>>>>>>>> of changes for Ignite 3.0 is huge.
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> So, I was wondering whether it makes sense to
> try
> > to
> > > > > > change
> > > > > > > > the
> > > > > > > > >>>>> old
> > > > > > > > >>>>>>>>>>>> codebase, or start with a new baseline and move
> > old
> > > > > pieces
> > > > > > > of
> > > > > > > > >>>>> code
> > > > > > > > >>>>>>>>> that
> > > > > > > > >>>>>>>>>> do
> > > > > > > > >>>>>>>>>>>> not require significant rework. Personally, I
> > would
> > > go
> > > > > > with
> > > > > > > > the
> > > > > > > > >>>>>>> second
> > > > > > > > >>>>>>>>>>>> option for the following reasons:
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> - We have a chance to shift the development
> > paradigm
> > > > in
> > > > > > the
> > > > > > > > >>>>> project
> > > > > > > > >>>>>>>>> and
> > > > > > > > >>>>>>>>>>>> introduce the practice of true unit-tests. In
> the
> > > new
> > > > > > > baseline
> > > > > > > > >>>> at
> > > > > > > > >>>>>> the
> > > > > > > > >>>>>>>>>>>> beginning there will be no ability to run an
> > > > end-to-end
> > > > > > > > >>>> scenario,
> > > > > > > > >>>>>>>>> thus
> > > > > > > > >>>>>>>>>> we
> > > > > > > > >>>>>>>>>>>> will be forced to write unit-tests. So far, such
> > > > > practice
> > > > > > > was
> > > > > > > > >>>>> hard
> > > > > > > > >>>>>> to
> > > > > > > > >>>>>>>>>>>> implement because of tight coupling between
> Ignite
> > > > > > > components
> > > > > > > > >>>> and
> > > > > > > > >>>>>>>>>> inability
> > > > > > > > >>>>>>>>>>>> to instantiate components without an instance of
> > > > > > > > KernalContext.
> > > > > > > > >>>>> For
> > > > > > > > >>>>>>>>>>>> example, we should be able to thoroughly test
> > > internal
> > > > > > > > >>>>> primitives,
> > > > > > > > >>>>>>>>>> such as
> > > > > > > > >>>>>>>>>>>> replication protocol (without actual
> > communication),
> > > > > > > > >>>> distributed
> > > > > > > > >>>>>>>>>>>> metastorage contracts, persistence layer, etc.
> > > > > > > > >>>>>>>>>>>> - We will significantly reduce the development
> > cycle
> > > > in
> > > > > > the
> > > > > > > > >>>>>> beginning
> > > > > > > > >>>>>>>>>>>> (right now the RunAll takes two hours of
> > > astronomical
> > > > > time
> > > > > > > > with
> > > > > > > > >>>>>> empty
> > > > > > > > >>>>>>>>>> TC;
> > > > > > > > >>>>>>>>>>>> in the new approach developer will be able to
> run
> > > ALL
> > > > > > tests
> > > > > > > > >>>>> locally
> > > > > > > > >>>>>>>>> in
> > > > > > > > >>>>>>>>>> a
> > > > > > > > >>>>>>>>>>>> matter of minutes)
> > > > > > > > >>>>>>>>>>>> - We can get rid of TC bot and enforce green TC
> by
> > > > > > > integrating
> > > > > > > > >>>> TC
> > > > > > > > >>>>>>>>> build
> > > > > > > > >>>>>>>>>>>> results with GitHub PRs (the same way Travis is
> > > > > currently
> > > > > > > > >>>>>> integrated
> > > > > > > > >>>>>>>>>> to PR
> > > > > > > > >>>>>>>>>>>> check). We should restrict PR merge without a TC
> > > check
> > > > > > > > >>>>>>>>>>>> - We will still have to re-write all tests, but
> > only
> > > > > once.
> > > > > > > If
> > > > > > > > >>>> we
> > > > > > > > >>>>>> try
> > > > > > > > >>>>>>>>> to
> > > > > > > > >>>>>>>>>>>> modify the old codebase, we would need to modify
> > all
> > > > the
> > > > > > > tests
> > > > > > > > >>>>> for
> > > > > > > > >>>>>>>>>> every
> > > > > > > > >>>>>>>>>>>> major change (public API change, configuration
> > > change)
> > > > > > > > >>>>>>>>>>>> - We will have fewer conflicts when working
> > > together.
> > > > > For
> > > > > > > > >>>>> example,
> > > > > > > > >>>>>> I
> > > > > > > > >>>>>>>>>>>> cannot imagine how one would merge two changes
> of
> > > > > getting
> > > > > > > rid
> > > > > > > > >>>> of
> > > > > > > > >>>>>>>>>>>> IgniteFuture and changes in replication
> protocol,
> > > for
> > > > > > > example
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> Technically, I would suggest creating a new
> > > repository
> > > > > for
> > > > > > > > >>>> Ignite
> > > > > > > > >>>>>> 3.0
> > > > > > > > >>>>>>>>>>>> (perhaps, a new clean branch, but a new repo
> looks
> > > > nicer
> > > > > > to
> > > > > > > > me)
> > > > > > > > >>>>>> and a
> > > > > > > > >>>>>>>>>> new
> > > > > > > > >>>>>>>>>>>> Ignite 3.0 TeamCity project.
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> While it may seem quite radical, I do believe
> that
> > > > this
> > > > > > > > >>>> approach
> > > > > > > > >>>>>> will
> > > > > > > > >>>>>>>>>> give
> > > > > > > > >>>>>>>>>>>> us more benefits than trying to make such major
> > > > changes
> > > > > in
> > > > > > > the
> > > > > > > > >>>>>>>>> existing
> > > > > > > > >>>>>>>>>>>> codebase. If needed, let's schedule a discord
> chat
> > > > like
> > > > > > > before
> > > > > > > > >>>> to
> > > > > > > > >>>>>>>>>> discuss
> > > > > > > > >>>>>>>>>>>> this.
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> WDYT?
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>
> > > > > > > > >>>>>
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>> --
> > > > > > > > >>>> Best regards,
> > > > > > > > >>>> Andrey V. Mashenkov
> > > > > > > > >>>>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
--
Best regards,
Alexei Scherbakov
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Sergey Chugunov <se...@gmail.com>.
Igniters,
I thought over Friday meeting ideas and concerns and summarized them in
these three points:
1. *Components design unification approach.* New proposed components
will be developed by different contributors, but they need to be unified
and should integrate with each other easily. To ensure that I suggest
calling an architecture group that will create design guidelines for all
components and high-level overview of overall architecture. How code is
split into components, what are component boundaries, how component
lifecycle works and what are its interfaces - all these and other questions
should be covered.
2. *Scope management.* Apache 3.0 should be implemented within a
reasonable time, so we need some procedure to decide whether a particular
feature should be dropped from the scope of 3.0 and postponed to 3.1
release. To do so I suggest to range all features by two parameters:
criticality for 3.0 and amount of breaking changes. 3.0 scope should
include features of high criticality AND features with a big amount of
breaking changes. All other features can be made optional.
3. *Development transparency.* Development of all components should be
made as transparent for everyone as possible. Any contributor should be
able to look over any component at any stage of development. To achieve
this I suggest to create a separate public repository dedicated for 3.0
development. It will make the code available for everyone but when
development of 3.0 is done we won't loose any stars of our current
repository as we merge dev repo into main one and drop dev.
Do these ideas make sense to you? Are there any concerns not covered by
these suggestions?
On Fri, Nov 6, 2020 at 7:36 PM Kseniya Romanova <ro...@gmail.com>
wrote:
> Here are the slides from Alexey Goncharuk. Let's think this over and
> continue on Monday:
>
> https://go.gridgain.com/rs/491-TWR-806/images/Ignite_3_Plans_and_development_process.pdf
>
> чт, 5 нояб. 2020 г. в 11:13, Anton Vinogradov <av...@apache.org>:
>
> > Folks,
> >
> > Should we perform cleanup work before (r)evolutional changes?
> > My huge proposal is to get rid of things which we don't need anyway
> > - local caches,
> > - strange tx modes,
> > - code overcomplexity because of RollingUpgrade feature never attended at
> > AI,
> > - etc,
> > before choosing the way.
> >
> > On Tue, Nov 3, 2020 at 3:31 PM Valentin Kulichenko <
> > valentin.kulichenko@gmail.com> wrote:
> >
> > > Ksenia, thanks for scheduling this on such short notice!
> > >
> > > As for the original topic, I do support Alexey's idea. We're not going
> to
> > > rewrite anything from scratch, as most of the components are going to
> be
> > > moved as-is or with minimal modifications. However, the changes that
> are
> > > proposed imply serious rework of the core parts of the code, which are
> > not
> > > properly decoupled from each other and from other parts. This makes the
> > > incremental approach borderline impossible. Developing in a new repo,
> > > however, addresses this concern. As a bonus, we can also refactor the
> > code,
> > > introduce better decoupling, get rid of kernel context, and develop
> unit
> > > tests (finally!).
> > >
> > > Basically, this proposal only affects the *process*, not the set of
> > changes
> > > we had discussed before. Ignite 3.0 is our unique chance to make things
> > > right.
> > >
> > > -Val
> > >
> > > On Tue, Nov 3, 2020 at 3:06 AM Kseniya Romanova <
> > romanova.ks.spb@gmail.com
> > > >
> > > wrote:
> > >
> > > > Pavel, all the interesting points will be anyway published here in
> > > English
> > > > (as the principal "if it's not on devlist it doesn't happened" is
> still
> > > > relevant). This is just a quick call for a group of developers. Later
> > we
> > > > can do a separate presentation of idea and discussion in English as
> we
> > > did
> > > > for the Ignite 3.0 draft of changes.
> > > >
> > > > вт, 3 нояб. 2020 г. в 13:52, Pavel Tupitsyn <pt...@apache.org>:
> > > >
> > > > > Kseniya,
> > > > >
> > > > > Thanks for scheduling this call.
> > > > > Do you think we can switch to English if non-Russian speaking
> > community
> > > > > members decide to join?
> > > > >
> > > > > On Tue, Nov 3, 2020 at 1:32 PM Kseniya Romanova <
> > > > romanova.ks.spb@gmail.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Let's do this community discussion open. Here's the link on zoom
> > call
> > > > in
> > > > > > Russian for Friday 6 PM:
> > > > > >
> > https://www.meetup.com/Moscow-Apache-Ignite-Meetup/events/274360378/
> > > > > >
> > > > > > вт, 3 нояб. 2020 г. в 12:49, Nikolay Izhikov <
> nizhikov@apache.org
> > >:
> > > > > >
> > > > > > > Time works for me.
> > > > > > >
> > > > > > > > 3 нояб. 2020 г., в 12:40, Alexey Goncharuk <
> > > > > alexey.goncharuk@gmail.com
> > > > > > >
> > > > > > > написал(а):
> > > > > > > >
> > > > > > > > Nikolay,
> > > > > > > >
> > > > > > > > I am up for the call. I will try to explain my reasoning in
> > > greater
> > > > > > > detail
> > > > > > > > and will be glad to hear the concerns. Will this Friday, Nov
> > 6th,
> > > > > work?
> > > > > > > >
> > > > > > > > вт, 3 нояб. 2020 г. в 10:09, Nikolay Izhikov <
> > > nizhikov@apache.org
> > > > >:
> > > > > > > >
> > > > > > > >> Igniters, should we have a call for this topic?
> > > > > > > >>
> > > > > > > >>> 2 нояб. 2020 г., в 18:53, Pavel Tupitsyn <
> > ptupitsyn@apache.org
> > > >
> > > > > > > >> написал(а):
> > > > > > > >>>
> > > > > > > >>>> not intend to rewrite everything from scratch
> > > > > > > >>>
> > > > > > > >>>> Every single test from Ignite 2.x should be moved to
> Ignite
> > 3
> > > > > > > >>>> regardless of how we choose to proceed.
> > > > > > > >>>
> > > > > > > >>> Alexey, thank you for the explanation, this addresses all
> of
> > my
> > > > > > > concerns.
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>> On Mon, Nov 2, 2020 at 6:43 PM Andrey Mashenkov <
> > > > > > > >> andrey.mashenkov@gmail.com>
> > > > > > > >>> wrote:
> > > > > > > >>>
> > > > > > > >>>> Hi, Igniters.
> > > > > > > >>>>
> > > > > > > >>>> * AFAIU, we need a new repo if we want to apply different
> > > > > > restrictions
> > > > > > > >> to
> > > > > > > >>>> pull requests,
> > > > > > > >>>> otherwise I see no difference for myself.
> > > > > > > >>>> E.g. make static analysis (do we have?), compile, styles,
> > and
> > > > > > javadoc
> > > > > > > >>>> checks mandatory.
> > > > > > > >>>>
> > > > > > > >>>> I think that relaxed requirements here will lead to bad
> > > product
> > > > > > > quality.
> > > > > > > >>>>
> > > > > > > >>>> * Agree with Pavel, we should 'keep' integrations tests
> > > somehow.
> > > > > > > >>>> During active development tests will be broken most of
> time,
> > > so,
> > > > > > > >>>> I'd port them e.g. suite-by-suite once we will have a
> stable
> > > and
> > > > > > > >> featured
> > > > > > > >>>> environment to run them and of course make test's code
> clear
> > > and
> > > > > > avoid
> > > > > > > >>>> bad/non-relevant ones.
> > > > > > > >>>>
> > > > > > > >>>> * I like bottom-up approach.
> > > > > > > >>>> With it we could make a better framework. I mean clear
> > > component
> > > > > > > >> lifecycle,
> > > > > > > >>>> component wiring mechanics, general methods to approach
> core
> > > > > > > components
> > > > > > > >>>> such as exchange/communication
> > > > > > > >>>> to avoid code mess like we have in ExchangeFuture with all
> > > these
> > > > > > > custom
> > > > > > > >>>> callbacks for each component, interfaces like
> > > > > > > >>>> PartitionsExchangeAware, IgniteChangeGlobalStateSupport
> and
> > > > > > > >>>> a pack of
> > > > > > > >>
> > start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected
> > > > > > > >>>> and so on in various unexpected places.
> > > > > > > >>>> Hope, we will be able to port most of the good code to the
> > new
> > > > > > > framework
> > > > > > > >>>> version.
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > > > >>>> On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk <
> > > > > > > >>>> alexey.goncharuk@gmail.com>
> > > > > > > >>>> wrote:
> > > > > > > >>>>
> > > > > > > >>>>> Nikolay, Pavel,
> > > > > > > >>>>>
> > > > > > > >>>>> Thanks for the feedback! First of all, I wanted to stress
> > > that
> > > > I
> > > > > do
> > > > > > > not
> > > > > > > >>>>> intend to rewrite everything from scratch (I never used
> > this
> > > > > > phrase).
> > > > > > > >>>> There
> > > > > > > >>>>> are significant parts of code that would be moved with
> > > minimal
> > > > > > > >>>>> modifications. Second, I never said that we will get rid
> of
> > > the
> > > > > old
> > > > > > > >> tests
> > > > > > > >>>>> codebase. Every single test from Ignite 2.x should be
> moved
> > > to
> > > > > > > Ignite 3
> > > > > > > >>>>> regardless of how we choose to proceed.
> > > > > > > >>>>>
> > > > > > > >>>>> My point is that for some parts of the code a clean
> > bottom-up
> > > > > > > >>>>> implementation will be cheaper in many ways. Let me give
> > you
> > > a
> > > > > few
> > > > > > > >>>> concrete
> > > > > > > >>>>> examples:
> > > > > > > >>>>>
> > > > > > > >>>>> - I think no one can object that we need a cleanly
> > separated
> > > > > > > >>>> persistence
> > > > > > > >>>>> layer for Ignite. There is a very raw draft IEP for this
> > > > > already.
> > > > > > On
> > > > > > > >>>> the
> > > > > > > >>>>> other hand, I think we also can agree that we need a
> > > > split-brain
> > > > > > > >>>>> resistant
> > > > > > > >>>>> replication protocol for caches. There is also an IEP
> for
> > > > this.
> > > > > > > >>>> Neither
> > > > > > > >>>>> of
> > > > > > > >>>>> the changes is a good fit for 2.x because they are
> likely
> > to
> > > > > > > >> introduce
> > > > > > > >>>>> breaking changes in the persistence layer, configuration
> > and
> > > > > > > >> behavior.
> > > > > > > >>>>> Additionally, these components are now tightly coupled,
> so
> > > > there
> > > > > > is
> > > > > > > >> no
> > > > > > > >>>>> way
> > > > > > > >>>>> these two changes can be implemented in parallel and
> then
> > > > merged
> > > > > > > >>>>> together
> > > > > > > >>>>> easily. So what we will end up with is having to
> implement
> > > > these
> > > > > > > >>>> changes
> > > > > > > >>>>> sequentially, fixing all existing tests twice, and
> > > essentially
> > > > > > > >>>> throwing
> > > > > > > >>>>> away half of the work done because the other part of the
> > > > change
> > > > > is
> > > > > > > >>>>> re-implemented
> > > > > > > >>>>> - Similar example goes with getting rid of
> > > > IgniteInternalFuture
> > > > > > and
> > > > > > > >>>>> replacing it with CompletableFuture, and any other
> change
> > > that
> > > > > > > >> touches
> > > > > > > >>>>> the
> > > > > > > >>>>> asynchronous part of the code.
> > > > > > > >>>>>
> > > > > > > >>>>> Third, I do not see how this choice affects the UX of
> > Ignite.
> > > > The
> > > > > > end
> > > > > > > >>>> user
> > > > > > > >>>>> experience must be fixed in the IEP regardless of the
> > > > development
> > > > > > > >> process
> > > > > > > >>>>> and the fact that we have gaps in this area in Ignite 2.x
> > > just
> > > > > > > confirms
> > > > > > > >>>>> that.
> > > > > > > >>>>>
> > > > > > > >>>>> Pavel, agree that a repo/branch is a technicality, I
> guess
> > if
> > > > > > > >>>> reformulate,
> > > > > > > >>>>> my point is that we might agree to have a single
> > development
> > > > > master
> > > > > > > >>>> branch
> > > > > > > >>>>> with 'disassembled' end-to-end functionality for some
> > period
> > > of
> > > > > > time
> > > > > > > to
> > > > > > > >>>>> speed up development, and re-assemble the core features
> > after
> > > > > > having
> > > > > > > >>>>> submodules tested independently.
> > > > > > > >>>>>
> > > > > > > >>>>> Nikolay,
> > > > > > > >>>>>> We have many features that have to evolve.
> > > > > > > >>>>>> Snapshots, rebalance, tooling, tracing, zookeeper
> support,
> > > > etc.
> > > > > > > >>>>> This is not very specific. In the end, resources are
> > limited
> > > > and
> > > > > we
> > > > > > > >> will
> > > > > > > >>>>> not be able to drive both tracks simultaneously,
> especially
> > > > > after a
> > > > > > > >>>> couple
> > > > > > > >>>>> of features having been implemented for Ignite 3.0. If
> > there
> > > > are
> > > > > > > indeed
> > > > > > > >>>>> some major changes that we want to do in Ignite 2.x
> instead
> > > of
> > > > > > > putting
> > > > > > > >>>>> effort into 3.0 - let's discuss them. I am just not aware
> > of
> > > > any,
> > > > > > > >> that's
> > > > > > > >>>>> why I am eager to move forward with Ignite 3.0.
> > > > > > > >>>>>
> > > > > > > >>>>>> We have bugs and issues that can be fixed in 2.x without
> > > > > breaking
> > > > > > > >>>> backward
> > > > > > > >>>>> compatibility.
> > > > > > > >>>>>> We have many users who are happy with the 2.x with all
> > it’s
> > > > > > issues.
> > > > > > > >>>>> These changes will be covered by end-to-end tests and
> > > migrated
> > > > to
> > > > > > > >> Ignite
> > > > > > > >>>>> 3.0, so I see no issues here.
> > > > > > > >>>>>
> > > > > > > >>>>> Finally, Anton & Nikolay
> > > > > > > >>>>> I do not have an estimate for this simply because the
> > > activity
> > > > is
> > > > > > > >>>>> community-driven and it depends on the number of people
> > > willing
> > > > > to
> > > > > > > >>>>> contribute. With the current pace, I would hope to have
> an
> > RC
> > > > of
> > > > > > > Ignite
> > > > > > > >>>> 3.0
> > > > > > > >>>>> to be ready by the end of 2021. My gut feeling is that by
> > > > moving
> > > > > > with
> > > > > > > >>>>> incremental changes, we will not be able to implement
> even
> > > half
> > > > > of
> > > > > > > the
> > > > > > > >>>>> wishlist by that time.
> > > > > > > >>>>> I doubt that releasing several major releases with
> breaking
> > > > > changes
> > > > > > > >> will
> > > > > > > >>>>> make Ignite users happy either because each upgrade will
> > cost
> > > > > > Ignite
> > > > > > > >>>> users
> > > > > > > >>>>> money, so the fewer major versions we release, the
> better.
> > > Thus
> > > > > my
> > > > > > > wish
> > > > > > > >>>> to
> > > > > > > >>>>> include all breaking changes in one release.
> > > > > > > >>>>>
> > > > > > > >>>>> I'll be now quiet for a while, let's see what other
> > community
> > > > > > members
> > > > > > > >>>>> think.
> > > > > > > >>>>>
> > > > > > > >>>>> пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <
> > > > > ptupitsyn@apache.org
> > > > > > >:
> > > > > > > >>>>>
> > > > > > > >>>>>> 1. Rewriting from scratch is never a good idea.
> > > > > > > >>>>>> We don't want to follow the path of Netscape and lose
> all
> > > our
> > > > > > users
> > > > > > > >>>>>> by the time we have a working 3.0 [1]
> > > > > > > >>>>>>
> > > > > > > >>>>>> 2. Not sure about new repo - seems like some pain and no
> > > gain,
> > > > > > > what's
> > > > > > > >>>> the
> > > > > > > >>>>>> problem with a branch?
> > > > > > > >>>>>>
> > > > > > > >>>>>> 3. We should keep existing integration tests when
> > possible.
> > > > > > > >>>>>> We have accumulated a lot of edge case knowledge over
> the
> > > > years,
> > > > > > > >>>>>> it is not a good idea to send all of that down the
> drain.
> > > > > > > >>>>>> Yes, integration tests are slow, but they are the most
> > > > valuable.
> > > > > > > >>>>>> I think we can move more stuff into nightly runs and
> have
> > a
> > > > fast
> > > > > > and
> > > > > > > >>>>> modern
> > > > > > > >>>>>> basic suite.
> > > > > > > >>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>>> Alexey, you are much more familiar with the Ignite core
> > > > codebase
> > > > > > > than
> > > > > > > >>>>> most
> > > > > > > >>>>>> of us,
> > > > > > > >>>>>> can you please explain in more detail which particular
> > > > feature,
> > > > > in
> > > > > > > >> your
> > > > > > > >>>>>> opinion,
> > > > > > > >>>>>> mandates this "start from scratch" approach?
> > > > > > > >>>>>> Is it really not possible at all to follow a less
> radical
> > > way?
> > > > > > > >>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>>> [1]
> > > > > > > >>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>>
> > > > > > > >>>>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
> > > > > > > >>>>>>
> > > > > > > >>>>>> On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <
> > > > > > nizhikov@apache.org
> > > > > > > >
> > > > > > > >>>>>> wrote:
> > > > > > > >>>>>>
> > > > > > > >>>>>>> Hello, Alexey.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> I think that «rewriting from scratch» approach has a
> high
> > > > risk
> > > > > to
> > > > > > > >>>> make
> > > > > > > >>>>>> new
> > > > > > > >>>>>>> features unusable.
> > > > > > > >>>>>>> At the time Ignite2 was started no-one wants to do bad
> UX
> > > or
> > > > > bad
> > > > > > > >>>>>> features.
> > > > > > > >>>>>>> Nevertheless, it happen.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> I think we can avoid it with the Ignite3 and successors
> > if
> > > we
> > > > > > will
> > > > > > > >>>> move
> > > > > > > >>>>>>> step by step without keeping backward compatibility
> > > > > > > >>>>>>> With the step by step approach, we can focus on each
> > > > component
> > > > > > > >>>>>> separately.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>> What new features are we planning to implement for
> > Ignite
> > > > 2.x?
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> We have many features that have to evolve.
> > > > > > > >>>>>>> Snapshots, rebalance, tooling, tracing, zookeeper
> > support,
> > > > etc.
> > > > > > > >>>>>>> We have bugs and issues that can be fixed in 2.x
> without
> > > > > breaking
> > > > > > > >>>>>> backward
> > > > > > > >>>>>>> compatibility.
> > > > > > > >>>>>>> We have many users who are happy with the 2.x with all
> > it’s
> > > > > > issues.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>> 2 нояб. 2020 г., в 14:09, Anton Vinogradov <
> > av@apache.org
> > > >
> > > > > > > >>>>> написал(а):
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> Alexey,
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> Do we have any estimates of how fast we'll be able to
> > gain
> > > > > > > >>>>>>> production-ready
> > > > > > > >>>>>>>> AI 3.0 in case of a "new repo" choice?
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk <
> > > > > > > >>>>>>> alexey.goncharuk@gmail.com>
> > > > > > > >>>>>>>> wrote:
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>>> Nikolay,
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> What new features are we planning to implement for
> > Ignite
> > > > > 2.x?
> > > > > > I
> > > > > > > >>>>> think
> > > > > > > >>>>>>> once
> > > > > > > >>>>>>>>> we commence working on Ignite 3.0, we should
> gradually
> > > > cease
> > > > > > the
> > > > > > > >>>>>>> activity
> > > > > > > >>>>>>>>> on Ignite 2.x to mere bugfixes because such parallel
> > > > > > development
> > > > > > > >>>>> will
> > > > > > > >>>>>> be
> > > > > > > >>>>>>>>> overwhelming regardless of how we choose to proceed.
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <
> > > > > > > nizhikov@apache.org
> > > > > > > >>>>> :
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>>> To be clear:
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>>> I would suggest creating a new repository for
> Ignite
> > > 3.0
> > > > > > > >>>>> (perhaps, a
> > > > > > > >>>>>>>>> new
> > > > > > > >>>>>>>>>> clean branch, but a new repo looks nicer to me) and
> a
> > > new
> > > > > > Ignite
> > > > > > > >>>>> 3.0
> > > > > > > >>>>>>>>>> TeamCity project.
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> +1 for new Team City project.
> > > > > > > >>>>>>>>>> +1 for new branch for Ignite3.
> > > > > > > >>>>>>>>>> -1 for new repo.
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <
> > > > > > > >>>> NIzhikov.dev@gmail.com
> > > > > > > >>>>>>
> > > > > > > >>>>>>>>>> написал(а):
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> Hello, Alexey.
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> I think it will hurt our project more than help.
> > > > > > > >>>>>>>>>>> Developing new features for 2 separate branches
> with
> > > the
> > > > > > > >>>> different
> > > > > > > >>>>>>> APIs
> > > > > > > >>>>>>>>>> and internal structure is overwhelming
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> Maybe we should relax a bit requirements for
> Ignite3?
> > > > > > > >>>>>>>>>>> Maybe we should move step by step and make Ignite3
> > with
> > > > new
> > > > > > > >>>>>>>>>> configuration than Ignite4 with new transactions,
> etc?
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <
> > > > > > > >>>>>>>>> alexey.goncharuk@gmail.com>
> > > > > > > >>>>>>>>>> написал(а):
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> Igniters,
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> I wanted to pitch a rather radical idea regarding
> > the
> > > > > Ignite
> > > > > > > >>>> 3.0
> > > > > > > >>>>>>>>>>>> development which has occurred to me some time
> ago.
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> We already have several IEPs targeted to Ignite
> 3.0
> > > > which
> > > > > > > imply
> > > > > > > >>>>>> major
> > > > > > > >>>>>>>>>>>> changes to the codebase (the change in replication
> > > > > protocol
> > > > > > > and
> > > > > > > >>>>>> thus
> > > > > > > >>>>>>>>>>>> transactions, change in binary format, updated
> > > > > metastorage,
> > > > > > > >>>> etc).
> > > > > > > >>>>>> We
> > > > > > > >>>>>>>>>> also
> > > > > > > >>>>>>>>>>>> planned significant changes in public APIs:
> > > > configuration
> > > > > > > >>>> format
> > > > > > > >>>>>>>>> change,
> > > > > > > >>>>>>>>>>>> improvements in cache APIs, SQL APIs, transaction
> > mode
> > > > > > rework.
> > > > > > > >>>>> The
> > > > > > > >>>>>>>>>> wishlist
> > > > > > > >>>>>>>>>>>> of changes for Ignite 3.0 is huge.
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> So, I was wondering whether it makes sense to try
> to
> > > > > change
> > > > > > > the
> > > > > > > >>>>> old
> > > > > > > >>>>>>>>>>>> codebase, or start with a new baseline and move
> old
> > > > pieces
> > > > > > of
> > > > > > > >>>>> code
> > > > > > > >>>>>>>>> that
> > > > > > > >>>>>>>>>> do
> > > > > > > >>>>>>>>>>>> not require significant rework. Personally, I
> would
> > go
> > > > > with
> > > > > > > the
> > > > > > > >>>>>>> second
> > > > > > > >>>>>>>>>>>> option for the following reasons:
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> - We have a chance to shift the development
> paradigm
> > > in
> > > > > the
> > > > > > > >>>>> project
> > > > > > > >>>>>>>>> and
> > > > > > > >>>>>>>>>>>> introduce the practice of true unit-tests. In the
> > new
> > > > > > baseline
> > > > > > > >>>> at
> > > > > > > >>>>>> the
> > > > > > > >>>>>>>>>>>> beginning there will be no ability to run an
> > > end-to-end
> > > > > > > >>>> scenario,
> > > > > > > >>>>>>>>> thus
> > > > > > > >>>>>>>>>> we
> > > > > > > >>>>>>>>>>>> will be forced to write unit-tests. So far, such
> > > > practice
> > > > > > was
> > > > > > > >>>>> hard
> > > > > > > >>>>>> to
> > > > > > > >>>>>>>>>>>> implement because of tight coupling between Ignite
> > > > > > components
> > > > > > > >>>> and
> > > > > > > >>>>>>>>>> inability
> > > > > > > >>>>>>>>>>>> to instantiate components without an instance of
> > > > > > > KernalContext.
> > > > > > > >>>>> For
> > > > > > > >>>>>>>>>>>> example, we should be able to thoroughly test
> > internal
> > > > > > > >>>>> primitives,
> > > > > > > >>>>>>>>>> such as
> > > > > > > >>>>>>>>>>>> replication protocol (without actual
> communication),
> > > > > > > >>>> distributed
> > > > > > > >>>>>>>>>>>> metastorage contracts, persistence layer, etc.
> > > > > > > >>>>>>>>>>>> - We will significantly reduce the development
> cycle
> > > in
> > > > > the
> > > > > > > >>>>>> beginning
> > > > > > > >>>>>>>>>>>> (right now the RunAll takes two hours of
> > astronomical
> > > > time
> > > > > > > with
> > > > > > > >>>>>> empty
> > > > > > > >>>>>>>>>> TC;
> > > > > > > >>>>>>>>>>>> in the new approach developer will be able to run
> > ALL
> > > > > tests
> > > > > > > >>>>> locally
> > > > > > > >>>>>>>>> in
> > > > > > > >>>>>>>>>> a
> > > > > > > >>>>>>>>>>>> matter of minutes)
> > > > > > > >>>>>>>>>>>> - We can get rid of TC bot and enforce green TC by
> > > > > > integrating
> > > > > > > >>>> TC
> > > > > > > >>>>>>>>> build
> > > > > > > >>>>>>>>>>>> results with GitHub PRs (the same way Travis is
> > > > currently
> > > > > > > >>>>>> integrated
> > > > > > > >>>>>>>>>> to PR
> > > > > > > >>>>>>>>>>>> check). We should restrict PR merge without a TC
> > check
> > > > > > > >>>>>>>>>>>> - We will still have to re-write all tests, but
> only
> > > > once.
> > > > > > If
> > > > > > > >>>> we
> > > > > > > >>>>>> try
> > > > > > > >>>>>>>>> to
> > > > > > > >>>>>>>>>>>> modify the old codebase, we would need to modify
> all
> > > the
> > > > > > tests
> > > > > > > >>>>> for
> > > > > > > >>>>>>>>>> every
> > > > > > > >>>>>>>>>>>> major change (public API change, configuration
> > change)
> > > > > > > >>>>>>>>>>>> - We will have fewer conflicts when working
> > together.
> > > > For
> > > > > > > >>>>> example,
> > > > > > > >>>>>> I
> > > > > > > >>>>>>>>>>>> cannot imagine how one would merge two changes of
> > > > getting
> > > > > > rid
> > > > > > > >>>> of
> > > > > > > >>>>>>>>>>>> IgniteFuture and changes in replication protocol,
> > for
> > > > > > example
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> Technically, I would suggest creating a new
> > repository
> > > > for
> > > > > > > >>>> Ignite
> > > > > > > >>>>>> 3.0
> > > > > > > >>>>>>>>>>>> (perhaps, a new clean branch, but a new repo looks
> > > nicer
> > > > > to
> > > > > > > me)
> > > > > > > >>>>>> and a
> > > > > > > >>>>>>>>>> new
> > > > > > > >>>>>>>>>>>> Ignite 3.0 TeamCity project.
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> While it may seem quite radical, I do believe that
> > > this
> > > > > > > >>>> approach
> > > > > > > >>>>>> will
> > > > > > > >>>>>>>>>> give
> > > > > > > >>>>>>>>>>>> us more benefits than trying to make such major
> > > changes
> > > > in
> > > > > > the
> > > > > > > >>>>>>>>> existing
> > > > > > > >>>>>>>>>>>> codebase. If needed, let's schedule a discord chat
> > > like
> > > > > > before
> > > > > > > >>>> to
> > > > > > > >>>>>>>>>> discuss
> > > > > > > >>>>>>>>>>>> this.
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> WDYT?
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>>
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > > > >>>> --
> > > > > > > >>>> Best regards,
> > > > > > > >>>> Andrey V. Mashenkov
> > > > > > > >>>>
> > > > > > > >>
> > > > > > > >>
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Kseniya Romanova <ro...@gmail.com>.
Here are the slides from Alexey Goncharuk. Let's think this over and
continue on Monday:
https://go.gridgain.com/rs/491-TWR-806/images/Ignite_3_Plans_and_development_process.pdf
чт, 5 нояб. 2020 г. в 11:13, Anton Vinogradov <av...@apache.org>:
> Folks,
>
> Should we perform cleanup work before (r)evolutional changes?
> My huge proposal is to get rid of things which we don't need anyway
> - local caches,
> - strange tx modes,
> - code overcomplexity because of RollingUpgrade feature never attended at
> AI,
> - etc,
> before choosing the way.
>
> On Tue, Nov 3, 2020 at 3:31 PM Valentin Kulichenko <
> valentin.kulichenko@gmail.com> wrote:
>
> > Ksenia, thanks for scheduling this on such short notice!
> >
> > As for the original topic, I do support Alexey's idea. We're not going to
> > rewrite anything from scratch, as most of the components are going to be
> > moved as-is or with minimal modifications. However, the changes that are
> > proposed imply serious rework of the core parts of the code, which are
> not
> > properly decoupled from each other and from other parts. This makes the
> > incremental approach borderline impossible. Developing in a new repo,
> > however, addresses this concern. As a bonus, we can also refactor the
> code,
> > introduce better decoupling, get rid of kernel context, and develop unit
> > tests (finally!).
> >
> > Basically, this proposal only affects the *process*, not the set of
> changes
> > we had discussed before. Ignite 3.0 is our unique chance to make things
> > right.
> >
> > -Val
> >
> > On Tue, Nov 3, 2020 at 3:06 AM Kseniya Romanova <
> romanova.ks.spb@gmail.com
> > >
> > wrote:
> >
> > > Pavel, all the interesting points will be anyway published here in
> > English
> > > (as the principal "if it's not on devlist it doesn't happened" is still
> > > relevant). This is just a quick call for a group of developers. Later
> we
> > > can do a separate presentation of idea and discussion in English as we
> > did
> > > for the Ignite 3.0 draft of changes.
> > >
> > > вт, 3 нояб. 2020 г. в 13:52, Pavel Tupitsyn <pt...@apache.org>:
> > >
> > > > Kseniya,
> > > >
> > > > Thanks for scheduling this call.
> > > > Do you think we can switch to English if non-Russian speaking
> community
> > > > members decide to join?
> > > >
> > > > On Tue, Nov 3, 2020 at 1:32 PM Kseniya Romanova <
> > > romanova.ks.spb@gmail.com
> > > > >
> > > > wrote:
> > > >
> > > > > Let's do this community discussion open. Here's the link on zoom
> call
> > > in
> > > > > Russian for Friday 6 PM:
> > > > >
> https://www.meetup.com/Moscow-Apache-Ignite-Meetup/events/274360378/
> > > > >
> > > > > вт, 3 нояб. 2020 г. в 12:49, Nikolay Izhikov <nizhikov@apache.org
> >:
> > > > >
> > > > > > Time works for me.
> > > > > >
> > > > > > > 3 нояб. 2020 г., в 12:40, Alexey Goncharuk <
> > > > alexey.goncharuk@gmail.com
> > > > > >
> > > > > > написал(а):
> > > > > > >
> > > > > > > Nikolay,
> > > > > > >
> > > > > > > I am up for the call. I will try to explain my reasoning in
> > greater
> > > > > > detail
> > > > > > > and will be glad to hear the concerns. Will this Friday, Nov
> 6th,
> > > > work?
> > > > > > >
> > > > > > > вт, 3 нояб. 2020 г. в 10:09, Nikolay Izhikov <
> > nizhikov@apache.org
> > > >:
> > > > > > >
> > > > > > >> Igniters, should we have a call for this topic?
> > > > > > >>
> > > > > > >>> 2 нояб. 2020 г., в 18:53, Pavel Tupitsyn <
> ptupitsyn@apache.org
> > >
> > > > > > >> написал(а):
> > > > > > >>>
> > > > > > >>>> not intend to rewrite everything from scratch
> > > > > > >>>
> > > > > > >>>> Every single test from Ignite 2.x should be moved to Ignite
> 3
> > > > > > >>>> regardless of how we choose to proceed.
> > > > > > >>>
> > > > > > >>> Alexey, thank you for the explanation, this addresses all of
> my
> > > > > > concerns.
> > > > > > >>>
> > > > > > >>>
> > > > > > >>>
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> On Mon, Nov 2, 2020 at 6:43 PM Andrey Mashenkov <
> > > > > > >> andrey.mashenkov@gmail.com>
> > > > > > >>> wrote:
> > > > > > >>>
> > > > > > >>>> Hi, Igniters.
> > > > > > >>>>
> > > > > > >>>> * AFAIU, we need a new repo if we want to apply different
> > > > > restrictions
> > > > > > >> to
> > > > > > >>>> pull requests,
> > > > > > >>>> otherwise I see no difference for myself.
> > > > > > >>>> E.g. make static analysis (do we have?), compile, styles,
> and
> > > > > javadoc
> > > > > > >>>> checks mandatory.
> > > > > > >>>>
> > > > > > >>>> I think that relaxed requirements here will lead to bad
> > product
> > > > > > quality.
> > > > > > >>>>
> > > > > > >>>> * Agree with Pavel, we should 'keep' integrations tests
> > somehow.
> > > > > > >>>> During active development tests will be broken most of time,
> > so,
> > > > > > >>>> I'd port them e.g. suite-by-suite once we will have a stable
> > and
> > > > > > >> featured
> > > > > > >>>> environment to run them and of course make test's code clear
> > and
> > > > > avoid
> > > > > > >>>> bad/non-relevant ones.
> > > > > > >>>>
> > > > > > >>>> * I like bottom-up approach.
> > > > > > >>>> With it we could make a better framework. I mean clear
> > component
> > > > > > >> lifecycle,
> > > > > > >>>> component wiring mechanics, general methods to approach core
> > > > > > components
> > > > > > >>>> such as exchange/communication
> > > > > > >>>> to avoid code mess like we have in ExchangeFuture with all
> > these
> > > > > > custom
> > > > > > >>>> callbacks for each component, interfaces like
> > > > > > >>>> PartitionsExchangeAware, IgniteChangeGlobalStateSupport and
> > > > > > >>>> a pack of
> > > > > > >>
> start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected
> > > > > > >>>> and so on in various unexpected places.
> > > > > > >>>> Hope, we will be able to port most of the good code to the
> new
> > > > > > framework
> > > > > > >>>> version.
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>> On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk <
> > > > > > >>>> alexey.goncharuk@gmail.com>
> > > > > > >>>> wrote:
> > > > > > >>>>
> > > > > > >>>>> Nikolay, Pavel,
> > > > > > >>>>>
> > > > > > >>>>> Thanks for the feedback! First of all, I wanted to stress
> > that
> > > I
> > > > do
> > > > > > not
> > > > > > >>>>> intend to rewrite everything from scratch (I never used
> this
> > > > > phrase).
> > > > > > >>>> There
> > > > > > >>>>> are significant parts of code that would be moved with
> > minimal
> > > > > > >>>>> modifications. Second, I never said that we will get rid of
> > the
> > > > old
> > > > > > >> tests
> > > > > > >>>>> codebase. Every single test from Ignite 2.x should be moved
> > to
> > > > > > Ignite 3
> > > > > > >>>>> regardless of how we choose to proceed.
> > > > > > >>>>>
> > > > > > >>>>> My point is that for some parts of the code a clean
> bottom-up
> > > > > > >>>>> implementation will be cheaper in many ways. Let me give
> you
> > a
> > > > few
> > > > > > >>>> concrete
> > > > > > >>>>> examples:
> > > > > > >>>>>
> > > > > > >>>>> - I think no one can object that we need a cleanly
> separated
> > > > > > >>>> persistence
> > > > > > >>>>> layer for Ignite. There is a very raw draft IEP for this
> > > > already.
> > > > > On
> > > > > > >>>> the
> > > > > > >>>>> other hand, I think we also can agree that we need a
> > > split-brain
> > > > > > >>>>> resistant
> > > > > > >>>>> replication protocol for caches. There is also an IEP for
> > > this.
> > > > > > >>>> Neither
> > > > > > >>>>> of
> > > > > > >>>>> the changes is a good fit for 2.x because they are likely
> to
> > > > > > >> introduce
> > > > > > >>>>> breaking changes in the persistence layer, configuration
> and
> > > > > > >> behavior.
> > > > > > >>>>> Additionally, these components are now tightly coupled, so
> > > there
> > > > > is
> > > > > > >> no
> > > > > > >>>>> way
> > > > > > >>>>> these two changes can be implemented in parallel and then
> > > merged
> > > > > > >>>>> together
> > > > > > >>>>> easily. So what we will end up with is having to implement
> > > these
> > > > > > >>>> changes
> > > > > > >>>>> sequentially, fixing all existing tests twice, and
> > essentially
> > > > > > >>>> throwing
> > > > > > >>>>> away half of the work done because the other part of the
> > > change
> > > > is
> > > > > > >>>>> re-implemented
> > > > > > >>>>> - Similar example goes with getting rid of
> > > IgniteInternalFuture
> > > > > and
> > > > > > >>>>> replacing it with CompletableFuture, and any other change
> > that
> > > > > > >> touches
> > > > > > >>>>> the
> > > > > > >>>>> asynchronous part of the code.
> > > > > > >>>>>
> > > > > > >>>>> Third, I do not see how this choice affects the UX of
> Ignite.
> > > The
> > > > > end
> > > > > > >>>> user
> > > > > > >>>>> experience must be fixed in the IEP regardless of the
> > > development
> > > > > > >> process
> > > > > > >>>>> and the fact that we have gaps in this area in Ignite 2.x
> > just
> > > > > > confirms
> > > > > > >>>>> that.
> > > > > > >>>>>
> > > > > > >>>>> Pavel, agree that a repo/branch is a technicality, I guess
> if
> > > > > > >>>> reformulate,
> > > > > > >>>>> my point is that we might agree to have a single
> development
> > > > master
> > > > > > >>>> branch
> > > > > > >>>>> with 'disassembled' end-to-end functionality for some
> period
> > of
> > > > > time
> > > > > > to
> > > > > > >>>>> speed up development, and re-assemble the core features
> after
> > > > > having
> > > > > > >>>>> submodules tested independently.
> > > > > > >>>>>
> > > > > > >>>>> Nikolay,
> > > > > > >>>>>> We have many features that have to evolve.
> > > > > > >>>>>> Snapshots, rebalance, tooling, tracing, zookeeper support,
> > > etc.
> > > > > > >>>>> This is not very specific. In the end, resources are
> limited
> > > and
> > > > we
> > > > > > >> will
> > > > > > >>>>> not be able to drive both tracks simultaneously, especially
> > > > after a
> > > > > > >>>> couple
> > > > > > >>>>> of features having been implemented for Ignite 3.0. If
> there
> > > are
> > > > > > indeed
> > > > > > >>>>> some major changes that we want to do in Ignite 2.x instead
> > of
> > > > > > putting
> > > > > > >>>>> effort into 3.0 - let's discuss them. I am just not aware
> of
> > > any,
> > > > > > >> that's
> > > > > > >>>>> why I am eager to move forward with Ignite 3.0.
> > > > > > >>>>>
> > > > > > >>>>>> We have bugs and issues that can be fixed in 2.x without
> > > > breaking
> > > > > > >>>> backward
> > > > > > >>>>> compatibility.
> > > > > > >>>>>> We have many users who are happy with the 2.x with all
> it’s
> > > > > issues.
> > > > > > >>>>> These changes will be covered by end-to-end tests and
> > migrated
> > > to
> > > > > > >> Ignite
> > > > > > >>>>> 3.0, so I see no issues here.
> > > > > > >>>>>
> > > > > > >>>>> Finally, Anton & Nikolay
> > > > > > >>>>> I do not have an estimate for this simply because the
> > activity
> > > is
> > > > > > >>>>> community-driven and it depends on the number of people
> > willing
> > > > to
> > > > > > >>>>> contribute. With the current pace, I would hope to have an
> RC
> > > of
> > > > > > Ignite
> > > > > > >>>> 3.0
> > > > > > >>>>> to be ready by the end of 2021. My gut feeling is that by
> > > moving
> > > > > with
> > > > > > >>>>> incremental changes, we will not be able to implement even
> > half
> > > > of
> > > > > > the
> > > > > > >>>>> wishlist by that time.
> > > > > > >>>>> I doubt that releasing several major releases with breaking
> > > > changes
> > > > > > >> will
> > > > > > >>>>> make Ignite users happy either because each upgrade will
> cost
> > > > > Ignite
> > > > > > >>>> users
> > > > > > >>>>> money, so the fewer major versions we release, the better.
> > Thus
> > > > my
> > > > > > wish
> > > > > > >>>> to
> > > > > > >>>>> include all breaking changes in one release.
> > > > > > >>>>>
> > > > > > >>>>> I'll be now quiet for a while, let's see what other
> community
> > > > > members
> > > > > > >>>>> think.
> > > > > > >>>>>
> > > > > > >>>>> пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <
> > > > ptupitsyn@apache.org
> > > > > >:
> > > > > > >>>>>
> > > > > > >>>>>> 1. Rewriting from scratch is never a good idea.
> > > > > > >>>>>> We don't want to follow the path of Netscape and lose all
> > our
> > > > > users
> > > > > > >>>>>> by the time we have a working 3.0 [1]
> > > > > > >>>>>>
> > > > > > >>>>>> 2. Not sure about new repo - seems like some pain and no
> > gain,
> > > > > > what's
> > > > > > >>>> the
> > > > > > >>>>>> problem with a branch?
> > > > > > >>>>>>
> > > > > > >>>>>> 3. We should keep existing integration tests when
> possible.
> > > > > > >>>>>> We have accumulated a lot of edge case knowledge over the
> > > years,
> > > > > > >>>>>> it is not a good idea to send all of that down the drain.
> > > > > > >>>>>> Yes, integration tests are slow, but they are the most
> > > valuable.
> > > > > > >>>>>> I think we can move more stuff into nightly runs and have
> a
> > > fast
> > > > > and
> > > > > > >>>>> modern
> > > > > > >>>>>> basic suite.
> > > > > > >>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>> Alexey, you are much more familiar with the Ignite core
> > > codebase
> > > > > > than
> > > > > > >>>>> most
> > > > > > >>>>>> of us,
> > > > > > >>>>>> can you please explain in more detail which particular
> > > feature,
> > > > in
> > > > > > >> your
> > > > > > >>>>>> opinion,
> > > > > > >>>>>> mandates this "start from scratch" approach?
> > > > > > >>>>>> Is it really not possible at all to follow a less radical
> > way?
> > > > > > >>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>> [1]
> > > > > > >>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>
> > > > > > >>>>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
> > > > > > >>>>>>
> > > > > > >>>>>> On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <
> > > > > nizhikov@apache.org
> > > > > > >
> > > > > > >>>>>> wrote:
> > > > > > >>>>>>
> > > > > > >>>>>>> Hello, Alexey.
> > > > > > >>>>>>>
> > > > > > >>>>>>> I think that «rewriting from scratch» approach has a high
> > > risk
> > > > to
> > > > > > >>>> make
> > > > > > >>>>>> new
> > > > > > >>>>>>> features unusable.
> > > > > > >>>>>>> At the time Ignite2 was started no-one wants to do bad UX
> > or
> > > > bad
> > > > > > >>>>>> features.
> > > > > > >>>>>>> Nevertheless, it happen.
> > > > > > >>>>>>>
> > > > > > >>>>>>> I think we can avoid it with the Ignite3 and successors
> if
> > we
> > > > > will
> > > > > > >>>> move
> > > > > > >>>>>>> step by step without keeping backward compatibility
> > > > > > >>>>>>> With the step by step approach, we can focus on each
> > > component
> > > > > > >>>>>> separately.
> > > > > > >>>>>>>
> > > > > > >>>>>>>> What new features are we planning to implement for
> Ignite
> > > 2.x?
> > > > > > >>>>>>>
> > > > > > >>>>>>> We have many features that have to evolve.
> > > > > > >>>>>>> Snapshots, rebalance, tooling, tracing, zookeeper
> support,
> > > etc.
> > > > > > >>>>>>> We have bugs and issues that can be fixed in 2.x without
> > > > breaking
> > > > > > >>>>>> backward
> > > > > > >>>>>>> compatibility.
> > > > > > >>>>>>> We have many users who are happy with the 2.x with all
> it’s
> > > > > issues.
> > > > > > >>>>>>>
> > > > > > >>>>>>>> 2 нояб. 2020 г., в 14:09, Anton Vinogradov <
> av@apache.org
> > >
> > > > > > >>>>> написал(а):
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Alexey,
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Do we have any estimates of how fast we'll be able to
> gain
> > > > > > >>>>>>> production-ready
> > > > > > >>>>>>>> AI 3.0 in case of a "new repo" choice?
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk <
> > > > > > >>>>>>> alexey.goncharuk@gmail.com>
> > > > > > >>>>>>>> wrote:
> > > > > > >>>>>>>>
> > > > > > >>>>>>>>> Nikolay,
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> What new features are we planning to implement for
> Ignite
> > > > 2.x?
> > > > > I
> > > > > > >>>>> think
> > > > > > >>>>>>> once
> > > > > > >>>>>>>>> we commence working on Ignite 3.0, we should gradually
> > > cease
> > > > > the
> > > > > > >>>>>>> activity
> > > > > > >>>>>>>>> on Ignite 2.x to mere bugfixes because such parallel
> > > > > development
> > > > > > >>>>> will
> > > > > > >>>>>> be
> > > > > > >>>>>>>>> overwhelming regardless of how we choose to proceed.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <
> > > > > > nizhikov@apache.org
> > > > > > >>>>> :
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>>> To be clear:
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>>> I would suggest creating a new repository for Ignite
> > 3.0
> > > > > > >>>>> (perhaps, a
> > > > > > >>>>>>>>> new
> > > > > > >>>>>>>>>> clean branch, but a new repo looks nicer to me) and a
> > new
> > > > > Ignite
> > > > > > >>>>> 3.0
> > > > > > >>>>>>>>>> TeamCity project.
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> +1 for new Team City project.
> > > > > > >>>>>>>>>> +1 for new branch for Ignite3.
> > > > > > >>>>>>>>>> -1 for new repo.
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <
> > > > > > >>>> NIzhikov.dev@gmail.com
> > > > > > >>>>>>
> > > > > > >>>>>>>>>> написал(а):
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> Hello, Alexey.
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> I think it will hurt our project more than help.
> > > > > > >>>>>>>>>>> Developing new features for 2 separate branches with
> > the
> > > > > > >>>> different
> > > > > > >>>>>>> APIs
> > > > > > >>>>>>>>>> and internal structure is overwhelming
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> Maybe we should relax a bit requirements for Ignite3?
> > > > > > >>>>>>>>>>> Maybe we should move step by step and make Ignite3
> with
> > > new
> > > > > > >>>>>>>>>> configuration than Ignite4 with new transactions, etc?
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <
> > > > > > >>>>>>>>> alexey.goncharuk@gmail.com>
> > > > > > >>>>>>>>>> написал(а):
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Igniters,
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> I wanted to pitch a rather radical idea regarding
> the
> > > > Ignite
> > > > > > >>>> 3.0
> > > > > > >>>>>>>>>>>> development which has occurred to me some time ago.
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> We already have several IEPs targeted to Ignite 3.0
> > > which
> > > > > > imply
> > > > > > >>>>>> major
> > > > > > >>>>>>>>>>>> changes to the codebase (the change in replication
> > > > protocol
> > > > > > and
> > > > > > >>>>>> thus
> > > > > > >>>>>>>>>>>> transactions, change in binary format, updated
> > > > metastorage,
> > > > > > >>>> etc).
> > > > > > >>>>>> We
> > > > > > >>>>>>>>>> also
> > > > > > >>>>>>>>>>>> planned significant changes in public APIs:
> > > configuration
> > > > > > >>>> format
> > > > > > >>>>>>>>> change,
> > > > > > >>>>>>>>>>>> improvements in cache APIs, SQL APIs, transaction
> mode
> > > > > rework.
> > > > > > >>>>> The
> > > > > > >>>>>>>>>> wishlist
> > > > > > >>>>>>>>>>>> of changes for Ignite 3.0 is huge.
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> So, I was wondering whether it makes sense to try to
> > > > change
> > > > > > the
> > > > > > >>>>> old
> > > > > > >>>>>>>>>>>> codebase, or start with a new baseline and move old
> > > pieces
> > > > > of
> > > > > > >>>>> code
> > > > > > >>>>>>>>> that
> > > > > > >>>>>>>>>> do
> > > > > > >>>>>>>>>>>> not require significant rework. Personally, I would
> go
> > > > with
> > > > > > the
> > > > > > >>>>>>> second
> > > > > > >>>>>>>>>>>> option for the following reasons:
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> - We have a chance to shift the development paradigm
> > in
> > > > the
> > > > > > >>>>> project
> > > > > > >>>>>>>>> and
> > > > > > >>>>>>>>>>>> introduce the practice of true unit-tests. In the
> new
> > > > > baseline
> > > > > > >>>> at
> > > > > > >>>>>> the
> > > > > > >>>>>>>>>>>> beginning there will be no ability to run an
> > end-to-end
> > > > > > >>>> scenario,
> > > > > > >>>>>>>>> thus
> > > > > > >>>>>>>>>> we
> > > > > > >>>>>>>>>>>> will be forced to write unit-tests. So far, such
> > > practice
> > > > > was
> > > > > > >>>>> hard
> > > > > > >>>>>> to
> > > > > > >>>>>>>>>>>> implement because of tight coupling between Ignite
> > > > > components
> > > > > > >>>> and
> > > > > > >>>>>>>>>> inability
> > > > > > >>>>>>>>>>>> to instantiate components without an instance of
> > > > > > KernalContext.
> > > > > > >>>>> For
> > > > > > >>>>>>>>>>>> example, we should be able to thoroughly test
> internal
> > > > > > >>>>> primitives,
> > > > > > >>>>>>>>>> such as
> > > > > > >>>>>>>>>>>> replication protocol (without actual communication),
> > > > > > >>>> distributed
> > > > > > >>>>>>>>>>>> metastorage contracts, persistence layer, etc.
> > > > > > >>>>>>>>>>>> - We will significantly reduce the development cycle
> > in
> > > > the
> > > > > > >>>>>> beginning
> > > > > > >>>>>>>>>>>> (right now the RunAll takes two hours of
> astronomical
> > > time
> > > > > > with
> > > > > > >>>>>> empty
> > > > > > >>>>>>>>>> TC;
> > > > > > >>>>>>>>>>>> in the new approach developer will be able to run
> ALL
> > > > tests
> > > > > > >>>>> locally
> > > > > > >>>>>>>>> in
> > > > > > >>>>>>>>>> a
> > > > > > >>>>>>>>>>>> matter of minutes)
> > > > > > >>>>>>>>>>>> - We can get rid of TC bot and enforce green TC by
> > > > > integrating
> > > > > > >>>> TC
> > > > > > >>>>>>>>> build
> > > > > > >>>>>>>>>>>> results with GitHub PRs (the same way Travis is
> > > currently
> > > > > > >>>>>> integrated
> > > > > > >>>>>>>>>> to PR
> > > > > > >>>>>>>>>>>> check). We should restrict PR merge without a TC
> check
> > > > > > >>>>>>>>>>>> - We will still have to re-write all tests, but only
> > > once.
> > > > > If
> > > > > > >>>> we
> > > > > > >>>>>> try
> > > > > > >>>>>>>>> to
> > > > > > >>>>>>>>>>>> modify the old codebase, we would need to modify all
> > the
> > > > > tests
> > > > > > >>>>> for
> > > > > > >>>>>>>>>> every
> > > > > > >>>>>>>>>>>> major change (public API change, configuration
> change)
> > > > > > >>>>>>>>>>>> - We will have fewer conflicts when working
> together.
> > > For
> > > > > > >>>>> example,
> > > > > > >>>>>> I
> > > > > > >>>>>>>>>>>> cannot imagine how one would merge two changes of
> > > getting
> > > > > rid
> > > > > > >>>> of
> > > > > > >>>>>>>>>>>> IgniteFuture and changes in replication protocol,
> for
> > > > > example
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Technically, I would suggest creating a new
> repository
> > > for
> > > > > > >>>> Ignite
> > > > > > >>>>>> 3.0
> > > > > > >>>>>>>>>>>> (perhaps, a new clean branch, but a new repo looks
> > nicer
> > > > to
> > > > > > me)
> > > > > > >>>>>> and a
> > > > > > >>>>>>>>>> new
> > > > > > >>>>>>>>>>>> Ignite 3.0 TeamCity project.
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> While it may seem quite radical, I do believe that
> > this
> > > > > > >>>> approach
> > > > > > >>>>>> will
> > > > > > >>>>>>>>>> give
> > > > > > >>>>>>>>>>>> us more benefits than trying to make such major
> > changes
> > > in
> > > > > the
> > > > > > >>>>>>>>> existing
> > > > > > >>>>>>>>>>>> codebase. If needed, let's schedule a discord chat
> > like
> > > > > before
> > > > > > >>>> to
> > > > > > >>>>>>>>>> discuss
> > > > > > >>>>>>>>>>>> this.
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> WDYT?
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>> --
> > > > > > >>>> Best regards,
> > > > > > >>>> Andrey V. Mashenkov
> > > > > > >>>>
> > > > > > >>
> > > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Anton Vinogradov <av...@apache.org>.
Folks,
Should we perform cleanup work before (r)evolutional changes?
My huge proposal is to get rid of things which we don't need anyway
- local caches,
- strange tx modes,
- code overcomplexity because of RollingUpgrade feature never attended at
AI,
- etc,
before choosing the way.
On Tue, Nov 3, 2020 at 3:31 PM Valentin Kulichenko <
valentin.kulichenko@gmail.com> wrote:
> Ksenia, thanks for scheduling this on such short notice!
>
> As for the original topic, I do support Alexey's idea. We're not going to
> rewrite anything from scratch, as most of the components are going to be
> moved as-is or with minimal modifications. However, the changes that are
> proposed imply serious rework of the core parts of the code, which are not
> properly decoupled from each other and from other parts. This makes the
> incremental approach borderline impossible. Developing in a new repo,
> however, addresses this concern. As a bonus, we can also refactor the code,
> introduce better decoupling, get rid of kernel context, and develop unit
> tests (finally!).
>
> Basically, this proposal only affects the *process*, not the set of changes
> we had discussed before. Ignite 3.0 is our unique chance to make things
> right.
>
> -Val
>
> On Tue, Nov 3, 2020 at 3:06 AM Kseniya Romanova <romanova.ks.spb@gmail.com
> >
> wrote:
>
> > Pavel, all the interesting points will be anyway published here in
> English
> > (as the principal "if it's not on devlist it doesn't happened" is still
> > relevant). This is just a quick call for a group of developers. Later we
> > can do a separate presentation of idea and discussion in English as we
> did
> > for the Ignite 3.0 draft of changes.
> >
> > вт, 3 нояб. 2020 г. в 13:52, Pavel Tupitsyn <pt...@apache.org>:
> >
> > > Kseniya,
> > >
> > > Thanks for scheduling this call.
> > > Do you think we can switch to English if non-Russian speaking community
> > > members decide to join?
> > >
> > > On Tue, Nov 3, 2020 at 1:32 PM Kseniya Romanova <
> > romanova.ks.spb@gmail.com
> > > >
> > > wrote:
> > >
> > > > Let's do this community discussion open. Here's the link on zoom call
> > in
> > > > Russian for Friday 6 PM:
> > > > https://www.meetup.com/Moscow-Apache-Ignite-Meetup/events/274360378/
> > > >
> > > > вт, 3 нояб. 2020 г. в 12:49, Nikolay Izhikov <ni...@apache.org>:
> > > >
> > > > > Time works for me.
> > > > >
> > > > > > 3 нояб. 2020 г., в 12:40, Alexey Goncharuk <
> > > alexey.goncharuk@gmail.com
> > > > >
> > > > > написал(а):
> > > > > >
> > > > > > Nikolay,
> > > > > >
> > > > > > I am up for the call. I will try to explain my reasoning in
> greater
> > > > > detail
> > > > > > and will be glad to hear the concerns. Will this Friday, Nov 6th,
> > > work?
> > > > > >
> > > > > > вт, 3 нояб. 2020 г. в 10:09, Nikolay Izhikov <
> nizhikov@apache.org
> > >:
> > > > > >
> > > > > >> Igniters, should we have a call for this topic?
> > > > > >>
> > > > > >>> 2 нояб. 2020 г., в 18:53, Pavel Tupitsyn <ptupitsyn@apache.org
> >
> > > > > >> написал(а):
> > > > > >>>
> > > > > >>>> not intend to rewrite everything from scratch
> > > > > >>>
> > > > > >>>> Every single test from Ignite 2.x should be moved to Ignite 3
> > > > > >>>> regardless of how we choose to proceed.
> > > > > >>>
> > > > > >>> Alexey, thank you for the explanation, this addresses all of my
> > > > > concerns.
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > > > > >>> On Mon, Nov 2, 2020 at 6:43 PM Andrey Mashenkov <
> > > > > >> andrey.mashenkov@gmail.com>
> > > > > >>> wrote:
> > > > > >>>
> > > > > >>>> Hi, Igniters.
> > > > > >>>>
> > > > > >>>> * AFAIU, we need a new repo if we want to apply different
> > > > restrictions
> > > > > >> to
> > > > > >>>> pull requests,
> > > > > >>>> otherwise I see no difference for myself.
> > > > > >>>> E.g. make static analysis (do we have?), compile, styles, and
> > > > javadoc
> > > > > >>>> checks mandatory.
> > > > > >>>>
> > > > > >>>> I think that relaxed requirements here will lead to bad
> product
> > > > > quality.
> > > > > >>>>
> > > > > >>>> * Agree with Pavel, we should 'keep' integrations tests
> somehow.
> > > > > >>>> During active development tests will be broken most of time,
> so,
> > > > > >>>> I'd port them e.g. suite-by-suite once we will have a stable
> and
> > > > > >> featured
> > > > > >>>> environment to run them and of course make test's code clear
> and
> > > > avoid
> > > > > >>>> bad/non-relevant ones.
> > > > > >>>>
> > > > > >>>> * I like bottom-up approach.
> > > > > >>>> With it we could make a better framework. I mean clear
> component
> > > > > >> lifecycle,
> > > > > >>>> component wiring mechanics, general methods to approach core
> > > > > components
> > > > > >>>> such as exchange/communication
> > > > > >>>> to avoid code mess like we have in ExchangeFuture with all
> these
> > > > > custom
> > > > > >>>> callbacks for each component, interfaces like
> > > > > >>>> PartitionsExchangeAware, IgniteChangeGlobalStateSupport and
> > > > > >>>> a pack of
> > > > > >> start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected
> > > > > >>>> and so on in various unexpected places.
> > > > > >>>> Hope, we will be able to port most of the good code to the new
> > > > > framework
> > > > > >>>> version.
> > > > > >>>>
> > > > > >>>>
> > > > > >>>>
> > > > > >>>> On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk <
> > > > > >>>> alexey.goncharuk@gmail.com>
> > > > > >>>> wrote:
> > > > > >>>>
> > > > > >>>>> Nikolay, Pavel,
> > > > > >>>>>
> > > > > >>>>> Thanks for the feedback! First of all, I wanted to stress
> that
> > I
> > > do
> > > > > not
> > > > > >>>>> intend to rewrite everything from scratch (I never used this
> > > > phrase).
> > > > > >>>> There
> > > > > >>>>> are significant parts of code that would be moved with
> minimal
> > > > > >>>>> modifications. Second, I never said that we will get rid of
> the
> > > old
> > > > > >> tests
> > > > > >>>>> codebase. Every single test from Ignite 2.x should be moved
> to
> > > > > Ignite 3
> > > > > >>>>> regardless of how we choose to proceed.
> > > > > >>>>>
> > > > > >>>>> My point is that for some parts of the code a clean bottom-up
> > > > > >>>>> implementation will be cheaper in many ways. Let me give you
> a
> > > few
> > > > > >>>> concrete
> > > > > >>>>> examples:
> > > > > >>>>>
> > > > > >>>>> - I think no one can object that we need a cleanly separated
> > > > > >>>> persistence
> > > > > >>>>> layer for Ignite. There is a very raw draft IEP for this
> > > already.
> > > > On
> > > > > >>>> the
> > > > > >>>>> other hand, I think we also can agree that we need a
> > split-brain
> > > > > >>>>> resistant
> > > > > >>>>> replication protocol for caches. There is also an IEP for
> > this.
> > > > > >>>> Neither
> > > > > >>>>> of
> > > > > >>>>> the changes is a good fit for 2.x because they are likely to
> > > > > >> introduce
> > > > > >>>>> breaking changes in the persistence layer, configuration and
> > > > > >> behavior.
> > > > > >>>>> Additionally, these components are now tightly coupled, so
> > there
> > > > is
> > > > > >> no
> > > > > >>>>> way
> > > > > >>>>> these two changes can be implemented in parallel and then
> > merged
> > > > > >>>>> together
> > > > > >>>>> easily. So what we will end up with is having to implement
> > these
> > > > > >>>> changes
> > > > > >>>>> sequentially, fixing all existing tests twice, and
> essentially
> > > > > >>>> throwing
> > > > > >>>>> away half of the work done because the other part of the
> > change
> > > is
> > > > > >>>>> re-implemented
> > > > > >>>>> - Similar example goes with getting rid of
> > IgniteInternalFuture
> > > > and
> > > > > >>>>> replacing it with CompletableFuture, and any other change
> that
> > > > > >> touches
> > > > > >>>>> the
> > > > > >>>>> asynchronous part of the code.
> > > > > >>>>>
> > > > > >>>>> Third, I do not see how this choice affects the UX of Ignite.
> > The
> > > > end
> > > > > >>>> user
> > > > > >>>>> experience must be fixed in the IEP regardless of the
> > development
> > > > > >> process
> > > > > >>>>> and the fact that we have gaps in this area in Ignite 2.x
> just
> > > > > confirms
> > > > > >>>>> that.
> > > > > >>>>>
> > > > > >>>>> Pavel, agree that a repo/branch is a technicality, I guess if
> > > > > >>>> reformulate,
> > > > > >>>>> my point is that we might agree to have a single development
> > > master
> > > > > >>>> branch
> > > > > >>>>> with 'disassembled' end-to-end functionality for some period
> of
> > > > time
> > > > > to
> > > > > >>>>> speed up development, and re-assemble the core features after
> > > > having
> > > > > >>>>> submodules tested independently.
> > > > > >>>>>
> > > > > >>>>> Nikolay,
> > > > > >>>>>> We have many features that have to evolve.
> > > > > >>>>>> Snapshots, rebalance, tooling, tracing, zookeeper support,
> > etc.
> > > > > >>>>> This is not very specific. In the end, resources are limited
> > and
> > > we
> > > > > >> will
> > > > > >>>>> not be able to drive both tracks simultaneously, especially
> > > after a
> > > > > >>>> couple
> > > > > >>>>> of features having been implemented for Ignite 3.0. If there
> > are
> > > > > indeed
> > > > > >>>>> some major changes that we want to do in Ignite 2.x instead
> of
> > > > > putting
> > > > > >>>>> effort into 3.0 - let's discuss them. I am just not aware of
> > any,
> > > > > >> that's
> > > > > >>>>> why I am eager to move forward with Ignite 3.0.
> > > > > >>>>>
> > > > > >>>>>> We have bugs and issues that can be fixed in 2.x without
> > > breaking
> > > > > >>>> backward
> > > > > >>>>> compatibility.
> > > > > >>>>>> We have many users who are happy with the 2.x with all it’s
> > > > issues.
> > > > > >>>>> These changes will be covered by end-to-end tests and
> migrated
> > to
> > > > > >> Ignite
> > > > > >>>>> 3.0, so I see no issues here.
> > > > > >>>>>
> > > > > >>>>> Finally, Anton & Nikolay
> > > > > >>>>> I do not have an estimate for this simply because the
> activity
> > is
> > > > > >>>>> community-driven and it depends on the number of people
> willing
> > > to
> > > > > >>>>> contribute. With the current pace, I would hope to have an RC
> > of
> > > > > Ignite
> > > > > >>>> 3.0
> > > > > >>>>> to be ready by the end of 2021. My gut feeling is that by
> > moving
> > > > with
> > > > > >>>>> incremental changes, we will not be able to implement even
> half
> > > of
> > > > > the
> > > > > >>>>> wishlist by that time.
> > > > > >>>>> I doubt that releasing several major releases with breaking
> > > changes
> > > > > >> will
> > > > > >>>>> make Ignite users happy either because each upgrade will cost
> > > > Ignite
> > > > > >>>> users
> > > > > >>>>> money, so the fewer major versions we release, the better.
> Thus
> > > my
> > > > > wish
> > > > > >>>> to
> > > > > >>>>> include all breaking changes in one release.
> > > > > >>>>>
> > > > > >>>>> I'll be now quiet for a while, let's see what other community
> > > > members
> > > > > >>>>> think.
> > > > > >>>>>
> > > > > >>>>> пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <
> > > ptupitsyn@apache.org
> > > > >:
> > > > > >>>>>
> > > > > >>>>>> 1. Rewriting from scratch is never a good idea.
> > > > > >>>>>> We don't want to follow the path of Netscape and lose all
> our
> > > > users
> > > > > >>>>>> by the time we have a working 3.0 [1]
> > > > > >>>>>>
> > > > > >>>>>> 2. Not sure about new repo - seems like some pain and no
> gain,
> > > > > what's
> > > > > >>>> the
> > > > > >>>>>> problem with a branch?
> > > > > >>>>>>
> > > > > >>>>>> 3. We should keep existing integration tests when possible.
> > > > > >>>>>> We have accumulated a lot of edge case knowledge over the
> > years,
> > > > > >>>>>> it is not a good idea to send all of that down the drain.
> > > > > >>>>>> Yes, integration tests are slow, but they are the most
> > valuable.
> > > > > >>>>>> I think we can move more stuff into nightly runs and have a
> > fast
> > > > and
> > > > > >>>>> modern
> > > > > >>>>>> basic suite.
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>> Alexey, you are much more familiar with the Ignite core
> > codebase
> > > > > than
> > > > > >>>>> most
> > > > > >>>>>> of us,
> > > > > >>>>>> can you please explain in more detail which particular
> > feature,
> > > in
> > > > > >> your
> > > > > >>>>>> opinion,
> > > > > >>>>>> mandates this "start from scratch" approach?
> > > > > >>>>>> Is it really not possible at all to follow a less radical
> way?
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>> [1]
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>
> > > > > >>>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
> > > > > >>>>>>
> > > > > >>>>>> On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <
> > > > nizhikov@apache.org
> > > > > >
> > > > > >>>>>> wrote:
> > > > > >>>>>>
> > > > > >>>>>>> Hello, Alexey.
> > > > > >>>>>>>
> > > > > >>>>>>> I think that «rewriting from scratch» approach has a high
> > risk
> > > to
> > > > > >>>> make
> > > > > >>>>>> new
> > > > > >>>>>>> features unusable.
> > > > > >>>>>>> At the time Ignite2 was started no-one wants to do bad UX
> or
> > > bad
> > > > > >>>>>> features.
> > > > > >>>>>>> Nevertheless, it happen.
> > > > > >>>>>>>
> > > > > >>>>>>> I think we can avoid it with the Ignite3 and successors if
> we
> > > > will
> > > > > >>>> move
> > > > > >>>>>>> step by step without keeping backward compatibility
> > > > > >>>>>>> With the step by step approach, we can focus on each
> > component
> > > > > >>>>>> separately.
> > > > > >>>>>>>
> > > > > >>>>>>>> What new features are we planning to implement for Ignite
> > 2.x?
> > > > > >>>>>>>
> > > > > >>>>>>> We have many features that have to evolve.
> > > > > >>>>>>> Snapshots, rebalance, tooling, tracing, zookeeper support,
> > etc.
> > > > > >>>>>>> We have bugs and issues that can be fixed in 2.x without
> > > breaking
> > > > > >>>>>> backward
> > > > > >>>>>>> compatibility.
> > > > > >>>>>>> We have many users who are happy with the 2.x with all it’s
> > > > issues.
> > > > > >>>>>>>
> > > > > >>>>>>>> 2 нояб. 2020 г., в 14:09, Anton Vinogradov <av@apache.org
> >
> > > > > >>>>> написал(а):
> > > > > >>>>>>>>
> > > > > >>>>>>>> Alexey,
> > > > > >>>>>>>>
> > > > > >>>>>>>> Do we have any estimates of how fast we'll be able to gain
> > > > > >>>>>>> production-ready
> > > > > >>>>>>>> AI 3.0 in case of a "new repo" choice?
> > > > > >>>>>>>>
> > > > > >>>>>>>> On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk <
> > > > > >>>>>>> alexey.goncharuk@gmail.com>
> > > > > >>>>>>>> wrote:
> > > > > >>>>>>>>
> > > > > >>>>>>>>> Nikolay,
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> What new features are we planning to implement for Ignite
> > > 2.x?
> > > > I
> > > > > >>>>> think
> > > > > >>>>>>> once
> > > > > >>>>>>>>> we commence working on Ignite 3.0, we should gradually
> > cease
> > > > the
> > > > > >>>>>>> activity
> > > > > >>>>>>>>> on Ignite 2.x to mere bugfixes because such parallel
> > > > development
> > > > > >>>>> will
> > > > > >>>>>> be
> > > > > >>>>>>>>> overwhelming regardless of how we choose to proceed.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <
> > > > > nizhikov@apache.org
> > > > > >>>>> :
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>> To be clear:
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>> I would suggest creating a new repository for Ignite
> 3.0
> > > > > >>>>> (perhaps, a
> > > > > >>>>>>>>> new
> > > > > >>>>>>>>>> clean branch, but a new repo looks nicer to me) and a
> new
> > > > Ignite
> > > > > >>>>> 3.0
> > > > > >>>>>>>>>> TeamCity project.
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> +1 for new Team City project.
> > > > > >>>>>>>>>> +1 for new branch for Ignite3.
> > > > > >>>>>>>>>> -1 for new repo.
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <
> > > > > >>>> NIzhikov.dev@gmail.com
> > > > > >>>>>>
> > > > > >>>>>>>>>> написал(а):
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> Hello, Alexey.
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> I think it will hurt our project more than help.
> > > > > >>>>>>>>>>> Developing new features for 2 separate branches with
> the
> > > > > >>>> different
> > > > > >>>>>>> APIs
> > > > > >>>>>>>>>> and internal structure is overwhelming
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> Maybe we should relax a bit requirements for Ignite3?
> > > > > >>>>>>>>>>> Maybe we should move step by step and make Ignite3 with
> > new
> > > > > >>>>>>>>>> configuration than Ignite4 with new transactions, etc?
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <
> > > > > >>>>>>>>> alexey.goncharuk@gmail.com>
> > > > > >>>>>>>>>> написал(а):
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> Igniters,
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> I wanted to pitch a rather radical idea regarding the
> > > Ignite
> > > > > >>>> 3.0
> > > > > >>>>>>>>>>>> development which has occurred to me some time ago.
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> We already have several IEPs targeted to Ignite 3.0
> > which
> > > > > imply
> > > > > >>>>>> major
> > > > > >>>>>>>>>>>> changes to the codebase (the change in replication
> > > protocol
> > > > > and
> > > > > >>>>>> thus
> > > > > >>>>>>>>>>>> transactions, change in binary format, updated
> > > metastorage,
> > > > > >>>> etc).
> > > > > >>>>>> We
> > > > > >>>>>>>>>> also
> > > > > >>>>>>>>>>>> planned significant changes in public APIs:
> > configuration
> > > > > >>>> format
> > > > > >>>>>>>>> change,
> > > > > >>>>>>>>>>>> improvements in cache APIs, SQL APIs, transaction mode
> > > > rework.
> > > > > >>>>> The
> > > > > >>>>>>>>>> wishlist
> > > > > >>>>>>>>>>>> of changes for Ignite 3.0 is huge.
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> So, I was wondering whether it makes sense to try to
> > > change
> > > > > the
> > > > > >>>>> old
> > > > > >>>>>>>>>>>> codebase, or start with a new baseline and move old
> > pieces
> > > > of
> > > > > >>>>> code
> > > > > >>>>>>>>> that
> > > > > >>>>>>>>>> do
> > > > > >>>>>>>>>>>> not require significant rework. Personally, I would go
> > > with
> > > > > the
> > > > > >>>>>>> second
> > > > > >>>>>>>>>>>> option for the following reasons:
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> - We have a chance to shift the development paradigm
> in
> > > the
> > > > > >>>>> project
> > > > > >>>>>>>>> and
> > > > > >>>>>>>>>>>> introduce the practice of true unit-tests. In the new
> > > > baseline
> > > > > >>>> at
> > > > > >>>>>> the
> > > > > >>>>>>>>>>>> beginning there will be no ability to run an
> end-to-end
> > > > > >>>> scenario,
> > > > > >>>>>>>>> thus
> > > > > >>>>>>>>>> we
> > > > > >>>>>>>>>>>> will be forced to write unit-tests. So far, such
> > practice
> > > > was
> > > > > >>>>> hard
> > > > > >>>>>> to
> > > > > >>>>>>>>>>>> implement because of tight coupling between Ignite
> > > > components
> > > > > >>>> and
> > > > > >>>>>>>>>> inability
> > > > > >>>>>>>>>>>> to instantiate components without an instance of
> > > > > KernalContext.
> > > > > >>>>> For
> > > > > >>>>>>>>>>>> example, we should be able to thoroughly test internal
> > > > > >>>>> primitives,
> > > > > >>>>>>>>>> such as
> > > > > >>>>>>>>>>>> replication protocol (without actual communication),
> > > > > >>>> distributed
> > > > > >>>>>>>>>>>> metastorage contracts, persistence layer, etc.
> > > > > >>>>>>>>>>>> - We will significantly reduce the development cycle
> in
> > > the
> > > > > >>>>>> beginning
> > > > > >>>>>>>>>>>> (right now the RunAll takes two hours of astronomical
> > time
> > > > > with
> > > > > >>>>>> empty
> > > > > >>>>>>>>>> TC;
> > > > > >>>>>>>>>>>> in the new approach developer will be able to run ALL
> > > tests
> > > > > >>>>> locally
> > > > > >>>>>>>>> in
> > > > > >>>>>>>>>> a
> > > > > >>>>>>>>>>>> matter of minutes)
> > > > > >>>>>>>>>>>> - We can get rid of TC bot and enforce green TC by
> > > > integrating
> > > > > >>>> TC
> > > > > >>>>>>>>> build
> > > > > >>>>>>>>>>>> results with GitHub PRs (the same way Travis is
> > currently
> > > > > >>>>>> integrated
> > > > > >>>>>>>>>> to PR
> > > > > >>>>>>>>>>>> check). We should restrict PR merge without a TC check
> > > > > >>>>>>>>>>>> - We will still have to re-write all tests, but only
> > once.
> > > > If
> > > > > >>>> we
> > > > > >>>>>> try
> > > > > >>>>>>>>> to
> > > > > >>>>>>>>>>>> modify the old codebase, we would need to modify all
> the
> > > > tests
> > > > > >>>>> for
> > > > > >>>>>>>>>> every
> > > > > >>>>>>>>>>>> major change (public API change, configuration change)
> > > > > >>>>>>>>>>>> - We will have fewer conflicts when working together.
> > For
> > > > > >>>>> example,
> > > > > >>>>>> I
> > > > > >>>>>>>>>>>> cannot imagine how one would merge two changes of
> > getting
> > > > rid
> > > > > >>>> of
> > > > > >>>>>>>>>>>> IgniteFuture and changes in replication protocol, for
> > > > example
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> Technically, I would suggest creating a new repository
> > for
> > > > > >>>> Ignite
> > > > > >>>>>> 3.0
> > > > > >>>>>>>>>>>> (perhaps, a new clean branch, but a new repo looks
> nicer
> > > to
> > > > > me)
> > > > > >>>>>> and a
> > > > > >>>>>>>>>> new
> > > > > >>>>>>>>>>>> Ignite 3.0 TeamCity project.
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> While it may seem quite radical, I do believe that
> this
> > > > > >>>> approach
> > > > > >>>>>> will
> > > > > >>>>>>>>>> give
> > > > > >>>>>>>>>>>> us more benefits than trying to make such major
> changes
> > in
> > > > the
> > > > > >>>>>>>>> existing
> > > > > >>>>>>>>>>>> codebase. If needed, let's schedule a discord chat
> like
> > > > before
> > > > > >>>> to
> > > > > >>>>>>>>>> discuss
> > > > > >>>>>>>>>>>> this.
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> WDYT?
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>
> > > > > >>>>>
> > > > > >>>>
> > > > > >>>>
> > > > > >>>> --
> > > > > >>>> Best regards,
> > > > > >>>> Andrey V. Mashenkov
> > > > > >>>>
> > > > > >>
> > > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Valentin Kulichenko <va...@gmail.com>.
Ksenia, thanks for scheduling this on such short notice!
As for the original topic, I do support Alexey's idea. We're not going to
rewrite anything from scratch, as most of the components are going to be
moved as-is or with minimal modifications. However, the changes that are
proposed imply serious rework of the core parts of the code, which are not
properly decoupled from each other and from other parts. This makes the
incremental approach borderline impossible. Developing in a new repo,
however, addresses this concern. As a bonus, we can also refactor the code,
introduce better decoupling, get rid of kernel context, and develop unit
tests (finally!).
Basically, this proposal only affects the *process*, not the set of changes
we had discussed before. Ignite 3.0 is our unique chance to make things
right.
-Val
On Tue, Nov 3, 2020 at 3:06 AM Kseniya Romanova <ro...@gmail.com>
wrote:
> Pavel, all the interesting points will be anyway published here in English
> (as the principal "if it's not on devlist it doesn't happened" is still
> relevant). This is just a quick call for a group of developers. Later we
> can do a separate presentation of idea and discussion in English as we did
> for the Ignite 3.0 draft of changes.
>
> вт, 3 нояб. 2020 г. в 13:52, Pavel Tupitsyn <pt...@apache.org>:
>
> > Kseniya,
> >
> > Thanks for scheduling this call.
> > Do you think we can switch to English if non-Russian speaking community
> > members decide to join?
> >
> > On Tue, Nov 3, 2020 at 1:32 PM Kseniya Romanova <
> romanova.ks.spb@gmail.com
> > >
> > wrote:
> >
> > > Let's do this community discussion open. Here's the link on zoom call
> in
> > > Russian for Friday 6 PM:
> > > https://www.meetup.com/Moscow-Apache-Ignite-Meetup/events/274360378/
> > >
> > > вт, 3 нояб. 2020 г. в 12:49, Nikolay Izhikov <ni...@apache.org>:
> > >
> > > > Time works for me.
> > > >
> > > > > 3 нояб. 2020 г., в 12:40, Alexey Goncharuk <
> > alexey.goncharuk@gmail.com
> > > >
> > > > написал(а):
> > > > >
> > > > > Nikolay,
> > > > >
> > > > > I am up for the call. I will try to explain my reasoning in greater
> > > > detail
> > > > > and will be glad to hear the concerns. Will this Friday, Nov 6th,
> > work?
> > > > >
> > > > > вт, 3 нояб. 2020 г. в 10:09, Nikolay Izhikov <nizhikov@apache.org
> >:
> > > > >
> > > > >> Igniters, should we have a call for this topic?
> > > > >>
> > > > >>> 2 нояб. 2020 г., в 18:53, Pavel Tupitsyn <pt...@apache.org>
> > > > >> написал(а):
> > > > >>>
> > > > >>>> not intend to rewrite everything from scratch
> > > > >>>
> > > > >>>> Every single test from Ignite 2.x should be moved to Ignite 3
> > > > >>>> regardless of how we choose to proceed.
> > > > >>>
> > > > >>> Alexey, thank you for the explanation, this addresses all of my
> > > > concerns.
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>> On Mon, Nov 2, 2020 at 6:43 PM Andrey Mashenkov <
> > > > >> andrey.mashenkov@gmail.com>
> > > > >>> wrote:
> > > > >>>
> > > > >>>> Hi, Igniters.
> > > > >>>>
> > > > >>>> * AFAIU, we need a new repo if we want to apply different
> > > restrictions
> > > > >> to
> > > > >>>> pull requests,
> > > > >>>> otherwise I see no difference for myself.
> > > > >>>> E.g. make static analysis (do we have?), compile, styles, and
> > > javadoc
> > > > >>>> checks mandatory.
> > > > >>>>
> > > > >>>> I think that relaxed requirements here will lead to bad product
> > > > quality.
> > > > >>>>
> > > > >>>> * Agree with Pavel, we should 'keep' integrations tests somehow.
> > > > >>>> During active development tests will be broken most of time, so,
> > > > >>>> I'd port them e.g. suite-by-suite once we will have a stable and
> > > > >> featured
> > > > >>>> environment to run them and of course make test's code clear and
> > > avoid
> > > > >>>> bad/non-relevant ones.
> > > > >>>>
> > > > >>>> * I like bottom-up approach.
> > > > >>>> With it we could make a better framework. I mean clear component
> > > > >> lifecycle,
> > > > >>>> component wiring mechanics, general methods to approach core
> > > > components
> > > > >>>> such as exchange/communication
> > > > >>>> to avoid code mess like we have in ExchangeFuture with all these
> > > > custom
> > > > >>>> callbacks for each component, interfaces like
> > > > >>>> PartitionsExchangeAware, IgniteChangeGlobalStateSupport and
> > > > >>>> a pack of
> > > > >> start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected
> > > > >>>> and so on in various unexpected places.
> > > > >>>> Hope, we will be able to port most of the good code to the new
> > > > framework
> > > > >>>> version.
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk <
> > > > >>>> alexey.goncharuk@gmail.com>
> > > > >>>> wrote:
> > > > >>>>
> > > > >>>>> Nikolay, Pavel,
> > > > >>>>>
> > > > >>>>> Thanks for the feedback! First of all, I wanted to stress that
> I
> > do
> > > > not
> > > > >>>>> intend to rewrite everything from scratch (I never used this
> > > phrase).
> > > > >>>> There
> > > > >>>>> are significant parts of code that would be moved with minimal
> > > > >>>>> modifications. Second, I never said that we will get rid of the
> > old
> > > > >> tests
> > > > >>>>> codebase. Every single test from Ignite 2.x should be moved to
> > > > Ignite 3
> > > > >>>>> regardless of how we choose to proceed.
> > > > >>>>>
> > > > >>>>> My point is that for some parts of the code a clean bottom-up
> > > > >>>>> implementation will be cheaper in many ways. Let me give you a
> > few
> > > > >>>> concrete
> > > > >>>>> examples:
> > > > >>>>>
> > > > >>>>> - I think no one can object that we need a cleanly separated
> > > > >>>> persistence
> > > > >>>>> layer for Ignite. There is a very raw draft IEP for this
> > already.
> > > On
> > > > >>>> the
> > > > >>>>> other hand, I think we also can agree that we need a
> split-brain
> > > > >>>>> resistant
> > > > >>>>> replication protocol for caches. There is also an IEP for
> this.
> > > > >>>> Neither
> > > > >>>>> of
> > > > >>>>> the changes is a good fit for 2.x because they are likely to
> > > > >> introduce
> > > > >>>>> breaking changes in the persistence layer, configuration and
> > > > >> behavior.
> > > > >>>>> Additionally, these components are now tightly coupled, so
> there
> > > is
> > > > >> no
> > > > >>>>> way
> > > > >>>>> these two changes can be implemented in parallel and then
> merged
> > > > >>>>> together
> > > > >>>>> easily. So what we will end up with is having to implement
> these
> > > > >>>> changes
> > > > >>>>> sequentially, fixing all existing tests twice, and essentially
> > > > >>>> throwing
> > > > >>>>> away half of the work done because the other part of the
> change
> > is
> > > > >>>>> re-implemented
> > > > >>>>> - Similar example goes with getting rid of
> IgniteInternalFuture
> > > and
> > > > >>>>> replacing it with CompletableFuture, and any other change that
> > > > >> touches
> > > > >>>>> the
> > > > >>>>> asynchronous part of the code.
> > > > >>>>>
> > > > >>>>> Third, I do not see how this choice affects the UX of Ignite.
> The
> > > end
> > > > >>>> user
> > > > >>>>> experience must be fixed in the IEP regardless of the
> development
> > > > >> process
> > > > >>>>> and the fact that we have gaps in this area in Ignite 2.x just
> > > > confirms
> > > > >>>>> that.
> > > > >>>>>
> > > > >>>>> Pavel, agree that a repo/branch is a technicality, I guess if
> > > > >>>> reformulate,
> > > > >>>>> my point is that we might agree to have a single development
> > master
> > > > >>>> branch
> > > > >>>>> with 'disassembled' end-to-end functionality for some period of
> > > time
> > > > to
> > > > >>>>> speed up development, and re-assemble the core features after
> > > having
> > > > >>>>> submodules tested independently.
> > > > >>>>>
> > > > >>>>> Nikolay,
> > > > >>>>>> We have many features that have to evolve.
> > > > >>>>>> Snapshots, rebalance, tooling, tracing, zookeeper support,
> etc.
> > > > >>>>> This is not very specific. In the end, resources are limited
> and
> > we
> > > > >> will
> > > > >>>>> not be able to drive both tracks simultaneously, especially
> > after a
> > > > >>>> couple
> > > > >>>>> of features having been implemented for Ignite 3.0. If there
> are
> > > > indeed
> > > > >>>>> some major changes that we want to do in Ignite 2.x instead of
> > > > putting
> > > > >>>>> effort into 3.0 - let's discuss them. I am just not aware of
> any,
> > > > >> that's
> > > > >>>>> why I am eager to move forward with Ignite 3.0.
> > > > >>>>>
> > > > >>>>>> We have bugs and issues that can be fixed in 2.x without
> > breaking
> > > > >>>> backward
> > > > >>>>> compatibility.
> > > > >>>>>> We have many users who are happy with the 2.x with all it’s
> > > issues.
> > > > >>>>> These changes will be covered by end-to-end tests and migrated
> to
> > > > >> Ignite
> > > > >>>>> 3.0, so I see no issues here.
> > > > >>>>>
> > > > >>>>> Finally, Anton & Nikolay
> > > > >>>>> I do not have an estimate for this simply because the activity
> is
> > > > >>>>> community-driven and it depends on the number of people willing
> > to
> > > > >>>>> contribute. With the current pace, I would hope to have an RC
> of
> > > > Ignite
> > > > >>>> 3.0
> > > > >>>>> to be ready by the end of 2021. My gut feeling is that by
> moving
> > > with
> > > > >>>>> incremental changes, we will not be able to implement even half
> > of
> > > > the
> > > > >>>>> wishlist by that time.
> > > > >>>>> I doubt that releasing several major releases with breaking
> > changes
> > > > >> will
> > > > >>>>> make Ignite users happy either because each upgrade will cost
> > > Ignite
> > > > >>>> users
> > > > >>>>> money, so the fewer major versions we release, the better. Thus
> > my
> > > > wish
> > > > >>>> to
> > > > >>>>> include all breaking changes in one release.
> > > > >>>>>
> > > > >>>>> I'll be now quiet for a while, let's see what other community
> > > members
> > > > >>>>> think.
> > > > >>>>>
> > > > >>>>> пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <
> > ptupitsyn@apache.org
> > > >:
> > > > >>>>>
> > > > >>>>>> 1. Rewriting from scratch is never a good idea.
> > > > >>>>>> We don't want to follow the path of Netscape and lose all our
> > > users
> > > > >>>>>> by the time we have a working 3.0 [1]
> > > > >>>>>>
> > > > >>>>>> 2. Not sure about new repo - seems like some pain and no gain,
> > > > what's
> > > > >>>> the
> > > > >>>>>> problem with a branch?
> > > > >>>>>>
> > > > >>>>>> 3. We should keep existing integration tests when possible.
> > > > >>>>>> We have accumulated a lot of edge case knowledge over the
> years,
> > > > >>>>>> it is not a good idea to send all of that down the drain.
> > > > >>>>>> Yes, integration tests are slow, but they are the most
> valuable.
> > > > >>>>>> I think we can move more stuff into nightly runs and have a
> fast
> > > and
> > > > >>>>> modern
> > > > >>>>>> basic suite.
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>> Alexey, you are much more familiar with the Ignite core
> codebase
> > > > than
> > > > >>>>> most
> > > > >>>>>> of us,
> > > > >>>>>> can you please explain in more detail which particular
> feature,
> > in
> > > > >> your
> > > > >>>>>> opinion,
> > > > >>>>>> mandates this "start from scratch" approach?
> > > > >>>>>> Is it really not possible at all to follow a less radical way?
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>> [1]
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>
> > > > >>>>
> > > > >>
> > > >
> > >
> >
> https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
> > > > >>>>>>
> > > > >>>>>> On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <
> > > nizhikov@apache.org
> > > > >
> > > > >>>>>> wrote:
> > > > >>>>>>
> > > > >>>>>>> Hello, Alexey.
> > > > >>>>>>>
> > > > >>>>>>> I think that «rewriting from scratch» approach has a high
> risk
> > to
> > > > >>>> make
> > > > >>>>>> new
> > > > >>>>>>> features unusable.
> > > > >>>>>>> At the time Ignite2 was started no-one wants to do bad UX or
> > bad
> > > > >>>>>> features.
> > > > >>>>>>> Nevertheless, it happen.
> > > > >>>>>>>
> > > > >>>>>>> I think we can avoid it with the Ignite3 and successors if we
> > > will
> > > > >>>> move
> > > > >>>>>>> step by step without keeping backward compatibility
> > > > >>>>>>> With the step by step approach, we can focus on each
> component
> > > > >>>>>> separately.
> > > > >>>>>>>
> > > > >>>>>>>> What new features are we planning to implement for Ignite
> 2.x?
> > > > >>>>>>>
> > > > >>>>>>> We have many features that have to evolve.
> > > > >>>>>>> Snapshots, rebalance, tooling, tracing, zookeeper support,
> etc.
> > > > >>>>>>> We have bugs and issues that can be fixed in 2.x without
> > breaking
> > > > >>>>>> backward
> > > > >>>>>>> compatibility.
> > > > >>>>>>> We have many users who are happy with the 2.x with all it’s
> > > issues.
> > > > >>>>>>>
> > > > >>>>>>>> 2 нояб. 2020 г., в 14:09, Anton Vinogradov <av...@apache.org>
> > > > >>>>> написал(а):
> > > > >>>>>>>>
> > > > >>>>>>>> Alexey,
> > > > >>>>>>>>
> > > > >>>>>>>> Do we have any estimates of how fast we'll be able to gain
> > > > >>>>>>> production-ready
> > > > >>>>>>>> AI 3.0 in case of a "new repo" choice?
> > > > >>>>>>>>
> > > > >>>>>>>> On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk <
> > > > >>>>>>> alexey.goncharuk@gmail.com>
> > > > >>>>>>>> wrote:
> > > > >>>>>>>>
> > > > >>>>>>>>> Nikolay,
> > > > >>>>>>>>>
> > > > >>>>>>>>> What new features are we planning to implement for Ignite
> > 2.x?
> > > I
> > > > >>>>> think
> > > > >>>>>>> once
> > > > >>>>>>>>> we commence working on Ignite 3.0, we should gradually
> cease
> > > the
> > > > >>>>>>> activity
> > > > >>>>>>>>> on Ignite 2.x to mere bugfixes because such parallel
> > > development
> > > > >>>>> will
> > > > >>>>>> be
> > > > >>>>>>>>> overwhelming regardless of how we choose to proceed.
> > > > >>>>>>>>>
> > > > >>>>>>>>> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <
> > > > nizhikov@apache.org
> > > > >>>>> :
> > > > >>>>>>>>>
> > > > >>>>>>>>>> To be clear:
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>> I would suggest creating a new repository for Ignite 3.0
> > > > >>>>> (perhaps, a
> > > > >>>>>>>>> new
> > > > >>>>>>>>>> clean branch, but a new repo looks nicer to me) and a new
> > > Ignite
> > > > >>>>> 3.0
> > > > >>>>>>>>>> TeamCity project.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> +1 for new Team City project.
> > > > >>>>>>>>>> +1 for new branch for Ignite3.
> > > > >>>>>>>>>> -1 for new repo.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <
> > > > >>>> NIzhikov.dev@gmail.com
> > > > >>>>>>
> > > > >>>>>>>>>> написал(а):
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Hello, Alexey.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> I think it will hurt our project more than help.
> > > > >>>>>>>>>>> Developing new features for 2 separate branches with the
> > > > >>>> different
> > > > >>>>>>> APIs
> > > > >>>>>>>>>> and internal structure is overwhelming
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Maybe we should relax a bit requirements for Ignite3?
> > > > >>>>>>>>>>> Maybe we should move step by step and make Ignite3 with
> new
> > > > >>>>>>>>>> configuration than Ignite4 with new transactions, etc?
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <
> > > > >>>>>>>>> alexey.goncharuk@gmail.com>
> > > > >>>>>>>>>> написал(а):
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Igniters,
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> I wanted to pitch a rather radical idea regarding the
> > Ignite
> > > > >>>> 3.0
> > > > >>>>>>>>>>>> development which has occurred to me some time ago.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> We already have several IEPs targeted to Ignite 3.0
> which
> > > > imply
> > > > >>>>>> major
> > > > >>>>>>>>>>>> changes to the codebase (the change in replication
> > protocol
> > > > and
> > > > >>>>>> thus
> > > > >>>>>>>>>>>> transactions, change in binary format, updated
> > metastorage,
> > > > >>>> etc).
> > > > >>>>>> We
> > > > >>>>>>>>>> also
> > > > >>>>>>>>>>>> planned significant changes in public APIs:
> configuration
> > > > >>>> format
> > > > >>>>>>>>> change,
> > > > >>>>>>>>>>>> improvements in cache APIs, SQL APIs, transaction mode
> > > rework.
> > > > >>>>> The
> > > > >>>>>>>>>> wishlist
> > > > >>>>>>>>>>>> of changes for Ignite 3.0 is huge.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> So, I was wondering whether it makes sense to try to
> > change
> > > > the
> > > > >>>>> old
> > > > >>>>>>>>>>>> codebase, or start with a new baseline and move old
> pieces
> > > of
> > > > >>>>> code
> > > > >>>>>>>>> that
> > > > >>>>>>>>>> do
> > > > >>>>>>>>>>>> not require significant rework. Personally, I would go
> > with
> > > > the
> > > > >>>>>>> second
> > > > >>>>>>>>>>>> option for the following reasons:
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> - We have a chance to shift the development paradigm in
> > the
> > > > >>>>> project
> > > > >>>>>>>>> and
> > > > >>>>>>>>>>>> introduce the practice of true unit-tests. In the new
> > > baseline
> > > > >>>> at
> > > > >>>>>> the
> > > > >>>>>>>>>>>> beginning there will be no ability to run an end-to-end
> > > > >>>> scenario,
> > > > >>>>>>>>> thus
> > > > >>>>>>>>>> we
> > > > >>>>>>>>>>>> will be forced to write unit-tests. So far, such
> practice
> > > was
> > > > >>>>> hard
> > > > >>>>>> to
> > > > >>>>>>>>>>>> implement because of tight coupling between Ignite
> > > components
> > > > >>>> and
> > > > >>>>>>>>>> inability
> > > > >>>>>>>>>>>> to instantiate components without an instance of
> > > > KernalContext.
> > > > >>>>> For
> > > > >>>>>>>>>>>> example, we should be able to thoroughly test internal
> > > > >>>>> primitives,
> > > > >>>>>>>>>> such as
> > > > >>>>>>>>>>>> replication protocol (without actual communication),
> > > > >>>> distributed
> > > > >>>>>>>>>>>> metastorage contracts, persistence layer, etc.
> > > > >>>>>>>>>>>> - We will significantly reduce the development cycle in
> > the
> > > > >>>>>> beginning
> > > > >>>>>>>>>>>> (right now the RunAll takes two hours of astronomical
> time
> > > > with
> > > > >>>>>> empty
> > > > >>>>>>>>>> TC;
> > > > >>>>>>>>>>>> in the new approach developer will be able to run ALL
> > tests
> > > > >>>>> locally
> > > > >>>>>>>>> in
> > > > >>>>>>>>>> a
> > > > >>>>>>>>>>>> matter of minutes)
> > > > >>>>>>>>>>>> - We can get rid of TC bot and enforce green TC by
> > > integrating
> > > > >>>> TC
> > > > >>>>>>>>> build
> > > > >>>>>>>>>>>> results with GitHub PRs (the same way Travis is
> currently
> > > > >>>>>> integrated
> > > > >>>>>>>>>> to PR
> > > > >>>>>>>>>>>> check). We should restrict PR merge without a TC check
> > > > >>>>>>>>>>>> - We will still have to re-write all tests, but only
> once.
> > > If
> > > > >>>> we
> > > > >>>>>> try
> > > > >>>>>>>>> to
> > > > >>>>>>>>>>>> modify the old codebase, we would need to modify all the
> > > tests
> > > > >>>>> for
> > > > >>>>>>>>>> every
> > > > >>>>>>>>>>>> major change (public API change, configuration change)
> > > > >>>>>>>>>>>> - We will have fewer conflicts when working together.
> For
> > > > >>>>> example,
> > > > >>>>>> I
> > > > >>>>>>>>>>>> cannot imagine how one would merge two changes of
> getting
> > > rid
> > > > >>>> of
> > > > >>>>>>>>>>>> IgniteFuture and changes in replication protocol, for
> > > example
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Technically, I would suggest creating a new repository
> for
> > > > >>>> Ignite
> > > > >>>>>> 3.0
> > > > >>>>>>>>>>>> (perhaps, a new clean branch, but a new repo looks nicer
> > to
> > > > me)
> > > > >>>>>> and a
> > > > >>>>>>>>>> new
> > > > >>>>>>>>>>>> Ignite 3.0 TeamCity project.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> While it may seem quite radical, I do believe that this
> > > > >>>> approach
> > > > >>>>>> will
> > > > >>>>>>>>>> give
> > > > >>>>>>>>>>>> us more benefits than trying to make such major changes
> in
> > > the
> > > > >>>>>>>>> existing
> > > > >>>>>>>>>>>> codebase. If needed, let's schedule a discord chat like
> > > before
> > > > >>>> to
> > > > >>>>>>>>>> discuss
> > > > >>>>>>>>>>>> this.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> WDYT?
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> --
> > > > >>>> Best regards,
> > > > >>>> Andrey V. Mashenkov
> > > > >>>>
> > > > >>
> > > > >>
> > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Kseniya Romanova <ro...@gmail.com>.
Pavel, all the interesting points will be anyway published here in English
(as the principal "if it's not on devlist it doesn't happened" is still
relevant). This is just a quick call for a group of developers. Later we
can do a separate presentation of idea and discussion in English as we did
for the Ignite 3.0 draft of changes.
вт, 3 нояб. 2020 г. в 13:52, Pavel Tupitsyn <pt...@apache.org>:
> Kseniya,
>
> Thanks for scheduling this call.
> Do you think we can switch to English if non-Russian speaking community
> members decide to join?
>
> On Tue, Nov 3, 2020 at 1:32 PM Kseniya Romanova <romanova.ks.spb@gmail.com
> >
> wrote:
>
> > Let's do this community discussion open. Here's the link on zoom call in
> > Russian for Friday 6 PM:
> > https://www.meetup.com/Moscow-Apache-Ignite-Meetup/events/274360378/
> >
> > вт, 3 нояб. 2020 г. в 12:49, Nikolay Izhikov <ni...@apache.org>:
> >
> > > Time works for me.
> > >
> > > > 3 нояб. 2020 г., в 12:40, Alexey Goncharuk <
> alexey.goncharuk@gmail.com
> > >
> > > написал(а):
> > > >
> > > > Nikolay,
> > > >
> > > > I am up for the call. I will try to explain my reasoning in greater
> > > detail
> > > > and will be glad to hear the concerns. Will this Friday, Nov 6th,
> work?
> > > >
> > > > вт, 3 нояб. 2020 г. в 10:09, Nikolay Izhikov <ni...@apache.org>:
> > > >
> > > >> Igniters, should we have a call for this topic?
> > > >>
> > > >>> 2 нояб. 2020 г., в 18:53, Pavel Tupitsyn <pt...@apache.org>
> > > >> написал(а):
> > > >>>
> > > >>>> not intend to rewrite everything from scratch
> > > >>>
> > > >>>> Every single test from Ignite 2.x should be moved to Ignite 3
> > > >>>> regardless of how we choose to proceed.
> > > >>>
> > > >>> Alexey, thank you for the explanation, this addresses all of my
> > > concerns.
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>> On Mon, Nov 2, 2020 at 6:43 PM Andrey Mashenkov <
> > > >> andrey.mashenkov@gmail.com>
> > > >>> wrote:
> > > >>>
> > > >>>> Hi, Igniters.
> > > >>>>
> > > >>>> * AFAIU, we need a new repo if we want to apply different
> > restrictions
> > > >> to
> > > >>>> pull requests,
> > > >>>> otherwise I see no difference for myself.
> > > >>>> E.g. make static analysis (do we have?), compile, styles, and
> > javadoc
> > > >>>> checks mandatory.
> > > >>>>
> > > >>>> I think that relaxed requirements here will lead to bad product
> > > quality.
> > > >>>>
> > > >>>> * Agree with Pavel, we should 'keep' integrations tests somehow.
> > > >>>> During active development tests will be broken most of time, so,
> > > >>>> I'd port them e.g. suite-by-suite once we will have a stable and
> > > >> featured
> > > >>>> environment to run them and of course make test's code clear and
> > avoid
> > > >>>> bad/non-relevant ones.
> > > >>>>
> > > >>>> * I like bottom-up approach.
> > > >>>> With it we could make a better framework. I mean clear component
> > > >> lifecycle,
> > > >>>> component wiring mechanics, general methods to approach core
> > > components
> > > >>>> such as exchange/communication
> > > >>>> to avoid code mess like we have in ExchangeFuture with all these
> > > custom
> > > >>>> callbacks for each component, interfaces like
> > > >>>> PartitionsExchangeAware, IgniteChangeGlobalStateSupport and
> > > >>>> a pack of
> > > >> start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected
> > > >>>> and so on in various unexpected places.
> > > >>>> Hope, we will be able to port most of the good code to the new
> > > framework
> > > >>>> version.
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk <
> > > >>>> alexey.goncharuk@gmail.com>
> > > >>>> wrote:
> > > >>>>
> > > >>>>> Nikolay, Pavel,
> > > >>>>>
> > > >>>>> Thanks for the feedback! First of all, I wanted to stress that I
> do
> > > not
> > > >>>>> intend to rewrite everything from scratch (I never used this
> > phrase).
> > > >>>> There
> > > >>>>> are significant parts of code that would be moved with minimal
> > > >>>>> modifications. Second, I never said that we will get rid of the
> old
> > > >> tests
> > > >>>>> codebase. Every single test from Ignite 2.x should be moved to
> > > Ignite 3
> > > >>>>> regardless of how we choose to proceed.
> > > >>>>>
> > > >>>>> My point is that for some parts of the code a clean bottom-up
> > > >>>>> implementation will be cheaper in many ways. Let me give you a
> few
> > > >>>> concrete
> > > >>>>> examples:
> > > >>>>>
> > > >>>>> - I think no one can object that we need a cleanly separated
> > > >>>> persistence
> > > >>>>> layer for Ignite. There is a very raw draft IEP for this
> already.
> > On
> > > >>>> the
> > > >>>>> other hand, I think we also can agree that we need a split-brain
> > > >>>>> resistant
> > > >>>>> replication protocol for caches. There is also an IEP for this.
> > > >>>> Neither
> > > >>>>> of
> > > >>>>> the changes is a good fit for 2.x because they are likely to
> > > >> introduce
> > > >>>>> breaking changes in the persistence layer, configuration and
> > > >> behavior.
> > > >>>>> Additionally, these components are now tightly coupled, so there
> > is
> > > >> no
> > > >>>>> way
> > > >>>>> these two changes can be implemented in parallel and then merged
> > > >>>>> together
> > > >>>>> easily. So what we will end up with is having to implement these
> > > >>>> changes
> > > >>>>> sequentially, fixing all existing tests twice, and essentially
> > > >>>> throwing
> > > >>>>> away half of the work done because the other part of the change
> is
> > > >>>>> re-implemented
> > > >>>>> - Similar example goes with getting rid of IgniteInternalFuture
> > and
> > > >>>>> replacing it with CompletableFuture, and any other change that
> > > >> touches
> > > >>>>> the
> > > >>>>> asynchronous part of the code.
> > > >>>>>
> > > >>>>> Third, I do not see how this choice affects the UX of Ignite. The
> > end
> > > >>>> user
> > > >>>>> experience must be fixed in the IEP regardless of the development
> > > >> process
> > > >>>>> and the fact that we have gaps in this area in Ignite 2.x just
> > > confirms
> > > >>>>> that.
> > > >>>>>
> > > >>>>> Pavel, agree that a repo/branch is a technicality, I guess if
> > > >>>> reformulate,
> > > >>>>> my point is that we might agree to have a single development
> master
> > > >>>> branch
> > > >>>>> with 'disassembled' end-to-end functionality for some period of
> > time
> > > to
> > > >>>>> speed up development, and re-assemble the core features after
> > having
> > > >>>>> submodules tested independently.
> > > >>>>>
> > > >>>>> Nikolay,
> > > >>>>>> We have many features that have to evolve.
> > > >>>>>> Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
> > > >>>>> This is not very specific. In the end, resources are limited and
> we
> > > >> will
> > > >>>>> not be able to drive both tracks simultaneously, especially
> after a
> > > >>>> couple
> > > >>>>> of features having been implemented for Ignite 3.0. If there are
> > > indeed
> > > >>>>> some major changes that we want to do in Ignite 2.x instead of
> > > putting
> > > >>>>> effort into 3.0 - let's discuss them. I am just not aware of any,
> > > >> that's
> > > >>>>> why I am eager to move forward with Ignite 3.0.
> > > >>>>>
> > > >>>>>> We have bugs and issues that can be fixed in 2.x without
> breaking
> > > >>>> backward
> > > >>>>> compatibility.
> > > >>>>>> We have many users who are happy with the 2.x with all it’s
> > issues.
> > > >>>>> These changes will be covered by end-to-end tests and migrated to
> > > >> Ignite
> > > >>>>> 3.0, so I see no issues here.
> > > >>>>>
> > > >>>>> Finally, Anton & Nikolay
> > > >>>>> I do not have an estimate for this simply because the activity is
> > > >>>>> community-driven and it depends on the number of people willing
> to
> > > >>>>> contribute. With the current pace, I would hope to have an RC of
> > > Ignite
> > > >>>> 3.0
> > > >>>>> to be ready by the end of 2021. My gut feeling is that by moving
> > with
> > > >>>>> incremental changes, we will not be able to implement even half
> of
> > > the
> > > >>>>> wishlist by that time.
> > > >>>>> I doubt that releasing several major releases with breaking
> changes
> > > >> will
> > > >>>>> make Ignite users happy either because each upgrade will cost
> > Ignite
> > > >>>> users
> > > >>>>> money, so the fewer major versions we release, the better. Thus
> my
> > > wish
> > > >>>> to
> > > >>>>> include all breaking changes in one release.
> > > >>>>>
> > > >>>>> I'll be now quiet for a while, let's see what other community
> > members
> > > >>>>> think.
> > > >>>>>
> > > >>>>> пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <
> ptupitsyn@apache.org
> > >:
> > > >>>>>
> > > >>>>>> 1. Rewriting from scratch is never a good idea.
> > > >>>>>> We don't want to follow the path of Netscape and lose all our
> > users
> > > >>>>>> by the time we have a working 3.0 [1]
> > > >>>>>>
> > > >>>>>> 2. Not sure about new repo - seems like some pain and no gain,
> > > what's
> > > >>>> the
> > > >>>>>> problem with a branch?
> > > >>>>>>
> > > >>>>>> 3. We should keep existing integration tests when possible.
> > > >>>>>> We have accumulated a lot of edge case knowledge over the years,
> > > >>>>>> it is not a good idea to send all of that down the drain.
> > > >>>>>> Yes, integration tests are slow, but they are the most valuable.
> > > >>>>>> I think we can move more stuff into nightly runs and have a fast
> > and
> > > >>>>> modern
> > > >>>>>> basic suite.
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> Alexey, you are much more familiar with the Ignite core codebase
> > > than
> > > >>>>> most
> > > >>>>>> of us,
> > > >>>>>> can you please explain in more detail which particular feature,
> in
> > > >> your
> > > >>>>>> opinion,
> > > >>>>>> mandates this "start from scratch" approach?
> > > >>>>>> Is it really not possible at all to follow a less radical way?
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> [1]
> > > >>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>
> > >
> >
> https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
> > > >>>>>>
> > > >>>>>> On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <
> > nizhikov@apache.org
> > > >
> > > >>>>>> wrote:
> > > >>>>>>
> > > >>>>>>> Hello, Alexey.
> > > >>>>>>>
> > > >>>>>>> I think that «rewriting from scratch» approach has a high risk
> to
> > > >>>> make
> > > >>>>>> new
> > > >>>>>>> features unusable.
> > > >>>>>>> At the time Ignite2 was started no-one wants to do bad UX or
> bad
> > > >>>>>> features.
> > > >>>>>>> Nevertheless, it happen.
> > > >>>>>>>
> > > >>>>>>> I think we can avoid it with the Ignite3 and successors if we
> > will
> > > >>>> move
> > > >>>>>>> step by step without keeping backward compatibility
> > > >>>>>>> With the step by step approach, we can focus on each component
> > > >>>>>> separately.
> > > >>>>>>>
> > > >>>>>>>> What new features are we planning to implement for Ignite 2.x?
> > > >>>>>>>
> > > >>>>>>> We have many features that have to evolve.
> > > >>>>>>> Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
> > > >>>>>>> We have bugs and issues that can be fixed in 2.x without
> breaking
> > > >>>>>> backward
> > > >>>>>>> compatibility.
> > > >>>>>>> We have many users who are happy with the 2.x with all it’s
> > issues.
> > > >>>>>>>
> > > >>>>>>>> 2 нояб. 2020 г., в 14:09, Anton Vinogradov <av...@apache.org>
> > > >>>>> написал(а):
> > > >>>>>>>>
> > > >>>>>>>> Alexey,
> > > >>>>>>>>
> > > >>>>>>>> Do we have any estimates of how fast we'll be able to gain
> > > >>>>>>> production-ready
> > > >>>>>>>> AI 3.0 in case of a "new repo" choice?
> > > >>>>>>>>
> > > >>>>>>>> On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk <
> > > >>>>>>> alexey.goncharuk@gmail.com>
> > > >>>>>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> Nikolay,
> > > >>>>>>>>>
> > > >>>>>>>>> What new features are we planning to implement for Ignite
> 2.x?
> > I
> > > >>>>> think
> > > >>>>>>> once
> > > >>>>>>>>> we commence working on Ignite 3.0, we should gradually cease
> > the
> > > >>>>>>> activity
> > > >>>>>>>>> on Ignite 2.x to mere bugfixes because such parallel
> > development
> > > >>>>> will
> > > >>>>>> be
> > > >>>>>>>>> overwhelming regardless of how we choose to proceed.
> > > >>>>>>>>>
> > > >>>>>>>>> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <
> > > nizhikov@apache.org
> > > >>>>> :
> > > >>>>>>>>>
> > > >>>>>>>>>> To be clear:
> > > >>>>>>>>>>
> > > >>>>>>>>>>> I would suggest creating a new repository for Ignite 3.0
> > > >>>>> (perhaps, a
> > > >>>>>>>>> new
> > > >>>>>>>>>> clean branch, but a new repo looks nicer to me) and a new
> > Ignite
> > > >>>>> 3.0
> > > >>>>>>>>>> TeamCity project.
> > > >>>>>>>>>>
> > > >>>>>>>>>> +1 for new Team City project.
> > > >>>>>>>>>> +1 for new branch for Ignite3.
> > > >>>>>>>>>> -1 for new repo.
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <
> > > >>>> NIzhikov.dev@gmail.com
> > > >>>>>>
> > > >>>>>>>>>> написал(а):
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Hello, Alexey.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I think it will hurt our project more than help.
> > > >>>>>>>>>>> Developing new features for 2 separate branches with the
> > > >>>> different
> > > >>>>>>> APIs
> > > >>>>>>>>>> and internal structure is overwhelming
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Maybe we should relax a bit requirements for Ignite3?
> > > >>>>>>>>>>> Maybe we should move step by step and make Ignite3 with new
> > > >>>>>>>>>> configuration than Ignite4 with new transactions, etc?
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <
> > > >>>>>>>>> alexey.goncharuk@gmail.com>
> > > >>>>>>>>>> написал(а):
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Igniters,
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> I wanted to pitch a rather radical idea regarding the
> Ignite
> > > >>>> 3.0
> > > >>>>>>>>>>>> development which has occurred to me some time ago.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> We already have several IEPs targeted to Ignite 3.0 which
> > > imply
> > > >>>>>> major
> > > >>>>>>>>>>>> changes to the codebase (the change in replication
> protocol
> > > and
> > > >>>>>> thus
> > > >>>>>>>>>>>> transactions, change in binary format, updated
> metastorage,
> > > >>>> etc).
> > > >>>>>> We
> > > >>>>>>>>>> also
> > > >>>>>>>>>>>> planned significant changes in public APIs: configuration
> > > >>>> format
> > > >>>>>>>>> change,
> > > >>>>>>>>>>>> improvements in cache APIs, SQL APIs, transaction mode
> > rework.
> > > >>>>> The
> > > >>>>>>>>>> wishlist
> > > >>>>>>>>>>>> of changes for Ignite 3.0 is huge.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> So, I was wondering whether it makes sense to try to
> change
> > > the
> > > >>>>> old
> > > >>>>>>>>>>>> codebase, or start with a new baseline and move old pieces
> > of
> > > >>>>> code
> > > >>>>>>>>> that
> > > >>>>>>>>>> do
> > > >>>>>>>>>>>> not require significant rework. Personally, I would go
> with
> > > the
> > > >>>>>>> second
> > > >>>>>>>>>>>> option for the following reasons:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> - We have a chance to shift the development paradigm in
> the
> > > >>>>> project
> > > >>>>>>>>> and
> > > >>>>>>>>>>>> introduce the practice of true unit-tests. In the new
> > baseline
> > > >>>> at
> > > >>>>>> the
> > > >>>>>>>>>>>> beginning there will be no ability to run an end-to-end
> > > >>>> scenario,
> > > >>>>>>>>> thus
> > > >>>>>>>>>> we
> > > >>>>>>>>>>>> will be forced to write unit-tests. So far, such practice
> > was
> > > >>>>> hard
> > > >>>>>> to
> > > >>>>>>>>>>>> implement because of tight coupling between Ignite
> > components
> > > >>>> and
> > > >>>>>>>>>> inability
> > > >>>>>>>>>>>> to instantiate components without an instance of
> > > KernalContext.
> > > >>>>> For
> > > >>>>>>>>>>>> example, we should be able to thoroughly test internal
> > > >>>>> primitives,
> > > >>>>>>>>>> such as
> > > >>>>>>>>>>>> replication protocol (without actual communication),
> > > >>>> distributed
> > > >>>>>>>>>>>> metastorage contracts, persistence layer, etc.
> > > >>>>>>>>>>>> - We will significantly reduce the development cycle in
> the
> > > >>>>>> beginning
> > > >>>>>>>>>>>> (right now the RunAll takes two hours of astronomical time
> > > with
> > > >>>>>> empty
> > > >>>>>>>>>> TC;
> > > >>>>>>>>>>>> in the new approach developer will be able to run ALL
> tests
> > > >>>>> locally
> > > >>>>>>>>> in
> > > >>>>>>>>>> a
> > > >>>>>>>>>>>> matter of minutes)
> > > >>>>>>>>>>>> - We can get rid of TC bot and enforce green TC by
> > integrating
> > > >>>> TC
> > > >>>>>>>>> build
> > > >>>>>>>>>>>> results with GitHub PRs (the same way Travis is currently
> > > >>>>>> integrated
> > > >>>>>>>>>> to PR
> > > >>>>>>>>>>>> check). We should restrict PR merge without a TC check
> > > >>>>>>>>>>>> - We will still have to re-write all tests, but only once.
> > If
> > > >>>> we
> > > >>>>>> try
> > > >>>>>>>>> to
> > > >>>>>>>>>>>> modify the old codebase, we would need to modify all the
> > tests
> > > >>>>> for
> > > >>>>>>>>>> every
> > > >>>>>>>>>>>> major change (public API change, configuration change)
> > > >>>>>>>>>>>> - We will have fewer conflicts when working together. For
> > > >>>>> example,
> > > >>>>>> I
> > > >>>>>>>>>>>> cannot imagine how one would merge two changes of getting
> > rid
> > > >>>> of
> > > >>>>>>>>>>>> IgniteFuture and changes in replication protocol, for
> > example
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Technically, I would suggest creating a new repository for
> > > >>>> Ignite
> > > >>>>>> 3.0
> > > >>>>>>>>>>>> (perhaps, a new clean branch, but a new repo looks nicer
> to
> > > me)
> > > >>>>>> and a
> > > >>>>>>>>>> new
> > > >>>>>>>>>>>> Ignite 3.0 TeamCity project.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> While it may seem quite radical, I do believe that this
> > > >>>> approach
> > > >>>>>> will
> > > >>>>>>>>>> give
> > > >>>>>>>>>>>> us more benefits than trying to make such major changes in
> > the
> > > >>>>>>>>> existing
> > > >>>>>>>>>>>> codebase. If needed, let's schedule a discord chat like
> > before
> > > >>>> to
> > > >>>>>>>>>> discuss
> > > >>>>>>>>>>>> this.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> WDYT?
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>>
> > > >>>> --
> > > >>>> Best regards,
> > > >>>> Andrey V. Mashenkov
> > > >>>>
> > > >>
> > > >>
> > >
> > >
> >
>
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Pavel Tupitsyn <pt...@apache.org>.
Kseniya,
Thanks for scheduling this call.
Do you think we can switch to English if non-Russian speaking community
members decide to join?
On Tue, Nov 3, 2020 at 1:32 PM Kseniya Romanova <ro...@gmail.com>
wrote:
> Let's do this community discussion open. Here's the link on zoom call in
> Russian for Friday 6 PM:
> https://www.meetup.com/Moscow-Apache-Ignite-Meetup/events/274360378/
>
> вт, 3 нояб. 2020 г. в 12:49, Nikolay Izhikov <ni...@apache.org>:
>
> > Time works for me.
> >
> > > 3 нояб. 2020 г., в 12:40, Alexey Goncharuk <alexey.goncharuk@gmail.com
> >
> > написал(а):
> > >
> > > Nikolay,
> > >
> > > I am up for the call. I will try to explain my reasoning in greater
> > detail
> > > and will be glad to hear the concerns. Will this Friday, Nov 6th, work?
> > >
> > > вт, 3 нояб. 2020 г. в 10:09, Nikolay Izhikov <ni...@apache.org>:
> > >
> > >> Igniters, should we have a call for this topic?
> > >>
> > >>> 2 нояб. 2020 г., в 18:53, Pavel Tupitsyn <pt...@apache.org>
> > >> написал(а):
> > >>>
> > >>>> not intend to rewrite everything from scratch
> > >>>
> > >>>> Every single test from Ignite 2.x should be moved to Ignite 3
> > >>>> regardless of how we choose to proceed.
> > >>>
> > >>> Alexey, thank you for the explanation, this addresses all of my
> > concerns.
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> On Mon, Nov 2, 2020 at 6:43 PM Andrey Mashenkov <
> > >> andrey.mashenkov@gmail.com>
> > >>> wrote:
> > >>>
> > >>>> Hi, Igniters.
> > >>>>
> > >>>> * AFAIU, we need a new repo if we want to apply different
> restrictions
> > >> to
> > >>>> pull requests,
> > >>>> otherwise I see no difference for myself.
> > >>>> E.g. make static analysis (do we have?), compile, styles, and
> javadoc
> > >>>> checks mandatory.
> > >>>>
> > >>>> I think that relaxed requirements here will lead to bad product
> > quality.
> > >>>>
> > >>>> * Agree with Pavel, we should 'keep' integrations tests somehow.
> > >>>> During active development tests will be broken most of time, so,
> > >>>> I'd port them e.g. suite-by-suite once we will have a stable and
> > >> featured
> > >>>> environment to run them and of course make test's code clear and
> avoid
> > >>>> bad/non-relevant ones.
> > >>>>
> > >>>> * I like bottom-up approach.
> > >>>> With it we could make a better framework. I mean clear component
> > >> lifecycle,
> > >>>> component wiring mechanics, general methods to approach core
> > components
> > >>>> such as exchange/communication
> > >>>> to avoid code mess like we have in ExchangeFuture with all these
> > custom
> > >>>> callbacks for each component, interfaces like
> > >>>> PartitionsExchangeAware, IgniteChangeGlobalStateSupport and
> > >>>> a pack of
> > >> start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected
> > >>>> and so on in various unexpected places.
> > >>>> Hope, we will be able to port most of the good code to the new
> > framework
> > >>>> version.
> > >>>>
> > >>>>
> > >>>>
> > >>>> On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk <
> > >>>> alexey.goncharuk@gmail.com>
> > >>>> wrote:
> > >>>>
> > >>>>> Nikolay, Pavel,
> > >>>>>
> > >>>>> Thanks for the feedback! First of all, I wanted to stress that I do
> > not
> > >>>>> intend to rewrite everything from scratch (I never used this
> phrase).
> > >>>> There
> > >>>>> are significant parts of code that would be moved with minimal
> > >>>>> modifications. Second, I never said that we will get rid of the old
> > >> tests
> > >>>>> codebase. Every single test from Ignite 2.x should be moved to
> > Ignite 3
> > >>>>> regardless of how we choose to proceed.
> > >>>>>
> > >>>>> My point is that for some parts of the code a clean bottom-up
> > >>>>> implementation will be cheaper in many ways. Let me give you a few
> > >>>> concrete
> > >>>>> examples:
> > >>>>>
> > >>>>> - I think no one can object that we need a cleanly separated
> > >>>> persistence
> > >>>>> layer for Ignite. There is a very raw draft IEP for this already.
> On
> > >>>> the
> > >>>>> other hand, I think we also can agree that we need a split-brain
> > >>>>> resistant
> > >>>>> replication protocol for caches. There is also an IEP for this.
> > >>>> Neither
> > >>>>> of
> > >>>>> the changes is a good fit for 2.x because they are likely to
> > >> introduce
> > >>>>> breaking changes in the persistence layer, configuration and
> > >> behavior.
> > >>>>> Additionally, these components are now tightly coupled, so there
> is
> > >> no
> > >>>>> way
> > >>>>> these two changes can be implemented in parallel and then merged
> > >>>>> together
> > >>>>> easily. So what we will end up with is having to implement these
> > >>>> changes
> > >>>>> sequentially, fixing all existing tests twice, and essentially
> > >>>> throwing
> > >>>>> away half of the work done because the other part of the change is
> > >>>>> re-implemented
> > >>>>> - Similar example goes with getting rid of IgniteInternalFuture
> and
> > >>>>> replacing it with CompletableFuture, and any other change that
> > >> touches
> > >>>>> the
> > >>>>> asynchronous part of the code.
> > >>>>>
> > >>>>> Third, I do not see how this choice affects the UX of Ignite. The
> end
> > >>>> user
> > >>>>> experience must be fixed in the IEP regardless of the development
> > >> process
> > >>>>> and the fact that we have gaps in this area in Ignite 2.x just
> > confirms
> > >>>>> that.
> > >>>>>
> > >>>>> Pavel, agree that a repo/branch is a technicality, I guess if
> > >>>> reformulate,
> > >>>>> my point is that we might agree to have a single development master
> > >>>> branch
> > >>>>> with 'disassembled' end-to-end functionality for some period of
> time
> > to
> > >>>>> speed up development, and re-assemble the core features after
> having
> > >>>>> submodules tested independently.
> > >>>>>
> > >>>>> Nikolay,
> > >>>>>> We have many features that have to evolve.
> > >>>>>> Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
> > >>>>> This is not very specific. In the end, resources are limited and we
> > >> will
> > >>>>> not be able to drive both tracks simultaneously, especially after a
> > >>>> couple
> > >>>>> of features having been implemented for Ignite 3.0. If there are
> > indeed
> > >>>>> some major changes that we want to do in Ignite 2.x instead of
> > putting
> > >>>>> effort into 3.0 - let's discuss them. I am just not aware of any,
> > >> that's
> > >>>>> why I am eager to move forward with Ignite 3.0.
> > >>>>>
> > >>>>>> We have bugs and issues that can be fixed in 2.x without breaking
> > >>>> backward
> > >>>>> compatibility.
> > >>>>>> We have many users who are happy with the 2.x with all it’s
> issues.
> > >>>>> These changes will be covered by end-to-end tests and migrated to
> > >> Ignite
> > >>>>> 3.0, so I see no issues here.
> > >>>>>
> > >>>>> Finally, Anton & Nikolay
> > >>>>> I do not have an estimate for this simply because the activity is
> > >>>>> community-driven and it depends on the number of people willing to
> > >>>>> contribute. With the current pace, I would hope to have an RC of
> > Ignite
> > >>>> 3.0
> > >>>>> to be ready by the end of 2021. My gut feeling is that by moving
> with
> > >>>>> incremental changes, we will not be able to implement even half of
> > the
> > >>>>> wishlist by that time.
> > >>>>> I doubt that releasing several major releases with breaking changes
> > >> will
> > >>>>> make Ignite users happy either because each upgrade will cost
> Ignite
> > >>>> users
> > >>>>> money, so the fewer major versions we release, the better. Thus my
> > wish
> > >>>> to
> > >>>>> include all breaking changes in one release.
> > >>>>>
> > >>>>> I'll be now quiet for a while, let's see what other community
> members
> > >>>>> think.
> > >>>>>
> > >>>>> пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <ptupitsyn@apache.org
> >:
> > >>>>>
> > >>>>>> 1. Rewriting from scratch is never a good idea.
> > >>>>>> We don't want to follow the path of Netscape and lose all our
> users
> > >>>>>> by the time we have a working 3.0 [1]
> > >>>>>>
> > >>>>>> 2. Not sure about new repo - seems like some pain and no gain,
> > what's
> > >>>> the
> > >>>>>> problem with a branch?
> > >>>>>>
> > >>>>>> 3. We should keep existing integration tests when possible.
> > >>>>>> We have accumulated a lot of edge case knowledge over the years,
> > >>>>>> it is not a good idea to send all of that down the drain.
> > >>>>>> Yes, integration tests are slow, but they are the most valuable.
> > >>>>>> I think we can move more stuff into nightly runs and have a fast
> and
> > >>>>> modern
> > >>>>>> basic suite.
> > >>>>>>
> > >>>>>>
> > >>>>>> Alexey, you are much more familiar with the Ignite core codebase
> > than
> > >>>>> most
> > >>>>>> of us,
> > >>>>>> can you please explain in more detail which particular feature, in
> > >> your
> > >>>>>> opinion,
> > >>>>>> mandates this "start from scratch" approach?
> > >>>>>> Is it really not possible at all to follow a less radical way?
> > >>>>>>
> > >>>>>>
> > >>>>>> [1]
> > >>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>
> >
> https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
> > >>>>>>
> > >>>>>> On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <
> nizhikov@apache.org
> > >
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>> Hello, Alexey.
> > >>>>>>>
> > >>>>>>> I think that «rewriting from scratch» approach has a high risk to
> > >>>> make
> > >>>>>> new
> > >>>>>>> features unusable.
> > >>>>>>> At the time Ignite2 was started no-one wants to do bad UX or bad
> > >>>>>> features.
> > >>>>>>> Nevertheless, it happen.
> > >>>>>>>
> > >>>>>>> I think we can avoid it with the Ignite3 and successors if we
> will
> > >>>> move
> > >>>>>>> step by step without keeping backward compatibility
> > >>>>>>> With the step by step approach, we can focus on each component
> > >>>>>> separately.
> > >>>>>>>
> > >>>>>>>> What new features are we planning to implement for Ignite 2.x?
> > >>>>>>>
> > >>>>>>> We have many features that have to evolve.
> > >>>>>>> Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
> > >>>>>>> We have bugs and issues that can be fixed in 2.x without breaking
> > >>>>>> backward
> > >>>>>>> compatibility.
> > >>>>>>> We have many users who are happy with the 2.x with all it’s
> issues.
> > >>>>>>>
> > >>>>>>>> 2 нояб. 2020 г., в 14:09, Anton Vinogradov <av...@apache.org>
> > >>>>> написал(а):
> > >>>>>>>>
> > >>>>>>>> Alexey,
> > >>>>>>>>
> > >>>>>>>> Do we have any estimates of how fast we'll be able to gain
> > >>>>>>> production-ready
> > >>>>>>>> AI 3.0 in case of a "new repo" choice?
> > >>>>>>>>
> > >>>>>>>> On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk <
> > >>>>>>> alexey.goncharuk@gmail.com>
> > >>>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> Nikolay,
> > >>>>>>>>>
> > >>>>>>>>> What new features are we planning to implement for Ignite 2.x?
> I
> > >>>>> think
> > >>>>>>> once
> > >>>>>>>>> we commence working on Ignite 3.0, we should gradually cease
> the
> > >>>>>>> activity
> > >>>>>>>>> on Ignite 2.x to mere bugfixes because such parallel
> development
> > >>>>> will
> > >>>>>> be
> > >>>>>>>>> overwhelming regardless of how we choose to proceed.
> > >>>>>>>>>
> > >>>>>>>>> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <
> > nizhikov@apache.org
> > >>>>> :
> > >>>>>>>>>
> > >>>>>>>>>> To be clear:
> > >>>>>>>>>>
> > >>>>>>>>>>> I would suggest creating a new repository for Ignite 3.0
> > >>>>> (perhaps, a
> > >>>>>>>>> new
> > >>>>>>>>>> clean branch, but a new repo looks nicer to me) and a new
> Ignite
> > >>>>> 3.0
> > >>>>>>>>>> TeamCity project.
> > >>>>>>>>>>
> > >>>>>>>>>> +1 for new Team City project.
> > >>>>>>>>>> +1 for new branch for Ignite3.
> > >>>>>>>>>> -1 for new repo.
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <
> > >>>> NIzhikov.dev@gmail.com
> > >>>>>>
> > >>>>>>>>>> написал(а):
> > >>>>>>>>>>>
> > >>>>>>>>>>> Hello, Alexey.
> > >>>>>>>>>>>
> > >>>>>>>>>>> I think it will hurt our project more than help.
> > >>>>>>>>>>> Developing new features for 2 separate branches with the
> > >>>> different
> > >>>>>>> APIs
> > >>>>>>>>>> and internal structure is overwhelming
> > >>>>>>>>>>>
> > >>>>>>>>>>> Maybe we should relax a bit requirements for Ignite3?
> > >>>>>>>>>>> Maybe we should move step by step and make Ignite3 with new
> > >>>>>>>>>> configuration than Ignite4 with new transactions, etc?
> > >>>>>>>>>>>
> > >>>>>>>>>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <
> > >>>>>>>>> alexey.goncharuk@gmail.com>
> > >>>>>>>>>> написал(а):
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Igniters,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I wanted to pitch a rather radical idea regarding the Ignite
> > >>>> 3.0
> > >>>>>>>>>>>> development which has occurred to me some time ago.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> We already have several IEPs targeted to Ignite 3.0 which
> > imply
> > >>>>>> major
> > >>>>>>>>>>>> changes to the codebase (the change in replication protocol
> > and
> > >>>>>> thus
> > >>>>>>>>>>>> transactions, change in binary format, updated metastorage,
> > >>>> etc).
> > >>>>>> We
> > >>>>>>>>>> also
> > >>>>>>>>>>>> planned significant changes in public APIs: configuration
> > >>>> format
> > >>>>>>>>> change,
> > >>>>>>>>>>>> improvements in cache APIs, SQL APIs, transaction mode
> rework.
> > >>>>> The
> > >>>>>>>>>> wishlist
> > >>>>>>>>>>>> of changes for Ignite 3.0 is huge.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> So, I was wondering whether it makes sense to try to change
> > the
> > >>>>> old
> > >>>>>>>>>>>> codebase, or start with a new baseline and move old pieces
> of
> > >>>>> code
> > >>>>>>>>> that
> > >>>>>>>>>> do
> > >>>>>>>>>>>> not require significant rework. Personally, I would go with
> > the
> > >>>>>>> second
> > >>>>>>>>>>>> option for the following reasons:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> - We have a chance to shift the development paradigm in the
> > >>>>> project
> > >>>>>>>>> and
> > >>>>>>>>>>>> introduce the practice of true unit-tests. In the new
> baseline
> > >>>> at
> > >>>>>> the
> > >>>>>>>>>>>> beginning there will be no ability to run an end-to-end
> > >>>> scenario,
> > >>>>>>>>> thus
> > >>>>>>>>>> we
> > >>>>>>>>>>>> will be forced to write unit-tests. So far, such practice
> was
> > >>>>> hard
> > >>>>>> to
> > >>>>>>>>>>>> implement because of tight coupling between Ignite
> components
> > >>>> and
> > >>>>>>>>>> inability
> > >>>>>>>>>>>> to instantiate components without an instance of
> > KernalContext.
> > >>>>> For
> > >>>>>>>>>>>> example, we should be able to thoroughly test internal
> > >>>>> primitives,
> > >>>>>>>>>> such as
> > >>>>>>>>>>>> replication protocol (without actual communication),
> > >>>> distributed
> > >>>>>>>>>>>> metastorage contracts, persistence layer, etc.
> > >>>>>>>>>>>> - We will significantly reduce the development cycle in the
> > >>>>>> beginning
> > >>>>>>>>>>>> (right now the RunAll takes two hours of astronomical time
> > with
> > >>>>>> empty
> > >>>>>>>>>> TC;
> > >>>>>>>>>>>> in the new approach developer will be able to run ALL tests
> > >>>>> locally
> > >>>>>>>>> in
> > >>>>>>>>>> a
> > >>>>>>>>>>>> matter of minutes)
> > >>>>>>>>>>>> - We can get rid of TC bot and enforce green TC by
> integrating
> > >>>> TC
> > >>>>>>>>> build
> > >>>>>>>>>>>> results with GitHub PRs (the same way Travis is currently
> > >>>>>> integrated
> > >>>>>>>>>> to PR
> > >>>>>>>>>>>> check). We should restrict PR merge without a TC check
> > >>>>>>>>>>>> - We will still have to re-write all tests, but only once.
> If
> > >>>> we
> > >>>>>> try
> > >>>>>>>>> to
> > >>>>>>>>>>>> modify the old codebase, we would need to modify all the
> tests
> > >>>>> for
> > >>>>>>>>>> every
> > >>>>>>>>>>>> major change (public API change, configuration change)
> > >>>>>>>>>>>> - We will have fewer conflicts when working together. For
> > >>>>> example,
> > >>>>>> I
> > >>>>>>>>>>>> cannot imagine how one would merge two changes of getting
> rid
> > >>>> of
> > >>>>>>>>>>>> IgniteFuture and changes in replication protocol, for
> example
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Technically, I would suggest creating a new repository for
> > >>>> Ignite
> > >>>>>> 3.0
> > >>>>>>>>>>>> (perhaps, a new clean branch, but a new repo looks nicer to
> > me)
> > >>>>>> and a
> > >>>>>>>>>> new
> > >>>>>>>>>>>> Ignite 3.0 TeamCity project.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> While it may seem quite radical, I do believe that this
> > >>>> approach
> > >>>>>> will
> > >>>>>>>>>> give
> > >>>>>>>>>>>> us more benefits than trying to make such major changes in
> the
> > >>>>>>>>> existing
> > >>>>>>>>>>>> codebase. If needed, let's schedule a discord chat like
> before
> > >>>> to
> > >>>>>>>>>> discuss
> > >>>>>>>>>>>> this.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> WDYT?
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>>
> > >>>> --
> > >>>> Best regards,
> > >>>> Andrey V. Mashenkov
> > >>>>
> > >>
> > >>
> >
> >
>
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Kseniya Romanova <ro...@gmail.com>.
Let's do this community discussion open. Here's the link on zoom call in
Russian for Friday 6 PM:
https://www.meetup.com/Moscow-Apache-Ignite-Meetup/events/274360378/
вт, 3 нояб. 2020 г. в 12:49, Nikolay Izhikov <ni...@apache.org>:
> Time works for me.
>
> > 3 нояб. 2020 г., в 12:40, Alexey Goncharuk <al...@gmail.com>
> написал(а):
> >
> > Nikolay,
> >
> > I am up for the call. I will try to explain my reasoning in greater
> detail
> > and will be glad to hear the concerns. Will this Friday, Nov 6th, work?
> >
> > вт, 3 нояб. 2020 г. в 10:09, Nikolay Izhikov <ni...@apache.org>:
> >
> >> Igniters, should we have a call for this topic?
> >>
> >>> 2 нояб. 2020 г., в 18:53, Pavel Tupitsyn <pt...@apache.org>
> >> написал(а):
> >>>
> >>>> not intend to rewrite everything from scratch
> >>>
> >>>> Every single test from Ignite 2.x should be moved to Ignite 3
> >>>> regardless of how we choose to proceed.
> >>>
> >>> Alexey, thank you for the explanation, this addresses all of my
> concerns.
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On Mon, Nov 2, 2020 at 6:43 PM Andrey Mashenkov <
> >> andrey.mashenkov@gmail.com>
> >>> wrote:
> >>>
> >>>> Hi, Igniters.
> >>>>
> >>>> * AFAIU, we need a new repo if we want to apply different restrictions
> >> to
> >>>> pull requests,
> >>>> otherwise I see no difference for myself.
> >>>> E.g. make static analysis (do we have?), compile, styles, and javadoc
> >>>> checks mandatory.
> >>>>
> >>>> I think that relaxed requirements here will lead to bad product
> quality.
> >>>>
> >>>> * Agree with Pavel, we should 'keep' integrations tests somehow.
> >>>> During active development tests will be broken most of time, so,
> >>>> I'd port them e.g. suite-by-suite once we will have a stable and
> >> featured
> >>>> environment to run them and of course make test's code clear and avoid
> >>>> bad/non-relevant ones.
> >>>>
> >>>> * I like bottom-up approach.
> >>>> With it we could make a better framework. I mean clear component
> >> lifecycle,
> >>>> component wiring mechanics, general methods to approach core
> components
> >>>> such as exchange/communication
> >>>> to avoid code mess like we have in ExchangeFuture with all these
> custom
> >>>> callbacks for each component, interfaces like
> >>>> PartitionsExchangeAware, IgniteChangeGlobalStateSupport and
> >>>> a pack of
> >> start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected
> >>>> and so on in various unexpected places.
> >>>> Hope, we will be able to port most of the good code to the new
> framework
> >>>> version.
> >>>>
> >>>>
> >>>>
> >>>> On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk <
> >>>> alexey.goncharuk@gmail.com>
> >>>> wrote:
> >>>>
> >>>>> Nikolay, Pavel,
> >>>>>
> >>>>> Thanks for the feedback! First of all, I wanted to stress that I do
> not
> >>>>> intend to rewrite everything from scratch (I never used this phrase).
> >>>> There
> >>>>> are significant parts of code that would be moved with minimal
> >>>>> modifications. Second, I never said that we will get rid of the old
> >> tests
> >>>>> codebase. Every single test from Ignite 2.x should be moved to
> Ignite 3
> >>>>> regardless of how we choose to proceed.
> >>>>>
> >>>>> My point is that for some parts of the code a clean bottom-up
> >>>>> implementation will be cheaper in many ways. Let me give you a few
> >>>> concrete
> >>>>> examples:
> >>>>>
> >>>>> - I think no one can object that we need a cleanly separated
> >>>> persistence
> >>>>> layer for Ignite. There is a very raw draft IEP for this already. On
> >>>> the
> >>>>> other hand, I think we also can agree that we need a split-brain
> >>>>> resistant
> >>>>> replication protocol for caches. There is also an IEP for this.
> >>>> Neither
> >>>>> of
> >>>>> the changes is a good fit for 2.x because they are likely to
> >> introduce
> >>>>> breaking changes in the persistence layer, configuration and
> >> behavior.
> >>>>> Additionally, these components are now tightly coupled, so there is
> >> no
> >>>>> way
> >>>>> these two changes can be implemented in parallel and then merged
> >>>>> together
> >>>>> easily. So what we will end up with is having to implement these
> >>>> changes
> >>>>> sequentially, fixing all existing tests twice, and essentially
> >>>> throwing
> >>>>> away half of the work done because the other part of the change is
> >>>>> re-implemented
> >>>>> - Similar example goes with getting rid of IgniteInternalFuture and
> >>>>> replacing it with CompletableFuture, and any other change that
> >> touches
> >>>>> the
> >>>>> asynchronous part of the code.
> >>>>>
> >>>>> Third, I do not see how this choice affects the UX of Ignite. The end
> >>>> user
> >>>>> experience must be fixed in the IEP regardless of the development
> >> process
> >>>>> and the fact that we have gaps in this area in Ignite 2.x just
> confirms
> >>>>> that.
> >>>>>
> >>>>> Pavel, agree that a repo/branch is a technicality, I guess if
> >>>> reformulate,
> >>>>> my point is that we might agree to have a single development master
> >>>> branch
> >>>>> with 'disassembled' end-to-end functionality for some period of time
> to
> >>>>> speed up development, and re-assemble the core features after having
> >>>>> submodules tested independently.
> >>>>>
> >>>>> Nikolay,
> >>>>>> We have many features that have to evolve.
> >>>>>> Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
> >>>>> This is not very specific. In the end, resources are limited and we
> >> will
> >>>>> not be able to drive both tracks simultaneously, especially after a
> >>>> couple
> >>>>> of features having been implemented for Ignite 3.0. If there are
> indeed
> >>>>> some major changes that we want to do in Ignite 2.x instead of
> putting
> >>>>> effort into 3.0 - let's discuss them. I am just not aware of any,
> >> that's
> >>>>> why I am eager to move forward with Ignite 3.0.
> >>>>>
> >>>>>> We have bugs and issues that can be fixed in 2.x without breaking
> >>>> backward
> >>>>> compatibility.
> >>>>>> We have many users who are happy with the 2.x with all it’s issues.
> >>>>> These changes will be covered by end-to-end tests and migrated to
> >> Ignite
> >>>>> 3.0, so I see no issues here.
> >>>>>
> >>>>> Finally, Anton & Nikolay
> >>>>> I do not have an estimate for this simply because the activity is
> >>>>> community-driven and it depends on the number of people willing to
> >>>>> contribute. With the current pace, I would hope to have an RC of
> Ignite
> >>>> 3.0
> >>>>> to be ready by the end of 2021. My gut feeling is that by moving with
> >>>>> incremental changes, we will not be able to implement even half of
> the
> >>>>> wishlist by that time.
> >>>>> I doubt that releasing several major releases with breaking changes
> >> will
> >>>>> make Ignite users happy either because each upgrade will cost Ignite
> >>>> users
> >>>>> money, so the fewer major versions we release, the better. Thus my
> wish
> >>>> to
> >>>>> include all breaking changes in one release.
> >>>>>
> >>>>> I'll be now quiet for a while, let's see what other community members
> >>>>> think.
> >>>>>
> >>>>> пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <pt...@apache.org>:
> >>>>>
> >>>>>> 1. Rewriting from scratch is never a good idea.
> >>>>>> We don't want to follow the path of Netscape and lose all our users
> >>>>>> by the time we have a working 3.0 [1]
> >>>>>>
> >>>>>> 2. Not sure about new repo - seems like some pain and no gain,
> what's
> >>>> the
> >>>>>> problem with a branch?
> >>>>>>
> >>>>>> 3. We should keep existing integration tests when possible.
> >>>>>> We have accumulated a lot of edge case knowledge over the years,
> >>>>>> it is not a good idea to send all of that down the drain.
> >>>>>> Yes, integration tests are slow, but they are the most valuable.
> >>>>>> I think we can move more stuff into nightly runs and have a fast and
> >>>>> modern
> >>>>>> basic suite.
> >>>>>>
> >>>>>>
> >>>>>> Alexey, you are much more familiar with the Ignite core codebase
> than
> >>>>> most
> >>>>>> of us,
> >>>>>> can you please explain in more detail which particular feature, in
> >> your
> >>>>>> opinion,
> >>>>>> mandates this "start from scratch" approach?
> >>>>>> Is it really not possible at all to follow a less radical way?
> >>>>>>
> >>>>>>
> >>>>>> [1]
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>
> https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
> >>>>>>
> >>>>>> On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <nizhikov@apache.org
> >
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Hello, Alexey.
> >>>>>>>
> >>>>>>> I think that «rewriting from scratch» approach has a high risk to
> >>>> make
> >>>>>> new
> >>>>>>> features unusable.
> >>>>>>> At the time Ignite2 was started no-one wants to do bad UX or bad
> >>>>>> features.
> >>>>>>> Nevertheless, it happen.
> >>>>>>>
> >>>>>>> I think we can avoid it with the Ignite3 and successors if we will
> >>>> move
> >>>>>>> step by step without keeping backward compatibility
> >>>>>>> With the step by step approach, we can focus on each component
> >>>>>> separately.
> >>>>>>>
> >>>>>>>> What new features are we planning to implement for Ignite 2.x?
> >>>>>>>
> >>>>>>> We have many features that have to evolve.
> >>>>>>> Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
> >>>>>>> We have bugs and issues that can be fixed in 2.x without breaking
> >>>>>> backward
> >>>>>>> compatibility.
> >>>>>>> We have many users who are happy with the 2.x with all it’s issues.
> >>>>>>>
> >>>>>>>> 2 нояб. 2020 г., в 14:09, Anton Vinogradov <av...@apache.org>
> >>>>> написал(а):
> >>>>>>>>
> >>>>>>>> Alexey,
> >>>>>>>>
> >>>>>>>> Do we have any estimates of how fast we'll be able to gain
> >>>>>>> production-ready
> >>>>>>>> AI 3.0 in case of a "new repo" choice?
> >>>>>>>>
> >>>>>>>> On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk <
> >>>>>>> alexey.goncharuk@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Nikolay,
> >>>>>>>>>
> >>>>>>>>> What new features are we planning to implement for Ignite 2.x? I
> >>>>> think
> >>>>>>> once
> >>>>>>>>> we commence working on Ignite 3.0, we should gradually cease the
> >>>>>>> activity
> >>>>>>>>> on Ignite 2.x to mere bugfixes because such parallel development
> >>>>> will
> >>>>>> be
> >>>>>>>>> overwhelming regardless of how we choose to proceed.
> >>>>>>>>>
> >>>>>>>>> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <
> nizhikov@apache.org
> >>>>> :
> >>>>>>>>>
> >>>>>>>>>> To be clear:
> >>>>>>>>>>
> >>>>>>>>>>> I would suggest creating a new repository for Ignite 3.0
> >>>>> (perhaps, a
> >>>>>>>>> new
> >>>>>>>>>> clean branch, but a new repo looks nicer to me) and a new Ignite
> >>>>> 3.0
> >>>>>>>>>> TeamCity project.
> >>>>>>>>>>
> >>>>>>>>>> +1 for new Team City project.
> >>>>>>>>>> +1 for new branch for Ignite3.
> >>>>>>>>>> -1 for new repo.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <
> >>>> NIzhikov.dev@gmail.com
> >>>>>>
> >>>>>>>>>> написал(а):
> >>>>>>>>>>>
> >>>>>>>>>>> Hello, Alexey.
> >>>>>>>>>>>
> >>>>>>>>>>> I think it will hurt our project more than help.
> >>>>>>>>>>> Developing new features for 2 separate branches with the
> >>>> different
> >>>>>>> APIs
> >>>>>>>>>> and internal structure is overwhelming
> >>>>>>>>>>>
> >>>>>>>>>>> Maybe we should relax a bit requirements for Ignite3?
> >>>>>>>>>>> Maybe we should move step by step and make Ignite3 with new
> >>>>>>>>>> configuration than Ignite4 with new transactions, etc?
> >>>>>>>>>>>
> >>>>>>>>>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <
> >>>>>>>>> alexey.goncharuk@gmail.com>
> >>>>>>>>>> написал(а):
> >>>>>>>>>>>>
> >>>>>>>>>>>> Igniters,
> >>>>>>>>>>>>
> >>>>>>>>>>>> I wanted to pitch a rather radical idea regarding the Ignite
> >>>> 3.0
> >>>>>>>>>>>> development which has occurred to me some time ago.
> >>>>>>>>>>>>
> >>>>>>>>>>>> We already have several IEPs targeted to Ignite 3.0 which
> imply
> >>>>>> major
> >>>>>>>>>>>> changes to the codebase (the change in replication protocol
> and
> >>>>>> thus
> >>>>>>>>>>>> transactions, change in binary format, updated metastorage,
> >>>> etc).
> >>>>>> We
> >>>>>>>>>> also
> >>>>>>>>>>>> planned significant changes in public APIs: configuration
> >>>> format
> >>>>>>>>> change,
> >>>>>>>>>>>> improvements in cache APIs, SQL APIs, transaction mode rework.
> >>>>> The
> >>>>>>>>>> wishlist
> >>>>>>>>>>>> of changes for Ignite 3.0 is huge.
> >>>>>>>>>>>>
> >>>>>>>>>>>> So, I was wondering whether it makes sense to try to change
> the
> >>>>> old
> >>>>>>>>>>>> codebase, or start with a new baseline and move old pieces of
> >>>>> code
> >>>>>>>>> that
> >>>>>>>>>> do
> >>>>>>>>>>>> not require significant rework. Personally, I would go with
> the
> >>>>>>> second
> >>>>>>>>>>>> option for the following reasons:
> >>>>>>>>>>>>
> >>>>>>>>>>>> - We have a chance to shift the development paradigm in the
> >>>>> project
> >>>>>>>>> and
> >>>>>>>>>>>> introduce the practice of true unit-tests. In the new baseline
> >>>> at
> >>>>>> the
> >>>>>>>>>>>> beginning there will be no ability to run an end-to-end
> >>>> scenario,
> >>>>>>>>> thus
> >>>>>>>>>> we
> >>>>>>>>>>>> will be forced to write unit-tests. So far, such practice was
> >>>>> hard
> >>>>>> to
> >>>>>>>>>>>> implement because of tight coupling between Ignite components
> >>>> and
> >>>>>>>>>> inability
> >>>>>>>>>>>> to instantiate components without an instance of
> KernalContext.
> >>>>> For
> >>>>>>>>>>>> example, we should be able to thoroughly test internal
> >>>>> primitives,
> >>>>>>>>>> such as
> >>>>>>>>>>>> replication protocol (without actual communication),
> >>>> distributed
> >>>>>>>>>>>> metastorage contracts, persistence layer, etc.
> >>>>>>>>>>>> - We will significantly reduce the development cycle in the
> >>>>>> beginning
> >>>>>>>>>>>> (right now the RunAll takes two hours of astronomical time
> with
> >>>>>> empty
> >>>>>>>>>> TC;
> >>>>>>>>>>>> in the new approach developer will be able to run ALL tests
> >>>>> locally
> >>>>>>>>> in
> >>>>>>>>>> a
> >>>>>>>>>>>> matter of minutes)
> >>>>>>>>>>>> - We can get rid of TC bot and enforce green TC by integrating
> >>>> TC
> >>>>>>>>> build
> >>>>>>>>>>>> results with GitHub PRs (the same way Travis is currently
> >>>>>> integrated
> >>>>>>>>>> to PR
> >>>>>>>>>>>> check). We should restrict PR merge without a TC check
> >>>>>>>>>>>> - We will still have to re-write all tests, but only once. If
> >>>> we
> >>>>>> try
> >>>>>>>>> to
> >>>>>>>>>>>> modify the old codebase, we would need to modify all the tests
> >>>>> for
> >>>>>>>>>> every
> >>>>>>>>>>>> major change (public API change, configuration change)
> >>>>>>>>>>>> - We will have fewer conflicts when working together. For
> >>>>> example,
> >>>>>> I
> >>>>>>>>>>>> cannot imagine how one would merge two changes of getting rid
> >>>> of
> >>>>>>>>>>>> IgniteFuture and changes in replication protocol, for example
> >>>>>>>>>>>>
> >>>>>>>>>>>> Technically, I would suggest creating a new repository for
> >>>> Ignite
> >>>>>> 3.0
> >>>>>>>>>>>> (perhaps, a new clean branch, but a new repo looks nicer to
> me)
> >>>>>> and a
> >>>>>>>>>> new
> >>>>>>>>>>>> Ignite 3.0 TeamCity project.
> >>>>>>>>>>>>
> >>>>>>>>>>>> While it may seem quite radical, I do believe that this
> >>>> approach
> >>>>>> will
> >>>>>>>>>> give
> >>>>>>>>>>>> us more benefits than trying to make such major changes in the
> >>>>>>>>> existing
> >>>>>>>>>>>> codebase. If needed, let's schedule a discord chat like before
> >>>> to
> >>>>>>>>>> discuss
> >>>>>>>>>>>> this.
> >>>>>>>>>>>>
> >>>>>>>>>>>> WDYT?
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> Best regards,
> >>>> Andrey V. Mashenkov
> >>>>
> >>
> >>
>
>
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Nikolay Izhikov <ni...@apache.org>.
Time works for me.
> 3 нояб. 2020 г., в 12:40, Alexey Goncharuk <al...@gmail.com> написал(а):
>
> Nikolay,
>
> I am up for the call. I will try to explain my reasoning in greater detail
> and will be glad to hear the concerns. Will this Friday, Nov 6th, work?
>
> вт, 3 нояб. 2020 г. в 10:09, Nikolay Izhikov <ni...@apache.org>:
>
>> Igniters, should we have a call for this topic?
>>
>>> 2 нояб. 2020 г., в 18:53, Pavel Tupitsyn <pt...@apache.org>
>> написал(а):
>>>
>>>> not intend to rewrite everything from scratch
>>>
>>>> Every single test from Ignite 2.x should be moved to Ignite 3
>>>> regardless of how we choose to proceed.
>>>
>>> Alexey, thank you for the explanation, this addresses all of my concerns.
>>>
>>>
>>>
>>>
>>>
>>> On Mon, Nov 2, 2020 at 6:43 PM Andrey Mashenkov <
>> andrey.mashenkov@gmail.com>
>>> wrote:
>>>
>>>> Hi, Igniters.
>>>>
>>>> * AFAIU, we need a new repo if we want to apply different restrictions
>> to
>>>> pull requests,
>>>> otherwise I see no difference for myself.
>>>> E.g. make static analysis (do we have?), compile, styles, and javadoc
>>>> checks mandatory.
>>>>
>>>> I think that relaxed requirements here will lead to bad product quality.
>>>>
>>>> * Agree with Pavel, we should 'keep' integrations tests somehow.
>>>> During active development tests will be broken most of time, so,
>>>> I'd port them e.g. suite-by-suite once we will have a stable and
>> featured
>>>> environment to run them and of course make test's code clear and avoid
>>>> bad/non-relevant ones.
>>>>
>>>> * I like bottom-up approach.
>>>> With it we could make a better framework. I mean clear component
>> lifecycle,
>>>> component wiring mechanics, general methods to approach core components
>>>> such as exchange/communication
>>>> to avoid code mess like we have in ExchangeFuture with all these custom
>>>> callbacks for each component, interfaces like
>>>> PartitionsExchangeAware, IgniteChangeGlobalStateSupport and
>>>> a pack of
>> start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected
>>>> and so on in various unexpected places.
>>>> Hope, we will be able to port most of the good code to the new framework
>>>> version.
>>>>
>>>>
>>>>
>>>> On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk <
>>>> alexey.goncharuk@gmail.com>
>>>> wrote:
>>>>
>>>>> Nikolay, Pavel,
>>>>>
>>>>> Thanks for the feedback! First of all, I wanted to stress that I do not
>>>>> intend to rewrite everything from scratch (I never used this phrase).
>>>> There
>>>>> are significant parts of code that would be moved with minimal
>>>>> modifications. Second, I never said that we will get rid of the old
>> tests
>>>>> codebase. Every single test from Ignite 2.x should be moved to Ignite 3
>>>>> regardless of how we choose to proceed.
>>>>>
>>>>> My point is that for some parts of the code a clean bottom-up
>>>>> implementation will be cheaper in many ways. Let me give you a few
>>>> concrete
>>>>> examples:
>>>>>
>>>>> - I think no one can object that we need a cleanly separated
>>>> persistence
>>>>> layer for Ignite. There is a very raw draft IEP for this already. On
>>>> the
>>>>> other hand, I think we also can agree that we need a split-brain
>>>>> resistant
>>>>> replication protocol for caches. There is also an IEP for this.
>>>> Neither
>>>>> of
>>>>> the changes is a good fit for 2.x because they are likely to
>> introduce
>>>>> breaking changes in the persistence layer, configuration and
>> behavior.
>>>>> Additionally, these components are now tightly coupled, so there is
>> no
>>>>> way
>>>>> these two changes can be implemented in parallel and then merged
>>>>> together
>>>>> easily. So what we will end up with is having to implement these
>>>> changes
>>>>> sequentially, fixing all existing tests twice, and essentially
>>>> throwing
>>>>> away half of the work done because the other part of the change is
>>>>> re-implemented
>>>>> - Similar example goes with getting rid of IgniteInternalFuture and
>>>>> replacing it with CompletableFuture, and any other change that
>> touches
>>>>> the
>>>>> asynchronous part of the code.
>>>>>
>>>>> Third, I do not see how this choice affects the UX of Ignite. The end
>>>> user
>>>>> experience must be fixed in the IEP regardless of the development
>> process
>>>>> and the fact that we have gaps in this area in Ignite 2.x just confirms
>>>>> that.
>>>>>
>>>>> Pavel, agree that a repo/branch is a technicality, I guess if
>>>> reformulate,
>>>>> my point is that we might agree to have a single development master
>>>> branch
>>>>> with 'disassembled' end-to-end functionality for some period of time to
>>>>> speed up development, and re-assemble the core features after having
>>>>> submodules tested independently.
>>>>>
>>>>> Nikolay,
>>>>>> We have many features that have to evolve.
>>>>>> Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
>>>>> This is not very specific. In the end, resources are limited and we
>> will
>>>>> not be able to drive both tracks simultaneously, especially after a
>>>> couple
>>>>> of features having been implemented for Ignite 3.0. If there are indeed
>>>>> some major changes that we want to do in Ignite 2.x instead of putting
>>>>> effort into 3.0 - let's discuss them. I am just not aware of any,
>> that's
>>>>> why I am eager to move forward with Ignite 3.0.
>>>>>
>>>>>> We have bugs and issues that can be fixed in 2.x without breaking
>>>> backward
>>>>> compatibility.
>>>>>> We have many users who are happy with the 2.x with all it’s issues.
>>>>> These changes will be covered by end-to-end tests and migrated to
>> Ignite
>>>>> 3.0, so I see no issues here.
>>>>>
>>>>> Finally, Anton & Nikolay
>>>>> I do not have an estimate for this simply because the activity is
>>>>> community-driven and it depends on the number of people willing to
>>>>> contribute. With the current pace, I would hope to have an RC of Ignite
>>>> 3.0
>>>>> to be ready by the end of 2021. My gut feeling is that by moving with
>>>>> incremental changes, we will not be able to implement even half of the
>>>>> wishlist by that time.
>>>>> I doubt that releasing several major releases with breaking changes
>> will
>>>>> make Ignite users happy either because each upgrade will cost Ignite
>>>> users
>>>>> money, so the fewer major versions we release, the better. Thus my wish
>>>> to
>>>>> include all breaking changes in one release.
>>>>>
>>>>> I'll be now quiet for a while, let's see what other community members
>>>>> think.
>>>>>
>>>>> пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <pt...@apache.org>:
>>>>>
>>>>>> 1. Rewriting from scratch is never a good idea.
>>>>>> We don't want to follow the path of Netscape and lose all our users
>>>>>> by the time we have a working 3.0 [1]
>>>>>>
>>>>>> 2. Not sure about new repo - seems like some pain and no gain, what's
>>>> the
>>>>>> problem with a branch?
>>>>>>
>>>>>> 3. We should keep existing integration tests when possible.
>>>>>> We have accumulated a lot of edge case knowledge over the years,
>>>>>> it is not a good idea to send all of that down the drain.
>>>>>> Yes, integration tests are slow, but they are the most valuable.
>>>>>> I think we can move more stuff into nightly runs and have a fast and
>>>>> modern
>>>>>> basic suite.
>>>>>>
>>>>>>
>>>>>> Alexey, you are much more familiar with the Ignite core codebase than
>>>>> most
>>>>>> of us,
>>>>>> can you please explain in more detail which particular feature, in
>> your
>>>>>> opinion,
>>>>>> mandates this "start from scratch" approach?
>>>>>> Is it really not possible at all to follow a less radical way?
>>>>>>
>>>>>>
>>>>>> [1]
>>>>>>
>>>>>>
>>>>>
>>>>
>> https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
>>>>>>
>>>>>> On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <ni...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> Hello, Alexey.
>>>>>>>
>>>>>>> I think that «rewriting from scratch» approach has a high risk to
>>>> make
>>>>>> new
>>>>>>> features unusable.
>>>>>>> At the time Ignite2 was started no-one wants to do bad UX or bad
>>>>>> features.
>>>>>>> Nevertheless, it happen.
>>>>>>>
>>>>>>> I think we can avoid it with the Ignite3 and successors if we will
>>>> move
>>>>>>> step by step without keeping backward compatibility
>>>>>>> With the step by step approach, we can focus on each component
>>>>>> separately.
>>>>>>>
>>>>>>>> What new features are we planning to implement for Ignite 2.x?
>>>>>>>
>>>>>>> We have many features that have to evolve.
>>>>>>> Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
>>>>>>> We have bugs and issues that can be fixed in 2.x without breaking
>>>>>> backward
>>>>>>> compatibility.
>>>>>>> We have many users who are happy with the 2.x with all it’s issues.
>>>>>>>
>>>>>>>> 2 нояб. 2020 г., в 14:09, Anton Vinogradov <av...@apache.org>
>>>>> написал(а):
>>>>>>>>
>>>>>>>> Alexey,
>>>>>>>>
>>>>>>>> Do we have any estimates of how fast we'll be able to gain
>>>>>>> production-ready
>>>>>>>> AI 3.0 in case of a "new repo" choice?
>>>>>>>>
>>>>>>>> On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk <
>>>>>>> alexey.goncharuk@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Nikolay,
>>>>>>>>>
>>>>>>>>> What new features are we planning to implement for Ignite 2.x? I
>>>>> think
>>>>>>> once
>>>>>>>>> we commence working on Ignite 3.0, we should gradually cease the
>>>>>>> activity
>>>>>>>>> on Ignite 2.x to mere bugfixes because such parallel development
>>>>> will
>>>>>> be
>>>>>>>>> overwhelming regardless of how we choose to proceed.
>>>>>>>>>
>>>>>>>>> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <nizhikov@apache.org
>>>>> :
>>>>>>>>>
>>>>>>>>>> To be clear:
>>>>>>>>>>
>>>>>>>>>>> I would suggest creating a new repository for Ignite 3.0
>>>>> (perhaps, a
>>>>>>>>> new
>>>>>>>>>> clean branch, but a new repo looks nicer to me) and a new Ignite
>>>>> 3.0
>>>>>>>>>> TeamCity project.
>>>>>>>>>>
>>>>>>>>>> +1 for new Team City project.
>>>>>>>>>> +1 for new branch for Ignite3.
>>>>>>>>>> -1 for new repo.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <
>>>> NIzhikov.dev@gmail.com
>>>>>>
>>>>>>>>>> написал(а):
>>>>>>>>>>>
>>>>>>>>>>> Hello, Alexey.
>>>>>>>>>>>
>>>>>>>>>>> I think it will hurt our project more than help.
>>>>>>>>>>> Developing new features for 2 separate branches with the
>>>> different
>>>>>>> APIs
>>>>>>>>>> and internal structure is overwhelming
>>>>>>>>>>>
>>>>>>>>>>> Maybe we should relax a bit requirements for Ignite3?
>>>>>>>>>>> Maybe we should move step by step and make Ignite3 with new
>>>>>>>>>> configuration than Ignite4 with new transactions, etc?
>>>>>>>>>>>
>>>>>>>>>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <
>>>>>>>>> alexey.goncharuk@gmail.com>
>>>>>>>>>> написал(а):
>>>>>>>>>>>>
>>>>>>>>>>>> Igniters,
>>>>>>>>>>>>
>>>>>>>>>>>> I wanted to pitch a rather radical idea regarding the Ignite
>>>> 3.0
>>>>>>>>>>>> development which has occurred to me some time ago.
>>>>>>>>>>>>
>>>>>>>>>>>> We already have several IEPs targeted to Ignite 3.0 which imply
>>>>>> major
>>>>>>>>>>>> changes to the codebase (the change in replication protocol and
>>>>>> thus
>>>>>>>>>>>> transactions, change in binary format, updated metastorage,
>>>> etc).
>>>>>> We
>>>>>>>>>> also
>>>>>>>>>>>> planned significant changes in public APIs: configuration
>>>> format
>>>>>>>>> change,
>>>>>>>>>>>> improvements in cache APIs, SQL APIs, transaction mode rework.
>>>>> The
>>>>>>>>>> wishlist
>>>>>>>>>>>> of changes for Ignite 3.0 is huge.
>>>>>>>>>>>>
>>>>>>>>>>>> So, I was wondering whether it makes sense to try to change the
>>>>> old
>>>>>>>>>>>> codebase, or start with a new baseline and move old pieces of
>>>>> code
>>>>>>>>> that
>>>>>>>>>> do
>>>>>>>>>>>> not require significant rework. Personally, I would go with the
>>>>>>> second
>>>>>>>>>>>> option for the following reasons:
>>>>>>>>>>>>
>>>>>>>>>>>> - We have a chance to shift the development paradigm in the
>>>>> project
>>>>>>>>> and
>>>>>>>>>>>> introduce the practice of true unit-tests. In the new baseline
>>>> at
>>>>>> the
>>>>>>>>>>>> beginning there will be no ability to run an end-to-end
>>>> scenario,
>>>>>>>>> thus
>>>>>>>>>> we
>>>>>>>>>>>> will be forced to write unit-tests. So far, such practice was
>>>>> hard
>>>>>> to
>>>>>>>>>>>> implement because of tight coupling between Ignite components
>>>> and
>>>>>>>>>> inability
>>>>>>>>>>>> to instantiate components without an instance of KernalContext.
>>>>> For
>>>>>>>>>>>> example, we should be able to thoroughly test internal
>>>>> primitives,
>>>>>>>>>> such as
>>>>>>>>>>>> replication protocol (without actual communication),
>>>> distributed
>>>>>>>>>>>> metastorage contracts, persistence layer, etc.
>>>>>>>>>>>> - We will significantly reduce the development cycle in the
>>>>>> beginning
>>>>>>>>>>>> (right now the RunAll takes two hours of astronomical time with
>>>>>> empty
>>>>>>>>>> TC;
>>>>>>>>>>>> in the new approach developer will be able to run ALL tests
>>>>> locally
>>>>>>>>> in
>>>>>>>>>> a
>>>>>>>>>>>> matter of minutes)
>>>>>>>>>>>> - We can get rid of TC bot and enforce green TC by integrating
>>>> TC
>>>>>>>>> build
>>>>>>>>>>>> results with GitHub PRs (the same way Travis is currently
>>>>>> integrated
>>>>>>>>>> to PR
>>>>>>>>>>>> check). We should restrict PR merge without a TC check
>>>>>>>>>>>> - We will still have to re-write all tests, but only once. If
>>>> we
>>>>>> try
>>>>>>>>> to
>>>>>>>>>>>> modify the old codebase, we would need to modify all the tests
>>>>> for
>>>>>>>>>> every
>>>>>>>>>>>> major change (public API change, configuration change)
>>>>>>>>>>>> - We will have fewer conflicts when working together. For
>>>>> example,
>>>>>> I
>>>>>>>>>>>> cannot imagine how one would merge two changes of getting rid
>>>> of
>>>>>>>>>>>> IgniteFuture and changes in replication protocol, for example
>>>>>>>>>>>>
>>>>>>>>>>>> Technically, I would suggest creating a new repository for
>>>> Ignite
>>>>>> 3.0
>>>>>>>>>>>> (perhaps, a new clean branch, but a new repo looks nicer to me)
>>>>>> and a
>>>>>>>>>> new
>>>>>>>>>>>> Ignite 3.0 TeamCity project.
>>>>>>>>>>>>
>>>>>>>>>>>> While it may seem quite radical, I do believe that this
>>>> approach
>>>>>> will
>>>>>>>>>> give
>>>>>>>>>>>> us more benefits than trying to make such major changes in the
>>>>>>>>> existing
>>>>>>>>>>>> codebase. If needed, let's schedule a discord chat like before
>>>> to
>>>>>>>>>> discuss
>>>>>>>>>>>> this.
>>>>>>>>>>>>
>>>>>>>>>>>> WDYT?
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Best regards,
>>>> Andrey V. Mashenkov
>>>>
>>
>>
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Alexey Goncharuk <al...@gmail.com>.
Nikolay,
I am up for the call. I will try to explain my reasoning in greater detail
and will be glad to hear the concerns. Will this Friday, Nov 6th, work?
вт, 3 нояб. 2020 г. в 10:09, Nikolay Izhikov <ni...@apache.org>:
> Igniters, should we have a call for this topic?
>
> > 2 нояб. 2020 г., в 18:53, Pavel Tupitsyn <pt...@apache.org>
> написал(а):
> >
> >> not intend to rewrite everything from scratch
> >
> >> Every single test from Ignite 2.x should be moved to Ignite 3
> >> regardless of how we choose to proceed.
> >
> > Alexey, thank you for the explanation, this addresses all of my concerns.
> >
> >
> >
> >
> >
> > On Mon, Nov 2, 2020 at 6:43 PM Andrey Mashenkov <
> andrey.mashenkov@gmail.com>
> > wrote:
> >
> >> Hi, Igniters.
> >>
> >> * AFAIU, we need a new repo if we want to apply different restrictions
> to
> >> pull requests,
> >> otherwise I see no difference for myself.
> >> E.g. make static analysis (do we have?), compile, styles, and javadoc
> >> checks mandatory.
> >>
> >> I think that relaxed requirements here will lead to bad product quality.
> >>
> >> * Agree with Pavel, we should 'keep' integrations tests somehow.
> >> During active development tests will be broken most of time, so,
> >> I'd port them e.g. suite-by-suite once we will have a stable and
> featured
> >> environment to run them and of course make test's code clear and avoid
> >> bad/non-relevant ones.
> >>
> >> * I like bottom-up approach.
> >> With it we could make a better framework. I mean clear component
> lifecycle,
> >> component wiring mechanics, general methods to approach core components
> >> such as exchange/communication
> >> to avoid code mess like we have in ExchangeFuture with all these custom
> >> callbacks for each component, interfaces like
> >> PartitionsExchangeAware, IgniteChangeGlobalStateSupport and
> >> a pack of
> start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected
> >> and so on in various unexpected places.
> >> Hope, we will be able to port most of the good code to the new framework
> >> version.
> >>
> >>
> >>
> >> On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk <
> >> alexey.goncharuk@gmail.com>
> >> wrote:
> >>
> >>> Nikolay, Pavel,
> >>>
> >>> Thanks for the feedback! First of all, I wanted to stress that I do not
> >>> intend to rewrite everything from scratch (I never used this phrase).
> >> There
> >>> are significant parts of code that would be moved with minimal
> >>> modifications. Second, I never said that we will get rid of the old
> tests
> >>> codebase. Every single test from Ignite 2.x should be moved to Ignite 3
> >>> regardless of how we choose to proceed.
> >>>
> >>> My point is that for some parts of the code a clean bottom-up
> >>> implementation will be cheaper in many ways. Let me give you a few
> >> concrete
> >>> examples:
> >>>
> >>> - I think no one can object that we need a cleanly separated
> >> persistence
> >>> layer for Ignite. There is a very raw draft IEP for this already. On
> >> the
> >>> other hand, I think we also can agree that we need a split-brain
> >>> resistant
> >>> replication protocol for caches. There is also an IEP for this.
> >> Neither
> >>> of
> >>> the changes is a good fit for 2.x because they are likely to
> introduce
> >>> breaking changes in the persistence layer, configuration and
> behavior.
> >>> Additionally, these components are now tightly coupled, so there is
> no
> >>> way
> >>> these two changes can be implemented in parallel and then merged
> >>> together
> >>> easily. So what we will end up with is having to implement these
> >> changes
> >>> sequentially, fixing all existing tests twice, and essentially
> >> throwing
> >>> away half of the work done because the other part of the change is
> >>> re-implemented
> >>> - Similar example goes with getting rid of IgniteInternalFuture and
> >>> replacing it with CompletableFuture, and any other change that
> touches
> >>> the
> >>> asynchronous part of the code.
> >>>
> >>> Third, I do not see how this choice affects the UX of Ignite. The end
> >> user
> >>> experience must be fixed in the IEP regardless of the development
> process
> >>> and the fact that we have gaps in this area in Ignite 2.x just confirms
> >>> that.
> >>>
> >>> Pavel, agree that a repo/branch is a technicality, I guess if
> >> reformulate,
> >>> my point is that we might agree to have a single development master
> >> branch
> >>> with 'disassembled' end-to-end functionality for some period of time to
> >>> speed up development, and re-assemble the core features after having
> >>> submodules tested independently.
> >>>
> >>> Nikolay,
> >>>> We have many features that have to evolve.
> >>>> Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
> >>> This is not very specific. In the end, resources are limited and we
> will
> >>> not be able to drive both tracks simultaneously, especially after a
> >> couple
> >>> of features having been implemented for Ignite 3.0. If there are indeed
> >>> some major changes that we want to do in Ignite 2.x instead of putting
> >>> effort into 3.0 - let's discuss them. I am just not aware of any,
> that's
> >>> why I am eager to move forward with Ignite 3.0.
> >>>
> >>>> We have bugs and issues that can be fixed in 2.x without breaking
> >> backward
> >>> compatibility.
> >>>> We have many users who are happy with the 2.x with all it’s issues.
> >>> These changes will be covered by end-to-end tests and migrated to
> Ignite
> >>> 3.0, so I see no issues here.
> >>>
> >>> Finally, Anton & Nikolay
> >>> I do not have an estimate for this simply because the activity is
> >>> community-driven and it depends on the number of people willing to
> >>> contribute. With the current pace, I would hope to have an RC of Ignite
> >> 3.0
> >>> to be ready by the end of 2021. My gut feeling is that by moving with
> >>> incremental changes, we will not be able to implement even half of the
> >>> wishlist by that time.
> >>> I doubt that releasing several major releases with breaking changes
> will
> >>> make Ignite users happy either because each upgrade will cost Ignite
> >> users
> >>> money, so the fewer major versions we release, the better. Thus my wish
> >> to
> >>> include all breaking changes in one release.
> >>>
> >>> I'll be now quiet for a while, let's see what other community members
> >>> think.
> >>>
> >>> пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <pt...@apache.org>:
> >>>
> >>>> 1. Rewriting from scratch is never a good idea.
> >>>> We don't want to follow the path of Netscape and lose all our users
> >>>> by the time we have a working 3.0 [1]
> >>>>
> >>>> 2. Not sure about new repo - seems like some pain and no gain, what's
> >> the
> >>>> problem with a branch?
> >>>>
> >>>> 3. We should keep existing integration tests when possible.
> >>>> We have accumulated a lot of edge case knowledge over the years,
> >>>> it is not a good idea to send all of that down the drain.
> >>>> Yes, integration tests are slow, but they are the most valuable.
> >>>> I think we can move more stuff into nightly runs and have a fast and
> >>> modern
> >>>> basic suite.
> >>>>
> >>>>
> >>>> Alexey, you are much more familiar with the Ignite core codebase than
> >>> most
> >>>> of us,
> >>>> can you please explain in more detail which particular feature, in
> your
> >>>> opinion,
> >>>> mandates this "start from scratch" approach?
> >>>> Is it really not possible at all to follow a less radical way?
> >>>>
> >>>>
> >>>> [1]
> >>>>
> >>>>
> >>>
> >>
> https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
> >>>>
> >>>> On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <ni...@apache.org>
> >>>> wrote:
> >>>>
> >>>>> Hello, Alexey.
> >>>>>
> >>>>> I think that «rewriting from scratch» approach has a high risk to
> >> make
> >>>> new
> >>>>> features unusable.
> >>>>> At the time Ignite2 was started no-one wants to do bad UX or bad
> >>>> features.
> >>>>> Nevertheless, it happen.
> >>>>>
> >>>>> I think we can avoid it with the Ignite3 and successors if we will
> >> move
> >>>>> step by step without keeping backward compatibility
> >>>>> With the step by step approach, we can focus on each component
> >>>> separately.
> >>>>>
> >>>>>> What new features are we planning to implement for Ignite 2.x?
> >>>>>
> >>>>> We have many features that have to evolve.
> >>>>> Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
> >>>>> We have bugs and issues that can be fixed in 2.x without breaking
> >>>> backward
> >>>>> compatibility.
> >>>>> We have many users who are happy with the 2.x with all it’s issues.
> >>>>>
> >>>>>> 2 нояб. 2020 г., в 14:09, Anton Vinogradov <av...@apache.org>
> >>> написал(а):
> >>>>>>
> >>>>>> Alexey,
> >>>>>>
> >>>>>> Do we have any estimates of how fast we'll be able to gain
> >>>>> production-ready
> >>>>>> AI 3.0 in case of a "new repo" choice?
> >>>>>>
> >>>>>> On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk <
> >>>>> alexey.goncharuk@gmail.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Nikolay,
> >>>>>>>
> >>>>>>> What new features are we planning to implement for Ignite 2.x? I
> >>> think
> >>>>> once
> >>>>>>> we commence working on Ignite 3.0, we should gradually cease the
> >>>>> activity
> >>>>>>> on Ignite 2.x to mere bugfixes because such parallel development
> >>> will
> >>>> be
> >>>>>>> overwhelming regardless of how we choose to proceed.
> >>>>>>>
> >>>>>>> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <nizhikov@apache.org
> >>> :
> >>>>>>>
> >>>>>>>> To be clear:
> >>>>>>>>
> >>>>>>>>> I would suggest creating a new repository for Ignite 3.0
> >>> (perhaps, a
> >>>>>>> new
> >>>>>>>> clean branch, but a new repo looks nicer to me) and a new Ignite
> >>> 3.0
> >>>>>>>> TeamCity project.
> >>>>>>>>
> >>>>>>>> +1 for new Team City project.
> >>>>>>>> +1 for new branch for Ignite3.
> >>>>>>>> -1 for new repo.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <
> >> NIzhikov.dev@gmail.com
> >>>>
> >>>>>>>> написал(а):
> >>>>>>>>>
> >>>>>>>>> Hello, Alexey.
> >>>>>>>>>
> >>>>>>>>> I think it will hurt our project more than help.
> >>>>>>>>> Developing new features for 2 separate branches with the
> >> different
> >>>>> APIs
> >>>>>>>> and internal structure is overwhelming
> >>>>>>>>>
> >>>>>>>>> Maybe we should relax a bit requirements for Ignite3?
> >>>>>>>>> Maybe we should move step by step and make Ignite3 with new
> >>>>>>>> configuration than Ignite4 with new transactions, etc?
> >>>>>>>>>
> >>>>>>>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <
> >>>>>>> alexey.goncharuk@gmail.com>
> >>>>>>>> написал(а):
> >>>>>>>>>>
> >>>>>>>>>> Igniters,
> >>>>>>>>>>
> >>>>>>>>>> I wanted to pitch a rather radical idea regarding the Ignite
> >> 3.0
> >>>>>>>>>> development which has occurred to me some time ago.
> >>>>>>>>>>
> >>>>>>>>>> We already have several IEPs targeted to Ignite 3.0 which imply
> >>>> major
> >>>>>>>>>> changes to the codebase (the change in replication protocol and
> >>>> thus
> >>>>>>>>>> transactions, change in binary format, updated metastorage,
> >> etc).
> >>>> We
> >>>>>>>> also
> >>>>>>>>>> planned significant changes in public APIs: configuration
> >> format
> >>>>>>> change,
> >>>>>>>>>> improvements in cache APIs, SQL APIs, transaction mode rework.
> >>> The
> >>>>>>>> wishlist
> >>>>>>>>>> of changes for Ignite 3.0 is huge.
> >>>>>>>>>>
> >>>>>>>>>> So, I was wondering whether it makes sense to try to change the
> >>> old
> >>>>>>>>>> codebase, or start with a new baseline and move old pieces of
> >>> code
> >>>>>>> that
> >>>>>>>> do
> >>>>>>>>>> not require significant rework. Personally, I would go with the
> >>>>> second
> >>>>>>>>>> option for the following reasons:
> >>>>>>>>>>
> >>>>>>>>>> - We have a chance to shift the development paradigm in the
> >>> project
> >>>>>>> and
> >>>>>>>>>> introduce the practice of true unit-tests. In the new baseline
> >> at
> >>>> the
> >>>>>>>>>> beginning there will be no ability to run an end-to-end
> >> scenario,
> >>>>>>> thus
> >>>>>>>> we
> >>>>>>>>>> will be forced to write unit-tests. So far, such practice was
> >>> hard
> >>>> to
> >>>>>>>>>> implement because of tight coupling between Ignite components
> >> and
> >>>>>>>> inability
> >>>>>>>>>> to instantiate components without an instance of KernalContext.
> >>> For
> >>>>>>>>>> example, we should be able to thoroughly test internal
> >>> primitives,
> >>>>>>>> such as
> >>>>>>>>>> replication protocol (without actual communication),
> >> distributed
> >>>>>>>>>> metastorage contracts, persistence layer, etc.
> >>>>>>>>>> - We will significantly reduce the development cycle in the
> >>>> beginning
> >>>>>>>>>> (right now the RunAll takes two hours of astronomical time with
> >>>> empty
> >>>>>>>> TC;
> >>>>>>>>>> in the new approach developer will be able to run ALL tests
> >>> locally
> >>>>>>> in
> >>>>>>>> a
> >>>>>>>>>> matter of minutes)
> >>>>>>>>>> - We can get rid of TC bot and enforce green TC by integrating
> >> TC
> >>>>>>> build
> >>>>>>>>>> results with GitHub PRs (the same way Travis is currently
> >>>> integrated
> >>>>>>>> to PR
> >>>>>>>>>> check). We should restrict PR merge without a TC check
> >>>>>>>>>> - We will still have to re-write all tests, but only once. If
> >> we
> >>>> try
> >>>>>>> to
> >>>>>>>>>> modify the old codebase, we would need to modify all the tests
> >>> for
> >>>>>>>> every
> >>>>>>>>>> major change (public API change, configuration change)
> >>>>>>>>>> - We will have fewer conflicts when working together. For
> >>> example,
> >>>> I
> >>>>>>>>>> cannot imagine how one would merge two changes of getting rid
> >> of
> >>>>>>>>>> IgniteFuture and changes in replication protocol, for example
> >>>>>>>>>>
> >>>>>>>>>> Technically, I would suggest creating a new repository for
> >> Ignite
> >>>> 3.0
> >>>>>>>>>> (perhaps, a new clean branch, but a new repo looks nicer to me)
> >>>> and a
> >>>>>>>> new
> >>>>>>>>>> Ignite 3.0 TeamCity project.
> >>>>>>>>>>
> >>>>>>>>>> While it may seem quite radical, I do believe that this
> >> approach
> >>>> will
> >>>>>>>> give
> >>>>>>>>>> us more benefits than trying to make such major changes in the
> >>>>>>> existing
> >>>>>>>>>> codebase. If needed, let's schedule a discord chat like before
> >> to
> >>>>>>>> discuss
> >>>>>>>>>> this.
> >>>>>>>>>>
> >>>>>>>>>> WDYT?
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>
> >>
> >> --
> >> Best regards,
> >> Andrey V. Mashenkov
> >>
>
>
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Nikolay Izhikov <ni...@apache.org>.
Igniters, should we have a call for this topic?
> 2 нояб. 2020 г., в 18:53, Pavel Tupitsyn <pt...@apache.org> написал(а):
>
>> not intend to rewrite everything from scratch
>
>> Every single test from Ignite 2.x should be moved to Ignite 3
>> regardless of how we choose to proceed.
>
> Alexey, thank you for the explanation, this addresses all of my concerns.
>
>
>
>
>
> On Mon, Nov 2, 2020 at 6:43 PM Andrey Mashenkov <an...@gmail.com>
> wrote:
>
>> Hi, Igniters.
>>
>> * AFAIU, we need a new repo if we want to apply different restrictions to
>> pull requests,
>> otherwise I see no difference for myself.
>> E.g. make static analysis (do we have?), compile, styles, and javadoc
>> checks mandatory.
>>
>> I think that relaxed requirements here will lead to bad product quality.
>>
>> * Agree with Pavel, we should 'keep' integrations tests somehow.
>> During active development tests will be broken most of time, so,
>> I'd port them e.g. suite-by-suite once we will have a stable and featured
>> environment to run them and of course make test's code clear and avoid
>> bad/non-relevant ones.
>>
>> * I like bottom-up approach.
>> With it we could make a better framework. I mean clear component lifecycle,
>> component wiring mechanics, general methods to approach core components
>> such as exchange/communication
>> to avoid code mess like we have in ExchangeFuture with all these custom
>> callbacks for each component, interfaces like
>> PartitionsExchangeAware, IgniteChangeGlobalStateSupport and
>> a pack of start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected
>> and so on in various unexpected places.
>> Hope, we will be able to port most of the good code to the new framework
>> version.
>>
>>
>>
>> On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk <
>> alexey.goncharuk@gmail.com>
>> wrote:
>>
>>> Nikolay, Pavel,
>>>
>>> Thanks for the feedback! First of all, I wanted to stress that I do not
>>> intend to rewrite everything from scratch (I never used this phrase).
>> There
>>> are significant parts of code that would be moved with minimal
>>> modifications. Second, I never said that we will get rid of the old tests
>>> codebase. Every single test from Ignite 2.x should be moved to Ignite 3
>>> regardless of how we choose to proceed.
>>>
>>> My point is that for some parts of the code a clean bottom-up
>>> implementation will be cheaper in many ways. Let me give you a few
>> concrete
>>> examples:
>>>
>>> - I think no one can object that we need a cleanly separated
>> persistence
>>> layer for Ignite. There is a very raw draft IEP for this already. On
>> the
>>> other hand, I think we also can agree that we need a split-brain
>>> resistant
>>> replication protocol for caches. There is also an IEP for this.
>> Neither
>>> of
>>> the changes is a good fit for 2.x because they are likely to introduce
>>> breaking changes in the persistence layer, configuration and behavior.
>>> Additionally, these components are now tightly coupled, so there is no
>>> way
>>> these two changes can be implemented in parallel and then merged
>>> together
>>> easily. So what we will end up with is having to implement these
>> changes
>>> sequentially, fixing all existing tests twice, and essentially
>> throwing
>>> away half of the work done because the other part of the change is
>>> re-implemented
>>> - Similar example goes with getting rid of IgniteInternalFuture and
>>> replacing it with CompletableFuture, and any other change that touches
>>> the
>>> asynchronous part of the code.
>>>
>>> Third, I do not see how this choice affects the UX of Ignite. The end
>> user
>>> experience must be fixed in the IEP regardless of the development process
>>> and the fact that we have gaps in this area in Ignite 2.x just confirms
>>> that.
>>>
>>> Pavel, agree that a repo/branch is a technicality, I guess if
>> reformulate,
>>> my point is that we might agree to have a single development master
>> branch
>>> with 'disassembled' end-to-end functionality for some period of time to
>>> speed up development, and re-assemble the core features after having
>>> submodules tested independently.
>>>
>>> Nikolay,
>>>> We have many features that have to evolve.
>>>> Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
>>> This is not very specific. In the end, resources are limited and we will
>>> not be able to drive both tracks simultaneously, especially after a
>> couple
>>> of features having been implemented for Ignite 3.0. If there are indeed
>>> some major changes that we want to do in Ignite 2.x instead of putting
>>> effort into 3.0 - let's discuss them. I am just not aware of any, that's
>>> why I am eager to move forward with Ignite 3.0.
>>>
>>>> We have bugs and issues that can be fixed in 2.x without breaking
>> backward
>>> compatibility.
>>>> We have many users who are happy with the 2.x with all it’s issues.
>>> These changes will be covered by end-to-end tests and migrated to Ignite
>>> 3.0, so I see no issues here.
>>>
>>> Finally, Anton & Nikolay
>>> I do not have an estimate for this simply because the activity is
>>> community-driven and it depends on the number of people willing to
>>> contribute. With the current pace, I would hope to have an RC of Ignite
>> 3.0
>>> to be ready by the end of 2021. My gut feeling is that by moving with
>>> incremental changes, we will not be able to implement even half of the
>>> wishlist by that time.
>>> I doubt that releasing several major releases with breaking changes will
>>> make Ignite users happy either because each upgrade will cost Ignite
>> users
>>> money, so the fewer major versions we release, the better. Thus my wish
>> to
>>> include all breaking changes in one release.
>>>
>>> I'll be now quiet for a while, let's see what other community members
>>> think.
>>>
>>> пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <pt...@apache.org>:
>>>
>>>> 1. Rewriting from scratch is never a good idea.
>>>> We don't want to follow the path of Netscape and lose all our users
>>>> by the time we have a working 3.0 [1]
>>>>
>>>> 2. Not sure about new repo - seems like some pain and no gain, what's
>> the
>>>> problem with a branch?
>>>>
>>>> 3. We should keep existing integration tests when possible.
>>>> We have accumulated a lot of edge case knowledge over the years,
>>>> it is not a good idea to send all of that down the drain.
>>>> Yes, integration tests are slow, but they are the most valuable.
>>>> I think we can move more stuff into nightly runs and have a fast and
>>> modern
>>>> basic suite.
>>>>
>>>>
>>>> Alexey, you are much more familiar with the Ignite core codebase than
>>> most
>>>> of us,
>>>> can you please explain in more detail which particular feature, in your
>>>> opinion,
>>>> mandates this "start from scratch" approach?
>>>> Is it really not possible at all to follow a less radical way?
>>>>
>>>>
>>>> [1]
>>>>
>>>>
>>>
>> https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
>>>>
>>>> On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <ni...@apache.org>
>>>> wrote:
>>>>
>>>>> Hello, Alexey.
>>>>>
>>>>> I think that «rewriting from scratch» approach has a high risk to
>> make
>>>> new
>>>>> features unusable.
>>>>> At the time Ignite2 was started no-one wants to do bad UX or bad
>>>> features.
>>>>> Nevertheless, it happen.
>>>>>
>>>>> I think we can avoid it with the Ignite3 and successors if we will
>> move
>>>>> step by step without keeping backward compatibility
>>>>> With the step by step approach, we can focus on each component
>>>> separately.
>>>>>
>>>>>> What new features are we planning to implement for Ignite 2.x?
>>>>>
>>>>> We have many features that have to evolve.
>>>>> Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
>>>>> We have bugs and issues that can be fixed in 2.x without breaking
>>>> backward
>>>>> compatibility.
>>>>> We have many users who are happy with the 2.x with all it’s issues.
>>>>>
>>>>>> 2 нояб. 2020 г., в 14:09, Anton Vinogradov <av...@apache.org>
>>> написал(а):
>>>>>>
>>>>>> Alexey,
>>>>>>
>>>>>> Do we have any estimates of how fast we'll be able to gain
>>>>> production-ready
>>>>>> AI 3.0 in case of a "new repo" choice?
>>>>>>
>>>>>> On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk <
>>>>> alexey.goncharuk@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Nikolay,
>>>>>>>
>>>>>>> What new features are we planning to implement for Ignite 2.x? I
>>> think
>>>>> once
>>>>>>> we commence working on Ignite 3.0, we should gradually cease the
>>>>> activity
>>>>>>> on Ignite 2.x to mere bugfixes because such parallel development
>>> will
>>>> be
>>>>>>> overwhelming regardless of how we choose to proceed.
>>>>>>>
>>>>>>> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <nizhikov@apache.org
>>> :
>>>>>>>
>>>>>>>> To be clear:
>>>>>>>>
>>>>>>>>> I would suggest creating a new repository for Ignite 3.0
>>> (perhaps, a
>>>>>>> new
>>>>>>>> clean branch, but a new repo looks nicer to me) and a new Ignite
>>> 3.0
>>>>>>>> TeamCity project.
>>>>>>>>
>>>>>>>> +1 for new Team City project.
>>>>>>>> +1 for new branch for Ignite3.
>>>>>>>> -1 for new repo.
>>>>>>>>
>>>>>>>>
>>>>>>>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <
>> NIzhikov.dev@gmail.com
>>>>
>>>>>>>> написал(а):
>>>>>>>>>
>>>>>>>>> Hello, Alexey.
>>>>>>>>>
>>>>>>>>> I think it will hurt our project more than help.
>>>>>>>>> Developing new features for 2 separate branches with the
>> different
>>>>> APIs
>>>>>>>> and internal structure is overwhelming
>>>>>>>>>
>>>>>>>>> Maybe we should relax a bit requirements for Ignite3?
>>>>>>>>> Maybe we should move step by step and make Ignite3 with new
>>>>>>>> configuration than Ignite4 with new transactions, etc?
>>>>>>>>>
>>>>>>>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <
>>>>>>> alexey.goncharuk@gmail.com>
>>>>>>>> написал(а):
>>>>>>>>>>
>>>>>>>>>> Igniters,
>>>>>>>>>>
>>>>>>>>>> I wanted to pitch a rather radical idea regarding the Ignite
>> 3.0
>>>>>>>>>> development which has occurred to me some time ago.
>>>>>>>>>>
>>>>>>>>>> We already have several IEPs targeted to Ignite 3.0 which imply
>>>> major
>>>>>>>>>> changes to the codebase (the change in replication protocol and
>>>> thus
>>>>>>>>>> transactions, change in binary format, updated metastorage,
>> etc).
>>>> We
>>>>>>>> also
>>>>>>>>>> planned significant changes in public APIs: configuration
>> format
>>>>>>> change,
>>>>>>>>>> improvements in cache APIs, SQL APIs, transaction mode rework.
>>> The
>>>>>>>> wishlist
>>>>>>>>>> of changes for Ignite 3.0 is huge.
>>>>>>>>>>
>>>>>>>>>> So, I was wondering whether it makes sense to try to change the
>>> old
>>>>>>>>>> codebase, or start with a new baseline and move old pieces of
>>> code
>>>>>>> that
>>>>>>>> do
>>>>>>>>>> not require significant rework. Personally, I would go with the
>>>>> second
>>>>>>>>>> option for the following reasons:
>>>>>>>>>>
>>>>>>>>>> - We have a chance to shift the development paradigm in the
>>> project
>>>>>>> and
>>>>>>>>>> introduce the practice of true unit-tests. In the new baseline
>> at
>>>> the
>>>>>>>>>> beginning there will be no ability to run an end-to-end
>> scenario,
>>>>>>> thus
>>>>>>>> we
>>>>>>>>>> will be forced to write unit-tests. So far, such practice was
>>> hard
>>>> to
>>>>>>>>>> implement because of tight coupling between Ignite components
>> and
>>>>>>>> inability
>>>>>>>>>> to instantiate components without an instance of KernalContext.
>>> For
>>>>>>>>>> example, we should be able to thoroughly test internal
>>> primitives,
>>>>>>>> such as
>>>>>>>>>> replication protocol (without actual communication),
>> distributed
>>>>>>>>>> metastorage contracts, persistence layer, etc.
>>>>>>>>>> - We will significantly reduce the development cycle in the
>>>> beginning
>>>>>>>>>> (right now the RunAll takes two hours of astronomical time with
>>>> empty
>>>>>>>> TC;
>>>>>>>>>> in the new approach developer will be able to run ALL tests
>>> locally
>>>>>>> in
>>>>>>>> a
>>>>>>>>>> matter of minutes)
>>>>>>>>>> - We can get rid of TC bot and enforce green TC by integrating
>> TC
>>>>>>> build
>>>>>>>>>> results with GitHub PRs (the same way Travis is currently
>>>> integrated
>>>>>>>> to PR
>>>>>>>>>> check). We should restrict PR merge without a TC check
>>>>>>>>>> - We will still have to re-write all tests, but only once. If
>> we
>>>> try
>>>>>>> to
>>>>>>>>>> modify the old codebase, we would need to modify all the tests
>>> for
>>>>>>>> every
>>>>>>>>>> major change (public API change, configuration change)
>>>>>>>>>> - We will have fewer conflicts when working together. For
>>> example,
>>>> I
>>>>>>>>>> cannot imagine how one would merge two changes of getting rid
>> of
>>>>>>>>>> IgniteFuture and changes in replication protocol, for example
>>>>>>>>>>
>>>>>>>>>> Technically, I would suggest creating a new repository for
>> Ignite
>>>> 3.0
>>>>>>>>>> (perhaps, a new clean branch, but a new repo looks nicer to me)
>>>> and a
>>>>>>>> new
>>>>>>>>>> Ignite 3.0 TeamCity project.
>>>>>>>>>>
>>>>>>>>>> While it may seem quite radical, I do believe that this
>> approach
>>>> will
>>>>>>>> give
>>>>>>>>>> us more benefits than trying to make such major changes in the
>>>>>>> existing
>>>>>>>>>> codebase. If needed, let's schedule a discord chat like before
>> to
>>>>>>>> discuss
>>>>>>>>>> this.
>>>>>>>>>>
>>>>>>>>>> WDYT?
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>>
>> --
>> Best regards,
>> Andrey V. Mashenkov
>>
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Pavel Tupitsyn <pt...@apache.org>.
> not intend to rewrite everything from scratch
> Every single test from Ignite 2.x should be moved to Ignite 3
> regardless of how we choose to proceed.
Alexey, thank you for the explanation, this addresses all of my concerns.
On Mon, Nov 2, 2020 at 6:43 PM Andrey Mashenkov <an...@gmail.com>
wrote:
> Hi, Igniters.
>
> * AFAIU, we need a new repo if we want to apply different restrictions to
> pull requests,
> otherwise I see no difference for myself.
> E.g. make static analysis (do we have?), compile, styles, and javadoc
> checks mandatory.
>
> I think that relaxed requirements here will lead to bad product quality.
>
> * Agree with Pavel, we should 'keep' integrations tests somehow.
> During active development tests will be broken most of time, so,
> I'd port them e.g. suite-by-suite once we will have a stable and featured
> environment to run them and of course make test's code clear and avoid
> bad/non-relevant ones.
>
> * I like bottom-up approach.
> With it we could make a better framework. I mean clear component lifecycle,
> component wiring mechanics, general methods to approach core components
> such as exchange/communication
> to avoid code mess like we have in ExchangeFuture with all these custom
> callbacks for each component, interfaces like
> PartitionsExchangeAware, IgniteChangeGlobalStateSupport and
> a pack of start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected
> and so on in various unexpected places.
> Hope, we will be able to port most of the good code to the new framework
> version.
>
>
>
> On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk <
> alexey.goncharuk@gmail.com>
> wrote:
>
> > Nikolay, Pavel,
> >
> > Thanks for the feedback! First of all, I wanted to stress that I do not
> > intend to rewrite everything from scratch (I never used this phrase).
> There
> > are significant parts of code that would be moved with minimal
> > modifications. Second, I never said that we will get rid of the old tests
> > codebase. Every single test from Ignite 2.x should be moved to Ignite 3
> > regardless of how we choose to proceed.
> >
> > My point is that for some parts of the code a clean bottom-up
> > implementation will be cheaper in many ways. Let me give you a few
> concrete
> > examples:
> >
> > - I think no one can object that we need a cleanly separated
> persistence
> > layer for Ignite. There is a very raw draft IEP for this already. On
> the
> > other hand, I think we also can agree that we need a split-brain
> > resistant
> > replication protocol for caches. There is also an IEP for this.
> Neither
> > of
> > the changes is a good fit for 2.x because they are likely to introduce
> > breaking changes in the persistence layer, configuration and behavior.
> > Additionally, these components are now tightly coupled, so there is no
> > way
> > these two changes can be implemented in parallel and then merged
> > together
> > easily. So what we will end up with is having to implement these
> changes
> > sequentially, fixing all existing tests twice, and essentially
> throwing
> > away half of the work done because the other part of the change is
> > re-implemented
> > - Similar example goes with getting rid of IgniteInternalFuture and
> > replacing it with CompletableFuture, and any other change that touches
> > the
> > asynchronous part of the code.
> >
> > Third, I do not see how this choice affects the UX of Ignite. The end
> user
> > experience must be fixed in the IEP regardless of the development process
> > and the fact that we have gaps in this area in Ignite 2.x just confirms
> > that.
> >
> > Pavel, agree that a repo/branch is a technicality, I guess if
> reformulate,
> > my point is that we might agree to have a single development master
> branch
> > with 'disassembled' end-to-end functionality for some period of time to
> > speed up development, and re-assemble the core features after having
> > submodules tested independently.
> >
> > Nikolay,
> > >We have many features that have to evolve.
> > >Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
> > This is not very specific. In the end, resources are limited and we will
> > not be able to drive both tracks simultaneously, especially after a
> couple
> > of features having been implemented for Ignite 3.0. If there are indeed
> > some major changes that we want to do in Ignite 2.x instead of putting
> > effort into 3.0 - let's discuss them. I am just not aware of any, that's
> > why I am eager to move forward with Ignite 3.0.
> >
> > >We have bugs and issues that can be fixed in 2.x without breaking
> backward
> > compatibility.
> > >We have many users who are happy with the 2.x with all it’s issues.
> > These changes will be covered by end-to-end tests and migrated to Ignite
> > 3.0, so I see no issues here.
> >
> > Finally, Anton & Nikolay
> > I do not have an estimate for this simply because the activity is
> > community-driven and it depends on the number of people willing to
> > contribute. With the current pace, I would hope to have an RC of Ignite
> 3.0
> > to be ready by the end of 2021. My gut feeling is that by moving with
> > incremental changes, we will not be able to implement even half of the
> > wishlist by that time.
> > I doubt that releasing several major releases with breaking changes will
> > make Ignite users happy either because each upgrade will cost Ignite
> users
> > money, so the fewer major versions we release, the better. Thus my wish
> to
> > include all breaking changes in one release.
> >
> > I'll be now quiet for a while, let's see what other community members
> > think.
> >
> > пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <pt...@apache.org>:
> >
> > > 1. Rewriting from scratch is never a good idea.
> > > We don't want to follow the path of Netscape and lose all our users
> > > by the time we have a working 3.0 [1]
> > >
> > > 2. Not sure about new repo - seems like some pain and no gain, what's
> the
> > > problem with a branch?
> > >
> > > 3. We should keep existing integration tests when possible.
> > > We have accumulated a lot of edge case knowledge over the years,
> > > it is not a good idea to send all of that down the drain.
> > > Yes, integration tests are slow, but they are the most valuable.
> > > I think we can move more stuff into nightly runs and have a fast and
> > modern
> > > basic suite.
> > >
> > >
> > > Alexey, you are much more familiar with the Ignite core codebase than
> > most
> > > of us,
> > > can you please explain in more detail which particular feature, in your
> > > opinion,
> > > mandates this "start from scratch" approach?
> > > Is it really not possible at all to follow a less radical way?
> > >
> > >
> > > [1]
> > >
> > >
> >
> https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
> > >
> > > On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <ni...@apache.org>
> > > wrote:
> > >
> > > > Hello, Alexey.
> > > >
> > > > I think that «rewriting from scratch» approach has a high risk to
> make
> > > new
> > > > features unusable.
> > > > At the time Ignite2 was started no-one wants to do bad UX or bad
> > > features.
> > > > Nevertheless, it happen.
> > > >
> > > > I think we can avoid it with the Ignite3 and successors if we will
> move
> > > > step by step without keeping backward compatibility
> > > > With the step by step approach, we can focus on each component
> > > separately.
> > > >
> > > > > What new features are we planning to implement for Ignite 2.x?
> > > >
> > > > We have many features that have to evolve.
> > > > Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
> > > > We have bugs and issues that can be fixed in 2.x without breaking
> > > backward
> > > > compatibility.
> > > > We have many users who are happy with the 2.x with all it’s issues.
> > > >
> > > > > 2 нояб. 2020 г., в 14:09, Anton Vinogradov <av...@apache.org>
> > написал(а):
> > > > >
> > > > > Alexey,
> > > > >
> > > > > Do we have any estimates of how fast we'll be able to gain
> > > > production-ready
> > > > > AI 3.0 in case of a "new repo" choice?
> > > > >
> > > > > On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk <
> > > > alexey.goncharuk@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> Nikolay,
> > > > >>
> > > > >> What new features are we planning to implement for Ignite 2.x? I
> > think
> > > > once
> > > > >> we commence working on Ignite 3.0, we should gradually cease the
> > > > activity
> > > > >> on Ignite 2.x to mere bugfixes because such parallel development
> > will
> > > be
> > > > >> overwhelming regardless of how we choose to proceed.
> > > > >>
> > > > >> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <nizhikov@apache.org
> >:
> > > > >>
> > > > >>> To be clear:
> > > > >>>
> > > > >>>> I would suggest creating a new repository for Ignite 3.0
> > (perhaps, a
> > > > >> new
> > > > >>> clean branch, but a new repo looks nicer to me) and a new Ignite
> > 3.0
> > > > >>> TeamCity project.
> > > > >>>
> > > > >>> +1 for new Team City project.
> > > > >>> +1 for new branch for Ignite3.
> > > > >>> -1 for new repo.
> > > > >>>
> > > > >>>
> > > > >>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <
> NIzhikov.dev@gmail.com
> > >
> > > > >>> написал(а):
> > > > >>>>
> > > > >>>> Hello, Alexey.
> > > > >>>>
> > > > >>>> I think it will hurt our project more than help.
> > > > >>>> Developing new features for 2 separate branches with the
> different
> > > > APIs
> > > > >>> and internal structure is overwhelming
> > > > >>>>
> > > > >>>> Maybe we should relax a bit requirements for Ignite3?
> > > > >>>> Maybe we should move step by step and make Ignite3 with new
> > > > >>> configuration than Ignite4 with new transactions, etc?
> > > > >>>>
> > > > >>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <
> > > > >> alexey.goncharuk@gmail.com>
> > > > >>> написал(а):
> > > > >>>>>
> > > > >>>>> Igniters,
> > > > >>>>>
> > > > >>>>> I wanted to pitch a rather radical idea regarding the Ignite
> 3.0
> > > > >>>>> development which has occurred to me some time ago.
> > > > >>>>>
> > > > >>>>> We already have several IEPs targeted to Ignite 3.0 which imply
> > > major
> > > > >>>>> changes to the codebase (the change in replication protocol and
> > > thus
> > > > >>>>> transactions, change in binary format, updated metastorage,
> etc).
> > > We
> > > > >>> also
> > > > >>>>> planned significant changes in public APIs: configuration
> format
> > > > >> change,
> > > > >>>>> improvements in cache APIs, SQL APIs, transaction mode rework.
> > The
> > > > >>> wishlist
> > > > >>>>> of changes for Ignite 3.0 is huge.
> > > > >>>>>
> > > > >>>>> So, I was wondering whether it makes sense to try to change the
> > old
> > > > >>>>> codebase, or start with a new baseline and move old pieces of
> > code
> > > > >> that
> > > > >>> do
> > > > >>>>> not require significant rework. Personally, I would go with the
> > > > second
> > > > >>>>> option for the following reasons:
> > > > >>>>>
> > > > >>>>> - We have a chance to shift the development paradigm in the
> > project
> > > > >> and
> > > > >>>>> introduce the practice of true unit-tests. In the new baseline
> at
> > > the
> > > > >>>>> beginning there will be no ability to run an end-to-end
> scenario,
> > > > >> thus
> > > > >>> we
> > > > >>>>> will be forced to write unit-tests. So far, such practice was
> > hard
> > > to
> > > > >>>>> implement because of tight coupling between Ignite components
> and
> > > > >>> inability
> > > > >>>>> to instantiate components without an instance of KernalContext.
> > For
> > > > >>>>> example, we should be able to thoroughly test internal
> > primitives,
> > > > >>> such as
> > > > >>>>> replication protocol (without actual communication),
> distributed
> > > > >>>>> metastorage contracts, persistence layer, etc.
> > > > >>>>> - We will significantly reduce the development cycle in the
> > > beginning
> > > > >>>>> (right now the RunAll takes two hours of astronomical time with
> > > empty
> > > > >>> TC;
> > > > >>>>> in the new approach developer will be able to run ALL tests
> > locally
> > > > >> in
> > > > >>> a
> > > > >>>>> matter of minutes)
> > > > >>>>> - We can get rid of TC bot and enforce green TC by integrating
> TC
> > > > >> build
> > > > >>>>> results with GitHub PRs (the same way Travis is currently
> > > integrated
> > > > >>> to PR
> > > > >>>>> check). We should restrict PR merge without a TC check
> > > > >>>>> - We will still have to re-write all tests, but only once. If
> we
> > > try
> > > > >> to
> > > > >>>>> modify the old codebase, we would need to modify all the tests
> > for
> > > > >>> every
> > > > >>>>> major change (public API change, configuration change)
> > > > >>>>> - We will have fewer conflicts when working together. For
> > example,
> > > I
> > > > >>>>> cannot imagine how one would merge two changes of getting rid
> of
> > > > >>>>> IgniteFuture and changes in replication protocol, for example
> > > > >>>>>
> > > > >>>>> Technically, I would suggest creating a new repository for
> Ignite
> > > 3.0
> > > > >>>>> (perhaps, a new clean branch, but a new repo looks nicer to me)
> > > and a
> > > > >>> new
> > > > >>>>> Ignite 3.0 TeamCity project.
> > > > >>>>>
> > > > >>>>> While it may seem quite radical, I do believe that this
> approach
> > > will
> > > > >>> give
> > > > >>>>> us more benefits than trying to make such major changes in the
> > > > >> existing
> > > > >>>>> codebase. If needed, let's schedule a discord chat like before
> to
> > > > >>> discuss
> > > > >>>>> this.
> > > > >>>>>
> > > > >>>>> WDYT?
> > > > >>>>
> > > > >>>
> > > > >>>
> > > > >>
> > > >
> > > >
> > >
> >
>
>
> --
> Best regards,
> Andrey V. Mashenkov
>
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Andrey Mashenkov <an...@gmail.com>.
Hi, Igniters.
* AFAIU, we need a new repo if we want to apply different restrictions to
pull requests,
otherwise I see no difference for myself.
E.g. make static analysis (do we have?), compile, styles, and javadoc
checks mandatory.
I think that relaxed requirements here will lead to bad product quality.
* Agree with Pavel, we should 'keep' integrations tests somehow.
During active development tests will be broken most of time, so,
I'd port them e.g. suite-by-suite once we will have a stable and featured
environment to run them and of course make test's code clear and avoid
bad/non-relevant ones.
* I like bottom-up approach.
With it we could make a better framework. I mean clear component lifecycle,
component wiring mechanics, general methods to approach core components
such as exchange/communication
to avoid code mess like we have in ExchangeFuture with all these custom
callbacks for each component, interfaces like
PartitionsExchangeAware, IgniteChangeGlobalStateSupport and
a pack of start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected
and so on in various unexpected places.
Hope, we will be able to port most of the good code to the new framework
version.
On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk <al...@gmail.com>
wrote:
> Nikolay, Pavel,
>
> Thanks for the feedback! First of all, I wanted to stress that I do not
> intend to rewrite everything from scratch (I never used this phrase). There
> are significant parts of code that would be moved with minimal
> modifications. Second, I never said that we will get rid of the old tests
> codebase. Every single test from Ignite 2.x should be moved to Ignite 3
> regardless of how we choose to proceed.
>
> My point is that for some parts of the code a clean bottom-up
> implementation will be cheaper in many ways. Let me give you a few concrete
> examples:
>
> - I think no one can object that we need a cleanly separated persistence
> layer for Ignite. There is a very raw draft IEP for this already. On the
> other hand, I think we also can agree that we need a split-brain
> resistant
> replication protocol for caches. There is also an IEP for this. Neither
> of
> the changes is a good fit for 2.x because they are likely to introduce
> breaking changes in the persistence layer, configuration and behavior.
> Additionally, these components are now tightly coupled, so there is no
> way
> these two changes can be implemented in parallel and then merged
> together
> easily. So what we will end up with is having to implement these changes
> sequentially, fixing all existing tests twice, and essentially throwing
> away half of the work done because the other part of the change is
> re-implemented
> - Similar example goes with getting rid of IgniteInternalFuture and
> replacing it with CompletableFuture, and any other change that touches
> the
> asynchronous part of the code.
>
> Third, I do not see how this choice affects the UX of Ignite. The end user
> experience must be fixed in the IEP regardless of the development process
> and the fact that we have gaps in this area in Ignite 2.x just confirms
> that.
>
> Pavel, agree that a repo/branch is a technicality, I guess if reformulate,
> my point is that we might agree to have a single development master branch
> with 'disassembled' end-to-end functionality for some period of time to
> speed up development, and re-assemble the core features after having
> submodules tested independently.
>
> Nikolay,
> >We have many features that have to evolve.
> >Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
> This is not very specific. In the end, resources are limited and we will
> not be able to drive both tracks simultaneously, especially after a couple
> of features having been implemented for Ignite 3.0. If there are indeed
> some major changes that we want to do in Ignite 2.x instead of putting
> effort into 3.0 - let's discuss them. I am just not aware of any, that's
> why I am eager to move forward with Ignite 3.0.
>
> >We have bugs and issues that can be fixed in 2.x without breaking backward
> compatibility.
> >We have many users who are happy with the 2.x with all it’s issues.
> These changes will be covered by end-to-end tests and migrated to Ignite
> 3.0, so I see no issues here.
>
> Finally, Anton & Nikolay
> I do not have an estimate for this simply because the activity is
> community-driven and it depends on the number of people willing to
> contribute. With the current pace, I would hope to have an RC of Ignite 3.0
> to be ready by the end of 2021. My gut feeling is that by moving with
> incremental changes, we will not be able to implement even half of the
> wishlist by that time.
> I doubt that releasing several major releases with breaking changes will
> make Ignite users happy either because each upgrade will cost Ignite users
> money, so the fewer major versions we release, the better. Thus my wish to
> include all breaking changes in one release.
>
> I'll be now quiet for a while, let's see what other community members
> think.
>
> пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <pt...@apache.org>:
>
> > 1. Rewriting from scratch is never a good idea.
> > We don't want to follow the path of Netscape and lose all our users
> > by the time we have a working 3.0 [1]
> >
> > 2. Not sure about new repo - seems like some pain and no gain, what's the
> > problem with a branch?
> >
> > 3. We should keep existing integration tests when possible.
> > We have accumulated a lot of edge case knowledge over the years,
> > it is not a good idea to send all of that down the drain.
> > Yes, integration tests are slow, but they are the most valuable.
> > I think we can move more stuff into nightly runs and have a fast and
> modern
> > basic suite.
> >
> >
> > Alexey, you are much more familiar with the Ignite core codebase than
> most
> > of us,
> > can you please explain in more detail which particular feature, in your
> > opinion,
> > mandates this "start from scratch" approach?
> > Is it really not possible at all to follow a less radical way?
> >
> >
> > [1]
> >
> >
> https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
> >
> > On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <ni...@apache.org>
> > wrote:
> >
> > > Hello, Alexey.
> > >
> > > I think that «rewriting from scratch» approach has a high risk to make
> > new
> > > features unusable.
> > > At the time Ignite2 was started no-one wants to do bad UX or bad
> > features.
> > > Nevertheless, it happen.
> > >
> > > I think we can avoid it with the Ignite3 and successors if we will move
> > > step by step without keeping backward compatibility
> > > With the step by step approach, we can focus on each component
> > separately.
> > >
> > > > What new features are we planning to implement for Ignite 2.x?
> > >
> > > We have many features that have to evolve.
> > > Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
> > > We have bugs and issues that can be fixed in 2.x without breaking
> > backward
> > > compatibility.
> > > We have many users who are happy with the 2.x with all it’s issues.
> > >
> > > > 2 нояб. 2020 г., в 14:09, Anton Vinogradov <av...@apache.org>
> написал(а):
> > > >
> > > > Alexey,
> > > >
> > > > Do we have any estimates of how fast we'll be able to gain
> > > production-ready
> > > > AI 3.0 in case of a "new repo" choice?
> > > >
> > > > On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk <
> > > alexey.goncharuk@gmail.com>
> > > > wrote:
> > > >
> > > >> Nikolay,
> > > >>
> > > >> What new features are we planning to implement for Ignite 2.x? I
> think
> > > once
> > > >> we commence working on Ignite 3.0, we should gradually cease the
> > > activity
> > > >> on Ignite 2.x to mere bugfixes because such parallel development
> will
> > be
> > > >> overwhelming regardless of how we choose to proceed.
> > > >>
> > > >> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <ni...@apache.org>:
> > > >>
> > > >>> To be clear:
> > > >>>
> > > >>>> I would suggest creating a new repository for Ignite 3.0
> (perhaps, a
> > > >> new
> > > >>> clean branch, but a new repo looks nicer to me) and a new Ignite
> 3.0
> > > >>> TeamCity project.
> > > >>>
> > > >>> +1 for new Team City project.
> > > >>> +1 for new branch for Ignite3.
> > > >>> -1 for new repo.
> > > >>>
> > > >>>
> > > >>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <NIzhikov.dev@gmail.com
> >
> > > >>> написал(а):
> > > >>>>
> > > >>>> Hello, Alexey.
> > > >>>>
> > > >>>> I think it will hurt our project more than help.
> > > >>>> Developing new features for 2 separate branches with the different
> > > APIs
> > > >>> and internal structure is overwhelming
> > > >>>>
> > > >>>> Maybe we should relax a bit requirements for Ignite3?
> > > >>>> Maybe we should move step by step and make Ignite3 with new
> > > >>> configuration than Ignite4 with new transactions, etc?
> > > >>>>
> > > >>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <
> > > >> alexey.goncharuk@gmail.com>
> > > >>> написал(а):
> > > >>>>>
> > > >>>>> Igniters,
> > > >>>>>
> > > >>>>> I wanted to pitch a rather radical idea regarding the Ignite 3.0
> > > >>>>> development which has occurred to me some time ago.
> > > >>>>>
> > > >>>>> We already have several IEPs targeted to Ignite 3.0 which imply
> > major
> > > >>>>> changes to the codebase (the change in replication protocol and
> > thus
> > > >>>>> transactions, change in binary format, updated metastorage, etc).
> > We
> > > >>> also
> > > >>>>> planned significant changes in public APIs: configuration format
> > > >> change,
> > > >>>>> improvements in cache APIs, SQL APIs, transaction mode rework.
> The
> > > >>> wishlist
> > > >>>>> of changes for Ignite 3.0 is huge.
> > > >>>>>
> > > >>>>> So, I was wondering whether it makes sense to try to change the
> old
> > > >>>>> codebase, or start with a new baseline and move old pieces of
> code
> > > >> that
> > > >>> do
> > > >>>>> not require significant rework. Personally, I would go with the
> > > second
> > > >>>>> option for the following reasons:
> > > >>>>>
> > > >>>>> - We have a chance to shift the development paradigm in the
> project
> > > >> and
> > > >>>>> introduce the practice of true unit-tests. In the new baseline at
> > the
> > > >>>>> beginning there will be no ability to run an end-to-end scenario,
> > > >> thus
> > > >>> we
> > > >>>>> will be forced to write unit-tests. So far, such practice was
> hard
> > to
> > > >>>>> implement because of tight coupling between Ignite components and
> > > >>> inability
> > > >>>>> to instantiate components without an instance of KernalContext.
> For
> > > >>>>> example, we should be able to thoroughly test internal
> primitives,
> > > >>> such as
> > > >>>>> replication protocol (without actual communication), distributed
> > > >>>>> metastorage contracts, persistence layer, etc.
> > > >>>>> - We will significantly reduce the development cycle in the
> > beginning
> > > >>>>> (right now the RunAll takes two hours of astronomical time with
> > empty
> > > >>> TC;
> > > >>>>> in the new approach developer will be able to run ALL tests
> locally
> > > >> in
> > > >>> a
> > > >>>>> matter of minutes)
> > > >>>>> - We can get rid of TC bot and enforce green TC by integrating TC
> > > >> build
> > > >>>>> results with GitHub PRs (the same way Travis is currently
> > integrated
> > > >>> to PR
> > > >>>>> check). We should restrict PR merge without a TC check
> > > >>>>> - We will still have to re-write all tests, but only once. If we
> > try
> > > >> to
> > > >>>>> modify the old codebase, we would need to modify all the tests
> for
> > > >>> every
> > > >>>>> major change (public API change, configuration change)
> > > >>>>> - We will have fewer conflicts when working together. For
> example,
> > I
> > > >>>>> cannot imagine how one would merge two changes of getting rid of
> > > >>>>> IgniteFuture and changes in replication protocol, for example
> > > >>>>>
> > > >>>>> Technically, I would suggest creating a new repository for Ignite
> > 3.0
> > > >>>>> (perhaps, a new clean branch, but a new repo looks nicer to me)
> > and a
> > > >>> new
> > > >>>>> Ignite 3.0 TeamCity project.
> > > >>>>>
> > > >>>>> While it may seem quite radical, I do believe that this approach
> > will
> > > >>> give
> > > >>>>> us more benefits than trying to make such major changes in the
> > > >> existing
> > > >>>>> codebase. If needed, let's schedule a discord chat like before to
> > > >>> discuss
> > > >>>>> this.
> > > >>>>>
> > > >>>>> WDYT?
> > > >>>>
> > > >>>
> > > >>>
> > > >>
> > >
> > >
> >
>
--
Best regards,
Andrey V. Mashenkov
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Alexey Goncharuk <al...@gmail.com>.
Nikolay, Pavel,
Thanks for the feedback! First of all, I wanted to stress that I do not
intend to rewrite everything from scratch (I never used this phrase). There
are significant parts of code that would be moved with minimal
modifications. Second, I never said that we will get rid of the old tests
codebase. Every single test from Ignite 2.x should be moved to Ignite 3
regardless of how we choose to proceed.
My point is that for some parts of the code a clean bottom-up
implementation will be cheaper in many ways. Let me give you a few concrete
examples:
- I think no one can object that we need a cleanly separated persistence
layer for Ignite. There is a very raw draft IEP for this already. On the
other hand, I think we also can agree that we need a split-brain resistant
replication protocol for caches. There is also an IEP for this. Neither of
the changes is a good fit for 2.x because they are likely to introduce
breaking changes in the persistence layer, configuration and behavior.
Additionally, these components are now tightly coupled, so there is no way
these two changes can be implemented in parallel and then merged together
easily. So what we will end up with is having to implement these changes
sequentially, fixing all existing tests twice, and essentially throwing
away half of the work done because the other part of the change is
re-implemented
- Similar example goes with getting rid of IgniteInternalFuture and
replacing it with CompletableFuture, and any other change that touches the
asynchronous part of the code.
Third, I do not see how this choice affects the UX of Ignite. The end user
experience must be fixed in the IEP regardless of the development process
and the fact that we have gaps in this area in Ignite 2.x just confirms
that.
Pavel, agree that a repo/branch is a technicality, I guess if reformulate,
my point is that we might agree to have a single development master branch
with 'disassembled' end-to-end functionality for some period of time to
speed up development, and re-assemble the core features after having
submodules tested independently.
Nikolay,
>We have many features that have to evolve.
>Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
This is not very specific. In the end, resources are limited and we will
not be able to drive both tracks simultaneously, especially after a couple
of features having been implemented for Ignite 3.0. If there are indeed
some major changes that we want to do in Ignite 2.x instead of putting
effort into 3.0 - let's discuss them. I am just not aware of any, that's
why I am eager to move forward with Ignite 3.0.
>We have bugs and issues that can be fixed in 2.x without breaking backward
compatibility.
>We have many users who are happy with the 2.x with all it’s issues.
These changes will be covered by end-to-end tests and migrated to Ignite
3.0, so I see no issues here.
Finally, Anton & Nikolay
I do not have an estimate for this simply because the activity is
community-driven and it depends on the number of people willing to
contribute. With the current pace, I would hope to have an RC of Ignite 3.0
to be ready by the end of 2021. My gut feeling is that by moving with
incremental changes, we will not be able to implement even half of the
wishlist by that time.
I doubt that releasing several major releases with breaking changes will
make Ignite users happy either because each upgrade will cost Ignite users
money, so the fewer major versions we release, the better. Thus my wish to
include all breaking changes in one release.
I'll be now quiet for a while, let's see what other community members think.
пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <pt...@apache.org>:
> 1. Rewriting from scratch is never a good idea.
> We don't want to follow the path of Netscape and lose all our users
> by the time we have a working 3.0 [1]
>
> 2. Not sure about new repo - seems like some pain and no gain, what's the
> problem with a branch?
>
> 3. We should keep existing integration tests when possible.
> We have accumulated a lot of edge case knowledge over the years,
> it is not a good idea to send all of that down the drain.
> Yes, integration tests are slow, but they are the most valuable.
> I think we can move more stuff into nightly runs and have a fast and modern
> basic suite.
>
>
> Alexey, you are much more familiar with the Ignite core codebase than most
> of us,
> can you please explain in more detail which particular feature, in your
> opinion,
> mandates this "start from scratch" approach?
> Is it really not possible at all to follow a less radical way?
>
>
> [1]
>
> https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
>
> On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <ni...@apache.org>
> wrote:
>
> > Hello, Alexey.
> >
> > I think that «rewriting from scratch» approach has a high risk to make
> new
> > features unusable.
> > At the time Ignite2 was started no-one wants to do bad UX or bad
> features.
> > Nevertheless, it happen.
> >
> > I think we can avoid it with the Ignite3 and successors if we will move
> > step by step without keeping backward compatibility
> > With the step by step approach, we can focus on each component
> separately.
> >
> > > What new features are we planning to implement for Ignite 2.x?
> >
> > We have many features that have to evolve.
> > Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
> > We have bugs and issues that can be fixed in 2.x without breaking
> backward
> > compatibility.
> > We have many users who are happy with the 2.x with all it’s issues.
> >
> > > 2 нояб. 2020 г., в 14:09, Anton Vinogradov <av...@apache.org> написал(а):
> > >
> > > Alexey,
> > >
> > > Do we have any estimates of how fast we'll be able to gain
> > production-ready
> > > AI 3.0 in case of a "new repo" choice?
> > >
> > > On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk <
> > alexey.goncharuk@gmail.com>
> > > wrote:
> > >
> > >> Nikolay,
> > >>
> > >> What new features are we planning to implement for Ignite 2.x? I think
> > once
> > >> we commence working on Ignite 3.0, we should gradually cease the
> > activity
> > >> on Ignite 2.x to mere bugfixes because such parallel development will
> be
> > >> overwhelming regardless of how we choose to proceed.
> > >>
> > >> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <ni...@apache.org>:
> > >>
> > >>> To be clear:
> > >>>
> > >>>> I would suggest creating a new repository for Ignite 3.0 (perhaps, a
> > >> new
> > >>> clean branch, but a new repo looks nicer to me) and a new Ignite 3.0
> > >>> TeamCity project.
> > >>>
> > >>> +1 for new Team City project.
> > >>> +1 for new branch for Ignite3.
> > >>> -1 for new repo.
> > >>>
> > >>>
> > >>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <NI...@gmail.com>
> > >>> написал(а):
> > >>>>
> > >>>> Hello, Alexey.
> > >>>>
> > >>>> I think it will hurt our project more than help.
> > >>>> Developing new features for 2 separate branches with the different
> > APIs
> > >>> and internal structure is overwhelming
> > >>>>
> > >>>> Maybe we should relax a bit requirements for Ignite3?
> > >>>> Maybe we should move step by step and make Ignite3 with new
> > >>> configuration than Ignite4 with new transactions, etc?
> > >>>>
> > >>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <
> > >> alexey.goncharuk@gmail.com>
> > >>> написал(а):
> > >>>>>
> > >>>>> Igniters,
> > >>>>>
> > >>>>> I wanted to pitch a rather radical idea regarding the Ignite 3.0
> > >>>>> development which has occurred to me some time ago.
> > >>>>>
> > >>>>> We already have several IEPs targeted to Ignite 3.0 which imply
> major
> > >>>>> changes to the codebase (the change in replication protocol and
> thus
> > >>>>> transactions, change in binary format, updated metastorage, etc).
> We
> > >>> also
> > >>>>> planned significant changes in public APIs: configuration format
> > >> change,
> > >>>>> improvements in cache APIs, SQL APIs, transaction mode rework. The
> > >>> wishlist
> > >>>>> of changes for Ignite 3.0 is huge.
> > >>>>>
> > >>>>> So, I was wondering whether it makes sense to try to change the old
> > >>>>> codebase, or start with a new baseline and move old pieces of code
> > >> that
> > >>> do
> > >>>>> not require significant rework. Personally, I would go with the
> > second
> > >>>>> option for the following reasons:
> > >>>>>
> > >>>>> - We have a chance to shift the development paradigm in the project
> > >> and
> > >>>>> introduce the practice of true unit-tests. In the new baseline at
> the
> > >>>>> beginning there will be no ability to run an end-to-end scenario,
> > >> thus
> > >>> we
> > >>>>> will be forced to write unit-tests. So far, such practice was hard
> to
> > >>>>> implement because of tight coupling between Ignite components and
> > >>> inability
> > >>>>> to instantiate components without an instance of KernalContext. For
> > >>>>> example, we should be able to thoroughly test internal primitives,
> > >>> such as
> > >>>>> replication protocol (without actual communication), distributed
> > >>>>> metastorage contracts, persistence layer, etc.
> > >>>>> - We will significantly reduce the development cycle in the
> beginning
> > >>>>> (right now the RunAll takes two hours of astronomical time with
> empty
> > >>> TC;
> > >>>>> in the new approach developer will be able to run ALL tests locally
> > >> in
> > >>> a
> > >>>>> matter of minutes)
> > >>>>> - We can get rid of TC bot and enforce green TC by integrating TC
> > >> build
> > >>>>> results with GitHub PRs (the same way Travis is currently
> integrated
> > >>> to PR
> > >>>>> check). We should restrict PR merge without a TC check
> > >>>>> - We will still have to re-write all tests, but only once. If we
> try
> > >> to
> > >>>>> modify the old codebase, we would need to modify all the tests for
> > >>> every
> > >>>>> major change (public API change, configuration change)
> > >>>>> - We will have fewer conflicts when working together. For example,
> I
> > >>>>> cannot imagine how one would merge two changes of getting rid of
> > >>>>> IgniteFuture and changes in replication protocol, for example
> > >>>>>
> > >>>>> Technically, I would suggest creating a new repository for Ignite
> 3.0
> > >>>>> (perhaps, a new clean branch, but a new repo looks nicer to me)
> and a
> > >>> new
> > >>>>> Ignite 3.0 TeamCity project.
> > >>>>>
> > >>>>> While it may seem quite radical, I do believe that this approach
> will
> > >>> give
> > >>>>> us more benefits than trying to make such major changes in the
> > >> existing
> > >>>>> codebase. If needed, let's schedule a discord chat like before to
> > >>> discuss
> > >>>>> this.
> > >>>>>
> > >>>>> WDYT?
> > >>>>
> > >>>
> > >>>
> > >>
> >
> >
>
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Pavel Tupitsyn <pt...@apache.org>.
1. Rewriting from scratch is never a good idea.
We don't want to follow the path of Netscape and lose all our users
by the time we have a working 3.0 [1]
2. Not sure about new repo - seems like some pain and no gain, what's the
problem with a branch?
3. We should keep existing integration tests when possible.
We have accumulated a lot of edge case knowledge over the years,
it is not a good idea to send all of that down the drain.
Yes, integration tests are slow, but they are the most valuable.
I think we can move more stuff into nightly runs and have a fast and modern
basic suite.
Alexey, you are much more familiar with the Ignite core codebase than most
of us,
can you please explain in more detail which particular feature, in your
opinion,
mandates this "start from scratch" approach?
Is it really not possible at all to follow a less radical way?
[1]
https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <ni...@apache.org> wrote:
> Hello, Alexey.
>
> I think that «rewriting from scratch» approach has a high risk to make new
> features unusable.
> At the time Ignite2 was started no-one wants to do bad UX or bad features.
> Nevertheless, it happen.
>
> I think we can avoid it with the Ignite3 and successors if we will move
> step by step without keeping backward compatibility
> With the step by step approach, we can focus on each component separately.
>
> > What new features are we planning to implement for Ignite 2.x?
>
> We have many features that have to evolve.
> Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
> We have bugs and issues that can be fixed in 2.x without breaking backward
> compatibility.
> We have many users who are happy with the 2.x with all it’s issues.
>
> > 2 нояб. 2020 г., в 14:09, Anton Vinogradov <av...@apache.org> написал(а):
> >
> > Alexey,
> >
> > Do we have any estimates of how fast we'll be able to gain
> production-ready
> > AI 3.0 in case of a "new repo" choice?
> >
> > On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk <
> alexey.goncharuk@gmail.com>
> > wrote:
> >
> >> Nikolay,
> >>
> >> What new features are we planning to implement for Ignite 2.x? I think
> once
> >> we commence working on Ignite 3.0, we should gradually cease the
> activity
> >> on Ignite 2.x to mere bugfixes because such parallel development will be
> >> overwhelming regardless of how we choose to proceed.
> >>
> >> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <ni...@apache.org>:
> >>
> >>> To be clear:
> >>>
> >>>> I would suggest creating a new repository for Ignite 3.0 (perhaps, a
> >> new
> >>> clean branch, but a new repo looks nicer to me) and a new Ignite 3.0
> >>> TeamCity project.
> >>>
> >>> +1 for new Team City project.
> >>> +1 for new branch for Ignite3.
> >>> -1 for new repo.
> >>>
> >>>
> >>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <NI...@gmail.com>
> >>> написал(а):
> >>>>
> >>>> Hello, Alexey.
> >>>>
> >>>> I think it will hurt our project more than help.
> >>>> Developing new features for 2 separate branches with the different
> APIs
> >>> and internal structure is overwhelming
> >>>>
> >>>> Maybe we should relax a bit requirements for Ignite3?
> >>>> Maybe we should move step by step and make Ignite3 with new
> >>> configuration than Ignite4 with new transactions, etc?
> >>>>
> >>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <
> >> alexey.goncharuk@gmail.com>
> >>> написал(а):
> >>>>>
> >>>>> Igniters,
> >>>>>
> >>>>> I wanted to pitch a rather radical idea regarding the Ignite 3.0
> >>>>> development which has occurred to me some time ago.
> >>>>>
> >>>>> We already have several IEPs targeted to Ignite 3.0 which imply major
> >>>>> changes to the codebase (the change in replication protocol and thus
> >>>>> transactions, change in binary format, updated metastorage, etc). We
> >>> also
> >>>>> planned significant changes in public APIs: configuration format
> >> change,
> >>>>> improvements in cache APIs, SQL APIs, transaction mode rework. The
> >>> wishlist
> >>>>> of changes for Ignite 3.0 is huge.
> >>>>>
> >>>>> So, I was wondering whether it makes sense to try to change the old
> >>>>> codebase, or start with a new baseline and move old pieces of code
> >> that
> >>> do
> >>>>> not require significant rework. Personally, I would go with the
> second
> >>>>> option for the following reasons:
> >>>>>
> >>>>> - We have a chance to shift the development paradigm in the project
> >> and
> >>>>> introduce the practice of true unit-tests. In the new baseline at the
> >>>>> beginning there will be no ability to run an end-to-end scenario,
> >> thus
> >>> we
> >>>>> will be forced to write unit-tests. So far, such practice was hard to
> >>>>> implement because of tight coupling between Ignite components and
> >>> inability
> >>>>> to instantiate components without an instance of KernalContext. For
> >>>>> example, we should be able to thoroughly test internal primitives,
> >>> such as
> >>>>> replication protocol (without actual communication), distributed
> >>>>> metastorage contracts, persistence layer, etc.
> >>>>> - We will significantly reduce the development cycle in the beginning
> >>>>> (right now the RunAll takes two hours of astronomical time with empty
> >>> TC;
> >>>>> in the new approach developer will be able to run ALL tests locally
> >> in
> >>> a
> >>>>> matter of minutes)
> >>>>> - We can get rid of TC bot and enforce green TC by integrating TC
> >> build
> >>>>> results with GitHub PRs (the same way Travis is currently integrated
> >>> to PR
> >>>>> check). We should restrict PR merge without a TC check
> >>>>> - We will still have to re-write all tests, but only once. If we try
> >> to
> >>>>> modify the old codebase, we would need to modify all the tests for
> >>> every
> >>>>> major change (public API change, configuration change)
> >>>>> - We will have fewer conflicts when working together. For example, I
> >>>>> cannot imagine how one would merge two changes of getting rid of
> >>>>> IgniteFuture and changes in replication protocol, for example
> >>>>>
> >>>>> Technically, I would suggest creating a new repository for Ignite 3.0
> >>>>> (perhaps, a new clean branch, but a new repo looks nicer to me) and a
> >>> new
> >>>>> Ignite 3.0 TeamCity project.
> >>>>>
> >>>>> While it may seem quite radical, I do believe that this approach will
> >>> give
> >>>>> us more benefits than trying to make such major changes in the
> >> existing
> >>>>> codebase. If needed, let's schedule a discord chat like before to
> >>> discuss
> >>>>> this.
> >>>>>
> >>>>> WDYT?
> >>>>
> >>>
> >>>
> >>
>
>
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Nikolay Izhikov <ni...@apache.org>.
Hello, Alexey.
I think that «rewriting from scratch» approach has a high risk to make new features unusable.
At the time Ignite2 was started no-one wants to do bad UX or bad features. Nevertheless, it happen.
I think we can avoid it with the Ignite3 and successors if we will move step by step without keeping backward compatibility
With the step by step approach, we can focus on each component separately.
> What new features are we planning to implement for Ignite 2.x?
We have many features that have to evolve.
Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
We have bugs and issues that can be fixed in 2.x without breaking backward compatibility.
We have many users who are happy with the 2.x with all it’s issues.
> 2 нояб. 2020 г., в 14:09, Anton Vinogradov <av...@apache.org> написал(а):
>
> Alexey,
>
> Do we have any estimates of how fast we'll be able to gain production-ready
> AI 3.0 in case of a "new repo" choice?
>
> On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk <al...@gmail.com>
> wrote:
>
>> Nikolay,
>>
>> What new features are we planning to implement for Ignite 2.x? I think once
>> we commence working on Ignite 3.0, we should gradually cease the activity
>> on Ignite 2.x to mere bugfixes because such parallel development will be
>> overwhelming regardless of how we choose to proceed.
>>
>> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <ni...@apache.org>:
>>
>>> To be clear:
>>>
>>>> I would suggest creating a new repository for Ignite 3.0 (perhaps, a
>> new
>>> clean branch, but a new repo looks nicer to me) and a new Ignite 3.0
>>> TeamCity project.
>>>
>>> +1 for new Team City project.
>>> +1 for new branch for Ignite3.
>>> -1 for new repo.
>>>
>>>
>>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <NI...@gmail.com>
>>> написал(а):
>>>>
>>>> Hello, Alexey.
>>>>
>>>> I think it will hurt our project more than help.
>>>> Developing new features for 2 separate branches with the different APIs
>>> and internal structure is overwhelming
>>>>
>>>> Maybe we should relax a bit requirements for Ignite3?
>>>> Maybe we should move step by step and make Ignite3 with new
>>> configuration than Ignite4 with new transactions, etc?
>>>>
>>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <
>> alexey.goncharuk@gmail.com>
>>> написал(а):
>>>>>
>>>>> Igniters,
>>>>>
>>>>> I wanted to pitch a rather radical idea regarding the Ignite 3.0
>>>>> development which has occurred to me some time ago.
>>>>>
>>>>> We already have several IEPs targeted to Ignite 3.0 which imply major
>>>>> changes to the codebase (the change in replication protocol and thus
>>>>> transactions, change in binary format, updated metastorage, etc). We
>>> also
>>>>> planned significant changes in public APIs: configuration format
>> change,
>>>>> improvements in cache APIs, SQL APIs, transaction mode rework. The
>>> wishlist
>>>>> of changes for Ignite 3.0 is huge.
>>>>>
>>>>> So, I was wondering whether it makes sense to try to change the old
>>>>> codebase, or start with a new baseline and move old pieces of code
>> that
>>> do
>>>>> not require significant rework. Personally, I would go with the second
>>>>> option for the following reasons:
>>>>>
>>>>> - We have a chance to shift the development paradigm in the project
>> and
>>>>> introduce the practice of true unit-tests. In the new baseline at the
>>>>> beginning there will be no ability to run an end-to-end scenario,
>> thus
>>> we
>>>>> will be forced to write unit-tests. So far, such practice was hard to
>>>>> implement because of tight coupling between Ignite components and
>>> inability
>>>>> to instantiate components without an instance of KernalContext. For
>>>>> example, we should be able to thoroughly test internal primitives,
>>> such as
>>>>> replication protocol (without actual communication), distributed
>>>>> metastorage contracts, persistence layer, etc.
>>>>> - We will significantly reduce the development cycle in the beginning
>>>>> (right now the RunAll takes two hours of astronomical time with empty
>>> TC;
>>>>> in the new approach developer will be able to run ALL tests locally
>> in
>>> a
>>>>> matter of minutes)
>>>>> - We can get rid of TC bot and enforce green TC by integrating TC
>> build
>>>>> results with GitHub PRs (the same way Travis is currently integrated
>>> to PR
>>>>> check). We should restrict PR merge without a TC check
>>>>> - We will still have to re-write all tests, but only once. If we try
>> to
>>>>> modify the old codebase, we would need to modify all the tests for
>>> every
>>>>> major change (public API change, configuration change)
>>>>> - We will have fewer conflicts when working together. For example, I
>>>>> cannot imagine how one would merge two changes of getting rid of
>>>>> IgniteFuture and changes in replication protocol, for example
>>>>>
>>>>> Technically, I would suggest creating a new repository for Ignite 3.0
>>>>> (perhaps, a new clean branch, but a new repo looks nicer to me) and a
>>> new
>>>>> Ignite 3.0 TeamCity project.
>>>>>
>>>>> While it may seem quite radical, I do believe that this approach will
>>> give
>>>>> us more benefits than trying to make such major changes in the
>> existing
>>>>> codebase. If needed, let's schedule a discord chat like before to
>>> discuss
>>>>> this.
>>>>>
>>>>> WDYT?
>>>>
>>>
>>>
>>
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Anton Vinogradov <av...@apache.org>.
Alexey,
Do we have any estimates of how fast we'll be able to gain production-ready
AI 3.0 in case of a "new repo" choice?
On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk <al...@gmail.com>
wrote:
> Nikolay,
>
> What new features are we planning to implement for Ignite 2.x? I think once
> we commence working on Ignite 3.0, we should gradually cease the activity
> on Ignite 2.x to mere bugfixes because such parallel development will be
> overwhelming regardless of how we choose to proceed.
>
> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <ni...@apache.org>:
>
> > To be clear:
> >
> > > I would suggest creating a new repository for Ignite 3.0 (perhaps, a
> new
> > clean branch, but a new repo looks nicer to me) and a new Ignite 3.0
> > TeamCity project.
> >
> > +1 for new Team City project.
> > +1 for new branch for Ignite3.
> > -1 for new repo.
> >
> >
> > > 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <NI...@gmail.com>
> > написал(а):
> > >
> > > Hello, Alexey.
> > >
> > > I think it will hurt our project more than help.
> > > Developing new features for 2 separate branches with the different APIs
> > and internal structure is overwhelming
> > >
> > > Maybe we should relax a bit requirements for Ignite3?
> > > Maybe we should move step by step and make Ignite3 with new
> > configuration than Ignite4 with new transactions, etc?
> > >
> > >> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <
> alexey.goncharuk@gmail.com>
> > написал(а):
> > >>
> > >> Igniters,
> > >>
> > >> I wanted to pitch a rather radical idea regarding the Ignite 3.0
> > >> development which has occurred to me some time ago.
> > >>
> > >> We already have several IEPs targeted to Ignite 3.0 which imply major
> > >> changes to the codebase (the change in replication protocol and thus
> > >> transactions, change in binary format, updated metastorage, etc). We
> > also
> > >> planned significant changes in public APIs: configuration format
> change,
> > >> improvements in cache APIs, SQL APIs, transaction mode rework. The
> > wishlist
> > >> of changes for Ignite 3.0 is huge.
> > >>
> > >> So, I was wondering whether it makes sense to try to change the old
> > >> codebase, or start with a new baseline and move old pieces of code
> that
> > do
> > >> not require significant rework. Personally, I would go with the second
> > >> option for the following reasons:
> > >>
> > >> - We have a chance to shift the development paradigm in the project
> and
> > >> introduce the practice of true unit-tests. In the new baseline at the
> > >> beginning there will be no ability to run an end-to-end scenario,
> thus
> > we
> > >> will be forced to write unit-tests. So far, such practice was hard to
> > >> implement because of tight coupling between Ignite components and
> > inability
> > >> to instantiate components without an instance of KernalContext. For
> > >> example, we should be able to thoroughly test internal primitives,
> > such as
> > >> replication protocol (without actual communication), distributed
> > >> metastorage contracts, persistence layer, etc.
> > >> - We will significantly reduce the development cycle in the beginning
> > >> (right now the RunAll takes two hours of astronomical time with empty
> > TC;
> > >> in the new approach developer will be able to run ALL tests locally
> in
> > a
> > >> matter of minutes)
> > >> - We can get rid of TC bot and enforce green TC by integrating TC
> build
> > >> results with GitHub PRs (the same way Travis is currently integrated
> > to PR
> > >> check). We should restrict PR merge without a TC check
> > >> - We will still have to re-write all tests, but only once. If we try
> to
> > >> modify the old codebase, we would need to modify all the tests for
> > every
> > >> major change (public API change, configuration change)
> > >> - We will have fewer conflicts when working together. For example, I
> > >> cannot imagine how one would merge two changes of getting rid of
> > >> IgniteFuture and changes in replication protocol, for example
> > >>
> > >> Technically, I would suggest creating a new repository for Ignite 3.0
> > >> (perhaps, a new clean branch, but a new repo looks nicer to me) and a
> > new
> > >> Ignite 3.0 TeamCity project.
> > >>
> > >> While it may seem quite radical, I do believe that this approach will
> > give
> > >> us more benefits than trying to make such major changes in the
> existing
> > >> codebase. If needed, let's schedule a discord chat like before to
> > discuss
> > >> this.
> > >>
> > >> WDYT?
> > >
> >
> >
>
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Alexey Goncharuk <al...@gmail.com>.
Nikolay,
What new features are we planning to implement for Ignite 2.x? I think once
we commence working on Ignite 3.0, we should gradually cease the activity
on Ignite 2.x to mere bugfixes because such parallel development will be
overwhelming regardless of how we choose to proceed.
пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <ni...@apache.org>:
> To be clear:
>
> > I would suggest creating a new repository for Ignite 3.0 (perhaps, a new
> clean branch, but a new repo looks nicer to me) and a new Ignite 3.0
> TeamCity project.
>
> +1 for new Team City project.
> +1 for new branch for Ignite3.
> -1 for new repo.
>
>
> > 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <NI...@gmail.com>
> написал(а):
> >
> > Hello, Alexey.
> >
> > I think it will hurt our project more than help.
> > Developing new features for 2 separate branches with the different APIs
> and internal structure is overwhelming
> >
> > Maybe we should relax a bit requirements for Ignite3?
> > Maybe we should move step by step and make Ignite3 with new
> configuration than Ignite4 with new transactions, etc?
> >
> >> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <al...@gmail.com>
> написал(а):
> >>
> >> Igniters,
> >>
> >> I wanted to pitch a rather radical idea regarding the Ignite 3.0
> >> development which has occurred to me some time ago.
> >>
> >> We already have several IEPs targeted to Ignite 3.0 which imply major
> >> changes to the codebase (the change in replication protocol and thus
> >> transactions, change in binary format, updated metastorage, etc). We
> also
> >> planned significant changes in public APIs: configuration format change,
> >> improvements in cache APIs, SQL APIs, transaction mode rework. The
> wishlist
> >> of changes for Ignite 3.0 is huge.
> >>
> >> So, I was wondering whether it makes sense to try to change the old
> >> codebase, or start with a new baseline and move old pieces of code that
> do
> >> not require significant rework. Personally, I would go with the second
> >> option for the following reasons:
> >>
> >> - We have a chance to shift the development paradigm in the project and
> >> introduce the practice of true unit-tests. In the new baseline at the
> >> beginning there will be no ability to run an end-to-end scenario, thus
> we
> >> will be forced to write unit-tests. So far, such practice was hard to
> >> implement because of tight coupling between Ignite components and
> inability
> >> to instantiate components without an instance of KernalContext. For
> >> example, we should be able to thoroughly test internal primitives,
> such as
> >> replication protocol (without actual communication), distributed
> >> metastorage contracts, persistence layer, etc.
> >> - We will significantly reduce the development cycle in the beginning
> >> (right now the RunAll takes two hours of astronomical time with empty
> TC;
> >> in the new approach developer will be able to run ALL tests locally in
> a
> >> matter of minutes)
> >> - We can get rid of TC bot and enforce green TC by integrating TC build
> >> results with GitHub PRs (the same way Travis is currently integrated
> to PR
> >> check). We should restrict PR merge without a TC check
> >> - We will still have to re-write all tests, but only once. If we try to
> >> modify the old codebase, we would need to modify all the tests for
> every
> >> major change (public API change, configuration change)
> >> - We will have fewer conflicts when working together. For example, I
> >> cannot imagine how one would merge two changes of getting rid of
> >> IgniteFuture and changes in replication protocol, for example
> >>
> >> Technically, I would suggest creating a new repository for Ignite 3.0
> >> (perhaps, a new clean branch, but a new repo looks nicer to me) and a
> new
> >> Ignite 3.0 TeamCity project.
> >>
> >> While it may seem quite radical, I do believe that this approach will
> give
> >> us more benefits than trying to make such major changes in the existing
> >> codebase. If needed, let's schedule a discord chat like before to
> discuss
> >> this.
> >>
> >> WDYT?
> >
>
>
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Nikolay Izhikov <ni...@apache.org>.
To be clear:
> I would suggest creating a new repository for Ignite 3.0 (perhaps, a new clean branch, but a new repo looks nicer to me) and a new Ignite 3.0 TeamCity project.
+1 for new Team City project.
+1 for new branch for Ignite3.
-1 for new repo.
> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <NI...@gmail.com> написал(а):
>
> Hello, Alexey.
>
> I think it will hurt our project more than help.
> Developing new features for 2 separate branches with the different APIs and internal structure is overwhelming
>
> Maybe we should relax a bit requirements for Ignite3?
> Maybe we should move step by step and make Ignite3 with new configuration than Ignite4 with new transactions, etc?
>
>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <al...@gmail.com> написал(а):
>>
>> Igniters,
>>
>> I wanted to pitch a rather radical idea regarding the Ignite 3.0
>> development which has occurred to me some time ago.
>>
>> We already have several IEPs targeted to Ignite 3.0 which imply major
>> changes to the codebase (the change in replication protocol and thus
>> transactions, change in binary format, updated metastorage, etc). We also
>> planned significant changes in public APIs: configuration format change,
>> improvements in cache APIs, SQL APIs, transaction mode rework. The wishlist
>> of changes for Ignite 3.0 is huge.
>>
>> So, I was wondering whether it makes sense to try to change the old
>> codebase, or start with a new baseline and move old pieces of code that do
>> not require significant rework. Personally, I would go with the second
>> option for the following reasons:
>>
>> - We have a chance to shift the development paradigm in the project and
>> introduce the practice of true unit-tests. In the new baseline at the
>> beginning there will be no ability to run an end-to-end scenario, thus we
>> will be forced to write unit-tests. So far, such practice was hard to
>> implement because of tight coupling between Ignite components and inability
>> to instantiate components without an instance of KernalContext. For
>> example, we should be able to thoroughly test internal primitives, such as
>> replication protocol (without actual communication), distributed
>> metastorage contracts, persistence layer, etc.
>> - We will significantly reduce the development cycle in the beginning
>> (right now the RunAll takes two hours of astronomical time with empty TC;
>> in the new approach developer will be able to run ALL tests locally in a
>> matter of minutes)
>> - We can get rid of TC bot and enforce green TC by integrating TC build
>> results with GitHub PRs (the same way Travis is currently integrated to PR
>> check). We should restrict PR merge without a TC check
>> - We will still have to re-write all tests, but only once. If we try to
>> modify the old codebase, we would need to modify all the tests for every
>> major change (public API change, configuration change)
>> - We will have fewer conflicts when working together. For example, I
>> cannot imagine how one would merge two changes of getting rid of
>> IgniteFuture and changes in replication protocol, for example
>>
>> Technically, I would suggest creating a new repository for Ignite 3.0
>> (perhaps, a new clean branch, but a new repo looks nicer to me) and a new
>> Ignite 3.0 TeamCity project.
>>
>> While it may seem quite radical, I do believe that this approach will give
>> us more benefits than trying to make such major changes in the existing
>> codebase. If needed, let's schedule a discord chat like before to discuss
>> this.
>>
>> WDYT?
>
Re: [DISCUSS] Ignite 3.0 development approach
Posted by Nikolay Izhikov <ni...@apache.org>.
Hello, Alexey.
I think it will hurt our project more than help.
Developing new features for 2 separate branches with the different APIs and internal structure is overwhelming
Maybe we should relax a bit requirements for Ignite3?
Maybe we should move step by step and make Ignite3 with new configuration than Ignite4 with new transactions, etc?
> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <al...@gmail.com> написал(а):
>
> Igniters,
>
> I wanted to pitch a rather radical idea regarding the Ignite 3.0
> development which has occurred to me some time ago.
>
> We already have several IEPs targeted to Ignite 3.0 which imply major
> changes to the codebase (the change in replication protocol and thus
> transactions, change in binary format, updated metastorage, etc). We also
> planned significant changes in public APIs: configuration format change,
> improvements in cache APIs, SQL APIs, transaction mode rework. The wishlist
> of changes for Ignite 3.0 is huge.
>
> So, I was wondering whether it makes sense to try to change the old
> codebase, or start with a new baseline and move old pieces of code that do
> not require significant rework. Personally, I would go with the second
> option for the following reasons:
>
> - We have a chance to shift the development paradigm in the project and
> introduce the practice of true unit-tests. In the new baseline at the
> beginning there will be no ability to run an end-to-end scenario, thus we
> will be forced to write unit-tests. So far, such practice was hard to
> implement because of tight coupling between Ignite components and inability
> to instantiate components without an instance of KernalContext. For
> example, we should be able to thoroughly test internal primitives, such as
> replication protocol (without actual communication), distributed
> metastorage contracts, persistence layer, etc.
> - We will significantly reduce the development cycle in the beginning
> (right now the RunAll takes two hours of astronomical time with empty TC;
> in the new approach developer will be able to run ALL tests locally in a
> matter of minutes)
> - We can get rid of TC bot and enforce green TC by integrating TC build
> results with GitHub PRs (the same way Travis is currently integrated to PR
> check). We should restrict PR merge without a TC check
> - We will still have to re-write all tests, but only once. If we try to
> modify the old codebase, we would need to modify all the tests for every
> major change (public API change, configuration change)
> - We will have fewer conflicts when working together. For example, I
> cannot imagine how one would merge two changes of getting rid of
> IgniteFuture and changes in replication protocol, for example
>
> Technically, I would suggest creating a new repository for Ignite 3.0
> (perhaps, a new clean branch, but a new repo looks nicer to me) and a new
> Ignite 3.0 TeamCity project.
>
> While it may seem quite radical, I do believe that this approach will give
> us more benefits than trying to make such major changes in the existing
> codebase. If needed, let's schedule a discord chat like before to discuss
> this.
>
> WDYT?