You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by Petr Ivanov <mr...@gmail.com> on 2020/08/27 09:16:52 UTC
IEP-52: Binary Delivery & Upgradability Enhancements
Hi, Igniters!
For Apache Ignite 3.0 I would like to propose vision of enhanced delivery and upgrade processes [1].
The key idea is to make main binary as slim as possible (delivering every optional component by demand only) while providing way to run each new version seamlessly without much of the efforts migrating data and configuration between upgrades.
I plan to create prototype based on current release of Apache Ignite (2.8.1 or 2.9.0 if it will be released soon) later in September.
[1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158873958
Re: IEP-52: Binary Delivery & Upgradability Enhancements
Posted by Ivan Daschinsky <iv...@gmail.com>.
I mean, that we should fix this issue, not just throw away zookeeper
discovery.
чт, 17 сент. 2020 г. в 16:44, Ivan Daschinsky <iv...@gmail.com>:
> I suggested to move TcpDiscoveryZookeeperIpFinder to ignite-extension.
> This tiny recipe-like class brings tons of dependency and has nothing
> common with ZookeeperDiscovery, except it name.
>
> ZookeeperDiscoveryImpl depends only on zookeeper-3.5.5.jar and
> zookeeper-jute-3.5.5.jar.
>
>
>
>
> чт, 17 сент. 2020 г. в 15:07, Ilya Kasnacheev <il...@gmail.com>:
>
>> Hello!
>>
>> The situation as follows:
>> libs/optional/ignite-kubernetes:
>> ignite-kubernetes-2.8.1.jar jackson-annotations-2.9.10.jar
>> jackson-core-2.9.10.jar jackson-databind-2.9.10.jar
>>
>> libs/optional/ignite-zookeeper:
>> curator-client-4.2.0.jar curator-x-discovery-4.2.0.jar
>> jackson-core-asl-1.9.13.jar log4j-1.2.17.jar
>> slf4j-log4j12-1.7.7.jar
>> curator-framework-4.2.0.jar guava-16.0.1.jar
>> jackson-mapper-asl-1.9.13.jar zookeeper-3.5.5.jar
>> curator-recipes-4.2.0.jar ignite-zookeeper-2.8.1.jar
>> slf4j-api-1.7.7.jar
>> zookeeper-jute-3.5.5.jar
>>
>> ignite-zookeeper just has way more dependencies than ignite-kubernetes,
>> and
>> their size is 5-fold.
>>
>> These dependencies are not just weight but also vulnerability surface.
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> чт, 17 сент. 2020 г. в 13:55, Ivan Daschinsky <iv...@gmail.com>:
>>
>> > Hi! I recently found that in slim binary release (at least in nightly
>> build
>> > of 2.9.0) there is no ignite-zookeeper module.
>> > But, for example, ignite-kubernetes is in.
>> > Petr, could you please clarify why this decision was made?
>> > We currently have only two discovery implementation in project and leave
>> > only one alternative looks not obvious for me.
>> >
>> > ср, 16 сент. 2020 г. в 17:04, Alexei Scherbakov <
>> > alexey.scherbakoff@gmail.com>:
>> >
>> > > Downloading additional jars will not 100% work in private networks,
>> so we
>> > > must provide full distro anyway.
>> > >
>> > > Also, I do not see any problems in downloading all dependencies,
>> because
>> > > their summary size is not too big.
>> > >
>> > > I suggest, in addition to tgz archive for manual deployment, provide
>> > Ignite
>> > > distribution as RPM (or other package manager(s)) for Linux (MSI for
>> > > Windows) to automate distribution upgrades, instead of reinventing the
>> > > wheel.
>> > >
>> > > пн, 31 авг. 2020 г. в 10:09, Valentin Kulichenko <
>> > > valentin.kulichenko@gmail.com>:
>> > >
>> > > > Hi Nikolay,
>> > > >
>> > > > Thanks for your feedback! Absolutely, the IEP is currently in the
>> draft
>> > > > mode - we will be adding more details going forward. Also, Petr
>> > mentioned
>> > > > that he wants to create a prototype which I think will make things
>> > > clearer
>> > > > for all of us.
>> > > >
>> > > > As for "download jars from the Internet on a production server",
>> that's
>> > > not
>> > > > necessarily what we propose. In case of Maven, artifacts are
>> downloaded
>> > > > from a specified repository, which doesn't have to be on the
>> Internet.
>> > > > Companies that have security concerns typically have a mirror/proxy
>> > > > containing approved artifacts only, already checked for viruses,
>> etc.
>> > > > Production servers use the proxy rather than public repositories.
>> > > >
>> > > > The concerns that you brought up are certainly valid. However, I
>> > believe
>> > > > that using Maven actually addresses them, because Maven (as any
>> other
>> > > > package/dependency manager for that matter) provides the
>> corresponding
>> > > > functionality out of the box.
>> > > >
>> > > > On the contrary, users who currently download our ZIP package are
>> > > *forced*
>> > > > to download modules and dependencies that they do not even need and
>> > will
>> > > > not use. In my experience, this is actually a bigger source of
>> concern.
>> > > >
>> > > > Anyway, let us add more details to the IEP, and we will go from
>> there.
>> > > >
>> > > > P.S. I'm also not sure I agree with your assessment of the embedded
>> > mode
>> > > > and thick clients, but that's probably a different discussion.
>> Please
>> > > feel
>> > > > free to create a new thread if you would like to discuss this.
>> > > >
>> > > > -Val
>> > > >
>> > > > On Sat, Aug 29, 2020 at 1:46 AM Nikolay Izhikov <
>> nizhikov@apache.org>
>> > > > wrote:
>> > > >
>> > > > > Also, guys.
>> > > > >
>> > > > > Can we make one step back and add some requirements to the IEP.
>> > > > > How we want to install and upgrade Ignite?
>> > > > > What specific issues we want to solve?
>> > > > >
>> > > > > With this input we can be on the same page during solution
>> > discussion.
>> > > > >
>> > > > > > 29 авг. 2020 г., в 11:41, Nikolay Izhikov <
>> NIzhikov.dev@gmail.com>
>> > > > > написал(а):
>> > > > > >
>> > > > > > Hello, Val.
>> > > > > >
>> > > > > > Sorry, but I sill don’t understand how «download jars from the
>> > > internet
>> > > > > on production server» approach helps us to solve the issues you
>> > > describe.
>> > > > > >
>> > > > > > Moreover, with this enhancement you want to add more
>> dependencies
>> > to
>> > > > > Ignite and create some security vulnerabilities:
>> > > > > > * Maven tool - new dependency. I doubt that and of the
>> > > production
>> > > > > server have it
>> > > > > > * Internet maven repository - many production servers
>> > restrict
>> > > > > outgoing internet connection to any addresses.
>> > > > > > It’s hard to understand why distributed DB should
>> > have
>> > > > the
>> > > > > ability to connect to some kind of address on the Internet.
>> > > > > > * All Ignite distributions can be compromised with the
>> hack
>> > of
>> > > > > some random maven repo or with the malicious proxy.
>> > > > > > * Note, that Ignite installation or upgrade procedure can
>> > > become
>> > > > > dead just because of some Maven repo down.
>> > > > > >
>> > > > > >
>> > > > > > I have 2 concerns:
>> > > > > >
>> > > > > > 1. We shouldn’t download any jars from the internet during
>> > production
>> > > > > deployment or upgrade procedure.
>> > > > > > 2. The IEP description, for now, is quite small for such
>> important
>> > > > > things as a way we distribute and upgrade Ignite.
>> > > > > > Petr, can you, please, make IEP more specific.
>> > > > > > I think we should add the following details:
>> > > > > > * Description of the typical Ignite deployment
>> > > procedure
>> > > > > including folder structure(config, persistence, .m2 and other
>> > folders)
>> > > > > > * This also should include examples of the
>> > bash
>> > > > > commands if any required.
>> > > > > > * Description of the typical upgrade(downgrade?)
>> > > > procedure
>> > > > > > * This also should include examples of the
>> > bash
>> > > > > commands if any required.
>> > > > > > * Description of «single tool responsible for all
>> > > > > operations».
>> > > > > > * all commands
>> > > > > > * all operations
>> > > > > > * all parameters
>> > > > > >
>> > > > > >> 1. Most paths are resolved relative to IGNITE_HOME, which
>> > > effectively
>> > > > > forces (or at least motivates) users to put everything related to
>> > > Ignite
>> > > > > under IGNITE_HOME (configs, additional JARs, etc.).
>> > > > > >
>> > > > > > AFAIU this issue doesn’t relate to the way Ignite distributed.
>> > > > > > We can and should fix this outside of this IEP.
>> > > > > >
>> > > > > >> 2. Modules are enabled/disabled manually by moving folders
>> around…
>> > > > this
>> > > > > is error-prone,
>> > > > > >
>> > > > > > Makes sense.
>> > > > > > Let’s fix this without «download jar from the internet»?
>> > > > > > We can have some kind of config with the enabled modules.
>> > > > > >
>> > > > > >> 3. Standalone nodes use JARs prepackaged in the ZIP file.
>> However,
>> > > if
>> > > > > you look at how this ZIP is built, you will notice that we don't
>> > > actually
>> > > > > include all transitive dependencies,
>> > > > > >
>> > > > > > We need to fix this.
>> > > > > > Let’s include all the required JARs to the distribution.
>> > > > > >
>> > > > > >> which means that an embedded node that uses Maven can have a
>> > > different
>> > > > > classpath.
>> > > > > >
>> > > > > > Embedded Ignite node is an error-prone design decision in the
>> first
>> > > > > place.
>> > > > > > We can’t fix it with the «download jars from the internet»
>> > approach.
>> > > > > > We should eliminate the client node as a thing and use a thin
>> > clients
>> > > > > instead of it.
>> > > > > >
>> > > > > >> 4. ignite.sh allows manual modifications (there are several
>> > > "Uncomment
>> > > > > this to..." lines). This complicates upgrades as well.
>> > > > > >
>> > > > > > Makes sense.
>> > > > > > Let’s fix this.
>> > > > > >
>> > > > > >> The whole purpose of the IEP is to fix the above, automate and
>> > > > simplify
>> > > > > operations for the users. The general idea is to create a single
>> tool
>> > > > > responsible for all those operations and use Maven under the hood
>> to
>> > > > > include/exclude modules.
>> > > > > >
>> > > > > > For now, IEP lacks the description of this tool.
>> > > > > >
>> > > > > >> 28 авг. 2020 г., в 22:35, Valentin Kulichenko <
>> > > > > valentin.kulichenko@gmail.com> написал(а):
>> > > > > >>
>> > > > > >> Hi Nikolay,
>> > > > > >>
>> > > > > >> Here are some of the issues that I see with the current binary
>> ZIP
>> > > > > package
>> > > > > >> that we create for every release.
>> > > > > >>
>> > > > > >> 1. Most paths are resolved relative to IGNITE_HOME, which
>> > > effectively
>> > > > > >> forces (or at least motivates) users to put everything related
>> to
>> > > > Ignite
>> > > > > >> under IGNITE_HOME (configs, additional JARs, etc.). Now, when a
>> > user
>> > > > > >> downloads and unzips a new version, they have to somehow merge
>> the
>> > > > > content
>> > > > > >> together - that's poor upgradability.
>> > > > > >> 2. Modules are enabled/disabled manually by moving folders
>> around.
>> > > Not
>> > > > > only
>> > > > > >> this is error-prone, but it also adds more issues to
>> > upgradability -
>> > > > you
>> > > > > >> have to apply these steps for every version you download.
>> > > > > >> 3. Standalone nodes use JARs prepackaged in the ZIP file.
>> However,
>> > > if
>> > > > > you
>> > > > > >> look at how this ZIP is built, you will notice that we don't
>> > > actually
>> > > > > >> include all transitive dependencies, which means that an
>> embedded
>> > > node
>> > > > > that
>> > > > > >> uses Maven can have a different classpath. It seems that we've
>> > been
>> > > > > lucky
>> > > > > >> so far as this hasn't caused any issues, but I still think the
>> > > > approach
>> > > > > is
>> > > > > >> wrong. The behavior should be consistent.
>> > > > > >> 4. ignite.sh allows manual modifications (there are several
>> > > "Uncomment
>> > > > > this
>> > > > > >> to..." lines). This complicates upgrades as well.
>> > > > > >>
>> > > > > >> The whole purpose of the IEP is to fix the above, automate and
>> > > > > >> simplify operations for the users. The general idea is to
>> create a
>> > > > > single
>> > > > > >> tool responsible for all those operations and use Maven under
>> the
>> > > hood
>> > > > > to
>> > > > > >> include/exclude modules.
>> > > > > >>
>> > > > > >> Exact commands and steps are still under discussion though. It
>> > looks
>> > > > > like
>> > > > > >> Petr is going to create a prototype that can be used as a
>> starting
>> > > > > point,
>> > > > > >> but if you have any suggestions or ideas in the meantime,
>> please
>> > > share
>> > > > > them
>> > > > > >> with us.
>> > > > > >>
>> > > > > >> -Val
>> > > > > >>
>> > > > > >> On Fri, Aug 28, 2020 at 12:31 AM Nikolay Izhikov <
>> > > nizhikov@apache.org
>> > > > >
>> > > > > >> wrote:
>> > > > > >>
>> > > > > >>> Hello, Igniters.
>> > > > > >>>
>> > > > > >>> Not sure if I understand the essence of the proposal at the
>> > moment.
>> > > > > >>> Can you, please, describe the advantages of the proposed way
>> from
>> > > the
>> > > > > user
>> > > > > >>> perspective?
>> > > > > >>>
>> > > > > >>> How the typical DevOps pipeline should look like with this
>> > > > enhancement?
>> > > > > >>>
>> > > > > >>> How I as a user can create a fully functional installation
>> > package
>> > > of
>> > > > > the
>> > > > > >>> Ignite?
>> > > > > >>> AFAIK downloading some artifacts from the internet straight to
>> > the
>> > > > > >>> production server is a security anti-pattern.
>> > > > > >>>
>> > > > > >>>
>> > > > > >>>> 28 авг. 2020 г., в 01:59, Denis Magda <dm...@apache.org>
>> > > > написал(а):
>> > > > > >>>>
>> > > > > >>>> Petr, thanks,
>> > > > > >>>>
>> > > > > >>>> There is also a collection of modules located in our
>> extensions
>> > > > > >>> repository:
>> > > > > >>>> https://github.com/apache/ignite-extensions
>> > > > > >>>>
>> > > > > >>>> @Saikat Maitra <sa...@gmail.com> is migrating all
>> our
>> > > > > existing
>> > > > > >>>> integrations to that repository and, once this is done, the
>> > > > extensions
>> > > > > >>> will
>> > > > > >>>> be released to Maven separately. Is it reasonable to expand
>> the
>> > > > scope
>> > > > > of
>> > > > > >>>> the IEP-52 and contemplate on how to install those
>> extensions?
>> > > > > >>>>
>> > > > > >>>> -
>> > > > > >>>> Denis
>> > > > > >>>>
>> > > > > >>>>
>> > > > > >>>> On Thu, Aug 27, 2020 at 3:40 PM Valentin Kulichenko <
>> > > > > >>>> valentin.kulichenko@gmail.com> wrote:
>> > > > > >>>>
>> > > > > >>>>> Hi Petr,
>> > > > > >>>>>
>> > > > > >>>>> The proposal makes sense to me - thanks for starting the
>> > > > discussion.
>> > > > > >>>>> Current installation and upgrade procedures involve a lot of
>> > > manual
>> > > > > >>> steps
>> > > > > >>>>> and quite error-prone, we need to automate and simplify
>> them as
>> > > > much
>> > > > > as
>> > > > > >>>>> possible.
>> > > > > >>>>>
>> > > > > >>>>> I believe we eventually should end up with a unified
>> > command-line
>> > > > > tool
>> > > > > >>>>> (ignitectl?) that will incorporate all the operations
>> > > > (enable/disable
>> > > > > >>>>> modules, start/stop, configuration updates, activation,
>> > baseline,
>> > > > > etc.).
>> > > > > >>>>> This IEP is a step in this direction.
>> > > > > >>>>>
>> > > > > >>>>> Looking forward to testing a prototype :)
>> > > > > >>>>>
>> > > > > >>>>> -Val
>> > > > > >>>>>
>> > > > > >>>>> On Thu, Aug 27, 2020 at 2:17 AM Petr Ivanov <
>> > mr.weider@gmail.com
>> > > >
>> > > > > >>> wrote:
>> > > > > >>>>>
>> > > > > >>>>>> Hi, Igniters!
>> > > > > >>>>>>
>> > > > > >>>>>>
>> > > > > >>>>>> For Apache Ignite 3.0 I would like to propose vision of
>> > enhanced
>> > > > > >>> delivery
>> > > > > >>>>>> and upgrade processes [1].
>> > > > > >>>>>> The key idea is to make main binary as slim as possible
>> > > > (delivering
>> > > > > >>> every
>> > > > > >>>>>> optional component by demand only) while providing way to
>> run
>> > > each
>> > > > > new
>> > > > > >>>>>> version seamlessly without much of the efforts migrating
>> data
>> > > and
>> > > > > >>>>>> configuration between upgrades.
>> > > > > >>>>>>
>> > > > > >>>>>> I plan to create prototype based on current release of
>> Apache
>> > > > Ignite
>> > > > > >>>>>> (2.8.1 or 2.9.0 if it will be released soon) later in
>> > September.
>> > > > > >>>>>>
>> > > > > >>>>>>
>> > > > > >>>>>>
>> > > > > >>>>>> [1]
>> > > > > >>>>>>
>> > > > > >>>>>
>> > > > > >>>
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158873958
>> > > > > >>>>>>
>> > > > > >>>>>>
>> > > > > >>>>>
>> > > > > >>>
>> > > > > >>>
>> > > > > >
>> > > > >
>> > > > >
>> > > >
>> > >
>> > >
>> > > --
>> > >
>> > > Best regards,
>> > > Alexei Scherbakov
>> > >
>> >
>> >
>> > --
>> > Sincerely yours, Ivan Daschinskiy
>> >
>>
>
>
> --
> Sincerely yours, Ivan Daschinskiy
>
--
Sincerely yours, Ivan Daschinskiy
Re: IEP-52: Binary Delivery & Upgradability Enhancements
Posted by Ivan Daschinsky <iv...@gmail.com>.
I suggested to move TcpDiscoveryZookeeperIpFinder to ignite-extension.
This tiny recipe-like class brings tons of dependency and has nothing
common with ZookeeperDiscovery, except it name.
ZookeeperDiscoveryImpl depends only on zookeeper-3.5.5.jar and
zookeeper-jute-3.5.5.jar.
чт, 17 сент. 2020 г. в 15:07, Ilya Kasnacheev <il...@gmail.com>:
> Hello!
>
> The situation as follows:
> libs/optional/ignite-kubernetes:
> ignite-kubernetes-2.8.1.jar jackson-annotations-2.9.10.jar
> jackson-core-2.9.10.jar jackson-databind-2.9.10.jar
>
> libs/optional/ignite-zookeeper:
> curator-client-4.2.0.jar curator-x-discovery-4.2.0.jar
> jackson-core-asl-1.9.13.jar log4j-1.2.17.jar
> slf4j-log4j12-1.7.7.jar
> curator-framework-4.2.0.jar guava-16.0.1.jar
> jackson-mapper-asl-1.9.13.jar zookeeper-3.5.5.jar
> curator-recipes-4.2.0.jar ignite-zookeeper-2.8.1.jar
> slf4j-api-1.7.7.jar
> zookeeper-jute-3.5.5.jar
>
> ignite-zookeeper just has way more dependencies than ignite-kubernetes, and
> their size is 5-fold.
>
> These dependencies are not just weight but also vulnerability surface.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> чт, 17 сент. 2020 г. в 13:55, Ivan Daschinsky <iv...@gmail.com>:
>
> > Hi! I recently found that in slim binary release (at least in nightly
> build
> > of 2.9.0) there is no ignite-zookeeper module.
> > But, for example, ignite-kubernetes is in.
> > Petr, could you please clarify why this decision was made?
> > We currently have only two discovery implementation in project and leave
> > only one alternative looks not obvious for me.
> >
> > ср, 16 сент. 2020 г. в 17:04, Alexei Scherbakov <
> > alexey.scherbakoff@gmail.com>:
> >
> > > Downloading additional jars will not 100% work in private networks, so
> we
> > > must provide full distro anyway.
> > >
> > > Also, I do not see any problems in downloading all dependencies,
> because
> > > their summary size is not too big.
> > >
> > > I suggest, in addition to tgz archive for manual deployment, provide
> > Ignite
> > > distribution as RPM (or other package manager(s)) for Linux (MSI for
> > > Windows) to automate distribution upgrades, instead of reinventing the
> > > wheel.
> > >
> > > пн, 31 авг. 2020 г. в 10:09, Valentin Kulichenko <
> > > valentin.kulichenko@gmail.com>:
> > >
> > > > Hi Nikolay,
> > > >
> > > > Thanks for your feedback! Absolutely, the IEP is currently in the
> draft
> > > > mode - we will be adding more details going forward. Also, Petr
> > mentioned
> > > > that he wants to create a prototype which I think will make things
> > > clearer
> > > > for all of us.
> > > >
> > > > As for "download jars from the Internet on a production server",
> that's
> > > not
> > > > necessarily what we propose. In case of Maven, artifacts are
> downloaded
> > > > from a specified repository, which doesn't have to be on the
> Internet.
> > > > Companies that have security concerns typically have a mirror/proxy
> > > > containing approved artifacts only, already checked for viruses, etc.
> > > > Production servers use the proxy rather than public repositories.
> > > >
> > > > The concerns that you brought up are certainly valid. However, I
> > believe
> > > > that using Maven actually addresses them, because Maven (as any other
> > > > package/dependency manager for that matter) provides the
> corresponding
> > > > functionality out of the box.
> > > >
> > > > On the contrary, users who currently download our ZIP package are
> > > *forced*
> > > > to download modules and dependencies that they do not even need and
> > will
> > > > not use. In my experience, this is actually a bigger source of
> concern.
> > > >
> > > > Anyway, let us add more details to the IEP, and we will go from
> there.
> > > >
> > > > P.S. I'm also not sure I agree with your assessment of the embedded
> > mode
> > > > and thick clients, but that's probably a different discussion. Please
> > > feel
> > > > free to create a new thread if you would like to discuss this.
> > > >
> > > > -Val
> > > >
> > > > On Sat, Aug 29, 2020 at 1:46 AM Nikolay Izhikov <nizhikov@apache.org
> >
> > > > wrote:
> > > >
> > > > > Also, guys.
> > > > >
> > > > > Can we make one step back and add some requirements to the IEP.
> > > > > How we want to install and upgrade Ignite?
> > > > > What specific issues we want to solve?
> > > > >
> > > > > With this input we can be on the same page during solution
> > discussion.
> > > > >
> > > > > > 29 авг. 2020 г., в 11:41, Nikolay Izhikov <
> NIzhikov.dev@gmail.com>
> > > > > написал(а):
> > > > > >
> > > > > > Hello, Val.
> > > > > >
> > > > > > Sorry, but I sill don’t understand how «download jars from the
> > > internet
> > > > > on production server» approach helps us to solve the issues you
> > > describe.
> > > > > >
> > > > > > Moreover, with this enhancement you want to add more dependencies
> > to
> > > > > Ignite and create some security vulnerabilities:
> > > > > > * Maven tool - new dependency. I doubt that and of the
> > > production
> > > > > server have it
> > > > > > * Internet maven repository - many production servers
> > restrict
> > > > > outgoing internet connection to any addresses.
> > > > > > It’s hard to understand why distributed DB should
> > have
> > > > the
> > > > > ability to connect to some kind of address on the Internet.
> > > > > > * All Ignite distributions can be compromised with the hack
> > of
> > > > > some random maven repo or with the malicious proxy.
> > > > > > * Note, that Ignite installation or upgrade procedure can
> > > become
> > > > > dead just because of some Maven repo down.
> > > > > >
> > > > > >
> > > > > > I have 2 concerns:
> > > > > >
> > > > > > 1. We shouldn’t download any jars from the internet during
> > production
> > > > > deployment or upgrade procedure.
> > > > > > 2. The IEP description, for now, is quite small for such
> important
> > > > > things as a way we distribute and upgrade Ignite.
> > > > > > Petr, can you, please, make IEP more specific.
> > > > > > I think we should add the following details:
> > > > > > * Description of the typical Ignite deployment
> > > procedure
> > > > > including folder structure(config, persistence, .m2 and other
> > folders)
> > > > > > * This also should include examples of the
> > bash
> > > > > commands if any required.
> > > > > > * Description of the typical upgrade(downgrade?)
> > > > procedure
> > > > > > * This also should include examples of the
> > bash
> > > > > commands if any required.
> > > > > > * Description of «single tool responsible for all
> > > > > operations».
> > > > > > * all commands
> > > > > > * all operations
> > > > > > * all parameters
> > > > > >
> > > > > >> 1. Most paths are resolved relative to IGNITE_HOME, which
> > > effectively
> > > > > forces (or at least motivates) users to put everything related to
> > > Ignite
> > > > > under IGNITE_HOME (configs, additional JARs, etc.).
> > > > > >
> > > > > > AFAIU this issue doesn’t relate to the way Ignite distributed.
> > > > > > We can and should fix this outside of this IEP.
> > > > > >
> > > > > >> 2. Modules are enabled/disabled manually by moving folders
> around…
> > > > this
> > > > > is error-prone,
> > > > > >
> > > > > > Makes sense.
> > > > > > Let’s fix this without «download jar from the internet»?
> > > > > > We can have some kind of config with the enabled modules.
> > > > > >
> > > > > >> 3. Standalone nodes use JARs prepackaged in the ZIP file.
> However,
> > > if
> > > > > you look at how this ZIP is built, you will notice that we don't
> > > actually
> > > > > include all transitive dependencies,
> > > > > >
> > > > > > We need to fix this.
> > > > > > Let’s include all the required JARs to the distribution.
> > > > > >
> > > > > >> which means that an embedded node that uses Maven can have a
> > > different
> > > > > classpath.
> > > > > >
> > > > > > Embedded Ignite node is an error-prone design decision in the
> first
> > > > > place.
> > > > > > We can’t fix it with the «download jars from the internet»
> > approach.
> > > > > > We should eliminate the client node as a thing and use a thin
> > clients
> > > > > instead of it.
> > > > > >
> > > > > >> 4. ignite.sh allows manual modifications (there are several
> > > "Uncomment
> > > > > this to..." lines). This complicates upgrades as well.
> > > > > >
> > > > > > Makes sense.
> > > > > > Let’s fix this.
> > > > > >
> > > > > >> The whole purpose of the IEP is to fix the above, automate and
> > > > simplify
> > > > > operations for the users. The general idea is to create a single
> tool
> > > > > responsible for all those operations and use Maven under the hood
> to
> > > > > include/exclude modules.
> > > > > >
> > > > > > For now, IEP lacks the description of this tool.
> > > > > >
> > > > > >> 28 авг. 2020 г., в 22:35, Valentin Kulichenko <
> > > > > valentin.kulichenko@gmail.com> написал(а):
> > > > > >>
> > > > > >> Hi Nikolay,
> > > > > >>
> > > > > >> Here are some of the issues that I see with the current binary
> ZIP
> > > > > package
> > > > > >> that we create for every release.
> > > > > >>
> > > > > >> 1. Most paths are resolved relative to IGNITE_HOME, which
> > > effectively
> > > > > >> forces (or at least motivates) users to put everything related
> to
> > > > Ignite
> > > > > >> under IGNITE_HOME (configs, additional JARs, etc.). Now, when a
> > user
> > > > > >> downloads and unzips a new version, they have to somehow merge
> the
> > > > > content
> > > > > >> together - that's poor upgradability.
> > > > > >> 2. Modules are enabled/disabled manually by moving folders
> around.
> > > Not
> > > > > only
> > > > > >> this is error-prone, but it also adds more issues to
> > upgradability -
> > > > you
> > > > > >> have to apply these steps for every version you download.
> > > > > >> 3. Standalone nodes use JARs prepackaged in the ZIP file.
> However,
> > > if
> > > > > you
> > > > > >> look at how this ZIP is built, you will notice that we don't
> > > actually
> > > > > >> include all transitive dependencies, which means that an
> embedded
> > > node
> > > > > that
> > > > > >> uses Maven can have a different classpath. It seems that we've
> > been
> > > > > lucky
> > > > > >> so far as this hasn't caused any issues, but I still think the
> > > > approach
> > > > > is
> > > > > >> wrong. The behavior should be consistent.
> > > > > >> 4. ignite.sh allows manual modifications (there are several
> > > "Uncomment
> > > > > this
> > > > > >> to..." lines). This complicates upgrades as well.
> > > > > >>
> > > > > >> The whole purpose of the IEP is to fix the above, automate and
> > > > > >> simplify operations for the users. The general idea is to
> create a
> > > > > single
> > > > > >> tool responsible for all those operations and use Maven under
> the
> > > hood
> > > > > to
> > > > > >> include/exclude modules.
> > > > > >>
> > > > > >> Exact commands and steps are still under discussion though. It
> > looks
> > > > > like
> > > > > >> Petr is going to create a prototype that can be used as a
> starting
> > > > > point,
> > > > > >> but if you have any suggestions or ideas in the meantime, please
> > > share
> > > > > them
> > > > > >> with us.
> > > > > >>
> > > > > >> -Val
> > > > > >>
> > > > > >> On Fri, Aug 28, 2020 at 12:31 AM Nikolay Izhikov <
> > > nizhikov@apache.org
> > > > >
> > > > > >> wrote:
> > > > > >>
> > > > > >>> Hello, Igniters.
> > > > > >>>
> > > > > >>> Not sure if I understand the essence of the proposal at the
> > moment.
> > > > > >>> Can you, please, describe the advantages of the proposed way
> from
> > > the
> > > > > user
> > > > > >>> perspective?
> > > > > >>>
> > > > > >>> How the typical DevOps pipeline should look like with this
> > > > enhancement?
> > > > > >>>
> > > > > >>> How I as a user can create a fully functional installation
> > package
> > > of
> > > > > the
> > > > > >>> Ignite?
> > > > > >>> AFAIK downloading some artifacts from the internet straight to
> > the
> > > > > >>> production server is a security anti-pattern.
> > > > > >>>
> > > > > >>>
> > > > > >>>> 28 авг. 2020 г., в 01:59, Denis Magda <dm...@apache.org>
> > > > написал(а):
> > > > > >>>>
> > > > > >>>> Petr, thanks,
> > > > > >>>>
> > > > > >>>> There is also a collection of modules located in our
> extensions
> > > > > >>> repository:
> > > > > >>>> https://github.com/apache/ignite-extensions
> > > > > >>>>
> > > > > >>>> @Saikat Maitra <sa...@gmail.com> is migrating all our
> > > > > existing
> > > > > >>>> integrations to that repository and, once this is done, the
> > > > extensions
> > > > > >>> will
> > > > > >>>> be released to Maven separately. Is it reasonable to expand
> the
> > > > scope
> > > > > of
> > > > > >>>> the IEP-52 and contemplate on how to install those extensions?
> > > > > >>>>
> > > > > >>>> -
> > > > > >>>> Denis
> > > > > >>>>
> > > > > >>>>
> > > > > >>>> On Thu, Aug 27, 2020 at 3:40 PM Valentin Kulichenko <
> > > > > >>>> valentin.kulichenko@gmail.com> wrote:
> > > > > >>>>
> > > > > >>>>> Hi Petr,
> > > > > >>>>>
> > > > > >>>>> The proposal makes sense to me - thanks for starting the
> > > > discussion.
> > > > > >>>>> Current installation and upgrade procedures involve a lot of
> > > manual
> > > > > >>> steps
> > > > > >>>>> and quite error-prone, we need to automate and simplify them
> as
> > > > much
> > > > > as
> > > > > >>>>> possible.
> > > > > >>>>>
> > > > > >>>>> I believe we eventually should end up with a unified
> > command-line
> > > > > tool
> > > > > >>>>> (ignitectl?) that will incorporate all the operations
> > > > (enable/disable
> > > > > >>>>> modules, start/stop, configuration updates, activation,
> > baseline,
> > > > > etc.).
> > > > > >>>>> This IEP is a step in this direction.
> > > > > >>>>>
> > > > > >>>>> Looking forward to testing a prototype :)
> > > > > >>>>>
> > > > > >>>>> -Val
> > > > > >>>>>
> > > > > >>>>> On Thu, Aug 27, 2020 at 2:17 AM Petr Ivanov <
> > mr.weider@gmail.com
> > > >
> > > > > >>> wrote:
> > > > > >>>>>
> > > > > >>>>>> Hi, Igniters!
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>> For Apache Ignite 3.0 I would like to propose vision of
> > enhanced
> > > > > >>> delivery
> > > > > >>>>>> and upgrade processes [1].
> > > > > >>>>>> The key idea is to make main binary as slim as possible
> > > > (delivering
> > > > > >>> every
> > > > > >>>>>> optional component by demand only) while providing way to
> run
> > > each
> > > > > new
> > > > > >>>>>> version seamlessly without much of the efforts migrating
> data
> > > and
> > > > > >>>>>> configuration between upgrades.
> > > > > >>>>>>
> > > > > >>>>>> I plan to create prototype based on current release of
> Apache
> > > > Ignite
> > > > > >>>>>> (2.8.1 or 2.9.0 if it will be released soon) later in
> > September.
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>> [1]
> > > > > >>>>>>
> > > > > >>>>>
> > > > > >>>
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158873958
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>
> > > > > >>>
> > > > > >>>
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > >
> > > Best regards,
> > > Alexei Scherbakov
> > >
> >
> >
> > --
> > Sincerely yours, Ivan Daschinskiy
> >
>
--
Sincerely yours, Ivan Daschinskiy
Re: IEP-52: Binary Delivery & Upgradability Enhancements
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
The situation as follows:
libs/optional/ignite-kubernetes:
ignite-kubernetes-2.8.1.jar jackson-annotations-2.9.10.jar
jackson-core-2.9.10.jar jackson-databind-2.9.10.jar
libs/optional/ignite-zookeeper:
curator-client-4.2.0.jar curator-x-discovery-4.2.0.jar
jackson-core-asl-1.9.13.jar log4j-1.2.17.jar
slf4j-log4j12-1.7.7.jar
curator-framework-4.2.0.jar guava-16.0.1.jar
jackson-mapper-asl-1.9.13.jar zookeeper-3.5.5.jar
curator-recipes-4.2.0.jar ignite-zookeeper-2.8.1.jar slf4j-api-1.7.7.jar
zookeeper-jute-3.5.5.jar
ignite-zookeeper just has way more dependencies than ignite-kubernetes, and
their size is 5-fold.
These dependencies are not just weight but also vulnerability surface.
Regards,
--
Ilya Kasnacheev
чт, 17 сент. 2020 г. в 13:55, Ivan Daschinsky <iv...@gmail.com>:
> Hi! I recently found that in slim binary release (at least in nightly build
> of 2.9.0) there is no ignite-zookeeper module.
> But, for example, ignite-kubernetes is in.
> Petr, could you please clarify why this decision was made?
> We currently have only two discovery implementation in project and leave
> only one alternative looks not obvious for me.
>
> ср, 16 сент. 2020 г. в 17:04, Alexei Scherbakov <
> alexey.scherbakoff@gmail.com>:
>
> > Downloading additional jars will not 100% work in private networks, so we
> > must provide full distro anyway.
> >
> > Also, I do not see any problems in downloading all dependencies, because
> > their summary size is not too big.
> >
> > I suggest, in addition to tgz archive for manual deployment, provide
> Ignite
> > distribution as RPM (or other package manager(s)) for Linux (MSI for
> > Windows) to automate distribution upgrades, instead of reinventing the
> > wheel.
> >
> > пн, 31 авг. 2020 г. в 10:09, Valentin Kulichenko <
> > valentin.kulichenko@gmail.com>:
> >
> > > Hi Nikolay,
> > >
> > > Thanks for your feedback! Absolutely, the IEP is currently in the draft
> > > mode - we will be adding more details going forward. Also, Petr
> mentioned
> > > that he wants to create a prototype which I think will make things
> > clearer
> > > for all of us.
> > >
> > > As for "download jars from the Internet on a production server", that's
> > not
> > > necessarily what we propose. In case of Maven, artifacts are downloaded
> > > from a specified repository, which doesn't have to be on the Internet.
> > > Companies that have security concerns typically have a mirror/proxy
> > > containing approved artifacts only, already checked for viruses, etc.
> > > Production servers use the proxy rather than public repositories.
> > >
> > > The concerns that you brought up are certainly valid. However, I
> believe
> > > that using Maven actually addresses them, because Maven (as any other
> > > package/dependency manager for that matter) provides the corresponding
> > > functionality out of the box.
> > >
> > > On the contrary, users who currently download our ZIP package are
> > *forced*
> > > to download modules and dependencies that they do not even need and
> will
> > > not use. In my experience, this is actually a bigger source of concern.
> > >
> > > Anyway, let us add more details to the IEP, and we will go from there.
> > >
> > > P.S. I'm also not sure I agree with your assessment of the embedded
> mode
> > > and thick clients, but that's probably a different discussion. Please
> > feel
> > > free to create a new thread if you would like to discuss this.
> > >
> > > -Val
> > >
> > > On Sat, Aug 29, 2020 at 1:46 AM Nikolay Izhikov <ni...@apache.org>
> > > wrote:
> > >
> > > > Also, guys.
> > > >
> > > > Can we make one step back and add some requirements to the IEP.
> > > > How we want to install and upgrade Ignite?
> > > > What specific issues we want to solve?
> > > >
> > > > With this input we can be on the same page during solution
> discussion.
> > > >
> > > > > 29 авг. 2020 г., в 11:41, Nikolay Izhikov <NI...@gmail.com>
> > > > написал(а):
> > > > >
> > > > > Hello, Val.
> > > > >
> > > > > Sorry, but I sill don’t understand how «download jars from the
> > internet
> > > > on production server» approach helps us to solve the issues you
> > describe.
> > > > >
> > > > > Moreover, with this enhancement you want to add more dependencies
> to
> > > > Ignite and create some security vulnerabilities:
> > > > > * Maven tool - new dependency. I doubt that and of the
> > production
> > > > server have it
> > > > > * Internet maven repository - many production servers
> restrict
> > > > outgoing internet connection to any addresses.
> > > > > It’s hard to understand why distributed DB should
> have
> > > the
> > > > ability to connect to some kind of address on the Internet.
> > > > > * All Ignite distributions can be compromised with the hack
> of
> > > > some random maven repo or with the malicious proxy.
> > > > > * Note, that Ignite installation or upgrade procedure can
> > become
> > > > dead just because of some Maven repo down.
> > > > >
> > > > >
> > > > > I have 2 concerns:
> > > > >
> > > > > 1. We shouldn’t download any jars from the internet during
> production
> > > > deployment or upgrade procedure.
> > > > > 2. The IEP description, for now, is quite small for such important
> > > > things as a way we distribute and upgrade Ignite.
> > > > > Petr, can you, please, make IEP more specific.
> > > > > I think we should add the following details:
> > > > > * Description of the typical Ignite deployment
> > procedure
> > > > including folder structure(config, persistence, .m2 and other
> folders)
> > > > > * This also should include examples of the
> bash
> > > > commands if any required.
> > > > > * Description of the typical upgrade(downgrade?)
> > > procedure
> > > > > * This also should include examples of the
> bash
> > > > commands if any required.
> > > > > * Description of «single tool responsible for all
> > > > operations».
> > > > > * all commands
> > > > > * all operations
> > > > > * all parameters
> > > > >
> > > > >> 1. Most paths are resolved relative to IGNITE_HOME, which
> > effectively
> > > > forces (or at least motivates) users to put everything related to
> > Ignite
> > > > under IGNITE_HOME (configs, additional JARs, etc.).
> > > > >
> > > > > AFAIU this issue doesn’t relate to the way Ignite distributed.
> > > > > We can and should fix this outside of this IEP.
> > > > >
> > > > >> 2. Modules are enabled/disabled manually by moving folders around…
> > > this
> > > > is error-prone,
> > > > >
> > > > > Makes sense.
> > > > > Let’s fix this without «download jar from the internet»?
> > > > > We can have some kind of config with the enabled modules.
> > > > >
> > > > >> 3. Standalone nodes use JARs prepackaged in the ZIP file. However,
> > if
> > > > you look at how this ZIP is built, you will notice that we don't
> > actually
> > > > include all transitive dependencies,
> > > > >
> > > > > We need to fix this.
> > > > > Let’s include all the required JARs to the distribution.
> > > > >
> > > > >> which means that an embedded node that uses Maven can have a
> > different
> > > > classpath.
> > > > >
> > > > > Embedded Ignite node is an error-prone design decision in the first
> > > > place.
> > > > > We can’t fix it with the «download jars from the internet»
> approach.
> > > > > We should eliminate the client node as a thing and use a thin
> clients
> > > > instead of it.
> > > > >
> > > > >> 4. ignite.sh allows manual modifications (there are several
> > "Uncomment
> > > > this to..." lines). This complicates upgrades as well.
> > > > >
> > > > > Makes sense.
> > > > > Let’s fix this.
> > > > >
> > > > >> The whole purpose of the IEP is to fix the above, automate and
> > > simplify
> > > > operations for the users. The general idea is to create a single tool
> > > > responsible for all those operations and use Maven under the hood to
> > > > include/exclude modules.
> > > > >
> > > > > For now, IEP lacks the description of this tool.
> > > > >
> > > > >> 28 авг. 2020 г., в 22:35, Valentin Kulichenko <
> > > > valentin.kulichenko@gmail.com> написал(а):
> > > > >>
> > > > >> Hi Nikolay,
> > > > >>
> > > > >> Here are some of the issues that I see with the current binary ZIP
> > > > package
> > > > >> that we create for every release.
> > > > >>
> > > > >> 1. Most paths are resolved relative to IGNITE_HOME, which
> > effectively
> > > > >> forces (or at least motivates) users to put everything related to
> > > Ignite
> > > > >> under IGNITE_HOME (configs, additional JARs, etc.). Now, when a
> user
> > > > >> downloads and unzips a new version, they have to somehow merge the
> > > > content
> > > > >> together - that's poor upgradability.
> > > > >> 2. Modules are enabled/disabled manually by moving folders around.
> > Not
> > > > only
> > > > >> this is error-prone, but it also adds more issues to
> upgradability -
> > > you
> > > > >> have to apply these steps for every version you download.
> > > > >> 3. Standalone nodes use JARs prepackaged in the ZIP file. However,
> > if
> > > > you
> > > > >> look at how this ZIP is built, you will notice that we don't
> > actually
> > > > >> include all transitive dependencies, which means that an embedded
> > node
> > > > that
> > > > >> uses Maven can have a different classpath. It seems that we've
> been
> > > > lucky
> > > > >> so far as this hasn't caused any issues, but I still think the
> > > approach
> > > > is
> > > > >> wrong. The behavior should be consistent.
> > > > >> 4. ignite.sh allows manual modifications (there are several
> > "Uncomment
> > > > this
> > > > >> to..." lines). This complicates upgrades as well.
> > > > >>
> > > > >> The whole purpose of the IEP is to fix the above, automate and
> > > > >> simplify operations for the users. The general idea is to create a
> > > > single
> > > > >> tool responsible for all those operations and use Maven under the
> > hood
> > > > to
> > > > >> include/exclude modules.
> > > > >>
> > > > >> Exact commands and steps are still under discussion though. It
> looks
> > > > like
> > > > >> Petr is going to create a prototype that can be used as a starting
> > > > point,
> > > > >> but if you have any suggestions or ideas in the meantime, please
> > share
> > > > them
> > > > >> with us.
> > > > >>
> > > > >> -Val
> > > > >>
> > > > >> On Fri, Aug 28, 2020 at 12:31 AM Nikolay Izhikov <
> > nizhikov@apache.org
> > > >
> > > > >> wrote:
> > > > >>
> > > > >>> Hello, Igniters.
> > > > >>>
> > > > >>> Not sure if I understand the essence of the proposal at the
> moment.
> > > > >>> Can you, please, describe the advantages of the proposed way from
> > the
> > > > user
> > > > >>> perspective?
> > > > >>>
> > > > >>> How the typical DevOps pipeline should look like with this
> > > enhancement?
> > > > >>>
> > > > >>> How I as a user can create a fully functional installation
> package
> > of
> > > > the
> > > > >>> Ignite?
> > > > >>> AFAIK downloading some artifacts from the internet straight to
> the
> > > > >>> production server is a security anti-pattern.
> > > > >>>
> > > > >>>
> > > > >>>> 28 авг. 2020 г., в 01:59, Denis Magda <dm...@apache.org>
> > > написал(а):
> > > > >>>>
> > > > >>>> Petr, thanks,
> > > > >>>>
> > > > >>>> There is also a collection of modules located in our extensions
> > > > >>> repository:
> > > > >>>> https://github.com/apache/ignite-extensions
> > > > >>>>
> > > > >>>> @Saikat Maitra <sa...@gmail.com> is migrating all our
> > > > existing
> > > > >>>> integrations to that repository and, once this is done, the
> > > extensions
> > > > >>> will
> > > > >>>> be released to Maven separately. Is it reasonable to expand the
> > > scope
> > > > of
> > > > >>>> the IEP-52 and contemplate on how to install those extensions?
> > > > >>>>
> > > > >>>> -
> > > > >>>> Denis
> > > > >>>>
> > > > >>>>
> > > > >>>> On Thu, Aug 27, 2020 at 3:40 PM Valentin Kulichenko <
> > > > >>>> valentin.kulichenko@gmail.com> wrote:
> > > > >>>>
> > > > >>>>> Hi Petr,
> > > > >>>>>
> > > > >>>>> The proposal makes sense to me - thanks for starting the
> > > discussion.
> > > > >>>>> Current installation and upgrade procedures involve a lot of
> > manual
> > > > >>> steps
> > > > >>>>> and quite error-prone, we need to automate and simplify them as
> > > much
> > > > as
> > > > >>>>> possible.
> > > > >>>>>
> > > > >>>>> I believe we eventually should end up with a unified
> command-line
> > > > tool
> > > > >>>>> (ignitectl?) that will incorporate all the operations
> > > (enable/disable
> > > > >>>>> modules, start/stop, configuration updates, activation,
> baseline,
> > > > etc.).
> > > > >>>>> This IEP is a step in this direction.
> > > > >>>>>
> > > > >>>>> Looking forward to testing a prototype :)
> > > > >>>>>
> > > > >>>>> -Val
> > > > >>>>>
> > > > >>>>> On Thu, Aug 27, 2020 at 2:17 AM Petr Ivanov <
> mr.weider@gmail.com
> > >
> > > > >>> wrote:
> > > > >>>>>
> > > > >>>>>> Hi, Igniters!
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>> For Apache Ignite 3.0 I would like to propose vision of
> enhanced
> > > > >>> delivery
> > > > >>>>>> and upgrade processes [1].
> > > > >>>>>> The key idea is to make main binary as slim as possible
> > > (delivering
> > > > >>> every
> > > > >>>>>> optional component by demand only) while providing way to run
> > each
> > > > new
> > > > >>>>>> version seamlessly without much of the efforts migrating data
> > and
> > > > >>>>>> configuration between upgrades.
> > > > >>>>>>
> > > > >>>>>> I plan to create prototype based on current release of Apache
> > > Ignite
> > > > >>>>>> (2.8.1 or 2.9.0 if it will be released soon) later in
> September.
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>> [1]
> > > > >>>>>>
> > > > >>>>>
> > > > >>>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158873958
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>
> > > > >>>
> > > > >>>
> > > > >
> > > >
> > > >
> > >
> >
> >
> > --
> >
> > Best regards,
> > Alexei Scherbakov
> >
>
>
> --
> Sincerely yours, Ivan Daschinskiy
>
Re: IEP-52: Binary Delivery & Upgradability Enhancements
Posted by Ivan Daschinsky <iv...@gmail.com>.
Hi! I recently found that in slim binary release (at least in nightly build
of 2.9.0) there is no ignite-zookeeper module.
But, for example, ignite-kubernetes is in.
Petr, could you please clarify why this decision was made?
We currently have only two discovery implementation in project and leave
only one alternative looks not obvious for me.
ср, 16 сент. 2020 г. в 17:04, Alexei Scherbakov <
alexey.scherbakoff@gmail.com>:
> Downloading additional jars will not 100% work in private networks, so we
> must provide full distro anyway.
>
> Also, I do not see any problems in downloading all dependencies, because
> their summary size is not too big.
>
> I suggest, in addition to tgz archive for manual deployment, provide Ignite
> distribution as RPM (or other package manager(s)) for Linux (MSI for
> Windows) to automate distribution upgrades, instead of reinventing the
> wheel.
>
> пн, 31 авг. 2020 г. в 10:09, Valentin Kulichenko <
> valentin.kulichenko@gmail.com>:
>
> > Hi Nikolay,
> >
> > Thanks for your feedback! Absolutely, the IEP is currently in the draft
> > mode - we will be adding more details going forward. Also, Petr mentioned
> > that he wants to create a prototype which I think will make things
> clearer
> > for all of us.
> >
> > As for "download jars from the Internet on a production server", that's
> not
> > necessarily what we propose. In case of Maven, artifacts are downloaded
> > from a specified repository, which doesn't have to be on the Internet.
> > Companies that have security concerns typically have a mirror/proxy
> > containing approved artifacts only, already checked for viruses, etc.
> > Production servers use the proxy rather than public repositories.
> >
> > The concerns that you brought up are certainly valid. However, I believe
> > that using Maven actually addresses them, because Maven (as any other
> > package/dependency manager for that matter) provides the corresponding
> > functionality out of the box.
> >
> > On the contrary, users who currently download our ZIP package are
> *forced*
> > to download modules and dependencies that they do not even need and will
> > not use. In my experience, this is actually a bigger source of concern.
> >
> > Anyway, let us add more details to the IEP, and we will go from there.
> >
> > P.S. I'm also not sure I agree with your assessment of the embedded mode
> > and thick clients, but that's probably a different discussion. Please
> feel
> > free to create a new thread if you would like to discuss this.
> >
> > -Val
> >
> > On Sat, Aug 29, 2020 at 1:46 AM Nikolay Izhikov <ni...@apache.org>
> > wrote:
> >
> > > Also, guys.
> > >
> > > Can we make one step back and add some requirements to the IEP.
> > > How we want to install and upgrade Ignite?
> > > What specific issues we want to solve?
> > >
> > > With this input we can be on the same page during solution discussion.
> > >
> > > > 29 авг. 2020 г., в 11:41, Nikolay Izhikov <NI...@gmail.com>
> > > написал(а):
> > > >
> > > > Hello, Val.
> > > >
> > > > Sorry, but I sill don’t understand how «download jars from the
> internet
> > > on production server» approach helps us to solve the issues you
> describe.
> > > >
> > > > Moreover, with this enhancement you want to add more dependencies to
> > > Ignite and create some security vulnerabilities:
> > > > * Maven tool - new dependency. I doubt that and of the
> production
> > > server have it
> > > > * Internet maven repository - many production servers restrict
> > > outgoing internet connection to any addresses.
> > > > It’s hard to understand why distributed DB should have
> > the
> > > ability to connect to some kind of address on the Internet.
> > > > * All Ignite distributions can be compromised with the hack of
> > > some random maven repo or with the malicious proxy.
> > > > * Note, that Ignite installation or upgrade procedure can
> become
> > > dead just because of some Maven repo down.
> > > >
> > > >
> > > > I have 2 concerns:
> > > >
> > > > 1. We shouldn’t download any jars from the internet during production
> > > deployment or upgrade procedure.
> > > > 2. The IEP description, for now, is quite small for such important
> > > things as a way we distribute and upgrade Ignite.
> > > > Petr, can you, please, make IEP more specific.
> > > > I think we should add the following details:
> > > > * Description of the typical Ignite deployment
> procedure
> > > including folder structure(config, persistence, .m2 and other folders)
> > > > * This also should include examples of the bash
> > > commands if any required.
> > > > * Description of the typical upgrade(downgrade?)
> > procedure
> > > > * This also should include examples of the bash
> > > commands if any required.
> > > > * Description of «single tool responsible for all
> > > operations».
> > > > * all commands
> > > > * all operations
> > > > * all parameters
> > > >
> > > >> 1. Most paths are resolved relative to IGNITE_HOME, which
> effectively
> > > forces (or at least motivates) users to put everything related to
> Ignite
> > > under IGNITE_HOME (configs, additional JARs, etc.).
> > > >
> > > > AFAIU this issue doesn’t relate to the way Ignite distributed.
> > > > We can and should fix this outside of this IEP.
> > > >
> > > >> 2. Modules are enabled/disabled manually by moving folders around…
> > this
> > > is error-prone,
> > > >
> > > > Makes sense.
> > > > Let’s fix this without «download jar from the internet»?
> > > > We can have some kind of config with the enabled modules.
> > > >
> > > >> 3. Standalone nodes use JARs prepackaged in the ZIP file. However,
> if
> > > you look at how this ZIP is built, you will notice that we don't
> actually
> > > include all transitive dependencies,
> > > >
> > > > We need to fix this.
> > > > Let’s include all the required JARs to the distribution.
> > > >
> > > >> which means that an embedded node that uses Maven can have a
> different
> > > classpath.
> > > >
> > > > Embedded Ignite node is an error-prone design decision in the first
> > > place.
> > > > We can’t fix it with the «download jars from the internet» approach.
> > > > We should eliminate the client node as a thing and use a thin clients
> > > instead of it.
> > > >
> > > >> 4. ignite.sh allows manual modifications (there are several
> "Uncomment
> > > this to..." lines). This complicates upgrades as well.
> > > >
> > > > Makes sense.
> > > > Let’s fix this.
> > > >
> > > >> The whole purpose of the IEP is to fix the above, automate and
> > simplify
> > > operations for the users. The general idea is to create a single tool
> > > responsible for all those operations and use Maven under the hood to
> > > include/exclude modules.
> > > >
> > > > For now, IEP lacks the description of this tool.
> > > >
> > > >> 28 авг. 2020 г., в 22:35, Valentin Kulichenko <
> > > valentin.kulichenko@gmail.com> написал(а):
> > > >>
> > > >> Hi Nikolay,
> > > >>
> > > >> Here are some of the issues that I see with the current binary ZIP
> > > package
> > > >> that we create for every release.
> > > >>
> > > >> 1. Most paths are resolved relative to IGNITE_HOME, which
> effectively
> > > >> forces (or at least motivates) users to put everything related to
> > Ignite
> > > >> under IGNITE_HOME (configs, additional JARs, etc.). Now, when a user
> > > >> downloads and unzips a new version, they have to somehow merge the
> > > content
> > > >> together - that's poor upgradability.
> > > >> 2. Modules are enabled/disabled manually by moving folders around.
> Not
> > > only
> > > >> this is error-prone, but it also adds more issues to upgradability -
> > you
> > > >> have to apply these steps for every version you download.
> > > >> 3. Standalone nodes use JARs prepackaged in the ZIP file. However,
> if
> > > you
> > > >> look at how this ZIP is built, you will notice that we don't
> actually
> > > >> include all transitive dependencies, which means that an embedded
> node
> > > that
> > > >> uses Maven can have a different classpath. It seems that we've been
> > > lucky
> > > >> so far as this hasn't caused any issues, but I still think the
> > approach
> > > is
> > > >> wrong. The behavior should be consistent.
> > > >> 4. ignite.sh allows manual modifications (there are several
> "Uncomment
> > > this
> > > >> to..." lines). This complicates upgrades as well.
> > > >>
> > > >> The whole purpose of the IEP is to fix the above, automate and
> > > >> simplify operations for the users. The general idea is to create a
> > > single
> > > >> tool responsible for all those operations and use Maven under the
> hood
> > > to
> > > >> include/exclude modules.
> > > >>
> > > >> Exact commands and steps are still under discussion though. It looks
> > > like
> > > >> Petr is going to create a prototype that can be used as a starting
> > > point,
> > > >> but if you have any suggestions or ideas in the meantime, please
> share
> > > them
> > > >> with us.
> > > >>
> > > >> -Val
> > > >>
> > > >> On Fri, Aug 28, 2020 at 12:31 AM Nikolay Izhikov <
> nizhikov@apache.org
> > >
> > > >> wrote:
> > > >>
> > > >>> Hello, Igniters.
> > > >>>
> > > >>> Not sure if I understand the essence of the proposal at the moment.
> > > >>> Can you, please, describe the advantages of the proposed way from
> the
> > > user
> > > >>> perspective?
> > > >>>
> > > >>> How the typical DevOps pipeline should look like with this
> > enhancement?
> > > >>>
> > > >>> How I as a user can create a fully functional installation package
> of
> > > the
> > > >>> Ignite?
> > > >>> AFAIK downloading some artifacts from the internet straight to the
> > > >>> production server is a security anti-pattern.
> > > >>>
> > > >>>
> > > >>>> 28 авг. 2020 г., в 01:59, Denis Magda <dm...@apache.org>
> > написал(а):
> > > >>>>
> > > >>>> Petr, thanks,
> > > >>>>
> > > >>>> There is also a collection of modules located in our extensions
> > > >>> repository:
> > > >>>> https://github.com/apache/ignite-extensions
> > > >>>>
> > > >>>> @Saikat Maitra <sa...@gmail.com> is migrating all our
> > > existing
> > > >>>> integrations to that repository and, once this is done, the
> > extensions
> > > >>> will
> > > >>>> be released to Maven separately. Is it reasonable to expand the
> > scope
> > > of
> > > >>>> the IEP-52 and contemplate on how to install those extensions?
> > > >>>>
> > > >>>> -
> > > >>>> Denis
> > > >>>>
> > > >>>>
> > > >>>> On Thu, Aug 27, 2020 at 3:40 PM Valentin Kulichenko <
> > > >>>> valentin.kulichenko@gmail.com> wrote:
> > > >>>>
> > > >>>>> Hi Petr,
> > > >>>>>
> > > >>>>> The proposal makes sense to me - thanks for starting the
> > discussion.
> > > >>>>> Current installation and upgrade procedures involve a lot of
> manual
> > > >>> steps
> > > >>>>> and quite error-prone, we need to automate and simplify them as
> > much
> > > as
> > > >>>>> possible.
> > > >>>>>
> > > >>>>> I believe we eventually should end up with a unified command-line
> > > tool
> > > >>>>> (ignitectl?) that will incorporate all the operations
> > (enable/disable
> > > >>>>> modules, start/stop, configuration updates, activation, baseline,
> > > etc.).
> > > >>>>> This IEP is a step in this direction.
> > > >>>>>
> > > >>>>> Looking forward to testing a prototype :)
> > > >>>>>
> > > >>>>> -Val
> > > >>>>>
> > > >>>>> On Thu, Aug 27, 2020 at 2:17 AM Petr Ivanov <mr.weider@gmail.com
> >
> > > >>> wrote:
> > > >>>>>
> > > >>>>>> Hi, Igniters!
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> For Apache Ignite 3.0 I would like to propose vision of enhanced
> > > >>> delivery
> > > >>>>>> and upgrade processes [1].
> > > >>>>>> The key idea is to make main binary as slim as possible
> > (delivering
> > > >>> every
> > > >>>>>> optional component by demand only) while providing way to run
> each
> > > new
> > > >>>>>> version seamlessly without much of the efforts migrating data
> and
> > > >>>>>> configuration between upgrades.
> > > >>>>>>
> > > >>>>>> I plan to create prototype based on current release of Apache
> > Ignite
> > > >>>>>> (2.8.1 or 2.9.0 if it will be released soon) later in September.
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> [1]
> > > >>>>>>
> > > >>>>>
> > > >>>
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158873958
> > > >>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>
> > > >>>
> > > >
> > >
> > >
> >
>
>
> --
>
> Best regards,
> Alexei Scherbakov
>
--
Sincerely yours, Ivan Daschinskiy
Re: IEP-52: Binary Delivery & Upgradability Enhancements
Posted by Alexei Scherbakov <al...@gmail.com>.
Downloading additional jars will not 100% work in private networks, so we
must provide full distro anyway.
Also, I do not see any problems in downloading all dependencies, because
their summary size is not too big.
I suggest, in addition to tgz archive for manual deployment, provide Ignite
distribution as RPM (or other package manager(s)) for Linux (MSI for
Windows) to automate distribution upgrades, instead of reinventing the
wheel.
пн, 31 авг. 2020 г. в 10:09, Valentin Kulichenko <
valentin.kulichenko@gmail.com>:
> Hi Nikolay,
>
> Thanks for your feedback! Absolutely, the IEP is currently in the draft
> mode - we will be adding more details going forward. Also, Petr mentioned
> that he wants to create a prototype which I think will make things clearer
> for all of us.
>
> As for "download jars from the Internet on a production server", that's not
> necessarily what we propose. In case of Maven, artifacts are downloaded
> from a specified repository, which doesn't have to be on the Internet.
> Companies that have security concerns typically have a mirror/proxy
> containing approved artifacts only, already checked for viruses, etc.
> Production servers use the proxy rather than public repositories.
>
> The concerns that you brought up are certainly valid. However, I believe
> that using Maven actually addresses them, because Maven (as any other
> package/dependency manager for that matter) provides the corresponding
> functionality out of the box.
>
> On the contrary, users who currently download our ZIP package are *forced*
> to download modules and dependencies that they do not even need and will
> not use. In my experience, this is actually a bigger source of concern.
>
> Anyway, let us add more details to the IEP, and we will go from there.
>
> P.S. I'm also not sure I agree with your assessment of the embedded mode
> and thick clients, but that's probably a different discussion. Please feel
> free to create a new thread if you would like to discuss this.
>
> -Val
>
> On Sat, Aug 29, 2020 at 1:46 AM Nikolay Izhikov <ni...@apache.org>
> wrote:
>
> > Also, guys.
> >
> > Can we make one step back and add some requirements to the IEP.
> > How we want to install and upgrade Ignite?
> > What specific issues we want to solve?
> >
> > With this input we can be on the same page during solution discussion.
> >
> > > 29 авг. 2020 г., в 11:41, Nikolay Izhikov <NI...@gmail.com>
> > написал(а):
> > >
> > > Hello, Val.
> > >
> > > Sorry, but I sill don’t understand how «download jars from the internet
> > on production server» approach helps us to solve the issues you describe.
> > >
> > > Moreover, with this enhancement you want to add more dependencies to
> > Ignite and create some security vulnerabilities:
> > > * Maven tool - new dependency. I doubt that and of the production
> > server have it
> > > * Internet maven repository - many production servers restrict
> > outgoing internet connection to any addresses.
> > > It’s hard to understand why distributed DB should have
> the
> > ability to connect to some kind of address on the Internet.
> > > * All Ignite distributions can be compromised with the hack of
> > some random maven repo or with the malicious proxy.
> > > * Note, that Ignite installation or upgrade procedure can become
> > dead just because of some Maven repo down.
> > >
> > >
> > > I have 2 concerns:
> > >
> > > 1. We shouldn’t download any jars from the internet during production
> > deployment or upgrade procedure.
> > > 2. The IEP description, for now, is quite small for such important
> > things as a way we distribute and upgrade Ignite.
> > > Petr, can you, please, make IEP more specific.
> > > I think we should add the following details:
> > > * Description of the typical Ignite deployment procedure
> > including folder structure(config, persistence, .m2 and other folders)
> > > * This also should include examples of the bash
> > commands if any required.
> > > * Description of the typical upgrade(downgrade?)
> procedure
> > > * This also should include examples of the bash
> > commands if any required.
> > > * Description of «single tool responsible for all
> > operations».
> > > * all commands
> > > * all operations
> > > * all parameters
> > >
> > >> 1. Most paths are resolved relative to IGNITE_HOME, which effectively
> > forces (or at least motivates) users to put everything related to Ignite
> > under IGNITE_HOME (configs, additional JARs, etc.).
> > >
> > > AFAIU this issue doesn’t relate to the way Ignite distributed.
> > > We can and should fix this outside of this IEP.
> > >
> > >> 2. Modules are enabled/disabled manually by moving folders around…
> this
> > is error-prone,
> > >
> > > Makes sense.
> > > Let’s fix this without «download jar from the internet»?
> > > We can have some kind of config with the enabled modules.
> > >
> > >> 3. Standalone nodes use JARs prepackaged in the ZIP file. However, if
> > you look at how this ZIP is built, you will notice that we don't actually
> > include all transitive dependencies,
> > >
> > > We need to fix this.
> > > Let’s include all the required JARs to the distribution.
> > >
> > >> which means that an embedded node that uses Maven can have a different
> > classpath.
> > >
> > > Embedded Ignite node is an error-prone design decision in the first
> > place.
> > > We can’t fix it with the «download jars from the internet» approach.
> > > We should eliminate the client node as a thing and use a thin clients
> > instead of it.
> > >
> > >> 4. ignite.sh allows manual modifications (there are several "Uncomment
> > this to..." lines). This complicates upgrades as well.
> > >
> > > Makes sense.
> > > Let’s fix this.
> > >
> > >> The whole purpose of the IEP is to fix the above, automate and
> simplify
> > operations for the users. The general idea is to create a single tool
> > responsible for all those operations and use Maven under the hood to
> > include/exclude modules.
> > >
> > > For now, IEP lacks the description of this tool.
> > >
> > >> 28 авг. 2020 г., в 22:35, Valentin Kulichenko <
> > valentin.kulichenko@gmail.com> написал(а):
> > >>
> > >> Hi Nikolay,
> > >>
> > >> Here are some of the issues that I see with the current binary ZIP
> > package
> > >> that we create for every release.
> > >>
> > >> 1. Most paths are resolved relative to IGNITE_HOME, which effectively
> > >> forces (or at least motivates) users to put everything related to
> Ignite
> > >> under IGNITE_HOME (configs, additional JARs, etc.). Now, when a user
> > >> downloads and unzips a new version, they have to somehow merge the
> > content
> > >> together - that's poor upgradability.
> > >> 2. Modules are enabled/disabled manually by moving folders around. Not
> > only
> > >> this is error-prone, but it also adds more issues to upgradability -
> you
> > >> have to apply these steps for every version you download.
> > >> 3. Standalone nodes use JARs prepackaged in the ZIP file. However, if
> > you
> > >> look at how this ZIP is built, you will notice that we don't actually
> > >> include all transitive dependencies, which means that an embedded node
> > that
> > >> uses Maven can have a different classpath. It seems that we've been
> > lucky
> > >> so far as this hasn't caused any issues, but I still think the
> approach
> > is
> > >> wrong. The behavior should be consistent.
> > >> 4. ignite.sh allows manual modifications (there are several "Uncomment
> > this
> > >> to..." lines). This complicates upgrades as well.
> > >>
> > >> The whole purpose of the IEP is to fix the above, automate and
> > >> simplify operations for the users. The general idea is to create a
> > single
> > >> tool responsible for all those operations and use Maven under the hood
> > to
> > >> include/exclude modules.
> > >>
> > >> Exact commands and steps are still under discussion though. It looks
> > like
> > >> Petr is going to create a prototype that can be used as a starting
> > point,
> > >> but if you have any suggestions or ideas in the meantime, please share
> > them
> > >> with us.
> > >>
> > >> -Val
> > >>
> > >> On Fri, Aug 28, 2020 at 12:31 AM Nikolay Izhikov <nizhikov@apache.org
> >
> > >> wrote:
> > >>
> > >>> Hello, Igniters.
> > >>>
> > >>> Not sure if I understand the essence of the proposal at the moment.
> > >>> Can you, please, describe the advantages of the proposed way from the
> > user
> > >>> perspective?
> > >>>
> > >>> How the typical DevOps pipeline should look like with this
> enhancement?
> > >>>
> > >>> How I as a user can create a fully functional installation package of
> > the
> > >>> Ignite?
> > >>> AFAIK downloading some artifacts from the internet straight to the
> > >>> production server is a security anti-pattern.
> > >>>
> > >>>
> > >>>> 28 авг. 2020 г., в 01:59, Denis Magda <dm...@apache.org>
> написал(а):
> > >>>>
> > >>>> Petr, thanks,
> > >>>>
> > >>>> There is also a collection of modules located in our extensions
> > >>> repository:
> > >>>> https://github.com/apache/ignite-extensions
> > >>>>
> > >>>> @Saikat Maitra <sa...@gmail.com> is migrating all our
> > existing
> > >>>> integrations to that repository and, once this is done, the
> extensions
> > >>> will
> > >>>> be released to Maven separately. Is it reasonable to expand the
> scope
> > of
> > >>>> the IEP-52 and contemplate on how to install those extensions?
> > >>>>
> > >>>> -
> > >>>> Denis
> > >>>>
> > >>>>
> > >>>> On Thu, Aug 27, 2020 at 3:40 PM Valentin Kulichenko <
> > >>>> valentin.kulichenko@gmail.com> wrote:
> > >>>>
> > >>>>> Hi Petr,
> > >>>>>
> > >>>>> The proposal makes sense to me - thanks for starting the
> discussion.
> > >>>>> Current installation and upgrade procedures involve a lot of manual
> > >>> steps
> > >>>>> and quite error-prone, we need to automate and simplify them as
> much
> > as
> > >>>>> possible.
> > >>>>>
> > >>>>> I believe we eventually should end up with a unified command-line
> > tool
> > >>>>> (ignitectl?) that will incorporate all the operations
> (enable/disable
> > >>>>> modules, start/stop, configuration updates, activation, baseline,
> > etc.).
> > >>>>> This IEP is a step in this direction.
> > >>>>>
> > >>>>> Looking forward to testing a prototype :)
> > >>>>>
> > >>>>> -Val
> > >>>>>
> > >>>>> On Thu, Aug 27, 2020 at 2:17 AM Petr Ivanov <mr...@gmail.com>
> > >>> wrote:
> > >>>>>
> > >>>>>> Hi, Igniters!
> > >>>>>>
> > >>>>>>
> > >>>>>> For Apache Ignite 3.0 I would like to propose vision of enhanced
> > >>> delivery
> > >>>>>> and upgrade processes [1].
> > >>>>>> The key idea is to make main binary as slim as possible
> (delivering
> > >>> every
> > >>>>>> optional component by demand only) while providing way to run each
> > new
> > >>>>>> version seamlessly without much of the efforts migrating data and
> > >>>>>> configuration between upgrades.
> > >>>>>>
> > >>>>>> I plan to create prototype based on current release of Apache
> Ignite
> > >>>>>> (2.8.1 or 2.9.0 if it will be released soon) later in September.
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> [1]
> > >>>>>>
> > >>>>>
> > >>>
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158873958
> > >>>>>>
> > >>>>>>
> > >>>>>
> > >>>
> > >>>
> > >
> >
> >
>
--
Best regards,
Alexei Scherbakov
Re: IEP-52: Binary Delivery & Upgradability Enhancements
Posted by Valentin Kulichenko <va...@gmail.com>.
Hi Nikolay,
Thanks for your feedback! Absolutely, the IEP is currently in the draft
mode - we will be adding more details going forward. Also, Petr mentioned
that he wants to create a prototype which I think will make things clearer
for all of us.
As for "download jars from the Internet on a production server", that's not
necessarily what we propose. In case of Maven, artifacts are downloaded
from a specified repository, which doesn't have to be on the Internet.
Companies that have security concerns typically have a mirror/proxy
containing approved artifacts only, already checked for viruses, etc.
Production servers use the proxy rather than public repositories.
The concerns that you brought up are certainly valid. However, I believe
that using Maven actually addresses them, because Maven (as any other
package/dependency manager for that matter) provides the corresponding
functionality out of the box.
On the contrary, users who currently download our ZIP package are *forced*
to download modules and dependencies that they do not even need and will
not use. In my experience, this is actually a bigger source of concern.
Anyway, let us add more details to the IEP, and we will go from there.
P.S. I'm also not sure I agree with your assessment of the embedded mode
and thick clients, but that's probably a different discussion. Please feel
free to create a new thread if you would like to discuss this.
-Val
On Sat, Aug 29, 2020 at 1:46 AM Nikolay Izhikov <ni...@apache.org> wrote:
> Also, guys.
>
> Can we make one step back and add some requirements to the IEP.
> How we want to install and upgrade Ignite?
> What specific issues we want to solve?
>
> With this input we can be on the same page during solution discussion.
>
> > 29 авг. 2020 г., в 11:41, Nikolay Izhikov <NI...@gmail.com>
> написал(а):
> >
> > Hello, Val.
> >
> > Sorry, but I sill don’t understand how «download jars from the internet
> on production server» approach helps us to solve the issues you describe.
> >
> > Moreover, with this enhancement you want to add more dependencies to
> Ignite and create some security vulnerabilities:
> > * Maven tool - new dependency. I doubt that and of the production
> server have it
> > * Internet maven repository - many production servers restrict
> outgoing internet connection to any addresses.
> > It’s hard to understand why distributed DB should have the
> ability to connect to some kind of address on the Internet.
> > * All Ignite distributions can be compromised with the hack of
> some random maven repo or with the malicious proxy.
> > * Note, that Ignite installation or upgrade procedure can become
> dead just because of some Maven repo down.
> >
> >
> > I have 2 concerns:
> >
> > 1. We shouldn’t download any jars from the internet during production
> deployment or upgrade procedure.
> > 2. The IEP description, for now, is quite small for such important
> things as a way we distribute and upgrade Ignite.
> > Petr, can you, please, make IEP more specific.
> > I think we should add the following details:
> > * Description of the typical Ignite deployment procedure
> including folder structure(config, persistence, .m2 and other folders)
> > * This also should include examples of the bash
> commands if any required.
> > * Description of the typical upgrade(downgrade?) procedure
> > * This also should include examples of the bash
> commands if any required.
> > * Description of «single tool responsible for all
> operations».
> > * all commands
> > * all operations
> > * all parameters
> >
> >> 1. Most paths are resolved relative to IGNITE_HOME, which effectively
> forces (or at least motivates) users to put everything related to Ignite
> under IGNITE_HOME (configs, additional JARs, etc.).
> >
> > AFAIU this issue doesn’t relate to the way Ignite distributed.
> > We can and should fix this outside of this IEP.
> >
> >> 2. Modules are enabled/disabled manually by moving folders around… this
> is error-prone,
> >
> > Makes sense.
> > Let’s fix this without «download jar from the internet»?
> > We can have some kind of config with the enabled modules.
> >
> >> 3. Standalone nodes use JARs prepackaged in the ZIP file. However, if
> you look at how this ZIP is built, you will notice that we don't actually
> include all transitive dependencies,
> >
> > We need to fix this.
> > Let’s include all the required JARs to the distribution.
> >
> >> which means that an embedded node that uses Maven can have a different
> classpath.
> >
> > Embedded Ignite node is an error-prone design decision in the first
> place.
> > We can’t fix it with the «download jars from the internet» approach.
> > We should eliminate the client node as a thing and use a thin clients
> instead of it.
> >
> >> 4. ignite.sh allows manual modifications (there are several "Uncomment
> this to..." lines). This complicates upgrades as well.
> >
> > Makes sense.
> > Let’s fix this.
> >
> >> The whole purpose of the IEP is to fix the above, automate and simplify
> operations for the users. The general idea is to create a single tool
> responsible for all those operations and use Maven under the hood to
> include/exclude modules.
> >
> > For now, IEP lacks the description of this tool.
> >
> >> 28 авг. 2020 г., в 22:35, Valentin Kulichenko <
> valentin.kulichenko@gmail.com> написал(а):
> >>
> >> Hi Nikolay,
> >>
> >> Here are some of the issues that I see with the current binary ZIP
> package
> >> that we create for every release.
> >>
> >> 1. Most paths are resolved relative to IGNITE_HOME, which effectively
> >> forces (or at least motivates) users to put everything related to Ignite
> >> under IGNITE_HOME (configs, additional JARs, etc.). Now, when a user
> >> downloads and unzips a new version, they have to somehow merge the
> content
> >> together - that's poor upgradability.
> >> 2. Modules are enabled/disabled manually by moving folders around. Not
> only
> >> this is error-prone, but it also adds more issues to upgradability - you
> >> have to apply these steps for every version you download.
> >> 3. Standalone nodes use JARs prepackaged in the ZIP file. However, if
> you
> >> look at how this ZIP is built, you will notice that we don't actually
> >> include all transitive dependencies, which means that an embedded node
> that
> >> uses Maven can have a different classpath. It seems that we've been
> lucky
> >> so far as this hasn't caused any issues, but I still think the approach
> is
> >> wrong. The behavior should be consistent.
> >> 4. ignite.sh allows manual modifications (there are several "Uncomment
> this
> >> to..." lines). This complicates upgrades as well.
> >>
> >> The whole purpose of the IEP is to fix the above, automate and
> >> simplify operations for the users. The general idea is to create a
> single
> >> tool responsible for all those operations and use Maven under the hood
> to
> >> include/exclude modules.
> >>
> >> Exact commands and steps are still under discussion though. It looks
> like
> >> Petr is going to create a prototype that can be used as a starting
> point,
> >> but if you have any suggestions or ideas in the meantime, please share
> them
> >> with us.
> >>
> >> -Val
> >>
> >> On Fri, Aug 28, 2020 at 12:31 AM Nikolay Izhikov <ni...@apache.org>
> >> wrote:
> >>
> >>> Hello, Igniters.
> >>>
> >>> Not sure if I understand the essence of the proposal at the moment.
> >>> Can you, please, describe the advantages of the proposed way from the
> user
> >>> perspective?
> >>>
> >>> How the typical DevOps pipeline should look like with this enhancement?
> >>>
> >>> How I as a user can create a fully functional installation package of
> the
> >>> Ignite?
> >>> AFAIK downloading some artifacts from the internet straight to the
> >>> production server is a security anti-pattern.
> >>>
> >>>
> >>>> 28 авг. 2020 г., в 01:59, Denis Magda <dm...@apache.org> написал(а):
> >>>>
> >>>> Petr, thanks,
> >>>>
> >>>> There is also a collection of modules located in our extensions
> >>> repository:
> >>>> https://github.com/apache/ignite-extensions
> >>>>
> >>>> @Saikat Maitra <sa...@gmail.com> is migrating all our
> existing
> >>>> integrations to that repository and, once this is done, the extensions
> >>> will
> >>>> be released to Maven separately. Is it reasonable to expand the scope
> of
> >>>> the IEP-52 and contemplate on how to install those extensions?
> >>>>
> >>>> -
> >>>> Denis
> >>>>
> >>>>
> >>>> On Thu, Aug 27, 2020 at 3:40 PM Valentin Kulichenko <
> >>>> valentin.kulichenko@gmail.com> wrote:
> >>>>
> >>>>> Hi Petr,
> >>>>>
> >>>>> The proposal makes sense to me - thanks for starting the discussion.
> >>>>> Current installation and upgrade procedures involve a lot of manual
> >>> steps
> >>>>> and quite error-prone, we need to automate and simplify them as much
> as
> >>>>> possible.
> >>>>>
> >>>>> I believe we eventually should end up with a unified command-line
> tool
> >>>>> (ignitectl?) that will incorporate all the operations (enable/disable
> >>>>> modules, start/stop, configuration updates, activation, baseline,
> etc.).
> >>>>> This IEP is a step in this direction.
> >>>>>
> >>>>> Looking forward to testing a prototype :)
> >>>>>
> >>>>> -Val
> >>>>>
> >>>>> On Thu, Aug 27, 2020 at 2:17 AM Petr Ivanov <mr...@gmail.com>
> >>> wrote:
> >>>>>
> >>>>>> Hi, Igniters!
> >>>>>>
> >>>>>>
> >>>>>> For Apache Ignite 3.0 I would like to propose vision of enhanced
> >>> delivery
> >>>>>> and upgrade processes [1].
> >>>>>> The key idea is to make main binary as slim as possible (delivering
> >>> every
> >>>>>> optional component by demand only) while providing way to run each
> new
> >>>>>> version seamlessly without much of the efforts migrating data and
> >>>>>> configuration between upgrades.
> >>>>>>
> >>>>>> I plan to create prototype based on current release of Apache Ignite
> >>>>>> (2.8.1 or 2.9.0 if it will be released soon) later in September.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> [1]
> >>>>>>
> >>>>>
> >>>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158873958
> >>>>>>
> >>>>>>
> >>>>>
> >>>
> >>>
> >
>
>
Re: IEP-52: Binary Delivery & Upgradability Enhancements
Posted by Nikolay Izhikov <ni...@apache.org>.
Also, guys.
Can we make one step back and add some requirements to the IEP.
How we want to install and upgrade Ignite?
What specific issues we want to solve?
With this input we can be on the same page during solution discussion.
> 29 авг. 2020 г., в 11:41, Nikolay Izhikov <NI...@gmail.com> написал(а):
>
> Hello, Val.
>
> Sorry, but I sill don’t understand how «download jars from the internet on production server» approach helps us to solve the issues you describe.
>
> Moreover, with this enhancement you want to add more dependencies to Ignite and create some security vulnerabilities:
> * Maven tool - new dependency. I doubt that and of the production server have it
> * Internet maven repository - many production servers restrict outgoing internet connection to any addresses.
> It’s hard to understand why distributed DB should have the ability to connect to some kind of address on the Internet.
> * All Ignite distributions can be compromised with the hack of some random maven repo or with the malicious proxy.
> * Note, that Ignite installation or upgrade procedure can become dead just because of some Maven repo down.
>
>
> I have 2 concerns:
>
> 1. We shouldn’t download any jars from the internet during production deployment or upgrade procedure.
> 2. The IEP description, for now, is quite small for such important things as a way we distribute and upgrade Ignite.
> Petr, can you, please, make IEP more specific.
> I think we should add the following details:
> * Description of the typical Ignite deployment procedure including folder structure(config, persistence, .m2 and other folders)
> * This also should include examples of the bash commands if any required.
> * Description of the typical upgrade(downgrade?) procedure
> * This also should include examples of the bash commands if any required.
> * Description of «single tool responsible for all operations».
> * all commands
> * all operations
> * all parameters
>
>> 1. Most paths are resolved relative to IGNITE_HOME, which effectively forces (or at least motivates) users to put everything related to Ignite under IGNITE_HOME (configs, additional JARs, etc.).
>
> AFAIU this issue doesn’t relate to the way Ignite distributed.
> We can and should fix this outside of this IEP.
>
>> 2. Modules are enabled/disabled manually by moving folders around… this is error-prone,
>
> Makes sense.
> Let’s fix this without «download jar from the internet»?
> We can have some kind of config with the enabled modules.
>
>> 3. Standalone nodes use JARs prepackaged in the ZIP file. However, if you look at how this ZIP is built, you will notice that we don't actually include all transitive dependencies,
>
> We need to fix this.
> Let’s include all the required JARs to the distribution.
>
>> which means that an embedded node that uses Maven can have a different classpath.
>
> Embedded Ignite node is an error-prone design decision in the first place.
> We can’t fix it with the «download jars from the internet» approach.
> We should eliminate the client node as a thing and use a thin clients instead of it.
>
>> 4. ignite.sh allows manual modifications (there are several "Uncomment this to..." lines). This complicates upgrades as well.
>
> Makes sense.
> Let’s fix this.
>
>> The whole purpose of the IEP is to fix the above, automate and simplify operations for the users. The general idea is to create a single tool responsible for all those operations and use Maven under the hood to include/exclude modules.
>
> For now, IEP lacks the description of this tool.
>
>> 28 авг. 2020 г., в 22:35, Valentin Kulichenko <va...@gmail.com> написал(а):
>>
>> Hi Nikolay,
>>
>> Here are some of the issues that I see with the current binary ZIP package
>> that we create for every release.
>>
>> 1. Most paths are resolved relative to IGNITE_HOME, which effectively
>> forces (or at least motivates) users to put everything related to Ignite
>> under IGNITE_HOME (configs, additional JARs, etc.). Now, when a user
>> downloads and unzips a new version, they have to somehow merge the content
>> together - that's poor upgradability.
>> 2. Modules are enabled/disabled manually by moving folders around. Not only
>> this is error-prone, but it also adds more issues to upgradability - you
>> have to apply these steps for every version you download.
>> 3. Standalone nodes use JARs prepackaged in the ZIP file. However, if you
>> look at how this ZIP is built, you will notice that we don't actually
>> include all transitive dependencies, which means that an embedded node that
>> uses Maven can have a different classpath. It seems that we've been lucky
>> so far as this hasn't caused any issues, but I still think the approach is
>> wrong. The behavior should be consistent.
>> 4. ignite.sh allows manual modifications (there are several "Uncomment this
>> to..." lines). This complicates upgrades as well.
>>
>> The whole purpose of the IEP is to fix the above, automate and
>> simplify operations for the users. The general idea is to create a single
>> tool responsible for all those operations and use Maven under the hood to
>> include/exclude modules.
>>
>> Exact commands and steps are still under discussion though. It looks like
>> Petr is going to create a prototype that can be used as a starting point,
>> but if you have any suggestions or ideas in the meantime, please share them
>> with us.
>>
>> -Val
>>
>> On Fri, Aug 28, 2020 at 12:31 AM Nikolay Izhikov <ni...@apache.org>
>> wrote:
>>
>>> Hello, Igniters.
>>>
>>> Not sure if I understand the essence of the proposal at the moment.
>>> Can you, please, describe the advantages of the proposed way from the user
>>> perspective?
>>>
>>> How the typical DevOps pipeline should look like with this enhancement?
>>>
>>> How I as a user can create a fully functional installation package of the
>>> Ignite?
>>> AFAIK downloading some artifacts from the internet straight to the
>>> production server is a security anti-pattern.
>>>
>>>
>>>> 28 авг. 2020 г., в 01:59, Denis Magda <dm...@apache.org> написал(а):
>>>>
>>>> Petr, thanks,
>>>>
>>>> There is also a collection of modules located in our extensions
>>> repository:
>>>> https://github.com/apache/ignite-extensions
>>>>
>>>> @Saikat Maitra <sa...@gmail.com> is migrating all our existing
>>>> integrations to that repository and, once this is done, the extensions
>>> will
>>>> be released to Maven separately. Is it reasonable to expand the scope of
>>>> the IEP-52 and contemplate on how to install those extensions?
>>>>
>>>> -
>>>> Denis
>>>>
>>>>
>>>> On Thu, Aug 27, 2020 at 3:40 PM Valentin Kulichenko <
>>>> valentin.kulichenko@gmail.com> wrote:
>>>>
>>>>> Hi Petr,
>>>>>
>>>>> The proposal makes sense to me - thanks for starting the discussion.
>>>>> Current installation and upgrade procedures involve a lot of manual
>>> steps
>>>>> and quite error-prone, we need to automate and simplify them as much as
>>>>> possible.
>>>>>
>>>>> I believe we eventually should end up with a unified command-line tool
>>>>> (ignitectl?) that will incorporate all the operations (enable/disable
>>>>> modules, start/stop, configuration updates, activation, baseline, etc.).
>>>>> This IEP is a step in this direction.
>>>>>
>>>>> Looking forward to testing a prototype :)
>>>>>
>>>>> -Val
>>>>>
>>>>> On Thu, Aug 27, 2020 at 2:17 AM Petr Ivanov <mr...@gmail.com>
>>> wrote:
>>>>>
>>>>>> Hi, Igniters!
>>>>>>
>>>>>>
>>>>>> For Apache Ignite 3.0 I would like to propose vision of enhanced
>>> delivery
>>>>>> and upgrade processes [1].
>>>>>> The key idea is to make main binary as slim as possible (delivering
>>> every
>>>>>> optional component by demand only) while providing way to run each new
>>>>>> version seamlessly without much of the efforts migrating data and
>>>>>> configuration between upgrades.
>>>>>>
>>>>>> I plan to create prototype based on current release of Apache Ignite
>>>>>> (2.8.1 or 2.9.0 if it will be released soon) later in September.
>>>>>>
>>>>>>
>>>>>>
>>>>>> [1]
>>>>>>
>>>>>
>>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158873958
>>>>>>
>>>>>>
>>>>>
>>>
>>>
>
Re: IEP-52: Binary Delivery & Upgradability Enhancements
Posted by Nikolay Izhikov <ni...@apache.org>.
Hello, Val.
Sorry, but I sill don’t understand how «download jars from the internet on production server» approach helps us to solve the issues you describe.
Moreover, with this enhancement you want to add more dependencies to Ignite and create some security vulnerabilities:
* Maven tool - new dependency. I doubt that and of the production server have it
* Internet maven repository - many production servers restrict outgoing internet connection to any addresses.
It’s hard to understand why distributed DB should have the ability to connect to some kind of address on the Internet.
* All Ignite distributions can be compromised with the hack of some random maven repo or with the malicious proxy.
* Note, that Ignite installation or upgrade procedure can become dead just because of some Maven repo down.
I have 2 concerns:
1. We shouldn’t download any jars from the internet during production deployment or upgrade procedure.
2. The IEP description, for now, is quite small for such important things as a way we distribute and upgrade Ignite.
Petr, can you, please, make IEP more specific.
I think we should add the following details:
* Description of the typical Ignite deployment procedure including folder structure(config, persistence, .m2 and other folders)
* This also should include examples of the bash commands if any required.
* Description of the typical upgrade(downgrade?) procedure
* This also should include examples of the bash commands if any required.
* Description of «single tool responsible for all operations».
* all commands
* all operations
* all parameters
> 1. Most paths are resolved relative to IGNITE_HOME, which effectively forces (or at least motivates) users to put everything related to Ignite under IGNITE_HOME (configs, additional JARs, etc.).
AFAIU this issue doesn’t relate to the way Ignite distributed.
We can and should fix this outside of this IEP.
> 2. Modules are enabled/disabled manually by moving folders around… this is error-prone,
Makes sense.
Let’s fix this without «download jar from the internet»?
We can have some kind of config with the enabled modules.
> 3. Standalone nodes use JARs prepackaged in the ZIP file. However, if you look at how this ZIP is built, you will notice that we don't actually include all transitive dependencies,
We need to fix this.
Let’s include all the required JARs to the distribution.
> which means that an embedded node that uses Maven can have a different classpath.
Embedded Ignite node is an error-prone design decision in the first place.
We can’t fix it with the «download jars from the internet» approach.
We should eliminate the client node as a thing and use a thin clients instead of it.
> 4. ignite.sh allows manual modifications (there are several "Uncomment this to..." lines). This complicates upgrades as well.
Makes sense.
Let’s fix this.
> The whole purpose of the IEP is to fix the above, automate and simplify operations for the users. The general idea is to create a single tool responsible for all those operations and use Maven under the hood to include/exclude modules.
For now, IEP lacks the description of this tool.
> 28 авг. 2020 г., в 22:35, Valentin Kulichenko <va...@gmail.com> написал(а):
>
> Hi Nikolay,
>
> Here are some of the issues that I see with the current binary ZIP package
> that we create for every release.
>
> 1. Most paths are resolved relative to IGNITE_HOME, which effectively
> forces (or at least motivates) users to put everything related to Ignite
> under IGNITE_HOME (configs, additional JARs, etc.). Now, when a user
> downloads and unzips a new version, they have to somehow merge the content
> together - that's poor upgradability.
> 2. Modules are enabled/disabled manually by moving folders around. Not only
> this is error-prone, but it also adds more issues to upgradability - you
> have to apply these steps for every version you download.
> 3. Standalone nodes use JARs prepackaged in the ZIP file. However, if you
> look at how this ZIP is built, you will notice that we don't actually
> include all transitive dependencies, which means that an embedded node that
> uses Maven can have a different classpath. It seems that we've been lucky
> so far as this hasn't caused any issues, but I still think the approach is
> wrong. The behavior should be consistent.
> 4. ignite.sh allows manual modifications (there are several "Uncomment this
> to..." lines). This complicates upgrades as well.
>
> The whole purpose of the IEP is to fix the above, automate and
> simplify operations for the users. The general idea is to create a single
> tool responsible for all those operations and use Maven under the hood to
> include/exclude modules.
>
> Exact commands and steps are still under discussion though. It looks like
> Petr is going to create a prototype that can be used as a starting point,
> but if you have any suggestions or ideas in the meantime, please share them
> with us.
>
> -Val
>
> On Fri, Aug 28, 2020 at 12:31 AM Nikolay Izhikov <ni...@apache.org>
> wrote:
>
>> Hello, Igniters.
>>
>> Not sure if I understand the essence of the proposal at the moment.
>> Can you, please, describe the advantages of the proposed way from the user
>> perspective?
>>
>> How the typical DevOps pipeline should look like with this enhancement?
>>
>> How I as a user can create a fully functional installation package of the
>> Ignite?
>> AFAIK downloading some artifacts from the internet straight to the
>> production server is a security anti-pattern.
>>
>>
>>> 28 авг. 2020 г., в 01:59, Denis Magda <dm...@apache.org> написал(а):
>>>
>>> Petr, thanks,
>>>
>>> There is also a collection of modules located in our extensions
>> repository:
>>> https://github.com/apache/ignite-extensions
>>>
>>> @Saikat Maitra <sa...@gmail.com> is migrating all our existing
>>> integrations to that repository and, once this is done, the extensions
>> will
>>> be released to Maven separately. Is it reasonable to expand the scope of
>>> the IEP-52 and contemplate on how to install those extensions?
>>>
>>> -
>>> Denis
>>>
>>>
>>> On Thu, Aug 27, 2020 at 3:40 PM Valentin Kulichenko <
>>> valentin.kulichenko@gmail.com> wrote:
>>>
>>>> Hi Petr,
>>>>
>>>> The proposal makes sense to me - thanks for starting the discussion.
>>>> Current installation and upgrade procedures involve a lot of manual
>> steps
>>>> and quite error-prone, we need to automate and simplify them as much as
>>>> possible.
>>>>
>>>> I believe we eventually should end up with a unified command-line tool
>>>> (ignitectl?) that will incorporate all the operations (enable/disable
>>>> modules, start/stop, configuration updates, activation, baseline, etc.).
>>>> This IEP is a step in this direction.
>>>>
>>>> Looking forward to testing a prototype :)
>>>>
>>>> -Val
>>>>
>>>> On Thu, Aug 27, 2020 at 2:17 AM Petr Ivanov <mr...@gmail.com>
>> wrote:
>>>>
>>>>> Hi, Igniters!
>>>>>
>>>>>
>>>>> For Apache Ignite 3.0 I would like to propose vision of enhanced
>> delivery
>>>>> and upgrade processes [1].
>>>>> The key idea is to make main binary as slim as possible (delivering
>> every
>>>>> optional component by demand only) while providing way to run each new
>>>>> version seamlessly without much of the efforts migrating data and
>>>>> configuration between upgrades.
>>>>>
>>>>> I plan to create prototype based on current release of Apache Ignite
>>>>> (2.8.1 or 2.9.0 if it will be released soon) later in September.
>>>>>
>>>>>
>>>>>
>>>>> [1]
>>>>>
>>>>
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158873958
>>>>>
>>>>>
>>>>
>>
>>
Re: IEP-52: Binary Delivery & Upgradability Enhancements
Posted by Valentin Kulichenko <va...@gmail.com>.
Hi Nikolay,
Here are some of the issues that I see with the current binary ZIP package
that we create for every release.
1. Most paths are resolved relative to IGNITE_HOME, which effectively
forces (or at least motivates) users to put everything related to Ignite
under IGNITE_HOME (configs, additional JARs, etc.). Now, when a user
downloads and unzips a new version, they have to somehow merge the content
together - that's poor upgradability.
2. Modules are enabled/disabled manually by moving folders around. Not only
this is error-prone, but it also adds more issues to upgradability - you
have to apply these steps for every version you download.
3. Standalone nodes use JARs prepackaged in the ZIP file. However, if you
look at how this ZIP is built, you will notice that we don't actually
include all transitive dependencies, which means that an embedded node that
uses Maven can have a different classpath. It seems that we've been lucky
so far as this hasn't caused any issues, but I still think the approach is
wrong. The behavior should be consistent.
4. ignite.sh allows manual modifications (there are several "Uncomment this
to..." lines). This complicates upgrades as well.
The whole purpose of the IEP is to fix the above, automate and
simplify operations for the users. The general idea is to create a single
tool responsible for all those operations and use Maven under the hood to
include/exclude modules.
Exact commands and steps are still under discussion though. It looks like
Petr is going to create a prototype that can be used as a starting point,
but if you have any suggestions or ideas in the meantime, please share them
with us.
-Val
On Fri, Aug 28, 2020 at 12:31 AM Nikolay Izhikov <ni...@apache.org>
wrote:
> Hello, Igniters.
>
> Not sure if I understand the essence of the proposal at the moment.
> Can you, please, describe the advantages of the proposed way from the user
> perspective?
>
> How the typical DevOps pipeline should look like with this enhancement?
>
> How I as a user can create a fully functional installation package of the
> Ignite?
> AFAIK downloading some artifacts from the internet straight to the
> production server is a security anti-pattern.
>
>
> > 28 авг. 2020 г., в 01:59, Denis Magda <dm...@apache.org> написал(а):
> >
> > Petr, thanks,
> >
> > There is also a collection of modules located in our extensions
> repository:
> > https://github.com/apache/ignite-extensions
> >
> > @Saikat Maitra <sa...@gmail.com> is migrating all our existing
> > integrations to that repository and, once this is done, the extensions
> will
> > be released to Maven separately. Is it reasonable to expand the scope of
> > the IEP-52 and contemplate on how to install those extensions?
> >
> > -
> > Denis
> >
> >
> > On Thu, Aug 27, 2020 at 3:40 PM Valentin Kulichenko <
> > valentin.kulichenko@gmail.com> wrote:
> >
> >> Hi Petr,
> >>
> >> The proposal makes sense to me - thanks for starting the discussion.
> >> Current installation and upgrade procedures involve a lot of manual
> steps
> >> and quite error-prone, we need to automate and simplify them as much as
> >> possible.
> >>
> >> I believe we eventually should end up with a unified command-line tool
> >> (ignitectl?) that will incorporate all the operations (enable/disable
> >> modules, start/stop, configuration updates, activation, baseline, etc.).
> >> This IEP is a step in this direction.
> >>
> >> Looking forward to testing a prototype :)
> >>
> >> -Val
> >>
> >> On Thu, Aug 27, 2020 at 2:17 AM Petr Ivanov <mr...@gmail.com>
> wrote:
> >>
> >>> Hi, Igniters!
> >>>
> >>>
> >>> For Apache Ignite 3.0 I would like to propose vision of enhanced
> delivery
> >>> and upgrade processes [1].
> >>> The key idea is to make main binary as slim as possible (delivering
> every
> >>> optional component by demand only) while providing way to run each new
> >>> version seamlessly without much of the efforts migrating data and
> >>> configuration between upgrades.
> >>>
> >>> I plan to create prototype based on current release of Apache Ignite
> >>> (2.8.1 or 2.9.0 if it will be released soon) later in September.
> >>>
> >>>
> >>>
> >>> [1]
> >>>
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158873958
> >>>
> >>>
> >>
>
>
Re: IEP-52: Binary Delivery & Upgradability Enhancements
Posted by Nikolay Izhikov <ni...@apache.org>.
Hello, Igniters.
Not sure if I understand the essence of the proposal at the moment.
Can you, please, describe the advantages of the proposed way from the user perspective?
How the typical DevOps pipeline should look like with this enhancement?
How I as a user can create a fully functional installation package of the Ignite?
AFAIK downloading some artifacts from the internet straight to the production server is a security anti-pattern.
> 28 авг. 2020 г., в 01:59, Denis Magda <dm...@apache.org> написал(а):
>
> Petr, thanks,
>
> There is also a collection of modules located in our extensions repository:
> https://github.com/apache/ignite-extensions
>
> @Saikat Maitra <sa...@gmail.com> is migrating all our existing
> integrations to that repository and, once this is done, the extensions will
> be released to Maven separately. Is it reasonable to expand the scope of
> the IEP-52 and contemplate on how to install those extensions?
>
> -
> Denis
>
>
> On Thu, Aug 27, 2020 at 3:40 PM Valentin Kulichenko <
> valentin.kulichenko@gmail.com> wrote:
>
>> Hi Petr,
>>
>> The proposal makes sense to me - thanks for starting the discussion.
>> Current installation and upgrade procedures involve a lot of manual steps
>> and quite error-prone, we need to automate and simplify them as much as
>> possible.
>>
>> I believe we eventually should end up with a unified command-line tool
>> (ignitectl?) that will incorporate all the operations (enable/disable
>> modules, start/stop, configuration updates, activation, baseline, etc.).
>> This IEP is a step in this direction.
>>
>> Looking forward to testing a prototype :)
>>
>> -Val
>>
>> On Thu, Aug 27, 2020 at 2:17 AM Petr Ivanov <mr...@gmail.com> wrote:
>>
>>> Hi, Igniters!
>>>
>>>
>>> For Apache Ignite 3.0 I would like to propose vision of enhanced delivery
>>> and upgrade processes [1].
>>> The key idea is to make main binary as slim as possible (delivering every
>>> optional component by demand only) while providing way to run each new
>>> version seamlessly without much of the efforts migrating data and
>>> configuration between upgrades.
>>>
>>> I plan to create prototype based on current release of Apache Ignite
>>> (2.8.1 or 2.9.0 if it will be released soon) later in September.
>>>
>>>
>>>
>>> [1]
>>>
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158873958
>>>
>>>
>>
Re: IEP-52: Binary Delivery & Upgradability Enhancements
Posted by Denis Magda <dm...@apache.org>.
Petr, thanks,
There is also a collection of modules located in our extensions repository:
https://github.com/apache/ignite-extensions
@Saikat Maitra <sa...@gmail.com> is migrating all our existing
integrations to that repository and, once this is done, the extensions will
be released to Maven separately. Is it reasonable to expand the scope of
the IEP-52 and contemplate on how to install those extensions?
-
Denis
On Thu, Aug 27, 2020 at 3:40 PM Valentin Kulichenko <
valentin.kulichenko@gmail.com> wrote:
> Hi Petr,
>
> The proposal makes sense to me - thanks for starting the discussion.
> Current installation and upgrade procedures involve a lot of manual steps
> and quite error-prone, we need to automate and simplify them as much as
> possible.
>
> I believe we eventually should end up with a unified command-line tool
> (ignitectl?) that will incorporate all the operations (enable/disable
> modules, start/stop, configuration updates, activation, baseline, etc.).
> This IEP is a step in this direction.
>
> Looking forward to testing a prototype :)
>
> -Val
>
> On Thu, Aug 27, 2020 at 2:17 AM Petr Ivanov <mr...@gmail.com> wrote:
>
> > Hi, Igniters!
> >
> >
> > For Apache Ignite 3.0 I would like to propose vision of enhanced delivery
> > and upgrade processes [1].
> > The key idea is to make main binary as slim as possible (delivering every
> > optional component by demand only) while providing way to run each new
> > version seamlessly without much of the efforts migrating data and
> > configuration between upgrades.
> >
> > I plan to create prototype based on current release of Apache Ignite
> > (2.8.1 or 2.9.0 if it will be released soon) later in September.
> >
> >
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158873958
> >
> >
>
Re: IEP-52: Binary Delivery & Upgradability Enhancements
Posted by Valentin Kulichenko <va...@gmail.com>.
Hi Petr,
The proposal makes sense to me - thanks for starting the discussion.
Current installation and upgrade procedures involve a lot of manual steps
and quite error-prone, we need to automate and simplify them as much as
possible.
I believe we eventually should end up with a unified command-line tool
(ignitectl?) that will incorporate all the operations (enable/disable
modules, start/stop, configuration updates, activation, baseline, etc.).
This IEP is a step in this direction.
Looking forward to testing a prototype :)
-Val
On Thu, Aug 27, 2020 at 2:17 AM Petr Ivanov <mr...@gmail.com> wrote:
> Hi, Igniters!
>
>
> For Apache Ignite 3.0 I would like to propose vision of enhanced delivery
> and upgrade processes [1].
> The key idea is to make main binary as slim as possible (delivering every
> optional component by demand only) while providing way to run each new
> version seamlessly without much of the efforts migrating data and
> configuration between upgrades.
>
> I plan to create prototype based on current release of Apache Ignite
> (2.8.1 or 2.9.0 if it will be released soon) later in September.
>
>
>
> [1]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158873958
>
>