You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by Alexey Goncharuk <al...@gmail.com> on 2020/10/06 14:19:39 UTC
Re: Apache Ignite 2.9.0 RELEASE [Time, Scope, Manager]
Saikat,
The plan sounds good to me! Do you have an idea for the timeline of the
module releases? Let me know if I can help in any way.
Also, I was looking into the modularization IEP and noticed that OSGI
module is discussed to be moved to the extensions, but there is no
corresponding ticket. Should I create one? I will be happy to help with
moving this module to extensions.
вт, 29 сент. 2020 г. в 03:03, Saikat Maitra <sa...@gmail.com>:
> Hi Alexey,
>
> All the modules as planned in IEP-36
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36:+Modularization
> are migrated to ignite-extensions repository.
>
> We can definitely plan to release ignite-extensions modules.
>
> I have a pending PR related to the kafka module and the PR is in review
> https://github.com/apache/ignite/pull/8222 but its corresponding PR has
> been merged in ignite-extensions repository.
>
> My thoughts / action items for the migration efforts and releases were
> as follows:
>
> 1. Merge the changes for https://github.com/apache/ignite/pull/8222
> 2. Fix the upload nightly packages task for ignite-core to help fix the
> travis build failure in ignite-extensions repository.
> 3. Review and update the README.md files for the ignite-extensions modules
> as required.
> 4. Update the docs for ignite-extensions modules in ignite-website.
> 5. Release each module separately and share updates.
>
> Please let me know if you have feedback.
>
> Regards,
> Saikat
>
>
>
>
>
>
>
>
> On Mon, Sep 28, 2020 at 7:36 AM Petr Ivanov <mr...@gmail.com> wrote:
>
>> It seems that gitbox.apache.org is not available on CI/CD.
>>
>> I will try to update VCS to GitHub.
>>
>>
>>
>>
>>
>> > On 28 Sep 2020, at 14:07, Alex Plehanov <pl...@gmail.com>
>> wrote:
>> >
>> > Guys,
>> >
>> > I've tried to build release candidate using TC [1]. But:
>> > 1. I can't choose a branch in this TC task (there is no such field).
>> > 2. On the "default" branch I got an error: Failed to collect changes,
>> > error: List remote refs failed:
>> > org.apache.http.conn.ConnectTimeoutException: Connect to
>> > gitbox.apache.org:443 [gitbox.apache.org/52.202.80.70] failed: connect
>> > timed out, VCS root: "GitBox [asf/ignite]" {instance id=300, parent
>> > internal id=74, parent id=GitBoxAsfIgnite, description: "
>> > https://gitbox.apache.org/repos/asf/ignite.git#refs/heads/master"}
>> >
>> > Is there some problem with this TC task? What am I doing wrong?
>> >
>> > [1] :
>> >
>> https://ci.ignite.apache.org/viewType.html?buildTypeId=Releases_ApacheIgniteMain_ReleaseBuild
>> >
>> > пн, 28 сент. 2020 г. в 13:15, Alexey Goncharuk <
>> alexey.goncharuk@gmail.com>:
>> >
>> >> Saikat, Nikolay,
>> >>
>> >> We have migrated a bunch of modules to ignite-extensions recently.
>> Given
>> >> that these modules will not be available in Ignite 2.9 anymore (will
>> >> they?), should we also plan to release the extensions?
>> >>
>> >> ср, 23 сент. 2020 г. в 21:49, Alex Plehanov <pl...@gmail.com>:
>> >>
>> >>> Igniters,
>> >>>
>> >>> I've prepared release notes for the 2.9 release [1]. Can anyone review
>> >> it,
>> >>> please?
>> >>>
>> >>> [1]: https://github.com/apache/ignite/pull/8273
>> >>>
>> >>> вт, 22 сент. 2020 г. в 09:39, Alex Plehanov <plehanov.alex@gmail.com
>> >:
>> >>>
>> >>>> Guys,
>> >>>>
>> >>>> I've filled the ticket with reproducer [1] for the discovery bug.
>> This
>> >>> bug
>> >>>> caused by [2] ticket. We discussed with Vladimir Steshin privately
>> and
>> >>>> decided to revert this ticket. I will do it today (after TC bot visa)
>> >> if
>> >>>> there are no objections.
>> >>>>
>> >>>> [1]: https://issues.apache.org/jira/browse/IGNITE-13465
>> >>>> [2]: https://issues.apache.org/jira/browse/IGNITE-13134
>> >>>>
>> >>>> пн, 21 сент. 2020 г. в 11:08, Alex Plehanov <plehanov.alex@gmail.com
>> >:
>> >>>>
>> >>>>> Guys,
>> >>>>>
>> >>>>> During internal testing, we've found a critical bug with
>> >>>>> discovery (cluster falls apart if two nodes segmented sequentially).
>> >>> This
>> >>>>> problem is not reproduced in 2.8.1. I think we should fix it
>> >>>>> before release. Under investigation now. I'll let you know when we
>> get
>> >>>>> something.
>> >>>>>
>> >>>>> чт, 17 сент. 2020 г. в 00:51, Andrey Gura <ag...@apache.org>:
>> >>>>>
>> >>>>>>> So what do you think? Should we proceed with a 'hacked' version of
>> >>> the
>> >>>>>> message factory in 2.9 and go for the runtime message generation in
>> >>> later
>> >>>>>> release, or keep the code clean and fix the regression in the next
>> >>> releases?
>> >>>>>>> Andrey, can you take a look at my change? I think it is fairly
>> >>>>>> straightforward and does not change the semantics, just skips the
>> >>> factory
>> >>>>>> closures for certain messages.
>> >>>>>>
>> >>>>>> IMHO 2.5% isn't too much especially because it isn't actual for all
>> >>>>>> workloads (I didn't get any significant drops during benchmarking).
>> >> So
>> >>>>>> I prefer the runtime generation in later releases.
>> >>>>>>
>> >>>>>> On Mon, Sep 14, 2020 at 12:41 PM Alexey Goncharuk
>> >>>>>> <al...@gmail.com> wrote:
>> >>>>>>>
>> >>>>>>> Alexey, Andrey, Igniters,
>> >>>>>>>
>> >>>>>>> So what do you think? Should we proceed with a 'hacked' version of
>> >>> the
>> >>>>>> message factory in 2.9 and go for the runtime message generation in
>> >>> later
>> >>>>>> release, or keep the code clean and fix the regression in the next
>> >>> releases?
>> >>>>>>> Andrey, can you take a look at my change? I think it is fairly
>> >>>>>> straightforward and does not change the semantics, just skips the
>> >>> factory
>> >>>>>> closures for certain messages.
>> >>>>>>>
>> >>>>>>> Personally, I would prefer fixing the regression given that we
>> also
>> >>>>>> introduced tracing in this release.
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> пт, 11 сент. 2020 г. в 12:09, Alex Plehanov <
>> >> plehanov.alex@gmail.com
>> >>>> :
>> >>>>>>>>
>> >>>>>>>> Alexey,
>> >>>>>>>>
>> >>>>>>>> We've benchmarked by yardstick commits 130376741bf vs ed52559eb95
>> >>> and
>> >>>>>> the performance of ed52559eb95 is better for about 2.5% on average
>> on
>> >>> our
>> >>>>>> environment (about the same results we got benchmarking 65c30ec6 vs
>> >>>>>> 0606f03d). We've made 24 runs for each commit of
>> >>>>>> IgnitePutTxImplicitBenchmark (we got maximum drop for 2.9 on this
>> >>>>>> benchmark), 200 seconds warmup, 300 seconds benchmark, 6 servers, 5
>> >>> clients
>> >>>>>> 50 threads each. Yardstick results for this configuration:
>> >>>>>>>> Commit 130376741bf: avg TPS=164096, avg latency=9173464 ns
>> >>>>>>>> Commit ed52559eb95: avg TPS=168283, avg latency=8945908 ns
>> >>>>>>>>
>> >>>>>>>> пт, 11 сент. 2020 г. в 09:51, Artem Budnikov <
>> >>>>>> a.budnikov.ignite@gmail.com>:
>> >>>>>>>>>
>> >>>>>>>>> Hi Everyone,
>> >>>>>>>>>
>> >>>>>>>>> I posted an instruction on how to publish the docs on
>> >>>>>> ignite.apache.org/docs [1]. When you finish with Ignite 2.9, you
>> can
>> >>>>>> update the docs by following the instruction. Unfortunately, I
>> won't
>> >> be
>> >>>>>> able to spend any time on this project any longer. You can send
>> your
>> >>> pull
>> >>>>>> requests and questions about the documentation to Denis Magda.
>> >>>>>>>>>
>> >>>>>>>>> -Artem
>> >>>>>>>>>
>> >>>>>>>>> [1] :
>> >>>>>> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Document
>> >>>>>>>>>
>> >>>>>>>>> On Thu, Sep 10, 2020 at 2:45 PM Alexey Goncharuk <
>> >>>>>> alexey.goncharuk@gmail.com> wrote:
>> >>>>>>>>>>
>> >>>>>>>>>> Alexey,
>> >>>>>>>>>>
>> >>>>>>>>>> I've tried to play with message factories locally, but
>> >>>>>> unfortunately, I
>> >>>>>>>>>> cannot spot the difference between old and new implementation
>> in
>> >>>>>>>>>> distributed benchmarks. I pushed an implementation of
>> >>>>>> MessageFactoryImpl
>> >>>>>>>>>> with the old switch statement to the ignite-2.9-revert-12568
>> >>> branch
>> >>>>>>>>>> (discussed this with Andrey Gura, the change should be
>> >> compatible
>> >>>>>> with the
>> >>>>>>>>>> new metrics as we still use the register() mechanics).
>> >>>>>>>>>>
>> >>>>>>>>>> Can you check if this change makes any difference
>> >> performance-wise
>> >>>>>> in your
>> >>>>>>>>>> environment? If yes, we can go with runtime code generation in
>> >> the
>> >>>>>> long
>> >>>>>>>>>> term: register classes and generate a dynamic message factory
>> >> with
>> >>>>>> a switch
>> >>>>>>>>>> statement once all messages are registered (not in 2.9 though,
>> >>>>>> obviously).
>> >>>>>>>>>>
>> >>>>>>>>>> ср, 9 сент. 2020 г. в 14:53, Alex Plehanov <
>> >>> plehanov.alex@gmail.com
>> >>>>>>> :
>> >>>>>>>>>>
>> >>>>>>>>>>> Hello guys,
>> >>>>>>>>>>>
>> >>>>>>>>>>> I've tried to optimize tracing implementation (ticket [1]), it
>> >>>>>> reduced the
>> >>>>>>>>>>> drop, but not completely removed it.
>> >>>>>>>>>>> Ivan Rakov, Alexander Lapin, can you please review the patch?
>> >>>>>>>>>>> Ivan Artiukhov, can you please benchmark the patch [2] against
>> >>>>>> 2.8.1
>> >>>>>>>>>>> release on your environment?
>> >>>>>>>>>>> With this patch on our environment, it's about a 3% drop left,
>> >>>>>> it's close
>> >>>>>>>>>>> to measurement error and I think such a drop is not a
>> >>>>>> showstopper. Guys,
>> >>>>>>>>>>> WDYT?
>> >>>>>>>>>>>
>> >>>>>>>>>>> Also, I found that compatibility is broken for JDBC thin
>> >> driver
>> >>>>>> between 2.8
>> >>>>>>>>>>> and 2.9 versions (ticket [3]). I think it's a blocker and
>> >> should
>> >>>>>> be
>> >>>>>>>>>>> fixed in 2.9. I've prepared the patch.
>> >>>>>>>>>>> Taras Ledkov, can you please review this patch?
>> >>>>>>>>>>>
>> >>>>>>>>>>> And one more ticket I propose to include into 2.9 [4] (NIO
>> >>> message
>> >>>>>>>>>>> send problem in some circumstances). I will cherry-pick it if
>> >>>>>> there is no
>> >>>>>>>>>>> objection.
>> >>>>>>>>>>>
>> >>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13411
>> >>>>>>>>>>> [2] https://github.com/apache/ignite/pull/8223
>> >>>>>>>>>>> [3] https://issues.apache.org/jira/browse/IGNITE-13414
>> >>>>>>>>>>> [4] https://issues.apache.org/jira/browse/IGNITE-13361
>> >>>>>>>>>>>
>> >>>>>>>>>>> пн, 7 сент. 2020 г. в 14:14, Maxim Muzafarov <
>> >> mmuzaf@apache.org
>> >>>> :
>> >>>>>>>>>>>
>> >>>>>>>>>>>> Alexey,
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> I propose to include [1] issue to the 2.9 release. Since
>> >> this
>> >>>>>> issue is
>> >>>>>>>>>>>> related to the new master key change functionality which
>> >>>>>> haven't been
>> >>>>>>>>>>>> released yet I think it will be safe to cherry-pick commit
>> >> to
>> >>>>>> the
>> >>>>>>>>>>>> release branch.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13390
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> On Tue, 1 Sep 2020 at 12:13, Nikolay Izhikov <
>> >>>>>> nizhikov@apache.org>
>> >>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Hello, Igniters.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Alexey, please, include one more Python thin client fix
>> >> [1]
>> >>>>>> into the
>> >>>>>>>>>>> 2.9
>> >>>>>>>>>>>> release
>> >>>>>>>>>>>>> It fixes kinda major issue - "Python client returns fields
>> >>> in
>> >>>>>> wrong
>> >>>>>>>>>>>> order since the 2 row when fields_count>10"
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-12809
>> >>>>>>>>>>>>> [2]
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>
>> >>>
>> >>
>> https://github.com/apache/ignite/commit/38025ee4167f05eaa2d6a2c5c2ab70c83a462cfc
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>> 31 авг. 2020 г., в 19:23, Alexey Goncharuk <
>> >>>>>>>>>>> alexey.goncharuk@gmail.com>
>> >>>>>>>>>>>> написал(а):
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Alexey, thanks, got it. I am not sure we can optimize
>> >>>>>> anything out of
>> >>>>>>>>>>>> the
>> >>>>>>>>>>>>>> message factory with suppliers (at least I have no ideas
>> >>>>>> right now),
>> >>>>>>>>>>> so
>> >>>>>>>>>>>>>> most likely the only move here is to switch back to the
>> >>>>>> switch
>> >>>>>>>>>>> approach
>> >>>>>>>>>>>>>> somehow preserving the metrics part. Probably, inlining
>> >>> the
>> >>>>>> Ignite
>> >>>>>>>>>>>> messages
>> >>>>>>>>>>>>>> to the IgniteMessageFactoryImpl should do the trick. Let
>> >>> me
>> >>>>>> explore
>> >>>>>>>>>>> the
>> >>>>>>>>>>>>>> code a bit.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> P.S. I am surprised by the impact this part makes for
>> >> the
>> >>>>>>>>>>> performance.
>> >>>>>>>>>>>>>> Message creation is indeed on the hot path, but a single
>> >>>>>> virtual call
>> >>>>>>>>>>>>>> should not make that much of a difference given the
>> >> amount
>> >>>>>> of other
>> >>>>>>>>>>>> work
>> >>>>>>>>>>>>>> happening during the message processing.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 18:33, Alex Plehanov <
>> >>>>>> plehanov.alex@gmail.com
>> >>>>>>>>>>>> :
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Alexey, sorry, I wrongly interpreted our benchmark
>> >>> results.
>> >>>>>>>>>>> Actually,
>> >>>>>>>>>>>> we
>> >>>>>>>>>>>>>>> were looking for a drop using bi-sect in the range
>> >>> between
>> >>>>>> e6a7f93
>> >>>>>>>>>>>> (first
>> >>>>>>>>>>>>>>> commit in the 2.9 branch after 2.8 branch cut) and
>> >>>>>> 6592dfa5 (last
>> >>>>>>>>>>>> commit in
>> >>>>>>>>>>>>>>> the 2.9 branch). And we found these two problematic
>> >>>>>> commits.
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Perhaps only IGNITE-13060 (Tracing) is responsible for
>> >> a
>> >>>>>> drop
>> >>>>>>>>>>> between
>> >>>>>>>>>>>>>>> 2.8.1 and 2.9 (we have benchmarked 2.8.1 vs 2.9 with
>> >>>>>> reverted
>> >>>>>>>>>>>> IGNITE-13060
>> >>>>>>>>>>>>>>> now and performance looks the same)
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Ticket IGNITE-12568 (MessageFactory refactoring) is not
>> >>>>>> related to
>> >>>>>>>>>>>> drop
>> >>>>>>>>>>>>>>> between 2.8.1 and 2.9, but still has some performance
>> >>>>>> problem, and
>> >>>>>>>>>>> we
>> >>>>>>>>>>>> can
>> >>>>>>>>>>>>>>> win back IGNITE-13060 drop by this ticket.
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Do we need more investigation on IGNITE-13060 or we
>> >> leave
>> >>>>>> it as is?
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> What should we do with IGNITE-12568 (MessageFactory
>> >>>>>> refactoring)?
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 13:25, Alexey Goncharuk <
>> >>>>>>>>>>>> alexey.goncharuk@gmail.com
>> >>>>>>>>>>>>>>>> :
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> Alexey,
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> While investigating, I found that IGNITE-12568 has an
>> >>>>>> incorrect fix
>> >>>>>>>>>>>>>>>> version and is actually present in ignite-2.8.1 branch
>> >>>>>> [1], so it
>> >>>>>>>>>>>> cannot be
>> >>>>>>>>>>>>>>>> the source of the drop against 2.8.1.
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> P.S. Looks like we need to enforce a more accurate
>> >> work
>> >>>>>> with fix
>> >>>>>>>>>>>> versions
>> >>>>>>>>>>>>>>>> or develop some sort of tooling to verify the fix
>> >>>>>> versions.
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> --AG
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> [1]
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>
>> >>>
>> >>
>> https://github.com/apache/ignite/commit/3e492bd23851856bbd0385c6a419892d0bba2a34
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 12:42, Alexey Goncharuk <
>> >>>>>>>>>>>> alexey.goncharuk@gmail.com
>> >>>>>>>>>>>>>>>>> :
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> пт, 28 авг. 2020 г. в 11:16, Alex Plehanov <
>> >>>>>>>>>>> plehanov.alex@gmail.com
>> >>>>>>>>>>>>> :
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> Guys,
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> We have benchmarked 2.9 without IGNITE-13060 and
>> >>>>>> IGNITE-12568
>> >>>>>>>>>>>> (reverted
>> >>>>>>>>>>>>>>>>>> it
>> >>>>>>>>>>>>>>>>>> locally) and got the same performance as on 2.8.1
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> IGNITE-13060 (Tracing) - some code was added to hot
>> >>>>>> paths, to
>> >>>>>>>>>>> trace
>> >>>>>>>>>>>>>>>>>> these
>> >>>>>>>>>>>>>>>>>> hot paths, it's clear why we have performance drop
>> >>> here.
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> IGNITE-12568 (MessageFactory refactoring) -
>> >>> switch/case
>> >>>>>> block was
>> >>>>>>>>>>>>>>>>>> refactored to an array of message suppliers. The
>> >>>>>> message factory
>> >>>>>>>>>>>> is on
>> >>>>>>>>>>>>>>>>>> the
>> >>>>>>>>>>>>>>>>>> hot path, which explains why this commit has an
>> >> impact
>> >>>>>> on total
>> >>>>>>>>>>>>>>>>>> performance.
>> >>>>>>>>>>>>>>>>>> I've checked JIT assembly output, done some JMH
>> >>>>>> microbenchmarks,
>> >>>>>>>>>>>> and
>> >>>>>>>>>>>>>>>>>> found
>> >>>>>>>>>>>>>>>>>> that old implementation of MessageFactory.create()
>> >>>>>> about 30-35%
>> >>>>>>>>>>>> faster
>> >>>>>>>>>>>>>>>>>> than
>> >>>>>>>>>>>>>>>>>> the new one. The reason - approach with switch/case
>> >>> can
>> >>>>>>>>>>> effectively
>> >>>>>>>>>>>>>>>>>> inline
>> >>>>>>>>>>>>>>>>>> message creation code, but with an array of
>> >> suppliers
>> >>>>>> relatively
>> >>>>>>>>>>>> heavy
>> >>>>>>>>>>>>>>>>>> "invokeinterface" cannot be skipped. I've tried to
>> >>>>>> rewrite the
>> >>>>>>>>>>> code
>> >>>>>>>>>>>>>>>>>> using
>> >>>>>>>>>>>>>>>>>> an abstract class for suppliers instead of an
>> >>> interface
>> >>>>>> (to
>> >>>>>>>>>>>>>>>>>> replace "invokeinterface" with the "invokevirtual"),
>> >>>>>> but it gives
>> >>>>>>>>>>>> back
>> >>>>>>>>>>>>>>>>>> only
>> >>>>>>>>>>>>>>>>>> 10% of method performance and in this case, code
>> >> looks
>> >>>>>> ugly
>> >>>>>>>>>>>> (lambdas
>> >>>>>>>>>>>>>>>>>> can't
>> >>>>>>>>>>>>>>>>>> be used). Currently, I can't find any more ways to
>> >>>>>> optimize the
>> >>>>>>>>>>>> current
>> >>>>>>>>>>>>>>>>>> approach (except return to the switch/case block).
>> >>>>>> Andrey Gura,
>> >>>>>>>>>>> as
>> >>>>>>>>>>>> the
>> >>>>>>>>>>>>>>>>>> author of IGNITE-12568, maybe you have some ideas
>> >>> about
>> >>>>>>>>>>>> optimization?
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> Perhaps we should revert IGNITE-12568, but there are
>> >>>>>> some metrics
>> >>>>>>>>>>>>>>>>>> already
>> >>>>>>>>>>>>>>>>>> created, which can't be rewritten using old message
>> >>>>>> factory
>> >>>>>>>>>>>>>>>>>> implementation
>> >>>>>>>>>>>>>>>>>> (IGNITE-12756). Guys, WDYT?
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> Alexey,
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> I see that IGNITE-12756 (metrics improvements) is
>> >>>>>> already released
>> >>>>>>>>>>>> in
>> >>>>>>>>>>>>>>>>> Ignite 2.8.1 while IGNITE-12568 (message factory) is
>> >>>>>> only present
>> >>>>>>>>>>>> in Ignite
>> >>>>>>>>>>>>>>>>> 2.9. Let's revert both IGNITE-12568 and whichever new
>> >>>>>> metrics
>> >>>>>>>>>>>> created for
>> >>>>>>>>>>>>>>>>> 2.9 that depend on the new message factory to unblock
>> >>>>>> the release
>> >>>>>>>>>>>> and deal
>> >>>>>>>>>>>>>>>>> with the optimizations in 2.10?
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>
>> >>>>>
>> >>>
>> >>
>>
>>
Re: Re[2]: Apache Ignite 2.9.0 RELEASE [Time, Scope, Manager]
Posted by Vladislav Pyatkov <vl...@gmail.com>.
This patch [1] interesting only a deployment where clients often
reconnecting (stopping/starting) to cluster.
In these cases Coordinator node could fail by OOM, even the cluster did not
use MVCC caches.
On Mon, Oct 12, 2020 at 1:41 PM Zhenya Stanilovsky
<ar...@mail.ru.invalid> wrote:
>
>
> without [2] and [3] we obtain unexpected fields in index creation and as a
> consequence buggy sql execution and pla of course.
>
>
>
> >Guys,
> >
> >I've found 3 more tickets which were targeted to 2.9 but merged only to
> the
> >master branch ([1], [2], [3]).
> >Do we need these tickets in 2.9? Are they critical?
> >
> >[1]: https://issues.apache.org/jira/browse/IGNITE-12350
> >[2]: https://issues.apache.org/jira/browse/IGNITE-13376
> >[3]: https://issues.apache.org/jira/browse/IGNITE-13280
> >
> >вс, 11 окт. 2020 г. в 16:11, Nikolay Izhikov < nizhikov@apache.org >:
> >
> >> Igniters,
> >>
> >> IGNITE-13553 fixed and cherry-picked to 2.9.
> >> We can continue with the release.
> >>
> >> > 10 окт. 2020 г., в 00:00, Denis Magda < dmagda@apache.org >
> написал(а):
> >> >
> >> > Nikolay,
> >> >
> >> > re: the minor improvements. I'm not against of including those if we
> can
> >> > prepare the docs before the vote starts. Presently, the docs are
> "frozen"
> >> > for the 2.9 release, but I can scratch some time and take part in the
> >> docs
> >> > review the next week.
> >> >
> >> > -
> >> > Denis
> >> >
> >> >
> >> > On Fri, Oct 9, 2020 at 12:40 AM Nikolay Izhikov < nizhikov@apache.org
> >
> >> wrote:
> >> >
> >> >> Hello, Igniters.
> >> >>
> >> >> I prepared a patch [1] for blocker ticket [2] - «Server node fail and
> >> >> stops in case wrong datatype put in indexed field»
> >> >> Can someone, please, help me with the review?
> >> >>
> >> >> [1] https://github.com/apache/ignite/pull/8330
> >> >> [2] https://issues.apache.org/jira/browse/IGNITE-13553
> >> >>
> >> >> ==================
> >> >>
> >> >> I propose to include several minor tickets to the scope of the 2.9
> that
> >> >> increase Ignite User Experience
> >> >>
> >> >> CMD tools improvements:
> >> >>
> >> >> IGNITE-13488 - Command to print metric value
> >> >> IGNITE-13426 - Command to print system view content
> >> >> IGNITE-13422 - Parameter to explicitly enable experimental commands
> >> >>
> >> >> IGNITE-13380 - Output IgniteSystemProperties via ignite.sh
> >> >>
> >> >> New system views:
> >> >>
> >> >> IGNITE-13409 Metastorage and DistributedMetastorage viewы.
> >> >> IGNITE-13408 BinaryMetadata view.
> >> >>
> >> >>> 9 окт. 2020 г., в 04:04, Denis Magda < dmagda@apache.org >
> написал(а):
> >> >>>
> >> >>> @Alex Plehanov < plehanov.alex@gmail.com >,
> >> >>>
> >> >>> If it still makes sense and not too late, you can cherry-pick the
> >> commit
> >> >>> with the new docs into the 2.9 branch:
> >> >>>
> >> >>
> >>
> https://github.com/apache/ignite/commit/073488ac97517bbaad9f6b94b781fc404646f191
> >> >>>
> >> >>> Anyway, it's not crucial as long as we agreed to make an exception
> for
> >> >> this
> >> >>> release. The docs were already published to the website and
> assembled
> >> >> from
> >> >>> the master. Once the vote passes, I'll make them visible and
> traceable
> >> >> from
> >> >>> the website menus.
> >> >>>
> >> >>> -
> >> >>> Denis
> >> >>>
> >> >>>
> >> >>> On Tue, Oct 6, 2020 at 7:20 AM Alexey Goncharuk <
> >> >> alexey.goncharuk@gmail.com >
> >> >>> wrote:
> >> >>>
> >> >>>> Saikat,
> >> >>>>
> >> >>>> The plan sounds good to me! Do you have an idea for the timeline of
> >> the
> >> >>>> module releases? Let me know if I can help in any way.
> >> >>>>
> >> >>>> Also, I was looking into the modularization IEP and noticed that
> OSGI
> >> >>>> module is discussed to be moved to the extensions, but there is no
> >> >>>> corresponding ticket. Should I create one? I will be happy to help
> >> with
> >> >>>> moving this module to extensions.
> >> >>>>
> >> >>>> вт, 29 сент. 2020 г. в 03:03, Saikat Maitra <
> saikat.maitra@gmail.com
> >> >:
> >> >>>>
> >> >>>>> Hi Alexey,
> >> >>>>>
> >> >>>>> All the modules as planned in IEP-36
> >> >>>>>
> >> >>>>
> >> >>
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36:+Modularization
> >> >>>>> are migrated to ignite-extensions repository.
> >> >>>>>
> >> >>>>> We can definitely plan to release ignite-extensions modules.
> >> >>>>>
> >> >>>>> I have a pending PR related to the kafka module and the PR is in
> >> review
> >> >>>>> https://github.com/apache/ignite/pull/8222 but its
> corresponding PR
> >> >> has
> >> >>>>> been merged in ignite-extensions repository.
> >> >>>>>
> >> >>>>> My thoughts / action items for the migration efforts and releases
> >> were
> >> >>>>> as follows:
> >> >>>>>
> >> >>>>> 1. Merge the changes for
> https://github.com/apache/ignite/pull/8222
> >> >>>>> 2. Fix the upload nightly packages task for ignite-core to help
> fix
> >> the
> >> >>>>> travis build failure in ignite-extensions repository.
> >> >>>>> 3. Review and update the README.md files for the ignite-extensions
> >> >>>> modules
> >> >>>>> as required.
> >> >>>>> 4. Update the docs for ignite-extensions modules in
> ignite-website.
> >> >>>>> 5. Release each module separately and share updates.
> >> >>>>>
> >> >>>>> Please let me know if you have feedback.
> >> >>>>>
> >> >>>>> Regards,
> >> >>>>> Saikat
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> On Mon, Sep 28, 2020 at 7:36 AM Petr Ivanov < mr.weider@gmail.com
> >
> >> >> wrote:
> >> >>>>>
> >> >>>>>> It seems that gitbox.apache.org is not available on CI/CD.
> >> >>>>>>
> >> >>>>>> I will try to update VCS to GitHub.
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>> On 28 Sep 2020, at 14:07, Alex Plehanov <
> plehanov.alex@gmail.com >
> >> >>>>>> wrote:
> >> >>>>>>>
> >> >>>>>>> Guys,
> >> >>>>>>>
> >> >>>>>>> I've tried to build release candidate using TC [1]. But:
> >> >>>>>>> 1. I can't choose a branch in this TC task (there is no such
> >> field).
> >> >>>>>>> 2. On the "default" branch I got an error: Failed to collect
> >> changes,
> >> >>>>>>> error: List remote refs failed:
> >> >>>>>>> org.apache.http.conn.ConnectTimeoutException: Connect to
> >> >>>>>>> gitbox.apache.org:443 [gitbox.apache.org/52.202.80.70] failed:
> >> >>>> connect
> >> >>>>>>> timed out, VCS root: "GitBox [asf/ignite]" {instance id=300,
> parent
> >> >>>>>>> internal id=74, parent id=GitBoxAsfIgnite, description: "
> >> >>>>>>>
> https://gitbox.apache.org/repos/asf/ignite.git#refs/heads/master "}
> >> >>>>>>>
> >> >>>>>>> Is there some problem with this TC task? What am I doing wrong?
> >> >>>>>>>
> >> >>>>>>> [1] :
> >> >>>>>>>
> >> >>>>>>
> >> >>>>
> >> >>
> >>
> https://ci.ignite.apache.org/viewType.html?buildTypeId=Releases_ApacheIgniteMain_ReleaseBuild
> >> >>>>>>>
> >> >>>>>>> пн, 28 сент. 2020 г. в 13:15, Alexey Goncharuk <
> >> >>>>>> alexey.goncharuk@gmail.com >:
> >> >>>>>>>
> >> >>>>>>>> Saikat, Nikolay,
> >> >>>>>>>>
> >> >>>>>>>> We have migrated a bunch of modules to ignite-extensions
> recently.
> >> >>>>>> Given
> >> >>>>>>>> that these modules will not be available in Ignite 2.9 anymore
> >> (will
> >> >>>>>>>> they?), should we also plan to release the extensions?
> >> >>>>>>>>
> >> >>>>>>>> ср, 23 сент. 2020 г. в 21:49, Alex Plehanov <
> >> >> plehanov.alex@gmail.com
> >> >>>>> :
> >> >>>>>>>>
> >> >>>>>>>>> Igniters,
> >> >>>>>>>>>
> >> >>>>>>>>> I've prepared release notes for the 2.9 release [1]. Can
> anyone
> >> >>>> review
> >> >>>>>>>> it,
> >> >>>>>>>>> please?
> >> >>>>>>>>>
> >> >>>>>>>>> [1]: https://github.com/apache/ignite/pull/8273
> >> >>>>>>>>>
> >> >>>>>>>>> вт, 22 сент. 2020 г. в 09:39, Alex Plehanov <
> >> >>>> plehanov.alex@gmail.com
> >> >>>>>>> :
> >> >>>>>>>>>
> >> >>>>>>>>>> Guys,
> >> >>>>>>>>>>
> >> >>>>>>>>>> I've filled the ticket with reproducer [1] for the discovery
> >> bug.
> >> >>>>>> This
> >> >>>>>>>>> bug
> >> >>>>>>>>>> caused by [2] ticket. We discussed with Vladimir Steshin
> >> privately
> >> >>>>>> and
> >> >>>>>>>>>> decided to revert this ticket. I will do it today (after TC
> bot
> >> >>>> visa)
> >> >>>>>>>> if
> >> >>>>>>>>>> there are no objections.
> >> >>>>>>>>>>
> >> >>>>>>>>>> [1]: https://issues.apache.org/jira/browse/IGNITE-13465
> >> >>>>>>>>>> [2]: https://issues.apache.org/jira/browse/IGNITE-13134
> >> >>>>>>>>>>
> >> >>>>>>>>>> пн, 21 сент. 2020 г. в 11:08, Alex Plehanov <
> >> >>>> plehanov.alex@gmail.com
> >> >>>>>>> :
> >> >>>>>>>>>>
> >> >>>>>>>>>>> Guys,
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> During internal testing, we've found a critical bug with
> >> >>>>>>>>>>> discovery (cluster falls apart if two nodes segmented
> >> >>>> sequentially).
> >> >>>>>>>>> This
> >> >>>>>>>>>>> problem is not reproduced in 2.8.1. I think we should fix it
> >> >>>>>>>>>>> before release. Under investigation now. I'll let you know
> when
> >> >> we
> >> >>>>>> get
> >> >>>>>>>>>>> something.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> чт, 17 сент. 2020 г. в 00:51, Andrey Gura <
> agura@apache.org >:
> >> >>>>>>>>>>>
> >> >>>>>>>>>>>>> So what do you think? Should we proceed with a 'hacked'
> >> version
> >> >>>> of
> >> >>>>>>>>> the
> >> >>>>>>>>>>>> message factory in 2.9 and go for the runtime message
> >> generation
> >> >>>> in
> >> >>>>>>>>> later
> >> >>>>>>>>>>>> release, or keep the code clean and fix the regression in
> the
> >> >>>> next
> >> >>>>>>>>> releases?
> >> >>>>>>>>>>>>> Andrey, can you take a look at my change? I think it is
> >> fairly
> >> >>>>>>>>>>>> straightforward and does not change the semantics, just
> skips
> >> >> the
> >> >>>>>>>>> factory
> >> >>>>>>>>>>>> closures for certain messages.
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> IMHO 2.5% isn't too much especially because it isn't actual
> >> for
> >> >>>> all
> >> >>>>>>>>>>>> workloads (I didn't get any significant drops during
> >> >>>> benchmarking).
> >> >>>>>>>> So
> >> >>>>>>>>>>>> I prefer the runtime generation in later releases.
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> On Mon, Sep 14, 2020 at 12:41 PM Alexey Goncharuk
> >> >>>>>>>>>>>> < alexey.goncharuk@gmail.com > wrote:
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>> Alexey, Andrey, Igniters,
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>> So what do you think? Should we proceed with a 'hacked'
> >> version
> >> >>>> of
> >> >>>>>>>>> the
> >> >>>>>>>>>>>> message factory in 2.9 and go for the runtime message
> >> generation
> >> >>>> in
> >> >>>>>>>>> later
> >> >>>>>>>>>>>> release, or keep the code clean and fix the regression in
> the
> >> >>>> next
> >> >>>>>>>>> releases?
> >> >>>>>>>>>>>>> Andrey, can you take a look at my change? I think it is
> >> fairly
> >> >>>>>>>>>>>> straightforward and does not change the semantics, just
> skips
> >> >> the
> >> >>>>>>>>> factory
> >> >>>>>>>>>>>> closures for certain messages.
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>> Personally, I would prefer fixing the regression given
> that
> >> we
> >> >>>>>> also
> >> >>>>>>>>>>>> introduced tracing in this release.
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>> пт, 11 сент. 2020 г. в 12:09, Alex Plehanov <
> >> >>>>>>>> plehanov.alex@gmail.com
> >> >>>>>>>>>> :
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> Alexey,
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> We've benchmarked by yardstick commits 130376741bf vs
> >> >>>> ed52559eb95
> >> >>>>>>>>> and
> >> >>>>>>>>>>>> the performance of ed52559eb95 is better for about 2.5% on
> >> >>>> average
> >> >>>>>> on
> >> >>>>>>>>> our
> >> >>>>>>>>>>>> environment (about the same results we got benchmarking
> >> 65c30ec6
> >> >>>> vs
> >> >>>>>>>>>>>> 0606f03d). We've made 24 runs for each commit of
> >> >>>>>>>>>>>> IgnitePutTxImplicitBenchmark (we got maximum drop for 2.9
> on
> >> >> this
> >> >>>>>>>>>>>> benchmark), 200 seconds warmup, 300 seconds benchmark, 6
> >> >>>> servers, 5
> >> >>>>>>>>> clients
> >> >>>>>>>>>>>> 50 threads each. Yardstick results for this configuration:
> >> >>>>>>>>>>>>>> Commit 130376741bf: avg TPS=164096, avg latency=9173464
> ns
> >> >>>>>>>>>>>>>> Commit ed52559eb95: avg TPS=168283, avg latency=8945908
> ns
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> пт, 11 сент. 2020 г. в 09:51, Artem Budnikov <
> >> >>>>>>>>>>>> a.budnikov.ignite@gmail.com >:
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> Hi Everyone,
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> I posted an instruction on how to publish the docs on
> >> >>>>>>>>>>>> ignite.apache.org/docs [1]. When you finish with Ignite
> 2.9,
> >> >> you
> >> >>>>>> can
> >> >>>>>>>>>>>> update the docs by following the instruction.
> Unfortunately, I
> >> >>>>>> won't
> >> >>>>>>>> be
> >> >>>>>>>>>>>> able to spend any time on this project any longer. You can
> >> send
> >> >>>>>> your
> >> >>>>>>>>> pull
> >> >>>>>>>>>>>> requests and questions about the documentation to Denis
> Magda.
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> -Artem
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> [1] :
> >> >>>>>>>>>>>>
> >> >>>>
> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Document
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> On Thu, Sep 10, 2020 at 2:45 PM Alexey Goncharuk <
> >> >>>>>>>>>>>> alexey.goncharuk@gmail.com > wrote:
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> Alexey,
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> I've tried to play with message factories locally, but
> >> >>>>>>>>>>>> unfortunately, I
> >> >>>>>>>>>>>>>>>> cannot spot the difference between old and new
> >> >> implementation
> >> >>>>>> in
> >> >>>>>>>>>>>>>>>> distributed benchmarks. I pushed an implementation of
> >> >>>>>>>>>>>> MessageFactoryImpl
> >> >>>>>>>>>>>>>>>> with the old switch statement to the
> >> ignite-2.9-revert-12568
> >> >>>>>>>>> branch
> >> >>>>>>>>>>>>>>>> (discussed this with Andrey Gura, the change should be
> >> >>>>>>>> compatible
> >> >>>>>>>>>>>> with the
> >> >>>>>>>>>>>>>>>> new metrics as we still use the register() mechanics).
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> Can you check if this change makes any difference
> >> >>>>>>>> performance-wise
> >> >>>>>>>>>>>> in your
> >> >>>>>>>>>>>>>>>> environment? If yes, we can go with runtime code
> >> generation
> >> >>>> in
> >> >>>>>>>> the
> >> >>>>>>>>>>>> long
> >> >>>>>>>>>>>>>>>> term: register classes and generate a dynamic message
> >> >> factory
> >> >>>>>>>> with
> >> >>>>>>>>>>>> a switch
> >> >>>>>>>>>>>>>>>> statement once all messages are registered (not in 2.9
> >> >>>> though,
> >> >>>>>>>>>>>> obviously).
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> ср, 9 сент. 2020 г. в 14:53, Alex Plehanov <
> >> >>>>>>>>> plehanov.alex@gmail.com
> >> >>>>>>>>>>>>> :
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>> Hello guys,
> >> >>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>> I've tried to optimize tracing implementation (ticket
> >> [1]),
> >> >>>> it
> >> >>>>>>>>>>>> reduced the
> >> >>>>>>>>>>>>>>>>> drop, but not completely removed it.
> >> >>>>>>>>>>>>>>>>> Ivan Rakov, Alexander Lapin, can you please review the
> >> >>>> patch?
> >> >>>>>>>>>>>>>>>>> Ivan Artiukhov, can you please benchmark the patch [2]
> >> >>>> against
> >> >>>>>>>>>>>> 2.8.1
> >> >>>>>>>>>>>>>>>>> release on your environment?
> >> >>>>>>>>>>>>>>>>> With this patch on our environment, it's about a 3%
> drop
> >> >>>> left,
> >> >>>>>>>>>>>> it's close
> >> >>>>>>>>>>>>>>>>> to measurement error and I think such a drop is not a
> >> >>>>>>>>>>>> showstopper. Guys,
> >> >>>>>>>>>>>>>>>>> WDYT?
> >> >>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>> Also, I found that compatibility is broken for JDBC
> thin
> >> >>>>>>>> driver
> >> >>>>>>>>>>>> between 2.8
> >> >>>>>>>>>>>>>>>>> and 2.9 versions (ticket [3]). I think it's a blocker
> and
> >> >>>>>>>> should
> >> >>>>>>>>>>>> be
> >> >>>>>>>>>>>>>>>>> fixed in 2.9. I've prepared the patch.
> >> >>>>>>>>>>>>>>>>> Taras Ledkov, can you please review this patch?
> >> >>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>> And one more ticket I propose to include into 2.9 [4]
> >> (NIO
> >> >>>>>>>>> message
> >> >>>>>>>>>>>>>>>>> send problem in some circumstances). I will
> cherry-pick
> >> it
> >> >>>> if
> >> >>>>>>>>>>>> there is no
> >> >>>>>>>>>>>>>>>>> objection.
> >> >>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>> [1]
> https://issues.apache.org/jira/browse/IGNITE-13411
> >> >>>>>>>>>>>>>>>>> [2] https://github.com/apache/ignite/pull/8223
> >> >>>>>>>>>>>>>>>>> [3]
> https://issues.apache.org/jira/browse/IGNITE-13414
> >> >>>>>>>>>>>>>>>>> [4]
> https://issues.apache.org/jira/browse/IGNITE-13361
> >> >>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>> пн, 7 сент. 2020 г. в 14:14, Maxim Muzafarov <
> >> >>>>>>>> mmuzaf@apache.org
> >> >>>>>>>>>> :
> >> >>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>> Alexey,
> >> >>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>> I propose to include [1] issue to the 2.9 release.
> Since
> >> >>>>>>>> this
> >> >>>>>>>>>>>> issue is
> >> >>>>>>>>>>>>>>>>>> related to the new master key change functionality
> which
> >> >>>>>>>>>>>> haven't been
> >> >>>>>>>>>>>>>>>>>> released yet I think it will be safe to cherry-pick
> >> commit
> >> >>>>>>>> to
> >> >>>>>>>>>>>> the
> >> >>>>>>>>>>>>>>>>>> release branch.
> >> >>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>> [1]
> https://issues.apache.org/jira/browse/IGNITE-13390
> >> >>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>> On Tue, 1 Sep 2020 at 12:13, Nikolay Izhikov <
> >> >>>>>>>>>>>> nizhikov@apache.org >
> >> >>>>>>>>>>>>>>>>> wrote:
> >> >>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>> Hello, Igniters.
> >> >>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>> Alexey, please, include one more Python thin client
> fix
> >> >>>>>>>> [1]
> >> >>>>>>>>>>>> into the
> >> >>>>>>>>>>>>>>>>> 2.9
> >> >>>>>>>>>>>>>>>>>> release
> >> >>>>>>>>>>>>>>>>>>> It fixes kinda major issue - "Python client returns
> >> >> fields
> >> >>>>>>>>> in
> >> >>>>>>>>>>>> wrong
> >> >>>>>>>>>>>>>>>>>> order since the 2 row when fields_count>10"
> >> >>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>> [1]
> https://issues.apache.org/jira/browse/IGNITE-12809
> >> >>>>>>>>>>>>>>>>>>> [2]
> >> >>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>
> >> >>>>>>>>>
> >> >>>>>>>>
> >> >>>>>>
> >> >>>>
> >> >>
> >>
> https://github.com/apache/ignite/commit/38025ee4167f05eaa2d6a2c5c2ab70c83a462cfc
> >> >>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>> 31 авг. 2020 г., в 19:23, Alexey Goncharuk <
> >> >>>>>>>>>>>>>>>>> alexey.goncharuk@gmail.com >
> >> >>>>>>>>>>>>>>>>>> написал(а):
> >> >>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>> Alexey, thanks, got it. I am not sure we can
> optimize
> >> >>>>>>>>>>>> anything out of
> >> >>>>>>>>>>>>>>>>>> the
> >> >>>>>>>>>>>>>>>>>>>> message factory with suppliers (at least I have no
> >> ideas
> >> >>>>>>>>>>>> right now),
> >> >>>>>>>>>>>>>>>>> so
> >> >>>>>>>>>>>>>>>>>>>> most likely the only move here is to switch back to
> >> the
> >> >>>>>>>>>>>> switch
> >> >>>>>>>>>>>>>>>>> approach
> >> >>>>>>>>>>>>>>>>>>>> somehow preserving the metrics part. Probably,
> >> inlining
> >> >>>>>>>>> the
> >> >>>>>>>>>>>> Ignite
> >> >>>>>>>>>>>>>>>>>> messages
> >> >>>>>>>>>>>>>>>>>>>> to the IgniteMessageFactoryImpl should do the
> trick.
> >> Let
> >> >>>>>>>>> me
> >> >>>>>>>>>>>> explore
> >> >>>>>>>>>>>>>>>>> the
> >> >>>>>>>>>>>>>>>>>>>> code a bit.
> >> >>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>> P.S. I am surprised by the impact this part makes
> for
> >> >>>>>>>> the
> >> >>>>>>>>>>>>>>>>> performance.
> >> >>>>>>>>>>>>>>>>>>>> Message creation is indeed on the hot path, but a
> >> single
> >> >>>>>>>>>>>> virtual call
> >> >>>>>>>>>>>>>>>>>>>> should not make that much of a difference given the
> >> >>>>>>>> amount
> >> >>>>>>>>>>>> of other
> >> >>>>>>>>>>>>>>>>>> work
> >> >>>>>>>>>>>>>>>>>>>> happening during the message processing.
> >> >>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 18:33, Alex Plehanov <
> >> >>>>>>>>>>>> plehanov.alex@gmail.com
> >> >>>>>>>>>>>>>>>>>> :
> >> >>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>> Alexey, sorry, I wrongly interpreted our benchmark
> >> >>>>>>>>> results.
> >> >>>>>>>>>>>>>>>>> Actually,
> >> >>>>>>>>>>>>>>>>>> we
> >> >>>>>>>>>>>>>>>>>>>>> were looking for a drop using bi-sect in the range
> >> >>>>>>>>> between
> >> >>>>>>>>>>>> e6a7f93
> >> >>>>>>>>>>>>>>>>>> (first
> >> >>>>>>>>>>>>>>>>>>>>> commit in the 2.9 branch after 2.8 branch cut) and
> >> >>>>>>>>>>>> 6592dfa5 (last
> >> >>>>>>>>>>>>>>>>>> commit in
> >> >>>>>>>>>>>>>>>>>>>>> the 2.9 branch). And we found these two
> problematic
> >> >>>>>>>>>>>> commits.
> >> >>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>> Perhaps only IGNITE-13060 (Tracing) is responsible
> >> for
> >> >>>>>>>> a
> >> >>>>>>>>>>>> drop
> >> >>>>>>>>>>>>>>>>> between
> >> >>>>>>>>>>>>>>>>>>>>> 2.8.1 and 2.9 (we have benchmarked 2.8.1 vs 2.9
> with
> >> >>>>>>>>>>>> reverted
> >> >>>>>>>>>>>>>>>>>> IGNITE-13060
> >> >>>>>>>>>>>>>>>>>>>>> now and performance looks the same)
> >> >>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>> Ticket IGNITE-12568 (MessageFactory refactoring)
> is
> >> not
> >> >>>>>>>>>>>> related to
> >> >>>>>>>>>>>>>>>>>> drop
> >> >>>>>>>>>>>>>>>>>>>>> between 2.8.1 and 2.9, but still has some
> performance
> >> >>>>>>>>>>>> problem, and
> >> >>>>>>>>>>>>>>>>> we
> >> >>>>>>>>>>>>>>>>>> can
> >> >>>>>>>>>>>>>>>>>>>>> win back IGNITE-13060 drop by this ticket.
> >> >>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>> Do we need more investigation on IGNITE-13060 or
> we
> >> >>>>>>>> leave
> >> >>>>>>>>>>>> it as is?
> >> >>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>> What should we do with IGNITE-12568
> (MessageFactory
> >> >>>>>>>>>>>> refactoring)?
> >> >>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 13:25, Alexey Goncharuk <
> >> >>>>>>>>>>>>>>>>>> alexey.goncharuk@gmail.com
> >> >>>>>>>>>>>>>>>>>>>>>> :
> >> >>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>> Alexey,
> >> >>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>> While investigating, I found that IGNITE-12568
> has
> >> an
> >> >>>>>>>>>>>> incorrect fix
> >> >>>>>>>>>>>>>>>>>>>>>> version and is actually present in ignite-2.8.1
> >> branch
> >> >>>>>>>>>>>> [1], so it
> >> >>>>>>>>>>>>>>>>>> cannot be
> >> >>>>>>>>>>>>>>>>>>>>>> the source of the drop against 2.8.1.
> >> >>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>> P.S. Looks like we need to enforce a more
> accurate
> >> >>>>>>>> work
> >> >>>>>>>>>>>> with fix
> >> >>>>>>>>>>>>>>>>>> versions
> >> >>>>>>>>>>>>>>>>>>>>>> or develop some sort of tooling to verify the fix
> >> >>>>>>>>>>>> versions.
> >> >>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>> --AG
> >> >>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>> [1]
> >> >>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>
> >> >>>>>>>>>
> >> >>>>>>>>
> >> >>>>>>
> >> >>>>
> >> >>
> >>
> https://github.com/apache/ignite/commit/3e492bd23851856bbd0385c6a419892d0bba2a34
> >> >>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 12:42, Alexey Goncharuk <
> >> >>>>>>>>>>>>>>>>>> alexey.goncharuk@gmail.com
> >> >>>>>>>>>>>>>>>>>>>>>>> :
> >> >>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>> пт, 28 авг. 2020 г. в 11:16, Alex Plehanov <
> >> >>>>>>>>>>>>>>>>> plehanov.alex@gmail.com
> >> >>>>>>>>>>>>>>>>>>> :
> >> >>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>>> Guys,
> >> >>>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>>> We have benchmarked 2.9 without IGNITE-13060
> and
> >> >>>>>>>>>>>> IGNITE-12568
> >> >>>>>>>>>>>>>>>>>> (reverted
> >> >>>>>>>>>>>>>>>>>>>>>>>> it
> >> >>>>>>>>>>>>>>>>>>>>>>>> locally) and got the same performance as on
> 2.8.1
> >> >>>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>>> IGNITE-13060 (Tracing) - some code was added to
> >> hot
> >> >>>>>>>>>>>> paths, to
> >> >>>>>>>>>>>>>>>>> trace
> >> >>>>>>>>>>>>>>>>>>>>>>>> these
> >> >>>>>>>>>>>>>>>>>>>>>>>> hot paths, it's clear why we have performance
> drop
> >> >>>>>>>>> here.
> >> >>>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>>> IGNITE-12568 (MessageFactory refactoring) -
> >> >>>>>>>>> switch/case
> >> >>>>>>>>>>>> block was
> >> >>>>>>>>>>>>>>>>>>>>>>>> refactored to an array of message suppliers.
> The
> >> >>>>>>>>>>>> message factory
> >> >>>>>>>>>>>>>>>>>> is on
> >> >>>>>>>>>>>>>>>>>>>>>>>> the
> >> >>>>>>>>>>>>>>>>>>>>>>>> hot path, which explains why this commit has an
> >> >>>>>>>> impact
> >> >>>>>>>>>>>> on total
> >> >>>>>>>>>>>>>>>>>>>>>>>> performance.
> >> >>>>>>>>>>>>>>>>>>>>>>>> I've checked JIT assembly output, done some JMH
> >> >>>>>>>>>>>> microbenchmarks,
> >> >>>>>>>>>>>>>>>>>> and
> >> >>>>>>>>>>>>>>>>>>>>>>>> found
> >> >>>>>>>>>>>>>>>>>>>>>>>> that old implementation of
> MessageFactory.create()
> >> >>>>>>>>>>>> about 30-35%
> >> >>>>>>>>>>>>>>>>>> faster
> >> >>>>>>>>>>>>>>>>>>>>>>>> than
> >> >>>>>>>>>>>>>>>>>>>>>>>> the new one. The reason - approach with
> >> switch/case
> >> >>>>>>>>> can
> >> >>>>>>>>>>>>>>>>> effectively
> >> >>>>>>>>>>>>>>>>>>>>>>>> inline
> >> >>>>>>>>>>>>>>>>>>>>>>>> message creation code, but with an array of
> >> >>>>>>>> suppliers
> >> >>>>>>>>>>>> relatively
> >> >>>>>>>>>>>>>>>>>> heavy
> >> >>>>>>>>>>>>>>>>>>>>>>>> "invokeinterface" cannot be skipped. I've
> tried to
> >> >>>>>>>>>>>> rewrite the
> >> >>>>>>>>>>>>>>>>> code
> >> >>>>>>>>>>>>>>>>>>>>>>>> using
> >> >>>>>>>>>>>>>>>>>>>>>>>> an abstract class for suppliers instead of an
> >> >>>>>>>>> interface
> >> >>>>>>>>>>>> (to
> >> >>>>>>>>>>>>>>>>>>>>>>>> replace "invokeinterface" with the
> >> "invokevirtual"),
> >> >>>>>>>>>>>> but it gives
> >> >>>>>>>>>>>>>>>>>> back
> >> >>>>>>>>>>>>>>>>>>>>>>>> only
> >> >>>>>>>>>>>>>>>>>>>>>>>> 10% of method performance and in this case,
> code
> >> >>>>>>>> looks
> >> >>>>>>>>>>>> ugly
> >> >>>>>>>>>>>>>>>>>> (lambdas
> >> >>>>>>>>>>>>>>>>>>>>>>>> can't
> >> >>>>>>>>>>>>>>>>>>>>>>>> be used). Currently, I can't find any more
> ways to
> >> >>>>>>>>>>>> optimize the
> >> >>>>>>>>>>>>>>>>>> current
> >> >>>>>>>>>>>>>>>>>>>>>>>> approach (except return to the switch/case
> block).
> >> >>>>>>>>>>>> Andrey Gura,
> >> >>>>>>>>>>>>>>>>> as
> >> >>>>>>>>>>>>>>>>>> the
> >> >>>>>>>>>>>>>>>>>>>>>>>> author of IGNITE-12568, maybe you have some
> ideas
> >> >>>>>>>>> about
> >> >>>>>>>>>>>>>>>>>> optimization?
> >> >>>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>>> Perhaps we should revert IGNITE-12568, but
> there
> >> are
> >> >>>>>>>>>>>> some metrics
> >> >>>>>>>>>>>>>>>>>>>>>>>> already
> >> >>>>>>>>>>>>>>>>>>>>>>>> created, which can't be rewritten using old
> >> message
> >> >>>>>>>>>>>> factory
> >> >>>>>>>>>>>>>>>>>>>>>>>> implementation
> >> >>>>>>>>>>>>>>>>>>>>>>>> (IGNITE-12756). Guys, WDYT?
> >> >>>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>> Alexey,
> >> >>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>> I see that IGNITE-12756 (metrics improvements)
> is
> >> >>>>>>>>>>>> already released
> >> >>>>>>>>>>>>>>>>>> in
> >> >>>>>>>>>>>>>>>>>>>>>>> Ignite 2.8.1 while IGNITE-12568 (message
> factory)
> >> is
> >> >>>>>>>>>>>> only present
> >> >>>>>>>>>>>>>>>>>> in Ignite
> >> >>>>>>>>>>>>>>>>>>>>>>> 2.9. Let's revert both IGNITE-12568 and
> whichever
> >> new
> >> >>>>>>>>>>>> metrics
> >> >>>>>>>>>>>>>>>>>> created for
> >> >>>>>>>>>>>>>>>>>>>>>>> 2.9 that depend on the new message factory to
> >> unblock
> >> >>>>>>>>>>>> the release
> >> >>>>>>>>>>>>>>>>>> and deal
> >> >>>>>>>>>>>>>>>>>>>>>>> with the optimizations in 2.10?
> >> >>>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>
> >> >>>>>>>>>
> >> >>>>>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>
> >> >>
> >> >>
> >>
> >>
>
--
Vladislav Pyatkov
Re[2]: Apache Ignite 2.9.0 RELEASE [Time, Scope, Manager]
Posted by Zhenya Stanilovsky <ar...@mail.ru.INVALID>.
without [2] and [3] we obtain unexpected fields in index creation and as a consequence buggy sql execution and pla of course.
>Guys,
>
>I've found 3 more tickets which were targeted to 2.9 but merged only to the
>master branch ([1], [2], [3]).
>Do we need these tickets in 2.9? Are they critical?
>
>[1]: https://issues.apache.org/jira/browse/IGNITE-12350
>[2]: https://issues.apache.org/jira/browse/IGNITE-13376
>[3]: https://issues.apache.org/jira/browse/IGNITE-13280
>
>вс, 11 окт. 2020 г. в 16:11, Nikolay Izhikov < nizhikov@apache.org >:
>
>> Igniters,
>>
>> IGNITE-13553 fixed and cherry-picked to 2.9.
>> We can continue with the release.
>>
>> > 10 окт. 2020 г., в 00:00, Denis Magda < dmagda@apache.org > написал(а):
>> >
>> > Nikolay,
>> >
>> > re: the minor improvements. I'm not against of including those if we can
>> > prepare the docs before the vote starts. Presently, the docs are "frozen"
>> > for the 2.9 release, but I can scratch some time and take part in the
>> docs
>> > review the next week.
>> >
>> > -
>> > Denis
>> >
>> >
>> > On Fri, Oct 9, 2020 at 12:40 AM Nikolay Izhikov < nizhikov@apache.org >
>> wrote:
>> >
>> >> Hello, Igniters.
>> >>
>> >> I prepared a patch [1] for blocker ticket [2] - «Server node fail and
>> >> stops in case wrong datatype put in indexed field»
>> >> Can someone, please, help me with the review?
>> >>
>> >> [1] https://github.com/apache/ignite/pull/8330
>> >> [2] https://issues.apache.org/jira/browse/IGNITE-13553
>> >>
>> >> ==================
>> >>
>> >> I propose to include several minor tickets to the scope of the 2.9 that
>> >> increase Ignite User Experience
>> >>
>> >> CMD tools improvements:
>> >>
>> >> IGNITE-13488 - Command to print metric value
>> >> IGNITE-13426 - Command to print system view content
>> >> IGNITE-13422 - Parameter to explicitly enable experimental commands
>> >>
>> >> IGNITE-13380 - Output IgniteSystemProperties via ignite.sh
>> >>
>> >> New system views:
>> >>
>> >> IGNITE-13409 Metastorage and DistributedMetastorage viewы.
>> >> IGNITE-13408 BinaryMetadata view.
>> >>
>> >>> 9 окт. 2020 г., в 04:04, Denis Magda < dmagda@apache.org > написал(а):
>> >>>
>> >>> @Alex Plehanov < plehanov.alex@gmail.com >,
>> >>>
>> >>> If it still makes sense and not too late, you can cherry-pick the
>> commit
>> >>> with the new docs into the 2.9 branch:
>> >>>
>> >>
>> https://github.com/apache/ignite/commit/073488ac97517bbaad9f6b94b781fc404646f191
>> >>>
>> >>> Anyway, it's not crucial as long as we agreed to make an exception for
>> >> this
>> >>> release. The docs were already published to the website and assembled
>> >> from
>> >>> the master. Once the vote passes, I'll make them visible and traceable
>> >> from
>> >>> the website menus.
>> >>>
>> >>> -
>> >>> Denis
>> >>>
>> >>>
>> >>> On Tue, Oct 6, 2020 at 7:20 AM Alexey Goncharuk <
>> >> alexey.goncharuk@gmail.com >
>> >>> wrote:
>> >>>
>> >>>> Saikat,
>> >>>>
>> >>>> The plan sounds good to me! Do you have an idea for the timeline of
>> the
>> >>>> module releases? Let me know if I can help in any way.
>> >>>>
>> >>>> Also, I was looking into the modularization IEP and noticed that OSGI
>> >>>> module is discussed to be moved to the extensions, but there is no
>> >>>> corresponding ticket. Should I create one? I will be happy to help
>> with
>> >>>> moving this module to extensions.
>> >>>>
>> >>>> вт, 29 сент. 2020 г. в 03:03, Saikat Maitra < saikat.maitra@gmail.com
>> >:
>> >>>>
>> >>>>> Hi Alexey,
>> >>>>>
>> >>>>> All the modules as planned in IEP-36
>> >>>>>
>> >>>>
>> >>
>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36:+Modularization
>> >>>>> are migrated to ignite-extensions repository.
>> >>>>>
>> >>>>> We can definitely plan to release ignite-extensions modules.
>> >>>>>
>> >>>>> I have a pending PR related to the kafka module and the PR is in
>> review
>> >>>>> https://github.com/apache/ignite/pull/8222 but its corresponding PR
>> >> has
>> >>>>> been merged in ignite-extensions repository.
>> >>>>>
>> >>>>> My thoughts / action items for the migration efforts and releases
>> were
>> >>>>> as follows:
>> >>>>>
>> >>>>> 1. Merge the changes for https://github.com/apache/ignite/pull/8222
>> >>>>> 2. Fix the upload nightly packages task for ignite-core to help fix
>> the
>> >>>>> travis build failure in ignite-extensions repository.
>> >>>>> 3. Review and update the README.md files for the ignite-extensions
>> >>>> modules
>> >>>>> as required.
>> >>>>> 4. Update the docs for ignite-extensions modules in ignite-website.
>> >>>>> 5. Release each module separately and share updates.
>> >>>>>
>> >>>>> Please let me know if you have feedback.
>> >>>>>
>> >>>>> Regards,
>> >>>>> Saikat
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> On Mon, Sep 28, 2020 at 7:36 AM Petr Ivanov < mr.weider@gmail.com >
>> >> wrote:
>> >>>>>
>> >>>>>> It seems that gitbox.apache.org is not available on CI/CD.
>> >>>>>>
>> >>>>>> I will try to update VCS to GitHub.
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>> On 28 Sep 2020, at 14:07, Alex Plehanov < plehanov.alex@gmail.com >
>> >>>>>> wrote:
>> >>>>>>>
>> >>>>>>> Guys,
>> >>>>>>>
>> >>>>>>> I've tried to build release candidate using TC [1]. But:
>> >>>>>>> 1. I can't choose a branch in this TC task (there is no such
>> field).
>> >>>>>>> 2. On the "default" branch I got an error: Failed to collect
>> changes,
>> >>>>>>> error: List remote refs failed:
>> >>>>>>> org.apache.http.conn.ConnectTimeoutException: Connect to
>> >>>>>>> gitbox.apache.org:443 [gitbox.apache.org/52.202.80.70] failed:
>> >>>> connect
>> >>>>>>> timed out, VCS root: "GitBox [asf/ignite]" {instance id=300, parent
>> >>>>>>> internal id=74, parent id=GitBoxAsfIgnite, description: "
>> >>>>>>> https://gitbox.apache.org/repos/asf/ignite.git#refs/heads/master "}
>> >>>>>>>
>> >>>>>>> Is there some problem with this TC task? What am I doing wrong?
>> >>>>>>>
>> >>>>>>> [1] :
>> >>>>>>>
>> >>>>>>
>> >>>>
>> >>
>> https://ci.ignite.apache.org/viewType.html?buildTypeId=Releases_ApacheIgniteMain_ReleaseBuild
>> >>>>>>>
>> >>>>>>> пн, 28 сент. 2020 г. в 13:15, Alexey Goncharuk <
>> >>>>>> alexey.goncharuk@gmail.com >:
>> >>>>>>>
>> >>>>>>>> Saikat, Nikolay,
>> >>>>>>>>
>> >>>>>>>> We have migrated a bunch of modules to ignite-extensions recently.
>> >>>>>> Given
>> >>>>>>>> that these modules will not be available in Ignite 2.9 anymore
>> (will
>> >>>>>>>> they?), should we also plan to release the extensions?
>> >>>>>>>>
>> >>>>>>>> ср, 23 сент. 2020 г. в 21:49, Alex Plehanov <
>> >> plehanov.alex@gmail.com
>> >>>>> :
>> >>>>>>>>
>> >>>>>>>>> Igniters,
>> >>>>>>>>>
>> >>>>>>>>> I've prepared release notes for the 2.9 release [1]. Can anyone
>> >>>> review
>> >>>>>>>> it,
>> >>>>>>>>> please?
>> >>>>>>>>>
>> >>>>>>>>> [1]: https://github.com/apache/ignite/pull/8273
>> >>>>>>>>>
>> >>>>>>>>> вт, 22 сент. 2020 г. в 09:39, Alex Plehanov <
>> >>>> plehanov.alex@gmail.com
>> >>>>>>> :
>> >>>>>>>>>
>> >>>>>>>>>> Guys,
>> >>>>>>>>>>
>> >>>>>>>>>> I've filled the ticket with reproducer [1] for the discovery
>> bug.
>> >>>>>> This
>> >>>>>>>>> bug
>> >>>>>>>>>> caused by [2] ticket. We discussed with Vladimir Steshin
>> privately
>> >>>>>> and
>> >>>>>>>>>> decided to revert this ticket. I will do it today (after TC bot
>> >>>> visa)
>> >>>>>>>> if
>> >>>>>>>>>> there are no objections.
>> >>>>>>>>>>
>> >>>>>>>>>> [1]: https://issues.apache.org/jira/browse/IGNITE-13465
>> >>>>>>>>>> [2]: https://issues.apache.org/jira/browse/IGNITE-13134
>> >>>>>>>>>>
>> >>>>>>>>>> пн, 21 сент. 2020 г. в 11:08, Alex Plehanov <
>> >>>> plehanov.alex@gmail.com
>> >>>>>>> :
>> >>>>>>>>>>
>> >>>>>>>>>>> Guys,
>> >>>>>>>>>>>
>> >>>>>>>>>>> During internal testing, we've found a critical bug with
>> >>>>>>>>>>> discovery (cluster falls apart if two nodes segmented
>> >>>> sequentially).
>> >>>>>>>>> This
>> >>>>>>>>>>> problem is not reproduced in 2.8.1. I think we should fix it
>> >>>>>>>>>>> before release. Under investigation now. I'll let you know when
>> >> we
>> >>>>>> get
>> >>>>>>>>>>> something.
>> >>>>>>>>>>>
>> >>>>>>>>>>> чт, 17 сент. 2020 г. в 00:51, Andrey Gura < agura@apache.org >:
>> >>>>>>>>>>>
>> >>>>>>>>>>>>> So what do you think? Should we proceed with a 'hacked'
>> version
>> >>>> of
>> >>>>>>>>> the
>> >>>>>>>>>>>> message factory in 2.9 and go for the runtime message
>> generation
>> >>>> in
>> >>>>>>>>> later
>> >>>>>>>>>>>> release, or keep the code clean and fix the regression in the
>> >>>> next
>> >>>>>>>>> releases?
>> >>>>>>>>>>>>> Andrey, can you take a look at my change? I think it is
>> fairly
>> >>>>>>>>>>>> straightforward and does not change the semantics, just skips
>> >> the
>> >>>>>>>>> factory
>> >>>>>>>>>>>> closures for certain messages.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> IMHO 2.5% isn't too much especially because it isn't actual
>> for
>> >>>> all
>> >>>>>>>>>>>> workloads (I didn't get any significant drops during
>> >>>> benchmarking).
>> >>>>>>>> So
>> >>>>>>>>>>>> I prefer the runtime generation in later releases.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> On Mon, Sep 14, 2020 at 12:41 PM Alexey Goncharuk
>> >>>>>>>>>>>> < alexey.goncharuk@gmail.com > wrote:
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Alexey, Andrey, Igniters,
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> So what do you think? Should we proceed with a 'hacked'
>> version
>> >>>> of
>> >>>>>>>>> the
>> >>>>>>>>>>>> message factory in 2.9 and go for the runtime message
>> generation
>> >>>> in
>> >>>>>>>>> later
>> >>>>>>>>>>>> release, or keep the code clean and fix the regression in the
>> >>>> next
>> >>>>>>>>> releases?
>> >>>>>>>>>>>>> Andrey, can you take a look at my change? I think it is
>> fairly
>> >>>>>>>>>>>> straightforward and does not change the semantics, just skips
>> >> the
>> >>>>>>>>> factory
>> >>>>>>>>>>>> closures for certain messages.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Personally, I would prefer fixing the regression given that
>> we
>> >>>>>> also
>> >>>>>>>>>>>> introduced tracing in this release.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> пт, 11 сент. 2020 г. в 12:09, Alex Plehanov <
>> >>>>>>>> plehanov.alex@gmail.com
>> >>>>>>>>>> :
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Alexey,
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> We've benchmarked by yardstick commits 130376741bf vs
>> >>>> ed52559eb95
>> >>>>>>>>> and
>> >>>>>>>>>>>> the performance of ed52559eb95 is better for about 2.5% on
>> >>>> average
>> >>>>>> on
>> >>>>>>>>> our
>> >>>>>>>>>>>> environment (about the same results we got benchmarking
>> 65c30ec6
>> >>>> vs
>> >>>>>>>>>>>> 0606f03d). We've made 24 runs for each commit of
>> >>>>>>>>>>>> IgnitePutTxImplicitBenchmark (we got maximum drop for 2.9 on
>> >> this
>> >>>>>>>>>>>> benchmark), 200 seconds warmup, 300 seconds benchmark, 6
>> >>>> servers, 5
>> >>>>>>>>> clients
>> >>>>>>>>>>>> 50 threads each. Yardstick results for this configuration:
>> >>>>>>>>>>>>>> Commit 130376741bf: avg TPS=164096, avg latency=9173464 ns
>> >>>>>>>>>>>>>> Commit ed52559eb95: avg TPS=168283, avg latency=8945908 ns
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> пт, 11 сент. 2020 г. в 09:51, Artem Budnikov <
>> >>>>>>>>>>>> a.budnikov.ignite@gmail.com >:
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Hi Everyone,
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> I posted an instruction on how to publish the docs on
>> >>>>>>>>>>>> ignite.apache.org/docs [1]. When you finish with Ignite 2.9,
>> >> you
>> >>>>>> can
>> >>>>>>>>>>>> update the docs by following the instruction. Unfortunately, I
>> >>>>>> won't
>> >>>>>>>> be
>> >>>>>>>>>>>> able to spend any time on this project any longer. You can
>> send
>> >>>>>> your
>> >>>>>>>>> pull
>> >>>>>>>>>>>> requests and questions about the documentation to Denis Magda.
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> -Artem
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> [1] :
>> >>>>>>>>>>>>
>> >>>> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Document
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> On Thu, Sep 10, 2020 at 2:45 PM Alexey Goncharuk <
>> >>>>>>>>>>>> alexey.goncharuk@gmail.com > wrote:
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> Alexey,
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> I've tried to play with message factories locally, but
>> >>>>>>>>>>>> unfortunately, I
>> >>>>>>>>>>>>>>>> cannot spot the difference between old and new
>> >> implementation
>> >>>>>> in
>> >>>>>>>>>>>>>>>> distributed benchmarks. I pushed an implementation of
>> >>>>>>>>>>>> MessageFactoryImpl
>> >>>>>>>>>>>>>>>> with the old switch statement to the
>> ignite-2.9-revert-12568
>> >>>>>>>>> branch
>> >>>>>>>>>>>>>>>> (discussed this with Andrey Gura, the change should be
>> >>>>>>>> compatible
>> >>>>>>>>>>>> with the
>> >>>>>>>>>>>>>>>> new metrics as we still use the register() mechanics).
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> Can you check if this change makes any difference
>> >>>>>>>> performance-wise
>> >>>>>>>>>>>> in your
>> >>>>>>>>>>>>>>>> environment? If yes, we can go with runtime code
>> generation
>> >>>> in
>> >>>>>>>> the
>> >>>>>>>>>>>> long
>> >>>>>>>>>>>>>>>> term: register classes and generate a dynamic message
>> >> factory
>> >>>>>>>> with
>> >>>>>>>>>>>> a switch
>> >>>>>>>>>>>>>>>> statement once all messages are registered (not in 2.9
>> >>>> though,
>> >>>>>>>>>>>> obviously).
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> ср, 9 сент. 2020 г. в 14:53, Alex Plehanov <
>> >>>>>>>>> plehanov.alex@gmail.com
>> >>>>>>>>>>>>> :
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> Hello guys,
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> I've tried to optimize tracing implementation (ticket
>> [1]),
>> >>>> it
>> >>>>>>>>>>>> reduced the
>> >>>>>>>>>>>>>>>>> drop, but not completely removed it.
>> >>>>>>>>>>>>>>>>> Ivan Rakov, Alexander Lapin, can you please review the
>> >>>> patch?
>> >>>>>>>>>>>>>>>>> Ivan Artiukhov, can you please benchmark the patch [2]
>> >>>> against
>> >>>>>>>>>>>> 2.8.1
>> >>>>>>>>>>>>>>>>> release on your environment?
>> >>>>>>>>>>>>>>>>> With this patch on our environment, it's about a 3% drop
>> >>>> left,
>> >>>>>>>>>>>> it's close
>> >>>>>>>>>>>>>>>>> to measurement error and I think such a drop is not a
>> >>>>>>>>>>>> showstopper. Guys,
>> >>>>>>>>>>>>>>>>> WDYT?
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> Also, I found that compatibility is broken for JDBC thin
>> >>>>>>>> driver
>> >>>>>>>>>>>> between 2.8
>> >>>>>>>>>>>>>>>>> and 2.9 versions (ticket [3]). I think it's a blocker and
>> >>>>>>>> should
>> >>>>>>>>>>>> be
>> >>>>>>>>>>>>>>>>> fixed in 2.9. I've prepared the patch.
>> >>>>>>>>>>>>>>>>> Taras Ledkov, can you please review this patch?
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> And one more ticket I propose to include into 2.9 [4]
>> (NIO
>> >>>>>>>>> message
>> >>>>>>>>>>>>>>>>> send problem in some circumstances). I will cherry-pick
>> it
>> >>>> if
>> >>>>>>>>>>>> there is no
>> >>>>>>>>>>>>>>>>> objection.
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13411
>> >>>>>>>>>>>>>>>>> [2] https://github.com/apache/ignite/pull/8223
>> >>>>>>>>>>>>>>>>> [3] https://issues.apache.org/jira/browse/IGNITE-13414
>> >>>>>>>>>>>>>>>>> [4] https://issues.apache.org/jira/browse/IGNITE-13361
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> пн, 7 сент. 2020 г. в 14:14, Maxim Muzafarov <
>> >>>>>>>> mmuzaf@apache.org
>> >>>>>>>>>> :
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> Alexey,
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> I propose to include [1] issue to the 2.9 release. Since
>> >>>>>>>> this
>> >>>>>>>>>>>> issue is
>> >>>>>>>>>>>>>>>>>> related to the new master key change functionality which
>> >>>>>>>>>>>> haven't been
>> >>>>>>>>>>>>>>>>>> released yet I think it will be safe to cherry-pick
>> commit
>> >>>>>>>> to
>> >>>>>>>>>>>> the
>> >>>>>>>>>>>>>>>>>> release branch.
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13390
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> On Tue, 1 Sep 2020 at 12:13, Nikolay Izhikov <
>> >>>>>>>>>>>> nizhikov@apache.org >
>> >>>>>>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> Hello, Igniters.
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> Alexey, please, include one more Python thin client fix
>> >>>>>>>> [1]
>> >>>>>>>>>>>> into the
>> >>>>>>>>>>>>>>>>> 2.9
>> >>>>>>>>>>>>>>>>>> release
>> >>>>>>>>>>>>>>>>>>> It fixes kinda major issue - "Python client returns
>> >> fields
>> >>>>>>>>> in
>> >>>>>>>>>>>> wrong
>> >>>>>>>>>>>>>>>>>> order since the 2 row when fields_count>10"
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-12809
>> >>>>>>>>>>>>>>>>>>> [2]
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>>>
>> >>
>> https://github.com/apache/ignite/commit/38025ee4167f05eaa2d6a2c5c2ab70c83a462cfc
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>> 31 авг. 2020 г., в 19:23, Alexey Goncharuk <
>> >>>>>>>>>>>>>>>>> alexey.goncharuk@gmail.com >
>> >>>>>>>>>>>>>>>>>> написал(а):
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>> Alexey, thanks, got it. I am not sure we can optimize
>> >>>>>>>>>>>> anything out of
>> >>>>>>>>>>>>>>>>>> the
>> >>>>>>>>>>>>>>>>>>>> message factory with suppliers (at least I have no
>> ideas
>> >>>>>>>>>>>> right now),
>> >>>>>>>>>>>>>>>>> so
>> >>>>>>>>>>>>>>>>>>>> most likely the only move here is to switch back to
>> the
>> >>>>>>>>>>>> switch
>> >>>>>>>>>>>>>>>>> approach
>> >>>>>>>>>>>>>>>>>>>> somehow preserving the metrics part. Probably,
>> inlining
>> >>>>>>>>> the
>> >>>>>>>>>>>> Ignite
>> >>>>>>>>>>>>>>>>>> messages
>> >>>>>>>>>>>>>>>>>>>> to the IgniteMessageFactoryImpl should do the trick.
>> Let
>> >>>>>>>>> me
>> >>>>>>>>>>>> explore
>> >>>>>>>>>>>>>>>>> the
>> >>>>>>>>>>>>>>>>>>>> code a bit.
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>> P.S. I am surprised by the impact this part makes for
>> >>>>>>>> the
>> >>>>>>>>>>>>>>>>> performance.
>> >>>>>>>>>>>>>>>>>>>> Message creation is indeed on the hot path, but a
>> single
>> >>>>>>>>>>>> virtual call
>> >>>>>>>>>>>>>>>>>>>> should not make that much of a difference given the
>> >>>>>>>> amount
>> >>>>>>>>>>>> of other
>> >>>>>>>>>>>>>>>>>> work
>> >>>>>>>>>>>>>>>>>>>> happening during the message processing.
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 18:33, Alex Plehanov <
>> >>>>>>>>>>>> plehanov.alex@gmail.com
>> >>>>>>>>>>>>>>>>>> :
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>> Alexey, sorry, I wrongly interpreted our benchmark
>> >>>>>>>>> results.
>> >>>>>>>>>>>>>>>>> Actually,
>> >>>>>>>>>>>>>>>>>> we
>> >>>>>>>>>>>>>>>>>>>>> were looking for a drop using bi-sect in the range
>> >>>>>>>>> between
>> >>>>>>>>>>>> e6a7f93
>> >>>>>>>>>>>>>>>>>> (first
>> >>>>>>>>>>>>>>>>>>>>> commit in the 2.9 branch after 2.8 branch cut) and
>> >>>>>>>>>>>> 6592dfa5 (last
>> >>>>>>>>>>>>>>>>>> commit in
>> >>>>>>>>>>>>>>>>>>>>> the 2.9 branch). And we found these two problematic
>> >>>>>>>>>>>> commits.
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>> Perhaps only IGNITE-13060 (Tracing) is responsible
>> for
>> >>>>>>>> a
>> >>>>>>>>>>>> drop
>> >>>>>>>>>>>>>>>>> between
>> >>>>>>>>>>>>>>>>>>>>> 2.8.1 and 2.9 (we have benchmarked 2.8.1 vs 2.9 with
>> >>>>>>>>>>>> reverted
>> >>>>>>>>>>>>>>>>>> IGNITE-13060
>> >>>>>>>>>>>>>>>>>>>>> now and performance looks the same)
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>> Ticket IGNITE-12568 (MessageFactory refactoring) is
>> not
>> >>>>>>>>>>>> related to
>> >>>>>>>>>>>>>>>>>> drop
>> >>>>>>>>>>>>>>>>>>>>> between 2.8.1 and 2.9, but still has some performance
>> >>>>>>>>>>>> problem, and
>> >>>>>>>>>>>>>>>>> we
>> >>>>>>>>>>>>>>>>>> can
>> >>>>>>>>>>>>>>>>>>>>> win back IGNITE-13060 drop by this ticket.
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>> Do we need more investigation on IGNITE-13060 or we
>> >>>>>>>> leave
>> >>>>>>>>>>>> it as is?
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>> What should we do with IGNITE-12568 (MessageFactory
>> >>>>>>>>>>>> refactoring)?
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 13:25, Alexey Goncharuk <
>> >>>>>>>>>>>>>>>>>> alexey.goncharuk@gmail.com
>> >>>>>>>>>>>>>>>>>>>>>> :
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>> Alexey,
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>> While investigating, I found that IGNITE-12568 has
>> an
>> >>>>>>>>>>>> incorrect fix
>> >>>>>>>>>>>>>>>>>>>>>> version and is actually present in ignite-2.8.1
>> branch
>> >>>>>>>>>>>> [1], so it
>> >>>>>>>>>>>>>>>>>> cannot be
>> >>>>>>>>>>>>>>>>>>>>>> the source of the drop against 2.8.1.
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>> P.S. Looks like we need to enforce a more accurate
>> >>>>>>>> work
>> >>>>>>>>>>>> with fix
>> >>>>>>>>>>>>>>>>>> versions
>> >>>>>>>>>>>>>>>>>>>>>> or develop some sort of tooling to verify the fix
>> >>>>>>>>>>>> versions.
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>> --AG
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>> [1]
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>>>
>> >>
>> https://github.com/apache/ignite/commit/3e492bd23851856bbd0385c6a419892d0bba2a34
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 12:42, Alexey Goncharuk <
>> >>>>>>>>>>>>>>>>>> alexey.goncharuk@gmail.com
>> >>>>>>>>>>>>>>>>>>>>>>> :
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>> пт, 28 авг. 2020 г. в 11:16, Alex Plehanov <
>> >>>>>>>>>>>>>>>>> plehanov.alex@gmail.com
>> >>>>>>>>>>>>>>>>>>> :
>> >>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>> Guys,
>> >>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>> We have benchmarked 2.9 without IGNITE-13060 and
>> >>>>>>>>>>>> IGNITE-12568
>> >>>>>>>>>>>>>>>>>> (reverted
>> >>>>>>>>>>>>>>>>>>>>>>>> it
>> >>>>>>>>>>>>>>>>>>>>>>>> locally) and got the same performance as on 2.8.1
>> >>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>> IGNITE-13060 (Tracing) - some code was added to
>> hot
>> >>>>>>>>>>>> paths, to
>> >>>>>>>>>>>>>>>>> trace
>> >>>>>>>>>>>>>>>>>>>>>>>> these
>> >>>>>>>>>>>>>>>>>>>>>>>> hot paths, it's clear why we have performance drop
>> >>>>>>>>> here.
>> >>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>> IGNITE-12568 (MessageFactory refactoring) -
>> >>>>>>>>> switch/case
>> >>>>>>>>>>>> block was
>> >>>>>>>>>>>>>>>>>>>>>>>> refactored to an array of message suppliers. The
>> >>>>>>>>>>>> message factory
>> >>>>>>>>>>>>>>>>>> is on
>> >>>>>>>>>>>>>>>>>>>>>>>> the
>> >>>>>>>>>>>>>>>>>>>>>>>> hot path, which explains why this commit has an
>> >>>>>>>> impact
>> >>>>>>>>>>>> on total
>> >>>>>>>>>>>>>>>>>>>>>>>> performance.
>> >>>>>>>>>>>>>>>>>>>>>>>> I've checked JIT assembly output, done some JMH
>> >>>>>>>>>>>> microbenchmarks,
>> >>>>>>>>>>>>>>>>>> and
>> >>>>>>>>>>>>>>>>>>>>>>>> found
>> >>>>>>>>>>>>>>>>>>>>>>>> that old implementation of MessageFactory.create()
>> >>>>>>>>>>>> about 30-35%
>> >>>>>>>>>>>>>>>>>> faster
>> >>>>>>>>>>>>>>>>>>>>>>>> than
>> >>>>>>>>>>>>>>>>>>>>>>>> the new one. The reason - approach with
>> switch/case
>> >>>>>>>>> can
>> >>>>>>>>>>>>>>>>> effectively
>> >>>>>>>>>>>>>>>>>>>>>>>> inline
>> >>>>>>>>>>>>>>>>>>>>>>>> message creation code, but with an array of
>> >>>>>>>> suppliers
>> >>>>>>>>>>>> relatively
>> >>>>>>>>>>>>>>>>>> heavy
>> >>>>>>>>>>>>>>>>>>>>>>>> "invokeinterface" cannot be skipped. I've tried to
>> >>>>>>>>>>>> rewrite the
>> >>>>>>>>>>>>>>>>> code
>> >>>>>>>>>>>>>>>>>>>>>>>> using
>> >>>>>>>>>>>>>>>>>>>>>>>> an abstract class for suppliers instead of an
>> >>>>>>>>> interface
>> >>>>>>>>>>>> (to
>> >>>>>>>>>>>>>>>>>>>>>>>> replace "invokeinterface" with the
>> "invokevirtual"),
>> >>>>>>>>>>>> but it gives
>> >>>>>>>>>>>>>>>>>> back
>> >>>>>>>>>>>>>>>>>>>>>>>> only
>> >>>>>>>>>>>>>>>>>>>>>>>> 10% of method performance and in this case, code
>> >>>>>>>> looks
>> >>>>>>>>>>>> ugly
>> >>>>>>>>>>>>>>>>>> (lambdas
>> >>>>>>>>>>>>>>>>>>>>>>>> can't
>> >>>>>>>>>>>>>>>>>>>>>>>> be used). Currently, I can't find any more ways to
>> >>>>>>>>>>>> optimize the
>> >>>>>>>>>>>>>>>>>> current
>> >>>>>>>>>>>>>>>>>>>>>>>> approach (except return to the switch/case block).
>> >>>>>>>>>>>> Andrey Gura,
>> >>>>>>>>>>>>>>>>> as
>> >>>>>>>>>>>>>>>>>> the
>> >>>>>>>>>>>>>>>>>>>>>>>> author of IGNITE-12568, maybe you have some ideas
>> >>>>>>>>> about
>> >>>>>>>>>>>>>>>>>> optimization?
>> >>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>> Perhaps we should revert IGNITE-12568, but there
>> are
>> >>>>>>>>>>>> some metrics
>> >>>>>>>>>>>>>>>>>>>>>>>> already
>> >>>>>>>>>>>>>>>>>>>>>>>> created, which can't be rewritten using old
>> message
>> >>>>>>>>>>>> factory
>> >>>>>>>>>>>>>>>>>>>>>>>> implementation
>> >>>>>>>>>>>>>>>>>>>>>>>> (IGNITE-12756). Guys, WDYT?
>> >>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>> Alexey,
>> >>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>> I see that IGNITE-12756 (metrics improvements) is
>> >>>>>>>>>>>> already released
>> >>>>>>>>>>>>>>>>>> in
>> >>>>>>>>>>>>>>>>>>>>>>> Ignite 2.8.1 while IGNITE-12568 (message factory)
>> is
>> >>>>>>>>>>>> only present
>> >>>>>>>>>>>>>>>>>> in Ignite
>> >>>>>>>>>>>>>>>>>>>>>>> 2.9. Let's revert both IGNITE-12568 and whichever
>> new
>> >>>>>>>>>>>> metrics
>> >>>>>>>>>>>>>>>>>> created for
>> >>>>>>>>>>>>>>>>>>>>>>> 2.9 that depend on the new message factory to
>> unblock
>> >>>>>>>>>>>> the release
>> >>>>>>>>>>>>>>>>>> and deal
>> >>>>>>>>>>>>>>>>>>>>>>> with the optimizations in 2.10?
>> >>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>
>> >>
>> >>
>>
>>
Re: Apache Ignite 2.9.0 RELEASE [Time, Scope, Manager]
Posted by Alexey Goncharuk <al...@gmail.com>.
Alexey,
For me, all three look quite critical as they: address a memory leak,
address a usability/performance issue with no workaround, and address a
regression in 2.8. If we want to limit the scope, I would still include the
memory leak fix, though.
Thoughts?
пн, 12 окт. 2020 г. в 13:07, Alex Plehanov <pl...@gmail.com>:
> Guys,
>
> I've found 3 more tickets which were targeted to 2.9 but merged only to the
> master branch ([1], [2], [3]).
> Do we need these tickets in 2.9? Are they critical?
>
> [1]: https://issues.apache.org/jira/browse/IGNITE-12350
> [2]: https://issues.apache.org/jira/browse/IGNITE-13376
> [3]: https://issues.apache.org/jira/browse/IGNITE-13280
>
> вс, 11 окт. 2020 г. в 16:11, Nikolay Izhikov <ni...@apache.org>:
>
> > Igniters,
> >
> > IGNITE-13553 fixed and cherry-picked to 2.9.
> > We can continue with the release.
> >
> > > 10 окт. 2020 г., в 00:00, Denis Magda <dm...@apache.org> написал(а):
> > >
> > > Nikolay,
> > >
> > > re: the minor improvements. I'm not against of including those if we
> can
> > > prepare the docs before the vote starts. Presently, the docs are
> "frozen"
> > > for the 2.9 release, but I can scratch some time and take part in the
> > docs
> > > review the next week.
> > >
> > > -
> > > Denis
> > >
> > >
> > > On Fri, Oct 9, 2020 at 12:40 AM Nikolay Izhikov <ni...@apache.org>
> > wrote:
> > >
> > >> Hello, Igniters.
> > >>
> > >> I prepared a patch [1] for blocker ticket [2] - «Server node fail and
> > >> stops in case wrong datatype put in indexed field»
> > >> Can someone, please, help me with the review?
> > >>
> > >> [1] https://github.com/apache/ignite/pull/8330
> > >> [2] https://issues.apache.org/jira/browse/IGNITE-13553
> > >>
> > >> ==================
> > >>
> > >> I propose to include several minor tickets to the scope of the 2.9
> that
> > >> increase Ignite User Experience
> > >>
> > >> CMD tools improvements:
> > >>
> > >> IGNITE-13488 - Command to print metric value
> > >> IGNITE-13426 - Command to print system view content
> > >> IGNITE-13422 - Parameter to explicitly enable experimental commands
> > >>
> > >> IGNITE-13380 - Output IgniteSystemProperties via ignite.sh
> > >>
> > >> New system views:
> > >>
> > >> IGNITE-13409 Metastorage and DistributedMetastorage viewы.
> > >> IGNITE-13408 BinaryMetadata view.
> > >>
> > >>> 9 окт. 2020 г., в 04:04, Denis Magda <dm...@apache.org> написал(а):
> > >>>
> > >>> @Alex Plehanov <pl...@gmail.com>,
> > >>>
> > >>> If it still makes sense and not too late, you can cherry-pick the
> > commit
> > >>> with the new docs into the 2.9 branch:
> > >>>
> > >>
> >
> https://github.com/apache/ignite/commit/073488ac97517bbaad9f6b94b781fc404646f191
> > >>>
> > >>> Anyway, it's not crucial as long as we agreed to make an exception
> for
> > >> this
> > >>> release. The docs were already published to the website and assembled
> > >> from
> > >>> the master. Once the vote passes, I'll make them visible and
> traceable
> > >> from
> > >>> the website menus.
> > >>>
> > >>> -
> > >>> Denis
> > >>>
> > >>>
> > >>> On Tue, Oct 6, 2020 at 7:20 AM Alexey Goncharuk <
> > >> alexey.goncharuk@gmail.com>
> > >>> wrote:
> > >>>
> > >>>> Saikat,
> > >>>>
> > >>>> The plan sounds good to me! Do you have an idea for the timeline of
> > the
> > >>>> module releases? Let me know if I can help in any way.
> > >>>>
> > >>>> Also, I was looking into the modularization IEP and noticed that
> OSGI
> > >>>> module is discussed to be moved to the extensions, but there is no
> > >>>> corresponding ticket. Should I create one? I will be happy to help
> > with
> > >>>> moving this module to extensions.
> > >>>>
> > >>>> вт, 29 сент. 2020 г. в 03:03, Saikat Maitra <
> saikat.maitra@gmail.com
> > >:
> > >>>>
> > >>>>> Hi Alexey,
> > >>>>>
> > >>>>> All the modules as planned in IEP-36
> > >>>>>
> > >>>>
> > >>
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36:+Modularization
> > >>>>> are migrated to ignite-extensions repository.
> > >>>>>
> > >>>>> We can definitely plan to release ignite-extensions modules.
> > >>>>>
> > >>>>> I have a pending PR related to the kafka module and the PR is in
> > review
> > >>>>> https://github.com/apache/ignite/pull/8222 but its corresponding
> PR
> > >> has
> > >>>>> been merged in ignite-extensions repository.
> > >>>>>
> > >>>>> My thoughts / action items for the migration efforts and releases
> > were
> > >>>>> as follows:
> > >>>>>
> > >>>>> 1. Merge the changes for
> https://github.com/apache/ignite/pull/8222
> > >>>>> 2. Fix the upload nightly packages task for ignite-core to help fix
> > the
> > >>>>> travis build failure in ignite-extensions repository.
> > >>>>> 3. Review and update the README.md files for the ignite-extensions
> > >>>> modules
> > >>>>> as required.
> > >>>>> 4. Update the docs for ignite-extensions modules in ignite-website.
> > >>>>> 5. Release each module separately and share updates.
> > >>>>>
> > >>>>> Please let me know if you have feedback.
> > >>>>>
> > >>>>> Regards,
> > >>>>> Saikat
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> On Mon, Sep 28, 2020 at 7:36 AM Petr Ivanov <mr...@gmail.com>
> > >> wrote:
> > >>>>>
> > >>>>>> It seems that gitbox.apache.org is not available on CI/CD.
> > >>>>>>
> > >>>>>> I will try to update VCS to GitHub.
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>>> On 28 Sep 2020, at 14:07, Alex Plehanov <plehanov.alex@gmail.com
> >
> > >>>>>> wrote:
> > >>>>>>>
> > >>>>>>> Guys,
> > >>>>>>>
> > >>>>>>> I've tried to build release candidate using TC [1]. But:
> > >>>>>>> 1. I can't choose a branch in this TC task (there is no such
> > field).
> > >>>>>>> 2. On the "default" branch I got an error: Failed to collect
> > changes,
> > >>>>>>> error: List remote refs failed:
> > >>>>>>> org.apache.http.conn.ConnectTimeoutException: Connect to
> > >>>>>>> gitbox.apache.org:443 [gitbox.apache.org/52.202.80.70] failed:
> > >>>> connect
> > >>>>>>> timed out, VCS root: "GitBox [asf/ignite]" {instance id=300,
> parent
> > >>>>>>> internal id=74, parent id=GitBoxAsfIgnite, description: "
> > >>>>>>> https://gitbox.apache.org/repos/asf/ignite.git#refs/heads/master
> "}
> > >>>>>>>
> > >>>>>>> Is there some problem with this TC task? What am I doing wrong?
> > >>>>>>>
> > >>>>>>> [1] :
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>
> >
> https://ci.ignite.apache.org/viewType.html?buildTypeId=Releases_ApacheIgniteMain_ReleaseBuild
> > >>>>>>>
> > >>>>>>> пн, 28 сент. 2020 г. в 13:15, Alexey Goncharuk <
> > >>>>>> alexey.goncharuk@gmail.com>:
> > >>>>>>>
> > >>>>>>>> Saikat, Nikolay,
> > >>>>>>>>
> > >>>>>>>> We have migrated a bunch of modules to ignite-extensions
> recently.
> > >>>>>> Given
> > >>>>>>>> that these modules will not be available in Ignite 2.9 anymore
> > (will
> > >>>>>>>> they?), should we also plan to release the extensions?
> > >>>>>>>>
> > >>>>>>>> ср, 23 сент. 2020 г. в 21:49, Alex Plehanov <
> > >> plehanov.alex@gmail.com
> > >>>>> :
> > >>>>>>>>
> > >>>>>>>>> Igniters,
> > >>>>>>>>>
> > >>>>>>>>> I've prepared release notes for the 2.9 release [1]. Can anyone
> > >>>> review
> > >>>>>>>> it,
> > >>>>>>>>> please?
> > >>>>>>>>>
> > >>>>>>>>> [1]: https://github.com/apache/ignite/pull/8273
> > >>>>>>>>>
> > >>>>>>>>> вт, 22 сент. 2020 г. в 09:39, Alex Plehanov <
> > >>>> plehanov.alex@gmail.com
> > >>>>>>> :
> > >>>>>>>>>
> > >>>>>>>>>> Guys,
> > >>>>>>>>>>
> > >>>>>>>>>> I've filled the ticket with reproducer [1] for the discovery
> > bug.
> > >>>>>> This
> > >>>>>>>>> bug
> > >>>>>>>>>> caused by [2] ticket. We discussed with Vladimir Steshin
> > privately
> > >>>>>> and
> > >>>>>>>>>> decided to revert this ticket. I will do it today (after TC
> bot
> > >>>> visa)
> > >>>>>>>> if
> > >>>>>>>>>> there are no objections.
> > >>>>>>>>>>
> > >>>>>>>>>> [1]: https://issues.apache.org/jira/browse/IGNITE-13465
> > >>>>>>>>>> [2]: https://issues.apache.org/jira/browse/IGNITE-13134
> > >>>>>>>>>>
> > >>>>>>>>>> пн, 21 сент. 2020 г. в 11:08, Alex Plehanov <
> > >>>> plehanov.alex@gmail.com
> > >>>>>>> :
> > >>>>>>>>>>
> > >>>>>>>>>>> Guys,
> > >>>>>>>>>>>
> > >>>>>>>>>>> During internal testing, we've found a critical bug with
> > >>>>>>>>>>> discovery (cluster falls apart if two nodes segmented
> > >>>> sequentially).
> > >>>>>>>>> This
> > >>>>>>>>>>> problem is not reproduced in 2.8.1. I think we should fix it
> > >>>>>>>>>>> before release. Under investigation now. I'll let you know
> when
> > >> we
> > >>>>>> get
> > >>>>>>>>>>> something.
> > >>>>>>>>>>>
> > >>>>>>>>>>> чт, 17 сент. 2020 г. в 00:51, Andrey Gura <agura@apache.org
> >:
> > >>>>>>>>>>>
> > >>>>>>>>>>>>> So what do you think? Should we proceed with a 'hacked'
> > version
> > >>>> of
> > >>>>>>>>> the
> > >>>>>>>>>>>> message factory in 2.9 and go for the runtime message
> > generation
> > >>>> in
> > >>>>>>>>> later
> > >>>>>>>>>>>> release, or keep the code clean and fix the regression in
> the
> > >>>> next
> > >>>>>>>>> releases?
> > >>>>>>>>>>>>> Andrey, can you take a look at my change? I think it is
> > fairly
> > >>>>>>>>>>>> straightforward and does not change the semantics, just
> skips
> > >> the
> > >>>>>>>>> factory
> > >>>>>>>>>>>> closures for certain messages.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> IMHO 2.5% isn't too much especially because it isn't actual
> > for
> > >>>> all
> > >>>>>>>>>>>> workloads (I didn't get any significant drops during
> > >>>> benchmarking).
> > >>>>>>>> So
> > >>>>>>>>>>>> I prefer the runtime generation in later releases.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Mon, Sep 14, 2020 at 12:41 PM Alexey Goncharuk
> > >>>>>>>>>>>> <al...@gmail.com> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Alexey, Andrey, Igniters,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> So what do you think? Should we proceed with a 'hacked'
> > version
> > >>>> of
> > >>>>>>>>> the
> > >>>>>>>>>>>> message factory in 2.9 and go for the runtime message
> > generation
> > >>>> in
> > >>>>>>>>> later
> > >>>>>>>>>>>> release, or keep the code clean and fix the regression in
> the
> > >>>> next
> > >>>>>>>>> releases?
> > >>>>>>>>>>>>> Andrey, can you take a look at my change? I think it is
> > fairly
> > >>>>>>>>>>>> straightforward and does not change the semantics, just
> skips
> > >> the
> > >>>>>>>>> factory
> > >>>>>>>>>>>> closures for certain messages.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Personally, I would prefer fixing the regression given that
> > we
> > >>>>>> also
> > >>>>>>>>>>>> introduced tracing in this release.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> пт, 11 сент. 2020 г. в 12:09, Alex Plehanov <
> > >>>>>>>> plehanov.alex@gmail.com
> > >>>>>>>>>> :
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Alexey,
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> We've benchmarked by yardstick commits 130376741bf vs
> > >>>> ed52559eb95
> > >>>>>>>>> and
> > >>>>>>>>>>>> the performance of ed52559eb95 is better for about 2.5% on
> > >>>> average
> > >>>>>> on
> > >>>>>>>>> our
> > >>>>>>>>>>>> environment (about the same results we got benchmarking
> > 65c30ec6
> > >>>> vs
> > >>>>>>>>>>>> 0606f03d). We've made 24 runs for each commit of
> > >>>>>>>>>>>> IgnitePutTxImplicitBenchmark (we got maximum drop for 2.9 on
> > >> this
> > >>>>>>>>>>>> benchmark), 200 seconds warmup, 300 seconds benchmark, 6
> > >>>> servers, 5
> > >>>>>>>>> clients
> > >>>>>>>>>>>> 50 threads each. Yardstick results for this configuration:
> > >>>>>>>>>>>>>> Commit 130376741bf: avg TPS=164096, avg latency=9173464 ns
> > >>>>>>>>>>>>>> Commit ed52559eb95: avg TPS=168283, avg latency=8945908 ns
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> пт, 11 сент. 2020 г. в 09:51, Artem Budnikov <
> > >>>>>>>>>>>> a.budnikov.ignite@gmail.com>:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Hi Everyone,
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> I posted an instruction on how to publish the docs on
> > >>>>>>>>>>>> ignite.apache.org/docs [1]. When you finish with Ignite
> 2.9,
> > >> you
> > >>>>>> can
> > >>>>>>>>>>>> update the docs by following the instruction.
> Unfortunately, I
> > >>>>>> won't
> > >>>>>>>> be
> > >>>>>>>>>>>> able to spend any time on this project any longer. You can
> > send
> > >>>>>> your
> > >>>>>>>>> pull
> > >>>>>>>>>>>> requests and questions about the documentation to Denis
> Magda.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> -Artem
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> [1] :
> > >>>>>>>>>>>>
> > >>>> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Document
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Thu, Sep 10, 2020 at 2:45 PM Alexey Goncharuk <
> > >>>>>>>>>>>> alexey.goncharuk@gmail.com> wrote:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Alexey,
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> I've tried to play with message factories locally, but
> > >>>>>>>>>>>> unfortunately, I
> > >>>>>>>>>>>>>>>> cannot spot the difference between old and new
> > >> implementation
> > >>>>>> in
> > >>>>>>>>>>>>>>>> distributed benchmarks. I pushed an implementation of
> > >>>>>>>>>>>> MessageFactoryImpl
> > >>>>>>>>>>>>>>>> with the old switch statement to the
> > ignite-2.9-revert-12568
> > >>>>>>>>> branch
> > >>>>>>>>>>>>>>>> (discussed this with Andrey Gura, the change should be
> > >>>>>>>> compatible
> > >>>>>>>>>>>> with the
> > >>>>>>>>>>>>>>>> new metrics as we still use the register() mechanics).
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Can you check if this change makes any difference
> > >>>>>>>> performance-wise
> > >>>>>>>>>>>> in your
> > >>>>>>>>>>>>>>>> environment? If yes, we can go with runtime code
> > generation
> > >>>> in
> > >>>>>>>> the
> > >>>>>>>>>>>> long
> > >>>>>>>>>>>>>>>> term: register classes and generate a dynamic message
> > >> factory
> > >>>>>>>> with
> > >>>>>>>>>>>> a switch
> > >>>>>>>>>>>>>>>> statement once all messages are registered (not in 2.9
> > >>>> though,
> > >>>>>>>>>>>> obviously).
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> ср, 9 сент. 2020 г. в 14:53, Alex Plehanov <
> > >>>>>>>>> plehanov.alex@gmail.com
> > >>>>>>>>>>>>> :
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Hello guys,
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> I've tried to optimize tracing implementation (ticket
> > [1]),
> > >>>> it
> > >>>>>>>>>>>> reduced the
> > >>>>>>>>>>>>>>>>> drop, but not completely removed it.
> > >>>>>>>>>>>>>>>>> Ivan Rakov, Alexander Lapin, can you please review the
> > >>>> patch?
> > >>>>>>>>>>>>>>>>> Ivan Artiukhov, can you please benchmark the patch [2]
> > >>>> against
> > >>>>>>>>>>>> 2.8.1
> > >>>>>>>>>>>>>>>>> release on your environment?
> > >>>>>>>>>>>>>>>>> With this patch on our environment, it's about a 3%
> drop
> > >>>> left,
> > >>>>>>>>>>>> it's close
> > >>>>>>>>>>>>>>>>> to measurement error and I think such a drop is not a
> > >>>>>>>>>>>> showstopper. Guys,
> > >>>>>>>>>>>>>>>>> WDYT?
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Also, I found that compatibility is broken for JDBC
> thin
> > >>>>>>>> driver
> > >>>>>>>>>>>> between 2.8
> > >>>>>>>>>>>>>>>>> and 2.9 versions (ticket [3]). I think it's a blocker
> and
> > >>>>>>>> should
> > >>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>> fixed in 2.9. I've prepared the patch.
> > >>>>>>>>>>>>>>>>> Taras Ledkov, can you please review this patch?
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> And one more ticket I propose to include into 2.9 [4]
> > (NIO
> > >>>>>>>>> message
> > >>>>>>>>>>>>>>>>> send problem in some circumstances). I will cherry-pick
> > it
> > >>>> if
> > >>>>>>>>>>>> there is no
> > >>>>>>>>>>>>>>>>> objection.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13411
> > >>>>>>>>>>>>>>>>> [2] https://github.com/apache/ignite/pull/8223
> > >>>>>>>>>>>>>>>>> [3] https://issues.apache.org/jira/browse/IGNITE-13414
> > >>>>>>>>>>>>>>>>> [4] https://issues.apache.org/jira/browse/IGNITE-13361
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> пн, 7 сент. 2020 г. в 14:14, Maxim Muzafarov <
> > >>>>>>>> mmuzaf@apache.org
> > >>>>>>>>>> :
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Alexey,
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> I propose to include [1] issue to the 2.9 release.
> Since
> > >>>>>>>> this
> > >>>>>>>>>>>> issue is
> > >>>>>>>>>>>>>>>>>> related to the new master key change functionality
> which
> > >>>>>>>>>>>> haven't been
> > >>>>>>>>>>>>>>>>>> released yet I think it will be safe to cherry-pick
> > commit
> > >>>>>>>> to
> > >>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>> release branch.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> [1]
> https://issues.apache.org/jira/browse/IGNITE-13390
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> On Tue, 1 Sep 2020 at 12:13, Nikolay Izhikov <
> > >>>>>>>>>>>> nizhikov@apache.org>
> > >>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Hello, Igniters.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Alexey, please, include one more Python thin client
> fix
> > >>>>>>>> [1]
> > >>>>>>>>>>>> into the
> > >>>>>>>>>>>>>>>>> 2.9
> > >>>>>>>>>>>>>>>>>> release
> > >>>>>>>>>>>>>>>>>>> It fixes kinda major issue - "Python client returns
> > >> fields
> > >>>>>>>>> in
> > >>>>>>>>>>>> wrong
> > >>>>>>>>>>>>>>>>>> order since the 2 row when fields_count>10"
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> [1]
> https://issues.apache.org/jira/browse/IGNITE-12809
> > >>>>>>>>>>>>>>>>>>> [2]
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>
> > >>
> >
> https://github.com/apache/ignite/commit/38025ee4167f05eaa2d6a2c5c2ab70c83a462cfc
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> 31 авг. 2020 г., в 19:23, Alexey Goncharuk <
> > >>>>>>>>>>>>>>>>> alexey.goncharuk@gmail.com>
> > >>>>>>>>>>>>>>>>>> написал(а):
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Alexey, thanks, got it. I am not sure we can
> optimize
> > >>>>>>>>>>>> anything out of
> > >>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>> message factory with suppliers (at least I have no
> > ideas
> > >>>>>>>>>>>> right now),
> > >>>>>>>>>>>>>>>>> so
> > >>>>>>>>>>>>>>>>>>>> most likely the only move here is to switch back to
> > the
> > >>>>>>>>>>>> switch
> > >>>>>>>>>>>>>>>>> approach
> > >>>>>>>>>>>>>>>>>>>> somehow preserving the metrics part. Probably,
> > inlining
> > >>>>>>>>> the
> > >>>>>>>>>>>> Ignite
> > >>>>>>>>>>>>>>>>>> messages
> > >>>>>>>>>>>>>>>>>>>> to the IgniteMessageFactoryImpl should do the trick.
> > Let
> > >>>>>>>>> me
> > >>>>>>>>>>>> explore
> > >>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>> code a bit.
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> P.S. I am surprised by the impact this part makes
> for
> > >>>>>>>> the
> > >>>>>>>>>>>>>>>>> performance.
> > >>>>>>>>>>>>>>>>>>>> Message creation is indeed on the hot path, but a
> > single
> > >>>>>>>>>>>> virtual call
> > >>>>>>>>>>>>>>>>>>>> should not make that much of a difference given the
> > >>>>>>>> amount
> > >>>>>>>>>>>> of other
> > >>>>>>>>>>>>>>>>>> work
> > >>>>>>>>>>>>>>>>>>>> happening during the message processing.
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 18:33, Alex Plehanov <
> > >>>>>>>>>>>> plehanov.alex@gmail.com
> > >>>>>>>>>>>>>>>>>> :
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Alexey, sorry, I wrongly interpreted our benchmark
> > >>>>>>>>> results.
> > >>>>>>>>>>>>>>>>> Actually,
> > >>>>>>>>>>>>>>>>>> we
> > >>>>>>>>>>>>>>>>>>>>> were looking for a drop using bi-sect in the range
> > >>>>>>>>> between
> > >>>>>>>>>>>> e6a7f93
> > >>>>>>>>>>>>>>>>>> (first
> > >>>>>>>>>>>>>>>>>>>>> commit in the 2.9 branch after 2.8 branch cut) and
> > >>>>>>>>>>>> 6592dfa5 (last
> > >>>>>>>>>>>>>>>>>> commit in
> > >>>>>>>>>>>>>>>>>>>>> the 2.9 branch). And we found these two problematic
> > >>>>>>>>>>>> commits.
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Perhaps only IGNITE-13060 (Tracing) is responsible
> > for
> > >>>>>>>> a
> > >>>>>>>>>>>> drop
> > >>>>>>>>>>>>>>>>> between
> > >>>>>>>>>>>>>>>>>>>>> 2.8.1 and 2.9 (we have benchmarked 2.8.1 vs 2.9
> with
> > >>>>>>>>>>>> reverted
> > >>>>>>>>>>>>>>>>>> IGNITE-13060
> > >>>>>>>>>>>>>>>>>>>>> now and performance looks the same)
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Ticket IGNITE-12568 (MessageFactory refactoring) is
> > not
> > >>>>>>>>>>>> related to
> > >>>>>>>>>>>>>>>>>> drop
> > >>>>>>>>>>>>>>>>>>>>> between 2.8.1 and 2.9, but still has some
> performance
> > >>>>>>>>>>>> problem, and
> > >>>>>>>>>>>>>>>>> we
> > >>>>>>>>>>>>>>>>>> can
> > >>>>>>>>>>>>>>>>>>>>> win back IGNITE-13060 drop by this ticket.
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Do we need more investigation on IGNITE-13060 or we
> > >>>>>>>> leave
> > >>>>>>>>>>>> it as is?
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> What should we do with IGNITE-12568 (MessageFactory
> > >>>>>>>>>>>> refactoring)?
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 13:25, Alexey Goncharuk <
> > >>>>>>>>>>>>>>>>>> alexey.goncharuk@gmail.com
> > >>>>>>>>>>>>>>>>>>>>>> :
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Alexey,
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> While investigating, I found that IGNITE-12568 has
> > an
> > >>>>>>>>>>>> incorrect fix
> > >>>>>>>>>>>>>>>>>>>>>> version and is actually present in ignite-2.8.1
> > branch
> > >>>>>>>>>>>> [1], so it
> > >>>>>>>>>>>>>>>>>> cannot be
> > >>>>>>>>>>>>>>>>>>>>>> the source of the drop against 2.8.1.
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> P.S. Looks like we need to enforce a more accurate
> > >>>>>>>> work
> > >>>>>>>>>>>> with fix
> > >>>>>>>>>>>>>>>>>> versions
> > >>>>>>>>>>>>>>>>>>>>>> or develop some sort of tooling to verify the fix
> > >>>>>>>>>>>> versions.
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> --AG
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> [1]
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>
> > >>
> >
> https://github.com/apache/ignite/commit/3e492bd23851856bbd0385c6a419892d0bba2a34
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 12:42, Alexey Goncharuk <
> > >>>>>>>>>>>>>>>>>> alexey.goncharuk@gmail.com
> > >>>>>>>>>>>>>>>>>>>>>>> :
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> пт, 28 авг. 2020 г. в 11:16, Alex Plehanov <
> > >>>>>>>>>>>>>>>>> plehanov.alex@gmail.com
> > >>>>>>>>>>>>>>>>>>> :
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Guys,
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> We have benchmarked 2.9 without IGNITE-13060 and
> > >>>>>>>>>>>> IGNITE-12568
> > >>>>>>>>>>>>>>>>>> (reverted
> > >>>>>>>>>>>>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>>>>>>>>> locally) and got the same performance as on
> 2.8.1
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> IGNITE-13060 (Tracing) - some code was added to
> > hot
> > >>>>>>>>>>>> paths, to
> > >>>>>>>>>>>>>>>>> trace
> > >>>>>>>>>>>>>>>>>>>>>>>> these
> > >>>>>>>>>>>>>>>>>>>>>>>> hot paths, it's clear why we have performance
> drop
> > >>>>>>>>> here.
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> IGNITE-12568 (MessageFactory refactoring) -
> > >>>>>>>>> switch/case
> > >>>>>>>>>>>> block was
> > >>>>>>>>>>>>>>>>>>>>>>>> refactored to an array of message suppliers. The
> > >>>>>>>>>>>> message factory
> > >>>>>>>>>>>>>>>>>> is on
> > >>>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>> hot path, which explains why this commit has an
> > >>>>>>>> impact
> > >>>>>>>>>>>> on total
> > >>>>>>>>>>>>>>>>>>>>>>>> performance.
> > >>>>>>>>>>>>>>>>>>>>>>>> I've checked JIT assembly output, done some JMH
> > >>>>>>>>>>>> microbenchmarks,
> > >>>>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>>>>> found
> > >>>>>>>>>>>>>>>>>>>>>>>> that old implementation of
> MessageFactory.create()
> > >>>>>>>>>>>> about 30-35%
> > >>>>>>>>>>>>>>>>>> faster
> > >>>>>>>>>>>>>>>>>>>>>>>> than
> > >>>>>>>>>>>>>>>>>>>>>>>> the new one. The reason - approach with
> > switch/case
> > >>>>>>>>> can
> > >>>>>>>>>>>>>>>>> effectively
> > >>>>>>>>>>>>>>>>>>>>>>>> inline
> > >>>>>>>>>>>>>>>>>>>>>>>> message creation code, but with an array of
> > >>>>>>>> suppliers
> > >>>>>>>>>>>> relatively
> > >>>>>>>>>>>>>>>>>> heavy
> > >>>>>>>>>>>>>>>>>>>>>>>> "invokeinterface" cannot be skipped. I've tried
> to
> > >>>>>>>>>>>> rewrite the
> > >>>>>>>>>>>>>>>>> code
> > >>>>>>>>>>>>>>>>>>>>>>>> using
> > >>>>>>>>>>>>>>>>>>>>>>>> an abstract class for suppliers instead of an
> > >>>>>>>>> interface
> > >>>>>>>>>>>> (to
> > >>>>>>>>>>>>>>>>>>>>>>>> replace "invokeinterface" with the
> > "invokevirtual"),
> > >>>>>>>>>>>> but it gives
> > >>>>>>>>>>>>>>>>>> back
> > >>>>>>>>>>>>>>>>>>>>>>>> only
> > >>>>>>>>>>>>>>>>>>>>>>>> 10% of method performance and in this case, code
> > >>>>>>>> looks
> > >>>>>>>>>>>> ugly
> > >>>>>>>>>>>>>>>>>> (lambdas
> > >>>>>>>>>>>>>>>>>>>>>>>> can't
> > >>>>>>>>>>>>>>>>>>>>>>>> be used). Currently, I can't find any more ways
> to
> > >>>>>>>>>>>> optimize the
> > >>>>>>>>>>>>>>>>>> current
> > >>>>>>>>>>>>>>>>>>>>>>>> approach (except return to the switch/case
> block).
> > >>>>>>>>>>>> Andrey Gura,
> > >>>>>>>>>>>>>>>>> as
> > >>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>> author of IGNITE-12568, maybe you have some
> ideas
> > >>>>>>>>> about
> > >>>>>>>>>>>>>>>>>> optimization?
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Perhaps we should revert IGNITE-12568, but there
> > are
> > >>>>>>>>>>>> some metrics
> > >>>>>>>>>>>>>>>>>>>>>>>> already
> > >>>>>>>>>>>>>>>>>>>>>>>> created, which can't be rewritten using old
> > message
> > >>>>>>>>>>>> factory
> > >>>>>>>>>>>>>>>>>>>>>>>> implementation
> > >>>>>>>>>>>>>>>>>>>>>>>> (IGNITE-12756). Guys, WDYT?
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Alexey,
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> I see that IGNITE-12756 (metrics improvements) is
> > >>>>>>>>>>>> already released
> > >>>>>>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>>>>>>> Ignite 2.8.1 while IGNITE-12568 (message factory)
> > is
> > >>>>>>>>>>>> only present
> > >>>>>>>>>>>>>>>>>> in Ignite
> > >>>>>>>>>>>>>>>>>>>>>>> 2.9. Let's revert both IGNITE-12568 and whichever
> > new
> > >>>>>>>>>>>> metrics
> > >>>>>>>>>>>>>>>>>> created for
> > >>>>>>>>>>>>>>>>>>>>>>> 2.9 that depend on the new message factory to
> > unblock
> > >>>>>>>>>>>> the release
> > >>>>>>>>>>>>>>>>>> and deal
> > >>>>>>>>>>>>>>>>>>>>>>> with the optimizations in 2.10?
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>
> > >>
> > >>
> >
> >
>
Re: Apache Ignite 2.9.0 RELEASE [Time, Scope, Manager]
Posted by Alex Plehanov <pl...@gmail.com>.
Guys,
I've found 3 more tickets which were targeted to 2.9 but merged only to the
master branch ([1], [2], [3]).
Do we need these tickets in 2.9? Are they critical?
[1]: https://issues.apache.org/jira/browse/IGNITE-12350
[2]: https://issues.apache.org/jira/browse/IGNITE-13376
[3]: https://issues.apache.org/jira/browse/IGNITE-13280
вс, 11 окт. 2020 г. в 16:11, Nikolay Izhikov <ni...@apache.org>:
> Igniters,
>
> IGNITE-13553 fixed and cherry-picked to 2.9.
> We can continue with the release.
>
> > 10 окт. 2020 г., в 00:00, Denis Magda <dm...@apache.org> написал(а):
> >
> > Nikolay,
> >
> > re: the minor improvements. I'm not against of including those if we can
> > prepare the docs before the vote starts. Presently, the docs are "frozen"
> > for the 2.9 release, but I can scratch some time and take part in the
> docs
> > review the next week.
> >
> > -
> > Denis
> >
> >
> > On Fri, Oct 9, 2020 at 12:40 AM Nikolay Izhikov <ni...@apache.org>
> wrote:
> >
> >> Hello, Igniters.
> >>
> >> I prepared a patch [1] for blocker ticket [2] - «Server node fail and
> >> stops in case wrong datatype put in indexed field»
> >> Can someone, please, help me with the review?
> >>
> >> [1] https://github.com/apache/ignite/pull/8330
> >> [2] https://issues.apache.org/jira/browse/IGNITE-13553
> >>
> >> ==================
> >>
> >> I propose to include several minor tickets to the scope of the 2.9 that
> >> increase Ignite User Experience
> >>
> >> CMD tools improvements:
> >>
> >> IGNITE-13488 - Command to print metric value
> >> IGNITE-13426 - Command to print system view content
> >> IGNITE-13422 - Parameter to explicitly enable experimental commands
> >>
> >> IGNITE-13380 - Output IgniteSystemProperties via ignite.sh
> >>
> >> New system views:
> >>
> >> IGNITE-13409 Metastorage and DistributedMetastorage viewы.
> >> IGNITE-13408 BinaryMetadata view.
> >>
> >>> 9 окт. 2020 г., в 04:04, Denis Magda <dm...@apache.org> написал(а):
> >>>
> >>> @Alex Plehanov <pl...@gmail.com>,
> >>>
> >>> If it still makes sense and not too late, you can cherry-pick the
> commit
> >>> with the new docs into the 2.9 branch:
> >>>
> >>
> https://github.com/apache/ignite/commit/073488ac97517bbaad9f6b94b781fc404646f191
> >>>
> >>> Anyway, it's not crucial as long as we agreed to make an exception for
> >> this
> >>> release. The docs were already published to the website and assembled
> >> from
> >>> the master. Once the vote passes, I'll make them visible and traceable
> >> from
> >>> the website menus.
> >>>
> >>> -
> >>> Denis
> >>>
> >>>
> >>> On Tue, Oct 6, 2020 at 7:20 AM Alexey Goncharuk <
> >> alexey.goncharuk@gmail.com>
> >>> wrote:
> >>>
> >>>> Saikat,
> >>>>
> >>>> The plan sounds good to me! Do you have an idea for the timeline of
> the
> >>>> module releases? Let me know if I can help in any way.
> >>>>
> >>>> Also, I was looking into the modularization IEP and noticed that OSGI
> >>>> module is discussed to be moved to the extensions, but there is no
> >>>> corresponding ticket. Should I create one? I will be happy to help
> with
> >>>> moving this module to extensions.
> >>>>
> >>>> вт, 29 сент. 2020 г. в 03:03, Saikat Maitra <saikat.maitra@gmail.com
> >:
> >>>>
> >>>>> Hi Alexey,
> >>>>>
> >>>>> All the modules as planned in IEP-36
> >>>>>
> >>>>
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36:+Modularization
> >>>>> are migrated to ignite-extensions repository.
> >>>>>
> >>>>> We can definitely plan to release ignite-extensions modules.
> >>>>>
> >>>>> I have a pending PR related to the kafka module and the PR is in
> review
> >>>>> https://github.com/apache/ignite/pull/8222 but its corresponding PR
> >> has
> >>>>> been merged in ignite-extensions repository.
> >>>>>
> >>>>> My thoughts / action items for the migration efforts and releases
> were
> >>>>> as follows:
> >>>>>
> >>>>> 1. Merge the changes for https://github.com/apache/ignite/pull/8222
> >>>>> 2. Fix the upload nightly packages task for ignite-core to help fix
> the
> >>>>> travis build failure in ignite-extensions repository.
> >>>>> 3. Review and update the README.md files for the ignite-extensions
> >>>> modules
> >>>>> as required.
> >>>>> 4. Update the docs for ignite-extensions modules in ignite-website.
> >>>>> 5. Release each module separately and share updates.
> >>>>>
> >>>>> Please let me know if you have feedback.
> >>>>>
> >>>>> Regards,
> >>>>> Saikat
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Mon, Sep 28, 2020 at 7:36 AM Petr Ivanov <mr...@gmail.com>
> >> wrote:
> >>>>>
> >>>>>> It seems that gitbox.apache.org is not available on CI/CD.
> >>>>>>
> >>>>>> I will try to update VCS to GitHub.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>> On 28 Sep 2020, at 14:07, Alex Plehanov <pl...@gmail.com>
> >>>>>> wrote:
> >>>>>>>
> >>>>>>> Guys,
> >>>>>>>
> >>>>>>> I've tried to build release candidate using TC [1]. But:
> >>>>>>> 1. I can't choose a branch in this TC task (there is no such
> field).
> >>>>>>> 2. On the "default" branch I got an error: Failed to collect
> changes,
> >>>>>>> error: List remote refs failed:
> >>>>>>> org.apache.http.conn.ConnectTimeoutException: Connect to
> >>>>>>> gitbox.apache.org:443 [gitbox.apache.org/52.202.80.70] failed:
> >>>> connect
> >>>>>>> timed out, VCS root: "GitBox [asf/ignite]" {instance id=300, parent
> >>>>>>> internal id=74, parent id=GitBoxAsfIgnite, description: "
> >>>>>>> https://gitbox.apache.org/repos/asf/ignite.git#refs/heads/master"}
> >>>>>>>
> >>>>>>> Is there some problem with this TC task? What am I doing wrong?
> >>>>>>>
> >>>>>>> [1] :
> >>>>>>>
> >>>>>>
> >>>>
> >>
> https://ci.ignite.apache.org/viewType.html?buildTypeId=Releases_ApacheIgniteMain_ReleaseBuild
> >>>>>>>
> >>>>>>> пн, 28 сент. 2020 г. в 13:15, Alexey Goncharuk <
> >>>>>> alexey.goncharuk@gmail.com>:
> >>>>>>>
> >>>>>>>> Saikat, Nikolay,
> >>>>>>>>
> >>>>>>>> We have migrated a bunch of modules to ignite-extensions recently.
> >>>>>> Given
> >>>>>>>> that these modules will not be available in Ignite 2.9 anymore
> (will
> >>>>>>>> they?), should we also plan to release the extensions?
> >>>>>>>>
> >>>>>>>> ср, 23 сент. 2020 г. в 21:49, Alex Plehanov <
> >> plehanov.alex@gmail.com
> >>>>> :
> >>>>>>>>
> >>>>>>>>> Igniters,
> >>>>>>>>>
> >>>>>>>>> I've prepared release notes for the 2.9 release [1]. Can anyone
> >>>> review
> >>>>>>>> it,
> >>>>>>>>> please?
> >>>>>>>>>
> >>>>>>>>> [1]: https://github.com/apache/ignite/pull/8273
> >>>>>>>>>
> >>>>>>>>> вт, 22 сент. 2020 г. в 09:39, Alex Plehanov <
> >>>> plehanov.alex@gmail.com
> >>>>>>> :
> >>>>>>>>>
> >>>>>>>>>> Guys,
> >>>>>>>>>>
> >>>>>>>>>> I've filled the ticket with reproducer [1] for the discovery
> bug.
> >>>>>> This
> >>>>>>>>> bug
> >>>>>>>>>> caused by [2] ticket. We discussed with Vladimir Steshin
> privately
> >>>>>> and
> >>>>>>>>>> decided to revert this ticket. I will do it today (after TC bot
> >>>> visa)
> >>>>>>>> if
> >>>>>>>>>> there are no objections.
> >>>>>>>>>>
> >>>>>>>>>> [1]: https://issues.apache.org/jira/browse/IGNITE-13465
> >>>>>>>>>> [2]: https://issues.apache.org/jira/browse/IGNITE-13134
> >>>>>>>>>>
> >>>>>>>>>> пн, 21 сент. 2020 г. в 11:08, Alex Plehanov <
> >>>> plehanov.alex@gmail.com
> >>>>>>> :
> >>>>>>>>>>
> >>>>>>>>>>> Guys,
> >>>>>>>>>>>
> >>>>>>>>>>> During internal testing, we've found a critical bug with
> >>>>>>>>>>> discovery (cluster falls apart if two nodes segmented
> >>>> sequentially).
> >>>>>>>>> This
> >>>>>>>>>>> problem is not reproduced in 2.8.1. I think we should fix it
> >>>>>>>>>>> before release. Under investigation now. I'll let you know when
> >> we
> >>>>>> get
> >>>>>>>>>>> something.
> >>>>>>>>>>>
> >>>>>>>>>>> чт, 17 сент. 2020 г. в 00:51, Andrey Gura <ag...@apache.org>:
> >>>>>>>>>>>
> >>>>>>>>>>>>> So what do you think? Should we proceed with a 'hacked'
> version
> >>>> of
> >>>>>>>>> the
> >>>>>>>>>>>> message factory in 2.9 and go for the runtime message
> generation
> >>>> in
> >>>>>>>>> later
> >>>>>>>>>>>> release, or keep the code clean and fix the regression in the
> >>>> next
> >>>>>>>>> releases?
> >>>>>>>>>>>>> Andrey, can you take a look at my change? I think it is
> fairly
> >>>>>>>>>>>> straightforward and does not change the semantics, just skips
> >> the
> >>>>>>>>> factory
> >>>>>>>>>>>> closures for certain messages.
> >>>>>>>>>>>>
> >>>>>>>>>>>> IMHO 2.5% isn't too much especially because it isn't actual
> for
> >>>> all
> >>>>>>>>>>>> workloads (I didn't get any significant drops during
> >>>> benchmarking).
> >>>>>>>> So
> >>>>>>>>>>>> I prefer the runtime generation in later releases.
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Mon, Sep 14, 2020 at 12:41 PM Alexey Goncharuk
> >>>>>>>>>>>> <al...@gmail.com> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Alexey, Andrey, Igniters,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> So what do you think? Should we proceed with a 'hacked'
> version
> >>>> of
> >>>>>>>>> the
> >>>>>>>>>>>> message factory in 2.9 and go for the runtime message
> generation
> >>>> in
> >>>>>>>>> later
> >>>>>>>>>>>> release, or keep the code clean and fix the regression in the
> >>>> next
> >>>>>>>>> releases?
> >>>>>>>>>>>>> Andrey, can you take a look at my change? I think it is
> fairly
> >>>>>>>>>>>> straightforward and does not change the semantics, just skips
> >> the
> >>>>>>>>> factory
> >>>>>>>>>>>> closures for certain messages.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Personally, I would prefer fixing the regression given that
> we
> >>>>>> also
> >>>>>>>>>>>> introduced tracing in this release.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> пт, 11 сент. 2020 г. в 12:09, Alex Plehanov <
> >>>>>>>> plehanov.alex@gmail.com
> >>>>>>>>>> :
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Alexey,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> We've benchmarked by yardstick commits 130376741bf vs
> >>>> ed52559eb95
> >>>>>>>>> and
> >>>>>>>>>>>> the performance of ed52559eb95 is better for about 2.5% on
> >>>> average
> >>>>>> on
> >>>>>>>>> our
> >>>>>>>>>>>> environment (about the same results we got benchmarking
> 65c30ec6
> >>>> vs
> >>>>>>>>>>>> 0606f03d). We've made 24 runs for each commit of
> >>>>>>>>>>>> IgnitePutTxImplicitBenchmark (we got maximum drop for 2.9 on
> >> this
> >>>>>>>>>>>> benchmark), 200 seconds warmup, 300 seconds benchmark, 6
> >>>> servers, 5
> >>>>>>>>> clients
> >>>>>>>>>>>> 50 threads each. Yardstick results for this configuration:
> >>>>>>>>>>>>>> Commit 130376741bf: avg TPS=164096, avg latency=9173464 ns
> >>>>>>>>>>>>>> Commit ed52559eb95: avg TPS=168283, avg latency=8945908 ns
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> пт, 11 сент. 2020 г. в 09:51, Artem Budnikov <
> >>>>>>>>>>>> a.budnikov.ignite@gmail.com>:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi Everyone,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I posted an instruction on how to publish the docs on
> >>>>>>>>>>>> ignite.apache.org/docs [1]. When you finish with Ignite 2.9,
> >> you
> >>>>>> can
> >>>>>>>>>>>> update the docs by following the instruction. Unfortunately, I
> >>>>>> won't
> >>>>>>>> be
> >>>>>>>>>>>> able to spend any time on this project any longer. You can
> send
> >>>>>> your
> >>>>>>>>> pull
> >>>>>>>>>>>> requests and questions about the documentation to Denis Magda.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> -Artem
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> [1] :
> >>>>>>>>>>>>
> >>>> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Document
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Thu, Sep 10, 2020 at 2:45 PM Alexey Goncharuk <
> >>>>>>>>>>>> alexey.goncharuk@gmail.com> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Alexey,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I've tried to play with message factories locally, but
> >>>>>>>>>>>> unfortunately, I
> >>>>>>>>>>>>>>>> cannot spot the difference between old and new
> >> implementation
> >>>>>> in
> >>>>>>>>>>>>>>>> distributed benchmarks. I pushed an implementation of
> >>>>>>>>>>>> MessageFactoryImpl
> >>>>>>>>>>>>>>>> with the old switch statement to the
> ignite-2.9-revert-12568
> >>>>>>>>> branch
> >>>>>>>>>>>>>>>> (discussed this with Andrey Gura, the change should be
> >>>>>>>> compatible
> >>>>>>>>>>>> with the
> >>>>>>>>>>>>>>>> new metrics as we still use the register() mechanics).
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Can you check if this change makes any difference
> >>>>>>>> performance-wise
> >>>>>>>>>>>> in your
> >>>>>>>>>>>>>>>> environment? If yes, we can go with runtime code
> generation
> >>>> in
> >>>>>>>> the
> >>>>>>>>>>>> long
> >>>>>>>>>>>>>>>> term: register classes and generate a dynamic message
> >> factory
> >>>>>>>> with
> >>>>>>>>>>>> a switch
> >>>>>>>>>>>>>>>> statement once all messages are registered (not in 2.9
> >>>> though,
> >>>>>>>>>>>> obviously).
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> ср, 9 сент. 2020 г. в 14:53, Alex Plehanov <
> >>>>>>>>> plehanov.alex@gmail.com
> >>>>>>>>>>>>> :
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hello guys,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I've tried to optimize tracing implementation (ticket
> [1]),
> >>>> it
> >>>>>>>>>>>> reduced the
> >>>>>>>>>>>>>>>>> drop, but not completely removed it.
> >>>>>>>>>>>>>>>>> Ivan Rakov, Alexander Lapin, can you please review the
> >>>> patch?
> >>>>>>>>>>>>>>>>> Ivan Artiukhov, can you please benchmark the patch [2]
> >>>> against
> >>>>>>>>>>>> 2.8.1
> >>>>>>>>>>>>>>>>> release on your environment?
> >>>>>>>>>>>>>>>>> With this patch on our environment, it's about a 3% drop
> >>>> left,
> >>>>>>>>>>>> it's close
> >>>>>>>>>>>>>>>>> to measurement error and I think such a drop is not a
> >>>>>>>>>>>> showstopper. Guys,
> >>>>>>>>>>>>>>>>> WDYT?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Also, I found that compatibility is broken for JDBC thin
> >>>>>>>> driver
> >>>>>>>>>>>> between 2.8
> >>>>>>>>>>>>>>>>> and 2.9 versions (ticket [3]). I think it's a blocker and
> >>>>>>>> should
> >>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>> fixed in 2.9. I've prepared the patch.
> >>>>>>>>>>>>>>>>> Taras Ledkov, can you please review this patch?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> And one more ticket I propose to include into 2.9 [4]
> (NIO
> >>>>>>>>> message
> >>>>>>>>>>>>>>>>> send problem in some circumstances). I will cherry-pick
> it
> >>>> if
> >>>>>>>>>>>> there is no
> >>>>>>>>>>>>>>>>> objection.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13411
> >>>>>>>>>>>>>>>>> [2] https://github.com/apache/ignite/pull/8223
> >>>>>>>>>>>>>>>>> [3] https://issues.apache.org/jira/browse/IGNITE-13414
> >>>>>>>>>>>>>>>>> [4] https://issues.apache.org/jira/browse/IGNITE-13361
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> пн, 7 сент. 2020 г. в 14:14, Maxim Muzafarov <
> >>>>>>>> mmuzaf@apache.org
> >>>>>>>>>> :
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Alexey,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I propose to include [1] issue to the 2.9 release. Since
> >>>>>>>> this
> >>>>>>>>>>>> issue is
> >>>>>>>>>>>>>>>>>> related to the new master key change functionality which
> >>>>>>>>>>>> haven't been
> >>>>>>>>>>>>>>>>>> released yet I think it will be safe to cherry-pick
> commit
> >>>>>>>> to
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> release branch.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13390
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Tue, 1 Sep 2020 at 12:13, Nikolay Izhikov <
> >>>>>>>>>>>> nizhikov@apache.org>
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Hello, Igniters.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Alexey, please, include one more Python thin client fix
> >>>>>>>> [1]
> >>>>>>>>>>>> into the
> >>>>>>>>>>>>>>>>> 2.9
> >>>>>>>>>>>>>>>>>> release
> >>>>>>>>>>>>>>>>>>> It fixes kinda major issue - "Python client returns
> >> fields
> >>>>>>>>> in
> >>>>>>>>>>>> wrong
> >>>>>>>>>>>>>>>>>> order since the 2 row when fields_count>10"
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-12809
> >>>>>>>>>>>>>>>>>>> [2]
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://github.com/apache/ignite/commit/38025ee4167f05eaa2d6a2c5c2ab70c83a462cfc
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> 31 авг. 2020 г., в 19:23, Alexey Goncharuk <
> >>>>>>>>>>>>>>>>> alexey.goncharuk@gmail.com>
> >>>>>>>>>>>>>>>>>> написал(а):
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Alexey, thanks, got it. I am not sure we can optimize
> >>>>>>>>>>>> anything out of
> >>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>> message factory with suppliers (at least I have no
> ideas
> >>>>>>>>>>>> right now),
> >>>>>>>>>>>>>>>>> so
> >>>>>>>>>>>>>>>>>>>> most likely the only move here is to switch back to
> the
> >>>>>>>>>>>> switch
> >>>>>>>>>>>>>>>>> approach
> >>>>>>>>>>>>>>>>>>>> somehow preserving the metrics part. Probably,
> inlining
> >>>>>>>>> the
> >>>>>>>>>>>> Ignite
> >>>>>>>>>>>>>>>>>> messages
> >>>>>>>>>>>>>>>>>>>> to the IgniteMessageFactoryImpl should do the trick.
> Let
> >>>>>>>>> me
> >>>>>>>>>>>> explore
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>> code a bit.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> P.S. I am surprised by the impact this part makes for
> >>>>>>>> the
> >>>>>>>>>>>>>>>>> performance.
> >>>>>>>>>>>>>>>>>>>> Message creation is indeed on the hot path, but a
> single
> >>>>>>>>>>>> virtual call
> >>>>>>>>>>>>>>>>>>>> should not make that much of a difference given the
> >>>>>>>> amount
> >>>>>>>>>>>> of other
> >>>>>>>>>>>>>>>>>> work
> >>>>>>>>>>>>>>>>>>>> happening during the message processing.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 18:33, Alex Plehanov <
> >>>>>>>>>>>> plehanov.alex@gmail.com
> >>>>>>>>>>>>>>>>>> :
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Alexey, sorry, I wrongly interpreted our benchmark
> >>>>>>>>> results.
> >>>>>>>>>>>>>>>>> Actually,
> >>>>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>>> were looking for a drop using bi-sect in the range
> >>>>>>>>> between
> >>>>>>>>>>>> e6a7f93
> >>>>>>>>>>>>>>>>>> (first
> >>>>>>>>>>>>>>>>>>>>> commit in the 2.9 branch after 2.8 branch cut) and
> >>>>>>>>>>>> 6592dfa5 (last
> >>>>>>>>>>>>>>>>>> commit in
> >>>>>>>>>>>>>>>>>>>>> the 2.9 branch). And we found these two problematic
> >>>>>>>>>>>> commits.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Perhaps only IGNITE-13060 (Tracing) is responsible
> for
> >>>>>>>> a
> >>>>>>>>>>>> drop
> >>>>>>>>>>>>>>>>> between
> >>>>>>>>>>>>>>>>>>>>> 2.8.1 and 2.9 (we have benchmarked 2.8.1 vs 2.9 with
> >>>>>>>>>>>> reverted
> >>>>>>>>>>>>>>>>>> IGNITE-13060
> >>>>>>>>>>>>>>>>>>>>> now and performance looks the same)
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Ticket IGNITE-12568 (MessageFactory refactoring) is
> not
> >>>>>>>>>>>> related to
> >>>>>>>>>>>>>>>>>> drop
> >>>>>>>>>>>>>>>>>>>>> between 2.8.1 and 2.9, but still has some performance
> >>>>>>>>>>>> problem, and
> >>>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>>>> win back IGNITE-13060 drop by this ticket.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Do we need more investigation on IGNITE-13060 or we
> >>>>>>>> leave
> >>>>>>>>>>>> it as is?
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> What should we do with IGNITE-12568 (MessageFactory
> >>>>>>>>>>>> refactoring)?
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 13:25, Alexey Goncharuk <
> >>>>>>>>>>>>>>>>>> alexey.goncharuk@gmail.com
> >>>>>>>>>>>>>>>>>>>>>> :
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Alexey,
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> While investigating, I found that IGNITE-12568 has
> an
> >>>>>>>>>>>> incorrect fix
> >>>>>>>>>>>>>>>>>>>>>> version and is actually present in ignite-2.8.1
> branch
> >>>>>>>>>>>> [1], so it
> >>>>>>>>>>>>>>>>>> cannot be
> >>>>>>>>>>>>>>>>>>>>>> the source of the drop against 2.8.1.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> P.S. Looks like we need to enforce a more accurate
> >>>>>>>> work
> >>>>>>>>>>>> with fix
> >>>>>>>>>>>>>>>>>> versions
> >>>>>>>>>>>>>>>>>>>>>> or develop some sort of tooling to verify the fix
> >>>>>>>>>>>> versions.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> --AG
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> [1]
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://github.com/apache/ignite/commit/3e492bd23851856bbd0385c6a419892d0bba2a34
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 12:42, Alexey Goncharuk <
> >>>>>>>>>>>>>>>>>> alexey.goncharuk@gmail.com
> >>>>>>>>>>>>>>>>>>>>>>> :
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> пт, 28 авг. 2020 г. в 11:16, Alex Plehanov <
> >>>>>>>>>>>>>>>>> plehanov.alex@gmail.com
> >>>>>>>>>>>>>>>>>>> :
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Guys,
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> We have benchmarked 2.9 without IGNITE-13060 and
> >>>>>>>>>>>> IGNITE-12568
> >>>>>>>>>>>>>>>>>> (reverted
> >>>>>>>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>>>> locally) and got the same performance as on 2.8.1
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> IGNITE-13060 (Tracing) - some code was added to
> hot
> >>>>>>>>>>>> paths, to
> >>>>>>>>>>>>>>>>> trace
> >>>>>>>>>>>>>>>>>>>>>>>> these
> >>>>>>>>>>>>>>>>>>>>>>>> hot paths, it's clear why we have performance drop
> >>>>>>>>> here.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> IGNITE-12568 (MessageFactory refactoring) -
> >>>>>>>>> switch/case
> >>>>>>>>>>>> block was
> >>>>>>>>>>>>>>>>>>>>>>>> refactored to an array of message suppliers. The
> >>>>>>>>>>>> message factory
> >>>>>>>>>>>>>>>>>> is on
> >>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>> hot path, which explains why this commit has an
> >>>>>>>> impact
> >>>>>>>>>>>> on total
> >>>>>>>>>>>>>>>>>>>>>>>> performance.
> >>>>>>>>>>>>>>>>>>>>>>>> I've checked JIT assembly output, done some JMH
> >>>>>>>>>>>> microbenchmarks,
> >>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>> found
> >>>>>>>>>>>>>>>>>>>>>>>> that old implementation of MessageFactory.create()
> >>>>>>>>>>>> about 30-35%
> >>>>>>>>>>>>>>>>>> faster
> >>>>>>>>>>>>>>>>>>>>>>>> than
> >>>>>>>>>>>>>>>>>>>>>>>> the new one. The reason - approach with
> switch/case
> >>>>>>>>> can
> >>>>>>>>>>>>>>>>> effectively
> >>>>>>>>>>>>>>>>>>>>>>>> inline
> >>>>>>>>>>>>>>>>>>>>>>>> message creation code, but with an array of
> >>>>>>>> suppliers
> >>>>>>>>>>>> relatively
> >>>>>>>>>>>>>>>>>> heavy
> >>>>>>>>>>>>>>>>>>>>>>>> "invokeinterface" cannot be skipped. I've tried to
> >>>>>>>>>>>> rewrite the
> >>>>>>>>>>>>>>>>> code
> >>>>>>>>>>>>>>>>>>>>>>>> using
> >>>>>>>>>>>>>>>>>>>>>>>> an abstract class for suppliers instead of an
> >>>>>>>>> interface
> >>>>>>>>>>>> (to
> >>>>>>>>>>>>>>>>>>>>>>>> replace "invokeinterface" with the
> "invokevirtual"),
> >>>>>>>>>>>> but it gives
> >>>>>>>>>>>>>>>>>> back
> >>>>>>>>>>>>>>>>>>>>>>>> only
> >>>>>>>>>>>>>>>>>>>>>>>> 10% of method performance and in this case, code
> >>>>>>>> looks
> >>>>>>>>>>>> ugly
> >>>>>>>>>>>>>>>>>> (lambdas
> >>>>>>>>>>>>>>>>>>>>>>>> can't
> >>>>>>>>>>>>>>>>>>>>>>>> be used). Currently, I can't find any more ways to
> >>>>>>>>>>>> optimize the
> >>>>>>>>>>>>>>>>>> current
> >>>>>>>>>>>>>>>>>>>>>>>> approach (except return to the switch/case block).
> >>>>>>>>>>>> Andrey Gura,
> >>>>>>>>>>>>>>>>> as
> >>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>> author of IGNITE-12568, maybe you have some ideas
> >>>>>>>>> about
> >>>>>>>>>>>>>>>>>> optimization?
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Perhaps we should revert IGNITE-12568, but there
> are
> >>>>>>>>>>>> some metrics
> >>>>>>>>>>>>>>>>>>>>>>>> already
> >>>>>>>>>>>>>>>>>>>>>>>> created, which can't be rewritten using old
> message
> >>>>>>>>>>>> factory
> >>>>>>>>>>>>>>>>>>>>>>>> implementation
> >>>>>>>>>>>>>>>>>>>>>>>> (IGNITE-12756). Guys, WDYT?
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Alexey,
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> I see that IGNITE-12756 (metrics improvements) is
> >>>>>>>>>>>> already released
> >>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>> Ignite 2.8.1 while IGNITE-12568 (message factory)
> is
> >>>>>>>>>>>> only present
> >>>>>>>>>>>>>>>>>> in Ignite
> >>>>>>>>>>>>>>>>>>>>>>> 2.9. Let's revert both IGNITE-12568 and whichever
> new
> >>>>>>>>>>>> metrics
> >>>>>>>>>>>>>>>>>> created for
> >>>>>>>>>>>>>>>>>>>>>>> 2.9 that depend on the new message factory to
> unblock
> >>>>>>>>>>>> the release
> >>>>>>>>>>>>>>>>>> and deal
> >>>>>>>>>>>>>>>>>>>>>>> with the optimizations in 2.10?
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>>
> >>>>
> >>
> >>
>
>
Re: Apache Ignite 2.9.0 RELEASE [Time, Scope, Manager]
Posted by Nikolay Izhikov <ni...@apache.org>.
Igniters,
IGNITE-13553 fixed and cherry-picked to 2.9.
We can continue with the release.
> 10 окт. 2020 г., в 00:00, Denis Magda <dm...@apache.org> написал(а):
>
> Nikolay,
>
> re: the minor improvements. I'm not against of including those if we can
> prepare the docs before the vote starts. Presently, the docs are "frozen"
> for the 2.9 release, but I can scratch some time and take part in the docs
> review the next week.
>
> -
> Denis
>
>
> On Fri, Oct 9, 2020 at 12:40 AM Nikolay Izhikov <ni...@apache.org> wrote:
>
>> Hello, Igniters.
>>
>> I prepared a patch [1] for blocker ticket [2] - «Server node fail and
>> stops in case wrong datatype put in indexed field»
>> Can someone, please, help me with the review?
>>
>> [1] https://github.com/apache/ignite/pull/8330
>> [2] https://issues.apache.org/jira/browse/IGNITE-13553
>>
>> ==================
>>
>> I propose to include several minor tickets to the scope of the 2.9 that
>> increase Ignite User Experience
>>
>> CMD tools improvements:
>>
>> IGNITE-13488 - Command to print metric value
>> IGNITE-13426 - Command to print system view content
>> IGNITE-13422 - Parameter to explicitly enable experimental commands
>>
>> IGNITE-13380 - Output IgniteSystemProperties via ignite.sh
>>
>> New system views:
>>
>> IGNITE-13409 Metastorage and DistributedMetastorage viewы.
>> IGNITE-13408 BinaryMetadata view.
>>
>>> 9 окт. 2020 г., в 04:04, Denis Magda <dm...@apache.org> написал(а):
>>>
>>> @Alex Plehanov <pl...@gmail.com>,
>>>
>>> If it still makes sense and not too late, you can cherry-pick the commit
>>> with the new docs into the 2.9 branch:
>>>
>> https://github.com/apache/ignite/commit/073488ac97517bbaad9f6b94b781fc404646f191
>>>
>>> Anyway, it's not crucial as long as we agreed to make an exception for
>> this
>>> release. The docs were already published to the website and assembled
>> from
>>> the master. Once the vote passes, I'll make them visible and traceable
>> from
>>> the website menus.
>>>
>>> -
>>> Denis
>>>
>>>
>>> On Tue, Oct 6, 2020 at 7:20 AM Alexey Goncharuk <
>> alexey.goncharuk@gmail.com>
>>> wrote:
>>>
>>>> Saikat,
>>>>
>>>> The plan sounds good to me! Do you have an idea for the timeline of the
>>>> module releases? Let me know if I can help in any way.
>>>>
>>>> Also, I was looking into the modularization IEP and noticed that OSGI
>>>> module is discussed to be moved to the extensions, but there is no
>>>> corresponding ticket. Should I create one? I will be happy to help with
>>>> moving this module to extensions.
>>>>
>>>> вт, 29 сент. 2020 г. в 03:03, Saikat Maitra <sa...@gmail.com>:
>>>>
>>>>> Hi Alexey,
>>>>>
>>>>> All the modules as planned in IEP-36
>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36:+Modularization
>>>>> are migrated to ignite-extensions repository.
>>>>>
>>>>> We can definitely plan to release ignite-extensions modules.
>>>>>
>>>>> I have a pending PR related to the kafka module and the PR is in review
>>>>> https://github.com/apache/ignite/pull/8222 but its corresponding PR
>> has
>>>>> been merged in ignite-extensions repository.
>>>>>
>>>>> My thoughts / action items for the migration efforts and releases were
>>>>> as follows:
>>>>>
>>>>> 1. Merge the changes for https://github.com/apache/ignite/pull/8222
>>>>> 2. Fix the upload nightly packages task for ignite-core to help fix the
>>>>> travis build failure in ignite-extensions repository.
>>>>> 3. Review and update the README.md files for the ignite-extensions
>>>> modules
>>>>> as required.
>>>>> 4. Update the docs for ignite-extensions modules in ignite-website.
>>>>> 5. Release each module separately and share updates.
>>>>>
>>>>> Please let me know if you have feedback.
>>>>>
>>>>> Regards,
>>>>> Saikat
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Sep 28, 2020 at 7:36 AM Petr Ivanov <mr...@gmail.com>
>> wrote:
>>>>>
>>>>>> It seems that gitbox.apache.org is not available on CI/CD.
>>>>>>
>>>>>> I will try to update VCS to GitHub.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> On 28 Sep 2020, at 14:07, Alex Plehanov <pl...@gmail.com>
>>>>>> wrote:
>>>>>>>
>>>>>>> Guys,
>>>>>>>
>>>>>>> I've tried to build release candidate using TC [1]. But:
>>>>>>> 1. I can't choose a branch in this TC task (there is no such field).
>>>>>>> 2. On the "default" branch I got an error: Failed to collect changes,
>>>>>>> error: List remote refs failed:
>>>>>>> org.apache.http.conn.ConnectTimeoutException: Connect to
>>>>>>> gitbox.apache.org:443 [gitbox.apache.org/52.202.80.70] failed:
>>>> connect
>>>>>>> timed out, VCS root: "GitBox [asf/ignite]" {instance id=300, parent
>>>>>>> internal id=74, parent id=GitBoxAsfIgnite, description: "
>>>>>>> https://gitbox.apache.org/repos/asf/ignite.git#refs/heads/master"}
>>>>>>>
>>>>>>> Is there some problem with this TC task? What am I doing wrong?
>>>>>>>
>>>>>>> [1] :
>>>>>>>
>>>>>>
>>>>
>> https://ci.ignite.apache.org/viewType.html?buildTypeId=Releases_ApacheIgniteMain_ReleaseBuild
>>>>>>>
>>>>>>> пн, 28 сент. 2020 г. в 13:15, Alexey Goncharuk <
>>>>>> alexey.goncharuk@gmail.com>:
>>>>>>>
>>>>>>>> Saikat, Nikolay,
>>>>>>>>
>>>>>>>> We have migrated a bunch of modules to ignite-extensions recently.
>>>>>> Given
>>>>>>>> that these modules will not be available in Ignite 2.9 anymore (will
>>>>>>>> they?), should we also plan to release the extensions?
>>>>>>>>
>>>>>>>> ср, 23 сент. 2020 г. в 21:49, Alex Plehanov <
>> plehanov.alex@gmail.com
>>>>> :
>>>>>>>>
>>>>>>>>> Igniters,
>>>>>>>>>
>>>>>>>>> I've prepared release notes for the 2.9 release [1]. Can anyone
>>>> review
>>>>>>>> it,
>>>>>>>>> please?
>>>>>>>>>
>>>>>>>>> [1]: https://github.com/apache/ignite/pull/8273
>>>>>>>>>
>>>>>>>>> вт, 22 сент. 2020 г. в 09:39, Alex Plehanov <
>>>> plehanov.alex@gmail.com
>>>>>>> :
>>>>>>>>>
>>>>>>>>>> Guys,
>>>>>>>>>>
>>>>>>>>>> I've filled the ticket with reproducer [1] for the discovery bug.
>>>>>> This
>>>>>>>>> bug
>>>>>>>>>> caused by [2] ticket. We discussed with Vladimir Steshin privately
>>>>>> and
>>>>>>>>>> decided to revert this ticket. I will do it today (after TC bot
>>>> visa)
>>>>>>>> if
>>>>>>>>>> there are no objections.
>>>>>>>>>>
>>>>>>>>>> [1]: https://issues.apache.org/jira/browse/IGNITE-13465
>>>>>>>>>> [2]: https://issues.apache.org/jira/browse/IGNITE-13134
>>>>>>>>>>
>>>>>>>>>> пн, 21 сент. 2020 г. в 11:08, Alex Plehanov <
>>>> plehanov.alex@gmail.com
>>>>>>> :
>>>>>>>>>>
>>>>>>>>>>> Guys,
>>>>>>>>>>>
>>>>>>>>>>> During internal testing, we've found a critical bug with
>>>>>>>>>>> discovery (cluster falls apart if two nodes segmented
>>>> sequentially).
>>>>>>>>> This
>>>>>>>>>>> problem is not reproduced in 2.8.1. I think we should fix it
>>>>>>>>>>> before release. Under investigation now. I'll let you know when
>> we
>>>>>> get
>>>>>>>>>>> something.
>>>>>>>>>>>
>>>>>>>>>>> чт, 17 сент. 2020 г. в 00:51, Andrey Gura <ag...@apache.org>:
>>>>>>>>>>>
>>>>>>>>>>>>> So what do you think? Should we proceed with a 'hacked' version
>>>> of
>>>>>>>>> the
>>>>>>>>>>>> message factory in 2.9 and go for the runtime message generation
>>>> in
>>>>>>>>> later
>>>>>>>>>>>> release, or keep the code clean and fix the regression in the
>>>> next
>>>>>>>>> releases?
>>>>>>>>>>>>> Andrey, can you take a look at my change? I think it is fairly
>>>>>>>>>>>> straightforward and does not change the semantics, just skips
>> the
>>>>>>>>> factory
>>>>>>>>>>>> closures for certain messages.
>>>>>>>>>>>>
>>>>>>>>>>>> IMHO 2.5% isn't too much especially because it isn't actual for
>>>> all
>>>>>>>>>>>> workloads (I didn't get any significant drops during
>>>> benchmarking).
>>>>>>>> So
>>>>>>>>>>>> I prefer the runtime generation in later releases.
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Sep 14, 2020 at 12:41 PM Alexey Goncharuk
>>>>>>>>>>>> <al...@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Alexey, Andrey, Igniters,
>>>>>>>>>>>>>
>>>>>>>>>>>>> So what do you think? Should we proceed with a 'hacked' version
>>>> of
>>>>>>>>> the
>>>>>>>>>>>> message factory in 2.9 and go for the runtime message generation
>>>> in
>>>>>>>>> later
>>>>>>>>>>>> release, or keep the code clean and fix the regression in the
>>>> next
>>>>>>>>> releases?
>>>>>>>>>>>>> Andrey, can you take a look at my change? I think it is fairly
>>>>>>>>>>>> straightforward and does not change the semantics, just skips
>> the
>>>>>>>>> factory
>>>>>>>>>>>> closures for certain messages.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Personally, I would prefer fixing the regression given that we
>>>>>> also
>>>>>>>>>>>> introduced tracing in this release.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> пт, 11 сент. 2020 г. в 12:09, Alex Plehanov <
>>>>>>>> plehanov.alex@gmail.com
>>>>>>>>>> :
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Alexey,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We've benchmarked by yardstick commits 130376741bf vs
>>>> ed52559eb95
>>>>>>>>> and
>>>>>>>>>>>> the performance of ed52559eb95 is better for about 2.5% on
>>>> average
>>>>>> on
>>>>>>>>> our
>>>>>>>>>>>> environment (about the same results we got benchmarking 65c30ec6
>>>> vs
>>>>>>>>>>>> 0606f03d). We've made 24 runs for each commit of
>>>>>>>>>>>> IgnitePutTxImplicitBenchmark (we got maximum drop for 2.9 on
>> this
>>>>>>>>>>>> benchmark), 200 seconds warmup, 300 seconds benchmark, 6
>>>> servers, 5
>>>>>>>>> clients
>>>>>>>>>>>> 50 threads each. Yardstick results for this configuration:
>>>>>>>>>>>>>> Commit 130376741bf: avg TPS=164096, avg latency=9173464 ns
>>>>>>>>>>>>>> Commit ed52559eb95: avg TPS=168283, avg latency=8945908 ns
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> пт, 11 сент. 2020 г. в 09:51, Artem Budnikov <
>>>>>>>>>>>> a.budnikov.ignite@gmail.com>:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I posted an instruction on how to publish the docs on
>>>>>>>>>>>> ignite.apache.org/docs [1]. When you finish with Ignite 2.9,
>> you
>>>>>> can
>>>>>>>>>>>> update the docs by following the instruction. Unfortunately, I
>>>>>> won't
>>>>>>>> be
>>>>>>>>>>>> able to spend any time on this project any longer. You can send
>>>>>> your
>>>>>>>>> pull
>>>>>>>>>>>> requests and questions about the documentation to Denis Magda.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -Artem
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [1] :
>>>>>>>>>>>>
>>>> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Document
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Thu, Sep 10, 2020 at 2:45 PM Alexey Goncharuk <
>>>>>>>>>>>> alexey.goncharuk@gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Alexey,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I've tried to play with message factories locally, but
>>>>>>>>>>>> unfortunately, I
>>>>>>>>>>>>>>>> cannot spot the difference between old and new
>> implementation
>>>>>> in
>>>>>>>>>>>>>>>> distributed benchmarks. I pushed an implementation of
>>>>>>>>>>>> MessageFactoryImpl
>>>>>>>>>>>>>>>> with the old switch statement to the ignite-2.9-revert-12568
>>>>>>>>> branch
>>>>>>>>>>>>>>>> (discussed this with Andrey Gura, the change should be
>>>>>>>> compatible
>>>>>>>>>>>> with the
>>>>>>>>>>>>>>>> new metrics as we still use the register() mechanics).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Can you check if this change makes any difference
>>>>>>>> performance-wise
>>>>>>>>>>>> in your
>>>>>>>>>>>>>>>> environment? If yes, we can go with runtime code generation
>>>> in
>>>>>>>> the
>>>>>>>>>>>> long
>>>>>>>>>>>>>>>> term: register classes and generate a dynamic message
>> factory
>>>>>>>> with
>>>>>>>>>>>> a switch
>>>>>>>>>>>>>>>> statement once all messages are registered (not in 2.9
>>>> though,
>>>>>>>>>>>> obviously).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ср, 9 сент. 2020 г. в 14:53, Alex Plehanov <
>>>>>>>>> plehanov.alex@gmail.com
>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hello guys,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I've tried to optimize tracing implementation (ticket [1]),
>>>> it
>>>>>>>>>>>> reduced the
>>>>>>>>>>>>>>>>> drop, but not completely removed it.
>>>>>>>>>>>>>>>>> Ivan Rakov, Alexander Lapin, can you please review the
>>>> patch?
>>>>>>>>>>>>>>>>> Ivan Artiukhov, can you please benchmark the patch [2]
>>>> against
>>>>>>>>>>>> 2.8.1
>>>>>>>>>>>>>>>>> release on your environment?
>>>>>>>>>>>>>>>>> With this patch on our environment, it's about a 3% drop
>>>> left,
>>>>>>>>>>>> it's close
>>>>>>>>>>>>>>>>> to measurement error and I think such a drop is not a
>>>>>>>>>>>> showstopper. Guys,
>>>>>>>>>>>>>>>>> WDYT?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Also, I found that compatibility is broken for JDBC thin
>>>>>>>> driver
>>>>>>>>>>>> between 2.8
>>>>>>>>>>>>>>>>> and 2.9 versions (ticket [3]). I think it's a blocker and
>>>>>>>> should
>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>> fixed in 2.9. I've prepared the patch.
>>>>>>>>>>>>>>>>> Taras Ledkov, can you please review this patch?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> And one more ticket I propose to include into 2.9 [4] (NIO
>>>>>>>>> message
>>>>>>>>>>>>>>>>> send problem in some circumstances). I will cherry-pick it
>>>> if
>>>>>>>>>>>> there is no
>>>>>>>>>>>>>>>>> objection.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13411
>>>>>>>>>>>>>>>>> [2] https://github.com/apache/ignite/pull/8223
>>>>>>>>>>>>>>>>> [3] https://issues.apache.org/jira/browse/IGNITE-13414
>>>>>>>>>>>>>>>>> [4] https://issues.apache.org/jira/browse/IGNITE-13361
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> пн, 7 сент. 2020 г. в 14:14, Maxim Muzafarov <
>>>>>>>> mmuzaf@apache.org
>>>>>>>>>> :
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Alexey,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I propose to include [1] issue to the 2.9 release. Since
>>>>>>>> this
>>>>>>>>>>>> issue is
>>>>>>>>>>>>>>>>>> related to the new master key change functionality which
>>>>>>>>>>>> haven't been
>>>>>>>>>>>>>>>>>> released yet I think it will be safe to cherry-pick commit
>>>>>>>> to
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> release branch.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13390
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Tue, 1 Sep 2020 at 12:13, Nikolay Izhikov <
>>>>>>>>>>>> nizhikov@apache.org>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hello, Igniters.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Alexey, please, include one more Python thin client fix
>>>>>>>> [1]
>>>>>>>>>>>> into the
>>>>>>>>>>>>>>>>> 2.9
>>>>>>>>>>>>>>>>>> release
>>>>>>>>>>>>>>>>>>> It fixes kinda major issue - "Python client returns
>> fields
>>>>>>>>> in
>>>>>>>>>>>> wrong
>>>>>>>>>>>>>>>>>> order since the 2 row when fields_count>10"
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-12809
>>>>>>>>>>>>>>>>>>> [2]
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> https://github.com/apache/ignite/commit/38025ee4167f05eaa2d6a2c5c2ab70c83a462cfc
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 31 авг. 2020 г., в 19:23, Alexey Goncharuk <
>>>>>>>>>>>>>>>>> alexey.goncharuk@gmail.com>
>>>>>>>>>>>>>>>>>> написал(а):
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Alexey, thanks, got it. I am not sure we can optimize
>>>>>>>>>>>> anything out of
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> message factory with suppliers (at least I have no ideas
>>>>>>>>>>>> right now),
>>>>>>>>>>>>>>>>> so
>>>>>>>>>>>>>>>>>>>> most likely the only move here is to switch back to the
>>>>>>>>>>>> switch
>>>>>>>>>>>>>>>>> approach
>>>>>>>>>>>>>>>>>>>> somehow preserving the metrics part. Probably, inlining
>>>>>>>>> the
>>>>>>>>>>>> Ignite
>>>>>>>>>>>>>>>>>> messages
>>>>>>>>>>>>>>>>>>>> to the IgniteMessageFactoryImpl should do the trick. Let
>>>>>>>>> me
>>>>>>>>>>>> explore
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> code a bit.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> P.S. I am surprised by the impact this part makes for
>>>>>>>> the
>>>>>>>>>>>>>>>>> performance.
>>>>>>>>>>>>>>>>>>>> Message creation is indeed on the hot path, but a single
>>>>>>>>>>>> virtual call
>>>>>>>>>>>>>>>>>>>> should not make that much of a difference given the
>>>>>>>> amount
>>>>>>>>>>>> of other
>>>>>>>>>>>>>>>>>> work
>>>>>>>>>>>>>>>>>>>> happening during the message processing.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 18:33, Alex Plehanov <
>>>>>>>>>>>> plehanov.alex@gmail.com
>>>>>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Alexey, sorry, I wrongly interpreted our benchmark
>>>>>>>>> results.
>>>>>>>>>>>>>>>>> Actually,
>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>> were looking for a drop using bi-sect in the range
>>>>>>>>> between
>>>>>>>>>>>> e6a7f93
>>>>>>>>>>>>>>>>>> (first
>>>>>>>>>>>>>>>>>>>>> commit in the 2.9 branch after 2.8 branch cut) and
>>>>>>>>>>>> 6592dfa5 (last
>>>>>>>>>>>>>>>>>> commit in
>>>>>>>>>>>>>>>>>>>>> the 2.9 branch). And we found these two problematic
>>>>>>>>>>>> commits.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Perhaps only IGNITE-13060 (Tracing) is responsible for
>>>>>>>> a
>>>>>>>>>>>> drop
>>>>>>>>>>>>>>>>> between
>>>>>>>>>>>>>>>>>>>>> 2.8.1 and 2.9 (we have benchmarked 2.8.1 vs 2.9 with
>>>>>>>>>>>> reverted
>>>>>>>>>>>>>>>>>> IGNITE-13060
>>>>>>>>>>>>>>>>>>>>> now and performance looks the same)
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Ticket IGNITE-12568 (MessageFactory refactoring) is not
>>>>>>>>>>>> related to
>>>>>>>>>>>>>>>>>> drop
>>>>>>>>>>>>>>>>>>>>> between 2.8.1 and 2.9, but still has some performance
>>>>>>>>>>>> problem, and
>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>> win back IGNITE-13060 drop by this ticket.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Do we need more investigation on IGNITE-13060 or we
>>>>>>>> leave
>>>>>>>>>>>> it as is?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> What should we do with IGNITE-12568 (MessageFactory
>>>>>>>>>>>> refactoring)?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 13:25, Alexey Goncharuk <
>>>>>>>>>>>>>>>>>> alexey.goncharuk@gmail.com
>>>>>>>>>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Alexey,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> While investigating, I found that IGNITE-12568 has an
>>>>>>>>>>>> incorrect fix
>>>>>>>>>>>>>>>>>>>>>> version and is actually present in ignite-2.8.1 branch
>>>>>>>>>>>> [1], so it
>>>>>>>>>>>>>>>>>> cannot be
>>>>>>>>>>>>>>>>>>>>>> the source of the drop against 2.8.1.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> P.S. Looks like we need to enforce a more accurate
>>>>>>>> work
>>>>>>>>>>>> with fix
>>>>>>>>>>>>>>>>>> versions
>>>>>>>>>>>>>>>>>>>>>> or develop some sort of tooling to verify the fix
>>>>>>>>>>>> versions.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> --AG
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> https://github.com/apache/ignite/commit/3e492bd23851856bbd0385c6a419892d0bba2a34
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 12:42, Alexey Goncharuk <
>>>>>>>>>>>>>>>>>> alexey.goncharuk@gmail.com
>>>>>>>>>>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> пт, 28 авг. 2020 г. в 11:16, Alex Plehanov <
>>>>>>>>>>>>>>>>> plehanov.alex@gmail.com
>>>>>>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Guys,
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> We have benchmarked 2.9 without IGNITE-13060 and
>>>>>>>>>>>> IGNITE-12568
>>>>>>>>>>>>>>>>>> (reverted
>>>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>> locally) and got the same performance as on 2.8.1
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> IGNITE-13060 (Tracing) - some code was added to hot
>>>>>>>>>>>> paths, to
>>>>>>>>>>>>>>>>> trace
>>>>>>>>>>>>>>>>>>>>>>>> these
>>>>>>>>>>>>>>>>>>>>>>>> hot paths, it's clear why we have performance drop
>>>>>>>>> here.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> IGNITE-12568 (MessageFactory refactoring) -
>>>>>>>>> switch/case
>>>>>>>>>>>> block was
>>>>>>>>>>>>>>>>>>>>>>>> refactored to an array of message suppliers. The
>>>>>>>>>>>> message factory
>>>>>>>>>>>>>>>>>> is on
>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> hot path, which explains why this commit has an
>>>>>>>> impact
>>>>>>>>>>>> on total
>>>>>>>>>>>>>>>>>>>>>>>> performance.
>>>>>>>>>>>>>>>>>>>>>>>> I've checked JIT assembly output, done some JMH
>>>>>>>>>>>> microbenchmarks,
>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>> found
>>>>>>>>>>>>>>>>>>>>>>>> that old implementation of MessageFactory.create()
>>>>>>>>>>>> about 30-35%
>>>>>>>>>>>>>>>>>> faster
>>>>>>>>>>>>>>>>>>>>>>>> than
>>>>>>>>>>>>>>>>>>>>>>>> the new one. The reason - approach with switch/case
>>>>>>>>> can
>>>>>>>>>>>>>>>>> effectively
>>>>>>>>>>>>>>>>>>>>>>>> inline
>>>>>>>>>>>>>>>>>>>>>>>> message creation code, but with an array of
>>>>>>>> suppliers
>>>>>>>>>>>> relatively
>>>>>>>>>>>>>>>>>> heavy
>>>>>>>>>>>>>>>>>>>>>>>> "invokeinterface" cannot be skipped. I've tried to
>>>>>>>>>>>> rewrite the
>>>>>>>>>>>>>>>>> code
>>>>>>>>>>>>>>>>>>>>>>>> using
>>>>>>>>>>>>>>>>>>>>>>>> an abstract class for suppliers instead of an
>>>>>>>>> interface
>>>>>>>>>>>> (to
>>>>>>>>>>>>>>>>>>>>>>>> replace "invokeinterface" with the "invokevirtual"),
>>>>>>>>>>>> but it gives
>>>>>>>>>>>>>>>>>> back
>>>>>>>>>>>>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>>>>>>>>>> 10% of method performance and in this case, code
>>>>>>>> looks
>>>>>>>>>>>> ugly
>>>>>>>>>>>>>>>>>> (lambdas
>>>>>>>>>>>>>>>>>>>>>>>> can't
>>>>>>>>>>>>>>>>>>>>>>>> be used). Currently, I can't find any more ways to
>>>>>>>>>>>> optimize the
>>>>>>>>>>>>>>>>>> current
>>>>>>>>>>>>>>>>>>>>>>>> approach (except return to the switch/case block).
>>>>>>>>>>>> Andrey Gura,
>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> author of IGNITE-12568, maybe you have some ideas
>>>>>>>>> about
>>>>>>>>>>>>>>>>>> optimization?
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Perhaps we should revert IGNITE-12568, but there are
>>>>>>>>>>>> some metrics
>>>>>>>>>>>>>>>>>>>>>>>> already
>>>>>>>>>>>>>>>>>>>>>>>> created, which can't be rewritten using old message
>>>>>>>>>>>> factory
>>>>>>>>>>>>>>>>>>>>>>>> implementation
>>>>>>>>>>>>>>>>>>>>>>>> (IGNITE-12756). Guys, WDYT?
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Alexey,
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I see that IGNITE-12756 (metrics improvements) is
>>>>>>>>>>>> already released
>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>> Ignite 2.8.1 while IGNITE-12568 (message factory) is
>>>>>>>>>>>> only present
>>>>>>>>>>>>>>>>>> in Ignite
>>>>>>>>>>>>>>>>>>>>>>> 2.9. Let's revert both IGNITE-12568 and whichever new
>>>>>>>>>>>> metrics
>>>>>>>>>>>>>>>>>> created for
>>>>>>>>>>>>>>>>>>>>>>> 2.9 that depend on the new message factory to unblock
>>>>>>>>>>>> the release
>>>>>>>>>>>>>>>>>> and deal
>>>>>>>>>>>>>>>>>>>>>>> with the optimizations in 2.10?
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>
>>
>>
Re: Apache Ignite 2.9.0 RELEASE [Time, Scope, Manager]
Posted by Denis Magda <dm...@apache.org>.
Nikolay,
re: the minor improvements. I'm not against of including those if we can
prepare the docs before the vote starts. Presently, the docs are "frozen"
for the 2.9 release, but I can scratch some time and take part in the docs
review the next week.
-
Denis
On Fri, Oct 9, 2020 at 12:40 AM Nikolay Izhikov <ni...@apache.org> wrote:
> Hello, Igniters.
>
> I prepared a patch [1] for blocker ticket [2] - «Server node fail and
> stops in case wrong datatype put in indexed field»
> Can someone, please, help me with the review?
>
> [1] https://github.com/apache/ignite/pull/8330
> [2] https://issues.apache.org/jira/browse/IGNITE-13553
>
> ==================
>
> I propose to include several minor tickets to the scope of the 2.9 that
> increase Ignite User Experience
>
> CMD tools improvements:
>
> IGNITE-13488 - Command to print metric value
> IGNITE-13426 - Command to print system view content
> IGNITE-13422 - Parameter to explicitly enable experimental commands
>
> IGNITE-13380 - Output IgniteSystemProperties via ignite.sh
>
> New system views:
>
> IGNITE-13409 Metastorage and DistributedMetastorage viewы.
> IGNITE-13408 BinaryMetadata view.
>
> > 9 окт. 2020 г., в 04:04, Denis Magda <dm...@apache.org> написал(а):
> >
> > @Alex Plehanov <pl...@gmail.com>,
> >
> > If it still makes sense and not too late, you can cherry-pick the commit
> > with the new docs into the 2.9 branch:
> >
> https://github.com/apache/ignite/commit/073488ac97517bbaad9f6b94b781fc404646f191
> >
> > Anyway, it's not crucial as long as we agreed to make an exception for
> this
> > release. The docs were already published to the website and assembled
> from
> > the master. Once the vote passes, I'll make them visible and traceable
> from
> > the website menus.
> >
> > -
> > Denis
> >
> >
> > On Tue, Oct 6, 2020 at 7:20 AM Alexey Goncharuk <
> alexey.goncharuk@gmail.com>
> > wrote:
> >
> >> Saikat,
> >>
> >> The plan sounds good to me! Do you have an idea for the timeline of the
> >> module releases? Let me know if I can help in any way.
> >>
> >> Also, I was looking into the modularization IEP and noticed that OSGI
> >> module is discussed to be moved to the extensions, but there is no
> >> corresponding ticket. Should I create one? I will be happy to help with
> >> moving this module to extensions.
> >>
> >> вт, 29 сент. 2020 г. в 03:03, Saikat Maitra <sa...@gmail.com>:
> >>
> >>> Hi Alexey,
> >>>
> >>> All the modules as planned in IEP-36
> >>>
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36:+Modularization
> >>> are migrated to ignite-extensions repository.
> >>>
> >>> We can definitely plan to release ignite-extensions modules.
> >>>
> >>> I have a pending PR related to the kafka module and the PR is in review
> >>> https://github.com/apache/ignite/pull/8222 but its corresponding PR
> has
> >>> been merged in ignite-extensions repository.
> >>>
> >>> My thoughts / action items for the migration efforts and releases were
> >>> as follows:
> >>>
> >>> 1. Merge the changes for https://github.com/apache/ignite/pull/8222
> >>> 2. Fix the upload nightly packages task for ignite-core to help fix the
> >>> travis build failure in ignite-extensions repository.
> >>> 3. Review and update the README.md files for the ignite-extensions
> >> modules
> >>> as required.
> >>> 4. Update the docs for ignite-extensions modules in ignite-website.
> >>> 5. Release each module separately and share updates.
> >>>
> >>> Please let me know if you have feedback.
> >>>
> >>> Regards,
> >>> Saikat
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On Mon, Sep 28, 2020 at 7:36 AM Petr Ivanov <mr...@gmail.com>
> wrote:
> >>>
> >>>> It seems that gitbox.apache.org is not available on CI/CD.
> >>>>
> >>>> I will try to update VCS to GitHub.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>> On 28 Sep 2020, at 14:07, Alex Plehanov <pl...@gmail.com>
> >>>> wrote:
> >>>>>
> >>>>> Guys,
> >>>>>
> >>>>> I've tried to build release candidate using TC [1]. But:
> >>>>> 1. I can't choose a branch in this TC task (there is no such field).
> >>>>> 2. On the "default" branch I got an error: Failed to collect changes,
> >>>>> error: List remote refs failed:
> >>>>> org.apache.http.conn.ConnectTimeoutException: Connect to
> >>>>> gitbox.apache.org:443 [gitbox.apache.org/52.202.80.70] failed:
> >> connect
> >>>>> timed out, VCS root: "GitBox [asf/ignite]" {instance id=300, parent
> >>>>> internal id=74, parent id=GitBoxAsfIgnite, description: "
> >>>>> https://gitbox.apache.org/repos/asf/ignite.git#refs/heads/master"}
> >>>>>
> >>>>> Is there some problem with this TC task? What am I doing wrong?
> >>>>>
> >>>>> [1] :
> >>>>>
> >>>>
> >>
> https://ci.ignite.apache.org/viewType.html?buildTypeId=Releases_ApacheIgniteMain_ReleaseBuild
> >>>>>
> >>>>> пн, 28 сент. 2020 г. в 13:15, Alexey Goncharuk <
> >>>> alexey.goncharuk@gmail.com>:
> >>>>>
> >>>>>> Saikat, Nikolay,
> >>>>>>
> >>>>>> We have migrated a bunch of modules to ignite-extensions recently.
> >>>> Given
> >>>>>> that these modules will not be available in Ignite 2.9 anymore (will
> >>>>>> they?), should we also plan to release the extensions?
> >>>>>>
> >>>>>> ср, 23 сент. 2020 г. в 21:49, Alex Plehanov <
> plehanov.alex@gmail.com
> >>> :
> >>>>>>
> >>>>>>> Igniters,
> >>>>>>>
> >>>>>>> I've prepared release notes for the 2.9 release [1]. Can anyone
> >> review
> >>>>>> it,
> >>>>>>> please?
> >>>>>>>
> >>>>>>> [1]: https://github.com/apache/ignite/pull/8273
> >>>>>>>
> >>>>>>> вт, 22 сент. 2020 г. в 09:39, Alex Plehanov <
> >> plehanov.alex@gmail.com
> >>>>> :
> >>>>>>>
> >>>>>>>> Guys,
> >>>>>>>>
> >>>>>>>> I've filled the ticket with reproducer [1] for the discovery bug.
> >>>> This
> >>>>>>> bug
> >>>>>>>> caused by [2] ticket. We discussed with Vladimir Steshin privately
> >>>> and
> >>>>>>>> decided to revert this ticket. I will do it today (after TC bot
> >> visa)
> >>>>>> if
> >>>>>>>> there are no objections.
> >>>>>>>>
> >>>>>>>> [1]: https://issues.apache.org/jira/browse/IGNITE-13465
> >>>>>>>> [2]: https://issues.apache.org/jira/browse/IGNITE-13134
> >>>>>>>>
> >>>>>>>> пн, 21 сент. 2020 г. в 11:08, Alex Plehanov <
> >> plehanov.alex@gmail.com
> >>>>> :
> >>>>>>>>
> >>>>>>>>> Guys,
> >>>>>>>>>
> >>>>>>>>> During internal testing, we've found a critical bug with
> >>>>>>>>> discovery (cluster falls apart if two nodes segmented
> >> sequentially).
> >>>>>>> This
> >>>>>>>>> problem is not reproduced in 2.8.1. I think we should fix it
> >>>>>>>>> before release. Under investigation now. I'll let you know when
> we
> >>>> get
> >>>>>>>>> something.
> >>>>>>>>>
> >>>>>>>>> чт, 17 сент. 2020 г. в 00:51, Andrey Gura <ag...@apache.org>:
> >>>>>>>>>
> >>>>>>>>>>> So what do you think? Should we proceed with a 'hacked' version
> >> of
> >>>>>>> the
> >>>>>>>>>> message factory in 2.9 and go for the runtime message generation
> >> in
> >>>>>>> later
> >>>>>>>>>> release, or keep the code clean and fix the regression in the
> >> next
> >>>>>>> releases?
> >>>>>>>>>>> Andrey, can you take a look at my change? I think it is fairly
> >>>>>>>>>> straightforward and does not change the semantics, just skips
> the
> >>>>>>> factory
> >>>>>>>>>> closures for certain messages.
> >>>>>>>>>>
> >>>>>>>>>> IMHO 2.5% isn't too much especially because it isn't actual for
> >> all
> >>>>>>>>>> workloads (I didn't get any significant drops during
> >> benchmarking).
> >>>>>> So
> >>>>>>>>>> I prefer the runtime generation in later releases.
> >>>>>>>>>>
> >>>>>>>>>> On Mon, Sep 14, 2020 at 12:41 PM Alexey Goncharuk
> >>>>>>>>>> <al...@gmail.com> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Alexey, Andrey, Igniters,
> >>>>>>>>>>>
> >>>>>>>>>>> So what do you think? Should we proceed with a 'hacked' version
> >> of
> >>>>>>> the
> >>>>>>>>>> message factory in 2.9 and go for the runtime message generation
> >> in
> >>>>>>> later
> >>>>>>>>>> release, or keep the code clean and fix the regression in the
> >> next
> >>>>>>> releases?
> >>>>>>>>>>> Andrey, can you take a look at my change? I think it is fairly
> >>>>>>>>>> straightforward and does not change the semantics, just skips
> the
> >>>>>>> factory
> >>>>>>>>>> closures for certain messages.
> >>>>>>>>>>>
> >>>>>>>>>>> Personally, I would prefer fixing the regression given that we
> >>>> also
> >>>>>>>>>> introduced tracing in this release.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> пт, 11 сент. 2020 г. в 12:09, Alex Plehanov <
> >>>>>> plehanov.alex@gmail.com
> >>>>>>>> :
> >>>>>>>>>>>>
> >>>>>>>>>>>> Alexey,
> >>>>>>>>>>>>
> >>>>>>>>>>>> We've benchmarked by yardstick commits 130376741bf vs
> >> ed52559eb95
> >>>>>>> and
> >>>>>>>>>> the performance of ed52559eb95 is better for about 2.5% on
> >> average
> >>>> on
> >>>>>>> our
> >>>>>>>>>> environment (about the same results we got benchmarking 65c30ec6
> >> vs
> >>>>>>>>>> 0606f03d). We've made 24 runs for each commit of
> >>>>>>>>>> IgnitePutTxImplicitBenchmark (we got maximum drop for 2.9 on
> this
> >>>>>>>>>> benchmark), 200 seconds warmup, 300 seconds benchmark, 6
> >> servers, 5
> >>>>>>> clients
> >>>>>>>>>> 50 threads each. Yardstick results for this configuration:
> >>>>>>>>>>>> Commit 130376741bf: avg TPS=164096, avg latency=9173464 ns
> >>>>>>>>>>>> Commit ed52559eb95: avg TPS=168283, avg latency=8945908 ns
> >>>>>>>>>>>>
> >>>>>>>>>>>> пт, 11 сент. 2020 г. в 09:51, Artem Budnikov <
> >>>>>>>>>> a.budnikov.ignite@gmail.com>:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Hi Everyone,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I posted an instruction on how to publish the docs on
> >>>>>>>>>> ignite.apache.org/docs [1]. When you finish with Ignite 2.9,
> you
> >>>> can
> >>>>>>>>>> update the docs by following the instruction. Unfortunately, I
> >>>> won't
> >>>>>> be
> >>>>>>>>>> able to spend any time on this project any longer. You can send
> >>>> your
> >>>>>>> pull
> >>>>>>>>>> requests and questions about the documentation to Denis Magda.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> -Artem
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> [1] :
> >>>>>>>>>>
> >> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Document
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Thu, Sep 10, 2020 at 2:45 PM Alexey Goncharuk <
> >>>>>>>>>> alexey.goncharuk@gmail.com> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Alexey,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I've tried to play with message factories locally, but
> >>>>>>>>>> unfortunately, I
> >>>>>>>>>>>>>> cannot spot the difference between old and new
> implementation
> >>>> in
> >>>>>>>>>>>>>> distributed benchmarks. I pushed an implementation of
> >>>>>>>>>> MessageFactoryImpl
> >>>>>>>>>>>>>> with the old switch statement to the ignite-2.9-revert-12568
> >>>>>>> branch
> >>>>>>>>>>>>>> (discussed this with Andrey Gura, the change should be
> >>>>>> compatible
> >>>>>>>>>> with the
> >>>>>>>>>>>>>> new metrics as we still use the register() mechanics).
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Can you check if this change makes any difference
> >>>>>> performance-wise
> >>>>>>>>>> in your
> >>>>>>>>>>>>>> environment? If yes, we can go with runtime code generation
> >> in
> >>>>>> the
> >>>>>>>>>> long
> >>>>>>>>>>>>>> term: register classes and generate a dynamic message
> factory
> >>>>>> with
> >>>>>>>>>> a switch
> >>>>>>>>>>>>>> statement once all messages are registered (not in 2.9
> >> though,
> >>>>>>>>>> obviously).
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> ср, 9 сент. 2020 г. в 14:53, Alex Plehanov <
> >>>>>>> plehanov.alex@gmail.com
> >>>>>>>>>>> :
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hello guys,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I've tried to optimize tracing implementation (ticket [1]),
> >> it
> >>>>>>>>>> reduced the
> >>>>>>>>>>>>>>> drop, but not completely removed it.
> >>>>>>>>>>>>>>> Ivan Rakov, Alexander Lapin, can you please review the
> >> patch?
> >>>>>>>>>>>>>>> Ivan Artiukhov, can you please benchmark the patch [2]
> >> against
> >>>>>>>>>> 2.8.1
> >>>>>>>>>>>>>>> release on your environment?
> >>>>>>>>>>>>>>> With this patch on our environment, it's about a 3% drop
> >> left,
> >>>>>>>>>> it's close
> >>>>>>>>>>>>>>> to measurement error and I think such a drop is not a
> >>>>>>>>>> showstopper. Guys,
> >>>>>>>>>>>>>>> WDYT?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Also, I found that compatibility is broken for JDBC thin
> >>>>>> driver
> >>>>>>>>>> between 2.8
> >>>>>>>>>>>>>>> and 2.9 versions (ticket [3]). I think it's a blocker and
> >>>>>> should
> >>>>>>>>>> be
> >>>>>>>>>>>>>>> fixed in 2.9. I've prepared the patch.
> >>>>>>>>>>>>>>> Taras Ledkov, can you please review this patch?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> And one more ticket I propose to include into 2.9 [4] (NIO
> >>>>>>> message
> >>>>>>>>>>>>>>> send problem in some circumstances). I will cherry-pick it
> >> if
> >>>>>>>>>> there is no
> >>>>>>>>>>>>>>> objection.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13411
> >>>>>>>>>>>>>>> [2] https://github.com/apache/ignite/pull/8223
> >>>>>>>>>>>>>>> [3] https://issues.apache.org/jira/browse/IGNITE-13414
> >>>>>>>>>>>>>>> [4] https://issues.apache.org/jira/browse/IGNITE-13361
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> пн, 7 сент. 2020 г. в 14:14, Maxim Muzafarov <
> >>>>>> mmuzaf@apache.org
> >>>>>>>> :
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Alexey,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I propose to include [1] issue to the 2.9 release. Since
> >>>>>> this
> >>>>>>>>>> issue is
> >>>>>>>>>>>>>>>> related to the new master key change functionality which
> >>>>>>>>>> haven't been
> >>>>>>>>>>>>>>>> released yet I think it will be safe to cherry-pick commit
> >>>>>> to
> >>>>>>>>>> the
> >>>>>>>>>>>>>>>> release branch.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13390
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Tue, 1 Sep 2020 at 12:13, Nikolay Izhikov <
> >>>>>>>>>> nizhikov@apache.org>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hello, Igniters.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Alexey, please, include one more Python thin client fix
> >>>>>> [1]
> >>>>>>>>>> into the
> >>>>>>>>>>>>>>> 2.9
> >>>>>>>>>>>>>>>> release
> >>>>>>>>>>>>>>>>> It fixes kinda major issue - "Python client returns
> fields
> >>>>>>> in
> >>>>>>>>>> wrong
> >>>>>>>>>>>>>>>> order since the 2 row when fields_count>10"
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-12809
> >>>>>>>>>>>>>>>>> [2]
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>
> https://github.com/apache/ignite/commit/38025ee4167f05eaa2d6a2c5c2ab70c83a462cfc
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> 31 авг. 2020 г., в 19:23, Alexey Goncharuk <
> >>>>>>>>>>>>>>> alexey.goncharuk@gmail.com>
> >>>>>>>>>>>>>>>> написал(а):
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Alexey, thanks, got it. I am not sure we can optimize
> >>>>>>>>>> anything out of
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> message factory with suppliers (at least I have no ideas
> >>>>>>>>>> right now),
> >>>>>>>>>>>>>>> so
> >>>>>>>>>>>>>>>>>> most likely the only move here is to switch back to the
> >>>>>>>>>> switch
> >>>>>>>>>>>>>>> approach
> >>>>>>>>>>>>>>>>>> somehow preserving the metrics part. Probably, inlining
> >>>>>>> the
> >>>>>>>>>> Ignite
> >>>>>>>>>>>>>>>> messages
> >>>>>>>>>>>>>>>>>> to the IgniteMessageFactoryImpl should do the trick. Let
> >>>>>>> me
> >>>>>>>>>> explore
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> code a bit.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> P.S. I am surprised by the impact this part makes for
> >>>>>> the
> >>>>>>>>>>>>>>> performance.
> >>>>>>>>>>>>>>>>>> Message creation is indeed on the hot path, but a single
> >>>>>>>>>> virtual call
> >>>>>>>>>>>>>>>>>> should not make that much of a difference given the
> >>>>>> amount
> >>>>>>>>>> of other
> >>>>>>>>>>>>>>>> work
> >>>>>>>>>>>>>>>>>> happening during the message processing.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 18:33, Alex Plehanov <
> >>>>>>>>>> plehanov.alex@gmail.com
> >>>>>>>>>>>>>>>> :
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Alexey, sorry, I wrongly interpreted our benchmark
> >>>>>>> results.
> >>>>>>>>>>>>>>> Actually,
> >>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>> were looking for a drop using bi-sect in the range
> >>>>>>> between
> >>>>>>>>>> e6a7f93
> >>>>>>>>>>>>>>>> (first
> >>>>>>>>>>>>>>>>>>> commit in the 2.9 branch after 2.8 branch cut) and
> >>>>>>>>>> 6592dfa5 (last
> >>>>>>>>>>>>>>>> commit in
> >>>>>>>>>>>>>>>>>>> the 2.9 branch). And we found these two problematic
> >>>>>>>>>> commits.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Perhaps only IGNITE-13060 (Tracing) is responsible for
> >>>>>> a
> >>>>>>>>>> drop
> >>>>>>>>>>>>>>> between
> >>>>>>>>>>>>>>>>>>> 2.8.1 and 2.9 (we have benchmarked 2.8.1 vs 2.9 with
> >>>>>>>>>> reverted
> >>>>>>>>>>>>>>>> IGNITE-13060
> >>>>>>>>>>>>>>>>>>> now and performance looks the same)
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Ticket IGNITE-12568 (MessageFactory refactoring) is not
> >>>>>>>>>> related to
> >>>>>>>>>>>>>>>> drop
> >>>>>>>>>>>>>>>>>>> between 2.8.1 and 2.9, but still has some performance
> >>>>>>>>>> problem, and
> >>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>> win back IGNITE-13060 drop by this ticket.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Do we need more investigation on IGNITE-13060 or we
> >>>>>> leave
> >>>>>>>>>> it as is?
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> What should we do with IGNITE-12568 (MessageFactory
> >>>>>>>>>> refactoring)?
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 13:25, Alexey Goncharuk <
> >>>>>>>>>>>>>>>> alexey.goncharuk@gmail.com
> >>>>>>>>>>>>>>>>>>>> :
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Alexey,
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> While investigating, I found that IGNITE-12568 has an
> >>>>>>>>>> incorrect fix
> >>>>>>>>>>>>>>>>>>>> version and is actually present in ignite-2.8.1 branch
> >>>>>>>>>> [1], so it
> >>>>>>>>>>>>>>>> cannot be
> >>>>>>>>>>>>>>>>>>>> the source of the drop against 2.8.1.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> P.S. Looks like we need to enforce a more accurate
> >>>>>> work
> >>>>>>>>>> with fix
> >>>>>>>>>>>>>>>> versions
> >>>>>>>>>>>>>>>>>>>> or develop some sort of tooling to verify the fix
> >>>>>>>>>> versions.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> --AG
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> [1]
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>
> https://github.com/apache/ignite/commit/3e492bd23851856bbd0385c6a419892d0bba2a34
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 12:42, Alexey Goncharuk <
> >>>>>>>>>>>>>>>> alexey.goncharuk@gmail.com
> >>>>>>>>>>>>>>>>>>>>> :
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> пт, 28 авг. 2020 г. в 11:16, Alex Plehanov <
> >>>>>>>>>>>>>>> plehanov.alex@gmail.com
> >>>>>>>>>>>>>>>>> :
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Guys,
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> We have benchmarked 2.9 without IGNITE-13060 and
> >>>>>>>>>> IGNITE-12568
> >>>>>>>>>>>>>>>> (reverted
> >>>>>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>> locally) and got the same performance as on 2.8.1
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> IGNITE-13060 (Tracing) - some code was added to hot
> >>>>>>>>>> paths, to
> >>>>>>>>>>>>>>> trace
> >>>>>>>>>>>>>>>>>>>>>> these
> >>>>>>>>>>>>>>>>>>>>>> hot paths, it's clear why we have performance drop
> >>>>>>> here.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> IGNITE-12568 (MessageFactory refactoring) -
> >>>>>>> switch/case
> >>>>>>>>>> block was
> >>>>>>>>>>>>>>>>>>>>>> refactored to an array of message suppliers. The
> >>>>>>>>>> message factory
> >>>>>>>>>>>>>>>> is on
> >>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>> hot path, which explains why this commit has an
> >>>>>> impact
> >>>>>>>>>> on total
> >>>>>>>>>>>>>>>>>>>>>> performance.
> >>>>>>>>>>>>>>>>>>>>>> I've checked JIT assembly output, done some JMH
> >>>>>>>>>> microbenchmarks,
> >>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>> found
> >>>>>>>>>>>>>>>>>>>>>> that old implementation of MessageFactory.create()
> >>>>>>>>>> about 30-35%
> >>>>>>>>>>>>>>>> faster
> >>>>>>>>>>>>>>>>>>>>>> than
> >>>>>>>>>>>>>>>>>>>>>> the new one. The reason - approach with switch/case
> >>>>>>> can
> >>>>>>>>>>>>>>> effectively
> >>>>>>>>>>>>>>>>>>>>>> inline
> >>>>>>>>>>>>>>>>>>>>>> message creation code, but with an array of
> >>>>>> suppliers
> >>>>>>>>>> relatively
> >>>>>>>>>>>>>>>> heavy
> >>>>>>>>>>>>>>>>>>>>>> "invokeinterface" cannot be skipped. I've tried to
> >>>>>>>>>> rewrite the
> >>>>>>>>>>>>>>> code
> >>>>>>>>>>>>>>>>>>>>>> using
> >>>>>>>>>>>>>>>>>>>>>> an abstract class for suppliers instead of an
> >>>>>>> interface
> >>>>>>>>>> (to
> >>>>>>>>>>>>>>>>>>>>>> replace "invokeinterface" with the "invokevirtual"),
> >>>>>>>>>> but it gives
> >>>>>>>>>>>>>>>> back
> >>>>>>>>>>>>>>>>>>>>>> only
> >>>>>>>>>>>>>>>>>>>>>> 10% of method performance and in this case, code
> >>>>>> looks
> >>>>>>>>>> ugly
> >>>>>>>>>>>>>>>> (lambdas
> >>>>>>>>>>>>>>>>>>>>>> can't
> >>>>>>>>>>>>>>>>>>>>>> be used). Currently, I can't find any more ways to
> >>>>>>>>>> optimize the
> >>>>>>>>>>>>>>>> current
> >>>>>>>>>>>>>>>>>>>>>> approach (except return to the switch/case block).
> >>>>>>>>>> Andrey Gura,
> >>>>>>>>>>>>>>> as
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>> author of IGNITE-12568, maybe you have some ideas
> >>>>>>> about
> >>>>>>>>>>>>>>>> optimization?
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Perhaps we should revert IGNITE-12568, but there are
> >>>>>>>>>> some metrics
> >>>>>>>>>>>>>>>>>>>>>> already
> >>>>>>>>>>>>>>>>>>>>>> created, which can't be rewritten using old message
> >>>>>>>>>> factory
> >>>>>>>>>>>>>>>>>>>>>> implementation
> >>>>>>>>>>>>>>>>>>>>>> (IGNITE-12756). Guys, WDYT?
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Alexey,
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> I see that IGNITE-12756 (metrics improvements) is
> >>>>>>>>>> already released
> >>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>> Ignite 2.8.1 while IGNITE-12568 (message factory) is
> >>>>>>>>>> only present
> >>>>>>>>>>>>>>>> in Ignite
> >>>>>>>>>>>>>>>>>>>>> 2.9. Let's revert both IGNITE-12568 and whichever new
> >>>>>>>>>> metrics
> >>>>>>>>>>>>>>>> created for
> >>>>>>>>>>>>>>>>>>>>> 2.9 that depend on the new message factory to unblock
> >>>>>>>>>> the release
> >>>>>>>>>>>>>>>> and deal
> >>>>>>>>>>>>>>>>>>>>> with the optimizations in 2.10?
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>>>
> >>
>
>
Re: Apache Ignite 2.9.0 RELEASE [Time, Scope, Manager]
Posted by Nikolay Izhikov <ni...@apache.org>.
Hello, Igniters.
I prepared a patch [1] for blocker ticket [2] - «Server node fail and stops in case wrong datatype put in indexed field»
Can someone, please, help me with the review?
[1] https://github.com/apache/ignite/pull/8330
[2] https://issues.apache.org/jira/browse/IGNITE-13553
==================
I propose to include several minor tickets to the scope of the 2.9 that increase Ignite User Experience
CMD tools improvements:
IGNITE-13488 - Command to print metric value
IGNITE-13426 - Command to print system view content
IGNITE-13422 - Parameter to explicitly enable experimental commands
IGNITE-13380 - Output IgniteSystemProperties via ignite.sh
New system views:
IGNITE-13409 Metastorage and DistributedMetastorage viewы.
IGNITE-13408 BinaryMetadata view.
> 9 окт. 2020 г., в 04:04, Denis Magda <dm...@apache.org> написал(а):
>
> @Alex Plehanov <pl...@gmail.com>,
>
> If it still makes sense and not too late, you can cherry-pick the commit
> with the new docs into the 2.9 branch:
> https://github.com/apache/ignite/commit/073488ac97517bbaad9f6b94b781fc404646f191
>
> Anyway, it's not crucial as long as we agreed to make an exception for this
> release. The docs were already published to the website and assembled from
> the master. Once the vote passes, I'll make them visible and traceable from
> the website menus.
>
> -
> Denis
>
>
> On Tue, Oct 6, 2020 at 7:20 AM Alexey Goncharuk <al...@gmail.com>
> wrote:
>
>> Saikat,
>>
>> The plan sounds good to me! Do you have an idea for the timeline of the
>> module releases? Let me know if I can help in any way.
>>
>> Also, I was looking into the modularization IEP and noticed that OSGI
>> module is discussed to be moved to the extensions, but there is no
>> corresponding ticket. Should I create one? I will be happy to help with
>> moving this module to extensions.
>>
>> вт, 29 сент. 2020 г. в 03:03, Saikat Maitra <sa...@gmail.com>:
>>
>>> Hi Alexey,
>>>
>>> All the modules as planned in IEP-36
>>>
>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36:+Modularization
>>> are migrated to ignite-extensions repository.
>>>
>>> We can definitely plan to release ignite-extensions modules.
>>>
>>> I have a pending PR related to the kafka module and the PR is in review
>>> https://github.com/apache/ignite/pull/8222 but its corresponding PR has
>>> been merged in ignite-extensions repository.
>>>
>>> My thoughts / action items for the migration efforts and releases were
>>> as follows:
>>>
>>> 1. Merge the changes for https://github.com/apache/ignite/pull/8222
>>> 2. Fix the upload nightly packages task for ignite-core to help fix the
>>> travis build failure in ignite-extensions repository.
>>> 3. Review and update the README.md files for the ignite-extensions
>> modules
>>> as required.
>>> 4. Update the docs for ignite-extensions modules in ignite-website.
>>> 5. Release each module separately and share updates.
>>>
>>> Please let me know if you have feedback.
>>>
>>> Regards,
>>> Saikat
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Mon, Sep 28, 2020 at 7:36 AM Petr Ivanov <mr...@gmail.com> wrote:
>>>
>>>> It seems that gitbox.apache.org is not available on CI/CD.
>>>>
>>>> I will try to update VCS to GitHub.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> On 28 Sep 2020, at 14:07, Alex Plehanov <pl...@gmail.com>
>>>> wrote:
>>>>>
>>>>> Guys,
>>>>>
>>>>> I've tried to build release candidate using TC [1]. But:
>>>>> 1. I can't choose a branch in this TC task (there is no such field).
>>>>> 2. On the "default" branch I got an error: Failed to collect changes,
>>>>> error: List remote refs failed:
>>>>> org.apache.http.conn.ConnectTimeoutException: Connect to
>>>>> gitbox.apache.org:443 [gitbox.apache.org/52.202.80.70] failed:
>> connect
>>>>> timed out, VCS root: "GitBox [asf/ignite]" {instance id=300, parent
>>>>> internal id=74, parent id=GitBoxAsfIgnite, description: "
>>>>> https://gitbox.apache.org/repos/asf/ignite.git#refs/heads/master"}
>>>>>
>>>>> Is there some problem with this TC task? What am I doing wrong?
>>>>>
>>>>> [1] :
>>>>>
>>>>
>> https://ci.ignite.apache.org/viewType.html?buildTypeId=Releases_ApacheIgniteMain_ReleaseBuild
>>>>>
>>>>> пн, 28 сент. 2020 г. в 13:15, Alexey Goncharuk <
>>>> alexey.goncharuk@gmail.com>:
>>>>>
>>>>>> Saikat, Nikolay,
>>>>>>
>>>>>> We have migrated a bunch of modules to ignite-extensions recently.
>>>> Given
>>>>>> that these modules will not be available in Ignite 2.9 anymore (will
>>>>>> they?), should we also plan to release the extensions?
>>>>>>
>>>>>> ср, 23 сент. 2020 г. в 21:49, Alex Plehanov <plehanov.alex@gmail.com
>>> :
>>>>>>
>>>>>>> Igniters,
>>>>>>>
>>>>>>> I've prepared release notes for the 2.9 release [1]. Can anyone
>> review
>>>>>> it,
>>>>>>> please?
>>>>>>>
>>>>>>> [1]: https://github.com/apache/ignite/pull/8273
>>>>>>>
>>>>>>> вт, 22 сент. 2020 г. в 09:39, Alex Plehanov <
>> plehanov.alex@gmail.com
>>>>> :
>>>>>>>
>>>>>>>> Guys,
>>>>>>>>
>>>>>>>> I've filled the ticket with reproducer [1] for the discovery bug.
>>>> This
>>>>>>> bug
>>>>>>>> caused by [2] ticket. We discussed with Vladimir Steshin privately
>>>> and
>>>>>>>> decided to revert this ticket. I will do it today (after TC bot
>> visa)
>>>>>> if
>>>>>>>> there are no objections.
>>>>>>>>
>>>>>>>> [1]: https://issues.apache.org/jira/browse/IGNITE-13465
>>>>>>>> [2]: https://issues.apache.org/jira/browse/IGNITE-13134
>>>>>>>>
>>>>>>>> пн, 21 сент. 2020 г. в 11:08, Alex Plehanov <
>> plehanov.alex@gmail.com
>>>>> :
>>>>>>>>
>>>>>>>>> Guys,
>>>>>>>>>
>>>>>>>>> During internal testing, we've found a critical bug with
>>>>>>>>> discovery (cluster falls apart if two nodes segmented
>> sequentially).
>>>>>>> This
>>>>>>>>> problem is not reproduced in 2.8.1. I think we should fix it
>>>>>>>>> before release. Under investigation now. I'll let you know when we
>>>> get
>>>>>>>>> something.
>>>>>>>>>
>>>>>>>>> чт, 17 сент. 2020 г. в 00:51, Andrey Gura <ag...@apache.org>:
>>>>>>>>>
>>>>>>>>>>> So what do you think? Should we proceed with a 'hacked' version
>> of
>>>>>>> the
>>>>>>>>>> message factory in 2.9 and go for the runtime message generation
>> in
>>>>>>> later
>>>>>>>>>> release, or keep the code clean and fix the regression in the
>> next
>>>>>>> releases?
>>>>>>>>>>> Andrey, can you take a look at my change? I think it is fairly
>>>>>>>>>> straightforward and does not change the semantics, just skips the
>>>>>>> factory
>>>>>>>>>> closures for certain messages.
>>>>>>>>>>
>>>>>>>>>> IMHO 2.5% isn't too much especially because it isn't actual for
>> all
>>>>>>>>>> workloads (I didn't get any significant drops during
>> benchmarking).
>>>>>> So
>>>>>>>>>> I prefer the runtime generation in later releases.
>>>>>>>>>>
>>>>>>>>>> On Mon, Sep 14, 2020 at 12:41 PM Alexey Goncharuk
>>>>>>>>>> <al...@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Alexey, Andrey, Igniters,
>>>>>>>>>>>
>>>>>>>>>>> So what do you think? Should we proceed with a 'hacked' version
>> of
>>>>>>> the
>>>>>>>>>> message factory in 2.9 and go for the runtime message generation
>> in
>>>>>>> later
>>>>>>>>>> release, or keep the code clean and fix the regression in the
>> next
>>>>>>> releases?
>>>>>>>>>>> Andrey, can you take a look at my change? I think it is fairly
>>>>>>>>>> straightforward and does not change the semantics, just skips the
>>>>>>> factory
>>>>>>>>>> closures for certain messages.
>>>>>>>>>>>
>>>>>>>>>>> Personally, I would prefer fixing the regression given that we
>>>> also
>>>>>>>>>> introduced tracing in this release.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> пт, 11 сент. 2020 г. в 12:09, Alex Plehanov <
>>>>>> plehanov.alex@gmail.com
>>>>>>>> :
>>>>>>>>>>>>
>>>>>>>>>>>> Alexey,
>>>>>>>>>>>>
>>>>>>>>>>>> We've benchmarked by yardstick commits 130376741bf vs
>> ed52559eb95
>>>>>>> and
>>>>>>>>>> the performance of ed52559eb95 is better for about 2.5% on
>> average
>>>> on
>>>>>>> our
>>>>>>>>>> environment (about the same results we got benchmarking 65c30ec6
>> vs
>>>>>>>>>> 0606f03d). We've made 24 runs for each commit of
>>>>>>>>>> IgnitePutTxImplicitBenchmark (we got maximum drop for 2.9 on this
>>>>>>>>>> benchmark), 200 seconds warmup, 300 seconds benchmark, 6
>> servers, 5
>>>>>>> clients
>>>>>>>>>> 50 threads each. Yardstick results for this configuration:
>>>>>>>>>>>> Commit 130376741bf: avg TPS=164096, avg latency=9173464 ns
>>>>>>>>>>>> Commit ed52559eb95: avg TPS=168283, avg latency=8945908 ns
>>>>>>>>>>>>
>>>>>>>>>>>> пт, 11 сент. 2020 г. в 09:51, Artem Budnikov <
>>>>>>>>>> a.budnikov.ignite@gmail.com>:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I posted an instruction on how to publish the docs on
>>>>>>>>>> ignite.apache.org/docs [1]. When you finish with Ignite 2.9, you
>>>> can
>>>>>>>>>> update the docs by following the instruction. Unfortunately, I
>>>> won't
>>>>>> be
>>>>>>>>>> able to spend any time on this project any longer. You can send
>>>> your
>>>>>>> pull
>>>>>>>>>> requests and questions about the documentation to Denis Magda.
>>>>>>>>>>>>>
>>>>>>>>>>>>> -Artem
>>>>>>>>>>>>>
>>>>>>>>>>>>> [1] :
>>>>>>>>>>
>> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Document
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Sep 10, 2020 at 2:45 PM Alexey Goncharuk <
>>>>>>>>>> alexey.goncharuk@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Alexey,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I've tried to play with message factories locally, but
>>>>>>>>>> unfortunately, I
>>>>>>>>>>>>>> cannot spot the difference between old and new implementation
>>>> in
>>>>>>>>>>>>>> distributed benchmarks. I pushed an implementation of
>>>>>>>>>> MessageFactoryImpl
>>>>>>>>>>>>>> with the old switch statement to the ignite-2.9-revert-12568
>>>>>>> branch
>>>>>>>>>>>>>> (discussed this with Andrey Gura, the change should be
>>>>>> compatible
>>>>>>>>>> with the
>>>>>>>>>>>>>> new metrics as we still use the register() mechanics).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Can you check if this change makes any difference
>>>>>> performance-wise
>>>>>>>>>> in your
>>>>>>>>>>>>>> environment? If yes, we can go with runtime code generation
>> in
>>>>>> the
>>>>>>>>>> long
>>>>>>>>>>>>>> term: register classes and generate a dynamic message factory
>>>>>> with
>>>>>>>>>> a switch
>>>>>>>>>>>>>> statement once all messages are registered (not in 2.9
>> though,
>>>>>>>>>> obviously).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ср, 9 сент. 2020 г. в 14:53, Alex Plehanov <
>>>>>>> plehanov.alex@gmail.com
>>>>>>>>>>> :
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hello guys,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I've tried to optimize tracing implementation (ticket [1]),
>> it
>>>>>>>>>> reduced the
>>>>>>>>>>>>>>> drop, but not completely removed it.
>>>>>>>>>>>>>>> Ivan Rakov, Alexander Lapin, can you please review the
>> patch?
>>>>>>>>>>>>>>> Ivan Artiukhov, can you please benchmark the patch [2]
>> against
>>>>>>>>>> 2.8.1
>>>>>>>>>>>>>>> release on your environment?
>>>>>>>>>>>>>>> With this patch on our environment, it's about a 3% drop
>> left,
>>>>>>>>>> it's close
>>>>>>>>>>>>>>> to measurement error and I think such a drop is not a
>>>>>>>>>> showstopper. Guys,
>>>>>>>>>>>>>>> WDYT?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Also, I found that compatibility is broken for JDBC thin
>>>>>> driver
>>>>>>>>>> between 2.8
>>>>>>>>>>>>>>> and 2.9 versions (ticket [3]). I think it's a blocker and
>>>>>> should
>>>>>>>>>> be
>>>>>>>>>>>>>>> fixed in 2.9. I've prepared the patch.
>>>>>>>>>>>>>>> Taras Ledkov, can you please review this patch?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> And one more ticket I propose to include into 2.9 [4] (NIO
>>>>>>> message
>>>>>>>>>>>>>>> send problem in some circumstances). I will cherry-pick it
>> if
>>>>>>>>>> there is no
>>>>>>>>>>>>>>> objection.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13411
>>>>>>>>>>>>>>> [2] https://github.com/apache/ignite/pull/8223
>>>>>>>>>>>>>>> [3] https://issues.apache.org/jira/browse/IGNITE-13414
>>>>>>>>>>>>>>> [4] https://issues.apache.org/jira/browse/IGNITE-13361
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> пн, 7 сент. 2020 г. в 14:14, Maxim Muzafarov <
>>>>>> mmuzaf@apache.org
>>>>>>>> :
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Alexey,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I propose to include [1] issue to the 2.9 release. Since
>>>>>> this
>>>>>>>>>> issue is
>>>>>>>>>>>>>>>> related to the new master key change functionality which
>>>>>>>>>> haven't been
>>>>>>>>>>>>>>>> released yet I think it will be safe to cherry-pick commit
>>>>>> to
>>>>>>>>>> the
>>>>>>>>>>>>>>>> release branch.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13390
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Tue, 1 Sep 2020 at 12:13, Nikolay Izhikov <
>>>>>>>>>> nizhikov@apache.org>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hello, Igniters.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Alexey, please, include one more Python thin client fix
>>>>>> [1]
>>>>>>>>>> into the
>>>>>>>>>>>>>>> 2.9
>>>>>>>>>>>>>>>> release
>>>>>>>>>>>>>>>>> It fixes kinda major issue - "Python client returns fields
>>>>>>> in
>>>>>>>>>> wrong
>>>>>>>>>>>>>>>> order since the 2 row when fields_count>10"
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-12809
>>>>>>>>>>>>>>>>> [2]
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>> https://github.com/apache/ignite/commit/38025ee4167f05eaa2d6a2c5c2ab70c83a462cfc
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 31 авг. 2020 г., в 19:23, Alexey Goncharuk <
>>>>>>>>>>>>>>> alexey.goncharuk@gmail.com>
>>>>>>>>>>>>>>>> написал(а):
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Alexey, thanks, got it. I am not sure we can optimize
>>>>>>>>>> anything out of
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> message factory with suppliers (at least I have no ideas
>>>>>>>>>> right now),
>>>>>>>>>>>>>>> so
>>>>>>>>>>>>>>>>>> most likely the only move here is to switch back to the
>>>>>>>>>> switch
>>>>>>>>>>>>>>> approach
>>>>>>>>>>>>>>>>>> somehow preserving the metrics part. Probably, inlining
>>>>>>> the
>>>>>>>>>> Ignite
>>>>>>>>>>>>>>>> messages
>>>>>>>>>>>>>>>>>> to the IgniteMessageFactoryImpl should do the trick. Let
>>>>>>> me
>>>>>>>>>> explore
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> code a bit.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> P.S. I am surprised by the impact this part makes for
>>>>>> the
>>>>>>>>>>>>>>> performance.
>>>>>>>>>>>>>>>>>> Message creation is indeed on the hot path, but a single
>>>>>>>>>> virtual call
>>>>>>>>>>>>>>>>>> should not make that much of a difference given the
>>>>>> amount
>>>>>>>>>> of other
>>>>>>>>>>>>>>>> work
>>>>>>>>>>>>>>>>>> happening during the message processing.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 18:33, Alex Plehanov <
>>>>>>>>>> plehanov.alex@gmail.com
>>>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Alexey, sorry, I wrongly interpreted our benchmark
>>>>>>> results.
>>>>>>>>>>>>>>> Actually,
>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>> were looking for a drop using bi-sect in the range
>>>>>>> between
>>>>>>>>>> e6a7f93
>>>>>>>>>>>>>>>> (first
>>>>>>>>>>>>>>>>>>> commit in the 2.9 branch after 2.8 branch cut) and
>>>>>>>>>> 6592dfa5 (last
>>>>>>>>>>>>>>>> commit in
>>>>>>>>>>>>>>>>>>> the 2.9 branch). And we found these two problematic
>>>>>>>>>> commits.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Perhaps only IGNITE-13060 (Tracing) is responsible for
>>>>>> a
>>>>>>>>>> drop
>>>>>>>>>>>>>>> between
>>>>>>>>>>>>>>>>>>> 2.8.1 and 2.9 (we have benchmarked 2.8.1 vs 2.9 with
>>>>>>>>>> reverted
>>>>>>>>>>>>>>>> IGNITE-13060
>>>>>>>>>>>>>>>>>>> now and performance looks the same)
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Ticket IGNITE-12568 (MessageFactory refactoring) is not
>>>>>>>>>> related to
>>>>>>>>>>>>>>>> drop
>>>>>>>>>>>>>>>>>>> between 2.8.1 and 2.9, but still has some performance
>>>>>>>>>> problem, and
>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>> win back IGNITE-13060 drop by this ticket.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Do we need more investigation on IGNITE-13060 or we
>>>>>> leave
>>>>>>>>>> it as is?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> What should we do with IGNITE-12568 (MessageFactory
>>>>>>>>>> refactoring)?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 13:25, Alexey Goncharuk <
>>>>>>>>>>>>>>>> alexey.goncharuk@gmail.com
>>>>>>>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Alexey,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> While investigating, I found that IGNITE-12568 has an
>>>>>>>>>> incorrect fix
>>>>>>>>>>>>>>>>>>>> version and is actually present in ignite-2.8.1 branch
>>>>>>>>>> [1], so it
>>>>>>>>>>>>>>>> cannot be
>>>>>>>>>>>>>>>>>>>> the source of the drop against 2.8.1.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> P.S. Looks like we need to enforce a more accurate
>>>>>> work
>>>>>>>>>> with fix
>>>>>>>>>>>>>>>> versions
>>>>>>>>>>>>>>>>>>>> or develop some sort of tooling to verify the fix
>>>>>>>>>> versions.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> --AG
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>> https://github.com/apache/ignite/commit/3e492bd23851856bbd0385c6a419892d0bba2a34
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 12:42, Alexey Goncharuk <
>>>>>>>>>>>>>>>> alexey.goncharuk@gmail.com
>>>>>>>>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> пт, 28 авг. 2020 г. в 11:16, Alex Plehanov <
>>>>>>>>>>>>>>> plehanov.alex@gmail.com
>>>>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Guys,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> We have benchmarked 2.9 without IGNITE-13060 and
>>>>>>>>>> IGNITE-12568
>>>>>>>>>>>>>>>> (reverted
>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>> locally) and got the same performance as on 2.8.1
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> IGNITE-13060 (Tracing) - some code was added to hot
>>>>>>>>>> paths, to
>>>>>>>>>>>>>>> trace
>>>>>>>>>>>>>>>>>>>>>> these
>>>>>>>>>>>>>>>>>>>>>> hot paths, it's clear why we have performance drop
>>>>>>> here.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> IGNITE-12568 (MessageFactory refactoring) -
>>>>>>> switch/case
>>>>>>>>>> block was
>>>>>>>>>>>>>>>>>>>>>> refactored to an array of message suppliers. The
>>>>>>>>>> message factory
>>>>>>>>>>>>>>>> is on
>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> hot path, which explains why this commit has an
>>>>>> impact
>>>>>>>>>> on total
>>>>>>>>>>>>>>>>>>>>>> performance.
>>>>>>>>>>>>>>>>>>>>>> I've checked JIT assembly output, done some JMH
>>>>>>>>>> microbenchmarks,
>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>> found
>>>>>>>>>>>>>>>>>>>>>> that old implementation of MessageFactory.create()
>>>>>>>>>> about 30-35%
>>>>>>>>>>>>>>>> faster
>>>>>>>>>>>>>>>>>>>>>> than
>>>>>>>>>>>>>>>>>>>>>> the new one. The reason - approach with switch/case
>>>>>>> can
>>>>>>>>>>>>>>> effectively
>>>>>>>>>>>>>>>>>>>>>> inline
>>>>>>>>>>>>>>>>>>>>>> message creation code, but with an array of
>>>>>> suppliers
>>>>>>>>>> relatively
>>>>>>>>>>>>>>>> heavy
>>>>>>>>>>>>>>>>>>>>>> "invokeinterface" cannot be skipped. I've tried to
>>>>>>>>>> rewrite the
>>>>>>>>>>>>>>> code
>>>>>>>>>>>>>>>>>>>>>> using
>>>>>>>>>>>>>>>>>>>>>> an abstract class for suppliers instead of an
>>>>>>> interface
>>>>>>>>>> (to
>>>>>>>>>>>>>>>>>>>>>> replace "invokeinterface" with the "invokevirtual"),
>>>>>>>>>> but it gives
>>>>>>>>>>>>>>>> back
>>>>>>>>>>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>>>>>>>> 10% of method performance and in this case, code
>>>>>> looks
>>>>>>>>>> ugly
>>>>>>>>>>>>>>>> (lambdas
>>>>>>>>>>>>>>>>>>>>>> can't
>>>>>>>>>>>>>>>>>>>>>> be used). Currently, I can't find any more ways to
>>>>>>>>>> optimize the
>>>>>>>>>>>>>>>> current
>>>>>>>>>>>>>>>>>>>>>> approach (except return to the switch/case block).
>>>>>>>>>> Andrey Gura,
>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> author of IGNITE-12568, maybe you have some ideas
>>>>>>> about
>>>>>>>>>>>>>>>> optimization?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Perhaps we should revert IGNITE-12568, but there are
>>>>>>>>>> some metrics
>>>>>>>>>>>>>>>>>>>>>> already
>>>>>>>>>>>>>>>>>>>>>> created, which can't be rewritten using old message
>>>>>>>>>> factory
>>>>>>>>>>>>>>>>>>>>>> implementation
>>>>>>>>>>>>>>>>>>>>>> (IGNITE-12756). Guys, WDYT?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Alexey,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I see that IGNITE-12756 (metrics improvements) is
>>>>>>>>>> already released
>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>> Ignite 2.8.1 while IGNITE-12568 (message factory) is
>>>>>>>>>> only present
>>>>>>>>>>>>>>>> in Ignite
>>>>>>>>>>>>>>>>>>>>> 2.9. Let's revert both IGNITE-12568 and whichever new
>>>>>>>>>> metrics
>>>>>>>>>>>>>>>> created for
>>>>>>>>>>>>>>>>>>>>> 2.9 that depend on the new message factory to unblock
>>>>>>>>>> the release
>>>>>>>>>>>>>>>> and deal
>>>>>>>>>>>>>>>>>>>>> with the optimizations in 2.10?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>>
>>
Re: Apache Ignite 2.9.0 RELEASE [Time, Scope, Manager]
Posted by Denis Magda <dm...@apache.org>.
@Alex Plehanov <pl...@gmail.com>,
If it still makes sense and not too late, you can cherry-pick the commit
with the new docs into the 2.9 branch:
https://github.com/apache/ignite/commit/073488ac97517bbaad9f6b94b781fc404646f191
Anyway, it's not crucial as long as we agreed to make an exception for this
release. The docs were already published to the website and assembled from
the master. Once the vote passes, I'll make them visible and traceable from
the website menus.
-
Denis
On Tue, Oct 6, 2020 at 7:20 AM Alexey Goncharuk <al...@gmail.com>
wrote:
> Saikat,
>
> The plan sounds good to me! Do you have an idea for the timeline of the
> module releases? Let me know if I can help in any way.
>
> Also, I was looking into the modularization IEP and noticed that OSGI
> module is discussed to be moved to the extensions, but there is no
> corresponding ticket. Should I create one? I will be happy to help with
> moving this module to extensions.
>
> вт, 29 сент. 2020 г. в 03:03, Saikat Maitra <sa...@gmail.com>:
>
> > Hi Alexey,
> >
> > All the modules as planned in IEP-36
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36:+Modularization
> > are migrated to ignite-extensions repository.
> >
> > We can definitely plan to release ignite-extensions modules.
> >
> > I have a pending PR related to the kafka module and the PR is in review
> > https://github.com/apache/ignite/pull/8222 but its corresponding PR has
> > been merged in ignite-extensions repository.
> >
> > My thoughts / action items for the migration efforts and releases were
> > as follows:
> >
> > 1. Merge the changes for https://github.com/apache/ignite/pull/8222
> > 2. Fix the upload nightly packages task for ignite-core to help fix the
> > travis build failure in ignite-extensions repository.
> > 3. Review and update the README.md files for the ignite-extensions
> modules
> > as required.
> > 4. Update the docs for ignite-extensions modules in ignite-website.
> > 5. Release each module separately and share updates.
> >
> > Please let me know if you have feedback.
> >
> > Regards,
> > Saikat
> >
> >
> >
> >
> >
> >
> >
> >
> > On Mon, Sep 28, 2020 at 7:36 AM Petr Ivanov <mr...@gmail.com> wrote:
> >
> >> It seems that gitbox.apache.org is not available on CI/CD.
> >>
> >> I will try to update VCS to GitHub.
> >>
> >>
> >>
> >>
> >>
> >> > On 28 Sep 2020, at 14:07, Alex Plehanov <pl...@gmail.com>
> >> wrote:
> >> >
> >> > Guys,
> >> >
> >> > I've tried to build release candidate using TC [1]. But:
> >> > 1. I can't choose a branch in this TC task (there is no such field).
> >> > 2. On the "default" branch I got an error: Failed to collect changes,
> >> > error: List remote refs failed:
> >> > org.apache.http.conn.ConnectTimeoutException: Connect to
> >> > gitbox.apache.org:443 [gitbox.apache.org/52.202.80.70] failed:
> connect
> >> > timed out, VCS root: "GitBox [asf/ignite]" {instance id=300, parent
> >> > internal id=74, parent id=GitBoxAsfIgnite, description: "
> >> > https://gitbox.apache.org/repos/asf/ignite.git#refs/heads/master"}
> >> >
> >> > Is there some problem with this TC task? What am I doing wrong?
> >> >
> >> > [1] :
> >> >
> >>
> https://ci.ignite.apache.org/viewType.html?buildTypeId=Releases_ApacheIgniteMain_ReleaseBuild
> >> >
> >> > пн, 28 сент. 2020 г. в 13:15, Alexey Goncharuk <
> >> alexey.goncharuk@gmail.com>:
> >> >
> >> >> Saikat, Nikolay,
> >> >>
> >> >> We have migrated a bunch of modules to ignite-extensions recently.
> >> Given
> >> >> that these modules will not be available in Ignite 2.9 anymore (will
> >> >> they?), should we also plan to release the extensions?
> >> >>
> >> >> ср, 23 сент. 2020 г. в 21:49, Alex Plehanov <plehanov.alex@gmail.com
> >:
> >> >>
> >> >>> Igniters,
> >> >>>
> >> >>> I've prepared release notes for the 2.9 release [1]. Can anyone
> review
> >> >> it,
> >> >>> please?
> >> >>>
> >> >>> [1]: https://github.com/apache/ignite/pull/8273
> >> >>>
> >> >>> вт, 22 сент. 2020 г. в 09:39, Alex Plehanov <
> plehanov.alex@gmail.com
> >> >:
> >> >>>
> >> >>>> Guys,
> >> >>>>
> >> >>>> I've filled the ticket with reproducer [1] for the discovery bug.
> >> This
> >> >>> bug
> >> >>>> caused by [2] ticket. We discussed with Vladimir Steshin privately
> >> and
> >> >>>> decided to revert this ticket. I will do it today (after TC bot
> visa)
> >> >> if
> >> >>>> there are no objections.
> >> >>>>
> >> >>>> [1]: https://issues.apache.org/jira/browse/IGNITE-13465
> >> >>>> [2]: https://issues.apache.org/jira/browse/IGNITE-13134
> >> >>>>
> >> >>>> пн, 21 сент. 2020 г. в 11:08, Alex Plehanov <
> plehanov.alex@gmail.com
> >> >:
> >> >>>>
> >> >>>>> Guys,
> >> >>>>>
> >> >>>>> During internal testing, we've found a critical bug with
> >> >>>>> discovery (cluster falls apart if two nodes segmented
> sequentially).
> >> >>> This
> >> >>>>> problem is not reproduced in 2.8.1. I think we should fix it
> >> >>>>> before release. Under investigation now. I'll let you know when we
> >> get
> >> >>>>> something.
> >> >>>>>
> >> >>>>> чт, 17 сент. 2020 г. в 00:51, Andrey Gura <ag...@apache.org>:
> >> >>>>>
> >> >>>>>>> So what do you think? Should we proceed with a 'hacked' version
> of
> >> >>> the
> >> >>>>>> message factory in 2.9 and go for the runtime message generation
> in
> >> >>> later
> >> >>>>>> release, or keep the code clean and fix the regression in the
> next
> >> >>> releases?
> >> >>>>>>> Andrey, can you take a look at my change? I think it is fairly
> >> >>>>>> straightforward and does not change the semantics, just skips the
> >> >>> factory
> >> >>>>>> closures for certain messages.
> >> >>>>>>
> >> >>>>>> IMHO 2.5% isn't too much especially because it isn't actual for
> all
> >> >>>>>> workloads (I didn't get any significant drops during
> benchmarking).
> >> >> So
> >> >>>>>> I prefer the runtime generation in later releases.
> >> >>>>>>
> >> >>>>>> On Mon, Sep 14, 2020 at 12:41 PM Alexey Goncharuk
> >> >>>>>> <al...@gmail.com> wrote:
> >> >>>>>>>
> >> >>>>>>> Alexey, Andrey, Igniters,
> >> >>>>>>>
> >> >>>>>>> So what do you think? Should we proceed with a 'hacked' version
> of
> >> >>> the
> >> >>>>>> message factory in 2.9 and go for the runtime message generation
> in
> >> >>> later
> >> >>>>>> release, or keep the code clean and fix the regression in the
> next
> >> >>> releases?
> >> >>>>>>> Andrey, can you take a look at my change? I think it is fairly
> >> >>>>>> straightforward and does not change the semantics, just skips the
> >> >>> factory
> >> >>>>>> closures for certain messages.
> >> >>>>>>>
> >> >>>>>>> Personally, I would prefer fixing the regression given that we
> >> also
> >> >>>>>> introduced tracing in this release.
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>> пт, 11 сент. 2020 г. в 12:09, Alex Plehanov <
> >> >> plehanov.alex@gmail.com
> >> >>>> :
> >> >>>>>>>>
> >> >>>>>>>> Alexey,
> >> >>>>>>>>
> >> >>>>>>>> We've benchmarked by yardstick commits 130376741bf vs
> ed52559eb95
> >> >>> and
> >> >>>>>> the performance of ed52559eb95 is better for about 2.5% on
> average
> >> on
> >> >>> our
> >> >>>>>> environment (about the same results we got benchmarking 65c30ec6
> vs
> >> >>>>>> 0606f03d). We've made 24 runs for each commit of
> >> >>>>>> IgnitePutTxImplicitBenchmark (we got maximum drop for 2.9 on this
> >> >>>>>> benchmark), 200 seconds warmup, 300 seconds benchmark, 6
> servers, 5
> >> >>> clients
> >> >>>>>> 50 threads each. Yardstick results for this configuration:
> >> >>>>>>>> Commit 130376741bf: avg TPS=164096, avg latency=9173464 ns
> >> >>>>>>>> Commit ed52559eb95: avg TPS=168283, avg latency=8945908 ns
> >> >>>>>>>>
> >> >>>>>>>> пт, 11 сент. 2020 г. в 09:51, Artem Budnikov <
> >> >>>>>> a.budnikov.ignite@gmail.com>:
> >> >>>>>>>>>
> >> >>>>>>>>> Hi Everyone,
> >> >>>>>>>>>
> >> >>>>>>>>> I posted an instruction on how to publish the docs on
> >> >>>>>> ignite.apache.org/docs [1]. When you finish with Ignite 2.9, you
> >> can
> >> >>>>>> update the docs by following the instruction. Unfortunately, I
> >> won't
> >> >> be
> >> >>>>>> able to spend any time on this project any longer. You can send
> >> your
> >> >>> pull
> >> >>>>>> requests and questions about the documentation to Denis Magda.
> >> >>>>>>>>>
> >> >>>>>>>>> -Artem
> >> >>>>>>>>>
> >> >>>>>>>>> [1] :
> >> >>>>>>
> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Document
> >> >>>>>>>>>
> >> >>>>>>>>> On Thu, Sep 10, 2020 at 2:45 PM Alexey Goncharuk <
> >> >>>>>> alexey.goncharuk@gmail.com> wrote:
> >> >>>>>>>>>>
> >> >>>>>>>>>> Alexey,
> >> >>>>>>>>>>
> >> >>>>>>>>>> I've tried to play with message factories locally, but
> >> >>>>>> unfortunately, I
> >> >>>>>>>>>> cannot spot the difference between old and new implementation
> >> in
> >> >>>>>>>>>> distributed benchmarks. I pushed an implementation of
> >> >>>>>> MessageFactoryImpl
> >> >>>>>>>>>> with the old switch statement to the ignite-2.9-revert-12568
> >> >>> branch
> >> >>>>>>>>>> (discussed this with Andrey Gura, the change should be
> >> >> compatible
> >> >>>>>> with the
> >> >>>>>>>>>> new metrics as we still use the register() mechanics).
> >> >>>>>>>>>>
> >> >>>>>>>>>> Can you check if this change makes any difference
> >> >> performance-wise
> >> >>>>>> in your
> >> >>>>>>>>>> environment? If yes, we can go with runtime code generation
> in
> >> >> the
> >> >>>>>> long
> >> >>>>>>>>>> term: register classes and generate a dynamic message factory
> >> >> with
> >> >>>>>> a switch
> >> >>>>>>>>>> statement once all messages are registered (not in 2.9
> though,
> >> >>>>>> obviously).
> >> >>>>>>>>>>
> >> >>>>>>>>>> ср, 9 сент. 2020 г. в 14:53, Alex Plehanov <
> >> >>> plehanov.alex@gmail.com
> >> >>>>>>> :
> >> >>>>>>>>>>
> >> >>>>>>>>>>> Hello guys,
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> I've tried to optimize tracing implementation (ticket [1]),
> it
> >> >>>>>> reduced the
> >> >>>>>>>>>>> drop, but not completely removed it.
> >> >>>>>>>>>>> Ivan Rakov, Alexander Lapin, can you please review the
> patch?
> >> >>>>>>>>>>> Ivan Artiukhov, can you please benchmark the patch [2]
> against
> >> >>>>>> 2.8.1
> >> >>>>>>>>>>> release on your environment?
> >> >>>>>>>>>>> With this patch on our environment, it's about a 3% drop
> left,
> >> >>>>>> it's close
> >> >>>>>>>>>>> to measurement error and I think such a drop is not a
> >> >>>>>> showstopper. Guys,
> >> >>>>>>>>>>> WDYT?
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> Also, I found that compatibility is broken for JDBC thin
> >> >> driver
> >> >>>>>> between 2.8
> >> >>>>>>>>>>> and 2.9 versions (ticket [3]). I think it's a blocker and
> >> >> should
> >> >>>>>> be
> >> >>>>>>>>>>> fixed in 2.9. I've prepared the patch.
> >> >>>>>>>>>>> Taras Ledkov, can you please review this patch?
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> And one more ticket I propose to include into 2.9 [4] (NIO
> >> >>> message
> >> >>>>>>>>>>> send problem in some circumstances). I will cherry-pick it
> if
> >> >>>>>> there is no
> >> >>>>>>>>>>> objection.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13411
> >> >>>>>>>>>>> [2] https://github.com/apache/ignite/pull/8223
> >> >>>>>>>>>>> [3] https://issues.apache.org/jira/browse/IGNITE-13414
> >> >>>>>>>>>>> [4] https://issues.apache.org/jira/browse/IGNITE-13361
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> пн, 7 сент. 2020 г. в 14:14, Maxim Muzafarov <
> >> >> mmuzaf@apache.org
> >> >>>> :
> >> >>>>>>>>>>>
> >> >>>>>>>>>>>> Alexey,
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> I propose to include [1] issue to the 2.9 release. Since
> >> >> this
> >> >>>>>> issue is
> >> >>>>>>>>>>>> related to the new master key change functionality which
> >> >>>>>> haven't been
> >> >>>>>>>>>>>> released yet I think it will be safe to cherry-pick commit
> >> >> to
> >> >>>>>> the
> >> >>>>>>>>>>>> release branch.
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13390
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> On Tue, 1 Sep 2020 at 12:13, Nikolay Izhikov <
> >> >>>>>> nizhikov@apache.org>
> >> >>>>>>>>>>> wrote:
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>> Hello, Igniters.
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>> Alexey, please, include one more Python thin client fix
> >> >> [1]
> >> >>>>>> into the
> >> >>>>>>>>>>> 2.9
> >> >>>>>>>>>>>> release
> >> >>>>>>>>>>>>> It fixes kinda major issue - "Python client returns fields
> >> >>> in
> >> >>>>>> wrong
> >> >>>>>>>>>>>> order since the 2 row when fields_count>10"
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-12809
> >> >>>>>>>>>>>>> [2]
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>
> >> >>>>>>
> >> >>>
> >> >>
> >>
> https://github.com/apache/ignite/commit/38025ee4167f05eaa2d6a2c5c2ab70c83a462cfc
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> 31 авг. 2020 г., в 19:23, Alexey Goncharuk <
> >> >>>>>>>>>>> alexey.goncharuk@gmail.com>
> >> >>>>>>>>>>>> написал(а):
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> Alexey, thanks, got it. I am not sure we can optimize
> >> >>>>>> anything out of
> >> >>>>>>>>>>>> the
> >> >>>>>>>>>>>>>> message factory with suppliers (at least I have no ideas
> >> >>>>>> right now),
> >> >>>>>>>>>>> so
> >> >>>>>>>>>>>>>> most likely the only move here is to switch back to the
> >> >>>>>> switch
> >> >>>>>>>>>>> approach
> >> >>>>>>>>>>>>>> somehow preserving the metrics part. Probably, inlining
> >> >>> the
> >> >>>>>> Ignite
> >> >>>>>>>>>>>> messages
> >> >>>>>>>>>>>>>> to the IgniteMessageFactoryImpl should do the trick. Let
> >> >>> me
> >> >>>>>> explore
> >> >>>>>>>>>>> the
> >> >>>>>>>>>>>>>> code a bit.
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> P.S. I am surprised by the impact this part makes for
> >> >> the
> >> >>>>>>>>>>> performance.
> >> >>>>>>>>>>>>>> Message creation is indeed on the hot path, but a single
> >> >>>>>> virtual call
> >> >>>>>>>>>>>>>> should not make that much of a difference given the
> >> >> amount
> >> >>>>>> of other
> >> >>>>>>>>>>>> work
> >> >>>>>>>>>>>>>> happening during the message processing.
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 18:33, Alex Plehanov <
> >> >>>>>> plehanov.alex@gmail.com
> >> >>>>>>>>>>>> :
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> Alexey, sorry, I wrongly interpreted our benchmark
> >> >>> results.
> >> >>>>>>>>>>> Actually,
> >> >>>>>>>>>>>> we
> >> >>>>>>>>>>>>>>> were looking for a drop using bi-sect in the range
> >> >>> between
> >> >>>>>> e6a7f93
> >> >>>>>>>>>>>> (first
> >> >>>>>>>>>>>>>>> commit in the 2.9 branch after 2.8 branch cut) and
> >> >>>>>> 6592dfa5 (last
> >> >>>>>>>>>>>> commit in
> >> >>>>>>>>>>>>>>> the 2.9 branch). And we found these two problematic
> >> >>>>>> commits.
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> Perhaps only IGNITE-13060 (Tracing) is responsible for
> >> >> a
> >> >>>>>> drop
> >> >>>>>>>>>>> between
> >> >>>>>>>>>>>>>>> 2.8.1 and 2.9 (we have benchmarked 2.8.1 vs 2.9 with
> >> >>>>>> reverted
> >> >>>>>>>>>>>> IGNITE-13060
> >> >>>>>>>>>>>>>>> now and performance looks the same)
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> Ticket IGNITE-12568 (MessageFactory refactoring) is not
> >> >>>>>> related to
> >> >>>>>>>>>>>> drop
> >> >>>>>>>>>>>>>>> between 2.8.1 and 2.9, but still has some performance
> >> >>>>>> problem, and
> >> >>>>>>>>>>> we
> >> >>>>>>>>>>>> can
> >> >>>>>>>>>>>>>>> win back IGNITE-13060 drop by this ticket.
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> Do we need more investigation on IGNITE-13060 or we
> >> >> leave
> >> >>>>>> it as is?
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> What should we do with IGNITE-12568 (MessageFactory
> >> >>>>>> refactoring)?
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 13:25, Alexey Goncharuk <
> >> >>>>>>>>>>>> alexey.goncharuk@gmail.com
> >> >>>>>>>>>>>>>>>> :
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> Alexey,
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> While investigating, I found that IGNITE-12568 has an
> >> >>>>>> incorrect fix
> >> >>>>>>>>>>>>>>>> version and is actually present in ignite-2.8.1 branch
> >> >>>>>> [1], so it
> >> >>>>>>>>>>>> cannot be
> >> >>>>>>>>>>>>>>>> the source of the drop against 2.8.1.
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> P.S. Looks like we need to enforce a more accurate
> >> >> work
> >> >>>>>> with fix
> >> >>>>>>>>>>>> versions
> >> >>>>>>>>>>>>>>>> or develop some sort of tooling to verify the fix
> >> >>>>>> versions.
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> --AG
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> [1]
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>
> >> >>>>>>
> >> >>>
> >> >>
> >>
> https://github.com/apache/ignite/commit/3e492bd23851856bbd0385c6a419892d0bba2a34
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 12:42, Alexey Goncharuk <
> >> >>>>>>>>>>>> alexey.goncharuk@gmail.com
> >> >>>>>>>>>>>>>>>>> :
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>> пт, 28 авг. 2020 г. в 11:16, Alex Plehanov <
> >> >>>>>>>>>>> plehanov.alex@gmail.com
> >> >>>>>>>>>>>>> :
> >> >>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>> Guys,
> >> >>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>> We have benchmarked 2.9 without IGNITE-13060 and
> >> >>>>>> IGNITE-12568
> >> >>>>>>>>>>>> (reverted
> >> >>>>>>>>>>>>>>>>>> it
> >> >>>>>>>>>>>>>>>>>> locally) and got the same performance as on 2.8.1
> >> >>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>> IGNITE-13060 (Tracing) - some code was added to hot
> >> >>>>>> paths, to
> >> >>>>>>>>>>> trace
> >> >>>>>>>>>>>>>>>>>> these
> >> >>>>>>>>>>>>>>>>>> hot paths, it's clear why we have performance drop
> >> >>> here.
> >> >>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>> IGNITE-12568 (MessageFactory refactoring) -
> >> >>> switch/case
> >> >>>>>> block was
> >> >>>>>>>>>>>>>>>>>> refactored to an array of message suppliers. The
> >> >>>>>> message factory
> >> >>>>>>>>>>>> is on
> >> >>>>>>>>>>>>>>>>>> the
> >> >>>>>>>>>>>>>>>>>> hot path, which explains why this commit has an
> >> >> impact
> >> >>>>>> on total
> >> >>>>>>>>>>>>>>>>>> performance.
> >> >>>>>>>>>>>>>>>>>> I've checked JIT assembly output, done some JMH
> >> >>>>>> microbenchmarks,
> >> >>>>>>>>>>>> and
> >> >>>>>>>>>>>>>>>>>> found
> >> >>>>>>>>>>>>>>>>>> that old implementation of MessageFactory.create()
> >> >>>>>> about 30-35%
> >> >>>>>>>>>>>> faster
> >> >>>>>>>>>>>>>>>>>> than
> >> >>>>>>>>>>>>>>>>>> the new one. The reason - approach with switch/case
> >> >>> can
> >> >>>>>>>>>>> effectively
> >> >>>>>>>>>>>>>>>>>> inline
> >> >>>>>>>>>>>>>>>>>> message creation code, but with an array of
> >> >> suppliers
> >> >>>>>> relatively
> >> >>>>>>>>>>>> heavy
> >> >>>>>>>>>>>>>>>>>> "invokeinterface" cannot be skipped. I've tried to
> >> >>>>>> rewrite the
> >> >>>>>>>>>>> code
> >> >>>>>>>>>>>>>>>>>> using
> >> >>>>>>>>>>>>>>>>>> an abstract class for suppliers instead of an
> >> >>> interface
> >> >>>>>> (to
> >> >>>>>>>>>>>>>>>>>> replace "invokeinterface" with the "invokevirtual"),
> >> >>>>>> but it gives
> >> >>>>>>>>>>>> back
> >> >>>>>>>>>>>>>>>>>> only
> >> >>>>>>>>>>>>>>>>>> 10% of method performance and in this case, code
> >> >> looks
> >> >>>>>> ugly
> >> >>>>>>>>>>>> (lambdas
> >> >>>>>>>>>>>>>>>>>> can't
> >> >>>>>>>>>>>>>>>>>> be used). Currently, I can't find any more ways to
> >> >>>>>> optimize the
> >> >>>>>>>>>>>> current
> >> >>>>>>>>>>>>>>>>>> approach (except return to the switch/case block).
> >> >>>>>> Andrey Gura,
> >> >>>>>>>>>>> as
> >> >>>>>>>>>>>> the
> >> >>>>>>>>>>>>>>>>>> author of IGNITE-12568, maybe you have some ideas
> >> >>> about
> >> >>>>>>>>>>>> optimization?
> >> >>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>> Perhaps we should revert IGNITE-12568, but there are
> >> >>>>>> some metrics
> >> >>>>>>>>>>>>>>>>>> already
> >> >>>>>>>>>>>>>>>>>> created, which can't be rewritten using old message
> >> >>>>>> factory
> >> >>>>>>>>>>>>>>>>>> implementation
> >> >>>>>>>>>>>>>>>>>> (IGNITE-12756). Guys, WDYT?
> >> >>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>> Alexey,
> >> >>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>> I see that IGNITE-12756 (metrics improvements) is
> >> >>>>>> already released
> >> >>>>>>>>>>>> in
> >> >>>>>>>>>>>>>>>>> Ignite 2.8.1 while IGNITE-12568 (message factory) is
> >> >>>>>> only present
> >> >>>>>>>>>>>> in Ignite
> >> >>>>>>>>>>>>>>>>> 2.9. Let's revert both IGNITE-12568 and whichever new
> >> >>>>>> metrics
> >> >>>>>>>>>>>> created for
> >> >>>>>>>>>>>>>>>>> 2.9 that depend on the new message factory to unblock
> >> >>>>>> the release
> >> >>>>>>>>>>>> and deal
> >> >>>>>>>>>>>>>>>>> with the optimizations in 2.10?
> >> >>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>
> >> >>>>>>
> >> >>>>>
> >> >>>
> >> >>
> >>
> >>
>
Re: Apache Ignite 2.9.0 RELEASE [Time, Scope, Manager]
Posted by Denis Magda <dm...@apache.org>.
@Alex Plehanov <pl...@gmail.com>,
I crossed off the step 6.3.4 (technical docs publication)
<https://cwiki.apache.org/confluence/display/IGNITE/Release+Process#ReleaseProcess-6.3.4.Publishtechnicaldocumentation>
of
the release process by adding the new docs to the website's menu. You can
find them by going to the "Resources"->"Documentation" menu item. I will be
unreachable early next week, but it feels like the 2.9 vote will pass;
thus, decided to proceed with this step.
Just in case, here is the documentation contribution and publication
process. Use it if you need to tweak anything:
https://cwiki.apache.org/confluence/display/IGNITE/How+to+Document#HowtoDocument-PublishingtotheWebsite
-
Denis
On Mon, Oct 12, 2020 at 12:57 PM Denis Magda <dm...@apache.org> wrote:
> Saikat, Alex,
>
> Based on your inputs here, it sounds like once Ignite 2.9 gets released,
> all the integrations that made their way to the extensions repo [1] will
> get stuck in limbo for some time. What I mean here:
>
> 1. While the users will be bumping up their ignite-core,
> ignite-indexing, etc. versions to 2.9, they won't find a 2.9 artifact for
> Kafka, Camel and other extensions
> 2. Also, the users won't find 1.0 versions of those extensions that
> also need to be released
>
> So, it's unclear how those users are supposed to upgrade to Ignite 2.9 if
> they use any of the extensions. Is it safe to use a 2.8 version of an
> extension? Or, should we release the 1.0 versions of all the migrated
> extensions together with 2.9?
>
>
> [1] https://github.com/apache/ignite-extensions/tree/master/modules
>
>
> -
> Denis
>
>
> On Tue, Oct 6, 2020 at 7:20 AM Alexey Goncharuk <
> alexey.goncharuk@gmail.com> wrote:
>
>> Saikat,
>>
>> The plan sounds good to me! Do you have an idea for the timeline of the
>> module releases? Let me know if I can help in any way.
>>
>> Also, I was looking into the modularization IEP and noticed that OSGI
>> module is discussed to be moved to the extensions, but there is no
>> corresponding ticket. Should I create one? I will be happy to help with
>> moving this module to extensions.
>>
>> вт, 29 сент. 2020 г. в 03:03, Saikat Maitra <sa...@gmail.com>:
>>
>> > Hi Alexey,
>> >
>> > All the modules as planned in IEP-36
>> >
>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36:+Modularization
>> > are migrated to ignite-extensions repository.
>> >
>> > We can definitely plan to release ignite-extensions modules.
>> >
>> > I have a pending PR related to the kafka module and the PR is in review
>> > https://github.com/apache/ignite/pull/8222 but its corresponding PR has
>> > been merged in ignite-extensions repository.
>> >
>> > My thoughts / action items for the migration efforts and releases were
>> > as follows:
>> >
>> > 1. Merge the changes for https://github.com/apache/ignite/pull/8222
>> > 2. Fix the upload nightly packages task for ignite-core to help fix the
>> > travis build failure in ignite-extensions repository.
>> > 3. Review and update the README.md files for the ignite-extensions
>> modules
>> > as required.
>> > 4. Update the docs for ignite-extensions modules in ignite-website.
>> > 5. Release each module separately and share updates.
>> >
>> > Please let me know if you have feedback.
>> >
>> > Regards,
>> > Saikat
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Mon, Sep 28, 2020 at 7:36 AM Petr Ivanov <mr...@gmail.com>
>> wrote:
>> >
>> >> It seems that gitbox.apache.org is not available on CI/CD.
>> >>
>> >> I will try to update VCS to GitHub.
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> > On 28 Sep 2020, at 14:07, Alex Plehanov <pl...@gmail.com>
>> >> wrote:
>> >> >
>> >> > Guys,
>> >> >
>> >> > I've tried to build release candidate using TC [1]. But:
>> >> > 1. I can't choose a branch in this TC task (there is no such field).
>> >> > 2. On the "default" branch I got an error: Failed to collect changes,
>> >> > error: List remote refs failed:
>> >> > org.apache.http.conn.ConnectTimeoutException: Connect to
>> >> > gitbox.apache.org:443 [gitbox.apache.org/52.202.80.70] failed:
>> connect
>> >> > timed out, VCS root: "GitBox [asf/ignite]" {instance id=300, parent
>> >> > internal id=74, parent id=GitBoxAsfIgnite, description: "
>> >> > https://gitbox.apache.org/repos/asf/ignite.git#refs/heads/master"}
>> >> >
>> >> > Is there some problem with this TC task? What am I doing wrong?
>> >> >
>> >> > [1] :
>> >> >
>> >>
>> https://ci.ignite.apache.org/viewType.html?buildTypeId=Releases_ApacheIgniteMain_ReleaseBuild
>> >> >
>> >> > пн, 28 сент. 2020 г. в 13:15, Alexey Goncharuk <
>> >> alexey.goncharuk@gmail.com>:
>> >> >
>> >> >> Saikat, Nikolay,
>> >> >>
>> >> >> We have migrated a bunch of modules to ignite-extensions recently.
>> >> Given
>> >> >> that these modules will not be available in Ignite 2.9 anymore (will
>> >> >> they?), should we also plan to release the extensions?
>> >> >>
>> >> >> ср, 23 сент. 2020 г. в 21:49, Alex Plehanov <
>> plehanov.alex@gmail.com>:
>> >> >>
>> >> >>> Igniters,
>> >> >>>
>> >> >>> I've prepared release notes for the 2.9 release [1]. Can anyone
>> review
>> >> >> it,
>> >> >>> please?
>> >> >>>
>> >> >>> [1]: https://github.com/apache/ignite/pull/8273
>> >> >>>
>> >> >>> вт, 22 сент. 2020 г. в 09:39, Alex Plehanov <
>> plehanov.alex@gmail.com
>> >> >:
>> >> >>>
>> >> >>>> Guys,
>> >> >>>>
>> >> >>>> I've filled the ticket with reproducer [1] for the discovery bug.
>> >> This
>> >> >>> bug
>> >> >>>> caused by [2] ticket. We discussed with Vladimir Steshin privately
>> >> and
>> >> >>>> decided to revert this ticket. I will do it today (after TC bot
>> visa)
>> >> >> if
>> >> >>>> there are no objections.
>> >> >>>>
>> >> >>>> [1]: https://issues.apache.org/jira/browse/IGNITE-13465
>> >> >>>> [2]: https://issues.apache.org/jira/browse/IGNITE-13134
>> >> >>>>
>> >> >>>> пн, 21 сент. 2020 г. в 11:08, Alex Plehanov <
>> plehanov.alex@gmail.com
>> >> >:
>> >> >>>>
>> >> >>>>> Guys,
>> >> >>>>>
>> >> >>>>> During internal testing, we've found a critical bug with
>> >> >>>>> discovery (cluster falls apart if two nodes segmented
>> sequentially).
>> >> >>> This
>> >> >>>>> problem is not reproduced in 2.8.1. I think we should fix it
>> >> >>>>> before release. Under investigation now. I'll let you know when
>> we
>> >> get
>> >> >>>>> something.
>> >> >>>>>
>> >> >>>>> чт, 17 сент. 2020 г. в 00:51, Andrey Gura <ag...@apache.org>:
>> >> >>>>>
>> >> >>>>>>> So what do you think? Should we proceed with a 'hacked'
>> version of
>> >> >>> the
>> >> >>>>>> message factory in 2.9 and go for the runtime message
>> generation in
>> >> >>> later
>> >> >>>>>> release, or keep the code clean and fix the regression in the
>> next
>> >> >>> releases?
>> >> >>>>>>> Andrey, can you take a look at my change? I think it is fairly
>> >> >>>>>> straightforward and does not change the semantics, just skips
>> the
>> >> >>> factory
>> >> >>>>>> closures for certain messages.
>> >> >>>>>>
>> >> >>>>>> IMHO 2.5% isn't too much especially because it isn't actual for
>> all
>> >> >>>>>> workloads (I didn't get any significant drops during
>> benchmarking).
>> >> >> So
>> >> >>>>>> I prefer the runtime generation in later releases.
>> >> >>>>>>
>> >> >>>>>> On Mon, Sep 14, 2020 at 12:41 PM Alexey Goncharuk
>> >> >>>>>> <al...@gmail.com> wrote:
>> >> >>>>>>>
>> >> >>>>>>> Alexey, Andrey, Igniters,
>> >> >>>>>>>
>> >> >>>>>>> So what do you think? Should we proceed with a 'hacked'
>> version of
>> >> >>> the
>> >> >>>>>> message factory in 2.9 and go for the runtime message
>> generation in
>> >> >>> later
>> >> >>>>>> release, or keep the code clean and fix the regression in the
>> next
>> >> >>> releases?
>> >> >>>>>>> Andrey, can you take a look at my change? I think it is fairly
>> >> >>>>>> straightforward and does not change the semantics, just skips
>> the
>> >> >>> factory
>> >> >>>>>> closures for certain messages.
>> >> >>>>>>>
>> >> >>>>>>> Personally, I would prefer fixing the regression given that we
>> >> also
>> >> >>>>>> introduced tracing in this release.
>> >> >>>>>>>
>> >> >>>>>>>
>> >> >>>>>>>
>> >> >>>>>>> пт, 11 сент. 2020 г. в 12:09, Alex Plehanov <
>> >> >> plehanov.alex@gmail.com
>> >> >>>> :
>> >> >>>>>>>>
>> >> >>>>>>>> Alexey,
>> >> >>>>>>>>
>> >> >>>>>>>> We've benchmarked by yardstick commits 130376741bf vs
>> ed52559eb95
>> >> >>> and
>> >> >>>>>> the performance of ed52559eb95 is better for about 2.5% on
>> average
>> >> on
>> >> >>> our
>> >> >>>>>> environment (about the same results we got benchmarking
>> 65c30ec6 vs
>> >> >>>>>> 0606f03d). We've made 24 runs for each commit of
>> >> >>>>>> IgnitePutTxImplicitBenchmark (we got maximum drop for 2.9 on
>> this
>> >> >>>>>> benchmark), 200 seconds warmup, 300 seconds benchmark, 6
>> servers, 5
>> >> >>> clients
>> >> >>>>>> 50 threads each. Yardstick results for this configuration:
>> >> >>>>>>>> Commit 130376741bf: avg TPS=164096, avg latency=9173464 ns
>> >> >>>>>>>> Commit ed52559eb95: avg TPS=168283, avg latency=8945908 ns
>> >> >>>>>>>>
>> >> >>>>>>>> пт, 11 сент. 2020 г. в 09:51, Artem Budnikov <
>> >> >>>>>> a.budnikov.ignite@gmail.com>:
>> >> >>>>>>>>>
>> >> >>>>>>>>> Hi Everyone,
>> >> >>>>>>>>>
>> >> >>>>>>>>> I posted an instruction on how to publish the docs on
>> >> >>>>>> ignite.apache.org/docs [1]. When you finish with Ignite 2.9,
>> you
>> >> can
>> >> >>>>>> update the docs by following the instruction. Unfortunately, I
>> >> won't
>> >> >> be
>> >> >>>>>> able to spend any time on this project any longer. You can send
>> >> your
>> >> >>> pull
>> >> >>>>>> requests and questions about the documentation to Denis Magda.
>> >> >>>>>>>>>
>> >> >>>>>>>>> -Artem
>> >> >>>>>>>>>
>> >> >>>>>>>>> [1] :
>> >> >>>>>>
>> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Document
>> >> >>>>>>>>>
>> >> >>>>>>>>> On Thu, Sep 10, 2020 at 2:45 PM Alexey Goncharuk <
>> >> >>>>>> alexey.goncharuk@gmail.com> wrote:
>> >> >>>>>>>>>>
>> >> >>>>>>>>>> Alexey,
>> >> >>>>>>>>>>
>> >> >>>>>>>>>> I've tried to play with message factories locally, but
>> >> >>>>>> unfortunately, I
>> >> >>>>>>>>>> cannot spot the difference between old and new
>> implementation
>> >> in
>> >> >>>>>>>>>> distributed benchmarks. I pushed an implementation of
>> >> >>>>>> MessageFactoryImpl
>> >> >>>>>>>>>> with the old switch statement to the ignite-2.9-revert-12568
>> >> >>> branch
>> >> >>>>>>>>>> (discussed this with Andrey Gura, the change should be
>> >> >> compatible
>> >> >>>>>> with the
>> >> >>>>>>>>>> new metrics as we still use the register() mechanics).
>> >> >>>>>>>>>>
>> >> >>>>>>>>>> Can you check if this change makes any difference
>> >> >> performance-wise
>> >> >>>>>> in your
>> >> >>>>>>>>>> environment? If yes, we can go with runtime code generation
>> in
>> >> >> the
>> >> >>>>>> long
>> >> >>>>>>>>>> term: register classes and generate a dynamic message
>> factory
>> >> >> with
>> >> >>>>>> a switch
>> >> >>>>>>>>>> statement once all messages are registered (not in 2.9
>> though,
>> >> >>>>>> obviously).
>> >> >>>>>>>>>>
>> >> >>>>>>>>>> ср, 9 сент. 2020 г. в 14:53, Alex Plehanov <
>> >> >>> plehanov.alex@gmail.com
>> >> >>>>>>> :
>> >> >>>>>>>>>>
>> >> >>>>>>>>>>> Hello guys,
>> >> >>>>>>>>>>>
>> >> >>>>>>>>>>> I've tried to optimize tracing implementation (ticket
>> [1]), it
>> >> >>>>>> reduced the
>> >> >>>>>>>>>>> drop, but not completely removed it.
>> >> >>>>>>>>>>> Ivan Rakov, Alexander Lapin, can you please review the
>> patch?
>> >> >>>>>>>>>>> Ivan Artiukhov, can you please benchmark the patch [2]
>> against
>> >> >>>>>> 2.8.1
>> >> >>>>>>>>>>> release on your environment?
>> >> >>>>>>>>>>> With this patch on our environment, it's about a 3% drop
>> left,
>> >> >>>>>> it's close
>> >> >>>>>>>>>>> to measurement error and I think such a drop is not a
>> >> >>>>>> showstopper. Guys,
>> >> >>>>>>>>>>> WDYT?
>> >> >>>>>>>>>>>
>> >> >>>>>>>>>>> Also, I found that compatibility is broken for JDBC thin
>> >> >> driver
>> >> >>>>>> between 2.8
>> >> >>>>>>>>>>> and 2.9 versions (ticket [3]). I think it's a blocker and
>> >> >> should
>> >> >>>>>> be
>> >> >>>>>>>>>>> fixed in 2.9. I've prepared the patch.
>> >> >>>>>>>>>>> Taras Ledkov, can you please review this patch?
>> >> >>>>>>>>>>>
>> >> >>>>>>>>>>> And one more ticket I propose to include into 2.9 [4] (NIO
>> >> >>> message
>> >> >>>>>>>>>>> send problem in some circumstances). I will cherry-pick it
>> if
>> >> >>>>>> there is no
>> >> >>>>>>>>>>> objection.
>> >> >>>>>>>>>>>
>> >> >>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13411
>> >> >>>>>>>>>>> [2] https://github.com/apache/ignite/pull/8223
>> >> >>>>>>>>>>> [3] https://issues.apache.org/jira/browse/IGNITE-13414
>> >> >>>>>>>>>>> [4] https://issues.apache.org/jira/browse/IGNITE-13361
>> >> >>>>>>>>>>>
>> >> >>>>>>>>>>> пн, 7 сент. 2020 г. в 14:14, Maxim Muzafarov <
>> >> >> mmuzaf@apache.org
>> >> >>>> :
>> >> >>>>>>>>>>>
>> >> >>>>>>>>>>>> Alexey,
>> >> >>>>>>>>>>>>
>> >> >>>>>>>>>>>> I propose to include [1] issue to the 2.9 release. Since
>> >> >> this
>> >> >>>>>> issue is
>> >> >>>>>>>>>>>> related to the new master key change functionality which
>> >> >>>>>> haven't been
>> >> >>>>>>>>>>>> released yet I think it will be safe to cherry-pick commit
>> >> >> to
>> >> >>>>>> the
>> >> >>>>>>>>>>>> release branch.
>> >> >>>>>>>>>>>>
>> >> >>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13390
>> >> >>>>>>>>>>>>
>> >> >>>>>>>>>>>> On Tue, 1 Sep 2020 at 12:13, Nikolay Izhikov <
>> >> >>>>>> nizhikov@apache.org>
>> >> >>>>>>>>>>> wrote:
>> >> >>>>>>>>>>>>>
>> >> >>>>>>>>>>>>> Hello, Igniters.
>> >> >>>>>>>>>>>>>
>> >> >>>>>>>>>>>>> Alexey, please, include one more Python thin client fix
>> >> >> [1]
>> >> >>>>>> into the
>> >> >>>>>>>>>>> 2.9
>> >> >>>>>>>>>>>> release
>> >> >>>>>>>>>>>>> It fixes kinda major issue - "Python client returns
>> fields
>> >> >>> in
>> >> >>>>>> wrong
>> >> >>>>>>>>>>>> order since the 2 row when fields_count>10"
>> >> >>>>>>>>>>>>>
>> >> >>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-12809
>> >> >>>>>>>>>>>>> [2]
>> >> >>>>>>>>>>>>
>> >> >>>>>>>>>>>
>> >> >>>>>>
>> >> >>>
>> >> >>
>> >>
>> https://github.com/apache/ignite/commit/38025ee4167f05eaa2d6a2c5c2ab70c83a462cfc
>> >> >>>>>>>>>>>>>
>> >> >>>>>>>>>>>>>> 31 авг. 2020 г., в 19:23, Alexey Goncharuk <
>> >> >>>>>>>>>>> alexey.goncharuk@gmail.com>
>> >> >>>>>>>>>>>> написал(а):
>> >> >>>>>>>>>>>>>>
>> >> >>>>>>>>>>>>>> Alexey, thanks, got it. I am not sure we can optimize
>> >> >>>>>> anything out of
>> >> >>>>>>>>>>>> the
>> >> >>>>>>>>>>>>>> message factory with suppliers (at least I have no ideas
>> >> >>>>>> right now),
>> >> >>>>>>>>>>> so
>> >> >>>>>>>>>>>>>> most likely the only move here is to switch back to the
>> >> >>>>>> switch
>> >> >>>>>>>>>>> approach
>> >> >>>>>>>>>>>>>> somehow preserving the metrics part. Probably, inlining
>> >> >>> the
>> >> >>>>>> Ignite
>> >> >>>>>>>>>>>> messages
>> >> >>>>>>>>>>>>>> to the IgniteMessageFactoryImpl should do the trick. Let
>> >> >>> me
>> >> >>>>>> explore
>> >> >>>>>>>>>>> the
>> >> >>>>>>>>>>>>>> code a bit.
>> >> >>>>>>>>>>>>>>
>> >> >>>>>>>>>>>>>> P.S. I am surprised by the impact this part makes for
>> >> >> the
>> >> >>>>>>>>>>> performance.
>> >> >>>>>>>>>>>>>> Message creation is indeed on the hot path, but a single
>> >> >>>>>> virtual call
>> >> >>>>>>>>>>>>>> should not make that much of a difference given the
>> >> >> amount
>> >> >>>>>> of other
>> >> >>>>>>>>>>>> work
>> >> >>>>>>>>>>>>>> happening during the message processing.
>> >> >>>>>>>>>>>>>>
>> >> >>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 18:33, Alex Plehanov <
>> >> >>>>>> plehanov.alex@gmail.com
>> >> >>>>>>>>>>>> :
>> >> >>>>>>>>>>>>>>
>> >> >>>>>>>>>>>>>>> Alexey, sorry, I wrongly interpreted our benchmark
>> >> >>> results.
>> >> >>>>>>>>>>> Actually,
>> >> >>>>>>>>>>>> we
>> >> >>>>>>>>>>>>>>> were looking for a drop using bi-sect in the range
>> >> >>> between
>> >> >>>>>> e6a7f93
>> >> >>>>>>>>>>>> (first
>> >> >>>>>>>>>>>>>>> commit in the 2.9 branch after 2.8 branch cut) and
>> >> >>>>>> 6592dfa5 (last
>> >> >>>>>>>>>>>> commit in
>> >> >>>>>>>>>>>>>>> the 2.9 branch). And we found these two problematic
>> >> >>>>>> commits.
>> >> >>>>>>>>>>>>>>>
>> >> >>>>>>>>>>>>>>> Perhaps only IGNITE-13060 (Tracing) is responsible for
>> >> >> a
>> >> >>>>>> drop
>> >> >>>>>>>>>>> between
>> >> >>>>>>>>>>>>>>> 2.8.1 and 2.9 (we have benchmarked 2.8.1 vs 2.9 with
>> >> >>>>>> reverted
>> >> >>>>>>>>>>>> IGNITE-13060
>> >> >>>>>>>>>>>>>>> now and performance looks the same)
>> >> >>>>>>>>>>>>>>>
>> >> >>>>>>>>>>>>>>> Ticket IGNITE-12568 (MessageFactory refactoring) is not
>> >> >>>>>> related to
>> >> >>>>>>>>>>>> drop
>> >> >>>>>>>>>>>>>>> between 2.8.1 and 2.9, but still has some performance
>> >> >>>>>> problem, and
>> >> >>>>>>>>>>> we
>> >> >>>>>>>>>>>> can
>> >> >>>>>>>>>>>>>>> win back IGNITE-13060 drop by this ticket.
>> >> >>>>>>>>>>>>>>>
>> >> >>>>>>>>>>>>>>> Do we need more investigation on IGNITE-13060 or we
>> >> >> leave
>> >> >>>>>> it as is?
>> >> >>>>>>>>>>>>>>>
>> >> >>>>>>>>>>>>>>> What should we do with IGNITE-12568 (MessageFactory
>> >> >>>>>> refactoring)?
>> >> >>>>>>>>>>>>>>>
>> >> >>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 13:25, Alexey Goncharuk <
>> >> >>>>>>>>>>>> alexey.goncharuk@gmail.com
>> >> >>>>>>>>>>>>>>>> :
>> >> >>>>>>>>>>>>>>>
>> >> >>>>>>>>>>>>>>>> Alexey,
>> >> >>>>>>>>>>>>>>>>
>> >> >>>>>>>>>>>>>>>> While investigating, I found that IGNITE-12568 has an
>> >> >>>>>> incorrect fix
>> >> >>>>>>>>>>>>>>>> version and is actually present in ignite-2.8.1 branch
>> >> >>>>>> [1], so it
>> >> >>>>>>>>>>>> cannot be
>> >> >>>>>>>>>>>>>>>> the source of the drop against 2.8.1.
>> >> >>>>>>>>>>>>>>>>
>> >> >>>>>>>>>>>>>>>> P.S. Looks like we need to enforce a more accurate
>> >> >> work
>> >> >>>>>> with fix
>> >> >>>>>>>>>>>> versions
>> >> >>>>>>>>>>>>>>>> or develop some sort of tooling to verify the fix
>> >> >>>>>> versions.
>> >> >>>>>>>>>>>>>>>>
>> >> >>>>>>>>>>>>>>>> --AG
>> >> >>>>>>>>>>>>>>>>
>> >> >>>>>>>>>>>>>>>> [1]
>> >> >>>>>>>>>>>>>>>>
>> >> >>>>>>>>>>>>
>> >> >>>>>>>>>>>
>> >> >>>>>>
>> >> >>>
>> >> >>
>> >>
>> https://github.com/apache/ignite/commit/3e492bd23851856bbd0385c6a419892d0bba2a34
>> >> >>>>>>>>>>>>>>>>
>> >> >>>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 12:42, Alexey Goncharuk <
>> >> >>>>>>>>>>>> alexey.goncharuk@gmail.com
>> >> >>>>>>>>>>>>>>>>> :
>> >> >>>>>>>>>>>>>>>>
>> >> >>>>>>>>>>>>>>>>> пт, 28 авг. 2020 г. в 11:16, Alex Plehanov <
>> >> >>>>>>>>>>> plehanov.alex@gmail.com
>> >> >>>>>>>>>>>>> :
>> >> >>>>>>>>>>>>>>>>>
>> >> >>>>>>>>>>>>>>>>>> Guys,
>> >> >>>>>>>>>>>>>>>>>>
>> >> >>>>>>>>>>>>>>>>>> We have benchmarked 2.9 without IGNITE-13060 and
>> >> >>>>>> IGNITE-12568
>> >> >>>>>>>>>>>> (reverted
>> >> >>>>>>>>>>>>>>>>>> it
>> >> >>>>>>>>>>>>>>>>>> locally) and got the same performance as on 2.8.1
>> >> >>>>>>>>>>>>>>>>>>
>> >> >>>>>>>>>>>>>>>>>> IGNITE-13060 (Tracing) - some code was added to hot
>> >> >>>>>> paths, to
>> >> >>>>>>>>>>> trace
>> >> >>>>>>>>>>>>>>>>>> these
>> >> >>>>>>>>>>>>>>>>>> hot paths, it's clear why we have performance drop
>> >> >>> here.
>> >> >>>>>>>>>>>>>>>>>>
>> >> >>>>>>>>>>>>>>>>>> IGNITE-12568 (MessageFactory refactoring) -
>> >> >>> switch/case
>> >> >>>>>> block was
>> >> >>>>>>>>>>>>>>>>>> refactored to an array of message suppliers. The
>> >> >>>>>> message factory
>> >> >>>>>>>>>>>> is on
>> >> >>>>>>>>>>>>>>>>>> the
>> >> >>>>>>>>>>>>>>>>>> hot path, which explains why this commit has an
>> >> >> impact
>> >> >>>>>> on total
>> >> >>>>>>>>>>>>>>>>>> performance.
>> >> >>>>>>>>>>>>>>>>>> I've checked JIT assembly output, done some JMH
>> >> >>>>>> microbenchmarks,
>> >> >>>>>>>>>>>> and
>> >> >>>>>>>>>>>>>>>>>> found
>> >> >>>>>>>>>>>>>>>>>> that old implementation of MessageFactory.create()
>> >> >>>>>> about 30-35%
>> >> >>>>>>>>>>>> faster
>> >> >>>>>>>>>>>>>>>>>> than
>> >> >>>>>>>>>>>>>>>>>> the new one. The reason - approach with switch/case
>> >> >>> can
>> >> >>>>>>>>>>> effectively
>> >> >>>>>>>>>>>>>>>>>> inline
>> >> >>>>>>>>>>>>>>>>>> message creation code, but with an array of
>> >> >> suppliers
>> >> >>>>>> relatively
>> >> >>>>>>>>>>>> heavy
>> >> >>>>>>>>>>>>>>>>>> "invokeinterface" cannot be skipped. I've tried to
>> >> >>>>>> rewrite the
>> >> >>>>>>>>>>> code
>> >> >>>>>>>>>>>>>>>>>> using
>> >> >>>>>>>>>>>>>>>>>> an abstract class for suppliers instead of an
>> >> >>> interface
>> >> >>>>>> (to
>> >> >>>>>>>>>>>>>>>>>> replace "invokeinterface" with the "invokevirtual"),
>> >> >>>>>> but it gives
>> >> >>>>>>>>>>>> back
>> >> >>>>>>>>>>>>>>>>>> only
>> >> >>>>>>>>>>>>>>>>>> 10% of method performance and in this case, code
>> >> >> looks
>> >> >>>>>> ugly
>> >> >>>>>>>>>>>> (lambdas
>> >> >>>>>>>>>>>>>>>>>> can't
>> >> >>>>>>>>>>>>>>>>>> be used). Currently, I can't find any more ways to
>> >> >>>>>> optimize the
>> >> >>>>>>>>>>>> current
>> >> >>>>>>>>>>>>>>>>>> approach (except return to the switch/case block).
>> >> >>>>>> Andrey Gura,
>> >> >>>>>>>>>>> as
>> >> >>>>>>>>>>>> the
>> >> >>>>>>>>>>>>>>>>>> author of IGNITE-12568, maybe you have some ideas
>> >> >>> about
>> >> >>>>>>>>>>>> optimization?
>> >> >>>>>>>>>>>>>>>>>>
>> >> >>>>>>>>>>>>>>>>>> Perhaps we should revert IGNITE-12568, but there are
>> >> >>>>>> some metrics
>> >> >>>>>>>>>>>>>>>>>> already
>> >> >>>>>>>>>>>>>>>>>> created, which can't be rewritten using old message
>> >> >>>>>> factory
>> >> >>>>>>>>>>>>>>>>>> implementation
>> >> >>>>>>>>>>>>>>>>>> (IGNITE-12756). Guys, WDYT?
>> >> >>>>>>>>>>>>>>>>>>
>> >> >>>>>>>>>>>>>>>>>
>> >> >>>>>>>>>>>>>>>>> Alexey,
>> >> >>>>>>>>>>>>>>>>>
>> >> >>>>>>>>>>>>>>>>> I see that IGNITE-12756 (metrics improvements) is
>> >> >>>>>> already released
>> >> >>>>>>>>>>>> in
>> >> >>>>>>>>>>>>>>>>> Ignite 2.8.1 while IGNITE-12568 (message factory) is
>> >> >>>>>> only present
>> >> >>>>>>>>>>>> in Ignite
>> >> >>>>>>>>>>>>>>>>> 2.9. Let's revert both IGNITE-12568 and whichever new
>> >> >>>>>> metrics
>> >> >>>>>>>>>>>> created for
>> >> >>>>>>>>>>>>>>>>> 2.9 that depend on the new message factory to unblock
>> >> >>>>>> the release
>> >> >>>>>>>>>>>> and deal
>> >> >>>>>>>>>>>>>>>>> with the optimizations in 2.10?
>> >> >>>>>>>>>>>>>>>>>
>> >> >>>>>>>>>>>>>>>>
>> >> >>>>>>>>>>>>>
>> >> >>>>>>>>>>>>
>> >> >>>>>>>>>>>
>> >> >>>>>>
>> >> >>>>>
>> >> >>>
>> >> >>
>> >>
>> >>
>>
>
Re: Apache Ignite 2.9.0 RELEASE [Time, Scope, Manager]
Posted by Alex Plehanov <pl...@gmail.com>.
Guys,
I've cherry-picked IGNITE-12350, IGNITE-13376, IGNITE-13280, and
documentation to 2.9.
Denis,
I see no benefits of a single vote for Ignite and extensions. I think it
will postpone Ignite 2.9 release. If some user uses some extension, he
still can continue to work with Ignite 2.8 release until ignite-extensions
is released. But those who don't use any extensions can gain benefits of
2.9 earlier.
Also, as far as I understand, there were no changes in extensions since
2.8. Only migration to the new repository. So it's possible to use Ignite
2.8 extensions with Ignite 2.9 core.
вт, 13 окт. 2020 г. в 07:34, Denis Magda <dm...@apache.org>:
> Is it possible/reasonable to run a single vote for Ignite 2.9 and all the
> migrated extensions? This time. @Alex Plehanov <pl...@gmail.com>,
> what do you think?
>
> If we decide to release the versions 1.0.0 of the extensions after 2.9 is
> out then it can last for a week or so...because we need to build, update
> docs, vote, publish.
>
> -
> Denis
>
>
> On Mon, Oct 12, 2020 at 7:19 PM Saikat Maitra <sa...@gmail.com>
> wrote:
>
>> Hi Denis,
>>
>> All the modules except OSGi have been migrated. Alexey is creating a
>> separate ticket for OSGi and we can migrate the same as well.
>>
>> My thoughts are Apache Ignite 2.9.0 and migrated extensions 1.0.0 to be
>> released together. I am thinking depending on the voting process and
>> testing cycles we can release the extensions 1.0.0 together with Apache
>> Ignite 2.9.0 or right after within days of Apache Ignite 2.9.0 release.
>>
>>
>> Regards,
>> Saikat
>>
>> On Mon, Oct 12, 2020 at 2:58 PM Denis Magda <dm...@apache.org> wrote:
>>
>>> Saikat, Alex,
>>>
>>> Based on your inputs here, it sounds like once Ignite 2.9 gets released,
>>> all the integrations that made their way to the extensions repo [1] will
>>> get stuck in limbo for some time. What I mean here:
>>>
>>> 1. While the users will be bumping up their ignite-core,
>>> ignite-indexing, etc. versions to 2.9, they won't find a 2.9 artifact for
>>> Kafka, Camel and other extensions
>>> 2. Also, the users won't find 1.0 versions of those extensions that
>>> also need to be released
>>>
>>> So, it's unclear how those users are supposed to upgrade to Ignite 2.9
>>> if they use any of the extensions. Is it safe to use a 2.8 version of an
>>> extension? Or, should we release the 1.0 versions of all the migrated
>>> extensions together with 2.9?
>>>
>>>
>>> [1] https://github.com/apache/ignite-extensions/tree/master/modules
>>>
>>>
>>> -
>>> Denis
>>>
>>>
>>> On Tue, Oct 6, 2020 at 7:20 AM Alexey Goncharuk <
>>> alexey.goncharuk@gmail.com> wrote:
>>>
>>>> Saikat,
>>>>
>>>> The plan sounds good to me! Do you have an idea for the timeline of the
>>>> module releases? Let me know if I can help in any way.
>>>>
>>>> Also, I was looking into the modularization IEP and noticed that OSGI
>>>> module is discussed to be moved to the extensions, but there is no
>>>> corresponding ticket. Should I create one? I will be happy to help with
>>>> moving this module to extensions.
>>>>
>>>> вт, 29 сент. 2020 г. в 03:03, Saikat Maitra <sa...@gmail.com>:
>>>>
>>>> > Hi Alexey,
>>>> >
>>>> > All the modules as planned in IEP-36
>>>> >
>>>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36:+Modularization
>>>> > are migrated to ignite-extensions repository.
>>>> >
>>>> > We can definitely plan to release ignite-extensions modules.
>>>> >
>>>> > I have a pending PR related to the kafka module and the PR is in
>>>> review
>>>> > https://github.com/apache/ignite/pull/8222 but its corresponding PR
>>>> has
>>>> > been merged in ignite-extensions repository.
>>>> >
>>>> > My thoughts / action items for the migration efforts and releases were
>>>> > as follows:
>>>> >
>>>> > 1. Merge the changes for https://github.com/apache/ignite/pull/8222
>>>> > 2. Fix the upload nightly packages task for ignite-core to help fix
>>>> the
>>>> > travis build failure in ignite-extensions repository.
>>>> > 3. Review and update the README.md files for the ignite-extensions
>>>> modules
>>>> > as required.
>>>> > 4. Update the docs for ignite-extensions modules in ignite-website.
>>>> > 5. Release each module separately and share updates.
>>>> >
>>>> > Please let me know if you have feedback.
>>>> >
>>>> > Regards,
>>>> > Saikat
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > On Mon, Sep 28, 2020 at 7:36 AM Petr Ivanov <mr...@gmail.com>
>>>> wrote:
>>>> >
>>>> >> It seems that gitbox.apache.org is not available on CI/CD.
>>>> >>
>>>> >> I will try to update VCS to GitHub.
>>>> >>
>>>> >>
>>>> >>
>>>> >>
>>>> >>
>>>> >> > On 28 Sep 2020, at 14:07, Alex Plehanov <pl...@gmail.com>
>>>> >> wrote:
>>>> >> >
>>>> >> > Guys,
>>>> >> >
>>>> >> > I've tried to build release candidate using TC [1]. But:
>>>> >> > 1. I can't choose a branch in this TC task (there is no such
>>>> field).
>>>> >> > 2. On the "default" branch I got an error: Failed to collect
>>>> changes,
>>>> >> > error: List remote refs failed:
>>>> >> > org.apache.http.conn.ConnectTimeoutException: Connect to
>>>> >> > gitbox.apache.org:443 [gitbox.apache.org/52.202.80.70] failed:
>>>> connect
>>>> >> > timed out, VCS root: "GitBox [asf/ignite]" {instance id=300, parent
>>>> >> > internal id=74, parent id=GitBoxAsfIgnite, description: "
>>>> >> > https://gitbox.apache.org/repos/asf/ignite.git#refs/heads/master"}
>>>> >> >
>>>> >> > Is there some problem with this TC task? What am I doing wrong?
>>>> >> >
>>>> >> > [1] :
>>>> >> >
>>>> >>
>>>> https://ci.ignite.apache.org/viewType.html?buildTypeId=Releases_ApacheIgniteMain_ReleaseBuild
>>>> >> >
>>>> >> > пн, 28 сент. 2020 г. в 13:15, Alexey Goncharuk <
>>>> >> alexey.goncharuk@gmail.com>:
>>>> >> >
>>>> >> >> Saikat, Nikolay,
>>>> >> >>
>>>> >> >> We have migrated a bunch of modules to ignite-extensions recently.
>>>> >> Given
>>>> >> >> that these modules will not be available in Ignite 2.9 anymore
>>>> (will
>>>> >> >> they?), should we also plan to release the extensions?
>>>> >> >>
>>>> >> >> ср, 23 сент. 2020 г. в 21:49, Alex Plehanov <
>>>> plehanov.alex@gmail.com>:
>>>> >> >>
>>>> >> >>> Igniters,
>>>> >> >>>
>>>> >> >>> I've prepared release notes for the 2.9 release [1]. Can anyone
>>>> review
>>>> >> >> it,
>>>> >> >>> please?
>>>> >> >>>
>>>> >> >>> [1]: https://github.com/apache/ignite/pull/8273
>>>> >> >>>
>>>> >> >>> вт, 22 сент. 2020 г. в 09:39, Alex Plehanov <
>>>> plehanov.alex@gmail.com
>>>> >> >:
>>>> >> >>>
>>>> >> >>>> Guys,
>>>> >> >>>>
>>>> >> >>>> I've filled the ticket with reproducer [1] for the discovery
>>>> bug.
>>>> >> This
>>>> >> >>> bug
>>>> >> >>>> caused by [2] ticket. We discussed with Vladimir Steshin
>>>> privately
>>>> >> and
>>>> >> >>>> decided to revert this ticket. I will do it today (after TC bot
>>>> visa)
>>>> >> >> if
>>>> >> >>>> there are no objections.
>>>> >> >>>>
>>>> >> >>>> [1]: https://issues.apache.org/jira/browse/IGNITE-13465
>>>> >> >>>> [2]: https://issues.apache.org/jira/browse/IGNITE-13134
>>>> >> >>>>
>>>> >> >>>> пн, 21 сент. 2020 г. в 11:08, Alex Plehanov <
>>>> plehanov.alex@gmail.com
>>>> >> >:
>>>> >> >>>>
>>>> >> >>>>> Guys,
>>>> >> >>>>>
>>>> >> >>>>> During internal testing, we've found a critical bug with
>>>> >> >>>>> discovery (cluster falls apart if two nodes segmented
>>>> sequentially).
>>>> >> >>> This
>>>> >> >>>>> problem is not reproduced in 2.8.1. I think we should fix it
>>>> >> >>>>> before release. Under investigation now. I'll let you know
>>>> when we
>>>> >> get
>>>> >> >>>>> something.
>>>> >> >>>>>
>>>> >> >>>>> чт, 17 сент. 2020 г. в 00:51, Andrey Gura <ag...@apache.org>:
>>>> >> >>>>>
>>>> >> >>>>>>> So what do you think? Should we proceed with a 'hacked'
>>>> version of
>>>> >> >>> the
>>>> >> >>>>>> message factory in 2.9 and go for the runtime message
>>>> generation in
>>>> >> >>> later
>>>> >> >>>>>> release, or keep the code clean and fix the regression in the
>>>> next
>>>> >> >>> releases?
>>>> >> >>>>>>> Andrey, can you take a look at my change? I think it is
>>>> fairly
>>>> >> >>>>>> straightforward and does not change the semantics, just skips
>>>> the
>>>> >> >>> factory
>>>> >> >>>>>> closures for certain messages.
>>>> >> >>>>>>
>>>> >> >>>>>> IMHO 2.5% isn't too much especially because it isn't actual
>>>> for all
>>>> >> >>>>>> workloads (I didn't get any significant drops during
>>>> benchmarking).
>>>> >> >> So
>>>> >> >>>>>> I prefer the runtime generation in later releases.
>>>> >> >>>>>>
>>>> >> >>>>>> On Mon, Sep 14, 2020 at 12:41 PM Alexey Goncharuk
>>>> >> >>>>>> <al...@gmail.com> wrote:
>>>> >> >>>>>>>
>>>> >> >>>>>>> Alexey, Andrey, Igniters,
>>>> >> >>>>>>>
>>>> >> >>>>>>> So what do you think? Should we proceed with a 'hacked'
>>>> version of
>>>> >> >>> the
>>>> >> >>>>>> message factory in 2.9 and go for the runtime message
>>>> generation in
>>>> >> >>> later
>>>> >> >>>>>> release, or keep the code clean and fix the regression in the
>>>> next
>>>> >> >>> releases?
>>>> >> >>>>>>> Andrey, can you take a look at my change? I think it is
>>>> fairly
>>>> >> >>>>>> straightforward and does not change the semantics, just skips
>>>> the
>>>> >> >>> factory
>>>> >> >>>>>> closures for certain messages.
>>>> >> >>>>>>>
>>>> >> >>>>>>> Personally, I would prefer fixing the regression given that
>>>> we
>>>> >> also
>>>> >> >>>>>> introduced tracing in this release.
>>>> >> >>>>>>>
>>>> >> >>>>>>>
>>>> >> >>>>>>>
>>>> >> >>>>>>> пт, 11 сент. 2020 г. в 12:09, Alex Plehanov <
>>>> >> >> plehanov.alex@gmail.com
>>>> >> >>>> :
>>>> >> >>>>>>>>
>>>> >> >>>>>>>> Alexey,
>>>> >> >>>>>>>>
>>>> >> >>>>>>>> We've benchmarked by yardstick commits 130376741bf vs
>>>> ed52559eb95
>>>> >> >>> and
>>>> >> >>>>>> the performance of ed52559eb95 is better for about 2.5% on
>>>> average
>>>> >> on
>>>> >> >>> our
>>>> >> >>>>>> environment (about the same results we got benchmarking
>>>> 65c30ec6 vs
>>>> >> >>>>>> 0606f03d). We've made 24 runs for each commit of
>>>> >> >>>>>> IgnitePutTxImplicitBenchmark (we got maximum drop for 2.9 on
>>>> this
>>>> >> >>>>>> benchmark), 200 seconds warmup, 300 seconds benchmark, 6
>>>> servers, 5
>>>> >> >>> clients
>>>> >> >>>>>> 50 threads each. Yardstick results for this configuration:
>>>> >> >>>>>>>> Commit 130376741bf: avg TPS=164096, avg latency=9173464 ns
>>>> >> >>>>>>>> Commit ed52559eb95: avg TPS=168283, avg latency=8945908 ns
>>>> >> >>>>>>>>
>>>> >> >>>>>>>> пт, 11 сент. 2020 г. в 09:51, Artem Budnikov <
>>>> >> >>>>>> a.budnikov.ignite@gmail.com>:
>>>> >> >>>>>>>>>
>>>> >> >>>>>>>>> Hi Everyone,
>>>> >> >>>>>>>>>
>>>> >> >>>>>>>>> I posted an instruction on how to publish the docs on
>>>> >> >>>>>> ignite.apache.org/docs [1]. When you finish with Ignite 2.9,
>>>> you
>>>> >> can
>>>> >> >>>>>> update the docs by following the instruction. Unfortunately, I
>>>> >> won't
>>>> >> >> be
>>>> >> >>>>>> able to spend any time on this project any longer. You can
>>>> send
>>>> >> your
>>>> >> >>> pull
>>>> >> >>>>>> requests and questions about the documentation to Denis Magda.
>>>> >> >>>>>>>>>
>>>> >> >>>>>>>>> -Artem
>>>> >> >>>>>>>>>
>>>> >> >>>>>>>>> [1] :
>>>> >> >>>>>>
>>>> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Document
>>>> >> >>>>>>>>>
>>>> >> >>>>>>>>> On Thu, Sep 10, 2020 at 2:45 PM Alexey Goncharuk <
>>>> >> >>>>>> alexey.goncharuk@gmail.com> wrote:
>>>> >> >>>>>>>>>>
>>>> >> >>>>>>>>>> Alexey,
>>>> >> >>>>>>>>>>
>>>> >> >>>>>>>>>> I've tried to play with message factories locally, but
>>>> >> >>>>>> unfortunately, I
>>>> >> >>>>>>>>>> cannot spot the difference between old and new
>>>> implementation
>>>> >> in
>>>> >> >>>>>>>>>> distributed benchmarks. I pushed an implementation of
>>>> >> >>>>>> MessageFactoryImpl
>>>> >> >>>>>>>>>> with the old switch statement to the
>>>> ignite-2.9-revert-12568
>>>> >> >>> branch
>>>> >> >>>>>>>>>> (discussed this with Andrey Gura, the change should be
>>>> >> >> compatible
>>>> >> >>>>>> with the
>>>> >> >>>>>>>>>> new metrics as we still use the register() mechanics).
>>>> >> >>>>>>>>>>
>>>> >> >>>>>>>>>> Can you check if this change makes any difference
>>>> >> >> performance-wise
>>>> >> >>>>>> in your
>>>> >> >>>>>>>>>> environment? If yes, we can go with runtime code
>>>> generation in
>>>> >> >> the
>>>> >> >>>>>> long
>>>> >> >>>>>>>>>> term: register classes and generate a dynamic message
>>>> factory
>>>> >> >> with
>>>> >> >>>>>> a switch
>>>> >> >>>>>>>>>> statement once all messages are registered (not in 2.9
>>>> though,
>>>> >> >>>>>> obviously).
>>>> >> >>>>>>>>>>
>>>> >> >>>>>>>>>> ср, 9 сент. 2020 г. в 14:53, Alex Plehanov <
>>>> >> >>> plehanov.alex@gmail.com
>>>> >> >>>>>>> :
>>>> >> >>>>>>>>>>
>>>> >> >>>>>>>>>>> Hello guys,
>>>> >> >>>>>>>>>>>
>>>> >> >>>>>>>>>>> I've tried to optimize tracing implementation (ticket
>>>> [1]), it
>>>> >> >>>>>> reduced the
>>>> >> >>>>>>>>>>> drop, but not completely removed it.
>>>> >> >>>>>>>>>>> Ivan Rakov, Alexander Lapin, can you please review the
>>>> patch?
>>>> >> >>>>>>>>>>> Ivan Artiukhov, can you please benchmark the patch [2]
>>>> against
>>>> >> >>>>>> 2.8.1
>>>> >> >>>>>>>>>>> release on your environment?
>>>> >> >>>>>>>>>>> With this patch on our environment, it's about a 3% drop
>>>> left,
>>>> >> >>>>>> it's close
>>>> >> >>>>>>>>>>> to measurement error and I think such a drop is not a
>>>> >> >>>>>> showstopper. Guys,
>>>> >> >>>>>>>>>>> WDYT?
>>>> >> >>>>>>>>>>>
>>>> >> >>>>>>>>>>> Also, I found that compatibility is broken for JDBC thin
>>>> >> >> driver
>>>> >> >>>>>> between 2.8
>>>> >> >>>>>>>>>>> and 2.9 versions (ticket [3]). I think it's a blocker and
>>>> >> >> should
>>>> >> >>>>>> be
>>>> >> >>>>>>>>>>> fixed in 2.9. I've prepared the patch.
>>>> >> >>>>>>>>>>> Taras Ledkov, can you please review this patch?
>>>> >> >>>>>>>>>>>
>>>> >> >>>>>>>>>>> And one more ticket I propose to include into 2.9 [4]
>>>> (NIO
>>>> >> >>> message
>>>> >> >>>>>>>>>>> send problem in some circumstances). I will cherry-pick
>>>> it if
>>>> >> >>>>>> there is no
>>>> >> >>>>>>>>>>> objection.
>>>> >> >>>>>>>>>>>
>>>> >> >>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13411
>>>> >> >>>>>>>>>>> [2] https://github.com/apache/ignite/pull/8223
>>>> >> >>>>>>>>>>> [3] https://issues.apache.org/jira/browse/IGNITE-13414
>>>> >> >>>>>>>>>>> [4] https://issues.apache.org/jira/browse/IGNITE-13361
>>>> >> >>>>>>>>>>>
>>>> >> >>>>>>>>>>> пн, 7 сент. 2020 г. в 14:14, Maxim Muzafarov <
>>>> >> >> mmuzaf@apache.org
>>>> >> >>>> :
>>>> >> >>>>>>>>>>>
>>>> >> >>>>>>>>>>>> Alexey,
>>>> >> >>>>>>>>>>>>
>>>> >> >>>>>>>>>>>> I propose to include [1] issue to the 2.9 release. Since
>>>> >> >> this
>>>> >> >>>>>> issue is
>>>> >> >>>>>>>>>>>> related to the new master key change functionality which
>>>> >> >>>>>> haven't been
>>>> >> >>>>>>>>>>>> released yet I think it will be safe to cherry-pick
>>>> commit
>>>> >> >> to
>>>> >> >>>>>> the
>>>> >> >>>>>>>>>>>> release branch.
>>>> >> >>>>>>>>>>>>
>>>> >> >>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13390
>>>> >> >>>>>>>>>>>>
>>>> >> >>>>>>>>>>>> On Tue, 1 Sep 2020 at 12:13, Nikolay Izhikov <
>>>> >> >>>>>> nizhikov@apache.org>
>>>> >> >>>>>>>>>>> wrote:
>>>> >> >>>>>>>>>>>>>
>>>> >> >>>>>>>>>>>>> Hello, Igniters.
>>>> >> >>>>>>>>>>>>>
>>>> >> >>>>>>>>>>>>> Alexey, please, include one more Python thin client fix
>>>> >> >> [1]
>>>> >> >>>>>> into the
>>>> >> >>>>>>>>>>> 2.9
>>>> >> >>>>>>>>>>>> release
>>>> >> >>>>>>>>>>>>> It fixes kinda major issue - "Python client returns
>>>> fields
>>>> >> >>> in
>>>> >> >>>>>> wrong
>>>> >> >>>>>>>>>>>> order since the 2 row when fields_count>10"
>>>> >> >>>>>>>>>>>>>
>>>> >> >>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-12809
>>>> >> >>>>>>>>>>>>> [2]
>>>> >> >>>>>>>>>>>>
>>>> >> >>>>>>>>>>>
>>>> >> >>>>>>
>>>> >> >>>
>>>> >> >>
>>>> >>
>>>> https://github.com/apache/ignite/commit/38025ee4167f05eaa2d6a2c5c2ab70c83a462cfc
>>>> >> >>>>>>>>>>>>>
>>>> >> >>>>>>>>>>>>>> 31 авг. 2020 г., в 19:23, Alexey Goncharuk <
>>>> >> >>>>>>>>>>> alexey.goncharuk@gmail.com>
>>>> >> >>>>>>>>>>>> написал(а):
>>>> >> >>>>>>>>>>>>>>
>>>> >> >>>>>>>>>>>>>> Alexey, thanks, got it. I am not sure we can optimize
>>>> >> >>>>>> anything out of
>>>> >> >>>>>>>>>>>> the
>>>> >> >>>>>>>>>>>>>> message factory with suppliers (at least I have no
>>>> ideas
>>>> >> >>>>>> right now),
>>>> >> >>>>>>>>>>> so
>>>> >> >>>>>>>>>>>>>> most likely the only move here is to switch back to
>>>> the
>>>> >> >>>>>> switch
>>>> >> >>>>>>>>>>> approach
>>>> >> >>>>>>>>>>>>>> somehow preserving the metrics part. Probably,
>>>> inlining
>>>> >> >>> the
>>>> >> >>>>>> Ignite
>>>> >> >>>>>>>>>>>> messages
>>>> >> >>>>>>>>>>>>>> to the IgniteMessageFactoryImpl should do the trick.
>>>> Let
>>>> >> >>> me
>>>> >> >>>>>> explore
>>>> >> >>>>>>>>>>> the
>>>> >> >>>>>>>>>>>>>> code a bit.
>>>> >> >>>>>>>>>>>>>>
>>>> >> >>>>>>>>>>>>>> P.S. I am surprised by the impact this part makes for
>>>> >> >> the
>>>> >> >>>>>>>>>>> performance.
>>>> >> >>>>>>>>>>>>>> Message creation is indeed on the hot path, but a
>>>> single
>>>> >> >>>>>> virtual call
>>>> >> >>>>>>>>>>>>>> should not make that much of a difference given the
>>>> >> >> amount
>>>> >> >>>>>> of other
>>>> >> >>>>>>>>>>>> work
>>>> >> >>>>>>>>>>>>>> happening during the message processing.
>>>> >> >>>>>>>>>>>>>>
>>>> >> >>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 18:33, Alex Plehanov <
>>>> >> >>>>>> plehanov.alex@gmail.com
>>>> >> >>>>>>>>>>>> :
>>>> >> >>>>>>>>>>>>>>
>>>> >> >>>>>>>>>>>>>>> Alexey, sorry, I wrongly interpreted our benchmark
>>>> >> >>> results.
>>>> >> >>>>>>>>>>> Actually,
>>>> >> >>>>>>>>>>>> we
>>>> >> >>>>>>>>>>>>>>> were looking for a drop using bi-sect in the range
>>>> >> >>> between
>>>> >> >>>>>> e6a7f93
>>>> >> >>>>>>>>>>>> (first
>>>> >> >>>>>>>>>>>>>>> commit in the 2.9 branch after 2.8 branch cut) and
>>>> >> >>>>>> 6592dfa5 (last
>>>> >> >>>>>>>>>>>> commit in
>>>> >> >>>>>>>>>>>>>>> the 2.9 branch). And we found these two problematic
>>>> >> >>>>>> commits.
>>>> >> >>>>>>>>>>>>>>>
>>>> >> >>>>>>>>>>>>>>> Perhaps only IGNITE-13060 (Tracing) is responsible
>>>> for
>>>> >> >> a
>>>> >> >>>>>> drop
>>>> >> >>>>>>>>>>> between
>>>> >> >>>>>>>>>>>>>>> 2.8.1 and 2.9 (we have benchmarked 2.8.1 vs 2.9 with
>>>> >> >>>>>> reverted
>>>> >> >>>>>>>>>>>> IGNITE-13060
>>>> >> >>>>>>>>>>>>>>> now and performance looks the same)
>>>> >> >>>>>>>>>>>>>>>
>>>> >> >>>>>>>>>>>>>>> Ticket IGNITE-12568 (MessageFactory refactoring) is
>>>> not
>>>> >> >>>>>> related to
>>>> >> >>>>>>>>>>>> drop
>>>> >> >>>>>>>>>>>>>>> between 2.8.1 and 2.9, but still has some performance
>>>> >> >>>>>> problem, and
>>>> >> >>>>>>>>>>> we
>>>> >> >>>>>>>>>>>> can
>>>> >> >>>>>>>>>>>>>>> win back IGNITE-13060 drop by this ticket.
>>>> >> >>>>>>>>>>>>>>>
>>>> >> >>>>>>>>>>>>>>> Do we need more investigation on IGNITE-13060 or we
>>>> >> >> leave
>>>> >> >>>>>> it as is?
>>>> >> >>>>>>>>>>>>>>>
>>>> >> >>>>>>>>>>>>>>> What should we do with IGNITE-12568 (MessageFactory
>>>> >> >>>>>> refactoring)?
>>>> >> >>>>>>>>>>>>>>>
>>>> >> >>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 13:25, Alexey Goncharuk <
>>>> >> >>>>>>>>>>>> alexey.goncharuk@gmail.com
>>>> >> >>>>>>>>>>>>>>>> :
>>>> >> >>>>>>>>>>>>>>>
>>>> >> >>>>>>>>>>>>>>>> Alexey,
>>>> >> >>>>>>>>>>>>>>>>
>>>> >> >>>>>>>>>>>>>>>> While investigating, I found that IGNITE-12568 has
>>>> an
>>>> >> >>>>>> incorrect fix
>>>> >> >>>>>>>>>>>>>>>> version and is actually present in ignite-2.8.1
>>>> branch
>>>> >> >>>>>> [1], so it
>>>> >> >>>>>>>>>>>> cannot be
>>>> >> >>>>>>>>>>>>>>>> the source of the drop against 2.8.1.
>>>> >> >>>>>>>>>>>>>>>>
>>>> >> >>>>>>>>>>>>>>>> P.S. Looks like we need to enforce a more accurate
>>>> >> >> work
>>>> >> >>>>>> with fix
>>>> >> >>>>>>>>>>>> versions
>>>> >> >>>>>>>>>>>>>>>> or develop some sort of tooling to verify the fix
>>>> >> >>>>>> versions.
>>>> >> >>>>>>>>>>>>>>>>
>>>> >> >>>>>>>>>>>>>>>> --AG
>>>> >> >>>>>>>>>>>>>>>>
>>>> >> >>>>>>>>>>>>>>>> [1]
>>>> >> >>>>>>>>>>>>>>>>
>>>> >> >>>>>>>>>>>>
>>>> >> >>>>>>>>>>>
>>>> >> >>>>>>
>>>> >> >>>
>>>> >> >>
>>>> >>
>>>> https://github.com/apache/ignite/commit/3e492bd23851856bbd0385c6a419892d0bba2a34
>>>> >> >>>>>>>>>>>>>>>>
>>>> >> >>>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 12:42, Alexey Goncharuk <
>>>> >> >>>>>>>>>>>> alexey.goncharuk@gmail.com
>>>> >> >>>>>>>>>>>>>>>>> :
>>>> >> >>>>>>>>>>>>>>>>
>>>> >> >>>>>>>>>>>>>>>>> пт, 28 авг. 2020 г. в 11:16, Alex Plehanov <
>>>> >> >>>>>>>>>>> plehanov.alex@gmail.com
>>>> >> >>>>>>>>>>>>> :
>>>> >> >>>>>>>>>>>>>>>>>
>>>> >> >>>>>>>>>>>>>>>>>> Guys,
>>>> >> >>>>>>>>>>>>>>>>>>
>>>> >> >>>>>>>>>>>>>>>>>> We have benchmarked 2.9 without IGNITE-13060 and
>>>> >> >>>>>> IGNITE-12568
>>>> >> >>>>>>>>>>>> (reverted
>>>> >> >>>>>>>>>>>>>>>>>> it
>>>> >> >>>>>>>>>>>>>>>>>> locally) and got the same performance as on 2.8.1
>>>> >> >>>>>>>>>>>>>>>>>>
>>>> >> >>>>>>>>>>>>>>>>>> IGNITE-13060 (Tracing) - some code was added to
>>>> hot
>>>> >> >>>>>> paths, to
>>>> >> >>>>>>>>>>> trace
>>>> >> >>>>>>>>>>>>>>>>>> these
>>>> >> >>>>>>>>>>>>>>>>>> hot paths, it's clear why we have performance drop
>>>> >> >>> here.
>>>> >> >>>>>>>>>>>>>>>>>>
>>>> >> >>>>>>>>>>>>>>>>>> IGNITE-12568 (MessageFactory refactoring) -
>>>> >> >>> switch/case
>>>> >> >>>>>> block was
>>>> >> >>>>>>>>>>>>>>>>>> refactored to an array of message suppliers. The
>>>> >> >>>>>> message factory
>>>> >> >>>>>>>>>>>> is on
>>>> >> >>>>>>>>>>>>>>>>>> the
>>>> >> >>>>>>>>>>>>>>>>>> hot path, which explains why this commit has an
>>>> >> >> impact
>>>> >> >>>>>> on total
>>>> >> >>>>>>>>>>>>>>>>>> performance.
>>>> >> >>>>>>>>>>>>>>>>>> I've checked JIT assembly output, done some JMH
>>>> >> >>>>>> microbenchmarks,
>>>> >> >>>>>>>>>>>> and
>>>> >> >>>>>>>>>>>>>>>>>> found
>>>> >> >>>>>>>>>>>>>>>>>> that old implementation of MessageFactory.create()
>>>> >> >>>>>> about 30-35%
>>>> >> >>>>>>>>>>>> faster
>>>> >> >>>>>>>>>>>>>>>>>> than
>>>> >> >>>>>>>>>>>>>>>>>> the new one. The reason - approach with
>>>> switch/case
>>>> >> >>> can
>>>> >> >>>>>>>>>>> effectively
>>>> >> >>>>>>>>>>>>>>>>>> inline
>>>> >> >>>>>>>>>>>>>>>>>> message creation code, but with an array of
>>>> >> >> suppliers
>>>> >> >>>>>> relatively
>>>> >> >>>>>>>>>>>> heavy
>>>> >> >>>>>>>>>>>>>>>>>> "invokeinterface" cannot be skipped. I've tried to
>>>> >> >>>>>> rewrite the
>>>> >> >>>>>>>>>>> code
>>>> >> >>>>>>>>>>>>>>>>>> using
>>>> >> >>>>>>>>>>>>>>>>>> an abstract class for suppliers instead of an
>>>> >> >>> interface
>>>> >> >>>>>> (to
>>>> >> >>>>>>>>>>>>>>>>>> replace "invokeinterface" with the
>>>> "invokevirtual"),
>>>> >> >>>>>> but it gives
>>>> >> >>>>>>>>>>>> back
>>>> >> >>>>>>>>>>>>>>>>>> only
>>>> >> >>>>>>>>>>>>>>>>>> 10% of method performance and in this case, code
>>>> >> >> looks
>>>> >> >>>>>> ugly
>>>> >> >>>>>>>>>>>> (lambdas
>>>> >> >>>>>>>>>>>>>>>>>> can't
>>>> >> >>>>>>>>>>>>>>>>>> be used). Currently, I can't find any more ways to
>>>> >> >>>>>> optimize the
>>>> >> >>>>>>>>>>>> current
>>>> >> >>>>>>>>>>>>>>>>>> approach (except return to the switch/case block).
>>>> >> >>>>>> Andrey Gura,
>>>> >> >>>>>>>>>>> as
>>>> >> >>>>>>>>>>>> the
>>>> >> >>>>>>>>>>>>>>>>>> author of IGNITE-12568, maybe you have some ideas
>>>> >> >>> about
>>>> >> >>>>>>>>>>>> optimization?
>>>> >> >>>>>>>>>>>>>>>>>>
>>>> >> >>>>>>>>>>>>>>>>>> Perhaps we should revert IGNITE-12568, but there
>>>> are
>>>> >> >>>>>> some metrics
>>>> >> >>>>>>>>>>>>>>>>>> already
>>>> >> >>>>>>>>>>>>>>>>>> created, which can't be rewritten using old
>>>> message
>>>> >> >>>>>> factory
>>>> >> >>>>>>>>>>>>>>>>>> implementation
>>>> >> >>>>>>>>>>>>>>>>>> (IGNITE-12756). Guys, WDYT?
>>>> >> >>>>>>>>>>>>>>>>>>
>>>> >> >>>>>>>>>>>>>>>>>
>>>> >> >>>>>>>>>>>>>>>>> Alexey,
>>>> >> >>>>>>>>>>>>>>>>>
>>>> >> >>>>>>>>>>>>>>>>> I see that IGNITE-12756 (metrics improvements) is
>>>> >> >>>>>> already released
>>>> >> >>>>>>>>>>>> in
>>>> >> >>>>>>>>>>>>>>>>> Ignite 2.8.1 while IGNITE-12568 (message factory)
>>>> is
>>>> >> >>>>>> only present
>>>> >> >>>>>>>>>>>> in Ignite
>>>> >> >>>>>>>>>>>>>>>>> 2.9. Let's revert both IGNITE-12568 and whichever
>>>> new
>>>> >> >>>>>> metrics
>>>> >> >>>>>>>>>>>> created for
>>>> >> >>>>>>>>>>>>>>>>> 2.9 that depend on the new message factory to
>>>> unblock
>>>> >> >>>>>> the release
>>>> >> >>>>>>>>>>>> and deal
>>>> >> >>>>>>>>>>>>>>>>> with the optimizations in 2.10?
>>>> >> >>>>>>>>>>>>>>>>>
>>>> >> >>>>>>>>>>>>>>>>
>>>> >> >>>>>>>>>>>>>
>>>> >> >>>>>>>>>>>>
>>>> >> >>>>>>>>>>>
>>>> >> >>>>>>
>>>> >> >>>>>
>>>> >> >>>
>>>> >> >>
>>>> >>
>>>> >>
>>>>
>>>
Re: Apache Ignite 2.9.0 RELEASE [Time, Scope, Manager]
Posted by Denis Magda <dm...@apache.org>.
Is it possible/reasonable to run a single vote for Ignite 2.9 and all the
migrated extensions? This time. @Alex Plehanov <pl...@gmail.com>,
what do you think?
If we decide to release the versions 1.0.0 of the extensions after 2.9 is
out then it can last for a week or so...because we need to build, update
docs, vote, publish.
-
Denis
On Mon, Oct 12, 2020 at 7:19 PM Saikat Maitra <sa...@gmail.com>
wrote:
> Hi Denis,
>
> All the modules except OSGi have been migrated. Alexey is creating a
> separate ticket for OSGi and we can migrate the same as well.
>
> My thoughts are Apache Ignite 2.9.0 and migrated extensions 1.0.0 to be
> released together. I am thinking depending on the voting process and
> testing cycles we can release the extensions 1.0.0 together with Apache
> Ignite 2.9.0 or right after within days of Apache Ignite 2.9.0 release.
>
>
> Regards,
> Saikat
>
> On Mon, Oct 12, 2020 at 2:58 PM Denis Magda <dm...@apache.org> wrote:
>
>> Saikat, Alex,
>>
>> Based on your inputs here, it sounds like once Ignite 2.9 gets released,
>> all the integrations that made their way to the extensions repo [1] will
>> get stuck in limbo for some time. What I mean here:
>>
>> 1. While the users will be bumping up their ignite-core,
>> ignite-indexing, etc. versions to 2.9, they won't find a 2.9 artifact for
>> Kafka, Camel and other extensions
>> 2. Also, the users won't find 1.0 versions of those extensions that
>> also need to be released
>>
>> So, it's unclear how those users are supposed to upgrade to Ignite 2.9 if
>> they use any of the extensions. Is it safe to use a 2.8 version of an
>> extension? Or, should we release the 1.0 versions of all the migrated
>> extensions together with 2.9?
>>
>>
>> [1] https://github.com/apache/ignite-extensions/tree/master/modules
>>
>>
>> -
>> Denis
>>
>>
>> On Tue, Oct 6, 2020 at 7:20 AM Alexey Goncharuk <
>> alexey.goncharuk@gmail.com> wrote:
>>
>>> Saikat,
>>>
>>> The plan sounds good to me! Do you have an idea for the timeline of the
>>> module releases? Let me know if I can help in any way.
>>>
>>> Also, I was looking into the modularization IEP and noticed that OSGI
>>> module is discussed to be moved to the extensions, but there is no
>>> corresponding ticket. Should I create one? I will be happy to help with
>>> moving this module to extensions.
>>>
>>> вт, 29 сент. 2020 г. в 03:03, Saikat Maitra <sa...@gmail.com>:
>>>
>>> > Hi Alexey,
>>> >
>>> > All the modules as planned in IEP-36
>>> >
>>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36:+Modularization
>>> > are migrated to ignite-extensions repository.
>>> >
>>> > We can definitely plan to release ignite-extensions modules.
>>> >
>>> > I have a pending PR related to the kafka module and the PR is in review
>>> > https://github.com/apache/ignite/pull/8222 but its corresponding PR
>>> has
>>> > been merged in ignite-extensions repository.
>>> >
>>> > My thoughts / action items for the migration efforts and releases were
>>> > as follows:
>>> >
>>> > 1. Merge the changes for https://github.com/apache/ignite/pull/8222
>>> > 2. Fix the upload nightly packages task for ignite-core to help fix the
>>> > travis build failure in ignite-extensions repository.
>>> > 3. Review and update the README.md files for the ignite-extensions
>>> modules
>>> > as required.
>>> > 4. Update the docs for ignite-extensions modules in ignite-website.
>>> > 5. Release each module separately and share updates.
>>> >
>>> > Please let me know if you have feedback.
>>> >
>>> > Regards,
>>> > Saikat
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > On Mon, Sep 28, 2020 at 7:36 AM Petr Ivanov <mr...@gmail.com>
>>> wrote:
>>> >
>>> >> It seems that gitbox.apache.org is not available on CI/CD.
>>> >>
>>> >> I will try to update VCS to GitHub.
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> > On 28 Sep 2020, at 14:07, Alex Plehanov <pl...@gmail.com>
>>> >> wrote:
>>> >> >
>>> >> > Guys,
>>> >> >
>>> >> > I've tried to build release candidate using TC [1]. But:
>>> >> > 1. I can't choose a branch in this TC task (there is no such field).
>>> >> > 2. On the "default" branch I got an error: Failed to collect
>>> changes,
>>> >> > error: List remote refs failed:
>>> >> > org.apache.http.conn.ConnectTimeoutException: Connect to
>>> >> > gitbox.apache.org:443 [gitbox.apache.org/52.202.80.70] failed:
>>> connect
>>> >> > timed out, VCS root: "GitBox [asf/ignite]" {instance id=300, parent
>>> >> > internal id=74, parent id=GitBoxAsfIgnite, description: "
>>> >> > https://gitbox.apache.org/repos/asf/ignite.git#refs/heads/master"}
>>> >> >
>>> >> > Is there some problem with this TC task? What am I doing wrong?
>>> >> >
>>> >> > [1] :
>>> >> >
>>> >>
>>> https://ci.ignite.apache.org/viewType.html?buildTypeId=Releases_ApacheIgniteMain_ReleaseBuild
>>> >> >
>>> >> > пн, 28 сент. 2020 г. в 13:15, Alexey Goncharuk <
>>> >> alexey.goncharuk@gmail.com>:
>>> >> >
>>> >> >> Saikat, Nikolay,
>>> >> >>
>>> >> >> We have migrated a bunch of modules to ignite-extensions recently.
>>> >> Given
>>> >> >> that these modules will not be available in Ignite 2.9 anymore
>>> (will
>>> >> >> they?), should we also plan to release the extensions?
>>> >> >>
>>> >> >> ср, 23 сент. 2020 г. в 21:49, Alex Plehanov <
>>> plehanov.alex@gmail.com>:
>>> >> >>
>>> >> >>> Igniters,
>>> >> >>>
>>> >> >>> I've prepared release notes for the 2.9 release [1]. Can anyone
>>> review
>>> >> >> it,
>>> >> >>> please?
>>> >> >>>
>>> >> >>> [1]: https://github.com/apache/ignite/pull/8273
>>> >> >>>
>>> >> >>> вт, 22 сент. 2020 г. в 09:39, Alex Plehanov <
>>> plehanov.alex@gmail.com
>>> >> >:
>>> >> >>>
>>> >> >>>> Guys,
>>> >> >>>>
>>> >> >>>> I've filled the ticket with reproducer [1] for the discovery bug.
>>> >> This
>>> >> >>> bug
>>> >> >>>> caused by [2] ticket. We discussed with Vladimir Steshin
>>> privately
>>> >> and
>>> >> >>>> decided to revert this ticket. I will do it today (after TC bot
>>> visa)
>>> >> >> if
>>> >> >>>> there are no objections.
>>> >> >>>>
>>> >> >>>> [1]: https://issues.apache.org/jira/browse/IGNITE-13465
>>> >> >>>> [2]: https://issues.apache.org/jira/browse/IGNITE-13134
>>> >> >>>>
>>> >> >>>> пн, 21 сент. 2020 г. в 11:08, Alex Plehanov <
>>> plehanov.alex@gmail.com
>>> >> >:
>>> >> >>>>
>>> >> >>>>> Guys,
>>> >> >>>>>
>>> >> >>>>> During internal testing, we've found a critical bug with
>>> >> >>>>> discovery (cluster falls apart if two nodes segmented
>>> sequentially).
>>> >> >>> This
>>> >> >>>>> problem is not reproduced in 2.8.1. I think we should fix it
>>> >> >>>>> before release. Under investigation now. I'll let you know when
>>> we
>>> >> get
>>> >> >>>>> something.
>>> >> >>>>>
>>> >> >>>>> чт, 17 сент. 2020 г. в 00:51, Andrey Gura <ag...@apache.org>:
>>> >> >>>>>
>>> >> >>>>>>> So what do you think? Should we proceed with a 'hacked'
>>> version of
>>> >> >>> the
>>> >> >>>>>> message factory in 2.9 and go for the runtime message
>>> generation in
>>> >> >>> later
>>> >> >>>>>> release, or keep the code clean and fix the regression in the
>>> next
>>> >> >>> releases?
>>> >> >>>>>>> Andrey, can you take a look at my change? I think it is fairly
>>> >> >>>>>> straightforward and does not change the semantics, just skips
>>> the
>>> >> >>> factory
>>> >> >>>>>> closures for certain messages.
>>> >> >>>>>>
>>> >> >>>>>> IMHO 2.5% isn't too much especially because it isn't actual
>>> for all
>>> >> >>>>>> workloads (I didn't get any significant drops during
>>> benchmarking).
>>> >> >> So
>>> >> >>>>>> I prefer the runtime generation in later releases.
>>> >> >>>>>>
>>> >> >>>>>> On Mon, Sep 14, 2020 at 12:41 PM Alexey Goncharuk
>>> >> >>>>>> <al...@gmail.com> wrote:
>>> >> >>>>>>>
>>> >> >>>>>>> Alexey, Andrey, Igniters,
>>> >> >>>>>>>
>>> >> >>>>>>> So what do you think? Should we proceed with a 'hacked'
>>> version of
>>> >> >>> the
>>> >> >>>>>> message factory in 2.9 and go for the runtime message
>>> generation in
>>> >> >>> later
>>> >> >>>>>> release, or keep the code clean and fix the regression in the
>>> next
>>> >> >>> releases?
>>> >> >>>>>>> Andrey, can you take a look at my change? I think it is fairly
>>> >> >>>>>> straightforward and does not change the semantics, just skips
>>> the
>>> >> >>> factory
>>> >> >>>>>> closures for certain messages.
>>> >> >>>>>>>
>>> >> >>>>>>> Personally, I would prefer fixing the regression given that we
>>> >> also
>>> >> >>>>>> introduced tracing in this release.
>>> >> >>>>>>>
>>> >> >>>>>>>
>>> >> >>>>>>>
>>> >> >>>>>>> пт, 11 сент. 2020 г. в 12:09, Alex Plehanov <
>>> >> >> plehanov.alex@gmail.com
>>> >> >>>> :
>>> >> >>>>>>>>
>>> >> >>>>>>>> Alexey,
>>> >> >>>>>>>>
>>> >> >>>>>>>> We've benchmarked by yardstick commits 130376741bf vs
>>> ed52559eb95
>>> >> >>> and
>>> >> >>>>>> the performance of ed52559eb95 is better for about 2.5% on
>>> average
>>> >> on
>>> >> >>> our
>>> >> >>>>>> environment (about the same results we got benchmarking
>>> 65c30ec6 vs
>>> >> >>>>>> 0606f03d). We've made 24 runs for each commit of
>>> >> >>>>>> IgnitePutTxImplicitBenchmark (we got maximum drop for 2.9 on
>>> this
>>> >> >>>>>> benchmark), 200 seconds warmup, 300 seconds benchmark, 6
>>> servers, 5
>>> >> >>> clients
>>> >> >>>>>> 50 threads each. Yardstick results for this configuration:
>>> >> >>>>>>>> Commit 130376741bf: avg TPS=164096, avg latency=9173464 ns
>>> >> >>>>>>>> Commit ed52559eb95: avg TPS=168283, avg latency=8945908 ns
>>> >> >>>>>>>>
>>> >> >>>>>>>> пт, 11 сент. 2020 г. в 09:51, Artem Budnikov <
>>> >> >>>>>> a.budnikov.ignite@gmail.com>:
>>> >> >>>>>>>>>
>>> >> >>>>>>>>> Hi Everyone,
>>> >> >>>>>>>>>
>>> >> >>>>>>>>> I posted an instruction on how to publish the docs on
>>> >> >>>>>> ignite.apache.org/docs [1]. When you finish with Ignite 2.9,
>>> you
>>> >> can
>>> >> >>>>>> update the docs by following the instruction. Unfortunately, I
>>> >> won't
>>> >> >> be
>>> >> >>>>>> able to spend any time on this project any longer. You can send
>>> >> your
>>> >> >>> pull
>>> >> >>>>>> requests and questions about the documentation to Denis Magda.
>>> >> >>>>>>>>>
>>> >> >>>>>>>>> -Artem
>>> >> >>>>>>>>>
>>> >> >>>>>>>>> [1] :
>>> >> >>>>>>
>>> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Document
>>> >> >>>>>>>>>
>>> >> >>>>>>>>> On Thu, Sep 10, 2020 at 2:45 PM Alexey Goncharuk <
>>> >> >>>>>> alexey.goncharuk@gmail.com> wrote:
>>> >> >>>>>>>>>>
>>> >> >>>>>>>>>> Alexey,
>>> >> >>>>>>>>>>
>>> >> >>>>>>>>>> I've tried to play with message factories locally, but
>>> >> >>>>>> unfortunately, I
>>> >> >>>>>>>>>> cannot spot the difference between old and new
>>> implementation
>>> >> in
>>> >> >>>>>>>>>> distributed benchmarks. I pushed an implementation of
>>> >> >>>>>> MessageFactoryImpl
>>> >> >>>>>>>>>> with the old switch statement to the
>>> ignite-2.9-revert-12568
>>> >> >>> branch
>>> >> >>>>>>>>>> (discussed this with Andrey Gura, the change should be
>>> >> >> compatible
>>> >> >>>>>> with the
>>> >> >>>>>>>>>> new metrics as we still use the register() mechanics).
>>> >> >>>>>>>>>>
>>> >> >>>>>>>>>> Can you check if this change makes any difference
>>> >> >> performance-wise
>>> >> >>>>>> in your
>>> >> >>>>>>>>>> environment? If yes, we can go with runtime code
>>> generation in
>>> >> >> the
>>> >> >>>>>> long
>>> >> >>>>>>>>>> term: register classes and generate a dynamic message
>>> factory
>>> >> >> with
>>> >> >>>>>> a switch
>>> >> >>>>>>>>>> statement once all messages are registered (not in 2.9
>>> though,
>>> >> >>>>>> obviously).
>>> >> >>>>>>>>>>
>>> >> >>>>>>>>>> ср, 9 сент. 2020 г. в 14:53, Alex Plehanov <
>>> >> >>> plehanov.alex@gmail.com
>>> >> >>>>>>> :
>>> >> >>>>>>>>>>
>>> >> >>>>>>>>>>> Hello guys,
>>> >> >>>>>>>>>>>
>>> >> >>>>>>>>>>> I've tried to optimize tracing implementation (ticket
>>> [1]), it
>>> >> >>>>>> reduced the
>>> >> >>>>>>>>>>> drop, but not completely removed it.
>>> >> >>>>>>>>>>> Ivan Rakov, Alexander Lapin, can you please review the
>>> patch?
>>> >> >>>>>>>>>>> Ivan Artiukhov, can you please benchmark the patch [2]
>>> against
>>> >> >>>>>> 2.8.1
>>> >> >>>>>>>>>>> release on your environment?
>>> >> >>>>>>>>>>> With this patch on our environment, it's about a 3% drop
>>> left,
>>> >> >>>>>> it's close
>>> >> >>>>>>>>>>> to measurement error and I think such a drop is not a
>>> >> >>>>>> showstopper. Guys,
>>> >> >>>>>>>>>>> WDYT?
>>> >> >>>>>>>>>>>
>>> >> >>>>>>>>>>> Also, I found that compatibility is broken for JDBC thin
>>> >> >> driver
>>> >> >>>>>> between 2.8
>>> >> >>>>>>>>>>> and 2.9 versions (ticket [3]). I think it's a blocker and
>>> >> >> should
>>> >> >>>>>> be
>>> >> >>>>>>>>>>> fixed in 2.9. I've prepared the patch.
>>> >> >>>>>>>>>>> Taras Ledkov, can you please review this patch?
>>> >> >>>>>>>>>>>
>>> >> >>>>>>>>>>> And one more ticket I propose to include into 2.9 [4] (NIO
>>> >> >>> message
>>> >> >>>>>>>>>>> send problem in some circumstances). I will cherry-pick
>>> it if
>>> >> >>>>>> there is no
>>> >> >>>>>>>>>>> objection.
>>> >> >>>>>>>>>>>
>>> >> >>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13411
>>> >> >>>>>>>>>>> [2] https://github.com/apache/ignite/pull/8223
>>> >> >>>>>>>>>>> [3] https://issues.apache.org/jira/browse/IGNITE-13414
>>> >> >>>>>>>>>>> [4] https://issues.apache.org/jira/browse/IGNITE-13361
>>> >> >>>>>>>>>>>
>>> >> >>>>>>>>>>> пн, 7 сент. 2020 г. в 14:14, Maxim Muzafarov <
>>> >> >> mmuzaf@apache.org
>>> >> >>>> :
>>> >> >>>>>>>>>>>
>>> >> >>>>>>>>>>>> Alexey,
>>> >> >>>>>>>>>>>>
>>> >> >>>>>>>>>>>> I propose to include [1] issue to the 2.9 release. Since
>>> >> >> this
>>> >> >>>>>> issue is
>>> >> >>>>>>>>>>>> related to the new master key change functionality which
>>> >> >>>>>> haven't been
>>> >> >>>>>>>>>>>> released yet I think it will be safe to cherry-pick
>>> commit
>>> >> >> to
>>> >> >>>>>> the
>>> >> >>>>>>>>>>>> release branch.
>>> >> >>>>>>>>>>>>
>>> >> >>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13390
>>> >> >>>>>>>>>>>>
>>> >> >>>>>>>>>>>> On Tue, 1 Sep 2020 at 12:13, Nikolay Izhikov <
>>> >> >>>>>> nizhikov@apache.org>
>>> >> >>>>>>>>>>> wrote:
>>> >> >>>>>>>>>>>>>
>>> >> >>>>>>>>>>>>> Hello, Igniters.
>>> >> >>>>>>>>>>>>>
>>> >> >>>>>>>>>>>>> Alexey, please, include one more Python thin client fix
>>> >> >> [1]
>>> >> >>>>>> into the
>>> >> >>>>>>>>>>> 2.9
>>> >> >>>>>>>>>>>> release
>>> >> >>>>>>>>>>>>> It fixes kinda major issue - "Python client returns
>>> fields
>>> >> >>> in
>>> >> >>>>>> wrong
>>> >> >>>>>>>>>>>> order since the 2 row when fields_count>10"
>>> >> >>>>>>>>>>>>>
>>> >> >>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-12809
>>> >> >>>>>>>>>>>>> [2]
>>> >> >>>>>>>>>>>>
>>> >> >>>>>>>>>>>
>>> >> >>>>>>
>>> >> >>>
>>> >> >>
>>> >>
>>> https://github.com/apache/ignite/commit/38025ee4167f05eaa2d6a2c5c2ab70c83a462cfc
>>> >> >>>>>>>>>>>>>
>>> >> >>>>>>>>>>>>>> 31 авг. 2020 г., в 19:23, Alexey Goncharuk <
>>> >> >>>>>>>>>>> alexey.goncharuk@gmail.com>
>>> >> >>>>>>>>>>>> написал(а):
>>> >> >>>>>>>>>>>>>>
>>> >> >>>>>>>>>>>>>> Alexey, thanks, got it. I am not sure we can optimize
>>> >> >>>>>> anything out of
>>> >> >>>>>>>>>>>> the
>>> >> >>>>>>>>>>>>>> message factory with suppliers (at least I have no
>>> ideas
>>> >> >>>>>> right now),
>>> >> >>>>>>>>>>> so
>>> >> >>>>>>>>>>>>>> most likely the only move here is to switch back to the
>>> >> >>>>>> switch
>>> >> >>>>>>>>>>> approach
>>> >> >>>>>>>>>>>>>> somehow preserving the metrics part. Probably, inlining
>>> >> >>> the
>>> >> >>>>>> Ignite
>>> >> >>>>>>>>>>>> messages
>>> >> >>>>>>>>>>>>>> to the IgniteMessageFactoryImpl should do the trick.
>>> Let
>>> >> >>> me
>>> >> >>>>>> explore
>>> >> >>>>>>>>>>> the
>>> >> >>>>>>>>>>>>>> code a bit.
>>> >> >>>>>>>>>>>>>>
>>> >> >>>>>>>>>>>>>> P.S. I am surprised by the impact this part makes for
>>> >> >> the
>>> >> >>>>>>>>>>> performance.
>>> >> >>>>>>>>>>>>>> Message creation is indeed on the hot path, but a
>>> single
>>> >> >>>>>> virtual call
>>> >> >>>>>>>>>>>>>> should not make that much of a difference given the
>>> >> >> amount
>>> >> >>>>>> of other
>>> >> >>>>>>>>>>>> work
>>> >> >>>>>>>>>>>>>> happening during the message processing.
>>> >> >>>>>>>>>>>>>>
>>> >> >>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 18:33, Alex Plehanov <
>>> >> >>>>>> plehanov.alex@gmail.com
>>> >> >>>>>>>>>>>> :
>>> >> >>>>>>>>>>>>>>
>>> >> >>>>>>>>>>>>>>> Alexey, sorry, I wrongly interpreted our benchmark
>>> >> >>> results.
>>> >> >>>>>>>>>>> Actually,
>>> >> >>>>>>>>>>>> we
>>> >> >>>>>>>>>>>>>>> were looking for a drop using bi-sect in the range
>>> >> >>> between
>>> >> >>>>>> e6a7f93
>>> >> >>>>>>>>>>>> (first
>>> >> >>>>>>>>>>>>>>> commit in the 2.9 branch after 2.8 branch cut) and
>>> >> >>>>>> 6592dfa5 (last
>>> >> >>>>>>>>>>>> commit in
>>> >> >>>>>>>>>>>>>>> the 2.9 branch). And we found these two problematic
>>> >> >>>>>> commits.
>>> >> >>>>>>>>>>>>>>>
>>> >> >>>>>>>>>>>>>>> Perhaps only IGNITE-13060 (Tracing) is responsible for
>>> >> >> a
>>> >> >>>>>> drop
>>> >> >>>>>>>>>>> between
>>> >> >>>>>>>>>>>>>>> 2.8.1 and 2.9 (we have benchmarked 2.8.1 vs 2.9 with
>>> >> >>>>>> reverted
>>> >> >>>>>>>>>>>> IGNITE-13060
>>> >> >>>>>>>>>>>>>>> now and performance looks the same)
>>> >> >>>>>>>>>>>>>>>
>>> >> >>>>>>>>>>>>>>> Ticket IGNITE-12568 (MessageFactory refactoring) is
>>> not
>>> >> >>>>>> related to
>>> >> >>>>>>>>>>>> drop
>>> >> >>>>>>>>>>>>>>> between 2.8.1 and 2.9, but still has some performance
>>> >> >>>>>> problem, and
>>> >> >>>>>>>>>>> we
>>> >> >>>>>>>>>>>> can
>>> >> >>>>>>>>>>>>>>> win back IGNITE-13060 drop by this ticket.
>>> >> >>>>>>>>>>>>>>>
>>> >> >>>>>>>>>>>>>>> Do we need more investigation on IGNITE-13060 or we
>>> >> >> leave
>>> >> >>>>>> it as is?
>>> >> >>>>>>>>>>>>>>>
>>> >> >>>>>>>>>>>>>>> What should we do with IGNITE-12568 (MessageFactory
>>> >> >>>>>> refactoring)?
>>> >> >>>>>>>>>>>>>>>
>>> >> >>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 13:25, Alexey Goncharuk <
>>> >> >>>>>>>>>>>> alexey.goncharuk@gmail.com
>>> >> >>>>>>>>>>>>>>>> :
>>> >> >>>>>>>>>>>>>>>
>>> >> >>>>>>>>>>>>>>>> Alexey,
>>> >> >>>>>>>>>>>>>>>>
>>> >> >>>>>>>>>>>>>>>> While investigating, I found that IGNITE-12568 has an
>>> >> >>>>>> incorrect fix
>>> >> >>>>>>>>>>>>>>>> version and is actually present in ignite-2.8.1
>>> branch
>>> >> >>>>>> [1], so it
>>> >> >>>>>>>>>>>> cannot be
>>> >> >>>>>>>>>>>>>>>> the source of the drop against 2.8.1.
>>> >> >>>>>>>>>>>>>>>>
>>> >> >>>>>>>>>>>>>>>> P.S. Looks like we need to enforce a more accurate
>>> >> >> work
>>> >> >>>>>> with fix
>>> >> >>>>>>>>>>>> versions
>>> >> >>>>>>>>>>>>>>>> or develop some sort of tooling to verify the fix
>>> >> >>>>>> versions.
>>> >> >>>>>>>>>>>>>>>>
>>> >> >>>>>>>>>>>>>>>> --AG
>>> >> >>>>>>>>>>>>>>>>
>>> >> >>>>>>>>>>>>>>>> [1]
>>> >> >>>>>>>>>>>>>>>>
>>> >> >>>>>>>>>>>>
>>> >> >>>>>>>>>>>
>>> >> >>>>>>
>>> >> >>>
>>> >> >>
>>> >>
>>> https://github.com/apache/ignite/commit/3e492bd23851856bbd0385c6a419892d0bba2a34
>>> >> >>>>>>>>>>>>>>>>
>>> >> >>>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 12:42, Alexey Goncharuk <
>>> >> >>>>>>>>>>>> alexey.goncharuk@gmail.com
>>> >> >>>>>>>>>>>>>>>>> :
>>> >> >>>>>>>>>>>>>>>>
>>> >> >>>>>>>>>>>>>>>>> пт, 28 авг. 2020 г. в 11:16, Alex Plehanov <
>>> >> >>>>>>>>>>> plehanov.alex@gmail.com
>>> >> >>>>>>>>>>>>> :
>>> >> >>>>>>>>>>>>>>>>>
>>> >> >>>>>>>>>>>>>>>>>> Guys,
>>> >> >>>>>>>>>>>>>>>>>>
>>> >> >>>>>>>>>>>>>>>>>> We have benchmarked 2.9 without IGNITE-13060 and
>>> >> >>>>>> IGNITE-12568
>>> >> >>>>>>>>>>>> (reverted
>>> >> >>>>>>>>>>>>>>>>>> it
>>> >> >>>>>>>>>>>>>>>>>> locally) and got the same performance as on 2.8.1
>>> >> >>>>>>>>>>>>>>>>>>
>>> >> >>>>>>>>>>>>>>>>>> IGNITE-13060 (Tracing) - some code was added to hot
>>> >> >>>>>> paths, to
>>> >> >>>>>>>>>>> trace
>>> >> >>>>>>>>>>>>>>>>>> these
>>> >> >>>>>>>>>>>>>>>>>> hot paths, it's clear why we have performance drop
>>> >> >>> here.
>>> >> >>>>>>>>>>>>>>>>>>
>>> >> >>>>>>>>>>>>>>>>>> IGNITE-12568 (MessageFactory refactoring) -
>>> >> >>> switch/case
>>> >> >>>>>> block was
>>> >> >>>>>>>>>>>>>>>>>> refactored to an array of message suppliers. The
>>> >> >>>>>> message factory
>>> >> >>>>>>>>>>>> is on
>>> >> >>>>>>>>>>>>>>>>>> the
>>> >> >>>>>>>>>>>>>>>>>> hot path, which explains why this commit has an
>>> >> >> impact
>>> >> >>>>>> on total
>>> >> >>>>>>>>>>>>>>>>>> performance.
>>> >> >>>>>>>>>>>>>>>>>> I've checked JIT assembly output, done some JMH
>>> >> >>>>>> microbenchmarks,
>>> >> >>>>>>>>>>>> and
>>> >> >>>>>>>>>>>>>>>>>> found
>>> >> >>>>>>>>>>>>>>>>>> that old implementation of MessageFactory.create()
>>> >> >>>>>> about 30-35%
>>> >> >>>>>>>>>>>> faster
>>> >> >>>>>>>>>>>>>>>>>> than
>>> >> >>>>>>>>>>>>>>>>>> the new one. The reason - approach with switch/case
>>> >> >>> can
>>> >> >>>>>>>>>>> effectively
>>> >> >>>>>>>>>>>>>>>>>> inline
>>> >> >>>>>>>>>>>>>>>>>> message creation code, but with an array of
>>> >> >> suppliers
>>> >> >>>>>> relatively
>>> >> >>>>>>>>>>>> heavy
>>> >> >>>>>>>>>>>>>>>>>> "invokeinterface" cannot be skipped. I've tried to
>>> >> >>>>>> rewrite the
>>> >> >>>>>>>>>>> code
>>> >> >>>>>>>>>>>>>>>>>> using
>>> >> >>>>>>>>>>>>>>>>>> an abstract class for suppliers instead of an
>>> >> >>> interface
>>> >> >>>>>> (to
>>> >> >>>>>>>>>>>>>>>>>> replace "invokeinterface" with the
>>> "invokevirtual"),
>>> >> >>>>>> but it gives
>>> >> >>>>>>>>>>>> back
>>> >> >>>>>>>>>>>>>>>>>> only
>>> >> >>>>>>>>>>>>>>>>>> 10% of method performance and in this case, code
>>> >> >> looks
>>> >> >>>>>> ugly
>>> >> >>>>>>>>>>>> (lambdas
>>> >> >>>>>>>>>>>>>>>>>> can't
>>> >> >>>>>>>>>>>>>>>>>> be used). Currently, I can't find any more ways to
>>> >> >>>>>> optimize the
>>> >> >>>>>>>>>>>> current
>>> >> >>>>>>>>>>>>>>>>>> approach (except return to the switch/case block).
>>> >> >>>>>> Andrey Gura,
>>> >> >>>>>>>>>>> as
>>> >> >>>>>>>>>>>> the
>>> >> >>>>>>>>>>>>>>>>>> author of IGNITE-12568, maybe you have some ideas
>>> >> >>> about
>>> >> >>>>>>>>>>>> optimization?
>>> >> >>>>>>>>>>>>>>>>>>
>>> >> >>>>>>>>>>>>>>>>>> Perhaps we should revert IGNITE-12568, but there
>>> are
>>> >> >>>>>> some metrics
>>> >> >>>>>>>>>>>>>>>>>> already
>>> >> >>>>>>>>>>>>>>>>>> created, which can't be rewritten using old message
>>> >> >>>>>> factory
>>> >> >>>>>>>>>>>>>>>>>> implementation
>>> >> >>>>>>>>>>>>>>>>>> (IGNITE-12756). Guys, WDYT?
>>> >> >>>>>>>>>>>>>>>>>>
>>> >> >>>>>>>>>>>>>>>>>
>>> >> >>>>>>>>>>>>>>>>> Alexey,
>>> >> >>>>>>>>>>>>>>>>>
>>> >> >>>>>>>>>>>>>>>>> I see that IGNITE-12756 (metrics improvements) is
>>> >> >>>>>> already released
>>> >> >>>>>>>>>>>> in
>>> >> >>>>>>>>>>>>>>>>> Ignite 2.8.1 while IGNITE-12568 (message factory) is
>>> >> >>>>>> only present
>>> >> >>>>>>>>>>>> in Ignite
>>> >> >>>>>>>>>>>>>>>>> 2.9. Let's revert both IGNITE-12568 and whichever
>>> new
>>> >> >>>>>> metrics
>>> >> >>>>>>>>>>>> created for
>>> >> >>>>>>>>>>>>>>>>> 2.9 that depend on the new message factory to
>>> unblock
>>> >> >>>>>> the release
>>> >> >>>>>>>>>>>> and deal
>>> >> >>>>>>>>>>>>>>>>> with the optimizations in 2.10?
>>> >> >>>>>>>>>>>>>>>>>
>>> >> >>>>>>>>>>>>>>>>
>>> >> >>>>>>>>>>>>>
>>> >> >>>>>>>>>>>>
>>> >> >>>>>>>>>>>
>>> >> >>>>>>
>>> >> >>>>>
>>> >> >>>
>>> >> >>
>>> >>
>>> >>
>>>
>>
Re: Apache Ignite 2.9.0 RELEASE [Time, Scope, Manager]
Posted by Saikat Maitra <sa...@gmail.com>.
Hi Denis,
All the modules except OSGi have been migrated. Alexey is creating a
separate ticket for OSGi and we can migrate the same as well.
My thoughts are Apache Ignite 2.9.0 and migrated extensions 1.0.0 to be
released together. I am thinking depending on the voting process and
testing cycles we can release the extensions 1.0.0 together with Apache
Ignite 2.9.0 or right after within days of Apache Ignite 2.9.0 release.
Regards,
Saikat
On Mon, Oct 12, 2020 at 2:58 PM Denis Magda <dm...@apache.org> wrote:
> Saikat, Alex,
>
> Based on your inputs here, it sounds like once Ignite 2.9 gets released,
> all the integrations that made their way to the extensions repo [1] will
> get stuck in limbo for some time. What I mean here:
>
> 1. While the users will be bumping up their ignite-core,
> ignite-indexing, etc. versions to 2.9, they won't find a 2.9 artifact for
> Kafka, Camel and other extensions
> 2. Also, the users won't find 1.0 versions of those extensions that
> also need to be released
>
> So, it's unclear how those users are supposed to upgrade to Ignite 2.9 if
> they use any of the extensions. Is it safe to use a 2.8 version of an
> extension? Or, should we release the 1.0 versions of all the migrated
> extensions together with 2.9?
>
>
> [1] https://github.com/apache/ignite-extensions/tree/master/modules
>
>
> -
> Denis
>
>
> On Tue, Oct 6, 2020 at 7:20 AM Alexey Goncharuk <
> alexey.goncharuk@gmail.com> wrote:
>
>> Saikat,
>>
>> The plan sounds good to me! Do you have an idea for the timeline of the
>> module releases? Let me know if I can help in any way.
>>
>> Also, I was looking into the modularization IEP and noticed that OSGI
>> module is discussed to be moved to the extensions, but there is no
>> corresponding ticket. Should I create one? I will be happy to help with
>> moving this module to extensions.
>>
>> вт, 29 сент. 2020 г. в 03:03, Saikat Maitra <sa...@gmail.com>:
>>
>> > Hi Alexey,
>> >
>> > All the modules as planned in IEP-36
>> >
>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36:+Modularization
>> > are migrated to ignite-extensions repository.
>> >
>> > We can definitely plan to release ignite-extensions modules.
>> >
>> > I have a pending PR related to the kafka module and the PR is in review
>> > https://github.com/apache/ignite/pull/8222 but its corresponding PR has
>> > been merged in ignite-extensions repository.
>> >
>> > My thoughts / action items for the migration efforts and releases were
>> > as follows:
>> >
>> > 1. Merge the changes for https://github.com/apache/ignite/pull/8222
>> > 2. Fix the upload nightly packages task for ignite-core to help fix the
>> > travis build failure in ignite-extensions repository.
>> > 3. Review and update the README.md files for the ignite-extensions
>> modules
>> > as required.
>> > 4. Update the docs for ignite-extensions modules in ignite-website.
>> > 5. Release each module separately and share updates.
>> >
>> > Please let me know if you have feedback.
>> >
>> > Regards,
>> > Saikat
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Mon, Sep 28, 2020 at 7:36 AM Petr Ivanov <mr...@gmail.com>
>> wrote:
>> >
>> >> It seems that gitbox.apache.org is not available on CI/CD.
>> >>
>> >> I will try to update VCS to GitHub.
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> > On 28 Sep 2020, at 14:07, Alex Plehanov <pl...@gmail.com>
>> >> wrote:
>> >> >
>> >> > Guys,
>> >> >
>> >> > I've tried to build release candidate using TC [1]. But:
>> >> > 1. I can't choose a branch in this TC task (there is no such field).
>> >> > 2. On the "default" branch I got an error: Failed to collect changes,
>> >> > error: List remote refs failed:
>> >> > org.apache.http.conn.ConnectTimeoutException: Connect to
>> >> > gitbox.apache.org:443 [gitbox.apache.org/52.202.80.70] failed:
>> connect
>> >> > timed out, VCS root: "GitBox [asf/ignite]" {instance id=300, parent
>> >> > internal id=74, parent id=GitBoxAsfIgnite, description: "
>> >> > https://gitbox.apache.org/repos/asf/ignite.git#refs/heads/master"}
>> >> >
>> >> > Is there some problem with this TC task? What am I doing wrong?
>> >> >
>> >> > [1] :
>> >> >
>> >>
>> https://ci.ignite.apache.org/viewType.html?buildTypeId=Releases_ApacheIgniteMain_ReleaseBuild
>> >> >
>> >> > пн, 28 сент. 2020 г. в 13:15, Alexey Goncharuk <
>> >> alexey.goncharuk@gmail.com>:
>> >> >
>> >> >> Saikat, Nikolay,
>> >> >>
>> >> >> We have migrated a bunch of modules to ignite-extensions recently.
>> >> Given
>> >> >> that these modules will not be available in Ignite 2.9 anymore (will
>> >> >> they?), should we also plan to release the extensions?
>> >> >>
>> >> >> ср, 23 сент. 2020 г. в 21:49, Alex Plehanov <
>> plehanov.alex@gmail.com>:
>> >> >>
>> >> >>> Igniters,
>> >> >>>
>> >> >>> I've prepared release notes for the 2.9 release [1]. Can anyone
>> review
>> >> >> it,
>> >> >>> please?
>> >> >>>
>> >> >>> [1]: https://github.com/apache/ignite/pull/8273
>> >> >>>
>> >> >>> вт, 22 сент. 2020 г. в 09:39, Alex Plehanov <
>> plehanov.alex@gmail.com
>> >> >:
>> >> >>>
>> >> >>>> Guys,
>> >> >>>>
>> >> >>>> I've filled the ticket with reproducer [1] for the discovery bug.
>> >> This
>> >> >>> bug
>> >> >>>> caused by [2] ticket. We discussed with Vladimir Steshin privately
>> >> and
>> >> >>>> decided to revert this ticket. I will do it today (after TC bot
>> visa)
>> >> >> if
>> >> >>>> there are no objections.
>> >> >>>>
>> >> >>>> [1]: https://issues.apache.org/jira/browse/IGNITE-13465
>> >> >>>> [2]: https://issues.apache.org/jira/browse/IGNITE-13134
>> >> >>>>
>> >> >>>> пн, 21 сент. 2020 г. в 11:08, Alex Plehanov <
>> plehanov.alex@gmail.com
>> >> >:
>> >> >>>>
>> >> >>>>> Guys,
>> >> >>>>>
>> >> >>>>> During internal testing, we've found a critical bug with
>> >> >>>>> discovery (cluster falls apart if two nodes segmented
>> sequentially).
>> >> >>> This
>> >> >>>>> problem is not reproduced in 2.8.1. I think we should fix it
>> >> >>>>> before release. Under investigation now. I'll let you know when
>> we
>> >> get
>> >> >>>>> something.
>> >> >>>>>
>> >> >>>>> чт, 17 сент. 2020 г. в 00:51, Andrey Gura <ag...@apache.org>:
>> >> >>>>>
>> >> >>>>>>> So what do you think? Should we proceed with a 'hacked'
>> version of
>> >> >>> the
>> >> >>>>>> message factory in 2.9 and go for the runtime message
>> generation in
>> >> >>> later
>> >> >>>>>> release, or keep the code clean and fix the regression in the
>> next
>> >> >>> releases?
>> >> >>>>>>> Andrey, can you take a look at my change? I think it is fairly
>> >> >>>>>> straightforward and does not change the semantics, just skips
>> the
>> >> >>> factory
>> >> >>>>>> closures for certain messages.
>> >> >>>>>>
>> >> >>>>>> IMHO 2.5% isn't too much especially because it isn't actual for
>> all
>> >> >>>>>> workloads (I didn't get any significant drops during
>> benchmarking).
>> >> >> So
>> >> >>>>>> I prefer the runtime generation in later releases.
>> >> >>>>>>
>> >> >>>>>> On Mon, Sep 14, 2020 at 12:41 PM Alexey Goncharuk
>> >> >>>>>> <al...@gmail.com> wrote:
>> >> >>>>>>>
>> >> >>>>>>> Alexey, Andrey, Igniters,
>> >> >>>>>>>
>> >> >>>>>>> So what do you think? Should we proceed with a 'hacked'
>> version of
>> >> >>> the
>> >> >>>>>> message factory in 2.9 and go for the runtime message
>> generation in
>> >> >>> later
>> >> >>>>>> release, or keep the code clean and fix the regression in the
>> next
>> >> >>> releases?
>> >> >>>>>>> Andrey, can you take a look at my change? I think it is fairly
>> >> >>>>>> straightforward and does not change the semantics, just skips
>> the
>> >> >>> factory
>> >> >>>>>> closures for certain messages.
>> >> >>>>>>>
>> >> >>>>>>> Personally, I would prefer fixing the regression given that we
>> >> also
>> >> >>>>>> introduced tracing in this release.
>> >> >>>>>>>
>> >> >>>>>>>
>> >> >>>>>>>
>> >> >>>>>>> пт, 11 сент. 2020 г. в 12:09, Alex Plehanov <
>> >> >> plehanov.alex@gmail.com
>> >> >>>> :
>> >> >>>>>>>>
>> >> >>>>>>>> Alexey,
>> >> >>>>>>>>
>> >> >>>>>>>> We've benchmarked by yardstick commits 130376741bf vs
>> ed52559eb95
>> >> >>> and
>> >> >>>>>> the performance of ed52559eb95 is better for about 2.5% on
>> average
>> >> on
>> >> >>> our
>> >> >>>>>> environment (about the same results we got benchmarking
>> 65c30ec6 vs
>> >> >>>>>> 0606f03d). We've made 24 runs for each commit of
>> >> >>>>>> IgnitePutTxImplicitBenchmark (we got maximum drop for 2.9 on
>> this
>> >> >>>>>> benchmark), 200 seconds warmup, 300 seconds benchmark, 6
>> servers, 5
>> >> >>> clients
>> >> >>>>>> 50 threads each. Yardstick results for this configuration:
>> >> >>>>>>>> Commit 130376741bf: avg TPS=164096, avg latency=9173464 ns
>> >> >>>>>>>> Commit ed52559eb95: avg TPS=168283, avg latency=8945908 ns
>> >> >>>>>>>>
>> >> >>>>>>>> пт, 11 сент. 2020 г. в 09:51, Artem Budnikov <
>> >> >>>>>> a.budnikov.ignite@gmail.com>:
>> >> >>>>>>>>>
>> >> >>>>>>>>> Hi Everyone,
>> >> >>>>>>>>>
>> >> >>>>>>>>> I posted an instruction on how to publish the docs on
>> >> >>>>>> ignite.apache.org/docs [1]. When you finish with Ignite 2.9,
>> you
>> >> can
>> >> >>>>>> update the docs by following the instruction. Unfortunately, I
>> >> won't
>> >> >> be
>> >> >>>>>> able to spend any time on this project any longer. You can send
>> >> your
>> >> >>> pull
>> >> >>>>>> requests and questions about the documentation to Denis Magda.
>> >> >>>>>>>>>
>> >> >>>>>>>>> -Artem
>> >> >>>>>>>>>
>> >> >>>>>>>>> [1] :
>> >> >>>>>>
>> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Document
>> >> >>>>>>>>>
>> >> >>>>>>>>> On Thu, Sep 10, 2020 at 2:45 PM Alexey Goncharuk <
>> >> >>>>>> alexey.goncharuk@gmail.com> wrote:
>> >> >>>>>>>>>>
>> >> >>>>>>>>>> Alexey,
>> >> >>>>>>>>>>
>> >> >>>>>>>>>> I've tried to play with message factories locally, but
>> >> >>>>>> unfortunately, I
>> >> >>>>>>>>>> cannot spot the difference between old and new
>> implementation
>> >> in
>> >> >>>>>>>>>> distributed benchmarks. I pushed an implementation of
>> >> >>>>>> MessageFactoryImpl
>> >> >>>>>>>>>> with the old switch statement to the ignite-2.9-revert-12568
>> >> >>> branch
>> >> >>>>>>>>>> (discussed this with Andrey Gura, the change should be
>> >> >> compatible
>> >> >>>>>> with the
>> >> >>>>>>>>>> new metrics as we still use the register() mechanics).
>> >> >>>>>>>>>>
>> >> >>>>>>>>>> Can you check if this change makes any difference
>> >> >> performance-wise
>> >> >>>>>> in your
>> >> >>>>>>>>>> environment? If yes, we can go with runtime code generation
>> in
>> >> >> the
>> >> >>>>>> long
>> >> >>>>>>>>>> term: register classes and generate a dynamic message
>> factory
>> >> >> with
>> >> >>>>>> a switch
>> >> >>>>>>>>>> statement once all messages are registered (not in 2.9
>> though,
>> >> >>>>>> obviously).
>> >> >>>>>>>>>>
>> >> >>>>>>>>>> ср, 9 сент. 2020 г. в 14:53, Alex Plehanov <
>> >> >>> plehanov.alex@gmail.com
>> >> >>>>>>> :
>> >> >>>>>>>>>>
>> >> >>>>>>>>>>> Hello guys,
>> >> >>>>>>>>>>>
>> >> >>>>>>>>>>> I've tried to optimize tracing implementation (ticket
>> [1]), it
>> >> >>>>>> reduced the
>> >> >>>>>>>>>>> drop, but not completely removed it.
>> >> >>>>>>>>>>> Ivan Rakov, Alexander Lapin, can you please review the
>> patch?
>> >> >>>>>>>>>>> Ivan Artiukhov, can you please benchmark the patch [2]
>> against
>> >> >>>>>> 2.8.1
>> >> >>>>>>>>>>> release on your environment?
>> >> >>>>>>>>>>> With this patch on our environment, it's about a 3% drop
>> left,
>> >> >>>>>> it's close
>> >> >>>>>>>>>>> to measurement error and I think such a drop is not a
>> >> >>>>>> showstopper. Guys,
>> >> >>>>>>>>>>> WDYT?
>> >> >>>>>>>>>>>
>> >> >>>>>>>>>>> Also, I found that compatibility is broken for JDBC thin
>> >> >> driver
>> >> >>>>>> between 2.8
>> >> >>>>>>>>>>> and 2.9 versions (ticket [3]). I think it's a blocker and
>> >> >> should
>> >> >>>>>> be
>> >> >>>>>>>>>>> fixed in 2.9. I've prepared the patch.
>> >> >>>>>>>>>>> Taras Ledkov, can you please review this patch?
>> >> >>>>>>>>>>>
>> >> >>>>>>>>>>> And one more ticket I propose to include into 2.9 [4] (NIO
>> >> >>> message
>> >> >>>>>>>>>>> send problem in some circumstances). I will cherry-pick it
>> if
>> >> >>>>>> there is no
>> >> >>>>>>>>>>> objection.
>> >> >>>>>>>>>>>
>> >> >>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13411
>> >> >>>>>>>>>>> [2] https://github.com/apache/ignite/pull/8223
>> >> >>>>>>>>>>> [3] https://issues.apache.org/jira/browse/IGNITE-13414
>> >> >>>>>>>>>>> [4] https://issues.apache.org/jira/browse/IGNITE-13361
>> >> >>>>>>>>>>>
>> >> >>>>>>>>>>> пн, 7 сент. 2020 г. в 14:14, Maxim Muzafarov <
>> >> >> mmuzaf@apache.org
>> >> >>>> :
>> >> >>>>>>>>>>>
>> >> >>>>>>>>>>>> Alexey,
>> >> >>>>>>>>>>>>
>> >> >>>>>>>>>>>> I propose to include [1] issue to the 2.9 release. Since
>> >> >> this
>> >> >>>>>> issue is
>> >> >>>>>>>>>>>> related to the new master key change functionality which
>> >> >>>>>> haven't been
>> >> >>>>>>>>>>>> released yet I think it will be safe to cherry-pick commit
>> >> >> to
>> >> >>>>>> the
>> >> >>>>>>>>>>>> release branch.
>> >> >>>>>>>>>>>>
>> >> >>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13390
>> >> >>>>>>>>>>>>
>> >> >>>>>>>>>>>> On Tue, 1 Sep 2020 at 12:13, Nikolay Izhikov <
>> >> >>>>>> nizhikov@apache.org>
>> >> >>>>>>>>>>> wrote:
>> >> >>>>>>>>>>>>>
>> >> >>>>>>>>>>>>> Hello, Igniters.
>> >> >>>>>>>>>>>>>
>> >> >>>>>>>>>>>>> Alexey, please, include one more Python thin client fix
>> >> >> [1]
>> >> >>>>>> into the
>> >> >>>>>>>>>>> 2.9
>> >> >>>>>>>>>>>> release
>> >> >>>>>>>>>>>>> It fixes kinda major issue - "Python client returns
>> fields
>> >> >>> in
>> >> >>>>>> wrong
>> >> >>>>>>>>>>>> order since the 2 row when fields_count>10"
>> >> >>>>>>>>>>>>>
>> >> >>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-12809
>> >> >>>>>>>>>>>>> [2]
>> >> >>>>>>>>>>>>
>> >> >>>>>>>>>>>
>> >> >>>>>>
>> >> >>>
>> >> >>
>> >>
>> https://github.com/apache/ignite/commit/38025ee4167f05eaa2d6a2c5c2ab70c83a462cfc
>> >> >>>>>>>>>>>>>
>> >> >>>>>>>>>>>>>> 31 авг. 2020 г., в 19:23, Alexey Goncharuk <
>> >> >>>>>>>>>>> alexey.goncharuk@gmail.com>
>> >> >>>>>>>>>>>> написал(а):
>> >> >>>>>>>>>>>>>>
>> >> >>>>>>>>>>>>>> Alexey, thanks, got it. I am not sure we can optimize
>> >> >>>>>> anything out of
>> >> >>>>>>>>>>>> the
>> >> >>>>>>>>>>>>>> message factory with suppliers (at least I have no ideas
>> >> >>>>>> right now),
>> >> >>>>>>>>>>> so
>> >> >>>>>>>>>>>>>> most likely the only move here is to switch back to the
>> >> >>>>>> switch
>> >> >>>>>>>>>>> approach
>> >> >>>>>>>>>>>>>> somehow preserving the metrics part. Probably, inlining
>> >> >>> the
>> >> >>>>>> Ignite
>> >> >>>>>>>>>>>> messages
>> >> >>>>>>>>>>>>>> to the IgniteMessageFactoryImpl should do the trick. Let
>> >> >>> me
>> >> >>>>>> explore
>> >> >>>>>>>>>>> the
>> >> >>>>>>>>>>>>>> code a bit.
>> >> >>>>>>>>>>>>>>
>> >> >>>>>>>>>>>>>> P.S. I am surprised by the impact this part makes for
>> >> >> the
>> >> >>>>>>>>>>> performance.
>> >> >>>>>>>>>>>>>> Message creation is indeed on the hot path, but a single
>> >> >>>>>> virtual call
>> >> >>>>>>>>>>>>>> should not make that much of a difference given the
>> >> >> amount
>> >> >>>>>> of other
>> >> >>>>>>>>>>>> work
>> >> >>>>>>>>>>>>>> happening during the message processing.
>> >> >>>>>>>>>>>>>>
>> >> >>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 18:33, Alex Plehanov <
>> >> >>>>>> plehanov.alex@gmail.com
>> >> >>>>>>>>>>>> :
>> >> >>>>>>>>>>>>>>
>> >> >>>>>>>>>>>>>>> Alexey, sorry, I wrongly interpreted our benchmark
>> >> >>> results.
>> >> >>>>>>>>>>> Actually,
>> >> >>>>>>>>>>>> we
>> >> >>>>>>>>>>>>>>> were looking for a drop using bi-sect in the range
>> >> >>> between
>> >> >>>>>> e6a7f93
>> >> >>>>>>>>>>>> (first
>> >> >>>>>>>>>>>>>>> commit in the 2.9 branch after 2.8 branch cut) and
>> >> >>>>>> 6592dfa5 (last
>> >> >>>>>>>>>>>> commit in
>> >> >>>>>>>>>>>>>>> the 2.9 branch). And we found these two problematic
>> >> >>>>>> commits.
>> >> >>>>>>>>>>>>>>>
>> >> >>>>>>>>>>>>>>> Perhaps only IGNITE-13060 (Tracing) is responsible for
>> >> >> a
>> >> >>>>>> drop
>> >> >>>>>>>>>>> between
>> >> >>>>>>>>>>>>>>> 2.8.1 and 2.9 (we have benchmarked 2.8.1 vs 2.9 with
>> >> >>>>>> reverted
>> >> >>>>>>>>>>>> IGNITE-13060
>> >> >>>>>>>>>>>>>>> now and performance looks the same)
>> >> >>>>>>>>>>>>>>>
>> >> >>>>>>>>>>>>>>> Ticket IGNITE-12568 (MessageFactory refactoring) is not
>> >> >>>>>> related to
>> >> >>>>>>>>>>>> drop
>> >> >>>>>>>>>>>>>>> between 2.8.1 and 2.9, but still has some performance
>> >> >>>>>> problem, and
>> >> >>>>>>>>>>> we
>> >> >>>>>>>>>>>> can
>> >> >>>>>>>>>>>>>>> win back IGNITE-13060 drop by this ticket.
>> >> >>>>>>>>>>>>>>>
>> >> >>>>>>>>>>>>>>> Do we need more investigation on IGNITE-13060 or we
>> >> >> leave
>> >> >>>>>> it as is?
>> >> >>>>>>>>>>>>>>>
>> >> >>>>>>>>>>>>>>> What should we do with IGNITE-12568 (MessageFactory
>> >> >>>>>> refactoring)?
>> >> >>>>>>>>>>>>>>>
>> >> >>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 13:25, Alexey Goncharuk <
>> >> >>>>>>>>>>>> alexey.goncharuk@gmail.com
>> >> >>>>>>>>>>>>>>>> :
>> >> >>>>>>>>>>>>>>>
>> >> >>>>>>>>>>>>>>>> Alexey,
>> >> >>>>>>>>>>>>>>>>
>> >> >>>>>>>>>>>>>>>> While investigating, I found that IGNITE-12568 has an
>> >> >>>>>> incorrect fix
>> >> >>>>>>>>>>>>>>>> version and is actually present in ignite-2.8.1 branch
>> >> >>>>>> [1], so it
>> >> >>>>>>>>>>>> cannot be
>> >> >>>>>>>>>>>>>>>> the source of the drop against 2.8.1.
>> >> >>>>>>>>>>>>>>>>
>> >> >>>>>>>>>>>>>>>> P.S. Looks like we need to enforce a more accurate
>> >> >> work
>> >> >>>>>> with fix
>> >> >>>>>>>>>>>> versions
>> >> >>>>>>>>>>>>>>>> or develop some sort of tooling to verify the fix
>> >> >>>>>> versions.
>> >> >>>>>>>>>>>>>>>>
>> >> >>>>>>>>>>>>>>>> --AG
>> >> >>>>>>>>>>>>>>>>
>> >> >>>>>>>>>>>>>>>> [1]
>> >> >>>>>>>>>>>>>>>>
>> >> >>>>>>>>>>>>
>> >> >>>>>>>>>>>
>> >> >>>>>>
>> >> >>>
>> >> >>
>> >>
>> https://github.com/apache/ignite/commit/3e492bd23851856bbd0385c6a419892d0bba2a34
>> >> >>>>>>>>>>>>>>>>
>> >> >>>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 12:42, Alexey Goncharuk <
>> >> >>>>>>>>>>>> alexey.goncharuk@gmail.com
>> >> >>>>>>>>>>>>>>>>> :
>> >> >>>>>>>>>>>>>>>>
>> >> >>>>>>>>>>>>>>>>> пт, 28 авг. 2020 г. в 11:16, Alex Plehanov <
>> >> >>>>>>>>>>> plehanov.alex@gmail.com
>> >> >>>>>>>>>>>>> :
>> >> >>>>>>>>>>>>>>>>>
>> >> >>>>>>>>>>>>>>>>>> Guys,
>> >> >>>>>>>>>>>>>>>>>>
>> >> >>>>>>>>>>>>>>>>>> We have benchmarked 2.9 without IGNITE-13060 and
>> >> >>>>>> IGNITE-12568
>> >> >>>>>>>>>>>> (reverted
>> >> >>>>>>>>>>>>>>>>>> it
>> >> >>>>>>>>>>>>>>>>>> locally) and got the same performance as on 2.8.1
>> >> >>>>>>>>>>>>>>>>>>
>> >> >>>>>>>>>>>>>>>>>> IGNITE-13060 (Tracing) - some code was added to hot
>> >> >>>>>> paths, to
>> >> >>>>>>>>>>> trace
>> >> >>>>>>>>>>>>>>>>>> these
>> >> >>>>>>>>>>>>>>>>>> hot paths, it's clear why we have performance drop
>> >> >>> here.
>> >> >>>>>>>>>>>>>>>>>>
>> >> >>>>>>>>>>>>>>>>>> IGNITE-12568 (MessageFactory refactoring) -
>> >> >>> switch/case
>> >> >>>>>> block was
>> >> >>>>>>>>>>>>>>>>>> refactored to an array of message suppliers. The
>> >> >>>>>> message factory
>> >> >>>>>>>>>>>> is on
>> >> >>>>>>>>>>>>>>>>>> the
>> >> >>>>>>>>>>>>>>>>>> hot path, which explains why this commit has an
>> >> >> impact
>> >> >>>>>> on total
>> >> >>>>>>>>>>>>>>>>>> performance.
>> >> >>>>>>>>>>>>>>>>>> I've checked JIT assembly output, done some JMH
>> >> >>>>>> microbenchmarks,
>> >> >>>>>>>>>>>> and
>> >> >>>>>>>>>>>>>>>>>> found
>> >> >>>>>>>>>>>>>>>>>> that old implementation of MessageFactory.create()
>> >> >>>>>> about 30-35%
>> >> >>>>>>>>>>>> faster
>> >> >>>>>>>>>>>>>>>>>> than
>> >> >>>>>>>>>>>>>>>>>> the new one. The reason - approach with switch/case
>> >> >>> can
>> >> >>>>>>>>>>> effectively
>> >> >>>>>>>>>>>>>>>>>> inline
>> >> >>>>>>>>>>>>>>>>>> message creation code, but with an array of
>> >> >> suppliers
>> >> >>>>>> relatively
>> >> >>>>>>>>>>>> heavy
>> >> >>>>>>>>>>>>>>>>>> "invokeinterface" cannot be skipped. I've tried to
>> >> >>>>>> rewrite the
>> >> >>>>>>>>>>> code
>> >> >>>>>>>>>>>>>>>>>> using
>> >> >>>>>>>>>>>>>>>>>> an abstract class for suppliers instead of an
>> >> >>> interface
>> >> >>>>>> (to
>> >> >>>>>>>>>>>>>>>>>> replace "invokeinterface" with the "invokevirtual"),
>> >> >>>>>> but it gives
>> >> >>>>>>>>>>>> back
>> >> >>>>>>>>>>>>>>>>>> only
>> >> >>>>>>>>>>>>>>>>>> 10% of method performance and in this case, code
>> >> >> looks
>> >> >>>>>> ugly
>> >> >>>>>>>>>>>> (lambdas
>> >> >>>>>>>>>>>>>>>>>> can't
>> >> >>>>>>>>>>>>>>>>>> be used). Currently, I can't find any more ways to
>> >> >>>>>> optimize the
>> >> >>>>>>>>>>>> current
>> >> >>>>>>>>>>>>>>>>>> approach (except return to the switch/case block).
>> >> >>>>>> Andrey Gura,
>> >> >>>>>>>>>>> as
>> >> >>>>>>>>>>>> the
>> >> >>>>>>>>>>>>>>>>>> author of IGNITE-12568, maybe you have some ideas
>> >> >>> about
>> >> >>>>>>>>>>>> optimization?
>> >> >>>>>>>>>>>>>>>>>>
>> >> >>>>>>>>>>>>>>>>>> Perhaps we should revert IGNITE-12568, but there are
>> >> >>>>>> some metrics
>> >> >>>>>>>>>>>>>>>>>> already
>> >> >>>>>>>>>>>>>>>>>> created, which can't be rewritten using old message
>> >> >>>>>> factory
>> >> >>>>>>>>>>>>>>>>>> implementation
>> >> >>>>>>>>>>>>>>>>>> (IGNITE-12756). Guys, WDYT?
>> >> >>>>>>>>>>>>>>>>>>
>> >> >>>>>>>>>>>>>>>>>
>> >> >>>>>>>>>>>>>>>>> Alexey,
>> >> >>>>>>>>>>>>>>>>>
>> >> >>>>>>>>>>>>>>>>> I see that IGNITE-12756 (metrics improvements) is
>> >> >>>>>> already released
>> >> >>>>>>>>>>>> in
>> >> >>>>>>>>>>>>>>>>> Ignite 2.8.1 while IGNITE-12568 (message factory) is
>> >> >>>>>> only present
>> >> >>>>>>>>>>>> in Ignite
>> >> >>>>>>>>>>>>>>>>> 2.9. Let's revert both IGNITE-12568 and whichever new
>> >> >>>>>> metrics
>> >> >>>>>>>>>>>> created for
>> >> >>>>>>>>>>>>>>>>> 2.9 that depend on the new message factory to unblock
>> >> >>>>>> the release
>> >> >>>>>>>>>>>> and deal
>> >> >>>>>>>>>>>>>>>>> with the optimizations in 2.10?
>> >> >>>>>>>>>>>>>>>>>
>> >> >>>>>>>>>>>>>>>>
>> >> >>>>>>>>>>>>>
>> >> >>>>>>>>>>>>
>> >> >>>>>>>>>>>
>> >> >>>>>>
>> >> >>>>>
>> >> >>>
>> >> >>
>> >>
>> >>
>>
>
Re: Apache Ignite 2.9.0 RELEASE [Time, Scope, Manager]
Posted by Denis Magda <dm...@apache.org>.
Saikat, Alex,
Based on your inputs here, it sounds like once Ignite 2.9 gets released,
all the integrations that made their way to the extensions repo [1] will
get stuck in limbo for some time. What I mean here:
1. While the users will be bumping up their ignite-core,
ignite-indexing, etc. versions to 2.9, they won't find a 2.9 artifact for
Kafka, Camel and other extensions
2. Also, the users won't find 1.0 versions of those extensions that also
need to be released
So, it's unclear how those users are supposed to upgrade to Ignite 2.9 if
they use any of the extensions. Is it safe to use a 2.8 version of an
extension? Or, should we release the 1.0 versions of all the migrated
extensions together with 2.9?
[1] https://github.com/apache/ignite-extensions/tree/master/modules
-
Denis
On Tue, Oct 6, 2020 at 7:20 AM Alexey Goncharuk <al...@gmail.com>
wrote:
> Saikat,
>
> The plan sounds good to me! Do you have an idea for the timeline of the
> module releases? Let me know if I can help in any way.
>
> Also, I was looking into the modularization IEP and noticed that OSGI
> module is discussed to be moved to the extensions, but there is no
> corresponding ticket. Should I create one? I will be happy to help with
> moving this module to extensions.
>
> вт, 29 сент. 2020 г. в 03:03, Saikat Maitra <sa...@gmail.com>:
>
> > Hi Alexey,
> >
> > All the modules as planned in IEP-36
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36:+Modularization
> > are migrated to ignite-extensions repository.
> >
> > We can definitely plan to release ignite-extensions modules.
> >
> > I have a pending PR related to the kafka module and the PR is in review
> > https://github.com/apache/ignite/pull/8222 but its corresponding PR has
> > been merged in ignite-extensions repository.
> >
> > My thoughts / action items for the migration efforts and releases were
> > as follows:
> >
> > 1. Merge the changes for https://github.com/apache/ignite/pull/8222
> > 2. Fix the upload nightly packages task for ignite-core to help fix the
> > travis build failure in ignite-extensions repository.
> > 3. Review and update the README.md files for the ignite-extensions
> modules
> > as required.
> > 4. Update the docs for ignite-extensions modules in ignite-website.
> > 5. Release each module separately and share updates.
> >
> > Please let me know if you have feedback.
> >
> > Regards,
> > Saikat
> >
> >
> >
> >
> >
> >
> >
> >
> > On Mon, Sep 28, 2020 at 7:36 AM Petr Ivanov <mr...@gmail.com> wrote:
> >
> >> It seems that gitbox.apache.org is not available on CI/CD.
> >>
> >> I will try to update VCS to GitHub.
> >>
> >>
> >>
> >>
> >>
> >> > On 28 Sep 2020, at 14:07, Alex Plehanov <pl...@gmail.com>
> >> wrote:
> >> >
> >> > Guys,
> >> >
> >> > I've tried to build release candidate using TC [1]. But:
> >> > 1. I can't choose a branch in this TC task (there is no such field).
> >> > 2. On the "default" branch I got an error: Failed to collect changes,
> >> > error: List remote refs failed:
> >> > org.apache.http.conn.ConnectTimeoutException: Connect to
> >> > gitbox.apache.org:443 [gitbox.apache.org/52.202.80.70] failed:
> connect
> >> > timed out, VCS root: "GitBox [asf/ignite]" {instance id=300, parent
> >> > internal id=74, parent id=GitBoxAsfIgnite, description: "
> >> > https://gitbox.apache.org/repos/asf/ignite.git#refs/heads/master"}
> >> >
> >> > Is there some problem with this TC task? What am I doing wrong?
> >> >
> >> > [1] :
> >> >
> >>
> https://ci.ignite.apache.org/viewType.html?buildTypeId=Releases_ApacheIgniteMain_ReleaseBuild
> >> >
> >> > пн, 28 сент. 2020 г. в 13:15, Alexey Goncharuk <
> >> alexey.goncharuk@gmail.com>:
> >> >
> >> >> Saikat, Nikolay,
> >> >>
> >> >> We have migrated a bunch of modules to ignite-extensions recently.
> >> Given
> >> >> that these modules will not be available in Ignite 2.9 anymore (will
> >> >> they?), should we also plan to release the extensions?
> >> >>
> >> >> ср, 23 сент. 2020 г. в 21:49, Alex Plehanov <plehanov.alex@gmail.com
> >:
> >> >>
> >> >>> Igniters,
> >> >>>
> >> >>> I've prepared release notes for the 2.9 release [1]. Can anyone
> review
> >> >> it,
> >> >>> please?
> >> >>>
> >> >>> [1]: https://github.com/apache/ignite/pull/8273
> >> >>>
> >> >>> вт, 22 сент. 2020 г. в 09:39, Alex Plehanov <
> plehanov.alex@gmail.com
> >> >:
> >> >>>
> >> >>>> Guys,
> >> >>>>
> >> >>>> I've filled the ticket with reproducer [1] for the discovery bug.
> >> This
> >> >>> bug
> >> >>>> caused by [2] ticket. We discussed with Vladimir Steshin privately
> >> and
> >> >>>> decided to revert this ticket. I will do it today (after TC bot
> visa)
> >> >> if
> >> >>>> there are no objections.
> >> >>>>
> >> >>>> [1]: https://issues.apache.org/jira/browse/IGNITE-13465
> >> >>>> [2]: https://issues.apache.org/jira/browse/IGNITE-13134
> >> >>>>
> >> >>>> пн, 21 сент. 2020 г. в 11:08, Alex Plehanov <
> plehanov.alex@gmail.com
> >> >:
> >> >>>>
> >> >>>>> Guys,
> >> >>>>>
> >> >>>>> During internal testing, we've found a critical bug with
> >> >>>>> discovery (cluster falls apart if two nodes segmented
> sequentially).
> >> >>> This
> >> >>>>> problem is not reproduced in 2.8.1. I think we should fix it
> >> >>>>> before release. Under investigation now. I'll let you know when we
> >> get
> >> >>>>> something.
> >> >>>>>
> >> >>>>> чт, 17 сент. 2020 г. в 00:51, Andrey Gura <ag...@apache.org>:
> >> >>>>>
> >> >>>>>>> So what do you think? Should we proceed with a 'hacked' version
> of
> >> >>> the
> >> >>>>>> message factory in 2.9 and go for the runtime message generation
> in
> >> >>> later
> >> >>>>>> release, or keep the code clean and fix the regression in the
> next
> >> >>> releases?
> >> >>>>>>> Andrey, can you take a look at my change? I think it is fairly
> >> >>>>>> straightforward and does not change the semantics, just skips the
> >> >>> factory
> >> >>>>>> closures for certain messages.
> >> >>>>>>
> >> >>>>>> IMHO 2.5% isn't too much especially because it isn't actual for
> all
> >> >>>>>> workloads (I didn't get any significant drops during
> benchmarking).
> >> >> So
> >> >>>>>> I prefer the runtime generation in later releases.
> >> >>>>>>
> >> >>>>>> On Mon, Sep 14, 2020 at 12:41 PM Alexey Goncharuk
> >> >>>>>> <al...@gmail.com> wrote:
> >> >>>>>>>
> >> >>>>>>> Alexey, Andrey, Igniters,
> >> >>>>>>>
> >> >>>>>>> So what do you think? Should we proceed with a 'hacked' version
> of
> >> >>> the
> >> >>>>>> message factory in 2.9 and go for the runtime message generation
> in
> >> >>> later
> >> >>>>>> release, or keep the code clean and fix the regression in the
> next
> >> >>> releases?
> >> >>>>>>> Andrey, can you take a look at my change? I think it is fairly
> >> >>>>>> straightforward and does not change the semantics, just skips the
> >> >>> factory
> >> >>>>>> closures for certain messages.
> >> >>>>>>>
> >> >>>>>>> Personally, I would prefer fixing the regression given that we
> >> also
> >> >>>>>> introduced tracing in this release.
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>> пт, 11 сент. 2020 г. в 12:09, Alex Plehanov <
> >> >> plehanov.alex@gmail.com
> >> >>>> :
> >> >>>>>>>>
> >> >>>>>>>> Alexey,
> >> >>>>>>>>
> >> >>>>>>>> We've benchmarked by yardstick commits 130376741bf vs
> ed52559eb95
> >> >>> and
> >> >>>>>> the performance of ed52559eb95 is better for about 2.5% on
> average
> >> on
> >> >>> our
> >> >>>>>> environment (about the same results we got benchmarking 65c30ec6
> vs
> >> >>>>>> 0606f03d). We've made 24 runs for each commit of
> >> >>>>>> IgnitePutTxImplicitBenchmark (we got maximum drop for 2.9 on this
> >> >>>>>> benchmark), 200 seconds warmup, 300 seconds benchmark, 6
> servers, 5
> >> >>> clients
> >> >>>>>> 50 threads each. Yardstick results for this configuration:
> >> >>>>>>>> Commit 130376741bf: avg TPS=164096, avg latency=9173464 ns
> >> >>>>>>>> Commit ed52559eb95: avg TPS=168283, avg latency=8945908 ns
> >> >>>>>>>>
> >> >>>>>>>> пт, 11 сент. 2020 г. в 09:51, Artem Budnikov <
> >> >>>>>> a.budnikov.ignite@gmail.com>:
> >> >>>>>>>>>
> >> >>>>>>>>> Hi Everyone,
> >> >>>>>>>>>
> >> >>>>>>>>> I posted an instruction on how to publish the docs on
> >> >>>>>> ignite.apache.org/docs [1]. When you finish with Ignite 2.9, you
> >> can
> >> >>>>>> update the docs by following the instruction. Unfortunately, I
> >> won't
> >> >> be
> >> >>>>>> able to spend any time on this project any longer. You can send
> >> your
> >> >>> pull
> >> >>>>>> requests and questions about the documentation to Denis Magda.
> >> >>>>>>>>>
> >> >>>>>>>>> -Artem
> >> >>>>>>>>>
> >> >>>>>>>>> [1] :
> >> >>>>>>
> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Document
> >> >>>>>>>>>
> >> >>>>>>>>> On Thu, Sep 10, 2020 at 2:45 PM Alexey Goncharuk <
> >> >>>>>> alexey.goncharuk@gmail.com> wrote:
> >> >>>>>>>>>>
> >> >>>>>>>>>> Alexey,
> >> >>>>>>>>>>
> >> >>>>>>>>>> I've tried to play with message factories locally, but
> >> >>>>>> unfortunately, I
> >> >>>>>>>>>> cannot spot the difference between old and new implementation
> >> in
> >> >>>>>>>>>> distributed benchmarks. I pushed an implementation of
> >> >>>>>> MessageFactoryImpl
> >> >>>>>>>>>> with the old switch statement to the ignite-2.9-revert-12568
> >> >>> branch
> >> >>>>>>>>>> (discussed this with Andrey Gura, the change should be
> >> >> compatible
> >> >>>>>> with the
> >> >>>>>>>>>> new metrics as we still use the register() mechanics).
> >> >>>>>>>>>>
> >> >>>>>>>>>> Can you check if this change makes any difference
> >> >> performance-wise
> >> >>>>>> in your
> >> >>>>>>>>>> environment? If yes, we can go with runtime code generation
> in
> >> >> the
> >> >>>>>> long
> >> >>>>>>>>>> term: register classes and generate a dynamic message factory
> >> >> with
> >> >>>>>> a switch
> >> >>>>>>>>>> statement once all messages are registered (not in 2.9
> though,
> >> >>>>>> obviously).
> >> >>>>>>>>>>
> >> >>>>>>>>>> ср, 9 сент. 2020 г. в 14:53, Alex Plehanov <
> >> >>> plehanov.alex@gmail.com
> >> >>>>>>> :
> >> >>>>>>>>>>
> >> >>>>>>>>>>> Hello guys,
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> I've tried to optimize tracing implementation (ticket [1]),
> it
> >> >>>>>> reduced the
> >> >>>>>>>>>>> drop, but not completely removed it.
> >> >>>>>>>>>>> Ivan Rakov, Alexander Lapin, can you please review the
> patch?
> >> >>>>>>>>>>> Ivan Artiukhov, can you please benchmark the patch [2]
> against
> >> >>>>>> 2.8.1
> >> >>>>>>>>>>> release on your environment?
> >> >>>>>>>>>>> With this patch on our environment, it's about a 3% drop
> left,
> >> >>>>>> it's close
> >> >>>>>>>>>>> to measurement error and I think such a drop is not a
> >> >>>>>> showstopper. Guys,
> >> >>>>>>>>>>> WDYT?
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> Also, I found that compatibility is broken for JDBC thin
> >> >> driver
> >> >>>>>> between 2.8
> >> >>>>>>>>>>> and 2.9 versions (ticket [3]). I think it's a blocker and
> >> >> should
> >> >>>>>> be
> >> >>>>>>>>>>> fixed in 2.9. I've prepared the patch.
> >> >>>>>>>>>>> Taras Ledkov, can you please review this patch?
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> And one more ticket I propose to include into 2.9 [4] (NIO
> >> >>> message
> >> >>>>>>>>>>> send problem in some circumstances). I will cherry-pick it
> if
> >> >>>>>> there is no
> >> >>>>>>>>>>> objection.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13411
> >> >>>>>>>>>>> [2] https://github.com/apache/ignite/pull/8223
> >> >>>>>>>>>>> [3] https://issues.apache.org/jira/browse/IGNITE-13414
> >> >>>>>>>>>>> [4] https://issues.apache.org/jira/browse/IGNITE-13361
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> пн, 7 сент. 2020 г. в 14:14, Maxim Muzafarov <
> >> >> mmuzaf@apache.org
> >> >>>> :
> >> >>>>>>>>>>>
> >> >>>>>>>>>>>> Alexey,
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> I propose to include [1] issue to the 2.9 release. Since
> >> >> this
> >> >>>>>> issue is
> >> >>>>>>>>>>>> related to the new master key change functionality which
> >> >>>>>> haven't been
> >> >>>>>>>>>>>> released yet I think it will be safe to cherry-pick commit
> >> >> to
> >> >>>>>> the
> >> >>>>>>>>>>>> release branch.
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13390
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> On Tue, 1 Sep 2020 at 12:13, Nikolay Izhikov <
> >> >>>>>> nizhikov@apache.org>
> >> >>>>>>>>>>> wrote:
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>> Hello, Igniters.
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>> Alexey, please, include one more Python thin client fix
> >> >> [1]
> >> >>>>>> into the
> >> >>>>>>>>>>> 2.9
> >> >>>>>>>>>>>> release
> >> >>>>>>>>>>>>> It fixes kinda major issue - "Python client returns fields
> >> >>> in
> >> >>>>>> wrong
> >> >>>>>>>>>>>> order since the 2 row when fields_count>10"
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-12809
> >> >>>>>>>>>>>>> [2]
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>
> >> >>>>>>
> >> >>>
> >> >>
> >>
> https://github.com/apache/ignite/commit/38025ee4167f05eaa2d6a2c5c2ab70c83a462cfc
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> 31 авг. 2020 г., в 19:23, Alexey Goncharuk <
> >> >>>>>>>>>>> alexey.goncharuk@gmail.com>
> >> >>>>>>>>>>>> написал(а):
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> Alexey, thanks, got it. I am not sure we can optimize
> >> >>>>>> anything out of
> >> >>>>>>>>>>>> the
> >> >>>>>>>>>>>>>> message factory with suppliers (at least I have no ideas
> >> >>>>>> right now),
> >> >>>>>>>>>>> so
> >> >>>>>>>>>>>>>> most likely the only move here is to switch back to the
> >> >>>>>> switch
> >> >>>>>>>>>>> approach
> >> >>>>>>>>>>>>>> somehow preserving the metrics part. Probably, inlining
> >> >>> the
> >> >>>>>> Ignite
> >> >>>>>>>>>>>> messages
> >> >>>>>>>>>>>>>> to the IgniteMessageFactoryImpl should do the trick. Let
> >> >>> me
> >> >>>>>> explore
> >> >>>>>>>>>>> the
> >> >>>>>>>>>>>>>> code a bit.
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> P.S. I am surprised by the impact this part makes for
> >> >> the
> >> >>>>>>>>>>> performance.
> >> >>>>>>>>>>>>>> Message creation is indeed on the hot path, but a single
> >> >>>>>> virtual call
> >> >>>>>>>>>>>>>> should not make that much of a difference given the
> >> >> amount
> >> >>>>>> of other
> >> >>>>>>>>>>>> work
> >> >>>>>>>>>>>>>> happening during the message processing.
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 18:33, Alex Plehanov <
> >> >>>>>> plehanov.alex@gmail.com
> >> >>>>>>>>>>>> :
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> Alexey, sorry, I wrongly interpreted our benchmark
> >> >>> results.
> >> >>>>>>>>>>> Actually,
> >> >>>>>>>>>>>> we
> >> >>>>>>>>>>>>>>> were looking for a drop using bi-sect in the range
> >> >>> between
> >> >>>>>> e6a7f93
> >> >>>>>>>>>>>> (first
> >> >>>>>>>>>>>>>>> commit in the 2.9 branch after 2.8 branch cut) and
> >> >>>>>> 6592dfa5 (last
> >> >>>>>>>>>>>> commit in
> >> >>>>>>>>>>>>>>> the 2.9 branch). And we found these two problematic
> >> >>>>>> commits.
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> Perhaps only IGNITE-13060 (Tracing) is responsible for
> >> >> a
> >> >>>>>> drop
> >> >>>>>>>>>>> between
> >> >>>>>>>>>>>>>>> 2.8.1 and 2.9 (we have benchmarked 2.8.1 vs 2.9 with
> >> >>>>>> reverted
> >> >>>>>>>>>>>> IGNITE-13060
> >> >>>>>>>>>>>>>>> now and performance looks the same)
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> Ticket IGNITE-12568 (MessageFactory refactoring) is not
> >> >>>>>> related to
> >> >>>>>>>>>>>> drop
> >> >>>>>>>>>>>>>>> between 2.8.1 and 2.9, but still has some performance
> >> >>>>>> problem, and
> >> >>>>>>>>>>> we
> >> >>>>>>>>>>>> can
> >> >>>>>>>>>>>>>>> win back IGNITE-13060 drop by this ticket.
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> Do we need more investigation on IGNITE-13060 or we
> >> >> leave
> >> >>>>>> it as is?
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> What should we do with IGNITE-12568 (MessageFactory
> >> >>>>>> refactoring)?
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 13:25, Alexey Goncharuk <
> >> >>>>>>>>>>>> alexey.goncharuk@gmail.com
> >> >>>>>>>>>>>>>>>> :
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> Alexey,
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> While investigating, I found that IGNITE-12568 has an
> >> >>>>>> incorrect fix
> >> >>>>>>>>>>>>>>>> version and is actually present in ignite-2.8.1 branch
> >> >>>>>> [1], so it
> >> >>>>>>>>>>>> cannot be
> >> >>>>>>>>>>>>>>>> the source of the drop against 2.8.1.
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> P.S. Looks like we need to enforce a more accurate
> >> >> work
> >> >>>>>> with fix
> >> >>>>>>>>>>>> versions
> >> >>>>>>>>>>>>>>>> or develop some sort of tooling to verify the fix
> >> >>>>>> versions.
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> --AG
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> [1]
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>
> >> >>>>>>
> >> >>>
> >> >>
> >>
> https://github.com/apache/ignite/commit/3e492bd23851856bbd0385c6a419892d0bba2a34
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 12:42, Alexey Goncharuk <
> >> >>>>>>>>>>>> alexey.goncharuk@gmail.com
> >> >>>>>>>>>>>>>>>>> :
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>> пт, 28 авг. 2020 г. в 11:16, Alex Plehanov <
> >> >>>>>>>>>>> plehanov.alex@gmail.com
> >> >>>>>>>>>>>>> :
> >> >>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>> Guys,
> >> >>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>> We have benchmarked 2.9 without IGNITE-13060 and
> >> >>>>>> IGNITE-12568
> >> >>>>>>>>>>>> (reverted
> >> >>>>>>>>>>>>>>>>>> it
> >> >>>>>>>>>>>>>>>>>> locally) and got the same performance as on 2.8.1
> >> >>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>> IGNITE-13060 (Tracing) - some code was added to hot
> >> >>>>>> paths, to
> >> >>>>>>>>>>> trace
> >> >>>>>>>>>>>>>>>>>> these
> >> >>>>>>>>>>>>>>>>>> hot paths, it's clear why we have performance drop
> >> >>> here.
> >> >>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>> IGNITE-12568 (MessageFactory refactoring) -
> >> >>> switch/case
> >> >>>>>> block was
> >> >>>>>>>>>>>>>>>>>> refactored to an array of message suppliers. The
> >> >>>>>> message factory
> >> >>>>>>>>>>>> is on
> >> >>>>>>>>>>>>>>>>>> the
> >> >>>>>>>>>>>>>>>>>> hot path, which explains why this commit has an
> >> >> impact
> >> >>>>>> on total
> >> >>>>>>>>>>>>>>>>>> performance.
> >> >>>>>>>>>>>>>>>>>> I've checked JIT assembly output, done some JMH
> >> >>>>>> microbenchmarks,
> >> >>>>>>>>>>>> and
> >> >>>>>>>>>>>>>>>>>> found
> >> >>>>>>>>>>>>>>>>>> that old implementation of MessageFactory.create()
> >> >>>>>> about 30-35%
> >> >>>>>>>>>>>> faster
> >> >>>>>>>>>>>>>>>>>> than
> >> >>>>>>>>>>>>>>>>>> the new one. The reason - approach with switch/case
> >> >>> can
> >> >>>>>>>>>>> effectively
> >> >>>>>>>>>>>>>>>>>> inline
> >> >>>>>>>>>>>>>>>>>> message creation code, but with an array of
> >> >> suppliers
> >> >>>>>> relatively
> >> >>>>>>>>>>>> heavy
> >> >>>>>>>>>>>>>>>>>> "invokeinterface" cannot be skipped. I've tried to
> >> >>>>>> rewrite the
> >> >>>>>>>>>>> code
> >> >>>>>>>>>>>>>>>>>> using
> >> >>>>>>>>>>>>>>>>>> an abstract class for suppliers instead of an
> >> >>> interface
> >> >>>>>> (to
> >> >>>>>>>>>>>>>>>>>> replace "invokeinterface" with the "invokevirtual"),
> >> >>>>>> but it gives
> >> >>>>>>>>>>>> back
> >> >>>>>>>>>>>>>>>>>> only
> >> >>>>>>>>>>>>>>>>>> 10% of method performance and in this case, code
> >> >> looks
> >> >>>>>> ugly
> >> >>>>>>>>>>>> (lambdas
> >> >>>>>>>>>>>>>>>>>> can't
> >> >>>>>>>>>>>>>>>>>> be used). Currently, I can't find any more ways to
> >> >>>>>> optimize the
> >> >>>>>>>>>>>> current
> >> >>>>>>>>>>>>>>>>>> approach (except return to the switch/case block).
> >> >>>>>> Andrey Gura,
> >> >>>>>>>>>>> as
> >> >>>>>>>>>>>> the
> >> >>>>>>>>>>>>>>>>>> author of IGNITE-12568, maybe you have some ideas
> >> >>> about
> >> >>>>>>>>>>>> optimization?
> >> >>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>> Perhaps we should revert IGNITE-12568, but there are
> >> >>>>>> some metrics
> >> >>>>>>>>>>>>>>>>>> already
> >> >>>>>>>>>>>>>>>>>> created, which can't be rewritten using old message
> >> >>>>>> factory
> >> >>>>>>>>>>>>>>>>>> implementation
> >> >>>>>>>>>>>>>>>>>> (IGNITE-12756). Guys, WDYT?
> >> >>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>> Alexey,
> >> >>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>> I see that IGNITE-12756 (metrics improvements) is
> >> >>>>>> already released
> >> >>>>>>>>>>>> in
> >> >>>>>>>>>>>>>>>>> Ignite 2.8.1 while IGNITE-12568 (message factory) is
> >> >>>>>> only present
> >> >>>>>>>>>>>> in Ignite
> >> >>>>>>>>>>>>>>>>> 2.9. Let's revert both IGNITE-12568 and whichever new
> >> >>>>>> metrics
> >> >>>>>>>>>>>> created for
> >> >>>>>>>>>>>>>>>>> 2.9 that depend on the new message factory to unblock
> >> >>>>>> the release
> >> >>>>>>>>>>>> and deal
> >> >>>>>>>>>>>>>>>>> with the optimizations in 2.10?
> >> >>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>
> >> >>>>>>
> >> >>>>>
> >> >>>
> >> >>
> >>
> >>
>
Re: Apache Ignite 2.9.0 RELEASE [Time, Scope, Manager]
Posted by Saikat Maitra <sa...@gmail.com>.
Hi Alexey,
Thank you for your email. Yes, I can certainly use help in the release
process. I am thinking around the month of Nov for the releases after
the In-Memory Computing Summit Virtual 2020 but any of the module can be
released before that timeline. I have a tech talk coming up on Oct 28 and I
may get a little occupied for the same.
https://www.imcsummit.org/2020/virtual/agenda/schedule/day-1
Also, I have not created any ticket for the OSGI module, please feel free
to create a ticket and I can help with the migration.
Regards,
Saikat
On Tue, Oct 6, 2020 at 9:19 AM Alexey Goncharuk <al...@gmail.com>
wrote:
> Saikat,
>
> The plan sounds good to me! Do you have an idea for the timeline of the
> module releases? Let me know if I can help in any way.
>
> Also, I was looking into the modularization IEP and noticed that OSGI
> module is discussed to be moved to the extensions, but there is no
> corresponding ticket. Should I create one? I will be happy to help with
> moving this module to extensions.
>
> вт, 29 сент. 2020 г. в 03:03, Saikat Maitra <sa...@gmail.com>:
>
>> Hi Alexey,
>>
>> All the modules as planned in IEP-36
>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36:+Modularization
>> are migrated to ignite-extensions repository.
>>
>> We can definitely plan to release ignite-extensions modules.
>>
>> I have a pending PR related to the kafka module and the PR is in review
>> https://github.com/apache/ignite/pull/8222 but its corresponding PR has
>> been merged in ignite-extensions repository.
>>
>> My thoughts / action items for the migration efforts and releases were
>> as follows:
>>
>> 1. Merge the changes for https://github.com/apache/ignite/pull/8222
>> 2. Fix the upload nightly packages task for ignite-core to help fix the
>> travis build failure in ignite-extensions repository.
>> 3. Review and update the README.md files for the ignite-extensions
>> modules as required.
>> 4. Update the docs for ignite-extensions modules in ignite-website.
>> 5. Release each module separately and share updates.
>>
>> Please let me know if you have feedback.
>>
>> Regards,
>> Saikat
>>
>>
>>
>>
>>
>>
>>
>>
>> On Mon, Sep 28, 2020 at 7:36 AM Petr Ivanov <mr...@gmail.com> wrote:
>>
>>> It seems that gitbox.apache.org is not available on CI/CD.
>>>
>>> I will try to update VCS to GitHub.
>>>
>>>
>>>
>>>
>>>
>>> > On 28 Sep 2020, at 14:07, Alex Plehanov <pl...@gmail.com>
>>> wrote:
>>> >
>>> > Guys,
>>> >
>>> > I've tried to build release candidate using TC [1]. But:
>>> > 1. I can't choose a branch in this TC task (there is no such field).
>>> > 2. On the "default" branch I got an error: Failed to collect changes,
>>> > error: List remote refs failed:
>>> > org.apache.http.conn.ConnectTimeoutException: Connect to
>>> > gitbox.apache.org:443 [gitbox.apache.org/52.202.80.70] failed: connect
>>> > timed out, VCS root: "GitBox [asf/ignite]" {instance id=300, parent
>>> > internal id=74, parent id=GitBoxAsfIgnite, description: "
>>> > https://gitbox.apache.org/repos/asf/ignite.git#refs/heads/master"}
>>> >
>>> > Is there some problem with this TC task? What am I doing wrong?
>>> >
>>> > [1] :
>>> >
>>> https://ci.ignite.apache.org/viewType.html?buildTypeId=Releases_ApacheIgniteMain_ReleaseBuild
>>> >
>>> > пн, 28 сент. 2020 г. в 13:15, Alexey Goncharuk <
>>> alexey.goncharuk@gmail.com>:
>>> >
>>> >> Saikat, Nikolay,
>>> >>
>>> >> We have migrated a bunch of modules to ignite-extensions recently.
>>> Given
>>> >> that these modules will not be available in Ignite 2.9 anymore (will
>>> >> they?), should we also plan to release the extensions?
>>> >>
>>> >> ср, 23 сент. 2020 г. в 21:49, Alex Plehanov <plehanov.alex@gmail.com
>>> >:
>>> >>
>>> >>> Igniters,
>>> >>>
>>> >>> I've prepared release notes for the 2.9 release [1]. Can anyone
>>> review
>>> >> it,
>>> >>> please?
>>> >>>
>>> >>> [1]: https://github.com/apache/ignite/pull/8273
>>> >>>
>>> >>> вт, 22 сент. 2020 г. в 09:39, Alex Plehanov <plehanov.alex@gmail.com
>>> >:
>>> >>>
>>> >>>> Guys,
>>> >>>>
>>> >>>> I've filled the ticket with reproducer [1] for the discovery bug.
>>> This
>>> >>> bug
>>> >>>> caused by [2] ticket. We discussed with Vladimir Steshin privately
>>> and
>>> >>>> decided to revert this ticket. I will do it today (after TC bot
>>> visa)
>>> >> if
>>> >>>> there are no objections.
>>> >>>>
>>> >>>> [1]: https://issues.apache.org/jira/browse/IGNITE-13465
>>> >>>> [2]: https://issues.apache.org/jira/browse/IGNITE-13134
>>> >>>>
>>> >>>> пн, 21 сент. 2020 г. в 11:08, Alex Plehanov <
>>> plehanov.alex@gmail.com>:
>>> >>>>
>>> >>>>> Guys,
>>> >>>>>
>>> >>>>> During internal testing, we've found a critical bug with
>>> >>>>> discovery (cluster falls apart if two nodes segmented
>>> sequentially).
>>> >>> This
>>> >>>>> problem is not reproduced in 2.8.1. I think we should fix it
>>> >>>>> before release. Under investigation now. I'll let you know when we
>>> get
>>> >>>>> something.
>>> >>>>>
>>> >>>>> чт, 17 сент. 2020 г. в 00:51, Andrey Gura <ag...@apache.org>:
>>> >>>>>
>>> >>>>>>> So what do you think? Should we proceed with a 'hacked' version
>>> of
>>> >>> the
>>> >>>>>> message factory in 2.9 and go for the runtime message generation
>>> in
>>> >>> later
>>> >>>>>> release, or keep the code clean and fix the regression in the next
>>> >>> releases?
>>> >>>>>>> Andrey, can you take a look at my change? I think it is fairly
>>> >>>>>> straightforward and does not change the semantics, just skips the
>>> >>> factory
>>> >>>>>> closures for certain messages.
>>> >>>>>>
>>> >>>>>> IMHO 2.5% isn't too much especially because it isn't actual for
>>> all
>>> >>>>>> workloads (I didn't get any significant drops during
>>> benchmarking).
>>> >> So
>>> >>>>>> I prefer the runtime generation in later releases.
>>> >>>>>>
>>> >>>>>> On Mon, Sep 14, 2020 at 12:41 PM Alexey Goncharuk
>>> >>>>>> <al...@gmail.com> wrote:
>>> >>>>>>>
>>> >>>>>>> Alexey, Andrey, Igniters,
>>> >>>>>>>
>>> >>>>>>> So what do you think? Should we proceed with a 'hacked' version
>>> of
>>> >>> the
>>> >>>>>> message factory in 2.9 and go for the runtime message generation
>>> in
>>> >>> later
>>> >>>>>> release, or keep the code clean and fix the regression in the next
>>> >>> releases?
>>> >>>>>>> Andrey, can you take a look at my change? I think it is fairly
>>> >>>>>> straightforward and does not change the semantics, just skips the
>>> >>> factory
>>> >>>>>> closures for certain messages.
>>> >>>>>>>
>>> >>>>>>> Personally, I would prefer fixing the regression given that we
>>> also
>>> >>>>>> introduced tracing in this release.
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> пт, 11 сент. 2020 г. в 12:09, Alex Plehanov <
>>> >> plehanov.alex@gmail.com
>>> >>>> :
>>> >>>>>>>>
>>> >>>>>>>> Alexey,
>>> >>>>>>>>
>>> >>>>>>>> We've benchmarked by yardstick commits 130376741bf vs
>>> ed52559eb95
>>> >>> and
>>> >>>>>> the performance of ed52559eb95 is better for about 2.5% on
>>> average on
>>> >>> our
>>> >>>>>> environment (about the same results we got benchmarking 65c30ec6
>>> vs
>>> >>>>>> 0606f03d). We've made 24 runs for each commit of
>>> >>>>>> IgnitePutTxImplicitBenchmark (we got maximum drop for 2.9 on this
>>> >>>>>> benchmark), 200 seconds warmup, 300 seconds benchmark, 6 servers,
>>> 5
>>> >>> clients
>>> >>>>>> 50 threads each. Yardstick results for this configuration:
>>> >>>>>>>> Commit 130376741bf: avg TPS=164096, avg latency=9173464 ns
>>> >>>>>>>> Commit ed52559eb95: avg TPS=168283, avg latency=8945908 ns
>>> >>>>>>>>
>>> >>>>>>>> пт, 11 сент. 2020 г. в 09:51, Artem Budnikov <
>>> >>>>>> a.budnikov.ignite@gmail.com>:
>>> >>>>>>>>>
>>> >>>>>>>>> Hi Everyone,
>>> >>>>>>>>>
>>> >>>>>>>>> I posted an instruction on how to publish the docs on
>>> >>>>>> ignite.apache.org/docs [1]. When you finish with Ignite 2.9, you
>>> can
>>> >>>>>> update the docs by following the instruction. Unfortunately, I
>>> won't
>>> >> be
>>> >>>>>> able to spend any time on this project any longer. You can send
>>> your
>>> >>> pull
>>> >>>>>> requests and questions about the documentation to Denis Magda.
>>> >>>>>>>>>
>>> >>>>>>>>> -Artem
>>> >>>>>>>>>
>>> >>>>>>>>> [1] :
>>> >>>>>>
>>> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Document
>>> >>>>>>>>>
>>> >>>>>>>>> On Thu, Sep 10, 2020 at 2:45 PM Alexey Goncharuk <
>>> >>>>>> alexey.goncharuk@gmail.com> wrote:
>>> >>>>>>>>>>
>>> >>>>>>>>>> Alexey,
>>> >>>>>>>>>>
>>> >>>>>>>>>> I've tried to play with message factories locally, but
>>> >>>>>> unfortunately, I
>>> >>>>>>>>>> cannot spot the difference between old and new implementation
>>> in
>>> >>>>>>>>>> distributed benchmarks. I pushed an implementation of
>>> >>>>>> MessageFactoryImpl
>>> >>>>>>>>>> with the old switch statement to the ignite-2.9-revert-12568
>>> >>> branch
>>> >>>>>>>>>> (discussed this with Andrey Gura, the change should be
>>> >> compatible
>>> >>>>>> with the
>>> >>>>>>>>>> new metrics as we still use the register() mechanics).
>>> >>>>>>>>>>
>>> >>>>>>>>>> Can you check if this change makes any difference
>>> >> performance-wise
>>> >>>>>> in your
>>> >>>>>>>>>> environment? If yes, we can go with runtime code generation in
>>> >> the
>>> >>>>>> long
>>> >>>>>>>>>> term: register classes and generate a dynamic message factory
>>> >> with
>>> >>>>>> a switch
>>> >>>>>>>>>> statement once all messages are registered (not in 2.9 though,
>>> >>>>>> obviously).
>>> >>>>>>>>>>
>>> >>>>>>>>>> ср, 9 сент. 2020 г. в 14:53, Alex Plehanov <
>>> >>> plehanov.alex@gmail.com
>>> >>>>>>> :
>>> >>>>>>>>>>
>>> >>>>>>>>>>> Hello guys,
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> I've tried to optimize tracing implementation (ticket [1]),
>>> it
>>> >>>>>> reduced the
>>> >>>>>>>>>>> drop, but not completely removed it.
>>> >>>>>>>>>>> Ivan Rakov, Alexander Lapin, can you please review the patch?
>>> >>>>>>>>>>> Ivan Artiukhov, can you please benchmark the patch [2]
>>> against
>>> >>>>>> 2.8.1
>>> >>>>>>>>>>> release on your environment?
>>> >>>>>>>>>>> With this patch on our environment, it's about a 3% drop
>>> left,
>>> >>>>>> it's close
>>> >>>>>>>>>>> to measurement error and I think such a drop is not a
>>> >>>>>> showstopper. Guys,
>>> >>>>>>>>>>> WDYT?
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> Also, I found that compatibility is broken for JDBC thin
>>> >> driver
>>> >>>>>> between 2.8
>>> >>>>>>>>>>> and 2.9 versions (ticket [3]). I think it's a blocker and
>>> >> should
>>> >>>>>> be
>>> >>>>>>>>>>> fixed in 2.9. I've prepared the patch.
>>> >>>>>>>>>>> Taras Ledkov, can you please review this patch?
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> And one more ticket I propose to include into 2.9 [4] (NIO
>>> >>> message
>>> >>>>>>>>>>> send problem in some circumstances). I will cherry-pick it if
>>> >>>>>> there is no
>>> >>>>>>>>>>> objection.
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13411
>>> >>>>>>>>>>> [2] https://github.com/apache/ignite/pull/8223
>>> >>>>>>>>>>> [3] https://issues.apache.org/jira/browse/IGNITE-13414
>>> >>>>>>>>>>> [4] https://issues.apache.org/jira/browse/IGNITE-13361
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> пн, 7 сент. 2020 г. в 14:14, Maxim Muzafarov <
>>> >> mmuzaf@apache.org
>>> >>>> :
>>> >>>>>>>>>>>
>>> >>>>>>>>>>>> Alexey,
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> I propose to include [1] issue to the 2.9 release. Since
>>> >> this
>>> >>>>>> issue is
>>> >>>>>>>>>>>> related to the new master key change functionality which
>>> >>>>>> haven't been
>>> >>>>>>>>>>>> released yet I think it will be safe to cherry-pick commit
>>> >> to
>>> >>>>>> the
>>> >>>>>>>>>>>> release branch.
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13390
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> On Tue, 1 Sep 2020 at 12:13, Nikolay Izhikov <
>>> >>>>>> nizhikov@apache.org>
>>> >>>>>>>>>>> wrote:
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> Hello, Igniters.
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> Alexey, please, include one more Python thin client fix
>>> >> [1]
>>> >>>>>> into the
>>> >>>>>>>>>>> 2.9
>>> >>>>>>>>>>>> release
>>> >>>>>>>>>>>>> It fixes kinda major issue - "Python client returns fields
>>> >>> in
>>> >>>>>> wrong
>>> >>>>>>>>>>>> order since the 2 row when fields_count>10"
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-12809
>>> >>>>>>>>>>>>> [2]
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>
>>> >>>>>>
>>> >>>
>>> >>
>>> https://github.com/apache/ignite/commit/38025ee4167f05eaa2d6a2c5c2ab70c83a462cfc
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>>> 31 авг. 2020 г., в 19:23, Alexey Goncharuk <
>>> >>>>>>>>>>> alexey.goncharuk@gmail.com>
>>> >>>>>>>>>>>> написал(а):
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>> Alexey, thanks, got it. I am not sure we can optimize
>>> >>>>>> anything out of
>>> >>>>>>>>>>>> the
>>> >>>>>>>>>>>>>> message factory with suppliers (at least I have no ideas
>>> >>>>>> right now),
>>> >>>>>>>>>>> so
>>> >>>>>>>>>>>>>> most likely the only move here is to switch back to the
>>> >>>>>> switch
>>> >>>>>>>>>>> approach
>>> >>>>>>>>>>>>>> somehow preserving the metrics part. Probably, inlining
>>> >>> the
>>> >>>>>> Ignite
>>> >>>>>>>>>>>> messages
>>> >>>>>>>>>>>>>> to the IgniteMessageFactoryImpl should do the trick. Let
>>> >>> me
>>> >>>>>> explore
>>> >>>>>>>>>>> the
>>> >>>>>>>>>>>>>> code a bit.
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>> P.S. I am surprised by the impact this part makes for
>>> >> the
>>> >>>>>>>>>>> performance.
>>> >>>>>>>>>>>>>> Message creation is indeed on the hot path, but a single
>>> >>>>>> virtual call
>>> >>>>>>>>>>>>>> should not make that much of a difference given the
>>> >> amount
>>> >>>>>> of other
>>> >>>>>>>>>>>> work
>>> >>>>>>>>>>>>>> happening during the message processing.
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 18:33, Alex Plehanov <
>>> >>>>>> plehanov.alex@gmail.com
>>> >>>>>>>>>>>> :
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>> Alexey, sorry, I wrongly interpreted our benchmark
>>> >>> results.
>>> >>>>>>>>>>> Actually,
>>> >>>>>>>>>>>> we
>>> >>>>>>>>>>>>>>> were looking for a drop using bi-sect in the range
>>> >>> between
>>> >>>>>> e6a7f93
>>> >>>>>>>>>>>> (first
>>> >>>>>>>>>>>>>>> commit in the 2.9 branch after 2.8 branch cut) and
>>> >>>>>> 6592dfa5 (last
>>> >>>>>>>>>>>> commit in
>>> >>>>>>>>>>>>>>> the 2.9 branch). And we found these two problematic
>>> >>>>>> commits.
>>> >>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>> Perhaps only IGNITE-13060 (Tracing) is responsible for
>>> >> a
>>> >>>>>> drop
>>> >>>>>>>>>>> between
>>> >>>>>>>>>>>>>>> 2.8.1 and 2.9 (we have benchmarked 2.8.1 vs 2.9 with
>>> >>>>>> reverted
>>> >>>>>>>>>>>> IGNITE-13060
>>> >>>>>>>>>>>>>>> now and performance looks the same)
>>> >>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>> Ticket IGNITE-12568 (MessageFactory refactoring) is not
>>> >>>>>> related to
>>> >>>>>>>>>>>> drop
>>> >>>>>>>>>>>>>>> between 2.8.1 and 2.9, but still has some performance
>>> >>>>>> problem, and
>>> >>>>>>>>>>> we
>>> >>>>>>>>>>>> can
>>> >>>>>>>>>>>>>>> win back IGNITE-13060 drop by this ticket.
>>> >>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>> Do we need more investigation on IGNITE-13060 or we
>>> >> leave
>>> >>>>>> it as is?
>>> >>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>> What should we do with IGNITE-12568 (MessageFactory
>>> >>>>>> refactoring)?
>>> >>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 13:25, Alexey Goncharuk <
>>> >>>>>>>>>>>> alexey.goncharuk@gmail.com
>>> >>>>>>>>>>>>>>>> :
>>> >>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>> Alexey,
>>> >>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>> While investigating, I found that IGNITE-12568 has an
>>> >>>>>> incorrect fix
>>> >>>>>>>>>>>>>>>> version and is actually present in ignite-2.8.1 branch
>>> >>>>>> [1], so it
>>> >>>>>>>>>>>> cannot be
>>> >>>>>>>>>>>>>>>> the source of the drop against 2.8.1.
>>> >>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>> P.S. Looks like we need to enforce a more accurate
>>> >> work
>>> >>>>>> with fix
>>> >>>>>>>>>>>> versions
>>> >>>>>>>>>>>>>>>> or develop some sort of tooling to verify the fix
>>> >>>>>> versions.
>>> >>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>> --AG
>>> >>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>> [1]
>>> >>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>
>>> >>>>>>
>>> >>>
>>> >>
>>> https://github.com/apache/ignite/commit/3e492bd23851856bbd0385c6a419892d0bba2a34
>>> >>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>> пн, 31 авг. 2020 г. в 12:42, Alexey Goncharuk <
>>> >>>>>>>>>>>> alexey.goncharuk@gmail.com
>>> >>>>>>>>>>>>>>>>> :
>>> >>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>> пт, 28 авг. 2020 г. в 11:16, Alex Plehanov <
>>> >>>>>>>>>>> plehanov.alex@gmail.com
>>> >>>>>>>>>>>>> :
>>> >>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>> Guys,
>>> >>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>> We have benchmarked 2.9 without IGNITE-13060 and
>>> >>>>>> IGNITE-12568
>>> >>>>>>>>>>>> (reverted
>>> >>>>>>>>>>>>>>>>>> it
>>> >>>>>>>>>>>>>>>>>> locally) and got the same performance as on 2.8.1
>>> >>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>> IGNITE-13060 (Tracing) - some code was added to hot
>>> >>>>>> paths, to
>>> >>>>>>>>>>> trace
>>> >>>>>>>>>>>>>>>>>> these
>>> >>>>>>>>>>>>>>>>>> hot paths, it's clear why we have performance drop
>>> >>> here.
>>> >>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>> IGNITE-12568 (MessageFactory refactoring) -
>>> >>> switch/case
>>> >>>>>> block was
>>> >>>>>>>>>>>>>>>>>> refactored to an array of message suppliers. The
>>> >>>>>> message factory
>>> >>>>>>>>>>>> is on
>>> >>>>>>>>>>>>>>>>>> the
>>> >>>>>>>>>>>>>>>>>> hot path, which explains why this commit has an
>>> >> impact
>>> >>>>>> on total
>>> >>>>>>>>>>>>>>>>>> performance.
>>> >>>>>>>>>>>>>>>>>> I've checked JIT assembly output, done some JMH
>>> >>>>>> microbenchmarks,
>>> >>>>>>>>>>>> and
>>> >>>>>>>>>>>>>>>>>> found
>>> >>>>>>>>>>>>>>>>>> that old implementation of MessageFactory.create()
>>> >>>>>> about 30-35%
>>> >>>>>>>>>>>> faster
>>> >>>>>>>>>>>>>>>>>> than
>>> >>>>>>>>>>>>>>>>>> the new one. The reason - approach with switch/case
>>> >>> can
>>> >>>>>>>>>>> effectively
>>> >>>>>>>>>>>>>>>>>> inline
>>> >>>>>>>>>>>>>>>>>> message creation code, but with an array of
>>> >> suppliers
>>> >>>>>> relatively
>>> >>>>>>>>>>>> heavy
>>> >>>>>>>>>>>>>>>>>> "invokeinterface" cannot be skipped. I've tried to
>>> >>>>>> rewrite the
>>> >>>>>>>>>>> code
>>> >>>>>>>>>>>>>>>>>> using
>>> >>>>>>>>>>>>>>>>>> an abstract class for suppliers instead of an
>>> >>> interface
>>> >>>>>> (to
>>> >>>>>>>>>>>>>>>>>> replace "invokeinterface" with the "invokevirtual"),
>>> >>>>>> but it gives
>>> >>>>>>>>>>>> back
>>> >>>>>>>>>>>>>>>>>> only
>>> >>>>>>>>>>>>>>>>>> 10% of method performance and in this case, code
>>> >> looks
>>> >>>>>> ugly
>>> >>>>>>>>>>>> (lambdas
>>> >>>>>>>>>>>>>>>>>> can't
>>> >>>>>>>>>>>>>>>>>> be used). Currently, I can't find any more ways to
>>> >>>>>> optimize the
>>> >>>>>>>>>>>> current
>>> >>>>>>>>>>>>>>>>>> approach (except return to the switch/case block).
>>> >>>>>> Andrey Gura,
>>> >>>>>>>>>>> as
>>> >>>>>>>>>>>> the
>>> >>>>>>>>>>>>>>>>>> author of IGNITE-12568, maybe you have some ideas
>>> >>> about
>>> >>>>>>>>>>>> optimization?
>>> >>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>> Perhaps we should revert IGNITE-12568, but there are
>>> >>>>>> some metrics
>>> >>>>>>>>>>>>>>>>>> already
>>> >>>>>>>>>>>>>>>>>> created, which can't be rewritten using old message
>>> >>>>>> factory
>>> >>>>>>>>>>>>>>>>>> implementation
>>> >>>>>>>>>>>>>>>>>> (IGNITE-12756). Guys, WDYT?
>>> >>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>> Alexey,
>>> >>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>> I see that IGNITE-12756 (metrics improvements) is
>>> >>>>>> already released
>>> >>>>>>>>>>>> in
>>> >>>>>>>>>>>>>>>>> Ignite 2.8.1 while IGNITE-12568 (message factory) is
>>> >>>>>> only present
>>> >>>>>>>>>>>> in Ignite
>>> >>>>>>>>>>>>>>>>> 2.9. Let's revert both IGNITE-12568 and whichever new
>>> >>>>>> metrics
>>> >>>>>>>>>>>> created for
>>> >>>>>>>>>>>>>>>>> 2.9 that depend on the new message factory to unblock
>>> >>>>>> the release
>>> >>>>>>>>>>>> and deal
>>> >>>>>>>>>>>>>>>>> with the optimizations in 2.10?
>>> >>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>
>>> >>>>>>
>>> >>>>>
>>> >>>
>>> >>
>>>
>>>