You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by Stepan Pilshchikov <pi...@gmail.com> on 2018/10/09 12:50:32 UTC
Re: Python thin client
Hi, all
Tried to compare new thin client performance and made similar benchmarks for
each one
And result not so good for python
Ticket with results and bench code:
https://issues.apache.org/jira/browse/IGNITE-9824
Py code src:
https://gist.github.com/pilshchikov/8aff4e30d83f8bac20c5a4a9c3917abb
Dmitry, please review results and bench code, maybe somthing wrong or it's
expected numbers?
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: Python thin client
Posted by Stepan Pilschikov <pi...@gmail.com>.
Oh, nice
Thanks for quick response
I've check against JS and PHP, now its works properly
Igor, can you approve this patch?
сб, 20 окт. 2018 г. в 0:57, Dmitry Melnichuk <dmitry.melnichuk@nobitlost.com
>:
> Stepan,
>
> My bad. I got the Decimal part of Ignite thin client protocol
> specification totally wrong.
>
> Just submitted a patch tested against Java thin client. Please evaluate.
>
> On 10/20/18 2:00 AM, Stepan Pilschikov wrote:
> > Dmitry,
> >
> > One more thing
> > Discover that Decimal data type can't be getting by others clients
> >
> > Make ticket with description:
> > https://issues.apache.org/jira/browse/IGNITE-9950
> >
> > Please, can you help to investigate this problem?
> >
> > ср, 17 окт. 2018 г. в 23:14, Stepan Pilschikov <
> pilshchikov.s.n@gmail.com>:
> >
> >> Dmitry,
> >>
> >> Great, now client works properly on all OS
> >> Thanks
> >>
> >> I hope PR will be merged soon
> >>
> >> Igor, can please help with that?
> >>
> >> ср, 17 окт. 2018 г. в 20:48, Dmitry Melnichuk <
> >> dmitry.melnichuk@nobitlost.com>:
> >>
> >>> Stepan,
> >>>
> >>> Sadly, it was a C `long` data type size mismatch between Windows and
> >>> other OSes. I investigated this issue and offered a pull request.
> >>>
> >>> https://github.com/apache/ignite/pull/5017
> >>>
> >>> On 10/17/18 6:29 PM, Stepan Pilschikov wrote:
> >>>> Dmitry,
> >>>>
> >>>> I've trying to use python thin client with Ignite 2.7 it dosn't work
> >>>> Please help to investigate
> >>>> Problem reproduced on several others machines
> >>>>
> >>>> Issue:
> >>>> https://issues.apache.org/jira/browse/IGNITE-9908
> >>>>
> >>>
> >>
> >
>
>
Re: Python thin client
Posted by Dmitry Melnichuk <dm...@nobitlost.com>.
Stepan,
My bad. I got the Decimal part of Ignite thin client protocol
specification totally wrong.
Just submitted a patch tested against Java thin client. Please evaluate.
On 10/20/18 2:00 AM, Stepan Pilschikov wrote:
> Dmitry,
>
> One more thing
> Discover that Decimal data type can't be getting by others clients
>
> Make ticket with description:
> https://issues.apache.org/jira/browse/IGNITE-9950
>
> Please, can you help to investigate this problem?
>
> ср, 17 окт. 2018 г. в 23:14, Stepan Pilschikov <pi...@gmail.com>:
>
>> Dmitry,
>>
>> Great, now client works properly on all OS
>> Thanks
>>
>> I hope PR will be merged soon
>>
>> Igor, can please help with that?
>>
>> ср, 17 окт. 2018 г. в 20:48, Dmitry Melnichuk <
>> dmitry.melnichuk@nobitlost.com>:
>>
>>> Stepan,
>>>
>>> Sadly, it was a C `long` data type size mismatch between Windows and
>>> other OSes. I investigated this issue and offered a pull request.
>>>
>>> https://github.com/apache/ignite/pull/5017
>>>
>>> On 10/17/18 6:29 PM, Stepan Pilschikov wrote:
>>>> Dmitry,
>>>>
>>>> I've trying to use python thin client with Ignite 2.7 it dosn't work
>>>> Please help to investigate
>>>> Problem reproduced on several others machines
>>>>
>>>> Issue:
>>>> https://issues.apache.org/jira/browse/IGNITE-9908
>>>>
>>>
>>
>
Re: Python thin client
Posted by Stepan Pilschikov <pi...@gmail.com>.
Dmitry,
One more thing
Discover that Decimal data type can't be getting by others clients
Make ticket with description:
https://issues.apache.org/jira/browse/IGNITE-9950
Please, can you help to investigate this problem?
ср, 17 окт. 2018 г. в 23:14, Stepan Pilschikov <pi...@gmail.com>:
> Dmitry,
>
> Great, now client works properly on all OS
> Thanks
>
> I hope PR will be merged soon
>
> Igor, can please help with that?
>
> ср, 17 окт. 2018 г. в 20:48, Dmitry Melnichuk <
> dmitry.melnichuk@nobitlost.com>:
>
>> Stepan,
>>
>> Sadly, it was a C `long` data type size mismatch between Windows and
>> other OSes. I investigated this issue and offered a pull request.
>>
>> https://github.com/apache/ignite/pull/5017
>>
>> On 10/17/18 6:29 PM, Stepan Pilschikov wrote:
>> > Dmitry,
>> >
>> > I've trying to use python thin client with Ignite 2.7 it dosn't work
>> > Please help to investigate
>> > Problem reproduced on several others machines
>> >
>> > Issue:
>> > https://issues.apache.org/jira/browse/IGNITE-9908
>> >
>>
>
Re: Python thin client
Posted by Stepan Pilschikov <pi...@gmail.com>.
Dmitry,
Great, now client works properly on all OS
Thanks
I hope PR will be merged soon
Igor, can please help with that?
ср, 17 окт. 2018 г. в 20:48, Dmitry Melnichuk <
dmitry.melnichuk@nobitlost.com>:
> Stepan,
>
> Sadly, it was a C `long` data type size mismatch between Windows and
> other OSes. I investigated this issue and offered a pull request.
>
> https://github.com/apache/ignite/pull/5017
>
> On 10/17/18 6:29 PM, Stepan Pilschikov wrote:
> > Dmitry,
> >
> > I've trying to use python thin client with Ignite 2.7 it dosn't work
> > Please help to investigate
> > Problem reproduced on several others machines
> >
> > Issue:
> > https://issues.apache.org/jira/browse/IGNITE-9908
> >
>
Re: Python thin client
Posted by Dmitry Melnichuk <dm...@nobitlost.com>.
Stepan,
Sadly, it was a C `long` data type size mismatch between Windows and
other OSes. I investigated this issue and offered a pull request.
https://github.com/apache/ignite/pull/5017
On 10/17/18 6:29 PM, Stepan Pilschikov wrote:
> Dmitry,
>
> I've trying to use python thin client with Ignite 2.7 it dosn't work
> Please help to investigate
> Problem reproduced on several others machines
>
> Issue:
> https://issues.apache.org/jira/browse/IGNITE-9908
>
Re: Python thin client
Posted by Stepan Pilschikov <pi...@gmail.com>.
Dmitry,
I've trying to use python thin client with Ignite 2.7 it dosn't work
Please help to investigate
Problem reproduced on several others machines
Issue:
https://issues.apache.org/jira/browse/IGNITE-9908
вт, 16 окт. 2018 г. в 4:36, Dmitry Melnichuk <dmitry.melnichuk@nobitlost.com
>:
> Igor,
>
> I do not have access to edit wiki pages, so please fill it.
>
> All the features are supported, except for:
>
> - Query API: ScanQuery with filter,
> - Binary API: Get object in BinaryObject form, BinaryObject building,
> - Additional Features: Best effort affinity,
> - Type registration: Enum registration, Type name registration.
>
> On 10/15/18 9:17 PM, Igor Sapego wrote:
> > Dmitry,
> >
> > Since Python thin client is now in master, can you please add it to our
> > "Thin clients features" wiki page, so we can track the feature parity
> > of our clients? Or just tell me which features are supported, so I can
> fill
> > it myself.
> >
> > [1] -
> > https://cwiki.apache.org/confluence/display/IGNITE/Thin+clients+features
> >
> > Best Regards,
> > Igor
> >
>
Re: Python thin client
Posted by Dmitry Melnichuk <dm...@nobitlost.com>.
Igor,
I do not have access to edit wiki pages, so please fill it.
All the features are supported, except for:
- Query API: ScanQuery with filter,
- Binary API: Get object in BinaryObject form, BinaryObject building,
- Additional Features: Best effort affinity,
- Type registration: Enum registration, Type name registration.
On 10/15/18 9:17 PM, Igor Sapego wrote:
> Dmitry,
>
> Since Python thin client is now in master, can you please add it to our
> "Thin clients features" wiki page, so we can track the feature parity
> of our clients? Or just tell me which features are supported, so I can fill
> it myself.
>
> [1] -
> https://cwiki.apache.org/confluence/display/IGNITE/Thin+clients+features
>
> Best Regards,
> Igor
>
Re: Python thin client
Posted by Igor Sapego <is...@apache.org>.
Dmitry,
Since Python thin client is now in master, can you please add it to our
"Thin clients features" wiki page, so we can track the feature parity
of our clients? Or just tell me which features are supported, so I can fill
it myself.
[1] -
https://cwiki.apache.org/confluence/display/IGNITE/Thin+clients+features
Best Regards,
Igor
On Mon, Oct 15, 2018 at 1:38 PM Igor Sapego <is...@apache.org> wrote:
> Hi,
>
> I've just merged Python thin client to the master and ignite-2.7
>
> Also, I like how the python's results improved and now are comparable to
> node.js.
>
> Great job, guys!
>
> Best Regards,
> Igor
>
>
> On Sat, Oct 13, 2018 at 3:53 PM Степан Пильщиков <
> pilshchikov.s.n@gmail.com> wrote:
>
>> Hi
>>
>> Made new run and client definitely become faster
>> I think its right direction for improvements
>>
>> Also rerun java bench with compiler=none option
>>
>> All results you can get in
>> https://issues.apache.org/jira/browse/IGNITE-9824
>> (last comment)
>>
>> сб, 13 окт. 2018 г. в 4:56, Pavel Petroshenko <pa...@petroshenko.com>:
>>
>> > Hi Stepan,
>> >
>> > Re your performance questions.
>> >
>> > Firstly, please pull out the latest sources from the ignite-7782 branch
>> as
>> > it has some performance optimizations recently implemented.
>> >
>> > Secondly, Hotspot is a pretty advanced dynamic adaptive compiler and
>> it's
>> > not fair to compare it to a purely interpreted Python language
>> > implementation. To get comparable results, please run your Java
>> benchmark
>> > with -Djava.compiler=NONE. It turns the dynamic compiler off at
>> runtime. I
>> > just did this exercise myself and got comparable scores.
>> >
>> > Please let me know if you still see any significant difference with the
>> > flag set.
>> >
>> > Thanks,
>> > P.
>> >
>> > On Fri, Oct 12, 2018 at 10:02 AM Dmitry Melnichuk <
>> > dmitry.melnichuk@nobitlost.com> wrote:
>> >
>> >> Igor,
>> >>
>> >> I fixed my files.
>> >>
>> >> Dmitry
>> >>
>> >> On 10/13/18 12:14 AM, Igor Sapego wrote:
>> >> > Dmitry,
>> >> >
>> >> > I've run the tests and it appears that there are a lot of files
>> without
>> >> > Apache
>> >> > licenses in Python client: [1] (See Unapproved licenses list).
>> >> >
>> >> > Can you either add headers or exclude file from the check, if you can
>> >> not
>> >> > add one? I.e. to this list: [2]
>> >> >
>> >> > [1] -
>> >> >
>> >>
>> https://ci.ignite.apache.org/repository/download/IgniteTests24Java8_LicensesHeaders/2069024:id/rat.zip%21/target/rat.txt
>> >> > [2] -
>> https://github.com/apache/ignite/blob/master/parent/pom.xml#L817
>> >> >
>> >> > Best Regards,
>> >> > Igor
>> >> >
>> >> >
>> >> > On Thu, Oct 11, 2018 at 1:13 PM Igor Sapego <is...@apache.org>
>> wrote:
>> >> >
>> >> >> Ok, I've filed a ticket [1] for the performance investigation and
>> >> targeted
>> >> >> it to 2.8. Feel free to assign it to yourself.
>> >> >>
>> >> >> I'm going to merge the ticket to master and ignite-2.7 branches by
>> the
>> >> >> end of the week, if no one have any objections.
>> >> >>
>> >> >> [1] - https://issues.apache.org/jira/browse/IGNITE-9850
>> >> >>
>> >> >> Best Regards,
>> >> >> Igor
>> >> >>
>> >> >>
>> >> >> On Thu, Oct 11, 2018 at 12:18 PM Dmitry Melnichuk <
>> >> >> dmitry.melnichuk@nobitlost.com> wrote:
>> >> >>
>> >> >>> Stepan,
>> >> >>>
>> >> >>> Thank you for answering my question and for the updated results.
>> >> >>>
>> >> >>> I have ran the profiler (cProfile) against the benchmark code you
>> >> >>> provided and have not found any particular bottleneck. One cycle
>> took
>> >> me
>> >> >>> 434 μs, which is the same order of magnitude as your result.
>> >> >>>
>> >> >>> I will need more time to analyze the performance of my client and
>> >> find a
>> >> >>> way to improve it. But is it a real stopper/blocker for the client
>> >> >>> itself? If there are no other issues, may we merge and release the
>> >> >>> client? Plus submit a jira-improvement to further investigate the
>> >> >>> performance of the Python platform.
>> >> >>>
>> >> >>> Dmitry
>> >> >>>
>> >> >>> On 10/11/18 4:19 PM, Степан Пильщиков wrote:
>> >> >>>> Dmitry,
>> >> >>>>
>> >> >>>> pip install -e from latest sources
>> >> >>>>
>> >> >>>> On Thu, 11 Oct 2018, 06:37 Dmitry Melnichuk, <
>> >> >>> dmitry.melnichuk@nobitlost.com>
>> >> >>>> wrote:
>> >> >>>>
>> >> >>>>> Stepan!
>> >> >>>>>
>> >> >>>>> Please tell me one thing about how you created the environment
>> for
>> >> your
>> >> >>>>> benchmarks, namely: how did you install the Python client. Did
>> you
>> >> just
>> >> >>>>> do `pip install pyignite`, or did you pull my working branch from
>> >> the
>> >> >>>>> downstream repository and did `pip install -e
>> >> >>>>> ignite/modules/platforms/python`?
>> >> >>>>>
>> >> >>>>> I highly recommend the latter, because PyPI version do not yet
>> >> include
>> >> >>>>> the latest improvements I did thanks to Dmitriy “qvad”
>> Sherstobitov.
>> >> >>>>> These improvements can have a noticeable positive effect on
>> speed.
>> >> >>>>>
>> >> >>>>> Dmitry
>> >> >>>>>
>> >> >>>>> On 10/10/18 11:06 PM, Степан Пильщиков wrote:
>> >> >>>>>> Dmitry,
>> >> >>>>>>
>> >> >>>>>> Thanks for review
>> >> >>>>>>
>> >> >>>>>> Agree with all suggestions.
>> >> >>>>>>
>> >> >>>>>> Made and run new benches without any redundant operations
>> (prints,
>> >> >>> time
>> >> >>>>>> calculation), only with "put" in "while true" cycle (almost)
>> >> >>>>>> Case description, environment, results and code in ticket
>> >> >>>>>> https://issues.apache.org/jira/browse/IGNITE-9824 (last
>> comment)
>> >> >>>>>>
>> >> >>>>>> Please review new metrics
>> >> >>>>>> Because picture is not changed so much, python still have
>> >> performance
>> >> >>>>> issues
>> >> >>>>>>
>> >> >>>>>
>> >> >>>>
>> >> >>>
>> >> >>>
>> >> >
>> >>
>> >>
>>
>
Re: Python thin client
Posted by Igor Sapego <is...@apache.org>.
Hi,
I've just merged Python thin client to the master and ignite-2.7
Also, I like how the python's results improved and now are comparable to
node.js.
Great job, guys!
Best Regards,
Igor
On Sat, Oct 13, 2018 at 3:53 PM Степан Пильщиков <pi...@gmail.com>
wrote:
> Hi
>
> Made new run and client definitely become faster
> I think its right direction for improvements
>
> Also rerun java bench with compiler=none option
>
> All results you can get in
> https://issues.apache.org/jira/browse/IGNITE-9824
> (last comment)
>
> сб, 13 окт. 2018 г. в 4:56, Pavel Petroshenko <pa...@petroshenko.com>:
>
> > Hi Stepan,
> >
> > Re your performance questions.
> >
> > Firstly, please pull out the latest sources from the ignite-7782 branch
> as
> > it has some performance optimizations recently implemented.
> >
> > Secondly, Hotspot is a pretty advanced dynamic adaptive compiler and it's
> > not fair to compare it to a purely interpreted Python language
> > implementation. To get comparable results, please run your Java benchmark
> > with -Djava.compiler=NONE. It turns the dynamic compiler off at runtime.
> I
> > just did this exercise myself and got comparable scores.
> >
> > Please let me know if you still see any significant difference with the
> > flag set.
> >
> > Thanks,
> > P.
> >
> > On Fri, Oct 12, 2018 at 10:02 AM Dmitry Melnichuk <
> > dmitry.melnichuk@nobitlost.com> wrote:
> >
> >> Igor,
> >>
> >> I fixed my files.
> >>
> >> Dmitry
> >>
> >> On 10/13/18 12:14 AM, Igor Sapego wrote:
> >> > Dmitry,
> >> >
> >> > I've run the tests and it appears that there are a lot of files
> without
> >> > Apache
> >> > licenses in Python client: [1] (See Unapproved licenses list).
> >> >
> >> > Can you either add headers or exclude file from the check, if you can
> >> not
> >> > add one? I.e. to this list: [2]
> >> >
> >> > [1] -
> >> >
> >>
> https://ci.ignite.apache.org/repository/download/IgniteTests24Java8_LicensesHeaders/2069024:id/rat.zip%21/target/rat.txt
> >> > [2] -
> https://github.com/apache/ignite/blob/master/parent/pom.xml#L817
> >> >
> >> > Best Regards,
> >> > Igor
> >> >
> >> >
> >> > On Thu, Oct 11, 2018 at 1:13 PM Igor Sapego <is...@apache.org>
> wrote:
> >> >
> >> >> Ok, I've filed a ticket [1] for the performance investigation and
> >> targeted
> >> >> it to 2.8. Feel free to assign it to yourself.
> >> >>
> >> >> I'm going to merge the ticket to master and ignite-2.7 branches by
> the
> >> >> end of the week, if no one have any objections.
> >> >>
> >> >> [1] - https://issues.apache.org/jira/browse/IGNITE-9850
> >> >>
> >> >> Best Regards,
> >> >> Igor
> >> >>
> >> >>
> >> >> On Thu, Oct 11, 2018 at 12:18 PM Dmitry Melnichuk <
> >> >> dmitry.melnichuk@nobitlost.com> wrote:
> >> >>
> >> >>> Stepan,
> >> >>>
> >> >>> Thank you for answering my question and for the updated results.
> >> >>>
> >> >>> I have ran the profiler (cProfile) against the benchmark code you
> >> >>> provided and have not found any particular bottleneck. One cycle
> took
> >> me
> >> >>> 434 μs, which is the same order of magnitude as your result.
> >> >>>
> >> >>> I will need more time to analyze the performance of my client and
> >> find a
> >> >>> way to improve it. But is it a real stopper/blocker for the client
> >> >>> itself? If there are no other issues, may we merge and release the
> >> >>> client? Plus submit a jira-improvement to further investigate the
> >> >>> performance of the Python platform.
> >> >>>
> >> >>> Dmitry
> >> >>>
> >> >>> On 10/11/18 4:19 PM, Степан Пильщиков wrote:
> >> >>>> Dmitry,
> >> >>>>
> >> >>>> pip install -e from latest sources
> >> >>>>
> >> >>>> On Thu, 11 Oct 2018, 06:37 Dmitry Melnichuk, <
> >> >>> dmitry.melnichuk@nobitlost.com>
> >> >>>> wrote:
> >> >>>>
> >> >>>>> Stepan!
> >> >>>>>
> >> >>>>> Please tell me one thing about how you created the environment for
> >> your
> >> >>>>> benchmarks, namely: how did you install the Python client. Did you
> >> just
> >> >>>>> do `pip install pyignite`, or did you pull my working branch from
> >> the
> >> >>>>> downstream repository and did `pip install -e
> >> >>>>> ignite/modules/platforms/python`?
> >> >>>>>
> >> >>>>> I highly recommend the latter, because PyPI version do not yet
> >> include
> >> >>>>> the latest improvements I did thanks to Dmitriy “qvad”
> Sherstobitov.
> >> >>>>> These improvements can have a noticeable positive effect on speed.
> >> >>>>>
> >> >>>>> Dmitry
> >> >>>>>
> >> >>>>> On 10/10/18 11:06 PM, Степан Пильщиков wrote:
> >> >>>>>> Dmitry,
> >> >>>>>>
> >> >>>>>> Thanks for review
> >> >>>>>>
> >> >>>>>> Agree with all suggestions.
> >> >>>>>>
> >> >>>>>> Made and run new benches without any redundant operations
> (prints,
> >> >>> time
> >> >>>>>> calculation), only with "put" in "while true" cycle (almost)
> >> >>>>>> Case description, environment, results and code in ticket
> >> >>>>>> https://issues.apache.org/jira/browse/IGNITE-9824 (last comment)
> >> >>>>>>
> >> >>>>>> Please review new metrics
> >> >>>>>> Because picture is not changed so much, python still have
> >> performance
> >> >>>>> issues
> >> >>>>>>
> >> >>>>>
> >> >>>>
> >> >>>
> >> >>>
> >> >
> >>
> >>
>
Re: Python thin client
Posted by Степан Пильщиков <pi...@gmail.com>.
Hi
Made new run and client definitely become faster
I think its right direction for improvements
Also rerun java bench with compiler=none option
All results you can get in https://issues.apache.org/jira/browse/IGNITE-9824
(last comment)
сб, 13 окт. 2018 г. в 4:56, Pavel Petroshenko <pa...@petroshenko.com>:
> Hi Stepan,
>
> Re your performance questions.
>
> Firstly, please pull out the latest sources from the ignite-7782 branch as
> it has some performance optimizations recently implemented.
>
> Secondly, Hotspot is a pretty advanced dynamic adaptive compiler and it's
> not fair to compare it to a purely interpreted Python language
> implementation. To get comparable results, please run your Java benchmark
> with -Djava.compiler=NONE. It turns the dynamic compiler off at runtime. I
> just did this exercise myself and got comparable scores.
>
> Please let me know if you still see any significant difference with the
> flag set.
>
> Thanks,
> P.
>
> On Fri, Oct 12, 2018 at 10:02 AM Dmitry Melnichuk <
> dmitry.melnichuk@nobitlost.com> wrote:
>
>> Igor,
>>
>> I fixed my files.
>>
>> Dmitry
>>
>> On 10/13/18 12:14 AM, Igor Sapego wrote:
>> > Dmitry,
>> >
>> > I've run the tests and it appears that there are a lot of files without
>> > Apache
>> > licenses in Python client: [1] (See Unapproved licenses list).
>> >
>> > Can you either add headers or exclude file from the check, if you can
>> not
>> > add one? I.e. to this list: [2]
>> >
>> > [1] -
>> >
>> https://ci.ignite.apache.org/repository/download/IgniteTests24Java8_LicensesHeaders/2069024:id/rat.zip%21/target/rat.txt
>> > [2] - https://github.com/apache/ignite/blob/master/parent/pom.xml#L817
>> >
>> > Best Regards,
>> > Igor
>> >
>> >
>> > On Thu, Oct 11, 2018 at 1:13 PM Igor Sapego <is...@apache.org> wrote:
>> >
>> >> Ok, I've filed a ticket [1] for the performance investigation and
>> targeted
>> >> it to 2.8. Feel free to assign it to yourself.
>> >>
>> >> I'm going to merge the ticket to master and ignite-2.7 branches by the
>> >> end of the week, if no one have any objections.
>> >>
>> >> [1] - https://issues.apache.org/jira/browse/IGNITE-9850
>> >>
>> >> Best Regards,
>> >> Igor
>> >>
>> >>
>> >> On Thu, Oct 11, 2018 at 12:18 PM Dmitry Melnichuk <
>> >> dmitry.melnichuk@nobitlost.com> wrote:
>> >>
>> >>> Stepan,
>> >>>
>> >>> Thank you for answering my question and for the updated results.
>> >>>
>> >>> I have ran the profiler (cProfile) against the benchmark code you
>> >>> provided and have not found any particular bottleneck. One cycle took
>> me
>> >>> 434 μs, which is the same order of magnitude as your result.
>> >>>
>> >>> I will need more time to analyze the performance of my client and
>> find a
>> >>> way to improve it. But is it a real stopper/blocker for the client
>> >>> itself? If there are no other issues, may we merge and release the
>> >>> client? Plus submit a jira-improvement to further investigate the
>> >>> performance of the Python platform.
>> >>>
>> >>> Dmitry
>> >>>
>> >>> On 10/11/18 4:19 PM, Степан Пильщиков wrote:
>> >>>> Dmitry,
>> >>>>
>> >>>> pip install -e from latest sources
>> >>>>
>> >>>> On Thu, 11 Oct 2018, 06:37 Dmitry Melnichuk, <
>> >>> dmitry.melnichuk@nobitlost.com>
>> >>>> wrote:
>> >>>>
>> >>>>> Stepan!
>> >>>>>
>> >>>>> Please tell me one thing about how you created the environment for
>> your
>> >>>>> benchmarks, namely: how did you install the Python client. Did you
>> just
>> >>>>> do `pip install pyignite`, or did you pull my working branch from
>> the
>> >>>>> downstream repository and did `pip install -e
>> >>>>> ignite/modules/platforms/python`?
>> >>>>>
>> >>>>> I highly recommend the latter, because PyPI version do not yet
>> include
>> >>>>> the latest improvements I did thanks to Dmitriy “qvad” Sherstobitov.
>> >>>>> These improvements can have a noticeable positive effect on speed.
>> >>>>>
>> >>>>> Dmitry
>> >>>>>
>> >>>>> On 10/10/18 11:06 PM, Степан Пильщиков wrote:
>> >>>>>> Dmitry,
>> >>>>>>
>> >>>>>> Thanks for review
>> >>>>>>
>> >>>>>> Agree with all suggestions.
>> >>>>>>
>> >>>>>> Made and run new benches without any redundant operations (prints,
>> >>> time
>> >>>>>> calculation), only with "put" in "while true" cycle (almost)
>> >>>>>> Case description, environment, results and code in ticket
>> >>>>>> https://issues.apache.org/jira/browse/IGNITE-9824 (last comment)
>> >>>>>>
>> >>>>>> Please review new metrics
>> >>>>>> Because picture is not changed so much, python still have
>> performance
>> >>>>> issues
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>>
>> >
>>
>>
Re: Python thin client
Posted by Pavel Petroshenko <pa...@petroshenko.com>.
Hi Stepan,
Re your performance questions.
Firstly, please pull out the latest sources from the ignite-7782 branch as
it has some performance optimizations recently implemented.
Secondly, Hotspot is a pretty advanced dynamic adaptive compiler and it's
not fair to compare it to a purely interpreted Python language
implementation. To get comparable results, please run your Java benchmark
with -Djava.compiler=NONE. It turns the dynamic compiler off at runtime. I
just did this exercise myself and got comparable scores.
Please let me know if you still see any significant difference with the
flag set.
Thanks,
P.
On Fri, Oct 12, 2018 at 10:02 AM Dmitry Melnichuk <
dmitry.melnichuk@nobitlost.com> wrote:
> Igor,
>
> I fixed my files.
>
> Dmitry
>
> On 10/13/18 12:14 AM, Igor Sapego wrote:
> > Dmitry,
> >
> > I've run the tests and it appears that there are a lot of files without
> > Apache
> > licenses in Python client: [1] (See Unapproved licenses list).
> >
> > Can you either add headers or exclude file from the check, if you can not
> > add one? I.e. to this list: [2]
> >
> > [1] -
> >
> https://ci.ignite.apache.org/repository/download/IgniteTests24Java8_LicensesHeaders/2069024:id/rat.zip%21/target/rat.txt
> > [2] - https://github.com/apache/ignite/blob/master/parent/pom.xml#L817
> >
> > Best Regards,
> > Igor
> >
> >
> > On Thu, Oct 11, 2018 at 1:13 PM Igor Sapego <is...@apache.org> wrote:
> >
> >> Ok, I've filed a ticket [1] for the performance investigation and
> targeted
> >> it to 2.8. Feel free to assign it to yourself.
> >>
> >> I'm going to merge the ticket to master and ignite-2.7 branches by the
> >> end of the week, if no one have any objections.
> >>
> >> [1] - https://issues.apache.org/jira/browse/IGNITE-9850
> >>
> >> Best Regards,
> >> Igor
> >>
> >>
> >> On Thu, Oct 11, 2018 at 12:18 PM Dmitry Melnichuk <
> >> dmitry.melnichuk@nobitlost.com> wrote:
> >>
> >>> Stepan,
> >>>
> >>> Thank you for answering my question and for the updated results.
> >>>
> >>> I have ran the profiler (cProfile) against the benchmark code you
> >>> provided and have not found any particular bottleneck. One cycle took
> me
> >>> 434 μs, which is the same order of magnitude as your result.
> >>>
> >>> I will need more time to analyze the performance of my client and find
> a
> >>> way to improve it. But is it a real stopper/blocker for the client
> >>> itself? If there are no other issues, may we merge and release the
> >>> client? Plus submit a jira-improvement to further investigate the
> >>> performance of the Python platform.
> >>>
> >>> Dmitry
> >>>
> >>> On 10/11/18 4:19 PM, Степан Пильщиков wrote:
> >>>> Dmitry,
> >>>>
> >>>> pip install -e from latest sources
> >>>>
> >>>> On Thu, 11 Oct 2018, 06:37 Dmitry Melnichuk, <
> >>> dmitry.melnichuk@nobitlost.com>
> >>>> wrote:
> >>>>
> >>>>> Stepan!
> >>>>>
> >>>>> Please tell me one thing about how you created the environment for
> your
> >>>>> benchmarks, namely: how did you install the Python client. Did you
> just
> >>>>> do `pip install pyignite`, or did you pull my working branch from the
> >>>>> downstream repository and did `pip install -e
> >>>>> ignite/modules/platforms/python`?
> >>>>>
> >>>>> I highly recommend the latter, because PyPI version do not yet
> include
> >>>>> the latest improvements I did thanks to Dmitriy “qvad” Sherstobitov.
> >>>>> These improvements can have a noticeable positive effect on speed.
> >>>>>
> >>>>> Dmitry
> >>>>>
> >>>>> On 10/10/18 11:06 PM, Степан Пильщиков wrote:
> >>>>>> Dmitry,
> >>>>>>
> >>>>>> Thanks for review
> >>>>>>
> >>>>>> Agree with all suggestions.
> >>>>>>
> >>>>>> Made and run new benches without any redundant operations (prints,
> >>> time
> >>>>>> calculation), only with "put" in "while true" cycle (almost)
> >>>>>> Case description, environment, results and code in ticket
> >>>>>> https://issues.apache.org/jira/browse/IGNITE-9824 (last comment)
> >>>>>>
> >>>>>> Please review new metrics
> >>>>>> Because picture is not changed so much, python still have
> performance
> >>>>> issues
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >
>
>
Re: Python thin client
Posted by Dmitry Melnichuk <dm...@nobitlost.com>.
Igor,
I fixed my files.
Dmitry
On 10/13/18 12:14 AM, Igor Sapego wrote:
> Dmitry,
>
> I've run the tests and it appears that there are a lot of files without
> Apache
> licenses in Python client: [1] (See Unapproved licenses list).
>
> Can you either add headers or exclude file from the check, if you can not
> add one? I.e. to this list: [2]
>
> [1] -
> https://ci.ignite.apache.org/repository/download/IgniteTests24Java8_LicensesHeaders/2069024:id/rat.zip%21/target/rat.txt
> [2] - https://github.com/apache/ignite/blob/master/parent/pom.xml#L817
>
> Best Regards,
> Igor
>
>
> On Thu, Oct 11, 2018 at 1:13 PM Igor Sapego <is...@apache.org> wrote:
>
>> Ok, I've filed a ticket [1] for the performance investigation and targeted
>> it to 2.8. Feel free to assign it to yourself.
>>
>> I'm going to merge the ticket to master and ignite-2.7 branches by the
>> end of the week, if no one have any objections.
>>
>> [1] - https://issues.apache.org/jira/browse/IGNITE-9850
>>
>> Best Regards,
>> Igor
>>
>>
>> On Thu, Oct 11, 2018 at 12:18 PM Dmitry Melnichuk <
>> dmitry.melnichuk@nobitlost.com> wrote:
>>
>>> Stepan,
>>>
>>> Thank you for answering my question and for the updated results.
>>>
>>> I have ran the profiler (cProfile) against the benchmark code you
>>> provided and have not found any particular bottleneck. One cycle took me
>>> 434 μs, which is the same order of magnitude as your result.
>>>
>>> I will need more time to analyze the performance of my client and find a
>>> way to improve it. But is it a real stopper/blocker for the client
>>> itself? If there are no other issues, may we merge and release the
>>> client? Plus submit a jira-improvement to further investigate the
>>> performance of the Python platform.
>>>
>>> Dmitry
>>>
>>> On 10/11/18 4:19 PM, Степан Пильщиков wrote:
>>>> Dmitry,
>>>>
>>>> pip install -e from latest sources
>>>>
>>>> On Thu, 11 Oct 2018, 06:37 Dmitry Melnichuk, <
>>> dmitry.melnichuk@nobitlost.com>
>>>> wrote:
>>>>
>>>>> Stepan!
>>>>>
>>>>> Please tell me one thing about how you created the environment for your
>>>>> benchmarks, namely: how did you install the Python client. Did you just
>>>>> do `pip install pyignite`, or did you pull my working branch from the
>>>>> downstream repository and did `pip install -e
>>>>> ignite/modules/platforms/python`?
>>>>>
>>>>> I highly recommend the latter, because PyPI version do not yet include
>>>>> the latest improvements I did thanks to Dmitriy “qvad” Sherstobitov.
>>>>> These improvements can have a noticeable positive effect on speed.
>>>>>
>>>>> Dmitry
>>>>>
>>>>> On 10/10/18 11:06 PM, Степан Пильщиков wrote:
>>>>>> Dmitry,
>>>>>>
>>>>>> Thanks for review
>>>>>>
>>>>>> Agree with all suggestions.
>>>>>>
>>>>>> Made and run new benches without any redundant operations (prints,
>>> time
>>>>>> calculation), only with "put" in "while true" cycle (almost)
>>>>>> Case description, environment, results and code in ticket
>>>>>> https://issues.apache.org/jira/browse/IGNITE-9824 (last comment)
>>>>>>
>>>>>> Please review new metrics
>>>>>> Because picture is not changed so much, python still have performance
>>>>> issues
>>>>>>
>>>>>
>>>>
>>>
>>>
>
Re: Python thin client
Posted by Igor Sapego <is...@apache.org>.
Dmitry,
I've run the tests and it appears that there are a lot of files without
Apache
licenses in Python client: [1] (See Unapproved licenses list).
Can you either add headers or exclude file from the check, if you can not
add one? I.e. to this list: [2]
[1] -
https://ci.ignite.apache.org/repository/download/IgniteTests24Java8_LicensesHeaders/2069024:id/rat.zip%21/target/rat.txt
[2] - https://github.com/apache/ignite/blob/master/parent/pom.xml#L817
Best Regards,
Igor
On Thu, Oct 11, 2018 at 1:13 PM Igor Sapego <is...@apache.org> wrote:
> Ok, I've filed a ticket [1] for the performance investigation and targeted
> it to 2.8. Feel free to assign it to yourself.
>
> I'm going to merge the ticket to master and ignite-2.7 branches by the
> end of the week, if no one have any objections.
>
> [1] - https://issues.apache.org/jira/browse/IGNITE-9850
>
> Best Regards,
> Igor
>
>
> On Thu, Oct 11, 2018 at 12:18 PM Dmitry Melnichuk <
> dmitry.melnichuk@nobitlost.com> wrote:
>
>> Stepan,
>>
>> Thank you for answering my question and for the updated results.
>>
>> I have ran the profiler (cProfile) against the benchmark code you
>> provided and have not found any particular bottleneck. One cycle took me
>> 434 μs, which is the same order of magnitude as your result.
>>
>> I will need more time to analyze the performance of my client and find a
>> way to improve it. But is it a real stopper/blocker for the client
>> itself? If there are no other issues, may we merge and release the
>> client? Plus submit a jira-improvement to further investigate the
>> performance of the Python platform.
>>
>> Dmitry
>>
>> On 10/11/18 4:19 PM, Степан Пильщиков wrote:
>> > Dmitry,
>> >
>> > pip install -e from latest sources
>> >
>> > On Thu, 11 Oct 2018, 06:37 Dmitry Melnichuk, <
>> dmitry.melnichuk@nobitlost.com>
>> > wrote:
>> >
>> >> Stepan!
>> >>
>> >> Please tell me one thing about how you created the environment for your
>> >> benchmarks, namely: how did you install the Python client. Did you just
>> >> do `pip install pyignite`, or did you pull my working branch from the
>> >> downstream repository and did `pip install -e
>> >> ignite/modules/platforms/python`?
>> >>
>> >> I highly recommend the latter, because PyPI version do not yet include
>> >> the latest improvements I did thanks to Dmitriy “qvad” Sherstobitov.
>> >> These improvements can have a noticeable positive effect on speed.
>> >>
>> >> Dmitry
>> >>
>> >> On 10/10/18 11:06 PM, Степан Пильщиков wrote:
>> >>> Dmitry,
>> >>>
>> >>> Thanks for review
>> >>>
>> >>> Agree with all suggestions.
>> >>>
>> >>> Made and run new benches without any redundant operations (prints,
>> time
>> >>> calculation), only with "put" in "while true" cycle (almost)
>> >>> Case description, environment, results and code in ticket
>> >>> https://issues.apache.org/jira/browse/IGNITE-9824 (last comment)
>> >>>
>> >>> Please review new metrics
>> >>> Because picture is not changed so much, python still have performance
>> >> issues
>> >>>
>> >>
>> >
>>
>>
Re: Python thin client
Posted by Igor Sapego <is...@apache.org>.
Ok, I've filed a ticket [1] for the performance investigation and targeted
it to 2.8. Feel free to assign it to yourself.
I'm going to merge the ticket to master and ignite-2.7 branches by the
end of the week, if no one have any objections.
[1] - https://issues.apache.org/jira/browse/IGNITE-9850
Best Regards,
Igor
On Thu, Oct 11, 2018 at 12:18 PM Dmitry Melnichuk <
dmitry.melnichuk@nobitlost.com> wrote:
> Stepan,
>
> Thank you for answering my question and for the updated results.
>
> I have ran the profiler (cProfile) against the benchmark code you
> provided and have not found any particular bottleneck. One cycle took me
> 434 μs, which is the same order of magnitude as your result.
>
> I will need more time to analyze the performance of my client and find a
> way to improve it. But is it a real stopper/blocker for the client
> itself? If there are no other issues, may we merge and release the
> client? Plus submit a jira-improvement to further investigate the
> performance of the Python platform.
>
> Dmitry
>
> On 10/11/18 4:19 PM, Степан Пильщиков wrote:
> > Dmitry,
> >
> > pip install -e from latest sources
> >
> > On Thu, 11 Oct 2018, 06:37 Dmitry Melnichuk, <
> dmitry.melnichuk@nobitlost.com>
> > wrote:
> >
> >> Stepan!
> >>
> >> Please tell me one thing about how you created the environment for your
> >> benchmarks, namely: how did you install the Python client. Did you just
> >> do `pip install pyignite`, or did you pull my working branch from the
> >> downstream repository and did `pip install -e
> >> ignite/modules/platforms/python`?
> >>
> >> I highly recommend the latter, because PyPI version do not yet include
> >> the latest improvements I did thanks to Dmitriy “qvad” Sherstobitov.
> >> These improvements can have a noticeable positive effect on speed.
> >>
> >> Dmitry
> >>
> >> On 10/10/18 11:06 PM, Степан Пильщиков wrote:
> >>> Dmitry,
> >>>
> >>> Thanks for review
> >>>
> >>> Agree with all suggestions.
> >>>
> >>> Made and run new benches without any redundant operations (prints, time
> >>> calculation), only with "put" in "while true" cycle (almost)
> >>> Case description, environment, results and code in ticket
> >>> https://issues.apache.org/jira/browse/IGNITE-9824 (last comment)
> >>>
> >>> Please review new metrics
> >>> Because picture is not changed so much, python still have performance
> >> issues
> >>>
> >>
> >
>
>
Re: Python thin client
Posted by Dmitry Melnichuk <dm...@nobitlost.com>.
Stepan,
Thank you for answering my question and for the updated results.
I have ran the profiler (cProfile) against the benchmark code you
provided and have not found any particular bottleneck. One cycle took me
434 μs, which is the same order of magnitude as your result.
I will need more time to analyze the performance of my client and find a
way to improve it. But is it a real stopper/blocker for the client
itself? If there are no other issues, may we merge and release the
client? Plus submit a jira-improvement to further investigate the
performance of the Python platform.
Dmitry
On 10/11/18 4:19 PM, Степан Пильщиков wrote:
> Dmitry,
>
> pip install -e from latest sources
>
> On Thu, 11 Oct 2018, 06:37 Dmitry Melnichuk, <dm...@nobitlost.com>
> wrote:
>
>> Stepan!
>>
>> Please tell me one thing about how you created the environment for your
>> benchmarks, namely: how did you install the Python client. Did you just
>> do `pip install pyignite`, or did you pull my working branch from the
>> downstream repository and did `pip install -e
>> ignite/modules/platforms/python`?
>>
>> I highly recommend the latter, because PyPI version do not yet include
>> the latest improvements I did thanks to Dmitriy “qvad” Sherstobitov.
>> These improvements can have a noticeable positive effect on speed.
>>
>> Dmitry
>>
>> On 10/10/18 11:06 PM, Степан Пильщиков wrote:
>>> Dmitry,
>>>
>>> Thanks for review
>>>
>>> Agree with all suggestions.
>>>
>>> Made and run new benches without any redundant operations (prints, time
>>> calculation), only with "put" in "while true" cycle (almost)
>>> Case description, environment, results and code in ticket
>>> https://issues.apache.org/jira/browse/IGNITE-9824 (last comment)
>>>
>>> Please review new metrics
>>> Because picture is not changed so much, python still have performance
>> issues
>>>
>>
>
Re: Python thin client
Posted by Степан Пильщиков <pi...@gmail.com>.
Dmitry,
pip install -e from latest sources
On Thu, 11 Oct 2018, 06:37 Dmitry Melnichuk, <dm...@nobitlost.com>
wrote:
> Stepan!
>
> Please tell me one thing about how you created the environment for your
> benchmarks, namely: how did you install the Python client. Did you just
> do `pip install pyignite`, or did you pull my working branch from the
> downstream repository and did `pip install -e
> ignite/modules/platforms/python`?
>
> I highly recommend the latter, because PyPI version do not yet include
> the latest improvements I did thanks to Dmitriy “qvad” Sherstobitov.
> These improvements can have a noticeable positive effect on speed.
>
> Dmitry
>
> On 10/10/18 11:06 PM, Степан Пильщиков wrote:
> > Dmitry,
> >
> > Thanks for review
> >
> > Agree with all suggestions.
> >
> > Made and run new benches without any redundant operations (prints, time
> > calculation), only with "put" in "while true" cycle (almost)
> > Case description, environment, results and code in ticket
> > https://issues.apache.org/jira/browse/IGNITE-9824 (last comment)
> >
> > Please review new metrics
> > Because picture is not changed so much, python still have performance
> issues
> >
>
Re: Python thin client
Posted by Dmitry Melnichuk <dm...@nobitlost.com>.
Stepan!
Please tell me one thing about how you created the environment for your
benchmarks, namely: how did you install the Python client. Did you just
do `pip install pyignite`, or did you pull my working branch from the
downstream repository and did `pip install -e
ignite/modules/platforms/python`?
I highly recommend the latter, because PyPI version do not yet include
the latest improvements I did thanks to Dmitriy “qvad” Sherstobitov.
These improvements can have a noticeable positive effect on speed.
Dmitry
On 10/10/18 11:06 PM, Степан Пильщиков wrote:
> Dmitry,
>
> Thanks for review
>
> Agree with all suggestions.
>
> Made and run new benches without any redundant operations (prints, time
> calculation), only with "put" in "while true" cycle (almost)
> Case description, environment, results and code in ticket
> https://issues.apache.org/jira/browse/IGNITE-9824 (last comment)
>
> Please review new metrics
> Because picture is not changed so much, python still have performance issues
>
Re: Python thin client
Posted by Степан Пильщиков <pi...@gmail.com>.
Dmitry,
Thanks for review
Agree with all suggestions.
Made and run new benches without any redundant operations (prints, time
calculation), only with "put" in "while true" cycle (almost)
Case description, environment, results and code in ticket
https://issues.apache.org/jira/browse/IGNITE-9824 (last comment)
Please review new metrics
Because picture is not changed so much, python still have performance issues
ср, 10 окт. 2018 г. в 12:30, Dmitry Melnichuk <
dmitry.melnichuk@nobitlost.com>:
> Stepan!
>
> Can you please update the benchmarks based on my suggestions earlier in
> this thread: use a cleaner time profiling for the loop, remove
> unnecessary operations (string formatting, rounding operations), stick
> to the primitive int value for put operations (agree with Vladimir,
> let's keep it simple). And let me know the results.
>
> On 10/10/18 5:46 PM, Vladimir Ozerov wrote:
> > Hi Dmitry,
> >
> > I agree with your comments on benchmarking code. As more fair
> > alternative we may measure time of loading N elements into the cache, so
> > that it will be necessary to call time() only twice. However, provided
> > that we have real network here, and according to numbers one PUT in
> > Python client requires about 0.3 msec, I hardly can imagine that call to
> > time() or round() may dominate in that response time. But I said, we can
> > easily rule that out by slight benchmark rewrite.
> >
> > As far as serialization, I would prefer to keep primitive objects at the
> > moment, because AFAIK the purpose of the benchmark was to assess
> > underlying infrastructure, looking for some obvious performance issues.
> > Every platform have sockets which use more or less the same API of
> > underlying OS. To the contrast, performance of serialization mechanics
> > may differ widely between platforms depending on implementation. E.g. in
> > Java we spend a lot of time on fine-grained tuning, applied a lot of
> > speculative optimizations. Add to this JIT nature of Java, and you will
> > hardly ever beat Java serialization engine from any interpreted
> > language. So primitive data types make good sense to me.
> >
> > At this point our goal is not make Python equally fast with other
> > platforms, but rather to understand why is it slower than others. Ignite
> > is a product which brings speed to end users. If they do no have speed,
> > they will not use Ignite. So performance questions are always of great
> > importance for us.
> >
> > Vladimir.
> >
> > On Wed, Oct 10, 2018 at 9:57 AM Dmitry Melnichuk
> > <dmitry.melnichuk@nobitlost.com <ma...@nobitlost.com>>
>
> > wrote:
> >
> > Hi, Stepan!
> >
> > I looked at the benchmark code and the overall methodics, discussed
> it
> > with fellow programmers, and came up with basically two remarks.
> >
> > First of all, I think, the key for the good (or, at least,
> > unobjectionable) measurement is to isolate the object being measured
> > from the influence of the measurement tool. The usual pattern we use
> in
> > Python looks like:
> >
> > ```
> > from time import time
> >
> >
> > bench_start = time()
> >
> > for _ in range(number_of_tests):
> > do_test()
> >
> > bench_end = time()
> >
> > print('Performance is {} tests per second'.format(
> > number_of_tests / (bench_end - bench_start)
> > ))
> > ```
> >
> > I think, you got the idea: the measurement consists almost solely of
> > the
> > time taken by our subject function `do_test`. As few other code as
> > possible influence the result.
> >
> > Now, let's take a look at your code:
> >
> > https://gist.github.com/pilshchikov/8aff4e30d83f8bac20c5a4a9c3917abb
> >
> > Ideally, the `while` loop should include only `cache.put(last_key,
> > some_precomputed_value)`. But instead it also includes:
> >
> > - a series of the `time()` calls, which could be mostly excluded from
> > the measured time, if the measurement done right; each call is
> probably
> > addresses the HPET device, or network time, or both,
> >
> > - some floating-point calculations, including `round()`, which is
> > hardly
> > necessary,
> >
> > - formatting and output of the intermediate result.
> >
> > I suppose the measurement influence can be quite significant here,
> but
> > it is at least more or less constant for each test.
> >
> > But if we look at the other benchmarks:
> >
> > https://gist.github.com/pilshchikov/8a4bdb03a8304136c22c9bf7217ee447
> > [Node.js]
> > https://gist.github.com/pilshchikov/b4351d78ad59e9cd923689c2e387bc80
> > [PHP]
> > https://gist.github.com/pilshchikov/08096c78b425e00166a2ffa2aa5f49ce
> > [Java]
> >
> > The extra code that influence the measurement is not equivalent
> across
> > all platforms. For example, PHP's `time()` is most probably lighter
> > than
> > Python `time()`, since it do not give out milliseconds and may
> address
> > RTC, not HPET. So the platform-to-platform comparison in your
> benchmark
> > does not look completely fair to me.
> >
> > The second remark concerns not the measurement procedure, but the
> > object
> > being measured.
> >
> > The only client operation being used is OP_CACHE_PUT with a payload
> > of a
> > primitive type. (BTW the type is `Long` in case of the Python client;
> > what about the other clients? Is it `Int`?) I afraid that such an
> > object, even being properly isolated from the measurement tool, would
> > show mostly the throughput of the underlying platform's sockets
> > implementation, not the performance of the client's code. To show the
> > potential of the thin client itself, more varied and fancy tasks
> > needed,
> > i. e. serialization/deserialization of the Complex objects.
> >
> > But it depends on the goal of the benchmarking. If showing off the
> raw
> > platform agility was intended, than this objection is removed.
> >
> > Dmitry
> >
> > On 10/9/18 10:50 PM, Stepan Pilshchikov wrote:
> > > Hi, all
> > >
> > > Tried to compare new thin client performance and made similar
> > benchmarks for
> > > each one
> > > And result not so good for python
> > >
> > > Ticket with results and bench code:
> > > https://issues.apache.org/jira/browse/IGNITE-9824
> > >
> > > Py code src:
> > >
> https://gist.github.com/pilshchikov/8aff4e30d83f8bac20c5a4a9c3917abb
> > >
> > > Dmitry, please review results and bench code, maybe somthing
> > wrong or it's
> > > expected numbers?
> > >
> > >
> > >
> > > --
> > > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
> > >
> >
>
>
Re: Python thin client
Posted by Dmitry Melnichuk <dm...@nobitlost.com>.
Stepan!
Can you please update the benchmarks based on my suggestions earlier in
this thread: use a cleaner time profiling for the loop, remove
unnecessary operations (string formatting, rounding operations), stick
to the primitive int value for put operations (agree with Vladimir,
let's keep it simple). And let me know the results.
On 10/10/18 5:46 PM, Vladimir Ozerov wrote:
> Hi Dmitry,
>
> I agree with your comments on benchmarking code. As more fair
> alternative we may measure time of loading N elements into the cache, so
> that it will be necessary to call time() only twice. However, provided
> that we have real network here, and according to numbers one PUT in
> Python client requires about 0.3 msec, I hardly can imagine that call to
> time() or round() may dominate in that response time. But I said, we can
> easily rule that out by slight benchmark rewrite.
>
> As far as serialization, I would prefer to keep primitive objects at the
> moment, because AFAIK the purpose of the benchmark was to assess
> underlying infrastructure, looking for some obvious performance issues.
> Every platform have sockets which use more or less the same API of
> underlying OS. To the contrast, performance of serialization mechanics
> may differ widely between platforms depending on implementation. E.g. in
> Java we spend a lot of time on fine-grained tuning, applied a lot of
> speculative optimizations. Add to this JIT nature of Java, and you will
> hardly ever beat Java serialization engine from any interpreted
> language. So primitive data types make good sense to me.
>
> At this point our goal is not make Python equally fast with other
> platforms, but rather to understand why is it slower than others. Ignite
> is a product which brings speed to end users. If they do no have speed,
> they will not use Ignite. So performance questions are always of great
> importance for us.
>
> Vladimir.
>
> On Wed, Oct 10, 2018 at 9:57 AM Dmitry Melnichuk
> <dmitry.melnichuk@nobitlost.com <ma...@nobitlost.com>>
> wrote:
>
> Hi, Stepan!
>
> I looked at the benchmark code and the overall methodics, discussed it
> with fellow programmers, and came up with basically two remarks.
>
> First of all, I think, the key for the good (or, at least,
> unobjectionable) measurement is to isolate the object being measured
> from the influence of the measurement tool. The usual pattern we use in
> Python looks like:
>
> ```
> from time import time
>
>
> bench_start = time()
>
> for _ in range(number_of_tests):
> do_test()
>
> bench_end = time()
>
> print('Performance is {} tests per second'.format(
> number_of_tests / (bench_end - bench_start)
> ))
> ```
>
> I think, you got the idea: the measurement consists almost solely of
> the
> time taken by our subject function `do_test`. As few other code as
> possible influence the result.
>
> Now, let's take a look at your code:
>
> https://gist.github.com/pilshchikov/8aff4e30d83f8bac20c5a4a9c3917abb
>
> Ideally, the `while` loop should include only `cache.put(last_key,
> some_precomputed_value)`. But instead it also includes:
>
> - a series of the `time()` calls, which could be mostly excluded from
> the measured time, if the measurement done right; each call is probably
> addresses the HPET device, or network time, or both,
>
> - some floating-point calculations, including `round()`, which is
> hardly
> necessary,
>
> - formatting and output of the intermediate result.
>
> I suppose the measurement influence can be quite significant here, but
> it is at least more or less constant for each test.
>
> But if we look at the other benchmarks:
>
> https://gist.github.com/pilshchikov/8a4bdb03a8304136c22c9bf7217ee447
> [Node.js]
> https://gist.github.com/pilshchikov/b4351d78ad59e9cd923689c2e387bc80
> [PHP]
> https://gist.github.com/pilshchikov/08096c78b425e00166a2ffa2aa5f49ce
> [Java]
>
> The extra code that influence the measurement is not equivalent across
> all platforms. For example, PHP's `time()` is most probably lighter
> than
> Python `time()`, since it do not give out milliseconds and may address
> RTC, not HPET. So the platform-to-platform comparison in your benchmark
> does not look completely fair to me.
>
> The second remark concerns not the measurement procedure, but the
> object
> being measured.
>
> The only client operation being used is OP_CACHE_PUT with a payload
> of a
> primitive type. (BTW the type is `Long` in case of the Python client;
> what about the other clients? Is it `Int`?) I afraid that such an
> object, even being properly isolated from the measurement tool, would
> show mostly the throughput of the underlying platform's sockets
> implementation, not the performance of the client's code. To show the
> potential of the thin client itself, more varied and fancy tasks
> needed,
> i. e. serialization/deserialization of the Complex objects.
>
> But it depends on the goal of the benchmarking. If showing off the raw
> platform agility was intended, than this objection is removed.
>
> Dmitry
>
> On 10/9/18 10:50 PM, Stepan Pilshchikov wrote:
> > Hi, all
> >
> > Tried to compare new thin client performance and made similar
> benchmarks for
> > each one
> > And result not so good for python
> >
> > Ticket with results and bench code:
> > https://issues.apache.org/jira/browse/IGNITE-9824
> >
> > Py code src:
> > https://gist.github.com/pilshchikov/8aff4e30d83f8bac20c5a4a9c3917abb
> >
> > Dmitry, please review results and bench code, maybe somthing
> wrong or it's
> > expected numbers?
> >
> >
> >
> > --
> > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
> >
>
Re: Python thin client
Posted by Vladimir Ozerov <vo...@gridgain.com>.
Hi Dmitry,
I agree with your comments on benchmarking code. As more fair alternative
we may measure time of loading N elements into the cache, so that it will
be necessary to call time() only twice. However, provided that we have real
network here, and according to numbers one PUT in Python client requires
about 0.3 msec, I hardly can imagine that call to time() or round() may
dominate in that response time. But I said, we can easily rule that out by
slight benchmark rewrite.
As far as serialization, I would prefer to keep primitive objects at the
moment, because AFAIK the purpose of the benchmark was to assess underlying
infrastructure, looking for some obvious performance issues. Every platform
have sockets which use more or less the same API of underlying OS. To the
contrast, performance of serialization mechanics may differ widely between
platforms depending on implementation. E.g. in Java we spend a lot of time
on fine-grained tuning, applied a lot of speculative optimizations. Add to
this JIT nature of Java, and you will hardly ever beat Java serialization
engine from any interpreted language. So primitive data types make good
sense to me.
At this point our goal is not make Python equally fast with other
platforms, but rather to understand why is it slower than others. Ignite is
a product which brings speed to end users. If they do no have speed, they
will not use Ignite. So performance questions are always of great
importance for us.
Vladimir.
On Wed, Oct 10, 2018 at 9:57 AM Dmitry Melnichuk <
dmitry.melnichuk@nobitlost.com> wrote:
> Hi, Stepan!
>
> I looked at the benchmark code and the overall methodics, discussed it
> with fellow programmers, and came up with basically two remarks.
>
> First of all, I think, the key for the good (or, at least,
> unobjectionable) measurement is to isolate the object being measured
> from the influence of the measurement tool. The usual pattern we use in
> Python looks like:
>
> ```
> from time import time
>
>
> bench_start = time()
>
> for _ in range(number_of_tests):
> do_test()
>
> bench_end = time()
>
> print('Performance is {} tests per second'.format(
> number_of_tests / (bench_end - bench_start)
> ))
> ```
>
> I think, you got the idea: the measurement consists almost solely of the
> time taken by our subject function `do_test`. As few other code as
> possible influence the result.
>
> Now, let's take a look at your code:
>
> https://gist.github.com/pilshchikov/8aff4e30d83f8bac20c5a4a9c3917abb
>
> Ideally, the `while` loop should include only `cache.put(last_key,
> some_precomputed_value)`. But instead it also includes:
>
> - a series of the `time()` calls, which could be mostly excluded from
> the measured time, if the measurement done right; each call is probably
> addresses the HPET device, or network time, or both,
>
> - some floating-point calculations, including `round()`, which is hardly
> necessary,
>
> - formatting and output of the intermediate result.
>
> I suppose the measurement influence can be quite significant here, but
> it is at least more or less constant for each test.
>
> But if we look at the other benchmarks:
>
> https://gist.github.com/pilshchikov/8a4bdb03a8304136c22c9bf7217ee447
> [Node.js]
> https://gist.github.com/pilshchikov/b4351d78ad59e9cd923689c2e387bc80 [PHP]
> https://gist.github.com/pilshchikov/08096c78b425e00166a2ffa2aa5f49ce
> [Java]
>
> The extra code that influence the measurement is not equivalent across
> all platforms. For example, PHP's `time()` is most probably lighter than
> Python `time()`, since it do not give out milliseconds and may address
> RTC, not HPET. So the platform-to-platform comparison in your benchmark
> does not look completely fair to me.
>
> The second remark concerns not the measurement procedure, but the object
> being measured.
>
> The only client operation being used is OP_CACHE_PUT with a payload of a
> primitive type. (BTW the type is `Long` in case of the Python client;
> what about the other clients? Is it `Int`?) I afraid that such an
> object, even being properly isolated from the measurement tool, would
> show mostly the throughput of the underlying platform's sockets
> implementation, not the performance of the client's code. To show the
> potential of the thin client itself, more varied and fancy tasks needed,
> i. e. serialization/deserialization of the Complex objects.
>
> But it depends on the goal of the benchmarking. If showing off the raw
> platform agility was intended, than this objection is removed.
>
> Dmitry
>
> On 10/9/18 10:50 PM, Stepan Pilshchikov wrote:
> > Hi, all
> >
> > Tried to compare new thin client performance and made similar benchmarks
> for
> > each one
> > And result not so good for python
> >
> > Ticket with results and bench code:
> > https://issues.apache.org/jira/browse/IGNITE-9824
> >
> > Py code src:
> > https://gist.github.com/pilshchikov/8aff4e30d83f8bac20c5a4a9c3917abb
> >
> > Dmitry, please review results and bench code, maybe somthing wrong or
> it's
> > expected numbers?
> >
> >
> >
> > --
> > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
> >
>
>
Re: Python thin client
Posted by Dmitry Melnichuk <dm...@nobitlost.com>.
Hi, Stepan!
I looked at the benchmark code and the overall methodics, discussed it
with fellow programmers, and came up with basically two remarks.
First of all, I think, the key for the good (or, at least,
unobjectionable) measurement is to isolate the object being measured
from the influence of the measurement tool. The usual pattern we use in
Python looks like:
```
from time import time
bench_start = time()
for _ in range(number_of_tests):
do_test()
bench_end = time()
print('Performance is {} tests per second'.format(
number_of_tests / (bench_end - bench_start)
))
```
I think, you got the idea: the measurement consists almost solely of the
time taken by our subject function `do_test`. As few other code as
possible influence the result.
Now, let's take a look at your code:
https://gist.github.com/pilshchikov/8aff4e30d83f8bac20c5a4a9c3917abb
Ideally, the `while` loop should include only `cache.put(last_key,
some_precomputed_value)`. But instead it also includes:
- a series of the `time()` calls, which could be mostly excluded from
the measured time, if the measurement done right; each call is probably
addresses the HPET device, or network time, or both,
- some floating-point calculations, including `round()`, which is hardly
necessary,
- formatting and output of the intermediate result.
I suppose the measurement influence can be quite significant here, but
it is at least more or less constant for each test.
But if we look at the other benchmarks:
https://gist.github.com/pilshchikov/8a4bdb03a8304136c22c9bf7217ee447
[Node.js]
https://gist.github.com/pilshchikov/b4351d78ad59e9cd923689c2e387bc80 [PHP]
https://gist.github.com/pilshchikov/08096c78b425e00166a2ffa2aa5f49ce [Java]
The extra code that influence the measurement is not equivalent across
all platforms. For example, PHP's `time()` is most probably lighter than
Python `time()`, since it do not give out milliseconds and may address
RTC, not HPET. So the platform-to-platform comparison in your benchmark
does not look completely fair to me.
The second remark concerns not the measurement procedure, but the object
being measured.
The only client operation being used is OP_CACHE_PUT with a payload of a
primitive type. (BTW the type is `Long` in case of the Python client;
what about the other clients? Is it `Int`?) I afraid that such an
object, even being properly isolated from the measurement tool, would
show mostly the throughput of the underlying platform's sockets
implementation, not the performance of the client's code. To show the
potential of the thin client itself, more varied and fancy tasks needed,
i. e. serialization/deserialization of the Complex objects.
But it depends on the goal of the benchmarking. If showing off the raw
platform agility was intended, than this objection is removed.
Dmitry
On 10/9/18 10:50 PM, Stepan Pilshchikov wrote:
> Hi, all
>
> Tried to compare new thin client performance and made similar benchmarks for
> each one
> And result not so good for python
>
> Ticket with results and bench code:
> https://issues.apache.org/jira/browse/IGNITE-9824
>
> Py code src:
> https://gist.github.com/pilshchikov/8aff4e30d83f8bac20c5a4a9c3917abb
>
> Dmitry, please review results and bench code, maybe somthing wrong or it's
> expected numbers?
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>