You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ratis.apache.org by runzhiwang <ru...@gmail.com> on 2020/12/14 08:35:06 UTC
Performance of ratis streaming compared to Ozone [Attach image]
Hi all,
I test the performance of ratis streaming. As the image shows, Raits
streaming can improve Ozone's throughput, memory usage, and stability
greatly.
[image: screenshot-1.png]
If you are interested with the test and want to reproduce it, please find
the details here:
https://docs.google.com/document/d/1TkR4qN1ymm9UY2nRLGlvL_TjI_gTabByDN5uCUEpJiM/edit
Thanks,
runzhiwang
Re: Performance of ratis streaming compared to Ozone [Attach image]
Posted by Rui Wang <am...@apache.org>.
Thanks runzhiwang for your explanation! It makes a lot of sense!
-Rui
On Mon, Dec 14, 2020 at 4:39 PM runzhiwang <ru...@gmail.com> wrote:
> Hi Rui,
> I did not post the comparison result of async api and streaming, because
> FileStore's asyncApi's implementation is different from Ozone.
> FileStore's asyncApi does not have any optimization, such as Ozone
> implement dataStateMachineCache to improve readStateMachineData,
> but FileStore's asyncApi need to read disk to read StateMachineData, so
> FileStore's asyncApi is slower than Ozone, and it's meaningless to
> compare FileStore's asyncApi and FileStore's streaming. Even though
> FileStore's streaming is faster than FileStore's asyncApi, we can not say
> FileStore's streaming is faster than Ozone. So I compare Ozone and
> FileStore's streaming directly, because the improvement is great, We
> can make sure FileStore's streaming is going to be a greate improvement on
> Ozone.
>
> Thanks,
> runzhiwang
>
>
> Rui Wang <am...@apache.org> 于2020年12月15日周二 上午4:58写道：
>
> > The improvements of Ratis streaming is solid based on the doc! Thanks
> > runzhiwang for your work!
> >
> > After walking through the performance doc, I think the comparison seemed
> to
> > happen between an Ozone cluster and a filestore?
> >
> > If so, my question is, will the Ozone cluster be too heavy to compare
> with
> > filestore, which is a simple implementation to write files? Do we need a
> > comparison for filesstore only between async api and streaming?
> >
> >
> > Hi Liyun,
> >
> > Currently the Ratis streaming write feature should be ready to use
> (though
> > improvements could keep happening). The read feature is under
> development.
> >
> > -Rui
> >
> > On Mon, Dec 14, 2020 at 8:02 AM Lin, Yiqun <yi...@ebay.com.invalid>
> > wrote:
> >
> > > Hi runzhiwang/ Tsz-Wo,
> > >
> > > Seems a very good improvement, I am interested for this.
> > > As I see the Ratis Streaming feature (RATIS-979) is still in progress
> > > state. Do you know current status of this feature and if it's ready and
> > > stable for Ozone to use now?
> > >
> > > Thanks,
> > > Yiqun
> > >
> > > On 2020/12/14, 5:40 PM, "Tsz Wo Sze" <sz...@gmail.com> wrote:
> > >
> > > External Email
> > >
> > > runzhiwang,
> > >
> > > Thanks a lot for working hard on Ratis Streaming and running the
> > > benchmarks! Ratis Streaming is going to be a big improvement in
> > Ozone.
> > >
> > > It is anticipated that Ratis Streaming can
> > > - improve the performance as shown in the benchmark results;
> > > - improve the scalability of Ozone since Streaming can reduce the
> > > number of
> > > pipelines in Ozone; and
> > > - reduce the memory requirement in datanodes.
> > >
> > > Regards,
> > > Tsz-Wo
> > >
> > > On Mon, Dec 14, 2020 at 4:35 PM runzhiwang <runzhiwang90@gmail.com
> >
> > > wrote:
> > >
> > > > Hi all,
> > > > I test the performance of ratis streaming. As the image shows,
> > Raits
> > > > streaming can improve Ozone's throughput, memory usage, and
> > stability
> > > > greatly.
> > > >
> > > > [image: screenshot-1.png]
> > > >
> > > > If you are interested with the test and want to reproduce it,
> > please
> > > find
> > > > the details here:
> > > >
> > > >
> > >
> >
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F1TkR4qN1ymm9UY2nRLGlvL_TjI_gTabByDN5uCUEpJiM%2Fedit&data=04%7C01%7Cyiqlin%40ebay.com%7C4a8c74e9a369402a3c8f08d8a014502e%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C637435356398333595%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=moz51OUIBm98n%2B%2FYdUtLT4Az0CE%2BgQ%2FnhpU47G5HWOY%3D&reserved=0
> > > >
> > > >
> > > > Thanks,
> > > > runzhiwang
> > > >
> > >
> > >
> >
>
Re: Performance of ratis streaming compared to Ozone [Attach image]
Posted by Rui Wang <am...@apache.org>.
Thanks runzhiwang for your explanation! It makes a lot of sense!
-Rui
On Mon, Dec 14, 2020 at 4:39 PM runzhiwang <ru...@gmail.com> wrote:
> Hi Rui,
> I did not post the comparison result of async api and streaming, because
> FileStore's asyncApi's implementation is different from Ozone.
> FileStore's asyncApi does not have any optimization, such as Ozone
> implement dataStateMachineCache to improve readStateMachineData,
> but FileStore's asyncApi need to read disk to read StateMachineData, so
> FileStore's asyncApi is slower than Ozone, and it's meaningless to
> compare FileStore's asyncApi and FileStore's streaming. Even though
> FileStore's streaming is faster than FileStore's asyncApi, we can not say
> FileStore's streaming is faster than Ozone. So I compare Ozone and
> FileStore's streaming directly, because the improvement is great, We
> can make sure FileStore's streaming is going to be a greate improvement on
> Ozone.
>
> Thanks,
> runzhiwang
>
>
> Rui Wang <am...@apache.org> 于2020年12月15日周二 上午4:58写道：
>
> > The improvements of Ratis streaming is solid based on the doc! Thanks
> > runzhiwang for your work!
> >
> > After walking through the performance doc, I think the comparison seemed
> to
> > happen between an Ozone cluster and a filestore?
> >
> > If so, my question is, will the Ozone cluster be too heavy to compare
> with
> > filestore, which is a simple implementation to write files? Do we need a
> > comparison for filesstore only between async api and streaming?
> >
> >
> > Hi Liyun,
> >
> > Currently the Ratis streaming write feature should be ready to use
> (though
> > improvements could keep happening). The read feature is under
> development.
> >
> > -Rui
> >
> > On Mon, Dec 14, 2020 at 8:02 AM Lin, Yiqun <yi...@ebay.com.invalid>
> > wrote:
> >
> > > Hi runzhiwang/ Tsz-Wo,
> > >
> > > Seems a very good improvement, I am interested for this.
> > > As I see the Ratis Streaming feature (RATIS-979) is still in progress
> > > state. Do you know current status of this feature and if it's ready and
> > > stable for Ozone to use now?
> > >
> > > Thanks,
> > > Yiqun
> > >
> > > On 2020/12/14, 5:40 PM, "Tsz Wo Sze" <sz...@gmail.com> wrote:
> > >
> > > External Email
> > >
> > > runzhiwang,
> > >
> > > Thanks a lot for working hard on Ratis Streaming and running the
> > > benchmarks! Ratis Streaming is going to be a big improvement in
> > Ozone.
> > >
> > > It is anticipated that Ratis Streaming can
> > > - improve the performance as shown in the benchmark results;
> > > - improve the scalability of Ozone since Streaming can reduce the
> > > number of
> > > pipelines in Ozone; and
> > > - reduce the memory requirement in datanodes.
> > >
> > > Regards,
> > > Tsz-Wo
> > >
> > > On Mon, Dec 14, 2020 at 4:35 PM runzhiwang <runzhiwang90@gmail.com
> >
> > > wrote:
> > >
> > > > Hi all,
> > > > I test the performance of ratis streaming. As the image shows,
> > Raits
> > > > streaming can improve Ozone's throughput, memory usage, and
> > stability
> > > > greatly.
> > > >
> > > > [image: screenshot-1.png]
> > > >
> > > > If you are interested with the test and want to reproduce it,
> > please
> > > find
> > > > the details here:
> > > >
> > > >
> > >
> >
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F1TkR4qN1ymm9UY2nRLGlvL_TjI_gTabByDN5uCUEpJiM%2Fedit&data=04%7C01%7Cyiqlin%40ebay.com%7C4a8c74e9a369402a3c8f08d8a014502e%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C637435356398333595%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=moz51OUIBm98n%2B%2FYdUtLT4Az0CE%2BgQ%2FnhpU47G5HWOY%3D&reserved=0
> > > >
> > > >
> > > > Thanks,
> > > > runzhiwang
> > > >
> > >
> > >
> >
>
Re: Performance of ratis streaming compared to Ozone [Attach image]
Posted by runzhiwang <ru...@gmail.com>.
Hi Rui,
I did not post the comparison result of async api and streaming, because
FileStore's asyncApi's implementation is different from Ozone.
FileStore's asyncApi does not have any optimization, such as Ozone
implement dataStateMachineCache to improve readStateMachineData,
but FileStore's asyncApi need to read disk to read StateMachineData, so
FileStore's asyncApi is slower than Ozone, and it's meaningless to
compare FileStore's asyncApi and FileStore's streaming. Even though
FileStore's streaming is faster than FileStore's asyncApi, we can not say
FileStore's streaming is faster than Ozone. So I compare Ozone and
FileStore's streaming directly, because the improvement is great, We
can make sure FileStore's streaming is going to be a greate improvement on
Ozone.
Thanks,
runzhiwang
Rui Wang <am...@apache.org> 于2020年12月15日周二 上午4:58写道：
> The improvements of Ratis streaming is solid based on the doc! Thanks
> runzhiwang for your work!
>
> After walking through the performance doc, I think the comparison seemed to
> happen between an Ozone cluster and a filestore?
>
> If so, my question is, will the Ozone cluster be too heavy to compare with
> filestore, which is a simple implementation to write files? Do we need a
> comparison for filesstore only between async api and streaming?
>
>
> Hi Liyun,
>
> Currently the Ratis streaming write feature should be ready to use (though
> improvements could keep happening). The read feature is under development.
>
> -Rui
>
> On Mon, Dec 14, 2020 at 8:02 AM Lin, Yiqun <yi...@ebay.com.invalid>
> wrote:
>
> > Hi runzhiwang/ Tsz-Wo,
> >
> > Seems a very good improvement, I am interested for this.
> > As I see the Ratis Streaming feature (RATIS-979) is still in progress
> > state. Do you know current status of this feature and if it's ready and
> > stable for Ozone to use now?
> >
> > Thanks,
> > Yiqun
> >
> > On 2020/12/14, 5:40 PM, "Tsz Wo Sze" <sz...@gmail.com> wrote:
> >
> > External Email
> >
> > runzhiwang,
> >
> > Thanks a lot for working hard on Ratis Streaming and running the
> > benchmarks! Ratis Streaming is going to be a big improvement in
> Ozone.
> >
> > It is anticipated that Ratis Streaming can
> > - improve the performance as shown in the benchmark results;
> > - improve the scalability of Ozone since Streaming can reduce the
> > number of
> > pipelines in Ozone; and
> > - reduce the memory requirement in datanodes.
> >
> > Regards,
> > Tsz-Wo
> >
> > On Mon, Dec 14, 2020 at 4:35 PM runzhiwang <ru...@gmail.com>
> > wrote:
> >
> > > Hi all,
> > > I test the performance of ratis streaming. As the image shows,
> Raits
> > > streaming can improve Ozone's throughput, memory usage, and
> stability
> > > greatly.
> > >
> > > [image: screenshot-1.png]
> > >
> > > If you are interested with the test and want to reproduce it,
> please
> > find
> > > the details here:
> > >
> > >
> >
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F1TkR4qN1ymm9UY2nRLGlvL_TjI_gTabByDN5uCUEpJiM%2Fedit&data=04%7C01%7Cyiqlin%40ebay.com%7C4a8c74e9a369402a3c8f08d8a014502e%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C637435356398333595%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=moz51OUIBm98n%2B%2FYdUtLT4Az0CE%2BgQ%2FnhpU47G5HWOY%3D&reserved=0
> > >
> > >
> > > Thanks,
> > > runzhiwang
> > >
> >
> >
>
Re: Performance of ratis streaming compared to Ozone [Attach image]
Posted by runzhiwang <ru...@gmail.com>.
Hi Rui,
I did not post the comparison result of async api and streaming, because
FileStore's asyncApi's implementation is different from Ozone.
FileStore's asyncApi does not have any optimization, such as Ozone
implement dataStateMachineCache to improve readStateMachineData,
but FileStore's asyncApi need to read disk to read StateMachineData, so
FileStore's asyncApi is slower than Ozone, and it's meaningless to
compare FileStore's asyncApi and FileStore's streaming. Even though
FileStore's streaming is faster than FileStore's asyncApi, we can not say
FileStore's streaming is faster than Ozone. So I compare Ozone and
FileStore's streaming directly, because the improvement is great, We
can make sure FileStore's streaming is going to be a greate improvement on
Ozone.
Thanks,
runzhiwang
Rui Wang <am...@apache.org> 于2020年12月15日周二 上午4:58写道：
> The improvements of Ratis streaming is solid based on the doc! Thanks
> runzhiwang for your work!
>
> After walking through the performance doc, I think the comparison seemed to
> happen between an Ozone cluster and a filestore?
>
> If so, my question is, will the Ozone cluster be too heavy to compare with
> filestore, which is a simple implementation to write files? Do we need a
> comparison for filesstore only between async api and streaming?
>
>
> Hi Liyun,
>
> Currently the Ratis streaming write feature should be ready to use (though
> improvements could keep happening). The read feature is under development.
>
> -Rui
>
> On Mon, Dec 14, 2020 at 8:02 AM Lin, Yiqun <yi...@ebay.com.invalid>
> wrote:
>
> > Hi runzhiwang/ Tsz-Wo,
> >
> > Seems a very good improvement, I am interested for this.
> > As I see the Ratis Streaming feature (RATIS-979) is still in progress
> > state. Do you know current status of this feature and if it's ready and
> > stable for Ozone to use now?
> >
> > Thanks,
> > Yiqun
> >
> > On 2020/12/14, 5:40 PM, "Tsz Wo Sze" <sz...@gmail.com> wrote:
> >
> > External Email
> >
> > runzhiwang,
> >
> > Thanks a lot for working hard on Ratis Streaming and running the
> > benchmarks! Ratis Streaming is going to be a big improvement in
> Ozone.
> >
> > It is anticipated that Ratis Streaming can
> > - improve the performance as shown in the benchmark results;
> > - improve the scalability of Ozone since Streaming can reduce the
> > number of
> > pipelines in Ozone; and
> > - reduce the memory requirement in datanodes.
> >
> > Regards,
> > Tsz-Wo
> >
> > On Mon, Dec 14, 2020 at 4:35 PM runzhiwang <ru...@gmail.com>
> > wrote:
> >
> > > Hi all,
> > > I test the performance of ratis streaming. As the image shows,
> Raits
> > > streaming can improve Ozone's throughput, memory usage, and
> stability
> > > greatly.
> > >
> > > [image: screenshot-1.png]
> > >
> > > If you are interested with the test and want to reproduce it,
> please
> > find
> > > the details here:
> > >
> > >
> >
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F1TkR4qN1ymm9UY2nRLGlvL_TjI_gTabByDN5uCUEpJiM%2Fedit&data=04%7C01%7Cyiqlin%40ebay.com%7C4a8c74e9a369402a3c8f08d8a014502e%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C637435356398333595%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=moz51OUIBm98n%2B%2FYdUtLT4Az0CE%2BgQ%2FnhpU47G5HWOY%3D&reserved=0
> > >
> > >
> > > Thanks,
> > > runzhiwang
> > >
> >
> >
>
Re: Performance of ratis streaming compared to Ozone [Attach image]
Posted by Rui Wang <am...@apache.org>.
The improvements of Ratis streaming is solid based on the doc! Thanks
runzhiwang for your work!
After walking through the performance doc, I think the comparison seemed to
happen between an Ozone cluster and a filestore?
If so, my question is, will the Ozone cluster be too heavy to compare with
filestore, which is a simple implementation to write files? Do we need a
comparison for filesstore only between async api and streaming?
Hi Liyun,
Currently the Ratis streaming write feature should be ready to use (though
improvements could keep happening). The read feature is under development.
-Rui
On Mon, Dec 14, 2020 at 8:02 AM Lin, Yiqun <yi...@ebay.com.invalid> wrote:
> Hi runzhiwang/ Tsz-Wo,
>
> Seems a very good improvement, I am interested for this.
> As I see the Ratis Streaming feature (RATIS-979) is still in progress
> state. Do you know current status of this feature and if it's ready and
> stable for Ozone to use now?
>
> Thanks,
> Yiqun
>
> On 2020/12/14, 5:40 PM, "Tsz Wo Sze" <sz...@gmail.com> wrote:
>
> External Email
>
> runzhiwang,
>
> Thanks a lot for working hard on Ratis Streaming and running the
> benchmarks! Ratis Streaming is going to be a big improvement in Ozone.
>
> It is anticipated that Ratis Streaming can
> - improve the performance as shown in the benchmark results;
> - improve the scalability of Ozone since Streaming can reduce the
> number of
> pipelines in Ozone; and
> - reduce the memory requirement in datanodes.
>
> Regards,
> Tsz-Wo
>
> On Mon, Dec 14, 2020 at 4:35 PM runzhiwang <ru...@gmail.com>
> wrote:
>
> > Hi all,
> > I test the performance of ratis streaming. As the image shows, Raits
> > streaming can improve Ozone's throughput, memory usage, and stability
> > greatly.
> >
> > [image: screenshot-1.png]
> >
> > If you are interested with the test and want to reproduce it, please
> find
> > the details here:
> >
> >
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F1TkR4qN1ymm9UY2nRLGlvL_TjI_gTabByDN5uCUEpJiM%2Fedit&data=04%7C01%7Cyiqlin%40ebay.com%7C4a8c74e9a369402a3c8f08d8a014502e%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C637435356398333595%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=moz51OUIBm98n%2B%2FYdUtLT4Az0CE%2BgQ%2FnhpU47G5HWOY%3D&reserved=0
> >
> >
> > Thanks,
> > runzhiwang
> >
>
>
Re: Performance of ratis streaming compared to Ozone [Attach image]
Posted by Rui Wang <am...@apache.org>.
The improvements of Ratis streaming is solid based on the doc! Thanks
runzhiwang for your work!
After walking through the performance doc, I think the comparison seemed to
happen between an Ozone cluster and a filestore?
If so, my question is, will the Ozone cluster be too heavy to compare with
filestore, which is a simple implementation to write files? Do we need a
comparison for filesstore only between async api and streaming?
Hi Liyun,
Currently the Ratis streaming write feature should be ready to use (though
improvements could keep happening). The read feature is under development.
-Rui
On Mon, Dec 14, 2020 at 8:02 AM Lin, Yiqun <yi...@ebay.com.invalid> wrote:
> Hi runzhiwang/ Tsz-Wo,
>
> Seems a very good improvement, I am interested for this.
> As I see the Ratis Streaming feature (RATIS-979) is still in progress
> state. Do you know current status of this feature and if it's ready and
> stable for Ozone to use now?
>
> Thanks,
> Yiqun
>
> On 2020/12/14, 5:40 PM, "Tsz Wo Sze" <sz...@gmail.com> wrote:
>
> External Email
>
> runzhiwang,
>
> Thanks a lot for working hard on Ratis Streaming and running the
> benchmarks! Ratis Streaming is going to be a big improvement in Ozone.
>
> It is anticipated that Ratis Streaming can
> - improve the performance as shown in the benchmark results;
> - improve the scalability of Ozone since Streaming can reduce the
> number of
> pipelines in Ozone; and
> - reduce the memory requirement in datanodes.
>
> Regards,
> Tsz-Wo
>
> On Mon, Dec 14, 2020 at 4:35 PM runzhiwang <ru...@gmail.com>
> wrote:
>
> > Hi all,
> > I test the performance of ratis streaming. As the image shows, Raits
> > streaming can improve Ozone's throughput, memory usage, and stability
> > greatly.
> >
> > [image: screenshot-1.png]
> >
> > If you are interested with the test and want to reproduce it, please
> find
> > the details here:
> >
> >
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F1TkR4qN1ymm9UY2nRLGlvL_TjI_gTabByDN5uCUEpJiM%2Fedit&data=04%7C01%7Cyiqlin%40ebay.com%7C4a8c74e9a369402a3c8f08d8a014502e%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C637435356398333595%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=moz51OUIBm98n%2B%2FYdUtLT4Az0CE%2BgQ%2FnhpU47G5HWOY%3D&reserved=0
> >
> >
> > Thanks,
> > runzhiwang
> >
>
>
Re: Performance of ratis streaming compared to Ozone [Attach image]
Posted by "Lin, Yiqun" <yi...@ebay.com.INVALID>.
Hi runzhiwang/ Tsz-Wo,
Seems a very good improvement, I am interested for this.
As I see the Ratis Streaming feature (RATIS-979) is still in progress state. Do you know current status of this feature and if it's ready and stable for Ozone to use now?
Thanks,
Yiqun
On 2020/12/14, 5:40 PM, "Tsz Wo Sze" <sz...@gmail.com> wrote:
External Email
runzhiwang,
Thanks a lot for working hard on Ratis Streaming and running the
benchmarks! Ratis Streaming is going to be a big improvement in Ozone.
It is anticipated that Ratis Streaming can
- improve the performance as shown in the benchmark results;
- improve the scalability of Ozone since Streaming can reduce the number of
pipelines in Ozone; and
- reduce the memory requirement in datanodes.
Regards,
Tsz-Wo
On Mon, Dec 14, 2020 at 4:35 PM runzhiwang <ru...@gmail.com> wrote:
> Hi all,
> I test the performance of ratis streaming. As the image shows, Raits
> streaming can improve Ozone's throughput, memory usage, and stability
> greatly.
>
> [image: screenshot-1.png]
>
> If you are interested with the test and want to reproduce it, please find
> the details here:
>
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F1TkR4qN1ymm9UY2nRLGlvL_TjI_gTabByDN5uCUEpJiM%2Fedit&data=04%7C01%7Cyiqlin%40ebay.com%7C4a8c74e9a369402a3c8f08d8a014502e%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C637435356398333595%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=moz51OUIBm98n%2B%2FYdUtLT4Az0CE%2BgQ%2FnhpU47G5HWOY%3D&reserved=0
>
>
> Thanks,
> runzhiwang
>
Re: Performance of ratis streaming compared to Ozone [Attach image]
Posted by "Lin, Yiqun" <yi...@ebay.com.INVALID>.
Hi runzhiwang/ Tsz-Wo,
Seems a very good improvement, I am interested for this.
As I see the Ratis Streaming feature (RATIS-979) is still in progress state. Do you know current status of this feature and if it's ready and stable for Ozone to use now?
Thanks,
Yiqun
On 2020/12/14, 5:40 PM, "Tsz Wo Sze" <sz...@gmail.com> wrote:
External Email
runzhiwang,
Thanks a lot for working hard on Ratis Streaming and running the
benchmarks! Ratis Streaming is going to be a big improvement in Ozone.
It is anticipated that Ratis Streaming can
- improve the performance as shown in the benchmark results;
- improve the scalability of Ozone since Streaming can reduce the number of
pipelines in Ozone; and
- reduce the memory requirement in datanodes.
Regards,
Tsz-Wo
On Mon, Dec 14, 2020 at 4:35 PM runzhiwang <ru...@gmail.com> wrote:
> Hi all,
> I test the performance of ratis streaming. As the image shows, Raits
> streaming can improve Ozone's throughput, memory usage, and stability
> greatly.
>
> [image: screenshot-1.png]
>
> If you are interested with the test and want to reproduce it, please find
> the details here:
>
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F1TkR4qN1ymm9UY2nRLGlvL_TjI_gTabByDN5uCUEpJiM%2Fedit&data=04%7C01%7Cyiqlin%40ebay.com%7C4a8c74e9a369402a3c8f08d8a014502e%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C637435356398333595%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=moz51OUIBm98n%2B%2FYdUtLT4Az0CE%2BgQ%2FnhpU47G5HWOY%3D&reserved=0
>
>
> Thanks,
> runzhiwang
>
Re: Performance of ratis streaming compared to Ozone [Attach image]
Posted by Tsz Wo Sze <sz...@gmail.com>.
runzhiwang,
Thanks a lot for working hard on Ratis Streaming and running the
benchmarks! Ratis Streaming is going to be a big improvement in Ozone.
It is anticipated that Ratis Streaming can
- improve the performance as shown in the benchmark results;
- improve the scalability of Ozone since Streaming can reduce the number of
pipelines in Ozone; and
- reduce the memory requirement in datanodes.
Regards,
Tsz-Wo
On Mon, Dec 14, 2020 at 4:35 PM runzhiwang <ru...@gmail.com> wrote:
> Hi all,
> I test the performance of ratis streaming. As the image shows, Raits
> streaming can improve Ozone's throughput, memory usage, and stability
> greatly.
>
> [image: screenshot-1.png]
>
> If you are interested with the test and want to reproduce it, please find
> the details here:
>
> https://docs.google.com/document/d/1TkR4qN1ymm9UY2nRLGlvL_TjI_gTabByDN5uCUEpJiM/edit
>
>
> Thanks,
> runzhiwang
>
Re: Performance of ratis streaming compared to Ozone [Attach image]
Posted by Tsz Wo Sze <sz...@gmail.com>.
runzhiwang,
Thanks a lot for working hard on Ratis Streaming and running the
benchmarks! Ratis Streaming is going to be a big improvement in Ozone.
It is anticipated that Ratis Streaming can
- improve the performance as shown in the benchmark results;
- improve the scalability of Ozone since Streaming can reduce the number of
pipelines in Ozone; and
- reduce the memory requirement in datanodes.
Regards,
Tsz-Wo
On Mon, Dec 14, 2020 at 4:35 PM runzhiwang <ru...@gmail.com> wrote:
> Hi all,
> I test the performance of ratis streaming. As the image shows, Raits
> streaming can improve Ozone's throughput, memory usage, and stability
> greatly.
>
> [image: screenshot-1.png]
>
> If you are interested with the test and want to reproduce it, please find
> the details here:
>
> https://docs.google.com/document/d/1TkR4qN1ymm9UY2nRLGlvL_TjI_gTabByDN5uCUEpJiM/edit
>
>
> Thanks,
> runzhiwang
>