You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@paimon.apache.org by 刘金辉 <96...@qq.com.INVALID> on 2025/01/15 11:02:10 UTC
回复： [DISCUSS]paimon support tencent cosn
hi，Jingsong teacher.
Here is some information，，please check thanks
Optimization methods of cosn:
1. Performance optimization:
a. Combined with the architecture of COS fusion bucket, separate metadata flow and data flow to achieve high-performance file semantic access
b. Based on COS distributed metadata engine, provide high-performance metadata access
c. Provide high-concurrency data flow through pre-reading strategy, block processing and other means
2. Integrate Tencent Cloud service role authentication capabilities to improve access security
3. Integrate Tencent Cloud log reporting capabilities, support task-level log reporting, and improve observability
S3 Architecture:
Cosn Architecture
liujinhui
刘金辉
965147871@qq.com
------------------ 原始邮件 ------------------
发件人: "dev" <xuanwo@apache.org>;
发送时间: 2025年1月9日(星期四) 下午4:20
收件人: "dev"<dev@paimon.apache.org>;
主题: Re: [DISCUSS]paimon support tencent cosn
Hi,
That's interesting to come across this discussion. Apache OpenDAL offers native support for COS (although we primarily utilize it for native IAM support and are not currently aware of any performance gaps). Perhaps we could consider adding a new file IO class based on OpenDAL to support this?
Additionally, OpenDAL provides native support for nearly all storage services, ranging from S3, Gcs to Azblob, which could be a valuable addition for us.
For more information about OpenDAL: https://github.com/apache/opendal
We also offer a native Java binding: https://github.com/apache/opendal/tree/main/bindings/java
On Thu, Jan 9, 2025, at 11:05, Jingsong Li wrote:
> Hi jinhui,
>
> Thanks for driving this, can you provide some information about "When
> using the s3 protocol to access the fusion bucket, the traffic will
> pass through the access layer twice", is there an official website?
>
> Best,
> Jingsong
>
> On Thu, Jan 9, 2025 at 10:56 AM 刘金辉 <965147871@qq.com.invalid> wrote:
>>
>> Introduction
>> Tencent COS is a famous object storage system provided by Tencent Corp. Hadoop-COS is a client that makes the upper computing systems based on HDFS be able to use the COS as its underlying storage system. tencent cloud is top 2 cloud vendors in China market and the object store COS （https://intl.cloud.tencent.com/product/cos） is widely used among China’s cloud users but now it is hard for hadoop user to access data laid on COS storage&nbsp;
>>
>>
>> We can use S3 SDK and COSN SDK to access COSN，but
>> 1. When using the s3 protocol to access the fusion bucket, the traffic will pass through the access layer twice, the traffic will be doubled, and the performance will be impaired. This problem will not occur when using cosn.
>> 2. Compared with cos ordinary buckets, fusion buckets have performance advantages in list, rename, truncate and other operations.
>> 3. If you must use the s3 protocol to access cos, we do not recommend using fusion buckets, then the s3 protocol + cos ordinary bucket will encounter some of the above performance issues.
>> 4. Other data lake frameworks also support access to cosn, oss, S3, etc.
>>
>>
>>
>>
>> Therefore, in order to allow more users to use paimon with high performance, it is necessary to support cosn access to paimon.
>>
>>
>>
>>
>>
>>
>> Please give your opinions. thanks
>>
>>
>>
>>
>>
>> 刘金辉
>> 965147871@qq.com
>>
>>
>>
>> &nbsp;
--
Xuanwo
https://xuanwo.io/
Re: [DISCUSS]paimon support tencent cosn
Posted by jialiang tan <ta...@gmail.com>.
Hi, jinhui.
Very thanks for explaining the difference between s3 and cosn.
I asked the Tencent Cloud and it is indeed. So just go head for it.
Best,
tanjialiang.
Jingsong Li <ji...@gmail.com> 于2025年1月15日周三 22:27写道：
> Hi, jinhui,
>
> Thanks for your information.
>
> I understood the difference.
>
> It is time to do vote if nobody has more concerns.
>
> Best,
> Jingsong
>
> Jingsong Li <ji...@gmail.com>于2025年1月15日 周三22:23写道：
>
> > Hi xuanwo, thanks for the suggestion.
> >
> > We are talking about getting rid of the middle layer, and use COSN
> > directly.
> >
> > So if OpenDAL has no difference to COSN, it is better to use it directly.
> >
> > Best,
> > Jingsong
> >
> > 刘金辉 <96...@qq.com.invalid>于2025年1月15日 周三19:02写道：
> >
> >> hi，Jingsong teacher.
> >> Here is some information，，please check thanks
> >> Optimization methods of cosn:
> >> 1. Performance optimization:
> >> a. Combined with the architecture of COS fusion bucket, separate
> metadata
> >> flow and data flow to achieve high-performance file semantic access
> >> b. Based on COS distributed metadata engine, provide high-performance
> >> metadata access
> >> c. Provide high-concurrency data flow through pre-reading strategy,
> block
> >> processing and other means
> >> 2. Integrate Tencent Cloud service role authentication capabilities to
> >> improve access security
> >> 3. Integrate Tencent Cloud log reporting capabilities, support
> task-level
> >> log reporting, and improve observability
> >>
> >>
> >>
> >>
> >>
> >> S3 Architecture:
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> Cosn Architecture
> >>
> >> liujinhui
> >>
> >>
> >>
> >>
> >> 刘金辉
> >> 965147871@qq.com
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> ------------------ 原始邮件 ------------------
> >> 发件人:
> >> "dev"
> >> <
> >> xuanwo@apache.org>;
> >> 发送时间: 2025年1月9日(星期四) 下午4:20
> >> 收件人: "dev"<dev@paimon.apache.org>;
> >>
> >> 主题: Re: [DISCUSS]paimon support tencent cosn
> >>
> >>
> >>
> >> Hi,
> >>
> >> That's interesting to come across this discussion. Apache OpenDAL offers
> >> native support for COS (although we primarily utilize it for native IAM
> >> support and are not currently aware of any performance gaps). Perhaps we
> >> could consider adding a new file IO class based on OpenDAL to support
> this?
> >>
> >> Additionally, OpenDAL provides native support for nearly all storage
> >> services, ranging from S3, Gcs to Azblob, which could be a valuable
> >> addition for us.
> >>
> >> For more information about OpenDAL: https://github.com/apache/opendal
> >> We also offer a native Java binding:
> >> https://github.com/apache/opendal/tree/main/bindings/java
> >>
> >> On Thu, Jan 9, 2025, at 11:05, Jingsong Li wrote:
> >> > Hi jinhui,
> >> >
> >> > Thanks for driving this, can you provide some information about
> "When
> >> > using the s3 protocol to access the fusion bucket, the traffic will
> >> > pass through the access layer twice", is there an official website?
> >> >
> >> > Best,
> >> > Jingsong
> >> >
> >> > On Thu, Jan 9, 2025 at 10:56 AM 刘金辉 <965147871@qq.com.invalid>
> >> wrote:
> >> >>
> >> >> Introduction
> >> >> Tencent COS is a famous object storage system provided by
> >> Tencent Corp. Hadoop-COS is a client that makes the upper computing
> systems
> >> based on HDFS be able to use the COS as its underlying storage system.
> >> tencent cloud is top 2 cloud vendors in China market and the object
> store
> >> COS （https://intl.cloud.tencent.com/product/cos） is widely used among
> >> China’s cloud users but now it is hard for hadoop user to access data
> laid
> >> on COS storage&nbsp;
> >> >>
> >> >>
> >> >> We can use S3 SDK and COSN SDK to access COSN，but
> >> >> 1. When using the s3 protocol to access the fusion bucket, the
> >> traffic will pass through the access layer twice, the traffic will be
> >> doubled, and the performance will be impaired. This problem will not
> occur
> >> when using cosn.
> >> >> 2. Compared with cos ordinary buckets, fusion buckets have
> >> performance advantages in list, rename, truncate and other operations.
> >> >> 3. If you must use the s3 protocol to access cos, we do not
> >> recommend using fusion buckets, then the s3 protocol + cos ordinary
> bucket
> >> will encounter some of the above performance issues.
> >> >> 4. Other data lake frameworks also support access to cosn, oss,
> >> S3, etc.
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> Therefore, in order to allow more users to use paimon with high
> >> performance, it is necessary to support cosn access to paimon.
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> Please give your opinions. thanks
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> 刘金辉
> >> >> 965147871@qq.com
> >> >>
> >> >>
> >> >>
> >> >> &nbsp;
> >>
> >> --
> >> Xuanwo
> >>
> >> https://xuanwo.io/
> >
> >
>
Re: [DISCUSS]paimon support tencent cosn
Posted by Jingsong Li <ji...@gmail.com>.
Hi, jinhui,
Thanks for your information.
I understood the difference.
It is time to do vote if nobody has more concerns.
Best,
Jingsong
Jingsong Li <ji...@gmail.com>于2025年1月15日 周三22:23写道：
> Hi xuanwo, thanks for the suggestion.
>
> We are talking about getting rid of the middle layer, and use COSN
> directly.
>
> So if OpenDAL has no difference to COSN, it is better to use it directly.
>
> Best,
> Jingsong
>
> 刘金辉 <96...@qq.com.invalid>于2025年1月15日 周三19:02写道：
>
>> hi，Jingsong teacher.
>> Here is some information，，please check thanks
>> Optimization methods of cosn:
>> 1. Performance optimization:
>> a. Combined with the architecture of COS fusion bucket, separate metadata
>> flow and data flow to achieve high-performance file semantic access
>> b. Based on COS distributed metadata engine, provide high-performance
>> metadata access
>> c. Provide high-concurrency data flow through pre-reading strategy, block
>> processing and other means
>> 2. Integrate Tencent Cloud service role authentication capabilities to
>> improve access security
>> 3. Integrate Tencent Cloud log reporting capabilities, support task-level
>> log reporting, and improve observability
>>
>>
>>
>>
>>
>> S3 Architecture:
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> Cosn Architecture
>>
>> liujinhui
>>
>>
>>
>>
>> 刘金辉
>> 965147871@qq.com
>>
>>
>>
>>
>>
>>
>>
>>
>> ------------------ 原始邮件 ------------------
>> 发件人:
>> "dev"
>> <
>> xuanwo@apache.org>;
>> 发送时间: 2025年1月9日(星期四) 下午4:20
>> 收件人: "dev"<dev@paimon.apache.org>;
>>
>> 主题: Re: [DISCUSS]paimon support tencent cosn
>>
>>
>>
>> Hi,
>>
>> That's interesting to come across this discussion. Apache OpenDAL offers
>> native support for COS (although we primarily utilize it for native IAM
>> support and are not currently aware of any performance gaps). Perhaps we
>> could consider adding a new file IO class based on OpenDAL to support this?
>>
>> Additionally, OpenDAL provides native support for nearly all storage
>> services, ranging from S3, Gcs to Azblob, which could be a valuable
>> addition for us.
>>
>> For more information about OpenDAL: https://github.com/apache/opendal
>> We also offer a native Java binding:
>> https://github.com/apache/opendal/tree/main/bindings/java
>>
>> On Thu, Jan 9, 2025, at 11:05, Jingsong Li wrote:
>> > Hi jinhui,
>> >
>> > Thanks for driving this, can you provide some information about "When
>> > using the s3 protocol to access the fusion bucket, the traffic will
>> > pass through the access layer twice", is there an official website?
>> >
>> > Best,
>> > Jingsong
>> >
>> > On Thu, Jan 9, 2025 at 10:56 AM 刘金辉 <965147871@qq.com.invalid>
>> wrote:
>> >>
>> >> Introduction
>> >> Tencent COS is a famous object storage system provided by
>> Tencent Corp. Hadoop-COS is a client that makes the upper computing systems
>> based on HDFS be able to use the COS as its underlying storage system.
>> tencent cloud is top 2 cloud vendors in China market and the object store
>> COS （https://intl.cloud.tencent.com/product/cos） is widely used among
>> China’s cloud users but now it is hard for hadoop user to access data laid
>> on COS storage&nbsp;
>> >>
>> >>
>> >> We can use S3 SDK and COSN SDK to access COSN，but
>> >> 1. When using the s3 protocol to access the fusion bucket, the
>> traffic will pass through the access layer twice, the traffic will be
>> doubled, and the performance will be impaired. This problem will not occur
>> when using cosn.
>> >> 2. Compared with cos ordinary buckets, fusion buckets have
>> performance advantages in list, rename, truncate and other operations.
>> >> 3. If you must use the s3 protocol to access cos, we do not
>> recommend using fusion buckets, then the s3 protocol + cos ordinary bucket
>> will encounter some of the above performance issues.
>> >> 4. Other data lake frameworks also support access to cosn, oss,
>> S3, etc.
>> >>
>> >>
>> >>
>> >>
>> >> Therefore, in order to allow more users to use paimon with high
>> performance, it is necessary to support cosn access to paimon.
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> Please give your opinions. thanks
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> 刘金辉
>> >> 965147871@qq.com
>> >>
>> >>
>> >>
>> >> &nbsp;
>>
>> --
>> Xuanwo
>>
>> https://xuanwo.io/
>
>
Re: [DISCUSS]paimon support tencent cosn
Posted by Jingsong Li <ji...@gmail.com>.
Hi xuanwo, thanks for the suggestion.
We are talking about getting rid of the middle layer, and use COSN directly.
So if OpenDAL has no difference to COSN, it is better to use it directly.
Best,
Jingsong
刘金辉 <96...@qq.com.invalid>于2025年1月15日 周三19:02写道：
> hi，Jingsong teacher.
> Here is some information，，please check thanks
> Optimization methods of cosn:
> 1. Performance optimization:
> a. Combined with the architecture of COS fusion bucket, separate metadata
> flow and data flow to achieve high-performance file semantic access
> b. Based on COS distributed metadata engine, provide high-performance
> metadata access
> c. Provide high-concurrency data flow through pre-reading strategy, block
> processing and other means
> 2. Integrate Tencent Cloud service role authentication capabilities to
> improve access security
> 3. Integrate Tencent Cloud log reporting capabilities, support task-level
> log reporting, and improve observability
>
>
>
>
>
> S3 Architecture:
>
>
>
>
>
>
>
>
>
>
> Cosn Architecture
>
> liujinhui
>
>
>
>
> 刘金辉
> 965147871@qq.com
>
>
>
>
>
>
>
>
> ------------------ 原始邮件 ------------------
> 发件人:
> "dev"
> <
> xuanwo@apache.org>;
> 发送时间: 2025年1月9日(星期四) 下午4:20
> 收件人: "dev"<dev@paimon.apache.org>;
>
> 主题: Re: [DISCUSS]paimon support tencent cosn
>
>
>
> Hi,
>
> That's interesting to come across this discussion. Apache OpenDAL offers
> native support for COS (although we primarily utilize it for native IAM
> support and are not currently aware of any performance gaps). Perhaps we
> could consider adding a new file IO class based on OpenDAL to support this?
>
> Additionally, OpenDAL provides native support for nearly all storage
> services, ranging from S3, Gcs to Azblob, which could be a valuable
> addition for us.
>
> For more information about OpenDAL: https://github.com/apache/opendal
> We also offer a native Java binding:
> https://github.com/apache/opendal/tree/main/bindings/java
>
> On Thu, Jan 9, 2025, at 11:05, Jingsong Li wrote:
> > Hi jinhui,
> >
> > Thanks for driving this, can you provide some information about "When
> > using the s3 protocol to access the fusion bucket, the traffic will
> > pass through the access layer twice", is there an official website?
> >
> > Best,
> > Jingsong
> >
> > On Thu, Jan 9, 2025 at 10:56 AM 刘金辉 <965147871@qq.com.invalid>
> wrote:
> >>
> >> Introduction
> >> Tencent COS is a famous object storage system provided by Tencent
> Corp. Hadoop-COS is a client that makes the upper computing systems based
> on HDFS be able to use the COS as its underlying storage system. tencent
> cloud is top 2 cloud vendors in China market and the object store COS （
> https://intl.cloud.tencent.com/product/cos） is widely used among China’s
> cloud users but now it is hard for hadoop user to access data laid on COS
> storage&nbsp;
> >>
> >>
> >> We can use S3 SDK and COSN SDK to access COSN，but
> >> 1. When using the s3 protocol to access the fusion bucket, the
> traffic will pass through the access layer twice, the traffic will be
> doubled, and the performance will be impaired. This problem will not occur
> when using cosn.
> >> 2. Compared with cos ordinary buckets, fusion buckets have
> performance advantages in list, rename, truncate and other operations.
> >> 3. If you must use the s3 protocol to access cos, we do not
> recommend using fusion buckets, then the s3 protocol + cos ordinary bucket
> will encounter some of the above performance issues.
> >> 4. Other data lake frameworks also support access to cosn, oss,
> S3, etc.
> >>
> >>
> >>
> >>
> >> Therefore, in order to allow more users to use paimon with high
> performance, it is necessary to support cosn access to paimon.
> >>
> >>
> >>
> >>
> >>
> >>
> >> Please give your opinions. thanks
> >>
> >>
> >>
> >>
> >>
> >> 刘金辉
> >> 965147871@qq.com
> >>
> >>
> >>
> >> &nbsp;
>
> --
> Xuanwo
>
> https://xuanwo.io/