You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Laurent Quérel <la...@gmail.com> on 2023/03/30 17:56:15 UTC
OpenTelemetry + Arrow
Hello everyone,
I was wondering if the Apache Arrow community would be interested in
featuring a two-part article series on their blog, discussing the
experiences and insights gained from an experimental version of the
OpenTelemetry protocol (OTLP) utilizing Apache Arrow. As the main author of
the OTLP Arrow specification
<https://github.com/lquerel/oteps/blob/main/text/0156-columnar-encoding.md>,
the reference implementation otlp-arrow-adapter
<https://github.com/f5/otel-arrow-adapter>, and the two articles (see links
below), I believe that fostering collaboration between open-source projects
like these is essential and mutually beneficial.
These articles would serve as a fitting complement to the three
introductory articles that Andrew Lamb and Raphael Taylor-Davies
co-authored. They delve into the practical aspects of integrating Apache
Arrow into an existing project, as well as the process of converting a
hierarchical data model into its Arrow representation. The first article
examines various mapping techniques for aligning an existing data model
with the corresponding Arrow representation, while the second article
explores an adaptive schema technique that I implemented in the library's
final version in greater depth. Although the second article is still under
development, the core framework description is already in place.
What are your thoughts on this proposal?
Article 1:
https://docs.google.com/document/d/11lG7Go2IgKOyW-RReBRW6r7HIdV1X7lu5WrDGlW5LbQ/edit?usp=sharing
Article 2 (WIP):
https://docs.google.com/document/d/1K2CqAtF4pZjpiVts8BOcq34sOcNgozvZ9ZZw-_zTv6I/edit?usp=sharing
Best regards,
Laurent Quérel
--
Laurent Quérel
Re: OpenTelemetry + Arrow
Posted by Laurent Quérel <la...@gmail.com>.
Thank you very much Andrew.
I should be able to work on the second article next week and I will follow
the same process.
Cheers, Laurent
On Tue, Apr 11, 2023 at 4:31 AM Andrew Lamb <al...@influxdata.com> wrote:
> The blog post is now live on the arrow site [1]
>
> Thanks again Laurent
>
> [1]:
>
> https://arrow.apache.org/blog/2023/04/11/our-journey-at-f5-with-apache-arrow-part-1/
>
> On Sun, Apr 2, 2023 at 9:07 PM Laurent Quérel <la...@gmail.com>
> wrote:
>
> > Hi Andrew,
> >
> > The feedback seems to be good so I created a PR.
> >
> > https://github.com/apache/arrow-site/pull/340
> >
> > Best regards,
> >
> > Laurent Querel
> >
> > On Thu, Mar 30, 2023 at 3:28 PM Laurent Quérel <laurent.querel@gmail.com
> >
> > wrote:
> >
> > > I'm glad to know that the article has been well-received. In the second
> > > article, I will allocate a dedicated section to summarize the various
> > > challenges encountered when using Arrow for this type of project.
> > >
> > > @Matt, I want to express my gratitude for your continuous support
> > > throughout this project. Your contributions and refinements to the
> Arrow
> > Go
> > > library have enabled me to make significant progress with minimal
> > obstacles.
> > >
> > > Best Regards, Laurent
> > >
> > > On Thu, Mar 30, 2023 at 2:24 PM Matt Topol <zo...@gmail.com>
> > wrote:
> > >
> > >> +1 (non -binding)
> > >>
> > >> I'm glad others on here are finding this as useful and interesting as
> I
> > >> did.
> > >>
> > >> Great job Laurent!
> > >>
> > >> --Matt
> > >>
> > >> On Thu, Mar 30, 2023, 3:26 PM Raphael Taylor-Davies
> > >> <r....@googlemail.com.invalid> wrote:
> > >>
> > >> > Hi Laurent,
> > >> >
> > >> > I gave the first blog post a read and I also really like it and
> would
> > be
> > >> > +1 on publishing it, nice work.
> > >> >
> > >> > I would also like to echo Will's sentiment that getting real-world
> > case
> > >> > studies for the more complex Arrow schemas is invaluable and will
> help
> > >> > drive improvements in this space, so thank you for driving this
> > forward.
> > >> >
> > >> > Kind Regards,
> > >> >
> > >> > Raphael
> > >> >
> > >> > On 30/03/2023 19:52, Will Jones wrote:
> > >> > > Hi Laurent,
> > >> > >
> > >> > > I have read the first post and I really like it. I'd be +1 on
> > >> publishing
> > >> > > these to the blog. I'm interested to read the second one when it's
> > >> > finished.
> > >> > >
> > >> > > IMO the blog could use more examples of using Arrow that's not
> > >> building a
> > >> > > data frame library / query engine, and I appreciate that this blog
> > >> > provides
> > >> > > advice for some of the trickier parts of working with complex
> Arrow
> > >> > > schemas. I think this will also provide a good concrete use case
> for
> > >> us
> > >> > to
> > >> > > think about improving the ecosystem's support for nested data.
> > >> > >
> > >> > > Best,
> > >> > >
> > >> > > Will Jones
> > >> > >
> > >> > > On Thu, Mar 30, 2023 at 10:56 AM Laurent Quérel <
> > >> > laurent.querel@gmail.com>
> > >> > > wrote:
> > >> > >
> > >> > >> Hello everyone,
> > >> > >>
> > >> > >> I was wondering if the Apache Arrow community would be interested
> > in
> > >> > >> featuring a two-part article series on their blog, discussing the
> > >> > >> experiences and insights gained from an experimental version of
> the
> > >> > >> OpenTelemetry protocol (OTLP) utilizing Apache Arrow. As the main
> > >> > author of
> > >> > >> the OTLP Arrow specification
> > >> > >> <
> > >> >
> > >>
> >
> https://github.com/lquerel/oteps/blob/main/text/0156-columnar-encoding.md
> > >> > >>> ,
> > >> > >> the reference implementation otlp-arrow-adapter
> > >> > >> <https://github.com/f5/otel-arrow-adapter>, and the two articles
> > >> (see
> > >> > >> links
> > >> > >> below), I believe that fostering collaboration between
> open-source
> > >> > projects
> > >> > >> like these is essential and mutually beneficial.
> > >> > >>
> > >> > >> These articles would serve as a fitting complement to the three
> > >> > >> introductory articles that Andrew Lamb and Raphael Taylor-Davies
> > >> > >> co-authored. They delve into the practical aspects of integrating
> > >> Apache
> > >> > >> Arrow into an existing project, as well as the process of
> > converting
> > >> a
> > >> > >> hierarchical data model into its Arrow representation. The first
> > >> article
> > >> > >> examines various mapping techniques for aligning an existing data
> > >> model
> > >> > >> with the corresponding Arrow representation, while the second
> > article
> > >> > >> explores an adaptive schema technique that I implemented in the
> > >> > library's
> > >> > >> final version in greater depth. Although the second article is
> > still
> > >> > under
> > >> > >> development, the core framework description is already in place.
> > >> > >>
> > >> > >> What are your thoughts on this proposal?
> > >> > >>
> > >> > >> Article 1:
> > >> > >>
> > >> > >>
> > >> >
> > >>
> >
> https://docs.google.com/document/d/11lG7Go2IgKOyW-RReBRW6r7HIdV1X7lu5WrDGlW5LbQ/edit?usp=sharing
> > >> > >>
> > >> > >> Article 2 (WIP):
> > >> > >>
> > >> > >>
> > >> >
> > >>
> >
> https://docs.google.com/document/d/1K2CqAtF4pZjpiVts8BOcq34sOcNgozvZ9ZZw-_zTv6I/edit?usp=sharing
> > >> > >>
> > >> > >>
> > >> > >> Best regards,
> > >> > >>
> > >> > >> Laurent Quérel
> > >> > >>
> > >> > >> --
> > >> > >> Laurent Quérel
> > >> > >>
> > >> >
> > >>
> > >
> > >
> > > --
> > > Laurent Quérel
> > >
> > >
> >
> >
> > --
> > Laurent Quérel
> >
>
--
Laurent Quérel
Re: OpenTelemetry + Arrow
Posted by Andrew Lamb <al...@influxdata.com>.
The blog post is now live on the arrow site [1]
Thanks again Laurent
[1]:
https://arrow.apache.org/blog/2023/04/11/our-journey-at-f5-with-apache-arrow-part-1/
On Sun, Apr 2, 2023 at 9:07 PM Laurent Quérel <la...@gmail.com>
wrote:
> Hi Andrew,
>
> The feedback seems to be good so I created a PR.
>
> https://github.com/apache/arrow-site/pull/340
>
> Best regards,
>
> Laurent Querel
>
> On Thu, Mar 30, 2023 at 3:28 PM Laurent Quérel <la...@gmail.com>
> wrote:
>
> > I'm glad to know that the article has been well-received. In the second
> > article, I will allocate a dedicated section to summarize the various
> > challenges encountered when using Arrow for this type of project.
> >
> > @Matt, I want to express my gratitude for your continuous support
> > throughout this project. Your contributions and refinements to the Arrow
> Go
> > library have enabled me to make significant progress with minimal
> obstacles.
> >
> > Best Regards, Laurent
> >
> > On Thu, Mar 30, 2023 at 2:24 PM Matt Topol <zo...@gmail.com>
> wrote:
> >
> >> +1 (non -binding)
> >>
> >> I'm glad others on here are finding this as useful and interesting as I
> >> did.
> >>
> >> Great job Laurent!
> >>
> >> --Matt
> >>
> >> On Thu, Mar 30, 2023, 3:26 PM Raphael Taylor-Davies
> >> <r....@googlemail.com.invalid> wrote:
> >>
> >> > Hi Laurent,
> >> >
> >> > I gave the first blog post a read and I also really like it and would
> be
> >> > +1 on publishing it, nice work.
> >> >
> >> > I would also like to echo Will's sentiment that getting real-world
> case
> >> > studies for the more complex Arrow schemas is invaluable and will help
> >> > drive improvements in this space, so thank you for driving this
> forward.
> >> >
> >> > Kind Regards,
> >> >
> >> > Raphael
> >> >
> >> > On 30/03/2023 19:52, Will Jones wrote:
> >> > > Hi Laurent,
> >> > >
> >> > > I have read the first post and I really like it. I'd be +1 on
> >> publishing
> >> > > these to the blog. I'm interested to read the second one when it's
> >> > finished.
> >> > >
> >> > > IMO the blog could use more examples of using Arrow that's not
> >> building a
> >> > > data frame library / query engine, and I appreciate that this blog
> >> > provides
> >> > > advice for some of the trickier parts of working with complex Arrow
> >> > > schemas. I think this will also provide a good concrete use case for
> >> us
> >> > to
> >> > > think about improving the ecosystem's support for nested data.
> >> > >
> >> > > Best,
> >> > >
> >> > > Will Jones
> >> > >
> >> > > On Thu, Mar 30, 2023 at 10:56 AM Laurent Quérel <
> >> > laurent.querel@gmail.com>
> >> > > wrote:
> >> > >
> >> > >> Hello everyone,
> >> > >>
> >> > >> I was wondering if the Apache Arrow community would be interested
> in
> >> > >> featuring a two-part article series on their blog, discussing the
> >> > >> experiences and insights gained from an experimental version of the
> >> > >> OpenTelemetry protocol (OTLP) utilizing Apache Arrow. As the main
> >> > author of
> >> > >> the OTLP Arrow specification
> >> > >> <
> >> >
> >>
> https://github.com/lquerel/oteps/blob/main/text/0156-columnar-encoding.md
> >> > >>> ,
> >> > >> the reference implementation otlp-arrow-adapter
> >> > >> <https://github.com/f5/otel-arrow-adapter>, and the two articles
> >> (see
> >> > >> links
> >> > >> below), I believe that fostering collaboration between open-source
> >> > projects
> >> > >> like these is essential and mutually beneficial.
> >> > >>
> >> > >> These articles would serve as a fitting complement to the three
> >> > >> introductory articles that Andrew Lamb and Raphael Taylor-Davies
> >> > >> co-authored. They delve into the practical aspects of integrating
> >> Apache
> >> > >> Arrow into an existing project, as well as the process of
> converting
> >> a
> >> > >> hierarchical data model into its Arrow representation. The first
> >> article
> >> > >> examines various mapping techniques for aligning an existing data
> >> model
> >> > >> with the corresponding Arrow representation, while the second
> article
> >> > >> explores an adaptive schema technique that I implemented in the
> >> > library's
> >> > >> final version in greater depth. Although the second article is
> still
> >> > under
> >> > >> development, the core framework description is already in place.
> >> > >>
> >> > >> What are your thoughts on this proposal?
> >> > >>
> >> > >> Article 1:
> >> > >>
> >> > >>
> >> >
> >>
> https://docs.google.com/document/d/11lG7Go2IgKOyW-RReBRW6r7HIdV1X7lu5WrDGlW5LbQ/edit?usp=sharing
> >> > >>
> >> > >> Article 2 (WIP):
> >> > >>
> >> > >>
> >> >
> >>
> https://docs.google.com/document/d/1K2CqAtF4pZjpiVts8BOcq34sOcNgozvZ9ZZw-_zTv6I/edit?usp=sharing
> >> > >>
> >> > >>
> >> > >> Best regards,
> >> > >>
> >> > >> Laurent Quérel
> >> > >>
> >> > >> --
> >> > >> Laurent Quérel
> >> > >>
> >> >
> >>
> >
> >
> > --
> > Laurent Quérel
> >
> >
>
>
> --
> Laurent Quérel
>
Re: OpenTelemetry + Arrow
Posted by Laurent Quérel <la...@gmail.com>.
Hi Andrew,
The feedback seems to be good so I created a PR.
https://github.com/apache/arrow-site/pull/340
Best regards,
Laurent Querel
On Thu, Mar 30, 2023 at 3:28 PM Laurent Quérel <la...@gmail.com>
wrote:
> I'm glad to know that the article has been well-received. In the second
> article, I will allocate a dedicated section to summarize the various
> challenges encountered when using Arrow for this type of project.
>
> @Matt, I want to express my gratitude for your continuous support
> throughout this project. Your contributions and refinements to the Arrow Go
> library have enabled me to make significant progress with minimal obstacles.
>
> Best Regards, Laurent
>
> On Thu, Mar 30, 2023 at 2:24 PM Matt Topol <zo...@gmail.com> wrote:
>
>> +1 (non -binding)
>>
>> I'm glad others on here are finding this as useful and interesting as I
>> did.
>>
>> Great job Laurent!
>>
>> --Matt
>>
>> On Thu, Mar 30, 2023, 3:26 PM Raphael Taylor-Davies
>> <r....@googlemail.com.invalid> wrote:
>>
>> > Hi Laurent,
>> >
>> > I gave the first blog post a read and I also really like it and would be
>> > +1 on publishing it, nice work.
>> >
>> > I would also like to echo Will's sentiment that getting real-world case
>> > studies for the more complex Arrow schemas is invaluable and will help
>> > drive improvements in this space, so thank you for driving this forward.
>> >
>> > Kind Regards,
>> >
>> > Raphael
>> >
>> > On 30/03/2023 19:52, Will Jones wrote:
>> > > Hi Laurent,
>> > >
>> > > I have read the first post and I really like it. I'd be +1 on
>> publishing
>> > > these to the blog. I'm interested to read the second one when it's
>> > finished.
>> > >
>> > > IMO the blog could use more examples of using Arrow that's not
>> building a
>> > > data frame library / query engine, and I appreciate that this blog
>> > provides
>> > > advice for some of the trickier parts of working with complex Arrow
>> > > schemas. I think this will also provide a good concrete use case for
>> us
>> > to
>> > > think about improving the ecosystem's support for nested data.
>> > >
>> > > Best,
>> > >
>> > > Will Jones
>> > >
>> > > On Thu, Mar 30, 2023 at 10:56 AM Laurent Quérel <
>> > laurent.querel@gmail.com>
>> > > wrote:
>> > >
>> > >> Hello everyone,
>> > >>
>> > >> I was wondering if the Apache Arrow community would be interested in
>> > >> featuring a two-part article series on their blog, discussing the
>> > >> experiences and insights gained from an experimental version of the
>> > >> OpenTelemetry protocol (OTLP) utilizing Apache Arrow. As the main
>> > author of
>> > >> the OTLP Arrow specification
>> > >> <
>> >
>> https://github.com/lquerel/oteps/blob/main/text/0156-columnar-encoding.md
>> > >>> ,
>> > >> the reference implementation otlp-arrow-adapter
>> > >> <https://github.com/f5/otel-arrow-adapter>, and the two articles
>> (see
>> > >> links
>> > >> below), I believe that fostering collaboration between open-source
>> > projects
>> > >> like these is essential and mutually beneficial.
>> > >>
>> > >> These articles would serve as a fitting complement to the three
>> > >> introductory articles that Andrew Lamb and Raphael Taylor-Davies
>> > >> co-authored. They delve into the practical aspects of integrating
>> Apache
>> > >> Arrow into an existing project, as well as the process of converting
>> a
>> > >> hierarchical data model into its Arrow representation. The first
>> article
>> > >> examines various mapping techniques for aligning an existing data
>> model
>> > >> with the corresponding Arrow representation, while the second article
>> > >> explores an adaptive schema technique that I implemented in the
>> > library's
>> > >> final version in greater depth. Although the second article is still
>> > under
>> > >> development, the core framework description is already in place.
>> > >>
>> > >> What are your thoughts on this proposal?
>> > >>
>> > >> Article 1:
>> > >>
>> > >>
>> >
>> https://docs.google.com/document/d/11lG7Go2IgKOyW-RReBRW6r7HIdV1X7lu5WrDGlW5LbQ/edit?usp=sharing
>> > >>
>> > >> Article 2 (WIP):
>> > >>
>> > >>
>> >
>> https://docs.google.com/document/d/1K2CqAtF4pZjpiVts8BOcq34sOcNgozvZ9ZZw-_zTv6I/edit?usp=sharing
>> > >>
>> > >>
>> > >> Best regards,
>> > >>
>> > >> Laurent Quérel
>> > >>
>> > >> --
>> > >> Laurent Quérel
>> > >>
>> >
>>
>
>
> --
> Laurent Quérel
>
>
--
Laurent Quérel
Re: OpenTelemetry + Arrow
Posted by Laurent Quérel <la...@gmail.com>.
I'm glad to know that the article has been well-received. In the second
article, I will allocate a dedicated section to summarize the various
challenges encountered when using Arrow for this type of project.
@Matt, I want to express my gratitude for your continuous support
throughout this project. Your contributions and refinements to the Arrow Go
library have enabled me to make significant progress with minimal obstacles.
Best Regards, Laurent
On Thu, Mar 30, 2023 at 2:24 PM Matt Topol <zo...@gmail.com> wrote:
> +1 (non -binding)
>
> I'm glad others on here are finding this as useful and interesting as I
> did.
>
> Great job Laurent!
>
> --Matt
>
> On Thu, Mar 30, 2023, 3:26 PM Raphael Taylor-Davies
> <r....@googlemail.com.invalid> wrote:
>
> > Hi Laurent,
> >
> > I gave the first blog post a read and I also really like it and would be
> > +1 on publishing it, nice work.
> >
> > I would also like to echo Will's sentiment that getting real-world case
> > studies for the more complex Arrow schemas is invaluable and will help
> > drive improvements in this space, so thank you for driving this forward.
> >
> > Kind Regards,
> >
> > Raphael
> >
> > On 30/03/2023 19:52, Will Jones wrote:
> > > Hi Laurent,
> > >
> > > I have read the first post and I really like it. I'd be +1 on
> publishing
> > > these to the blog. I'm interested to read the second one when it's
> > finished.
> > >
> > > IMO the blog could use more examples of using Arrow that's not
> building a
> > > data frame library / query engine, and I appreciate that this blog
> > provides
> > > advice for some of the trickier parts of working with complex Arrow
> > > schemas. I think this will also provide a good concrete use case for us
> > to
> > > think about improving the ecosystem's support for nested data.
> > >
> > > Best,
> > >
> > > Will Jones
> > >
> > > On Thu, Mar 30, 2023 at 10:56 AM Laurent Quérel <
> > laurent.querel@gmail.com>
> > > wrote:
> > >
> > >> Hello everyone,
> > >>
> > >> I was wondering if the Apache Arrow community would be interested in
> > >> featuring a two-part article series on their blog, discussing the
> > >> experiences and insights gained from an experimental version of the
> > >> OpenTelemetry protocol (OTLP) utilizing Apache Arrow. As the main
> > author of
> > >> the OTLP Arrow specification
> > >> <
> >
> https://github.com/lquerel/oteps/blob/main/text/0156-columnar-encoding.md
> > >>> ,
> > >> the reference implementation otlp-arrow-adapter
> > >> <https://github.com/f5/otel-arrow-adapter>, and the two articles (see
> > >> links
> > >> below), I believe that fostering collaboration between open-source
> > projects
> > >> like these is essential and mutually beneficial.
> > >>
> > >> These articles would serve as a fitting complement to the three
> > >> introductory articles that Andrew Lamb and Raphael Taylor-Davies
> > >> co-authored. They delve into the practical aspects of integrating
> Apache
> > >> Arrow into an existing project, as well as the process of converting a
> > >> hierarchical data model into its Arrow representation. The first
> article
> > >> examines various mapping techniques for aligning an existing data
> model
> > >> with the corresponding Arrow representation, while the second article
> > >> explores an adaptive schema technique that I implemented in the
> > library's
> > >> final version in greater depth. Although the second article is still
> > under
> > >> development, the core framework description is already in place.
> > >>
> > >> What are your thoughts on this proposal?
> > >>
> > >> Article 1:
> > >>
> > >>
> >
> https://docs.google.com/document/d/11lG7Go2IgKOyW-RReBRW6r7HIdV1X7lu5WrDGlW5LbQ/edit?usp=sharing
> > >>
> > >> Article 2 (WIP):
> > >>
> > >>
> >
> https://docs.google.com/document/d/1K2CqAtF4pZjpiVts8BOcq34sOcNgozvZ9ZZw-_zTv6I/edit?usp=sharing
> > >>
> > >>
> > >> Best regards,
> > >>
> > >> Laurent Quérel
> > >>
> > >> --
> > >> Laurent Quérel
> > >>
> >
>
--
Laurent Quérel
Re: OpenTelemetry + Arrow
Posted by Matt Topol <zo...@gmail.com>.
+1 (non -binding)
I'm glad others on here are finding this as useful and interesting as I did.
Great job Laurent!
--Matt
On Thu, Mar 30, 2023, 3:26 PM Raphael Taylor-Davies
<r....@googlemail.com.invalid> wrote:
> Hi Laurent,
>
> I gave the first blog post a read and I also really like it and would be
> +1 on publishing it, nice work.
>
> I would also like to echo Will's sentiment that getting real-world case
> studies for the more complex Arrow schemas is invaluable and will help
> drive improvements in this space, so thank you for driving this forward.
>
> Kind Regards,
>
> Raphael
>
> On 30/03/2023 19:52, Will Jones wrote:
> > Hi Laurent,
> >
> > I have read the first post and I really like it. I'd be +1 on publishing
> > these to the blog. I'm interested to read the second one when it's
> finished.
> >
> > IMO the blog could use more examples of using Arrow that's not building a
> > data frame library / query engine, and I appreciate that this blog
> provides
> > advice for some of the trickier parts of working with complex Arrow
> > schemas. I think this will also provide a good concrete use case for us
> to
> > think about improving the ecosystem's support for nested data.
> >
> > Best,
> >
> > Will Jones
> >
> > On Thu, Mar 30, 2023 at 10:56 AM Laurent Quérel <
> laurent.querel@gmail.com>
> > wrote:
> >
> >> Hello everyone,
> >>
> >> I was wondering if the Apache Arrow community would be interested in
> >> featuring a two-part article series on their blog, discussing the
> >> experiences and insights gained from an experimental version of the
> >> OpenTelemetry protocol (OTLP) utilizing Apache Arrow. As the main
> author of
> >> the OTLP Arrow specification
> >> <
> https://github.com/lquerel/oteps/blob/main/text/0156-columnar-encoding.md
> >>> ,
> >> the reference implementation otlp-arrow-adapter
> >> <https://github.com/f5/otel-arrow-adapter>, and the two articles (see
> >> links
> >> below), I believe that fostering collaboration between open-source
> projects
> >> like these is essential and mutually beneficial.
> >>
> >> These articles would serve as a fitting complement to the three
> >> introductory articles that Andrew Lamb and Raphael Taylor-Davies
> >> co-authored. They delve into the practical aspects of integrating Apache
> >> Arrow into an existing project, as well as the process of converting a
> >> hierarchical data model into its Arrow representation. The first article
> >> examines various mapping techniques for aligning an existing data model
> >> with the corresponding Arrow representation, while the second article
> >> explores an adaptive schema technique that I implemented in the
> library's
> >> final version in greater depth. Although the second article is still
> under
> >> development, the core framework description is already in place.
> >>
> >> What are your thoughts on this proposal?
> >>
> >> Article 1:
> >>
> >>
> https://docs.google.com/document/d/11lG7Go2IgKOyW-RReBRW6r7HIdV1X7lu5WrDGlW5LbQ/edit?usp=sharing
> >>
> >> Article 2 (WIP):
> >>
> >>
> https://docs.google.com/document/d/1K2CqAtF4pZjpiVts8BOcq34sOcNgozvZ9ZZw-_zTv6I/edit?usp=sharing
> >>
> >>
> >> Best regards,
> >>
> >> Laurent Quérel
> >>
> >> --
> >> Laurent Quérel
> >>
>
Re: OpenTelemetry + Arrow
Posted by Raphael Taylor-Davies <r....@googlemail.com.INVALID>.
Hi Laurent,
I gave the first blog post a read and I also really like it and would be
+1 on publishing it, nice work.
I would also like to echo Will's sentiment that getting real-world case
studies for the more complex Arrow schemas is invaluable and will help
drive improvements in this space, so thank you for driving this forward.
Kind Regards,
Raphael
On 30/03/2023 19:52, Will Jones wrote:
> Hi Laurent,
>
> I have read the first post and I really like it. I'd be +1 on publishing
> these to the blog. I'm interested to read the second one when it's finished.
>
> IMO the blog could use more examples of using Arrow that's not building a
> data frame library / query engine, and I appreciate that this blog provides
> advice for some of the trickier parts of working with complex Arrow
> schemas. I think this will also provide a good concrete use case for us to
> think about improving the ecosystem's support for nested data.
>
> Best,
>
> Will Jones
>
> On Thu, Mar 30, 2023 at 10:56 AM Laurent Quérel <la...@gmail.com>
> wrote:
>
>> Hello everyone,
>>
>> I was wondering if the Apache Arrow community would be interested in
>> featuring a two-part article series on their blog, discussing the
>> experiences and insights gained from an experimental version of the
>> OpenTelemetry protocol (OTLP) utilizing Apache Arrow. As the main author of
>> the OTLP Arrow specification
>> <https://github.com/lquerel/oteps/blob/main/text/0156-columnar-encoding.md
>>> ,
>> the reference implementation otlp-arrow-adapter
>> <https://github.com/f5/otel-arrow-adapter>, and the two articles (see
>> links
>> below), I believe that fostering collaboration between open-source projects
>> like these is essential and mutually beneficial.
>>
>> These articles would serve as a fitting complement to the three
>> introductory articles that Andrew Lamb and Raphael Taylor-Davies
>> co-authored. They delve into the practical aspects of integrating Apache
>> Arrow into an existing project, as well as the process of converting a
>> hierarchical data model into its Arrow representation. The first article
>> examines various mapping techniques for aligning an existing data model
>> with the corresponding Arrow representation, while the second article
>> explores an adaptive schema technique that I implemented in the library's
>> final version in greater depth. Although the second article is still under
>> development, the core framework description is already in place.
>>
>> What are your thoughts on this proposal?
>>
>> Article 1:
>>
>> https://docs.google.com/document/d/11lG7Go2IgKOyW-RReBRW6r7HIdV1X7lu5WrDGlW5LbQ/edit?usp=sharing
>>
>> Article 2 (WIP):
>>
>> https://docs.google.com/document/d/1K2CqAtF4pZjpiVts8BOcq34sOcNgozvZ9ZZw-_zTv6I/edit?usp=sharing
>>
>>
>> Best regards,
>>
>> Laurent Quérel
>>
>> --
>> Laurent Quérel
>>
Re: OpenTelemetry + Arrow
Posted by Will Jones <wi...@gmail.com>.
Hi Laurent,
I have read the first post and I really like it. I'd be +1 on publishing
these to the blog. I'm interested to read the second one when it's finished.
IMO the blog could use more examples of using Arrow that's not building a
data frame library / query engine, and I appreciate that this blog provides
advice for some of the trickier parts of working with complex Arrow
schemas. I think this will also provide a good concrete use case for us to
think about improving the ecosystem's support for nested data.
Best,
Will Jones
On Thu, Mar 30, 2023 at 10:56 AM Laurent Quérel <la...@gmail.com>
wrote:
> Hello everyone,
>
> I was wondering if the Apache Arrow community would be interested in
> featuring a two-part article series on their blog, discussing the
> experiences and insights gained from an experimental version of the
> OpenTelemetry protocol (OTLP) utilizing Apache Arrow. As the main author of
> the OTLP Arrow specification
> <https://github.com/lquerel/oteps/blob/main/text/0156-columnar-encoding.md
> >,
> the reference implementation otlp-arrow-adapter
> <https://github.com/f5/otel-arrow-adapter>, and the two articles (see
> links
> below), I believe that fostering collaboration between open-source projects
> like these is essential and mutually beneficial.
>
> These articles would serve as a fitting complement to the three
> introductory articles that Andrew Lamb and Raphael Taylor-Davies
> co-authored. They delve into the practical aspects of integrating Apache
> Arrow into an existing project, as well as the process of converting a
> hierarchical data model into its Arrow representation. The first article
> examines various mapping techniques for aligning an existing data model
> with the corresponding Arrow representation, while the second article
> explores an adaptive schema technique that I implemented in the library's
> final version in greater depth. Although the second article is still under
> development, the core framework description is already in place.
>
> What are your thoughts on this proposal?
>
> Article 1:
>
> https://docs.google.com/document/d/11lG7Go2IgKOyW-RReBRW6r7HIdV1X7lu5WrDGlW5LbQ/edit?usp=sharing
>
> Article 2 (WIP):
>
> https://docs.google.com/document/d/1K2CqAtF4pZjpiVts8BOcq34sOcNgozvZ9ZZw-_zTv6I/edit?usp=sharing
>
>
> Best regards,
>
> Laurent Quérel
>
> --
> Laurent Quérel
>