You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Laurent Quérel <la...@gmail.com> on 2023/04/03 01:07:18 UTC
Re: OpenTelemetry + Arrow
Hi Andrew,
The feedback seems to be good so I created a PR.
https://github.com/apache/arrow-site/pull/340
Best regards,
Laurent Querel
On Thu, Mar 30, 2023 at 3:28 PM Laurent Quérel <la...@gmail.com>
wrote:
> I'm glad to know that the article has been well-received. In the second
> article, I will allocate a dedicated section to summarize the various
> challenges encountered when using Arrow for this type of project.
>
> @Matt, I want to express my gratitude for your continuous support
> throughout this project. Your contributions and refinements to the Arrow Go
> library have enabled me to make significant progress with minimal obstacles.
>
> Best Regards, Laurent
>
> On Thu, Mar 30, 2023 at 2:24 PM Matt Topol <zo...@gmail.com> wrote:
>
>> +1 (non -binding)
>>
>> I'm glad others on here are finding this as useful and interesting as I
>> did.
>>
>> Great job Laurent!
>>
>> --Matt
>>
>> On Thu, Mar 30, 2023, 3:26 PM Raphael Taylor-Davies
>> <r....@googlemail.com.invalid> wrote:
>>
>> > Hi Laurent,
>> >
>> > I gave the first blog post a read and I also really like it and would be
>> > +1 on publishing it, nice work.
>> >
>> > I would also like to echo Will's sentiment that getting real-world case
>> > studies for the more complex Arrow schemas is invaluable and will help
>> > drive improvements in this space, so thank you for driving this forward.
>> >
>> > Kind Regards,
>> >
>> > Raphael
>> >
>> > On 30/03/2023 19:52, Will Jones wrote:
>> > > Hi Laurent,
>> > >
>> > > I have read the first post and I really like it. I'd be +1 on
>> publishing
>> > > these to the blog. I'm interested to read the second one when it's
>> > finished.
>> > >
>> > > IMO the blog could use more examples of using Arrow that's not
>> building a
>> > > data frame library / query engine, and I appreciate that this blog
>> > provides
>> > > advice for some of the trickier parts of working with complex Arrow
>> > > schemas. I think this will also provide a good concrete use case for
>> us
>> > to
>> > > think about improving the ecosystem's support for nested data.
>> > >
>> > > Best,
>> > >
>> > > Will Jones
>> > >
>> > > On Thu, Mar 30, 2023 at 10:56 AM Laurent Quérel <
>> > laurent.querel@gmail.com>
>> > > wrote:
>> > >
>> > >> Hello everyone,
>> > >>
>> > >> I was wondering if the Apache Arrow community would be interested in
>> > >> featuring a two-part article series on their blog, discussing the
>> > >> experiences and insights gained from an experimental version of the
>> > >> OpenTelemetry protocol (OTLP) utilizing Apache Arrow. As the main
>> > author of
>> > >> the OTLP Arrow specification
>> > >> <
>> >
>> https://github.com/lquerel/oteps/blob/main/text/0156-columnar-encoding.md
>> > >>> ,
>> > >> the reference implementation otlp-arrow-adapter
>> > >> <https://github.com/f5/otel-arrow-adapter>, and the two articles
>> (see
>> > >> links
>> > >> below), I believe that fostering collaboration between open-source
>> > projects
>> > >> like these is essential and mutually beneficial.
>> > >>
>> > >> These articles would serve as a fitting complement to the three
>> > >> introductory articles that Andrew Lamb and Raphael Taylor-Davies
>> > >> co-authored. They delve into the practical aspects of integrating
>> Apache
>> > >> Arrow into an existing project, as well as the process of converting
>> a
>> > >> hierarchical data model into its Arrow representation. The first
>> article
>> > >> examines various mapping techniques for aligning an existing data
>> model
>> > >> with the corresponding Arrow representation, while the second article
>> > >> explores an adaptive schema technique that I implemented in the
>> > library's
>> > >> final version in greater depth. Although the second article is still
>> > under
>> > >> development, the core framework description is already in place.
>> > >>
>> > >> What are your thoughts on this proposal?
>> > >>
>> > >> Article 1:
>> > >>
>> > >>
>> >
>> https://docs.google.com/document/d/11lG7Go2IgKOyW-RReBRW6r7HIdV1X7lu5WrDGlW5LbQ/edit?usp=sharing
>> > >>
>> > >> Article 2 (WIP):
>> > >>
>> > >>
>> >
>> https://docs.google.com/document/d/1K2CqAtF4pZjpiVts8BOcq34sOcNgozvZ9ZZw-_zTv6I/edit?usp=sharing
>> > >>
>> > >>
>> > >> Best regards,
>> > >>
>> > >> Laurent Quérel
>> > >>
>> > >> --
>> > >> Laurent Quérel
>> > >>
>> >
>>
>
>
> --
> Laurent Quérel
>
>
--
Laurent Quérel
Re: OpenTelemetry + Arrow
Posted by Laurent Quérel <la...@gmail.com>.
Thank you very much Andrew.
I should be able to work on the second article next week and I will follow
the same process.
Cheers, Laurent
On Tue, Apr 11, 2023 at 4:31 AM Andrew Lamb <al...@influxdata.com> wrote:
> The blog post is now live on the arrow site [1]
>
> Thanks again Laurent
>
> [1]:
>
> https://arrow.apache.org/blog/2023/04/11/our-journey-at-f5-with-apache-arrow-part-1/
>
> On Sun, Apr 2, 2023 at 9:07 PM Laurent Quérel <la...@gmail.com>
> wrote:
>
> > Hi Andrew,
> >
> > The feedback seems to be good so I created a PR.
> >
> > https://github.com/apache/arrow-site/pull/340
> >
> > Best regards,
> >
> > Laurent Querel
> >
> > On Thu, Mar 30, 2023 at 3:28 PM Laurent Quérel <laurent.querel@gmail.com
> >
> > wrote:
> >
> > > I'm glad to know that the article has been well-received. In the second
> > > article, I will allocate a dedicated section to summarize the various
> > > challenges encountered when using Arrow for this type of project.
> > >
> > > @Matt, I want to express my gratitude for your continuous support
> > > throughout this project. Your contributions and refinements to the
> Arrow
> > Go
> > > library have enabled me to make significant progress with minimal
> > obstacles.
> > >
> > > Best Regards, Laurent
> > >
> > > On Thu, Mar 30, 2023 at 2:24 PM Matt Topol <zo...@gmail.com>
> > wrote:
> > >
> > >> +1 (non -binding)
> > >>
> > >> I'm glad others on here are finding this as useful and interesting as
> I
> > >> did.
> > >>
> > >> Great job Laurent!
> > >>
> > >> --Matt
> > >>
> > >> On Thu, Mar 30, 2023, 3:26 PM Raphael Taylor-Davies
> > >> <r....@googlemail.com.invalid> wrote:
> > >>
> > >> > Hi Laurent,
> > >> >
> > >> > I gave the first blog post a read and I also really like it and
> would
> > be
> > >> > +1 on publishing it, nice work.
> > >> >
> > >> > I would also like to echo Will's sentiment that getting real-world
> > case
> > >> > studies for the more complex Arrow schemas is invaluable and will
> help
> > >> > drive improvements in this space, so thank you for driving this
> > forward.
> > >> >
> > >> > Kind Regards,
> > >> >
> > >> > Raphael
> > >> >
> > >> > On 30/03/2023 19:52, Will Jones wrote:
> > >> > > Hi Laurent,
> > >> > >
> > >> > > I have read the first post and I really like it. I'd be +1 on
> > >> publishing
> > >> > > these to the blog. I'm interested to read the second one when it's
> > >> > finished.
> > >> > >
> > >> > > IMO the blog could use more examples of using Arrow that's not
> > >> building a
> > >> > > data frame library / query engine, and I appreciate that this blog
> > >> > provides
> > >> > > advice for some of the trickier parts of working with complex
> Arrow
> > >> > > schemas. I think this will also provide a good concrete use case
> for
> > >> us
> > >> > to
> > >> > > think about improving the ecosystem's support for nested data.
> > >> > >
> > >> > > Best,
> > >> > >
> > >> > > Will Jones
> > >> > >
> > >> > > On Thu, Mar 30, 2023 at 10:56 AM Laurent Quérel <
> > >> > laurent.querel@gmail.com>
> > >> > > wrote:
> > >> > >
> > >> > >> Hello everyone,
> > >> > >>
> > >> > >> I was wondering if the Apache Arrow community would be interested
> > in
> > >> > >> featuring a two-part article series on their blog, discussing the
> > >> > >> experiences and insights gained from an experimental version of
> the
> > >> > >> OpenTelemetry protocol (OTLP) utilizing Apache Arrow. As the main
> > >> > author of
> > >> > >> the OTLP Arrow specification
> > >> > >> <
> > >> >
> > >>
> >
> https://github.com/lquerel/oteps/blob/main/text/0156-columnar-encoding.md
> > >> > >>> ,
> > >> > >> the reference implementation otlp-arrow-adapter
> > >> > >> <https://github.com/f5/otel-arrow-adapter>, and the two articles
> > >> (see
> > >> > >> links
> > >> > >> below), I believe that fostering collaboration between
> open-source
> > >> > projects
> > >> > >> like these is essential and mutually beneficial.
> > >> > >>
> > >> > >> These articles would serve as a fitting complement to the three
> > >> > >> introductory articles that Andrew Lamb and Raphael Taylor-Davies
> > >> > >> co-authored. They delve into the practical aspects of integrating
> > >> Apache
> > >> > >> Arrow into an existing project, as well as the process of
> > converting
> > >> a
> > >> > >> hierarchical data model into its Arrow representation. The first
> > >> article
> > >> > >> examines various mapping techniques for aligning an existing data
> > >> model
> > >> > >> with the corresponding Arrow representation, while the second
> > article
> > >> > >> explores an adaptive schema technique that I implemented in the
> > >> > library's
> > >> > >> final version in greater depth. Although the second article is
> > still
> > >> > under
> > >> > >> development, the core framework description is already in place.
> > >> > >>
> > >> > >> What are your thoughts on this proposal?
> > >> > >>
> > >> > >> Article 1:
> > >> > >>
> > >> > >>
> > >> >
> > >>
> >
> https://docs.google.com/document/d/11lG7Go2IgKOyW-RReBRW6r7HIdV1X7lu5WrDGlW5LbQ/edit?usp=sharing
> > >> > >>
> > >> > >> Article 2 (WIP):
> > >> > >>
> > >> > >>
> > >> >
> > >>
> >
> https://docs.google.com/document/d/1K2CqAtF4pZjpiVts8BOcq34sOcNgozvZ9ZZw-_zTv6I/edit?usp=sharing
> > >> > >>
> > >> > >>
> > >> > >> Best regards,
> > >> > >>
> > >> > >> Laurent Quérel
> > >> > >>
> > >> > >> --
> > >> > >> Laurent Quérel
> > >> > >>
> > >> >
> > >>
> > >
> > >
> > > --
> > > Laurent Quérel
> > >
> > >
> >
> >
> > --
> > Laurent Quérel
> >
>
--
Laurent Quérel
Re: OpenTelemetry + Arrow
Posted by Andrew Lamb <al...@influxdata.com>.
The blog post is now live on the arrow site [1]
Thanks again Laurent
[1]:
https://arrow.apache.org/blog/2023/04/11/our-journey-at-f5-with-apache-arrow-part-1/
On Sun, Apr 2, 2023 at 9:07 PM Laurent Quérel <la...@gmail.com>
wrote:
> Hi Andrew,
>
> The feedback seems to be good so I created a PR.
>
> https://github.com/apache/arrow-site/pull/340
>
> Best regards,
>
> Laurent Querel
>
> On Thu, Mar 30, 2023 at 3:28 PM Laurent Quérel <la...@gmail.com>
> wrote:
>
> > I'm glad to know that the article has been well-received. In the second
> > article, I will allocate a dedicated section to summarize the various
> > challenges encountered when using Arrow for this type of project.
> >
> > @Matt, I want to express my gratitude for your continuous support
> > throughout this project. Your contributions and refinements to the Arrow
> Go
> > library have enabled me to make significant progress with minimal
> obstacles.
> >
> > Best Regards, Laurent
> >
> > On Thu, Mar 30, 2023 at 2:24 PM Matt Topol <zo...@gmail.com>
> wrote:
> >
> >> +1 (non -binding)
> >>
> >> I'm glad others on here are finding this as useful and interesting as I
> >> did.
> >>
> >> Great job Laurent!
> >>
> >> --Matt
> >>
> >> On Thu, Mar 30, 2023, 3:26 PM Raphael Taylor-Davies
> >> <r....@googlemail.com.invalid> wrote:
> >>
> >> > Hi Laurent,
> >> >
> >> > I gave the first blog post a read and I also really like it and would
> be
> >> > +1 on publishing it, nice work.
> >> >
> >> > I would also like to echo Will's sentiment that getting real-world
> case
> >> > studies for the more complex Arrow schemas is invaluable and will help
> >> > drive improvements in this space, so thank you for driving this
> forward.
> >> >
> >> > Kind Regards,
> >> >
> >> > Raphael
> >> >
> >> > On 30/03/2023 19:52, Will Jones wrote:
> >> > > Hi Laurent,
> >> > >
> >> > > I have read the first post and I really like it. I'd be +1 on
> >> publishing
> >> > > these to the blog. I'm interested to read the second one when it's
> >> > finished.
> >> > >
> >> > > IMO the blog could use more examples of using Arrow that's not
> >> building a
> >> > > data frame library / query engine, and I appreciate that this blog
> >> > provides
> >> > > advice for some of the trickier parts of working with complex Arrow
> >> > > schemas. I think this will also provide a good concrete use case for
> >> us
> >> > to
> >> > > think about improving the ecosystem's support for nested data.
> >> > >
> >> > > Best,
> >> > >
> >> > > Will Jones
> >> > >
> >> > > On Thu, Mar 30, 2023 at 10:56 AM Laurent Quérel <
> >> > laurent.querel@gmail.com>
> >> > > wrote:
> >> > >
> >> > >> Hello everyone,
> >> > >>
> >> > >> I was wondering if the Apache Arrow community would be interested
> in
> >> > >> featuring a two-part article series on their blog, discussing the
> >> > >> experiences and insights gained from an experimental version of the
> >> > >> OpenTelemetry protocol (OTLP) utilizing Apache Arrow. As the main
> >> > author of
> >> > >> the OTLP Arrow specification
> >> > >> <
> >> >
> >>
> https://github.com/lquerel/oteps/blob/main/text/0156-columnar-encoding.md
> >> > >>> ,
> >> > >> the reference implementation otlp-arrow-adapter
> >> > >> <https://github.com/f5/otel-arrow-adapter>, and the two articles
> >> (see
> >> > >> links
> >> > >> below), I believe that fostering collaboration between open-source
> >> > projects
> >> > >> like these is essential and mutually beneficial.
> >> > >>
> >> > >> These articles would serve as a fitting complement to the three
> >> > >> introductory articles that Andrew Lamb and Raphael Taylor-Davies
> >> > >> co-authored. They delve into the practical aspects of integrating
> >> Apache
> >> > >> Arrow into an existing project, as well as the process of
> converting
> >> a
> >> > >> hierarchical data model into its Arrow representation. The first
> >> article
> >> > >> examines various mapping techniques for aligning an existing data
> >> model
> >> > >> with the corresponding Arrow representation, while the second
> article
> >> > >> explores an adaptive schema technique that I implemented in the
> >> > library's
> >> > >> final version in greater depth. Although the second article is
> still
> >> > under
> >> > >> development, the core framework description is already in place.
> >> > >>
> >> > >> What are your thoughts on this proposal?
> >> > >>
> >> > >> Article 1:
> >> > >>
> >> > >>
> >> >
> >>
> https://docs.google.com/document/d/11lG7Go2IgKOyW-RReBRW6r7HIdV1X7lu5WrDGlW5LbQ/edit?usp=sharing
> >> > >>
> >> > >> Article 2 (WIP):
> >> > >>
> >> > >>
> >> >
> >>
> https://docs.google.com/document/d/1K2CqAtF4pZjpiVts8BOcq34sOcNgozvZ9ZZw-_zTv6I/edit?usp=sharing
> >> > >>
> >> > >>
> >> > >> Best regards,
> >> > >>
> >> > >> Laurent Quérel
> >> > >>
> >> > >> --
> >> > >> Laurent Quérel
> >> > >>
> >> >
> >>
> >
> >
> > --
> > Laurent Quérel
> >
> >
>
>
> --
> Laurent Quérel
>