One Configuration, Many File Write Formats

[Damon Douglas <do...@gmail.com>]

Hello Everyone,

I hope you are doing well.  The following design document proposes, via a
single configuration, a producer of a Beam File writing transform
supporting multiple formats.
bit.ly/fileioschematransformwriteprovider

For those new to Beam and Schema, I've added a final section of
suggested pre-requisite reading.  It's important that everyone can
participate in the conversation at any level of experience, even if this is
the first day learning Beam.  *Please feel invited to let me know anything
that isn't clear so this document can strive to include everyone.*

*My personal thoughts on the proposal's value*

I've witnessed many smart people and teams argue and divide over the
subject of a programming language.  Beam multi-language support allows us
to join transforms written in various languages, currently Java, Python,
Go, and experimentally TypeScript into a single unified pipeline.  It's
Beam's schema and processing of these objects, called Rows, that allow this
unification possible.  The aforementioned proposal continues this vision
for producing file and object system sinks via a single language agnostic
configuration and supporting provider.

Ada Lovelace dreamed of a machine that processed objects instead of just
numbers, so that they might produce music and the human things of life.
Through Beam schema awareness, let us live Ada's dream and join our
multiple languages so that we may end our strife and produce the valuable
stuff of life.

Sincerely,

Damon

Re: One Configuration, Many File Write Formats

[Damon Douglas via dev <de...@beam.apache.org>]

Thank you so much, Ahmed.  I look forward to reading your helpful comments!

On Thu, Nov 17, 2022 at 7:36 AM Ahmed Abualsaud via dev <de...@beam.apache.org>
wrote:

> Thanks for drafting this Damon! I left some comments on the doc. It's
> really cool that users can go to one source with a specified file format
> (json, avro, xml, csv, parquet) and retrieve the relevant file writing
> PTransform. I also like how the same configuration can be re-used for
> different file formats. Looking forward to seeing its implementation :)
>
> Ahmed
>
> On Mon, Nov 14, 2022 at 11:53 AM Damon Douglas <do...@gmail.com>
> wrote:
>
>> Hello Everyone,
>>
>> I hope you are doing well.  The following design document proposes, via a
>> single configuration, a producer of a Beam File writing transform
>> supporting multiple formats.
>> bit.ly/fileioschematransformwriteprovider
>>
>> For those new to Beam and Schema, I've added a final section of
>> suggested pre-requisite reading.  It's important that everyone can
>> participate in the conversation at any level of experience, even if this is
>> the first day learning Beam.  *Please feel invited to let me know
>> anything that isn't clear so this document can strive to include everyone.*
>>
>> *My personal thoughts on the proposal's value*
>>
>> I've witnessed many smart people and teams argue and divide over the
>> subject of a programming language.  Beam multi-language support allows us
>> to join transforms written in various languages, currently Java, Python,
>> Go, and experimentally TypeScript into a single unified pipeline.  It's
>> Beam's schema and processing of these objects, called Rows, that allow this
>> unification possible.  The aforementioned proposal continues this vision
>> for producing file and object system sinks via a single language agnostic
>> configuration and supporting provider.
>>
>> Ada Lovelace dreamed of a machine that processed objects instead of just
>> numbers, so that they might produce music and the human things of life.
>> Through Beam schema awareness, let us live Ada's dream and join our
>> multiple languages so that we may end our strife and produce the valuable
>> stuff of life.
>>
>> Sincerely,
>>
>> Damon
>>
>

Re: One Configuration, Many File Write Formats

[Ahmed Abualsaud via dev <de...@beam.apache.org>]

Thanks for drafting this Damon! I left some comments on the doc. It's
really cool that users can go to one source with a specified file format
(json, avro, xml, csv, parquet) and retrieve the relevant file writing
PTransform. I also like how the same configuration can be re-used for
different file formats. Looking forward to seeing its implementation :)

Ahmed

On Mon, Nov 14, 2022 at 11:53 AM Damon Douglas <do...@gmail.com>
wrote:

> Hello Everyone,
>
> I hope you are doing well.  The following design document proposes, via a
> single configuration, a producer of a Beam File writing transform
> supporting multiple formats.
> bit.ly/fileioschematransformwriteprovider
>
> For those new to Beam and Schema, I've added a final section of
> suggested pre-requisite reading.  It's important that everyone can
> participate in the conversation at any level of experience, even if this is
> the first day learning Beam.  *Please feel invited to let me know
> anything that isn't clear so this document can strive to include everyone.*
>
> *My personal thoughts on the proposal's value*
>
> I've witnessed many smart people and teams argue and divide over the
> subject of a programming language.  Beam multi-language support allows us
> to join transforms written in various languages, currently Java, Python,
> Go, and experimentally TypeScript into a single unified pipeline.  It's
> Beam's schema and processing of these objects, called Rows, that allow this
> unification possible.  The aforementioned proposal continues this vision
> for producing file and object system sinks via a single language agnostic
> configuration and supporting provider.
>
> Ada Lovelace dreamed of a machine that processed objects instead of just
> numbers, so that they might produce music and the human things of life.
> Through Beam schema awareness, let us live Ada's dream and join our
> multiple languages so that we may end our strife and produce the valuable
> stuff of life.
>
> Sincerely,
>
> Damon
>