You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@avro.apache.org by Anirudha Jadhav <an...@nyu.edu> on 2015/08/31 21:46:20 UTC
Avro Tranformation Language / Avro Graph
Hello All,
I would like to introduce you all to a project which we have been working
on using Avro and get some feedback.
1. AvroGraph
------------
We have created an avro to graphml serializer / deserializer. This allows
us to visualize avro schemas in a graph to understand the relation between
all the data points. This will later lead to creation of lineage graphs
among other things
- Implementation
o similar to json serializer / deserializer
o Apache Tinkerpop is used as a graph library and can be used to persist
to a variety of graph stores.
o support for scheme evolution between multiple version of the avro
schemas
o lot of unit tests and documentation
2. Avro Transformation Language
-------------------------------
This is YAML based specification that will transform a data in a source
schema to a target schema. For this we introduce a "transform node" to join
the two schemas
- The following operations can be done during the source to target data
transformations
o Copy source leaves to target leaves
o Copy source parent nodes to target parent nodes, only if the sub
graphs have the same structure.
o Concatenate source nodes and copy to a target node
o User-defined operations on the transforms
o Extract certain leaves from the source and call an external end point
for data manipulation eg. Spark / Http
Let me know how/if these components would benefit the apache avro project
and accordingly we would like to contribute it to the apache avro project.
-Ani
Re: Avro Tranformation Language / Avro Graph
Posted by Anirudha Jadhav <an...@nyu.edu>.
I need to go through approval process to release source, I wanted some feed
back while that process in in progress. Let me know if you have any
question or comments that I can address
- Anirudha Jadhav
On Mon, Aug 31, 2015 at 4:09 PM, Prajwal Tuladhar <pr...@infynyxx.com> wrote:
> Hi Anirudha,
>
> Is it possible to view the source of those projects?
>
> On Mon, Aug 31, 2015 at 7:46 PM, Anirudha Jadhav <an...@nyu.edu> wrote:
>
>> Hello All,
>>
>> I would like to introduce you all to a project which we have been working
>> on using Avro and get some feedback.
>>
>> 1. AvroGraph
>> ------------
>> We have created an avro to graphml serializer / deserializer. This allows
>> us to visualize avro schemas in a graph to understand the relation between
>> all the data points. This will later lead to creation of lineage graphs
>> among other things
>> - Implementation
>> o similar to json serializer / deserializer
>> o Apache Tinkerpop is used as a graph library and can be used to
>> persist to a variety of graph stores.
>> o support for scheme evolution between multiple version of the avro
>> schemas
>> o lot of unit tests and documentation
>>
>> 2. Avro Transformation Language
>> -------------------------------
>> This is YAML based specification that will transform a data in a source
>> schema to a target schema. For this we introduce a "transform node" to join
>> the two schemas
>> - The following operations can be done during the source to target data
>> transformations
>> o Copy source leaves to target leaves
>> o Copy source parent nodes to target parent nodes, only if the sub
>> graphs have the same structure.
>> o Concatenate source nodes and copy to a target node
>> o User-defined operations on the transforms
>> o Extract certain leaves from the source and call an external end
>> point for data manipulation eg. Spark / Http
>>
>>
>> Let me know how/if these components would benefit the apache avro project
>> and accordingly we would like to contribute it to the apache avro project.
>>
>> -Ani
>>
>
>
>
> --
> --
> Cheers,
> Praj
>
--
Anirudha P. Jadhav
Re: Avro Tranformation Language / Avro Graph
Posted by Anirudha Jadhav <an...@nyu.edu>.
I need to go through approval process to release source, I wanted some feed
back while that process in in progress. Let me know if you have any
question or comments that I can address
- Anirudha Jadhav
On Mon, Aug 31, 2015 at 4:09 PM, Prajwal Tuladhar <pr...@infynyxx.com> wrote:
> Hi Anirudha,
>
> Is it possible to view the source of those projects?
>
> On Mon, Aug 31, 2015 at 7:46 PM, Anirudha Jadhav <an...@nyu.edu> wrote:
>
>> Hello All,
>>
>> I would like to introduce you all to a project which we have been working
>> on using Avro and get some feedback.
>>
>> 1. AvroGraph
>> ------------
>> We have created an avro to graphml serializer / deserializer. This allows
>> us to visualize avro schemas in a graph to understand the relation between
>> all the data points. This will later lead to creation of lineage graphs
>> among other things
>> - Implementation
>> o similar to json serializer / deserializer
>> o Apache Tinkerpop is used as a graph library and can be used to
>> persist to a variety of graph stores.
>> o support for scheme evolution between multiple version of the avro
>> schemas
>> o lot of unit tests and documentation
>>
>> 2. Avro Transformation Language
>> -------------------------------
>> This is YAML based specification that will transform a data in a source
>> schema to a target schema. For this we introduce a "transform node" to join
>> the two schemas
>> - The following operations can be done during the source to target data
>> transformations
>> o Copy source leaves to target leaves
>> o Copy source parent nodes to target parent nodes, only if the sub
>> graphs have the same structure.
>> o Concatenate source nodes and copy to a target node
>> o User-defined operations on the transforms
>> o Extract certain leaves from the source and call an external end
>> point for data manipulation eg. Spark / Http
>>
>>
>> Let me know how/if these components would benefit the apache avro project
>> and accordingly we would like to contribute it to the apache avro project.
>>
>> -Ani
>>
>
>
>
> --
> --
> Cheers,
> Praj
>
--
Anirudha P. Jadhav
Re: Avro Tranformation Language / Avro Graph
Posted by Anirudha Jadhav <an...@nyu.edu>.
I need to go through approval process to release source, I wanted some feed
back while that process in in progress. Let me know if you have any
question or comments that I can address
- Anirudha Jadhav
On Mon, Aug 31, 2015 at 4:09 PM, Prajwal Tuladhar <pr...@infynyxx.com> wrote:
> Hi Anirudha,
>
> Is it possible to view the source of those projects?
>
> On Mon, Aug 31, 2015 at 7:46 PM, Anirudha Jadhav <an...@nyu.edu> wrote:
>
>> Hello All,
>>
>> I would like to introduce you all to a project which we have been working
>> on using Avro and get some feedback.
>>
>> 1. AvroGraph
>> ------------
>> We have created an avro to graphml serializer / deserializer. This allows
>> us to visualize avro schemas in a graph to understand the relation between
>> all the data points. This will later lead to creation of lineage graphs
>> among other things
>> - Implementation
>> o similar to json serializer / deserializer
>> o Apache Tinkerpop is used as a graph library and can be used to
>> persist to a variety of graph stores.
>> o support for scheme evolution between multiple version of the avro
>> schemas
>> o lot of unit tests and documentation
>>
>> 2. Avro Transformation Language
>> -------------------------------
>> This is YAML based specification that will transform a data in a source
>> schema to a target schema. For this we introduce a "transform node" to join
>> the two schemas
>> - The following operations can be done during the source to target data
>> transformations
>> o Copy source leaves to target leaves
>> o Copy source parent nodes to target parent nodes, only if the sub
>> graphs have the same structure.
>> o Concatenate source nodes and copy to a target node
>> o User-defined operations on the transforms
>> o Extract certain leaves from the source and call an external end
>> point for data manipulation eg. Spark / Http
>>
>>
>> Let me know how/if these components would benefit the apache avro project
>> and accordingly we would like to contribute it to the apache avro project.
>>
>> -Ani
>>
>
>
>
> --
> --
> Cheers,
> Praj
>
--
Anirudha P. Jadhav
Re: Avro Tranformation Language / Avro Graph
Posted by Prajwal Tuladhar <pr...@infynyxx.com>.
Hi Anirudha,
Is it possible to view the source of those projects?
On Mon, Aug 31, 2015 at 7:46 PM, Anirudha Jadhav <an...@nyu.edu> wrote:
> Hello All,
>
> I would like to introduce you all to a project which we have been working
> on using Avro and get some feedback.
>
> 1. AvroGraph
> ------------
> We have created an avro to graphml serializer / deserializer. This allows
> us to visualize avro schemas in a graph to understand the relation between
> all the data points. This will later lead to creation of lineage graphs
> among other things
> - Implementation
> o similar to json serializer / deserializer
> o Apache Tinkerpop is used as a graph library and can be used to persist
> to a variety of graph stores.
> o support for scheme evolution between multiple version of the avro
> schemas
> o lot of unit tests and documentation
>
> 2. Avro Transformation Language
> -------------------------------
> This is YAML based specification that will transform a data in a source
> schema to a target schema. For this we introduce a "transform node" to join
> the two schemas
> - The following operations can be done during the source to target data
> transformations
> o Copy source leaves to target leaves
> o Copy source parent nodes to target parent nodes, only if the sub
> graphs have the same structure.
> o Concatenate source nodes and copy to a target node
> o User-defined operations on the transforms
> o Extract certain leaves from the source and call an external end point
> for data manipulation eg. Spark / Http
>
>
> Let me know how/if these components would benefit the apache avro project
> and accordingly we would like to contribute it to the apache avro project.
>
> -Ani
>
--
--
Cheers,
Praj