You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@avro.apache.org by Bill Graham <bi...@gmail.com> on 2011/08/16 22:38:37 UTC
Compiling multiple input schemas
Hi,
With Avro-874, multiple inter-dependent schema files can be parsed. I've
written a patch to the SpecificCompilerTool to allow the same when producing
java from multiple schemas that I'd like to contribute for consistency if
there's interest. It allows you to pass multiple input files like this:
$ java -cp avro-tools-1.6.0.jar org.apache.avro.tool.Main compile schema
input/position.avsc,input/player.avsc tmp/
While I was at it, it seemed useful to parse an entire directory of schema
files as well so I implemented this:
$ java -cp avro-tools-1.6.0.jar org.apache.avro.tool.Main compile schema
input/ tmp/
The latter approach will not work properly for files with dependencies,
since the file order probably isn't in reverse dependency order. If that's
the case, a combination of files and directories can be used to force an
ordering. So if b depends on a and other files depend on either of them you
could do this:
$ java -cp avro-tools-1.6.0.jar org.apache.avro.tool.Main compile schema
input/a.avsc,input/b.avsc,input/ tmp/
Let me know if some or all of this seems useful to contribute. The first
example is really the main one that I need. I've done the same for Protocol
as well btw.
thanks,
Bill
Re: Compiling multiple input schemas
Posted by Doug Cutting <cu...@apache.org>.
+1
This sounds like a useful addition to me.
Instead of a comma, might you use a space, so that the last parameter on
the command line was the output directory and all preceding are inputs?
Doug
On 08/16/2011 01:38 PM, Bill Graham wrote:
> Hi,
>
> With Avro-874, multiple inter-dependent schema files can be parsed. I've
> written a patch to the SpecificCompilerTool to allow the same when
> producing java from multiple schemas that I'd like to contribute for
> consistency if there's interest. It allows you to pass multiple input
> files like this:
>
> $ java -cp avro-tools-1.6.0.jar org.apache.avro.tool.Main compile schema
> input/position.avsc,input/player.avsc tmp/
>
> While I was at it, it seemed useful to parse an entire directory of
> schema files as well so I implemented this:
>
> $ java -cp avro-tools-1.6.0.jar org.apache.avro.tool.Main compile schema
> input/ tmp/
>
> The latter approach will not work properly for files with dependencies,
> since the file order probably isn't in reverse dependency order. If
> that's the case, a combination of files and directories can be used to
> force an ordering. So if b depends on a and other files depend on either
> of them you could do this:
>
> $ java -cp avro-tools-1.6.0.jar org.apache.avro.tool.Main compile schema
> input/a.avsc,input/b.avsc,input/ tmp/
>
> Let me know if some or all of this seems useful to contribute. The first
> example is really the main one that I need. I've done the same for
> Protocol as well btw.
>
> thanks,
> Bill
Re: Compiling multiple input schemas
Posted by Bill Graham <bi...@gmail.com>.
Works for me:
https://issues.apache.org/jira/browse/AVRO-877
On Tue, Aug 16, 2011 at 3:00 PM, Doug Cutting <cu...@apache.org> wrote:
> On 08/16/2011 02:52 PM, Scott Carey wrote:
> > That would be incompatible, since we reversed argument order. But it
> > would be more like other unix command line tools that take lists of
> > files and output results somewhere else.
> >
> > (Or as I see Doug has just replied — the last argument can be the
> > destination)
>
> Destination last is like Unix 'cp' and 'mv' and is back-compatible.
>
> Doug
>
Re: Compiling multiple input schemas
Posted by Doug Cutting <cu...@apache.org>.
On 08/16/2011 02:52 PM, Scott Carey wrote:
> That would be incompatible, since we reversed argument order. But it
> would be more like other unix command line tools that take lists of
> files and output results somewhere else.
>
> (Or as I see Doug has just replied — the last argument can be the
> destination)
Destination last is like Unix 'cp' and 'mv' and is back-compatible.
Doug
Re: Compiling multiple input schemas
Posted by Scott Carey <sc...@apache.org>.
What about leveraging shell expansion?
This would mean we would need inverse syntax, like tar or zip ( destination,
list of sources in reverse dependency order )
Then your examples are
avro-tools-1.6.0.jar compile schema tmp/ input/position.avsc
input/player.avsc
avro-tools-1.6.0.jar compile schema tmp/ input/*
That would be incompatible, since we reversed argument order. But it would
be more like other unix command line tools that take lists of files and
output results somewhere else.
(Or as I see Doug has just replied the last argument can be the
destination)
On 8/16/11 1:38 PM, "Bill Graham" <bi...@gmail.com> wrote:
> Hi,
>
> With Avro-874, multiple inter-dependent schema files can be parsed. I've
> written a patch to the SpecificCompilerTool to allow the same when producing
> java from multiple schemas that I'd like to contribute for consistency if
> there's interest. It allows you to pass multiple input files like this:
>
> $ java -cp avro-tools-1.6.0.jar org.apache.avro.tool.Main compile schema
> input/position.avsc,input/player.avsc tmp/
>
> While I was at it, it seemed useful to parse an entire directory of schema
> files as well so I implemented this:
>
> $ java -cp avro-tools-1.6.0.jar org.apache.avro.tool.Main compile schema
> input/ tmp/
>
> The latter approach will not work properly for files with dependencies, since
> the file order probably isn't in reverse dependency order. If that's the case,
> a combination of files and directories can be used to force an ordering. So if
> b depends on a and other files depend on either of them you could do this:
>
> $ java -cp avro-tools-1.6.0.jar org.apache.avro.tool.Main compile schema
> input/a.avsc,input/b.avsc,input/ tmp/
>
> Let me know if some or all of this seems useful to contribute. The first
> example is really the main one that I need. I've done the same for Protocol as
> well btw.
>
> thanks,
> Bill