You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "Bill Graham (JIRA)" <ji...@apache.org> on 2011/08/17 00:21:27 UTC
[jira] [Created] (AVRO-877) Support compiling multiple input
schemas and protocols
Support compiling multiple input schemas and protocols
------------------------------------------------------
Key: AVRO-877
URL: https://issues.apache.org/jira/browse/AVRO-877
Project: Avro
Issue Type: Improvement
Reporter: Bill Graham
Assignee: Bill Graham
With Avro-874, multiple inter-dependent schema files can be parsed. {{SpecificCompilerTool}} should be changed to allow the same when producing java from multiple schemas or protocols. Proposed syntax change is to allow multiple files or directories to be passed (space-delimited) before the last token, which is the output path.
- To compile multiple files where {{b}} depends on {{a}}:
{noformat}
$ java -cp avro-tools-x.x.x.jar org.apache.avro.tool.Main \
compile schema input/a.avsc input/b.avsc output
{noformat}
- To compile files from multiple directories:
{noformat}
$ java -cp avro-tools-x.x.x.jar org.apache.avro.tool.Main \
compile schema input1 input2 output
{noformat}
- To compile {{b}} which depends on {{a}} and all the other content in {{input}} that might depend on {{a}} or {{b}}:
{noformat}
$ java -cp avro-tools-x.x.x.jar org.apache.avro.tool.Main \
compile schema input/a.avsc input/b.avsc input output
{noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-877) Support compiling multiple input
schemas and protocols
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087848#comment-13087848 ]
Doug Cutting commented on AVRO-877:
-----------------------------------
Looks good in general.
I think the System.out statements should probably be System.err. We should only use System.out from command line programs for output data.
Also, should we add a test or two for the new functionality?
> Support compiling multiple input schemas and protocols
> ------------------------------------------------------
>
> Key: AVRO-877
> URL: https://issues.apache.org/jira/browse/AVRO-877
> Project: Avro
> Issue Type: Improvement
> Reporter: Bill Graham
> Assignee: Bill Graham
> Attachments: AVRO-877_1.patch
>
>
> With AVRO-872, multiple inter-dependent schema files can be parsed. {{SpecificCompilerTool}} should be changed to allow the same when producing java from multiple schemas or protocols. Proposed syntax change is to allow multiple files or directories to be passed (space-delimited) before the last token, which is the output path.
> - To compile multiple files where {{b}} depends on {{a}}:
> {noformat}
> $ java -cp avro-tools-x.x.x.jar org.apache.avro.tool.Main \
> compile schema input/a.avsc input/b.avsc output
> {noformat}
> - To compile files from multiple directories:
> {noformat}
> $ java -cp avro-tools-x.x.x.jar org.apache.avro.tool.Main \
> compile schema input1 input2 output
> {noformat}
> - To compile {{b}} which depends on {{a}} and all the other content in {{input}} that might depend on {{a}} or {{b}}:
> {noformat}
> $ java -cp avro-tools-x.x.x.jar org.apache.avro.tool.Main \
> compile schema input/a.avsc input/b.avsc input output
> {noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-877) Support compiling multiple input
schemas and protocols
Posted by "Bill Graham (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bill Graham updated AVRO-877:
-----------------------------
Description:
With AVRO-872, multiple inter-dependent schema files can be parsed. {{SpecificCompilerTool}} should be changed to allow the same when producing java from multiple schemas or protocols. Proposed syntax change is to allow multiple files or directories to be passed (space-delimited) before the last token, which is the output path.
- To compile multiple files where {{b}} depends on {{a}}:
{noformat}
$ java -cp avro-tools-x.x.x.jar org.apache.avro.tool.Main \
compile schema input/a.avsc input/b.avsc output
{noformat}
- To compile files from multiple directories:
{noformat}
$ java -cp avro-tools-x.x.x.jar org.apache.avro.tool.Main \
compile schema input1 input2 output
{noformat}
- To compile {{b}} which depends on {{a}} and all the other content in {{input}} that might depend on {{a}} or {{b}}:
{noformat}
$ java -cp avro-tools-x.x.x.jar org.apache.avro.tool.Main \
compile schema input/a.avsc input/b.avsc input output
{noformat}
was:
With AVRO-874, multiple inter-dependent schema files can be parsed. {{SpecificCompilerTool}} should be changed to allow the same when producing java from multiple schemas or protocols. Proposed syntax change is to allow multiple files or directories to be passed (space-delimited) before the last token, which is the output path.
- To compile multiple files where {{b}} depends on {{a}}:
{noformat}
$ java -cp avro-tools-x.x.x.jar org.apache.avro.tool.Main \
compile schema input/a.avsc input/b.avsc output
{noformat}
- To compile files from multiple directories:
{noformat}
$ java -cp avro-tools-x.x.x.jar org.apache.avro.tool.Main \
compile schema input1 input2 output
{noformat}
- To compile {{b}} which depends on {{a}} and all the other content in {{input}} that might depend on {{a}} or {{b}}:
{noformat}
$ java -cp avro-tools-x.x.x.jar org.apache.avro.tool.Main \
compile schema input/a.avsc input/b.avsc input output
{noformat}
> Support compiling multiple input schemas and protocols
> ------------------------------------------------------
>
> Key: AVRO-877
> URL: https://issues.apache.org/jira/browse/AVRO-877
> Project: Avro
> Issue Type: Improvement
> Reporter: Bill Graham
> Assignee: Bill Graham
> Attachments: AVRO-877_1.patch
>
>
> With AVRO-872, multiple inter-dependent schema files can be parsed. {{SpecificCompilerTool}} should be changed to allow the same when producing java from multiple schemas or protocols. Proposed syntax change is to allow multiple files or directories to be passed (space-delimited) before the last token, which is the output path.
> - To compile multiple files where {{b}} depends on {{a}}:
> {noformat}
> $ java -cp avro-tools-x.x.x.jar org.apache.avro.tool.Main \
> compile schema input/a.avsc input/b.avsc output
> {noformat}
> - To compile files from multiple directories:
> {noformat}
> $ java -cp avro-tools-x.x.x.jar org.apache.avro.tool.Main \
> compile schema input1 input2 output
> {noformat}
> - To compile {{b}} which depends on {{a}} and all the other content in {{input}} that might depend on {{a}} or {{b}}:
> {noformat}
> $ java -cp avro-tools-x.x.x.jar org.apache.avro.tool.Main \
> compile schema input/a.avsc input/b.avsc input output
> {noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-877) Support compiling multiple input
schemas and protocols
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated AVRO-877:
------------------------------
Resolution: Fixed
Fix Version/s: (was: 1.6.0)
1.5.3
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I just committed this. Thanks, Bill!
> Support compiling multiple input schemas and protocols
> ------------------------------------------------------
>
> Key: AVRO-877
> URL: https://issues.apache.org/jira/browse/AVRO-877
> Project: Avro
> Issue Type: Improvement
> Reporter: Bill Graham
> Assignee: Bill Graham
> Fix For: 1.5.3
>
> Attachments: AVRO-877_1.patch, AVRO-877_2.patch
>
>
> With AVRO-872, multiple inter-dependent schema files can be parsed. {{SpecificCompilerTool}} should be changed to allow the same when producing java from multiple schemas or protocols. Proposed syntax change is to allow multiple files or directories to be passed (space-delimited) before the last token, which is the output path.
> - To compile multiple files where {{b}} depends on {{a}}:
> {noformat}
> $ java -cp avro-tools-x.x.x.jar org.apache.avro.tool.Main \
> compile schema input/a.avsc input/b.avsc output
> {noformat}
> - To compile files from multiple directories:
> {noformat}
> $ java -cp avro-tools-x.x.x.jar org.apache.avro.tool.Main \
> compile schema input1 input2 output
> {noformat}
> - To compile {{b}} which depends on {{a}} and all the other content in {{input}} that might depend on {{a}} or {{b}}:
> {noformat}
> $ java -cp avro-tools-x.x.x.jar org.apache.avro.tool.Main \
> compile schema input/a.avsc input/b.avsc input output
> {noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-877) Support compiling multiple input
schemas and protocols
Posted by "Bill Graham (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bill Graham updated AVRO-877:
-----------------------------
Attachment: AVRO-877_2.patch
Attached is a patch with unit tests. I'm just verifying a textual match between expected and generated Java src. Let me know if I should change the location of the input, expected output or generated output, amongst other things. Currently they are as follows, respectively:
{noformat}
lang/java/tools/
src/test/compiler/input
src/test/compiler/output
target/test/compiler/output
{noformat}
> Support compiling multiple input schemas and protocols
> ------------------------------------------------------
>
> Key: AVRO-877
> URL: https://issues.apache.org/jira/browse/AVRO-877
> Project: Avro
> Issue Type: Improvement
> Reporter: Bill Graham
> Assignee: Bill Graham
> Fix For: 1.6.0
>
> Attachments: AVRO-877_1.patch, AVRO-877_2.patch
>
>
> With AVRO-872, multiple inter-dependent schema files can be parsed. {{SpecificCompilerTool}} should be changed to allow the same when producing java from multiple schemas or protocols. Proposed syntax change is to allow multiple files or directories to be passed (space-delimited) before the last token, which is the output path.
> - To compile multiple files where {{b}} depends on {{a}}:
> {noformat}
> $ java -cp avro-tools-x.x.x.jar org.apache.avro.tool.Main \
> compile schema input/a.avsc input/b.avsc output
> {noformat}
> - To compile files from multiple directories:
> {noformat}
> $ java -cp avro-tools-x.x.x.jar org.apache.avro.tool.Main \
> compile schema input1 input2 output
> {noformat}
> - To compile {{b}} which depends on {{a}} and all the other content in {{input}} that might depend on {{a}} or {{b}}:
> {noformat}
> $ java -cp avro-tools-x.x.x.jar org.apache.avro.tool.Main \
> compile schema input/a.avsc input/b.avsc input output
> {noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-877) Support compiling multiple input
schemas and protocols
Posted by "Bill Graham (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bill Graham updated AVRO-877:
-----------------------------
Description:
With AVRO-874, multiple inter-dependent schema files can be parsed. {{SpecificCompilerTool}} should be changed to allow the same when producing java from multiple schemas or protocols. Proposed syntax change is to allow multiple files or directories to be passed (space-delimited) before the last token, which is the output path.
- To compile multiple files where {{b}} depends on {{a}}:
{noformat}
$ java -cp avro-tools-x.x.x.jar org.apache.avro.tool.Main \
compile schema input/a.avsc input/b.avsc output
{noformat}
- To compile files from multiple directories:
{noformat}
$ java -cp avro-tools-x.x.x.jar org.apache.avro.tool.Main \
compile schema input1 input2 output
{noformat}
- To compile {{b}} which depends on {{a}} and all the other content in {{input}} that might depend on {{a}} or {{b}}:
{noformat}
$ java -cp avro-tools-x.x.x.jar org.apache.avro.tool.Main \
compile schema input/a.avsc input/b.avsc input output
{noformat}
was:
With Avro-874, multiple inter-dependent schema files can be parsed. {{SpecificCompilerTool}} should be changed to allow the same when producing java from multiple schemas or protocols. Proposed syntax change is to allow multiple files or directories to be passed (space-delimited) before the last token, which is the output path.
- To compile multiple files where {{b}} depends on {{a}}:
{noformat}
$ java -cp avro-tools-x.x.x.jar org.apache.avro.tool.Main \
compile schema input/a.avsc input/b.avsc output
{noformat}
- To compile files from multiple directories:
{noformat}
$ java -cp avro-tools-x.x.x.jar org.apache.avro.tool.Main \
compile schema input1 input2 output
{noformat}
- To compile {{b}} which depends on {{a}} and all the other content in {{input}} that might depend on {{a}} or {{b}}:
{noformat}
$ java -cp avro-tools-x.x.x.jar org.apache.avro.tool.Main \
compile schema input/a.avsc input/b.avsc input output
{noformat}
> Support compiling multiple input schemas and protocols
> ------------------------------------------------------
>
> Key: AVRO-877
> URL: https://issues.apache.org/jira/browse/AVRO-877
> Project: Avro
> Issue Type: Improvement
> Reporter: Bill Graham
> Assignee: Bill Graham
> Attachments: AVRO-877_1.patch
>
>
> With AVRO-874, multiple inter-dependent schema files can be parsed. {{SpecificCompilerTool}} should be changed to allow the same when producing java from multiple schemas or protocols. Proposed syntax change is to allow multiple files or directories to be passed (space-delimited) before the last token, which is the output path.
> - To compile multiple files where {{b}} depends on {{a}}:
> {noformat}
> $ java -cp avro-tools-x.x.x.jar org.apache.avro.tool.Main \
> compile schema input/a.avsc input/b.avsc output
> {noformat}
> - To compile files from multiple directories:
> {noformat}
> $ java -cp avro-tools-x.x.x.jar org.apache.avro.tool.Main \
> compile schema input1 input2 output
> {noformat}
> - To compile {{b}} which depends on {{a}} and all the other content in {{input}} that might depend on {{a}} or {{b}}:
> {noformat}
> $ java -cp avro-tools-x.x.x.jar org.apache.avro.tool.Main \
> compile schema input/a.avsc input/b.avsc input output
> {noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-877) Support compiling multiple input
schemas and protocols
Posted by "Bill Graham (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bill Graham updated AVRO-877:
-----------------------------
Attachment: AVRO-877_1.patch
Attaching first patch. Note that the parser of Protocol didn't get updated as part of AVRO-874 so it still doesn't handle interdependencies. That should probably be a separate JIRA.
> Support compiling multiple input schemas and protocols
> ------------------------------------------------------
>
> Key: AVRO-877
> URL: https://issues.apache.org/jira/browse/AVRO-877
> Project: Avro
> Issue Type: Improvement
> Reporter: Bill Graham
> Assignee: Bill Graham
> Attachments: AVRO-877_1.patch
>
>
> With Avro-874, multiple inter-dependent schema files can be parsed. {{SpecificCompilerTool}} should be changed to allow the same when producing java from multiple schemas or protocols. Proposed syntax change is to allow multiple files or directories to be passed (space-delimited) before the last token, which is the output path.
> - To compile multiple files where {{b}} depends on {{a}}:
> {noformat}
> $ java -cp avro-tools-x.x.x.jar org.apache.avro.tool.Main \
> compile schema input/a.avsc input/b.avsc output
> {noformat}
> - To compile files from multiple directories:
> {noformat}
> $ java -cp avro-tools-x.x.x.jar org.apache.avro.tool.Main \
> compile schema input1 input2 output
> {noformat}
> - To compile {{b}} which depends on {{a}} and all the other content in {{input}} that might depend on {{a}} or {{b}}:
> {noformat}
> $ java -cp avro-tools-x.x.x.jar org.apache.avro.tool.Main \
> compile schema input/a.avsc input/b.avsc input output
> {noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-877) Support compiling multiple input
schemas and protocols
Posted by "Bill Graham (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bill Graham updated AVRO-877:
-----------------------------
Release Note: Schema compiler now accepts multiple input files and/or paths.
Status: Patch Available (was: Open)
> Support compiling multiple input schemas and protocols
> ------------------------------------------------------
>
> Key: AVRO-877
> URL: https://issues.apache.org/jira/browse/AVRO-877
> Project: Avro
> Issue Type: Improvement
> Reporter: Bill Graham
> Assignee: Bill Graham
> Attachments: AVRO-877_1.patch
>
>
> With AVRO-874, multiple inter-dependent schema files can be parsed. {{SpecificCompilerTool}} should be changed to allow the same when producing java from multiple schemas or protocols. Proposed syntax change is to allow multiple files or directories to be passed (space-delimited) before the last token, which is the output path.
> - To compile multiple files where {{b}} depends on {{a}}:
> {noformat}
> $ java -cp avro-tools-x.x.x.jar org.apache.avro.tool.Main \
> compile schema input/a.avsc input/b.avsc output
> {noformat}
> - To compile files from multiple directories:
> {noformat}
> $ java -cp avro-tools-x.x.x.jar org.apache.avro.tool.Main \
> compile schema input1 input2 output
> {noformat}
> - To compile {{b}} which depends on {{a}} and all the other content in {{input}} that might depend on {{a}} or {{b}}:
> {noformat}
> $ java -cp avro-tools-x.x.x.jar org.apache.avro.tool.Main \
> compile schema input/a.avsc input/b.avsc input output
> {noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-877) Support compiling multiple input
schemas and protocols
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13089712#comment-13089712 ]
Doug Cutting commented on AVRO-877:
-----------------------------------
We might just run the compiler over some inputs and check that the expected output files are created. So a test case would be a list of input files (in the src/ tree) and the list of expected files that will be created (in the target/ tree).
For extra credit we could compile these output files to check them with javac.
http://download.oracle.com/javase/6/docs/api/javax/tools/JavaCompiler.html
Another approach is like TestIdl, where the compiler output is also in the test tree, and we check the generated output against that.
> Support compiling multiple input schemas and protocols
> ------------------------------------------------------
>
> Key: AVRO-877
> URL: https://issues.apache.org/jira/browse/AVRO-877
> Project: Avro
> Issue Type: Improvement
> Reporter: Bill Graham
> Assignee: Bill Graham
> Attachments: AVRO-877_1.patch
>
>
> With AVRO-872, multiple inter-dependent schema files can be parsed. {{SpecificCompilerTool}} should be changed to allow the same when producing java from multiple schemas or protocols. Proposed syntax change is to allow multiple files or directories to be passed (space-delimited) before the last token, which is the output path.
> - To compile multiple files where {{b}} depends on {{a}}:
> {noformat}
> $ java -cp avro-tools-x.x.x.jar org.apache.avro.tool.Main \
> compile schema input/a.avsc input/b.avsc output
> {noformat}
> - To compile files from multiple directories:
> {noformat}
> $ java -cp avro-tools-x.x.x.jar org.apache.avro.tool.Main \
> compile schema input1 input2 output
> {noformat}
> - To compile {{b}} which depends on {{a}} and all the other content in {{input}} that might depend on {{a}} or {{b}}:
> {noformat}
> $ java -cp avro-tools-x.x.x.jar org.apache.avro.tool.Main \
> compile schema input/a.avsc input/b.avsc input output
> {noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-877) Support compiling multiple input
schemas and protocols
Posted by "Bill Graham (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087862#comment-13087862 ]
Bill Graham commented on AVRO-877:
----------------------------------
Yeah, the unit tests in the compiler module are pretty sparse. At what level do you think we should test the new functionality? We could do a true unit test to verify that parsing and input file generation works properly for the change in how we now handle input paths/files. Or we could do more of a functional test that the compiler tool generates valid code for some inputs. Asserting the validity of generated code from a running VM sounds tricky though...
> Support compiling multiple input schemas and protocols
> ------------------------------------------------------
>
> Key: AVRO-877
> URL: https://issues.apache.org/jira/browse/AVRO-877
> Project: Avro
> Issue Type: Improvement
> Reporter: Bill Graham
> Assignee: Bill Graham
> Attachments: AVRO-877_1.patch
>
>
> With AVRO-872, multiple inter-dependent schema files can be parsed. {{SpecificCompilerTool}} should be changed to allow the same when producing java from multiple schemas or protocols. Proposed syntax change is to allow multiple files or directories to be passed (space-delimited) before the last token, which is the output path.
> - To compile multiple files where {{b}} depends on {{a}}:
> {noformat}
> $ java -cp avro-tools-x.x.x.jar org.apache.avro.tool.Main \
> compile schema input/a.avsc input/b.avsc output
> {noformat}
> - To compile files from multiple directories:
> {noformat}
> $ java -cp avro-tools-x.x.x.jar org.apache.avro.tool.Main \
> compile schema input1 input2 output
> {noformat}
> - To compile {{b}} which depends on {{a}} and all the other content in {{input}} that might depend on {{a}} or {{b}}:
> {noformat}
> $ java -cp avro-tools-x.x.x.jar org.apache.avro.tool.Main \
> compile schema input/a.avsc input/b.avsc input output
> {noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira