You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "Sharmarke Aden (JIRA)" <ji...@apache.org> on 2012/11/03 01:20:13 UTC
[jira] [Created] (AVRO-1188) External Schema Imports via AVSC
Schema
Sharmarke Aden created AVRO-1188:
------------------------------------
Summary: External Schema Imports via AVSC Schema
Key: AVRO-1188
URL: https://issues.apache.org/jira/browse/AVRO-1188
Project: Avro
Issue Type: New Feature
Components: java
Reporter: Sharmarke Aden
There is no way for ".avsc" schema files to import types (i.e records, enums, etc) in external schema files. There's tremendous benefit in being able to do this as it would all for the sharing of common types between multiple schema files. Here's a use case that illustrates the typical usecase of this feature request.
Suppose we have an enum called "Privacy" that we would like to share between multiple schemas:
{code}
//privacy.avsc
{
"type": "enum",
"name": "Privacy",
"symbols" : ["Public", "Private"]
}
{code}
Now, if this feature was implemented one could import the above type into other schema files by doing something like this:
{code}
//the post.avsc
{
"type": "record",
"name": "Post",
"imports": ["privacy.avsc"] //you can import one or more schemas
"fields": [{
"name": "privacy",
"type": [ "null", "Privacy"], //use imported Privacy type
"default": "Private"
}
]
}
{code}
Here's another schema file that also has a similar privacy concern:
{code}
//the event.avsc is another schema that also imports the privacy type
{
"type": "record",
"name": "Event",
"imports": ["privacy.avsc"] //it also imports the privacy schemas
"fields": [{
"name": "privacy",
"type": [ "null", "Privacy"], //use imported Privacy type
"default": "Public"
}
]
}
{code}
IDL files are able to import external schemas and protocols and it would be very beneficial if schema files could import other schema files.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1188) External Schema Imports via AVSC
Schema
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13490966#comment-13490966 ]
Doug Cutting commented on AVRO-1188:
------------------------------------
> What do you think of the idea of having an explicit imports directory that's passed via command-line/maven which is processed/compiled before any other schemas?
I think that would be a good approach.
> External Schema Imports via AVSC Schema
> ---------------------------------------
>
> Key: AVRO-1188
> URL: https://issues.apache.org/jira/browse/AVRO-1188
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Sharmarke Aden
>
> There is no way for ".avsc" schema files to import types (i.e records, enums, etc) in external schema files. There's tremendous benefit in being able to do this as it would all for the sharing of common types between multiple schema files. Here's a use case that illustrates the typical usecase of this feature request.
> Suppose we have an enum called "Privacy" that we would like to share between multiple schemas:
> {code}
> //privacy.avsc
> {
> "type": "enum",
> "name": "Privacy",
> "symbols" : ["Public", "Private"]
> }
> {code}
> Now, if this feature was implemented one could import the above type into other schema files by doing something like this:
> {code}
> //the post.avsc
> {
> "type": "record",
> "name": "Post",
> "imports": ["privacy.avsc"] //you can import one or more schemas
> "fields": [{
> "name": "privacy",
> "type": [ "null", "Privacy"], //use imported Privacy type
> "default": "Private"
> }
> ]
> }
> {code}
> Here's another schema file that also has a similar privacy concern:
> {code}
> //the event.avsc is another schema that also imports the privacy type
> {
> "type": "record",
> "name": "Event",
> "imports": ["privacy.avsc"] //it also imports the privacy schemas
> "fields": [{
> "name": "privacy",
> "type": [ "null", "Privacy"], //use imported Privacy type
> "default": "Public"
> }
> ]
> }
> {code}
> IDL files are able to import external schemas and protocols and it would be very beneficial if schema files could import other schema files.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1188) External Schema Imports via AVSC
Schema
Posted by "Sharmarke Aden (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13498250#comment-13498250 ]
Sharmarke Aden commented on AVRO-1188:
--------------------------------------
Yes, I will take a look at it.
On a side note, is there any reason why there aren't any tests for the plugin Mojos?
> External Schema Imports via AVSC Schema
> ---------------------------------------
>
> Key: AVRO-1188
> URL: https://issues.apache.org/jira/browse/AVRO-1188
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Sharmarke Aden
> Assignee: Sharmarke Aden
> Fix For: 1.7.3
>
> Attachments: Avro-1188.tar.gz, Avro-1188.tar.gz, vcs-diff1160361655737792386.patch, vcs-diff2916139350460140957.patch, vcs-diff4277815358664835838.patch, vcs-diff6739872835137179667.patch
>
>
> There is no way for ".avsc" schema files to import types (i.e records, enums, etc) in external schema files. There's tremendous benefit in being able to do this as it would allow the sharing of common types between multiple schema files. Here's a use case that illustrates the typical usecase of this feature request.
> Suppose we have an enum called "Privacy" that we would like to share between multiple schemas:
> {code}
> //privacy.avsc
> {
> "type": "enum",
> "name": "Privacy",
> "symbols" : ["Public", "Private"]
> }
> {code}
> Now, if this feature was implemented one could import the above type into other schema files by doing something like this:
> {code}
> //the post.avsc
> {
> "type": "record",
> "name": "Post",
> "imports": ["privacy.avsc"] //you can import one or more schemas
> "fields": [{
> "name": "privacy",
> "type": [ "null", "Privacy"], //use imported Privacy type
> "default": "Private"
> }
> ]
> }
> {code}
> Here's another schema file that also has a similar privacy concern:
> {code}
> //the event.avsc is another schema that also imports the privacy type
> {
> "type": "record",
> "name": "Event",
> "imports": ["privacy.avsc"] //it also imports the privacy schemas
> "fields": [{
> "name": "privacy",
> "type": [ "null", "Privacy"], //use imported Privacy type
> "default": "Public"
> }
> ]
> }
> {code}
> IDL files are able to import external schemas and protocols and it would be very beneficial if schema files could import other schema files.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1188) External Schema Imports via AVSC
Schema
Posted by "Sharmarke Aden (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495378#comment-13495378 ]
Sharmarke Aden commented on AVRO-1188:
--------------------------------------
Updated the patch based on Tom's valuable feedback. Please note that I also changed the plugin coUpdated the patch based on Tom's feedback. Please note that I also changed the plugin "importedFiles" configuration to just "imports"
> External Schema Imports via AVSC Schema
> ---------------------------------------
>
> Key: AVRO-1188
> URL: https://issues.apache.org/jira/browse/AVRO-1188
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Sharmarke Aden
> Assignee: Sharmarke Aden
> Attachments: Avro-1188.tar.gz, vcs-diff1160361655737792386.patch, vcs-diff2916139350460140957.patch, vcs-diff4277815358664835838.patch, vcs-diff6739872835137179667.patch
>
>
> There is no way for ".avsc" schema files to import types (i.e records, enums, etc) in external schema files. There's tremendous benefit in being able to do this as it would allow the sharing of common types between multiple schema files. Here's a use case that illustrates the typical usecase of this feature request.
> Suppose we have an enum called "Privacy" that we would like to share between multiple schemas:
> {code}
> //privacy.avsc
> {
> "type": "enum",
> "name": "Privacy",
> "symbols" : ["Public", "Private"]
> }
> {code}
> Now, if this feature was implemented one could import the above type into other schema files by doing something like this:
> {code}
> //the post.avsc
> {
> "type": "record",
> "name": "Post",
> "imports": ["privacy.avsc"] //you can import one or more schemas
> "fields": [{
> "name": "privacy",
> "type": [ "null", "Privacy"], //use imported Privacy type
> "default": "Private"
> }
> ]
> }
> {code}
> Here's another schema file that also has a similar privacy concern:
> {code}
> //the event.avsc is another schema that also imports the privacy type
> {
> "type": "record",
> "name": "Event",
> "imports": ["privacy.avsc"] //it also imports the privacy schemas
> "fields": [{
> "name": "privacy",
> "type": [ "null", "Privacy"], //use imported Privacy type
> "default": "Public"
> }
> ]
> }
> {code}
> IDL files are able to import external schemas and protocols and it would be very beneficial if schema files could import other schema files.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-1188) External Schema Imports via AVSC
Schema
Posted by "Sharmarke Aden (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sharmarke Aden updated AVRO-1188:
---------------------------------
Attachment: vcs-diff4277815358664835838.patch
> External Schema Imports via AVSC Schema
> ---------------------------------------
>
> Key: AVRO-1188
> URL: https://issues.apache.org/jira/browse/AVRO-1188
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Sharmarke Aden
> Attachments: vcs-diff4277815358664835838.patch
>
>
> There is no way for ".avsc" schema files to import types (i.e records, enums, etc) in external schema files. There's tremendous benefit in being able to do this as it would allow the sharing of common types between multiple schema files. Here's a use case that illustrates the typical usecase of this feature request.
> Suppose we have an enum called "Privacy" that we would like to share between multiple schemas:
> {code}
> //privacy.avsc
> {
> "type": "enum",
> "name": "Privacy",
> "symbols" : ["Public", "Private"]
> }
> {code}
> Now, if this feature was implemented one could import the above type into other schema files by doing something like this:
> {code}
> //the post.avsc
> {
> "type": "record",
> "name": "Post",
> "imports": ["privacy.avsc"] //you can import one or more schemas
> "fields": [{
> "name": "privacy",
> "type": [ "null", "Privacy"], //use imported Privacy type
> "default": "Private"
> }
> ]
> }
> {code}
> Here's another schema file that also has a similar privacy concern:
> {code}
> //the event.avsc is another schema that also imports the privacy type
> {
> "type": "record",
> "name": "Event",
> "imports": ["privacy.avsc"] //it also imports the privacy schemas
> "fields": [{
> "name": "privacy",
> "type": [ "null", "Privacy"], //use imported Privacy type
> "default": "Public"
> }
> ]
> }
> {code}
> IDL files are able to import external schemas and protocols and it would be very beneficial if schema files could import other schema files.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (AVRO-1188) External Schema Imports via AVSC
Schema
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White resolved AVRO-1188.
-----------------------------
Resolution: Fixed
Fix Version/s: 1.7.3
Hadoop Flags: Reviewed
+1 I just committed this. Thanks, Sharmarke!
> External Schema Imports via AVSC Schema
> ---------------------------------------
>
> Key: AVRO-1188
> URL: https://issues.apache.org/jira/browse/AVRO-1188
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Sharmarke Aden
> Assignee: Sharmarke Aden
> Fix For: 1.7.3
>
> Attachments: Avro-1188.tar.gz, Avro-1188.tar.gz, vcs-diff1160361655737792386.patch, vcs-diff2916139350460140957.patch, vcs-diff4277815358664835838.patch, vcs-diff6739872835137179667.patch
>
>
> There is no way for ".avsc" schema files to import types (i.e records, enums, etc) in external schema files. There's tremendous benefit in being able to do this as it would allow the sharing of common types between multiple schema files. Here's a use case that illustrates the typical usecase of this feature request.
> Suppose we have an enum called "Privacy" that we would like to share between multiple schemas:
> {code}
> //privacy.avsc
> {
> "type": "enum",
> "name": "Privacy",
> "symbols" : ["Public", "Private"]
> }
> {code}
> Now, if this feature was implemented one could import the above type into other schema files by doing something like this:
> {code}
> //the post.avsc
> {
> "type": "record",
> "name": "Post",
> "imports": ["privacy.avsc"] //you can import one or more schemas
> "fields": [{
> "name": "privacy",
> "type": [ "null", "Privacy"], //use imported Privacy type
> "default": "Private"
> }
> ]
> }
> {code}
> Here's another schema file that also has a similar privacy concern:
> {code}
> //the event.avsc is another schema that also imports the privacy type
> {
> "type": "record",
> "name": "Event",
> "imports": ["privacy.avsc"] //it also imports the privacy schemas
> "fields": [{
> "name": "privacy",
> "type": [ "null", "Privacy"], //use imported Privacy type
> "default": "Public"
> }
> ]
> }
> {code}
> IDL files are able to import external schemas and protocols and it would be very beneficial if schema files could import other schema files.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1188) External Schema Imports via AVSC
Schema
Posted by "Sharmarke Aden (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495019#comment-13495019 ]
Sharmarke Aden commented on AVRO-1188:
--------------------------------------
This patch also displays the same symptoms as AVRO-983 patch. Having a single Schema.Parser causes the IPC module to fail with "Can't redefine: org.apache.avro.ipc.MD5"
> External Schema Imports via AVSC Schema
> ---------------------------------------
>
> Key: AVRO-1188
> URL: https://issues.apache.org/jira/browse/AVRO-1188
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Sharmarke Aden
> Attachments: Avro-1188.tar.gz, vcs-diff2916139350460140957.patch, vcs-diff4277815358664835838.patch
>
>
> There is no way for ".avsc" schema files to import types (i.e records, enums, etc) in external schema files. There's tremendous benefit in being able to do this as it would allow the sharing of common types between multiple schema files. Here's a use case that illustrates the typical usecase of this feature request.
> Suppose we have an enum called "Privacy" that we would like to share between multiple schemas:
> {code}
> //privacy.avsc
> {
> "type": "enum",
> "name": "Privacy",
> "symbols" : ["Public", "Private"]
> }
> {code}
> Now, if this feature was implemented one could import the above type into other schema files by doing something like this:
> {code}
> //the post.avsc
> {
> "type": "record",
> "name": "Post",
> "imports": ["privacy.avsc"] //you can import one or more schemas
> "fields": [{
> "name": "privacy",
> "type": [ "null", "Privacy"], //use imported Privacy type
> "default": "Private"
> }
> ]
> }
> {code}
> Here's another schema file that also has a similar privacy concern:
> {code}
> //the event.avsc is another schema that also imports the privacy type
> {
> "type": "record",
> "name": "Event",
> "imports": ["privacy.avsc"] //it also imports the privacy schemas
> "fields": [{
> "name": "privacy",
> "type": [ "null", "Privacy"], //use imported Privacy type
> "default": "Public"
> }
> ]
> }
> {code}
> IDL files are able to import external schemas and protocols and it would be very beneficial if schema files could import other schema files.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1188) External Schema Imports via AVSC
Schema
Posted by "Sharmarke Aden (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13497481#comment-13497481 ]
Sharmarke Aden commented on AVRO-1188:
--------------------------------------
I would very much like to add unit tests but I'm not sure where they should be added.
> External Schema Imports via AVSC Schema
> ---------------------------------------
>
> Key: AVRO-1188
> URL: https://issues.apache.org/jira/browse/AVRO-1188
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Sharmarke Aden
> Assignee: Sharmarke Aden
> Fix For: 1.7.3
>
> Attachments: Avro-1188.tar.gz, Avro-1188.tar.gz, vcs-diff1160361655737792386.patch, vcs-diff2916139350460140957.patch, vcs-diff4277815358664835838.patch, vcs-diff6739872835137179667.patch
>
>
> There is no way for ".avsc" schema files to import types (i.e records, enums, etc) in external schema files. There's tremendous benefit in being able to do this as it would allow the sharing of common types between multiple schema files. Here's a use case that illustrates the typical usecase of this feature request.
> Suppose we have an enum called "Privacy" that we would like to share between multiple schemas:
> {code}
> //privacy.avsc
> {
> "type": "enum",
> "name": "Privacy",
> "symbols" : ["Public", "Private"]
> }
> {code}
> Now, if this feature was implemented one could import the above type into other schema files by doing something like this:
> {code}
> //the post.avsc
> {
> "type": "record",
> "name": "Post",
> "imports": ["privacy.avsc"] //you can import one or more schemas
> "fields": [{
> "name": "privacy",
> "type": [ "null", "Privacy"], //use imported Privacy type
> "default": "Private"
> }
> ]
> }
> {code}
> Here's another schema file that also has a similar privacy concern:
> {code}
> //the event.avsc is another schema that also imports the privacy type
> {
> "type": "record",
> "name": "Event",
> "imports": ["privacy.avsc"] //it also imports the privacy schemas
> "fields": [{
> "name": "privacy",
> "type": [ "null", "Privacy"], //use imported Privacy type
> "default": "Public"
> }
> ]
> }
> {code}
> IDL files are able to import external schemas and protocols and it would be very beneficial if schema files could import other schema files.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1188) External Schema Imports via AVSC
Schema
Posted by "Sharmarke Aden (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13490916#comment-13490916 ]
Sharmarke Aden commented on AVRO-1188:
--------------------------------------
Looking at AVRO-983 it seems that this issue is somewhat of a duplicate issue. I searched for a similar issue but didn't have luck finding it. Yes, utilization of the order technique will work for us though I wonder if there should be better way of schema imports based on previously loaded/compiled schemas without having to rely on the order of files/directorys. What do you think of the idea of having an explicit imports directory that's passed via command-line/maven which is processed/compiled before any other schemas?
In maven one would define a configuration property called <importsDirectory/> that points to a directory containing schemas that should be processed first. Perhaps this directory could default to "src/main/avro/imports" which is treated as a special reserved directory by the maven plugin. The maven plugin declaration would look like this:
{code}
<plugin>
<groupId>org.apache.avro</groupId>
<artifactId>avro-maven-plugin</artifactId>
<executions>
<execution>
<goals>
<goal>schema</goal>
</goals>
</execution>
</executions>
<configuration>
<importsDirectory>${basedir}/src/main/avro/imports</importsDirectory>
</configuration>
</plugin>
{code}
I am not familiar with the command-line tool but I would imagine a command-line option called "-imports" can be added that allows a user to specify an import directory that gets processed before other schemas.
{code}
java -jar avro-tools-XXX.jar -imports "/schema/import/directory" ...
{code}
> External Schema Imports via AVSC Schema
> ---------------------------------------
>
> Key: AVRO-1188
> URL: https://issues.apache.org/jira/browse/AVRO-1188
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Sharmarke Aden
>
> There is no way for ".avsc" schema files to import types (i.e records, enums, etc) in external schema files. There's tremendous benefit in being able to do this as it would all for the sharing of common types between multiple schema files. Here's a use case that illustrates the typical usecase of this feature request.
> Suppose we have an enum called "Privacy" that we would like to share between multiple schemas:
> {code}
> //privacy.avsc
> {
> "type": "enum",
> "name": "Privacy",
> "symbols" : ["Public", "Private"]
> }
> {code}
> Now, if this feature was implemented one could import the above type into other schema files by doing something like this:
> {code}
> //the post.avsc
> {
> "type": "record",
> "name": "Post",
> "imports": ["privacy.avsc"] //you can import one or more schemas
> "fields": [{
> "name": "privacy",
> "type": [ "null", "Privacy"], //use imported Privacy type
> "default": "Private"
> }
> ]
> }
> {code}
> Here's another schema file that also has a similar privacy concern:
> {code}
> //the event.avsc is another schema that also imports the privacy type
> {
> "type": "record",
> "name": "Event",
> "imports": ["privacy.avsc"] //it also imports the privacy schemas
> "fields": [{
> "name": "privacy",
> "type": [ "null", "Privacy"], //use imported Privacy type
> "default": "Public"
> }
> ]
> }
> {code}
> IDL files are able to import external schemas and protocols and it would be very beneficial if schema files could import other schema files.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-1188) External Schema Imports via AVSC
Schema
Posted by "Sharmarke Aden (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sharmarke Aden updated AVRO-1188:
---------------------------------
Attachment: Avro-1188.tar.gz
Here is a simple project that demonstrates the import feature.
> External Schema Imports via AVSC Schema
> ---------------------------------------
>
> Key: AVRO-1188
> URL: https://issues.apache.org/jira/browse/AVRO-1188
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Sharmarke Aden
> Attachments: Avro-1188.tar.gz, vcs-diff4277815358664835838.patch
>
>
> There is no way for ".avsc" schema files to import types (i.e records, enums, etc) in external schema files. There's tremendous benefit in being able to do this as it would allow the sharing of common types between multiple schema files. Here's a use case that illustrates the typical usecase of this feature request.
> Suppose we have an enum called "Privacy" that we would like to share between multiple schemas:
> {code}
> //privacy.avsc
> {
> "type": "enum",
> "name": "Privacy",
> "symbols" : ["Public", "Private"]
> }
> {code}
> Now, if this feature was implemented one could import the above type into other schema files by doing something like this:
> {code}
> //the post.avsc
> {
> "type": "record",
> "name": "Post",
> "imports": ["privacy.avsc"] //you can import one or more schemas
> "fields": [{
> "name": "privacy",
> "type": [ "null", "Privacy"], //use imported Privacy type
> "default": "Private"
> }
> ]
> }
> {code}
> Here's another schema file that also has a similar privacy concern:
> {code}
> //the event.avsc is another schema that also imports the privacy type
> {
> "type": "record",
> "name": "Event",
> "imports": ["privacy.avsc"] //it also imports the privacy schemas
> "fields": [{
> "name": "privacy",
> "type": [ "null", "Privacy"], //use imported Privacy type
> "default": "Public"
> }
> ]
> }
> {code}
> IDL files are able to import external schemas and protocols and it would be very beneficial if schema files could import other schema files.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-1188) External Schema Imports via AVSC
Schema
Posted by "Sharmarke Aden (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sharmarke Aden updated AVRO-1188:
---------------------------------
Attachment: vcs-diff2916139350460140957.patch
> External Schema Imports via AVSC Schema
> ---------------------------------------
>
> Key: AVRO-1188
> URL: https://issues.apache.org/jira/browse/AVRO-1188
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Sharmarke Aden
> Attachments: Avro-1188.tar.gz, vcs-diff2916139350460140957.patch, vcs-diff4277815358664835838.patch
>
>
> There is no way for ".avsc" schema files to import types (i.e records, enums, etc) in external schema files. There's tremendous benefit in being able to do this as it would allow the sharing of common types between multiple schema files. Here's a use case that illustrates the typical usecase of this feature request.
> Suppose we have an enum called "Privacy" that we would like to share between multiple schemas:
> {code}
> //privacy.avsc
> {
> "type": "enum",
> "name": "Privacy",
> "symbols" : ["Public", "Private"]
> }
> {code}
> Now, if this feature was implemented one could import the above type into other schema files by doing something like this:
> {code}
> //the post.avsc
> {
> "type": "record",
> "name": "Post",
> "imports": ["privacy.avsc"] //you can import one or more schemas
> "fields": [{
> "name": "privacy",
> "type": [ "null", "Privacy"], //use imported Privacy type
> "default": "Private"
> }
> ]
> }
> {code}
> Here's another schema file that also has a similar privacy concern:
> {code}
> //the event.avsc is another schema that also imports the privacy type
> {
> "type": "record",
> "name": "Event",
> "imports": ["privacy.avsc"] //it also imports the privacy schemas
> "fields": [{
> "name": "privacy",
> "type": [ "null", "Privacy"], //use imported Privacy type
> "default": "Public"
> }
> ]
> }
> {code}
> IDL files are able to import external schemas and protocols and it would be very beneficial if schema files could import other schema files.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1188) External Schema Imports via AVSC
Schema
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13498397#comment-13498397 ]
Doug Cutting commented on AVRO-1188:
------------------------------------
> is there any reason why there aren't any tests for the plugin Mojos?
No good reason. They were implemented as a part of the conversion from Ant to Maven and no tests were added then.
> External Schema Imports via AVSC Schema
> ---------------------------------------
>
> Key: AVRO-1188
> URL: https://issues.apache.org/jira/browse/AVRO-1188
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Sharmarke Aden
> Assignee: Sharmarke Aden
> Fix For: 1.7.3
>
> Attachments: Avro-1188.tar.gz, Avro-1188.tar.gz, vcs-diff1160361655737792386.patch, vcs-diff2916139350460140957.patch, vcs-diff4277815358664835838.patch, vcs-diff6739872835137179667.patch
>
>
> There is no way for ".avsc" schema files to import types (i.e records, enums, etc) in external schema files. There's tremendous benefit in being able to do this as it would allow the sharing of common types between multiple schema files. Here's a use case that illustrates the typical usecase of this feature request.
> Suppose we have an enum called "Privacy" that we would like to share between multiple schemas:
> {code}
> //privacy.avsc
> {
> "type": "enum",
> "name": "Privacy",
> "symbols" : ["Public", "Private"]
> }
> {code}
> Now, if this feature was implemented one could import the above type into other schema files by doing something like this:
> {code}
> //the post.avsc
> {
> "type": "record",
> "name": "Post",
> "imports": ["privacy.avsc"] //you can import one or more schemas
> "fields": [{
> "name": "privacy",
> "type": [ "null", "Privacy"], //use imported Privacy type
> "default": "Private"
> }
> ]
> }
> {code}
> Here's another schema file that also has a similar privacy concern:
> {code}
> //the event.avsc is another schema that also imports the privacy type
> {
> "type": "record",
> "name": "Event",
> "imports": ["privacy.avsc"] //it also imports the privacy schemas
> "fields": [{
> "name": "privacy",
> "type": [ "null", "Privacy"], //use imported Privacy type
> "default": "Public"
> }
> ]
> }
> {code}
> IDL files are able to import external schemas and protocols and it would be very beneficial if schema files could import other schema files.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-1188) External Schema Imports via AVSC
Schema
Posted by "Sharmarke Aden (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sharmarke Aden updated AVRO-1188:
---------------------------------
Attachment: vcs-diff1160361655737792386.patch
> External Schema Imports via AVSC Schema
> ---------------------------------------
>
> Key: AVRO-1188
> URL: https://issues.apache.org/jira/browse/AVRO-1188
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Sharmarke Aden
> Attachments: Avro-1188.tar.gz, vcs-diff1160361655737792386.patch, vcs-diff2916139350460140957.patch, vcs-diff4277815358664835838.patch
>
>
> There is no way for ".avsc" schema files to import types (i.e records, enums, etc) in external schema files. There's tremendous benefit in being able to do this as it would allow the sharing of common types between multiple schema files. Here's a use case that illustrates the typical usecase of this feature request.
> Suppose we have an enum called "Privacy" that we would like to share between multiple schemas:
> {code}
> //privacy.avsc
> {
> "type": "enum",
> "name": "Privacy",
> "symbols" : ["Public", "Private"]
> }
> {code}
> Now, if this feature was implemented one could import the above type into other schema files by doing something like this:
> {code}
> //the post.avsc
> {
> "type": "record",
> "name": "Post",
> "imports": ["privacy.avsc"] //you can import one or more schemas
> "fields": [{
> "name": "privacy",
> "type": [ "null", "Privacy"], //use imported Privacy type
> "default": "Private"
> }
> ]
> }
> {code}
> Here's another schema file that also has a similar privacy concern:
> {code}
> //the event.avsc is another schema that also imports the privacy type
> {
> "type": "record",
> "name": "Event",
> "imports": ["privacy.avsc"] //it also imports the privacy schemas
> "fields": [{
> "name": "privacy",
> "type": [ "null", "Privacy"], //use imported Privacy type
> "default": "Public"
> }
> ]
> }
> {code}
> IDL files are able to import external schemas and protocols and it would be very beneficial if schema files could import other schema files.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-1188) External Schema Imports via AVSC
Schema
Posted by "Sharmarke Aden (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sharmarke Aden updated AVRO-1188:
---------------------------------
Description:
There is no way for ".avsc" schema files to import types (i.e records, enums, etc) in external schema files. There's tremendous benefit in being able to do this as it would allow the sharing of common types between multiple schema files. Here's a use case that illustrates the typical usecase of this feature request.
Suppose we have an enum called "Privacy" that we would like to share between multiple schemas:
{code}
//privacy.avsc
{
"type": "enum",
"name": "Privacy",
"symbols" : ["Public", "Private"]
}
{code}
Now, if this feature was implemented one could import the above type into other schema files by doing something like this:
{code}
//the post.avsc
{
"type": "record",
"name": "Post",
"imports": ["privacy.avsc"] //you can import one or more schemas
"fields": [{
"name": "privacy",
"type": [ "null", "Privacy"], //use imported Privacy type
"default": "Private"
}
]
}
{code}
Here's another schema file that also has a similar privacy concern:
{code}
//the event.avsc is another schema that also imports the privacy type
{
"type": "record",
"name": "Event",
"imports": ["privacy.avsc"] //it also imports the privacy schemas
"fields": [{
"name": "privacy",
"type": [ "null", "Privacy"], //use imported Privacy type
"default": "Public"
}
]
}
{code}
IDL files are able to import external schemas and protocols and it would be very beneficial if schema files could import other schema files.
was:
There is no way for ".avsc" schema files to import types (i.e records, enums, etc) in external schema files. There's tremendous benefit in being able to do this as it would all for the sharing of common types between multiple schema files. Here's a use case that illustrates the typical usecase of this feature request.
Suppose we have an enum called "Privacy" that we would like to share between multiple schemas:
{code}
//privacy.avsc
{
"type": "enum",
"name": "Privacy",
"symbols" : ["Public", "Private"]
}
{code}
Now, if this feature was implemented one could import the above type into other schema files by doing something like this:
{code}
//the post.avsc
{
"type": "record",
"name": "Post",
"imports": ["privacy.avsc"] //you can import one or more schemas
"fields": [{
"name": "privacy",
"type": [ "null", "Privacy"], //use imported Privacy type
"default": "Private"
}
]
}
{code}
Here's another schema file that also has a similar privacy concern:
{code}
//the event.avsc is another schema that also imports the privacy type
{
"type": "record",
"name": "Event",
"imports": ["privacy.avsc"] //it also imports the privacy schemas
"fields": [{
"name": "privacy",
"type": [ "null", "Privacy"], //use imported Privacy type
"default": "Public"
}
]
}
{code}
IDL files are able to import external schemas and protocols and it would be very beneficial if schema files could import other schema files.
> External Schema Imports via AVSC Schema
> ---------------------------------------
>
> Key: AVRO-1188
> URL: https://issues.apache.org/jira/browse/AVRO-1188
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Sharmarke Aden
>
> There is no way for ".avsc" schema files to import types (i.e records, enums, etc) in external schema files. There's tremendous benefit in being able to do this as it would allow the sharing of common types between multiple schema files. Here's a use case that illustrates the typical usecase of this feature request.
> Suppose we have an enum called "Privacy" that we would like to share between multiple schemas:
> {code}
> //privacy.avsc
> {
> "type": "enum",
> "name": "Privacy",
> "symbols" : ["Public", "Private"]
> }
> {code}
> Now, if this feature was implemented one could import the above type into other schema files by doing something like this:
> {code}
> //the post.avsc
> {
> "type": "record",
> "name": "Post",
> "imports": ["privacy.avsc"] //you can import one or more schemas
> "fields": [{
> "name": "privacy",
> "type": [ "null", "Privacy"], //use imported Privacy type
> "default": "Private"
> }
> ]
> }
> {code}
> Here's another schema file that also has a similar privacy concern:
> {code}
> //the event.avsc is another schema that also imports the privacy type
> {
> "type": "record",
> "name": "Event",
> "imports": ["privacy.avsc"] //it also imports the privacy schemas
> "fields": [{
> "name": "privacy",
> "type": [ "null", "Privacy"], //use imported Privacy type
> "default": "Public"
> }
> ]
> }
> {code}
> IDL files are able to import external schemas and protocols and it would be very beneficial if schema files could import other schema files.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1188) External Schema Imports via AVSC
Schema
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13490809#comment-13490809 ]
Doug Cutting commented on AVRO-1188:
------------------------------------
In many cases we require that schemas are standalone, that they do not have external references. For example, the schema included in a data file or transmitted to a remote server in RPC must be standalone. For this reason we've generally suggested that folks use some sort of pre-processor when they need to include schemas. The IDL compiler is one such pre-processor but one might also reasonably use cpp, m4 or some other pre-processor for this.
Note however that the Java schema parser remembers all named schemas previously parsed by the same schema. Thus if one first parses your privacy.avsc above then post.avsc and/or event.avsc with the same parser, then the effect you desire is achieved.
http://avro.apache.org/docs/current/api/java/org/apache/avro/Schema.Parser.html
This facility is used, e.g., by the command-line schema compiler. Schemas are processed in the order listed on the command line so that later schemas may depend on earlier schemas. AVRO-983 changes Avro's Maven plugin so that schemas are processed by a single parser in directory-traversal order, permitting dependencies.
Would either of these features work for you? If not, can you tell more about your use case?
> External Schema Imports via AVSC Schema
> ---------------------------------------
>
> Key: AVRO-1188
> URL: https://issues.apache.org/jira/browse/AVRO-1188
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Sharmarke Aden
>
> There is no way for ".avsc" schema files to import types (i.e records, enums, etc) in external schema files. There's tremendous benefit in being able to do this as it would all for the sharing of common types between multiple schema files. Here's a use case that illustrates the typical usecase of this feature request.
> Suppose we have an enum called "Privacy" that we would like to share between multiple schemas:
> {code}
> //privacy.avsc
> {
> "type": "enum",
> "name": "Privacy",
> "symbols" : ["Public", "Private"]
> }
> {code}
> Now, if this feature was implemented one could import the above type into other schema files by doing something like this:
> {code}
> //the post.avsc
> {
> "type": "record",
> "name": "Post",
> "imports": ["privacy.avsc"] //you can import one or more schemas
> "fields": [{
> "name": "privacy",
> "type": [ "null", "Privacy"], //use imported Privacy type
> "default": "Private"
> }
> ]
> }
> {code}
> Here's another schema file that also has a similar privacy concern:
> {code}
> //the event.avsc is another schema that also imports the privacy type
> {
> "type": "record",
> "name": "Event",
> "imports": ["privacy.avsc"] //it also imports the privacy schemas
> "fields": [{
> "name": "privacy",
> "type": [ "null", "Privacy"], //use imported Privacy type
> "default": "Public"
> }
> ]
> }
> {code}
> IDL files are able to import external schemas and protocols and it would be very beneficial if schema files could import other schema files.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1188) External Schema Imports via AVSC
Schema
Posted by "Sharmarke Aden (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13498867#comment-13498867 ]
Sharmarke Aden commented on AVRO-1188:
--------------------------------------
I have created AVRO-1207 and submitted a patch for it. It adds tests for all the mojos and it also tests schema imports.
> External Schema Imports via AVSC Schema
> ---------------------------------------
>
> Key: AVRO-1188
> URL: https://issues.apache.org/jira/browse/AVRO-1188
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Sharmarke Aden
> Assignee: Sharmarke Aden
> Fix For: 1.7.3
>
> Attachments: Avro-1188.tar.gz, Avro-1188.tar.gz, vcs-diff1160361655737792386.patch, vcs-diff2916139350460140957.patch, vcs-diff4277815358664835838.patch, vcs-diff6739872835137179667.patch
>
>
> There is no way for ".avsc" schema files to import types (i.e records, enums, etc) in external schema files. There's tremendous benefit in being able to do this as it would allow the sharing of common types between multiple schema files. Here's a use case that illustrates the typical usecase of this feature request.
> Suppose we have an enum called "Privacy" that we would like to share between multiple schemas:
> {code}
> //privacy.avsc
> {
> "type": "enum",
> "name": "Privacy",
> "symbols" : ["Public", "Private"]
> }
> {code}
> Now, if this feature was implemented one could import the above type into other schema files by doing something like this:
> {code}
> //the post.avsc
> {
> "type": "record",
> "name": "Post",
> "imports": ["privacy.avsc"] //you can import one or more schemas
> "fields": [{
> "name": "privacy",
> "type": [ "null", "Privacy"], //use imported Privacy type
> "default": "Private"
> }
> ]
> }
> {code}
> Here's another schema file that also has a similar privacy concern:
> {code}
> //the event.avsc is another schema that also imports the privacy type
> {
> "type": "record",
> "name": "Event",
> "imports": ["privacy.avsc"] //it also imports the privacy schemas
> "fields": [{
> "name": "privacy",
> "type": [ "null", "Privacy"], //use imported Privacy type
> "default": "Public"
> }
> ]
> }
> {code}
> IDL files are able to import external schemas and protocols and it would be very beneficial if schema files could import other schema files.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1188) External Schema Imports via AVSC
Schema
Posted by "Sharmarke Aden (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13494954#comment-13494954 ]
Sharmarke Aden commented on AVRO-1188:
--------------------------------------
This patch adds the ability to import directories or files to the maven plugin
> External Schema Imports via AVSC Schema
> ---------------------------------------
>
> Key: AVRO-1188
> URL: https://issues.apache.org/jira/browse/AVRO-1188
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Sharmarke Aden
> Attachments: vcs-diff4277815358664835838.patch
>
>
> There is no way for ".avsc" schema files to import types (i.e records, enums, etc) in external schema files. There's tremendous benefit in being able to do this as it would allow the sharing of common types between multiple schema files. Here's a use case that illustrates the typical usecase of this feature request.
> Suppose we have an enum called "Privacy" that we would like to share between multiple schemas:
> {code}
> //privacy.avsc
> {
> "type": "enum",
> "name": "Privacy",
> "symbols" : ["Public", "Private"]
> }
> {code}
> Now, if this feature was implemented one could import the above type into other schema files by doing something like this:
> {code}
> //the post.avsc
> {
> "type": "record",
> "name": "Post",
> "imports": ["privacy.avsc"] //you can import one or more schemas
> "fields": [{
> "name": "privacy",
> "type": [ "null", "Privacy"], //use imported Privacy type
> "default": "Private"
> }
> ]
> }
> {code}
> Here's another schema file that also has a similar privacy concern:
> {code}
> //the event.avsc is another schema that also imports the privacy type
> {
> "type": "record",
> "name": "Event",
> "imports": ["privacy.avsc"] //it also imports the privacy schemas
> "fields": [{
> "name": "privacy",
> "type": [ "null", "Privacy"], //use imported Privacy type
> "default": "Public"
> }
> ]
> }
> {code}
> IDL files are able to import external schemas and protocols and it would be very beneficial if schema files could import other schema files.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1188) External Schema Imports via AVSC
Schema
Posted by "Sharmarke Aden (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13496465#comment-13496465 ]
Sharmarke Aden commented on AVRO-1188:
--------------------------------------
My pleasure. Thanks you guys for your guidance.
> External Schema Imports via AVSC Schema
> ---------------------------------------
>
> Key: AVRO-1188
> URL: https://issues.apache.org/jira/browse/AVRO-1188
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Sharmarke Aden
> Assignee: Sharmarke Aden
> Fix For: 1.7.3
>
> Attachments: Avro-1188.tar.gz, Avro-1188.tar.gz, vcs-diff1160361655737792386.patch, vcs-diff2916139350460140957.patch, vcs-diff4277815358664835838.patch, vcs-diff6739872835137179667.patch
>
>
> There is no way for ".avsc" schema files to import types (i.e records, enums, etc) in external schema files. There's tremendous benefit in being able to do this as it would allow the sharing of common types between multiple schema files. Here's a use case that illustrates the typical usecase of this feature request.
> Suppose we have an enum called "Privacy" that we would like to share between multiple schemas:
> {code}
> //privacy.avsc
> {
> "type": "enum",
> "name": "Privacy",
> "symbols" : ["Public", "Private"]
> }
> {code}
> Now, if this feature was implemented one could import the above type into other schema files by doing something like this:
> {code}
> //the post.avsc
> {
> "type": "record",
> "name": "Post",
> "imports": ["privacy.avsc"] //you can import one or more schemas
> "fields": [{
> "name": "privacy",
> "type": [ "null", "Privacy"], //use imported Privacy type
> "default": "Private"
> }
> ]
> }
> {code}
> Here's another schema file that also has a similar privacy concern:
> {code}
> //the event.avsc is another schema that also imports the privacy type
> {
> "type": "record",
> "name": "Event",
> "imports": ["privacy.avsc"] //it also imports the privacy schemas
> "fields": [{
> "name": "privacy",
> "type": [ "null", "Privacy"], //use imported Privacy type
> "default": "Public"
> }
> ]
> }
> {code}
> IDL files are able to import external schemas and protocols and it would be very beneficial if schema files could import other schema files.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1188) External Schema Imports via AVSC
Schema
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13497540#comment-13497540 ]
Doug Cutting commented on AVRO-1188:
------------------------------------
We should perhaps add directories lang/java/maven-plugin/src/test/avro/{input,output,include} then write a test that's like TestIdl.java and TestSpecificCompilerTool.java, which compiles schemas in the input directory, specifying the include directory, and compares the output to what's in the output directory.
> External Schema Imports via AVSC Schema
> ---------------------------------------
>
> Key: AVRO-1188
> URL: https://issues.apache.org/jira/browse/AVRO-1188
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Sharmarke Aden
> Assignee: Sharmarke Aden
> Fix For: 1.7.3
>
> Attachments: Avro-1188.tar.gz, Avro-1188.tar.gz, vcs-diff1160361655737792386.patch, vcs-diff2916139350460140957.patch, vcs-diff4277815358664835838.patch, vcs-diff6739872835137179667.patch
>
>
> There is no way for ".avsc" schema files to import types (i.e records, enums, etc) in external schema files. There's tremendous benefit in being able to do this as it would allow the sharing of common types between multiple schema files. Here's a use case that illustrates the typical usecase of this feature request.
> Suppose we have an enum called "Privacy" that we would like to share between multiple schemas:
> {code}
> //privacy.avsc
> {
> "type": "enum",
> "name": "Privacy",
> "symbols" : ["Public", "Private"]
> }
> {code}
> Now, if this feature was implemented one could import the above type into other schema files by doing something like this:
> {code}
> //the post.avsc
> {
> "type": "record",
> "name": "Post",
> "imports": ["privacy.avsc"] //you can import one or more schemas
> "fields": [{
> "name": "privacy",
> "type": [ "null", "Privacy"], //use imported Privacy type
> "default": "Private"
> }
> ]
> }
> {code}
> Here's another schema file that also has a similar privacy concern:
> {code}
> //the event.avsc is another schema that also imports the privacy type
> {
> "type": "record",
> "name": "Event",
> "imports": ["privacy.avsc"] //it also imports the privacy schemas
> "fields": [{
> "name": "privacy",
> "type": [ "null", "Privacy"], //use imported Privacy type
> "default": "Public"
> }
> ]
> }
> {code}
> IDL files are able to import external schemas and protocols and it would be very beneficial if schema files could import other schema files.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-1188) External Schema Imports via AVSC
Schema
Posted by "Sharmarke Aden (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sharmarke Aden updated AVRO-1188:
---------------------------------
Attachment: vcs-diff6739872835137179667.patch
> External Schema Imports via AVSC Schema
> ---------------------------------------
>
> Key: AVRO-1188
> URL: https://issues.apache.org/jira/browse/AVRO-1188
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Sharmarke Aden
> Assignee: Sharmarke Aden
> Attachments: Avro-1188.tar.gz, vcs-diff1160361655737792386.patch, vcs-diff2916139350460140957.patch, vcs-diff4277815358664835838.patch, vcs-diff6739872835137179667.patch
>
>
> There is no way for ".avsc" schema files to import types (i.e records, enums, etc) in external schema files. There's tremendous benefit in being able to do this as it would allow the sharing of common types between multiple schema files. Here's a use case that illustrates the typical usecase of this feature request.
> Suppose we have an enum called "Privacy" that we would like to share between multiple schemas:
> {code}
> //privacy.avsc
> {
> "type": "enum",
> "name": "Privacy",
> "symbols" : ["Public", "Private"]
> }
> {code}
> Now, if this feature was implemented one could import the above type into other schema files by doing something like this:
> {code}
> //the post.avsc
> {
> "type": "record",
> "name": "Post",
> "imports": ["privacy.avsc"] //you can import one or more schemas
> "fields": [{
> "name": "privacy",
> "type": [ "null", "Privacy"], //use imported Privacy type
> "default": "Private"
> }
> ]
> }
> {code}
> Here's another schema file that also has a similar privacy concern:
> {code}
> //the event.avsc is another schema that also imports the privacy type
> {
> "type": "record",
> "name": "Event",
> "imports": ["privacy.avsc"] //it also imports the privacy schemas
> "fields": [{
> "name": "privacy",
> "type": [ "null", "Privacy"], //use imported Privacy type
> "default": "Public"
> }
> ]
> }
> {code}
> IDL files are able to import external schemas and protocols and it would be very beneficial if schema files could import other schema files.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1188) External Schema Imports via AVSC
Schema
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495292#comment-13495292 ]
Tom White commented on AVRO-1188:
---------------------------------
Thanks for the patch, Sharmarke. I tried using the patch with a small project I was testing AVRO-983 with and it worked fine. A few comments on the patch:
* Using a static Schema.Parser could potentially cause problems if the plugin is used more than once in a Maven project, since only one parser instance is used and it would retain references to earlier schemas. Instead, make it a non-static instance variable.
* If importing files then project.addCompileSourceRoot is called twice with the same output directory. I'm not sure if this is a problem, but it might be better to call it just once.
* Nit: declaration of "File file" can be moved to the line where the object is first instantiated. This occurs in two places in the patch.
> External Schema Imports via AVSC Schema
> ---------------------------------------
>
> Key: AVRO-1188
> URL: https://issues.apache.org/jira/browse/AVRO-1188
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Sharmarke Aden
> Attachments: Avro-1188.tar.gz, vcs-diff1160361655737792386.patch, vcs-diff2916139350460140957.patch, vcs-diff4277815358664835838.patch
>
>
> There is no way for ".avsc" schema files to import types (i.e records, enums, etc) in external schema files. There's tremendous benefit in being able to do this as it would allow the sharing of common types between multiple schema files. Here's a use case that illustrates the typical usecase of this feature request.
> Suppose we have an enum called "Privacy" that we would like to share between multiple schemas:
> {code}
> //privacy.avsc
> {
> "type": "enum",
> "name": "Privacy",
> "symbols" : ["Public", "Private"]
> }
> {code}
> Now, if this feature was implemented one could import the above type into other schema files by doing something like this:
> {code}
> //the post.avsc
> {
> "type": "record",
> "name": "Post",
> "imports": ["privacy.avsc"] //you can import one or more schemas
> "fields": [{
> "name": "privacy",
> "type": [ "null", "Privacy"], //use imported Privacy type
> "default": "Private"
> }
> ]
> }
> {code}
> Here's another schema file that also has a similar privacy concern:
> {code}
> //the event.avsc is another schema that also imports the privacy type
> {
> "type": "record",
> "name": "Event",
> "imports": ["privacy.avsc"] //it also imports the privacy schemas
> "fields": [{
> "name": "privacy",
> "type": [ "null", "Privacy"], //use imported Privacy type
> "default": "Public"
> }
> ]
> }
> {code}
> IDL files are able to import external schemas and protocols and it would be very beneficial if schema files could import other schema files.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (AVRO-1188) External Schema Imports via AVSC
Schema
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White reassigned AVRO-1188:
-------------------------------
Assignee: Sharmarke Aden
> External Schema Imports via AVSC Schema
> ---------------------------------------
>
> Key: AVRO-1188
> URL: https://issues.apache.org/jira/browse/AVRO-1188
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Sharmarke Aden
> Assignee: Sharmarke Aden
> Attachments: Avro-1188.tar.gz, vcs-diff1160361655737792386.patch, vcs-diff2916139350460140957.patch, vcs-diff4277815358664835838.patch
>
>
> There is no way for ".avsc" schema files to import types (i.e records, enums, etc) in external schema files. There's tremendous benefit in being able to do this as it would allow the sharing of common types between multiple schema files. Here's a use case that illustrates the typical usecase of this feature request.
> Suppose we have an enum called "Privacy" that we would like to share between multiple schemas:
> {code}
> //privacy.avsc
> {
> "type": "enum",
> "name": "Privacy",
> "symbols" : ["Public", "Private"]
> }
> {code}
> Now, if this feature was implemented one could import the above type into other schema files by doing something like this:
> {code}
> //the post.avsc
> {
> "type": "record",
> "name": "Post",
> "imports": ["privacy.avsc"] //you can import one or more schemas
> "fields": [{
> "name": "privacy",
> "type": [ "null", "Privacy"], //use imported Privacy type
> "default": "Private"
> }
> ]
> }
> {code}
> Here's another schema file that also has a similar privacy concern:
> {code}
> //the event.avsc is another schema that also imports the privacy type
> {
> "type": "record",
> "name": "Event",
> "imports": ["privacy.avsc"] //it also imports the privacy schemas
> "fields": [{
> "name": "privacy",
> "type": [ "null", "Privacy"], //use imported Privacy type
> "default": "Public"
> }
> ]
> }
> {code}
> IDL files are able to import external schemas and protocols and it would be very beneficial if schema files could import other schema files.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1188) External Schema Imports via AVSC
Schema
Posted by "Sharmarke Aden (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495235#comment-13495235 ]
Sharmarke Aden commented on AVRO-1188:
--------------------------------------
This patch makes everything hunky-dory and backward compatible by checking to see if the user opted in to the importing of files. If user specified importedFiles then a single schema parser is used otherwise we do things the old way and create a new schema parser every time.
> External Schema Imports via AVSC Schema
> ---------------------------------------
>
> Key: AVRO-1188
> URL: https://issues.apache.org/jira/browse/AVRO-1188
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Sharmarke Aden
> Attachments: Avro-1188.tar.gz, vcs-diff1160361655737792386.patch, vcs-diff2916139350460140957.patch, vcs-diff4277815358664835838.patch
>
>
> There is no way for ".avsc" schema files to import types (i.e records, enums, etc) in external schema files. There's tremendous benefit in being able to do this as it would allow the sharing of common types between multiple schema files. Here's a use case that illustrates the typical usecase of this feature request.
> Suppose we have an enum called "Privacy" that we would like to share between multiple schemas:
> {code}
> //privacy.avsc
> {
> "type": "enum",
> "name": "Privacy",
> "symbols" : ["Public", "Private"]
> }
> {code}
> Now, if this feature was implemented one could import the above type into other schema files by doing something like this:
> {code}
> //the post.avsc
> {
> "type": "record",
> "name": "Post",
> "imports": ["privacy.avsc"] //you can import one or more schemas
> "fields": [{
> "name": "privacy",
> "type": [ "null", "Privacy"], //use imported Privacy type
> "default": "Private"
> }
> ]
> }
> {code}
> Here's another schema file that also has a similar privacy concern:
> {code}
> //the event.avsc is another schema that also imports the privacy type
> {
> "type": "record",
> "name": "Event",
> "imports": ["privacy.avsc"] //it also imports the privacy schemas
> "fields": [{
> "name": "privacy",
> "type": [ "null", "Privacy"], //use imported Privacy type
> "default": "Public"
> }
> ]
> }
> {code}
> IDL files are able to import external schemas and protocols and it would be very beneficial if schema files could import other schema files.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1188) External Schema Imports via AVSC
Schema
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13497918#comment-13497918 ]
Tom White commented on AVRO-1188:
---------------------------------
Yes, we should have a test for this - I was a bit hasty in my commit, sorry! I didn't think there were any tests for the avro-maven-plugin, but in retrospect I realise that the ipc and mapred modules are using it, so they are tests of sorts for the plugin already. Sharmarke, are you able take a look at this?
> External Schema Imports via AVSC Schema
> ---------------------------------------
>
> Key: AVRO-1188
> URL: https://issues.apache.org/jira/browse/AVRO-1188
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Sharmarke Aden
> Assignee: Sharmarke Aden
> Fix For: 1.7.3
>
> Attachments: Avro-1188.tar.gz, Avro-1188.tar.gz, vcs-diff1160361655737792386.patch, vcs-diff2916139350460140957.patch, vcs-diff4277815358664835838.patch, vcs-diff6739872835137179667.patch
>
>
> There is no way for ".avsc" schema files to import types (i.e records, enums, etc) in external schema files. There's tremendous benefit in being able to do this as it would allow the sharing of common types between multiple schema files. Here's a use case that illustrates the typical usecase of this feature request.
> Suppose we have an enum called "Privacy" that we would like to share between multiple schemas:
> {code}
> //privacy.avsc
> {
> "type": "enum",
> "name": "Privacy",
> "symbols" : ["Public", "Private"]
> }
> {code}
> Now, if this feature was implemented one could import the above type into other schema files by doing something like this:
> {code}
> //the post.avsc
> {
> "type": "record",
> "name": "Post",
> "imports": ["privacy.avsc"] //you can import one or more schemas
> "fields": [{
> "name": "privacy",
> "type": [ "null", "Privacy"], //use imported Privacy type
> "default": "Private"
> }
> ]
> }
> {code}
> Here's another schema file that also has a similar privacy concern:
> {code}
> //the event.avsc is another schema that also imports the privacy type
> {
> "type": "record",
> "name": "Event",
> "imports": ["privacy.avsc"] //it also imports the privacy schemas
> "fields": [{
> "name": "privacy",
> "type": [ "null", "Privacy"], //use imported Privacy type
> "default": "Public"
> }
> ]
> }
> {code}
> IDL files are able to import external schemas and protocols and it would be very beneficial if schema files could import other schema files.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-1188) External Schema Imports via AVSC
Schema
Posted by "Sharmarke Aden (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sharmarke Aden updated AVRO-1188:
---------------------------------
Attachment: Avro-1188.tar.gz
Here is an updated project that demonstrates usage of the import functionality. The plugin deceleration essentially looks like this:
{code}
<plugin>
<groupId>org.apache.avro</groupId>
<artifactId>avro-maven-plugin</artifactId>
<executions>
<execution>
<goals>
<goal>schema</goal>
</goals>
</execution>
</executions>
<configuration>
<imports>
<!-- import a directory -->
<import>${basedir}/src/main/avro/imports</import>
<!-- import a file -->
<import>${basedir}/src/main/avro/directImport/PrivacyDirectImport.avsc</import>
</imports>
</configuration>
</plugin>
{code}
> External Schema Imports via AVSC Schema
> ---------------------------------------
>
> Key: AVRO-1188
> URL: https://issues.apache.org/jira/browse/AVRO-1188
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Sharmarke Aden
> Assignee: Sharmarke Aden
> Attachments: Avro-1188.tar.gz, Avro-1188.tar.gz, vcs-diff1160361655737792386.patch, vcs-diff2916139350460140957.patch, vcs-diff4277815358664835838.patch, vcs-diff6739872835137179667.patch
>
>
> There is no way for ".avsc" schema files to import types (i.e records, enums, etc) in external schema files. There's tremendous benefit in being able to do this as it would allow the sharing of common types between multiple schema files. Here's a use case that illustrates the typical usecase of this feature request.
> Suppose we have an enum called "Privacy" that we would like to share between multiple schemas:
> {code}
> //privacy.avsc
> {
> "type": "enum",
> "name": "Privacy",
> "symbols" : ["Public", "Private"]
> }
> {code}
> Now, if this feature was implemented one could import the above type into other schema files by doing something like this:
> {code}
> //the post.avsc
> {
> "type": "record",
> "name": "Post",
> "imports": ["privacy.avsc"] //you can import one or more schemas
> "fields": [{
> "name": "privacy",
> "type": [ "null", "Privacy"], //use imported Privacy type
> "default": "Private"
> }
> ]
> }
> {code}
> Here's another schema file that also has a similar privacy concern:
> {code}
> //the event.avsc is another schema that also imports the privacy type
> {
> "type": "record",
> "name": "Event",
> "imports": ["privacy.avsc"] //it also imports the privacy schemas
> "fields": [{
> "name": "privacy",
> "type": [ "null", "Privacy"], //use imported Privacy type
> "default": "Public"
> }
> ]
> }
> {code}
> IDL files are able to import external schemas and protocols and it would be very beneficial if schema files could import other schema files.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1188) External Schema Imports via AVSC
Schema
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13497436#comment-13497436 ]
Doug Cutting commented on AVRO-1188:
------------------------------------
Shouldn't we add a test? I didn't see one in the commit.
> External Schema Imports via AVSC Schema
> ---------------------------------------
>
> Key: AVRO-1188
> URL: https://issues.apache.org/jira/browse/AVRO-1188
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Sharmarke Aden
> Assignee: Sharmarke Aden
> Fix For: 1.7.3
>
> Attachments: Avro-1188.tar.gz, Avro-1188.tar.gz, vcs-diff1160361655737792386.patch, vcs-diff2916139350460140957.patch, vcs-diff4277815358664835838.patch, vcs-diff6739872835137179667.patch
>
>
> There is no way for ".avsc" schema files to import types (i.e records, enums, etc) in external schema files. There's tremendous benefit in being able to do this as it would allow the sharing of common types between multiple schema files. Here's a use case that illustrates the typical usecase of this feature request.
> Suppose we have an enum called "Privacy" that we would like to share between multiple schemas:
> {code}
> //privacy.avsc
> {
> "type": "enum",
> "name": "Privacy",
> "symbols" : ["Public", "Private"]
> }
> {code}
> Now, if this feature was implemented one could import the above type into other schema files by doing something like this:
> {code}
> //the post.avsc
> {
> "type": "record",
> "name": "Post",
> "imports": ["privacy.avsc"] //you can import one or more schemas
> "fields": [{
> "name": "privacy",
> "type": [ "null", "Privacy"], //use imported Privacy type
> "default": "Private"
> }
> ]
> }
> {code}
> Here's another schema file that also has a similar privacy concern:
> {code}
> //the event.avsc is another schema that also imports the privacy type
> {
> "type": "record",
> "name": "Event",
> "imports": ["privacy.avsc"] //it also imports the privacy schemas
> "fields": [{
> "name": "privacy",
> "type": [ "null", "Privacy"], //use imported Privacy type
> "default": "Public"
> }
> ]
> }
> {code}
> IDL files are able to import external schemas and protocols and it would be very beneficial if schema files could import other schema files.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1188) External Schema Imports via AVSC
Schema
Posted by "Sharmarke Aden (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495018#comment-13495018 ]
Sharmarke Aden commented on AVRO-1188:
--------------------------------------
Updated patch which gaurds against NPE if importedFiles is not defined
> External Schema Imports via AVSC Schema
> ---------------------------------------
>
> Key: AVRO-1188
> URL: https://issues.apache.org/jira/browse/AVRO-1188
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Sharmarke Aden
> Attachments: Avro-1188.tar.gz, vcs-diff2916139350460140957.patch, vcs-diff4277815358664835838.patch
>
>
> There is no way for ".avsc" schema files to import types (i.e records, enums, etc) in external schema files. There's tremendous benefit in being able to do this as it would allow the sharing of common types between multiple schema files. Here's a use case that illustrates the typical usecase of this feature request.
> Suppose we have an enum called "Privacy" that we would like to share between multiple schemas:
> {code}
> //privacy.avsc
> {
> "type": "enum",
> "name": "Privacy",
> "symbols" : ["Public", "Private"]
> }
> {code}
> Now, if this feature was implemented one could import the above type into other schema files by doing something like this:
> {code}
> //the post.avsc
> {
> "type": "record",
> "name": "Post",
> "imports": ["privacy.avsc"] //you can import one or more schemas
> "fields": [{
> "name": "privacy",
> "type": [ "null", "Privacy"], //use imported Privacy type
> "default": "Private"
> }
> ]
> }
> {code}
> Here's another schema file that also has a similar privacy concern:
> {code}
> //the event.avsc is another schema that also imports the privacy type
> {
> "type": "record",
> "name": "Event",
> "imports": ["privacy.avsc"] //it also imports the privacy schemas
> "fields": [{
> "name": "privacy",
> "type": [ "null", "Privacy"], //use imported Privacy type
> "default": "Public"
> }
> ]
> }
> {code}
> IDL files are able to import external schemas and protocols and it would be very beneficial if schema files could import other schema files.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira