You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by GitBox <gi...@apache.org> on 2020/05/16 08:08:53 UTC

[GitHub] [avro] maccamlc opened a new pull request #885: Avro 2621 maven custom logical types

maccamlc opened a new pull request #885:
URL: https://github.com/apache/avro/pull/885


   Change is based on top of https://github.com/apache/avro/pull/883 (allow fixed/enum type logical type conversions to be used in specific compiler, and added test with custom enum type logical type and conversion)
   
   ### Jira
   
   - [x] My PR addresses the following [Avro Jira](https://issues.apache.org/jira/browse/AVRO-2621) issues and references them in the PR title.
   
   ### Tests
   
   - [x] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason:
   
   Integration Tests added that showing a custom logical type being used, and working along with a custom conversions.
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines. In addition, my commits follow the guidelines from "[How to write a good git commit message](https://chris.beams.io/posts/git-commit/)":
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes how to use it.
     - All the public functions and the classes in the PR contain Javadoc that explain what it does
   
   Added an option to maven plugin, along with a description.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [avro] maccamlc commented on a change in pull request #885: AVRO-2840 maven custom logical types

Posted by GitBox <gi...@apache.org>.
maccamlc commented on a change in pull request #885:
URL: https://github.com/apache/avro/pull/885#discussion_r711494098



##########
File path: lang/java/compiler/src/test/java/org/apache/avro/compiler/specific/TestSpecificCompiler.java
##########
@@ -516,6 +521,36 @@ public void testNullableTypesJavaUnbox() throws Exception {
         "java.lang.Boolean");
   }
 
+  @Test
+  public void testGetUsedCustomLogicalTypeFactories() throws Exception {
+    LogicalTypes.register("string-constant", new StringCustomLogicalTypeFactory());
+
+    SpecificCompiler compiler = createCompiler();
+    compiler.setEnableDecimalLogicalType(true);
+
+    final Schema schema = new Schema.Parser().parse(
+        "{\"type\":\"record\",\"name\":\"NestedLogicalTypesRecord\",\"namespace\":\"org.apache.avro.codegentest.testdata\",\"doc\":\"Test nested types with logical types in generated Java classes\",\"fields\":[{\"name\":\"nestedRecord\",\"type\":{\"type\":\"record\",\"name\":\"NestedRecord\",\"fields\":[{\"name\":\"nullableDateField\",\"type\":[\"null\",{\"type\":\"int\",\"logicalType\":\"date\"}]}]}},{\"name\":\"myLogical\",\"type\":{\"type\":\"string\",\"logicalType\":\"string-constant\"}}]}");

Review comment:
       Updated 👍 

##########
File path: lang/java/compiler/src/test/java/org/apache/avro/compiler/specific/TestSpecificCompiler.java
##########
@@ -516,6 +521,36 @@ public void testNullableTypesJavaUnbox() throws Exception {
         "java.lang.Boolean");
   }
 
+  @Test
+  public void testGetUsedCustomLogicalTypeFactories() throws Exception {
+    LogicalTypes.register("string-constant", new StringCustomLogicalTypeFactory());
+
+    SpecificCompiler compiler = createCompiler();
+    compiler.setEnableDecimalLogicalType(true);
+
+    final Schema schema = new Schema.Parser().parse(
+        "{\"type\":\"record\",\"name\":\"NestedLogicalTypesRecord\",\"namespace\":\"org.apache.avro.codegentest.testdata\",\"doc\":\"Test nested types with logical types in generated Java classes\",\"fields\":[{\"name\":\"nestedRecord\",\"type\":{\"type\":\"record\",\"name\":\"NestedRecord\",\"fields\":[{\"name\":\"nullableDateField\",\"type\":[\"null\",{\"type\":\"int\",\"logicalType\":\"date\"}]}]}},{\"name\":\"myLogical\",\"type\":{\"type\":\"string\",\"logicalType\":\"string-constant\"}}]}");
+
+    final Set<String> usedCustomLogicalTypeFactories = compiler.getUsedCustomLogicalTypeFactories(schema);
+    Assert.assertEquals(1, usedCustomLogicalTypeFactories.size());
+    Assert.assertEquals("org.apache.avro.compiler.specific.TestSpecificCompiler.StringCustomLogicalTypeFactory",
+        usedCustomLogicalTypeFactories.iterator().next());
+  }
+
+  @Test
+  public void testEmptyGetUsedCustomLogicalTypeFactories() throws Exception {
+    LogicalTypes.register("string-constant", new StringCustomLogicalTypeFactory());
+
+    SpecificCompiler compiler = createCompiler();
+    compiler.setEnableDecimalLogicalType(true);
+
+    final Schema schema = new Schema.Parser().parse(
+        "{\"type\":\"record\",\"name\":\"NestedLogicalTypesRecord\",\"namespace\":\"org.apache.avro.codegentest.testdata\",\"doc\":\"Test nested types with logical types in generated Java classes\",\"fields\":[{\"name\":\"nestedRecord\",\"type\":{\"type\":\"record\",\"name\":\"NestedRecord\",\"fields\":[{\"name\":\"nullableDateField\",\"type\":[\"null\",{\"type\":\"int\",\"logicalType\":\"date\"}]}]}}]}");

Review comment:
       Updated 👍 

##########
File path: lang/java/compiler/src/test/java/org/apache/avro/compiler/specific/TestSpecificCompiler.java
##########
@@ -860,4 +895,11 @@ public void testAdditionalToolsAreInjectedIntoTemplate() throws Exception {
     assertEquals(1, itWorksFound);
   }
 
+  public static class StringCustomLogicalTypeFactory implements LogicalTypes.LogicalTypeFactory {
+    @Override
+    public LogicalType fromSchema(Schema schema) {
+      return new LogicalType("string-constant");

Review comment:
       Updated 👍 

##########
File path: lang/java/integration-test/codegen-test/src/test/resources/avro/custom_conversion_logical_types_enum.avsc
##########
@@ -0,0 +1,19 @@
+{"namespace": "org.apache.avro.codegentest.testdata",

Review comment:
       Updated 👍 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [avro] maccamlc commented on pull request #885: AVRO-2840 maven custom logical types

Posted by GitBox <gi...@apache.org>.
maccamlc commented on pull request #885:
URL: https://github.com/apache/avro/pull/885#issuecomment-730015808


   @icecreamhead no problem, will do it today


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [avro] maccamlc commented on pull request #885: AVRO-2840 maven custom logical types

Posted by GitBox <gi...@apache.org>.
maccamlc commented on pull request #885:
URL: https://github.com/apache/avro/pull/885#issuecomment-634422984


   Just pushed an update. 
   
   Some PRs were merged today, including https://github.com/apache/avro/pull/535 that was a duplicate of this attempt.
   
   The strategy was similar, but still different.
   
   Updated this PR to:
   
   * Fix broken tests in the build after changes to SpecificCompilerTool.java
   * Allow the auto generated code to auto register the used logical types that have been registered with the specific compiler


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [avro] RyanSkraba merged pull request #885: AVRO-2840 maven custom logical types

Posted by GitBox <gi...@apache.org>.
RyanSkraba merged pull request #885:
URL: https://github.com/apache/avro/pull/885


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@avro.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [avro] icecreamhead commented on pull request #885: AVRO-2840 maven custom logical types

Posted by GitBox <gi...@apache.org>.
icecreamhead commented on pull request #885:
URL: https://github.com/apache/avro/pull/885#issuecomment-729749685


   @maccamlc Can you rebase so we can try to get this in?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [avro] maccamlc edited a comment on pull request #885: AVRO-2840 maven custom logical types

Posted by GitBox <gi...@apache.org>.
maccamlc edited a comment on pull request #885:
URL: https://github.com/apache/avro/pull/885#issuecomment-640462476


   > Hello! I'm afraid I've gotten a bit mixed up on what's happening with [AVRO-2493](https://issues.apache.org/jira/browse/AVRO-2493) and [AVRO-2621](https://issues.apache.org/jira/browse/AVRO-2621) -- what is this PR changing now that [AVRO-2493](https://issues.apache.org/jira/browse/AVRO-2493) has been implemented? Are custom logical types still unusable with the avro maven plugin?
   
   Initially I was unaware of [AVRO-2493](https://issues.apache.org/jira/browse/AVRO-2493) and was looking to implement a solution against a different ticket. 
   
   When saw this was merged, for me, the missing part was simply auto registering the used logical type in the Specific code generated classes at [SpecificCompiler#getUsedCustomLogicalTypeFactories](https://github.com/apache/avro/pull/885/files#diff-1131527472ff7e3626a7e126568fedc8), and also added a shortcut to register the [Logical Type](https://github.com/apache/avro/pull/885/files#diff-95cda30d5633f1e967fbe8e873f7a667) using the new type name.
   
   From my point of view on the usages, if you have auto generated a class using a logical type, then you probably always expect to also use it at runtime (and likely also have a custom conversion implemented that needs to be present at compile and runtime), so removing the need to manually register the types of startup. But if other peoples use cases are different, potentially this could be behind a flag?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [avro] maccamlc commented on a change in pull request #885: AVRO-2840 maven custom logical types

Posted by GitBox <gi...@apache.org>.
maccamlc commented on a change in pull request #885:
URL: https://github.com/apache/avro/pull/885#discussion_r711493615



##########
File path: lang/java/avro/src/test/java/org/apache/avro/TestLogicalType.java
##########
@@ -226,6 +228,73 @@ public void testLogicalTypeInSchemaEquals() {
     assertEqualsFalse("Different logical type", schema1, schema3);
   }
 
+  @Test
+  public void testRegisterLogicalTypeThrowsIfTypeNameInconsistent() {
+    assertThrows("Should error if type name is inconsistent", IllegalArgumentException.class,
+        "Provided logicalTypeName 'name' does not match factory typeName 'different'", () -> {
+          LogicalTypes.register("name", new LogicalTypes.LogicalTypeFactory() {
+            @Override
+            public LogicalType fromSchema(Schema schema) {
+              return LogicalTypes.date();
+            }
+
+            @Override
+            public String getTypeName() {
+              return "different";
+            }
+          });
+          return null;
+        });
+  }
+
+  @Test
+  public void testRegisterLogicalTypeWithName() {
+    final LogicalTypes.LogicalTypeFactory factory = new LogicalTypes.LogicalTypeFactory() {
+      @Override
+      public LogicalType fromSchema(Schema schema) {
+        return LogicalTypes.date();
+      }
+
+      @Override
+      public String getTypeName() {
+        return "same";
+      }
+    };
+
+    LogicalTypes.register("same", factory);
+
+    MatcherAssert.assertThat(LogicalTypes.getCustomRegisteredTypes(), IsMapContaining.hasEntry("same", factory));
+  }
+
+  @Test
+  public void testRegisterLogicalTypeWithFactoryName() {
+    final LogicalTypes.LogicalTypeFactory factory = new LogicalTypes.LogicalTypeFactory() {
+      @Override
+      public LogicalType fromSchema(Schema schema) {
+        return LogicalTypes.date();
+      }
+
+      @Override
+      public String getTypeName() {
+        return "factory";
+      }
+    };
+
+    LogicalTypes.register(factory);
+
+    MatcherAssert.assertThat(LogicalTypes.getCustomRegisteredTypes(), IsMapContaining.hasEntry("factory", factory));
+  }
+
+  @Test
+  public void testRegisterLogicalTypeWithFactoryNameNotProvidedWithoutError() {
+    final LogicalTypes.LogicalTypeFactory factory = schema -> LogicalTypes.date();
+
+    LogicalTypes.register("logicalTypeName", factory);
+
+    MatcherAssert.assertThat(LogicalTypes.getCustomRegisteredTypes(),
+        IsMapContaining.hasEntry("logicalTypeName", factory));
+  }
+

Review comment:
       Reworked




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [avro] maccamlc commented on pull request #885: AVRO-2840 maven custom logical types

Posted by GitBox <gi...@apache.org>.
maccamlc commented on pull request #885:
URL: https://github.com/apache/avro/pull/885#issuecomment-921600096


   @RyanSkraba @icecreamhead @helderdiniz fwiw I have updated this PR on top of current master. Any feedback welcome as would like to get this merged, if possible.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [avro] RyanSkraba commented on pull request #885: AVRO-2840 maven custom logical types

Posted by GitBox <gi...@apache.org>.
RyanSkraba commented on pull request #885:
URL: https://github.com/apache/avro/pull/885#issuecomment-640452277


   Hello!  I'm afraid I've gotten a bit mixed up on what's happening with AVRO-2493 and AVRO-2621 -- what is this PR changing now that AVRO-2493 has been implemented?  Are custom logical types still unusable with the avro maven plugin?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [avro] maccamlc commented on a change in pull request #885: AVRO-2840 maven custom logical types

Posted by GitBox <gi...@apache.org>.
maccamlc commented on a change in pull request #885:
URL: https://github.com/apache/avro/pull/885#discussion_r711495284



##########
File path: lang/java/avro/src/main/java/org/apache/avro/LogicalTypes.java
##########
@@ -51,9 +65,27 @@ default String getTypeName() {
   public static void register(String logicalTypeName, LogicalTypeFactory factory) {
     Objects.requireNonNull(logicalTypeName, "Logical type name cannot be null");
     Objects.requireNonNull(factory, "Logical type factory cannot be null");
+
+    try {
+      String factoryTypeName = factory.getTypeName();
+      if (!logicalTypeName.equals(factoryTypeName)) {
+        throw new IllegalArgumentException(String.format(

Review comment:
       Just in case if there is a difference, updated the Specific Compiler to use the method that takes the string key and the factory when generating code.
   
   Technically shouldn't be an issue, as to use a custom logic type factory in the maven plugin, it requires the type name. But seems safer this way for now to me




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [avro] maccamlc commented on a change in pull request #885: AVRO-2840 maven custom logical types

Posted by GitBox <gi...@apache.org>.
maccamlc commented on a change in pull request #885:
URL: https://github.com/apache/avro/pull/885#discussion_r711493820



##########
File path: lang/java/compiler/src/main/java/org/apache/avro/compiler/specific/SpecificCompiler.java
##########
@@ -282,37 +287,51 @@ public void addCustomConversion(Class<?> conversionClass) {
 
   public Collection<String> getUsedConversionClasses(Schema schema) {
     Collection<String> result = new HashSet<>();
-    for (Conversion<?> conversion : getUsedConversions(schema, new HashSet<>(), new HashSet<>())) {
+    for (Conversion<?> conversion : getUsedConversions(schema)) {
       result.add(conversion.getClass().getCanonicalName());
     }
     return result;
   }
 
-  private Set<Conversion<?>> getUsedConversions(Schema schema, Set<Conversion<?>> result, Set<Schema> seenSchemas) {
+  public Set<String> getUsedCustomLogicalTypeFactories(Schema schema) {
+    final Set<String> logicalTypeNames = getUsedLogicalTypes(schema).stream().map(LogicalType::getName)
+        .collect(Collectors.toSet());
+
+    return LogicalTypes.getCustomRegisteredTypes().entrySet().stream()
+        .filter(entry -> logicalTypeNames.contains(entry.getKey()))
+        .map(entry -> entry.getValue().getClass().getCanonicalName()).collect(Collectors.toSet());
+  }
+
+  private void getUsedTypes(Schema schema, Set<Conversion<?>> conversionResults, Set<LogicalType> logicalTypeResults,

Review comment:
       Updated 👍 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [avro] maccamlc edited a comment on pull request #885: AVRO-2840 maven custom logical types

Posted by GitBox <gi...@apache.org>.
maccamlc edited a comment on pull request #885:
URL: https://github.com/apache/avro/pull/885#issuecomment-919083998


   @RyanSkraba I am happy to rework this or get up to date again, as I still don't believe this is supported yet. Or could a new approach can be used to support this same thing?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [avro] maccamlc commented on pull request #885: AVRO-2840 maven custom logical types

Posted by GitBox <gi...@apache.org>.
maccamlc commented on pull request #885:
URL: https://github.com/apache/avro/pull/885#issuecomment-640462476


   > Hello! I'm afraid I've gotten a bit mixed up on what's happening with [AVRO-2493](https://issues.apache.org/jira/browse/AVRO-2493) and [AVRO-2621](https://issues.apache.org/jira/browse/AVRO-2621) -- what is this PR changing now that [AVRO-2493](https://issues.apache.org/jira/browse/AVRO-2493) has been implemented? Are custom logical types still unusable with the avro maven plugin?
   
   Initially I was unaware of [AVRO-2493](https://issues.apache.org/jira/browse/AVRO-2493) and was looking to implement a solution against a different ticket. 
   
   When saw this was merged, for me, the missing part was simply auto registering the used logical type in the Specific code generated classes at [SpecificCompiler#getUsedCustomLogicalTypeFactories](https://github.com/apache/avro/pull/885/files#diff-1131527472ff7e3626a7e126568fedc8), and also added a shortcut to register the [Logical Type](https://github.com/apache/avro/pull/885/files#diff-95cda30d5633f1e967fbe8e873f7a667) using the new type name.
   
   From my point of view on the usages, if you have auto generated a class using a logical type, then you probably always expect to also use it at runtime, so removing the need to manually register the types of startup. But if other peoples use cases are different, potentially this could be behind a flag?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [avro] maccamlc commented on pull request #885: AVRO-2840 maven custom logical types

Posted by GitBox <gi...@apache.org>.
maccamlc commented on pull request #885:
URL: https://github.com/apache/avro/pull/885#issuecomment-640331276


   @RyanSkraba would something like this be useful on 1.10.0 release, or keep what currently been added there in other PR?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [avro] martin-g commented on a change in pull request #885: AVRO-2840 maven custom logical types

Posted by GitBox <gi...@apache.org>.
martin-g commented on a change in pull request #885:
URL: https://github.com/apache/avro/pull/885#discussion_r710947377



##########
File path: lang/java/avro/src/main/java/org/apache/avro/LogicalTypes.java
##########
@@ -51,9 +65,27 @@ default String getTypeName() {
   public static void register(String logicalTypeName, LogicalTypeFactory factory) {
     Objects.requireNonNull(logicalTypeName, "Logical type name cannot be null");
     Objects.requireNonNull(factory, "Logical type factory cannot be null");
+
+    try {
+      String factoryTypeName = factory.getTypeName();
+      if (!logicalTypeName.equals(factoryTypeName)) {
+        throw new IllegalArgumentException(String.format(

Review comment:
       This might break some old users of this API.

##########
File path: lang/java/avro/src/main/java/org/apache/avro/LogicalTypes.java
##########
@@ -51,9 +65,27 @@ default String getTypeName() {
   public static void register(String logicalTypeName, LogicalTypeFactory factory) {
     Objects.requireNonNull(logicalTypeName, "Logical type name cannot be null");
     Objects.requireNonNull(factory, "Logical type factory cannot be null");
+
+    try {
+      String factoryTypeName = factory.getTypeName();
+      if (!logicalTypeName.equals(factoryTypeName)) {
+        throw new IllegalArgumentException(String.format(
+            "Provided logicalTypeName '%s' does not match factory typeName '%s'", logicalTypeName, factoryTypeName));
+      }
+    } catch (UnsupportedOperationException ignore) {
+      // ignore exception, as by default this value has not been provided

Review comment:
       Which value is not provided ? It is not very clear.
   It seems only `factory.getTypeName();` can throw it. And it looks like a hack to avoid the new protection.

##########
File path: lang/java/avro/src/test/java/org/apache/avro/TestLogicalType.java
##########
@@ -226,6 +228,73 @@ public void testLogicalTypeInSchemaEquals() {
     assertEqualsFalse("Different logical type", schema1, schema3);
   }
 
+  @Test
+  public void testRegisterLogicalTypeThrowsIfTypeNameInconsistent() {
+    assertThrows("Should error if type name is inconsistent", IllegalArgumentException.class,
+        "Provided logicalTypeName 'name' does not match factory typeName 'different'", () -> {
+          LogicalTypes.register("name", new LogicalTypes.LogicalTypeFactory() {
+            @Override
+            public LogicalType fromSchema(Schema schema) {
+              return LogicalTypes.date();
+            }
+
+            @Override
+            public String getTypeName() {
+              return "different";
+            }
+          });
+          return null;
+        });
+  }
+
+  @Test
+  public void testRegisterLogicalTypeWithName() {
+    final LogicalTypes.LogicalTypeFactory factory = new LogicalTypes.LogicalTypeFactory() {
+      @Override
+      public LogicalType fromSchema(Schema schema) {
+        return LogicalTypes.date();
+      }
+
+      @Override
+      public String getTypeName() {
+        return "same";
+      }
+    };
+
+    LogicalTypes.register("same", factory);
+
+    MatcherAssert.assertThat(LogicalTypes.getCustomRegisteredTypes(), IsMapContaining.hasEntry("same", factory));
+  }
+
+  @Test
+  public void testRegisterLogicalTypeWithFactoryName() {
+    final LogicalTypes.LogicalTypeFactory factory = new LogicalTypes.LogicalTypeFactory() {
+      @Override
+      public LogicalType fromSchema(Schema schema) {
+        return LogicalTypes.date();
+      }
+
+      @Override
+      public String getTypeName() {
+        return "factory";
+      }
+    };
+
+    LogicalTypes.register(factory);
+
+    MatcherAssert.assertThat(LogicalTypes.getCustomRegisteredTypes(), IsMapContaining.hasEntry("factory", factory));
+  }
+
+  @Test
+  public void testRegisterLogicalTypeWithFactoryNameNotProvidedWithoutError() {
+    final LogicalTypes.LogicalTypeFactory factory = schema -> LogicalTypes.date();
+
+    LogicalTypes.register("logicalTypeName", factory);
+
+    MatcherAssert.assertThat(LogicalTypes.getCustomRegisteredTypes(),
+        IsMapContaining.hasEntry("logicalTypeName", factory));
+  }
+

Review comment:
       It is clear to me now!
   I'll let others decide whether the improvement (the names check) is allowed now. I see its value but as I said it might break someone's app.

##########
File path: lang/java/compiler/src/test/java/org/apache/avro/compiler/specific/TestSpecificCompiler.java
##########
@@ -300,8 +305,8 @@ public void testSettingOutputCharacterEncoding() throws Exception {
     is.close(); // close input stream otherwise delete might fail
     if (!this.outputFile.delete()) {
       throw new IllegalStateException("unable to delete " + this.outputFile); // delete otherwise compiler might not
-                                                                              // overwrite because src timestamp hasn't
-                                                                              // changed.
+      // overwrite because src timestamp hasn't

Review comment:
       Please restore the formatting or move the whole comment on the previous line

##########
File path: lang/java/compiler/src/test/java/org/apache/avro/compiler/specific/TestSpecificCompiler.java
##########
@@ -516,6 +521,36 @@ public void testNullableTypesJavaUnbox() throws Exception {
         "java.lang.Boolean");
   }
 
+  @Test
+  public void testGetUsedCustomLogicalTypeFactories() throws Exception {
+    LogicalTypes.register("string-constant", new StringCustomLogicalTypeFactory());
+
+    SpecificCompiler compiler = createCompiler();
+    compiler.setEnableDecimalLogicalType(true);
+
+    final Schema schema = new Schema.Parser().parse(
+        "{\"type\":\"record\",\"name\":\"NestedLogicalTypesRecord\",\"namespace\":\"org.apache.avro.codegentest.testdata\",\"doc\":\"Test nested types with logical types in generated Java classes\",\"fields\":[{\"name\":\"nestedRecord\",\"type\":{\"type\":\"record\",\"name\":\"NestedRecord\",\"fields\":[{\"name\":\"nullableDateField\",\"type\":[\"null\",{\"type\":\"int\",\"logicalType\":\"date\"}]}]}},{\"name\":\"myLogical\",\"type\":{\"type\":\"string\",\"logicalType\":\"string-constant\"}}]}");

Review comment:
       let's split this line into several shorter concatenated ones

##########
File path: lang/java/compiler/src/test/java/org/apache/avro/compiler/specific/TestSpecificCompiler.java
##########
@@ -860,4 +895,11 @@ public void testAdditionalToolsAreInjectedIntoTemplate() throws Exception {
     assertEquals(1, itWorksFound);
   }
 
+  public static class StringCustomLogicalTypeFactory implements LogicalTypes.LogicalTypeFactory {
+    @Override
+    public LogicalType fromSchema(Schema schema) {
+      return new LogicalType("string-constant");

Review comment:
       `"string-custom"` ?!

##########
File path: lang/java/compiler/src/main/java/org/apache/avro/compiler/specific/SpecificCompiler.java
##########
@@ -282,37 +287,51 @@ public void addCustomConversion(Class<?> conversionClass) {
 
   public Collection<String> getUsedConversionClasses(Schema schema) {
     Collection<String> result = new HashSet<>();
-    for (Conversion<?> conversion : getUsedConversions(schema, new HashSet<>(), new HashSet<>())) {
+    for (Conversion<?> conversion : getUsedConversions(schema)) {
       result.add(conversion.getClass().getCanonicalName());
     }
     return result;
   }
 
-  private Set<Conversion<?>> getUsedConversions(Schema schema, Set<Conversion<?>> result, Set<Schema> seenSchemas) {
+  public Set<String> getUsedCustomLogicalTypeFactories(Schema schema) {
+    final Set<String> logicalTypeNames = getUsedLogicalTypes(schema).stream().map(LogicalType::getName)
+        .collect(Collectors.toSet());
+
+    return LogicalTypes.getCustomRegisteredTypes().entrySet().stream()
+        .filter(entry -> logicalTypeNames.contains(entry.getKey()))
+        .map(entry -> entry.getValue().getClass().getCanonicalName()).collect(Collectors.toSet());
+  }
+
+  private void getUsedTypes(Schema schema, Set<Conversion<?>> conversionResults, Set<LogicalType> logicalTypeResults,

Review comment:
       Since the method's return type is `void` I would suggest to rename it to `collectUsedTypes`.

##########
File path: lang/java/integration-test/codegen-test/src/test/resources/avro/custom_conversion_logical_types_enum.avsc
##########
@@ -0,0 +1,19 @@
+{"namespace": "org.apache.avro.codegentest.testdata",

Review comment:
       nit: move the namespace on a new line

##########
File path: lang/java/compiler/src/test/java/org/apache/avro/compiler/specific/TestSpecificCompiler.java
##########
@@ -516,6 +521,36 @@ public void testNullableTypesJavaUnbox() throws Exception {
         "java.lang.Boolean");
   }
 
+  @Test
+  public void testGetUsedCustomLogicalTypeFactories() throws Exception {
+    LogicalTypes.register("string-constant", new StringCustomLogicalTypeFactory());
+
+    SpecificCompiler compiler = createCompiler();
+    compiler.setEnableDecimalLogicalType(true);
+
+    final Schema schema = new Schema.Parser().parse(
+        "{\"type\":\"record\",\"name\":\"NestedLogicalTypesRecord\",\"namespace\":\"org.apache.avro.codegentest.testdata\",\"doc\":\"Test nested types with logical types in generated Java classes\",\"fields\":[{\"name\":\"nestedRecord\",\"type\":{\"type\":\"record\",\"name\":\"NestedRecord\",\"fields\":[{\"name\":\"nullableDateField\",\"type\":[\"null\",{\"type\":\"int\",\"logicalType\":\"date\"}]}]}},{\"name\":\"myLogical\",\"type\":{\"type\":\"string\",\"logicalType\":\"string-constant\"}}]}");
+
+    final Set<String> usedCustomLogicalTypeFactories = compiler.getUsedCustomLogicalTypeFactories(schema);
+    Assert.assertEquals(1, usedCustomLogicalTypeFactories.size());
+    Assert.assertEquals("org.apache.avro.compiler.specific.TestSpecificCompiler.StringCustomLogicalTypeFactory",
+        usedCustomLogicalTypeFactories.iterator().next());
+  }
+
+  @Test
+  public void testEmptyGetUsedCustomLogicalTypeFactories() throws Exception {
+    LogicalTypes.register("string-constant", new StringCustomLogicalTypeFactory());
+
+    SpecificCompiler compiler = createCompiler();
+    compiler.setEnableDecimalLogicalType(true);
+
+    final Schema schema = new Schema.Parser().parse(
+        "{\"type\":\"record\",\"name\":\"NestedLogicalTypesRecord\",\"namespace\":\"org.apache.avro.codegentest.testdata\",\"doc\":\"Test nested types with logical types in generated Java classes\",\"fields\":[{\"name\":\"nestedRecord\",\"type\":{\"type\":\"record\",\"name\":\"NestedRecord\",\"fields\":[{\"name\":\"nullableDateField\",\"type\":[\"null\",{\"type\":\"int\",\"logicalType\":\"date\"}]}]}}]}");

Review comment:
       line too long




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [avro] maccamlc commented on pull request #885: AVRO-2840 maven custom logical types

Posted by GitBox <gi...@apache.org>.
maccamlc commented on pull request #885:
URL: https://github.com/apache/avro/pull/885#issuecomment-919083998


   @RyanSkraba I am happy to rework this if a different approach is required. I still don't believe this is supported yet.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [avro] maccamlc edited a comment on pull request #885: AVRO-2840 maven custom logical types

Posted by GitBox <gi...@apache.org>.
maccamlc edited a comment on pull request #885:
URL: https://github.com/apache/avro/pull/885#issuecomment-634422984


   @RyanSkraba I just pushed an update. 
   
   Some PRs were merged today, including https://github.com/apache/avro/pull/535 that was a duplicate of this attempt.
   
   The strategy was similar, but still different.
   
   Updated this PR to:
   
   * Fix broken tests in the build after changes to SpecificCompilerTool.java
   * Allow the auto generated code to auto register the used logical types that have been registered with the specific compiler


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [avro] icecreamhead commented on pull request #885: AVRO-2840 maven custom logical types

Posted by GitBox <gi...@apache.org>.
icecreamhead commented on pull request #885:
URL: https://github.com/apache/avro/pull/885#issuecomment-731199562


   @RyanSkraba ?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [avro] maccamlc commented on a change in pull request #885: AVRO-2840 maven custom logical types

Posted by GitBox <gi...@apache.org>.
maccamlc commented on a change in pull request #885:
URL: https://github.com/apache/avro/pull/885#discussion_r711493550



##########
File path: lang/java/avro/src/main/java/org/apache/avro/LogicalTypes.java
##########
@@ -51,9 +65,27 @@ default String getTypeName() {
   public static void register(String logicalTypeName, LogicalTypeFactory factory) {
     Objects.requireNonNull(logicalTypeName, "Logical type name cannot be null");
     Objects.requireNonNull(factory, "Logical type factory cannot be null");
+
+    try {
+      String factoryTypeName = factory.getTypeName();
+      if (!logicalTypeName.equals(factoryTypeName)) {
+        throw new IllegalArgumentException(String.format(
+            "Provided logicalTypeName '%s' does not match factory typeName '%s'", logicalTypeName, factoryTypeName));
+      }
+    } catch (UnsupportedOperationException ignore) {
+      // ignore exception, as by default this value has not been provided

Review comment:
       Reworked. No exception, but might be worth to still log this. Eventually might want to settle on providing the value in the LogTypeFactory only? But now it should always use the value passed through in this method if they differ.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [avro] RyanSkraba commented on a change in pull request #885: AVRO-2840 maven custom logical types

Posted by GitBox <gi...@apache.org>.
RyanSkraba commented on a change in pull request #885:
URL: https://github.com/apache/avro/pull/885#discussion_r433315009



##########
File path: lang/java/compiler/src/main/java/org/apache/avro/compiler/specific/SpecificCompiler.java
##########
@@ -299,6 +304,24 @@ public void addCustomConversion(Class<?> conversionClass) {
     return result;
   }
 
+  @SuppressWarnings("unchecked")
+  public Set<String> getUsedCustomLogicalTypeFactories(Schema schema) {
+    final Map<String, LogicalTypes.LogicalTypeFactory> registeredTypesMap;
+    try {
+      final java.lang.reflect.Field registeredTypesField = LogicalTypes.class.getDeclaredField("REGISTERED_TYPES");

Review comment:
       I'd prefer to avoid this private access via reflection!  Should we just expose the registered types?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [avro] helderdiniz commented on pull request #885: AVRO-2840 maven custom logical types

Posted by GitBox <gi...@apache.org>.
helderdiniz commented on pull request #885:
URL: https://github.com/apache/avro/pull/885#issuecomment-814247329


   Could this see the light of day? Quite a bummer that the gradle plugin has this, but this one doesn't.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [avro] icecreamhead commented on pull request #885: AVRO-2840 maven custom logical types

Posted by GitBox <gi...@apache.org>.
icecreamhead commented on pull request #885:
URL: https://github.com/apache/avro/pull/885#issuecomment-728013536


   Hello, I recently stumbled across this exact issue when attempting to register a custom conversion. What do we need to do to get this moving again?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [avro] maccamlc commented on a change in pull request #885: AVRO-2840 maven custom logical types

Posted by GitBox <gi...@apache.org>.
maccamlc commented on a change in pull request #885:
URL: https://github.com/apache/avro/pull/885#discussion_r711494002



##########
File path: lang/java/compiler/src/test/java/org/apache/avro/compiler/specific/TestSpecificCompiler.java
##########
@@ -300,8 +305,8 @@ public void testSettingOutputCharacterEncoding() throws Exception {
     is.close(); // close input stream otherwise delete might fail
     if (!this.outputFile.delete()) {
       throw new IllegalStateException("unable to delete " + this.outputFile); // delete otherwise compiler might not
-                                                                              // overwrite because src timestamp hasn't
-                                                                              // changed.
+      // overwrite because src timestamp hasn't

Review comment:
       Not sure how that occurred, but reverted




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [avro] maccamlc commented on a change in pull request #885: AVRO-2840 maven custom logical types

Posted by GitBox <gi...@apache.org>.
maccamlc commented on a change in pull request #885:
URL: https://github.com/apache/avro/pull/885#discussion_r711480943



##########
File path: lang/java/avro/src/main/java/org/apache/avro/LogicalTypes.java
##########
@@ -51,9 +65,27 @@ default String getTypeName() {
   public static void register(String logicalTypeName, LogicalTypeFactory factory) {
     Objects.requireNonNull(logicalTypeName, "Logical type name cannot be null");
     Objects.requireNonNull(factory, "Logical type factory cannot be null");
+
+    try {
+      String factoryTypeName = factory.getTypeName();
+      if (!logicalTypeName.equals(factoryTypeName)) {
+        throw new IllegalArgumentException(String.format(

Review comment:
       Changed to a log message. Might be useful if this can be cleaned up at some point, as seems could be confusing that TypeName can be defined in the Factory, but if a different name is used to Register, that will be used instead.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [avro] icecreamhead commented on pull request #885: AVRO-2840 maven custom logical types

Posted by GitBox <gi...@apache.org>.
icecreamhead commented on pull request #885:
URL: https://github.com/apache/avro/pull/885#issuecomment-921612156


   @maccamlc looks great to me, shame there seems to be very few maintainers on this repo :( 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [avro] maccamlc commented on pull request #885: AVRO-2840 maven custom logical types

Posted by GitBox <gi...@apache.org>.
maccamlc commented on pull request #885:
URL: https://github.com/apache/avro/pull/885#issuecomment-636316896


   > @RyanSkraba I just pushed an update.
   > 
   > Some PRs were merged today, including #535 that was a duplicate of this attempt.
   > 
   > The strategy was similar, but still different.
   > 
   > Updated this PR to:
   > 
   > * Fix broken tests in the build after changes to SpecificCompilerTool.java
   > * Allow the auto generated code to auto register the used logical types that have been registered with the specific compiler
   
   Rebased onto master where the broken test was fixed


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [avro] maccamlc commented on pull request #885: AVRO-2840 maven custom logical types

Posted by GitBox <gi...@apache.org>.
maccamlc commented on pull request #885:
URL: https://github.com/apache/avro/pull/885#issuecomment-730071811


   @RyanSkraba @icecreamhead I have rebased this on top of master. Ready to review again / merge


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org