You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/08/19 08:01:53 UTC
[GitHub] [hudi] UZi5136225 opened a new pull request #1984: [HUDI-1200] Fix NullPointerException, KeyGenerator does not work
UZi5136225 opened a new pull request #1984:
URL: https://github.com/apache/hudi/pull/1984
## What is the purpose of the pull request
Fix NullPointerException, KeyGenerator does not work
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on pull request #1984: [HUDI-1200] Fix NullPointerException, CustomKeyGenerator does not work
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on pull request #1984:
URL: https://github.com/apache/hudi/pull/1984#issuecomment-683899921
@pratyakshsharma can you please take a look at this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pratyakshsharma commented on a change in pull request #1984: [HUDI-1200] Fix NullPointerException, CustomKeyGenerator does not work
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on a change in pull request #1984:
URL: https://github.com/apache/hudi/pull/1984#discussion_r485151075
##########
File path: hudi-spark/src/main/java/org/apache/hudi/keygen/KeyGenerator.java
##########
@@ -41,7 +41,7 @@
private static final String STRUCT_NAME = "hoodieRowTopLevelField";
private static final String NAMESPACE = "hoodieRow";
- protected transient TypedProperties config;
+ protected TypedProperties config;
Review comment:
@bhasudha @vinothchandar what was the purpose behind keeping this config object here as transient? I see it has been like this from the very beginning. There is a workaround solution also which only affects CustomKeyGenerator class but removing transient keyword from here actually seems the easier solution. :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pratyakshsharma commented on pull request #1984: [HUDI-1200] Fix NullPointerException, CustomKeyGenerator does not work
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on pull request #1984:
URL: https://github.com/apache/hudi/pull/1984#issuecomment-693578602
@liujinhui1994 Making the TypedProperties variable non transient works. Another work around can be https://github.com/apache/hudi/pull/2093
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] liujinhui1994 commented on a change in pull request #1984: [HUDI-1200] Fix NullPointerException, CustomKeyGenerator does not work
Posted by GitBox <gi...@apache.org>.
liujinhui1994 commented on a change in pull request #1984:
URL: https://github.com/apache/hudi/pull/1984#discussion_r474439210
##########
File path: hudi-spark/src/main/java/org/apache/hudi/keygen/KeyGenerator.java
##########
@@ -41,7 +41,7 @@
private static final String STRUCT_NAME = "hoodieRowTopLevelField";
private static final String NAMESPACE = "hoodieRow";
- protected transient TypedProperties config;
+ protected TypedProperties config;
Review comment:
Ok i understand what you think, let me try other solutions
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pratyakshsharma commented on pull request #1984: [HUDI-1200] Fix NullPointerException, CustomKeyGenerator does not work
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on pull request #1984:
URL: https://github.com/apache/hudi/pull/1984#issuecomment-687900186
@ashishmgofficial Apologies for the delay, will check this today and circle back.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on pull request #1984: [HUDI-1200] Fix NullPointerException, CustomKeyGenerator does not work
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on pull request #1984:
URL: https://github.com/apache/hudi/pull/1984#issuecomment-688581291
@pratyakshsharma I would like to get this into 0.6.1 if possible. please prioritize accordingly
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bhasudha commented on a change in pull request #1984: [HUDI-1200] Fix NullPointerException, CustomKeyGenerator does not work
Posted by GitBox <gi...@apache.org>.
bhasudha commented on a change in pull request #1984:
URL: https://github.com/apache/hudi/pull/1984#discussion_r478582473
##########
File path: hudi-spark/src/main/java/org/apache/hudi/keygen/KeyGenerator.java
##########
@@ -41,7 +41,7 @@
private static final String STRUCT_NAME = "hoodieRowTopLevelField";
private static final String NAMESPACE = "hoodieRow";
- protected transient TypedProperties config;
+ protected TypedProperties config;
Review comment:
I think so too. @liujinhui1994 can you try with Hudi 0.6.0 and see if that helps? We fixed some serialization issues there wrt KeyGenerators.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] UZi5136225 commented on pull request #1984: [HUDI-1200] Fix NullPointerException, KeyGenerator does not work
Posted by GitBox <gi...@apache.org>.
UZi5136225 commented on pull request #1984:
URL: https://github.com/apache/hudi/pull/1984#issuecomment-676055412
please help review
@bhasudha @yanghua
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pratyakshsharma commented on a change in pull request #1984: [HUDI-1200] Fix NullPointerException, CustomKeyGenerator does not work
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on a change in pull request #1984:
URL: https://github.com/apache/hudi/pull/1984#discussion_r485151075
##########
File path: hudi-spark/src/main/java/org/apache/hudi/keygen/KeyGenerator.java
##########
@@ -41,7 +41,7 @@
private static final String STRUCT_NAME = "hoodieRowTopLevelField";
private static final String NAMESPACE = "hoodieRow";
- protected transient TypedProperties config;
+ protected TypedProperties config;
Review comment:
@bhasudha @vinothchandar what was the purpose behind keeping this config object here as transient? There is a workaround solution also which only affects CustomKeyGenerator class but removing transient keyword from here actually seems the easier solution. :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] liujinhui1994 closed pull request #1984: [HUDI-1200] Fix NullPointerException, CustomKeyGenerator does not work
Posted by GitBox <gi...@apache.org>.
liujinhui1994 closed pull request #1984:
URL: https://github.com/apache/hudi/pull/1984
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pratyakshsharma commented on a change in pull request #1984: [HUDI-1200] Fix NullPointerException, CustomKeyGenerator does not work
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on a change in pull request #1984:
URL: https://github.com/apache/hudi/pull/1984#discussion_r485182381
##########
File path: hudi-spark/src/main/java/org/apache/hudi/keygen/KeyGenerator.java
##########
@@ -41,7 +41,7 @@
private static final String STRUCT_NAME = "hoodieRowTopLevelField";
private static final String NAMESPACE = "hoodieRow";
- protected transient TypedProperties config;
+ protected TypedProperties config;
Review comment:
If we do not want to make config object as transient, we have 2 choices -
1. introduce a new TypedProperties instance in CustomKeyGenerator
2. introduce a new TimestampBasedKeyGenerator instance in CustomKeyGenerator, which gets initialised only if needed.
Please suggest.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] wangxianghu commented on a change in pull request #1984: [HUDI-1200] Fix NullPointerException, CustomKeyGenerator does not work
Posted by GitBox <gi...@apache.org>.
wangxianghu commented on a change in pull request #1984:
URL: https://github.com/apache/hudi/pull/1984#discussion_r479085868
##########
File path: hudi-spark/src/main/java/org/apache/hudi/keygen/KeyGenerator.java
##########
@@ -41,7 +41,7 @@
private static final String STRUCT_NAME = "hoodieRowTopLevelField";
private static final String NAMESPACE = "hoodieRow";
- protected transient TypedProperties config;
+ protected TypedProperties config;
Review comment:
> I think so too. @liujinhui1994 can you try with Hudi 0.6.0 and see if that helps? We fixed some serialization issues there wrt KeyGenerators.
I tried both `TimestampBasedKeyGenerator` and `CustomKeyGenerator` with 0.6.0 (in my IDEA) `TimestampBasedKeyGenerator` is ok, while `CustomKeyGenerator` still has NPE.
Remove `transient ` can resolve this problem.
`Caused by: java.lang.NullPointerException
at org.apache.hudi.keygen.SimpleKeyGenerator.<init>(SimpleKeyGenerator.java:35)
at org.apache.hudi.keygen.CustomKeyGenerator.getRecordKey(CustomKeyGenerator.java:128)
at org.apache.hudi.keygen.BuiltinKeyGenerator.getKey(BuiltinKeyGenerator.java:75)
at org.apache.hudi.HoodieSparkSqlWriter$$anonfun$1.apply(HoodieSparkSqlWriter.scala:143)
at org.apache.hudi.HoodieSparkSqlWriter$$anonfun$1.apply(HoodieSparkSqlWriter.scala:139)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ashishmgofficial commented on pull request #1984: [HUDI-1200] Fix NullPointerException, CustomKeyGenerator does not work
Posted by GitBox <gi...@apache.org>.
ashishmgofficial commented on pull request #1984:
URL: https://github.com/apache/hudi/pull/1984#issuecomment-686388820
Hi Team, any updates on this. Wanted this fix in 0.6.0 as Im having this as part of a POC . @pratyakshsharma @bhasudha
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] liujinhui1994 commented on a change in pull request #1984: [HUDI-1200] Fix NullPointerException, CustomKeyGenerator does not work
Posted by GitBox <gi...@apache.org>.
liujinhui1994 commented on a change in pull request #1984:
URL: https://github.com/apache/hudi/pull/1984#discussion_r474439210
##########
File path: hudi-spark/src/main/java/org/apache/hudi/keygen/KeyGenerator.java
##########
@@ -41,7 +41,7 @@
private static final String STRUCT_NAME = "hoodieRowTopLevelField";
private static final String NAMESPACE = "hoodieRow";
- protected transient TypedProperties config;
+ protected TypedProperties config;
Review comment:
Ok i understand what you think, let me try other solutions
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pratyakshsharma commented on pull request #1984: [HUDI-1200] Fix NullPointerException, CustomKeyGenerator does not work
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on pull request #1984:
URL: https://github.com/apache/hudi/pull/1984#issuecomment-689053835
> @pratyakshsharma I would like to get this into 0.6.1 if possible. please prioritize accordingly
Checking this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pratyakshsharma commented on pull request #1984: [HUDI-1200] Fix NullPointerException, CustomKeyGenerator does not work
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on pull request #1984:
URL: https://github.com/apache/hudi/pull/1984#issuecomment-684474047
Ack, will check this today.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] wangxianghu commented on a change in pull request #1984: [HUDI-1200] Fix NullPointerException, CustomKeyGenerator does not work
Posted by GitBox <gi...@apache.org>.
wangxianghu commented on a change in pull request #1984:
URL: https://github.com/apache/hudi/pull/1984#discussion_r475038373
##########
File path: hudi-spark/src/main/java/org/apache/hudi/keygen/KeyGenerator.java
##########
@@ -41,7 +41,7 @@
private static final String STRUCT_NAME = "hoodieRowTopLevelField";
private static final String NAMESPACE = "hoodieRow";
- protected transient TypedProperties config;
+ protected TypedProperties config;
Review comment:
> This is intended to be transient so that we don't serialize config in KeyGenerator s. Lets keep it that way. Since the issue is specifically with CustomKeyGenerator we should resolve there.
Hi @bhasudha , CustomKeyGenerator extends KeyGenerator, maybe this NPE is caused by the config(lost in the process of serialization) in KeyGenerator. @liujinhui1994 is that what you mean?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] liujinhui1994 closed pull request #1984: [HUDI-1200] Fix NullPointerException, CustomKeyGenerator does not work
Posted by GitBox <gi...@apache.org>.
liujinhui1994 closed pull request #1984:
URL: https://github.com/apache/hudi/pull/1984
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] wangxianghu commented on a change in pull request #1984: [HUDI-1200] Fix NullPointerException, CustomKeyGenerator does not work
Posted by GitBox <gi...@apache.org>.
wangxianghu commented on a change in pull request #1984:
URL: https://github.com/apache/hudi/pull/1984#discussion_r479085868
##########
File path: hudi-spark/src/main/java/org/apache/hudi/keygen/KeyGenerator.java
##########
@@ -41,7 +41,7 @@
private static final String STRUCT_NAME = "hoodieRowTopLevelField";
private static final String NAMESPACE = "hoodieRow";
- protected transient TypedProperties config;
+ protected TypedProperties config;
Review comment:
> I think so too. @liujinhui1994 can you try with Hudi 0.6.0 and see if that helps? We fixed some serialization issues there wrt KeyGenerators.
I tried Both `TimestampBasedKeyGenerator` and `CustomKeyGenerator` with 0.6.0 (in my IDEA) `TimestampBasedKeyGenerator` is ok, while `CustomKeyGenerator` still has NPE.
Remove `transient ` can resolve this problem.
`Caused by: java.lang.NullPointerException
at org.apache.hudi.keygen.SimpleKeyGenerator.<init>(SimpleKeyGenerator.java:35)
at org.apache.hudi.keygen.CustomKeyGenerator.getRecordKey(CustomKeyGenerator.java:128)
at org.apache.hudi.keygen.BuiltinKeyGenerator.getKey(BuiltinKeyGenerator.java:75)
at org.apache.hudi.HoodieSparkSqlWriter$$anonfun$1.apply(HoodieSparkSqlWriter.scala:143)
at org.apache.hudi.HoodieSparkSqlWriter$$anonfun$1.apply(HoodieSparkSqlWriter.scala:139)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] liujinhui1994 commented on pull request #1984: [HUDI-1200] Fix NullPointerException, CustomKeyGenerator does not work
Posted by GitBox <gi...@apache.org>.
liujinhui1994 commented on pull request #1984:
URL: https://github.com/apache/hudi/pull/1984#issuecomment-693467930
@pratyakshsharma Do you have any suggestions for this PR? According to the content of this PR, I resolved the error
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] wangxianghu commented on a change in pull request #1984: [HUDI-1200] Fix NullPointerException, CustomKeyGenerator does not work
Posted by GitBox <gi...@apache.org>.
wangxianghu commented on a change in pull request #1984:
URL: https://github.com/apache/hudi/pull/1984#discussion_r475038373
##########
File path: hudi-spark/src/main/java/org/apache/hudi/keygen/KeyGenerator.java
##########
@@ -41,7 +41,7 @@
private static final String STRUCT_NAME = "hoodieRowTopLevelField";
private static final String NAMESPACE = "hoodieRow";
- protected transient TypedProperties config;
+ protected TypedProperties config;
Review comment:
> This is intended to be transient so that we don't serialize config in KeyGenerator s. Lets keep it that way. Since the issue is specifically with CustomKeyGenerator we should resolve there.
Hi @bhasudha , CustomKeyGenerator extends KeyGenerator, maybe this NPE is caused by the config(erased in the process of serialization) in KeyGenerator. @liujinhui1994 is that what you mean?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bhasudha commented on a change in pull request #1984: [HUDI-1200] Fix NullPointerException, CustomKeyGenerator does not work
Posted by GitBox <gi...@apache.org>.
bhasudha commented on a change in pull request #1984:
URL: https://github.com/apache/hudi/pull/1984#discussion_r473255513
##########
File path: hudi-spark/src/main/java/org/apache/hudi/keygen/KeyGenerator.java
##########
@@ -41,7 +41,7 @@
private static final String STRUCT_NAME = "hoodieRowTopLevelField";
private static final String NAMESPACE = "hoodieRow";
- protected transient TypedProperties config;
+ protected TypedProperties config;
Review comment:
This is intended to be transient so that we don't serialize config in KeyGenerator s. Lets keep it that way. Since the issue is specifically with CustomKeyGenerator we should resolve there.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] liujinhui1994 commented on a change in pull request #1984: [HUDI-1200] Fix NullPointerException, CustomKeyGenerator does not work
Posted by GitBox <gi...@apache.org>.
liujinhui1994 commented on a change in pull request #1984:
URL: https://github.com/apache/hudi/pull/1984#discussion_r474448203
##########
File path: hudi-spark/src/main/java/org/apache/hudi/keygen/KeyGenerator.java
##########
@@ -41,7 +41,7 @@
private static final String STRUCT_NAME = "hoodieRowTopLevelField";
private static final String NAMESPACE = "hoodieRow";
- protected transient TypedProperties config;
+ protected TypedProperties config;
Review comment:
Can you let me know what you think? Can you describe it in detail @bhasudha
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org