You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2020/07/15 16:58:35 UTC
[GitHub] [hive] miklosgergely opened a new pull request #1258: HIVE-23857: Fix HiveParser 'code too large' problem (Miklos Gergely, reviewed by David Mollitor)
miklosgergely opened a new pull request #1258:
URL: https://github.com/apache/hive/pull/1258
HivePasrser.g can not be extended anymore as adding any more tokens leads to a "code too large" problem, because the compiled code size would exceed 65536 bytes. The real solution would be to introduce anltr4, in the meantime it can be fixed be moving the tokenNames variable into a separate file.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] miklosgergely commented on a change in pull request #1258: HIVE-23857: Fix HiveParser 'code too large' problem (Miklos Gergely, reviewed by David Mollitor)
Posted by GitBox <gi...@apache.org>.
miklosgergely commented on a change in pull request #1258:
URL: https://github.com/apache/hive/pull/1258#discussion_r455352739
##########
File path: parser/bin/fixHiveParser.sh
##########
@@ -0,0 +1,44 @@
+#!/bin/bash
+
+# This is a temporary solution for the issue of the "code too large" problem related to HiveParser.java
+# We got to a point where adding anything to the antlr files lead to an issue about having a HiveParser.java that can not be compiled due to the compiled code size limitation in java (maximum 65536 bytes), so to avoid it we temorarly add this script to remove the huge tokenNames array into a separate file.
+# The real solution would be to switch to antlr 4
+
+tokenFile="target/generated-sources/antlr3/org/apache/hadoop/hive/ql/parse/HiveParserTokens.java"
+input="target/generated-sources/antlr3/org/apache/hadoop/hive/ql/parse/HiveParser.java"
+output="target/generated-sources/antlr3/org/apache/hadoop/hive/ql/parse/HiveParser.java-fixed"
+
+rm $tokenFile > /dev/null 2>&1
+rm $output > /dev/null 2>&1
+
+echo "package org.apache.hadoop.hive.ql.parse;" >> $tokenFile
+echo "" >> $tokenFile
+echo "public class HiveParserTokens {" >> $tokenFile
+
+state="STAY"
+while IFS= read -r line
Review comment:
Thank you @t3rmin4t0r, I've modified the patch using awk.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] miklosgergely merged pull request #1258: HIVE-23857: Fix HiveParser 'code too large' problem (Miklos Gergely, reviewed by David Mollitor)
Posted by GitBox <gi...@apache.org>.
miklosgergely merged pull request #1258:
URL: https://github.com/apache/hive/pull/1258
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] t3rmin4t0r commented on a change in pull request #1258: HIVE-23857: Fix HiveParser 'code too large' problem (Miklos Gergely, reviewed by David Mollitor)
Posted by GitBox <gi...@apache.org>.
t3rmin4t0r commented on a change in pull request #1258:
URL: https://github.com/apache/hive/pull/1258#discussion_r455302352
##########
File path: parser/bin/fixHiveParser.sh
##########
@@ -0,0 +1,44 @@
+#!/bin/bash
+
+# This is a temporary solution for the issue of the "code too large" problem related to HiveParser.java
+# We got to a point where adding anything to the antlr files lead to an issue about having a HiveParser.java that can not be compiled due to the compiled code size limitation in java (maximum 65536 bytes), so to avoid it we temorarly add this script to remove the huge tokenNames array into a separate file.
+# The real solution would be to switch to antlr 4
+
+tokenFile="target/generated-sources/antlr3/org/apache/hadoop/hive/ql/parse/HiveParserTokens.java"
+input="target/generated-sources/antlr3/org/apache/hadoop/hive/ql/parse/HiveParser.java"
+output="target/generated-sources/antlr3/org/apache/hadoop/hive/ql/parse/HiveParser.java-fixed"
+
+rm $tokenFile > /dev/null 2>&1
+rm $output > /dev/null 2>&1
+
+echo "package org.apache.hadoop.hive.ql.parse;" >> $tokenFile
+echo "" >> $tokenFile
+echo "public class HiveParserTokens {" >> $tokenFile
+
+state="STAY"
+while IFS= read -r line
Review comment:
Looks like AWK reinvented in bash
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] belugabehr commented on pull request #1258: HIVE-23857: Fix HiveParser 'code too large' problem (Miklos Gergely, reviewed by David Mollitor)
Posted by GitBox <gi...@apache.org>.
belugabehr commented on pull request #1258:
URL: https://github.com/apache/hive/pull/1258#issuecomment-658902129
I once looked at this and was trying to avoid this route, but I guess it's all we can do unless someone wants to take up the mantle of going to ANTL4.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org