You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by vundela <gi...@git.apache.org> on 2016/09/07 21:14:47 UTC
[GitHub] spark pull request #15003: [MINOR][SQL] Stripping extra new lines from the g...
GitHub user vundela opened a pull request:
https://github.com/apache/spark/pull/15003
[MINOR][SQL] Stripping extra new lines from the generator code
## What changes were proposed in this pull request?
Here is the simple unit test
test("Extra new lines in generated code") {
GeneratePredicate.generate(EqualTo(Literal(1), Literal(1)))
}
Generated code without fix
/* 001 */ public SpecificPredicate generate(Object[] references) {
/* 002 */ return new SpecificPredicate(references);
/* 003 */ }
/* 004 */
/* 005 */ class SpecificPredicate extends org.apache.spark.sql.catalyst.expressions.codegen.Predicate {
/* 006 */ private final Object[] references;
/* 007 */
/* 008 */
/* 009 */
/* 010 */ public SpecificPredicate(Object[] references) {
/* 011 */ this.references = references;
/* 012 */
/* 013 */ }
/* 014 */
/* 015 */ public boolean eval(InternalRow i) {
/* 016 */
/* 017 */ boolean isNull = false;
/* 018 */
/* 019 */
/* 020 */ boolean value = false;
/* 021 */ value = 1 == 1;
/* 022 */ return !false && value;
/* 023 */ }
/* 024 */ }
Generated code with fix
/* 001 */ public SpecificPredicate generate(Object[] references) {
/* 002 */ return new SpecificPredicate(references);
/* 003 */ }
/* 004 */
/* 005 */ class SpecificPredicate extends org.apache.spark.sql.catalyst.expressions.codegen.Predicate {
/* 006 */ private final Object[] references;
/* 007 */
/* 008 */ public SpecificPredicate(Object[] references) {
/* 009 */ this.references = references;
/* 010 */
/* 011 */ }
/* 012 */
/* 013 */ public boolean eval(InternalRow i) {
/* 014 */ boolean isNull = false;
/* 015 */
/* 016 */ boolean value = false;
/* 017 */ value = 1 == 1;
/* 018 */ return !false && value;
/* 019 */ }
/* 020 */ }
## How was this patch tested?
Manually tested the generated code and it strips the extra new lines.
(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/vundela/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/15003.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #15003
----
commit 7ace69ec3cb06bee9fa87b08c3a2038d027e0506
Author: Srinivasa Reddy Vundela <vs...@cloudera.com>
Date: 2016-09-07T21:04:42Z
[MINOR][SQL] Stripping extra new lines from the generator code
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15003: [MINOR][SQL] Stripping extra new lines from the generato...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/15003
The 64k limit wouldn't be affected by whitespace in the source code.
I bet some of the code generation could be optimized. That may be a janino issue, and is certainly separate.
Yes the logs are shorter but the question is readability, and generally whitespace helps readability.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15003: [MINOR][SQL] Stripping extra new lines from the generato...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/15003
I see, does it help readability though to strip the newlines? they make the logs scroll longer but I would have thought they're a nice to have when trying to read them. Are there are large stretches of redundant lines?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15003: [MINOR][SQL] Stripping extra new lines from the generato...
Posted by vundela <gi...@git.apache.org>.
Github user vundela commented on the issue:
https://github.com/apache/spark/pull/15003
Agreed @srowen, I will close this PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15003: [MINOR][SQL] Stripping extra new lines from the generato...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/15003
What does this improve though? It won't matter to the compiled code. It also doesn't save generating a version with those newlines.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15003: [MINOR][SQL] Stripping extra new lines from the generato...
Posted by vundela <gi...@git.apache.org>.
Github user vundela commented on the issue:
https://github.com/apache/spark/pull/15003
@srowen, Its mostly for code readability. One of the customer got >64kb JVM code size limit and I see lot of new lines. Thought it would give better readability when code is in logs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15003: [MINOR][SQL] Stripping extra new lines from the generato...
Posted by vundela <gi...@git.apache.org>.
Github user vundela commented on the issue:
https://github.com/apache/spark/pull/15003
Confirmed that comments and extra new lines in method body will not lead to 64KB limit exception.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15003: [MINOR][SQL] Stripping extra new lines from the generato...
Posted by vundela <gi...@git.apache.org>.
Github user vundela commented on the issue:
https://github.com/apache/spark/pull/15003
I am also seeing lot of redundant code. I did the grep for single if condition on a variable got98 and here is the output.
boolean got98 = false;
if (!got98) {
got98 = true;
if (!got98) {
boolean got98 = false;
if (!got98) {
got98 = true;
if (!got98) {
boolean got98 = false;
if (!got98) {
got98 = true;
if (!got98) {
boolean got98 = false;
if (!got98) {
got98 = true;
if (!got98) {
boolean got98 = false;
if (!got98) {
got98 = true;
if (!got98) {
boolean got98 = false;
if (!got98) {
got98 = true;
if (!got98) {
boolean got98 = false;
if (!got98) {
got98 = true;
if (!got98) {
boolean got98 = false;
if (!got98) {
got98 = true;
if (!got98) {
boolean got98 = false;
if (!got98) {
got98 = true;
if (!got98) {
boolean got98 = false;
if (!got98) {
got98 = true;
if (!got98) {
boolean got98 = false;
if (!got98) {
got98 = true;
if (!got98) {
boolean got98 = false;
if (!got98) {
got98 = true;
if (!got98) {
Also a quick look into the generated code of if blocks seems to be same with human eye.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15003: [MINOR][SQL] Stripping extra new lines from the generato...
Posted by vundela <gi...@git.apache.org>.
Github user vundela commented on the issue:
https://github.com/apache/spark/pull/15003
Definitely log size will be lesser. When I look into the generated code for caseWhen expressions with UDF's, majority was comments and extra new lines. I don't know whether JVM considers the comments and extra new lines when it throws the exception. If yes, this might help. I went through the generated code again and it seems readability is good.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15003: [MINOR][SQL] Stripping extra new lines from the g...
Posted by vundela <gi...@git.apache.org>.
Github user vundela closed the pull request at:
https://github.com/apache/spark/pull/15003
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org