You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Sailesh Mukil (Code Review)" <ge...@cloudera.org> on 2016/12/06 08:10:01 UTC

[Impala-ASF-CR] IMPALA-4594: WriteSlot and CodegenWriteSlot handle escaped NULL slots differently

Sailesh Mukil has uploaded a new change for review.

  http://gerrit.cloudera.org:8080/5377

Change subject: IMPALA-4594: WriteSlot and CodegenWriteSlot handle escaped NULL slots differently
......................................................................

IMPALA-4594: WriteSlot and CodegenWriteSlot handle escaped NULL slots differently

CodegenWriteSlot() receives negative length values for the lengths of
the slots passed to it if the slots contains escape characters. This
differs from the behavior of WriteSlot() where the length passed to
it is always positive.

Due to this, the codegen generated code to check if the slot is NULL
or not never marks a slot NULL, when the slot contains the default
NULL pattern '\N' because the length for that slot is negative (due
to the escape character) and the IsNullString() and
IrGenericIsNullString() functions always returns false for these
cases, even though the slot is a NULL slot.

Things worked fine because of some incorrect code in CodegenWriteSlot
that marks invalid slots and NULL slots as NULL. Due to this code,
invalid slots were not marked as invalid and did not return an error.
Instead they were just sliently marked as NULL.

This patch makes sure that only positive lengths are passed to
CodegenWriteSlot() so that NULL checking is correct and it also
makes sure that invalid slots are not silently marked as NULL.

Testing: Re-enabled an older hdfs-scan-node-errors test. Formatted
it to fit new error message format after IMPALA-3859 and IMPALA-3895.

Change-Id: I858e427ad7c2b2da8c2bb657be06b7443655781f
---
M be/src/exec/hdfs-scanner.cc
M be/src/exec/text-converter.cc
M testdata/workloads/functional-query/queries/DataErrorsTest/hdfs-scan-node-errors.test
3 files changed, 68 insertions(+), 66 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/77/5377/1
-- 
To view, visit http://gerrit.cloudera.org:8080/5377
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I858e427ad7c2b2da8c2bb657be06b7443655781f
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Sailesh Mukil <sa...@cloudera.com>

[Impala-ASF-CR] IMPALA-4594: WriteSlot and CodegenWriteSlot handle escaped NULL slots differently

Posted by "Sailesh Mukil (Code Review)" <ge...@cloudera.org>.
Sailesh Mukil has uploaded a new patch set (#2).

Change subject: IMPALA-4594: WriteSlot and CodegenWriteSlot handle escaped NULL slots differently
......................................................................

IMPALA-4594: WriteSlot and CodegenWriteSlot handle escaped NULL slots differently

CodegenWriteSlot() receives negative length values for the lengths of
the slots passed to it if the slots contains escape characters. This
differs from the behavior of WriteSlot() where the length passed to
it is always positive.

Due to this, the codegen generated code to check if the slot is NULL
or not never marks a slot NULL, when the slot contains the default
NULL pattern '\N' because the length for that slot is negative (due
to the escape character) and the IsNullString() and
IrGenericIsNullString() functions always returns false for these
cases, even though the slot is a NULL slot.

Things worked fine because of some incorrect code in CodegenWriteSlot
that marks invalid slots and NULL slots as NULL. Due to this code,
invalid slots were not marked as invalid and did not return an error.
Instead they were just sliently marked as NULL.

This patch makes sure that only positive lengths are passed to
CodegenWriteSlot() so that NULL checking is correct and it also
makes sure that invalid slots are not silently marked as NULL.

Testing: Re-enabled an older hdfs-scan-node-errors test. Formatted
it to fit new error message format after IMPALA-3859 and IMPALA-3895.

Change-Id: I858e427ad7c2b2da8c2bb657be06b7443655781f
---
M be/src/exec/hdfs-scanner.cc
M be/src/exec/text-converter.cc
M testdata/workloads/functional-query/queries/DataErrorsTest/hdfs-scan-node-errors.test
3 files changed, 73 insertions(+), 66 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/77/5377/2
-- 
To view, visit http://gerrit.cloudera.org:8080/5377
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I858e427ad7c2b2da8c2bb657be06b7443655781f
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Sailesh Mukil <sa...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sa...@cloudera.com>

[Impala-ASF-CR] IMPALA-4594: WriteSlot and CodegenWriteSlot handle escaped NULL slots differently

Posted by "Sailesh Mukil (Code Review)" <ge...@cloudera.org>.
Sailesh Mukil has uploaded a new patch set (#3).

Change subject: IMPALA-4594: WriteSlot and CodegenWriteSlot handle escaped NULL slots differently
......................................................................

IMPALA-4594: WriteSlot and CodegenWriteSlot handle escaped NULL slots differently

CodegenWriteSlot() receives negative length values for the lengths of
the slots passed to it if the slots contain escape characters. (This
is currently only for non-string types, as we do not codegen string
types with escaped characters). This differs from the behavior of
WriteSlot() where the length passed to it is always positive.

Due to this, the codegen generated code to check if the slot is NULL
or not never marks a slot NULL, when the slot contains the default
NULL pattern '\N' because the length for that slot is negative (due
to the escape character) and the IsNullString() and
IrGenericIsNullString() functions always returns false for these
cases, even though the slot is a NULL slot.

Things worked fine because of some incorrect code in CodegenWriteSlot
that marks invalid slots and NULL slots as NULL. Due to this code,
invalid slots were not marked as invalid and did not return an error.
Instead they were just sliently marked as NULL.

This patch makes sure that only positive lengths are passed to
CodegenWriteSlot() so that NULL checking is correct and it also
makes sure that invalid slots are not silently marked as NULL.

Testing: Re-enabled an older hdfs-scan-node-errors test. Formatted
it to fit new error message format after IMPALA-3859 and IMPALA-3895.

Change-Id: I858e427ad7c2b2da8c2bb657be06b7443655781f
---
M be/src/exec/hdfs-scanner.cc
M be/src/exec/text-converter.cc
M testdata/workloads/functional-query/queries/DataErrorsTest/hdfs-scan-node-errors.test
3 files changed, 74 insertions(+), 68 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/77/5377/3
-- 
To view, visit http://gerrit.cloudera.org:8080/5377
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I858e427ad7c2b2da8c2bb657be06b7443655781f
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Sailesh Mukil <sa...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sa...@cloudera.com>

[Impala-ASF-CR] IMPALA-4594: WriteSlot and CodegenWriteSlot handle escaped NULL slots differently

Posted by "Sailesh Mukil (Code Review)" <ge...@cloudera.org>.
Sailesh Mukil has posted comments on this change.

Change subject: IMPALA-4594: WriteSlot and CodegenWriteSlot handle escaped NULL slots differently
......................................................................


Patch Set 3:

(5 comments)

http://gerrit.cloudera.org:8080/#/c/5377/2//COMMIT_MSG
Commit Message:

PS2, Line 10:  the slots contain escape characters. 
> May help to clarify that the problem shows up when the tuple only contains 
Done


http://gerrit.cloudera.org:8080/#/c/5377/2/be/src/exec/hdfs-scanner.cc
File be/src/exec/hdfs-scanner.cc:

PS2, Line 460: // Convert length to positive if it is negative.
> I wonder if this should be emitted for text scanner only. I believe we only
The non-codegen'd equivalent doesn't make that distinction, so it should be safe to do this here.
https://github.com/apache/incubator-impala/blob/master/be/src/exec/hdfs-scanner.cc#L235-L238


PS2, Line 468: positive_len = buil
> abs_len  or positive_len ?
Done


PS2, Line 469: ones_compliment_len, codegen->GetIntConstant(TYPE_INT, 1),
> nit: long line
Done


http://gerrit.cloudera.org:8080/#/c/5377/2/be/src/exec/text-converter.cc
File be/src/exec/text-converter.cc:

PS2, Line 171:     Value* null_len = builder.CreateICmpEQ(
             :         args[2], codegen->GetIntConst
> Remove.
Done


-- 
To view, visit http://gerrit.cloudera.org:8080/5377
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I858e427ad7c2b2da8c2bb657be06b7443655781f
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Sailesh Mukil <sa...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sa...@cloudera.com>
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4594: WriteSlot and CodegenWriteSlot handle escaped NULL slots differently

Posted by "Sailesh Mukil (Code Review)" <ge...@cloudera.org>.
Sailesh Mukil has posted comments on this change.

Change subject: IMPALA-4594: WriteSlot and CodegenWriteSlot handle escaped NULL slots differently
......................................................................


Patch Set 4: Code-Review+1

(7 comments)

Carry Michael's +1.

http://gerrit.cloudera.org:8080/#/c/5377/3//COMMIT_MSG
Commit Message:

PS3, Line 7: escaped NULL
> does this mean '\N', or can it be something else? if it means that, let's j
It mostly defaults to '\N' but Impala allows for the flexibility for it to be anything else too. And even if it was something else, it wouldn't have worked prior to this patch.


PS3, Line 9: receives
> this could use more explanation as to where these values come from and wher
Done


Line 20: The IrIsNullString() and IrGenericIsNullString() functions are
> i don't understand the explanation given the code you're changing in Codege
Done


PS3, Line 22: They are called by
> Was confused by this statement since you go on to say that invalid slots di
Yes, I've clarified that now.


PS3, Line 23: h means 
> what does this mean, given that you just said the code to check for NULL sl
The check for NULL slots didn't work where it was intended to work. It was marked as NULL accidentally elsewhere. I've clarified this para now.


http://gerrit.cloudera.org:8080/#/c/5377/3/be/src/exec/hdfs-scanner.cc
File be/src/exec/hdfs-scanner.cc:

Line 460:       // Convert length to positive if it is negative. Negative lengths are assigned to
> where do we explain what negative lengths mean? i think a reminder of what 
Done


PS3, Line 461: 
> I think this would be clearer to say: strings with escape charaters.
Done


-- 
To view, visit http://gerrit.cloudera.org:8080/5377
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I858e427ad7c2b2da8c2bb657be06b7443655781f
Gerrit-PatchSet: 4
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Sailesh Mukil <sa...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sa...@cloudera.com>
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4594: WriteSlot and CodegenWriteSlot handle escaped NULL slots differently

Posted by "Michael Ho (Code Review)" <ge...@cloudera.org>.
Michael Ho has posted comments on this change.

Change subject: IMPALA-4594: WriteSlot and CodegenWriteSlot handle escaped NULL slots differently
......................................................................


Patch Set 2:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/5377/2/be/src/exec/hdfs-scanner.cc
File be/src/exec/hdfs-scanner.cc:

PS2, Line 468: twos_compliment_len
abs_len  or positive_len ?


PS2, Line 469: ones_compliment_len, codegen->GetIntConstant(TYPE_INT, 1), "twos_compliment_len");
nit: long line


http://gerrit.cloudera.org:8080/#/c/5377/2/be/src/exec/text-converter.cc
File be/src/exec/text-converter.cc:

PS2, Line 171:     // The len can be less than 0 if the field contained an escape character which
             :     // is only valid for string cols.
Remove.


-- 
To view, visit http://gerrit.cloudera.org:8080/5377
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I858e427ad7c2b2da8c2bb657be06b7443655781f
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Sailesh Mukil <sa...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sa...@cloudera.com>
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4594: WriteSlot and CodegenWriteSlot handle escaped NULL slots differently

Posted by "Michael Ho (Code Review)" <ge...@cloudera.org>.
Michael Ho has posted comments on this change.

Change subject: IMPALA-4594: WriteSlot and CodegenWriteSlot handle escaped NULL slots differently
......................................................................


Patch Set 3: Code-Review+1

-- 
To view, visit http://gerrit.cloudera.org:8080/5377
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I858e427ad7c2b2da8c2bb657be06b7443655781f
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Sailesh Mukil <sa...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sa...@cloudera.com>
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-4594: WriteSlot and CodegenWriteSlot handle escaped NULL slots differently

Posted by "Sailesh Mukil (Code Review)" <ge...@cloudera.org>.
Hello Michael Ho,

I'd like you to reexamine a change.  Please visit

    http://gerrit.cloudera.org:8080/5377

to look at the new patch set (#4).

Change subject: IMPALA-4594: WriteSlot and CodegenWriteSlot handle escaped NULL slots differently
......................................................................

IMPALA-4594: WriteSlot and CodegenWriteSlot handle escaped NULL slots differently

CodegenWriteSlot() receives negative length values for the lengths of
the slots passed to it if the slots contain escape characters. (This
is currently only for non-string types, as we do not codegen string
types with escaped characters). The DelimitedTextParser is responsible
for identifying escape characters and assigning the negative lengths
appropriately. CodegenWriteCompleteTuple() passes this length to
CodegenWriteSlot() as it is. This differs from the behavior of
WriteSlot() where the length passed to it is always positive, as all
the callers of WriteSlot() make sure of that (including
WriteCompleteTuple()).

The IrIsNullString() and IrGenericIsNullString() functions are
responsibe for checking if the given data contains a NULL pattern.
They are called by CodegenWriteSlot(). A NULL pattern usually contains
an escaped character which means that the length of that slot will be
a negative length. However, the IrIsNullString() and
IrGenericIsNullString() that take the same length argument from
CodegenWriteSlot() always expect a positive length argument. So, no
slots were ever marked as NULL by these NULL-checking functions when
codegen was enabled.

NULL slots were still detected accidentally because of some incorrect
code in CodegenWriteSlot() that marked invalid slots and NULL slots as
NULL. Therefore, due to this code, even invalid slots were not marked
as invalid and did not return an error. Instead they were just
sliently marked as NULL.

This patch makes sure that only positive lengths are passed to
CodegenWriteSlot() so that NULL checking is correct and it also
makes sure that invalid slots are not silently marked as NULL.

Testing: Re-enabled an older hdfs-scan-node-errors test. Formatted
it to fit new error message format after IMPALA-3859 and IMPALA-3895.

Change-Id: I858e427ad7c2b2da8c2bb657be06b7443655781f
---
M be/src/exec/hdfs-scanner.cc
M be/src/exec/text-converter.cc
M testdata/workloads/functional-query/queries/DataErrorsTest/hdfs-scan-node-errors.test
3 files changed, 75 insertions(+), 68 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/77/5377/4
-- 
To view, visit http://gerrit.cloudera.org:8080/5377
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I858e427ad7c2b2da8c2bb657be06b7443655781f
Gerrit-PatchSet: 4
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Sailesh Mukil <sa...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sa...@cloudera.com>

[Impala-ASF-CR] IMPALA-4594: WriteSlot and CodegenWriteSlot handle escaped NULL slots differently

Posted by "Michael Ho (Code Review)" <ge...@cloudera.org>.
Michael Ho has posted comments on this change.

Change subject: IMPALA-4594: WriteSlot and CodegenWriteSlot handle escaped NULL slots differently
......................................................................


Patch Set 2:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/5377/2//COMMIT_MSG
Commit Message:

PS2, Line 10:  the slots contains escape characters.
May help to clarify that the problem shows up when the tuple only contains non-string slots. We don't use CodegenWriteSlot() if there is any string slots and the escape character is not null (https://github.com/apache/incubator-impala/blob/master/be/src/exec/hdfs-scanner.cc#L139).


http://gerrit.cloudera.org:8080/#/c/5377/2/be/src/exec/hdfs-scanner.cc
File be/src/exec/hdfs-scanner.cc:

PS2, Line 460: // Convert length to positive if it is negative.
I wonder if this should be emitted for text scanner only. I believe we only set length to negative if process_escapes is true in DelimitedTextParser::AddColumn(). Or do you think it's a good line of defense in case there are bugs in other part of the scanner ?


-- 
To view, visit http://gerrit.cloudera.org:8080/5377
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I858e427ad7c2b2da8c2bb657be06b7443655781f
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Sailesh Mukil <sa...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sa...@cloudera.com>
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4594: WriteSlot and CodegenWriteSlot handle escaped NULL slots differently

Posted by "Internal Jenkins (Code Review)" <ge...@cloudera.org>.
Internal Jenkins has posted comments on this change.

Change subject: IMPALA-4594: WriteSlot and CodegenWriteSlot handle escaped NULL slots differently
......................................................................


Patch Set 5: Verified+1

-- 
To view, visit http://gerrit.cloudera.org:8080/5377
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I858e427ad7c2b2da8c2bb657be06b7443655781f
Gerrit-PatchSet: 5
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Sailesh Mukil <sa...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Internal Jenkins
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sa...@cloudera.com>
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-4594: WriteSlot and CodegenWriteSlot handle escaped NULL slots differently

Posted by "Internal Jenkins (Code Review)" <ge...@cloudera.org>.
Internal Jenkins has submitted this change and it was merged.

Change subject: IMPALA-4594: WriteSlot and CodegenWriteSlot handle escaped NULL slots differently
......................................................................


IMPALA-4594: WriteSlot and CodegenWriteSlot handle escaped NULL slots differently

CodegenWriteSlot() receives negative length values for the lengths of
the slots passed to it if the slots contain escape characters. (This
is currently only for non-string types, as we do not codegen string
types with escaped characters). The DelimitedTextParser is responsible
for identifying escape characters and assigning the negative lengths
appropriately. CodegenWriteCompleteTuple() passes this length to
CodegenWriteSlot() as it is. This differs from the behavior of
WriteSlot() where the length passed to it is always positive, as all
the callers of WriteSlot() make sure of that (including
WriteCompleteTuple()).

The IrIsNullString() and IrGenericIsNullString() functions are
responsibe for checking if the given data contains a NULL pattern.
They are called by CodegenWriteSlot(). A NULL pattern usually contains
an escaped character which means that the length of that slot will be
a negative length. However, the IrIsNullString() and
IrGenericIsNullString() that take the same length argument from
CodegenWriteSlot() always expect a positive length argument. So, no
slots were ever marked as NULL by these NULL-checking functions when
codegen was enabled.

NULL slots were still detected accidentally because of some incorrect
code in CodegenWriteSlot() that marked invalid slots and NULL slots as
NULL. Therefore, due to this code, even invalid slots were not marked
as invalid and did not return an error. Instead they were just
sliently marked as NULL.

This patch makes sure that only positive lengths are passed to
CodegenWriteSlot() so that NULL checking is correct and it also
makes sure that invalid slots are not silently marked as NULL.

Testing: Re-enabled an older hdfs-scan-node-errors test. Formatted
it to fit new error message format after IMPALA-3859 and IMPALA-3895.

Change-Id: I858e427ad7c2b2da8c2bb657be06b7443655781f
Reviewed-on: http://gerrit.cloudera.org:8080/5377
Reviewed-by: Sailesh Mukil <sa...@cloudera.com>
Tested-by: Internal Jenkins
---
M be/src/exec/hdfs-scanner.cc
M be/src/exec/text-converter.cc
M testdata/workloads/functional-query/queries/DataErrorsTest/hdfs-scan-node-errors.test
3 files changed, 75 insertions(+), 68 deletions(-)

Approvals:
  Internal Jenkins: Verified
  Sailesh Mukil: Looks good to me, approved



-- 
To view, visit http://gerrit.cloudera.org:8080/5377
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I858e427ad7c2b2da8c2bb657be06b7443655781f
Gerrit-PatchSet: 6
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Sailesh Mukil <sa...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Internal Jenkins
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sa...@cloudera.com>

[Impala-ASF-CR] IMPALA-4594: WriteSlot and CodegenWriteSlot handle escaped NULL slots differently

Posted by "Dan Hecht (Code Review)" <ge...@cloudera.org>.
Dan Hecht has posted comments on this change.

Change subject: IMPALA-4594: WriteSlot and CodegenWriteSlot handle escaped NULL slots differently
......................................................................


Patch Set 3:

(7 comments)

http://gerrit.cloudera.org:8080/#/c/5377/3//COMMIT_MSG
Commit Message:

PS3, Line 7: escaped NULL
does this mean '\N', or can it be something else? if it means that, let's just say that (I wasn't sure what it meant at first).


PS3, Line 9: receives
this could use more explanation as to where these values come from and where this convention is defined (is it DelimitedTextParser?)


Line 20: cases, even though the slot is a NULL slot.
i don't understand the explanation given the code you're changing in CodegenWriteSlot used to mark anything <= 0 as NULL.

The code change itself looks fine though, it's just hard to follow this explanation.  Could you try to clarify it?


PS3, Line 22: Things worked fine
Was confused by this statement since you go on to say that invalid slots did not return an error.  Do you mean "NULL slots worked..." here?


PS3, Line 23: and NULL
what does this mean, given that you just said the code to check for NULL slots doesn't work?


http://gerrit.cloudera.org:8080/#/c/5377/3/be/src/exec/hdfs-scanner.cc
File be/src/exec/hdfs-scanner.cc:

Line 460:       // Convert length to positive if it is negative.
where do we explain what negative lengths mean? i think a reminder of what it means would be nice here. You can figure it out by reading the TODO but I think it would be good to say so directly.


PS3, Line 461: escaped strings
I think this would be clearer to say: strings with escape charaters.
Or: strings that need unescaping.


-- 
To view, visit http://gerrit.cloudera.org:8080/5377
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I858e427ad7c2b2da8c2bb657be06b7443655781f
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Sailesh Mukil <sa...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sa...@cloudera.com>
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4594: WriteSlot and CodegenWriteSlot handle escaped NULL slots differently

Posted by "Sailesh Mukil (Code Review)" <ge...@cloudera.org>.
Sailesh Mukil has posted comments on this change.

Change subject: IMPALA-4594: WriteSlot and CodegenWriteSlot handle escaped NULL slots differently
......................................................................


Patch Set 1:

> Uploaded patch set 1.

Ran a private build and it passed successfully:
http://sandbox.jenkins.cloudera.com/job/impala-private-build-and-test/4784/

-- 
To view, visit http://gerrit.cloudera.org:8080/5377
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I858e427ad7c2b2da8c2bb657be06b7443655781f
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Sailesh Mukil <sa...@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sa...@cloudera.com>
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-4594: WriteSlot and CodegenWriteSlot handle escaped NULL slots differently

Posted by "Sailesh Mukil (Code Review)" <ge...@cloudera.org>.
Sailesh Mukil has posted comments on this change.

Change subject: IMPALA-4594: WriteSlot and CodegenWriteSlot handle escaped NULL slots differently
......................................................................


Patch Set 2:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/5377/1/be/src/exec/hdfs-scanner.cc
File be/src/exec/hdfs-scanner.cc:

PS1, Line 463:       // 'need_escape' bool to CodegenWriteSlot(), since we are making the length positive
             :       // here.
> May be  (~x + 1) to be safe ? LLVM probably figures that out during optimiz
Done


-- 
To view, visit http://gerrit.cloudera.org:8080/5377
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I858e427ad7c2b2da8c2bb657be06b7443655781f
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Sailesh Mukil <sa...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sa...@cloudera.com>
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4594: WriteSlot and CodegenWriteSlot handle escaped NULL slots differently

Posted by "Michael Ho (Code Review)" <ge...@cloudera.org>.
Michael Ho has posted comments on this change.

Change subject: IMPALA-4594: WriteSlot and CodegenWriteSlot handle escaped NULL slots differently
......................................................................


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/5377/1/be/src/exec/hdfs-scanner.cc
File be/src/exec/hdfs-scanner.cc:

PS1, Line 463:       Value* inverse_sign_len = builder.CreateMul(len,
             :           codegen->GetIntConstant(TYPE_INT, -1), "inverse_sign_len");
May be  (~x + 1) to be safe ? LLVM probably figures that out during optimization (hopefully).


-- 
To view, visit http://gerrit.cloudera.org:8080/5377
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I858e427ad7c2b2da8c2bb657be06b7443655781f
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Sailesh Mukil <sa...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sa...@cloudera.com>
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4594: WriteSlot and CodegenWriteSlot handle escaped NULL slots differently

Posted by "Dan Hecht (Code Review)" <ge...@cloudera.org>.
Dan Hecht has posted comments on this change.

Change subject: IMPALA-4594: WriteSlot and CodegenWriteSlot handle escaped NULL slots differently
......................................................................


Patch Set 4: Code-Review+2

-- 
To view, visit http://gerrit.cloudera.org:8080/5377
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I858e427ad7c2b2da8c2bb657be06b7443655781f
Gerrit-PatchSet: 4
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Sailesh Mukil <sa...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sa...@cloudera.com>
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-4594: WriteSlot and CodegenWriteSlot handle escaped NULL slots differently

Posted by "Sailesh Mukil (Code Review)" <ge...@cloudera.org>.
Sailesh Mukil has posted comments on this change.

Change subject: IMPALA-4594: WriteSlot and CodegenWriteSlot handle escaped NULL slots differently
......................................................................


Patch Set 5: Code-Review+2

Rebase, carry +2.

-- 
To view, visit http://gerrit.cloudera.org:8080/5377
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I858e427ad7c2b2da8c2bb657be06b7443655781f
Gerrit-PatchSet: 5
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Sailesh Mukil <sa...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sa...@cloudera.com>
Gerrit-HasComments: No