[jira] [Commented] (TRAFODION-2615) TMUDF returns wrong results with small numeric inputs

["Hans Zeller (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TRAFODION-2615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16006906#comment-16006906 ] 

Hans Zeller commented on TRAFODION-2615:
----------------------------------------

The problem is that the new TINYINT type sometimes is stored in 1 byte, other times in 2 bytes. The TMUDF code makes the assumption that we can compute the storage size from the precision. I need to fix this in the binder and make sure the storage size matches the storage size that the TMUDF code will compute.

> TMUDF returns wrong results with small numeric inputs
> -----------------------------------------------------
>
>                 Key: TRAFODION-2615
>                 URL: https://issues.apache.org/jira/browse/TRAFODION-2615
>             Project: Apache Trafodion
>          Issue Type: Bug
>          Components: sql-cmp
>    Affects Versions: 2.1-incubating
>            Reporter: Hans Zeller
>            Assignee: Hans Zeller
>             Fix For: 2.2-incubating
>
>         Attachments: EchoUdf.cpp
>
>
> I got an incorrect result with this query:
> {noformat}
> >>select * from udf(echo(table(select * from (values (1), (22), (3), (4)) t(x))));
> X     
> ------
>      1
>   -234
>      3
>    772
> --- 4 row(s) selected.
> >>
> {noformat}
> Note that -234 if FF16 (and 16 is hex for 22) and that 772 is 304, so it's a junk byte followed by the true values 22 and 4 that is returned.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)