You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/12/01 02:58:27 UTC

[GitHub] [beam] agrawal-siddharth opened a new pull request, #24447: Add integer to NUMERIC and BIGNUMERIC conversion support

agrawal-siddharth opened a new pull request, #24447:
URL: https://github.com/apache/beam/pull/24447

   Support integer values to be provided as input for columns of type NUMERIC and BIGNUMERIC.
   
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
    - [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
    - [ ] Update `CHANGES.md` with noteworthy changes.
    - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
   
   To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   [![Build python source distribution and wheels](https://github.com/apache/beam/workflows/Build%20python%20source%20distribution%20and%20wheels/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
   [![Python tests](https://github.com/apache/beam/workflows/Python%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Java tests](https://github.com/apache/beam/workflows/Java%20Tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Go tests](https://github.com/apache/beam/workflows/Go%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] Abacn commented on a diff in pull request #24447: Add integer to NUMERIC and BIGNUMERIC conversion support

Posted by GitBox <gi...@apache.org>.
Abacn commented on code in PR #24447:
URL: https://github.com/apache/beam/pull/24447#discussion_r1037409787


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowToStorageApiProto.java:
##########
@@ -681,7 +681,7 @@ private static void fieldDescriptorFromTableField(
               new BigDecimal((String) value));
         } else if (value instanceof BigDecimal) {
           return BigDecimalByteStringEncoder.encodeToBigNumericByteString(((BigDecimal) value));
-        } else if (value instanceof Double || value instanceof Float) {

Review Comment:
   Sorry, I meant the conversion of Long to double value `BigDecimal.valueOf(((Number) value).doubleValue())` could lose precision here. Long has max 2^63-1 up to 18 digit of significance, however Double has 16 digit of significance. A safe way is something like this
   ```
   else if (value instanceof Double || value instanceof Float) {
             return BigDecimalByteStringEncoder.encodeToNumericByteString(
                 BigDecimal.valueOf(((Number) value).doubleValue()));
           }
   ekse if (value instanceof java.lang.Short || value instanceof java.lang.Integer || value instanceof java.lang.Long) {
             return BigDecimalByteStringEncoder.encodeToNumericByteString(
                 BigDecimal.valueOf(((Number) value).longValue()));
           }
   ```
   We may also want to add test to check Long value of 18 digits do not lose precision. This use case is possible as raw timestamp can be backed by a Long.
   
   my original comment about the unsupported Number refers to some known direct subclasses of Number here: https://docs.oracle.com/javase/8/docs/api/java/lang/Number.html



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] Abacn merged pull request #24447: Add integer to NUMERIC and BIGNUMERIC conversion support

Posted by GitBox <gi...@apache.org>.
Abacn merged PR #24447:
URL: https://github.com/apache/beam/pull/24447


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] agrawal-siddharth commented on a diff in pull request #24447: Add integer to NUMERIC and BIGNUMERIC conversion support

Posted by GitBox <gi...@apache.org>.
agrawal-siddharth commented on code in PR #24447:
URL: https://github.com/apache/beam/pull/24447#discussion_r1037452677


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowToStorageApiProto.java:
##########
@@ -681,7 +681,7 @@ private static void fieldDescriptorFromTableField(
               new BigDecimal((String) value));
         } else if (value instanceof BigDecimal) {
           return BigDecimalByteStringEncoder.encodeToBigNumericByteString(((BigDecimal) value));
-        } else if (value instanceof Double || value instanceof Float) {

Review Comment:
   I broke out handling of short/integer/long as you suggested. I have also revised the test case to include a value with 18 digits which demonstrates it could not be handled by a double.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] Abacn commented on a diff in pull request #24447: Add integer to NUMERIC and BIGNUMERIC conversion support

Posted by GitBox <gi...@apache.org>.
Abacn commented on code in PR #24447:
URL: https://github.com/apache/beam/pull/24447#discussion_r1037285983


##########
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowToStorageApiProtoTest.java:
##########
@@ -576,8 +576,8 @@ public void testNestedFromTableSchema() {
                   new TableCell().setV("00:52:07.123456"),
                   new TableCell().setV("2019-08-16T00:52:07.123456"),
                   new TableCell().setV("2019-08-16"),
-                  new TableCell().setV("23.4"),

Review Comment:
   I see, I think we should add a test case instead of replace one. Now the conversion of double to decimal becomes untested



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] agrawal-siddharth commented on pull request #24447: Add integer to NUMERIC and BIGNUMERIC conversion support

Posted by GitBox <gi...@apache.org>.
agrawal-siddharth commented on PR #24447:
URL: https://github.com/apache/beam/pull/24447#issuecomment-1333944538

   For more context, this fix is similar to that made in https://github.com/apache/beam/pull/17779. It extends support for integer input values to table columns of type NUMERIC and BIGNUMERIC. The fix was motivated by a bug report from a customer.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] Abacn commented on a diff in pull request #24447: Add integer to NUMERIC and BIGNUMERIC conversion support

Posted by GitBox <gi...@apache.org>.
Abacn commented on code in PR #24447:
URL: https://github.com/apache/beam/pull/24447#discussion_r1037290890


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowToStorageApiProto.java:
##########
@@ -681,7 +681,7 @@ private static void fieldDescriptorFromTableField(
               new BigDecimal((String) value));
         } else if (value instanceof BigDecimal) {
           return BigDecimalByteStringEncoder.encodeToBigNumericByteString(((BigDecimal) value));
-        } else if (value instanceof Double || value instanceof Float) {

Review Comment:
   Thanks. Understand. However we should be careful to convert all Number (other than BigDecimal) to Double here as the it can lose precision. Still think we should add a condition `else if (value instanceof Integer or value instanceof Long) {` 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] Abacn commented on a diff in pull request #24447: Add integer to NUMERIC and BIGNUMERIC conversion support

Posted by GitBox <gi...@apache.org>.
Abacn commented on code in PR #24447:
URL: https://github.com/apache/beam/pull/24447#discussion_r1037411019


##########
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowToStorageApiProtoTest.java:
##########
@@ -576,8 +576,8 @@ public void testNestedFromTableSchema() {
                   new TableCell().setV("00:52:07.123456"),
                   new TableCell().setV("2019-08-16T00:52:07.123456"),
                   new TableCell().setV("2019-08-16"),
-                  new TableCell().setV("23.4"),

Review Comment:
   Thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] Abacn commented on a diff in pull request #24447: Add integer to NUMERIC and BIGNUMERIC conversion support

Posted by GitBox <gi...@apache.org>.
Abacn commented on code in PR #24447:
URL: https://github.com/apache/beam/pull/24447#discussion_r1037196896


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowToStorageApiProto.java:
##########
@@ -681,7 +681,7 @@ private static void fieldDescriptorFromTableField(
               new BigDecimal((String) value));
         } else if (value instanceof BigDecimal) {
           return BigDecimalByteStringEncoder.encodeToBigNumericByteString(((BigDecimal) value));
-        } else if (value instanceof Double || value instanceof Float) {

Review Comment:
   We still can have unsupported Number instance, is that right?



##########
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowToStorageApiProtoTest.java:
##########
@@ -576,8 +576,8 @@ public void testNestedFromTableSchema() {
                   new TableCell().setV("00:52:07.123456"),
                   new TableCell().setV("2019-08-16T00:52:07.123456"),
                   new TableCell().setV("2019-08-16"),
-                  new TableCell().setV("23.4"),

Review Comment:
   why does this get changed, any breaking change?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] agrawal-siddharth commented on a diff in pull request #24447: Add integer to NUMERIC and BIGNUMERIC conversion support

Posted by GitBox <gi...@apache.org>.
agrawal-siddharth commented on code in PR #24447:
URL: https://github.com/apache/beam/pull/24447#discussion_r1037393102


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowToStorageApiProto.java:
##########
@@ -681,7 +681,7 @@ private static void fieldDescriptorFromTableField(
               new BigDecimal((String) value));
         } else if (value instanceof BigDecimal) {
           return BigDecimalByteStringEncoder.encodeToBigNumericByteString(((BigDecimal) value));
-        } else if (value instanceof Double || value instanceof Float) {

Review Comment:
   I have reduced the scope of the change by simply adding Integer and Long alonside Double and Float.



##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowToStorageApiProto.java:
##########
@@ -681,7 +681,7 @@ private static void fieldDescriptorFromTableField(
               new BigDecimal((String) value));
         } else if (value instanceof BigDecimal) {
           return BigDecimalByteStringEncoder.encodeToBigNumericByteString(((BigDecimal) value));
-        } else if (value instanceof Double || value instanceof Float) {

Review Comment:
   I have reduced the scope of the change by simply adding Integer and Long alongside Double and Float.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] agrawal-siddharth commented on a diff in pull request #24447: Add integer to NUMERIC and BIGNUMERIC conversion support

Posted by GitBox <gi...@apache.org>.
agrawal-siddharth commented on code in PR #24447:
URL: https://github.com/apache/beam/pull/24447#discussion_r1037393852


##########
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowToStorageApiProtoTest.java:
##########
@@ -576,8 +576,8 @@ public void testNestedFromTableSchema() {
                   new TableCell().setV("00:52:07.123456"),
                   new TableCell().setV("2019-08-16T00:52:07.123456"),
                   new TableCell().setV("2019-08-16"),
-                  new TableCell().setV("23.4"),

Review Comment:
   I have restored the original test case and added new ones to exercise integers.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] agrawal-siddharth commented on a diff in pull request #24447: Add integer to NUMERIC and BIGNUMERIC conversion support

Posted by GitBox <gi...@apache.org>.
agrawal-siddharth commented on code in PR #24447:
URL: https://github.com/apache/beam/pull/24447#discussion_r1037248700


##########
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowToStorageApiProtoTest.java:
##########
@@ -576,8 +576,8 @@ public void testNestedFromTableSchema() {
                   new TableCell().setV("00:52:07.123456"),
                   new TableCell().setV("2019-08-16T00:52:07.123456"),
                   new TableCell().setV("2019-08-16"),
-                  new TableCell().setV("23.4"),

Review Comment:
   I simply updated the test to pass in an integer rather than string. This helps to demonstrate the problem, and exercises the fix. This is not a breaking change, both strings and numbers are still supported as input.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] agrawal-siddharth commented on a diff in pull request #24447: Add integer to NUMERIC and BIGNUMERIC conversion support

Posted by GitBox <gi...@apache.org>.
agrawal-siddharth commented on code in PR #24447:
URL: https://github.com/apache/beam/pull/24447#discussion_r1037248700


##########
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowToStorageApiProtoTest.java:
##########
@@ -576,8 +576,8 @@ public void testNestedFromTableSchema() {
                   new TableCell().setV("00:52:07.123456"),
                   new TableCell().setV("2019-08-16T00:52:07.123456"),
                   new TableCell().setV("2019-08-16"),
-                  new TableCell().setV("23.4"),

Review Comment:
   I simply updated the test to pass in an integer rather than string. This helps to demonstrate the problem, and exercises the fix.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] Abacn commented on pull request #24447: Add integer to NUMERIC and BIGNUMERIC conversion support

Posted by GitBox <gi...@apache.org>.
Abacn commented on PR #24447:
URL: https://github.com/apache/beam/pull/24447#issuecomment-1334436617

   Related to (BEAM-14529) Beam BQIO not accepting integer values for BQ table fields of type ~Float64~Numeric/BigNumeric.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] github-actions[bot] commented on pull request #24447: Add integer to NUMERIC and BIGNUMERIC conversion support

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #24447:
URL: https://github.com/apache/beam/pull/24447#issuecomment-1333110185

   Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`:
   
   R: @kennknowles for label java.
   R: @Abacn for label io.
   
   Available commands:
   - `stop reviewer notifications` - opt out of the automated review tooling
   - `remind me after tests pass` - tag the comment author after tests pass
   - `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
   
   The PR bot will only process comments in the main thread (not review comments).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] agrawal-siddharth commented on a diff in pull request #24447: Add integer to NUMERIC and BIGNUMERIC conversion support

Posted by GitBox <gi...@apache.org>.
agrawal-siddharth commented on code in PR #24447:
URL: https://github.com/apache/beam/pull/24447#discussion_r1037252390


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowToStorageApiProto.java:
##########
@@ -681,7 +681,7 @@ private static void fieldDescriptorFromTableField(
               new BigDecimal((String) value));
         } else if (value instanceof BigDecimal) {
           return BigDecimalByteStringEncoder.encodeToBigNumericByteString(((BigDecimal) value));
-        } else if (value instanceof Double || value instanceof Float) {

Review Comment:
   I'm not sure what you mean. We accept all Number formats now, rather than only Double or Float. This enables us to accept integer values as input.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] Abacn commented on a diff in pull request #24447: Add integer to NUMERIC and BIGNUMERIC conversion support

Posted by GitBox <gi...@apache.org>.
Abacn commented on code in PR #24447:
URL: https://github.com/apache/beam/pull/24447#discussion_r1037409787


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowToStorageApiProto.java:
##########
@@ -681,7 +681,7 @@ private static void fieldDescriptorFromTableField(
               new BigDecimal((String) value));
         } else if (value instanceof BigDecimal) {
           return BigDecimalByteStringEncoder.encodeToBigNumericByteString(((BigDecimal) value));
-        } else if (value instanceof Double || value instanceof Float) {

Review Comment:
   Sorry, I meant the conversion of Long to double value `BigDecimal.valueOf(((Number) value).doubleValue())` could lose precision here. Long has max 2^63-1 up to 18 digit of significance, however Double has 16 digit of significance. A safe way is something like this
   ```
   else if (value instanceof Double || value instanceof Float) {
             return BigDecimalByteStringEncoder.encodeToNumericByteString(
                 BigDecimal.valueOf(((Number) value).doubleValue()));
           }
   ekse if (value instanceof java.lang.Short || value instanceof java.lang.Integer || value instanceof java.lang.Long) {
             return BigDecimalByteStringEncoder.encodeToNumericByteString(
                 BigDecimal.valueOf(((Number) value).longValue()));
           }
   ```
   
   my original comment about the unsupported Number refers to some known direct subclasses of Number here: https://docs.oracle.com/javase/8/docs/api/java/lang/Number.html



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org