You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "zeroshade (via GitHub)" <gi...@apache.org> on 2023/10/26 16:51:38 UTC

[PR] GH-38477: [Go] Fixing decimal 128 rounding issue [arrow]

zeroshade opened a new pull request, #38478:
URL: https://github.com/apache/arrow/pull/38478

   <!--
   Thanks for opening a pull request!
   If this is your first pull request you can find detailed information on how 
   to contribute here:
     * [New Contributor's Guide](https://arrow.apache.org/docs/dev/developers/guide/step_by_step/pr_lifecycle.html#reviews-and-merge-of-the-pull-request)
     * [Contributing Overview](https://arrow.apache.org/docs/dev/developers/overview.html)
   
   
   If this is not a [minor PR](https://github.com/apache/arrow/blob/main/CONTRIBUTING.md#Minor-Fixes). Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
   
   Opening GitHub issues ahead of time contributes to the [Openness](http://theapacheway.com/open/#:~:text=Openness%20allows%20new%20users%20the,must%20happen%20in%20the%20open.) of the Apache Arrow project.
   
   Then could you also rename the pull request title in the following format?
   
       GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
   
   or
   
       MINOR: [${COMPONENT}] ${SUMMARY}
   
   In the case of PARQUET issues on JIRA the title also supports:
   
       PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
   
   -->
   
   ### Rationale for this change
   Fixing an off-by-one rounding issue with decimal128 by ensuring proper precision handling.
   
   <!--
    Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
    Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.  
   -->
   
   
   ### Are these changes tested?
   The test case which reproduced the rounding issue has been added as a unit test.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38477: [Go] Fixing decimal 128 rounding issue [arrow]

Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on PR #38478:
URL: https://github.com/apache/arrow/pull/38478#issuecomment-1783024268

   I'll take a look, though technically `128` for the precision would be incorrect because a 128 bit value can only have 127 bits of precision + 1 sign bit. That said, it may not correlate exactly due to the differences between the `math/big` package vs how we handle decimal128


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38477: [Go] Fixing decimal 128 rounding issue [arrow]

Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on PR #38478:
URL: https://github.com/apache/arrow/pull/38478#issuecomment-1781491446

   CC @kuerbiskarton 
   
   I've used your reproducer as a unit test here to ensure it fixes the issue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38477: [Go] Fixing decimal 128 rounding issue [arrow]

Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade merged PR #38478:
URL: https://github.com/apache/arrow/pull/38478


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38477: [Go] Fixing decimal 128 rounding issue [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38478:
URL: https://github.com/apache/arrow/pull/38478#issuecomment-1781491051

   :warning: GitHub issue #38477 **has been automatically assigned in GitHub** to PR creator.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38477: [Go] Fixing decimal 128 rounding issue [arrow]

Posted by "sstemmle (via GitHub)" <gi...@apache.org>.
sstemmle commented on PR #38478:
URL: https://github.com/apache/arrow/pull/38478#issuecomment-1782505446

   Thanks for working on this @zeroshade. The current version is still failing for e.g. `9323406071781562130.6457232358109488923`. The following piece of code can be used to check a range of different input values.
   ```
   func TestFromStringDecimal128(t *testing.T) {
   	rq := require.New(t)
   
   	const maxprec = 38
   
   	for {
   		scale := int(rand.Int31()) % (maxprec + 1)
   
   		s := ""
   		if maxprec-scale > 0 {
   			for i := 0; i < maxprec-scale; i++ {
   				s += randDigit()
   			}
   		} else {
   			s += "0"
   		}
   
   		if scale > 0 {
   			s += "."
   
   			for i := 0; i < scale; i++ {
   				s += randDigit()
   			}
   		}
   
   		num, err := decimal128.FromString(s, maxprec, int32(scale))
   		rq.NoErrorf(err, "s=%s, scale=%d", s, scale)
   
   		actualCoeff := num.BigInt()
   
   		expectedCoeff, _ := (&big.Int{}).SetString(strings.Replace(s, ".", "", -1), 10)
   		rq.Equal(expectedCoeff.Bytes(), actualCoeff.Bytes(), s)
   	}
   }
   
   func randDigit() string {
   	return strconv.Itoa(int(rand.Int31() % 10))
   }
   ```
   
   I think one needs to adjust the precision provided to `big.ParseFloat()` to 128. See https://stackoverflow.com/questions/30688422/is-the-most-significant-decimal-digits-precision-that-can-be-converted-to-binary .
   
   The significant (in the above sense) decimal digits giving a floating point precision are given by ⌊(N-1) log10(2)⌋.
   To get 38 digits, 127 is not enough, but 128 is needed. For 76 digits 255 is enough.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38477: [Go] Fixing decimal 128 rounding issue [arrow]

Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #38478:
URL: https://github.com/apache/arrow/pull/38478#issuecomment-1811051705

   After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit bb7ffaf0bd0661baba872c3fe1500369f26241bd.
   
   There were no benchmark performance regressions. 🎉
   
   The [full Conbench report](https://github.com/apache/arrow/runs/18679212078) has more details.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38477: [Go] Fixing decimal 128 rounding issue [arrow]

Posted by "kuerbiskarton (via GitHub)" <gi...@apache.org>.
kuerbiskarton commented on PR #38478:
URL: https://github.com/apache/arrow/pull/38478#issuecomment-1783008787

   @zeroshade we're using https://github.com/kuerbiskarton/arrowtests/blob/master/strconv/strconv_test.go to verify the correctness of the string functions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org