[jira] [Created] (AVRO-3902) 64bit long support for JavaScript

["Philippe Ozil (Jira)" <ji...@apache.org>]

Philippe Ozil created AVRO-3902:
-----------------------------------

             Summary: 64bit long support for JavaScript
                 Key: AVRO-3902
                 URL: https://issues.apache.org/jira/browse/AVRO-3902
             Project: Apache Avro
          Issue Type: Bug
          Components: javascript, js
    Affects Versions: 1.11.3
            Reporter: Philippe Ozil


The latest version of avro-js cannot deserialize large numeric values that fit in a 64 bit Long. It's limited to the values that fit in the [maximum safe integer in JavaScript|https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Number/MAX_SAFE_INTEGER] (2^53 – 1).

Attempting to read larger values such as 1698100138468892700 triggers a deserialization error:

?? Error: invalid "long": 1698100138468892700??

 

The avro-js documentation provides a solution for this: [specifying a custom Long type|https://github.com/apache/avro/blob/main/lang/js/doc/Advanced-usage.md#custom-long-types] however, this documentation page is not easy to find and can easily be missed.

I understand that you may not want to change the default behavior of the library for backward compatibility reasons but could you at least mention this important limitation in the main documentation page?

 

Also, there's now a native JS solution for supporting large values in long type: [BigInt|https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/BigInt]. I believe that this should be the preferred approach since it doesn't introduce extra dependencies.

This is the native long-safe implementation (inspired from [avsc|https://github.com/mtth/avsc/wiki/Advanced-usage#custom-long-types]):
{code:js}
const longType = avro.types.LongType.using({
  fromBuffer: (buf) => buf.readBigInt64LE(),
  toBuffer: (n) => {
    const buf = Buffer.alloc(8);
    buf.writeBigInt64LE(n);
    return buf;
  },
  fromJSON: BigInt,
  toJSON: Number,
  isValid: (n) => typeof n === 'bigint',
  compare: (n1, n2) => { return n1 === n2 ? 0 : (n1 < n2 ? -1 : 1); }
});{code}
 

This is an alternate implementation that I wrote. It uses regular Number whenever possible and falls back to BigInt when the value exceeds the max safe value for Number. This could be a good built-in alternative for avro-js in which we avoid throwing a deserialization error.
{code:js}
const CUSTOM_LONG_AVRO_TYPE = avro.types.LongType.using({
    fromBuffer: (buf) => {
        const big = buf.readBigInt64LE();
        if (big > Number.MAX_SAFE_INTEGER) {
            return big;
        }
        return Number(BigInt.asIntN(64, big));
    },
    toBuffer: (n) => {
        const buf = Buffer.alloc(8);
        if (n instanceof BigInt) {
            buf.writeBigInt64LE(n);
        } else {
            buf.writeBigInt64LE(BigInt(n));
        }
        return buf;
    },
    fromJSON: BigInt,
    toJSON: Number,
    isValid: (n) => {
        const type = typeof n;
        return type === 'bigint' || type === 'number';
    },
    compare: (n1, n2) => {
        return n1 === n2 ? 0 : n1 < n2 ? -1 : 1;
    }
});{code}
 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)