[GitHub] [arrow] denisgursky opened a new issue, #14256: Is there a file size limit for the JS SDK?

[GitBox <gi...@apache.org>]

denisgursky opened a new issue, #14256:
URL: https://github.com/apache/arrow/issues/14256

   ```javascript
   const fs = require("fs");
   const { tableFromIPC, RecordBatchReader } = require("apache-arrow");
   
   const filePath = "./arraydata.arrow";
   
   const stream = fs.createReadStream(filePath);
   const reader = RecordBatchReader.from(stream);
   
   (async function () {
     const table = await tableFromIPC(reader);
   
     console.log("numRows", table.numRows);
     console.log("first row", table.get(0).toArray());
   })();
   
   ```
   
   The code above prints:
   ```
   numRows 140000000
   first row [ undefined, undefined ]
   ```
   
   `numRows` is correct, but the values are coming out as `undefined`.
   
   `./arraydata.arrow` is generated by the following python code.
   
   ```python
   import pyarrow as pa
   
   nums1 = [42]
   nums2 = [42.42]
   mil = 1000000
   
   for n in range(1, 140 * mil):
     nums1.append(n)
     nums2.append(1 / n)
   
   arr1 = pa.array(nums1)
   arr2 = pa.array(nums2)
   
   schema = pa.schema([
     pa.field('nums1', arr1.type),
     pa.field('nums2', arr2.type),
   ])
   
   with pa.OSFile('arraydata.arrow', 'wb') as sink:
     with pa.ipc.new_file(sink, schema=schema) as writer:
       batch = pa.record_batch([arr1, arr2], schema=schema)
       writer.write(batch)
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] ljluestc commented on issue #14256: Is there a file size limit for the JS SDK?

["ljluestc (via GitHub)" <gi...@apache.org>]

ljluestc commented on issue #14256:
URL: https://github.com/apache/arrow/issues/14256#issuecomment-1661615437

   ```
   const fs = require("fs");
   const { tableFromChunks, RecordBatchStreamReader } = require("apache-arrow");
   
   const filePath = "./arraydata.arrow";
   const chunkSize = 100000; // Set an appropriate chunk size based on available memory
   
   const stream = fs.createReadStream(filePath);
   const reader = new RecordBatchStreamReader(stream);
   
   (async function () {
     let chunks = [];
     for await (const chunk of tableFromChunks(reader, chunkSize)) {
       chunks.push(chunk);
     }
   
     // Concatenate the chunks to form the complete table
     const table = chunks.reduce((acc, chunk) => acc.concat(chunk));
   
     console.log("numRows", table.numRows);
     console.log("first row", table.get(0).toArray());
   })();
   
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org