[GitHub] [cordova-plugin-file] LightMind commented on issue #364: Large file save freeze app for seconds

[GitBox <gi...@apache.org>]

LightMind commented on issue #364:
URL: https://github.com/apache/cordova-plugin-file/issues/364#issuecomment-773977157


   We were running into some of the same issues. We try to download files via XMLHttpRequest, and write the contents via a fileWriter. Writing files larger than a few megabytes would consistently crash the whole application. We switched to writing the files in chunks, which alleviated the problem, but I was still confused why writing files less than 50MiB would result in crashes or white-screens.
   
   I profiled with chromes debugging tools, and found that the garbage collector is running like crazy, and the function uses a lot of memory. You can see here, that while invoking `.writeFile` that the memory usage goes from 11MiB to about 40MiB, even though we are writing 1MiB chunks of binary data.
   ![Screenshot 2021-02-03 at 13 58 41](https://user-images.githubusercontent.com/1789905/107011007-5d6bca80-6797-11eb-879a-42ebc7c0ba2d.png)
   
   As already mentioned by @breautek, this part in cordova is really slow: 
   `args[i] = base64.fromArrayBuffer(args[i]);`
   
   I wanted to understand what is happening here:
   Taking a look at the base64 module in cordova-common:
   
   ```
   function uint8ToBase64 (rawData) {
       var numBytes = rawData.byteLength;
       var output = '';
       var segment;
       var table = b64_12bitTable();
       for (var i = 0; i < numBytes - 2; i += 3) {
           segment = (rawData[i] << 16) + (rawData[i + 1] << 8) + rawData[i + 2];
           output += table[segment >> 12];
           output += table[segment & 0xfff];
       }
       if (numBytes - i === 2) {
           segment = (rawData[i] << 16) + (rawData[i + 1] << 8);
           output += table[segment >> 12];
           output += b64_6bit[(segment & 0xfff) >> 6];
           output += '=';
       } else if (numBytes - i === 1) {
           segment = (rawData[i] << 16);
           output += table[segment >> 12];
           output += '==';
       }
       return output;
   }
   ```
   
   Ouch. JavaScript has immutable strings. Using `+=` to concat the `output` creates new strings in memory all the time. This means if we need to convert `n` bytes to base64, we create at least `2 * n/3` new strings, each bigger than the last. This ends up costing `O(n^2)` bytes of memory.   ( "AA", "AABB", "AABBCC" ... all the way to the full string )
   
   Note that if you want to write 1MiB, you would have to allocate several gigabytes of memory!
   
   This explains the memory usage, and why writing binary files is slow. The garbage collector needs to clean up millions of strings, and if it can't keep up, we run out of memory.
   
   This not only affects writing files, but ANY cordova plugin that wants to pass binary data from JS to the native layer.
   
   The conversion via `FileReader.readAsDataURL` that @digaus used is infinitely better than the cordova-common algorithm.
   
   
   
   I thought it would be reasonable to support `FileWriter.write(buffer: ArrayBuffer)`, and let it do the chunking and conversion to base64, to bypass the cordova algorithm. Then users of this plugin would not need to implement custom chunking code.
   
   So I started to change the `FileWriter.js` code. The files are written to disk, but I must have made a mistake somewhere, because the files seem corrupted ( I have removed the prefix from the readAsDataURL result ).
   
   I am also not sure why @digaus solution works. When I pass a string to FileWriter.write(), it should write the strings as-is to disk, instead of interpreting it as binary data encoded in base64?
   In `www/FileWriter.js`, we need to pass `isBinary = true` here:
   ```
   }, 'File', 'write', [this.localURL, data, this.position, isBinary]);
   ```
   In order to get `LocalFileSystem.java writeToFileAtURL()`  to decode it to binary data.
   
   Does `FileSystem.writeFile` behave differently? 
   
   
   I will spend some more time on this. I would really like to solve this issue and make a PR, once I get this to work.
   
   
   
   
   
   
   
   
   
   
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@cordova.apache.org
For additional commands, e-mail: issues-help@cordova.apache.org