You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2019/10/27 19:33:00 UTC
[jira] [Commented] (AVRO-2562) Expose the JavaScript BlockEncoder's
piped stream
[ https://issues.apache.org/jira/browse/AVRO-2562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16960663#comment-16960663 ]
ASF subversion and git services commented on AVRO-2562:
-------------------------------------------------------
Commit a684f3e0b1859ba05c17b18407c2e9905701079a in avro's branch refs/heads/master from Kengo Seki
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=a684f3e ]
AVRO-2562: Expose the JavaScript BlockEncoder's piped stream (#650)
> Expose the JavaScript BlockEncoder's piped stream
> -------------------------------------------------
>
> Key: AVRO-2562
> URL: https://issues.apache.org/jira/browse/AVRO-2562
> Project: Apache Avro
> Issue Type: New Feature
> Components: javascript
> Reporter: Kengo Seki
> Assignee: Kengo Seki
> Priority: Major
>
> The "createFileEncoder" test for the JavaScript implementation is flaky. It sometimes fails in docker mode, as follows:
> {code}
> ~/repos/avro$ git checkout master
> ~/repos/avro$ ./build.sh docker
> sekikn@13d38772b8e6:~/avro$ ./build.sh veryclean
> sekikn@13d38772b8e6:~/avro$ cd lang/js
> sekikn@13d38772b8e6:~/avro/lang/js$ node -v
> v6.17.1
> sekikn@13d38772b8e6:~/avro/lang/js$ npm i
> sekikn@13d38772b8e6:~/avro/lang/js$ for i in $(seq 1 100); do echo "#${i}"; if ! npm t; then break; fi; done
> #1
> > avro-js@1.9.0 test /home/sekikn/avro/lang/js
> > mocha
> ․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․
> ․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․
> ․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․
> 381 passing (557ms)
> (snip)
> #7
> > avro-js@1.9.0 test /home/sekikn/avro/lang/js
> > mocha
> ․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․
> ․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․
> ․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․
> 380 passing (554ms)
> 1 failing
> 1) files createFileEncoder:
> Uncaught AssertionError: 0 == 2
> + expected - actual
> -0
> +2
>
> at BlockDecoder.<anonymous> (test/test_files.js:574:18)
> at endReadableNT (_stream_readable.js:978:12)
> at _combinedTickCallback (internal/process/next_tick.js:80:11)
> at process._tickCallback (internal/process/next_tick.js:104:9)
> npm ERR! Test failed. See above for more details.
> {code}
> And it always fails with Node.js v8.10.0, which is installed by apt-get on Ubuntu 18.04 LTS.
> {code}
> ~/repos/avro$ git checkout master
> ~/repos/avro$ cd lang/js
> ~/repos/avro/lang/js$ ./build.sh test
> > avro-js@1.9.0 cover /home/sekikn/repos/avro/lang/js
> > istanbul cover _mocha
> ․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․
> ․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․
> ․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․
> 380 passing (537ms)
> 1 failing
> 1) files createFileEncoder:
> Uncaught AssertionError [ERR_ASSERTION]: 0 == 2
> + expected - actual
> -0
> +2
>
> at BlockDecoder.<anonymous> (test/test_files.js:574:18)
> at endReadableNT (_stream_readable.js:1064:12)
> at _combinedTickCallback (internal/process/next_tick.js:138:11)
> at process._tickCallback (internal/process/next_tick.js:180:9)
> =============================================================================
> Writing coverage object [/home/sekikn/repos/avro/lang/js/coverage/coverage.json]
> Writing coverage reports at [/home/sekikn/repos/avro/lang/js/coverage]
> =============================================================================
> =============================== Coverage summary ===============================
> Statements : 100% ( 2529/2529 )
> Branches : 100% ( 996/996 )
> Functions : 100% ( 414/414 )
> Lines : 100% ( 2443/2443 )
> ================================================================================
> npm ERR! Linux 5.0.0-25-generic
> npm ERR! argv "/usr/bin/node" "/usr/bin/npm" "run" "cover"
> npm ERR! node v8.10.0
> npm ERR! npm v3.5.2
> npm ERR! code ELIFECYCLE
> npm ERR! avro-js@1.9.0 cover: `istanbul cover _mocha`
> npm ERR! Exit status 1
> npm ERR!
> npm ERR! Failed at the avro-js@1.9.0 cover script 'istanbul cover _mocha'.
> npm ERR! Make sure you have the latest version of node.js and npm installed.
> npm ERR! If you do, this is most likely a problem with the avro-js package,
> npm ERR! not with npm itself.
> npm ERR! Tell the author that this fails on your system:
> npm ERR! istanbul cover _mocha
> npm ERR! You can get information on how to open an issue for this project with:
> npm ERR! npm bugs avro-js
> npm ERR! Or if that isn't available, you can get their info via:
> npm ERR! npm owner ls avro-js
> npm ERR! There is likely additional logging output above.
> npm ERR! Please include the following file with any support request:
> npm ERR! /home/sekikn/repos/avro/lang/js/npm-debug.log
> {code}
> In this test, {{createFileDecoder()}} is called in the event listener for the {{'finish'}} event emitted from the {{encoder}} object.
> {code:title=test/test_files.js}
> 553 test('createFileEncoder', function (cb) {
> 554 var type = createType({
> 555 type: 'record',
> 556 name: 'Person',
> 557 fields: [
> 558 {name: 'name', type: 'string'},
> 559 {name: 'age', type: 'int'}
> 560 ]
> 561 });
> 562 var path = tmp.fileSync().name;
> 563 var encoder = files.createFileEncoder(path, type);
> 564 encoder.write({name: 'Ann', age: 32});
> 565 encoder.end({name: 'Bob', age: 33});
> 566 var n = 0;
> 567 encoder.on('finish', function () {
> 568 files.createFileDecoder(path)
> 569 .on('data', function (obj) {
> 570 n++;
> 571 assert(type.isValid(obj));
> 572 })
> 573 .on('end', function () {
> 574 assert.equal(n, 2);
> 575 cb();
> 576 });
> 577 });
> 578 });
> {code}
> And the {{encoder}} object has the underlying writable stream created by {{createWriteStream()}}.
> {code:title=lib/files.js}
> 562 function createFileEncoder(path, schema, opts) {
> 563 var encoder = new BlockEncoder(schema, opts);
> 564 encoder.pipe(fs.createWriteStream(path, {defaultEncoding: 'binary'}));
> 565 return encoder;
> 566 }
> {code}
> In such a situation, the caller should wait for the underlying stream's {{'finish'}} event rather than {{encoder}}'s. Because when the {{encoder}} has finished to write and the caller catch its {{'finish'}} event, the underlying stream perhaps doesn't finished to write into the disk yet (cf. https://stackoverflow.com/questions/40864496/detecting-the-end-of-a-piped-stream-operation).
> I think BlockEncoder should expose a reference to the underlying piped stream, so that users can listen its event (typically {{'finish'}}).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)