You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@apex.apache.org by Aaron Bossert <aa...@punchcyber.com> on 2018/06/11 19:03:49 UTC
Re: Kryo version and default serializer
I know it has been a little while since I brought this up...I pulled down
the 3.7.1-RELEASE branch from GitHub in order to test out the change to
Kryo v. 4.0.2 and encountered no problems while building, however, when I
try to do mvn install, there are several failed tests. I am hoping, that
other than the obvious (to me, anyway) errors such as not finding
yarn-site.xml on my laptop which does not have hadoop installed, someone
may be able to see what is going on?
-------------------------------------------------------------------------------
Test set: com.datatorrent.stram.StramRecoveryTest
-------------------------------------------------------------------------------
Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 6.372 sec
<<< FAILURE! - in com.datatorrent.stram.StramRecoveryTest
testWriteAheadLog(com.datatorrent.stram.StramRecoveryTest) Time elapsed:
0.115 sec <<< FAILURE!
java.lang.AssertionError: flush count expected:<1> but was:<2>
at
com.datatorrent.stram.StramRecoveryTest.testWriteAheadLog(StramRecoveryTest.java:326)
-------------------------------------------------------------------------------
Test set: com.datatorrent.stram.engine.StatsTest
-------------------------------------------------------------------------------
Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 21.894 sec
<<< FAILURE! - in com.datatorrent.stram.engine.StatsTest
testQueueSizeForContainerLocalOperators(com.datatorrent.stram.engine.StatsTest)
Time elapsed: 3.266 sec <<< FAILURE!
java.lang.AssertionError: Validate input port queue size -1
at
com.datatorrent.stram.engine.StatsTest.baseTestForQueueSize(StatsTest.java:270)
at
com.datatorrent.stram.engine.StatsTest.testQueueSizeForContainerLocalOperators(StatsTest.java:285)
-------------------------------------------------------------------------------
Test set: com.datatorrent.stram.StramMiniClusterTest
-------------------------------------------------------------------------------
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.002 sec
<<< FAILURE! - in com.datatorrent.stram.StramMiniClusterTest
com.datatorrent.stram.StramMiniClusterTest Time elapsed: 0.002 sec <<<
ERROR!
java.io.FileNotFoundException:
/Users/mbossert/Library/Mobile%20Documents/com~apple~CloudDocs/Punch/IdeaProjects/apex-core/engine/target/test-classes/yarn-site.xml
(No such file or directory)
at
com.datatorrent.stram.StramMiniClusterTest.setup(StramMiniClusterTest.java:151)
-------------------------------------------------------------------------------
Test set: com.datatorrent.stram.StramLocalClusterTest
-------------------------------------------------------------------------------
Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 15.081 sec
<<< FAILURE! - in com.datatorrent.stram.StramLocalClusterTest
testAppPath(com.datatorrent.stram.StramLocalClusterTest) Time elapsed:
0.138 sec <<< FAILURE!
java.lang.AssertionError: checkpoint path exists
at
com.datatorrent.stram.StramLocalClusterTest.testAppPath(StramLocalClusterTest.java:413)
testLocalClusterInitShutdown(com.datatorrent.stram.StramLocalClusterTest)
Time elapsed: 5.191 sec <<< FAILURE!
java.lang.AssertionError:
./target/com.datatorrent.stram.StramLocalClusterTest-testLocalClusterInitShutdown.out
exists
at
com.datatorrent.stram.StramLocalClusterTest.testLocalClusterInitShutdown(StramLocalClusterTest.java:117)
-------------------------------------------------------------------------------
Test set: com.datatorrent.stram.client.AppPackageTest
-------------------------------------------------------------------------------
Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.105 sec
<<< FAILURE! - in com.datatorrent.stram.client.AppPackageTest
testAppPackage(com.datatorrent.stram.client.AppPackageTest) Time elapsed:
0 sec <<< FAILURE!
java.lang.AssertionError: Number of applications expected:<2> but was:<0>
at
com.datatorrent.stram.client.AppPackageTest.testAppPackage(AppPackageTest.java:91)
testAppLevelRequiredAndDefaultProperties(com.datatorrent.stram.client.AppPackageTest)
Time elapsed: 0.001 sec <<< FAILURE!
java.lang.AssertionError: Should consist of an app called MyFirstApplication
at
com.datatorrent.stram.client.AppPackageTest.testAppLevelRequiredAndDefaultProperties(AppPackageTest.java:171)
-------------------------------------------------------------------------------
Test set: com.datatorrent.stram.cli.ApexCliTest
-------------------------------------------------------------------------------
Tests run: 7, Failures: 2, Errors: 2, Skipped: 0, Time elapsed: 0.938 sec
<<< FAILURE! - in com.datatorrent.stram.cli.ApexCliTest
testAppFromOnlyConfigPackage(com.datatorrent.stram.cli.ApexCliTest) Time
elapsed: 0.104 sec <<< ERROR!
com.datatorrent.stram.cli.ApexCli$CliException: No applications in
Application Package
at
com.datatorrent.stram.cli.ApexCliTest.testAppFromOnlyConfigPackage(ApexCliTest.java:235)
testLaunchAppPackagePropertyPrecedence(com.datatorrent.stram.cli.ApexCliTest)
Time elapsed: 0.045 sec <<< FAILURE!
org.junit.ComparisonFailure: expected:<[app]-default> but
was:<[package]-default>
at
com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePropertyPrecedence(ApexCliTest.java:143)
testLaunchAppPackageParametersWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
Time elapsed: 0.045 sec <<< ERROR!
com.datatorrent.stram.cli.ApexCli$CliException: No applications in
Application Package matching "MyFirstApplication"
at
com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackageParametersWithConfigPackage(ApexCliTest.java:155)
testLaunchAppPackagePrecedenceWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
Time elapsed: 0.058 sec <<< FAILURE!
org.junit.ComparisonFailure: expected:<[app]-default> but
was:<[package]-default>
at
com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePrecedenceWithConfigPackage(ApexCliTest.java:204)
On Thu, May 17, 2018 at 1:31 PM, Aaron Bossert <aa...@punchcyber.com> wrote:
> Thanks, Thomas! I have implemented the fix you suggested
> (@FieldSerializer.BInd) and also created a JIRA ticket:
> https://issues.apache.org/jira/browse/APEXCORE-816
>
> On Thu, May 17, 2018 at 1:31 AM, Thomas Weise <th...@apache.org> wrote:
>
>> Hi,
>>
>> It is not necessary to use a different stream codec when you have control
>> over the class that is causing the error. You can annotate a different
>> field serializer instead, using the @FieldSerializer.Bind annotation. Here
>> is an example:
>>
>> https://github.com/apache/apex-malhar/blob/2fe2903bfe65055c7
>> fd361a97549b6e7e12d4289/library/src/main/java/org/
>> apache/apex/malhar/lib/window/WindowOption.java#L50
>>
>> It would be good to upgrade Kryo to 4.x in apex-core master. Would you
>> like to create a JIRA for it? (http://apex.apache.org/contributing.html)
>>
>> Thanks,
>> Thomas
>>
>>
>>
>>
>>
>> On Wed, May 16, 2018 at 10:31 AM, Aaron Bossert <aa...@punchcyber.com>
>> wrote:
>>
>>> I am writing an application that includes a custom object. One of the
>>> fields is of type Instant and seems to not agree with the version of Kryo
>>> that is used in Apex (see exception below). I now that kryo 4.X supports
>>> serializing Instant fields, but for the life of me, cannot seem to find a
>>> way to either get Apex to use a newer version of Kryo. Can anyone help?
>>> Perhaps setting a different streamcodec that uses the newer version of kryo
>>> would work, but I cannot seem to find an example of how to set a different
>>> streamcodec despite references to the two (Kryo and Java) that ship with
>>> Apex.
>>>
>>> ERROR com.datatorrent.stram.codec.DefaultStatefulStreamCodec -
>>> Catastrophic Error: Execution halted due to Kryo exception!
>>> com.esotericsoftware.kryo.KryoException: Class cannot be created
>>> (missing no-arg constructor): java.time.Instant
>>> Serialization trace:
>>> systemTime (com.punchcyber.streaming.pipeline.apex.datatypes.evtx.EvtxE
>>> vent)
>>> at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.n
>>> ewInstantiatorOf(Kryo.java:1228)
>>> at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
>>> at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
>>> at com.esotericsoftware.kryo.serializers.FieldSerializer.create
>>> (FieldSerializer.java:547)
>>> at com.esotericsoftware.kryo.serializers.FieldSerializer.read(F
>>> ieldSerializer.java:523)
>>> at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730)
>>> at com.esotericsoftware.kryo.serializers.ObjectField.read(Objec
>>> tField.java:113)
>>> at com.esotericsoftware.kryo.serializers.FieldSerializer.read(F
>>> ieldSerializer.java:528)
>>> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
>>> at com.datatorrent.stram.codec.DefaultStatefulStreamCodec.fromD
>>> ataStatePair(DefaultStatefulStreamCodec.java:98)
>>> at com.datatorrent.stram.stream.BufferServerSubscriber$BufferRe
>>> servoir.processPayload(BufferServerSubscriber.java:391)
>>> at com.datatorrent.stram.stream.BufferServerSubscriber$BufferRe
>>> servoir.sweep(BufferServerSubscriber.java:339)
>>> at com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:269)
>>> at com.datatorrent.stram.engine.StreamingContainer$2.run(Stream
>>> ingContainer.java:1429)
>>>
>>>
>>> --
>>>
>>> M. Aaron Bossert
>>> (571) 242-4021
>>> Punch Cyber Analytics Group
>>>
>>>
>>>
>>
>
>
> --
>
> M. Aaron Bossert
> (571) 242-4021
> Punch Cyber Analytics Group
>
>
>
--
M. Aaron Bossert
(571) 242-4021
Punch Cyber Analytics Group
Re: Kryo version and default serializer
Posted by Vlad Rozov <vr...@apache.org>.
Hi Aaron,
Nice to see more contributors to the Apex! Welcome.
I'd suggest that we stop this e-mail thread and continue discussion on
the other dev@apex only e-mail thread. Usually users@apex is not a good
choice to discuss dependency upgrades.
Please be patient, majority of Apex and Apache committers contribute at
their free time and there may be delays in response.
Thank you,
Vlad
On 6/19/18 20:02, Aaron Bossert wrote:
> cool! I'm on it.
>
> On Tue, Jun 19, 2018 at 11:00 PM Pramod Immaneni
> <pramod.immaneni@gmail.com <ma...@gmail.com>> wrote:
>
> Hi Aaron,
>
> That would be great if you can pitch in wherever you can. Let us
> know how we can help. Also, I was going to try the
> new kryo version later today and see if there are any internal
> errors that are bubbling up to those test failures but not getting
> logged.
>
> Thanks
>
> On Tue, Jun 19, 2018 at 7:54 PM Aaron Bossert
> <aaron@punchcyber.com <ma...@punchcyber.com>> wrote:
>
> P.S. I am happy to roll up my sleeves, but from my initial
> attempt, errors
> I was seeing were not obviously related to the Kryo version
> update (though
> I am certain they are...at least indirectly because they only
> show up when
> the version is changed), in terms of tracking down where the
> changes would
> be required...I am not really familiar with the inner workings
> of Apex yet,
> though this would definitely be a good excuse to learn...
>
> I also saw a bunch of annoying warnings when building related
> to use of
> deprecated methods/classes within the com.datatorrent.*
> namespace...I'd be
> happy to start tracking some of those down too if it would be
> helpful...
>
> On Tue, Jun 19, 2018 at 10:48 PM Aaron Bossert
> <aaron@punchcyber.com <ma...@punchcyber.com>> wrote:
>
> > Pramod,
> >
> > In a nutshell, yes. I can set a @FieldSerializer to the
> fields in
> > question...the only one I have run into thus far is
> Instant...but it would
> > be great to not need to annotate each field this way...Also,
> I could see
> > this becoming problematic should I encounter more than a
> couple of fields
> > that are not serializable using the older version of Kryo
> (but presumably
> > would work with a more current one)...though admitedly, it
> would not be
> > that big a deal in the interim. It would be much cleaner to
> have an update
> > to the most current version of Kryo, IMHO...though I
> understand that is
> > something of a lift. By the way, Kryo 5 just came out
> yesterday, so if
> > there is to be effort expended, it might be good to go with
> the latest
> > version...though, heads up, I tried it out and there seem to
> be quite a few
> > changes that will be needed...new methods, and definitely
> some replaced
> > ones...
> >
> >
> >
> > On Tue, Jun 19, 2018 at 10:37 PM Pramod Immaneni <
> > pramod.immaneni@gmail.com
> <ma...@gmail.com>> wrote:
> >
> >> Hi Aaron,
> >>
> >> While we are debugging the test failures on dev, I didn't fully
> >> understand the last question in your email. Looks like you
> mentioned that
> >> the workaround for adding the @FieldSerializer to the field
> is working. Are
> >> you looking for an alternative to this workaround and
> trying to set a
> >> default serializer for all fields of type Instant (the
> field type you
> >> mentioned in the earlier email) so that you don't have to
> set the
> >> annotation each time?
> >>
> >> Thanks
> >>
> >> On Tue, Jun 19, 2018 at 12:59 PM Aaron Bossert
> <aaron@punchcyber.com <ma...@punchcyber.com>>
> >> wrote:
> >>
> >>> I sent an e-mail to the dev list...but have not heard any
> responses.
> >>> How active is that list? Do you know? Basically, I have a
> workaround that
> >>> will deal with those "things" that Kryo v 2.24.0 will not
> handle (but 4.0.2
> >>> or 5.0.0-RC1 would) using the @FieldSerializer ...but this
> is not ideal. I
> >>> have two options left: go it alone and try to update Kryo
> in the apex-core,
> >>> or simply change the default serializer. The hiccup I am
> running into is
> >>> that I don't see how (from the documentation) to set the
> default
> >>> serializer...perhaps I am missing it. Can someone point
> me to where that
> >>> is in the docs?
> >>>
> >>> On Mon, Jun 11, 2018 at 4:20 PM Aaron Bossert
> <aaron@punchcyber.com <ma...@punchcyber.com>>
> >>> wrote:
> >>>
> >>>> Ah, you make a good point about the dev list...I am using
> IntelliJ on a
> >>>> Mac. When I tried building without any changes, I also
> encountered
> >>>> failures. I can re-run the non-modified sources to do a
> diff and see if
> >>>> the errors were different...
> >>>>
> >>>> Sent from my iPhone
> >>>>
> >>>> On Jun 11, 2018, at 15:11, Pramod Immaneni
> <pramod.immaneni@gmail.com <ma...@gmail.com>>
> >>>> wrote:
> >>>>
> >>>> Aaron,
> >>>>
> >>>> The question might be best served on the
> dev@apex.apache.org <ma...@apex.apache.org> mailing
> >>>> list as you are starting to make changes to the sources.
> Nothing obvious
> >>>> jumps out to be, you don't need hadoop on your system to
> build the sources
> >>>> successfully with all the tests passing. The test
> failures you are seeing
> >>>> may or may not be related to the kryo change. What
> happens when you try to
> >>>> build the original sources with changing the Kryo
> version? What is your
> >>>> environment, mac, linux?
> >>>>
> >>>> On Mon, Jun 11, 2018 at 12:03 PM Aaron Bossert
> <aaron@punchcyber.com <ma...@punchcyber.com>>
> >>>> wrote:
> >>>>
> >>>>> I know it has been a little while since I brought this
> up...I pulled
> >>>>> down the 3.7.1-RELEASE branch from GitHub in order to
> test out the change
> >>>>> to Kryo v. 4.0.2 and encountered no problems while
> building, however, when
> >>>>> I try to do mvn install, there are several failed
> tests. I am hoping, that
> >>>>> other than the obvious (to me, anyway) errors such as
> not finding
> >>>>> yarn-site.xml on my laptop which does not have hadoop
> installed, someone
> >>>>> may be able to see what is going on?
> >>>>>
> >>>>>
> >>>>>
> -------------------------------------------------------------------------------
> >>>>> Test set: com.datatorrent.stram.StramRecoveryTest
> >>>>>
> >>>>>
> -------------------------------------------------------------------------------
> >>>>> Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time
> elapsed: 6.372
> >>>>> sec <<< FAILURE! - in
> com.datatorrent.stram.StramRecoveryTest
> >>>>>
> testWriteAheadLog(com.datatorrent.stram.StramRecoveryTest) Time
> >>>>> elapsed: 0.115 sec <<< FAILURE!
> >>>>> java.lang.AssertionError: flush count expected:<1> but
> was:<2>
> >>>>> at
> >>>>>
> com.datatorrent.stram.StramRecoveryTest.testWriteAheadLog(StramRecoveryTest.java:326)
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> -------------------------------------------------------------------------------
> >>>>> Test set: com.datatorrent.stram.engine.StatsTest
> >>>>>
> >>>>>
> -------------------------------------------------------------------------------
> >>>>> Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time
> elapsed: 21.894
> >>>>> sec <<< FAILURE! - in com.datatorrent.stram.engine.StatsTest
> >>>>>
> testQueueSizeForContainerLocalOperators(com.datatorrent.stram.engine.StatsTest)
> >>>>> Time elapsed: 3.266 sec <<< FAILURE!
> >>>>> java.lang.AssertionError: Validate input port queue size -1
> >>>>> at
> >>>>>
> com.datatorrent.stram.engine.StatsTest.baseTestForQueueSize(StatsTest.java:270)
> >>>>> at
> >>>>>
> com.datatorrent.stram.engine.StatsTest.testQueueSizeForContainerLocalOperators(StatsTest.java:285)
> >>>>>
> >>>>>
> >>>>>
> -------------------------------------------------------------------------------
> >>>>> Test set: com.datatorrent.stram.StramMiniClusterTest
> >>>>>
> >>>>>
> -------------------------------------------------------------------------------
> >>>>> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time
> elapsed: 0.002
> >>>>> sec <<< FAILURE! - in
> com.datatorrent.stram.StramMiniClusterTest
> >>>>> com.datatorrent.stram.StramMiniClusterTest Time
> elapsed: 0.002 sec
> >>>>> <<< ERROR!
> >>>>> java.io.FileNotFoundException:
> >>>>>
> /Users/mbossert/Library/Mobile%20Documents/com~apple~CloudDocs/Punch/IdeaProjects/apex-core/engine/target/test-classes/yarn-site.xml
> >>>>> (No such file or directory)
> >>>>> at
> >>>>>
> com.datatorrent.stram.StramMiniClusterTest.setup(StramMiniClusterTest.java:151)
> >>>>>
> >>>>>
> >>>>>
> -------------------------------------------------------------------------------
> >>>>> Test set: com.datatorrent.stram.StramLocalClusterTest
> >>>>>
> >>>>>
> -------------------------------------------------------------------------------
> >>>>> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time
> elapsed: 15.081
> >>>>> sec <<< FAILURE! - in
> com.datatorrent.stram.StramLocalClusterTest
> >>>>> testAppPath(com.datatorrent.stram.StramLocalClusterTest)
> Time
> >>>>> elapsed: 0.138 sec <<< FAILURE!
> >>>>> java.lang.AssertionError: checkpoint path exists
> >>>>> at
> >>>>>
> com.datatorrent.stram.StramLocalClusterTest.testAppPath(StramLocalClusterTest.java:413)
> >>>>>
> >>>>>
> testLocalClusterInitShutdown(com.datatorrent.stram.StramLocalClusterTest)
> >>>>> Time elapsed: 5.191 sec <<< FAILURE!
> >>>>> java.lang.AssertionError:
> >>>>>
> ./target/com.datatorrent.stram.StramLocalClusterTest-testLocalClusterInitShutdown.out
> >>>>> exists
> >>>>> at
> >>>>>
> com.datatorrent.stram.StramLocalClusterTest.testLocalClusterInitShutdown(StramLocalClusterTest.java:117)
> >>>>>
> >>>>>
> >>>>>
> -------------------------------------------------------------------------------
> >>>>> Test set: com.datatorrent.stram.client.AppPackageTest
> >>>>>
> >>>>>
> -------------------------------------------------------------------------------
> >>>>> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time
> elapsed: 0.105
> >>>>> sec <<< FAILURE! - in
> com.datatorrent.stram.client.AppPackageTest
> >>>>>
> testAppPackage(com.datatorrent.stram.client.AppPackageTest) Time
> >>>>> elapsed: 0 sec <<< FAILURE!
> >>>>> java.lang.AssertionError: Number of applications
> expected:<2> but
> >>>>> was:<0>
> >>>>> at
> >>>>>
> com.datatorrent.stram.client.AppPackageTest.testAppPackage(AppPackageTest.java:91)
> >>>>>
> >>>>>
> testAppLevelRequiredAndDefaultProperties(com.datatorrent.stram.client.AppPackageTest)
> >>>>> Time elapsed: 0.001 sec <<< FAILURE!
> >>>>> java.lang.AssertionError: Should consist of an app called
> >>>>> MyFirstApplication
> >>>>> at
> >>>>>
> com.datatorrent.stram.client.AppPackageTest.testAppLevelRequiredAndDefaultProperties(AppPackageTest.java:171)
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> -------------------------------------------------------------------------------
> >>>>> Test set: com.datatorrent.stram.cli.ApexCliTest
> >>>>>
> >>>>>
> -------------------------------------------------------------------------------
> >>>>> Tests run: 7, Failures: 2, Errors: 2, Skipped: 0, Time
> elapsed: 0.938
> >>>>> sec <<< FAILURE! - in com.datatorrent.stram.cli.ApexCliTest
> >>>>>
> testAppFromOnlyConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
> >>>>> Time elapsed: 0.104 sec <<< ERROR!
> >>>>> com.datatorrent.stram.cli.ApexCli$CliException: No
> applications in
> >>>>> Application Package
> >>>>> at
> >>>>>
> com.datatorrent.stram.cli.ApexCliTest.testAppFromOnlyConfigPackage(ApexCliTest.java:235)
> >>>>>
> >>>>>
> testLaunchAppPackagePropertyPrecedence(com.datatorrent.stram.cli.ApexCliTest)
> >>>>> Time elapsed: 0.045 sec <<< FAILURE!
> >>>>> org.junit.ComparisonFailure: expected:<[app]-default> but
> >>>>> was:<[package]-default>
> >>>>> at
> >>>>>
> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePropertyPrecedence(ApexCliTest.java:143)
> >>>>>
> >>>>>
> testLaunchAppPackageParametersWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
> >>>>> Time elapsed: 0.045 sec <<< ERROR!
> >>>>> com.datatorrent.stram.cli.ApexCli$CliException: No
> applications in
> >>>>> Application Package matching "MyFirstApplication"
> >>>>> at
> >>>>>
> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackageParametersWithConfigPackage(ApexCliTest.java:155)
> >>>>>
> >>>>>
> testLaunchAppPackagePrecedenceWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
> >>>>> Time elapsed: 0.058 sec <<< FAILURE!
> >>>>> org.junit.ComparisonFailure: expected:<[app]-default> but
> >>>>> was:<[package]-default>
> >>>>> at
> >>>>>
> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePrecedenceWithConfigPackage(ApexCliTest.java:204)
> >>>>>
> >>>>>
> >>>>> On Thu, May 17, 2018 at 1:31 PM, Aaron Bossert
> <aaron@punchcyber.com <ma...@punchcyber.com>>
> >>>>> wrote:
> >>>>>
> >>>>>> Thanks, Thomas! I have implemented the fix you suggested
> >>>>>> (@FieldSerializer.BInd) and also created a JIRA ticket:
> >>>>>> https://issues.apache.org/jira/browse/APEXCORE-816
> >>>>>>
> >>>>>> On Thu, May 17, 2018 at 1:31 AM, Thomas Weise
> <thw@apache.org <ma...@apache.org>> wrote:
> >>>>>>
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> It is not necessary to use a different stream codec
> when you have
> >>>>>>> control over the class that is causing the error. You
> can annotate a
> >>>>>>> different field serializer instead, using the
> @FieldSerializer.Bind
> >>>>>>> annotation. Here is an example:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> https://github.com/apache/apex-malhar/blob/2fe2903bfe65055c7fd361a97549b6e7e12d4289/library/src/main/java/org/apache/apex/malhar/lib/window/WindowOption.java#L50
> >>>>>>>
> >>>>>>> It would be good to upgrade Kryo to 4.x in apex-core
> master. Would
> >>>>>>> you like to create a JIRA for it? (
> >>>>>>> http://apex.apache.org/contributing.html)
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> Thomas
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Wed, May 16, 2018 at 10:31 AM, Aaron Bossert <
> >>>>>>> aaron@punchcyber.com <ma...@punchcyber.com>> wrote:
> >>>>>>>
> >>>>>>>> I am writing an application that includes a custom
> object. One of
> >>>>>>>> the fields is of type Instant and seems to not agree
> with the version of
> >>>>>>>> Kryo that is used in Apex (see exception below). I
> now that kryo 4.X
> >>>>>>>> supports serializing Instant fields, but for the life
> of me, cannot seem to
> >>>>>>>> find a way to either get Apex to use a newer version
> of Kryo. Can anyone
> >>>>>>>> help? Perhaps setting a different streamcodec that
> uses the newer version
> >>>>>>>> of kryo would work, but I cannot seem to find an
> example of how to set a
> >>>>>>>> different streamcodec despite references to the two
> (Kryo and Java) that
> >>>>>>>> ship with Apex.
> >>>>>>>>
> >>>>>>>> ERROR
> com.datatorrent.stram.codec.DefaultStatefulStreamCodec -
> >>>>>>>> Catastrophic Error: Execution halted due to Kryo
> exception!
> >>>>>>>> com.esotericsoftware.kryo.KryoException: Class cannot
> be created
> >>>>>>>> (missing no-arg constructor): java.time.Instant
> >>>>>>>> Serialization trace:
> >>>>>>>> systemTime
> >>>>>>>>
> (com.punchcyber.streaming.pipeline.apex.datatypes.evtx.EvtxEvent)
> >>>>>>>> at
> >>>>>>>>
> com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
> >>>>>>>> at
> com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
> >>>>>>>> at
> com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
> >>>>>>>> at
> >>>>>>>>
> com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
> >>>>>>>> at
> >>>>>>>>
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
> >>>>>>>> at
> com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730)
> >>>>>>>> at
> >>>>>>>>
> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113)
> >>>>>>>> at
> >>>>>>>>
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
> >>>>>>>> at
> com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
> >>>>>>>> at com.datatorrent.stram.codec.De
> <http://com.datatorrent.stram.codec.De>
> >>>>>>>>
> faultStatefulStreamCodec.fromDataStatePair(DefaultStatefulStreamCodec.java:98)
> >>>>>>>> at
> >>>>>>>>
> com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.processPayload(BufferServerSubscriber.java:391)
> >>>>>>>> at
> >>>>>>>>
> com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.sweep(BufferServerSubscriber.java:339)
> >>>>>>>> at
> >>>>>>>>
> com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:269)
> >>>>>>>> at
> >>>>>>>>
> com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1429)
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>>
> >>>>>>>> M. Aaron Bossert
> >>>>>>>> (571) 242-4021
> >>>>>>>> Punch Cyber Analytics Group
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>>
> >>>>>> M. Aaron Bossert
> >>>>>> (571) 242-4021
> >>>>>> Punch Cyber Analytics Group
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>>
> >>>>> M. Aaron Bossert
> >>>>> (571) 242-4021
> >>>>> Punch Cyber Analytics Group
> >>>>>
> >>>>>
> >>>>>
> >>>
> >>> --
> >>>
> >>> M. Aaron Bossert
> >>> (571) 242-4021
> >>> Punch Cyber Analytics Group
> >>>
> >>>
> >>>
> >
> > --
> >
> > M. Aaron Bossert
> > (571) 242-4021
> > Punch Cyber Analytics Group
> >
> >
> >
>
> --
>
> M. Aaron Bossert
> (571) 242-4021
> Punch Cyber Analytics Group
>
>
>
> --
>
> M. Aaron Bossert
> (571) 242-4021
> Punch Cyber Analytics Group
>
>
Re: Kryo version and default serializer
Posted by Vlad Rozov <vr...@apache.org>.
Hi Aaron,
Nice to see more contributors to the Apex! Welcome.
I'd suggest that we stop this e-mail thread and continue discussion on
the other dev@apex only e-mail thread. Usually users@apex is not a good
choice to discuss dependency upgrades.
Please be patient, majority of Apex and Apache committers contribute at
their free time and there may be delays in response.
Thank you,
Vlad
On 6/19/18 20:02, Aaron Bossert wrote:
> cool! I'm on it.
>
> On Tue, Jun 19, 2018 at 11:00 PM Pramod Immaneni
> <pramod.immaneni@gmail.com <ma...@gmail.com>> wrote:
>
> Hi Aaron,
>
> That would be great if you can pitch in wherever you can. Let us
> know how we can help. Also, I was going to try the
> new kryo version later today and see if there are any internal
> errors that are bubbling up to those test failures but not getting
> logged.
>
> Thanks
>
> On Tue, Jun 19, 2018 at 7:54 PM Aaron Bossert
> <aaron@punchcyber.com <ma...@punchcyber.com>> wrote:
>
> P.S. I am happy to roll up my sleeves, but from my initial
> attempt, errors
> I was seeing were not obviously related to the Kryo version
> update (though
> I am certain they are...at least indirectly because they only
> show up when
> the version is changed), in terms of tracking down where the
> changes would
> be required...I am not really familiar with the inner workings
> of Apex yet,
> though this would definitely be a good excuse to learn...
>
> I also saw a bunch of annoying warnings when building related
> to use of
> deprecated methods/classes within the com.datatorrent.*
> namespace...I'd be
> happy to start tracking some of those down too if it would be
> helpful...
>
> On Tue, Jun 19, 2018 at 10:48 PM Aaron Bossert
> <aaron@punchcyber.com <ma...@punchcyber.com>> wrote:
>
> > Pramod,
> >
> > In a nutshell, yes. I can set a @FieldSerializer to the
> fields in
> > question...the only one I have run into thus far is
> Instant...but it would
> > be great to not need to annotate each field this way...Also,
> I could see
> > this becoming problematic should I encounter more than a
> couple of fields
> > that are not serializable using the older version of Kryo
> (but presumably
> > would work with a more current one)...though admitedly, it
> would not be
> > that big a deal in the interim. It would be much cleaner to
> have an update
> > to the most current version of Kryo, IMHO...though I
> understand that is
> > something of a lift. By the way, Kryo 5 just came out
> yesterday, so if
> > there is to be effort expended, it might be good to go with
> the latest
> > version...though, heads up, I tried it out and there seem to
> be quite a few
> > changes that will be needed...new methods, and definitely
> some replaced
> > ones...
> >
> >
> >
> > On Tue, Jun 19, 2018 at 10:37 PM Pramod Immaneni <
> > pramod.immaneni@gmail.com
> <ma...@gmail.com>> wrote:
> >
> >> Hi Aaron,
> >>
> >> While we are debugging the test failures on dev, I didn't fully
> >> understand the last question in your email. Looks like you
> mentioned that
> >> the workaround for adding the @FieldSerializer to the field
> is working. Are
> >> you looking for an alternative to this workaround and
> trying to set a
> >> default serializer for all fields of type Instant (the
> field type you
> >> mentioned in the earlier email) so that you don't have to
> set the
> >> annotation each time?
> >>
> >> Thanks
> >>
> >> On Tue, Jun 19, 2018 at 12:59 PM Aaron Bossert
> <aaron@punchcyber.com <ma...@punchcyber.com>>
> >> wrote:
> >>
> >>> I sent an e-mail to the dev list...but have not heard any
> responses.
> >>> How active is that list? Do you know? Basically, I have a
> workaround that
> >>> will deal with those "things" that Kryo v 2.24.0 will not
> handle (but 4.0.2
> >>> or 5.0.0-RC1 would) using the @FieldSerializer ...but this
> is not ideal. I
> >>> have two options left: go it alone and try to update Kryo
> in the apex-core,
> >>> or simply change the default serializer. The hiccup I am
> running into is
> >>> that I don't see how (from the documentation) to set the
> default
> >>> serializer...perhaps I am missing it. Can someone point
> me to where that
> >>> is in the docs?
> >>>
> >>> On Mon, Jun 11, 2018 at 4:20 PM Aaron Bossert
> <aaron@punchcyber.com <ma...@punchcyber.com>>
> >>> wrote:
> >>>
> >>>> Ah, you make a good point about the dev list...I am using
> IntelliJ on a
> >>>> Mac. When I tried building without any changes, I also
> encountered
> >>>> failures. I can re-run the non-modified sources to do a
> diff and see if
> >>>> the errors were different...
> >>>>
> >>>> Sent from my iPhone
> >>>>
> >>>> On Jun 11, 2018, at 15:11, Pramod Immaneni
> <pramod.immaneni@gmail.com <ma...@gmail.com>>
> >>>> wrote:
> >>>>
> >>>> Aaron,
> >>>>
> >>>> The question might be best served on the
> dev@apex.apache.org <ma...@apex.apache.org> mailing
> >>>> list as you are starting to make changes to the sources.
> Nothing obvious
> >>>> jumps out to be, you don't need hadoop on your system to
> build the sources
> >>>> successfully with all the tests passing. The test
> failures you are seeing
> >>>> may or may not be related to the kryo change. What
> happens when you try to
> >>>> build the original sources with changing the Kryo
> version? What is your
> >>>> environment, mac, linux?
> >>>>
> >>>> On Mon, Jun 11, 2018 at 12:03 PM Aaron Bossert
> <aaron@punchcyber.com <ma...@punchcyber.com>>
> >>>> wrote:
> >>>>
> >>>>> I know it has been a little while since I brought this
> up...I pulled
> >>>>> down the 3.7.1-RELEASE branch from GitHub in order to
> test out the change
> >>>>> to Kryo v. 4.0.2 and encountered no problems while
> building, however, when
> >>>>> I try to do mvn install, there are several failed
> tests. I am hoping, that
> >>>>> other than the obvious (to me, anyway) errors such as
> not finding
> >>>>> yarn-site.xml on my laptop which does not have hadoop
> installed, someone
> >>>>> may be able to see what is going on?
> >>>>>
> >>>>>
> >>>>>
> -------------------------------------------------------------------------------
> >>>>> Test set: com.datatorrent.stram.StramRecoveryTest
> >>>>>
> >>>>>
> -------------------------------------------------------------------------------
> >>>>> Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time
> elapsed: 6.372
> >>>>> sec <<< FAILURE! - in
> com.datatorrent.stram.StramRecoveryTest
> >>>>>
> testWriteAheadLog(com.datatorrent.stram.StramRecoveryTest) Time
> >>>>> elapsed: 0.115 sec <<< FAILURE!
> >>>>> java.lang.AssertionError: flush count expected:<1> but
> was:<2>
> >>>>> at
> >>>>>
> com.datatorrent.stram.StramRecoveryTest.testWriteAheadLog(StramRecoveryTest.java:326)
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> -------------------------------------------------------------------------------
> >>>>> Test set: com.datatorrent.stram.engine.StatsTest
> >>>>>
> >>>>>
> -------------------------------------------------------------------------------
> >>>>> Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time
> elapsed: 21.894
> >>>>> sec <<< FAILURE! - in com.datatorrent.stram.engine.StatsTest
> >>>>>
> testQueueSizeForContainerLocalOperators(com.datatorrent.stram.engine.StatsTest)
> >>>>> Time elapsed: 3.266 sec <<< FAILURE!
> >>>>> java.lang.AssertionError: Validate input port queue size -1
> >>>>> at
> >>>>>
> com.datatorrent.stram.engine.StatsTest.baseTestForQueueSize(StatsTest.java:270)
> >>>>> at
> >>>>>
> com.datatorrent.stram.engine.StatsTest.testQueueSizeForContainerLocalOperators(StatsTest.java:285)
> >>>>>
> >>>>>
> >>>>>
> -------------------------------------------------------------------------------
> >>>>> Test set: com.datatorrent.stram.StramMiniClusterTest
> >>>>>
> >>>>>
> -------------------------------------------------------------------------------
> >>>>> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time
> elapsed: 0.002
> >>>>> sec <<< FAILURE! - in
> com.datatorrent.stram.StramMiniClusterTest
> >>>>> com.datatorrent.stram.StramMiniClusterTest Time
> elapsed: 0.002 sec
> >>>>> <<< ERROR!
> >>>>> java.io.FileNotFoundException:
> >>>>>
> /Users/mbossert/Library/Mobile%20Documents/com~apple~CloudDocs/Punch/IdeaProjects/apex-core/engine/target/test-classes/yarn-site.xml
> >>>>> (No such file or directory)
> >>>>> at
> >>>>>
> com.datatorrent.stram.StramMiniClusterTest.setup(StramMiniClusterTest.java:151)
> >>>>>
> >>>>>
> >>>>>
> -------------------------------------------------------------------------------
> >>>>> Test set: com.datatorrent.stram.StramLocalClusterTest
> >>>>>
> >>>>>
> -------------------------------------------------------------------------------
> >>>>> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time
> elapsed: 15.081
> >>>>> sec <<< FAILURE! - in
> com.datatorrent.stram.StramLocalClusterTest
> >>>>> testAppPath(com.datatorrent.stram.StramLocalClusterTest)
> Time
> >>>>> elapsed: 0.138 sec <<< FAILURE!
> >>>>> java.lang.AssertionError: checkpoint path exists
> >>>>> at
> >>>>>
> com.datatorrent.stram.StramLocalClusterTest.testAppPath(StramLocalClusterTest.java:413)
> >>>>>
> >>>>>
> testLocalClusterInitShutdown(com.datatorrent.stram.StramLocalClusterTest)
> >>>>> Time elapsed: 5.191 sec <<< FAILURE!
> >>>>> java.lang.AssertionError:
> >>>>>
> ./target/com.datatorrent.stram.StramLocalClusterTest-testLocalClusterInitShutdown.out
> >>>>> exists
> >>>>> at
> >>>>>
> com.datatorrent.stram.StramLocalClusterTest.testLocalClusterInitShutdown(StramLocalClusterTest.java:117)
> >>>>>
> >>>>>
> >>>>>
> -------------------------------------------------------------------------------
> >>>>> Test set: com.datatorrent.stram.client.AppPackageTest
> >>>>>
> >>>>>
> -------------------------------------------------------------------------------
> >>>>> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time
> elapsed: 0.105
> >>>>> sec <<< FAILURE! - in
> com.datatorrent.stram.client.AppPackageTest
> >>>>>
> testAppPackage(com.datatorrent.stram.client.AppPackageTest) Time
> >>>>> elapsed: 0 sec <<< FAILURE!
> >>>>> java.lang.AssertionError: Number of applications
> expected:<2> but
> >>>>> was:<0>
> >>>>> at
> >>>>>
> com.datatorrent.stram.client.AppPackageTest.testAppPackage(AppPackageTest.java:91)
> >>>>>
> >>>>>
> testAppLevelRequiredAndDefaultProperties(com.datatorrent.stram.client.AppPackageTest)
> >>>>> Time elapsed: 0.001 sec <<< FAILURE!
> >>>>> java.lang.AssertionError: Should consist of an app called
> >>>>> MyFirstApplication
> >>>>> at
> >>>>>
> com.datatorrent.stram.client.AppPackageTest.testAppLevelRequiredAndDefaultProperties(AppPackageTest.java:171)
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> -------------------------------------------------------------------------------
> >>>>> Test set: com.datatorrent.stram.cli.ApexCliTest
> >>>>>
> >>>>>
> -------------------------------------------------------------------------------
> >>>>> Tests run: 7, Failures: 2, Errors: 2, Skipped: 0, Time
> elapsed: 0.938
> >>>>> sec <<< FAILURE! - in com.datatorrent.stram.cli.ApexCliTest
> >>>>>
> testAppFromOnlyConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
> >>>>> Time elapsed: 0.104 sec <<< ERROR!
> >>>>> com.datatorrent.stram.cli.ApexCli$CliException: No
> applications in
> >>>>> Application Package
> >>>>> at
> >>>>>
> com.datatorrent.stram.cli.ApexCliTest.testAppFromOnlyConfigPackage(ApexCliTest.java:235)
> >>>>>
> >>>>>
> testLaunchAppPackagePropertyPrecedence(com.datatorrent.stram.cli.ApexCliTest)
> >>>>> Time elapsed: 0.045 sec <<< FAILURE!
> >>>>> org.junit.ComparisonFailure: expected:<[app]-default> but
> >>>>> was:<[package]-default>
> >>>>> at
> >>>>>
> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePropertyPrecedence(ApexCliTest.java:143)
> >>>>>
> >>>>>
> testLaunchAppPackageParametersWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
> >>>>> Time elapsed: 0.045 sec <<< ERROR!
> >>>>> com.datatorrent.stram.cli.ApexCli$CliException: No
> applications in
> >>>>> Application Package matching "MyFirstApplication"
> >>>>> at
> >>>>>
> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackageParametersWithConfigPackage(ApexCliTest.java:155)
> >>>>>
> >>>>>
> testLaunchAppPackagePrecedenceWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
> >>>>> Time elapsed: 0.058 sec <<< FAILURE!
> >>>>> org.junit.ComparisonFailure: expected:<[app]-default> but
> >>>>> was:<[package]-default>
> >>>>> at
> >>>>>
> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePrecedenceWithConfigPackage(ApexCliTest.java:204)
> >>>>>
> >>>>>
> >>>>> On Thu, May 17, 2018 at 1:31 PM, Aaron Bossert
> <aaron@punchcyber.com <ma...@punchcyber.com>>
> >>>>> wrote:
> >>>>>
> >>>>>> Thanks, Thomas! I have implemented the fix you suggested
> >>>>>> (@FieldSerializer.BInd) and also created a JIRA ticket:
> >>>>>> https://issues.apache.org/jira/browse/APEXCORE-816
> >>>>>>
> >>>>>> On Thu, May 17, 2018 at 1:31 AM, Thomas Weise
> <thw@apache.org <ma...@apache.org>> wrote:
> >>>>>>
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> It is not necessary to use a different stream codec
> when you have
> >>>>>>> control over the class that is causing the error. You
> can annotate a
> >>>>>>> different field serializer instead, using the
> @FieldSerializer.Bind
> >>>>>>> annotation. Here is an example:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> https://github.com/apache/apex-malhar/blob/2fe2903bfe65055c7fd361a97549b6e7e12d4289/library/src/main/java/org/apache/apex/malhar/lib/window/WindowOption.java#L50
> >>>>>>>
> >>>>>>> It would be good to upgrade Kryo to 4.x in apex-core
> master. Would
> >>>>>>> you like to create a JIRA for it? (
> >>>>>>> http://apex.apache.org/contributing.html)
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> Thomas
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Wed, May 16, 2018 at 10:31 AM, Aaron Bossert <
> >>>>>>> aaron@punchcyber.com <ma...@punchcyber.com>> wrote:
> >>>>>>>
> >>>>>>>> I am writing an application that includes a custom
> object. One of
> >>>>>>>> the fields is of type Instant and seems to not agree
> with the version of
> >>>>>>>> Kryo that is used in Apex (see exception below). I
> now that kryo 4.X
> >>>>>>>> supports serializing Instant fields, but for the life
> of me, cannot seem to
> >>>>>>>> find a way to either get Apex to use a newer version
> of Kryo. Can anyone
> >>>>>>>> help? Perhaps setting a different streamcodec that
> uses the newer version
> >>>>>>>> of kryo would work, but I cannot seem to find an
> example of how to set a
> >>>>>>>> different streamcodec despite references to the two
> (Kryo and Java) that
> >>>>>>>> ship with Apex.
> >>>>>>>>
> >>>>>>>> ERROR
> com.datatorrent.stram.codec.DefaultStatefulStreamCodec -
> >>>>>>>> Catastrophic Error: Execution halted due to Kryo
> exception!
> >>>>>>>> com.esotericsoftware.kryo.KryoException: Class cannot
> be created
> >>>>>>>> (missing no-arg constructor): java.time.Instant
> >>>>>>>> Serialization trace:
> >>>>>>>> systemTime
> >>>>>>>>
> (com.punchcyber.streaming.pipeline.apex.datatypes.evtx.EvtxEvent)
> >>>>>>>> at
> >>>>>>>>
> com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
> >>>>>>>> at
> com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
> >>>>>>>> at
> com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
> >>>>>>>> at
> >>>>>>>>
> com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
> >>>>>>>> at
> >>>>>>>>
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
> >>>>>>>> at
> com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730)
> >>>>>>>> at
> >>>>>>>>
> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113)
> >>>>>>>> at
> >>>>>>>>
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
> >>>>>>>> at
> com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
> >>>>>>>> at com.datatorrent.stram.codec.De
> <http://com.datatorrent.stram.codec.De>
> >>>>>>>>
> faultStatefulStreamCodec.fromDataStatePair(DefaultStatefulStreamCodec.java:98)
> >>>>>>>> at
> >>>>>>>>
> com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.processPayload(BufferServerSubscriber.java:391)
> >>>>>>>> at
> >>>>>>>>
> com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.sweep(BufferServerSubscriber.java:339)
> >>>>>>>> at
> >>>>>>>>
> com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:269)
> >>>>>>>> at
> >>>>>>>>
> com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1429)
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>>
> >>>>>>>> M. Aaron Bossert
> >>>>>>>> (571) 242-4021
> >>>>>>>> Punch Cyber Analytics Group
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>>
> >>>>>> M. Aaron Bossert
> >>>>>> (571) 242-4021
> >>>>>> Punch Cyber Analytics Group
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>>
> >>>>> M. Aaron Bossert
> >>>>> (571) 242-4021
> >>>>> Punch Cyber Analytics Group
> >>>>>
> >>>>>
> >>>>>
> >>>
> >>> --
> >>>
> >>> M. Aaron Bossert
> >>> (571) 242-4021
> >>> Punch Cyber Analytics Group
> >>>
> >>>
> >>>
> >
> > --
> >
> > M. Aaron Bossert
> > (571) 242-4021
> > Punch Cyber Analytics Group
> >
> >
> >
>
> --
>
> M. Aaron Bossert
> (571) 242-4021
> Punch Cyber Analytics Group
>
>
>
> --
>
> M. Aaron Bossert
> (571) 242-4021
> Punch Cyber Analytics Group
>
>
Re: Kryo version and default serializer
Posted by Aaron Bossert <aa...@punchcyber.com>.
cool! I'm on it.
On Tue, Jun 19, 2018 at 11:00 PM Pramod Immaneni <pr...@gmail.com>
wrote:
> Hi Aaron,
>
> That would be great if you can pitch in wherever you can. Let us know how
> we can help. Also, I was going to try the new kryo version later today and
> see if there are any internal errors that are bubbling up to those test
> failures but not getting logged.
>
> Thanks
>
> On Tue, Jun 19, 2018 at 7:54 PM Aaron Bossert <aa...@punchcyber.com>
> wrote:
>
>> P.S. I am happy to roll up my sleeves, but from my initial attempt,
>> errors
>> I was seeing were not obviously related to the Kryo version update (though
>> I am certain they are...at least indirectly because they only show up when
>> the version is changed), in terms of tracking down where the changes would
>> be required...I am not really familiar with the inner workings of Apex
>> yet,
>> though this would definitely be a good excuse to learn...
>>
>> I also saw a bunch of annoying warnings when building related to use of
>> deprecated methods/classes within the com.datatorrent.* namespace...I'd be
>> happy to start tracking some of those down too if it would be helpful...
>>
>> On Tue, Jun 19, 2018 at 10:48 PM Aaron Bossert <aa...@punchcyber.com>
>> wrote:
>>
>> > Pramod,
>> >
>> > In a nutshell, yes. I can set a @FieldSerializer to the fields in
>> > question...the only one I have run into thus far is Instant...but it
>> would
>> > be great to not need to annotate each field this way...Also, I could see
>> > this becoming problematic should I encounter more than a couple of
>> fields
>> > that are not serializable using the older version of Kryo (but
>> presumably
>> > would work with a more current one)...though admitedly, it would not be
>> > that big a deal in the interim. It would be much cleaner to have an
>> update
>> > to the most current version of Kryo, IMHO...though I understand that is
>> > something of a lift. By the way, Kryo 5 just came out yesterday, so if
>> > there is to be effort expended, it might be good to go with the latest
>> > version...though, heads up, I tried it out and there seem to be quite a
>> few
>> > changes that will be needed...new methods, and definitely some replaced
>> > ones...
>> >
>> >
>> >
>> > On Tue, Jun 19, 2018 at 10:37 PM Pramod Immaneni <
>> > pramod.immaneni@gmail.com> wrote:
>> >
>> >> Hi Aaron,
>> >>
>> >> While we are debugging the test failures on dev, I didn't fully
>> >> understand the last question in your email. Looks like you mentioned
>> that
>> >> the workaround for adding the @FieldSerializer to the field is
>> working. Are
>> >> you looking for an alternative to this workaround and trying to set a
>> >> default serializer for all fields of type Instant (the field type you
>> >> mentioned in the earlier email) so that you don't have to set the
>> >> annotation each time?
>> >>
>> >> Thanks
>> >>
>> >> On Tue, Jun 19, 2018 at 12:59 PM Aaron Bossert <aa...@punchcyber.com>
>> >> wrote:
>> >>
>> >>> I sent an e-mail to the dev list...but have not heard any responses.
>> >>> How active is that list? Do you know? Basically, I have a
>> workaround that
>> >>> will deal with those "things" that Kryo v 2.24.0 will not handle (but
>> 4.0.2
>> >>> or 5.0.0-RC1 would) using the @FieldSerializer ...but this is not
>> ideal. I
>> >>> have two options left: go it alone and try to update Kryo in the
>> apex-core,
>> >>> or simply change the default serializer. The hiccup I am running
>> into is
>> >>> that I don't see how (from the documentation) to set the default
>> >>> serializer...perhaps I am missing it. Can someone point me to where
>> that
>> >>> is in the docs?
>> >>>
>> >>> On Mon, Jun 11, 2018 at 4:20 PM Aaron Bossert <aa...@punchcyber.com>
>> >>> wrote:
>> >>>
>> >>>> Ah, you make a good point about the dev list...I am using IntelliJ
>> on a
>> >>>> Mac. When I tried building without any changes, I also encountered
>> >>>> failures. I can re-run the non-modified sources to do a diff and
>> see if
>> >>>> the errors were different...
>> >>>>
>> >>>> Sent from my iPhone
>> >>>>
>> >>>> On Jun 11, 2018, at 15:11, Pramod Immaneni <
>> pramod.immaneni@gmail.com>
>> >>>> wrote:
>> >>>>
>> >>>> Aaron,
>> >>>>
>> >>>> The question might be best served on the dev@apex.apache.org mailing
>> >>>> list as you are starting to make changes to the sources. Nothing
>> obvious
>> >>>> jumps out to be, you don't need hadoop on your system to build the
>> sources
>> >>>> successfully with all the tests passing. The test failures you are
>> seeing
>> >>>> may or may not be related to the kryo change. What happens when you
>> try to
>> >>>> build the original sources with changing the Kryo version? What is
>> your
>> >>>> environment, mac, linux?
>> >>>>
>> >>>> On Mon, Jun 11, 2018 at 12:03 PM Aaron Bossert <aaron@punchcyber.com
>> >
>> >>>> wrote:
>> >>>>
>> >>>>> I know it has been a little while since I brought this up...I pulled
>> >>>>> down the 3.7.1-RELEASE branch from GitHub in order to test out the
>> change
>> >>>>> to Kryo v. 4.0.2 and encountered no problems while building,
>> however, when
>> >>>>> I try to do mvn install, there are several failed tests. I am
>> hoping, that
>> >>>>> other than the obvious (to me, anyway) errors such as not finding
>> >>>>> yarn-site.xml on my laptop which does not have hadoop installed,
>> someone
>> >>>>> may be able to see what is going on?
>> >>>>>
>> >>>>>
>> >>>>>
>> -------------------------------------------------------------------------------
>> >>>>> Test set: com.datatorrent.stram.StramRecoveryTest
>> >>>>>
>> >>>>>
>> -------------------------------------------------------------------------------
>> >>>>> Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
>> 6.372
>> >>>>> sec <<< FAILURE! - in com.datatorrent.stram.StramRecoveryTest
>> >>>>> testWriteAheadLog(com.datatorrent.stram.StramRecoveryTest) Time
>> >>>>> elapsed: 0.115 sec <<< FAILURE!
>> >>>>> java.lang.AssertionError: flush count expected:<1> but was:<2>
>> >>>>> at
>> >>>>>
>> com.datatorrent.stram.StramRecoveryTest.testWriteAheadLog(StramRecoveryTest.java:326)
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> -------------------------------------------------------------------------------
>> >>>>> Test set: com.datatorrent.stram.engine.StatsTest
>> >>>>>
>> >>>>>
>> -------------------------------------------------------------------------------
>> >>>>> Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
>> 21.894
>> >>>>> sec <<< FAILURE! - in com.datatorrent.stram.engine.StatsTest
>> >>>>>
>> testQueueSizeForContainerLocalOperators(com.datatorrent.stram.engine.StatsTest)
>> >>>>> Time elapsed: 3.266 sec <<< FAILURE!
>> >>>>> java.lang.AssertionError: Validate input port queue size -1
>> >>>>> at
>> >>>>>
>> com.datatorrent.stram.engine.StatsTest.baseTestForQueueSize(StatsTest.java:270)
>> >>>>> at
>> >>>>>
>> com.datatorrent.stram.engine.StatsTest.testQueueSizeForContainerLocalOperators(StatsTest.java:285)
>> >>>>>
>> >>>>>
>> >>>>>
>> -------------------------------------------------------------------------------
>> >>>>> Test set: com.datatorrent.stram.StramMiniClusterTest
>> >>>>>
>> >>>>>
>> -------------------------------------------------------------------------------
>> >>>>> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
>> 0.002
>> >>>>> sec <<< FAILURE! - in com.datatorrent.stram.StramMiniClusterTest
>> >>>>> com.datatorrent.stram.StramMiniClusterTest Time elapsed: 0.002 sec
>> >>>>> <<< ERROR!
>> >>>>> java.io.FileNotFoundException:
>> >>>>>
>> /Users/mbossert/Library/Mobile%20Documents/com~apple~CloudDocs/Punch/IdeaProjects/apex-core/engine/target/test-classes/yarn-site.xml
>> >>>>> (No such file or directory)
>> >>>>> at
>> >>>>>
>> com.datatorrent.stram.StramMiniClusterTest.setup(StramMiniClusterTest.java:151)
>> >>>>>
>> >>>>>
>> >>>>>
>> -------------------------------------------------------------------------------
>> >>>>> Test set: com.datatorrent.stram.StramLocalClusterTest
>> >>>>>
>> >>>>>
>> -------------------------------------------------------------------------------
>> >>>>> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed:
>> 15.081
>> >>>>> sec <<< FAILURE! - in com.datatorrent.stram.StramLocalClusterTest
>> >>>>> testAppPath(com.datatorrent.stram.StramLocalClusterTest) Time
>> >>>>> elapsed: 0.138 sec <<< FAILURE!
>> >>>>> java.lang.AssertionError: checkpoint path exists
>> >>>>> at
>> >>>>>
>> com.datatorrent.stram.StramLocalClusterTest.testAppPath(StramLocalClusterTest.java:413)
>> >>>>>
>> >>>>>
>> testLocalClusterInitShutdown(com.datatorrent.stram.StramLocalClusterTest)
>> >>>>> Time elapsed: 5.191 sec <<< FAILURE!
>> >>>>> java.lang.AssertionError:
>> >>>>>
>> ./target/com.datatorrent.stram.StramLocalClusterTest-testLocalClusterInitShutdown.out
>> >>>>> exists
>> >>>>> at
>> >>>>>
>> com.datatorrent.stram.StramLocalClusterTest.testLocalClusterInitShutdown(StramLocalClusterTest.java:117)
>> >>>>>
>> >>>>>
>> >>>>>
>> -------------------------------------------------------------------------------
>> >>>>> Test set: com.datatorrent.stram.client.AppPackageTest
>> >>>>>
>> >>>>>
>> -------------------------------------------------------------------------------
>> >>>>> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed:
>> 0.105
>> >>>>> sec <<< FAILURE! - in com.datatorrent.stram.client.AppPackageTest
>> >>>>> testAppPackage(com.datatorrent.stram.client.AppPackageTest) Time
>> >>>>> elapsed: 0 sec <<< FAILURE!
>> >>>>> java.lang.AssertionError: Number of applications expected:<2> but
>> >>>>> was:<0>
>> >>>>> at
>> >>>>>
>> com.datatorrent.stram.client.AppPackageTest.testAppPackage(AppPackageTest.java:91)
>> >>>>>
>> >>>>>
>> testAppLevelRequiredAndDefaultProperties(com.datatorrent.stram.client.AppPackageTest)
>> >>>>> Time elapsed: 0.001 sec <<< FAILURE!
>> >>>>> java.lang.AssertionError: Should consist of an app called
>> >>>>> MyFirstApplication
>> >>>>> at
>> >>>>>
>> com.datatorrent.stram.client.AppPackageTest.testAppLevelRequiredAndDefaultProperties(AppPackageTest.java:171)
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> -------------------------------------------------------------------------------
>> >>>>> Test set: com.datatorrent.stram.cli.ApexCliTest
>> >>>>>
>> >>>>>
>> -------------------------------------------------------------------------------
>> >>>>> Tests run: 7, Failures: 2, Errors: 2, Skipped: 0, Time elapsed:
>> 0.938
>> >>>>> sec <<< FAILURE! - in com.datatorrent.stram.cli.ApexCliTest
>> >>>>> testAppFromOnlyConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
>> >>>>> Time elapsed: 0.104 sec <<< ERROR!
>> >>>>> com.datatorrent.stram.cli.ApexCli$CliException: No applications in
>> >>>>> Application Package
>> >>>>> at
>> >>>>>
>> com.datatorrent.stram.cli.ApexCliTest.testAppFromOnlyConfigPackage(ApexCliTest.java:235)
>> >>>>>
>> >>>>>
>> testLaunchAppPackagePropertyPrecedence(com.datatorrent.stram.cli.ApexCliTest)
>> >>>>> Time elapsed: 0.045 sec <<< FAILURE!
>> >>>>> org.junit.ComparisonFailure: expected:<[app]-default> but
>> >>>>> was:<[package]-default>
>> >>>>> at
>> >>>>>
>> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePropertyPrecedence(ApexCliTest.java:143)
>> >>>>>
>> >>>>>
>> testLaunchAppPackageParametersWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
>> >>>>> Time elapsed: 0.045 sec <<< ERROR!
>> >>>>> com.datatorrent.stram.cli.ApexCli$CliException: No applications in
>> >>>>> Application Package matching "MyFirstApplication"
>> >>>>> at
>> >>>>>
>> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackageParametersWithConfigPackage(ApexCliTest.java:155)
>> >>>>>
>> >>>>>
>> testLaunchAppPackagePrecedenceWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
>> >>>>> Time elapsed: 0.058 sec <<< FAILURE!
>> >>>>> org.junit.ComparisonFailure: expected:<[app]-default> but
>> >>>>> was:<[package]-default>
>> >>>>> at
>> >>>>>
>> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePrecedenceWithConfigPackage(ApexCliTest.java:204)
>> >>>>>
>> >>>>>
>> >>>>> On Thu, May 17, 2018 at 1:31 PM, Aaron Bossert <
>> aaron@punchcyber.com>
>> >>>>> wrote:
>> >>>>>
>> >>>>>> Thanks, Thomas! I have implemented the fix you suggested
>> >>>>>> (@FieldSerializer.BInd) and also created a JIRA ticket:
>> >>>>>> https://issues.apache.org/jira/browse/APEXCORE-816
>> >>>>>>
>> >>>>>> On Thu, May 17, 2018 at 1:31 AM, Thomas Weise <th...@apache.org>
>> wrote:
>> >>>>>>
>> >>>>>>> Hi,
>> >>>>>>>
>> >>>>>>> It is not necessary to use a different stream codec when you have
>> >>>>>>> control over the class that is causing the error. You can
>> annotate a
>> >>>>>>> different field serializer instead, using the
>> @FieldSerializer.Bind
>> >>>>>>> annotation. Here is an example:
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> https://github.com/apache/apex-malhar/blob/2fe2903bfe65055c7fd361a97549b6e7e12d4289/library/src/main/java/org/apache/apex/malhar/lib/window/WindowOption.java#L50
>> >>>>>>>
>> >>>>>>> It would be good to upgrade Kryo to 4.x in apex-core master. Would
>> >>>>>>> you like to create a JIRA for it? (
>> >>>>>>> http://apex.apache.org/contributing.html)
>> >>>>>>>
>> >>>>>>> Thanks,
>> >>>>>>> Thomas
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> On Wed, May 16, 2018 at 10:31 AM, Aaron Bossert <
>> >>>>>>> aaron@punchcyber.com> wrote:
>> >>>>>>>
>> >>>>>>>> I am writing an application that includes a custom object. One
>> of
>> >>>>>>>> the fields is of type Instant and seems to not agree with the
>> version of
>> >>>>>>>> Kryo that is used in Apex (see exception below). I now that
>> kryo 4.X
>> >>>>>>>> supports serializing Instant fields, but for the life of me,
>> cannot seem to
>> >>>>>>>> find a way to either get Apex to use a newer version of Kryo.
>> Can anyone
>> >>>>>>>> help? Perhaps setting a different streamcodec that uses the
>> newer version
>> >>>>>>>> of kryo would work, but I cannot seem to find an example of how
>> to set a
>> >>>>>>>> different streamcodec despite references to the two (Kryo and
>> Java) that
>> >>>>>>>> ship with Apex.
>> >>>>>>>>
>> >>>>>>>> ERROR com.datatorrent.stram.codec.DefaultStatefulStreamCodec -
>> >>>>>>>> Catastrophic Error: Execution halted due to Kryo exception!
>> >>>>>>>> com.esotericsoftware.kryo.KryoException: Class cannot be created
>> >>>>>>>> (missing no-arg constructor): java.time.Instant
>> >>>>>>>> Serialization trace:
>> >>>>>>>> systemTime
>> >>>>>>>> (com.punchcyber.streaming.pipeline.apex.datatypes.evtx.EvtxEvent)
>> >>>>>>>> at
>> >>>>>>>>
>> com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
>> >>>>>>>> at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
>> >>>>>>>> at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
>> >>>>>>>> at
>> >>>>>>>>
>> com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
>> >>>>>>>> at
>> >>>>>>>>
>> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
>> >>>>>>>> at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730)
>> >>>>>>>> at
>> >>>>>>>>
>> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113)
>> >>>>>>>> at
>> >>>>>>>>
>> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
>> >>>>>>>> at
>> com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
>> >>>>>>>> at com.datatorrent.stram.codec.De
>> >>>>>>>>
>> faultStatefulStreamCodec.fromDataStatePair(DefaultStatefulStreamCodec.java:98)
>> >>>>>>>> at
>> >>>>>>>>
>> com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.processPayload(BufferServerSubscriber.java:391)
>> >>>>>>>> at
>> >>>>>>>>
>> com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.sweep(BufferServerSubscriber.java:339)
>> >>>>>>>> at
>> >>>>>>>>
>> com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:269)
>> >>>>>>>> at
>> >>>>>>>>
>> com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1429)
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> --
>> >>>>>>>>
>> >>>>>>>> M. Aaron Bossert
>> >>>>>>>> (571) 242-4021
>> >>>>>>>> Punch Cyber Analytics Group
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> --
>> >>>>>>
>> >>>>>> M. Aaron Bossert
>> >>>>>> (571) 242-4021
>> >>>>>> Punch Cyber Analytics Group
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>
>> >>>>>
>> >>>>> --
>> >>>>>
>> >>>>> M. Aaron Bossert
>> >>>>> (571) 242-4021
>> >>>>> Punch Cyber Analytics Group
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>
>> >>> --
>> >>>
>> >>> M. Aaron Bossert
>> >>> (571) 242-4021
>> >>> Punch Cyber Analytics Group
>> >>>
>> >>>
>> >>>
>> >
>> > --
>> >
>> > M. Aaron Bossert
>> > (571) 242-4021
>> > Punch Cyber Analytics Group
>> >
>> >
>> >
>>
>> --
>>
>> M. Aaron Bossert
>> (571) 242-4021
>> Punch Cyber Analytics Group
>>
>
--
M. Aaron Bossert
(571) 242-4021
Punch Cyber Analytics Group
Re: Kryo version and default serializer
Posted by Aaron Bossert <aa...@punchcyber.com>.
cool! I'm on it.
On Tue, Jun 19, 2018 at 11:00 PM Pramod Immaneni <pr...@gmail.com>
wrote:
> Hi Aaron,
>
> That would be great if you can pitch in wherever you can. Let us know how
> we can help. Also, I was going to try the new kryo version later today and
> see if there are any internal errors that are bubbling up to those test
> failures but not getting logged.
>
> Thanks
>
> On Tue, Jun 19, 2018 at 7:54 PM Aaron Bossert <aa...@punchcyber.com>
> wrote:
>
>> P.S. I am happy to roll up my sleeves, but from my initial attempt,
>> errors
>> I was seeing were not obviously related to the Kryo version update (though
>> I am certain they are...at least indirectly because they only show up when
>> the version is changed), in terms of tracking down where the changes would
>> be required...I am not really familiar with the inner workings of Apex
>> yet,
>> though this would definitely be a good excuse to learn...
>>
>> I also saw a bunch of annoying warnings when building related to use of
>> deprecated methods/classes within the com.datatorrent.* namespace...I'd be
>> happy to start tracking some of those down too if it would be helpful...
>>
>> On Tue, Jun 19, 2018 at 10:48 PM Aaron Bossert <aa...@punchcyber.com>
>> wrote:
>>
>> > Pramod,
>> >
>> > In a nutshell, yes. I can set a @FieldSerializer to the fields in
>> > question...the only one I have run into thus far is Instant...but it
>> would
>> > be great to not need to annotate each field this way...Also, I could see
>> > this becoming problematic should I encounter more than a couple of
>> fields
>> > that are not serializable using the older version of Kryo (but
>> presumably
>> > would work with a more current one)...though admitedly, it would not be
>> > that big a deal in the interim. It would be much cleaner to have an
>> update
>> > to the most current version of Kryo, IMHO...though I understand that is
>> > something of a lift. By the way, Kryo 5 just came out yesterday, so if
>> > there is to be effort expended, it might be good to go with the latest
>> > version...though, heads up, I tried it out and there seem to be quite a
>> few
>> > changes that will be needed...new methods, and definitely some replaced
>> > ones...
>> >
>> >
>> >
>> > On Tue, Jun 19, 2018 at 10:37 PM Pramod Immaneni <
>> > pramod.immaneni@gmail.com> wrote:
>> >
>> >> Hi Aaron,
>> >>
>> >> While we are debugging the test failures on dev, I didn't fully
>> >> understand the last question in your email. Looks like you mentioned
>> that
>> >> the workaround for adding the @FieldSerializer to the field is
>> working. Are
>> >> you looking for an alternative to this workaround and trying to set a
>> >> default serializer for all fields of type Instant (the field type you
>> >> mentioned in the earlier email) so that you don't have to set the
>> >> annotation each time?
>> >>
>> >> Thanks
>> >>
>> >> On Tue, Jun 19, 2018 at 12:59 PM Aaron Bossert <aa...@punchcyber.com>
>> >> wrote:
>> >>
>> >>> I sent an e-mail to the dev list...but have not heard any responses.
>> >>> How active is that list? Do you know? Basically, I have a
>> workaround that
>> >>> will deal with those "things" that Kryo v 2.24.0 will not handle (but
>> 4.0.2
>> >>> or 5.0.0-RC1 would) using the @FieldSerializer ...but this is not
>> ideal. I
>> >>> have two options left: go it alone and try to update Kryo in the
>> apex-core,
>> >>> or simply change the default serializer. The hiccup I am running
>> into is
>> >>> that I don't see how (from the documentation) to set the default
>> >>> serializer...perhaps I am missing it. Can someone point me to where
>> that
>> >>> is in the docs?
>> >>>
>> >>> On Mon, Jun 11, 2018 at 4:20 PM Aaron Bossert <aa...@punchcyber.com>
>> >>> wrote:
>> >>>
>> >>>> Ah, you make a good point about the dev list...I am using IntelliJ
>> on a
>> >>>> Mac. When I tried building without any changes, I also encountered
>> >>>> failures. I can re-run the non-modified sources to do a diff and
>> see if
>> >>>> the errors were different...
>> >>>>
>> >>>> Sent from my iPhone
>> >>>>
>> >>>> On Jun 11, 2018, at 15:11, Pramod Immaneni <
>> pramod.immaneni@gmail.com>
>> >>>> wrote:
>> >>>>
>> >>>> Aaron,
>> >>>>
>> >>>> The question might be best served on the dev@apex.apache.org mailing
>> >>>> list as you are starting to make changes to the sources. Nothing
>> obvious
>> >>>> jumps out to be, you don't need hadoop on your system to build the
>> sources
>> >>>> successfully with all the tests passing. The test failures you are
>> seeing
>> >>>> may or may not be related to the kryo change. What happens when you
>> try to
>> >>>> build the original sources with changing the Kryo version? What is
>> your
>> >>>> environment, mac, linux?
>> >>>>
>> >>>> On Mon, Jun 11, 2018 at 12:03 PM Aaron Bossert <aaron@punchcyber.com
>> >
>> >>>> wrote:
>> >>>>
>> >>>>> I know it has been a little while since I brought this up...I pulled
>> >>>>> down the 3.7.1-RELEASE branch from GitHub in order to test out the
>> change
>> >>>>> to Kryo v. 4.0.2 and encountered no problems while building,
>> however, when
>> >>>>> I try to do mvn install, there are several failed tests. I am
>> hoping, that
>> >>>>> other than the obvious (to me, anyway) errors such as not finding
>> >>>>> yarn-site.xml on my laptop which does not have hadoop installed,
>> someone
>> >>>>> may be able to see what is going on?
>> >>>>>
>> >>>>>
>> >>>>>
>> -------------------------------------------------------------------------------
>> >>>>> Test set: com.datatorrent.stram.StramRecoveryTest
>> >>>>>
>> >>>>>
>> -------------------------------------------------------------------------------
>> >>>>> Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
>> 6.372
>> >>>>> sec <<< FAILURE! - in com.datatorrent.stram.StramRecoveryTest
>> >>>>> testWriteAheadLog(com.datatorrent.stram.StramRecoveryTest) Time
>> >>>>> elapsed: 0.115 sec <<< FAILURE!
>> >>>>> java.lang.AssertionError: flush count expected:<1> but was:<2>
>> >>>>> at
>> >>>>>
>> com.datatorrent.stram.StramRecoveryTest.testWriteAheadLog(StramRecoveryTest.java:326)
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> -------------------------------------------------------------------------------
>> >>>>> Test set: com.datatorrent.stram.engine.StatsTest
>> >>>>>
>> >>>>>
>> -------------------------------------------------------------------------------
>> >>>>> Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
>> 21.894
>> >>>>> sec <<< FAILURE! - in com.datatorrent.stram.engine.StatsTest
>> >>>>>
>> testQueueSizeForContainerLocalOperators(com.datatorrent.stram.engine.StatsTest)
>> >>>>> Time elapsed: 3.266 sec <<< FAILURE!
>> >>>>> java.lang.AssertionError: Validate input port queue size -1
>> >>>>> at
>> >>>>>
>> com.datatorrent.stram.engine.StatsTest.baseTestForQueueSize(StatsTest.java:270)
>> >>>>> at
>> >>>>>
>> com.datatorrent.stram.engine.StatsTest.testQueueSizeForContainerLocalOperators(StatsTest.java:285)
>> >>>>>
>> >>>>>
>> >>>>>
>> -------------------------------------------------------------------------------
>> >>>>> Test set: com.datatorrent.stram.StramMiniClusterTest
>> >>>>>
>> >>>>>
>> -------------------------------------------------------------------------------
>> >>>>> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
>> 0.002
>> >>>>> sec <<< FAILURE! - in com.datatorrent.stram.StramMiniClusterTest
>> >>>>> com.datatorrent.stram.StramMiniClusterTest Time elapsed: 0.002 sec
>> >>>>> <<< ERROR!
>> >>>>> java.io.FileNotFoundException:
>> >>>>>
>> /Users/mbossert/Library/Mobile%20Documents/com~apple~CloudDocs/Punch/IdeaProjects/apex-core/engine/target/test-classes/yarn-site.xml
>> >>>>> (No such file or directory)
>> >>>>> at
>> >>>>>
>> com.datatorrent.stram.StramMiniClusterTest.setup(StramMiniClusterTest.java:151)
>> >>>>>
>> >>>>>
>> >>>>>
>> -------------------------------------------------------------------------------
>> >>>>> Test set: com.datatorrent.stram.StramLocalClusterTest
>> >>>>>
>> >>>>>
>> -------------------------------------------------------------------------------
>> >>>>> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed:
>> 15.081
>> >>>>> sec <<< FAILURE! - in com.datatorrent.stram.StramLocalClusterTest
>> >>>>> testAppPath(com.datatorrent.stram.StramLocalClusterTest) Time
>> >>>>> elapsed: 0.138 sec <<< FAILURE!
>> >>>>> java.lang.AssertionError: checkpoint path exists
>> >>>>> at
>> >>>>>
>> com.datatorrent.stram.StramLocalClusterTest.testAppPath(StramLocalClusterTest.java:413)
>> >>>>>
>> >>>>>
>> testLocalClusterInitShutdown(com.datatorrent.stram.StramLocalClusterTest)
>> >>>>> Time elapsed: 5.191 sec <<< FAILURE!
>> >>>>> java.lang.AssertionError:
>> >>>>>
>> ./target/com.datatorrent.stram.StramLocalClusterTest-testLocalClusterInitShutdown.out
>> >>>>> exists
>> >>>>> at
>> >>>>>
>> com.datatorrent.stram.StramLocalClusterTest.testLocalClusterInitShutdown(StramLocalClusterTest.java:117)
>> >>>>>
>> >>>>>
>> >>>>>
>> -------------------------------------------------------------------------------
>> >>>>> Test set: com.datatorrent.stram.client.AppPackageTest
>> >>>>>
>> >>>>>
>> -------------------------------------------------------------------------------
>> >>>>> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed:
>> 0.105
>> >>>>> sec <<< FAILURE! - in com.datatorrent.stram.client.AppPackageTest
>> >>>>> testAppPackage(com.datatorrent.stram.client.AppPackageTest) Time
>> >>>>> elapsed: 0 sec <<< FAILURE!
>> >>>>> java.lang.AssertionError: Number of applications expected:<2> but
>> >>>>> was:<0>
>> >>>>> at
>> >>>>>
>> com.datatorrent.stram.client.AppPackageTest.testAppPackage(AppPackageTest.java:91)
>> >>>>>
>> >>>>>
>> testAppLevelRequiredAndDefaultProperties(com.datatorrent.stram.client.AppPackageTest)
>> >>>>> Time elapsed: 0.001 sec <<< FAILURE!
>> >>>>> java.lang.AssertionError: Should consist of an app called
>> >>>>> MyFirstApplication
>> >>>>> at
>> >>>>>
>> com.datatorrent.stram.client.AppPackageTest.testAppLevelRequiredAndDefaultProperties(AppPackageTest.java:171)
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> -------------------------------------------------------------------------------
>> >>>>> Test set: com.datatorrent.stram.cli.ApexCliTest
>> >>>>>
>> >>>>>
>> -------------------------------------------------------------------------------
>> >>>>> Tests run: 7, Failures: 2, Errors: 2, Skipped: 0, Time elapsed:
>> 0.938
>> >>>>> sec <<< FAILURE! - in com.datatorrent.stram.cli.ApexCliTest
>> >>>>> testAppFromOnlyConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
>> >>>>> Time elapsed: 0.104 sec <<< ERROR!
>> >>>>> com.datatorrent.stram.cli.ApexCli$CliException: No applications in
>> >>>>> Application Package
>> >>>>> at
>> >>>>>
>> com.datatorrent.stram.cli.ApexCliTest.testAppFromOnlyConfigPackage(ApexCliTest.java:235)
>> >>>>>
>> >>>>>
>> testLaunchAppPackagePropertyPrecedence(com.datatorrent.stram.cli.ApexCliTest)
>> >>>>> Time elapsed: 0.045 sec <<< FAILURE!
>> >>>>> org.junit.ComparisonFailure: expected:<[app]-default> but
>> >>>>> was:<[package]-default>
>> >>>>> at
>> >>>>>
>> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePropertyPrecedence(ApexCliTest.java:143)
>> >>>>>
>> >>>>>
>> testLaunchAppPackageParametersWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
>> >>>>> Time elapsed: 0.045 sec <<< ERROR!
>> >>>>> com.datatorrent.stram.cli.ApexCli$CliException: No applications in
>> >>>>> Application Package matching "MyFirstApplication"
>> >>>>> at
>> >>>>>
>> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackageParametersWithConfigPackage(ApexCliTest.java:155)
>> >>>>>
>> >>>>>
>> testLaunchAppPackagePrecedenceWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
>> >>>>> Time elapsed: 0.058 sec <<< FAILURE!
>> >>>>> org.junit.ComparisonFailure: expected:<[app]-default> but
>> >>>>> was:<[package]-default>
>> >>>>> at
>> >>>>>
>> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePrecedenceWithConfigPackage(ApexCliTest.java:204)
>> >>>>>
>> >>>>>
>> >>>>> On Thu, May 17, 2018 at 1:31 PM, Aaron Bossert <
>> aaron@punchcyber.com>
>> >>>>> wrote:
>> >>>>>
>> >>>>>> Thanks, Thomas! I have implemented the fix you suggested
>> >>>>>> (@FieldSerializer.BInd) and also created a JIRA ticket:
>> >>>>>> https://issues.apache.org/jira/browse/APEXCORE-816
>> >>>>>>
>> >>>>>> On Thu, May 17, 2018 at 1:31 AM, Thomas Weise <th...@apache.org>
>> wrote:
>> >>>>>>
>> >>>>>>> Hi,
>> >>>>>>>
>> >>>>>>> It is not necessary to use a different stream codec when you have
>> >>>>>>> control over the class that is causing the error. You can
>> annotate a
>> >>>>>>> different field serializer instead, using the
>> @FieldSerializer.Bind
>> >>>>>>> annotation. Here is an example:
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> https://github.com/apache/apex-malhar/blob/2fe2903bfe65055c7fd361a97549b6e7e12d4289/library/src/main/java/org/apache/apex/malhar/lib/window/WindowOption.java#L50
>> >>>>>>>
>> >>>>>>> It would be good to upgrade Kryo to 4.x in apex-core master. Would
>> >>>>>>> you like to create a JIRA for it? (
>> >>>>>>> http://apex.apache.org/contributing.html)
>> >>>>>>>
>> >>>>>>> Thanks,
>> >>>>>>> Thomas
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> On Wed, May 16, 2018 at 10:31 AM, Aaron Bossert <
>> >>>>>>> aaron@punchcyber.com> wrote:
>> >>>>>>>
>> >>>>>>>> I am writing an application that includes a custom object. One
>> of
>> >>>>>>>> the fields is of type Instant and seems to not agree with the
>> version of
>> >>>>>>>> Kryo that is used in Apex (see exception below). I now that
>> kryo 4.X
>> >>>>>>>> supports serializing Instant fields, but for the life of me,
>> cannot seem to
>> >>>>>>>> find a way to either get Apex to use a newer version of Kryo.
>> Can anyone
>> >>>>>>>> help? Perhaps setting a different streamcodec that uses the
>> newer version
>> >>>>>>>> of kryo would work, but I cannot seem to find an example of how
>> to set a
>> >>>>>>>> different streamcodec despite references to the two (Kryo and
>> Java) that
>> >>>>>>>> ship with Apex.
>> >>>>>>>>
>> >>>>>>>> ERROR com.datatorrent.stram.codec.DefaultStatefulStreamCodec -
>> >>>>>>>> Catastrophic Error: Execution halted due to Kryo exception!
>> >>>>>>>> com.esotericsoftware.kryo.KryoException: Class cannot be created
>> >>>>>>>> (missing no-arg constructor): java.time.Instant
>> >>>>>>>> Serialization trace:
>> >>>>>>>> systemTime
>> >>>>>>>> (com.punchcyber.streaming.pipeline.apex.datatypes.evtx.EvtxEvent)
>> >>>>>>>> at
>> >>>>>>>>
>> com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
>> >>>>>>>> at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
>> >>>>>>>> at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
>> >>>>>>>> at
>> >>>>>>>>
>> com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
>> >>>>>>>> at
>> >>>>>>>>
>> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
>> >>>>>>>> at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730)
>> >>>>>>>> at
>> >>>>>>>>
>> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113)
>> >>>>>>>> at
>> >>>>>>>>
>> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
>> >>>>>>>> at
>> com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
>> >>>>>>>> at com.datatorrent.stram.codec.De
>> >>>>>>>>
>> faultStatefulStreamCodec.fromDataStatePair(DefaultStatefulStreamCodec.java:98)
>> >>>>>>>> at
>> >>>>>>>>
>> com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.processPayload(BufferServerSubscriber.java:391)
>> >>>>>>>> at
>> >>>>>>>>
>> com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.sweep(BufferServerSubscriber.java:339)
>> >>>>>>>> at
>> >>>>>>>>
>> com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:269)
>> >>>>>>>> at
>> >>>>>>>>
>> com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1429)
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> --
>> >>>>>>>>
>> >>>>>>>> M. Aaron Bossert
>> >>>>>>>> (571) 242-4021
>> >>>>>>>> Punch Cyber Analytics Group
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> --
>> >>>>>>
>> >>>>>> M. Aaron Bossert
>> >>>>>> (571) 242-4021
>> >>>>>> Punch Cyber Analytics Group
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>
>> >>>>>
>> >>>>> --
>> >>>>>
>> >>>>> M. Aaron Bossert
>> >>>>> (571) 242-4021
>> >>>>> Punch Cyber Analytics Group
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>
>> >>> --
>> >>>
>> >>> M. Aaron Bossert
>> >>> (571) 242-4021
>> >>> Punch Cyber Analytics Group
>> >>>
>> >>>
>> >>>
>> >
>> > --
>> >
>> > M. Aaron Bossert
>> > (571) 242-4021
>> > Punch Cyber Analytics Group
>> >
>> >
>> >
>>
>> --
>>
>> M. Aaron Bossert
>> (571) 242-4021
>> Punch Cyber Analytics Group
>>
>
--
M. Aaron Bossert
(571) 242-4021
Punch Cyber Analytics Group
Re: Kryo version and default serializer
Posted by Pramod Immaneni <pr...@gmail.com>.
Hi Aaron,
That would be great if you can pitch in wherever you can. Let us know how
we can help. Also, I was going to try the new kryo version later today and
see if there are any internal errors that are bubbling up to those test
failures but not getting logged.
Thanks
On Tue, Jun 19, 2018 at 7:54 PM Aaron Bossert <aa...@punchcyber.com> wrote:
> P.S. I am happy to roll up my sleeves, but from my initial attempt, errors
> I was seeing were not obviously related to the Kryo version update (though
> I am certain they are...at least indirectly because they only show up when
> the version is changed), in terms of tracking down where the changes would
> be required...I am not really familiar with the inner workings of Apex yet,
> though this would definitely be a good excuse to learn...
>
> I also saw a bunch of annoying warnings when building related to use of
> deprecated methods/classes within the com.datatorrent.* namespace...I'd be
> happy to start tracking some of those down too if it would be helpful...
>
> On Tue, Jun 19, 2018 at 10:48 PM Aaron Bossert <aa...@punchcyber.com>
> wrote:
>
> > Pramod,
> >
> > In a nutshell, yes. I can set a @FieldSerializer to the fields in
> > question...the only one I have run into thus far is Instant...but it
> would
> > be great to not need to annotate each field this way...Also, I could see
> > this becoming problematic should I encounter more than a couple of fields
> > that are not serializable using the older version of Kryo (but presumably
> > would work with a more current one)...though admitedly, it would not be
> > that big a deal in the interim. It would be much cleaner to have an
> update
> > to the most current version of Kryo, IMHO...though I understand that is
> > something of a lift. By the way, Kryo 5 just came out yesterday, so if
> > there is to be effort expended, it might be good to go with the latest
> > version...though, heads up, I tried it out and there seem to be quite a
> few
> > changes that will be needed...new methods, and definitely some replaced
> > ones...
> >
> >
> >
> > On Tue, Jun 19, 2018 at 10:37 PM Pramod Immaneni <
> > pramod.immaneni@gmail.com> wrote:
> >
> >> Hi Aaron,
> >>
> >> While we are debugging the test failures on dev, I didn't fully
> >> understand the last question in your email. Looks like you mentioned
> that
> >> the workaround for adding the @FieldSerializer to the field is working.
> Are
> >> you looking for an alternative to this workaround and trying to set a
> >> default serializer for all fields of type Instant (the field type you
> >> mentioned in the earlier email) so that you don't have to set the
> >> annotation each time?
> >>
> >> Thanks
> >>
> >> On Tue, Jun 19, 2018 at 12:59 PM Aaron Bossert <aa...@punchcyber.com>
> >> wrote:
> >>
> >>> I sent an e-mail to the dev list...but have not heard any responses.
> >>> How active is that list? Do you know? Basically, I have a workaround
> that
> >>> will deal with those "things" that Kryo v 2.24.0 will not handle (but
> 4.0.2
> >>> or 5.0.0-RC1 would) using the @FieldSerializer ...but this is not
> ideal. I
> >>> have two options left: go it alone and try to update Kryo in the
> apex-core,
> >>> or simply change the default serializer. The hiccup I am running into
> is
> >>> that I don't see how (from the documentation) to set the default
> >>> serializer...perhaps I am missing it. Can someone point me to where
> that
> >>> is in the docs?
> >>>
> >>> On Mon, Jun 11, 2018 at 4:20 PM Aaron Bossert <aa...@punchcyber.com>
> >>> wrote:
> >>>
> >>>> Ah, you make a good point about the dev list...I am using IntelliJ on
> a
> >>>> Mac. When I tried building without any changes, I also encountered
> >>>> failures. I can re-run the non-modified sources to do a diff and see
> if
> >>>> the errors were different...
> >>>>
> >>>> Sent from my iPhone
> >>>>
> >>>> On Jun 11, 2018, at 15:11, Pramod Immaneni <pramod.immaneni@gmail.com
> >
> >>>> wrote:
> >>>>
> >>>> Aaron,
> >>>>
> >>>> The question might be best served on the dev@apex.apache.org mailing
> >>>> list as you are starting to make changes to the sources. Nothing
> obvious
> >>>> jumps out to be, you don't need hadoop on your system to build the
> sources
> >>>> successfully with all the tests passing. The test failures you are
> seeing
> >>>> may or may not be related to the kryo change. What happens when you
> try to
> >>>> build the original sources with changing the Kryo version? What is
> your
> >>>> environment, mac, linux?
> >>>>
> >>>> On Mon, Jun 11, 2018 at 12:03 PM Aaron Bossert <aa...@punchcyber.com>
> >>>> wrote:
> >>>>
> >>>>> I know it has been a little while since I brought this up...I pulled
> >>>>> down the 3.7.1-RELEASE branch from GitHub in order to test out the
> change
> >>>>> to Kryo v. 4.0.2 and encountered no problems while building,
> however, when
> >>>>> I try to do mvn install, there are several failed tests. I am
> hoping, that
> >>>>> other than the obvious (to me, anyway) errors such as not finding
> >>>>> yarn-site.xml on my laptop which does not have hadoop installed,
> someone
> >>>>> may be able to see what is going on?
> >>>>>
> >>>>>
> >>>>>
> -------------------------------------------------------------------------------
> >>>>> Test set: com.datatorrent.stram.StramRecoveryTest
> >>>>>
> >>>>>
> -------------------------------------------------------------------------------
> >>>>> Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 6.372
> >>>>> sec <<< FAILURE! - in com.datatorrent.stram.StramRecoveryTest
> >>>>> testWriteAheadLog(com.datatorrent.stram.StramRecoveryTest) Time
> >>>>> elapsed: 0.115 sec <<< FAILURE!
> >>>>> java.lang.AssertionError: flush count expected:<1> but was:<2>
> >>>>> at
> >>>>>
> com.datatorrent.stram.StramRecoveryTest.testWriteAheadLog(StramRecoveryTest.java:326)
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> -------------------------------------------------------------------------------
> >>>>> Test set: com.datatorrent.stram.engine.StatsTest
> >>>>>
> >>>>>
> -------------------------------------------------------------------------------
> >>>>> Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 21.894
> >>>>> sec <<< FAILURE! - in com.datatorrent.stram.engine.StatsTest
> >>>>>
> testQueueSizeForContainerLocalOperators(com.datatorrent.stram.engine.StatsTest)
> >>>>> Time elapsed: 3.266 sec <<< FAILURE!
> >>>>> java.lang.AssertionError: Validate input port queue size -1
> >>>>> at
> >>>>>
> com.datatorrent.stram.engine.StatsTest.baseTestForQueueSize(StatsTest.java:270)
> >>>>> at
> >>>>>
> com.datatorrent.stram.engine.StatsTest.testQueueSizeForContainerLocalOperators(StatsTest.java:285)
> >>>>>
> >>>>>
> >>>>>
> -------------------------------------------------------------------------------
> >>>>> Test set: com.datatorrent.stram.StramMiniClusterTest
> >>>>>
> >>>>>
> -------------------------------------------------------------------------------
> >>>>> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.002
> >>>>> sec <<< FAILURE! - in com.datatorrent.stram.StramMiniClusterTest
> >>>>> com.datatorrent.stram.StramMiniClusterTest Time elapsed: 0.002 sec
> >>>>> <<< ERROR!
> >>>>> java.io.FileNotFoundException:
> >>>>>
> /Users/mbossert/Library/Mobile%20Documents/com~apple~CloudDocs/Punch/IdeaProjects/apex-core/engine/target/test-classes/yarn-site.xml
> >>>>> (No such file or directory)
> >>>>> at
> >>>>>
> com.datatorrent.stram.StramMiniClusterTest.setup(StramMiniClusterTest.java:151)
> >>>>>
> >>>>>
> >>>>>
> -------------------------------------------------------------------------------
> >>>>> Test set: com.datatorrent.stram.StramLocalClusterTest
> >>>>>
> >>>>>
> -------------------------------------------------------------------------------
> >>>>> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed:
> 15.081
> >>>>> sec <<< FAILURE! - in com.datatorrent.stram.StramLocalClusterTest
> >>>>> testAppPath(com.datatorrent.stram.StramLocalClusterTest) Time
> >>>>> elapsed: 0.138 sec <<< FAILURE!
> >>>>> java.lang.AssertionError: checkpoint path exists
> >>>>> at
> >>>>>
> com.datatorrent.stram.StramLocalClusterTest.testAppPath(StramLocalClusterTest.java:413)
> >>>>>
> >>>>>
> testLocalClusterInitShutdown(com.datatorrent.stram.StramLocalClusterTest)
> >>>>> Time elapsed: 5.191 sec <<< FAILURE!
> >>>>> java.lang.AssertionError:
> >>>>>
> ./target/com.datatorrent.stram.StramLocalClusterTest-testLocalClusterInitShutdown.out
> >>>>> exists
> >>>>> at
> >>>>>
> com.datatorrent.stram.StramLocalClusterTest.testLocalClusterInitShutdown(StramLocalClusterTest.java:117)
> >>>>>
> >>>>>
> >>>>>
> -------------------------------------------------------------------------------
> >>>>> Test set: com.datatorrent.stram.client.AppPackageTest
> >>>>>
> >>>>>
> -------------------------------------------------------------------------------
> >>>>> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.105
> >>>>> sec <<< FAILURE! - in com.datatorrent.stram.client.AppPackageTest
> >>>>> testAppPackage(com.datatorrent.stram.client.AppPackageTest) Time
> >>>>> elapsed: 0 sec <<< FAILURE!
> >>>>> java.lang.AssertionError: Number of applications expected:<2> but
> >>>>> was:<0>
> >>>>> at
> >>>>>
> com.datatorrent.stram.client.AppPackageTest.testAppPackage(AppPackageTest.java:91)
> >>>>>
> >>>>>
> testAppLevelRequiredAndDefaultProperties(com.datatorrent.stram.client.AppPackageTest)
> >>>>> Time elapsed: 0.001 sec <<< FAILURE!
> >>>>> java.lang.AssertionError: Should consist of an app called
> >>>>> MyFirstApplication
> >>>>> at
> >>>>>
> com.datatorrent.stram.client.AppPackageTest.testAppLevelRequiredAndDefaultProperties(AppPackageTest.java:171)
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> -------------------------------------------------------------------------------
> >>>>> Test set: com.datatorrent.stram.cli.ApexCliTest
> >>>>>
> >>>>>
> -------------------------------------------------------------------------------
> >>>>> Tests run: 7, Failures: 2, Errors: 2, Skipped: 0, Time elapsed: 0.938
> >>>>> sec <<< FAILURE! - in com.datatorrent.stram.cli.ApexCliTest
> >>>>> testAppFromOnlyConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
> >>>>> Time elapsed: 0.104 sec <<< ERROR!
> >>>>> com.datatorrent.stram.cli.ApexCli$CliException: No applications in
> >>>>> Application Package
> >>>>> at
> >>>>>
> com.datatorrent.stram.cli.ApexCliTest.testAppFromOnlyConfigPackage(ApexCliTest.java:235)
> >>>>>
> >>>>>
> testLaunchAppPackagePropertyPrecedence(com.datatorrent.stram.cli.ApexCliTest)
> >>>>> Time elapsed: 0.045 sec <<< FAILURE!
> >>>>> org.junit.ComparisonFailure: expected:<[app]-default> but
> >>>>> was:<[package]-default>
> >>>>> at
> >>>>>
> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePropertyPrecedence(ApexCliTest.java:143)
> >>>>>
> >>>>>
> testLaunchAppPackageParametersWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
> >>>>> Time elapsed: 0.045 sec <<< ERROR!
> >>>>> com.datatorrent.stram.cli.ApexCli$CliException: No applications in
> >>>>> Application Package matching "MyFirstApplication"
> >>>>> at
> >>>>>
> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackageParametersWithConfigPackage(ApexCliTest.java:155)
> >>>>>
> >>>>>
> testLaunchAppPackagePrecedenceWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
> >>>>> Time elapsed: 0.058 sec <<< FAILURE!
> >>>>> org.junit.ComparisonFailure: expected:<[app]-default> but
> >>>>> was:<[package]-default>
> >>>>> at
> >>>>>
> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePrecedenceWithConfigPackage(ApexCliTest.java:204)
> >>>>>
> >>>>>
> >>>>> On Thu, May 17, 2018 at 1:31 PM, Aaron Bossert <aaron@punchcyber.com
> >
> >>>>> wrote:
> >>>>>
> >>>>>> Thanks, Thomas! I have implemented the fix you suggested
> >>>>>> (@FieldSerializer.BInd) and also created a JIRA ticket:
> >>>>>> https://issues.apache.org/jira/browse/APEXCORE-816
> >>>>>>
> >>>>>> On Thu, May 17, 2018 at 1:31 AM, Thomas Weise <th...@apache.org>
> wrote:
> >>>>>>
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> It is not necessary to use a different stream codec when you have
> >>>>>>> control over the class that is causing the error. You can annotate
> a
> >>>>>>> different field serializer instead, using the @FieldSerializer.Bind
> >>>>>>> annotation. Here is an example:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> https://github.com/apache/apex-malhar/blob/2fe2903bfe65055c7fd361a97549b6e7e12d4289/library/src/main/java/org/apache/apex/malhar/lib/window/WindowOption.java#L50
> >>>>>>>
> >>>>>>> It would be good to upgrade Kryo to 4.x in apex-core master. Would
> >>>>>>> you like to create a JIRA for it? (
> >>>>>>> http://apex.apache.org/contributing.html)
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> Thomas
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Wed, May 16, 2018 at 10:31 AM, Aaron Bossert <
> >>>>>>> aaron@punchcyber.com> wrote:
> >>>>>>>
> >>>>>>>> I am writing an application that includes a custom object. One of
> >>>>>>>> the fields is of type Instant and seems to not agree with the
> version of
> >>>>>>>> Kryo that is used in Apex (see exception below). I now that kryo
> 4.X
> >>>>>>>> supports serializing Instant fields, but for the life of me,
> cannot seem to
> >>>>>>>> find a way to either get Apex to use a newer version of Kryo.
> Can anyone
> >>>>>>>> help? Perhaps setting a different streamcodec that uses the
> newer version
> >>>>>>>> of kryo would work, but I cannot seem to find an example of how
> to set a
> >>>>>>>> different streamcodec despite references to the two (Kryo and
> Java) that
> >>>>>>>> ship with Apex.
> >>>>>>>>
> >>>>>>>> ERROR com.datatorrent.stram.codec.DefaultStatefulStreamCodec -
> >>>>>>>> Catastrophic Error: Execution halted due to Kryo exception!
> >>>>>>>> com.esotericsoftware.kryo.KryoException: Class cannot be created
> >>>>>>>> (missing no-arg constructor): java.time.Instant
> >>>>>>>> Serialization trace:
> >>>>>>>> systemTime
> >>>>>>>> (com.punchcyber.streaming.pipeline.apex.datatypes.evtx.EvtxEvent)
> >>>>>>>> at
> >>>>>>>>
> com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
> >>>>>>>> at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
> >>>>>>>> at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
> >>>>>>>> at
> >>>>>>>>
> com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
> >>>>>>>> at
> >>>>>>>>
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
> >>>>>>>> at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730)
> >>>>>>>> at
> >>>>>>>>
> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113)
> >>>>>>>> at
> >>>>>>>>
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
> >>>>>>>> at
> com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
> >>>>>>>> at com.datatorrent.stram.codec.De
> >>>>>>>>
> faultStatefulStreamCodec.fromDataStatePair(DefaultStatefulStreamCodec.java:98)
> >>>>>>>> at
> >>>>>>>>
> com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.processPayload(BufferServerSubscriber.java:391)
> >>>>>>>> at
> >>>>>>>>
> com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.sweep(BufferServerSubscriber.java:339)
> >>>>>>>> at
> >>>>>>>> com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:269)
> >>>>>>>> at
> >>>>>>>>
> com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1429)
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>>
> >>>>>>>> M. Aaron Bossert
> >>>>>>>> (571) 242-4021
> >>>>>>>> Punch Cyber Analytics Group
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>>
> >>>>>> M. Aaron Bossert
> >>>>>> (571) 242-4021
> >>>>>> Punch Cyber Analytics Group
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>>
> >>>>> M. Aaron Bossert
> >>>>> (571) 242-4021
> >>>>> Punch Cyber Analytics Group
> >>>>>
> >>>>>
> >>>>>
> >>>
> >>> --
> >>>
> >>> M. Aaron Bossert
> >>> (571) 242-4021
> >>> Punch Cyber Analytics Group
> >>>
> >>>
> >>>
> >
> > --
> >
> > M. Aaron Bossert
> > (571) 242-4021
> > Punch Cyber Analytics Group
> >
> >
> >
>
> --
>
> M. Aaron Bossert
> (571) 242-4021
> Punch Cyber Analytics Group
>
Re: Kryo version and default serializer
Posted by Pramod Immaneni <pr...@gmail.com>.
Hi Aaron,
That would be great if you can pitch in wherever you can. Let us know how
we can help. Also, I was going to try the new kryo version later today and
see if there are any internal errors that are bubbling up to those test
failures but not getting logged.
Thanks
On Tue, Jun 19, 2018 at 7:54 PM Aaron Bossert <aa...@punchcyber.com> wrote:
> P.S. I am happy to roll up my sleeves, but from my initial attempt, errors
> I was seeing were not obviously related to the Kryo version update (though
> I am certain they are...at least indirectly because they only show up when
> the version is changed), in terms of tracking down where the changes would
> be required...I am not really familiar with the inner workings of Apex yet,
> though this would definitely be a good excuse to learn...
>
> I also saw a bunch of annoying warnings when building related to use of
> deprecated methods/classes within the com.datatorrent.* namespace...I'd be
> happy to start tracking some of those down too if it would be helpful...
>
> On Tue, Jun 19, 2018 at 10:48 PM Aaron Bossert <aa...@punchcyber.com>
> wrote:
>
> > Pramod,
> >
> > In a nutshell, yes. I can set a @FieldSerializer to the fields in
> > question...the only one I have run into thus far is Instant...but it
> would
> > be great to not need to annotate each field this way...Also, I could see
> > this becoming problematic should I encounter more than a couple of fields
> > that are not serializable using the older version of Kryo (but presumably
> > would work with a more current one)...though admitedly, it would not be
> > that big a deal in the interim. It would be much cleaner to have an
> update
> > to the most current version of Kryo, IMHO...though I understand that is
> > something of a lift. By the way, Kryo 5 just came out yesterday, so if
> > there is to be effort expended, it might be good to go with the latest
> > version...though, heads up, I tried it out and there seem to be quite a
> few
> > changes that will be needed...new methods, and definitely some replaced
> > ones...
> >
> >
> >
> > On Tue, Jun 19, 2018 at 10:37 PM Pramod Immaneni <
> > pramod.immaneni@gmail.com> wrote:
> >
> >> Hi Aaron,
> >>
> >> While we are debugging the test failures on dev, I didn't fully
> >> understand the last question in your email. Looks like you mentioned
> that
> >> the workaround for adding the @FieldSerializer to the field is working.
> Are
> >> you looking for an alternative to this workaround and trying to set a
> >> default serializer for all fields of type Instant (the field type you
> >> mentioned in the earlier email) so that you don't have to set the
> >> annotation each time?
> >>
> >> Thanks
> >>
> >> On Tue, Jun 19, 2018 at 12:59 PM Aaron Bossert <aa...@punchcyber.com>
> >> wrote:
> >>
> >>> I sent an e-mail to the dev list...but have not heard any responses.
> >>> How active is that list? Do you know? Basically, I have a workaround
> that
> >>> will deal with those "things" that Kryo v 2.24.0 will not handle (but
> 4.0.2
> >>> or 5.0.0-RC1 would) using the @FieldSerializer ...but this is not
> ideal. I
> >>> have two options left: go it alone and try to update Kryo in the
> apex-core,
> >>> or simply change the default serializer. The hiccup I am running into
> is
> >>> that I don't see how (from the documentation) to set the default
> >>> serializer...perhaps I am missing it. Can someone point me to where
> that
> >>> is in the docs?
> >>>
> >>> On Mon, Jun 11, 2018 at 4:20 PM Aaron Bossert <aa...@punchcyber.com>
> >>> wrote:
> >>>
> >>>> Ah, you make a good point about the dev list...I am using IntelliJ on
> a
> >>>> Mac. When I tried building without any changes, I also encountered
> >>>> failures. I can re-run the non-modified sources to do a diff and see
> if
> >>>> the errors were different...
> >>>>
> >>>> Sent from my iPhone
> >>>>
> >>>> On Jun 11, 2018, at 15:11, Pramod Immaneni <pramod.immaneni@gmail.com
> >
> >>>> wrote:
> >>>>
> >>>> Aaron,
> >>>>
> >>>> The question might be best served on the dev@apex.apache.org mailing
> >>>> list as you are starting to make changes to the sources. Nothing
> obvious
> >>>> jumps out to be, you don't need hadoop on your system to build the
> sources
> >>>> successfully with all the tests passing. The test failures you are
> seeing
> >>>> may or may not be related to the kryo change. What happens when you
> try to
> >>>> build the original sources with changing the Kryo version? What is
> your
> >>>> environment, mac, linux?
> >>>>
> >>>> On Mon, Jun 11, 2018 at 12:03 PM Aaron Bossert <aa...@punchcyber.com>
> >>>> wrote:
> >>>>
> >>>>> I know it has been a little while since I brought this up...I pulled
> >>>>> down the 3.7.1-RELEASE branch from GitHub in order to test out the
> change
> >>>>> to Kryo v. 4.0.2 and encountered no problems while building,
> however, when
> >>>>> I try to do mvn install, there are several failed tests. I am
> hoping, that
> >>>>> other than the obvious (to me, anyway) errors such as not finding
> >>>>> yarn-site.xml on my laptop which does not have hadoop installed,
> someone
> >>>>> may be able to see what is going on?
> >>>>>
> >>>>>
> >>>>>
> -------------------------------------------------------------------------------
> >>>>> Test set: com.datatorrent.stram.StramRecoveryTest
> >>>>>
> >>>>>
> -------------------------------------------------------------------------------
> >>>>> Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 6.372
> >>>>> sec <<< FAILURE! - in com.datatorrent.stram.StramRecoveryTest
> >>>>> testWriteAheadLog(com.datatorrent.stram.StramRecoveryTest) Time
> >>>>> elapsed: 0.115 sec <<< FAILURE!
> >>>>> java.lang.AssertionError: flush count expected:<1> but was:<2>
> >>>>> at
> >>>>>
> com.datatorrent.stram.StramRecoveryTest.testWriteAheadLog(StramRecoveryTest.java:326)
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> -------------------------------------------------------------------------------
> >>>>> Test set: com.datatorrent.stram.engine.StatsTest
> >>>>>
> >>>>>
> -------------------------------------------------------------------------------
> >>>>> Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 21.894
> >>>>> sec <<< FAILURE! - in com.datatorrent.stram.engine.StatsTest
> >>>>>
> testQueueSizeForContainerLocalOperators(com.datatorrent.stram.engine.StatsTest)
> >>>>> Time elapsed: 3.266 sec <<< FAILURE!
> >>>>> java.lang.AssertionError: Validate input port queue size -1
> >>>>> at
> >>>>>
> com.datatorrent.stram.engine.StatsTest.baseTestForQueueSize(StatsTest.java:270)
> >>>>> at
> >>>>>
> com.datatorrent.stram.engine.StatsTest.testQueueSizeForContainerLocalOperators(StatsTest.java:285)
> >>>>>
> >>>>>
> >>>>>
> -------------------------------------------------------------------------------
> >>>>> Test set: com.datatorrent.stram.StramMiniClusterTest
> >>>>>
> >>>>>
> -------------------------------------------------------------------------------
> >>>>> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.002
> >>>>> sec <<< FAILURE! - in com.datatorrent.stram.StramMiniClusterTest
> >>>>> com.datatorrent.stram.StramMiniClusterTest Time elapsed: 0.002 sec
> >>>>> <<< ERROR!
> >>>>> java.io.FileNotFoundException:
> >>>>>
> /Users/mbossert/Library/Mobile%20Documents/com~apple~CloudDocs/Punch/IdeaProjects/apex-core/engine/target/test-classes/yarn-site.xml
> >>>>> (No such file or directory)
> >>>>> at
> >>>>>
> com.datatorrent.stram.StramMiniClusterTest.setup(StramMiniClusterTest.java:151)
> >>>>>
> >>>>>
> >>>>>
> -------------------------------------------------------------------------------
> >>>>> Test set: com.datatorrent.stram.StramLocalClusterTest
> >>>>>
> >>>>>
> -------------------------------------------------------------------------------
> >>>>> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed:
> 15.081
> >>>>> sec <<< FAILURE! - in com.datatorrent.stram.StramLocalClusterTest
> >>>>> testAppPath(com.datatorrent.stram.StramLocalClusterTest) Time
> >>>>> elapsed: 0.138 sec <<< FAILURE!
> >>>>> java.lang.AssertionError: checkpoint path exists
> >>>>> at
> >>>>>
> com.datatorrent.stram.StramLocalClusterTest.testAppPath(StramLocalClusterTest.java:413)
> >>>>>
> >>>>>
> testLocalClusterInitShutdown(com.datatorrent.stram.StramLocalClusterTest)
> >>>>> Time elapsed: 5.191 sec <<< FAILURE!
> >>>>> java.lang.AssertionError:
> >>>>>
> ./target/com.datatorrent.stram.StramLocalClusterTest-testLocalClusterInitShutdown.out
> >>>>> exists
> >>>>> at
> >>>>>
> com.datatorrent.stram.StramLocalClusterTest.testLocalClusterInitShutdown(StramLocalClusterTest.java:117)
> >>>>>
> >>>>>
> >>>>>
> -------------------------------------------------------------------------------
> >>>>> Test set: com.datatorrent.stram.client.AppPackageTest
> >>>>>
> >>>>>
> -------------------------------------------------------------------------------
> >>>>> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.105
> >>>>> sec <<< FAILURE! - in com.datatorrent.stram.client.AppPackageTest
> >>>>> testAppPackage(com.datatorrent.stram.client.AppPackageTest) Time
> >>>>> elapsed: 0 sec <<< FAILURE!
> >>>>> java.lang.AssertionError: Number of applications expected:<2> but
> >>>>> was:<0>
> >>>>> at
> >>>>>
> com.datatorrent.stram.client.AppPackageTest.testAppPackage(AppPackageTest.java:91)
> >>>>>
> >>>>>
> testAppLevelRequiredAndDefaultProperties(com.datatorrent.stram.client.AppPackageTest)
> >>>>> Time elapsed: 0.001 sec <<< FAILURE!
> >>>>> java.lang.AssertionError: Should consist of an app called
> >>>>> MyFirstApplication
> >>>>> at
> >>>>>
> com.datatorrent.stram.client.AppPackageTest.testAppLevelRequiredAndDefaultProperties(AppPackageTest.java:171)
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> -------------------------------------------------------------------------------
> >>>>> Test set: com.datatorrent.stram.cli.ApexCliTest
> >>>>>
> >>>>>
> -------------------------------------------------------------------------------
> >>>>> Tests run: 7, Failures: 2, Errors: 2, Skipped: 0, Time elapsed: 0.938
> >>>>> sec <<< FAILURE! - in com.datatorrent.stram.cli.ApexCliTest
> >>>>> testAppFromOnlyConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
> >>>>> Time elapsed: 0.104 sec <<< ERROR!
> >>>>> com.datatorrent.stram.cli.ApexCli$CliException: No applications in
> >>>>> Application Package
> >>>>> at
> >>>>>
> com.datatorrent.stram.cli.ApexCliTest.testAppFromOnlyConfigPackage(ApexCliTest.java:235)
> >>>>>
> >>>>>
> testLaunchAppPackagePropertyPrecedence(com.datatorrent.stram.cli.ApexCliTest)
> >>>>> Time elapsed: 0.045 sec <<< FAILURE!
> >>>>> org.junit.ComparisonFailure: expected:<[app]-default> but
> >>>>> was:<[package]-default>
> >>>>> at
> >>>>>
> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePropertyPrecedence(ApexCliTest.java:143)
> >>>>>
> >>>>>
> testLaunchAppPackageParametersWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
> >>>>> Time elapsed: 0.045 sec <<< ERROR!
> >>>>> com.datatorrent.stram.cli.ApexCli$CliException: No applications in
> >>>>> Application Package matching "MyFirstApplication"
> >>>>> at
> >>>>>
> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackageParametersWithConfigPackage(ApexCliTest.java:155)
> >>>>>
> >>>>>
> testLaunchAppPackagePrecedenceWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
> >>>>> Time elapsed: 0.058 sec <<< FAILURE!
> >>>>> org.junit.ComparisonFailure: expected:<[app]-default> but
> >>>>> was:<[package]-default>
> >>>>> at
> >>>>>
> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePrecedenceWithConfigPackage(ApexCliTest.java:204)
> >>>>>
> >>>>>
> >>>>> On Thu, May 17, 2018 at 1:31 PM, Aaron Bossert <aaron@punchcyber.com
> >
> >>>>> wrote:
> >>>>>
> >>>>>> Thanks, Thomas! I have implemented the fix you suggested
> >>>>>> (@FieldSerializer.BInd) and also created a JIRA ticket:
> >>>>>> https://issues.apache.org/jira/browse/APEXCORE-816
> >>>>>>
> >>>>>> On Thu, May 17, 2018 at 1:31 AM, Thomas Weise <th...@apache.org>
> wrote:
> >>>>>>
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> It is not necessary to use a different stream codec when you have
> >>>>>>> control over the class that is causing the error. You can annotate
> a
> >>>>>>> different field serializer instead, using the @FieldSerializer.Bind
> >>>>>>> annotation. Here is an example:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> https://github.com/apache/apex-malhar/blob/2fe2903bfe65055c7fd361a97549b6e7e12d4289/library/src/main/java/org/apache/apex/malhar/lib/window/WindowOption.java#L50
> >>>>>>>
> >>>>>>> It would be good to upgrade Kryo to 4.x in apex-core master. Would
> >>>>>>> you like to create a JIRA for it? (
> >>>>>>> http://apex.apache.org/contributing.html)
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> Thomas
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Wed, May 16, 2018 at 10:31 AM, Aaron Bossert <
> >>>>>>> aaron@punchcyber.com> wrote:
> >>>>>>>
> >>>>>>>> I am writing an application that includes a custom object. One of
> >>>>>>>> the fields is of type Instant and seems to not agree with the
> version of
> >>>>>>>> Kryo that is used in Apex (see exception below). I now that kryo
> 4.X
> >>>>>>>> supports serializing Instant fields, but for the life of me,
> cannot seem to
> >>>>>>>> find a way to either get Apex to use a newer version of Kryo.
> Can anyone
> >>>>>>>> help? Perhaps setting a different streamcodec that uses the
> newer version
> >>>>>>>> of kryo would work, but I cannot seem to find an example of how
> to set a
> >>>>>>>> different streamcodec despite references to the two (Kryo and
> Java) that
> >>>>>>>> ship with Apex.
> >>>>>>>>
> >>>>>>>> ERROR com.datatorrent.stram.codec.DefaultStatefulStreamCodec -
> >>>>>>>> Catastrophic Error: Execution halted due to Kryo exception!
> >>>>>>>> com.esotericsoftware.kryo.KryoException: Class cannot be created
> >>>>>>>> (missing no-arg constructor): java.time.Instant
> >>>>>>>> Serialization trace:
> >>>>>>>> systemTime
> >>>>>>>> (com.punchcyber.streaming.pipeline.apex.datatypes.evtx.EvtxEvent)
> >>>>>>>> at
> >>>>>>>>
> com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
> >>>>>>>> at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
> >>>>>>>> at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
> >>>>>>>> at
> >>>>>>>>
> com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
> >>>>>>>> at
> >>>>>>>>
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
> >>>>>>>> at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730)
> >>>>>>>> at
> >>>>>>>>
> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113)
> >>>>>>>> at
> >>>>>>>>
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
> >>>>>>>> at
> com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
> >>>>>>>> at com.datatorrent.stram.codec.De
> >>>>>>>>
> faultStatefulStreamCodec.fromDataStatePair(DefaultStatefulStreamCodec.java:98)
> >>>>>>>> at
> >>>>>>>>
> com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.processPayload(BufferServerSubscriber.java:391)
> >>>>>>>> at
> >>>>>>>>
> com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.sweep(BufferServerSubscriber.java:339)
> >>>>>>>> at
> >>>>>>>> com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:269)
> >>>>>>>> at
> >>>>>>>>
> com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1429)
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>>
> >>>>>>>> M. Aaron Bossert
> >>>>>>>> (571) 242-4021
> >>>>>>>> Punch Cyber Analytics Group
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>>
> >>>>>> M. Aaron Bossert
> >>>>>> (571) 242-4021
> >>>>>> Punch Cyber Analytics Group
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>>
> >>>>> M. Aaron Bossert
> >>>>> (571) 242-4021
> >>>>> Punch Cyber Analytics Group
> >>>>>
> >>>>>
> >>>>>
> >>>
> >>> --
> >>>
> >>> M. Aaron Bossert
> >>> (571) 242-4021
> >>> Punch Cyber Analytics Group
> >>>
> >>>
> >>>
> >
> > --
> >
> > M. Aaron Bossert
> > (571) 242-4021
> > Punch Cyber Analytics Group
> >
> >
> >
>
> --
>
> M. Aaron Bossert
> (571) 242-4021
> Punch Cyber Analytics Group
>
Re: Kryo version and default serializer
Posted by Aaron Bossert <aa...@punchcyber.com>.
P.S. I am happy to roll up my sleeves, but from my initial attempt, errors
I was seeing were not obviously related to the Kryo version update (though
I am certain they are...at least indirectly because they only show up when
the version is changed), in terms of tracking down where the changes would
be required...I am not really familiar with the inner workings of Apex yet,
though this would definitely be a good excuse to learn...
I also saw a bunch of annoying warnings when building related to use of
deprecated methods/classes within the com.datatorrent.* namespace...I'd be
happy to start tracking some of those down too if it would be helpful...
On Tue, Jun 19, 2018 at 10:48 PM Aaron Bossert <aa...@punchcyber.com> wrote:
> Pramod,
>
> In a nutshell, yes. I can set a @FieldSerializer to the fields in
> question...the only one I have run into thus far is Instant...but it would
> be great to not need to annotate each field this way...Also, I could see
> this becoming problematic should I encounter more than a couple of fields
> that are not serializable using the older version of Kryo (but presumably
> would work with a more current one)...though admitedly, it would not be
> that big a deal in the interim. It would be much cleaner to have an update
> to the most current version of Kryo, IMHO...though I understand that is
> something of a lift. By the way, Kryo 5 just came out yesterday, so if
> there is to be effort expended, it might be good to go with the latest
> version...though, heads up, I tried it out and there seem to be quite a few
> changes that will be needed...new methods, and definitely some replaced
> ones...
>
>
>
> On Tue, Jun 19, 2018 at 10:37 PM Pramod Immaneni <
> pramod.immaneni@gmail.com> wrote:
>
>> Hi Aaron,
>>
>> While we are debugging the test failures on dev, I didn't fully
>> understand the last question in your email. Looks like you mentioned that
>> the workaround for adding the @FieldSerializer to the field is working. Are
>> you looking for an alternative to this workaround and trying to set a
>> default serializer for all fields of type Instant (the field type you
>> mentioned in the earlier email) so that you don't have to set the
>> annotation each time?
>>
>> Thanks
>>
>> On Tue, Jun 19, 2018 at 12:59 PM Aaron Bossert <aa...@punchcyber.com>
>> wrote:
>>
>>> I sent an e-mail to the dev list...but have not heard any responses.
>>> How active is that list? Do you know? Basically, I have a workaround that
>>> will deal with those "things" that Kryo v 2.24.0 will not handle (but 4.0.2
>>> or 5.0.0-RC1 would) using the @FieldSerializer ...but this is not ideal. I
>>> have two options left: go it alone and try to update Kryo in the apex-core,
>>> or simply change the default serializer. The hiccup I am running into is
>>> that I don't see how (from the documentation) to set the default
>>> serializer...perhaps I am missing it. Can someone point me to where that
>>> is in the docs?
>>>
>>> On Mon, Jun 11, 2018 at 4:20 PM Aaron Bossert <aa...@punchcyber.com>
>>> wrote:
>>>
>>>> Ah, you make a good point about the dev list...I am using IntelliJ on a
>>>> Mac. When I tried building without any changes, I also encountered
>>>> failures. I can re-run the non-modified sources to do a diff and see if
>>>> the errors were different...
>>>>
>>>> Sent from my iPhone
>>>>
>>>> On Jun 11, 2018, at 15:11, Pramod Immaneni <pr...@gmail.com>
>>>> wrote:
>>>>
>>>> Aaron,
>>>>
>>>> The question might be best served on the dev@apex.apache.org mailing
>>>> list as you are starting to make changes to the sources. Nothing obvious
>>>> jumps out to be, you don't need hadoop on your system to build the sources
>>>> successfully with all the tests passing. The test failures you are seeing
>>>> may or may not be related to the kryo change. What happens when you try to
>>>> build the original sources with changing the Kryo version? What is your
>>>> environment, mac, linux?
>>>>
>>>> On Mon, Jun 11, 2018 at 12:03 PM Aaron Bossert <aa...@punchcyber.com>
>>>> wrote:
>>>>
>>>>> I know it has been a little while since I brought this up...I pulled
>>>>> down the 3.7.1-RELEASE branch from GitHub in order to test out the change
>>>>> to Kryo v. 4.0.2 and encountered no problems while building, however, when
>>>>> I try to do mvn install, there are several failed tests. I am hoping, that
>>>>> other than the obvious (to me, anyway) errors such as not finding
>>>>> yarn-site.xml on my laptop which does not have hadoop installed, someone
>>>>> may be able to see what is going on?
>>>>>
>>>>>
>>>>> -------------------------------------------------------------------------------
>>>>> Test set: com.datatorrent.stram.StramRecoveryTest
>>>>>
>>>>> -------------------------------------------------------------------------------
>>>>> Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 6.372
>>>>> sec <<< FAILURE! - in com.datatorrent.stram.StramRecoveryTest
>>>>> testWriteAheadLog(com.datatorrent.stram.StramRecoveryTest) Time
>>>>> elapsed: 0.115 sec <<< FAILURE!
>>>>> java.lang.AssertionError: flush count expected:<1> but was:<2>
>>>>> at
>>>>> com.datatorrent.stram.StramRecoveryTest.testWriteAheadLog(StramRecoveryTest.java:326)
>>>>>
>>>>>
>>>>>
>>>>> -------------------------------------------------------------------------------
>>>>> Test set: com.datatorrent.stram.engine.StatsTest
>>>>>
>>>>> -------------------------------------------------------------------------------
>>>>> Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 21.894
>>>>> sec <<< FAILURE! - in com.datatorrent.stram.engine.StatsTest
>>>>> testQueueSizeForContainerLocalOperators(com.datatorrent.stram.engine.StatsTest)
>>>>> Time elapsed: 3.266 sec <<< FAILURE!
>>>>> java.lang.AssertionError: Validate input port queue size -1
>>>>> at
>>>>> com.datatorrent.stram.engine.StatsTest.baseTestForQueueSize(StatsTest.java:270)
>>>>> at
>>>>> com.datatorrent.stram.engine.StatsTest.testQueueSizeForContainerLocalOperators(StatsTest.java:285)
>>>>>
>>>>>
>>>>> -------------------------------------------------------------------------------
>>>>> Test set: com.datatorrent.stram.StramMiniClusterTest
>>>>>
>>>>> -------------------------------------------------------------------------------
>>>>> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.002
>>>>> sec <<< FAILURE! - in com.datatorrent.stram.StramMiniClusterTest
>>>>> com.datatorrent.stram.StramMiniClusterTest Time elapsed: 0.002 sec
>>>>> <<< ERROR!
>>>>> java.io.FileNotFoundException:
>>>>> /Users/mbossert/Library/Mobile%20Documents/com~apple~CloudDocs/Punch/IdeaProjects/apex-core/engine/target/test-classes/yarn-site.xml
>>>>> (No such file or directory)
>>>>> at
>>>>> com.datatorrent.stram.StramMiniClusterTest.setup(StramMiniClusterTest.java:151)
>>>>>
>>>>>
>>>>> -------------------------------------------------------------------------------
>>>>> Test set: com.datatorrent.stram.StramLocalClusterTest
>>>>>
>>>>> -------------------------------------------------------------------------------
>>>>> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 15.081
>>>>> sec <<< FAILURE! - in com.datatorrent.stram.StramLocalClusterTest
>>>>> testAppPath(com.datatorrent.stram.StramLocalClusterTest) Time
>>>>> elapsed: 0.138 sec <<< FAILURE!
>>>>> java.lang.AssertionError: checkpoint path exists
>>>>> at
>>>>> com.datatorrent.stram.StramLocalClusterTest.testAppPath(StramLocalClusterTest.java:413)
>>>>>
>>>>> testLocalClusterInitShutdown(com.datatorrent.stram.StramLocalClusterTest)
>>>>> Time elapsed: 5.191 sec <<< FAILURE!
>>>>> java.lang.AssertionError:
>>>>> ./target/com.datatorrent.stram.StramLocalClusterTest-testLocalClusterInitShutdown.out
>>>>> exists
>>>>> at
>>>>> com.datatorrent.stram.StramLocalClusterTest.testLocalClusterInitShutdown(StramLocalClusterTest.java:117)
>>>>>
>>>>>
>>>>> -------------------------------------------------------------------------------
>>>>> Test set: com.datatorrent.stram.client.AppPackageTest
>>>>>
>>>>> -------------------------------------------------------------------------------
>>>>> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.105
>>>>> sec <<< FAILURE! - in com.datatorrent.stram.client.AppPackageTest
>>>>> testAppPackage(com.datatorrent.stram.client.AppPackageTest) Time
>>>>> elapsed: 0 sec <<< FAILURE!
>>>>> java.lang.AssertionError: Number of applications expected:<2> but
>>>>> was:<0>
>>>>> at
>>>>> com.datatorrent.stram.client.AppPackageTest.testAppPackage(AppPackageTest.java:91)
>>>>>
>>>>> testAppLevelRequiredAndDefaultProperties(com.datatorrent.stram.client.AppPackageTest)
>>>>> Time elapsed: 0.001 sec <<< FAILURE!
>>>>> java.lang.AssertionError: Should consist of an app called
>>>>> MyFirstApplication
>>>>> at
>>>>> com.datatorrent.stram.client.AppPackageTest.testAppLevelRequiredAndDefaultProperties(AppPackageTest.java:171)
>>>>>
>>>>>
>>>>>
>>>>> -------------------------------------------------------------------------------
>>>>> Test set: com.datatorrent.stram.cli.ApexCliTest
>>>>>
>>>>> -------------------------------------------------------------------------------
>>>>> Tests run: 7, Failures: 2, Errors: 2, Skipped: 0, Time elapsed: 0.938
>>>>> sec <<< FAILURE! - in com.datatorrent.stram.cli.ApexCliTest
>>>>> testAppFromOnlyConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
>>>>> Time elapsed: 0.104 sec <<< ERROR!
>>>>> com.datatorrent.stram.cli.ApexCli$CliException: No applications in
>>>>> Application Package
>>>>> at
>>>>> com.datatorrent.stram.cli.ApexCliTest.testAppFromOnlyConfigPackage(ApexCliTest.java:235)
>>>>>
>>>>> testLaunchAppPackagePropertyPrecedence(com.datatorrent.stram.cli.ApexCliTest)
>>>>> Time elapsed: 0.045 sec <<< FAILURE!
>>>>> org.junit.ComparisonFailure: expected:<[app]-default> but
>>>>> was:<[package]-default>
>>>>> at
>>>>> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePropertyPrecedence(ApexCliTest.java:143)
>>>>>
>>>>> testLaunchAppPackageParametersWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
>>>>> Time elapsed: 0.045 sec <<< ERROR!
>>>>> com.datatorrent.stram.cli.ApexCli$CliException: No applications in
>>>>> Application Package matching "MyFirstApplication"
>>>>> at
>>>>> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackageParametersWithConfigPackage(ApexCliTest.java:155)
>>>>>
>>>>> testLaunchAppPackagePrecedenceWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
>>>>> Time elapsed: 0.058 sec <<< FAILURE!
>>>>> org.junit.ComparisonFailure: expected:<[app]-default> but
>>>>> was:<[package]-default>
>>>>> at
>>>>> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePrecedenceWithConfigPackage(ApexCliTest.java:204)
>>>>>
>>>>>
>>>>> On Thu, May 17, 2018 at 1:31 PM, Aaron Bossert <aa...@punchcyber.com>
>>>>> wrote:
>>>>>
>>>>>> Thanks, Thomas! I have implemented the fix you suggested
>>>>>> (@FieldSerializer.BInd) and also created a JIRA ticket:
>>>>>> https://issues.apache.org/jira/browse/APEXCORE-816
>>>>>>
>>>>>> On Thu, May 17, 2018 at 1:31 AM, Thomas Weise <th...@apache.org> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> It is not necessary to use a different stream codec when you have
>>>>>>> control over the class that is causing the error. You can annotate a
>>>>>>> different field serializer instead, using the @FieldSerializer.Bind
>>>>>>> annotation. Here is an example:
>>>>>>>
>>>>>>>
>>>>>>> https://github.com/apache/apex-malhar/blob/2fe2903bfe65055c7fd361a97549b6e7e12d4289/library/src/main/java/org/apache/apex/malhar/lib/window/WindowOption.java#L50
>>>>>>>
>>>>>>> It would be good to upgrade Kryo to 4.x in apex-core master. Would
>>>>>>> you like to create a JIRA for it? (
>>>>>>> http://apex.apache.org/contributing.html)
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Thomas
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, May 16, 2018 at 10:31 AM, Aaron Bossert <
>>>>>>> aaron@punchcyber.com> wrote:
>>>>>>>
>>>>>>>> I am writing an application that includes a custom object. One of
>>>>>>>> the fields is of type Instant and seems to not agree with the version of
>>>>>>>> Kryo that is used in Apex (see exception below). I now that kryo 4.X
>>>>>>>> supports serializing Instant fields, but for the life of me, cannot seem to
>>>>>>>> find a way to either get Apex to use a newer version of Kryo. Can anyone
>>>>>>>> help? Perhaps setting a different streamcodec that uses the newer version
>>>>>>>> of kryo would work, but I cannot seem to find an example of how to set a
>>>>>>>> different streamcodec despite references to the two (Kryo and Java) that
>>>>>>>> ship with Apex.
>>>>>>>>
>>>>>>>> ERROR com.datatorrent.stram.codec.DefaultStatefulStreamCodec -
>>>>>>>> Catastrophic Error: Execution halted due to Kryo exception!
>>>>>>>> com.esotericsoftware.kryo.KryoException: Class cannot be created
>>>>>>>> (missing no-arg constructor): java.time.Instant
>>>>>>>> Serialization trace:
>>>>>>>> systemTime
>>>>>>>> (com.punchcyber.streaming.pipeline.apex.datatypes.evtx.EvtxEvent)
>>>>>>>> at
>>>>>>>> com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
>>>>>>>> at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
>>>>>>>> at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
>>>>>>>> at
>>>>>>>> com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
>>>>>>>> at
>>>>>>>> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
>>>>>>>> at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730)
>>>>>>>> at
>>>>>>>> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113)
>>>>>>>> at
>>>>>>>> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
>>>>>>>> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
>>>>>>>> at com.datatorrent.stram.codec.De
>>>>>>>> faultStatefulStreamCodec.fromDataStatePair(DefaultStatefulStreamCodec.java:98)
>>>>>>>> at
>>>>>>>> com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.processPayload(BufferServerSubscriber.java:391)
>>>>>>>> at
>>>>>>>> com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.sweep(BufferServerSubscriber.java:339)
>>>>>>>> at
>>>>>>>> com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:269)
>>>>>>>> at
>>>>>>>> com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1429)
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>> M. Aaron Bossert
>>>>>>>> (571) 242-4021
>>>>>>>> Punch Cyber Analytics Group
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> M. Aaron Bossert
>>>>>> (571) 242-4021
>>>>>> Punch Cyber Analytics Group
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> M. Aaron Bossert
>>>>> (571) 242-4021
>>>>> Punch Cyber Analytics Group
>>>>>
>>>>>
>>>>>
>>>
>>> --
>>>
>>> M. Aaron Bossert
>>> (571) 242-4021
>>> Punch Cyber Analytics Group
>>>
>>>
>>>
>
> --
>
> M. Aaron Bossert
> (571) 242-4021
> Punch Cyber Analytics Group
>
>
>
--
M. Aaron Bossert
(571) 242-4021
Punch Cyber Analytics Group
Re: Kryo version and default serializer
Posted by Aaron Bossert <aa...@punchcyber.com>.
P.S. I am happy to roll up my sleeves, but from my initial attempt, errors
I was seeing were not obviously related to the Kryo version update (though
I am certain they are...at least indirectly because they only show up when
the version is changed), in terms of tracking down where the changes would
be required...I am not really familiar with the inner workings of Apex yet,
though this would definitely be a good excuse to learn...
I also saw a bunch of annoying warnings when building related to use of
deprecated methods/classes within the com.datatorrent.* namespace...I'd be
happy to start tracking some of those down too if it would be helpful...
On Tue, Jun 19, 2018 at 10:48 PM Aaron Bossert <aa...@punchcyber.com> wrote:
> Pramod,
>
> In a nutshell, yes. I can set a @FieldSerializer to the fields in
> question...the only one I have run into thus far is Instant...but it would
> be great to not need to annotate each field this way...Also, I could see
> this becoming problematic should I encounter more than a couple of fields
> that are not serializable using the older version of Kryo (but presumably
> would work with a more current one)...though admitedly, it would not be
> that big a deal in the interim. It would be much cleaner to have an update
> to the most current version of Kryo, IMHO...though I understand that is
> something of a lift. By the way, Kryo 5 just came out yesterday, so if
> there is to be effort expended, it might be good to go with the latest
> version...though, heads up, I tried it out and there seem to be quite a few
> changes that will be needed...new methods, and definitely some replaced
> ones...
>
>
>
> On Tue, Jun 19, 2018 at 10:37 PM Pramod Immaneni <
> pramod.immaneni@gmail.com> wrote:
>
>> Hi Aaron,
>>
>> While we are debugging the test failures on dev, I didn't fully
>> understand the last question in your email. Looks like you mentioned that
>> the workaround for adding the @FieldSerializer to the field is working. Are
>> you looking for an alternative to this workaround and trying to set a
>> default serializer for all fields of type Instant (the field type you
>> mentioned in the earlier email) so that you don't have to set the
>> annotation each time?
>>
>> Thanks
>>
>> On Tue, Jun 19, 2018 at 12:59 PM Aaron Bossert <aa...@punchcyber.com>
>> wrote:
>>
>>> I sent an e-mail to the dev list...but have not heard any responses.
>>> How active is that list? Do you know? Basically, I have a workaround that
>>> will deal with those "things" that Kryo v 2.24.0 will not handle (but 4.0.2
>>> or 5.0.0-RC1 would) using the @FieldSerializer ...but this is not ideal. I
>>> have two options left: go it alone and try to update Kryo in the apex-core,
>>> or simply change the default serializer. The hiccup I am running into is
>>> that I don't see how (from the documentation) to set the default
>>> serializer...perhaps I am missing it. Can someone point me to where that
>>> is in the docs?
>>>
>>> On Mon, Jun 11, 2018 at 4:20 PM Aaron Bossert <aa...@punchcyber.com>
>>> wrote:
>>>
>>>> Ah, you make a good point about the dev list...I am using IntelliJ on a
>>>> Mac. When I tried building without any changes, I also encountered
>>>> failures. I can re-run the non-modified sources to do a diff and see if
>>>> the errors were different...
>>>>
>>>> Sent from my iPhone
>>>>
>>>> On Jun 11, 2018, at 15:11, Pramod Immaneni <pr...@gmail.com>
>>>> wrote:
>>>>
>>>> Aaron,
>>>>
>>>> The question might be best served on the dev@apex.apache.org mailing
>>>> list as you are starting to make changes to the sources. Nothing obvious
>>>> jumps out to be, you don't need hadoop on your system to build the sources
>>>> successfully with all the tests passing. The test failures you are seeing
>>>> may or may not be related to the kryo change. What happens when you try to
>>>> build the original sources with changing the Kryo version? What is your
>>>> environment, mac, linux?
>>>>
>>>> On Mon, Jun 11, 2018 at 12:03 PM Aaron Bossert <aa...@punchcyber.com>
>>>> wrote:
>>>>
>>>>> I know it has been a little while since I brought this up...I pulled
>>>>> down the 3.7.1-RELEASE branch from GitHub in order to test out the change
>>>>> to Kryo v. 4.0.2 and encountered no problems while building, however, when
>>>>> I try to do mvn install, there are several failed tests. I am hoping, that
>>>>> other than the obvious (to me, anyway) errors such as not finding
>>>>> yarn-site.xml on my laptop which does not have hadoop installed, someone
>>>>> may be able to see what is going on?
>>>>>
>>>>>
>>>>> -------------------------------------------------------------------------------
>>>>> Test set: com.datatorrent.stram.StramRecoveryTest
>>>>>
>>>>> -------------------------------------------------------------------------------
>>>>> Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 6.372
>>>>> sec <<< FAILURE! - in com.datatorrent.stram.StramRecoveryTest
>>>>> testWriteAheadLog(com.datatorrent.stram.StramRecoveryTest) Time
>>>>> elapsed: 0.115 sec <<< FAILURE!
>>>>> java.lang.AssertionError: flush count expected:<1> but was:<2>
>>>>> at
>>>>> com.datatorrent.stram.StramRecoveryTest.testWriteAheadLog(StramRecoveryTest.java:326)
>>>>>
>>>>>
>>>>>
>>>>> -------------------------------------------------------------------------------
>>>>> Test set: com.datatorrent.stram.engine.StatsTest
>>>>>
>>>>> -------------------------------------------------------------------------------
>>>>> Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 21.894
>>>>> sec <<< FAILURE! - in com.datatorrent.stram.engine.StatsTest
>>>>> testQueueSizeForContainerLocalOperators(com.datatorrent.stram.engine.StatsTest)
>>>>> Time elapsed: 3.266 sec <<< FAILURE!
>>>>> java.lang.AssertionError: Validate input port queue size -1
>>>>> at
>>>>> com.datatorrent.stram.engine.StatsTest.baseTestForQueueSize(StatsTest.java:270)
>>>>> at
>>>>> com.datatorrent.stram.engine.StatsTest.testQueueSizeForContainerLocalOperators(StatsTest.java:285)
>>>>>
>>>>>
>>>>> -------------------------------------------------------------------------------
>>>>> Test set: com.datatorrent.stram.StramMiniClusterTest
>>>>>
>>>>> -------------------------------------------------------------------------------
>>>>> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.002
>>>>> sec <<< FAILURE! - in com.datatorrent.stram.StramMiniClusterTest
>>>>> com.datatorrent.stram.StramMiniClusterTest Time elapsed: 0.002 sec
>>>>> <<< ERROR!
>>>>> java.io.FileNotFoundException:
>>>>> /Users/mbossert/Library/Mobile%20Documents/com~apple~CloudDocs/Punch/IdeaProjects/apex-core/engine/target/test-classes/yarn-site.xml
>>>>> (No such file or directory)
>>>>> at
>>>>> com.datatorrent.stram.StramMiniClusterTest.setup(StramMiniClusterTest.java:151)
>>>>>
>>>>>
>>>>> -------------------------------------------------------------------------------
>>>>> Test set: com.datatorrent.stram.StramLocalClusterTest
>>>>>
>>>>> -------------------------------------------------------------------------------
>>>>> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 15.081
>>>>> sec <<< FAILURE! - in com.datatorrent.stram.StramLocalClusterTest
>>>>> testAppPath(com.datatorrent.stram.StramLocalClusterTest) Time
>>>>> elapsed: 0.138 sec <<< FAILURE!
>>>>> java.lang.AssertionError: checkpoint path exists
>>>>> at
>>>>> com.datatorrent.stram.StramLocalClusterTest.testAppPath(StramLocalClusterTest.java:413)
>>>>>
>>>>> testLocalClusterInitShutdown(com.datatorrent.stram.StramLocalClusterTest)
>>>>> Time elapsed: 5.191 sec <<< FAILURE!
>>>>> java.lang.AssertionError:
>>>>> ./target/com.datatorrent.stram.StramLocalClusterTest-testLocalClusterInitShutdown.out
>>>>> exists
>>>>> at
>>>>> com.datatorrent.stram.StramLocalClusterTest.testLocalClusterInitShutdown(StramLocalClusterTest.java:117)
>>>>>
>>>>>
>>>>> -------------------------------------------------------------------------------
>>>>> Test set: com.datatorrent.stram.client.AppPackageTest
>>>>>
>>>>> -------------------------------------------------------------------------------
>>>>> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.105
>>>>> sec <<< FAILURE! - in com.datatorrent.stram.client.AppPackageTest
>>>>> testAppPackage(com.datatorrent.stram.client.AppPackageTest) Time
>>>>> elapsed: 0 sec <<< FAILURE!
>>>>> java.lang.AssertionError: Number of applications expected:<2> but
>>>>> was:<0>
>>>>> at
>>>>> com.datatorrent.stram.client.AppPackageTest.testAppPackage(AppPackageTest.java:91)
>>>>>
>>>>> testAppLevelRequiredAndDefaultProperties(com.datatorrent.stram.client.AppPackageTest)
>>>>> Time elapsed: 0.001 sec <<< FAILURE!
>>>>> java.lang.AssertionError: Should consist of an app called
>>>>> MyFirstApplication
>>>>> at
>>>>> com.datatorrent.stram.client.AppPackageTest.testAppLevelRequiredAndDefaultProperties(AppPackageTest.java:171)
>>>>>
>>>>>
>>>>>
>>>>> -------------------------------------------------------------------------------
>>>>> Test set: com.datatorrent.stram.cli.ApexCliTest
>>>>>
>>>>> -------------------------------------------------------------------------------
>>>>> Tests run: 7, Failures: 2, Errors: 2, Skipped: 0, Time elapsed: 0.938
>>>>> sec <<< FAILURE! - in com.datatorrent.stram.cli.ApexCliTest
>>>>> testAppFromOnlyConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
>>>>> Time elapsed: 0.104 sec <<< ERROR!
>>>>> com.datatorrent.stram.cli.ApexCli$CliException: No applications in
>>>>> Application Package
>>>>> at
>>>>> com.datatorrent.stram.cli.ApexCliTest.testAppFromOnlyConfigPackage(ApexCliTest.java:235)
>>>>>
>>>>> testLaunchAppPackagePropertyPrecedence(com.datatorrent.stram.cli.ApexCliTest)
>>>>> Time elapsed: 0.045 sec <<< FAILURE!
>>>>> org.junit.ComparisonFailure: expected:<[app]-default> but
>>>>> was:<[package]-default>
>>>>> at
>>>>> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePropertyPrecedence(ApexCliTest.java:143)
>>>>>
>>>>> testLaunchAppPackageParametersWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
>>>>> Time elapsed: 0.045 sec <<< ERROR!
>>>>> com.datatorrent.stram.cli.ApexCli$CliException: No applications in
>>>>> Application Package matching "MyFirstApplication"
>>>>> at
>>>>> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackageParametersWithConfigPackage(ApexCliTest.java:155)
>>>>>
>>>>> testLaunchAppPackagePrecedenceWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
>>>>> Time elapsed: 0.058 sec <<< FAILURE!
>>>>> org.junit.ComparisonFailure: expected:<[app]-default> but
>>>>> was:<[package]-default>
>>>>> at
>>>>> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePrecedenceWithConfigPackage(ApexCliTest.java:204)
>>>>>
>>>>>
>>>>> On Thu, May 17, 2018 at 1:31 PM, Aaron Bossert <aa...@punchcyber.com>
>>>>> wrote:
>>>>>
>>>>>> Thanks, Thomas! I have implemented the fix you suggested
>>>>>> (@FieldSerializer.BInd) and also created a JIRA ticket:
>>>>>> https://issues.apache.org/jira/browse/APEXCORE-816
>>>>>>
>>>>>> On Thu, May 17, 2018 at 1:31 AM, Thomas Weise <th...@apache.org> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> It is not necessary to use a different stream codec when you have
>>>>>>> control over the class that is causing the error. You can annotate a
>>>>>>> different field serializer instead, using the @FieldSerializer.Bind
>>>>>>> annotation. Here is an example:
>>>>>>>
>>>>>>>
>>>>>>> https://github.com/apache/apex-malhar/blob/2fe2903bfe65055c7fd361a97549b6e7e12d4289/library/src/main/java/org/apache/apex/malhar/lib/window/WindowOption.java#L50
>>>>>>>
>>>>>>> It would be good to upgrade Kryo to 4.x in apex-core master. Would
>>>>>>> you like to create a JIRA for it? (
>>>>>>> http://apex.apache.org/contributing.html)
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Thomas
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, May 16, 2018 at 10:31 AM, Aaron Bossert <
>>>>>>> aaron@punchcyber.com> wrote:
>>>>>>>
>>>>>>>> I am writing an application that includes a custom object. One of
>>>>>>>> the fields is of type Instant and seems to not agree with the version of
>>>>>>>> Kryo that is used in Apex (see exception below). I now that kryo 4.X
>>>>>>>> supports serializing Instant fields, but for the life of me, cannot seem to
>>>>>>>> find a way to either get Apex to use a newer version of Kryo. Can anyone
>>>>>>>> help? Perhaps setting a different streamcodec that uses the newer version
>>>>>>>> of kryo would work, but I cannot seem to find an example of how to set a
>>>>>>>> different streamcodec despite references to the two (Kryo and Java) that
>>>>>>>> ship with Apex.
>>>>>>>>
>>>>>>>> ERROR com.datatorrent.stram.codec.DefaultStatefulStreamCodec -
>>>>>>>> Catastrophic Error: Execution halted due to Kryo exception!
>>>>>>>> com.esotericsoftware.kryo.KryoException: Class cannot be created
>>>>>>>> (missing no-arg constructor): java.time.Instant
>>>>>>>> Serialization trace:
>>>>>>>> systemTime
>>>>>>>> (com.punchcyber.streaming.pipeline.apex.datatypes.evtx.EvtxEvent)
>>>>>>>> at
>>>>>>>> com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
>>>>>>>> at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
>>>>>>>> at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
>>>>>>>> at
>>>>>>>> com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
>>>>>>>> at
>>>>>>>> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
>>>>>>>> at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730)
>>>>>>>> at
>>>>>>>> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113)
>>>>>>>> at
>>>>>>>> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
>>>>>>>> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
>>>>>>>> at com.datatorrent.stram.codec.De
>>>>>>>> faultStatefulStreamCodec.fromDataStatePair(DefaultStatefulStreamCodec.java:98)
>>>>>>>> at
>>>>>>>> com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.processPayload(BufferServerSubscriber.java:391)
>>>>>>>> at
>>>>>>>> com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.sweep(BufferServerSubscriber.java:339)
>>>>>>>> at
>>>>>>>> com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:269)
>>>>>>>> at
>>>>>>>> com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1429)
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>> M. Aaron Bossert
>>>>>>>> (571) 242-4021
>>>>>>>> Punch Cyber Analytics Group
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> M. Aaron Bossert
>>>>>> (571) 242-4021
>>>>>> Punch Cyber Analytics Group
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> M. Aaron Bossert
>>>>> (571) 242-4021
>>>>> Punch Cyber Analytics Group
>>>>>
>>>>>
>>>>>
>>>
>>> --
>>>
>>> M. Aaron Bossert
>>> (571) 242-4021
>>> Punch Cyber Analytics Group
>>>
>>>
>>>
>
> --
>
> M. Aaron Bossert
> (571) 242-4021
> Punch Cyber Analytics Group
>
>
>
--
M. Aaron Bossert
(571) 242-4021
Punch Cyber Analytics Group
Re: Kryo version and default serializer
Posted by Aaron Bossert <aa...@punchcyber.com>.
Pramod,
In a nutshell, yes. I can set a @FieldSerializer to the fields in
question...the only one I have run into thus far is Instant...but it would
be great to not need to annotate each field this way...Also, I could see
this becoming problematic should I encounter more than a couple of fields
that are not serializable using the older version of Kryo (but presumably
would work with a more current one)...though admitedly, it would not be
that big a deal in the interim. It would be much cleaner to have an update
to the most current version of Kryo, IMHO...though I understand that is
something of a lift. By the way, Kryo 5 just came out yesterday, so if
there is to be effort expended, it might be good to go with the latest
version...though, heads up, I tried it out and there seem to be quite a few
changes that will be needed...new methods, and definitely some replaced
ones...
On Tue, Jun 19, 2018 at 10:37 PM Pramod Immaneni <pr...@gmail.com>
wrote:
> Hi Aaron,
>
> While we are debugging the test failures on dev, I didn't fully understand
> the last question in your email. Looks like you mentioned that the
> workaround for adding the @FieldSerializer to the field is working. Are you
> looking for an alternative to this workaround and trying to set a default
> serializer for all fields of type Instant (the field type you mentioned in
> the earlier email) so that you don't have to set the annotation each time?
>
> Thanks
>
> On Tue, Jun 19, 2018 at 12:59 PM Aaron Bossert <aa...@punchcyber.com>
> wrote:
>
>> I sent an e-mail to the dev list...but have not heard any responses. How
>> active is that list? Do you know? Basically, I have a workaround that
>> will deal with those "things" that Kryo v 2.24.0 will not handle (but 4.0.2
>> or 5.0.0-RC1 would) using the @FieldSerializer ...but this is not ideal. I
>> have two options left: go it alone and try to update Kryo in the apex-core,
>> or simply change the default serializer. The hiccup I am running into is
>> that I don't see how (from the documentation) to set the default
>> serializer...perhaps I am missing it. Can someone point me to where that
>> is in the docs?
>>
>> On Mon, Jun 11, 2018 at 4:20 PM Aaron Bossert <aa...@punchcyber.com>
>> wrote:
>>
>>> Ah, you make a good point about the dev list...I am using IntelliJ on a
>>> Mac. When I tried building without any changes, I also encountered
>>> failures. I can re-run the non-modified sources to do a diff and see if
>>> the errors were different...
>>>
>>> Sent from my iPhone
>>>
>>> On Jun 11, 2018, at 15:11, Pramod Immaneni <pr...@gmail.com>
>>> wrote:
>>>
>>> Aaron,
>>>
>>> The question might be best served on the dev@apex.apache.org mailing
>>> list as you are starting to make changes to the sources. Nothing obvious
>>> jumps out to be, you don't need hadoop on your system to build the sources
>>> successfully with all the tests passing. The test failures you are seeing
>>> may or may not be related to the kryo change. What happens when you try to
>>> build the original sources with changing the Kryo version? What is your
>>> environment, mac, linux?
>>>
>>> On Mon, Jun 11, 2018 at 12:03 PM Aaron Bossert <aa...@punchcyber.com>
>>> wrote:
>>>
>>>> I know it has been a little while since I brought this up...I pulled
>>>> down the 3.7.1-RELEASE branch from GitHub in order to test out the change
>>>> to Kryo v. 4.0.2 and encountered no problems while building, however, when
>>>> I try to do mvn install, there are several failed tests. I am hoping, that
>>>> other than the obvious (to me, anyway) errors such as not finding
>>>> yarn-site.xml on my laptop which does not have hadoop installed, someone
>>>> may be able to see what is going on?
>>>>
>>>>
>>>> -------------------------------------------------------------------------------
>>>> Test set: com.datatorrent.stram.StramRecoveryTest
>>>>
>>>> -------------------------------------------------------------------------------
>>>> Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 6.372
>>>> sec <<< FAILURE! - in com.datatorrent.stram.StramRecoveryTest
>>>> testWriteAheadLog(com.datatorrent.stram.StramRecoveryTest) Time
>>>> elapsed: 0.115 sec <<< FAILURE!
>>>> java.lang.AssertionError: flush count expected:<1> but was:<2>
>>>> at
>>>> com.datatorrent.stram.StramRecoveryTest.testWriteAheadLog(StramRecoveryTest.java:326)
>>>>
>>>>
>>>>
>>>> -------------------------------------------------------------------------------
>>>> Test set: com.datatorrent.stram.engine.StatsTest
>>>>
>>>> -------------------------------------------------------------------------------
>>>> Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 21.894
>>>> sec <<< FAILURE! - in com.datatorrent.stram.engine.StatsTest
>>>> testQueueSizeForContainerLocalOperators(com.datatorrent.stram.engine.StatsTest)
>>>> Time elapsed: 3.266 sec <<< FAILURE!
>>>> java.lang.AssertionError: Validate input port queue size -1
>>>> at
>>>> com.datatorrent.stram.engine.StatsTest.baseTestForQueueSize(StatsTest.java:270)
>>>> at
>>>> com.datatorrent.stram.engine.StatsTest.testQueueSizeForContainerLocalOperators(StatsTest.java:285)
>>>>
>>>>
>>>> -------------------------------------------------------------------------------
>>>> Test set: com.datatorrent.stram.StramMiniClusterTest
>>>>
>>>> -------------------------------------------------------------------------------
>>>> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.002
>>>> sec <<< FAILURE! - in com.datatorrent.stram.StramMiniClusterTest
>>>> com.datatorrent.stram.StramMiniClusterTest Time elapsed: 0.002 sec
>>>> <<< ERROR!
>>>> java.io.FileNotFoundException:
>>>> /Users/mbossert/Library/Mobile%20Documents/com~apple~CloudDocs/Punch/IdeaProjects/apex-core/engine/target/test-classes/yarn-site.xml
>>>> (No such file or directory)
>>>> at
>>>> com.datatorrent.stram.StramMiniClusterTest.setup(StramMiniClusterTest.java:151)
>>>>
>>>>
>>>> -------------------------------------------------------------------------------
>>>> Test set: com.datatorrent.stram.StramLocalClusterTest
>>>>
>>>> -------------------------------------------------------------------------------
>>>> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 15.081
>>>> sec <<< FAILURE! - in com.datatorrent.stram.StramLocalClusterTest
>>>> testAppPath(com.datatorrent.stram.StramLocalClusterTest) Time elapsed:
>>>> 0.138 sec <<< FAILURE!
>>>> java.lang.AssertionError: checkpoint path exists
>>>> at
>>>> com.datatorrent.stram.StramLocalClusterTest.testAppPath(StramLocalClusterTest.java:413)
>>>>
>>>> testLocalClusterInitShutdown(com.datatorrent.stram.StramLocalClusterTest)
>>>> Time elapsed: 5.191 sec <<< FAILURE!
>>>> java.lang.AssertionError:
>>>> ./target/com.datatorrent.stram.StramLocalClusterTest-testLocalClusterInitShutdown.out
>>>> exists
>>>> at
>>>> com.datatorrent.stram.StramLocalClusterTest.testLocalClusterInitShutdown(StramLocalClusterTest.java:117)
>>>>
>>>>
>>>> -------------------------------------------------------------------------------
>>>> Test set: com.datatorrent.stram.client.AppPackageTest
>>>>
>>>> -------------------------------------------------------------------------------
>>>> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.105
>>>> sec <<< FAILURE! - in com.datatorrent.stram.client.AppPackageTest
>>>> testAppPackage(com.datatorrent.stram.client.AppPackageTest) Time
>>>> elapsed: 0 sec <<< FAILURE!
>>>> java.lang.AssertionError: Number of applications expected:<2> but
>>>> was:<0>
>>>> at
>>>> com.datatorrent.stram.client.AppPackageTest.testAppPackage(AppPackageTest.java:91)
>>>>
>>>> testAppLevelRequiredAndDefaultProperties(com.datatorrent.stram.client.AppPackageTest)
>>>> Time elapsed: 0.001 sec <<< FAILURE!
>>>> java.lang.AssertionError: Should consist of an app called
>>>> MyFirstApplication
>>>> at
>>>> com.datatorrent.stram.client.AppPackageTest.testAppLevelRequiredAndDefaultProperties(AppPackageTest.java:171)
>>>>
>>>>
>>>>
>>>> -------------------------------------------------------------------------------
>>>> Test set: com.datatorrent.stram.cli.ApexCliTest
>>>>
>>>> -------------------------------------------------------------------------------
>>>> Tests run: 7, Failures: 2, Errors: 2, Skipped: 0, Time elapsed: 0.938
>>>> sec <<< FAILURE! - in com.datatorrent.stram.cli.ApexCliTest
>>>> testAppFromOnlyConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
>>>> Time elapsed: 0.104 sec <<< ERROR!
>>>> com.datatorrent.stram.cli.ApexCli$CliException: No applications in
>>>> Application Package
>>>> at
>>>> com.datatorrent.stram.cli.ApexCliTest.testAppFromOnlyConfigPackage(ApexCliTest.java:235)
>>>>
>>>> testLaunchAppPackagePropertyPrecedence(com.datatorrent.stram.cli.ApexCliTest)
>>>> Time elapsed: 0.045 sec <<< FAILURE!
>>>> org.junit.ComparisonFailure: expected:<[app]-default> but
>>>> was:<[package]-default>
>>>> at
>>>> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePropertyPrecedence(ApexCliTest.java:143)
>>>>
>>>> testLaunchAppPackageParametersWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
>>>> Time elapsed: 0.045 sec <<< ERROR!
>>>> com.datatorrent.stram.cli.ApexCli$CliException: No applications in
>>>> Application Package matching "MyFirstApplication"
>>>> at
>>>> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackageParametersWithConfigPackage(ApexCliTest.java:155)
>>>>
>>>> testLaunchAppPackagePrecedenceWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
>>>> Time elapsed: 0.058 sec <<< FAILURE!
>>>> org.junit.ComparisonFailure: expected:<[app]-default> but
>>>> was:<[package]-default>
>>>> at
>>>> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePrecedenceWithConfigPackage(ApexCliTest.java:204)
>>>>
>>>>
>>>> On Thu, May 17, 2018 at 1:31 PM, Aaron Bossert <aa...@punchcyber.com>
>>>> wrote:
>>>>
>>>>> Thanks, Thomas! I have implemented the fix you suggested
>>>>> (@FieldSerializer.BInd) and also created a JIRA ticket:
>>>>> https://issues.apache.org/jira/browse/APEXCORE-816
>>>>>
>>>>> On Thu, May 17, 2018 at 1:31 AM, Thomas Weise <th...@apache.org> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> It is not necessary to use a different stream codec when you have
>>>>>> control over the class that is causing the error. You can annotate a
>>>>>> different field serializer instead, using the @FieldSerializer.Bind
>>>>>> annotation. Here is an example:
>>>>>>
>>>>>>
>>>>>> https://github.com/apache/apex-malhar/blob/2fe2903bfe65055c7fd361a97549b6e7e12d4289/library/src/main/java/org/apache/apex/malhar/lib/window/WindowOption.java#L50
>>>>>>
>>>>>> It would be good to upgrade Kryo to 4.x in apex-core master. Would
>>>>>> you like to create a JIRA for it? (
>>>>>> http://apex.apache.org/contributing.html)
>>>>>>
>>>>>> Thanks,
>>>>>> Thomas
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, May 16, 2018 at 10:31 AM, Aaron Bossert <aaron@punchcyber.com
>>>>>> > wrote:
>>>>>>
>>>>>>> I am writing an application that includes a custom object. One of
>>>>>>> the fields is of type Instant and seems to not agree with the version of
>>>>>>> Kryo that is used in Apex (see exception below). I now that kryo 4.X
>>>>>>> supports serializing Instant fields, but for the life of me, cannot seem to
>>>>>>> find a way to either get Apex to use a newer version of Kryo. Can anyone
>>>>>>> help? Perhaps setting a different streamcodec that uses the newer version
>>>>>>> of kryo would work, but I cannot seem to find an example of how to set a
>>>>>>> different streamcodec despite references to the two (Kryo and Java) that
>>>>>>> ship with Apex.
>>>>>>>
>>>>>>> ERROR com.datatorrent.stram.codec.DefaultStatefulStreamCodec -
>>>>>>> Catastrophic Error: Execution halted due to Kryo exception!
>>>>>>> com.esotericsoftware.kryo.KryoException: Class cannot be created
>>>>>>> (missing no-arg constructor): java.time.Instant
>>>>>>> Serialization trace:
>>>>>>> systemTime
>>>>>>> (com.punchcyber.streaming.pipeline.apex.datatypes.evtx.EvtxEvent)
>>>>>>> at
>>>>>>> com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
>>>>>>> at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
>>>>>>> at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
>>>>>>> at
>>>>>>> com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
>>>>>>> at
>>>>>>> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
>>>>>>> at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730)
>>>>>>> at
>>>>>>> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113)
>>>>>>> at
>>>>>>> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
>>>>>>> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
>>>>>>> at com.datatorrent.stram.codec.De
>>>>>>> faultStatefulStreamCodec.fromDataStatePair(DefaultStatefulStreamCodec.java:98)
>>>>>>> at
>>>>>>> com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.processPayload(BufferServerSubscriber.java:391)
>>>>>>> at
>>>>>>> com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.sweep(BufferServerSubscriber.java:339)
>>>>>>> at com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:269)
>>>>>>> at
>>>>>>> com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1429)
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> M. Aaron Bossert
>>>>>>> (571) 242-4021
>>>>>>> Punch Cyber Analytics Group
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> M. Aaron Bossert
>>>>> (571) 242-4021
>>>>> Punch Cyber Analytics Group
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> M. Aaron Bossert
>>>> (571) 242-4021
>>>> Punch Cyber Analytics Group
>>>>
>>>>
>>>>
>>
>> --
>>
>> M. Aaron Bossert
>> (571) 242-4021
>> Punch Cyber Analytics Group
>>
>>
>>
--
M. Aaron Bossert
(571) 242-4021
Punch Cyber Analytics Group
Re: Kryo version and default serializer
Posted by Aaron Bossert <aa...@punchcyber.com>.
Pramod,
In a nutshell, yes. I can set a @FieldSerializer to the fields in
question...the only one I have run into thus far is Instant...but it would
be great to not need to annotate each field this way...Also, I could see
this becoming problematic should I encounter more than a couple of fields
that are not serializable using the older version of Kryo (but presumably
would work with a more current one)...though admitedly, it would not be
that big a deal in the interim. It would be much cleaner to have an update
to the most current version of Kryo, IMHO...though I understand that is
something of a lift. By the way, Kryo 5 just came out yesterday, so if
there is to be effort expended, it might be good to go with the latest
version...though, heads up, I tried it out and there seem to be quite a few
changes that will be needed...new methods, and definitely some replaced
ones...
On Tue, Jun 19, 2018 at 10:37 PM Pramod Immaneni <pr...@gmail.com>
wrote:
> Hi Aaron,
>
> While we are debugging the test failures on dev, I didn't fully understand
> the last question in your email. Looks like you mentioned that the
> workaround for adding the @FieldSerializer to the field is working. Are you
> looking for an alternative to this workaround and trying to set a default
> serializer for all fields of type Instant (the field type you mentioned in
> the earlier email) so that you don't have to set the annotation each time?
>
> Thanks
>
> On Tue, Jun 19, 2018 at 12:59 PM Aaron Bossert <aa...@punchcyber.com>
> wrote:
>
>> I sent an e-mail to the dev list...but have not heard any responses. How
>> active is that list? Do you know? Basically, I have a workaround that
>> will deal with those "things" that Kryo v 2.24.0 will not handle (but 4.0.2
>> or 5.0.0-RC1 would) using the @FieldSerializer ...but this is not ideal. I
>> have two options left: go it alone and try to update Kryo in the apex-core,
>> or simply change the default serializer. The hiccup I am running into is
>> that I don't see how (from the documentation) to set the default
>> serializer...perhaps I am missing it. Can someone point me to where that
>> is in the docs?
>>
>> On Mon, Jun 11, 2018 at 4:20 PM Aaron Bossert <aa...@punchcyber.com>
>> wrote:
>>
>>> Ah, you make a good point about the dev list...I am using IntelliJ on a
>>> Mac. When I tried building without any changes, I also encountered
>>> failures. I can re-run the non-modified sources to do a diff and see if
>>> the errors were different...
>>>
>>> Sent from my iPhone
>>>
>>> On Jun 11, 2018, at 15:11, Pramod Immaneni <pr...@gmail.com>
>>> wrote:
>>>
>>> Aaron,
>>>
>>> The question might be best served on the dev@apex.apache.org mailing
>>> list as you are starting to make changes to the sources. Nothing obvious
>>> jumps out to be, you don't need hadoop on your system to build the sources
>>> successfully with all the tests passing. The test failures you are seeing
>>> may or may not be related to the kryo change. What happens when you try to
>>> build the original sources with changing the Kryo version? What is your
>>> environment, mac, linux?
>>>
>>> On Mon, Jun 11, 2018 at 12:03 PM Aaron Bossert <aa...@punchcyber.com>
>>> wrote:
>>>
>>>> I know it has been a little while since I brought this up...I pulled
>>>> down the 3.7.1-RELEASE branch from GitHub in order to test out the change
>>>> to Kryo v. 4.0.2 and encountered no problems while building, however, when
>>>> I try to do mvn install, there are several failed tests. I am hoping, that
>>>> other than the obvious (to me, anyway) errors such as not finding
>>>> yarn-site.xml on my laptop which does not have hadoop installed, someone
>>>> may be able to see what is going on?
>>>>
>>>>
>>>> -------------------------------------------------------------------------------
>>>> Test set: com.datatorrent.stram.StramRecoveryTest
>>>>
>>>> -------------------------------------------------------------------------------
>>>> Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 6.372
>>>> sec <<< FAILURE! - in com.datatorrent.stram.StramRecoveryTest
>>>> testWriteAheadLog(com.datatorrent.stram.StramRecoveryTest) Time
>>>> elapsed: 0.115 sec <<< FAILURE!
>>>> java.lang.AssertionError: flush count expected:<1> but was:<2>
>>>> at
>>>> com.datatorrent.stram.StramRecoveryTest.testWriteAheadLog(StramRecoveryTest.java:326)
>>>>
>>>>
>>>>
>>>> -------------------------------------------------------------------------------
>>>> Test set: com.datatorrent.stram.engine.StatsTest
>>>>
>>>> -------------------------------------------------------------------------------
>>>> Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 21.894
>>>> sec <<< FAILURE! - in com.datatorrent.stram.engine.StatsTest
>>>> testQueueSizeForContainerLocalOperators(com.datatorrent.stram.engine.StatsTest)
>>>> Time elapsed: 3.266 sec <<< FAILURE!
>>>> java.lang.AssertionError: Validate input port queue size -1
>>>> at
>>>> com.datatorrent.stram.engine.StatsTest.baseTestForQueueSize(StatsTest.java:270)
>>>> at
>>>> com.datatorrent.stram.engine.StatsTest.testQueueSizeForContainerLocalOperators(StatsTest.java:285)
>>>>
>>>>
>>>> -------------------------------------------------------------------------------
>>>> Test set: com.datatorrent.stram.StramMiniClusterTest
>>>>
>>>> -------------------------------------------------------------------------------
>>>> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.002
>>>> sec <<< FAILURE! - in com.datatorrent.stram.StramMiniClusterTest
>>>> com.datatorrent.stram.StramMiniClusterTest Time elapsed: 0.002 sec
>>>> <<< ERROR!
>>>> java.io.FileNotFoundException:
>>>> /Users/mbossert/Library/Mobile%20Documents/com~apple~CloudDocs/Punch/IdeaProjects/apex-core/engine/target/test-classes/yarn-site.xml
>>>> (No such file or directory)
>>>> at
>>>> com.datatorrent.stram.StramMiniClusterTest.setup(StramMiniClusterTest.java:151)
>>>>
>>>>
>>>> -------------------------------------------------------------------------------
>>>> Test set: com.datatorrent.stram.StramLocalClusterTest
>>>>
>>>> -------------------------------------------------------------------------------
>>>> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 15.081
>>>> sec <<< FAILURE! - in com.datatorrent.stram.StramLocalClusterTest
>>>> testAppPath(com.datatorrent.stram.StramLocalClusterTest) Time elapsed:
>>>> 0.138 sec <<< FAILURE!
>>>> java.lang.AssertionError: checkpoint path exists
>>>> at
>>>> com.datatorrent.stram.StramLocalClusterTest.testAppPath(StramLocalClusterTest.java:413)
>>>>
>>>> testLocalClusterInitShutdown(com.datatorrent.stram.StramLocalClusterTest)
>>>> Time elapsed: 5.191 sec <<< FAILURE!
>>>> java.lang.AssertionError:
>>>> ./target/com.datatorrent.stram.StramLocalClusterTest-testLocalClusterInitShutdown.out
>>>> exists
>>>> at
>>>> com.datatorrent.stram.StramLocalClusterTest.testLocalClusterInitShutdown(StramLocalClusterTest.java:117)
>>>>
>>>>
>>>> -------------------------------------------------------------------------------
>>>> Test set: com.datatorrent.stram.client.AppPackageTest
>>>>
>>>> -------------------------------------------------------------------------------
>>>> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.105
>>>> sec <<< FAILURE! - in com.datatorrent.stram.client.AppPackageTest
>>>> testAppPackage(com.datatorrent.stram.client.AppPackageTest) Time
>>>> elapsed: 0 sec <<< FAILURE!
>>>> java.lang.AssertionError: Number of applications expected:<2> but
>>>> was:<0>
>>>> at
>>>> com.datatorrent.stram.client.AppPackageTest.testAppPackage(AppPackageTest.java:91)
>>>>
>>>> testAppLevelRequiredAndDefaultProperties(com.datatorrent.stram.client.AppPackageTest)
>>>> Time elapsed: 0.001 sec <<< FAILURE!
>>>> java.lang.AssertionError: Should consist of an app called
>>>> MyFirstApplication
>>>> at
>>>> com.datatorrent.stram.client.AppPackageTest.testAppLevelRequiredAndDefaultProperties(AppPackageTest.java:171)
>>>>
>>>>
>>>>
>>>> -------------------------------------------------------------------------------
>>>> Test set: com.datatorrent.stram.cli.ApexCliTest
>>>>
>>>> -------------------------------------------------------------------------------
>>>> Tests run: 7, Failures: 2, Errors: 2, Skipped: 0, Time elapsed: 0.938
>>>> sec <<< FAILURE! - in com.datatorrent.stram.cli.ApexCliTest
>>>> testAppFromOnlyConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
>>>> Time elapsed: 0.104 sec <<< ERROR!
>>>> com.datatorrent.stram.cli.ApexCli$CliException: No applications in
>>>> Application Package
>>>> at
>>>> com.datatorrent.stram.cli.ApexCliTest.testAppFromOnlyConfigPackage(ApexCliTest.java:235)
>>>>
>>>> testLaunchAppPackagePropertyPrecedence(com.datatorrent.stram.cli.ApexCliTest)
>>>> Time elapsed: 0.045 sec <<< FAILURE!
>>>> org.junit.ComparisonFailure: expected:<[app]-default> but
>>>> was:<[package]-default>
>>>> at
>>>> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePropertyPrecedence(ApexCliTest.java:143)
>>>>
>>>> testLaunchAppPackageParametersWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
>>>> Time elapsed: 0.045 sec <<< ERROR!
>>>> com.datatorrent.stram.cli.ApexCli$CliException: No applications in
>>>> Application Package matching "MyFirstApplication"
>>>> at
>>>> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackageParametersWithConfigPackage(ApexCliTest.java:155)
>>>>
>>>> testLaunchAppPackagePrecedenceWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
>>>> Time elapsed: 0.058 sec <<< FAILURE!
>>>> org.junit.ComparisonFailure: expected:<[app]-default> but
>>>> was:<[package]-default>
>>>> at
>>>> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePrecedenceWithConfigPackage(ApexCliTest.java:204)
>>>>
>>>>
>>>> On Thu, May 17, 2018 at 1:31 PM, Aaron Bossert <aa...@punchcyber.com>
>>>> wrote:
>>>>
>>>>> Thanks, Thomas! I have implemented the fix you suggested
>>>>> (@FieldSerializer.BInd) and also created a JIRA ticket:
>>>>> https://issues.apache.org/jira/browse/APEXCORE-816
>>>>>
>>>>> On Thu, May 17, 2018 at 1:31 AM, Thomas Weise <th...@apache.org> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> It is not necessary to use a different stream codec when you have
>>>>>> control over the class that is causing the error. You can annotate a
>>>>>> different field serializer instead, using the @FieldSerializer.Bind
>>>>>> annotation. Here is an example:
>>>>>>
>>>>>>
>>>>>> https://github.com/apache/apex-malhar/blob/2fe2903bfe65055c7fd361a97549b6e7e12d4289/library/src/main/java/org/apache/apex/malhar/lib/window/WindowOption.java#L50
>>>>>>
>>>>>> It would be good to upgrade Kryo to 4.x in apex-core master. Would
>>>>>> you like to create a JIRA for it? (
>>>>>> http://apex.apache.org/contributing.html)
>>>>>>
>>>>>> Thanks,
>>>>>> Thomas
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, May 16, 2018 at 10:31 AM, Aaron Bossert <aaron@punchcyber.com
>>>>>> > wrote:
>>>>>>
>>>>>>> I am writing an application that includes a custom object. One of
>>>>>>> the fields is of type Instant and seems to not agree with the version of
>>>>>>> Kryo that is used in Apex (see exception below). I now that kryo 4.X
>>>>>>> supports serializing Instant fields, but for the life of me, cannot seem to
>>>>>>> find a way to either get Apex to use a newer version of Kryo. Can anyone
>>>>>>> help? Perhaps setting a different streamcodec that uses the newer version
>>>>>>> of kryo would work, but I cannot seem to find an example of how to set a
>>>>>>> different streamcodec despite references to the two (Kryo and Java) that
>>>>>>> ship with Apex.
>>>>>>>
>>>>>>> ERROR com.datatorrent.stram.codec.DefaultStatefulStreamCodec -
>>>>>>> Catastrophic Error: Execution halted due to Kryo exception!
>>>>>>> com.esotericsoftware.kryo.KryoException: Class cannot be created
>>>>>>> (missing no-arg constructor): java.time.Instant
>>>>>>> Serialization trace:
>>>>>>> systemTime
>>>>>>> (com.punchcyber.streaming.pipeline.apex.datatypes.evtx.EvtxEvent)
>>>>>>> at
>>>>>>> com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
>>>>>>> at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
>>>>>>> at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
>>>>>>> at
>>>>>>> com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
>>>>>>> at
>>>>>>> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
>>>>>>> at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730)
>>>>>>> at
>>>>>>> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113)
>>>>>>> at
>>>>>>> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
>>>>>>> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
>>>>>>> at com.datatorrent.stram.codec.De
>>>>>>> faultStatefulStreamCodec.fromDataStatePair(DefaultStatefulStreamCodec.java:98)
>>>>>>> at
>>>>>>> com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.processPayload(BufferServerSubscriber.java:391)
>>>>>>> at
>>>>>>> com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.sweep(BufferServerSubscriber.java:339)
>>>>>>> at com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:269)
>>>>>>> at
>>>>>>> com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1429)
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> M. Aaron Bossert
>>>>>>> (571) 242-4021
>>>>>>> Punch Cyber Analytics Group
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> M. Aaron Bossert
>>>>> (571) 242-4021
>>>>> Punch Cyber Analytics Group
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> M. Aaron Bossert
>>>> (571) 242-4021
>>>> Punch Cyber Analytics Group
>>>>
>>>>
>>>>
>>
>> --
>>
>> M. Aaron Bossert
>> (571) 242-4021
>> Punch Cyber Analytics Group
>>
>>
>>
--
M. Aaron Bossert
(571) 242-4021
Punch Cyber Analytics Group
Re: Kryo version and default serializer
Posted by Pramod Immaneni <pr...@gmail.com>.
Hi Aaron,
While we are debugging the test failures on dev, I didn't fully understand
the last question in your email. Looks like you mentioned that the
workaround for adding the @FieldSerializer to the field is working. Are you
looking for an alternative to this workaround and trying to set a default
serializer for all fields of type Instant (the field type you mentioned in
the earlier email) so that you don't have to set the annotation each time?
Thanks
On Tue, Jun 19, 2018 at 12:59 PM Aaron Bossert <aa...@punchcyber.com> wrote:
> I sent an e-mail to the dev list...but have not heard any responses. How
> active is that list? Do you know? Basically, I have a workaround that
> will deal with those "things" that Kryo v 2.24.0 will not handle (but 4.0.2
> or 5.0.0-RC1 would) using the @FieldSerializer ...but this is not ideal. I
> have two options left: go it alone and try to update Kryo in the apex-core,
> or simply change the default serializer. The hiccup I am running into is
> that I don't see how (from the documentation) to set the default
> serializer...perhaps I am missing it. Can someone point me to where that
> is in the docs?
>
> On Mon, Jun 11, 2018 at 4:20 PM Aaron Bossert <aa...@punchcyber.com>
> wrote:
>
>> Ah, you make a good point about the dev list...I am using IntelliJ on a
>> Mac. When I tried building without any changes, I also encountered
>> failures. I can re-run the non-modified sources to do a diff and see if
>> the errors were different...
>>
>> Sent from my iPhone
>>
>> On Jun 11, 2018, at 15:11, Pramod Immaneni <pr...@gmail.com>
>> wrote:
>>
>> Aaron,
>>
>> The question might be best served on the dev@apex.apache.org mailing
>> list as you are starting to make changes to the sources. Nothing obvious
>> jumps out to be, you don't need hadoop on your system to build the sources
>> successfully with all the tests passing. The test failures you are seeing
>> may or may not be related to the kryo change. What happens when you try to
>> build the original sources with changing the Kryo version? What is your
>> environment, mac, linux?
>>
>> On Mon, Jun 11, 2018 at 12:03 PM Aaron Bossert <aa...@punchcyber.com>
>> wrote:
>>
>>> I know it has been a little while since I brought this up...I pulled
>>> down the 3.7.1-RELEASE branch from GitHub in order to test out the change
>>> to Kryo v. 4.0.2 and encountered no problems while building, however, when
>>> I try to do mvn install, there are several failed tests. I am hoping, that
>>> other than the obvious (to me, anyway) errors such as not finding
>>> yarn-site.xml on my laptop which does not have hadoop installed, someone
>>> may be able to see what is going on?
>>>
>>>
>>> -------------------------------------------------------------------------------
>>> Test set: com.datatorrent.stram.StramRecoveryTest
>>>
>>> -------------------------------------------------------------------------------
>>> Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 6.372
>>> sec <<< FAILURE! - in com.datatorrent.stram.StramRecoveryTest
>>> testWriteAheadLog(com.datatorrent.stram.StramRecoveryTest) Time
>>> elapsed: 0.115 sec <<< FAILURE!
>>> java.lang.AssertionError: flush count expected:<1> but was:<2>
>>> at
>>> com.datatorrent.stram.StramRecoveryTest.testWriteAheadLog(StramRecoveryTest.java:326)
>>>
>>>
>>>
>>> -------------------------------------------------------------------------------
>>> Test set: com.datatorrent.stram.engine.StatsTest
>>>
>>> -------------------------------------------------------------------------------
>>> Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 21.894
>>> sec <<< FAILURE! - in com.datatorrent.stram.engine.StatsTest
>>> testQueueSizeForContainerLocalOperators(com.datatorrent.stram.engine.StatsTest)
>>> Time elapsed: 3.266 sec <<< FAILURE!
>>> java.lang.AssertionError: Validate input port queue size -1
>>> at
>>> com.datatorrent.stram.engine.StatsTest.baseTestForQueueSize(StatsTest.java:270)
>>> at
>>> com.datatorrent.stram.engine.StatsTest.testQueueSizeForContainerLocalOperators(StatsTest.java:285)
>>>
>>>
>>> -------------------------------------------------------------------------------
>>> Test set: com.datatorrent.stram.StramMiniClusterTest
>>>
>>> -------------------------------------------------------------------------------
>>> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.002
>>> sec <<< FAILURE! - in com.datatorrent.stram.StramMiniClusterTest
>>> com.datatorrent.stram.StramMiniClusterTest Time elapsed: 0.002 sec <<<
>>> ERROR!
>>> java.io.FileNotFoundException:
>>> /Users/mbossert/Library/Mobile%20Documents/com~apple~CloudDocs/Punch/IdeaProjects/apex-core/engine/target/test-classes/yarn-site.xml
>>> (No such file or directory)
>>> at
>>> com.datatorrent.stram.StramMiniClusterTest.setup(StramMiniClusterTest.java:151)
>>>
>>>
>>> -------------------------------------------------------------------------------
>>> Test set: com.datatorrent.stram.StramLocalClusterTest
>>>
>>> -------------------------------------------------------------------------------
>>> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 15.081
>>> sec <<< FAILURE! - in com.datatorrent.stram.StramLocalClusterTest
>>> testAppPath(com.datatorrent.stram.StramLocalClusterTest) Time elapsed:
>>> 0.138 sec <<< FAILURE!
>>> java.lang.AssertionError: checkpoint path exists
>>> at
>>> com.datatorrent.stram.StramLocalClusterTest.testAppPath(StramLocalClusterTest.java:413)
>>>
>>> testLocalClusterInitShutdown(com.datatorrent.stram.StramLocalClusterTest)
>>> Time elapsed: 5.191 sec <<< FAILURE!
>>> java.lang.AssertionError:
>>> ./target/com.datatorrent.stram.StramLocalClusterTest-testLocalClusterInitShutdown.out
>>> exists
>>> at
>>> com.datatorrent.stram.StramLocalClusterTest.testLocalClusterInitShutdown(StramLocalClusterTest.java:117)
>>>
>>>
>>> -------------------------------------------------------------------------------
>>> Test set: com.datatorrent.stram.client.AppPackageTest
>>>
>>> -------------------------------------------------------------------------------
>>> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.105
>>> sec <<< FAILURE! - in com.datatorrent.stram.client.AppPackageTest
>>> testAppPackage(com.datatorrent.stram.client.AppPackageTest) Time
>>> elapsed: 0 sec <<< FAILURE!
>>> java.lang.AssertionError: Number of applications expected:<2> but was:<0>
>>> at
>>> com.datatorrent.stram.client.AppPackageTest.testAppPackage(AppPackageTest.java:91)
>>>
>>> testAppLevelRequiredAndDefaultProperties(com.datatorrent.stram.client.AppPackageTest)
>>> Time elapsed: 0.001 sec <<< FAILURE!
>>> java.lang.AssertionError: Should consist of an app called
>>> MyFirstApplication
>>> at
>>> com.datatorrent.stram.client.AppPackageTest.testAppLevelRequiredAndDefaultProperties(AppPackageTest.java:171)
>>>
>>>
>>>
>>> -------------------------------------------------------------------------------
>>> Test set: com.datatorrent.stram.cli.ApexCliTest
>>>
>>> -------------------------------------------------------------------------------
>>> Tests run: 7, Failures: 2, Errors: 2, Skipped: 0, Time elapsed: 0.938
>>> sec <<< FAILURE! - in com.datatorrent.stram.cli.ApexCliTest
>>> testAppFromOnlyConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
>>> Time elapsed: 0.104 sec <<< ERROR!
>>> com.datatorrent.stram.cli.ApexCli$CliException: No applications in
>>> Application Package
>>> at
>>> com.datatorrent.stram.cli.ApexCliTest.testAppFromOnlyConfigPackage(ApexCliTest.java:235)
>>>
>>> testLaunchAppPackagePropertyPrecedence(com.datatorrent.stram.cli.ApexCliTest)
>>> Time elapsed: 0.045 sec <<< FAILURE!
>>> org.junit.ComparisonFailure: expected:<[app]-default> but
>>> was:<[package]-default>
>>> at
>>> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePropertyPrecedence(ApexCliTest.java:143)
>>>
>>> testLaunchAppPackageParametersWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
>>> Time elapsed: 0.045 sec <<< ERROR!
>>> com.datatorrent.stram.cli.ApexCli$CliException: No applications in
>>> Application Package matching "MyFirstApplication"
>>> at
>>> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackageParametersWithConfigPackage(ApexCliTest.java:155)
>>>
>>> testLaunchAppPackagePrecedenceWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
>>> Time elapsed: 0.058 sec <<< FAILURE!
>>> org.junit.ComparisonFailure: expected:<[app]-default> but
>>> was:<[package]-default>
>>> at
>>> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePrecedenceWithConfigPackage(ApexCliTest.java:204)
>>>
>>>
>>> On Thu, May 17, 2018 at 1:31 PM, Aaron Bossert <aa...@punchcyber.com>
>>> wrote:
>>>
>>>> Thanks, Thomas! I have implemented the fix you suggested
>>>> (@FieldSerializer.BInd) and also created a JIRA ticket:
>>>> https://issues.apache.org/jira/browse/APEXCORE-816
>>>>
>>>> On Thu, May 17, 2018 at 1:31 AM, Thomas Weise <th...@apache.org> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> It is not necessary to use a different stream codec when you have
>>>>> control over the class that is causing the error. You can annotate a
>>>>> different field serializer instead, using the @FieldSerializer.Bind
>>>>> annotation. Here is an example:
>>>>>
>>>>>
>>>>> https://github.com/apache/apex-malhar/blob/2fe2903bfe65055c7fd361a97549b6e7e12d4289/library/src/main/java/org/apache/apex/malhar/lib/window/WindowOption.java#L50
>>>>>
>>>>> It would be good to upgrade Kryo to 4.x in apex-core master. Would you
>>>>> like to create a JIRA for it? (
>>>>> http://apex.apache.org/contributing.html)
>>>>>
>>>>> Thanks,
>>>>> Thomas
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Wed, May 16, 2018 at 10:31 AM, Aaron Bossert <aa...@punchcyber.com>
>>>>> wrote:
>>>>>
>>>>>> I am writing an application that includes a custom object. One of
>>>>>> the fields is of type Instant and seems to not agree with the version of
>>>>>> Kryo that is used in Apex (see exception below). I now that kryo 4.X
>>>>>> supports serializing Instant fields, but for the life of me, cannot seem to
>>>>>> find a way to either get Apex to use a newer version of Kryo. Can anyone
>>>>>> help? Perhaps setting a different streamcodec that uses the newer version
>>>>>> of kryo would work, but I cannot seem to find an example of how to set a
>>>>>> different streamcodec despite references to the two (Kryo and Java) that
>>>>>> ship with Apex.
>>>>>>
>>>>>> ERROR com.datatorrent.stram.codec.DefaultStatefulStreamCodec -
>>>>>> Catastrophic Error: Execution halted due to Kryo exception!
>>>>>> com.esotericsoftware.kryo.KryoException: Class cannot be created
>>>>>> (missing no-arg constructor): java.time.Instant
>>>>>> Serialization trace:
>>>>>> systemTime
>>>>>> (com.punchcyber.streaming.pipeline.apex.datatypes.evtx.EvtxEvent)
>>>>>> at
>>>>>> com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
>>>>>> at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
>>>>>> at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
>>>>>> at
>>>>>> com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
>>>>>> at
>>>>>> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
>>>>>> at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730)
>>>>>> at
>>>>>> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113)
>>>>>> at
>>>>>> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
>>>>>> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
>>>>>> at com.datatorrent.stram.codec.De
>>>>>> faultStatefulStreamCodec.fromDataStatePair(DefaultStatefulStreamCodec.java:98)
>>>>>> at
>>>>>> com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.processPayload(BufferServerSubscriber.java:391)
>>>>>> at
>>>>>> com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.sweep(BufferServerSubscriber.java:339)
>>>>>> at com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:269)
>>>>>> at
>>>>>> com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1429)
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> M. Aaron Bossert
>>>>>> (571) 242-4021
>>>>>> Punch Cyber Analytics Group
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> M. Aaron Bossert
>>>> (571) 242-4021
>>>> Punch Cyber Analytics Group
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> M. Aaron Bossert
>>> (571) 242-4021
>>> Punch Cyber Analytics Group
>>>
>>>
>>>
>
> --
>
> M. Aaron Bossert
> (571) 242-4021
> Punch Cyber Analytics Group
>
>
>
Re: Kryo version and default serializer
Posted by Pramod Immaneni <pr...@gmail.com>.
Hi Aaron,
While we are debugging the test failures on dev, I didn't fully understand
the last question in your email. Looks like you mentioned that the
workaround for adding the @FieldSerializer to the field is working. Are you
looking for an alternative to this workaround and trying to set a default
serializer for all fields of type Instant (the field type you mentioned in
the earlier email) so that you don't have to set the annotation each time?
Thanks
On Tue, Jun 19, 2018 at 12:59 PM Aaron Bossert <aa...@punchcyber.com> wrote:
> I sent an e-mail to the dev list...but have not heard any responses. How
> active is that list? Do you know? Basically, I have a workaround that
> will deal with those "things" that Kryo v 2.24.0 will not handle (but 4.0.2
> or 5.0.0-RC1 would) using the @FieldSerializer ...but this is not ideal. I
> have two options left: go it alone and try to update Kryo in the apex-core,
> or simply change the default serializer. The hiccup I am running into is
> that I don't see how (from the documentation) to set the default
> serializer...perhaps I am missing it. Can someone point me to where that
> is in the docs?
>
> On Mon, Jun 11, 2018 at 4:20 PM Aaron Bossert <aa...@punchcyber.com>
> wrote:
>
>> Ah, you make a good point about the dev list...I am using IntelliJ on a
>> Mac. When I tried building without any changes, I also encountered
>> failures. I can re-run the non-modified sources to do a diff and see if
>> the errors were different...
>>
>> Sent from my iPhone
>>
>> On Jun 11, 2018, at 15:11, Pramod Immaneni <pr...@gmail.com>
>> wrote:
>>
>> Aaron,
>>
>> The question might be best served on the dev@apex.apache.org mailing
>> list as you are starting to make changes to the sources. Nothing obvious
>> jumps out to be, you don't need hadoop on your system to build the sources
>> successfully with all the tests passing. The test failures you are seeing
>> may or may not be related to the kryo change. What happens when you try to
>> build the original sources with changing the Kryo version? What is your
>> environment, mac, linux?
>>
>> On Mon, Jun 11, 2018 at 12:03 PM Aaron Bossert <aa...@punchcyber.com>
>> wrote:
>>
>>> I know it has been a little while since I brought this up...I pulled
>>> down the 3.7.1-RELEASE branch from GitHub in order to test out the change
>>> to Kryo v. 4.0.2 and encountered no problems while building, however, when
>>> I try to do mvn install, there are several failed tests. I am hoping, that
>>> other than the obvious (to me, anyway) errors such as not finding
>>> yarn-site.xml on my laptop which does not have hadoop installed, someone
>>> may be able to see what is going on?
>>>
>>>
>>> -------------------------------------------------------------------------------
>>> Test set: com.datatorrent.stram.StramRecoveryTest
>>>
>>> -------------------------------------------------------------------------------
>>> Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 6.372
>>> sec <<< FAILURE! - in com.datatorrent.stram.StramRecoveryTest
>>> testWriteAheadLog(com.datatorrent.stram.StramRecoveryTest) Time
>>> elapsed: 0.115 sec <<< FAILURE!
>>> java.lang.AssertionError: flush count expected:<1> but was:<2>
>>> at
>>> com.datatorrent.stram.StramRecoveryTest.testWriteAheadLog(StramRecoveryTest.java:326)
>>>
>>>
>>>
>>> -------------------------------------------------------------------------------
>>> Test set: com.datatorrent.stram.engine.StatsTest
>>>
>>> -------------------------------------------------------------------------------
>>> Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 21.894
>>> sec <<< FAILURE! - in com.datatorrent.stram.engine.StatsTest
>>> testQueueSizeForContainerLocalOperators(com.datatorrent.stram.engine.StatsTest)
>>> Time elapsed: 3.266 sec <<< FAILURE!
>>> java.lang.AssertionError: Validate input port queue size -1
>>> at
>>> com.datatorrent.stram.engine.StatsTest.baseTestForQueueSize(StatsTest.java:270)
>>> at
>>> com.datatorrent.stram.engine.StatsTest.testQueueSizeForContainerLocalOperators(StatsTest.java:285)
>>>
>>>
>>> -------------------------------------------------------------------------------
>>> Test set: com.datatorrent.stram.StramMiniClusterTest
>>>
>>> -------------------------------------------------------------------------------
>>> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.002
>>> sec <<< FAILURE! - in com.datatorrent.stram.StramMiniClusterTest
>>> com.datatorrent.stram.StramMiniClusterTest Time elapsed: 0.002 sec <<<
>>> ERROR!
>>> java.io.FileNotFoundException:
>>> /Users/mbossert/Library/Mobile%20Documents/com~apple~CloudDocs/Punch/IdeaProjects/apex-core/engine/target/test-classes/yarn-site.xml
>>> (No such file or directory)
>>> at
>>> com.datatorrent.stram.StramMiniClusterTest.setup(StramMiniClusterTest.java:151)
>>>
>>>
>>> -------------------------------------------------------------------------------
>>> Test set: com.datatorrent.stram.StramLocalClusterTest
>>>
>>> -------------------------------------------------------------------------------
>>> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 15.081
>>> sec <<< FAILURE! - in com.datatorrent.stram.StramLocalClusterTest
>>> testAppPath(com.datatorrent.stram.StramLocalClusterTest) Time elapsed:
>>> 0.138 sec <<< FAILURE!
>>> java.lang.AssertionError: checkpoint path exists
>>> at
>>> com.datatorrent.stram.StramLocalClusterTest.testAppPath(StramLocalClusterTest.java:413)
>>>
>>> testLocalClusterInitShutdown(com.datatorrent.stram.StramLocalClusterTest)
>>> Time elapsed: 5.191 sec <<< FAILURE!
>>> java.lang.AssertionError:
>>> ./target/com.datatorrent.stram.StramLocalClusterTest-testLocalClusterInitShutdown.out
>>> exists
>>> at
>>> com.datatorrent.stram.StramLocalClusterTest.testLocalClusterInitShutdown(StramLocalClusterTest.java:117)
>>>
>>>
>>> -------------------------------------------------------------------------------
>>> Test set: com.datatorrent.stram.client.AppPackageTest
>>>
>>> -------------------------------------------------------------------------------
>>> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.105
>>> sec <<< FAILURE! - in com.datatorrent.stram.client.AppPackageTest
>>> testAppPackage(com.datatorrent.stram.client.AppPackageTest) Time
>>> elapsed: 0 sec <<< FAILURE!
>>> java.lang.AssertionError: Number of applications expected:<2> but was:<0>
>>> at
>>> com.datatorrent.stram.client.AppPackageTest.testAppPackage(AppPackageTest.java:91)
>>>
>>> testAppLevelRequiredAndDefaultProperties(com.datatorrent.stram.client.AppPackageTest)
>>> Time elapsed: 0.001 sec <<< FAILURE!
>>> java.lang.AssertionError: Should consist of an app called
>>> MyFirstApplication
>>> at
>>> com.datatorrent.stram.client.AppPackageTest.testAppLevelRequiredAndDefaultProperties(AppPackageTest.java:171)
>>>
>>>
>>>
>>> -------------------------------------------------------------------------------
>>> Test set: com.datatorrent.stram.cli.ApexCliTest
>>>
>>> -------------------------------------------------------------------------------
>>> Tests run: 7, Failures: 2, Errors: 2, Skipped: 0, Time elapsed: 0.938
>>> sec <<< FAILURE! - in com.datatorrent.stram.cli.ApexCliTest
>>> testAppFromOnlyConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
>>> Time elapsed: 0.104 sec <<< ERROR!
>>> com.datatorrent.stram.cli.ApexCli$CliException: No applications in
>>> Application Package
>>> at
>>> com.datatorrent.stram.cli.ApexCliTest.testAppFromOnlyConfigPackage(ApexCliTest.java:235)
>>>
>>> testLaunchAppPackagePropertyPrecedence(com.datatorrent.stram.cli.ApexCliTest)
>>> Time elapsed: 0.045 sec <<< FAILURE!
>>> org.junit.ComparisonFailure: expected:<[app]-default> but
>>> was:<[package]-default>
>>> at
>>> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePropertyPrecedence(ApexCliTest.java:143)
>>>
>>> testLaunchAppPackageParametersWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
>>> Time elapsed: 0.045 sec <<< ERROR!
>>> com.datatorrent.stram.cli.ApexCli$CliException: No applications in
>>> Application Package matching "MyFirstApplication"
>>> at
>>> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackageParametersWithConfigPackage(ApexCliTest.java:155)
>>>
>>> testLaunchAppPackagePrecedenceWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
>>> Time elapsed: 0.058 sec <<< FAILURE!
>>> org.junit.ComparisonFailure: expected:<[app]-default> but
>>> was:<[package]-default>
>>> at
>>> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePrecedenceWithConfigPackage(ApexCliTest.java:204)
>>>
>>>
>>> On Thu, May 17, 2018 at 1:31 PM, Aaron Bossert <aa...@punchcyber.com>
>>> wrote:
>>>
>>>> Thanks, Thomas! I have implemented the fix you suggested
>>>> (@FieldSerializer.BInd) and also created a JIRA ticket:
>>>> https://issues.apache.org/jira/browse/APEXCORE-816
>>>>
>>>> On Thu, May 17, 2018 at 1:31 AM, Thomas Weise <th...@apache.org> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> It is not necessary to use a different stream codec when you have
>>>>> control over the class that is causing the error. You can annotate a
>>>>> different field serializer instead, using the @FieldSerializer.Bind
>>>>> annotation. Here is an example:
>>>>>
>>>>>
>>>>> https://github.com/apache/apex-malhar/blob/2fe2903bfe65055c7fd361a97549b6e7e12d4289/library/src/main/java/org/apache/apex/malhar/lib/window/WindowOption.java#L50
>>>>>
>>>>> It would be good to upgrade Kryo to 4.x in apex-core master. Would you
>>>>> like to create a JIRA for it? (
>>>>> http://apex.apache.org/contributing.html)
>>>>>
>>>>> Thanks,
>>>>> Thomas
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Wed, May 16, 2018 at 10:31 AM, Aaron Bossert <aa...@punchcyber.com>
>>>>> wrote:
>>>>>
>>>>>> I am writing an application that includes a custom object. One of
>>>>>> the fields is of type Instant and seems to not agree with the version of
>>>>>> Kryo that is used in Apex (see exception below). I now that kryo 4.X
>>>>>> supports serializing Instant fields, but for the life of me, cannot seem to
>>>>>> find a way to either get Apex to use a newer version of Kryo. Can anyone
>>>>>> help? Perhaps setting a different streamcodec that uses the newer version
>>>>>> of kryo would work, but I cannot seem to find an example of how to set a
>>>>>> different streamcodec despite references to the two (Kryo and Java) that
>>>>>> ship with Apex.
>>>>>>
>>>>>> ERROR com.datatorrent.stram.codec.DefaultStatefulStreamCodec -
>>>>>> Catastrophic Error: Execution halted due to Kryo exception!
>>>>>> com.esotericsoftware.kryo.KryoException: Class cannot be created
>>>>>> (missing no-arg constructor): java.time.Instant
>>>>>> Serialization trace:
>>>>>> systemTime
>>>>>> (com.punchcyber.streaming.pipeline.apex.datatypes.evtx.EvtxEvent)
>>>>>> at
>>>>>> com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
>>>>>> at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
>>>>>> at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
>>>>>> at
>>>>>> com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
>>>>>> at
>>>>>> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
>>>>>> at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730)
>>>>>> at
>>>>>> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113)
>>>>>> at
>>>>>> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
>>>>>> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
>>>>>> at com.datatorrent.stram.codec.De
>>>>>> faultStatefulStreamCodec.fromDataStatePair(DefaultStatefulStreamCodec.java:98)
>>>>>> at
>>>>>> com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.processPayload(BufferServerSubscriber.java:391)
>>>>>> at
>>>>>> com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.sweep(BufferServerSubscriber.java:339)
>>>>>> at com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:269)
>>>>>> at
>>>>>> com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1429)
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> M. Aaron Bossert
>>>>>> (571) 242-4021
>>>>>> Punch Cyber Analytics Group
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> M. Aaron Bossert
>>>> (571) 242-4021
>>>> Punch Cyber Analytics Group
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> M. Aaron Bossert
>>> (571) 242-4021
>>> Punch Cyber Analytics Group
>>>
>>>
>>>
>
> --
>
> M. Aaron Bossert
> (571) 242-4021
> Punch Cyber Analytics Group
>
>
>
Re: Kryo version and default serializer
Posted by Aaron Bossert <aa...@punchcyber.com>.
I sent an e-mail to the dev list...but have not heard any responses. How
active is that list? Do you know? Basically, I have a workaround that
will deal with those "things" that Kryo v 2.24.0 will not handle (but 4.0.2
or 5.0.0-RC1 would) using the @FieldSerializer ...but this is not ideal. I
have two options left: go it alone and try to update Kryo in the apex-core,
or simply change the default serializer. The hiccup I am running into is
that I don't see how (from the documentation) to set the default
serializer...perhaps I am missing it. Can someone point me to where that
is in the docs?
On Mon, Jun 11, 2018 at 4:20 PM Aaron Bossert <aa...@punchcyber.com> wrote:
> Ah, you make a good point about the dev list...I am using IntelliJ on a
> Mac. When I tried building without any changes, I also encountered
> failures. I can re-run the non-modified sources to do a diff and see if
> the errors were different...
>
> Sent from my iPhone
>
> On Jun 11, 2018, at 15:11, Pramod Immaneni <pr...@gmail.com>
> wrote:
>
> Aaron,
>
> The question might be best served on the dev@apex.apache.org mailing list
> as you are starting to make changes to the sources. Nothing obvious jumps
> out to be, you don't need hadoop on your system to build the sources
> successfully with all the tests passing. The test failures you are seeing
> may or may not be related to the kryo change. What happens when you try to
> build the original sources with changing the Kryo version? What is your
> environment, mac, linux?
>
> On Mon, Jun 11, 2018 at 12:03 PM Aaron Bossert <aa...@punchcyber.com>
> wrote:
>
>> I know it has been a little while since I brought this up...I pulled down
>> the 3.7.1-RELEASE branch from GitHub in order to test out the change to
>> Kryo v. 4.0.2 and encountered no problems while building, however, when I
>> try to do mvn install, there are several failed tests. I am hoping, that
>> other than the obvious (to me, anyway) errors such as not finding
>> yarn-site.xml on my laptop which does not have hadoop installed, someone
>> may be able to see what is going on?
>>
>>
>> -------------------------------------------------------------------------------
>> Test set: com.datatorrent.stram.StramRecoveryTest
>>
>> -------------------------------------------------------------------------------
>> Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 6.372 sec
>> <<< FAILURE! - in com.datatorrent.stram.StramRecoveryTest
>> testWriteAheadLog(com.datatorrent.stram.StramRecoveryTest) Time elapsed:
>> 0.115 sec <<< FAILURE!
>> java.lang.AssertionError: flush count expected:<1> but was:<2>
>> at
>> com.datatorrent.stram.StramRecoveryTest.testWriteAheadLog(StramRecoveryTest.java:326)
>>
>>
>>
>> -------------------------------------------------------------------------------
>> Test set: com.datatorrent.stram.engine.StatsTest
>>
>> -------------------------------------------------------------------------------
>> Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 21.894
>> sec <<< FAILURE! - in com.datatorrent.stram.engine.StatsTest
>> testQueueSizeForContainerLocalOperators(com.datatorrent.stram.engine.StatsTest)
>> Time elapsed: 3.266 sec <<< FAILURE!
>> java.lang.AssertionError: Validate input port queue size -1
>> at
>> com.datatorrent.stram.engine.StatsTest.baseTestForQueueSize(StatsTest.java:270)
>> at
>> com.datatorrent.stram.engine.StatsTest.testQueueSizeForContainerLocalOperators(StatsTest.java:285)
>>
>>
>> -------------------------------------------------------------------------------
>> Test set: com.datatorrent.stram.StramMiniClusterTest
>>
>> -------------------------------------------------------------------------------
>> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.002 sec
>> <<< FAILURE! - in com.datatorrent.stram.StramMiniClusterTest
>> com.datatorrent.stram.StramMiniClusterTest Time elapsed: 0.002 sec <<<
>> ERROR!
>> java.io.FileNotFoundException:
>> /Users/mbossert/Library/Mobile%20Documents/com~apple~CloudDocs/Punch/IdeaProjects/apex-core/engine/target/test-classes/yarn-site.xml
>> (No such file or directory)
>> at
>> com.datatorrent.stram.StramMiniClusterTest.setup(StramMiniClusterTest.java:151)
>>
>>
>> -------------------------------------------------------------------------------
>> Test set: com.datatorrent.stram.StramLocalClusterTest
>>
>> -------------------------------------------------------------------------------
>> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 15.081
>> sec <<< FAILURE! - in com.datatorrent.stram.StramLocalClusterTest
>> testAppPath(com.datatorrent.stram.StramLocalClusterTest) Time elapsed:
>> 0.138 sec <<< FAILURE!
>> java.lang.AssertionError: checkpoint path exists
>> at
>> com.datatorrent.stram.StramLocalClusterTest.testAppPath(StramLocalClusterTest.java:413)
>>
>> testLocalClusterInitShutdown(com.datatorrent.stram.StramLocalClusterTest)
>> Time elapsed: 5.191 sec <<< FAILURE!
>> java.lang.AssertionError:
>> ./target/com.datatorrent.stram.StramLocalClusterTest-testLocalClusterInitShutdown.out
>> exists
>> at
>> com.datatorrent.stram.StramLocalClusterTest.testLocalClusterInitShutdown(StramLocalClusterTest.java:117)
>>
>>
>> -------------------------------------------------------------------------------
>> Test set: com.datatorrent.stram.client.AppPackageTest
>>
>> -------------------------------------------------------------------------------
>> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.105 sec
>> <<< FAILURE! - in com.datatorrent.stram.client.AppPackageTest
>> testAppPackage(com.datatorrent.stram.client.AppPackageTest) Time
>> elapsed: 0 sec <<< FAILURE!
>> java.lang.AssertionError: Number of applications expected:<2> but was:<0>
>> at
>> com.datatorrent.stram.client.AppPackageTest.testAppPackage(AppPackageTest.java:91)
>>
>> testAppLevelRequiredAndDefaultProperties(com.datatorrent.stram.client.AppPackageTest)
>> Time elapsed: 0.001 sec <<< FAILURE!
>> java.lang.AssertionError: Should consist of an app called
>> MyFirstApplication
>> at
>> com.datatorrent.stram.client.AppPackageTest.testAppLevelRequiredAndDefaultProperties(AppPackageTest.java:171)
>>
>>
>>
>> -------------------------------------------------------------------------------
>> Test set: com.datatorrent.stram.cli.ApexCliTest
>>
>> -------------------------------------------------------------------------------
>> Tests run: 7, Failures: 2, Errors: 2, Skipped: 0, Time elapsed: 0.938 sec
>> <<< FAILURE! - in com.datatorrent.stram.cli.ApexCliTest
>> testAppFromOnlyConfigPackage(com.datatorrent.stram.cli.ApexCliTest) Time
>> elapsed: 0.104 sec <<< ERROR!
>> com.datatorrent.stram.cli.ApexCli$CliException: No applications in
>> Application Package
>> at
>> com.datatorrent.stram.cli.ApexCliTest.testAppFromOnlyConfigPackage(ApexCliTest.java:235)
>>
>> testLaunchAppPackagePropertyPrecedence(com.datatorrent.stram.cli.ApexCliTest)
>> Time elapsed: 0.045 sec <<< FAILURE!
>> org.junit.ComparisonFailure: expected:<[app]-default> but
>> was:<[package]-default>
>> at
>> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePropertyPrecedence(ApexCliTest.java:143)
>>
>> testLaunchAppPackageParametersWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
>> Time elapsed: 0.045 sec <<< ERROR!
>> com.datatorrent.stram.cli.ApexCli$CliException: No applications in
>> Application Package matching "MyFirstApplication"
>> at
>> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackageParametersWithConfigPackage(ApexCliTest.java:155)
>>
>> testLaunchAppPackagePrecedenceWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
>> Time elapsed: 0.058 sec <<< FAILURE!
>> org.junit.ComparisonFailure: expected:<[app]-default> but
>> was:<[package]-default>
>> at
>> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePrecedenceWithConfigPackage(ApexCliTest.java:204)
>>
>>
>> On Thu, May 17, 2018 at 1:31 PM, Aaron Bossert <aa...@punchcyber.com>
>> wrote:
>>
>>> Thanks, Thomas! I have implemented the fix you suggested
>>> (@FieldSerializer.BInd) and also created a JIRA ticket:
>>> https://issues.apache.org/jira/browse/APEXCORE-816
>>>
>>> On Thu, May 17, 2018 at 1:31 AM, Thomas Weise <th...@apache.org> wrote:
>>>
>>>> Hi,
>>>>
>>>> It is not necessary to use a different stream codec when you have
>>>> control over the class that is causing the error. You can annotate a
>>>> different field serializer instead, using the @FieldSerializer.Bind
>>>> annotation. Here is an example:
>>>>
>>>>
>>>> https://github.com/apache/apex-malhar/blob/2fe2903bfe65055c7fd361a97549b6e7e12d4289/library/src/main/java/org/apache/apex/malhar/lib/window/WindowOption.java#L50
>>>>
>>>> It would be good to upgrade Kryo to 4.x in apex-core master. Would you
>>>> like to create a JIRA for it? (http://apex.apache.org/contributing.html
>>>> )
>>>>
>>>> Thanks,
>>>> Thomas
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Wed, May 16, 2018 at 10:31 AM, Aaron Bossert <aa...@punchcyber.com>
>>>> wrote:
>>>>
>>>>> I am writing an application that includes a custom object. One of the
>>>>> fields is of type Instant and seems to not agree with the version of Kryo
>>>>> that is used in Apex (see exception below). I now that kryo 4.X supports
>>>>> serializing Instant fields, but for the life of me, cannot seem to find a
>>>>> way to either get Apex to use a newer version of Kryo. Can anyone help?
>>>>> Perhaps setting a different streamcodec that uses the newer version of kryo
>>>>> would work, but I cannot seem to find an example of how to set a different
>>>>> streamcodec despite references to the two (Kryo and Java) that ship with
>>>>> Apex.
>>>>>
>>>>> ERROR com.datatorrent.stram.codec.DefaultStatefulStreamCodec -
>>>>> Catastrophic Error: Execution halted due to Kryo exception!
>>>>> com.esotericsoftware.kryo.KryoException: Class cannot be created
>>>>> (missing no-arg constructor): java.time.Instant
>>>>> Serialization trace:
>>>>> systemTime
>>>>> (com.punchcyber.streaming.pipeline.apex.datatypes.evtx.EvtxEvent)
>>>>> at
>>>>> com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
>>>>> at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
>>>>> at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
>>>>> at
>>>>> com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
>>>>> at
>>>>> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
>>>>> at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730)
>>>>> at
>>>>> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113)
>>>>> at
>>>>> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
>>>>> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
>>>>> at com.datatorrent.stram.codec.De
>>>>> faultStatefulStreamCodec.fromDataStatePair(DefaultStatefulStreamCodec.java:98)
>>>>> at
>>>>> com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.processPayload(BufferServerSubscriber.java:391)
>>>>> at
>>>>> com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.sweep(BufferServerSubscriber.java:339)
>>>>> at com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:269)
>>>>> at
>>>>> com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1429)
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> M. Aaron Bossert
>>>>> (571) 242-4021
>>>>> Punch Cyber Analytics Group
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> M. Aaron Bossert
>>> (571) 242-4021
>>> Punch Cyber Analytics Group
>>>
>>>
>>>
>>
>>
>> --
>>
>> M. Aaron Bossert
>> (571) 242-4021
>> Punch Cyber Analytics Group
>>
>>
>>
--
M. Aaron Bossert
(571) 242-4021
Punch Cyber Analytics Group
Re: Kryo version and default serializer
Posted by Aaron Bossert <aa...@punchcyber.com>.
I sent an e-mail to the dev list...but have not heard any responses. How
active is that list? Do you know? Basically, I have a workaround that
will deal with those "things" that Kryo v 2.24.0 will not handle (but 4.0.2
or 5.0.0-RC1 would) using the @FieldSerializer ...but this is not ideal. I
have two options left: go it alone and try to update Kryo in the apex-core,
or simply change the default serializer. The hiccup I am running into is
that I don't see how (from the documentation) to set the default
serializer...perhaps I am missing it. Can someone point me to where that
is in the docs?
On Mon, Jun 11, 2018 at 4:20 PM Aaron Bossert <aa...@punchcyber.com> wrote:
> Ah, you make a good point about the dev list...I am using IntelliJ on a
> Mac. When I tried building without any changes, I also encountered
> failures. I can re-run the non-modified sources to do a diff and see if
> the errors were different...
>
> Sent from my iPhone
>
> On Jun 11, 2018, at 15:11, Pramod Immaneni <pr...@gmail.com>
> wrote:
>
> Aaron,
>
> The question might be best served on the dev@apex.apache.org mailing list
> as you are starting to make changes to the sources. Nothing obvious jumps
> out to be, you don't need hadoop on your system to build the sources
> successfully with all the tests passing. The test failures you are seeing
> may or may not be related to the kryo change. What happens when you try to
> build the original sources with changing the Kryo version? What is your
> environment, mac, linux?
>
> On Mon, Jun 11, 2018 at 12:03 PM Aaron Bossert <aa...@punchcyber.com>
> wrote:
>
>> I know it has been a little while since I brought this up...I pulled down
>> the 3.7.1-RELEASE branch from GitHub in order to test out the change to
>> Kryo v. 4.0.2 and encountered no problems while building, however, when I
>> try to do mvn install, there are several failed tests. I am hoping, that
>> other than the obvious (to me, anyway) errors such as not finding
>> yarn-site.xml on my laptop which does not have hadoop installed, someone
>> may be able to see what is going on?
>>
>>
>> -------------------------------------------------------------------------------
>> Test set: com.datatorrent.stram.StramRecoveryTest
>>
>> -------------------------------------------------------------------------------
>> Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 6.372 sec
>> <<< FAILURE! - in com.datatorrent.stram.StramRecoveryTest
>> testWriteAheadLog(com.datatorrent.stram.StramRecoveryTest) Time elapsed:
>> 0.115 sec <<< FAILURE!
>> java.lang.AssertionError: flush count expected:<1> but was:<2>
>> at
>> com.datatorrent.stram.StramRecoveryTest.testWriteAheadLog(StramRecoveryTest.java:326)
>>
>>
>>
>> -------------------------------------------------------------------------------
>> Test set: com.datatorrent.stram.engine.StatsTest
>>
>> -------------------------------------------------------------------------------
>> Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 21.894
>> sec <<< FAILURE! - in com.datatorrent.stram.engine.StatsTest
>> testQueueSizeForContainerLocalOperators(com.datatorrent.stram.engine.StatsTest)
>> Time elapsed: 3.266 sec <<< FAILURE!
>> java.lang.AssertionError: Validate input port queue size -1
>> at
>> com.datatorrent.stram.engine.StatsTest.baseTestForQueueSize(StatsTest.java:270)
>> at
>> com.datatorrent.stram.engine.StatsTest.testQueueSizeForContainerLocalOperators(StatsTest.java:285)
>>
>>
>> -------------------------------------------------------------------------------
>> Test set: com.datatorrent.stram.StramMiniClusterTest
>>
>> -------------------------------------------------------------------------------
>> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.002 sec
>> <<< FAILURE! - in com.datatorrent.stram.StramMiniClusterTest
>> com.datatorrent.stram.StramMiniClusterTest Time elapsed: 0.002 sec <<<
>> ERROR!
>> java.io.FileNotFoundException:
>> /Users/mbossert/Library/Mobile%20Documents/com~apple~CloudDocs/Punch/IdeaProjects/apex-core/engine/target/test-classes/yarn-site.xml
>> (No such file or directory)
>> at
>> com.datatorrent.stram.StramMiniClusterTest.setup(StramMiniClusterTest.java:151)
>>
>>
>> -------------------------------------------------------------------------------
>> Test set: com.datatorrent.stram.StramLocalClusterTest
>>
>> -------------------------------------------------------------------------------
>> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 15.081
>> sec <<< FAILURE! - in com.datatorrent.stram.StramLocalClusterTest
>> testAppPath(com.datatorrent.stram.StramLocalClusterTest) Time elapsed:
>> 0.138 sec <<< FAILURE!
>> java.lang.AssertionError: checkpoint path exists
>> at
>> com.datatorrent.stram.StramLocalClusterTest.testAppPath(StramLocalClusterTest.java:413)
>>
>> testLocalClusterInitShutdown(com.datatorrent.stram.StramLocalClusterTest)
>> Time elapsed: 5.191 sec <<< FAILURE!
>> java.lang.AssertionError:
>> ./target/com.datatorrent.stram.StramLocalClusterTest-testLocalClusterInitShutdown.out
>> exists
>> at
>> com.datatorrent.stram.StramLocalClusterTest.testLocalClusterInitShutdown(StramLocalClusterTest.java:117)
>>
>>
>> -------------------------------------------------------------------------------
>> Test set: com.datatorrent.stram.client.AppPackageTest
>>
>> -------------------------------------------------------------------------------
>> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.105 sec
>> <<< FAILURE! - in com.datatorrent.stram.client.AppPackageTest
>> testAppPackage(com.datatorrent.stram.client.AppPackageTest) Time
>> elapsed: 0 sec <<< FAILURE!
>> java.lang.AssertionError: Number of applications expected:<2> but was:<0>
>> at
>> com.datatorrent.stram.client.AppPackageTest.testAppPackage(AppPackageTest.java:91)
>>
>> testAppLevelRequiredAndDefaultProperties(com.datatorrent.stram.client.AppPackageTest)
>> Time elapsed: 0.001 sec <<< FAILURE!
>> java.lang.AssertionError: Should consist of an app called
>> MyFirstApplication
>> at
>> com.datatorrent.stram.client.AppPackageTest.testAppLevelRequiredAndDefaultProperties(AppPackageTest.java:171)
>>
>>
>>
>> -------------------------------------------------------------------------------
>> Test set: com.datatorrent.stram.cli.ApexCliTest
>>
>> -------------------------------------------------------------------------------
>> Tests run: 7, Failures: 2, Errors: 2, Skipped: 0, Time elapsed: 0.938 sec
>> <<< FAILURE! - in com.datatorrent.stram.cli.ApexCliTest
>> testAppFromOnlyConfigPackage(com.datatorrent.stram.cli.ApexCliTest) Time
>> elapsed: 0.104 sec <<< ERROR!
>> com.datatorrent.stram.cli.ApexCli$CliException: No applications in
>> Application Package
>> at
>> com.datatorrent.stram.cli.ApexCliTest.testAppFromOnlyConfigPackage(ApexCliTest.java:235)
>>
>> testLaunchAppPackagePropertyPrecedence(com.datatorrent.stram.cli.ApexCliTest)
>> Time elapsed: 0.045 sec <<< FAILURE!
>> org.junit.ComparisonFailure: expected:<[app]-default> but
>> was:<[package]-default>
>> at
>> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePropertyPrecedence(ApexCliTest.java:143)
>>
>> testLaunchAppPackageParametersWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
>> Time elapsed: 0.045 sec <<< ERROR!
>> com.datatorrent.stram.cli.ApexCli$CliException: No applications in
>> Application Package matching "MyFirstApplication"
>> at
>> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackageParametersWithConfigPackage(ApexCliTest.java:155)
>>
>> testLaunchAppPackagePrecedenceWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
>> Time elapsed: 0.058 sec <<< FAILURE!
>> org.junit.ComparisonFailure: expected:<[app]-default> but
>> was:<[package]-default>
>> at
>> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePrecedenceWithConfigPackage(ApexCliTest.java:204)
>>
>>
>> On Thu, May 17, 2018 at 1:31 PM, Aaron Bossert <aa...@punchcyber.com>
>> wrote:
>>
>>> Thanks, Thomas! I have implemented the fix you suggested
>>> (@FieldSerializer.BInd) and also created a JIRA ticket:
>>> https://issues.apache.org/jira/browse/APEXCORE-816
>>>
>>> On Thu, May 17, 2018 at 1:31 AM, Thomas Weise <th...@apache.org> wrote:
>>>
>>>> Hi,
>>>>
>>>> It is not necessary to use a different stream codec when you have
>>>> control over the class that is causing the error. You can annotate a
>>>> different field serializer instead, using the @FieldSerializer.Bind
>>>> annotation. Here is an example:
>>>>
>>>>
>>>> https://github.com/apache/apex-malhar/blob/2fe2903bfe65055c7fd361a97549b6e7e12d4289/library/src/main/java/org/apache/apex/malhar/lib/window/WindowOption.java#L50
>>>>
>>>> It would be good to upgrade Kryo to 4.x in apex-core master. Would you
>>>> like to create a JIRA for it? (http://apex.apache.org/contributing.html
>>>> )
>>>>
>>>> Thanks,
>>>> Thomas
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Wed, May 16, 2018 at 10:31 AM, Aaron Bossert <aa...@punchcyber.com>
>>>> wrote:
>>>>
>>>>> I am writing an application that includes a custom object. One of the
>>>>> fields is of type Instant and seems to not agree with the version of Kryo
>>>>> that is used in Apex (see exception below). I now that kryo 4.X supports
>>>>> serializing Instant fields, but for the life of me, cannot seem to find a
>>>>> way to either get Apex to use a newer version of Kryo. Can anyone help?
>>>>> Perhaps setting a different streamcodec that uses the newer version of kryo
>>>>> would work, but I cannot seem to find an example of how to set a different
>>>>> streamcodec despite references to the two (Kryo and Java) that ship with
>>>>> Apex.
>>>>>
>>>>> ERROR com.datatorrent.stram.codec.DefaultStatefulStreamCodec -
>>>>> Catastrophic Error: Execution halted due to Kryo exception!
>>>>> com.esotericsoftware.kryo.KryoException: Class cannot be created
>>>>> (missing no-arg constructor): java.time.Instant
>>>>> Serialization trace:
>>>>> systemTime
>>>>> (com.punchcyber.streaming.pipeline.apex.datatypes.evtx.EvtxEvent)
>>>>> at
>>>>> com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
>>>>> at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
>>>>> at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
>>>>> at
>>>>> com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
>>>>> at
>>>>> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
>>>>> at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730)
>>>>> at
>>>>> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113)
>>>>> at
>>>>> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
>>>>> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
>>>>> at com.datatorrent.stram.codec.De
>>>>> faultStatefulStreamCodec.fromDataStatePair(DefaultStatefulStreamCodec.java:98)
>>>>> at
>>>>> com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.processPayload(BufferServerSubscriber.java:391)
>>>>> at
>>>>> com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.sweep(BufferServerSubscriber.java:339)
>>>>> at com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:269)
>>>>> at
>>>>> com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1429)
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> M. Aaron Bossert
>>>>> (571) 242-4021
>>>>> Punch Cyber Analytics Group
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> M. Aaron Bossert
>>> (571) 242-4021
>>> Punch Cyber Analytics Group
>>>
>>>
>>>
>>
>>
>> --
>>
>> M. Aaron Bossert
>> (571) 242-4021
>> Punch Cyber Analytics Group
>>
>>
>>
--
M. Aaron Bossert
(571) 242-4021
Punch Cyber Analytics Group
Re: Kryo version and default serializer
Posted by Aaron Bossert <aa...@punchcyber.com>.
Ah, you make a good point about the dev list...I am using IntelliJ on a Mac. When I tried building without any changes, I also encountered failures. I can re-run the non-modified sources to do a diff and see if the errors were different...
Sent from my iPhone
> On Jun 11, 2018, at 15:11, Pramod Immaneni <pr...@gmail.com> wrote:
>
> Aaron,
>
> The question might be best served on the dev@apex.apache.org mailing list as you are starting to make changes to the sources. Nothing obvious jumps out to be, you don't need hadoop on your system to build the sources successfully with all the tests passing. The test failures you are seeing may or may not be related to the kryo change. What happens when you try to build the original sources with changing the Kryo version? What is your environment, mac, linux?
>
>> On Mon, Jun 11, 2018 at 12:03 PM Aaron Bossert <aa...@punchcyber.com> wrote:
>> I know it has been a little while since I brought this up...I pulled down the 3.7.1-RELEASE branch from GitHub in order to test out the change to Kryo v. 4.0.2 and encountered no problems while building, however, when I try to do mvn install, there are several failed tests. I am hoping, that other than the obvious (to me, anyway) errors such as not finding yarn-site.xml on my laptop which does not have hadoop installed, someone may be able to see what is going on?
>>
>> -------------------------------------------------------------------------------
>> Test set: com.datatorrent.stram.StramRecoveryTest
>> -------------------------------------------------------------------------------
>> Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 6.372 sec <<< FAILURE! - in com.datatorrent.stram.StramRecoveryTest
>> testWriteAheadLog(com.datatorrent.stram.StramRecoveryTest) Time elapsed: 0.115 sec <<< FAILURE!
>> java.lang.AssertionError: flush count expected:<1> but was:<2>
>> at com.datatorrent.stram.StramRecoveryTest.testWriteAheadLog(StramRecoveryTest.java:326)
>>
>>
>> -------------------------------------------------------------------------------
>> Test set: com.datatorrent.stram.engine.StatsTest
>> -------------------------------------------------------------------------------
>> Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 21.894 sec <<< FAILURE! - in com.datatorrent.stram.engine.StatsTest
>> testQueueSizeForContainerLocalOperators(com.datatorrent.stram.engine.StatsTest) Time elapsed: 3.266 sec <<< FAILURE!
>> java.lang.AssertionError: Validate input port queue size -1
>> at com.datatorrent.stram.engine.StatsTest.baseTestForQueueSize(StatsTest.java:270)
>> at com.datatorrent.stram.engine.StatsTest.testQueueSizeForContainerLocalOperators(StatsTest.java:285)
>>
>> -------------------------------------------------------------------------------
>> Test set: com.datatorrent.stram.StramMiniClusterTest
>> -------------------------------------------------------------------------------
>> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.002 sec <<< FAILURE! - in com.datatorrent.stram.StramMiniClusterTest
>> com.datatorrent.stram.StramMiniClusterTest Time elapsed: 0.002 sec <<< ERROR!
>> java.io.FileNotFoundException: /Users/mbossert/Library/Mobile%20Documents/com~apple~CloudDocs/Punch/IdeaProjects/apex-core/engine/target/test-classes/yarn-site.xml (No such file or directory)
>> at com.datatorrent.stram.StramMiniClusterTest.setup(StramMiniClusterTest.java:151)
>>
>> -------------------------------------------------------------------------------
>> Test set: com.datatorrent.stram.StramLocalClusterTest
>> -------------------------------------------------------------------------------
>> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 15.081 sec <<< FAILURE! - in com.datatorrent.stram.StramLocalClusterTest
>> testAppPath(com.datatorrent.stram.StramLocalClusterTest) Time elapsed: 0.138 sec <<< FAILURE!
>> java.lang.AssertionError: checkpoint path exists
>> at com.datatorrent.stram.StramLocalClusterTest.testAppPath(StramLocalClusterTest.java:413)
>>
>> testLocalClusterInitShutdown(com.datatorrent.stram.StramLocalClusterTest) Time elapsed: 5.191 sec <<< FAILURE!
>> java.lang.AssertionError: ./target/com.datatorrent.stram.StramLocalClusterTest-testLocalClusterInitShutdown.out exists
>> at com.datatorrent.stram.StramLocalClusterTest.testLocalClusterInitShutdown(StramLocalClusterTest.java:117)
>>
>> -------------------------------------------------------------------------------
>> Test set: com.datatorrent.stram.client.AppPackageTest
>> -------------------------------------------------------------------------------
>> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.105 sec <<< FAILURE! - in com.datatorrent.stram.client.AppPackageTest
>> testAppPackage(com.datatorrent.stram.client.AppPackageTest) Time elapsed: 0 sec <<< FAILURE!
>> java.lang.AssertionError: Number of applications expected:<2> but was:<0>
>> at com.datatorrent.stram.client.AppPackageTest.testAppPackage(AppPackageTest.java:91)
>>
>> testAppLevelRequiredAndDefaultProperties(com.datatorrent.stram.client.AppPackageTest) Time elapsed: 0.001 sec <<< FAILURE!
>> java.lang.AssertionError: Should consist of an app called MyFirstApplication
>> at com.datatorrent.stram.client.AppPackageTest.testAppLevelRequiredAndDefaultProperties(AppPackageTest.java:171)
>>
>>
>> -------------------------------------------------------------------------------
>> Test set: com.datatorrent.stram.cli.ApexCliTest
>> -------------------------------------------------------------------------------
>> Tests run: 7, Failures: 2, Errors: 2, Skipped: 0, Time elapsed: 0.938 sec <<< FAILURE! - in com.datatorrent.stram.cli.ApexCliTest
>> testAppFromOnlyConfigPackage(com.datatorrent.stram.cli.ApexCliTest) Time elapsed: 0.104 sec <<< ERROR!
>> com.datatorrent.stram.cli.ApexCli$CliException: No applications in Application Package
>> at com.datatorrent.stram.cli.ApexCliTest.testAppFromOnlyConfigPackage(ApexCliTest.java:235)
>>
>> testLaunchAppPackagePropertyPrecedence(com.datatorrent.stram.cli.ApexCliTest) Time elapsed: 0.045 sec <<< FAILURE!
>> org.junit.ComparisonFailure: expected:<[app]-default> but was:<[package]-default>
>> at com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePropertyPrecedence(ApexCliTest.java:143)
>>
>> testLaunchAppPackageParametersWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest) Time elapsed: 0.045 sec <<< ERROR!
>> com.datatorrent.stram.cli.ApexCli$CliException: No applications in Application Package matching "MyFirstApplication"
>> at com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackageParametersWithConfigPackage(ApexCliTest.java:155)
>>
>> testLaunchAppPackagePrecedenceWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest) Time elapsed: 0.058 sec <<< FAILURE!
>> org.junit.ComparisonFailure: expected:<[app]-default> but was:<[package]-default>
>> at com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePrecedenceWithConfigPackage(ApexCliTest.java:204)
>>
>>
>>> On Thu, May 17, 2018 at 1:31 PM, Aaron Bossert <aa...@punchcyber.com> wrote:
>>> Thanks, Thomas! I have implemented the fix you suggested (@FieldSerializer.BInd) and also created a JIRA ticket: https://issues.apache.org/jira/browse/APEXCORE-816
>>>
>>>> On Thu, May 17, 2018 at 1:31 AM, Thomas Weise <th...@apache.org> wrote:
>>>> Hi,
>>>>
>>>> It is not necessary to use a different stream codec when you have control over the class that is causing the error. You can annotate a different field serializer instead, using the @FieldSerializer.Bind annotation. Here is an example:
>>>>
>>>> https://github.com/apache/apex-malhar/blob/2fe2903bfe65055c7fd361a97549b6e7e12d4289/library/src/main/java/org/apache/apex/malhar/lib/window/WindowOption.java#L50
>>>>
>>>> It would be good to upgrade Kryo to 4.x in apex-core master. Would you like to create a JIRA for it? (http://apex.apache.org/contributing.html)
>>>>
>>>> Thanks,
>>>> Thomas
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> On Wed, May 16, 2018 at 10:31 AM, Aaron Bossert <aa...@punchcyber.com> wrote:
>>>>> I am writing an application that includes a custom object. One of the fields is of type Instant and seems to not agree with the version of Kryo that is used in Apex (see exception below). I now that kryo 4.X supports serializing Instant fields, but for the life of me, cannot seem to find a way to either get Apex to use a newer version of Kryo. Can anyone help? Perhaps setting a different streamcodec that uses the newer version of kryo would work, but I cannot seem to find an example of how to set a different streamcodec despite references to the two (Kryo and Java) that ship with Apex.
>>>>>
>>>>> ERROR com.datatorrent.stram.codec.DefaultStatefulStreamCodec - Catastrophic Error: Execution halted due to Kryo exception!
>>>>> com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): java.time.Instant
>>>>> Serialization trace:
>>>>> systemTime (com.punchcyber.streaming.pipeline.apex.datatypes.evtx.EvtxEvent)
>>>>> at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
>>>>> at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
>>>>> at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
>>>>> at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
>>>>> at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
>>>>> at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730)
>>>>> at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113)
>>>>> at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
>>>>> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
>>>>> at com.datatorrent.stram.codec.DefaultStatefulStreamCodec.fromDataStatePair(DefaultStatefulStreamCodec.java:98)
>>>>> at com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.processPayload(BufferServerSubscriber.java:391)
>>>>> at com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.sweep(BufferServerSubscriber.java:339)
>>>>> at com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:269)
>>>>> at com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1429)
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> M. Aaron Bossert
>>>>> (571) 242-4021
>>>>> Punch Cyber Analytics Group
>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>>
>>> M. Aaron Bossert
>>> (571) 242-4021
>>> Punch Cyber Analytics Group
>>>
>>>
>>
>>
>>
>> --
>>
>> M. Aaron Bossert
>> (571) 242-4021
>> Punch Cyber Analytics Group
>>
>>
Re: Kryo version and default serializer
Posted by Pramod Immaneni <pr...@gmail.com>.
Aaron,
The question might be best served on the dev@apex.apache.org mailing list
as you are starting to make changes to the sources. Nothing obvious jumps
out to be, you don't need hadoop on your system to build the sources
successfully with all the tests passing. The test failures you are seeing
may or may not be related to the kryo change. What happens when you try to
build the original sources with changing the Kryo version? What is your
environment, mac, linux?
On Mon, Jun 11, 2018 at 12:03 PM Aaron Bossert <aa...@punchcyber.com> wrote:
> I know it has been a little while since I brought this up...I pulled down
> the 3.7.1-RELEASE branch from GitHub in order to test out the change to
> Kryo v. 4.0.2 and encountered no problems while building, however, when I
> try to do mvn install, there are several failed tests. I am hoping, that
> other than the obvious (to me, anyway) errors such as not finding
> yarn-site.xml on my laptop which does not have hadoop installed, someone
> may be able to see what is going on?
>
>
> -------------------------------------------------------------------------------
> Test set: com.datatorrent.stram.StramRecoveryTest
>
> -------------------------------------------------------------------------------
> Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 6.372 sec
> <<< FAILURE! - in com.datatorrent.stram.StramRecoveryTest
> testWriteAheadLog(com.datatorrent.stram.StramRecoveryTest) Time elapsed:
> 0.115 sec <<< FAILURE!
> java.lang.AssertionError: flush count expected:<1> but was:<2>
> at
> com.datatorrent.stram.StramRecoveryTest.testWriteAheadLog(StramRecoveryTest.java:326)
>
>
>
> -------------------------------------------------------------------------------
> Test set: com.datatorrent.stram.engine.StatsTest
>
> -------------------------------------------------------------------------------
> Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 21.894 sec
> <<< FAILURE! - in com.datatorrent.stram.engine.StatsTest
> testQueueSizeForContainerLocalOperators(com.datatorrent.stram.engine.StatsTest)
> Time elapsed: 3.266 sec <<< FAILURE!
> java.lang.AssertionError: Validate input port queue size -1
> at
> com.datatorrent.stram.engine.StatsTest.baseTestForQueueSize(StatsTest.java:270)
> at
> com.datatorrent.stram.engine.StatsTest.testQueueSizeForContainerLocalOperators(StatsTest.java:285)
>
>
> -------------------------------------------------------------------------------
> Test set: com.datatorrent.stram.StramMiniClusterTest
>
> -------------------------------------------------------------------------------
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.002 sec
> <<< FAILURE! - in com.datatorrent.stram.StramMiniClusterTest
> com.datatorrent.stram.StramMiniClusterTest Time elapsed: 0.002 sec <<<
> ERROR!
> java.io.FileNotFoundException:
> /Users/mbossert/Library/Mobile%20Documents/com~apple~CloudDocs/Punch/IdeaProjects/apex-core/engine/target/test-classes/yarn-site.xml
> (No such file or directory)
> at
> com.datatorrent.stram.StramMiniClusterTest.setup(StramMiniClusterTest.java:151)
>
>
> -------------------------------------------------------------------------------
> Test set: com.datatorrent.stram.StramLocalClusterTest
>
> -------------------------------------------------------------------------------
> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 15.081 sec
> <<< FAILURE! - in com.datatorrent.stram.StramLocalClusterTest
> testAppPath(com.datatorrent.stram.StramLocalClusterTest) Time elapsed:
> 0.138 sec <<< FAILURE!
> java.lang.AssertionError: checkpoint path exists
> at
> com.datatorrent.stram.StramLocalClusterTest.testAppPath(StramLocalClusterTest.java:413)
>
> testLocalClusterInitShutdown(com.datatorrent.stram.StramLocalClusterTest)
> Time elapsed: 5.191 sec <<< FAILURE!
> java.lang.AssertionError:
> ./target/com.datatorrent.stram.StramLocalClusterTest-testLocalClusterInitShutdown.out
> exists
> at
> com.datatorrent.stram.StramLocalClusterTest.testLocalClusterInitShutdown(StramLocalClusterTest.java:117)
>
>
> -------------------------------------------------------------------------------
> Test set: com.datatorrent.stram.client.AppPackageTest
>
> -------------------------------------------------------------------------------
> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.105 sec
> <<< FAILURE! - in com.datatorrent.stram.client.AppPackageTest
> testAppPackage(com.datatorrent.stram.client.AppPackageTest) Time elapsed:
> 0 sec <<< FAILURE!
> java.lang.AssertionError: Number of applications expected:<2> but was:<0>
> at
> com.datatorrent.stram.client.AppPackageTest.testAppPackage(AppPackageTest.java:91)
>
> testAppLevelRequiredAndDefaultProperties(com.datatorrent.stram.client.AppPackageTest)
> Time elapsed: 0.001 sec <<< FAILURE!
> java.lang.AssertionError: Should consist of an app called
> MyFirstApplication
> at
> com.datatorrent.stram.client.AppPackageTest.testAppLevelRequiredAndDefaultProperties(AppPackageTest.java:171)
>
>
>
> -------------------------------------------------------------------------------
> Test set: com.datatorrent.stram.cli.ApexCliTest
>
> -------------------------------------------------------------------------------
> Tests run: 7, Failures: 2, Errors: 2, Skipped: 0, Time elapsed: 0.938 sec
> <<< FAILURE! - in com.datatorrent.stram.cli.ApexCliTest
> testAppFromOnlyConfigPackage(com.datatorrent.stram.cli.ApexCliTest) Time
> elapsed: 0.104 sec <<< ERROR!
> com.datatorrent.stram.cli.ApexCli$CliException: No applications in
> Application Package
> at
> com.datatorrent.stram.cli.ApexCliTest.testAppFromOnlyConfigPackage(ApexCliTest.java:235)
>
> testLaunchAppPackagePropertyPrecedence(com.datatorrent.stram.cli.ApexCliTest)
> Time elapsed: 0.045 sec <<< FAILURE!
> org.junit.ComparisonFailure: expected:<[app]-default> but
> was:<[package]-default>
> at
> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePropertyPrecedence(ApexCliTest.java:143)
>
> testLaunchAppPackageParametersWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
> Time elapsed: 0.045 sec <<< ERROR!
> com.datatorrent.stram.cli.ApexCli$CliException: No applications in
> Application Package matching "MyFirstApplication"
> at
> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackageParametersWithConfigPackage(ApexCliTest.java:155)
>
> testLaunchAppPackagePrecedenceWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
> Time elapsed: 0.058 sec <<< FAILURE!
> org.junit.ComparisonFailure: expected:<[app]-default> but
> was:<[package]-default>
> at
> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePrecedenceWithConfigPackage(ApexCliTest.java:204)
>
>
> On Thu, May 17, 2018 at 1:31 PM, Aaron Bossert <aa...@punchcyber.com>
> wrote:
>
>> Thanks, Thomas! I have implemented the fix you suggested
>> (@FieldSerializer.BInd) and also created a JIRA ticket:
>> https://issues.apache.org/jira/browse/APEXCORE-816
>>
>> On Thu, May 17, 2018 at 1:31 AM, Thomas Weise <th...@apache.org> wrote:
>>
>>> Hi,
>>>
>>> It is not necessary to use a different stream codec when you have
>>> control over the class that is causing the error. You can annotate a
>>> different field serializer instead, using the @FieldSerializer.Bind
>>> annotation. Here is an example:
>>>
>>>
>>> https://github.com/apache/apex-malhar/blob/2fe2903bfe65055c7fd361a97549b6e7e12d4289/library/src/main/java/org/apache/apex/malhar/lib/window/WindowOption.java#L50
>>>
>>> It would be good to upgrade Kryo to 4.x in apex-core master. Would you
>>> like to create a JIRA for it? (http://apex.apache.org/contributing.html)
>>>
>>> Thanks,
>>> Thomas
>>>
>>>
>>>
>>>
>>>
>>> On Wed, May 16, 2018 at 10:31 AM, Aaron Bossert <aa...@punchcyber.com>
>>> wrote:
>>>
>>>> I am writing an application that includes a custom object. One of the
>>>> fields is of type Instant and seems to not agree with the version of Kryo
>>>> that is used in Apex (see exception below). I now that kryo 4.X supports
>>>> serializing Instant fields, but for the life of me, cannot seem to find a
>>>> way to either get Apex to use a newer version of Kryo. Can anyone help?
>>>> Perhaps setting a different streamcodec that uses the newer version of kryo
>>>> would work, but I cannot seem to find an example of how to set a different
>>>> streamcodec despite references to the two (Kryo and Java) that ship with
>>>> Apex.
>>>>
>>>> ERROR com.datatorrent.stram.codec.DefaultStatefulStreamCodec -
>>>> Catastrophic Error: Execution halted due to Kryo exception!
>>>> com.esotericsoftware.kryo.KryoException: Class cannot be created
>>>> (missing no-arg constructor): java.time.Instant
>>>> Serialization trace:
>>>> systemTime
>>>> (com.punchcyber.streaming.pipeline.apex.datatypes.evtx.EvtxEvent)
>>>> at
>>>> com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
>>>> at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
>>>> at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
>>>> at
>>>> com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
>>>> at
>>>> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
>>>> at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730)
>>>> at
>>>> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113)
>>>> at
>>>> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
>>>> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
>>>> at com.datatorrent.stram.codec.De
>>>> faultStatefulStreamCodec.fromDataStatePair(DefaultStatefulStreamCodec.java:98)
>>>> at
>>>> com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.processPayload(BufferServerSubscriber.java:391)
>>>> at
>>>> com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.sweep(BufferServerSubscriber.java:339)
>>>> at com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:269)
>>>> at
>>>> com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1429)
>>>>
>>>>
>>>> --
>>>>
>>>> M. Aaron Bossert
>>>> (571) 242-4021
>>>> Punch Cyber Analytics Group
>>>>
>>>>
>>>>
>>>
>>
>>
>> --
>>
>> M. Aaron Bossert
>> (571) 242-4021
>> Punch Cyber Analytics Group
>>
>>
>>
>
>
> --
>
> M. Aaron Bossert
> (571) 242-4021
> Punch Cyber Analytics Group
>
>
>