You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by Jeremiah Adams <JA...@helixeducation.com> on 2019/03/06 16:14:20 UTC

Re: [POSSIBLE PHISHING] Task Partition Commit Failed After Upgrade

It appears that the issue is related to the KafkaCheckpointLogKey.java constructor. grouperFactoryClassName here is null.  THe documentation indicates that task.name.grouper.factory config setting has a default value of 	org.apache.samza.container.grouper.task.GroupByContainerCountFactory. I wouldn't expect it to be null here.

If I specify GroupByContainerCountFactory for the task.name.grouper.factory in my properties file, I get a NoSuchMethodException:

Exception in thread "main" java.lang.InstantiationException: org.apache.samza.container.grouper.task.GroupByContainerCount
        at java.lang.Class.newInstance(Class.java:427)
        at org.apache.samza.util.Util$.getObj(Util.scala:80)
        at org.apache.samza.coordinator.JobModelManager$.readJobModel(JobModelManager.scala:261)
        at org.apache.samza.coordinator.JobModelManager$.getJobModelManager(JobModelManager.scala:155)
        at org.apache.samza.coordinator.JobModelManager$.apply(JobModelManager.scala:117)
        at org.apache.samza.coordinator.JobModelManager.apply(JobModelManager.scala)
        at org.apache.samza.clustermanager.ClusterBasedJobCoordinator.buildJobModelManager(ClusterBasedJobCoordinator.java:241)
        at org.apache.samza.clustermanager.ClusterBasedJobCoordinator.<init>(ClusterBasedJobCoordinator.java:152)
        at org.apache.samza.clustermanager.ClusterBasedJobCoordinator.main(ClusterBasedJobCoordinator.java:297)
Caused by: java.lang.NoSuchMethodException: org.apache.samza.container.grouper.task.GroupByContainerCount.<init>()
        at java.lang.Class.getConstructor0(Class.java:3082)
        at java.lang.Class.newInstance(Class.java:412)
        ... 8 more

 

Jeremiah Adams
Software Engineer
www.helixeducation.com <http://www.helixeducation.com/>
Blog <http://www.helixeducation.com/blog/> | Twitter <https://twitter.com/HelixEducation> | Facebook <https://www.facebook.com/HelixEducation> | LinkedIn <http://www.linkedin.com/company/3609946>
 

On 3/4/19, 2:48 PM, "Jeremiah Adams" <JA...@helixeducation.com> wrote:

    I am updating dependencies and moving from Samza V0.13.0 to V0.14.0.  I develop locally using the grid app in the hello-samza project to spin up local yarn/zookeeper/kafka instances.
    
    Grid is running these versions:
    kafka_2.11-0.10.2.1.tgz
    hadoop-2.6.1.tar.gz
    zookeeper-3.4.3.tar.gz
    
    
    My job is now failing with the NPE below. anyone have ideas on the cause of this error?
    
    
    2019-03-04 14:13:49 AsyncRunLoop [ERROR] Task Partition 0 commit failed
    java.lang.NullPointerException
            at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:782)
            at org.apache.samza.checkpoint.kafka.KafkaCheckpointLogKey.<init>(KafkaCheckpointLogKey.java:46)
            at org.apache.samza.checkpoint.kafka.KafkaCheckpointManager.writeCheckpoint(KafkaCheckpointManager.scala:136)
            at org.apache.samza.checkpoint.OffsetManager.writeCheckpoint(OffsetManager.scala:259)
            at org.apache.samza.container.TaskInstance.commit(TaskInstance.scala:205)
            at org.apache.samza.task.AsyncRunLoop$AsyncTaskWorker$5.run(AsyncRunLoop.java:494)
            at org.apache.samza.task.AsyncRunLoop$AsyncTaskWorker.commit(AsyncRunLoop.java:513)
            at org.apache.samza.task.AsyncRunLoop$AsyncTaskWorker.run(AsyncRunLoop.java:379)
            at org.apache.samza.task.AsyncRunLoop$AsyncTaskWorker.access$300(AsyncRunLoop.java:314)
            at org.apache.samza.task.AsyncRunLoop.runTasks(AsyncRunLoop.java:228)
            at org.apache.samza.task.AsyncRunLoop.run(AsyncRunLoop.java:157)
            at org.apache.samza.container.SamzaContainer.run(SamzaContainer.scala:728)
            at org.apache.samza.runtime.LocalContainerRunner.run(LocalContainerRunner.java:102)
            at org.apache.samza.runtime.LocalContainerRunner.main(LocalContainerRunner.java:147)
    2019-03-04 14:13:49 AsyncRunLoop [ERROR] Caught throwable and stopping run loop
    
    
    
    Jeremiah Adams
    Software Engineer
    https://url.emailprotection.link/?ahfhEufaAWbezBrUFPG98ZJcterGfIerU3ZwsA3Gv_C0~<https://url.emailprotection.link/?a49H2rNGIIBtQOw6md8OcHp-qKE3Xn2gNiZ3dlqAeSDA~>
    Blog<https://url.emailprotection.link/?a49H2rNGIIBtQOw6md8OcHgFEZu-KYuiu8doY66NWwmmyWxz7kC-27Yfnbdgd2wyh5gjXUa6LMT_NRXsj1g1VVg~~> | Twitter<https://url.emailprotection.link/?a0Q7ct5_6cOdbJ86kpWB0zx6RbtgugTVC7lU_W7za50jLdZQGpLgVlR1V06zckSaM5oOKb6QBo46Qp9xt0Tt7Aw~~> | Facebook<https://url.emailprotection.link/?aAmyAO_nS_C1aDgBLeKyGTu0tksTt1_mn2PcS8KJXNJPM04iRHKgX96qGgENV-dMSER5wl8zDVRr3RsS0OmcF9A~~> | LinkedIn<https://url.emailprotection.link/?aanlcNI-cN74Gdz-TD332xAl6lHu7TRNICWoHUFjYf-KlBjrCGHoYR65b3rl-OyW10nWFv6hwYvUSoVHL4b3vGA~~>
    
    


Re: [POSSIBLE PHISHING] Task Partition Commit Failed After Upgrade

Posted by Jeremiah Adams <JA...@helixeducation.com>.
Prateek, 

Migrating to v1.0.0 went off without a hitch. Thanks.

 

Jeremiah Adams
Software Engineer
www.helixeducation.com <http://www.helixeducation.com/>
Blog <http://www.helixeducation.com/blog/> | Twitter <https://twitter.com/HelixEducation> | Facebook <https://www.facebook.com/HelixEducation> | LinkedIn <http://www.linkedin.com/company/3609946>
 

On 3/8/19, 9:03 AM, "Jeremiah Adams" <JA...@helixeducation.com> wrote:

    Thanks for checking Prateek. There’s a similar thread in the group. I decided to follow advice there and am working on migrating to V1.0.0 instead of 0.14.
    
    
    
    Jeremiah Adams
    Software Engineer
    https://url.emailprotection.link/?bM9S-3pRw1lv8pYfwa-TwdjElP4W2K6b9vP5Crz22L_YcgsRJ-13h-OgPZSwFtU7GSNTDi1z-jdaRvWESRhtTVA~~<https://url.emailprotection.link/?basKr9vk92a8vVw0XMnK5bmaSKuBc0AuEZ7YasYc7Df8YVt3SYmcjmLWdKMWzAAINWlUUA33ebGI7pSoTl9cg1g~~>
    Blog<https://url.emailprotection.link/?basKr9vk92a8vVw0XMnK5bmaSKuBc0AuEZ7YasYc7Df-lAcqG1fqHPpNw-wd9z7HtUJeCG5_8UjCf2mHtn6C_zQ~~> | Twitter<https://url.emailprotection.link/?bVO2q0UXR235wN_yOnM0FjqITPdBYMD3reLGNddq-zPV5ChMQK9JwV4Be-QnrbRoXpJl8IcknAqKzYtA3RABKww~~> | Facebook<https://url.emailprotection.link/?bUU7m4NfMS_EWGtH1yojBHX9sWZ6uxVdT1eQUkmU5vWY01WFZiS2KJ-c9iLIncdHB7Uw1lRYCprEEpPPQCdiK6Q~~> | LinkedIn<https://url.emailprotection.link/?b0ZQfJ1pZYnASyoShs9MJI46-r1lxPhA-JS5VSkR7so-DFP0_HxbOo2LsajGOaoYXxb1ZCOMAu7hZscPCnIKWpXz0cpgQ386SnNHjPcwsu4z90mzBkuwoZc6YxOCzMGA0>
    
    
    From: Prateek Maheshwari <pr...@gmail.com>
    Date: Thursday, March 7, 2019 at 2:57 PM
    To: "dev@samza.apache.org" <de...@samza.apache.org>, Jeremiah Adams <JA...@helixeducation.com>
    Subject: Re: [POSSIBLE PHISHING] Task Partition Commit Failed After Upgrade
    
    Jeremiah, were you able to resolve this issue?
    
    - Prateek
    
    On Wed, Mar 6, 2019 at 10:08 AM Prateek Maheshwari <pr...@gmail.com>> wrote:
    Hi Jeremiah,
    
    The configuration you want to look for is: 'job.systemstreampartition.grouper.factory'. It should default to: 'org.apache.samza.container.grouper.stream.GroupByPartitionFactory'.
    Can you check if you see this value in the configuration logged by SamzaContainer during container start? You can grep for: "Using configuration".
    
    For context, there are two groupers for a Samza job. One that groups input partitions into tasks (this one), and one that groups tasks into containers (the one you mentioned above).
    
    Thanks,
    Prateek
    
    
    
    On Wed, Mar 6, 2019 at 8:14 AM Jeremiah Adams <JA...@helixeducation.com>> wrote:
    It appears that the issue is related to the KafkaCheckpointLogKey.java constructor. grouperFactoryClassName here is null.  THe documentation indicates that task.name.grouper.factory config setting has a default value of     org.apache.samza.container.grouper.task.GroupByContainerCountFactory. I wouldn't expect it to be null here.
    
    If I specify GroupByContainerCountFactory for the task.name.grouper.factory in my properties file, I get a NoSuchMethodException:
    
    Exception in thread "main" java.lang.InstantiationException: org.apache.samza.container.grouper.task.GroupByContainerCount
            at java.lang.Class.newInstance(Class.java:427)
            at org.apache.samza.util.Util$.getObj(Util.scala:80)
            at org.apache.samza.coordinator.JobModelManager$.readJobModel(JobModelManager.scala:261)
            at org.apache.samza.coordinator.JobModelManager$.getJobModelManager(JobModelManager.scala:155)
            at org.apache.samza.coordinator.JobModelManager$.apply(JobModelManager.scala:117)
            at org.apache.samza.coordinator.JobModelManager.apply(JobModelManager.scala)
            at org.apache.samza.clustermanager.ClusterBasedJobCoordinator.buildJobModelManager(ClusterBasedJobCoordinator.java:241)
            at org.apache.samza.clustermanager.ClusterBasedJobCoordinator.<init>(ClusterBasedJobCoordinator.java:152)
            at org.apache.samza.clustermanager.ClusterBasedJobCoordinator.main(ClusterBasedJobCoordinator.java:297)
    Caused by: java.lang.NoSuchMethodException: org.apache.samza.container.grouper.task.GroupByContainerCount.<init>()
            at java.lang.Class.getConstructor0(Class.java:3082)
            at java.lang.Class.newInstance(Class.java:412)
            ... 8 more
    
    
    
    Jeremiah Adams
    Software Engineer
    https://url.emailprotection.link/?bM9S-3pRw1lv8pYfwa-TwdjElP4W2K6b9vP5Crz22L_YcgsRJ-13h-OgPZSwFtU7GSNTDi1z-jdaRvWESRhtTVA~~<https://url.emailprotection.link/?basKr9vk92a8vVw0XMnK5bnWsxM_w6KChRx8CY_UgrU5RmcwzgGL3Po63B7rJIXeNyMBLKYpptY6Rl-f5kb6p2A~~> <https://url.emailprotection.link/?basKr9vk92a8vVw0XMnK5bmaSKuBc0AuEZ7YasYc7Df8YVt3SYmcjmLWdKMWzAAINWlUUA33ebGI7pSoTl9cg1g~~<https://url.emailprotection.link/?basKr9vk92a8vVw0XMnK5bmaSKuBc0AuEZ7YasYc7Df8YVt3SYmcjmLWdKMWzAAINWlUUA33ebGI7pSoTl9cg1g~~>>
    Blog <https://url.emailprotection.link/?basKr9vk92a8vVw0XMnK5bmaSKuBc0AuEZ7YasYc7Df-lAcqG1fqHPpNw-wd9z7HtUJeCG5_8UjCf2mHtn6C_zQ~~<https://url.emailprotection.link/?basKr9vk92a8vVw0XMnK5bmaSKuBc0AuEZ7YasYc7Df-lAcqG1fqHPpNw-wd9z7HtUJeCG5_8UjCf2mHtn6C_zQ~~>> | Twitter <https://url.emailprotection.link/?bVO2q0UXR235wN_yOnM0FjqITPdBYMD3reLGNddq-zPV5ChMQK9JwV4Be-QnrbRoXpJl8IcknAqKzYtA3RABKww~~<https://url.emailprotection.link/?bVO2q0UXR235wN_yOnM0FjqITPdBYMD3reLGNddq-zPV5ChMQK9JwV4Be-QnrbRoXpJl8IcknAqKzYtA3RABKww~~>> | Facebook <https://url.emailprotection.link/?bUU7m4NfMS_EWGtH1yojBHX9sWZ6uxVdT1eQUkmU5vWY01WFZiS2KJ-c9iLIncdHB7Uw1lRYCprEEpPPQCdiK6Q~~<https://url.emailprotection.link/?bUU7m4NfMS_EWGtH1yojBHX9sWZ6uxVdT1eQUkmU5vWY01WFZiS2KJ-c9iLIncdHB7Uw1lRYCprEEpPPQCdiK6Q~~>> | LinkedIn <https://url.emailprotection.link/?b0ZQfJ1pZYnASyoShs9MJI46-r1lxPhA-JS5VSkR7so-DFP0_HxbOo2LsajGOaoYXxb1ZCOMAu7hZscPCnIKWpXz0cpgQ386SnNHjPcwsu4z90mzBkuwoZc6YxOCzMGA0<https://url.emailprotection.link/?b0ZQfJ1pZYnASyoShs9MJI46-r1lxPhA-JS5VSkR7so-DFP0_HxbOo2LsajGOaoYXxb1ZCOMAu7hZscPCnIKWpXz0cpgQ386SnNHjPcwsu4z90mzBkuwoZc6YxOCzMGA0>>
    
    
    On 3/4/19, 2:48 PM, "Jeremiah Adams" <JA...@helixeducation.com>> wrote:
    
        I am updating dependencies and moving from Samza V0.13.0 to V0.14.0.  I develop locally using the grid app in the hello-samza project to spin up local yarn/zookeeper/kafka instances.
    
        Grid is running these versions:
        kafka_2.11-0.10.2.1.tgz
        hadoop-2.6.1.tar.gz
        zookeeper-3.4.3.tar.gz
    
    
        My job is now failing with the NPE below. anyone have ideas on the cause of this error?
    
    
        2019-03-04 14:13:49 AsyncRunLoop [ERROR] Task Partition 0 commit failed
        java.lang.NullPointerException
                at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:782)
                at org.apache.samza.checkpoint.kafka.KafkaCheckpointLogKey.<init>(KafkaCheckpointLogKey.java:46)
                at org.apache.samza.checkpoint.kafka.KafkaCheckpointManager.writeCheckpoint(KafkaCheckpointManager.scala:136)
                at org.apache.samza.checkpoint.OffsetManager.writeCheckpoint(OffsetManager.scala:259)
                at org.apache.samza.container.TaskInstance.commit(TaskInstance.scala:205)
                at org.apache.samza.task.AsyncRunLoop$AsyncTaskWorker$5.run(AsyncRunLoop.java:494)
                at org.apache.samza.task.AsyncRunLoop$AsyncTaskWorker.commit(AsyncRunLoop.java:513)
                at org.apache.samza.task.AsyncRunLoop$AsyncTaskWorker.run(AsyncRunLoop.java:379)
                at org.apache.samza.task.AsyncRunLoop$AsyncTaskWorker.access$300(AsyncRunLoop.java:314)
                at org.apache.samza.task.AsyncRunLoop.runTasks(AsyncRunLoop.java:228)
                at org.apache.samza.task.AsyncRunLoop.run(AsyncRunLoop.java:157)
                at org.apache.samza.container.SamzaContainer.run(SamzaContainer.scala:728)
                at org.apache.samza.runtime.LocalContainerRunner.run(LocalContainerRunner.java:102)
                at org.apache.samza.runtime.LocalContainerRunner.main(LocalContainerRunner.java:147)
        2019-03-04 14:13:49 AsyncRunLoop [ERROR] Caught throwable and stopping run loop
    
    
    
        Jeremiah Adams
        Software Engineer
        https://url.emailprotection.link/?bM9S-3pRw1lv8pYfwa-TwdjElP4W2K6b9vP5Crz22L_YcgsRJ-13h-OgPZSwFtU7GSNTDi1z-jdaRvWESRhtTVA~~<https://url.emailprotection.link/?bM9S-3pRw1lv8pYfwa-TwdjElP4W2K6b9vP5Crz22L_YcgsRJ-13h-OgPZSwFtU7GSNTDi1z-jdaRvWESRhtTVA~~><https://url.emailprotection.link/?basKr9vk92a8vVw0XMnK5bmaSKuBc0AuEZ7YasYc7Df8YVt3SYmcjmLWdKMWzAAINWlUUA33ebGI7pSoTl9cg1g~~<https://url.emailprotection.link/?basKr9vk92a8vVw0XMnK5bmaSKuBc0AuEZ7YasYc7Df8YVt3SYmcjmLWdKMWzAAINWlUUA33ebGI7pSoTl9cg1g~~>>
        Blog<https://url.emailprotection.link/?basKr9vk92a8vVw0XMnK5bmaSKuBc0AuEZ7YasYc7Df-lAcqG1fqHPpNw-wd9z7HtUJeCG5_8UjCf2mHtn6C_zQ~~<https://url.emailprotection.link/?basKr9vk92a8vVw0XMnK5bmaSKuBc0AuEZ7YasYc7Df-lAcqG1fqHPpNw-wd9z7HtUJeCG5_8UjCf2mHtn6C_zQ~~>> | Twitter<https://url.emailprotection.link/?bVO2q0UXR235wN_yOnM0FjqITPdBYMD3reLGNddq-zPV5ChMQK9JwV4Be-QnrbRoXpJl8IcknAqKzYtA3RABKww~~<https://url.emailprotection.link/?bVO2q0UXR235wN_yOnM0FjqITPdBYMD3reLGNddq-zPV5ChMQK9JwV4Be-QnrbRoXpJl8IcknAqKzYtA3RABKww~~>> | Facebook<https://url.emailprotection.link/?bUU7m4NfMS_EWGtH1yojBHX9sWZ6uxVdT1eQUkmU5vWY01WFZiS2KJ-c9iLIncdHB7Uw1lRYCprEEpPPQCdiK6Q~~<https://url.emailprotection.link/?bUU7m4NfMS_EWGtH1yojBHX9sWZ6uxVdT1eQUkmU5vWY01WFZiS2KJ-c9iLIncdHB7Uw1lRYCprEEpPPQCdiK6Q~~>> | LinkedIn<https://url.emailprotection.link/?b0ZQfJ1pZYnASyoShs9MJI46-r1lxPhA-JS5VSkR7so-DFP0_HxbOo2LsajGOaoYXxb1ZCOMAu7hZscPCnIKWpXz0cpgQ386SnNHjPcwsu4z90mzBkuwoZc6YxOCzMGA0<https://url.emailprotection.link/?b0ZQfJ1pZYnASyoShs9MJI46-r1lxPhA-JS5VSkR7so-DFP0_HxbOo2LsajGOaoYXxb1ZCOMAu7hZscPCnIKWpXz0cpgQ386SnNHjPcwsu4z90mzBkuwoZc6YxOCzMGA0>>
    
    
    


Re: [POSSIBLE PHISHING] Task Partition Commit Failed After Upgrade

Posted by Jeremiah Adams <JA...@helixeducation.com>.
Thanks for checking Prateek. There’s a similar thread in the group. I decided to follow advice there and am working on migrating to V1.0.0 instead of 0.14.



Jeremiah Adams
Software Engineer
www.helixeducation.com<http://www.helixeducation.com/>
Blog<http://www.helixeducation.com/blog/> | Twitter<https://twitter.com/HelixEducation> | Facebook<https://www.facebook.com/HelixEducation> | LinkedIn<http://www.linkedin.com/company/3609946>


From: Prateek Maheshwari <pr...@gmail.com>
Date: Thursday, March 7, 2019 at 2:57 PM
To: "dev@samza.apache.org" <de...@samza.apache.org>, Jeremiah Adams <JA...@helixeducation.com>
Subject: Re: [POSSIBLE PHISHING] Task Partition Commit Failed After Upgrade

Jeremiah, were you able to resolve this issue?

- Prateek

On Wed, Mar 6, 2019 at 10:08 AM Prateek Maheshwari <pr...@gmail.com>> wrote:
Hi Jeremiah,

The configuration you want to look for is: 'job.systemstreampartition.grouper.factory'. It should default to: 'org.apache.samza.container.grouper.stream.GroupByPartitionFactory'.
Can you check if you see this value in the configuration logged by SamzaContainer during container start? You can grep for: "Using configuration".

For context, there are two groupers for a Samza job. One that groups input partitions into tasks (this one), and one that groups tasks into containers (the one you mentioned above).

Thanks,
Prateek



On Wed, Mar 6, 2019 at 8:14 AM Jeremiah Adams <JA...@helixeducation.com>> wrote:
It appears that the issue is related to the KafkaCheckpointLogKey.java constructor. grouperFactoryClassName here is null.  THe documentation indicates that task.name.grouper.factory config setting has a default value of     org.apache.samza.container.grouper.task.GroupByContainerCountFactory. I wouldn't expect it to be null here.

If I specify GroupByContainerCountFactory for the task.name.grouper.factory in my properties file, I get a NoSuchMethodException:

Exception in thread "main" java.lang.InstantiationException: org.apache.samza.container.grouper.task.GroupByContainerCount
        at java.lang.Class.newInstance(Class.java:427)
        at org.apache.samza.util.Util$.getObj(Util.scala:80)
        at org.apache.samza.coordinator.JobModelManager$.readJobModel(JobModelManager.scala:261)
        at org.apache.samza.coordinator.JobModelManager$.getJobModelManager(JobModelManager.scala:155)
        at org.apache.samza.coordinator.JobModelManager$.apply(JobModelManager.scala:117)
        at org.apache.samza.coordinator.JobModelManager.apply(JobModelManager.scala)
        at org.apache.samza.clustermanager.ClusterBasedJobCoordinator.buildJobModelManager(ClusterBasedJobCoordinator.java:241)
        at org.apache.samza.clustermanager.ClusterBasedJobCoordinator.<init>(ClusterBasedJobCoordinator.java:152)
        at org.apache.samza.clustermanager.ClusterBasedJobCoordinator.main(ClusterBasedJobCoordinator.java:297)
Caused by: java.lang.NoSuchMethodException: org.apache.samza.container.grouper.task.GroupByContainerCount.<init>()
        at java.lang.Class.getConstructor0(Class.java:3082)
        at java.lang.Class.newInstance(Class.java:412)
        ... 8 more



Jeremiah Adams
Software Engineer
www.helixeducation.com<https://url.emailprotection.link/?basKr9vk92a8vVw0XMnK5bnWsxM_w6KChRx8CY_UgrU5RmcwzgGL3Po63B7rJIXeNyMBLKYpptY6Rl-f5kb6p2A~~> <http://www.helixeducation.com/<https://url.emailprotection.link/?basKr9vk92a8vVw0XMnK5bmaSKuBc0AuEZ7YasYc7Df8YVt3SYmcjmLWdKMWzAAINWlUUA33ebGI7pSoTl9cg1g~~>>
Blog <http://www.helixeducation.com/blog/<https://url.emailprotection.link/?basKr9vk92a8vVw0XMnK5bmaSKuBc0AuEZ7YasYc7Df-lAcqG1fqHPpNw-wd9z7HtUJeCG5_8UjCf2mHtn6C_zQ~~>> | Twitter <https://twitter.com/HelixEducation<https://url.emailprotection.link/?bVO2q0UXR235wN_yOnM0FjqITPdBYMD3reLGNddq-zPV5ChMQK9JwV4Be-QnrbRoXpJl8IcknAqKzYtA3RABKww~~>> | Facebook <https://www.facebook.com/HelixEducation<https://url.emailprotection.link/?bUU7m4NfMS_EWGtH1yojBHX9sWZ6uxVdT1eQUkmU5vWY01WFZiS2KJ-c9iLIncdHB7Uw1lRYCprEEpPPQCdiK6Q~~>> | LinkedIn <http://www.linkedin.com/company/3609946<https://url.emailprotection.link/?b0ZQfJ1pZYnASyoShs9MJI46-r1lxPhA-JS5VSkR7so-DFP0_HxbOo2LsajGOaoYXxb1ZCOMAu7hZscPCnIKWpXz0cpgQ386SnNHjPcwsu4z90mzBkuwoZc6YxOCzMGA0>>


On 3/4/19, 2:48 PM, "Jeremiah Adams" <JA...@helixeducation.com>> wrote:

    I am updating dependencies and moving from Samza V0.13.0 to V0.14.0.  I develop locally using the grid app in the hello-samza project to spin up local yarn/zookeeper/kafka instances.

    Grid is running these versions:
    kafka_2.11-0.10.2.1.tgz
    hadoop-2.6.1.tar.gz
    zookeeper-3.4.3.tar.gz


    My job is now failing with the NPE below. anyone have ideas on the cause of this error?


    2019-03-04 14:13:49 AsyncRunLoop [ERROR] Task Partition 0 commit failed
    java.lang.NullPointerException
            at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:782)
            at org.apache.samza.checkpoint.kafka.KafkaCheckpointLogKey.<init>(KafkaCheckpointLogKey.java:46)
            at org.apache.samza.checkpoint.kafka.KafkaCheckpointManager.writeCheckpoint(KafkaCheckpointManager.scala:136)
            at org.apache.samza.checkpoint.OffsetManager.writeCheckpoint(OffsetManager.scala:259)
            at org.apache.samza.container.TaskInstance.commit(TaskInstance.scala:205)
            at org.apache.samza.task.AsyncRunLoop$AsyncTaskWorker$5.run(AsyncRunLoop.java:494)
            at org.apache.samza.task.AsyncRunLoop$AsyncTaskWorker.commit(AsyncRunLoop.java:513)
            at org.apache.samza.task.AsyncRunLoop$AsyncTaskWorker.run(AsyncRunLoop.java:379)
            at org.apache.samza.task.AsyncRunLoop$AsyncTaskWorker.access$300(AsyncRunLoop.java:314)
            at org.apache.samza.task.AsyncRunLoop.runTasks(AsyncRunLoop.java:228)
            at org.apache.samza.task.AsyncRunLoop.run(AsyncRunLoop.java:157)
            at org.apache.samza.container.SamzaContainer.run(SamzaContainer.scala:728)
            at org.apache.samza.runtime.LocalContainerRunner.run(LocalContainerRunner.java:102)
            at org.apache.samza.runtime.LocalContainerRunner.main(LocalContainerRunner.java:147)
    2019-03-04 14:13:49 AsyncRunLoop [ERROR] Caught throwable and stopping run loop



    Jeremiah Adams
    Software Engineer
    https://url.emailprotection.link/?ahfhEufaAWbezBrUFPG98ZJcterGfIerU3ZwsA3Gv_C0~<https://url.emailprotection.link/?bM9S-3pRw1lv8pYfwa-TwdjElP4W2K6b9vP5Crz22L_YcgsRJ-13h-OgPZSwFtU7GSNTDi1z-jdaRvWESRhtTVA~~><https://url.emailprotection.link/?a49H2rNGIIBtQOw6md8OcHp-qKE3Xn2gNiZ3dlqAeSDA~<https://url.emailprotection.link/?basKr9vk92a8vVw0XMnK5bmaSKuBc0AuEZ7YasYc7Df8YVt3SYmcjmLWdKMWzAAINWlUUA33ebGI7pSoTl9cg1g~~>>
    Blog<https://url.emailprotection.link/?a49H2rNGIIBtQOw6md8OcHgFEZu-KYuiu8doY66NWwmmyWxz7kC-27Yfnbdgd2wyh5gjXUa6LMT_NRXsj1g1VVg~~<https://url.emailprotection.link/?basKr9vk92a8vVw0XMnK5bmaSKuBc0AuEZ7YasYc7Df-lAcqG1fqHPpNw-wd9z7HtUJeCG5_8UjCf2mHtn6C_zQ~~>> | Twitter<https://url.emailprotection.link/?a0Q7ct5_6cOdbJ86kpWB0zx6RbtgugTVC7lU_W7za50jLdZQGpLgVlR1V06zckSaM5oOKb6QBo46Qp9xt0Tt7Aw~~<https://url.emailprotection.link/?bVO2q0UXR235wN_yOnM0FjqITPdBYMD3reLGNddq-zPV5ChMQK9JwV4Be-QnrbRoXpJl8IcknAqKzYtA3RABKww~~>> | Facebook<https://url.emailprotection.link/?aAmyAO_nS_C1aDgBLeKyGTu0tksTt1_mn2PcS8KJXNJPM04iRHKgX96qGgENV-dMSER5wl8zDVRr3RsS0OmcF9A~~<https://url.emailprotection.link/?bUU7m4NfMS_EWGtH1yojBHX9sWZ6uxVdT1eQUkmU5vWY01WFZiS2KJ-c9iLIncdHB7Uw1lRYCprEEpPPQCdiK6Q~~>> | LinkedIn<https://url.emailprotection.link/?aanlcNI-cN74Gdz-TD332xAl6lHu7TRNICWoHUFjYf-KlBjrCGHoYR65b3rl-OyW10nWFv6hwYvUSoVHL4b3vGA~~<https://url.emailprotection.link/?b0ZQfJ1pZYnASyoShs9MJI46-r1lxPhA-JS5VSkR7so-DFP0_HxbOo2LsajGOaoYXxb1ZCOMAu7hZscPCnIKWpXz0cpgQ386SnNHjPcwsu4z90mzBkuwoZc6YxOCzMGA0>>



Re: [POSSIBLE PHISHING] Task Partition Commit Failed After Upgrade

Posted by Prateek Maheshwari <pr...@gmail.com>.
Jeremiah, were you able to resolve this issue?

- Prateek

On Wed, Mar 6, 2019 at 10:08 AM Prateek Maheshwari <pr...@gmail.com>
wrote:

> Hi Jeremiah,
>
> The configuration you want to look for is:
> 'job.systemstreampartition.grouper.factory'. It should default to:
> 'org.apache.samza.container.grouper.stream.GroupByPartitionFactory'.
> Can you check if you see this value in the configuration logged by
> SamzaContainer during container start? You can grep for: "Using
> configuration".
>
> For context, there are two groupers for a Samza job. One that groups input
> partitions into tasks (this one), and one that groups tasks into containers
> (the one you mentioned above).
>
> Thanks,
> Prateek
>
>
>
> On Wed, Mar 6, 2019 at 8:14 AM Jeremiah Adams <JA...@helixeducation.com>
> wrote:
>
>> It appears that the issue is related to the KafkaCheckpointLogKey.java
>> constructor. grouperFactoryClassName here is null.  THe documentation
>> indicates that task.name.grouper.factory config setting has a default value
>> of
>>  org.apache.samza.container.grouper.task.GroupByContainerCountFactory. I
>> wouldn't expect it to be null here.
>>
>> If I specify GroupByContainerCountFactory for the
>> task.name.grouper.factory in my properties file, I get a
>> NoSuchMethodException:
>>
>> Exception in thread "main" java.lang.InstantiationException:
>> org.apache.samza.container.grouper.task.GroupByContainerCount
>>         at java.lang.Class.newInstance(Class.java:427)
>>         at org.apache.samza.util.Util$.getObj(Util.scala:80)
>>         at
>> org.apache.samza.coordinator.JobModelManager$.readJobModel(JobModelManager.scala:261)
>>         at
>> org.apache.samza.coordinator.JobModelManager$.getJobModelManager(JobModelManager.scala:155)
>>         at
>> org.apache.samza.coordinator.JobModelManager$.apply(JobModelManager.scala:117)
>>         at
>> org.apache.samza.coordinator.JobModelManager.apply(JobModelManager.scala)
>>         at
>> org.apache.samza.clustermanager.ClusterBasedJobCoordinator.buildJobModelManager(ClusterBasedJobCoordinator.java:241)
>>         at
>> org.apache.samza.clustermanager.ClusterBasedJobCoordinator.<init>(ClusterBasedJobCoordinator.java:152)
>>         at
>> org.apache.samza.clustermanager.ClusterBasedJobCoordinator.main(ClusterBasedJobCoordinator.java:297)
>> Caused by: java.lang.NoSuchMethodException:
>> org.apache.samza.container.grouper.task.GroupByContainerCount.<init>()
>>         at java.lang.Class.getConstructor0(Class.java:3082)
>>         at java.lang.Class.newInstance(Class.java:412)
>>         ... 8 more
>>
>>
>>
>> Jeremiah Adams
>> Software Engineer
>> www.helixeducation.com <http://www.helixeducation.com/>
>> Blog <http://www.helixeducation.com/blog/> | Twitter <
>> https://twitter.com/HelixEducation> | Facebook <
>> https://www.facebook.com/HelixEducation> | LinkedIn <
>> http://www.linkedin.com/company/3609946>
>>
>>
>> On 3/4/19, 2:48 PM, "Jeremiah Adams" <JA...@helixeducation.com> wrote:
>>
>>     I am updating dependencies and moving from Samza V0.13.0 to V0.14.0.
>> I develop locally using the grid app in the hello-samza project to spin up
>> local yarn/zookeeper/kafka instances.
>>
>>     Grid is running these versions:
>>     kafka_2.11-0.10.2.1.tgz
>>     hadoop-2.6.1.tar.gz
>>     zookeeper-3.4.3.tar.gz
>>
>>
>>     My job is now failing with the NPE below. anyone have ideas on the
>> cause of this error?
>>
>>
>>     2019-03-04 14:13:49 AsyncRunLoop [ERROR] Task Partition 0 commit
>> failed
>>     java.lang.NullPointerException
>>             at
>> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:782)
>>             at
>> org.apache.samza.checkpoint.kafka.KafkaCheckpointLogKey.<init>(KafkaCheckpointLogKey.java:46)
>>             at
>> org.apache.samza.checkpoint.kafka.KafkaCheckpointManager.writeCheckpoint(KafkaCheckpointManager.scala:136)
>>             at
>> org.apache.samza.checkpoint.OffsetManager.writeCheckpoint(OffsetManager.scala:259)
>>             at
>> org.apache.samza.container.TaskInstance.commit(TaskInstance.scala:205)
>>             at
>> org.apache.samza.task.AsyncRunLoop$AsyncTaskWorker$5.run(AsyncRunLoop.java:494)
>>             at
>> org.apache.samza.task.AsyncRunLoop$AsyncTaskWorker.commit(AsyncRunLoop.java:513)
>>             at
>> org.apache.samza.task.AsyncRunLoop$AsyncTaskWorker.run(AsyncRunLoop.java:379)
>>             at
>> org.apache.samza.task.AsyncRunLoop$AsyncTaskWorker.access$300(AsyncRunLoop.java:314)
>>             at
>> org.apache.samza.task.AsyncRunLoop.runTasks(AsyncRunLoop.java:228)
>>             at
>> org.apache.samza.task.AsyncRunLoop.run(AsyncRunLoop.java:157)
>>             at
>> org.apache.samza.container.SamzaContainer.run(SamzaContainer.scala:728)
>>             at
>> org.apache.samza.runtime.LocalContainerRunner.run(LocalContainerRunner.java:102)
>>             at
>> org.apache.samza.runtime.LocalContainerRunner.main(LocalContainerRunner.java:147)
>>     2019-03-04 14:13:49 AsyncRunLoop [ERROR] Caught throwable and
>> stopping run loop
>>
>>
>>
>>     Jeremiah Adams
>>     Software Engineer
>>
>> https://url.emailprotection.link/?ahfhEufaAWbezBrUFPG98ZJcterGfIerU3ZwsA3Gv_C0~
>> <
>> https://url.emailprotection.link/?a49H2rNGIIBtQOw6md8OcHp-qKE3Xn2gNiZ3dlqAeSDA~
>> >
>>     Blog<
>> https://url.emailprotection.link/?a49H2rNGIIBtQOw6md8OcHgFEZu-KYuiu8doY66NWwmmyWxz7kC-27Yfnbdgd2wyh5gjXUa6LMT_NRXsj1g1VVg~~>
>> | Twitter<
>> https://url.emailprotection.link/?a0Q7ct5_6cOdbJ86kpWB0zx6RbtgugTVC7lU_W7za50jLdZQGpLgVlR1V06zckSaM5oOKb6QBo46Qp9xt0Tt7Aw~~>
>> | Facebook<
>> https://url.emailprotection.link/?aAmyAO_nS_C1aDgBLeKyGTu0tksTt1_mn2PcS8KJXNJPM04iRHKgX96qGgENV-dMSER5wl8zDVRr3RsS0OmcF9A~~>
>> | LinkedIn<
>> https://url.emailprotection.link/?aanlcNI-cN74Gdz-TD332xAl6lHu7TRNICWoHUFjYf-KlBjrCGHoYR65b3rl-OyW10nWFv6hwYvUSoVHL4b3vGA~~
>> >
>>
>>
>>
>>

Re: [POSSIBLE PHISHING] Task Partition Commit Failed After Upgrade

Posted by Prateek Maheshwari <pr...@gmail.com>.
Hi Jeremiah,

The configuration you want to look for is:
'job.systemstreampartition.grouper.factory'. It should default to:
'org.apache.samza.container.grouper.stream.GroupByPartitionFactory'.
Can you check if you see this value in the configuration logged by
SamzaContainer during container start? You can grep for: "Using
configuration".

For context, there are two groupers for a Samza job. One that groups input
partitions into tasks (this one), and one that groups tasks into containers
(the one you mentioned above).

Thanks,
Prateek



On Wed, Mar 6, 2019 at 8:14 AM Jeremiah Adams <JA...@helixeducation.com>
wrote:

> It appears that the issue is related to the KafkaCheckpointLogKey.java
> constructor. grouperFactoryClassName here is null.  THe documentation
> indicates that task.name.grouper.factory config setting has a default value
> of
>  org.apache.samza.container.grouper.task.GroupByContainerCountFactory. I
> wouldn't expect it to be null here.
>
> If I specify GroupByContainerCountFactory for the
> task.name.grouper.factory in my properties file, I get a
> NoSuchMethodException:
>
> Exception in thread "main" java.lang.InstantiationException:
> org.apache.samza.container.grouper.task.GroupByContainerCount
>         at java.lang.Class.newInstance(Class.java:427)
>         at org.apache.samza.util.Util$.getObj(Util.scala:80)
>         at
> org.apache.samza.coordinator.JobModelManager$.readJobModel(JobModelManager.scala:261)
>         at
> org.apache.samza.coordinator.JobModelManager$.getJobModelManager(JobModelManager.scala:155)
>         at
> org.apache.samza.coordinator.JobModelManager$.apply(JobModelManager.scala:117)
>         at
> org.apache.samza.coordinator.JobModelManager.apply(JobModelManager.scala)
>         at
> org.apache.samza.clustermanager.ClusterBasedJobCoordinator.buildJobModelManager(ClusterBasedJobCoordinator.java:241)
>         at
> org.apache.samza.clustermanager.ClusterBasedJobCoordinator.<init>(ClusterBasedJobCoordinator.java:152)
>         at
> org.apache.samza.clustermanager.ClusterBasedJobCoordinator.main(ClusterBasedJobCoordinator.java:297)
> Caused by: java.lang.NoSuchMethodException:
> org.apache.samza.container.grouper.task.GroupByContainerCount.<init>()
>         at java.lang.Class.getConstructor0(Class.java:3082)
>         at java.lang.Class.newInstance(Class.java:412)
>         ... 8 more
>
>
>
> Jeremiah Adams
> Software Engineer
> www.helixeducation.com <http://www.helixeducation.com/>
> Blog <http://www.helixeducation.com/blog/> | Twitter <
> https://twitter.com/HelixEducation> | Facebook <
> https://www.facebook.com/HelixEducation> | LinkedIn <
> http://www.linkedin.com/company/3609946>
>
>
> On 3/4/19, 2:48 PM, "Jeremiah Adams" <JA...@helixeducation.com> wrote:
>
>     I am updating dependencies and moving from Samza V0.13.0 to V0.14.0.
> I develop locally using the grid app in the hello-samza project to spin up
> local yarn/zookeeper/kafka instances.
>
>     Grid is running these versions:
>     kafka_2.11-0.10.2.1.tgz
>     hadoop-2.6.1.tar.gz
>     zookeeper-3.4.3.tar.gz
>
>
>     My job is now failing with the NPE below. anyone have ideas on the
> cause of this error?
>
>
>     2019-03-04 14:13:49 AsyncRunLoop [ERROR] Task Partition 0 commit failed
>     java.lang.NullPointerException
>             at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:782)
>             at
> org.apache.samza.checkpoint.kafka.KafkaCheckpointLogKey.<init>(KafkaCheckpointLogKey.java:46)
>             at
> org.apache.samza.checkpoint.kafka.KafkaCheckpointManager.writeCheckpoint(KafkaCheckpointManager.scala:136)
>             at
> org.apache.samza.checkpoint.OffsetManager.writeCheckpoint(OffsetManager.scala:259)
>             at
> org.apache.samza.container.TaskInstance.commit(TaskInstance.scala:205)
>             at
> org.apache.samza.task.AsyncRunLoop$AsyncTaskWorker$5.run(AsyncRunLoop.java:494)
>             at
> org.apache.samza.task.AsyncRunLoop$AsyncTaskWorker.commit(AsyncRunLoop.java:513)
>             at
> org.apache.samza.task.AsyncRunLoop$AsyncTaskWorker.run(AsyncRunLoop.java:379)
>             at
> org.apache.samza.task.AsyncRunLoop$AsyncTaskWorker.access$300(AsyncRunLoop.java:314)
>             at
> org.apache.samza.task.AsyncRunLoop.runTasks(AsyncRunLoop.java:228)
>             at
> org.apache.samza.task.AsyncRunLoop.run(AsyncRunLoop.java:157)
>             at
> org.apache.samza.container.SamzaContainer.run(SamzaContainer.scala:728)
>             at
> org.apache.samza.runtime.LocalContainerRunner.run(LocalContainerRunner.java:102)
>             at
> org.apache.samza.runtime.LocalContainerRunner.main(LocalContainerRunner.java:147)
>     2019-03-04 14:13:49 AsyncRunLoop [ERROR] Caught throwable and stopping
> run loop
>
>
>
>     Jeremiah Adams
>     Software Engineer
>
> https://url.emailprotection.link/?ahfhEufaAWbezBrUFPG98ZJcterGfIerU3ZwsA3Gv_C0~
> <
> https://url.emailprotection.link/?a49H2rNGIIBtQOw6md8OcHp-qKE3Xn2gNiZ3dlqAeSDA~
> >
>     Blog<
> https://url.emailprotection.link/?a49H2rNGIIBtQOw6md8OcHgFEZu-KYuiu8doY66NWwmmyWxz7kC-27Yfnbdgd2wyh5gjXUa6LMT_NRXsj1g1VVg~~>
> | Twitter<
> https://url.emailprotection.link/?a0Q7ct5_6cOdbJ86kpWB0zx6RbtgugTVC7lU_W7za50jLdZQGpLgVlR1V06zckSaM5oOKb6QBo46Qp9xt0Tt7Aw~~>
> | Facebook<
> https://url.emailprotection.link/?aAmyAO_nS_C1aDgBLeKyGTu0tksTt1_mn2PcS8KJXNJPM04iRHKgX96qGgENV-dMSER5wl8zDVRr3RsS0OmcF9A~~>
> | LinkedIn<
> https://url.emailprotection.link/?aanlcNI-cN74Gdz-TD332xAl6lHu7TRNICWoHUFjYf-KlBjrCGHoYR65b3rl-OyW10nWFv6hwYvUSoVHL4b3vGA~~
> >
>
>
>
>