[jira] [Comment Edited] (KAFKA-12594) Self-join produces no ouput, and prevents other streams from producing output

["Jamie Brandon (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-12594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17312802#comment-17312802 ] 

Jamie Brandon edited comment on KAFKA-12594 at 4/1/21, 1:16 AM:
----------------------------------------------------------------

I'm also noticing that when I just run the identity pipeline, it works but it occasionally drops some records. Not sure if this is related or should be a separate issue.

{code:bash}
[nix-shell:~/streaming-consistency/kafka-streams]$ wc -l tmp/*transactions
 999514 tmp/accepted_transactions
 1000000 tmp/transactions
 1999514 total

[nix-shell:~/streaming-consistency/kafka-streams]$ cat tmp/transactions | cut -d',' -f 1 | cut -d' ' -f 2 > in

[nix-shell:~/streaming-consistency/kafka-streams]$ cat tmp/accepted_transactions | cut -d',' -f 1 | cut -d':' -f 2 > out

[nix-shell:~/streaming-consistency/kafka-streams]$ diff in out | wc -l
 487

[nix-shell:~/streaming-consistency/kafka-streams]$ diff in out | head
 25313,25798d25312
 < 25312
 < 25313
 < 25314
 < 25315
 < 25316
 < 25317
 < 25318
 < 25319
 < 25320
 
[nix-shell:~/streaming-consistency/kafka-streams]$ diff in out | tail
 < 25788
 < 25789
 < 25790
 < 25791
 < 25792
 < 25793
 < 25794
 < 25795
 < 25796
 < 25797
{code}

Happens around 1 in 5 runs. Not always the same range of records dropped.


was (Author: jamii):
I'm also noticing that when I just run the identity pipeline, it works but it occasionally drops some records. Not sure if this is related or should be a separate issue.

{code:bash}
[nix-shell:~/streaming-consistency/kafka-streams]$ wc -l tmp/*transactions
 999514 tmp/accepted_transactions
 1000000 tmp/transactions
 1999514 total

[nix-shell:~/streaming-consistency/kafka-streams]$ cat tmp/transactions | cut -d',' -f 1 | cut -d' ' -f 2 > in

[nix-shell:~/streaming-consistency/kafka-streams]$ cat tmp/accepted_transactions | cut -d',' -f 1 | cut -d':' -f 2 > out

[nix-shell:~/streaming-consistency/kafka-streams]$ diff in out | wc -l
 487

[nix-shell:~/streaming-consistency/kafka-streams]$ diff in out | head
 25313,25798d25312
 < 25312
 < 25313
 < 25314
 < 25315
 < 25316
 < 25317
 < 25318
 < 25319
 < 25320
 
[nix-shell:~/streaming-consistency/kafka-streams]$ diff in out | tail
 < 25788
 < 25789
 < 25790
 < 25791
 < 25792
 < 25793
 < 25794
 < 25795
 < 25796
 < 25797
{code}

> Self-join produces no ouput, and prevents other streams from producing output
> -----------------------------------------------------------------------------
>
>                 Key: KAFKA-12594
>                 URL: https://issues.apache.org/jira/browse/KAFKA-12594
>             Project: Kafka
>          Issue Type: Bug
>          Components: streams
>    Affects Versions: 2.7.0
>         Environment: https://github.com/jamii/streaming-consistency/blob/c1f504e73141405ee6cd0c7f217604d643babf81/pkgs.nix
> [nix-shell:~/streaming-consistency/kafka-streams]$ java -version
> openjdk version "1.8.0_265"
> OpenJDK Runtime Environment (build 1.8.0_265-ga)
> OpenJDK 64-Bit Server VM (build 25.265-bga, mixed mode)
> [nix-shell:~/streaming-consistency/kafka-streams]$ nix-info
> system: "x86_64-linux", multi-user?: yes, version: nix-env (Nix) 2.3.10, channels(jamie): "", channels(root): "nixos-20.09.3554.f8929dce13e", nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos
>            Reporter: Jamie Brandon
>            Priority: Major
>
> I have a simple kafka-streams example which just reads in some transactions and writes them back out again.
> https://github.com/jamii/streaming-consistency/blob/c1f504e73141405ee6cd0c7f217604d643babf81/kafka-streams/src/main/java/Demo.java
> This works fine and I see the right output in 'accepted_transactions'.
> If I uncomment the left join at line 58, then not only do I not get any output for the left join, but I don't get any output in 'accepted_transactions' either.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)