You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Steve Jerman <st...@kloudspot.com> on 2017/03/24 01:11:08 UTC
Question Regarding a sink..
Hi,
I have a sink writing data to InfluxDB. I’ve noticed that the sink gets multiple copies of upstream records..
Why does this happen, and how can I avoid it… ?
Below is a trace …showing 2 records (I have a parallelism of two) for each record in the ‘.printToError’ for the same stream.
Any help/suggestions appreciated.
Steve
1> Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=2F:A4:BD:56:EC:4D, sessionTime=15000}]
2> Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=17:8E:FC:7E:F7:20, sessionTime=20000}]
1> Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=40:07:2D:CB:41:07, sessionTime=5000}]
2> Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=28:24:0B:B6:42:CA, sessionTime=5000}]
1> Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=8F:13:AC:4A:DA:93, sessionTime=5000}]
2> Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=43:7D:8A:D4:7D:D7, sessionTime=5000}]
2> Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=77:CD:BD:48:EE:D8, sessionTime=5000}]
18:41:27,351 INFO com.kloudspot.flink.sink.InfluxSink - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=2F:A4:BD:56:EC:4D, sessionTime=15000}]
18:41:27,351 INFO com.kloudspot.flink.sink.InfluxSink - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=2F:A4:BD:56:EC:4D, sessionTime=15000}]
18:41:27,351 INFO com.kloudspot.flink.sink.InfluxSink - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=17:8E:FC:7E:F7:20, sessionTime=20000}]
18:41:27,351 INFO com.kloudspot.flink.sink.InfluxSink - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=17:8E:FC:7E:F7:20, sessionTime=20000}]
18:41:27,351 INFO com.kloudspot.flink.sink.InfluxSink - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=40:07:2D:CB:41:07, sessionTime=5000}]
18:41:27,351 INFO com.kloudspot.flink.sink.InfluxSink - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=40:07:2D:CB:41:07, sessionTime=5000}]
18:41:27,351 INFO com.kloudspot.flink.sink.InfluxSink - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=28:24:0B:B6:42:CA, sessionTime=5000}]
18:41:27,351 INFO com.kloudspot.flink.sink.InfluxSink - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=28:24:0B:B6:42:CA, sessionTime=5000}]
18:41:27,352 INFO com.kloudspot.flink.sink.InfluxSink - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=8F:13:AC:4A:DA:93, sessionTime=5000}]
18:41:27,352 INFO com.kloudspot.flink.sink.InfluxSink - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=8F:13:AC:4A:DA:93, sessionTime=5000}]
18:41:27,352 INFO com.kloudspot.flink.sink.InfluxSink - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=43:7D:8A:D4:7D:D7, sessionTime=5000}]
18:41:27,352 INFO com.kloudspot.flink.sink.InfluxSink - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=43:7D:8A:D4:7D:D7, sessionTime=5000}]
18:41:27,352 INFO com.kloudspot.flink.sink.InfluxSink - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=77:CD:BD:48:EE:D8, sessionTime=5000}]
18:41:27,352 INFO com.kloudspot.flink.sink.InfluxSink - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=77:CD:BD:48:EE:D8, sessionTime=5000}]
Re: Question Regarding a sink..
Posted by "Tzu-Li (Gordon) Tai" <tz...@apache.org>.
Hi Steve,
This normally shouldn’t happen, unless there simply is two copies of the data.
What is the source of the topology? Also, this might be obvious, but if you have broadcasted your input stream to the sink, then each sink instance would then get all records in the input stream.
Cheers,
Gordon
On March 24, 2017 at 9:11:35 AM, Steve Jerman (steve@kloudspot.com) wrote:
Hi,
I have a sink writing data to InfluxDB. I’ve noticed that the sink gets multiple copies of upstream records..
Why does this happen, and how can I avoid it… ?
Below is a trace …showing 2 records (I have a parallelism of two) for each record in the ‘.printToError’ for the same stream.
Any help/suggestions appreciated.
Steve
1> Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=2F:A4:BD:56:EC:4D, sessionTime=15000}]
2> Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=17:8E:FC:7E:F7:20, sessionTime=20000}]
1> Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=40:07:2D:CB:41:07, sessionTime=5000}]
2> Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=28:24:0B:B6:42:CA, sessionTime=5000}]
1> Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=8F:13:AC:4A:DA:93, sessionTime=5000}]
2> Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=43:7D:8A:D4:7D:D7, sessionTime=5000}]
2> Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=77:CD:BD:48:EE:D8, sessionTime=5000}]
18:41:27,351 INFO com.kloudspot.flink.sink.InfluxSink - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=2F:A4:BD:56:EC:4D, sessionTime=15000}]
18:41:27,351 INFO com.kloudspot.flink.sink.InfluxSink - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=2F:A4:BD:56:EC:4D, sessionTime=15000}]
18:41:27,351 INFO com.kloudspot.flink.sink.InfluxSink - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=17:8E:FC:7E:F7:20, sessionTime=20000}]
18:41:27,351 INFO com.kloudspot.flink.sink.InfluxSink - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=17:8E:FC:7E:F7:20, sessionTime=20000}]
18:41:27,351 INFO com.kloudspot.flink.sink.InfluxSink - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=40:07:2D:CB:41:07, sessionTime=5000}]
18:41:27,351 INFO com.kloudspot.flink.sink.InfluxSink - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=40:07:2D:CB:41:07, sessionTime=5000}]
18:41:27,351 INFO com.kloudspot.flink.sink.InfluxSink - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=28:24:0B:B6:42:CA, sessionTime=5000}]
18:41:27,351 INFO com.kloudspot.flink.sink.InfluxSink - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=28:24:0B:B6:42:CA, sessionTime=5000}]
18:41:27,352 INFO com.kloudspot.flink.sink.InfluxSink - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=8F:13:AC:4A:DA:93, sessionTime=5000}]
18:41:27,352 INFO com.kloudspot.flink.sink.InfluxSink - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=8F:13:AC:4A:DA:93, sessionTime=5000}]
18:41:27,352 INFO com.kloudspot.flink.sink.InfluxSink - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=43:7D:8A:D4:7D:D7, sessionTime=5000}]
18:41:27,352 INFO com.kloudspot.flink.sink.InfluxSink - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=43:7D:8A:D4:7D:D7, sessionTime=5000}]
18:41:27,352 INFO com.kloudspot.flink.sink.InfluxSink - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=77:CD:BD:48:EE:D8, sessionTime=5000}]
18:41:27,352 INFO com.kloudspot.flink.sink.InfluxSink - Point [name=Presence, time=1490316086000, tags={ap=1OPUA7IKN9, associated=false, dwellTime=less_than_fifteen_mins, location=1OPUA7IKN9, returned=false, sessionEnd=false}, precision=MILLISECONDS, fields={client=77:CD:BD:48:EE:D8, sessionTime=5000}]