You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flume.apache.org by "Ralph Goers (Jira)" <ji...@apache.org> on 2022/06/01 19:02:00 UTC
[jira] [Updated] (FLUME-3423) Taildir source will throw unnecessay exception when restart agent
[ https://issues.apache.org/jira/browse/FLUME-3423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ralph Goers updated FLUME-3423:
-------------------------------
Fix Version/s: (was: 1.10.0)
> Taildir source will throw unnecessay exception when restart agent
> -----------------------------------------------------------------
>
> Key: FLUME-3423
> URL: https://issues.apache.org/jira/browse/FLUME-3423
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: 1.9.0
> Reporter: Jasper D
> Priority: Trivial
> Labels: patch
> Original Estimate: 5m
> Time Spent: 50m
> Remaining Estimate: 0h
>
> h2. *Phenomenon*
> When we use Taildir source, it will auto create an empty tail position file at start time. But if the filegroup directory is empty, it has no file to tail and then we restart agent, it will throw an EOFException. Although it has little effect on functionality, it will annoy users.
> h2. *Recur*
> This problem can esaily recur.
> We can use the example.conf from flume user guide directly.
> {code:java}
> //代码占位符
> a1.sources = r1
> a1.channels = c1
> a1.sources.r1.type = TAILDIR
> a1.sources.r1.channels = c1
> a1.sources.r1.positionFile = /var/log/flume/taildir_position.json
> a1.sources.r1.filegroups = f1
> a1.sources.r1.filegroups.f1 = /var/log/test1/*.log
> a1.sources.ri.maxBatchCount = 1000
> a1.channels.c1.type = memory
> a1.channels.c1.capacity = 1000
> a1.channels.c1.transactionCapacity = 100
> a1.sinks.k1.type = logger
> a1.sinks.k1.channel = c1{code}
> and start flume with it
> {code:java}
> //代码占位符
> bin/flume-ng agent --conf conf --conf-file conf/example.conf --name a1 -Dflume.root.logger=INFO,console {code}
> then ctrl + c stop it.
> Just wait a moment, and then start flume again, then it will throw an exception like below.
> {code:java}
> //代码占位符
> ERROR [lifecycleSupervisor-1-1] (org.apache.flume.source.taildir.ReliableTaildirEventReader.loadPositionFile:147) - Failed loading positionFile: /var/log/flume/taildir_position.json
> java.io.EOFException: End of input at line 1 column 1
> at com.google.gson.stream.JsonReader.nextNonWhitespace(JsonReader.java:954) ~[gson-2.2.2.jar:?]
> at com.google.gson.stream.JsonReader.nextValue(JsonReader.java:771) ~[gson-2.2.2.jar:?]
> at com.google.gson.stream.JsonReader.peek(JsonReader.java:367) ~[gson-2.2.2.jar:?]
> at com.google.gson.stream.JsonReader.expect(JsonReader.java:337) ~[gson-2.2.2.jar:?]
> at com.google.gson.stream.JsonReader.beginArray(JsonReader.java:306) ~[gson-2.2.2.jar:?]
> at org.apache.flume.source.taildir.ReliableTaildirEventReader.loadPositionFile(ReliableTaildirEventReader.java:111) [flume-taildir-source-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
> at org.apache.flume.source.taildir.ReliableTaildirEventReader.<init>(ReliableTaildirEventReader.java:96) [flume-taildir-source-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
> at org.apache.flume.source.taildir.ReliableTaildirEventReader.<init>(ReliableTaildirEventReader.java:49) [flume-taildir-source-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
> at org.apache.flume.source.taildir.ReliableTaildirEventReader$Builder.build(ReliableTaildirEventReader.java:355) [flume-taildir-source-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
> at org.apache.flume.source.taildir.TaildirSource.start(TaildirSource.java:105) [flume-taildir-source-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
> at org.apache.flume.source.PollableSourceRunner.start(PollableSourceRunner.java:71) [flume-ng-core-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
> at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:249) [flume-ng-core-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0-292]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0-292]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0-292]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0-292]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0-292]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0-292]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0-292] {code}
> h2. Fix
> We just add a condition in ReliableTaildirEventReader constructor.
> The pr/patch will be submitted as as shown below.
> Or let it still exist and using warning log instead of exception thrown is better?
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@flume.apache.org
For additional commands, e-mail: issues-help@flume.apache.org