You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Andy Yang <ay...@yahoo.ca> on 2014/11/03 17:09:19 UTC
FlumeTools/FileChannelIntegrityTool
Has anyone ever used the File Channel Integrity Tool released as part of Flume 1.4.0? There is absolutely no documentation on it but anoutstanding JIRA issue from last year to write some.
I've been poking it with a stick as follows:
java -cp "/usr/lib/flume-ng/lib/*" org.apache.flume.tools.FlumeToolsMain FCINTEGRITYTOOL -l /path/to/.flume/file-channel/data/log-xxx
but it returns far too quickly with no output, and no apparent change in the data file.
I'm hoping it will help with a corrupt file channel error:
java.lang.IllegalStateException: Channel closed [channel=c_xxx]. Due to java.io.IOException: Unable to read next Transaction from log file /path/to/.flume/file-channel/data/log-xxx at offset 513457598
...
Caused by: com.google.protobuf.InvalidProtocolBufferException: While parsing a protocol message, the input ended unexpectedly in the middle of a field. This could mean either than the input has been truncated or that an embedded message misreported its own length.
Thanks,
Andy
Re: FlumeTools/FileChannelIntegrityTool
Posted by Hari Shreedharan <hs...@cloudera.com>.
What version of Flume is this? Can you use the current trunk or 1.5? Did you run out of disk space at some point? or truncate the file?
Thanks,
Hari
On Mon, Nov 3, 2014 at 1:23 PM, Andy Yang <ay...@yahoo.ca> wrote:
> Hi Hari,
> Yes, I managed to get the tool to actually do something:
> java -cp "/usr/lib/flume-ng/lib/*" org.apache.flume.tools.FlumeToolsMain FCINTEGRITYTOOL -l /path/to/.flume/file-channel/data
> but it looks like it just gives me the same exception that Flume itself throws when trying to restart:
> Exception in thread "main" java.io.IOException: Unable to read next Transaction from log file /path/to/.flume/file-channel/data/log-xxx at offset 513457598
> at org.apache.flume.channel.file.LogFile$SequentialReader.next(LogFile.java:628)
> at org.apache.flume.tools.FileChannelIntegrityTool.run(FileChannelIntegrityTool.java:87)
> at org.apache.flume.tools.FlumeToolsMain.run(FlumeToolsMain.java:64)
> at org.apache.flume.tools.FlumeToolsMain.main(FlumeToolsMain.java:31)
> Caused by: com.google.protobuf.InvalidProtocolBufferException: While parsing a protocol message, the input ended unexpectedly in the middle of a field. This could mean either than the input has been truncated or that an embedded message misreported its own length.
> at com.google.protobuf.InvalidProtocolBufferException.truncatedMessage(InvalidProtocolBufferException.java:49)
> at com.google.protobuf.CodedInputStream.refillBuffer(CodedInputStream.java:711)
> at com.google.protobuf.CodedInputStream.readRawBytes(CodedInputStream.java:776)
> at com.google.protobuf.CodedInputStream.readBytes(CodedInputStream.java:294)
> at com.google.protobuf.UnknownFieldSet$Builder.mergeFieldFrom(UnknownFieldSet.java:484)
> at com.google.protobuf.UnknownFieldSet$Builder.mergeFrom(UnknownFieldSet.java:461)
> at com.google.protobuf.UnknownFieldSet$Builder.mergeFrom(UnknownFieldSet.java:579)
> at com.google.protobuf.UnknownFieldSet$Builder.mergeFrom(UnknownFieldSet.java:280)
> at com.google.protobuf.CodedInputStream.readGroup(CodedInputStream.java:240)
> at com.google.protobuf.UnknownFieldSet$Builder.mergeFieldFrom(UnknownFieldSet.java:488)
> at com.google.protobuf.GeneratedMessage$Builder.parseUnknownField(GeneratedMessage.java:438)
> at org.apache.flume.channel.file.proto.ProtosFactory$Put$Builder.mergeFrom(ProtosFactory.java:3444)
> at org.apache.flume.channel.file.proto.ProtosFactory$Put$Builder.mergeFrom(ProtosFactory.java:3300)
> at com.google.protobuf.AbstractMessageLite$Builder.mergeFrom(AbstractMessageLite.java:212)
> at com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:746)
> at com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:238)
> at com.google.protobuf.AbstractMessageLite$Builder.mergeDelimitedFrom(AbstractMessageLite.java:282)
> at com.google.protobuf.AbstractMessage$Builder.mergeDelimitedFrom(AbstractMessage.java:760)
> at com.google.protobuf.AbstractMessageLite$Builder.mergeDelimitedFrom(AbstractMessageLite.java:288)
> at com.google.protobuf.AbstractMessage$Builder.mergeDelimitedFrom(AbstractMessage.java:752)
> at org.apache.flume.channel.file.proto.ProtosFactory$Put.parseDelimitedFrom(ProtosFactory.java:3257)
> at org.apache.flume.channel.file.Put.readProtos(Put.java:98)
> at org.apache.flume.channel.file.TransactionEventRecord.fromByteArray(TransactionEventRecord.java:204)
> at org.apache.flume.channel.file.LogFileV3$SequentialReader.doNext(LogFileV3.java:359)
> at org.apache.flume.channel.file.LogFile$SequentialReader.next(LogFile.java:624)
> ... 3 more
> So, I guess "Unable to read next transaction" due to "While parsing a protocol message, the input ended unexpectedly in the middle of a field" is not the kind of file channel integrity problem the tool can fix.
> Which leads me to ask, "is there anything else we can do with this file channel beyond discarding all the data currently persisted within?"
> Thanks,
> Andy
> On Monday, November 3, 2014 2:05 PM, Hari Shreedharan <hs...@cloudera.com> wrote:
>
> You need to specify only the data directories(comma-separated), the channel will look at all the files in the directory.
> Thanks,
> Hari
> On Mon, Nov 3, 2014 at 8:11 AM, Andy Yang <ay...@yahoo.ca> wrote:
> Has anyone ever used the File Channel Integrity Tool released as part of Flume 1.4.0? There is absolutely no documentation on it but an outstanding JIRA issue from last year to write some.
>>
>>I've been poking it with a stick as follows:
>>
>>java -cp "/usr/lib/flume-ng/lib/*" org.apache.flume.tools.FlumeToolsMain FCINTEGRITYTOOL -l /path/to/.flume/file-channel/data/log-xxx
>>
>>but it returns far too quickly with no output, and no apparent change in the data file.
>>
>>I'm hoping it will help with a corrupt file channel error:
>>
>>java.lang.IllegalStateException: Channel closed [channel=c_xxx]. Due to java.io.IOException: Unable to read next Transaction from log file /path/to/.flume/file-channel/data/log-xxx at offset 513457598
>>...
>>Caused by: com.google.protobuf.InvalidProtocolBufferException: While parsing a protocol message, the input ended unexpectedly in the middle of a field. This could mean either than the input has been truncated or that an embedded message misreported its own length.
>>
>>
>>Thanks,
>>Andy
>>
>>
>>
Re: FlumeTools/FileChannelIntegrityTool
Posted by Andy Yang <ay...@yahoo.ca>.
Hi Hari,
Yes, I managed to get the tool to actually do something:
java -cp "/usr/lib/flume-ng/lib/*" org.apache.flume.tools.FlumeToolsMain FCINTEGRITYTOOL -l /path/to/.flume/file-channel/data
but it looks like it just gives me the same exception that Flume itself throws when trying to restart:
Exception in thread "main" java.io.IOException: Unable to read next Transaction from log file /path/to/.flume/file-channel/data/log-xxx at offset 513457598
at org.apache.flume.channel.file.LogFile$SequentialReader.next(LogFile.java:628)
at org.apache.flume.tools.FileChannelIntegrityTool.run(FileChannelIntegrityTool.java:87)
at org.apache.flume.tools.FlumeToolsMain.run(FlumeToolsMain.java:64)
at org.apache.flume.tools.FlumeToolsMain.main(FlumeToolsMain.java:31)
Caused by: com.google.protobuf.InvalidProtocolBufferException: While parsing a protocol message, the input ended unexpectedly in the middle of a field. This could mean either than the input has been truncated or that an embedded message misreported its own length.
at com.google.protobuf.InvalidProtocolBufferException.truncatedMessage(InvalidProtocolBufferException.java:49)
at com.google.protobuf.CodedInputStream.refillBuffer(CodedInputStream.java:711)
at com.google.protobuf.CodedInputStream.readRawBytes(CodedInputStream.java:776)
at com.google.protobuf.CodedInputStream.readBytes(CodedInputStream.java:294)
at com.google.protobuf.UnknownFieldSet$Builder.mergeFieldFrom(UnknownFieldSet.java:484)
at com.google.protobuf.UnknownFieldSet$Builder.mergeFrom(UnknownFieldSet.java:461)
at com.google.protobuf.UnknownFieldSet$Builder.mergeFrom(UnknownFieldSet.java:579)
at com.google.protobuf.UnknownFieldSet$Builder.mergeFrom(UnknownFieldSet.java:280)
at com.google.protobuf.CodedInputStream.readGroup(CodedInputStream.java:240)
at com.google.protobuf.UnknownFieldSet$Builder.mergeFieldFrom(UnknownFieldSet.java:488)
at com.google.protobuf.GeneratedMessage$Builder.parseUnknownField(GeneratedMessage.java:438)
at org.apache.flume.channel.file.proto.ProtosFactory$Put$Builder.mergeFrom(ProtosFactory.java:3444)
at org.apache.flume.channel.file.proto.ProtosFactory$Put$Builder.mergeFrom(ProtosFactory.java:3300)
at com.google.protobuf.AbstractMessageLite$Builder.mergeFrom(AbstractMessageLite.java:212)
at com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:746)
at com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:238)
at com.google.protobuf.AbstractMessageLite$Builder.mergeDelimitedFrom(AbstractMessageLite.java:282)
at com.google.protobuf.AbstractMessage$Builder.mergeDelimitedFrom(AbstractMessage.java:760)
at com.google.protobuf.AbstractMessageLite$Builder.mergeDelimitedFrom(AbstractMessageLite.java:288)
at com.google.protobuf.AbstractMessage$Builder.mergeDelimitedFrom(AbstractMessage.java:752)
at org.apache.flume.channel.file.proto.ProtosFactory$Put.parseDelimitedFrom(ProtosFactory.java:3257)
at org.apache.flume.channel.file.Put.readProtos(Put.java:98)
at org.apache.flume.channel.file.TransactionEventRecord.fromByteArray(TransactionEventRecord.java:204)
at org.apache.flume.channel.file.LogFileV3$SequentialReader.doNext(LogFileV3.java:359)
at org.apache.flume.channel.file.LogFile$SequentialReader.next(LogFile.java:624)
... 3 more
So, I guess "Unable to read next transaction" due to "While parsing a protocol message, the input ended unexpectedly in the middle of a field" is not the kind of file channel integrity problem the tool can fix.
Which leads me to ask, "is there anything else we can do with this file channel beyond discarding all the data currently persisted within?"
Thanks,
Andy
On Monday, November 3, 2014 2:05 PM, Hari Shreedharan <hs...@cloudera.com> wrote:
You need to specify only the data directories(comma-separated), the channel will look at all the files in the directory.
Thanks,
Hari
On Mon, Nov 3, 2014 at 8:11 AM, Andy Yang <ay...@yahoo.ca> wrote:
Has anyone ever used the File Channel Integrity Tool released as part of Flume 1.4.0? There is absolutely no documentation on it but an outstanding JIRA issue from last year to write some.
>
>I've been poking it with a stick as follows:
>
>java -cp "/usr/lib/flume-ng/lib/*" org.apache.flume.tools.FlumeToolsMain FCINTEGRITYTOOL -l /path/to/.flume/file-channel/data/log-xxx
>
>but it returns far too quickly with no output, and no apparent change in the data file.
>
>I'm hoping it will help with a corrupt file channel error:
>
>java.lang.IllegalStateException: Channel closed [channel=c_xxx]. Due to java.io.IOException: Unable to read next Transaction from log file /path/to/.flume/file-channel/data/log-xxx at offset 513457598
>...
>Caused by: com.google.protobuf.InvalidProtocolBufferException: While parsing a protocol message, the input ended unexpectedly in the middle of a field. This could mean either than the input has been truncated or that an embedded message misreported its own length.
>
>
>Thanks,
>Andy
>
>
>
Re: FlumeTools/FileChannelIntegrityTool
Posted by Hari Shreedharan <hs...@cloudera.com>.
You need to specify only the data directories(comma-separated), the channel will look at all the files in the directory.
Thanks,
Hari
On Mon, Nov 3, 2014 at 8:11 AM, Andy Yang <ay...@yahoo.ca> wrote:
> Has anyone ever used the File Channel Integrity Tool released as part of Flume 1.4.0? There is absolutely no documentation on it but anoutstanding JIRA issue from last year to write some.
>
> I've been poking it with a stick as follows:
>
> java -cp "/usr/lib/flume-ng/lib/*" org.apache.flume.tools.FlumeToolsMain FCINTEGRITYTOOL -l /path/to/.flume/file-channel/data/log-xxx
>
> but it returns far too quickly with no output, and no apparent change in the data file.
>
> I'm hoping it will help with a corrupt file channel error:
>
> java.lang.IllegalStateException: Channel closed [channel=c_xxx]. Due to java.io.IOException: Unable to read next Transaction from log file /path/to/.flume/file-channel/data/log-xxx at offset 513457598
> ...
> Caused by: com.google.protobuf.InvalidProtocolBufferException: While parsing a protocol message, the input ended unexpectedly in the middle of a field. This could mean either than the input has been truncated or that an embedded message misreported its own length.
> Thanks,
> Andy