[jira] [Commented] (FLUME-2315) org.apache.flume.sink.solr.morphline.BlobDeserializer is unable to handle empty streams

["wolfgang hoschek (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLUME-2315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13895507#comment-13895507 ] 

wolfgang hoschek commented on FLUME-2315:
-----------------------------------------

What's the exception you are seeing in production?

The underlying BlobDeserializer returns a null event for an empty stream in order to signal that empty streams should be ignored. Is this a problem?

The testEmptyStream test suggested above fails because validateMiniParse() test helper currently isn't designed for empty streams. If you wanted to make it handle empty stream you'd change it to read something like this:

{code}
  private void validateMiniParse(EventDeserializer des) throws IOException {
    Event evt;

    des.mark();
    evt = des.readEvent();
    if (mini.length() != 0) {
      assertEquals(new String(evt.getBody()), mini);
    } else {
      assertNull(evt);
    }
    des.reset(); // reset!

    evt = des.readEvent();
    if (mini.length() != 0) {
      assertEquals("data should be repeated, " +
          "because we reset() the stream", new String(evt.getBody()), mini);
    } else {
      assertNull(evt);
    }

    evt = des.readEvent();
    assertNull("Event should be null because there are no lines " +
        "left to read", evt);

    des.mark();
    des.close();
  }
{code}


> org.apache.flume.sink.solr.morphline.BlobDeserializer is unable to handle empty streams
> ---------------------------------------------------------------------------------------
>
>                 Key: FLUME-2315
>                 URL: https://issues.apache.org/jira/browse/FLUME-2315
>             Project: Flume
>          Issue Type: Bug
>          Components: Sinks+Sources
>    Affects Versions: v1.4.0
>            Reporter: Muhammad Ehsan ul Haque
>             Fix For: v1.4.0
>
>
> org.apache.flume.sink.solr.morphline.BlobDeserializer does not handles empty streams correctly. For example if the deserializer is used with a spooling directory containing empty files then the empty files will not be consumed and through exception.
> Also following test will also fail in org.apache.flume.sink.solr.morphline.TestBlobDeserializer
> {code}
>   @Test
>   public void testEmptyStream() throws IOException {
>     mini = "";
>     ResettableInputStream in = new ResettableTestStringInputStream(mini);
>     EventDeserializer des = new BlobDeserializer(new Context(), in);
>     validateMiniParse(des);
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)