You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Mike Drob (JIRA)" <ji...@apache.org> on 2016/01/20 23:46:39 UTC

[jira] [Commented] (SOLR-8575) Fix HDFSLogReader replay status numbers and a performance bug where we can reopen FSDataInputStream too often.

    [ https://issues.apache.org/jira/browse/SOLR-8575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15109639#comment-15109639 ] 

Mike Drob commented on SOLR-8575:
---------------------------------

I had talked to [~andrew.wang] about this maybe a month ago and he suggested that if you want to get the updated size from the file then you have to use hsync with the length update flag\[1] using an HdfsOutputStream (not FSDataOutputStream like we use).

Using an internally stored length is probably better anyway, though.

\[1]: https://github.com/apache/hadoop/blob/2ec438e8f7cd77cb48fd1264781e60a48e331908/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsDataOutputStream.java#L105

> Fix HDFSLogReader replay status numbers and a performance bug where we can reopen FSDataInputStream too often.
> --------------------------------------------------------------------------------------------------------------
>
>                 Key: SOLR-8575
>                 URL: https://issues.apache.org/jira/browse/SOLR-8575
>             Project: Solr
>          Issue Type: Bug
>            Reporter: Mark Miller
>            Assignee: Mark Miller
>         Attachments: SOLR-8575.patch
>
>
> [~pdvorak@cloudera.com] noticed some funny transaction log replay status logging a while back:
> active=true starting pos=444978 current pos=2855956 current size=16262 % read=17562
> active=true starting pos=444978 current pos=5748869 current size=16262 % read=35352
> 17562% read? Current size does not change as expected in this case?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org