[jira] [Commented] (KAFKA-6831) FileStreamSink is very slow

["Prabhat Verma (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-6831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16459466#comment-16459466 ] 

Prabhat Verma commented on KAFKA-6831:
--------------------------------------

Any help please ...

> FileStreamSink is very slow
> ---------------------------
>
>                 Key: KAFKA-6831
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6831
>             Project: Kafka
>          Issue Type: Test
>          Components: consumer
>    Affects Versions: 1.1.0
>            Reporter: Prabhat Verma
>            Priority: Major
>             Fix For: 1.1.0
>
>
> Hi Team,
>  
> I am very new in kafka. My project requirement is fetch data from source location and place it in other other location (consumer location). I am using FileStreamSink class to perform above action.
> I am using Linux machine having memory of 32 GB. 
> When i start FIleStreamSink , It is syncing to consumer location very very slowly. Not sure why it is taking 2000 message at a time and then sync it. After that it wait for few second then sync again. This waiting time increases per run .
>  
> I am processing 600K message but it took 1 hrs to process only 60K message.
>  
> Below are my config details : 
>  
> connect-file-sink.property
> Name = local-file
> Connector.class = FileStreamSource
> task.max=20
> file=/d/d1/kafka/destination/outfile.txt
> topic=abc_partion_20
> connect-file-source.property
> Name = local-file
> Connector.class = FileStreamSource
> task.max=20
> file=/d/d1/kafka/source/infile.txt
> topic=abc_partion_20
>  
> Can you please help ?
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)