You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@kafka.apache.org by "Prabhat Verma (JIRA)" <ji...@apache.org> on 2018/04/27 01:20:00 UTC

[jira] [Created] (KAFKA-6831) FileStreamSink is very slow

Prabhat Verma created KAFKA-6831:
------------------------------------

             Summary: FileStreamSink is very slow
                 Key: KAFKA-6831
                 URL: https://issues.apache.org/jira/browse/KAFKA-6831
             Project: Kafka
          Issue Type: Test
          Components: consumer
    Affects Versions: 1.1.0
            Reporter: Prabhat Verma
             Fix For: 1.1.0


Hi Team,

 

I am very new in kafka. My project requirement is fetch data from source location and place it in other other location (consumer location). I am using FileStreamSink class to perform above action.

I am using Linux machine having memory of 32 GB. 

When i start FIleStreamSink , It is syncing to consumer location very very slowly. Not sure why it is taking 2000 message at a time and then sync it. After that it wait for few second then sync again. This waiting time increases per run .

 

I am processing 600K message but it took 1 hrs to process only 60K message.

 

Below are my config details : 

 

connect-file-sink.property

Name = local-file

Connector.class = FileStreamSource

task.max=20

file=/d/d1/kafka/destination/outfile.txt

topic=abc_partion_20

connect-file-source.property

Name = local-file

Connector.class = FileStreamSource

task.max=20

file=/d/d1/kafka/source/infile.txt

topic=abc_partion_20

 

Can you please help ?

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)