[jira] [Resolved] (KAFKA-6831) FileStreamSink is very slow

["Randall Hauch (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-6831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Randall Hauch resolved KAFKA-6831.
----------------------------------
       Resolution: Invalid
    Fix Version/s:     (was: 1.1.0)

[~vrmprabhat], first of all, these kinds of questions are better asked through the user mailing list or other online resources. But to get you started, Kafka Connect uses normal producers and consumers under the covers, so be sure that you set the consumer settings in the Connect worker configuration to handle traffic like yours. It's often a balancing act of batch and message sizes to handle your throughputs. A few commonly used consumer properties include {{fetch.min.bytes}}, {{max.partition.fetch.bytes}}, {{fetch.max.bytes}}, and {{fetch.message.max.bytes}}; these might be too small given the message size, # of partitions, throughput, etc., and might be instructing Connect to consume too many small batches.


> FileStreamSink is very slow
> ---------------------------
>
>                 Key: KAFKA-6831
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6831
>             Project: Kafka
>          Issue Type: Test
>          Components: consumer
>    Affects Versions: 1.1.0
>            Reporter: Prabhat Verma
>            Priority: Major
>
> Hi Team,
>  
> I am very new in kafka. My project requirement is fetch data from source location and place it in other other location (consumer location). I am using FileStreamSink class to perform above action.
> I am using Linux machine having memory of 32 GB. 
> When i start FIleStreamSink , It is syncing to consumer location very very slowly. Not sure why it is taking 2000 message at a time and then sync it. After that it wait for few second then sync again. This waiting time increases per run .
>  
> I am processing 600K message but it took 1 hrs to process only 60K message.
>  
> Below are my config details : 
>  
> connect-file-sink.property
> Name = local-file
> Connector.class = FileStreamSource
> task.max=20
> file=/d/d1/kafka/destination/outfile.txt
> topic=abc_partion_20
> connect-file-source.property
> Name = local-file
> Connector.class = FileStreamSource
> task.max=20
> file=/d/d1/kafka/source/infile.txt
> topic=abc_partion_20
>  
> Can you please help ?
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)