[jira] [Commented] (FLUME-2380) Encrypted files from spooldir to S3 using file channel

["Ashish Paliwal (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLUME-2380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13988583#comment-13988583 ] 

Ashish Paliwal commented on FLUME-2380:
---------------------------------------

Can you please ask this question on user Mailing List (http://flume.apache.org/mailinglists.html)

> Encrypted files from spooldir to S3 using file channel
> ------------------------------------------------------
>
>                 Key: FLUME-2380
>                 URL: https://issues.apache.org/jira/browse/FLUME-2380
>             Project: Flume
>          Issue Type: Question
>          Components: File Channel
>         Environment: Linux \ S3 
>            Reporter: MX
>            Priority: Blocker
>
> 0down votefavorite
>  
> I am a newbie to flume tring to move data from spooldir to S3, I am successfully able to move the data and roll the files into a bucket. I need to encrypt the data loaded into S3. I tried the following configuration in flume , log files tells me that "Encryption is enabled" but the files are written in S3 without encryption. Can someone tell me if flume encryption works with the config?
>  
> flume.conf
> -----------
> agent.sources = eventlog
> agent.channels = file_channel
> agent.sinks = s3sink
> # Define / Configure source
> agent.sources.eventlog.type = spooldir
> agent.sources.eventlog.spoolDir = /test/data
> agent.sources.eventlog.restart = true
> agent.sources.eventlog.batchSize = 1000
> #agent.sources.eventlog.type = seq
> # HDFS sinks
> agent.sinks.s3sink.type = hdfs
> agent.sinks.s3sink.hdfs.path = s3n://<ACCESSKEY>:<SECRETKEY>@<BUCKET>
> agent.sinks.s3sink.hdfs.fileType = DataStream
> agent.sinks.s3sink.hdfs.filePrefix = event
> agent.sinks.s3sink.hdfs.fileSuffix = .log
> agent.sinks.s3sink.hdfs.writeFormat = Text 
> agent.sinks.s3sink.hdfs.batchSize = 1000
> agent.sinks.s3sink.hdfs.rollSize = 10485760
> agent.sinks.s3sink.hdfs.rollCount = 0
> agent.sinks.s3sink.hdfs.rollInterval = 60
> agent.sinks.s3sink.hdfs.threadsPoolSize = 100
> # Use a channel which buffers events in memory
> agent.channels.file_channel.type = file
> #25GB
> agent.channels.fileChannel.maxFileSize = 25000000000
> agent.channels.file_channel.checkpointDir = /var/flume/checkpoint
> agent.channels.file_channel.dataDirs = /<directory>/data
> agent.channels.file_channel.capacity = 50000000
> agent.channels.file_channel.transactionCapacity = 10000
> agent.channels.file_channel.encryption.activeKey = key-1
> agent.channels.file_channel.encryption.cipherProvider = AESCTRNOPADDING
> #agent.channels.file_channel.encryption.keyProvider = key-provider-1
> agent.channels.file_channel.encryption.keyProvider = JCEKSFILE
> agent.channels.file_channel.encryption.keyProvider.keys = key-1
> agent.channels.file_channel.encryption.keyProvider.keyStoreFile = /<directory>/my.keystore
> agent.channels.file_channel.encryption.keyProvider.keyStorePasswordFile = /<directory>/my.keystore.password 
> # Bind the source and sink to the channel
> agent.sources.eventlog.channels = file_channel
> agent.sinks.s3sink.channel = file_channel



--
This message was sent by Atlassian JIRA
(v6.2#6252)