You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@flume.apache.org by Brock Noland <br...@cloudera.com> on 2012/09/06 17:41:21 UTC

Review Request: FLUME-1424: File Channel should support encryption

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6946/
-----------------------------------------------------------

Review request for Flume.


Description
-------

Patch which resolves FLUME-1424, it's on top of the latest FLUME-1487 patch. Additionally, it changes the output format slightly so we would not want to make a release with only FLUME-1487 since files written byt 1487 are not readable by 1424. We could add yet another version to get around this, but since both patches will be committed to trunk in the same major release, it makes sense to make them a pair.


Below is some sample configuration:


Generating a key with a password seperate from the key store password:

    keytool -genseckey -alias key-0 -keypass keyPassword -keyalg AES \
      -keysize 192 -validity 9000 -keystore test.keystore \
      -storetype jceks -storepass keyStorePassword

Generating a key with the password the same as the key store password:      

    keytool -genseckey -alias key-1 -keyalg AES -keysize 192 -validity 9000 \
      -keystore src/test/resources/test.keystore -storetype jceks \
      -storepass keyStorePassword

Here is a basic configuration with key-0 using the same password as the key store:

agent.channels.ch-0.encryption.keyProvider = JCEKSFILE
agent.channels.ch-0.encryption.keyAlias = key-0
agent.channels.ch-0.encryption.cipherProvider = AESCTRNOPADDING
agent.channels.ch-0.encryption.keyStoreFile = /path/to/my.keystore
agent.channels.ch-0.encryption.keyStorePasswordFile = /path/to/my.keystore.password
agent.channels.ch-0.encryption.keys = key-0

Let's say you have aged key-0 out and new files should be encrypted with key-1:

agent.channels.ch-0.encryption.keyProvider = JCEKSFILE
agent.channels.ch-0.encryption.keyAlias = key-1
agent.channels.ch-0.encryption.cipherProvider = AESCTRNOPADDING
agent.channels.ch-0.encryption.keyStoreFile = /path/to/my.keystore
agent.channels.ch-0.encryption.keyStorePasswordFile = /path/to/my.keystore.password
agent.channels.ch-0.encryption.keys = key-0 key-1

The same scenerio as above, however key-0 has it's own password:

agent.channels.ch-0.encryption.keyProvider = JCEKSFILE
agent.channels.ch-0.encryption.keyAlias = key-1
agent.channels.ch-0.encryption.cipherProvider = AESCTRNOPADDING
agent.channels.ch-0.encryption.keyStoreFile = /path/to/my.keystore
agent.channels.ch-0.encryption.keyStorePasswordFile = /path/to/my.keystore.password
agent.channels.ch-0.encryption.key-0 = /path/to/key-0.password
agent.channels.ch-0.encryption.keys = key-0 key-1


This addresses bug FLUME-1424.
    https://issues.apache.org/jira/browse/FLUME-1424


Diffs
-----

  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/CheckpointRebuilder.java 32b5324 
  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java 5d588ea 
  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java e13ecc4 
  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java 2867fc7 
  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFileFactory.java PRE-CREATION 
  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFileV2.java PRE-CREATION 
  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFileV3.java PRE-CREATION 
  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java 4e908ec 
  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java 67c68f8 
  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/encryption/AESCTRNoPaddingProvider.java PRE-CREATION 
  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/encryption/CipherProvider.java PRE-CREATION 
  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/encryption/CipherProviderFactory.java PRE-CREATION 
  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/encryption/CipherProviderType.java PRE-CREATION 
  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/encryption/EncryptionConfiguration.java PRE-CREATION 
  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/encryption/JCEFileKeyProvider.java PRE-CREATION 
  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/encryption/KeyProvider.java PRE-CREATION 
  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/encryption/KeyProviderFactory.java PRE-CREATION 
  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/encryption/KeyProviderType.java PRE-CREATION 
  flume-ng-channels/flume-file-channel/src/main/proto/filechannel.proto PRE-CREATION 
  flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java 35521d1 
  flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java 193cd2b 
  flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecordV3.java PRE-CREATION 
  flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java e64f856 
  flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/encryption/CipherProviderTestSuite.java PRE-CREATION 
  flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/encryption/EncryptionTestUtils.java PRE-CREATION 
  flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/encryption/TestAESCTRNoPaddingProvider.java PRE-CREATION 
  flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/encryption/TestFileChannelEncryption.java PRE-CREATION 
  flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/encryption/TestJCEFileKeyProvider.java PRE-CREATION 
  flume-ng-channels/flume-file-channel/src/test/resources/test.keystore PRE-CREATION 

Diff: https://reviews.apache.org/r/6946/diff/


Testing
-------

Unit tests added and pass.


Thanks,

Brock Noland

Re: Review Request: FLUME-1424: File Channel should support encryption

Posted by Mike Percy <mp...@apache.org>.


> On Sept. 7, 2012, 10 p.m., Mike Percy wrote:
> > This is good stuff Brock. I'd like to go ahead and commit this, just one thing I'd like to change first:
> > 
> > agent.channels.ch-0.encryption.keys = key-0 key-1
> > agent.channels.ch-0.encryption.key-0 = /path/to/key-0.password
> > 
> > is not consistent with most config conventions we have tright now, I think it should be:
> > 
> > agent.channels.ch-0.encryption.keys.key-0.passwordFile = /path/to/key-0.password

I mean:

agent.channels.ch-0.encryption.keys = key-0 key-1
agent.channels.ch-0.encryption.keys.key-0.passwordFile = /path/to/key-0.password


- Mike


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6946/#review11187
-----------------------------------------------------------


On Sept. 7, 2012, 9:34 p.m., Brock Noland wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/6946/
> -----------------------------------------------------------
> 
> (Updated Sept. 7, 2012, 9:34 p.m.)
> 
> 
> Review request for Flume.
> 
> 
> Description
> -------
> 
> Patch which resolves FLUME-1424, it's on top of the latest FLUME-1487 patch. Additionally, it changes the output format slightly so we would not want to make a release with only FLUME-1487 since files written by 1487 are not readable by 1424. We could add yet another version to get around this, but since both patches will be committed to trunk in the same major release, it makes sense to make them a pair.
> 
> The reason for this change was to make our read interface the same as our write interface. So now we read/write a ByteBuffer and then handle it however we need to.
> 
> 
> Below is some sample configuration:
> 
> 
> Generating a key with a password seperate from the key store password:
> 
>     keytool -genseckey -alias key-0 -keypass keyPassword -keyalg AES \
>       -keysize 192 -validity 9000 -keystore test.keystore \
>       -storetype jceks -storepass keyStorePassword
> 
> Generating a key with the password the same as the key store password:      
> 
>     keytool -genseckey -alias key-1 -keyalg AES -keysize 192 -validity 9000 \
>       -keystore src/test/resources/test.keystore -storetype jceks \
>       -storepass keyStorePassword
> 
> Here is a basic configuration with key-0 using the same password as the key store:
> 
> agent.channels.ch-0.encryption.keyProvider = JCEKSFILE
> agent.channels.ch-0.encryption.keyAlias = key-0
> agent.channels.ch-0.encryption.cipherProvider = AESCTRNOPADDING
> agent.channels.ch-0.encryption.keyStoreFile = /path/to/my.keystore
> agent.channels.ch-0.encryption.keyStorePasswordFile = /path/to/my.keystore.password
> agent.channels.ch-0.encryption.keys = key-0
> 
> Let's say you have aged key-0 out and new files should be encrypted with key-1:
> 
> agent.channels.ch-0.encryption.keyProvider = JCEKSFILE
> agent.channels.ch-0.encryption.keyAlias = key-1
> agent.channels.ch-0.encryption.cipherProvider = AESCTRNOPADDING
> agent.channels.ch-0.encryption.keyStoreFile = /path/to/my.keystore
> agent.channels.ch-0.encryption.keyStorePasswordFile = /path/to/my.keystore.password
> agent.channels.ch-0.encryption.keys = key-0 key-1
> 
> The same scenerio as above, however key-0 has it's own password:
> 
> agent.channels.ch-0.encryption.keyProvider = JCEKSFILE
> agent.channels.ch-0.encryption.keyAlias = key-1
> agent.channels.ch-0.encryption.cipherProvider = AESCTRNOPADDING
> agent.channels.ch-0.encryption.keyStoreFile = /path/to/my.keystore
> agent.channels.ch-0.encryption.keyStorePasswordFile = /path/to/my.keystore.password
> agent.channels.ch-0.encryption.key-0 = /path/to/key-0.password
> agent.channels.ch-0.encryption.keys = key-0 key-1
> 
> 
> This addresses bug FLUME-1424.
>     https://issues.apache.org/jira/browse/FLUME-1424
> 
> 
> Diffs
> -----
> 
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/CheckpointRebuilder.java 32b5324 
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java 5d588ea 
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java e13ecc4 
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java 2867fc7 
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFileFactory.java PRE-CREATION 
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFileV2.java PRE-CREATION 
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFileV3.java PRE-CREATION 
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java 4e908ec 
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java 67c68f8 
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/encryption/AESCTRNoPaddingProvider.java PRE-CREATION 
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/encryption/CipherProvider.java PRE-CREATION 
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/encryption/CipherProviderFactory.java PRE-CREATION 
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/encryption/CipherProviderType.java PRE-CREATION 
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/encryption/EncryptionConfiguration.java PRE-CREATION 
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/encryption/JCEFileKeyProvider.java PRE-CREATION 
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/encryption/KeyProvider.java PRE-CREATION 
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/encryption/KeyProviderFactory.java PRE-CREATION 
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/encryption/KeyProviderType.java PRE-CREATION 
>   flume-ng-channels/flume-file-channel/src/main/proto/filechannel.proto PRE-CREATION 
>   flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java 35521d1 
>   flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java 193cd2b 
>   flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecordV3.java PRE-CREATION 
>   flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java e64f856 
>   flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/encryption/CipherProviderTestSuite.java PRE-CREATION 
>   flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/encryption/EncryptionTestUtils.java PRE-CREATION 
>   flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/encryption/TestAESCTRNoPaddingProvider.java PRE-CREATION 
>   flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/encryption/TestFileChannelEncryption.java PRE-CREATION 
>   flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/encryption/TestJCEFileKeyProvider.java PRE-CREATION 
>   flume-ng-channels/flume-file-channel/src/test/resources/test.keystore PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/6946/diff/
> 
> 
> Testing
> -------
> 
> Unit tests added and pass.
> 
> 
> Thanks,
> 
> Brock Noland
> 
>

Re: Review Request: FLUME-1424: File Channel should support encryption

Posted by Mike Percy <mp...@apache.org>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6946/#review11187
-----------------------------------------------------------


This is good stuff Brock. I'd like to go ahead and commit this, just one thing I'd like to change first:

agent.channels.ch-0.encryption.keys = key-0 key-1
agent.channels.ch-0.encryption.key-0 = /path/to/key-0.password

is not consistent with most config conventions we have tright now, I think it should be:

agent.channels.ch-0.encryption.keys.key-0.passwordFile = /path/to/key-0.password

- Mike Percy


On Sept. 7, 2012, 9:34 p.m., Brock Noland wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/6946/
> -----------------------------------------------------------
> 
> (Updated Sept. 7, 2012, 9:34 p.m.)
> 
> 
> Review request for Flume.
> 
> 
> Description
> -------
> 
> Patch which resolves FLUME-1424, it's on top of the latest FLUME-1487 patch. Additionally, it changes the output format slightly so we would not want to make a release with only FLUME-1487 since files written by 1487 are not readable by 1424. We could add yet another version to get around this, but since both patches will be committed to trunk in the same major release, it makes sense to make them a pair.
> 
> The reason for this change was to make our read interface the same as our write interface. So now we read/write a ByteBuffer and then handle it however we need to.
> 
> 
> Below is some sample configuration:
> 
> 
> Generating a key with a password seperate from the key store password:
> 
>     keytool -genseckey -alias key-0 -keypass keyPassword -keyalg AES \
>       -keysize 192 -validity 9000 -keystore test.keystore \
>       -storetype jceks -storepass keyStorePassword
> 
> Generating a key with the password the same as the key store password:      
> 
>     keytool -genseckey -alias key-1 -keyalg AES -keysize 192 -validity 9000 \
>       -keystore src/test/resources/test.keystore -storetype jceks \
>       -storepass keyStorePassword
> 
> Here is a basic configuration with key-0 using the same password as the key store:
> 
> agent.channels.ch-0.encryption.keyProvider = JCEKSFILE
> agent.channels.ch-0.encryption.keyAlias = key-0
> agent.channels.ch-0.encryption.cipherProvider = AESCTRNOPADDING
> agent.channels.ch-0.encryption.keyStoreFile = /path/to/my.keystore
> agent.channels.ch-0.encryption.keyStorePasswordFile = /path/to/my.keystore.password
> agent.channels.ch-0.encryption.keys = key-0
> 
> Let's say you have aged key-0 out and new files should be encrypted with key-1:
> 
> agent.channels.ch-0.encryption.keyProvider = JCEKSFILE
> agent.channels.ch-0.encryption.keyAlias = key-1
> agent.channels.ch-0.encryption.cipherProvider = AESCTRNOPADDING
> agent.channels.ch-0.encryption.keyStoreFile = /path/to/my.keystore
> agent.channels.ch-0.encryption.keyStorePasswordFile = /path/to/my.keystore.password
> agent.channels.ch-0.encryption.keys = key-0 key-1
> 
> The same scenerio as above, however key-0 has it's own password:
> 
> agent.channels.ch-0.encryption.keyProvider = JCEKSFILE
> agent.channels.ch-0.encryption.keyAlias = key-1
> agent.channels.ch-0.encryption.cipherProvider = AESCTRNOPADDING
> agent.channels.ch-0.encryption.keyStoreFile = /path/to/my.keystore
> agent.channels.ch-0.encryption.keyStorePasswordFile = /path/to/my.keystore.password
> agent.channels.ch-0.encryption.key-0 = /path/to/key-0.password
> agent.channels.ch-0.encryption.keys = key-0 key-1
> 
> 
> This addresses bug FLUME-1424.
>     https://issues.apache.org/jira/browse/FLUME-1424
> 
> 
> Diffs
> -----
> 
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/CheckpointRebuilder.java 32b5324 
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java 5d588ea 
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java e13ecc4 
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java 2867fc7 
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFileFactory.java PRE-CREATION 
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFileV2.java PRE-CREATION 
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFileV3.java PRE-CREATION 
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java 4e908ec 
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java 67c68f8 
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/encryption/AESCTRNoPaddingProvider.java PRE-CREATION 
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/encryption/CipherProvider.java PRE-CREATION 
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/encryption/CipherProviderFactory.java PRE-CREATION 
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/encryption/CipherProviderType.java PRE-CREATION 
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/encryption/EncryptionConfiguration.java PRE-CREATION 
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/encryption/JCEFileKeyProvider.java PRE-CREATION 
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/encryption/KeyProvider.java PRE-CREATION 
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/encryption/KeyProviderFactory.java PRE-CREATION 
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/encryption/KeyProviderType.java PRE-CREATION 
>   flume-ng-channels/flume-file-channel/src/main/proto/filechannel.proto PRE-CREATION 
>   flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java 35521d1 
>   flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java 193cd2b 
>   flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecordV3.java PRE-CREATION 
>   flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java e64f856 
>   flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/encryption/CipherProviderTestSuite.java PRE-CREATION 
>   flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/encryption/EncryptionTestUtils.java PRE-CREATION 
>   flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/encryption/TestAESCTRNoPaddingProvider.java PRE-CREATION 
>   flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/encryption/TestFileChannelEncryption.java PRE-CREATION 
>   flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/encryption/TestJCEFileKeyProvider.java PRE-CREATION 
>   flume-ng-channels/flume-file-channel/src/test/resources/test.keystore PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/6946/diff/
> 
> 
> Testing
> -------
> 
> Unit tests added and pass.
> 
> 
> Thanks,
> 
> Brock Noland
> 
>