You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/06/28 05:16:54 UTC
[GitHub] [pulsar-client-go] GPrabhudas opened a new pull request #555: add base package for encryption
GPrabhudas opened a new pull request #555:
URL: https://github.com/apache/pulsar-client-go/pull/555
Breakdown of PR [#552 ](https://github.com/apache/pulsar-client-go/pull/552)
This PR includes base interface and default implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-go] GPrabhudas commented on a change in pull request #555: add base package for encryption
Posted by GitBox <gi...@apache.org>.
GPrabhudas commented on a change in pull request #555:
URL: https://github.com/apache/pulsar-client-go/pull/555#discussion_r662017200
##########
File path: pulsar/crypto/message_crypto.go
##########
@@ -0,0 +1,39 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package crypto
+
+// MessageCrypto implement this interface to encrypt and decrypt messages
+type MessageCrypto interface {
+ // AddPublicKeyCipher
+ AddPublicKeyCipher(keyNames []string, keyReader KeyReader) error
+
+ // RemoveKeyCipher remove the key indentified by the keyname from the list
+ RemoveKeyCipher(keyName string) bool
+
+ /*
Review comment:
Done.
##########
File path: pulsar/crypto/crypto_failure_action.go
##########
@@ -0,0 +1,43 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package crypto
+
+const (
+ // ProducerCryptoFailureActionFail this is the default option to fail send if crypto operation fails.
+ ProducerCryptoFailureActionFail = iota
+
+ // ProducerCryptoFailureActionFailSend ingnore crypto failure and proceed with sending unencrypted message.
+ ProducerCryptoFailureActionFailSend
Review comment:
done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-go] GPrabhudas commented on a change in pull request #555: add base package for encryption
Posted by GitBox <gi...@apache.org>.
GPrabhudas commented on a change in pull request #555:
URL: https://github.com/apache/pulsar-client-go/pull/555#discussion_r660230094
##########
File path: pulsar/crypto/message_metadata.go
##########
@@ -0,0 +1,122 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package crypto
+
+import (
+ pb "github.com/apache/pulsar-client-go/pulsar/internal/pulsar_proto"
+)
+
+// MessageMetadataSupplier wrapper implementation around message metadata
+type MessageMetadataSupplier interface {
+ // GetEncryptionKeys read all the encryption keys from the MessageMetadata
+ GetEncryptionKeys() []EncryptionKeyInfo
+
+ // UpsertEncryptionkey add new or update existing EncryptionKeys in to the MessageMetadata
+ UpsertEncryptionkey(EncryptionKeyInfo)
+
+ // GetEncryptionParam read the ecryption parameter from the MessageMetadata
+ GetEncryptionParam() []byte
+
+ // SetEncryptionParam set encryption parameter in to the MessageMetadata
+ SetEncryptionParam([]byte)
Review comment:
Yes they are needed.
1. Producer will update `MessageMetadata` with encryption parameter and encrypted data key.
2. Consumer will try to read the same info from the `MessageMetadata` to decrypt the payload.
There is need to have this setters here since we are not directly exposing `proto package`.
If client(user) wants to have his own implementation of `MessageCrypto` interface, in such case he won't be able to access `MessageMetadata` directly because it is an `internal` package.
So adding this interface and exposing necessary methods to read `encrypted data key and encryption parameter` from the `MessageMetadata` that are required for decryption of payload.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-go] GPrabhudas commented on a change in pull request #555: add base package for encryption
Posted by GitBox <gi...@apache.org>.
GPrabhudas commented on a change in pull request #555:
URL: https://github.com/apache/pulsar-client-go/pull/555#discussion_r660237713
##########
File path: pulsar/crypto/default_message_crypto.go
##########
@@ -0,0 +1,379 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package crypto
+
+import (
+ gocrypto "crypto"
+ "crypto/aes"
+ "crypto/cipher"
+ "crypto/md5"
+ "crypto/rand"
+ "crypto/rsa"
+ "crypto/sha1"
+ "crypto/x509"
+ "encoding/pem"
+ "errors"
+ "fmt"
+ "sync"
+
+ "github.com/apache/pulsar-client-go/pulsar/log"
+)
+
+// DefaultMessageCrypto implmentation of the interface MessageCryto
+type DefaultMessageCrypto struct {
+ // data key which is used to encrypt/decrypt messages
+ dataKey []byte
+
+ // LoadingCache used by the consumer to cache already decrypted key
+ loadingCache sync.Map // map[string][]byte
+
+ encryptedDataKeyMap sync.Map // map[string]EncryptionKeyInfo
+
+ logCtx string
+
+ logger log.Logger
+
+ cipherLock sync.Mutex
+
+ encryptLock sync.Mutex
+}
+
+const (
+ errGeneratingNonce = "failed to generate new nonce"
Review comment:
done.
##########
File path: pulsar/crypto/default_message_crypto.go
##########
@@ -0,0 +1,379 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package crypto
+
+import (
+ gocrypto "crypto"
+ "crypto/aes"
+ "crypto/cipher"
+ "crypto/md5"
+ "crypto/rand"
+ "crypto/rsa"
+ "crypto/sha1"
+ "crypto/x509"
+ "encoding/pem"
+ "errors"
+ "fmt"
+ "sync"
+
+ "github.com/apache/pulsar-client-go/pulsar/log"
+)
+
+// DefaultMessageCrypto implmentation of the interface MessageCryto
+type DefaultMessageCrypto struct {
+ // data key which is used to encrypt/decrypt messages
+ dataKey []byte
+
+ // LoadingCache used by the consumer to cache already decrypted key
+ loadingCache sync.Map // map[string][]byte
+
+ encryptedDataKeyMap sync.Map // map[string]EncryptionKeyInfo
+
+ logCtx string
+
+ logger log.Logger
+
+ cipherLock sync.Mutex
+
+ encryptLock sync.Mutex
+}
+
+const (
+ errGeneratingNonce = "failed to generate new nonce"
+ errEmptyKeyNameOrKeyReader = "keyname or keyreader is null"
+ errParseRSAPubKey = "unable to parse RSA public key"
+ errParseRSAPriKey = "unable to parse RSA private key"
+ errEncryptionKeyInfo = "Failed to get EncryptionKeyInfo for key %v"
+ errEncryptedDataKeyNotFound = "%v Failed to find encrypted Data key for key %v"
+ errAESCipherCreation = "failed to create AES cipher"
+ errGCMCreation = "failed to create gcm"
+ errDataKeyDecryption = "unable to decrypt data key"
+ errDecodePriKey = "failed to decode private key"
+ errDecodePubKey = "failed to decode public key"
+)
+
+// NewDefaultMessageCrypto get the instance of message crypto
+func NewDefaultMessageCrypto(logCtx string, keyGenNeeded bool, logger log.Logger) (*DefaultMessageCrypto, error) {
+
+ d := &DefaultMessageCrypto{
+ logCtx: logCtx,
+ loadingCache: sync.Map{},
+ encryptedDataKeyMap: sync.Map{},
+ logger: logger,
+ }
+
+ if keyGenNeeded {
+ key, err := generateDataKey()
+ if err != nil {
+ return d, err
+ }
+ d.dataKey = key
+ }
+
+ return d, nil
+}
+
+// AddPublicKeyCipher encrypt data key using keyCrypto and cache
+func (d *DefaultMessageCrypto) AddPublicKeyCipher(keyNames []string, keyReader KeyReader) error {
+ key, err := generateDataKey()
+ if err != nil {
+ return err
+ }
+
+ d.dataKey = key
+ for _, keyName := range keyNames {
+ err := d.addPublicKeyCipher(keyName, keyReader)
+ if err != nil {
+ return err
+ }
+ }
+ return nil
+}
+
+func (d *DefaultMessageCrypto) addPublicKeyCipher(keyName string, keyReader KeyReader) error {
+ d.cipherLock.Lock()
+ defer d.cipherLock.Unlock()
+ if keyName == "" || keyReader == nil {
+ return fmt.Errorf(errEmptyKeyNameOrKeyReader)
+ }
+
+ // read the public key and its info using keyReader
+ keyInfo, err := keyReader.GetPublicKey(keyName, nil)
+ if err != nil {
+ return err
+ }
+
+ parsedKey, err := d.loadPublicKey(keyInfo.GetValue())
+ if err != nil {
+ return err
+ }
+
+ // try to cast to RSA key
+ rsaPubKey, ok := parsedKey.(*rsa.PublicKey)
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-go] GPrabhudas commented on a change in pull request #555: add base package for encryption
Posted by GitBox <gi...@apache.org>.
GPrabhudas commented on a change in pull request #555:
URL: https://github.com/apache/pulsar-client-go/pull/555#discussion_r660231758
##########
File path: pulsar/crypto/message_metadata.go
##########
@@ -0,0 +1,122 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package crypto
+
+import (
+ pb "github.com/apache/pulsar-client-go/pulsar/internal/pulsar_proto"
+)
+
+// MessageMetadataSupplier wrapper implementation around message metadata
+type MessageMetadataSupplier interface {
Review comment:
Goal : Is to provide client(user) to set/read the required information in/from `MessageMetadata`.
Meaningful name => I think it is meaningful since it just providing a way to read some info from internal `MessageMetadata`
`MessageMetadata` => this is from the `proto` which is an internal package and client can access it directly and this class implemented required methods to read/write MessageMetadata.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-go] GPrabhudas commented on a change in pull request #555: add base package for encryption
Posted by GitBox <gi...@apache.org>.
GPrabhudas commented on a change in pull request #555:
URL: https://github.com/apache/pulsar-client-go/pull/555#discussion_r660227097
##########
File path: pulsar/crypto/encryption_key_Info.go
##########
@@ -0,0 +1,64 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package crypto
+
+// EncryptionKeyInfo
+type EncryptionKeyInfo struct {
+ metadata map[string]string
+ value []byte
+ key string
+}
+
+// NewEncryptionKeyInfo
+func NewEncryptionKeyInfo(key string, value []byte, metadata map[string]string) *EncryptionKeyInfo {
+ return &EncryptionKeyInfo{
+ metadata: metadata,
+ key: key,
+ value: value,
+ }
+}
+
+// GetKey get key
+func (eci *EncryptionKeyInfo) GetKey() string {
+ return eci.key
+}
+
+// SetKey set key
+func (eci *EncryptionKeyInfo) SetKey(key string) {
Review comment:
No need of setters. Removed them
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-go] cckellogg commented on a change in pull request #555: add base package for encryption
Posted by GitBox <gi...@apache.org>.
cckellogg commented on a change in pull request #555:
URL: https://github.com/apache/pulsar-client-go/pull/555#discussion_r660206561
##########
File path: pulsar/crypto/encryption_key_Info.go
##########
@@ -0,0 +1,64 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package crypto
+
+// EncryptionKeyInfo
+type EncryptionKeyInfo struct {
+ metadata map[string]string
+ value []byte
+ key string
+}
+
+// NewEncryptionKeyInfo
+func NewEncryptionKeyInfo(key string, value []byte, metadata map[string]string) *EncryptionKeyInfo {
+ return &EncryptionKeyInfo{
+ metadata: metadata,
+ key: key,
+ value: value,
+ }
+}
+
+// GetKey get key
+func (eci *EncryptionKeyInfo) GetKey() string {
+ return eci.key
+}
+
+// SetKey set key
+func (eci *EncryptionKeyInfo) SetKey(key string) {
Review comment:
Why are set methods needed? Should we make this immutable and if changes are needed create a new struct.
##########
File path: pulsar/crypto/default_message_crypto.go
##########
@@ -0,0 +1,379 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package crypto
+
+import (
+ gocrypto "crypto"
+ "crypto/aes"
+ "crypto/cipher"
+ "crypto/md5"
+ "crypto/rand"
+ "crypto/rsa"
+ "crypto/sha1"
+ "crypto/x509"
+ "encoding/pem"
+ "errors"
+ "fmt"
+ "sync"
+
+ "github.com/apache/pulsar-client-go/pulsar/log"
+)
+
+// DefaultMessageCrypto implmentation of the interface MessageCryto
+type DefaultMessageCrypto struct {
+ // data key which is used to encrypt/decrypt messages
+ dataKey []byte
+
+ // LoadingCache used by the consumer to cache already decrypted key
+ loadingCache sync.Map // map[string][]byte
+
+ encryptedDataKeyMap sync.Map // map[string]EncryptionKeyInfo
+
+ logCtx string
+
+ logger log.Logger
+
+ cipherLock sync.Mutex
+
+ encryptLock sync.Mutex
+}
+
+const (
+ errGeneratingNonce = "failed to generate new nonce"
+ errEmptyKeyNameOrKeyReader = "keyname or keyreader is null"
+ errParseRSAPubKey = "unable to parse RSA public key"
+ errParseRSAPriKey = "unable to parse RSA private key"
+ errEncryptionKeyInfo = "Failed to get EncryptionKeyInfo for key %v"
+ errEncryptedDataKeyNotFound = "%v Failed to find encrypted Data key for key %v"
+ errAESCipherCreation = "failed to create AES cipher"
+ errGCMCreation = "failed to create gcm"
+ errDataKeyDecryption = "unable to decrypt data key"
+ errDecodePriKey = "failed to decode private key"
+ errDecodePubKey = "failed to decode public key"
+)
+
+// NewDefaultMessageCrypto get the instance of message crypto
+func NewDefaultMessageCrypto(logCtx string, keyGenNeeded bool, logger log.Logger) (*DefaultMessageCrypto, error) {
+
+ d := &DefaultMessageCrypto{
+ logCtx: logCtx,
+ loadingCache: sync.Map{},
+ encryptedDataKeyMap: sync.Map{},
+ logger: logger,
+ }
+
+ if keyGenNeeded {
+ key, err := generateDataKey()
+ if err != nil {
+ return d, err
+ }
+ d.dataKey = key
+ }
+
+ return d, nil
+}
+
+// AddPublicKeyCipher encrypt data key using keyCrypto and cache
+func (d *DefaultMessageCrypto) AddPublicKeyCipher(keyNames []string, keyReader KeyReader) error {
+ key, err := generateDataKey()
+ if err != nil {
+ return err
+ }
+
+ d.dataKey = key
+ for _, keyName := range keyNames {
+ err := d.addPublicKeyCipher(keyName, keyReader)
+ if err != nil {
+ return err
+ }
+ }
+ return nil
+}
+
+func (d *DefaultMessageCrypto) addPublicKeyCipher(keyName string, keyReader KeyReader) error {
+ d.cipherLock.Lock()
+ defer d.cipherLock.Unlock()
+ if keyName == "" || keyReader == nil {
+ return fmt.Errorf(errEmptyKeyNameOrKeyReader)
+ }
+
+ // read the public key and its info using keyReader
+ keyInfo, err := keyReader.GetPublicKey(keyName, nil)
+ if err != nil {
+ return err
+ }
+
+ parsedKey, err := d.loadPublicKey(keyInfo.GetValue())
+ if err != nil {
+ return err
+ }
+
+ // try to cast to RSA key
+ rsaPubKey, ok := parsedKey.(*rsa.PublicKey)
Review comment:
Why check for RSA here? And if that is only supported should we return an error like expected a RSA key or only RSA keys are supported?
##########
File path: pulsar/crypto/crypto_key_reader.go
##########
@@ -0,0 +1,28 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package crypto
+
+// KeyReader implement this interface to read and provide public & private keys
+// key pair can be RSA, ECDSA
+type KeyReader interface {
+ // GetPublicKey get public key that is be used by the producer to encrypt data key
+ GetPublicKey(keyName string, metadata map[string]string) (*EncryptionKeyInfo, error)
+
+ // GetPrivateKey get private key that is used by the consumer to decrypt data key
+ GetPrivateKey(keyName string, metadata map[string]string) (*EncryptionKeyInfo, error)
Review comment:
Nitpick generally go code style does not use get for values. It would be PublicKey() and PrivateKey() instead. Same for other parts fo the code.
##########
File path: pulsar/crypto/pri_key_rsa.pem
##########
@@ -0,0 +1,39 @@
+-----BEGIN RSA PRIVATE KEY-----
Review comment:
move add these test keys file to a pulsar/crypto/testdata directory
https://golang.org/pkg/cmd/go/internal/test/
```
The go tool will ignore a directory named "testdata", making it available
to hold ancillary data needed by the tests.
```
##########
File path: pulsar/crypto/crypto_key_reader.go
##########
@@ -0,0 +1,28 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package crypto
+
+// KeyReader implement this interface to read and provide public & private keys
+// key pair can be RSA, ECDSA
+type KeyReader interface {
+ // GetPublicKey get public key that is be used by the producer to encrypt data key
+ GetPublicKey(keyName string, metadata map[string]string) (*EncryptionKeyInfo, error)
Review comment:
Where does the key and metadata come from? the message properties?
##########
File path: pulsar/crypto/encryption_key_Info.go
##########
@@ -0,0 +1,64 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package crypto
+
+// EncryptionKeyInfo
+type EncryptionKeyInfo struct {
+ metadata map[string]string
+ value []byte
+ key string
+}
+
+// NewEncryptionKeyInfo
+func NewEncryptionKeyInfo(key string, value []byte, metadata map[string]string) *EncryptionKeyInfo {
+ return &EncryptionKeyInfo{
+ metadata: metadata,
+ key: key,
+ value: value,
+ }
+}
+
+// GetKey get key
+func (eci *EncryptionKeyInfo) GetKey() string {
Review comment:
When i think of key i think of the actual bytes of the key what about just these functions
```
Name() string
Key() []byte
Meta() map[string]string
```
##########
File path: pulsar/crypto/message_metadata.go
##########
@@ -0,0 +1,122 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package crypto
+
+import (
+ pb "github.com/apache/pulsar-client-go/pulsar/internal/pulsar_proto"
+)
+
+// MessageMetadataSupplier wrapper implementation around message metadata
+type MessageMetadataSupplier interface {
+ // GetEncryptionKeys read all the encryption keys from the MessageMetadata
+ GetEncryptionKeys() []EncryptionKeyInfo
+
+ // UpsertEncryptionkey add new or update existing EncryptionKeys in to the MessageMetadata
+ UpsertEncryptionkey(EncryptionKeyInfo)
+
+ // GetEncryptionParam read the ecryption parameter from the MessageMetadata
+ GetEncryptionParam() []byte
+
+ // SetEncryptionParam set encryption parameter in to the MessageMetadata
+ SetEncryptionParam([]byte)
Review comment:
Are setters need in this interface?
##########
File path: pulsar/crypto/crypto_failure_action.go
##########
@@ -0,0 +1,46 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package crypto
+
+type ProducerCryptoFailureAction int
+type ConsumerCryptoFailureAction int
+
+const (
Review comment:
Since there aren't really enums in go to namespace things maybe we should use long names to make it easier to reason about in the code?
ProducerCryptoFailureActionFail or ProducerFailureActionSend
ProducerCryptoFailureActionSend or ProducerFailureActionFail
Thoughts?
##########
File path: pulsar/crypto/default_message_crypto.go
##########
@@ -0,0 +1,379 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package crypto
+
+import (
+ gocrypto "crypto"
+ "crypto/aes"
+ "crypto/cipher"
+ "crypto/md5"
+ "crypto/rand"
+ "crypto/rsa"
+ "crypto/sha1"
+ "crypto/x509"
+ "encoding/pem"
+ "errors"
+ "fmt"
+ "sync"
+
+ "github.com/apache/pulsar-client-go/pulsar/log"
+)
+
+// DefaultMessageCrypto implmentation of the interface MessageCryto
+type DefaultMessageCrypto struct {
+ // data key which is used to encrypt/decrypt messages
+ dataKey []byte
+
+ // LoadingCache used by the consumer to cache already decrypted key
+ loadingCache sync.Map // map[string][]byte
+
+ encryptedDataKeyMap sync.Map // map[string]EncryptionKeyInfo
+
+ logCtx string
+
+ logger log.Logger
+
+ cipherLock sync.Mutex
+
+ encryptLock sync.Mutex
+}
+
+const (
+ errGeneratingNonce = "failed to generate new nonce"
Review comment:
create error vars for strings that have no formatting and no need to create formatting constants if they are only used once. It's easier to get context when looking at the code if the formatting is done where the log/error is happening.
```
var (
errEmptyKeyNameOrKeyReader = fmt.Errorf("keyname or keyreader is not provided"),
...
)
```
##########
File path: pulsar/crypto/message_metadata.go
##########
@@ -0,0 +1,122 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package crypto
+
+import (
+ pb "github.com/apache/pulsar-client-go/pulsar/internal/pulsar_proto"
+)
+
+// MessageMetadataSupplier wrapper implementation around message metadata
+type MessageMetadataSupplier interface {
Review comment:
What is the goal of this class and am wondering if can come up with more meaningful name?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-go] GPrabhudas commented on a change in pull request #555: add base package for encryption
Posted by GitBox <gi...@apache.org>.
GPrabhudas commented on a change in pull request #555:
URL: https://github.com/apache/pulsar-client-go/pull/555#discussion_r660219757
##########
File path: pulsar/crypto/default_message_crypto.go
##########
@@ -0,0 +1,379 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package crypto
+
+import (
+ gocrypto "crypto"
+ "crypto/aes"
+ "crypto/cipher"
+ "crypto/md5"
+ "crypto/rand"
+ "crypto/rsa"
+ "crypto/sha1"
+ "crypto/x509"
+ "encoding/pem"
+ "errors"
+ "fmt"
+ "sync"
+
+ "github.com/apache/pulsar-client-go/pulsar/log"
+)
+
+// DefaultMessageCrypto implmentation of the interface MessageCryto
+type DefaultMessageCrypto struct {
+ // data key which is used to encrypt/decrypt messages
+ dataKey []byte
+
+ // LoadingCache used by the consumer to cache already decrypted key
+ loadingCache sync.Map // map[string][]byte
+
+ encryptedDataKeyMap sync.Map // map[string]EncryptionKeyInfo
+
+ logCtx string
+
+ logger log.Logger
+
+ cipherLock sync.Mutex
+
+ encryptLock sync.Mutex
+}
+
+const (
+ errGeneratingNonce = "failed to generate new nonce"
+ errEmptyKeyNameOrKeyReader = "keyname or keyreader is null"
+ errParseRSAPubKey = "unable to parse RSA public key"
+ errParseRSAPriKey = "unable to parse RSA private key"
+ errEncryptionKeyInfo = "Failed to get EncryptionKeyInfo for key %v"
+ errEncryptedDataKeyNotFound = "%v Failed to find encrypted Data key for key %v"
+ errAESCipherCreation = "failed to create AES cipher"
+ errGCMCreation = "failed to create gcm"
+ errDataKeyDecryption = "unable to decrypt data key"
+ errDecodePriKey = "failed to decode private key"
+ errDecodePubKey = "failed to decode public key"
+)
+
+// NewDefaultMessageCrypto get the instance of message crypto
+func NewDefaultMessageCrypto(logCtx string, keyGenNeeded bool, logger log.Logger) (*DefaultMessageCrypto, error) {
+
+ d := &DefaultMessageCrypto{
+ logCtx: logCtx,
+ loadingCache: sync.Map{},
+ encryptedDataKeyMap: sync.Map{},
+ logger: logger,
+ }
+
+ if keyGenNeeded {
+ key, err := generateDataKey()
+ if err != nil {
+ return d, err
+ }
+ d.dataKey = key
+ }
+
+ return d, nil
+}
+
+// AddPublicKeyCipher encrypt data key using keyCrypto and cache
+func (d *DefaultMessageCrypto) AddPublicKeyCipher(keyNames []string, keyReader KeyReader) error {
+ key, err := generateDataKey()
+ if err != nil {
+ return err
+ }
+
+ d.dataKey = key
+ for _, keyName := range keyNames {
+ err := d.addPublicKeyCipher(keyName, keyReader)
+ if err != nil {
+ return err
+ }
+ }
+ return nil
+}
+
+func (d *DefaultMessageCrypto) addPublicKeyCipher(keyName string, keyReader KeyReader) error {
+ d.cipherLock.Lock()
+ defer d.cipherLock.Unlock()
+ if keyName == "" || keyReader == nil {
+ return fmt.Errorf(errEmptyKeyNameOrKeyReader)
+ }
+
+ // read the public key and its info using keyReader
+ keyInfo, err := keyReader.GetPublicKey(keyName, nil)
+ if err != nil {
+ return err
+ }
+
+ parsedKey, err := d.loadPublicKey(keyInfo.GetValue())
+ if err != nil {
+ return err
+ }
+
+ // try to cast to RSA key
+ rsaPubKey, ok := parsedKey.(*rsa.PublicKey)
Review comment:
Yes currently only supporting RSA Keys and will update it accordingly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-go] GPrabhudas commented on a change in pull request #555: add base package for encryption
Posted by GitBox <gi...@apache.org>.
GPrabhudas commented on a change in pull request #555:
URL: https://github.com/apache/pulsar-client-go/pull/555#discussion_r660231758
##########
File path: pulsar/crypto/message_metadata.go
##########
@@ -0,0 +1,122 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package crypto
+
+import (
+ pb "github.com/apache/pulsar-client-go/pulsar/internal/pulsar_proto"
+)
+
+// MessageMetadataSupplier wrapper implementation around message metadata
+type MessageMetadataSupplier interface {
Review comment:
Goal : Is to provide client(user) to set/read the required information in/from `MessageMetadata`.
Meaningful name => I think it is meaningful since it just providing a way to read some info from internal `MessageMetadata`
`MessageMetadata` => this is from the `proto` which is an internal package and client can not access it directly and this class implemented required methods to read/write MessageMetadata.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-go] GPrabhudas commented on a change in pull request #555: add base package for encryption
Posted by GitBox <gi...@apache.org>.
GPrabhudas commented on a change in pull request #555:
URL: https://github.com/apache/pulsar-client-go/pull/555#discussion_r660234887
##########
File path: pulsar/crypto/crypto_failure_action.go
##########
@@ -0,0 +1,46 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package crypto
+
+type ProducerCryptoFailureAction int
+type ConsumerCryptoFailureAction int
+
+const (
Review comment:
And it is good have long name like `ProducerCryptoFailureActionFail` instead of `ProducerFailureActionSend`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-go] cckellogg commented on a change in pull request #555: add base package for encryption
Posted by GitBox <gi...@apache.org>.
cckellogg commented on a change in pull request #555:
URL: https://github.com/apache/pulsar-client-go/pull/555#discussion_r660206561
##########
File path: pulsar/crypto/encryption_key_Info.go
##########
@@ -0,0 +1,64 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package crypto
+
+// EncryptionKeyInfo
+type EncryptionKeyInfo struct {
+ metadata map[string]string
+ value []byte
+ key string
+}
+
+// NewEncryptionKeyInfo
+func NewEncryptionKeyInfo(key string, value []byte, metadata map[string]string) *EncryptionKeyInfo {
+ return &EncryptionKeyInfo{
+ metadata: metadata,
+ key: key,
+ value: value,
+ }
+}
+
+// GetKey get key
+func (eci *EncryptionKeyInfo) GetKey() string {
+ return eci.key
+}
+
+// SetKey set key
+func (eci *EncryptionKeyInfo) SetKey(key string) {
Review comment:
Why are set methods needed? Should we make this immutable and if changes are needed create a new struct.
##########
File path: pulsar/crypto/default_message_crypto.go
##########
@@ -0,0 +1,379 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package crypto
+
+import (
+ gocrypto "crypto"
+ "crypto/aes"
+ "crypto/cipher"
+ "crypto/md5"
+ "crypto/rand"
+ "crypto/rsa"
+ "crypto/sha1"
+ "crypto/x509"
+ "encoding/pem"
+ "errors"
+ "fmt"
+ "sync"
+
+ "github.com/apache/pulsar-client-go/pulsar/log"
+)
+
+// DefaultMessageCrypto implmentation of the interface MessageCryto
+type DefaultMessageCrypto struct {
+ // data key which is used to encrypt/decrypt messages
+ dataKey []byte
+
+ // LoadingCache used by the consumer to cache already decrypted key
+ loadingCache sync.Map // map[string][]byte
+
+ encryptedDataKeyMap sync.Map // map[string]EncryptionKeyInfo
+
+ logCtx string
+
+ logger log.Logger
+
+ cipherLock sync.Mutex
+
+ encryptLock sync.Mutex
+}
+
+const (
+ errGeneratingNonce = "failed to generate new nonce"
+ errEmptyKeyNameOrKeyReader = "keyname or keyreader is null"
+ errParseRSAPubKey = "unable to parse RSA public key"
+ errParseRSAPriKey = "unable to parse RSA private key"
+ errEncryptionKeyInfo = "Failed to get EncryptionKeyInfo for key %v"
+ errEncryptedDataKeyNotFound = "%v Failed to find encrypted Data key for key %v"
+ errAESCipherCreation = "failed to create AES cipher"
+ errGCMCreation = "failed to create gcm"
+ errDataKeyDecryption = "unable to decrypt data key"
+ errDecodePriKey = "failed to decode private key"
+ errDecodePubKey = "failed to decode public key"
+)
+
+// NewDefaultMessageCrypto get the instance of message crypto
+func NewDefaultMessageCrypto(logCtx string, keyGenNeeded bool, logger log.Logger) (*DefaultMessageCrypto, error) {
+
+ d := &DefaultMessageCrypto{
+ logCtx: logCtx,
+ loadingCache: sync.Map{},
+ encryptedDataKeyMap: sync.Map{},
+ logger: logger,
+ }
+
+ if keyGenNeeded {
+ key, err := generateDataKey()
+ if err != nil {
+ return d, err
+ }
+ d.dataKey = key
+ }
+
+ return d, nil
+}
+
+// AddPublicKeyCipher encrypt data key using keyCrypto and cache
+func (d *DefaultMessageCrypto) AddPublicKeyCipher(keyNames []string, keyReader KeyReader) error {
+ key, err := generateDataKey()
+ if err != nil {
+ return err
+ }
+
+ d.dataKey = key
+ for _, keyName := range keyNames {
+ err := d.addPublicKeyCipher(keyName, keyReader)
+ if err != nil {
+ return err
+ }
+ }
+ return nil
+}
+
+func (d *DefaultMessageCrypto) addPublicKeyCipher(keyName string, keyReader KeyReader) error {
+ d.cipherLock.Lock()
+ defer d.cipherLock.Unlock()
+ if keyName == "" || keyReader == nil {
+ return fmt.Errorf(errEmptyKeyNameOrKeyReader)
+ }
+
+ // read the public key and its info using keyReader
+ keyInfo, err := keyReader.GetPublicKey(keyName, nil)
+ if err != nil {
+ return err
+ }
+
+ parsedKey, err := d.loadPublicKey(keyInfo.GetValue())
+ if err != nil {
+ return err
+ }
+
+ // try to cast to RSA key
+ rsaPubKey, ok := parsedKey.(*rsa.PublicKey)
Review comment:
Why check for RSA here? And if that is only supported should we return an error like expected a RSA key or only RSA keys are supported?
##########
File path: pulsar/crypto/crypto_key_reader.go
##########
@@ -0,0 +1,28 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package crypto
+
+// KeyReader implement this interface to read and provide public & private keys
+// key pair can be RSA, ECDSA
+type KeyReader interface {
+ // GetPublicKey get public key that is be used by the producer to encrypt data key
+ GetPublicKey(keyName string, metadata map[string]string) (*EncryptionKeyInfo, error)
+
+ // GetPrivateKey get private key that is used by the consumer to decrypt data key
+ GetPrivateKey(keyName string, metadata map[string]string) (*EncryptionKeyInfo, error)
Review comment:
Nitpick generally go code style does not use get for values. It would be PublicKey() and PrivateKey() instead. Same for other parts fo the code.
##########
File path: pulsar/crypto/pri_key_rsa.pem
##########
@@ -0,0 +1,39 @@
+-----BEGIN RSA PRIVATE KEY-----
Review comment:
move add these test keys file to a pulsar/crypto/testdata directory
https://golang.org/pkg/cmd/go/internal/test/
```
The go tool will ignore a directory named "testdata", making it available
to hold ancillary data needed by the tests.
```
##########
File path: pulsar/crypto/crypto_key_reader.go
##########
@@ -0,0 +1,28 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package crypto
+
+// KeyReader implement this interface to read and provide public & private keys
+// key pair can be RSA, ECDSA
+type KeyReader interface {
+ // GetPublicKey get public key that is be used by the producer to encrypt data key
+ GetPublicKey(keyName string, metadata map[string]string) (*EncryptionKeyInfo, error)
Review comment:
Where does the key and metadata come from? the message properties?
##########
File path: pulsar/crypto/encryption_key_Info.go
##########
@@ -0,0 +1,64 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package crypto
+
+// EncryptionKeyInfo
+type EncryptionKeyInfo struct {
+ metadata map[string]string
+ value []byte
+ key string
+}
+
+// NewEncryptionKeyInfo
+func NewEncryptionKeyInfo(key string, value []byte, metadata map[string]string) *EncryptionKeyInfo {
+ return &EncryptionKeyInfo{
+ metadata: metadata,
+ key: key,
+ value: value,
+ }
+}
+
+// GetKey get key
+func (eci *EncryptionKeyInfo) GetKey() string {
Review comment:
When i think of key i think of the actual bytes of the key what about just these functions
```
Name() string
Key() []byte
Meta() map[string]string
```
##########
File path: pulsar/crypto/message_metadata.go
##########
@@ -0,0 +1,122 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package crypto
+
+import (
+ pb "github.com/apache/pulsar-client-go/pulsar/internal/pulsar_proto"
+)
+
+// MessageMetadataSupplier wrapper implementation around message metadata
+type MessageMetadataSupplier interface {
+ // GetEncryptionKeys read all the encryption keys from the MessageMetadata
+ GetEncryptionKeys() []EncryptionKeyInfo
+
+ // UpsertEncryptionkey add new or update existing EncryptionKeys in to the MessageMetadata
+ UpsertEncryptionkey(EncryptionKeyInfo)
+
+ // GetEncryptionParam read the ecryption parameter from the MessageMetadata
+ GetEncryptionParam() []byte
+
+ // SetEncryptionParam set encryption parameter in to the MessageMetadata
+ SetEncryptionParam([]byte)
Review comment:
Are setters need in this interface?
##########
File path: pulsar/crypto/crypto_failure_action.go
##########
@@ -0,0 +1,46 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package crypto
+
+type ProducerCryptoFailureAction int
+type ConsumerCryptoFailureAction int
+
+const (
Review comment:
Since there aren't really enums in go to namespace things maybe we should use long names to make it easier to reason about in the code?
ProducerCryptoFailureActionFail or ProducerFailureActionSend
ProducerCryptoFailureActionSend or ProducerFailureActionFail
Thoughts?
##########
File path: pulsar/crypto/default_message_crypto.go
##########
@@ -0,0 +1,379 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package crypto
+
+import (
+ gocrypto "crypto"
+ "crypto/aes"
+ "crypto/cipher"
+ "crypto/md5"
+ "crypto/rand"
+ "crypto/rsa"
+ "crypto/sha1"
+ "crypto/x509"
+ "encoding/pem"
+ "errors"
+ "fmt"
+ "sync"
+
+ "github.com/apache/pulsar-client-go/pulsar/log"
+)
+
+// DefaultMessageCrypto implmentation of the interface MessageCryto
+type DefaultMessageCrypto struct {
+ // data key which is used to encrypt/decrypt messages
+ dataKey []byte
+
+ // LoadingCache used by the consumer to cache already decrypted key
+ loadingCache sync.Map // map[string][]byte
+
+ encryptedDataKeyMap sync.Map // map[string]EncryptionKeyInfo
+
+ logCtx string
+
+ logger log.Logger
+
+ cipherLock sync.Mutex
+
+ encryptLock sync.Mutex
+}
+
+const (
+ errGeneratingNonce = "failed to generate new nonce"
Review comment:
create error vars for strings that have no formatting and no need to create formatting constants if they are only used once. It's easier to get context when looking at the code if the formatting is done where the log/error is happening.
```
var (
errEmptyKeyNameOrKeyReader = fmt.Errorf("keyname or keyreader is not provided"),
...
)
```
##########
File path: pulsar/crypto/message_metadata.go
##########
@@ -0,0 +1,122 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package crypto
+
+import (
+ pb "github.com/apache/pulsar-client-go/pulsar/internal/pulsar_proto"
+)
+
+// MessageMetadataSupplier wrapper implementation around message metadata
+type MessageMetadataSupplier interface {
Review comment:
What is the goal of this class and am wondering if can come up with more meaningful name?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-go] GPrabhudas commented on a change in pull request #555: add base package for encryption
Posted by GitBox <gi...@apache.org>.
GPrabhudas commented on a change in pull request #555:
URL: https://github.com/apache/pulsar-client-go/pull/555#discussion_r660232863
##########
File path: pulsar/crypto/crypto_key_reader.go
##########
@@ -0,0 +1,28 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package crypto
+
+// KeyReader implement this interface to read and provide public & private keys
+// key pair can be RSA, ECDSA
+type KeyReader interface {
+ // GetPublicKey get public key that is be used by the producer to encrypt data key
+ GetPublicKey(keyName string, metadata map[string]string) (*EncryptionKeyInfo, error)
+
+ // GetPrivateKey get private key that is used by the consumer to decrypt data key
+ GetPrivateKey(keyName string, metadata map[string]string) (*EncryptionKeyInfo, error)
Review comment:
Got it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-go] GPrabhudas commented on a change in pull request #555: add base package for encryption
Posted by GitBox <gi...@apache.org>.
GPrabhudas commented on a change in pull request #555:
URL: https://github.com/apache/pulsar-client-go/pull/555#discussion_r660219757
##########
File path: pulsar/crypto/default_message_crypto.go
##########
@@ -0,0 +1,379 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package crypto
+
+import (
+ gocrypto "crypto"
+ "crypto/aes"
+ "crypto/cipher"
+ "crypto/md5"
+ "crypto/rand"
+ "crypto/rsa"
+ "crypto/sha1"
+ "crypto/x509"
+ "encoding/pem"
+ "errors"
+ "fmt"
+ "sync"
+
+ "github.com/apache/pulsar-client-go/pulsar/log"
+)
+
+// DefaultMessageCrypto implmentation of the interface MessageCryto
+type DefaultMessageCrypto struct {
+ // data key which is used to encrypt/decrypt messages
+ dataKey []byte
+
+ // LoadingCache used by the consumer to cache already decrypted key
+ loadingCache sync.Map // map[string][]byte
+
+ encryptedDataKeyMap sync.Map // map[string]EncryptionKeyInfo
+
+ logCtx string
+
+ logger log.Logger
+
+ cipherLock sync.Mutex
+
+ encryptLock sync.Mutex
+}
+
+const (
+ errGeneratingNonce = "failed to generate new nonce"
+ errEmptyKeyNameOrKeyReader = "keyname or keyreader is null"
+ errParseRSAPubKey = "unable to parse RSA public key"
+ errParseRSAPriKey = "unable to parse RSA private key"
+ errEncryptionKeyInfo = "Failed to get EncryptionKeyInfo for key %v"
+ errEncryptedDataKeyNotFound = "%v Failed to find encrypted Data key for key %v"
+ errAESCipherCreation = "failed to create AES cipher"
+ errGCMCreation = "failed to create gcm"
+ errDataKeyDecryption = "unable to decrypt data key"
+ errDecodePriKey = "failed to decode private key"
+ errDecodePubKey = "failed to decode public key"
+)
+
+// NewDefaultMessageCrypto get the instance of message crypto
+func NewDefaultMessageCrypto(logCtx string, keyGenNeeded bool, logger log.Logger) (*DefaultMessageCrypto, error) {
+
+ d := &DefaultMessageCrypto{
+ logCtx: logCtx,
+ loadingCache: sync.Map{},
+ encryptedDataKeyMap: sync.Map{},
+ logger: logger,
+ }
+
+ if keyGenNeeded {
+ key, err := generateDataKey()
+ if err != nil {
+ return d, err
+ }
+ d.dataKey = key
+ }
+
+ return d, nil
+}
+
+// AddPublicKeyCipher encrypt data key using keyCrypto and cache
+func (d *DefaultMessageCrypto) AddPublicKeyCipher(keyNames []string, keyReader KeyReader) error {
+ key, err := generateDataKey()
+ if err != nil {
+ return err
+ }
+
+ d.dataKey = key
+ for _, keyName := range keyNames {
+ err := d.addPublicKeyCipher(keyName, keyReader)
+ if err != nil {
+ return err
+ }
+ }
+ return nil
+}
+
+func (d *DefaultMessageCrypto) addPublicKeyCipher(keyName string, keyReader KeyReader) error {
+ d.cipherLock.Lock()
+ defer d.cipherLock.Unlock()
+ if keyName == "" || keyReader == nil {
+ return fmt.Errorf(errEmptyKeyNameOrKeyReader)
+ }
+
+ // read the public key and its info using keyReader
+ keyInfo, err := keyReader.GetPublicKey(keyName, nil)
+ if err != nil {
+ return err
+ }
+
+ parsedKey, err := d.loadPublicKey(keyInfo.GetValue())
+ if err != nil {
+ return err
+ }
+
+ // try to cast to RSA key
+ rsaPubKey, ok := parsedKey.(*rsa.PublicKey)
Review comment:
Yes currently only supporting RSA Keys and will update it accordingly
##########
File path: pulsar/crypto/crypto_key_reader.go
##########
@@ -0,0 +1,28 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package crypto
+
+// KeyReader implement this interface to read and provide public & private keys
+// key pair can be RSA, ECDSA
+type KeyReader interface {
+ // GetPublicKey get public key that is be used by the producer to encrypt data key
+ GetPublicKey(keyName string, metadata map[string]string) (*EncryptionKeyInfo, error)
Review comment:
Key name is something user configures while creating producer in `ProducerOptions`
##########
File path: pulsar/crypto/encryption_key_Info.go
##########
@@ -0,0 +1,64 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package crypto
+
+// EncryptionKeyInfo
+type EncryptionKeyInfo struct {
+ metadata map[string]string
+ value []byte
+ key string
+}
+
+// NewEncryptionKeyInfo
+func NewEncryptionKeyInfo(key string, value []byte, metadata map[string]string) *EncryptionKeyInfo {
+ return &EncryptionKeyInfo{
+ metadata: metadata,
+ key: key,
+ value: value,
+ }
+}
+
+// GetKey get key
+func (eci *EncryptionKeyInfo) GetKey() string {
+ return eci.key
+}
+
+// SetKey set key
+func (eci *EncryptionKeyInfo) SetKey(key string) {
Review comment:
No need of setters. Removed them
##########
File path: pulsar/crypto/encryption_key_Info.go
##########
@@ -0,0 +1,64 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package crypto
+
+// EncryptionKeyInfo
+type EncryptionKeyInfo struct {
+ metadata map[string]string
+ value []byte
+ key string
+}
+
+// NewEncryptionKeyInfo
+func NewEncryptionKeyInfo(key string, value []byte, metadata map[string]string) *EncryptionKeyInfo {
+ return &EncryptionKeyInfo{
+ metadata: metadata,
+ key: key,
+ value: value,
+ }
+}
+
+// GetKey get key
+func (eci *EncryptionKeyInfo) GetKey() string {
Review comment:
done
##########
File path: pulsar/crypto/message_metadata.go
##########
@@ -0,0 +1,122 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package crypto
+
+import (
+ pb "github.com/apache/pulsar-client-go/pulsar/internal/pulsar_proto"
+)
+
+// MessageMetadataSupplier wrapper implementation around message metadata
+type MessageMetadataSupplier interface {
+ // GetEncryptionKeys read all the encryption keys from the MessageMetadata
+ GetEncryptionKeys() []EncryptionKeyInfo
+
+ // UpsertEncryptionkey add new or update existing EncryptionKeys in to the MessageMetadata
+ UpsertEncryptionkey(EncryptionKeyInfo)
+
+ // GetEncryptionParam read the ecryption parameter from the MessageMetadata
+ GetEncryptionParam() []byte
+
+ // SetEncryptionParam set encryption parameter in to the MessageMetadata
+ SetEncryptionParam([]byte)
Review comment:
Yes they are needed.
1. Producer will update `MessageMetadata` with encryption parameter and encrypted data key.
2. Consumer will try to read the same info from the `MessageMetadata` to decrypt the payload.
There is need to have this setters here since we are not directly exposing `proto package`.
If client(user) wants to have his own implementation of `MessageCrypto` interface, in such case he won't be able to access `MessageMetadata` directly because it is an `internal` package.
So adding this interface and exposing necessary methods to read `encrypted data key and encryption parameter` from the `MessageMetadata` that are required for decryption of payload.
##########
File path: pulsar/crypto/message_metadata.go
##########
@@ -0,0 +1,122 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package crypto
+
+import (
+ pb "github.com/apache/pulsar-client-go/pulsar/internal/pulsar_proto"
+)
+
+// MessageMetadataSupplier wrapper implementation around message metadata
+type MessageMetadataSupplier interface {
Review comment:
Goal : Is to provide client(user) to set/read the required information in/from `MessageMetadata`.
Meaningful name => I think it is meaningful since it just providing a way to read some info from internal `MessageMetadata`
`MessageMetadata` => this is from the `proto` which is an internal package and client can access it directly and this class implemented required methods to read/write MessageMetadata.
##########
File path: pulsar/crypto/pri_key_rsa.pem
##########
@@ -0,0 +1,39 @@
+-----BEGIN RSA PRIVATE KEY-----
Review comment:
done.
##########
File path: pulsar/crypto/crypto_key_reader.go
##########
@@ -0,0 +1,28 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package crypto
+
+// KeyReader implement this interface to read and provide public & private keys
+// key pair can be RSA, ECDSA
+type KeyReader interface {
+ // GetPublicKey get public key that is be used by the producer to encrypt data key
+ GetPublicKey(keyName string, metadata map[string]string) (*EncryptionKeyInfo, error)
+
+ // GetPrivateKey get private key that is used by the consumer to decrypt data key
+ GetPrivateKey(keyName string, metadata map[string]string) (*EncryptionKeyInfo, error)
Review comment:
Got it.
##########
File path: pulsar/crypto/crypto_failure_action.go
##########
@@ -0,0 +1,46 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package crypto
+
+type ProducerCryptoFailureAction int
+type ConsumerCryptoFailureAction int
+
+const (
Review comment:
I tried to use them as enum, but yeah I think it make sense to have a long names instead.
##########
File path: pulsar/crypto/crypto_failure_action.go
##########
@@ -0,0 +1,46 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package crypto
+
+type ProducerCryptoFailureAction int
+type ConsumerCryptoFailureAction int
+
+const (
Review comment:
And it is good have long name like `ProducerCryptoFailureActionFail` instead of `ProducerFailureActionSend`
##########
File path: pulsar/crypto/message_metadata.go
##########
@@ -0,0 +1,122 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package crypto
+
+import (
+ pb "github.com/apache/pulsar-client-go/pulsar/internal/pulsar_proto"
+)
+
+// MessageMetadataSupplier wrapper implementation around message metadata
+type MessageMetadataSupplier interface {
Review comment:
Goal : Is to provide client(user) to set/read the required information in/from `MessageMetadata`.
Meaningful name => I think it is meaningful since it just providing a way to read some info from internal `MessageMetadata`
`MessageMetadata` => this is from the `proto` which is an internal package and client can not access it directly and this class implemented required methods to read/write MessageMetadata.
##########
File path: pulsar/crypto/default_message_crypto.go
##########
@@ -0,0 +1,379 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package crypto
+
+import (
+ gocrypto "crypto"
+ "crypto/aes"
+ "crypto/cipher"
+ "crypto/md5"
+ "crypto/rand"
+ "crypto/rsa"
+ "crypto/sha1"
+ "crypto/x509"
+ "encoding/pem"
+ "errors"
+ "fmt"
+ "sync"
+
+ "github.com/apache/pulsar-client-go/pulsar/log"
+)
+
+// DefaultMessageCrypto implmentation of the interface MessageCryto
+type DefaultMessageCrypto struct {
+ // data key which is used to encrypt/decrypt messages
+ dataKey []byte
+
+ // LoadingCache used by the consumer to cache already decrypted key
+ loadingCache sync.Map // map[string][]byte
+
+ encryptedDataKeyMap sync.Map // map[string]EncryptionKeyInfo
+
+ logCtx string
+
+ logger log.Logger
+
+ cipherLock sync.Mutex
+
+ encryptLock sync.Mutex
+}
+
+const (
+ errGeneratingNonce = "failed to generate new nonce"
Review comment:
done.
##########
File path: pulsar/crypto/default_message_crypto.go
##########
@@ -0,0 +1,379 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package crypto
+
+import (
+ gocrypto "crypto"
+ "crypto/aes"
+ "crypto/cipher"
+ "crypto/md5"
+ "crypto/rand"
+ "crypto/rsa"
+ "crypto/sha1"
+ "crypto/x509"
+ "encoding/pem"
+ "errors"
+ "fmt"
+ "sync"
+
+ "github.com/apache/pulsar-client-go/pulsar/log"
+)
+
+// DefaultMessageCrypto implmentation of the interface MessageCryto
+type DefaultMessageCrypto struct {
+ // data key which is used to encrypt/decrypt messages
+ dataKey []byte
+
+ // LoadingCache used by the consumer to cache already decrypted key
+ loadingCache sync.Map // map[string][]byte
+
+ encryptedDataKeyMap sync.Map // map[string]EncryptionKeyInfo
+
+ logCtx string
+
+ logger log.Logger
+
+ cipherLock sync.Mutex
+
+ encryptLock sync.Mutex
+}
+
+const (
+ errGeneratingNonce = "failed to generate new nonce"
+ errEmptyKeyNameOrKeyReader = "keyname or keyreader is null"
+ errParseRSAPubKey = "unable to parse RSA public key"
+ errParseRSAPriKey = "unable to parse RSA private key"
+ errEncryptionKeyInfo = "Failed to get EncryptionKeyInfo for key %v"
+ errEncryptedDataKeyNotFound = "%v Failed to find encrypted Data key for key %v"
+ errAESCipherCreation = "failed to create AES cipher"
+ errGCMCreation = "failed to create gcm"
+ errDataKeyDecryption = "unable to decrypt data key"
+ errDecodePriKey = "failed to decode private key"
+ errDecodePubKey = "failed to decode public key"
+)
+
+// NewDefaultMessageCrypto get the instance of message crypto
+func NewDefaultMessageCrypto(logCtx string, keyGenNeeded bool, logger log.Logger) (*DefaultMessageCrypto, error) {
+
+ d := &DefaultMessageCrypto{
+ logCtx: logCtx,
+ loadingCache: sync.Map{},
+ encryptedDataKeyMap: sync.Map{},
+ logger: logger,
+ }
+
+ if keyGenNeeded {
+ key, err := generateDataKey()
+ if err != nil {
+ return d, err
+ }
+ d.dataKey = key
+ }
+
+ return d, nil
+}
+
+// AddPublicKeyCipher encrypt data key using keyCrypto and cache
+func (d *DefaultMessageCrypto) AddPublicKeyCipher(keyNames []string, keyReader KeyReader) error {
+ key, err := generateDataKey()
+ if err != nil {
+ return err
+ }
+
+ d.dataKey = key
+ for _, keyName := range keyNames {
+ err := d.addPublicKeyCipher(keyName, keyReader)
+ if err != nil {
+ return err
+ }
+ }
+ return nil
+}
+
+func (d *DefaultMessageCrypto) addPublicKeyCipher(keyName string, keyReader KeyReader) error {
+ d.cipherLock.Lock()
+ defer d.cipherLock.Unlock()
+ if keyName == "" || keyReader == nil {
+ return fmt.Errorf(errEmptyKeyNameOrKeyReader)
+ }
+
+ // read the public key and its info using keyReader
+ keyInfo, err := keyReader.GetPublicKey(keyName, nil)
+ if err != nil {
+ return err
+ }
+
+ parsedKey, err := d.loadPublicKey(keyInfo.GetValue())
+ if err != nil {
+ return err
+ }
+
+ // try to cast to RSA key
+ rsaPubKey, ok := parsedKey.(*rsa.PublicKey)
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-go] GPrabhudas commented on a change in pull request #555: add base package for encryption
Posted by GitBox <gi...@apache.org>.
GPrabhudas commented on a change in pull request #555:
URL: https://github.com/apache/pulsar-client-go/pull/555#discussion_r660234181
##########
File path: pulsar/crypto/crypto_failure_action.go
##########
@@ -0,0 +1,46 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package crypto
+
+type ProducerCryptoFailureAction int
+type ConsumerCryptoFailureAction int
+
+const (
Review comment:
I tried to use them as enum, but yeah I think it make sense to have a long names instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-go] GPrabhudas commented on pull request #555: add base package for encryption
Posted by GitBox <gi...@apache.org>.
GPrabhudas commented on pull request #555:
URL: https://github.com/apache/pulsar-client-go/pull/555#issuecomment-871965689
> Ok makes sense. For my understanding what's the difference between UpsertEncryptionkey and SetEncryptionParam and when would one or the other be used?
[EncryptionParam](https://github.com/apache/pulsar-client-go/blob/master/pulsar/internal/pulsar_proto/PulsarApi.pb.go#L1361) => It is a `nonce or padding` used while encrypting and we need to pass same `nonce` while decrypting payload. So `SetEncryptionParam` sets the `nonce` in `MessageMetadata`.
[EncryptionKeys](https://github.com/apache/pulsar-client-go/blob/master/pulsar/internal/pulsar_proto/PulsarApi.pb.go#L1357) => It is the Info about data key used to encrypt the payload . `UpsertEncryptionkey` sets/updates the encrypted `data key` and it's metadata if any in the `MessageMetadata`.
Producer uses `UpsertEncryptionkey` and `SetEncryptionParam` to update MessageMetadata after encrypting payload.
Consumer uses `EncryptionKeys` and `EncryptionParam` to read from MessageMetadata for decrypting payload.
@cckellogg
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-go] cckellogg commented on a change in pull request #555: add base package for encryption
Posted by GitBox <gi...@apache.org>.
cckellogg commented on a change in pull request #555:
URL: https://github.com/apache/pulsar-client-go/pull/555#discussion_r661799669
##########
File path: pulsar/crypto/message_crypto.go
##########
@@ -0,0 +1,39 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package crypto
+
+// MessageCrypto implement this interface to encrypt and decrypt messages
+type MessageCrypto interface {
+ // AddPublicKeyCipher
+ AddPublicKeyCipher(keyNames []string, keyReader KeyReader) error
+
+ // RemoveKeyCipher remove the key indentified by the keyname from the list
+ RemoveKeyCipher(keyName string) bool
+
+ /*
Review comment:
Nitpick use `//` for each line in multiline comments
##########
File path: pulsar/crypto/message_metadata.go
##########
@@ -0,0 +1,122 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package crypto
+
+import (
+ pb "github.com/apache/pulsar-client-go/pulsar/internal/pulsar_proto"
+)
+
+// MessageMetadataSupplier wrapper implementation around message metadata
+type MessageMetadataSupplier interface {
+ // GetEncryptionKeys read all the encryption keys from the MessageMetadata
+ GetEncryptionKeys() []EncryptionKeyInfo
+
+ // UpsertEncryptionkey add new or update existing EncryptionKeys in to the MessageMetadata
+ UpsertEncryptionkey(EncryptionKeyInfo)
+
+ // GetEncryptionParam read the ecryption parameter from the MessageMetadata
+ GetEncryptionParam() []byte
+
+ // SetEncryptionParam set encryption parameter in to the MessageMetadata
+ SetEncryptionParam([]byte)
Review comment:
Ok makes sense. For my understanding what's the difference between `UpsertEncryptionkey` and `SetEncryptionParam` and when would one or the other be used?
##########
File path: pulsar/crypto/crypto_failure_action.go
##########
@@ -0,0 +1,43 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package crypto
+
+const (
+ // ProducerCryptoFailureActionFail this is the default option to fail send if crypto operation fails.
+ ProducerCryptoFailureActionFail = iota
+
+ // ProducerCryptoFailureActionFailSend ingnore crypto failure and proceed with sending unencrypted message.
+ ProducerCryptoFailureActionFailSend
Review comment:
nit `ProducerCryptoFailureActionSend`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-go] wolfstudy merged pull request #555: add base package for encryption
Posted by GitBox <gi...@apache.org>.
wolfstudy merged pull request #555:
URL: https://github.com/apache/pulsar-client-go/pull/555
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-go] GPrabhudas commented on a change in pull request #555: add base package for encryption
Posted by GitBox <gi...@apache.org>.
GPrabhudas commented on a change in pull request #555:
URL: https://github.com/apache/pulsar-client-go/pull/555#discussion_r660227129
##########
File path: pulsar/crypto/encryption_key_Info.go
##########
@@ -0,0 +1,64 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package crypto
+
+// EncryptionKeyInfo
+type EncryptionKeyInfo struct {
+ metadata map[string]string
+ value []byte
+ key string
+}
+
+// NewEncryptionKeyInfo
+func NewEncryptionKeyInfo(key string, value []byte, metadata map[string]string) *EncryptionKeyInfo {
+ return &EncryptionKeyInfo{
+ metadata: metadata,
+ key: key,
+ value: value,
+ }
+}
+
+// GetKey get key
+func (eci *EncryptionKeyInfo) GetKey() string {
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-go] GPrabhudas commented on a change in pull request #555: add base package for encryption
Posted by GitBox <gi...@apache.org>.
GPrabhudas commented on a change in pull request #555:
URL: https://github.com/apache/pulsar-client-go/pull/555#discussion_r660232478
##########
File path: pulsar/crypto/pri_key_rsa.pem
##########
@@ -0,0 +1,39 @@
+-----BEGIN RSA PRIVATE KEY-----
Review comment:
done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-go] GPrabhudas commented on a change in pull request #555: add base package for encryption
Posted by GitBox <gi...@apache.org>.
GPrabhudas commented on a change in pull request #555:
URL: https://github.com/apache/pulsar-client-go/pull/555#discussion_r660220502
##########
File path: pulsar/crypto/crypto_key_reader.go
##########
@@ -0,0 +1,28 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package crypto
+
+// KeyReader implement this interface to read and provide public & private keys
+// key pair can be RSA, ECDSA
+type KeyReader interface {
+ // GetPublicKey get public key that is be used by the producer to encrypt data key
+ GetPublicKey(keyName string, metadata map[string]string) (*EncryptionKeyInfo, error)
Review comment:
Key name is something user configures while creating producer in `ProducerOptions`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org