You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by jjmeyer0 <gi...@git.apache.org> on 2017/07/10 14:44:15 UTC
[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.
GitHub user jjmeyer0 opened a pull request:
https://github.com/apache/metron/pull/641
METRON-539: added HASH function for stellar.
## Contributor Comments
Added a hash function to stellar. Many unit tests were created. Along with these tests I opened the Stellar terminal and did the following:
### Start Stellar
```bash
mvn exec:java -Dexec.mainClass="org.apache.metron.stellar.common.shell.StellarShell" -pl metron-stellar/stellar-common/
```
### Test `HASH` Function
Below is the output of some of the tests I ran. To get all available inputs for the second argument you can call `Security.getAlgorithms("MessageDigest")` and print out the array. However, 'SHA-1', 'MD5', and 'SHA-256' are required by all Java platforms. Please see [MessageDigest](https://docs.oracle.com/javase/8/docs/api/java/security/MessageDigest.html) and [Security](https://docs.oracle.com/javase/8/docs/api/java/security/Security.html).
```bash
[Stellar]>>> Functions loaded, you may refer to functions now...
[Stellar]>>> some_string := 'my string to hash'
[Stellar]>>> HASH(some_string, 'md5')
E08863A4BE232F166C350F45C9DAA586
[Stellar]>>> some_serializable_object := [ 1, 2, 3 ]
[Stellar]>>> HASH(some_serializable_object, 'MD5')
2383CD011F1DA06A173ADE2F3F99CF62
[Stellar]>>>
```
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions.
Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
```
mvn -q clean integration-test install && build_utils/verify_licenses.sh
```
- [x] Have you written or updated unit tests and or integration tests to verify your changes?
- [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`:
```
cd site-book
mvn site
```
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jjmeyer0/incubator-metron METRON-539
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/641.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #641
----
commit 8de6390825e3a17de226ace4764eb6a9d4cec609
Author: JJ <jj...@gmail.com>
Date: 2017-07-10T12:34:02Z
METRON-539: added HASH function for stellar.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/641#discussion_r126810799
--- Diff: metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/HashFunctions.java ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang3.SerializationUtils;
+import org.apache.metron.stellar.dsl.BaseStellarFunction;
+import org.apache.metron.stellar.dsl.Stellar;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.List;
+
+public class HashFunctions {
+ @Stellar(
+ name = "HASH",
+ description = "Hashes a given value using the given hashing algorithm and returns a hex encoded string. This function only hashes " +
+ "strings and values that implement java.io.Serializable.",
+ params = {
+ "toHash - value to hash.",
+ "hashType - A valid string representation of an algorithm supported by java.security.MessageDigest.",
+ },
+ returns = "A hex representation of a hashed value using the given hashing algorithm. If either argument is " +
+ "null then null will be returned. If the type of 'toHash' is neither a string nor of type java.io.Serializable, " +
+ "then null is returned."
+ )
+ public static class Hash extends BaseStellarFunction {
+
+ @Override
+ public Object apply(final List<Object> args) {
+ if (args == null || args.size() != 2) {
+ throw new IllegalArgumentException("Invalid number of arguments: " + (args == null ? 0 : args.size()));
+ }
+
+ final Object toHash = args.get(0);
+ final Object hashType = args.get(1);
+
+ if (toHash == null || hashType == null) {
+ return null;
+ }
+
+ final MessageDigest messageDigest = getMessageDigest(hashType.toString());
+
+ if (toHash instanceof String) {
+ return getHashAsHex(messageDigest, toHash.toString().getBytes(StandardCharsets.UTF_8));
+ } else if (toHash instanceof Serializable) {
+ final byte[] serialized = SerializationUtils.serialize((Serializable) toHash);
+ return getHashAsHex(messageDigest, serialized);
+ }
+
+ return null;
+ }
+
+ private MessageDigest getMessageDigest(String hashType) {
+ try {
+ return MessageDigest.getInstance(hashType);
+ } catch (NoSuchAlgorithmException e) {
+ throw new IllegalArgumentException("Invalid hash type: " + hashType);
+ }
+ }
+
--- End diff --
Why is this Object and not String? Can't we just be clear and use the real type?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.
Posted by mattf-horton <gi...@git.apache.org>.
Github user mattf-horton commented on a diff in the pull request:
https://github.com/apache/metron/pull/641#discussion_r126828810
--- Diff: metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/HashFunctions.java ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang3.SerializationUtils;
+import org.apache.metron.stellar.dsl.BaseStellarFunction;
+import org.apache.metron.stellar.dsl.Stellar;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.List;
+
+public class HashFunctions {
+ @Stellar(
+ name = "HASH",
--- End diff --
Still, if it's not serializable, what should HASH do? This may be moot, since all normal objects Stellar can get its hands on are, I think, Serializable. The exceptions may indeed be appropriate to not be able to hash.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron issue #641: METRON-539: added HASH function for stellar.
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/641
I just want to chime in and say thanks for the contribution @jjmeyer0 !. :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.
Posted by mattf-horton <gi...@git.apache.org>.
Github user mattf-horton commented on a diff in the pull request:
https://github.com/apache/metron/pull/641#discussion_r126481330
--- Diff: metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/HashFunctionsTest.java ---
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang.SerializationUtils;
+import org.junit.Test;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.security.Security;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.metron.stellar.common.utils.StellarProcessorUtils.run;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNull;
+
+public class HashFunctionsTest {
+ final HashFunctions.Hash hash = new HashFunctions.Hash();
+
+ @Test(expected = IllegalArgumentException.class)
+ public void nullArgumentListShouldThrowException() throws Exception {
+ hash.apply(null);
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void emptyArgumentListShouldThrowException() throws Exception {
+ hash.apply(Collections.emptyList());
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void singleArgumentListShouldThrowException() throws Exception {
+ hash.apply(Collections.singletonList("some value."));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void argumentListWithMoreThanTwoValuesShouldThrowException3() throws Exception {
+ hash.apply(Arrays.asList("1", "2", "3"));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void argumentListWithMoreThanTwoValuesShouldThrowException4() throws Exception {
+ hash.apply(Arrays.asList("1", "2", "3", "4"));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void invalidAlgorithmArgumentShouldThrowException() throws Exception {
+ hash.apply(Arrays.asList("value to hash", "invalidAlgorithm"));
+ }
+
+ @Test
+ public void invalidNullAlgorithmArgumentShouldThrowException() throws Exception {
+ assertNull(hash.apply(Arrays.asList("value to hash", null)));
+ }
+
+ @Test
+ public void nullInputForValueToHashShouldProperlyThrowException() throws Exception {
+ assertNull(hash.apply(Arrays.asList(null, "md5")));
+ }
+
+ @Test
+ public void allAlgorithmsForMessageDigestShouldBeAbleToHash() throws Exception {
+ final String valueToHash = "My value to hash";
+ final Set<String> algorithms = Security.getAlgorithms("MessageDigest");
+
+ algorithms.forEach(algorithm -> {
+ try {
+ final MessageDigest m = MessageDigest.getInstance(algorithm);
+ m.update(valueToHash.getBytes(StandardCharsets.UTF_8));
+
+ assertEquals(BaseEncoding.base16().encode(m.digest()), hash.apply(Arrays.asList(valueToHash, algorithm)));
+ } catch (NoSuchAlgorithmException e) {
+ throw new RuntimeException(e);
+ }
+ });
+ }
+
+ @Test
+ public void allAlgorithmsForMessageDigestShouldBeAbleToHashDirectStellarCall() throws Exception {
+ final String valueToHash = "My value to hash";
+ final Set<String> algorithms = Security.getAlgorithms("MessageDigest");
+
+ algorithms.forEach(algorithm -> {
+ try {
+ final Object actual = run("HASH('" + valueToHash + "', '" + algorithm + "')", Collections.emptyMap());
+
+ final MessageDigest m = MessageDigest.getInstance(algorithm);
+ m.update(valueToHash.getBytes(StandardCharsets.UTF_8));
+
+ assertEquals(BaseEncoding.base16().encode(m.digest()), actual);
+ } catch (NoSuchAlgorithmException e) {
+ throw new RuntimeException(e);
+ }
+ });
+ }
+
+ @Test
+ public void nonStringValueThatIsSerializableShouldBeSerializedAndThenEncodedUsingHex() throws Exception {
--- End diff --
This is another nit, but I'll share the thought anyway: A more typical (and shorter) name would be "testNonStringValueThatIsSerializable". Being a "test", it is understood that the normative result is to properly do whatever it should do without error. Similarly, the next two items could more briefly be named "testHashFunctionsWithVariablesAsInput" and "testStellarFunctionWhereOnlyHashTypeIsAVariable".
The last one, "aNonNullObjectThatDoesNotImplementSerializableShouldReturnAValueOfNull" is for a non-normative case, so it provides useful information as is. It could still be shorter though. How about "testNonSerializableObjectReturnsNull"?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron issue #641: METRON-539: added HASH function for stellar.
Posted by JonZeolla <gi...@git.apache.org>.
Github user JonZeolla commented on the issue:
https://github.com/apache/metron/pull/641
Yeah, this is exactly what I had in mind when I put the ticket in, thanks @jjmeyer0 .
I would love to get some LSH and CCTP hashes added, but I think that's more reasonable as a follow on.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron issue #641: METRON-539: added HASH function for stellar.
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/641
Ok, I'm +1 on this by inspection as soon as we have the conflict resolved. Great job, @jjmeyer0
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.
Posted by mattf-horton <gi...@git.apache.org>.
Github user mattf-horton commented on a diff in the pull request:
https://github.com/apache/metron/pull/641#discussion_r126475943
--- Diff: metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/HashFunctionsTest.java ---
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang.SerializationUtils;
+import org.junit.Test;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.security.Security;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.metron.stellar.common.utils.StellarProcessorUtils.run;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNull;
+
+public class HashFunctionsTest {
+ final HashFunctions.Hash hash = new HashFunctions.Hash();
+
+ @Test(expected = IllegalArgumentException.class)
+ public void nullArgumentListShouldThrowException() throws Exception {
+ hash.apply(null);
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void emptyArgumentListShouldThrowException() throws Exception {
+ hash.apply(Collections.emptyList());
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void singleArgumentListShouldThrowException() throws Exception {
+ hash.apply(Collections.singletonList("some value."));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void argumentListWithMoreThanTwoValuesShouldThrowException3() throws Exception {
+ hash.apply(Arrays.asList("1", "2", "3"));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void argumentListWithMoreThanTwoValuesShouldThrowException4() throws Exception {
+ hash.apply(Arrays.asList("1", "2", "3", "4"));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void invalidAlgorithmArgumentShouldThrowException() throws Exception {
+ hash.apply(Arrays.asList("value to hash", "invalidAlgorithm"));
+ }
+
+ @Test
+ public void invalidNullAlgorithmArgumentShouldThrowException() throws Exception {
+ assertNull(hash.apply(Arrays.asList("value to hash", null)));
+ }
+
+ @Test
+ public void nullInputForValueToHashShouldProperlyThrowException() throws Exception {
+ assertNull(hash.apply(Arrays.asList(null, "md5")));
+ }
+
+ @Test
+ public void allAlgorithmsForMessageDigestShouldBeAbleToHash() throws Exception {
+ final String valueToHash = "My value to hash";
+ final Set<String> algorithms = Security.getAlgorithms("MessageDigest");
+
+ algorithms.forEach(algorithm -> {
+ try {
+ final MessageDigest m = MessageDigest.getInstance(algorithm);
+ m.update(valueToHash.getBytes(StandardCharsets.UTF_8));
+
+ assertEquals(BaseEncoding.base16().encode(m.digest()), hash.apply(Arrays.asList(valueToHash, algorithm)));
+ } catch (NoSuchAlgorithmException e) {
+ throw new RuntimeException(e);
--- End diff --
Unless there's some junit subtlety that isn't occurring to me right now, it would be cleaner to declare this testcase as "throws NoSuchAlgorithmException", and then there's no need for the try/catch context.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.
Posted by mattf-horton <gi...@git.apache.org>.
Github user mattf-horton commented on a diff in the pull request:
https://github.com/apache/metron/pull/641#discussion_r126473615
--- Diff: metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/HashFunctionsTest.java ---
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang.SerializationUtils;
+import org.junit.Test;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.security.Security;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.metron.stellar.common.utils.StellarProcessorUtils.run;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNull;
+
+public class HashFunctionsTest {
+ final HashFunctions.Hash hash = new HashFunctions.Hash();
+
+ @Test(expected = IllegalArgumentException.class)
+ public void nullArgumentListShouldThrowException() throws Exception {
+ hash.apply(null);
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void emptyArgumentListShouldThrowException() throws Exception {
+ hash.apply(Collections.emptyList());
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void singleArgumentListShouldThrowException() throws Exception {
+ hash.apply(Collections.singletonList("some value."));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void argumentListWithMoreThanTwoValuesShouldThrowException3() throws Exception {
+ hash.apply(Arrays.asList("1", "2", "3"));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void argumentListWithMoreThanTwoValuesShouldThrowException4() throws Exception {
+ hash.apply(Arrays.asList("1", "2", "3", "4"));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void invalidAlgorithmArgumentShouldThrowException() throws Exception {
+ hash.apply(Arrays.asList("value to hash", "invalidAlgorithm"));
+ }
+
+ @Test
+ public void invalidNullAlgorithmArgumentShouldThrowException() throws Exception {
+ assertNull(hash.apply(Arrays.asList("value to hash", null)));
+ }
--- End diff --
Need "@Test(expected = IllegalArgumentException.class)" on this one too, don't we?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/641#discussion_r126810010
--- Diff: metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/HashFunctions.java ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang3.SerializationUtils;
+import org.apache.metron.stellar.dsl.BaseStellarFunction;
+import org.apache.metron.stellar.dsl.Stellar;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.List;
+
+public class HashFunctions {
+ @Stellar(
+ name = "HASH",
--- End diff --
I am not sure that java.io.Serializable should bleed through here. While we may limit ourselves in some places to having stellar conform to java limitations and behaviors, and other things, we don't really limit ourselves explicitly to java types. The idea that the user 'knows' the type of the entity in a stellar statement in java.... doesn't seem right.
While this limitation may be true and enforced, I am not sure it should be expressed this way explicitly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/641#discussion_r126831146
--- Diff: metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/HashFunctions.java ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang3.SerializationUtils;
+import org.apache.metron.stellar.dsl.BaseStellarFunction;
+import org.apache.metron.stellar.dsl.Stellar;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.List;
+
+public class HashFunctions {
+ @Stellar(
+ name = "HASH",
+ description = "Hashes a given value using the given hashing algorithm and returns a hex encoded string. This function only hashes " +
+ "strings and values that implement java.io.Serializable.",
+ params = {
+ "toHash - value to hash.",
+ "hashType - A valid string representation of an algorithm supported by java.security.MessageDigest.",
+ },
+ returns = "A hex representation of a hashed value using the given hashing algorithm. If either argument is " +
+ "null then null will be returned. If the type of 'toHash' is neither a string nor of type java.io.Serializable, " +
+ "then null is returned."
+ )
+ public static class Hash extends BaseStellarFunction {
+
+ @Override
+ public Object apply(final List<Object> args) {
+ if (args == null || args.size() != 2) {
+ throw new IllegalArgumentException("Invalid number of arguments: " + (args == null ? 0 : args.size()));
+ }
+
+ final Object toHash = args.get(0);
+ final Object hashType = args.get(1);
+
+ if (toHash == null || hashType == null) {
+ return null;
+ }
+
+ final MessageDigest messageDigest = getMessageDigest(hashType.toString());
+
+ if (toHash instanceof String) {
+ return getHashAsHex(messageDigest, toHash.toString().getBytes(StandardCharsets.UTF_8));
+ } else if (toHash instanceof Serializable) {
+ final byte[] serialized = SerializationUtils.serialize((Serializable) toHash);
+ return getHashAsHex(messageDigest, serialized);
+ }
+
+ return null;
+ }
+
+ private MessageDigest getMessageDigest(String hashType) {
--- End diff --
I am not sure I understand :( right over my head.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/641#discussion_r126577935
--- Diff: metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/HashFunctions.java ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang3.SerializationUtils;
+import org.apache.metron.stellar.dsl.BaseStellarFunction;
+import org.apache.metron.stellar.dsl.Stellar;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.List;
+
+public class HashFunctions {
+ @Stellar(
+ name = "HASH",
+ description = "Hashes a given value using the given hashing algorithm and returns a hex encoded string. This function only hashes " +
+ "strings and values that implement java.io.Serializable.",
+ params = {
+ "toHash - value to hash.",
+ "hashType - A valid string representation of an algorithm supported by java.security.MessageDigest.",
+ },
+ returns = "A hex representation of a hashed value using the given hashing algorithm. If either argument is " +
+ "null then null will be returned. If the type of 'toHash' is neither a string nor of type java.io.Serializable, " +
+ "then null is returned."
+ )
+ public static class Hash extends BaseStellarFunction {
--- End diff --
I have no preference. I picked the string thinking of the return of the call being for 'information' and display only. Like from the shell. I think that we should do the 'get' no params calls the same though, i'd change to list if that is what makes sense
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/641#discussion_r126812613
--- Diff: metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/HashFunctions.java ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang3.SerializationUtils;
+import org.apache.metron.stellar.dsl.BaseStellarFunction;
+import org.apache.metron.stellar.dsl.Stellar;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.List;
+
+public class HashFunctions {
+ @Stellar(
+ name = "HASH",
+ description = "Hashes a given value using the given hashing algorithm and returns a hex encoded string. This function only hashes " +
+ "strings and values that implement java.io.Serializable.",
+ params = {
+ "toHash - value to hash.",
+ "hashType - A valid string representation of an algorithm supported by java.security.MessageDigest.",
+ },
+ returns = "A hex representation of a hashed value using the given hashing algorithm. If either argument is " +
+ "null then null will be returned. If the type of 'toHash' is neither a string nor of type java.io.Serializable, " +
+ "then null is returned."
+ )
+ public static class Hash extends BaseStellarFunction {
+
+ @Override
+ public Object apply(final List<Object> args) {
+ if (args == null || args.size() != 2) {
+ throw new IllegalArgumentException("Invalid number of arguments: " + (args == null ? 0 : args.size()));
+ }
+
+ final Object toHash = args.get(0);
+ final Object hashType = args.get(1);
+
+ if (toHash == null || hashType == null) {
+ return null;
+ }
+
+ final MessageDigest messageDigest = getMessageDigest(hashType.toString());
+
+ if (toHash instanceof String) {
+ return getHashAsHex(messageDigest, toHash.toString().getBytes(StandardCharsets.UTF_8));
+ } else if (toHash instanceof Serializable) {
+ final byte[] serialized = SerializationUtils.serialize((Serializable) toHash);
+ return getHashAsHex(messageDigest, serialized);
+ }
+
+ return null;
+ }
+
+ private MessageDigest getMessageDigest(String hashType) {
--- End diff --
Then again, I don't usually think of using the Stellar static classes on their own, or from 'other' stellar functions, maybe I'm not thinking of that correctly
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/641#discussion_r126836440
--- Diff: metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/HashFunctions.java ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang3.SerializationUtils;
+import org.apache.metron.stellar.dsl.BaseStellarFunction;
+import org.apache.metron.stellar.dsl.Stellar;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.List;
+
+public class HashFunctions {
+ @Stellar(
+ name = "HASH",
+ description = "Hashes a given value using the given hashing algorithm and returns a hex encoded string. This function only hashes " +
+ "strings and values that implement java.io.Serializable.",
+ params = {
+ "toHash - value to hash.",
+ "hashType - A valid string representation of an algorithm supported by java.security.MessageDigest.",
+ },
+ returns = "A hex representation of a hashed value using the given hashing algorithm. If either argument is " +
+ "null then null will be returned. If the type of 'toHash' is neither a string nor of type java.io.Serializable, " +
+ "then null is returned."
+ )
+ public static class Hash extends BaseStellarFunction {
+
+ @Override
+ public Object apply(final List<Object> args) {
+ if (args == null || args.size() != 2) {
+ throw new IllegalArgumentException("Invalid number of arguments: " + (args == null ? 0 : args.size()));
+ }
+
+ final Object toHash = args.get(0);
+ final Object hashType = args.get(1);
+
+ if (toHash == null || hashType == null) {
+ return null;
+ }
+
+ final MessageDigest messageDigest = getMessageDigest(hashType.toString());
+
+ if (toHash instanceof String) {
+ return getHashAsHex(messageDigest, toHash.toString().getBytes(StandardCharsets.UTF_8));
+ } else if (toHash instanceof Serializable) {
+ final byte[] serialized = SerializationUtils.serialize((Serializable) toHash);
+ return getHashAsHex(messageDigest, serialized);
+ }
+
+ return null;
+ }
+
+ private MessageDigest getMessageDigest(String hashType) {
--- End diff --
I have this [Encodings.java](https://github.com/apache/metron/pull/642/files#diff-21626fcded1721297dd93114ccfd5ba1) and [EncodingFunctions.java](https://github.com/apache/metron/pull/642/files#diff-295147ddc21d558ddd7af7f31d9bb8ca) fresh in my mind.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.
Posted by mattf-horton <gi...@git.apache.org>.
Github user mattf-horton commented on a diff in the pull request:
https://github.com/apache/metron/pull/641#discussion_r126829075
--- Diff: metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/HashFunctions.java ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang3.SerializationUtils;
+import org.apache.metron.stellar.dsl.BaseStellarFunction;
+import org.apache.metron.stellar.dsl.Stellar;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.List;
+
+public class HashFunctions {
+ @Stellar(
+ name = "HASH",
+ description = "Hashes a given value using the given hashing algorithm and returns a hex encoded string. This function only hashes " +
+ "strings and values that implement java.io.Serializable.",
+ params = {
+ "toHash - value to hash.",
+ "hashType - A valid string representation of an algorithm supported by java.security.MessageDigest.",
+ },
+ returns = "A hex representation of a hashed value using the given hashing algorithm. If either argument is " +
+ "null then null will be returned. If the type of 'toHash' is neither a string nor of type java.io.Serializable, " +
+ "then null is returned."
+ )
+ public static class Hash extends BaseStellarFunction {
+
+ @Override
+ public Object apply(final List<Object> args) {
+ if (args == null || args.size() != 2) {
+ throw new IllegalArgumentException("Invalid number of arguments: " + (args == null ? 0 : args.size()));
+ }
+
+ final Object toHash = args.get(0);
+ final Object hashType = args.get(1);
+
+ if (toHash == null || hashType == null) {
+ return null;
+ }
+
+ final MessageDigest messageDigest = getMessageDigest(hashType.toString());
+
+ if (toHash instanceof String) {
+ return getHashAsHex(messageDigest, toHash.toString().getBytes(StandardCharsets.UTF_8));
+ } else if (toHash instanceof Serializable) {
+ final byte[] serialized = SerializationUtils.serialize((Serializable) toHash);
+ return getHashAsHex(messageDigest, serialized);
+ }
+
+ return null;
+ }
+
+ private MessageDigest getMessageDigest(String hashType) {
--- End diff --
If we expose MessageDigest objects, that would be a good example of non-Serializable objects it is reasonable to be unable to hash :-)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.
Posted by jjmeyer0 <gi...@git.apache.org>.
Github user jjmeyer0 commented on a diff in the pull request:
https://github.com/apache/metron/pull/641#discussion_r126926090
--- Diff: metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/HashFunctionsTest.java ---
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang.SerializationUtils;
+import org.junit.Test;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.security.Security;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.metron.stellar.common.utils.StellarProcessorUtils.run;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNull;
+
+public class HashFunctionsTest {
+ final HashFunctions.Hash hash = new HashFunctions.Hash();
+
+ @Test(expected = IllegalArgumentException.class)
+ public void nullArgumentListShouldThrowException() throws Exception {
+ hash.apply(null);
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void emptyArgumentListShouldThrowException() throws Exception {
+ hash.apply(Collections.emptyList());
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void singleArgumentListShouldThrowException() throws Exception {
+ hash.apply(Collections.singletonList("some value."));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void argumentListWithMoreThanTwoValuesShouldThrowException3() throws Exception {
+ hash.apply(Arrays.asList("1", "2", "3"));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void argumentListWithMoreThanTwoValuesShouldThrowException4() throws Exception {
+ hash.apply(Arrays.asList("1", "2", "3", "4"));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void invalidAlgorithmArgumentShouldThrowException() throws Exception {
+ hash.apply(Arrays.asList("value to hash", "invalidAlgorithm"));
+ }
+
+ @Test
+ public void invalidNullAlgorithmArgumentShouldThrowException() throws Exception {
+ assertNull(hash.apply(Arrays.asList("value to hash", null)));
+ }
+
+ @Test
+ public void nullInputForValueToHashShouldProperlyThrowException() throws Exception {
+ assertNull(hash.apply(Arrays.asList(null, "md5")));
+ }
--- End diff --
oh boy, @mattf-horton that makes much more sense. forgive me i had a long day yesterday. I had a moment haha
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/641#discussion_r126831664
--- Diff: metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/HashFunctions.java ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang3.SerializationUtils;
+import org.apache.metron.stellar.dsl.BaseStellarFunction;
+import org.apache.metron.stellar.dsl.Stellar;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.List;
+
+public class HashFunctions {
+ @Stellar(
+ name = "HASH",
--- End diff --
My point isn't that the behavior or restriction in practice should change, but that the parameter documentation should be different.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.
Posted by mattf-horton <gi...@git.apache.org>.
Github user mattf-horton commented on a diff in the pull request:
https://github.com/apache/metron/pull/641#discussion_r126835535
--- Diff: metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/HashFunctions.java ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang3.SerializationUtils;
+import org.apache.metron.stellar.dsl.BaseStellarFunction;
+import org.apache.metron.stellar.dsl.Stellar;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.List;
+
+public class HashFunctions {
+ @Stellar(
+ name = "HASH",
+ description = "Hashes a given value using the given hashing algorithm and returns a hex encoded string. This function only hashes " +
+ "strings and values that implement java.io.Serializable.",
+ params = {
+ "toHash - value to hash.",
+ "hashType - A valid string representation of an algorithm supported by java.security.MessageDigest.",
+ },
+ returns = "A hex representation of a hashed value using the given hashing algorithm. If either argument is " +
+ "null then null will be returned. If the type of 'toHash' is neither a string nor of type java.io.Serializable, " +
+ "then null is returned."
+ )
+ public static class Hash extends BaseStellarFunction {
+
+ @Override
+ public Object apply(final List<Object> args) {
+ if (args == null || args.size() != 2) {
+ throw new IllegalArgumentException("Invalid number of arguments: " + (args == null ? 0 : args.size()));
+ }
+
+ final Object toHash = args.get(0);
+ final Object hashType = args.get(1);
+
+ if (toHash == null || hashType == null) {
+ return null;
+ }
+
+ final MessageDigest messageDigest = getMessageDigest(hashType.toString());
+
+ if (toHash instanceof String) {
+ return getHashAsHex(messageDigest, toHash.toString().getBytes(StandardCharsets.UTF_8));
+ } else if (toHash instanceof Serializable) {
+ final byte[] serialized = SerializationUtils.serialize((Serializable) toHash);
+ return getHashAsHex(messageDigest, serialized);
+ }
+
+ return null;
+ }
+
+ private MessageDigest getMessageDigest(String hashType) {
--- End diff --
Never mind, I think I misunderstood you again. You were probably just proposing the code be in a separate directory instead of in the generic stellar/dsl/functions, right? If so then I'm in left field here. Prob didn't get enough sleep last night...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.
Posted by mattf-horton <gi...@git.apache.org>.
Github user mattf-horton commented on a diff in the pull request:
https://github.com/apache/metron/pull/641#discussion_r126849833
--- Diff: metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/HashFunctionsTest.java ---
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang.SerializationUtils;
+import org.junit.Test;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.security.Security;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.metron.stellar.common.utils.StellarProcessorUtils.run;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNull;
+
+public class HashFunctionsTest {
+ final HashFunctions.Hash hash = new HashFunctions.Hash();
+
+ @Test(expected = IllegalArgumentException.class)
+ public void nullArgumentListShouldThrowException() throws Exception {
+ hash.apply(null);
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void emptyArgumentListShouldThrowException() throws Exception {
+ hash.apply(Collections.emptyList());
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void singleArgumentListShouldThrowException() throws Exception {
+ hash.apply(Collections.singletonList("some value."));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void argumentListWithMoreThanTwoValuesShouldThrowException3() throws Exception {
+ hash.apply(Arrays.asList("1", "2", "3"));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void argumentListWithMoreThanTwoValuesShouldThrowException4() throws Exception {
+ hash.apply(Arrays.asList("1", "2", "3", "4"));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void invalidAlgorithmArgumentShouldThrowException() throws Exception {
+ hash.apply(Arrays.asList("value to hash", "invalidAlgorithm"));
+ }
+
+ @Test
+ public void invalidNullAlgorithmArgumentShouldThrowException() throws Exception {
+ assertNull(hash.apply(Arrays.asList("value to hash", null)));
+ }
+
+ @Test
+ public void nullInputForValueToHashShouldProperlyThrowException() throws Exception {
+ assertNull(hash.apply(Arrays.asList(null, "md5")));
+ }
--- End diff --
Um, but if the hash values are always supposed to be of a certain length, then pad that out to as many zero digits as necessary to be conforming.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.
Posted by jjmeyer0 <gi...@git.apache.org>.
Github user jjmeyer0 commented on a diff in the pull request:
https://github.com/apache/metron/pull/641#discussion_r126841963
--- Diff: metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/HashFunctionsTest.java ---
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang.SerializationUtils;
+import org.junit.Test;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.security.Security;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.metron.stellar.common.utils.StellarProcessorUtils.run;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNull;
+
+public class HashFunctionsTest {
+ final HashFunctions.Hash hash = new HashFunctions.Hash();
+
+ @Test(expected = IllegalArgumentException.class)
+ public void nullArgumentListShouldThrowException() throws Exception {
+ hash.apply(null);
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void emptyArgumentListShouldThrowException() throws Exception {
+ hash.apply(Collections.emptyList());
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void singleArgumentListShouldThrowException() throws Exception {
+ hash.apply(Collections.singletonList("some value."));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void argumentListWithMoreThanTwoValuesShouldThrowException3() throws Exception {
+ hash.apply(Arrays.asList("1", "2", "3"));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void argumentListWithMoreThanTwoValuesShouldThrowException4() throws Exception {
+ hash.apply(Arrays.asList("1", "2", "3", "4"));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void invalidAlgorithmArgumentShouldThrowException() throws Exception {
+ hash.apply(Arrays.asList("value to hash", "invalidAlgorithm"));
+ }
+
+ @Test
+ public void invalidNullAlgorithmArgumentShouldThrowException() throws Exception {
+ assertNull(hash.apply(Arrays.asList("value to hash", null)));
+ }
+
+ @Test
+ public void nullInputForValueToHashShouldProperlyThrowException() throws Exception {
+ assertNull(hash.apply(Arrays.asList(null, "md5")));
+ }
--- End diff --
@mattf-horton just to clarify. The tests `invalidNullAlgorithmArgumentShouldThrowException` and `nullInputForValueToHashShouldProperlyThrowException` do not through exceptions. They return null. I originally had them throw exceptions, but changed it to return null, but forgot to change the names. I'll address this.
That being said, are you saying you to return a hex encoded string of "\u0000" when `toHash` is null?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/641#discussion_r126810315
--- Diff: metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/HashFunctions.java ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang3.SerializationUtils;
+import org.apache.metron.stellar.dsl.BaseStellarFunction;
+import org.apache.metron.stellar.dsl.Stellar;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.List;
+
+public class HashFunctions {
+ @Stellar(
+ name = "HASH",
+ description = "Hashes a given value using the given hashing algorithm and returns a hex encoded string. This function only hashes " +
+ "strings and values that implement java.io.Serializable.",
+ params = {
+ "toHash - value to hash.",
+ "hashType - A valid string representation of an algorithm supported by java.security.MessageDigest.",
+ },
--- End diff --
may want to call out that it is a string of the hex value
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.
Posted by mattf-horton <gi...@git.apache.org>.
Github user mattf-horton commented on a diff in the pull request:
https://github.com/apache/metron/pull/641#discussion_r126848506
--- Diff: metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/HashFunctionsTest.java ---
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang.SerializationUtils;
+import org.junit.Test;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.security.Security;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.metron.stellar.common.utils.StellarProcessorUtils.run;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNull;
+
+public class HashFunctionsTest {
+ final HashFunctions.Hash hash = new HashFunctions.Hash();
+
+ @Test(expected = IllegalArgumentException.class)
+ public void nullArgumentListShouldThrowException() throws Exception {
+ hash.apply(null);
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void emptyArgumentListShouldThrowException() throws Exception {
+ hash.apply(Collections.emptyList());
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void singleArgumentListShouldThrowException() throws Exception {
+ hash.apply(Collections.singletonList("some value."));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void argumentListWithMoreThanTwoValuesShouldThrowException3() throws Exception {
+ hash.apply(Arrays.asList("1", "2", "3"));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void argumentListWithMoreThanTwoValuesShouldThrowException4() throws Exception {
+ hash.apply(Arrays.asList("1", "2", "3", "4"));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void invalidAlgorithmArgumentShouldThrowException() throws Exception {
+ hash.apply(Arrays.asList("value to hash", "invalidAlgorithm"));
+ }
+
+ @Test
+ public void invalidNullAlgorithmArgumentShouldThrowException() throws Exception {
+ assertNull(hash.apply(Arrays.asList("value to hash", null)));
+ }
+
+ @Test
+ public void nullInputForValueToHashShouldProperlyThrowException() throws Exception {
+ assertNull(hash.apply(Arrays.asList(null, "md5")));
+ }
+
+ @Test
+ public void allAlgorithmsForMessageDigestShouldBeAbleToHash() throws Exception {
+ final String valueToHash = "My value to hash";
+ final Set<String> algorithms = Security.getAlgorithms("MessageDigest");
+
+ algorithms.forEach(algorithm -> {
+ try {
+ final MessageDigest m = MessageDigest.getInstance(algorithm);
+ m.update(valueToHash.getBytes(StandardCharsets.UTF_8));
+
+ assertEquals(BaseEncoding.base16().encode(m.digest()), hash.apply(Arrays.asList(valueToHash, algorithm)));
+ } catch (NoSuchAlgorithmException e) {
+ throw new RuntimeException(e);
+ }
+ });
+ }
+
+ @Test
+ public void allAlgorithmsForMessageDigestShouldBeAbleToHashDirectStellarCall() throws Exception {
+ final String valueToHash = "My value to hash";
+ final Set<String> algorithms = Security.getAlgorithms("MessageDigest");
+
+ algorithms.forEach(algorithm -> {
+ try {
+ final Object actual = run("HASH('" + valueToHash + "', '" + algorithm + "')", Collections.emptyMap());
+
+ final MessageDigest m = MessageDigest.getInstance(algorithm);
+ m.update(valueToHash.getBytes(StandardCharsets.UTF_8));
+
+ assertEquals(BaseEncoding.base16().encode(m.digest()), actual);
+ } catch (NoSuchAlgorithmException e) {
+ throw new RuntimeException(e);
+ }
+ });
+ }
+
+ @Test
+ public void nonStringValueThatIsSerializableShouldBeSerializedAndThenEncodedUsingHex() throws Exception {
--- End diff --
As noted, it's a nit. Do as you choose, it won't interfere with a +1 from me, after the substantive items are taken care of. My inclination, though, is that verbiage (beyond what's needed for clarity) should go in comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.
Posted by jjmeyer0 <gi...@git.apache.org>.
Github user jjmeyer0 commented on a diff in the pull request:
https://github.com/apache/metron/pull/641#discussion_r126589803
--- Diff: metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/HashFunctions.java ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang3.SerializationUtils;
+import org.apache.metron.stellar.dsl.BaseStellarFunction;
+import org.apache.metron.stellar.dsl.Stellar;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.List;
+
+public class HashFunctions {
+ @Stellar(
+ name = "HASH",
+ description = "Hashes a given value using the given hashing algorithm and returns a hex encoded string. This function only hashes " +
+ "strings and values that implement java.io.Serializable.",
+ params = {
+ "toHash - value to hash.",
+ "hashType - A valid string representation of an algorithm supported by java.security.MessageDigest.",
+ },
+ returns = "A hex representation of a hashed value using the given hashing algorithm. If either argument is " +
+ "null then null will be returned. If the type of 'toHash' is neither a string nor of type java.io.Serializable, " +
+ "then null is returned."
+ )
+ public static class Hash extends BaseStellarFunction {
--- End diff --
@ottobackwards I'll make sure to stay consistent with what you have.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/641#discussion_r126836524
--- Diff: metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/HashFunctions.java ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang3.SerializationUtils;
+import org.apache.metron.stellar.dsl.BaseStellarFunction;
+import org.apache.metron.stellar.dsl.Stellar;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.List;
+
+public class HashFunctions {
+ @Stellar(
+ name = "HASH",
+ description = "Hashes a given value using the given hashing algorithm and returns a hex encoded string. This function only hashes " +
+ "strings and values that implement java.io.Serializable.",
+ params = {
+ "toHash - value to hash.",
+ "hashType - A valid string representation of an algorithm supported by java.security.MessageDigest.",
+ },
+ returns = "A hex representation of a hashed value using the given hashing algorithm. If either argument is " +
+ "null then null will be returned. If the type of 'toHash' is neither a string nor of type java.io.Serializable, " +
+ "then null is returned."
+ )
+ public static class Hash extends BaseStellarFunction {
+
+ @Override
+ public Object apply(final List<Object> args) {
+ if (args == null || args.size() != 2) {
+ throw new IllegalArgumentException("Invalid number of arguments: " + (args == null ? 0 : args.size()));
+ }
+
+ final Object toHash = args.get(0);
+ final Object hashType = args.get(1);
+
+ if (toHash == null || hashType == null) {
+ return null;
+ }
+
+ final MessageDigest messageDigest = getMessageDigest(hashType.toString());
+
+ if (toHash instanceof String) {
+ return getHashAsHex(messageDigest, toHash.toString().getBytes(StandardCharsets.UTF_8));
+ } else if (toHash instanceof Serializable) {
+ final byte[] serialized = SerializationUtils.serialize((Serializable) toHash);
+ return getHashAsHex(messageDigest, serialized);
+ }
+
+ return null;
+ }
+
+ private MessageDigest getMessageDigest(String hashType) {
--- End diff --
if you were on irc we wound't have these problems ;)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.
Posted by jjmeyer0 <gi...@git.apache.org>.
Github user jjmeyer0 commented on a diff in the pull request:
https://github.com/apache/metron/pull/641#discussion_r126838847
--- Diff: metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/HashFunctionsTest.java ---
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang.SerializationUtils;
+import org.junit.Test;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.security.Security;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.metron.stellar.common.utils.StellarProcessorUtils.run;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNull;
+
+public class HashFunctionsTest {
+ final HashFunctions.Hash hash = new HashFunctions.Hash();
+
+ @Test(expected = IllegalArgumentException.class)
+ public void nullArgumentListShouldThrowException() throws Exception {
+ hash.apply(null);
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void emptyArgumentListShouldThrowException() throws Exception {
+ hash.apply(Collections.emptyList());
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void singleArgumentListShouldThrowException() throws Exception {
+ hash.apply(Collections.singletonList("some value."));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void argumentListWithMoreThanTwoValuesShouldThrowException3() throws Exception {
+ hash.apply(Arrays.asList("1", "2", "3"));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void argumentListWithMoreThanTwoValuesShouldThrowException4() throws Exception {
+ hash.apply(Arrays.asList("1", "2", "3", "4"));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void invalidAlgorithmArgumentShouldThrowException() throws Exception {
+ hash.apply(Arrays.asList("value to hash", "invalidAlgorithm"));
+ }
+
+ @Test
+ public void invalidNullAlgorithmArgumentShouldThrowException() throws Exception {
+ assertNull(hash.apply(Arrays.asList("value to hash", null)));
+ }
+
+ @Test
+ public void nullInputForValueToHashShouldProperlyThrowException() throws Exception {
+ assertNull(hash.apply(Arrays.asList(null, "md5")));
+ }
+
+ @Test
+ public void allAlgorithmsForMessageDigestShouldBeAbleToHash() throws Exception {
+ final String valueToHash = "My value to hash";
+ final Set<String> algorithms = Security.getAlgorithms("MessageDigest");
+
+ algorithms.forEach(algorithm -> {
+ try {
+ final MessageDigest m = MessageDigest.getInstance(algorithm);
+ m.update(valueToHash.getBytes(StandardCharsets.UTF_8));
+
+ assertEquals(BaseEncoding.base16().encode(m.digest()), hash.apply(Arrays.asList(valueToHash, algorithm)));
+ } catch (NoSuchAlgorithmException e) {
+ throw new RuntimeException(e);
--- End diff --
@mattf-horton thanks for the clarification. I never took the time to read more into it. You're comment made me take the time. thanks again!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/641#discussion_r126812125
--- Diff: metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/HashFunctions.java ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang3.SerializationUtils;
+import org.apache.metron.stellar.dsl.BaseStellarFunction;
+import org.apache.metron.stellar.dsl.Stellar;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.List;
+
+public class HashFunctions {
+ @Stellar(
+ name = "HASH",
+ description = "Hashes a given value using the given hashing algorithm and returns a hex encoded string. This function only hashes " +
+ "strings and values that implement java.io.Serializable.",
+ params = {
+ "toHash - value to hash.",
+ "hashType - A valid string representation of an algorithm supported by java.security.MessageDigest.",
+ },
+ returns = "A hex representation of a hashed value using the given hashing algorithm. If either argument is " +
+ "null then null will be returned. If the type of 'toHash' is neither a string nor of type java.io.Serializable, " +
+ "then null is returned."
+ )
+ public static class Hash extends BaseStellarFunction {
+
+ @Override
+ public Object apply(final List<Object> args) {
+ if (args == null || args.size() != 2) {
+ throw new IllegalArgumentException("Invalid number of arguments: " + (args == null ? 0 : args.size()));
+ }
+
+ final Object toHash = args.get(0);
+ final Object hashType = args.get(1);
+
+ if (toHash == null || hashType == null) {
+ return null;
+ }
+
+ final MessageDigest messageDigest = getMessageDigest(hashType.toString());
+
+ if (toHash instanceof String) {
+ return getHashAsHex(messageDigest, toHash.toString().getBytes(StandardCharsets.UTF_8));
+ } else if (toHash instanceof Serializable) {
+ final byte[] serialized = SerializationUtils.serialize((Serializable) toHash);
+ return getHashAsHex(messageDigest, serialized);
+ }
+
+ return null;
+ }
+
+ private MessageDigest getMessageDigest(String hashType) {
--- End diff --
Is it possible that the MessageDigest and hashing functions may be usable to other Stellar functions? I think they may. If so, then putting them into a stellar/common/utils/hashing/xxxx area would be better, then you are just making Stellar functions exposing those functions, and not 'burying' them in the class.
Disclaimer: this is what I did with the Encodings helper class, for base32,64,binary,hex encodings
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/641#discussion_r126579811
--- Diff: metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/HashFunctions.java ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang3.SerializationUtils;
+import org.apache.metron.stellar.dsl.BaseStellarFunction;
+import org.apache.metron.stellar.dsl.Stellar;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.List;
+
+public class HashFunctions {
+ @Stellar(
+ name = "HASH",
+ description = "Hashes a given value using the given hashing algorithm and returns a hex encoded string. This function only hashes " +
+ "strings and values that implement java.io.Serializable.",
+ params = {
+ "toHash - value to hash.",
+ "hashType - A valid string representation of an algorithm supported by java.security.MessageDigest.",
+ },
+ returns = "A hex representation of a hashed value using the given hashing algorithm. If either argument is " +
+ "null then null will be returned. If the type of 'toHash' is neither a string nor of type java.io.Serializable, " +
+ "then null is returned."
+ )
+ public static class Hash extends BaseStellarFunction {
--- End diff --
I changed mine to GET_ instead of LIST_ and return a List<String>
So we shall be consistent.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.
Posted by mattf-horton <gi...@git.apache.org>.
Github user mattf-horton commented on a diff in the pull request:
https://github.com/apache/metron/pull/641#discussion_r126573313
--- Diff: metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/HashFunctions.java ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang3.SerializationUtils;
+import org.apache.metron.stellar.dsl.BaseStellarFunction;
+import org.apache.metron.stellar.dsl.Stellar;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.List;
+
+public class HashFunctions {
+ @Stellar(
+ name = "HASH",
+ description = "Hashes a given value using the given hashing algorithm and returns a hex encoded string. This function only hashes " +
+ "strings and values that implement java.io.Serializable.",
+ params = {
+ "toHash - value to hash.",
+ "hashType - A valid string representation of an algorithm supported by java.security.MessageDigest.",
+ },
+ returns = "A hex representation of a hashed value using the given hashing algorithm. If either argument is " +
+ "null then null will be returned. If the type of 'toHash' is neither a string nor of type java.io.Serializable, " +
+ "then null is returned."
+ )
+ public static class Hash extends BaseStellarFunction {
--- End diff --
Well, SPLIT and JOIN let you move back and forth between the two formats, but wouldn't the List format be the most naturally usable one?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/metron/pull/641
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.
Posted by mattf-horton <gi...@git.apache.org>.
Github user mattf-horton commented on a diff in the pull request:
https://github.com/apache/metron/pull/641#discussion_r126474912
--- Diff: metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/HashFunctions.java ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang3.SerializationUtils;
+import org.apache.metron.stellar.dsl.BaseStellarFunction;
+import org.apache.metron.stellar.dsl.Stellar;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.List;
+
+public class HashFunctions {
+ @Stellar(
+ name = "HASH",
+ description = "Hashes a given value using the given hashing algorithm and returns a hex encoded string. This function only hashes " +
+ "strings and values that implement java.io.Serializable.",
+ params = {
+ "toHash - value to hash.",
+ "hashType - A valid string representation of an algorithm supported by java.security.MessageDigest.",
+ },
+ returns = "A hex representation of a hashed value using the given hashing algorithm. If either argument is " +
+ "null then null will be returned. If the type of 'toHash' is neither a string nor of type java.io.Serializable, " +
+ "then null is returned."
+ )
+ public static class Hash extends BaseStellarFunction {
--- End diff --
Nice use of general Java hash algorithm i/f. Suggest you also add a convenience Stellar function to return the available list of hash algs in current environment. Perhaps GET_HASHES_AVAILABLE() -> List<String>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron issue #641: METRON-539: added HASH function for stellar.
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/641
<img class="emoji" title=":lgtm:" alt=":lgtm:" align="absmiddle" src="https://reviewable.io/lgtm.png" height="20" width="61"/>
---
*Comments from [Reviewable](https://reviewable.io:443/reviews/apache/metron/641#-:-KrWliAsg4-eqkjCEHxu:bnfp4nl)*
<!-- Sent from Reviewable.io -->
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/641#discussion_r126566101
--- Diff: metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/HashFunctions.java ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang3.SerializationUtils;
+import org.apache.metron.stellar.dsl.BaseStellarFunction;
+import org.apache.metron.stellar.dsl.Stellar;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.List;
+
+public class HashFunctions {
+ @Stellar(
+ name = "HASH",
+ description = "Hashes a given value using the given hashing algorithm and returns a hex encoded string. This function only hashes " +
+ "strings and values that implement java.io.Serializable.",
+ params = {
+ "toHash - value to hash.",
+ "hashType - A valid string representation of an algorithm supported by java.security.MessageDigest.",
+ },
+ returns = "A hex representation of a hashed value using the given hashing algorithm. If either argument is " +
+ "null then null will be returned. If the type of 'toHash' is neither a string nor of type java.io.Serializable, " +
+ "then null is returned."
+ )
+ public static class Hash extends BaseStellarFunction {
--- End diff --
My pr for encoding has this, but it returns a comma delimited string
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.
Posted by jjmeyer0 <gi...@git.apache.org>.
Github user jjmeyer0 commented on a diff in the pull request:
https://github.com/apache/metron/pull/641#discussion_r126838736
--- Diff: metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/HashFunctionsTest.java ---
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang.SerializationUtils;
+import org.junit.Test;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.security.Security;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.metron.stellar.common.utils.StellarProcessorUtils.run;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNull;
+
+public class HashFunctionsTest {
+ final HashFunctions.Hash hash = new HashFunctions.Hash();
+
+ @Test(expected = IllegalArgumentException.class)
+ public void nullArgumentListShouldThrowException() throws Exception {
+ hash.apply(null);
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void emptyArgumentListShouldThrowException() throws Exception {
+ hash.apply(Collections.emptyList());
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void singleArgumentListShouldThrowException() throws Exception {
+ hash.apply(Collections.singletonList("some value."));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void argumentListWithMoreThanTwoValuesShouldThrowException3() throws Exception {
+ hash.apply(Arrays.asList("1", "2", "3"));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void argumentListWithMoreThanTwoValuesShouldThrowException4() throws Exception {
+ hash.apply(Arrays.asList("1", "2", "3", "4"));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void invalidAlgorithmArgumentShouldThrowException() throws Exception {
+ hash.apply(Arrays.asList("value to hash", "invalidAlgorithm"));
+ }
+
+ @Test
+ public void invalidNullAlgorithmArgumentShouldThrowException() throws Exception {
+ assertNull(hash.apply(Arrays.asList("value to hash", null)));
+ }
+
+ @Test
+ public void nullInputForValueToHashShouldProperlyThrowException() throws Exception {
+ assertNull(hash.apply(Arrays.asList(null, "md5")));
+ }
+
+ @Test
+ public void allAlgorithmsForMessageDigestShouldBeAbleToHash() throws Exception {
+ final String valueToHash = "My value to hash";
+ final Set<String> algorithms = Security.getAlgorithms("MessageDigest");
+
+ algorithms.forEach(algorithm -> {
+ try {
+ final MessageDigest m = MessageDigest.getInstance(algorithm);
+ m.update(valueToHash.getBytes(StandardCharsets.UTF_8));
+
+ assertEquals(BaseEncoding.base16().encode(m.digest()), hash.apply(Arrays.asList(valueToHash, algorithm)));
+ } catch (NoSuchAlgorithmException e) {
+ throw new RuntimeException(e);
+ }
+ });
+ }
+
+ @Test
+ public void allAlgorithmsForMessageDigestShouldBeAbleToHashDirectStellarCall() throws Exception {
+ final String valueToHash = "My value to hash";
+ final Set<String> algorithms = Security.getAlgorithms("MessageDigest");
+
+ algorithms.forEach(algorithm -> {
+ try {
+ final Object actual = run("HASH('" + valueToHash + "', '" + algorithm + "')", Collections.emptyMap());
+
+ final MessageDigest m = MessageDigest.getInstance(algorithm);
+ m.update(valueToHash.getBytes(StandardCharsets.UTF_8));
+
+ assertEquals(BaseEncoding.base16().encode(m.digest()), actual);
+ } catch (NoSuchAlgorithmException e) {
+ throw new RuntimeException(e);
+ }
+ });
+ }
+
+ @Test
+ public void nonStringValueThatIsSerializableShouldBeSerializedAndThenEncodedUsingHex() throws Exception {
--- End diff --
@mattf-horton I understand that many of these are long. This is somewhat intentional on my part. I personally don't like the `test*` pattern. IIRC this is the way junit used to pick up tests. Since this isn't necessarily needed anymore, I prefer to use these longer form names. My reason is, when I write tests, I actually write a lot these before I actually start coding. It helps me think through things. That being said, I do agree that some of them are a quite long, and I will re-word them. Also, I'm not completely tied to it. If you still prefer the shorter "test*" let me know and I will update it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.
Posted by mattf-horton <gi...@git.apache.org>.
Github user mattf-horton commented on a diff in the pull request:
https://github.com/apache/metron/pull/641#discussion_r126560557
--- Diff: metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/HashFunctionsTest.java ---
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang.SerializationUtils;
+import org.junit.Test;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.security.Security;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.metron.stellar.common.utils.StellarProcessorUtils.run;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNull;
+
+public class HashFunctionsTest {
+ final HashFunctions.Hash hash = new HashFunctions.Hash();
+
+ @Test(expected = IllegalArgumentException.class)
+ public void nullArgumentListShouldThrowException() throws Exception {
+ hash.apply(null);
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void emptyArgumentListShouldThrowException() throws Exception {
+ hash.apply(Collections.emptyList());
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void singleArgumentListShouldThrowException() throws Exception {
+ hash.apply(Collections.singletonList("some value."));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void argumentListWithMoreThanTwoValuesShouldThrowException3() throws Exception {
+ hash.apply(Arrays.asList("1", "2", "3"));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void argumentListWithMoreThanTwoValuesShouldThrowException4() throws Exception {
+ hash.apply(Arrays.asList("1", "2", "3", "4"));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void invalidAlgorithmArgumentShouldThrowException() throws Exception {
+ hash.apply(Arrays.asList("value to hash", "invalidAlgorithm"));
+ }
+
+ @Test
+ public void invalidNullAlgorithmArgumentShouldThrowException() throws Exception {
+ assertNull(hash.apply(Arrays.asList("value to hash", null)));
+ }
+
+ @Test
+ public void nullInputForValueToHashShouldProperlyThrowException() throws Exception {
+ assertNull(hash.apply(Arrays.asList(null, "md5")));
+ }
+
+ @Test
+ public void allAlgorithmsForMessageDigestShouldBeAbleToHash() throws Exception {
+ final String valueToHash = "My value to hash";
+ final Set<String> algorithms = Security.getAlgorithms("MessageDigest");
+
+ algorithms.forEach(algorithm -> {
+ try {
+ final MessageDigest m = MessageDigest.getInstance(algorithm);
+ m.update(valueToHash.getBytes(StandardCharsets.UTF_8));
+
+ assertEquals(BaseEncoding.base16().encode(m.digest()), hash.apply(Arrays.asList(valueToHash, algorithm)));
+ } catch (NoSuchAlgorithmException e) {
+ throw new RuntimeException(e);
--- End diff --
@jjmeyer0 , quite right! Thanks, I learned something :-)
For those not already familiar with this, it isn't that it's a lambda, it's that it's a lambda being used as the Consumer argument to a .forEach call, and Consumer.accept() is contractually specified to not throw checked exceptions.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.
Posted by jjmeyer0 <gi...@git.apache.org>.
Github user jjmeyer0 commented on a diff in the pull request:
https://github.com/apache/metron/pull/641#discussion_r126524234
--- Diff: metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/HashFunctionsTest.java ---
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang.SerializationUtils;
+import org.junit.Test;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.security.Security;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.metron.stellar.common.utils.StellarProcessorUtils.run;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNull;
+
+public class HashFunctionsTest {
+ final HashFunctions.Hash hash = new HashFunctions.Hash();
+
+ @Test(expected = IllegalArgumentException.class)
+ public void nullArgumentListShouldThrowException() throws Exception {
+ hash.apply(null);
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void emptyArgumentListShouldThrowException() throws Exception {
+ hash.apply(Collections.emptyList());
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void singleArgumentListShouldThrowException() throws Exception {
+ hash.apply(Collections.singletonList("some value."));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void argumentListWithMoreThanTwoValuesShouldThrowException3() throws Exception {
+ hash.apply(Arrays.asList("1", "2", "3"));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void argumentListWithMoreThanTwoValuesShouldThrowException4() throws Exception {
+ hash.apply(Arrays.asList("1", "2", "3", "4"));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void invalidAlgorithmArgumentShouldThrowException() throws Exception {
+ hash.apply(Arrays.asList("value to hash", "invalidAlgorithm"));
+ }
+
+ @Test
+ public void invalidNullAlgorithmArgumentShouldThrowException() throws Exception {
+ assertNull(hash.apply(Arrays.asList("value to hash", null)));
+ }
+
+ @Test
+ public void nullInputForValueToHashShouldProperlyThrowException() throws Exception {
+ assertNull(hash.apply(Arrays.asList(null, "md5")));
+ }
+
+ @Test
+ public void allAlgorithmsForMessageDigestShouldBeAbleToHash() throws Exception {
+ final String valueToHash = "My value to hash";
+ final Set<String> algorithms = Security.getAlgorithms("MessageDigest");
+
+ algorithms.forEach(algorithm -> {
+ try {
+ final MessageDigest m = MessageDigest.getInstance(algorithm);
+ m.update(valueToHash.getBytes(StandardCharsets.UTF_8));
+
+ assertEquals(BaseEncoding.base16().encode(m.digest()), hash.apply(Arrays.asList(valueToHash, algorithm)));
+ } catch (NoSuchAlgorithmException e) {
+ throw new RuntimeException(e);
--- End diff --
@mattf-horton since I'm using a lambda, I don't believe this is possible since `NoSuchAlgorithmException` is a checked exception.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.
Posted by mattf-horton <gi...@git.apache.org>.
Github user mattf-horton commented on a diff in the pull request:
https://github.com/apache/metron/pull/641#discussion_r126752781
--- Diff: metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/HashFunctions.java ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang3.SerializationUtils;
+import org.apache.metron.stellar.dsl.BaseStellarFunction;
+import org.apache.metron.stellar.dsl.Stellar;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.List;
+
+public class HashFunctions {
+ @Stellar(
+ name = "HASH",
+ description = "Hashes a given value using the given hashing algorithm and returns a hex encoded string. This function only hashes " +
+ "strings and values that implement java.io.Serializable.",
+ params = {
+ "toHash - value to hash.",
+ "hashType - A valid string representation of an algorithm supported by java.security.MessageDigest.",
+ },
+ returns = "A hex representation of a hashed value using the given hashing algorithm. If either argument is " +
+ "null then null will be returned. If the type of 'toHash' is neither a string nor of type java.io.Serializable, " +
+ "then null is returned."
+ )
+ public static class Hash extends BaseStellarFunction {
--- End diff --
Thanks @ottobackwards , I agree.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.
Posted by mattf-horton <gi...@git.apache.org>.
Github user mattf-horton commented on a diff in the pull request:
https://github.com/apache/metron/pull/641#discussion_r126849654
--- Diff: metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/HashFunctionsTest.java ---
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang.SerializationUtils;
+import org.junit.Test;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.security.Security;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.metron.stellar.common.utils.StellarProcessorUtils.run;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNull;
+
+public class HashFunctionsTest {
+ final HashFunctions.Hash hash = new HashFunctions.Hash();
+
+ @Test(expected = IllegalArgumentException.class)
+ public void nullArgumentListShouldThrowException() throws Exception {
+ hash.apply(null);
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void emptyArgumentListShouldThrowException() throws Exception {
+ hash.apply(Collections.emptyList());
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void singleArgumentListShouldThrowException() throws Exception {
+ hash.apply(Collections.singletonList("some value."));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void argumentListWithMoreThanTwoValuesShouldThrowException3() throws Exception {
+ hash.apply(Arrays.asList("1", "2", "3"));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void argumentListWithMoreThanTwoValuesShouldThrowException4() throws Exception {
+ hash.apply(Arrays.asList("1", "2", "3", "4"));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void invalidAlgorithmArgumentShouldThrowException() throws Exception {
+ hash.apply(Arrays.asList("value to hash", "invalidAlgorithm"));
+ }
+
+ @Test
+ public void invalidNullAlgorithmArgumentShouldThrowException() throws Exception {
+ assertNull(hash.apply(Arrays.asList("value to hash", null)));
+ }
+
+ @Test
+ public void nullInputForValueToHashShouldProperlyThrowException() throws Exception {
+ assertNull(hash.apply(Arrays.asList(null, "md5")));
+ }
--- End diff --
Ah, I was trusting your very detailed testcase names :-) If they don't throw exceptions then of course the annotation should be consistent.
I remain of the opinion that it would be good for HASH to be able to return a conforming result for null objects, ie "00" instead of null. But this is mostly an aesthetic judgment on my part, so I could be talked out of it.
You gave me a headache by prefixing the "00" string with the unicode symbol :-) and no, "\u0000" would not be the desirable return value; that would be the unicode codepoint for a single null character. Yuck! But I shouldn't have used the hex prefix "0x" either. Since we are returning unadorned string values representing hex values, it should be "00" (a pair of zero digit characters in a String of length 2).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.
Posted by mattf-horton <gi...@git.apache.org>.
Github user mattf-horton commented on a diff in the pull request:
https://github.com/apache/metron/pull/641#discussion_r126476908
--- Diff: metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/HashFunctionsTest.java ---
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang.SerializationUtils;
+import org.junit.Test;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.security.Security;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.metron.stellar.common.utils.StellarProcessorUtils.run;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNull;
+
+public class HashFunctionsTest {
+ final HashFunctions.Hash hash = new HashFunctions.Hash();
+
+ @Test(expected = IllegalArgumentException.class)
+ public void nullArgumentListShouldThrowException() throws Exception {
+ hash.apply(null);
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void emptyArgumentListShouldThrowException() throws Exception {
+ hash.apply(Collections.emptyList());
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void singleArgumentListShouldThrowException() throws Exception {
+ hash.apply(Collections.singletonList("some value."));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void argumentListWithMoreThanTwoValuesShouldThrowException3() throws Exception {
+ hash.apply(Arrays.asList("1", "2", "3"));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void argumentListWithMoreThanTwoValuesShouldThrowException4() throws Exception {
+ hash.apply(Arrays.asList("1", "2", "3", "4"));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void invalidAlgorithmArgumentShouldThrowException() throws Exception {
+ hash.apply(Arrays.asList("value to hash", "invalidAlgorithm"));
+ }
+
+ @Test
+ public void invalidNullAlgorithmArgumentShouldThrowException() throws Exception {
+ assertNull(hash.apply(Arrays.asList("value to hash", null)));
+ }
+
+ @Test
+ public void nullInputForValueToHashShouldProperlyThrowException() throws Exception {
+ assertNull(hash.apply(Arrays.asList(null, "md5")));
+ }
+
+ @Test
+ public void allAlgorithmsForMessageDigestShouldBeAbleToHash() throws Exception {
+ final String valueToHash = "My value to hash";
+ final Set<String> algorithms = Security.getAlgorithms("MessageDigest");
+
+ algorithms.forEach(algorithm -> {
+ try {
+ final MessageDigest m = MessageDigest.getInstance(algorithm);
+ m.update(valueToHash.getBytes(StandardCharsets.UTF_8));
+
+ assertEquals(BaseEncoding.base16().encode(m.digest()), hash.apply(Arrays.asList(valueToHash, algorithm)));
+ } catch (NoSuchAlgorithmException e) {
+ throw new RuntimeException(e);
+ }
+ });
+ }
+
+ @Test
+ public void allAlgorithmsForMessageDigestShouldBeAbleToHashDirectStellarCall() throws Exception {
+ final String valueToHash = "My value to hash";
+ final Set<String> algorithms = Security.getAlgorithms("MessageDigest");
+
+ algorithms.forEach(algorithm -> {
+ try {
+ final Object actual = run("HASH('" + valueToHash + "', '" + algorithm + "')", Collections.emptyMap());
+
+ final MessageDigest m = MessageDigest.getInstance(algorithm);
--- End diff --
Nit: call this "expected" instead of "m" would read more clearly, here and in each test cases below.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.
Posted by mattf-horton <gi...@git.apache.org>.
Github user mattf-horton commented on a diff in the pull request:
https://github.com/apache/metron/pull/641#discussion_r126834205
--- Diff: metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/HashFunctions.java ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang3.SerializationUtils;
+import org.apache.metron.stellar.dsl.BaseStellarFunction;
+import org.apache.metron.stellar.dsl.Stellar;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.List;
+
+public class HashFunctions {
+ @Stellar(
+ name = "HASH",
--- End diff --
Ah, sorry I didn't understand this was only a comment about documentation. I guess you're right; since most objects are indeed serializable, maybe it's okay to let them discover the occasional object that isn't (if any).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.
Posted by mattf-horton <gi...@git.apache.org>.
Github user mattf-horton commented on a diff in the pull request:
https://github.com/apache/metron/pull/641#discussion_r126474167
--- Diff: metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/HashFunctionsTest.java ---
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang.SerializationUtils;
+import org.junit.Test;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.security.Security;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.metron.stellar.common.utils.StellarProcessorUtils.run;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNull;
+
+public class HashFunctionsTest {
+ final HashFunctions.Hash hash = new HashFunctions.Hash();
+
+ @Test(expected = IllegalArgumentException.class)
+ public void nullArgumentListShouldThrowException() throws Exception {
+ hash.apply(null);
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void emptyArgumentListShouldThrowException() throws Exception {
+ hash.apply(Collections.emptyList());
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void singleArgumentListShouldThrowException() throws Exception {
+ hash.apply(Collections.singletonList("some value."));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void argumentListWithMoreThanTwoValuesShouldThrowException3() throws Exception {
+ hash.apply(Arrays.asList("1", "2", "3"));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void argumentListWithMoreThanTwoValuesShouldThrowException4() throws Exception {
+ hash.apply(Arrays.asList("1", "2", "3", "4"));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void invalidAlgorithmArgumentShouldThrowException() throws Exception {
+ hash.apply(Arrays.asList("value to hash", "invalidAlgorithm"));
+ }
+
+ @Test
+ public void invalidNullAlgorithmArgumentShouldThrowException() throws Exception {
+ assertNull(hash.apply(Arrays.asList("value to hash", null)));
+ }
+
+ @Test
+ public void nullInputForValueToHashShouldProperlyThrowException() throws Exception {
+ assertNull(hash.apply(Arrays.asList(null, "md5")));
+ }
--- End diff --
Not sure I agree about this one. There's a lot of cases, including Stellar validation, where functions can get fed null arguments when you might not expect it. What if we establish a norm that hash of null returns 0x00, for all hashing algorithms?
If you leave this one as is, remove "Properly" from testcase name and add "@Test(expected = IllegalArgumentException.class)".
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---