You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by GitBox <gi...@apache.org> on 2020/03/02 18:58:18 UTC
[GitHub] [nifi] Xeinn opened a new pull request #4104: NIFI-7159
Xeinn opened a new pull request #4104: NIFI-7159
URL: https://github.com/apache/nifi/pull/4104
Aligning the implementation of DECIMAL RecordFieldType with other types
Introduce RecordFieldType of DECIMAL for BigDecimal values
Adding RecordFieldType DECIMAL to ResultSetRecordSet
Introduce RecordFieldType DECIMAL into DataTypeUtils
Added missing BIGINT type to convertSimpleIfPossible method
Added missing type BIGINT to convertField method
Added missing type BIGINT to writeFieldForType
Added missing type BIGINT to parseFieldForType
Added missing type BIGINT to parseStringForType
Potentially breaking change of behaviour for Kudu Service.
Switched decimal from treatment as string to treatment as decimal
Added big decimal support for ElasticSearch
Added big decimal support for hbase processors
Added big decimal support for prometheus reporting
Added big decimal support to solr
Big decimal support in site to site reporting
Core support for big decimal
Fix issue with new toBigDecimal method
Issue where method was not correctly handling the situation where a
BigDecimal values was received as input.
Update test to reflect use of BigDecimal
Changes to unit tests to support DECIMAL type
Fix bug where NaN Double reported from jvm metrics not handled
BigDecimal implementation for mongo processor
Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
#### Description of PR
_Enables X functionality; fixes bug NIFI-7159._
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [Y] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [Y] Does your PR title start with **NIFI-XXXX** where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [Y] Has your PR been rebased against the latest commit within the target branch (typically `master`)?
- [Y] Is your initial contribution a single, squashed commit? _Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not `squash` or use `--force` when pushing to allow for clean monitoring of changes._
### For code changes:
- [Y] Have you ensured that the full suite of tests is executed via `mvn -Pcontrib-check clean install` at the root `nifi` folder?
- [Y] Have you written or updated unit tests to verify your changes?
- [Y] Have you verified that the full build is successful on both JDK 8 and JDK 11?
- [N/A] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [N/A] If applicable, have you updated the `LICENSE` file, including the main `LICENSE` file under `nifi-assembly`?
- [N/A] If applicable, have you updated the `NOTICE` file, including the main `NOTICE` file found under `nifi-assembly`?
- [N/A] If adding new Properties, have you added `.displayName` in addition to .name (programmatic access) for each of the new properties?
### For documentation related changes:
- [N/A] Have you ensured that format looks appropriate for the output in which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] MikeThomsen commented on issue #4104: NIFI-7159
Posted by GitBox <gi...@apache.org>.
MikeThomsen commented on issue #4104: NIFI-7159
URL: https://github.com/apache/nifi/pull/4104#issuecomment-593629623
Will start reviewing tonight.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] tpalfy commented on a change in pull request #4104: NIFI-7159
Posted by GitBox <gi...@apache.org>.
tpalfy commented on a change in pull request #4104: NIFI-7159
URL: https://github.com/apache/nifi/pull/4104#discussion_r389051423
##########
File path: nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/record/type/DecimalDataType.java
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.serialization.record.type;
+
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+
+public class DecimalDataType extends DataType {
+
+ private final int precision;
+ private final int scale;
+
+ public DecimalDataType(int precision, int scale) {
+ super(RecordFieldType.DECIMAL, null);
+ this.precision = precision;
+ this.scale = scale;
+ }
+
+ public int getPrecision() {
+ return precision;
+ }
+
+ public int getScale() {
+ return scale;
+ }
+
+ @Override
+ public RecordFieldType getFieldType() {
+ return RecordFieldType.DECIMAL;
+ }
+
+ @Override
+ public int hashCode() {
+ return 31 + 41 * getFieldType().hashCode() + 41 * Integer.hashCode(precision) * Integer.hashCode(scale); // Need to confirm this hash * hash is correct
+ }
+
+ @Override
+ public boolean equals(final Object obj) {
+ if (obj == this) {
+ return true;
+ }
+ if (obj == null) {
+ return false;
+ }
+ if (!(obj instanceof DecimalDataType)) {
+ return false;
+ }
+
+ final DecimalDataType other = (DecimalDataType) obj;
+ return precision == other.getPrecision() && scale == other.getScale();
+ }
+
+ @Override
+ public String toString() {
+ return "DECIMAL" + Integer.toString(precision) + Integer.toString(scale);
Review comment:
Might be a good idea to add underscores (for example) so it looks like this:
`DECIMAL_11_11`
instead of this:
`DECIMAL1111`
Btw `Integer.toString` is redundant with the string literal plus '+' operator (makes it a bit less readable).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] Xeinn commented on issue #4104: NIFI-7159
Posted by GitBox <gi...@apache.org>.
Xeinn commented on issue #4104: NIFI-7159
URL: https://github.com/apache/nifi/pull/4104#issuecomment-593683120
Hi Mike,
Yes, was just looking at that.
Any thoughts? I’m thinking that the update should be it’s own processor now, eg nifi-mongodb-3.8-processors, as I don’t know of another clean way to handle the situation conditionally based on the profile.
Kicking myself for not spotting it sooner
Regards
Chris
Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10
________________________________
From: Mike <no...@github.com>
Sent: Monday, March 2, 2020 11:33:00 PM
To: apache/nifi <ni...@noreply.github.com>
Cc: Chris Manniex <cm...@hotmail.com>; Author <au...@noreply.github.com>
Subject: Re: [apache/nifi] NIFI-7159 (#4104)
Getting build errors now related to Decimal128.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fnifi%2Fpull%2F4104%3Femail_source%3Dnotifications%26email_token%3DAFHXE7DOBBWD6RRABMOK62DRFQ62ZA5CNFSM4K72OIGKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENRM4JA%23issuecomment-593677860&data=02%7C01%7C%7Cfd71e4b983b94947aa7008d7bf020be3%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637187887811668553&sdata=7FkyXDxK2g021ZkBw4G37UzfAF%2BxRzeqK1%2F1eDOEqg8%3D&reserved=0>, or unsubscribe<https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAFHXE7FBYHP424L77HJ7VK3RFQ62ZANCNFSM4K72OIGA&data=02%7C01%7C%7Cfd71e4b983b94947aa7008d7bf020be3%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637187887811678538&sdata=ZiS6b5VxBipH%2F3AgHwL7NicCbae5%2Be8lv7KGZwFqf%2B0%3D&reserved=0>.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] MikeThomsen commented on issue #4104: NIFI-7159 - Add
support for some data types in various processors
Posted by GitBox <gi...@apache.org>.
MikeThomsen commented on issue #4104: NIFI-7159 - Add support for some data types in various processors
URL: https://github.com/apache/nifi/pull/4104#issuecomment-602097123
@markap14 could you take a look at the precision part and give me your opinion?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] joewitt commented on issue #4104: NIFI-7159 - Add support
for some data types in various processors
Posted by GitBox <gi...@apache.org>.
joewitt commented on issue #4104: NIFI-7159 - Add support for some data types in various processors
URL: https://github.com/apache/nifi/pull/4104#issuecomment-602188614
All the builds including windows should pass and failures need to be looked into.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] MikeThomsen commented on issue #4104: NIFI-7159
Posted by GitBox <gi...@apache.org>.
MikeThomsen commented on issue #4104: NIFI-7159
URL: https://github.com/apache/nifi/pull/4104#issuecomment-593630124
You have a checkstyle violation in your changes `nifi-record`. See the output for why the build failed:
```
[WARNING] src/main/java/org/apache/nifi/serialization/record/RecordFieldType.java:[291] (regexp) RegexpSinglelineJava: Line has trailing whitespace.
[WARNING] src/main/java/org/apache/nifi/serialization/record/RecordFieldType.java:[292] (javadoc) NonEmptyAtclauseDescription: At-clause should have a non-empty description.
[WARNING] src/main/java/org/apache/nifi/serialization/record/RecordFieldType.java:[294] (regexp) RegexpSinglelineJava: Line has trailing whitespace.
[WARNING] src/main/java/org/apache/nifi/serialization/record/RecordFieldType.java:[298] (regexp) RegexpSinglelineJava: Line has trailing whitespace.
[WARNING] src/main/java/org/apache/nifi/serialization/record/RecordFieldType.java:[300,5] (whitespace) FileTabCharacter: Line contains a tab character.
[WARNING] src/main/java/org/apache/nifi/serialization/record/RecordFieldType.java:[301,5] (whitespace) FileTabCharacter: Line contains a tab character.
[WARNING] src/main/java/org/apache/nifi/serialization/record/RecordFieldType.java:[302,5] (whitespace) FileTabCharacter: Line contains a tab character.
[WARNING] src/main/java/org/apache/nifi/serialization/record/RecordFieldType.java:[303] (regexp) RegexpSinglelineJava: Line has trailing whitespace.
[WARNING] src/main/java/org/apache/nifi/serialization/record/RecordFieldType.java:[303,5] (whitespace) FileTabCharacter: Line contains a tab character.
[WARNING] src/main/java/org/apache/nifi/serialization/record/RecordFieldType.java:[304,5] (whitespace) FileTabCharacter: Line contains a tab character.
[WARNING] src/main/java/org/apache/nifi/serialization/record/RecordFieldType.java:[306] (regexp) RegexpSinglelineJava: Line has trailing whitespace.
[WARNING] src/main/java/org/apache/nifi/serialization/record/RecordFieldType.java:[307] (regexp) RegexpSinglelineJava: Line has trailing whitespace.
[WARNING] src/main/java/org/apache/nifi/serialization/record/util/DataTypeUtils.java:[464,17] (whitespace) FileTabCharacter: Line contains a tab character.
[WARNING] src/main/java/org/apache/nifi/serialization/record/type/DecimalDataType.java:[16] (regexp) RegexpSinglelineJava: Line has trailing whitespace.
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-checkstyle-plugin:3.1.0:check (check-style) on project nifi-record: You have 15 Checkstyle violations. -> [Help 1]
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] MikeThomsen commented on issue #4104: NIFI-7159
Posted by GitBox <gi...@apache.org>.
MikeThomsen commented on issue #4104: NIFI-7159
URL: https://github.com/apache/nifi/pull/4104#issuecomment-593691339
I think we can use this and a little reflection to get around that one explicit reference to Decimal128
https://stackoverflow.com/questions/52870627/get-mongodb-driver-version-programmatically
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] tpalfy commented on a change in pull request #4104: NIFI-7159
Posted by GitBox <gi...@apache.org>.
tpalfy commented on a change in pull request #4104: NIFI-7159
URL: https://github.com/apache/nifi/pull/4104#discussion_r389097315
##########
File path: nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/record/util/DataTypeUtils.java
##########
@@ -34,6 +34,7 @@
import java.io.InputStream;
import java.io.Reader;
import java.lang.reflect.Array;
+import java.math.BigDecimal;
Review comment:
Currently
```java
DataTypeUtils.findMostSuitableType(
new BigDecimal("11.110"),
Arrays.asList(RecordFieldType.DECIMAL.getDecimalDataType(5, 3)),
Function.identity()
);
```
correctly returns `Optional.of(RecordFieldType.DECIMAL.getDecimalDataType(5, 3));`, but
```java
DataTypeUtils.findMostSuitableType(
new BigDecimal("1.110"),
Arrays.asList(RecordFieldType.DECIMAL.getDecimalDataType(5, 3)),
Function.identity()
);
```
("1.110" instead of "11.111") returns `Optional.empty();`
This is because the inferred type DECIMAL(4,3) is not equal to the provided DECIMAL(5,3) so the `getWiderType` method is called which doesn't handle cases where the two types are the same in general, only differ in their details.
Probably the `public static Optional<DataType> getWiderType(final DataType thisDataType, final DataType otherDataType)` should handle two different DECIMAL types by returning another DECIMAL that is the superset of the two in terms of scale and precision.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] Xeinn commented on issue #4104: NIFI-7159
Posted by GitBox <gi...@apache.org>.
Xeinn commented on issue #4104: NIFI-7159
URL: https://github.com/apache/nifi/pull/4104#issuecomment-593706129
Thanks Mike,
Had a quick look at that and can’t seem to find any reference to the class MongoDriverVersion in any of the mongo-java-driver jars (checked 3.2.2 and 3.8.0 so far)
Or are you suggesting we query the driver version at runtime and then use Reflection to create and instance of Decimal128 using the classname?
Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10
________________________________
From: Mike <no...@github.com>
Sent: Tuesday, March 3, 2020 12:01:48 AM
To: apache/nifi <ni...@noreply.github.com>
Cc: Chris Manniex <cm...@hotmail.com>; Author <au...@noreply.github.com>
Subject: Re: [apache/nifi] NIFI-7159 (#4104)
I think we can use this and a little reflection to get around that one explicit reference to Decimal128
https://stackoverflow.com/questions/52870627/get-mongodb-driver-version-programmatically<https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstackoverflow.com%2Fquestions%2F52870627%2Fget-mongodb-driver-version-programmatically&data=02%7C01%7C%7C0253ae36f38c4f94a8ae08d7bf061247%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637187905099936240&sdata=3dXwi4Vk4cEE5%2Bnq%2F4RwY8SpdjBYDZvXwFaazFdvR5c%3D&reserved=0>
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fnifi%2Fpull%2F4104%3Femail_source%3Dnotifications%26email_token%3DAFHXE7FHL6IXL5WFIWQUKITRFRCGZA5CNFSM4K72OIGKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENRQFSY%23issuecomment-593691339&data=02%7C01%7C%7C0253ae36f38c4f94a8ae08d7bf061247%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637187905099946254&sdata=SHYb%2BQ2V0hyD8%2FbXwM1iEI43UJkh1RsYV3k8yVS4LmM%3D&reserved=0>, or unsubscribe<https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAFHXE7GPNHPQK22T52PQZF3RFRCGZANCNFSM4K72OIGA&data=02%7C01%7C%7C0253ae36f38c4f94a8ae08d7bf061247%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637187905099956256&sdata=kEWEPOwherm0Z0gkwKbJ1oopI10F7ISGe8QNRTzddUo%3D&reserved=0>.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] Xeinn commented on issue #4104: NIFI-7159 - Add support for
some data types in various processors
Posted by GitBox <gi...@apache.org>.
Xeinn commented on issue #4104: NIFI-7159 - Add support for some data types in various processors
URL: https://github.com/apache/nifi/pull/4104#issuecomment-599368815
Thanks Mike,
That’ll help.
Will tackle it this evening and shout if I have problems/questions
Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10
________________________________
From: Mike <no...@github.com>
Sent: Sunday, March 15, 2020 1:21:09 PM
To: apache/nifi <ni...@noreply.github.com>
Cc: Chris Manniex <cm...@hotmail.com>; Mention <me...@noreply.github.com>
Subject: Re: [apache/nifi] NIFI-7159 - Add support for some data types in various processors (#4104)
@Xeinn<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FXeinn&data=02%7C01%7C%7C2263074bc9c546305bd208d7c8e3ba1a%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637198752708391176&sdata=QP6Ul1JuqCThkQQg3S01T8aZqsjpe1roKp0QiBetcqw%3D&reserved=0> there's a merge conflict now. Instructions follow if you're newish to git. Otherwise ignore.
To resolve do the following:
1. Pull apache/nifi master into your master.
2. Rebase (git rebase master) your branch off of master again.
3. Follow the prompts that git gives you. Be careful to read what it says.
4. git push origin --force BRANCH.
Don't git pull apache/nifi master into your branch.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fnifi%2Fpull%2F4104%23issuecomment-599209694&data=02%7C01%7C%7C2263074bc9c546305bd208d7c8e3ba1a%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637198752708401167&sdata=qtGlAFv8ejfLeZ86havjZ0d7n3A9ss4pXT0mNNPdhSs%3D&reserved=0>, or unsubscribe<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAFHXE7CDMKCJ5NRHQYJ2C73RHTI4LANCNFSM4K72OIGA&data=02%7C01%7C%7C2263074bc9c546305bd208d7c8e3ba1a%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637198752708411162&sdata=hYnvlM5lYzNdYTUaM%2BpZjC2BzBC1nQEEoCTEq1ZP6eA%3D&reserved=0>.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] MikeThomsen commented on issue #4104: NIFI-7159 - Add
support for some data types in various processors
Posted by GitBox <gi...@apache.org>.
MikeThomsen commented on issue #4104: NIFI-7159 - Add support for some data types in various processors
URL: https://github.com/apache/nifi/pull/4104#issuecomment-599209694
@Xeinn there's a merge conflict now. Instructions follow if you're newish to git. Otherwise ignore.
To resolve do the following:
1. Pull apache/nifi master into your master.
2. Rebase (git rebase master) your branch off of master again.
3. Follow the prompts that git gives you. Be careful to read what it says.
4. git push origin --force BRANCH.
**Don't** git pull apache/nifi master into your branch.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] Xeinn commented on issue #4104: NIFI-7159
Posted by GitBox <gi...@apache.org>.
Xeinn commented on issue #4104: NIFI-7159
URL: https://github.com/apache/nifi/pull/4104#issuecomment-595001978
Sorry, it's 3am here, just read first line and replied... missed rest of content on email(was on my phone) will read through in morning.
Regards
Chris
On 5 Mar 2020 12:06 am, Mike <no...@github.com> wrote:
@MikeThomsen commented on this pull request.
Looks pretty good, but there is definitely some scope creep that will require others' input.
________________________________
In nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttpRecord.java<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fnifi%2Fpull%2F4104%23discussion_r388006141&data=02%7C01%7C%7Ce644906b5c724171b30c08d7c0991093%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637189635942475012&sdata=XWa%2Fpsj%2FaDh6zjpZ%2FPkQeMcP6DdqRQhRsBz%2BowUECEQ%3D&reserved=0>:
> @@ -669,6 +669,9 @@ private void writeValue(final JsonGenerator generator, final Object value, final
}
break;
}
+ case DECIMAL:
I think this should actually be a double because that's the maximum that Elasticsearch supports. See here:
https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.elastic.co%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fmapping-types.html&data=02%7C01%7C%7Ce644906b5c724171b30c08d7c0991093%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637189635942475012&sdata=YHV3U6dbWF7ijTcGE3dGZ0aS9YdugbjqGz7hHH8WEUA%3D&reserved=0>
________________________________
In nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/PutHBaseRecord.java<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fnifi%2Fpull%2F4104%23discussion_r388006605&data=02%7C01%7C%7Ce644906b5c724171b30c08d7c0991093%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637189635942485007&sdata=vYgcjLLPE4yK7WbHMmORO1xcAB8mOPlTsng1Usvv%2B3U%3D&reserved=0>:
> @@ -341,6 +341,8 @@ protected PutFlowFile createPut(ProcessSession session, ProcessContext context,
case BOOLEAN:
retVal = clientService.toBytes(record.getAsBoolean(field));
break;
+ case DECIMAL:
+ // Decimal to be treated as the same as double
It should be broken down into a byte array like the other types.
________________________________
In nifi-nar-bundles/nifi-hive-bundle/nifi-hive3-processors/src/main/java/org/apache/hadoop/hive/ql/io/orc/NiFiOrcUtils.java<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fnifi%2Fpull%2F4104%23discussion_r388006930&data=02%7C01%7C%7Ce644906b5c724171b30c08d7c0991093%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637189635942495005&sdata=TngnYOop4kHdaO7c1PdgTSXptjYjhgo0Nnpr73IyT8o%3D&reserved=0>:
> @@ -104,6 +105,10 @@ public static Object convertToORCObject(TypeInfo typeInfo, Object o, final boole
if (o instanceof Double) {
return new DoubleWritable((double) o);
}
+ // Map BigDecimal to a Double type - this should be improved to map to Hive Decimal type
+ if (o instanceof BigDecimal) {
There were some runtime issues with unit tests related to Orc. @mattyb149<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmattyb149&data=02%7C01%7C%7Ce644906b5c724171b30c08d7c0991093%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637189635942505006&sdata=Uh48wNNGrMxwezJ0oeePBSIONqX0dwztEAcpOVU%2FOEE%3D&reserved=0> @bbende<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fbbende&data=02%7C01%7C%7Ce644906b5c724171b30c08d7c0991093%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637189635942514998&sdata=AbgWHeiA8yuR5lnje6Q7bf21PfGg2QKjkG0krwgLLEc%3D&reserved=0> @ijokarumawak<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fijokarumawak&data=02%7C01%7C%7Ce644906b5c724171b30c08d7c0991093%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637189635942524997&sdata=2DM3oYjo4Se9I0akbixrRG2kW4bYpw4yzWdnA6mEIFY%3D&reserved=0> do y'all have any insight into this ORC conversion?
________________________________
In nifi-nar-bundles/nifi-kudu-bundle/nifi-kudu-controller-service/src/main/java/org/apache/nifi/controller/kudu/KuduLookupService.java<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fnifi%2Fpull%2F4104%23discussion_r388007299&data=02%7C01%7C%7Ce644906b5c724171b30c08d7c0991093%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637189635942534986&sdata=%2Bk9ocVYWhw0TcbIHz5mCcBXnQHyhLSgPy1mive51I9E%3D&reserved=0>:
> @@ -317,7 +317,7 @@ private RecordSchema kuduSchemaToNiFiSchema(Schema kuduTableSchema, List<String>
case BINARY:
case STRING:
case DECIMAL:
- fields.add(new RecordField(cs.getName(), RecordFieldType.STRING.getDataType()));
+ fields.add(new RecordField(cs.getName(), RecordFieldType.DECIMAL.getDataType()));
I don't know anything about Kudu, so I may put this ticket on hold and ask for input from nifi-dev on a few of these other components (ex Hive)
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fnifi%2Fpull%2F4104%3Femail_source%3Dnotifications%26email_token%3DAFHXE7CHGMESCDOAGOUXIWDRF3UIRA5CNFSM4K72OIGKYY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOCYA3SRY%23pullrequestreview-369211719&data=02%7C01%7C%7Ce644906b5c724171b30c08d7c0991093%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637189635942544983&sdata=3W2eDNRRhvMVdVOP9qOfw4pSbtEJvncNuRDHenYYGms%3D&reserved=0>, or unsubscribe<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAFHXE7EJVGYFFGBPKHRJK4LRF3UIRANCNFSM4K72OIGA&data=02%7C01%7C%7Ce644906b5c724171b30c08d7c0991093%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637189635942554981&sdata=SdrT1%2FeIhXaeUKASG%2BMsf9qsCyOXdnve6%2BsF1E0on68%3D&reserved=0>.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] MikeThomsen commented on a change in pull request #4104:
NIFI-7159
Posted by GitBox <gi...@apache.org>.
MikeThomsen commented on a change in pull request #4104: NIFI-7159
URL: https://github.com/apache/nifi/pull/4104#discussion_r388006141
##########
File path: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttpRecord.java
##########
@@ -669,6 +669,9 @@ private void writeValue(final JsonGenerator generator, final Object value, final
}
break;
}
+ case DECIMAL:
Review comment:
I think this should actually be a double because that's the maximum that Elasticsearch supports. See here:
https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] Xeinn commented on issue #4104: NIFI-7159
Posted by GitBox <gi...@apache.org>.
Xeinn commented on issue #4104: NIFI-7159
URL: https://github.com/apache/nifi/pull/4104#issuecomment-597106942
Thanks for the feedback.
Good idea will be making some changes as per Mike’s feedback later in the week so will incorporate the below then.
Regards
Chris
Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10
________________________________
From: tpalfy <no...@github.com>
Sent: Friday, March 6, 2020 7:31:43 PM
To: apache/nifi <ni...@noreply.github.com>
Cc: Chris Manniex <cm...@hotmail.com>; Author <au...@noreply.github.com>
Subject: Re: [apache/nifi] NIFI-7159 (#4104)
@tpalfy requested changes on this pull request.
________________________________
In nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/record/type/DecimalDataType.java<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fnifi%2Fpull%2F4104%23discussion_r389051423&data=02%7C01%7C%7Cb6ae8c600b2441190d0408d7c20500ed%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637191199047595864&sdata=yejHbBKmWv0PfQHtTJFaDrgRy0NCyByfe%2FCSbLI1cJU%3D&reserved=0>:
> + return true;
+ }
+ if (obj == null) {
+ return false;
+ }
+ if (!(obj instanceof DecimalDataType)) {
+ return false;
+ }
+
+ final DecimalDataType other = (DecimalDataType) obj;
+ return precision == other.getPrecision() && scale == other.getScale();
+ }
+
+ @Override
+ public String toString() {
+ return "DECIMAL" + Integer.toString(precision) + Integer.toString(scale);
Might be a good idea to add underscores (for example) so it looks like this:
DECIMAL_11_11
instead of this:
DECIMAL1111
Btw Integer.toString is redundant with the string literal plus '+' operator (makes it a bit less readable).
________________________________
In nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/record/util/DataTypeUtils.java<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fnifi%2Fpull%2F4104%23discussion_r389052612&data=02%7C01%7C%7Cb6ae8c600b2441190d0408d7c20500ed%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637191199047605858&sdata=oAr4Ue8tgmCg0LLB5mi1yghxl9kYgY1OblG9YLjyfTs%3D&reserved=0>:
> @@ -1237,10 +1246,38 @@ public static Double toDouble(final Object value, final String fieldName) {
throw new IllegalTypeConversionException("Cannot convert value [" + value + "] of type " + value.getClass() + " to Double for field " + fieldName);
}
+ public static BigDecimal toBigDecimal(final Object value, final String fieldName) {
+ if (value == null) {
Shouldn't we cover all the Number cases?
I think new BigDecimal(value.toString()); might be enough.
(Can't think of an example where the string representation of a Number would not be good.)
________________________________
In nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/record/RecordFieldType.java<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fnifi%2Fpull%2F4104%23discussion_r389092840&data=02%7C01%7C%7Cb6ae8c600b2441190d0408d7c20500ed%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637191199047605858&sdata=oOrZRC7UnSHPpsuDAe51e6gy2O%2BgHY%2BIe%2Bk1hff0piM%3D&reserved=0>:
> @@ -280,6 +286,25 @@ public DataType getRecordDataType(final RecordSchema childSchema) {
return new RecordDataType(childSchema);
}
+ /**
+ * Returns a data type that represents a DECIMAL value with given precision and scale
+ *
+ * @param precision indicates the expected number of digits before the decimal point
Precision is the number of all digits.
________________________________
In nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/record/util/DataTypeUtils.java<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fnifi%2Fpull%2F4104%23discussion_r389097315&data=02%7C01%7C%7Cb6ae8c600b2441190d0408d7c20500ed%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637191199047615853&sdata=ztQpc0iqHV7rQndHeBKvWxQhosmnOOxINZfGdv910os%3D&reserved=0>:
> @@ -34,6 +34,7 @@
import java.io.InputStream;
import java.io.Reader;
import java.lang.reflect.Array;
+import java.math.BigDecimal;
Currently
DataTypeUtils.findMostSuitableType(
new BigDecimal("11.110"),
Arrays.asList(RecordFieldType.DECIMAL.getDecimalDataType(5, 3)),
Function.identity()
);
correctly returns Optional.of(RecordFieldType.DECIMAL.getDecimalDataType(5, 3));, but
DataTypeUtils.findMostSuitableType(
new BigDecimal("1.110"),
Arrays.asList(RecordFieldType.DECIMAL.getDecimalDataType(5, 3)),
Function.identity()
);
("1.110" instead of "11.111") returns Optional.empty();
This is because the inferred type DECIMAL(4,2) is not equal to the provided DECIMAL(5,2) so the getWiderType method is called which doesn't handle cases where the two types are the same in general, only differ in their details.
Probably the public static Optional<DataType> getWiderType(final DataType thisDataType, final DataType otherDataType) should handle two different DECIMAL types by returning another DECIMAL that is the superset of the two in terms of scale and precision.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fnifi%2Fpull%2F4104%3Femail_source%3Dnotifications%26email_token%3DAFHXE7DKUBOFT4EWJBAGEADRGFFR7A5CNFSM4K72OIGKYY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOCYKZB2Y%23pullrequestreview-370512107&data=02%7C01%7C%7Cb6ae8c600b2441190d0408d7c20500ed%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637191199047615853&sdata=LU88VM5%2Fw%2FMrEzyJUdVu5yOxlkVlwOg2sRdz5JmyvGo%3D&reserved=0>, or unsubscribe<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAFHXE7EJCF5ODMWQD2DE64LRGFFR7ANCNFSM4K72OIGA&data=02%7C01%7C%7Cb6ae8c600b2441190d0408d7c20500ed%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637191199047615853&sdata=f2DOBtkZihGP3GAS%2FKlmBymYTT%2B6lVgI7RWjo74SipY%3D&reserved=0>.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] MikeThomsen commented on a change in pull request #4104:
NIFI-7159
Posted by GitBox <gi...@apache.org>.
MikeThomsen commented on a change in pull request #4104: NIFI-7159
URL: https://github.com/apache/nifi/pull/4104#discussion_r388006605
##########
File path: nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/PutHBaseRecord.java
##########
@@ -341,6 +341,8 @@ protected PutFlowFile createPut(ProcessSession session, ProcessContext context,
case BOOLEAN:
retVal = clientService.toBytes(record.getAsBoolean(field));
break;
+ case DECIMAL:
+ // Decimal to be treated as the same as double
Review comment:
It should be broken down into a byte array like the other types.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] joewitt commented on issue #4104: NIFI-7159 - Add support
for some data types in various processors
Posted by GitBox <gi...@apache.org>.
joewitt commented on issue #4104: NIFI-7159 - Add support for some data types in various processors
URL: https://github.com/apache/nifi/pull/4104#issuecomment-602192509
Please be specific about which tests on windows have timing issues. A ton of effort has gone into the tests on all platforms. A large number of tests were not written to work on windows but none of those that I've found are related to timing issues but instead are due to actual 'test wont work on windows' issues. The issue that occurred in this case appears to be a corrupt zip file. Reran the jobs
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] MikeThomsen commented on a change in pull request #4104:
NIFI-7159
Posted by GitBox <gi...@apache.org>.
MikeThomsen commented on a change in pull request #4104: NIFI-7159
URL: https://github.com/apache/nifi/pull/4104#discussion_r388007299
##########
File path: nifi-nar-bundles/nifi-kudu-bundle/nifi-kudu-controller-service/src/main/java/org/apache/nifi/controller/kudu/KuduLookupService.java
##########
@@ -317,7 +317,7 @@ private RecordSchema kuduSchemaToNiFiSchema(Schema kuduTableSchema, List<String>
case BINARY:
case STRING:
case DECIMAL:
- fields.add(new RecordField(cs.getName(), RecordFieldType.STRING.getDataType()));
+ fields.add(new RecordField(cs.getName(), RecordFieldType.DECIMAL.getDataType()));
Review comment:
I don't know anything about Kudu, so I may put this ticket on hold and ask for input from nifi-dev on a few of these other components (ex Hive)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] tpalfy commented on a change in pull request #4104: NIFI-7159
Posted by GitBox <gi...@apache.org>.
tpalfy commented on a change in pull request #4104: NIFI-7159
URL: https://github.com/apache/nifi/pull/4104#discussion_r389092840
##########
File path: nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/record/RecordFieldType.java
##########
@@ -280,6 +286,25 @@ public DataType getRecordDataType(final RecordSchema childSchema) {
return new RecordDataType(childSchema);
}
+ /**
+ * Returns a data type that represents a DECIMAL value with given precision and scale
+ *
+ * @param precision indicates the expected number of digits before the decimal point
Review comment:
Precision is the number of all digits.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] MikeThomsen commented on issue #4104: NIFI-7159 - Add
support for some data types in various processors
Posted by GitBox <gi...@apache.org>.
MikeThomsen commented on issue #4104: NIFI-7159 - Add support for some data types in various processors
URL: https://github.com/apache/nifi/pull/4104#issuecomment-602196249
> but instead are due to actual 'test wont work on windows' issues.
That's pretty much what I was getting at.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] tpalfy commented on a change in pull request #4104: NIFI-7159
Posted by GitBox <gi...@apache.org>.
tpalfy commented on a change in pull request #4104: NIFI-7159
URL: https://github.com/apache/nifi/pull/4104#discussion_r389052612
##########
File path: nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/record/util/DataTypeUtils.java
##########
@@ -1237,10 +1246,38 @@ public static Double toDouble(final Object value, final String fieldName) {
throw new IllegalTypeConversionException("Cannot convert value [" + value + "] of type " + value.getClass() + " to Double for field " + fieldName);
}
+ public static BigDecimal toBigDecimal(final Object value, final String fieldName) {
+ if (value == null) {
Review comment:
Shouldn't we cover all the `Number` cases?
I think `new BigDecimal(value.toString());` might be enough.
(Can't think of an example where the string representation of a `Number` would not be good.)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] MikeThomsen commented on issue #4104: NIFI-7159
Posted by GitBox <gi...@apache.org>.
MikeThomsen commented on issue #4104: NIFI-7159
URL: https://github.com/apache/nifi/pull/4104#issuecomment-593709915
Yeah, the javadocs are sketchy on that class, but there's gotta be some way where we can look at the driver, find it out if it's `>= 3.8` and then use reflection so that older driver support will still work.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] MikeThomsen commented on issue #4104: NIFI-7159 - Add
support for some data types in various processors
Posted by GitBox <gi...@apache.org>.
MikeThomsen commented on issue #4104: NIFI-7159 - Add support for some data types in various processors
URL: https://github.com/apache/nifi/pull/4104#issuecomment-600106854
As a rule of thumb, if the Linux build passes we can merge. Will try to find some time to review.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] Xeinn commented on issue #4104: NIFI-7159 - Add support for
some data types in various processors
Posted by GitBox <gi...@apache.org>.
Xeinn commented on issue #4104: NIFI-7159 - Add support for some data types in various processors
URL: https://github.com/apache/nifi/pull/4104#issuecomment-602193188
Thanks Joe,
Will look into it. Previous commit had all jobs run fine, after rebasing the issues appeared. Understand that it’s better to be certain of cause.
Regards
Chris
Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10
________________________________
From: Joe Witt <no...@github.com>
Sent: Sunday, March 22, 2020 12:30:24 PM
To: apache/nifi <ni...@noreply.github.com>
Cc: Chris Manniex <cm...@hotmail.com>; Mention <me...@noreply.github.com>
Subject: Re: [apache/nifi] NIFI-7159 - Add support for some data types in various processors (#4104)
Please be specific about which tests on windows have timing issues. A ton of effort has gone into the tests on all platforms. A large number of tests were not written to work on windows but none of those that I've found are related to timing issues but instead are due to actual 'test wont work on windows' issues. The issue that occurred in this case appears to be a corrupt zip file. Reran the jobs
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fnifi%2Fpull%2F4104%23issuecomment-602192509&data=02%7C01%7C%7Ce6d382ee92424a37566108d7ce5ccc03%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637204770255370792&sdata=Ln1DYRH%2FJ%2Fke%2FxRIqc6DKCQfz7U%2FLHXUfLBqpDeLUa0%3D&reserved=0>, or unsubscribe<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAFHXE7HPE3UMHLTBTRV4WVLRIYAGBANCNFSM4K72OIGA&data=02%7C01%7C%7Ce6d382ee92424a37566108d7ce5ccc03%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637204770255380779&sdata=S1gvBDj%2FW%2BxPJuh6nZTl0cd1FL%2FNJ%2BZhML3TVkUd02c%3D&reserved=0>.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] Xeinn commented on issue #4104: NIFI-7159
Posted by GitBox <gi...@apache.org>.
Xeinn commented on issue #4104: NIFI-7159
URL: https://github.com/apache/nifi/pull/4104#issuecomment-595001438
I had thought that might be the case with a change to the core types.
Think there will be some work still to do to allow for good backward compatibility.
What should the next steps be?
Regards
Chris
On 5 Mar 2020 12:06 am, Mike <no...@github.com> wrote:
@MikeThomsen commented on this pull request.
Looks pretty good, but there is definitely some scope creep that will require others' input.
________________________________
In nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttpRecord.java<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fnifi%2Fpull%2F4104%23discussion_r388006141&data=02%7C01%7C%7Ce644906b5c724171b30c08d7c0991093%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637189635942475012&sdata=XWa%2Fpsj%2FaDh6zjpZ%2FPkQeMcP6DdqRQhRsBz%2BowUECEQ%3D&reserved=0>:
> @@ -669,6 +669,9 @@ private void writeValue(final JsonGenerator generator, final Object value, final
}
break;
}
+ case DECIMAL:
I think this should actually be a double because that's the maximum that Elasticsearch supports. See here:
https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.elastic.co%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fmapping-types.html&data=02%7C01%7C%7Ce644906b5c724171b30c08d7c0991093%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637189635942475012&sdata=YHV3U6dbWF7ijTcGE3dGZ0aS9YdugbjqGz7hHH8WEUA%3D&reserved=0>
________________________________
In nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/PutHBaseRecord.java<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fnifi%2Fpull%2F4104%23discussion_r388006605&data=02%7C01%7C%7Ce644906b5c724171b30c08d7c0991093%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637189635942485007&sdata=vYgcjLLPE4yK7WbHMmORO1xcAB8mOPlTsng1Usvv%2B3U%3D&reserved=0>:
> @@ -341,6 +341,8 @@ protected PutFlowFile createPut(ProcessSession session, ProcessContext context,
case BOOLEAN:
retVal = clientService.toBytes(record.getAsBoolean(field));
break;
+ case DECIMAL:
+ // Decimal to be treated as the same as double
It should be broken down into a byte array like the other types.
________________________________
In nifi-nar-bundles/nifi-hive-bundle/nifi-hive3-processors/src/main/java/org/apache/hadoop/hive/ql/io/orc/NiFiOrcUtils.java<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fnifi%2Fpull%2F4104%23discussion_r388006930&data=02%7C01%7C%7Ce644906b5c724171b30c08d7c0991093%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637189635942495005&sdata=TngnYOop4kHdaO7c1PdgTSXptjYjhgo0Nnpr73IyT8o%3D&reserved=0>:
> @@ -104,6 +105,10 @@ public static Object convertToORCObject(TypeInfo typeInfo, Object o, final boole
if (o instanceof Double) {
return new DoubleWritable((double) o);
}
+ // Map BigDecimal to a Double type - this should be improved to map to Hive Decimal type
+ if (o instanceof BigDecimal) {
There were some runtime issues with unit tests related to Orc. @mattyb149<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmattyb149&data=02%7C01%7C%7Ce644906b5c724171b30c08d7c0991093%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637189635942505006&sdata=Uh48wNNGrMxwezJ0oeePBSIONqX0dwztEAcpOVU%2FOEE%3D&reserved=0> @bbende<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fbbende&data=02%7C01%7C%7Ce644906b5c724171b30c08d7c0991093%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637189635942514998&sdata=AbgWHeiA8yuR5lnje6Q7bf21PfGg2QKjkG0krwgLLEc%3D&reserved=0> @ijokarumawak<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fijokarumawak&data=02%7C01%7C%7Ce644906b5c724171b30c08d7c0991093%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637189635942524997&sdata=2DM3oYjo4Se9I0akbixrRG2kW4bYpw4yzWdnA6mEIFY%3D&reserved=0> do y'all have any insight into this ORC conversion?
________________________________
In nifi-nar-bundles/nifi-kudu-bundle/nifi-kudu-controller-service/src/main/java/org/apache/nifi/controller/kudu/KuduLookupService.java<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fnifi%2Fpull%2F4104%23discussion_r388007299&data=02%7C01%7C%7Ce644906b5c724171b30c08d7c0991093%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637189635942534986&sdata=%2Bk9ocVYWhw0TcbIHz5mCcBXnQHyhLSgPy1mive51I9E%3D&reserved=0>:
> @@ -317,7 +317,7 @@ private RecordSchema kuduSchemaToNiFiSchema(Schema kuduTableSchema, List<String>
case BINARY:
case STRING:
case DECIMAL:
- fields.add(new RecordField(cs.getName(), RecordFieldType.STRING.getDataType()));
+ fields.add(new RecordField(cs.getName(), RecordFieldType.DECIMAL.getDataType()));
I don't know anything about Kudu, so I may put this ticket on hold and ask for input from nifi-dev on a few of these other components (ex Hive)
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fnifi%2Fpull%2F4104%3Femail_source%3Dnotifications%26email_token%3DAFHXE7CHGMESCDOAGOUXIWDRF3UIRA5CNFSM4K72OIGKYY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOCYA3SRY%23pullrequestreview-369211719&data=02%7C01%7C%7Ce644906b5c724171b30c08d7c0991093%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637189635942544983&sdata=3W2eDNRRhvMVdVOP9qOfw4pSbtEJvncNuRDHenYYGms%3D&reserved=0>, or unsubscribe<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAFHXE7EJVGYFFGBPKHRJK4LRF3UIRANCNFSM4K72OIGA&data=02%7C01%7C%7Ce644906b5c724171b30c08d7c0991093%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637189635942554981&sdata=SdrT1%2FeIhXaeUKASG%2BMsf9qsCyOXdnve6%2BsF1E0on68%3D&reserved=0>.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] MikeThomsen commented on a change in pull request #4104:
NIFI-7159
Posted by GitBox <gi...@apache.org>.
MikeThomsen commented on a change in pull request #4104: NIFI-7159
URL: https://github.com/apache/nifi/pull/4104#discussion_r388006930
##########
File path: nifi-nar-bundles/nifi-hive-bundle/nifi-hive3-processors/src/main/java/org/apache/hadoop/hive/ql/io/orc/NiFiOrcUtils.java
##########
@@ -104,6 +105,10 @@ public static Object convertToORCObject(TypeInfo typeInfo, Object o, final boole
if (o instanceof Double) {
return new DoubleWritable((double) o);
}
+ // Map BigDecimal to a Double type - this should be improved to map to Hive Decimal type
+ if (o instanceof BigDecimal) {
Review comment:
There were some runtime issues with unit tests related to Orc. @mattyb149 @bbende @ijokarumawak do y'all have any insight into this ORC conversion?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] Xeinn commented on issue #4104: NIFI-7159
Posted by GitBox <gi...@apache.org>.
Xeinn commented on issue #4104: NIFI-7159
URL: https://github.com/apache/nifi/pull/4104#issuecomment-593714060
Hmmm not finding anything that I would class as a reliable source for driver version that’s available from the Mongo Driver.
I may be overlooking something obvious however.
There is the option of a slightly more implied check by verifying the presence of Decimal128?
Will pick up in the morning when I’ve got a bit more coffee in me 😊 thanks for the help so far, I’m sure we’ll have it cracked by tomorrow.
Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10
________________________________
From: Mike <no...@github.com>
Sent: Tuesday, March 3, 2020 12:54:54 AM
To: apache/nifi <ni...@noreply.github.com>
Cc: Chris Manniex <cm...@hotmail.com>; Author <au...@noreply.github.com>
Subject: Re: [apache/nifi] NIFI-7159 (#4104)
Yeah, the javadocs are sketchy on that class, but there's gotta be some way where we can look at the driver, find it out if it's >= 3.8 and then use reflection so that older driver support will still work.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fnifi%2Fpull%2F4104%3Femail_source%3Dnotifications%26email_token%3DAFHXE7DNUOFVMERHDXAS6RDRFRIN5A5CNFSM4K72OIGKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENRUWWY%23issuecomment-593709915&data=02%7C01%7C%7Cb00f111ed1784593059008d7bf0d7dfe%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637187936968703202&sdata=gZolhXppkoQcrBUpKALLpcQ%2BKe8ASo7QAjoL16BsmyU%3D&reserved=0>, or unsubscribe<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAFHXE7GHAMKDIBXIW7W3YQTRFRIN5ANCNFSM4K72OIGA&data=02%7C01%7C%7Cb00f111ed1784593059008d7bf0d7dfe%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637187936968713213&sdata=9nnXRNKpEr7Gd7j61o%2Fi7w2%2Bs0oe1708PJvXHPrEqpk%3D&reserved=0>.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] Xeinn commented on issue #4104: NIFI-7159 - Add support for
some data types in various processors
Posted by GitBox <gi...@apache.org>.
Xeinn commented on issue #4104: NIFI-7159 - Add support for some data types in various processors
URL: https://github.com/apache/nifi/pull/4104#issuecomment-602161276
@MikeThomsen
Hi Mike,
Had a quick look at the unit tests and can see what you mean. Recall having a bit of debate with myself over how to handle this when initially implementing - there may be some scope for improvement.
Basically we have two different areas of "precision & scale"
A schema can define a type with precision and scale but then values manipulated from the records will also have a precision and scale. The DataType represented by DecimalDataType stores the schema defined precision and scale - but the values themselves, internally represented by the BigDecimal type take on the precision and scale of the value held.
So a string of "100.00" assigned to a BigDecimal would have precision of 5 and scale of 2
I guess the question is do we want to enforce the schema defined precision and scale throughout the handling of values and where necessary truncate/round values so they adhere to the schema defined precision and scale?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] MikeThomsen commented on issue #4104: NIFI-7159 - Add
support for some data types in various processors
Posted by GitBox <gi...@apache.org>.
MikeThomsen commented on issue #4104: NIFI-7159 - Add support for some data types in various processors
URL: https://github.com/apache/nifi/pull/4104#issuecomment-602181635
I'll try to find some time today to play around with the APIs to see what validation is even possible. The first thing that comes to mind is whether or not the Avro precision and scale values are not just advice versus hard limits that are validated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] tpalfy commented on a change in pull request #4104: NIFI-7159
Posted by GitBox <gi...@apache.org>.
tpalfy commented on a change in pull request #4104: NIFI-7159
URL: https://github.com/apache/nifi/pull/4104#discussion_r389097315
##########
File path: nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/record/util/DataTypeUtils.java
##########
@@ -34,6 +34,7 @@
import java.io.InputStream;
import java.io.Reader;
import java.lang.reflect.Array;
+import java.math.BigDecimal;
Review comment:
Currently
```java
DataTypeUtils.findMostSuitableType(
new BigDecimal("11.110"),
Arrays.asList(RecordFieldType.DECIMAL.getDecimalDataType(5, 3)),
Function.identity()
);
```
correctly returns `Optional.of(RecordFieldType.DECIMAL.getDecimalDataType(5, 3));`, but
```java
DataTypeUtils.findMostSuitableType(
new BigDecimal("1.110"),
Arrays.asList(RecordFieldType.DECIMAL.getDecimalDataType(5, 3)),
Function.identity()
);
```
("1.110" instead of "11.111") returns `Optional.empty();`
This is because the inferred type DECIMAL(4,2) is not equal to the provided DECIMAL(5,2) so the `getWiderType` method is called which doesn't handle cases where the two types are the same in general, only differ in their details.
Probably the `public static Optional<DataType> getWiderType(final DataType thisDataType, final DataType otherDataType)` should handle two different DECIMAL types by returning another DECIMAL that is the superset of the two in terms of scale and precision.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] Xeinn commented on issue #4104: NIFI-7159 - Add support for
some data types in various processors
Posted by GitBox <gi...@apache.org>.
Xeinn commented on issue #4104: NIFI-7159 - Add support for some data types in various processors
URL: https://github.com/apache/nifi/pull/4104#issuecomment-599916838
@MikeThomsen @tpalfy
Seem to have some issues reported from the Windows build. Not sure if they are related to something I have done or results of commits I've rebased from master.
However I've pushed the changes as suggested above. Would appreciate you checking them again especially the hbase changes.
Question. Should I look to add parameters to affected processors to allow users to select to turn on handling of Decimal type and try where not enabled to maintain current behaviour?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] MikeThomsen commented on issue #4104: NIFI-7159
Posted by GitBox <gi...@apache.org>.
MikeThomsen commented on issue #4104: NIFI-7159
URL: https://github.com/apache/nifi/pull/4104#issuecomment-593677860
Getting build errors now related to Decimal128.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] MikeThomsen commented on issue #4104: NIFI-7159 - Add
support for some data types in various processors
Posted by GitBox <gi...@apache.org>.
MikeThomsen commented on issue #4104: NIFI-7159 - Add support for some data types in various processors
URL: https://github.com/apache/nifi/pull/4104#issuecomment-602191215
@joewitt ok, but we may need to reevaluate some of the tests because some of them have had timing issues on Windows that don't appear on a *NIX
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services