You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Bilal Bektas <Bi...@obase.com> on 2021/07/28 09:35:33 UTC

QueryDatabaseTable - Default Decimal Scale

Hi Dev Team,

"Default Decimal Scale" which is one of the properties of QueryDatabaseTable has worked correctly before NiFi 1.13.x.

Example situation:
[cid:image001.png@01D783A3.5438DE80]

Version

Avro Logical Type (Same DB (MS SQL 2012+) & Same Table & Same Column)

Work Correctly?

12.1

{"name":"occupancy","type":["null",{"type":"bytes","logicalType":"decimal","precision":8,"scale":0}]}

Yes

13.2

{"name":"occupancy","type":["null",{"type":"bytes","logicalType":"decimal","precision":8,"scale":8}]}

No

14.0

{"name":"occupancy","type":["null",{"type":"bytes","logicalType":"decimal","precision":8,"scale":8}]}

No


As you can see situation above, QueryDatabaseTable use "Default Decimal Precision" instead of "Default Decimal Scale".

I have checked nifi source code in github<https://github.com/apache/nifi/blob/7fe8858d22f580ad9dcd96ebafb59dbe5e73f5cb/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/QueryDatabaseTable.java#L127> and I have seen the below situation (defaultScale equals to VARIABLE_REGISTRY_ONLY_DEFAULT_PRECISION instead of VARIABLE_REGISTRY_ONLY_DEFAULT_SCALE):

@Override
protected SqlWriter configureSqlWriter(ProcessSession session, ProcessContext context) {
    final String tableName = context.getProperty(TABLE_NAME).evaluateAttributeExpressions().getValue();
    final boolean convertNamesForAvro = context.getProperty(NORMALIZE_NAMES_FOR_AVRO).asBoolean();
    final Boolean useAvroLogicalTypes = context.getProperty(USE_AVRO_LOGICAL_TYPES).asBoolean();
    final Integer maxRowsPerFlowFile = context.getProperty(MAX_ROWS_PER_FLOW_FILE).evaluateAttributeExpressions().asInteger();
    final Integer defaultPrecision = context.getProperty(VARIABLE_REGISTRY_ONLY_DEFAULT_PRECISION).evaluateAttributeExpressions().asInteger();
    final Integer defaultScale = context.getProperty(VARIABLE_REGISTRY_ONLY_DEFAULT_PRECISION).evaluateAttributeExpressions().asInteger();


Could you help in order to resolve this situation?

Thank you in advance,


obase
TEL: +90216 527 30 00
FAX: +90216 527 31 11
[http://www.obase.com/images/signature/home.png]<http://www.obase.com> [http://www.obase.com/images/signature/facebook.png] <https://www.facebook.com/obasesocial>  [http://www.obase.com/images/signature/twitter.png] <https://twitter.com/obasesocial>  [http://www.obase.com/images/signature/linkedin.png] <https://tr.linkedin.com/in/obase>
[http://www.obase.com/images/signature/obase25tr.png]<http://www.obase.com>

Bu elektronik posta ve onunla iletilen b?t?n dosyalar sadece g?ndericisi tarafindan almasi amaclanan yetkili gercek ya da t?zel kisinin kullanimi icindir. Eger s?z konusu yetkili alici degilseniz bu elektronik postanin icerigini aciklamaniz, kopyalamaniz, y?nlendirmeniz ve kullanmaniz kesinlikle yasaktir ve bu elektronik postayi derhal silmeniz gerekmektedir. OBASE bu mesajin icerdigi bilgilerin do?rulu?u veya eksiksiz oldugu konusunda herhangi bir garanti vermemektedir. Bu nedenle bu bilgilerin ne sekilde olursa olsun iceriginden, iletilmesinden, alinmasindan ve saklanmasindan sorumlu degildir. Bu mesajdaki g?r?sler yalnizca g?nderen kisiye aittir ve OBASE g?r?slerini yansitmayabilir.

Bu e-posta bilinen b?t?n bilgisayar vir?slerine karsi taranmistir.

This e-mail and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you are not the intended recipient you are hereby notified that any dissemination, forwarding, copying or use of any of the information is strictly prohibited, and the e-mail should immediately be deleted. OBASE makes no warranty as to the accuracy or completeness of any information contained in this message and hereby excludes any liability of any kind for the information contained therein or for the information transmission, recepxion, storage or use of such in any way whatsoever. The opinions expressed in this message belong to sender alone and may not necessarily reflect the opinions of OBASE.

This e-mail has been scanned for all known computer viruses.

RE: QueryDatabaseTable - Default Decimal Scale

Posted by Bilal Bektas <Bi...@obase.com>.
Thank you for quick response and solution, Pierre. I will have been waiting fix version in order to update our NiFi systems.

Best wishes,

--Bilal


-----Original Message-----
From: Pierre Villard <pi...@gmail.com> 
Sent: 28 Temmuz 2021 Çarşamba 13:41
To: dev@nifi.apache.org
Subject: Re: QueryDatabaseTable - Default Decimal Scale

Filed https://issues.apache.org/jira/browse/NIFI-8951
and submitted https://github.com/apache/nifi/pull/5256

Thanks,
Pierre

Le mer. 28 juil. 2021 à 12:19, Pierre Villard <pi...@gmail.com> a écrit :

> Hi Bilal,
>
> Your observation is correct, I'll file a JIRA and fix this. Thanks for 
> reporting it.
>
> Thanks,
> Pierre
>
> Le mer. 28 juil. 2021 à 11:36, Bilal Bektas <Bi...@obase.com> a 
> écrit :
>
>> Hi Dev Team,
>>
>>
>>
>> “Default Decimal Scale” which is one of the properties of 
>> QueryDatabaseTable has worked correctly before NiFi 1.13.x.
>>
>>
>>
>> Example situation:
>>
>>
>>
>> *Version*
>>
>> *Avro Logical Type (Same DB (MS SQL 2012+) & Same Table & Same 
>> Column)*
>>
>> *Work Correctly?*
>>
>> 12.1
>>
>>
>> {"name":"occupancy","type":["null",{"type":"bytes","logicalType":"dec
>> imal","precision":8,"scale":0}]}
>>
>> Yes
>>
>> 13.2
>>
>>
>> {"name":"occupancy","type":["null",{"type":"bytes","logicalType":"dec
>> imal","precision":8,
>> "scale":8}]}
>>
>> No
>>
>> 14.0
>>
>>
>> {"name":"occupancy","type":["null",{"type":"bytes","logicalType":"dec
>> imal","precision":8,
>> "scale":8}]}
>>
>> No
>>
>>
>>
>> As you can see situation above, QueryDatabaseTable use "Default 
>> Decimal Precision" instead of “Default Decimal Scale”.
>>
>>
>>
>> I have checked nifi source code in github 
>> <https://github.com/apache/nifi/blob/7fe8858d22f580ad9dcd96ebafb59dbe
>> 5e73f5cb/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processo
>> rs/src/main/java/org/apache/nifi/processors/standard/QueryDatabaseTab
>> le.java#L127> and I have seen the below situation (defaultScale 
>> equals to
>> VARIABLE_REGISTRY_ONLY_DEFAULT_*PRECISION* instead of
>> VARIABLE_REGISTRY_ONLY_DEFAULT_*SCALE*):
>>
>>
>>
>> @Override
>>
>> protected SqlWriter configureSqlWriter*(*ProcessSession session*,* 
>> ProcessContext context*)* *{*
>>
>>     final String tableName *=* context*.*getProperty*(*TABLE_NAME*).*
>> evaluateAttributeExpressions*().*getValue*();*
>>
>>     final boolean convertNamesForAvro *=* context*.*getProperty*(*
>> NORMALIZE_NAMES_FOR_AVRO*).*asBoolean*();*
>>
>>     final Boolean useAvroLogicalTypes *=* context*.*getProperty*(*
>> USE_AVRO_LOGICAL_TYPES*).*asBoolean*();*
>>
>>     final Integer maxRowsPerFlowFile *=* context*.*getProperty*(*
>> MAX_ROWS_PER_FLOW_FILE*).*evaluateAttributeExpressions*().*asInteger*
>> ();*
>>
>>     final Integer defaultPrecision *=* context*.*getProperty*(* 
>> VARIABLE_REGISTRY_ONLY_DEFAULT_PRECISION*).*evaluateAttributeExpressi
>> ons
>> *().*asInteger*();*
>>
>>     final Integer defaultScale *=* context*.*getProperty*(* 
>> VARIABLE_REGISTRY_ONLY_DEFAULT_PRECISION*).*evaluateAttributeExpressi
>> ons
>> *().*asInteger*();*
>>
>>
>>
>> Could you help in order to resolve this situation?
>>
>>
>>
>> Thank you in advance,
>>
>>
>>
>> obase
>> TEL: +90216 527 30 00
>> FAX: +90216 527 31 11
>> <http://www.obase.com> <https://www.facebook.com/obasesocial>
>> <https://twitter.com/obasesocial> <https://tr.linkedin.com/in/obase>
>> <http://www.obase.com>
>>
>> Bu elektronik posta ve onunla iletilen bütün dosyalar sadece 
>> göndericisi tarafindan almasi amaclanan yetkili gercek ya da tüzel 
>> kisinin kullanimi icindir. Eger söz konusu yetkili alici degilseniz 
>> bu elektronik postanin icerigini aciklamaniz, kopyalamaniz, 
>> yönlendirmeniz ve kullanmaniz kesinlikle yasaktir ve bu elektronik postayi derhal silmeniz gerekmektedir.
>> OBASE bu mesajin icerdigi bilgilerin doğruluğu veya eksiksiz oldugu 
>> konusunda herhangi bir garanti vermemektedir. Bu nedenle bu 
>> bilgilerin ne sekilde olursa olsun iceriginden, iletilmesinden, 
>> alinmasindan ve saklanmasindan sorumlu degildir. Bu mesajdaki 
>> görüsler yalnizca gönderen kisiye aittir ve OBASE görüslerini yansitmayabilir.
>>
>> Bu e-posta bilinen bütün bilgisayar virüslerine karsi taranmistir.
>>
>> This e-mail and any files transmitted with it are confidential and 
>> intended solely for the use of the individual or entity to whom they 
>> are addressed. If you are not the intended recipient you are hereby 
>> notified that any dissemination, forwarding, copying or use of any of 
>> the information is strictly prohibited, and the e-mail should 
>> immediately be deleted. OBASE makes no warranty as to the accuracy or 
>> completeness of any information contained in this message and hereby 
>> excludes any liability of any kind for the information contained 
>> therein or for the information transmission, recepxion, storage or 
>> use of such in any way whatsoever. The opinions expressed in this 
>> message belong to sender alone and may not necessarily reflect the opinions of OBASE.
>>
>> This e-mail has been scanned for all known computer viruses.
>>
>

Re: QueryDatabaseTable - Default Decimal Scale

Posted by Pierre Villard <pi...@gmail.com>.
Filed https://issues.apache.org/jira/browse/NIFI-8951
and submitted https://github.com/apache/nifi/pull/5256

Thanks,
Pierre

Le mer. 28 juil. 2021 à 12:19, Pierre Villard <pi...@gmail.com>
a écrit :

> Hi Bilal,
>
> Your observation is correct, I'll file a JIRA and fix this. Thanks for
> reporting it.
>
> Thanks,
> Pierre
>
> Le mer. 28 juil. 2021 à 11:36, Bilal Bektas <Bi...@obase.com> a
> écrit :
>
>> Hi Dev Team,
>>
>>
>>
>> “Default Decimal Scale” which is one of the properties of
>> QueryDatabaseTable has worked correctly before NiFi 1.13.x.
>>
>>
>>
>> Example situation:
>>
>>
>>
>> *Version*
>>
>> *Avro Logical Type (Same DB (MS SQL 2012+) & Same Table & Same Column)*
>>
>> *Work Correctly?*
>>
>> 12.1
>>
>>
>> {"name":"occupancy","type":["null",{"type":"bytes","logicalType":"decimal","precision":8,"scale":0}]}
>>
>> Yes
>>
>> 13.2
>>
>>
>> {"name":"occupancy","type":["null",{"type":"bytes","logicalType":"decimal","precision":8,
>> "scale":8}]}
>>
>> No
>>
>> 14.0
>>
>>
>> {"name":"occupancy","type":["null",{"type":"bytes","logicalType":"decimal","precision":8,
>> "scale":8}]}
>>
>> No
>>
>>
>>
>> As you can see situation above, QueryDatabaseTable use "Default Decimal
>> Precision" instead of “Default Decimal Scale”.
>>
>>
>>
>> I have checked nifi source code in github
>> <https://github.com/apache/nifi/blob/7fe8858d22f580ad9dcd96ebafb59dbe5e73f5cb/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/QueryDatabaseTable.java#L127>
>> and I have seen the below situation (defaultScale equals to
>> VARIABLE_REGISTRY_ONLY_DEFAULT_*PRECISION* instead of
>> VARIABLE_REGISTRY_ONLY_DEFAULT_*SCALE*):
>>
>>
>>
>> @Override
>>
>> protected SqlWriter configureSqlWriter*(*ProcessSession session*,*
>> ProcessContext context*)* *{*
>>
>>     final String tableName *=* context*.*getProperty*(*TABLE_NAME*).*
>> evaluateAttributeExpressions*().*getValue*();*
>>
>>     final boolean convertNamesForAvro *=* context*.*getProperty*(*
>> NORMALIZE_NAMES_FOR_AVRO*).*asBoolean*();*
>>
>>     final Boolean useAvroLogicalTypes *=* context*.*getProperty*(*
>> USE_AVRO_LOGICAL_TYPES*).*asBoolean*();*
>>
>>     final Integer maxRowsPerFlowFile *=* context*.*getProperty*(*
>> MAX_ROWS_PER_FLOW_FILE*).*evaluateAttributeExpressions*().*asInteger*();*
>>
>>     final Integer defaultPrecision *=* context*.*getProperty*(*
>> VARIABLE_REGISTRY_ONLY_DEFAULT_PRECISION*).*evaluateAttributeExpressions
>> *().*asInteger*();*
>>
>>     final Integer defaultScale *=* context*.*getProperty*(*
>> VARIABLE_REGISTRY_ONLY_DEFAULT_PRECISION*).*evaluateAttributeExpressions
>> *().*asInteger*();*
>>
>>
>>
>> Could you help in order to resolve this situation?
>>
>>
>>
>> Thank you in advance,
>>
>>
>>
>> obase
>> TEL: +90216 527 30 00
>> FAX: +90216 527 31 11
>> <http://www.obase.com> <https://www.facebook.com/obasesocial>
>> <https://twitter.com/obasesocial> <https://tr.linkedin.com/in/obase>
>> <http://www.obase.com>
>>
>> Bu elektronik posta ve onunla iletilen bütün dosyalar sadece göndericisi
>> tarafindan almasi amaclanan yetkili gercek ya da tüzel kisinin kullanimi
>> icindir. Eger söz konusu yetkili alici degilseniz bu elektronik postanin
>> icerigini aciklamaniz, kopyalamaniz, yönlendirmeniz ve kullanmaniz
>> kesinlikle yasaktir ve bu elektronik postayi derhal silmeniz gerekmektedir.
>> OBASE bu mesajin icerdigi bilgilerin doğruluğu veya eksiksiz oldugu
>> konusunda herhangi bir garanti vermemektedir. Bu nedenle bu bilgilerin ne
>> sekilde olursa olsun iceriginden, iletilmesinden, alinmasindan ve
>> saklanmasindan sorumlu degildir. Bu mesajdaki görüsler yalnizca gönderen
>> kisiye aittir ve OBASE görüslerini yansitmayabilir.
>>
>> Bu e-posta bilinen bütün bilgisayar virüslerine karsi taranmistir.
>>
>> This e-mail and any files transmitted with it are confidential and
>> intended solely for the use of the individual or entity to whom they are
>> addressed. If you are not the intended recipient you are hereby notified
>> that any dissemination, forwarding, copying or use of any of the
>> information is strictly prohibited, and the e-mail should immediately be
>> deleted. OBASE makes no warranty as to the accuracy or completeness of any
>> information contained in this message and hereby excludes any liability of
>> any kind for the information contained therein or for the information
>> transmission, recepxion, storage or use of such in any way whatsoever. The
>> opinions expressed in this message belong to sender alone and may not
>> necessarily reflect the opinions of OBASE.
>>
>> This e-mail has been scanned for all known computer viruses.
>>
>

Re: QueryDatabaseTable - Default Decimal Scale

Posted by Pierre Villard <pi...@gmail.com>.
Hi Bilal,

Your observation is correct, I'll file a JIRA and fix this. Thanks for
reporting it.

Thanks,
Pierre

Le mer. 28 juil. 2021 à 11:36, Bilal Bektas <Bi...@obase.com> a
écrit :

> Hi Dev Team,
>
>
>
> “Default Decimal Scale” which is one of the properties of
> QueryDatabaseTable has worked correctly before NiFi 1.13.x.
>
>
>
> Example situation:
>
>
>
> *Version*
>
> *Avro Logical Type (Same DB (MS SQL 2012+) & Same Table & Same Column)*
>
> *Work Correctly?*
>
> 12.1
>
>
> {"name":"occupancy","type":["null",{"type":"bytes","logicalType":"decimal","precision":8,"scale":0}]}
>
> Yes
>
> 13.2
>
>
> {"name":"occupancy","type":["null",{"type":"bytes","logicalType":"decimal","precision":8,
> "scale":8}]}
>
> No
>
> 14.0
>
>
> {"name":"occupancy","type":["null",{"type":"bytes","logicalType":"decimal","precision":8,
> "scale":8}]}
>
> No
>
>
>
> As you can see situation above, QueryDatabaseTable use "Default Decimal
> Precision" instead of “Default Decimal Scale”.
>
>
>
> I have checked nifi source code in github
> <https://github.com/apache/nifi/blob/7fe8858d22f580ad9dcd96ebafb59dbe5e73f5cb/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/QueryDatabaseTable.java#L127>
> and I have seen the below situation (defaultScale equals to
> VARIABLE_REGISTRY_ONLY_DEFAULT_*PRECISION* instead of
> VARIABLE_REGISTRY_ONLY_DEFAULT_*SCALE*):
>
>
>
> @Override
>
> protected SqlWriter configureSqlWriter*(*ProcessSession session*,*
> ProcessContext context*)* *{*
>
>     final String tableName *=* context*.*getProperty*(*TABLE_NAME*).*
> evaluateAttributeExpressions*().*getValue*();*
>
>     final boolean convertNamesForAvro *=* context*.*getProperty*(*
> NORMALIZE_NAMES_FOR_AVRO*).*asBoolean*();*
>
>     final Boolean useAvroLogicalTypes *=* context*.*getProperty*(*
> USE_AVRO_LOGICAL_TYPES*).*asBoolean*();*
>
>     final Integer maxRowsPerFlowFile *=* context*.*getProperty*(*
> MAX_ROWS_PER_FLOW_FILE*).*evaluateAttributeExpressions*().*asInteger*();*
>
>     final Integer defaultPrecision *=* context*.*getProperty*(*
> VARIABLE_REGISTRY_ONLY_DEFAULT_PRECISION*).*evaluateAttributeExpressions
> *().*asInteger*();*
>
>     final Integer defaultScale *=* context*.*getProperty*(*
> VARIABLE_REGISTRY_ONLY_DEFAULT_PRECISION*).*evaluateAttributeExpressions
> *().*asInteger*();*
>
>
>
> Could you help in order to resolve this situation?
>
>
>
> Thank you in advance,
>
>
>
> obase
> TEL: +90216 527 30 00
> FAX: +90216 527 31 11
> <http://www.obase.com> <https://www.facebook.com/obasesocial>
> <https://twitter.com/obasesocial> <https://tr.linkedin.com/in/obase>
> <http://www.obase.com>
>
> Bu elektronik posta ve onunla iletilen bütün dosyalar sadece göndericisi
> tarafindan almasi amaclanan yetkili gercek ya da tüzel kisinin kullanimi
> icindir. Eger söz konusu yetkili alici degilseniz bu elektronik postanin
> icerigini aciklamaniz, kopyalamaniz, yönlendirmeniz ve kullanmaniz
> kesinlikle yasaktir ve bu elektronik postayi derhal silmeniz gerekmektedir.
> OBASE bu mesajin icerdigi bilgilerin doğruluğu veya eksiksiz oldugu
> konusunda herhangi bir garanti vermemektedir. Bu nedenle bu bilgilerin ne
> sekilde olursa olsun iceriginden, iletilmesinden, alinmasindan ve
> saklanmasindan sorumlu degildir. Bu mesajdaki görüsler yalnizca gönderen
> kisiye aittir ve OBASE görüslerini yansitmayabilir.
>
> Bu e-posta bilinen bütün bilgisayar virüslerine karsi taranmistir.
>
> This e-mail and any files transmitted with it are confidential and
> intended solely for the use of the individual or entity to whom they are
> addressed. If you are not the intended recipient you are hereby notified
> that any dissemination, forwarding, copying or use of any of the
> information is strictly prohibited, and the e-mail should immediately be
> deleted. OBASE makes no warranty as to the accuracy or completeness of any
> information contained in this message and hereby excludes any liability of
> any kind for the information contained therein or for the information
> transmission, recepxion, storage or use of such in any way whatsoever. The
> opinions expressed in this message belong to sender alone and may not
> necessarily reflect the opinions of OBASE.
>
> This e-mail has been scanned for all known computer viruses.
>