You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Pierre Villard <pi...@gmail.com> on 2021/01/29 18:22:30 UTC
Re: Connectivity issue (Nifi to postgres)
Hi,
We'll need more information like the error message, stack trace, versions
you're using for NiFi / Registry / Postgres, configuration, etc.
Pierre
Le ven. 29 janv. 2021 à 16:48, Arora, Meghna <me...@hpe.com> a
écrit :
> Hello Team,
> We are facing some issue in NIFI connection.
>
>
> JDBC Driver of postgres is not connecting.
> Please let me know how to resolve this issue.
>
>
> Regards
> Meghna
>
RE: Connectivity issue (Nifi to postgres)
Posted by "Arora, Meghna" <me...@hpe.com>.
Hello Pierre,
We have data in a schema and then multiple tables. so we have few doubts on that
1. Which processer is to be best used for bulk load
2. Which processor to be used to Insert update record from one center to another in incremental form.
3. Which process is to be used for update deletion format.
We are having data in 3 different locations (Delhi, Bangalore and Lucknow) and from all locations data has to come to Delhi location. All inserts and update are to be done from Delhi.
We are using nifi of version 1.12.1 and DB is in postgres version 12.12.
Nifi configerations has been attached in onedrive.
https://hpe-my.sharepoint.com/:f:/p/meghna_arora/EmHrg__NKRpKsmPAv_BDG8YBhDYTSz4Dn23MLGxjmgpREg?e=Y4gGF6
Password: Hpe@!23
Right now we test and it is giving some error "ERROR: column "created_on" is of type timestamp without time zone but expression is of type character varying"
Can you please look into it.
Regards
Meghna Arora
[https://attachments.office.net/owa/meghna.arora%40hpe.com/service.svc/s/GetAttachmentThumbnail?id=AAMkADIyNDgzODc5LTU2NjctNDVmNS04ZmViLTQ5NTZlYzc5ODNhOABGAAAAAAC8Ib0GF8fSQKF5%2Fq7H9aFLBwD3XB7kcFaLS6dFCvhJoQB1AAAAAAEPAACkZDZm4whhRrbLHA0d403dAAAAAOslAAABEgAQAO6RTZxOc4hCnN8NvW%2FFw38%3D&thumbnailType=2&token=eyJhbGciOiJSUzI1NiIsImtpZCI6IjMwODE3OUNFNUY0QjUyRTc4QjJEQjg5NjZCQUY0RUNDMzcyN0FFRUUiLCJ0eXAiOiJKV1QiLCJ4NXQiOiJNSUY1emw5TFV1ZUxMYmlXYTY5T3pEY25ydTQifQ.eyJvcmlnaW4iOiJodHRwczovL291dGxvb2sub2ZmaWNlLmNvbSIsInVjIjoiODU3YTllOWZmOTVmNGNiMzhlOTU2MGE2OGFkN2E2YTgiLCJzaWduaW5fc3RhdGUiOiJbXCJrbXNpXCJdIiwidmVyIjoiRXhjaGFuZ2UuQ2FsbGJhY2suVjEiLCJhcHBjdHhzZW5kZXIiOiJPd2FEb3dubG9hZEAxMDViMjA2MS1iNjY5LTRiMzEtOTJhYy0yNGQzMDRkMTk1ZGMiLCJpc3NyaW5nIjoiV1ciLCJhcHBjdHgiOiJ7XCJtc2V4Y2hwcm90XCI6XCJvd2FcIixcInB1aWRcIjpcIjExNTM4MDExMTY2NTAxMzkwNzNcIixcInNjb3BlXCI6XCJPd2FEb3dubG9hZFwiLFwib2lkXCI6XCIzYTNlNWMyMS02MjkyLTQ4ZGYtYjliZS1jOTcyOTUzNjZmNDlcIixcInByaW1hcnlzaWRcIjpcIlMtMS01LTIxLTQxNjM0NTY4MDctNDA3MTQwNTc3NS0yMjI5OTk0MTctODA0OTA0MVwifSIsIm5iZiI6MTYxMzY0NDU3OSwiZXhwIjoxNjEzNjQ1MTc5LCJpc3MiOiIwMDAwMDAwMi0wMDAwLTBmZjEtY2UwMC0wMDAwMDAwMDAwMDBAMTA1YjIwNjEtYjY2OS00YjMxLTkyYWMtMjRkMzA0ZDE5NWRjIiwiYXVkIjoiMDAwMDAwMDItMDAwMC0wZmYxLWNlMDAtMDAwMDAwMDAwMDAwL2F0dGFjaG1lbnRzLm9mZmljZS5uZXRAMTA1YjIwNjEtYjY2OS00YjMxLTkyYWMtMjRkMzA0ZDE5NWRjIiwiaGFwcCI6Im93YSJ9.UsbBPbOidWFW0XPytopAMq4nNSi6MqEyUJrxYAQpWnWWNF8cWJhZ-3vKRiJZKntOIfUcxYLpoDHGKUGE1k7xMUBeCHu0B-Z1nFzmtacNQ2MQaMIAidCHinuPA7X3GKDLQvxx0y7s3XyWBKF_4ERbWiQjc5DOMafmnVv7sEMAatb_6Xb9XVmYmb2fSADE3T_Ovbl_punlWSh-M-Jka2DdQXlGnRIv5_4VePmKBVZjdtPQ7Q1oBmtUjT98F7ZjOEM2FsxxkJ6tTIrwem2yEWGITS_OfcQM8SAdl6i7rMiu-XJuxZGRnmZ2EIPa0T5rY0-8Kdd2RBfLaaNA_TG1lCGWbw&X-OWA-CANARY=zhVPKf0AjkyDT3niYio4MYBFiVv509gYi5xUCf57jDnG_CYlZwgQBlr8sULLakIgD4xpgFtP10I.&owa=outlook.office.com&scriptVer=20210208002.03&animation=true]
From: Pierre Villard <pi...@gmail.com>
Sent: Tuesday, February 16, 2021 5:43 PM
To: Arora, Meghna <me...@hpe.com>
Cc: dev@nifi.apache.org; Poddar, Rakesh Kumar <ra...@hpe.com>; Bhattacharjee, Koushik <ko...@hpe.com>; Kiri, Aarti <aa...@hpe.com>
Subject: Re: Connectivity issue (Nifi to postgres)
Hi,
As said before, we need more information. We don't see the images you're adding to your messages and I don't recommend attaching zip files to messages here. Can please add your logs/images/etc on something like Gist by Github [1]. Also, please describe the use case, the processors you are using, the configuration of the processors, what is your flow, NiFi version you have, how many nodes, etc. The more info you give, the more likely someone would be willing to spend some of their time to help you.
Thanks,
Pierre
[1] https://gist.github.com/
Le mar. 16 févr. 2021 à 16:04, Arora, Meghna <me...@hpe.com>> a écrit :
Hello Team,
Any update on this.
We have to process 3TB records and an incremental value of 1 lakh records from source to destination. we are getting following error.
LOGS:
2021-02-16 15:48:48,496 ERROR [Cleanup Archive for default] o.a.n.c.repository.FileSystemRepository
java.lang.OutOfMemoryError: GC overhead limit exceeded
2021-02-16 15:49:55,755 ERROR [Framework Task Thread Thread-1] o.a.n.c.s.StandardProcessScheduler Failed to run Framework Task Expire FlowFiles due to java.lang.OutOfMemoryError: GC overhead limit exceeded
As an option I am trying to explore Query Record to solve problem. Is it correct?
Regards
Meghna
________________________________
From: Arora, Meghna
Sent: Thursday, February 4, 2021 6:22 PM
To: dev@nifi.apache.org<ma...@nifi.apache.org> <de...@nifi.apache.org>>; pierre.villard.fr@gmail.com<ma...@gmail.com> <pi...@gmail.com>>
Cc: Poddar, Rakesh Kumar <ra...@hpe.com>>; Bhattacharjee, Koushik <ko...@hpe.com>>
Subject: RE: Connectivity issue (Nifi to postgres)
Hello Pierre,
Logs has been attached for same.
Regards
Meghna Arora
[https://attachments.office.net/owa/meghna.arora%40hpe.com/service.svc/s/GetAttachmentThumbnail?id=AAMkADIyNDgzODc5LTU2NjctNDVmNS04ZmViLTQ5NTZlYzc5ODNhOABGAAAAAAC8Ib0GF8fSQKF5%2Fq7H9aFLBwD3XB7kcFaLS6dFCvhJoQB1AAAAAAEPAACkZDZm4whhRrbLHA0d403dAAAAAOslAAABEgAQAO6RTZxOc4hCnN8NvW%2FFw38%3D&thumbnailType=2&token=eyJhbGciOiJSUzI1NiIsImtpZCI6IjMwODE3OUNFNUY0QjUyRTc4QjJEQjg5NjZCQUY0RUNDMzcyN0FFRUUiLCJ0eXAiOiJKV1QiLCJ4NXQiOiJNSUY1emw5TFV1ZUxMYmlXYTY5T3pEY25ydTQifQ.eyJvcmlnaW4iOiJodHRwczovL291dGxvb2sub2ZmaWNlLmNvbSIsInVjIjoiODU3YTllOWZmOTVmNGNiMzhlOTU2MGE2OGFkN2E2YTgiLCJzaWduaW5fc3RhdGUiOiJbXCJrbXNpXCJdIiwidmVyIjoiRXhjaGFuZ2UuQ2FsbGJhY2suVjEiLCJhcHBjdHhzZW5kZXIiOiJPd2FEb3dubG9hZEAxMDViMjA2MS1iNjY5LTRiMzEtOTJhYy0yNGQzMDRkMTk1ZGMiLCJpc3NyaW5nIjoiV1ciLCJhcHBjdHgiOiJ7XCJtc2V4Y2hwcm90XCI6XCJvd2FcIixcInB1aWRcIjpcIjExNTM4MDExMTY2NTAxMzkwNzNcIixcInNjb3BlXCI6XCJPd2FEb3dubG9hZFwiLFwib2lkXCI6XCIzYTNlNWMyMS02MjkyLTQ4ZGYtYjliZS1jOTcyOTUzNjZmNDlcIixcInByaW1hcnlzaWRcIjpcIlMtMS01LTIxLTQxNjM0NTY4MDctNDA3MTQwNTc3NS0yMjI5OTk0MTctODA0OTA0MVwifSIsIm5iZiI6MTYxMzY0NDU3OSwiZXhwIjoxNjEzNjQ1MTc5LCJpc3MiOiIwMDAwMDAwMi0wMDAwLTBmZjEtY2UwMC0wMDAwMDAwMDAwMDBAMTA1YjIwNjEtYjY2OS00YjMxLTkyYWMtMjRkMzA0ZDE5NWRjIiwiYXVkIjoiMDAwMDAwMDItMDAwMC0wZmYxLWNlMDAtMDAwMDAwMDAwMDAwL2F0dGFjaG1lbnRzLm9mZmljZS5uZXRAMTA1YjIwNjEtYjY2OS00YjMxLTkyYWMtMjRkMzA0ZDE5NWRjIiwiaGFwcCI6Im93YSJ9.UsbBPbOidWFW0XPytopAMq4nNSi6MqEyUJrxYAQpWnWWNF8cWJhZ-3vKRiJZKntOIfUcxYLpoDHGKUGE1k7xMUBeCHu0B-Z1nFzmtacNQ2MQaMIAidCHinuPA7X3GKDLQvxx0y7s3XyWBKF_4ERbWiQjc5DOMafmnVv7sEMAatb_6Xb9XVmYmb2fSADE3T_Ovbl_punlWSh-M-Jka2DdQXlGnRIv5_4VePmKBVZjdtPQ7Q1oBmtUjT98F7ZjOEM2FsxxkJ6tTIrwem2yEWGITS_OfcQM8SAdl6i7rMiu-XJuxZGRnmZ2EIPa0T5rY0-8Kdd2RBfLaaNA_TG1lCGWbw&X-OWA-CANARY=zhVPKf0AjkyDT3niYio4MYBFiVv509gYi5xUCf57jDnG_CYlZwgQBlr8sULLakIgD4xpgFtP10I.&owa=outlook.office.com&scriptVer=20210208002.03&animation=true]
________________________________
From: Pierre Villard <pi...@gmail.com>>
Sent: 29 January 2021 23:52
To: dev@nifi.apache.org<ma...@nifi.apache.org> <de...@nifi.apache.org>>
Cc: issues@nifi.apache.org<ma...@nifi.apache.org> <is...@nifi.apache.org>>; users@nifi.apache.org<ma...@nifi.apache.org> <us...@nifi.apache.org>>; Bhattacharjee, Koushik <ko...@hpe.com>>; Poddar, Rakesh Kumar <ra...@hpe.com>>
Subject: Re: Connectivity issue (Nifi to postgres)
Hi,
We'll need more information like the error message, stack trace, versions you're using for NiFi / Registry / Postgres, configuration, etc.
Pierre
Le ven. 29 janv. 2021 à 16:48, Arora, Meghna <me...@hpe.com>> a écrit :
Hello Team,
We are facing some issue in NIFI connection.
JDBC Driver of postgres is not connecting.
Please let me know how to resolve this issue.
Regards
Meghna
RE: Connectivity issue (Nifi to postgres)
Posted by "Arora, Meghna" <me...@hpe.com>.
Please review.
Regards
Meghna Arora
[cid:image001.png@01D7048C.27598630]
From: Pierre Villard <pi...@gmail.com>
Sent: Tuesday, February 16, 2021 5:43 PM
To: Arora, Meghna <me...@hpe.com>
Cc: dev@nifi.apache.org; Poddar, Rakesh Kumar <ra...@hpe.com>; Bhattacharjee, Koushik <ko...@hpe.com>; Kiri, Aarti <aa...@hpe.com>
Subject: Re: Connectivity issue (Nifi to postgres)
Hi,
As said before, we need more information. We don't see the images you're adding to your messages and I don't recommend attaching zip files to messages here. Can please add your logs/images/etc on something like Gist by Github [1]. Also, please describe the use case, the processors you are using, the configuration of the processors, what is your flow, NiFi version you have, how many nodes, etc. The more info you give, the more likely someone would be willing to spend some of their time to help you.
Thanks,
Pierre
[1] https://gist.github.com/
Le mar. 16 févr. 2021 à 16:04, Arora, Meghna <me...@hpe.com>> a écrit :
Hello Team,
Any update on this.
We have to process 3TB records and an incremental value of 1 lakh records from source to destination. we are getting following error.
LOGS:
2021-02-16 15:48:48,496 ERROR [Cleanup Archive for default] o.a.n.c.repository.FileSystemRepository
java.lang.OutOfMemoryError: GC overhead limit exceeded
2021-02-16 15:49:55,755 ERROR [Framework Task Thread Thread-1] o.a.n.c.s.StandardProcessScheduler Failed to run Framework Task Expire FlowFiles due to java.lang.OutOfMemoryError: GC overhead limit exceeded
As an option I am trying to explore Query Record to solve problem. Is it correct?
Regards
Meghna
________________________________
From: Arora, Meghna
Sent: Thursday, February 4, 2021 6:22 PM
To: dev@nifi.apache.org<ma...@nifi.apache.org> <de...@nifi.apache.org>>; pierre.villard.fr@gmail.com<ma...@gmail.com> <pi...@gmail.com>>
Cc: Poddar, Rakesh Kumar <ra...@hpe.com>>; Bhattacharjee, Koushik <ko...@hpe.com>>
Subject: RE: Connectivity issue (Nifi to postgres)
Hello Pierre,
Logs has been attached for same.
Regards
Meghna Arora
[cid:image001.png@01D7048C.27598630]
________________________________
From: Pierre Villard <pi...@gmail.com>>
Sent: 29 January 2021 23:52
To: dev@nifi.apache.org<ma...@nifi.apache.org> <de...@nifi.apache.org>>
Cc: issues@nifi.apache.org<ma...@nifi.apache.org> <is...@nifi.apache.org>>; users@nifi.apache.org<ma...@nifi.apache.org> <us...@nifi.apache.org>>; Bhattacharjee, Koushik <ko...@hpe.com>>; Poddar, Rakesh Kumar <ra...@hpe.com>>
Subject: Re: Connectivity issue (Nifi to postgres)
Hi,
We'll need more information like the error message, stack trace, versions you're using for NiFi / Registry / Postgres, configuration, etc.
Pierre
Le ven. 29 janv. 2021 à 16:48, Arora, Meghna <me...@hpe.com>> a écrit :
Hello Team,
We are facing some issue in NIFI connection.
JDBC Driver of postgres is not connecting.
Please let me know how to resolve this issue.
Regards
Meghna
Re: Connectivity issue (Nifi to postgres)
Posted by Pierre Villard <pi...@gmail.com>.
Hi,
As said before, we need more information. We don't see the images you're
adding to your messages and I don't recommend attaching zip files to
messages here. Can please add your logs/images/etc on something like Gist
by Github [1]. Also, please describe the use case, the processors you are
using, the configuration of the processors, what is your flow, NiFi version
you have, how many nodes, etc. The more info you give, the more likely
someone would be willing to spend some of their time to help you.
Thanks,
Pierre
[1] https://gist.github.com/
Le mar. 16 févr. 2021 à 16:04, Arora, Meghna <me...@hpe.com> a
écrit :
> Hello Team,
> Any update on this.
> We have to process 3TB records and an incremental value of 1 lakh records
> from source to destination. we are getting following error.
> LOGS:
> *2021-02-16 15:48:48,496 ERROR [Cleanup Archive for default]
> o.a.n.c.repository.FileSystemRepository*
> *java.lang.OutOfMemoryError: GC overhead limit exceeded*
> *2021-02-16 15:49:55,755 ERROR [Framework Task Thread Thread-1]
> o.a.n.c.s.StandardProcessScheduler Failed to run Framework Task Expire
> FlowFiles due to java.lang.OutOfMemoryError: GC overhead limit exceeded*
>
> As an option I am trying to explore Query Record to solve problem. Is it
> correct?
>
> Regards
> Meghna
>
>
> ------------------------------
> *From:* Arora, Meghna
> *Sent:* Thursday, February 4, 2021 6:22 PM
> *To:* dev@nifi.apache.org <de...@nifi.apache.org>;
> pierre.villard.fr@gmail.com <pi...@gmail.com>
> *Cc:* Poddar, Rakesh Kumar <ra...@hpe.com>; Bhattacharjee,
> Koushik <ko...@hpe.com>
> *Subject:* RE: Connectivity issue (Nifi to postgres)
>
>
> Hello Pierre,
>
> Logs has been attached for same.
>
>
>
> Regards
>
> Meghna Arora
>
>
> ------------------------------
>
> *From:* Pierre Villard <pi...@gmail.com>
> *Sent:* 29 January 2021 23:52
> *To:* dev@nifi.apache.org <de...@nifi.apache.org>
> *Cc:* issues@nifi.apache.org <is...@nifi.apache.org>;
> users@nifi.apache.org <us...@nifi.apache.org>; Bhattacharjee, Koushik <
> koushik.bhattacharjee@hpe.com>; Poddar, Rakesh Kumar <
> rakesh-kumar.poddar@hpe.com>
> *Subject:* Re: Connectivity issue (Nifi to postgres)
>
>
>
> Hi,
>
>
>
> We'll need more information like the error message, stack trace, versions
> you're using for NiFi / Registry / Postgres, configuration, etc.
>
>
>
> Pierre
>
>
>
> Le ven. 29 janv. 2021 à 16:48, Arora, Meghna <me...@hpe.com> a
> écrit :
>
> Hello Team,
>
> We are facing some issue in NIFI connection.
>
>
>
>
>
> JDBC Driver of postgres is not connecting.
>
> Please let me know how to resolve this issue.
>
>
>
>
>
> Regards
>
> Meghna
>
>
Re: Connectivity issue (Nifi to postgres)
Posted by "Arora, Meghna" <me...@hpe.com>.
Hello Team,
Any update on this.
We have to process 3TB records and an incremental value of 1 lakh records from source to destination. we are getting following error.
LOGS:
2021-02-16 15:48:48,496 ERROR [Cleanup Archive for default] o.a.n.c.repository.FileSystemRepository
java.lang.OutOfMemoryError: GC overhead limit exceeded
2021-02-16 15:49:55,755 ERROR [Framework Task Thread Thread-1] o.a.n.c.s.StandardProcessScheduler Failed to run Framework Task Expire FlowFiles due to java.lang.OutOfMemoryError: GC overhead limit exceeded
As an option I am trying to explore Query Record to solve problem. Is it correct?
Regards
Meghna
________________________________
From: Arora, Meghna
Sent: Thursday, February 4, 2021 6:22 PM
To: dev@nifi.apache.org <de...@nifi.apache.org>; pierre.villard.fr@gmail.com <pi...@gmail.com>
Cc: Poddar, Rakesh Kumar <ra...@hpe.com>; Bhattacharjee, Koushik <ko...@hpe.com>
Subject: RE: Connectivity issue (Nifi to postgres)
Hello Pierre,
Logs has been attached for same.
Regards
Meghna Arora
[cid:image001.png@01D6FB0D.C48CB270]
________________________________
From: Pierre Villard <pi...@gmail.com>>
Sent: 29 January 2021 23:52
To: dev@nifi.apache.org<ma...@nifi.apache.org> <de...@nifi.apache.org>>
Cc: issues@nifi.apache.org<ma...@nifi.apache.org> <is...@nifi.apache.org>>; users@nifi.apache.org<ma...@nifi.apache.org> <us...@nifi.apache.org>>; Bhattacharjee, Koushik <ko...@hpe.com>>; Poddar, Rakesh Kumar <ra...@hpe.com>>
Subject: Re: Connectivity issue (Nifi to postgres)
Hi,
We'll need more information like the error message, stack trace, versions you're using for NiFi / Registry / Postgres, configuration, etc.
Pierre
Le ven. 29 janv. 2021 à 16:48, Arora, Meghna <me...@hpe.com>> a écrit :
Hello Team,
We are facing some issue in NIFI connection.
JDBC Driver of postgres is not connecting.
Please let me know how to resolve this issue.
Regards
Meghna
RE: Connectivity issue (Nifi to postgres)
Posted by "Arora, Meghna" <me...@hpe.com>.
Hello Pierre,
Logs has been attached for same.
Regards
Meghna Arora
[cid:image001.png@01D6FB0D.C48CB270]
________________________________
From: Pierre Villard <pi...@gmail.com>>
Sent: 29 January 2021 23:52
To: dev@nifi.apache.org<ma...@nifi.apache.org> <de...@nifi.apache.org>>
Cc: issues@nifi.apache.org<ma...@nifi.apache.org> <is...@nifi.apache.org>>; users@nifi.apache.org<ma...@nifi.apache.org> <us...@nifi.apache.org>>; Bhattacharjee, Koushik <ko...@hpe.com>>; Poddar, Rakesh Kumar <ra...@hpe.com>>
Subject: Re: Connectivity issue (Nifi to postgres)
Hi,
We'll need more information like the error message, stack trace, versions you're using for NiFi / Registry / Postgres, configuration, etc.
Pierre
Le ven. 29 janv. 2021 à 16:48, Arora, Meghna <me...@hpe.com>> a écrit :
Hello Team,
We are facing some issue in NIFI connection.
JDBC Driver of postgres is not connecting.
Please let me know how to resolve this issue.
Regards
Meghna