You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "tpalfy (via GitHub)" <gi...@apache.org> on 2023/01/27 15:56:12 UTC
[GitHub] [nifi] tpalfy opened a new pull request, #6900: NIFI-11107 In ConsumeIMAP and ConsumePOP3 added support for OAUTH based authorization.
tpalfy opened a new pull request, #6900:
URL: https://github.com/apache/nifi/pull/6900
<!-- Licensed to the Apache Software Foundation (ASF) under one or more -->
<!-- contributor license agreements. See the NOTICE file distributed with -->
<!-- this work for additional information regarding copyright ownership. -->
<!-- The ASF licenses this file to You under the Apache License, Version 2.0 -->
<!-- (the "License"); you may not use this file except in compliance with -->
<!-- the License. You may obtain a copy of the License at -->
<!-- http://www.apache.org/licenses/LICENSE-2.0 -->
<!-- Unless required by applicable law or agreed to in writing, software -->
<!-- distributed under the License is distributed on an "AS IS" BASIS, -->
<!-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -->
<!-- See the License for the specific language governing permissions and -->
<!-- limitations under the License. -->
# Summary
[NIFI-11107](https://issues.apache.org/jira/browse/NIFI-11107)
# Tracking
Please complete the following tracking steps prior to pull request creation.
### Issue Tracking
- [ ] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created
### Pull Request Tracking
- [ ] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-00000`
- [ ] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-00000`
### Pull Request Formatting
- [ ] Pull Request based on current revision of the `main` branch
- [ ] Pull Request refers to a feature branch with one commit containing changes
# Verification
Please indicate the verification steps performed prior to pull request creation.
### Build
- [ ] Build completed using `mvn clean install -P contrib-check`
- [ ] JDK 8
- [ ] JDK 11
- [ ] JDK 17
### Licensing
- [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html)
- [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files
### Documentation
- [ ] Documentation formatting appears as expected in rendered files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11107 In ConsumeIMAP and ConsumePOP3 added support for OAUTH based authorization. [nifi]
Posted by "tpalfy (via GitHub)" <gi...@apache.org>.
tpalfy commented on PR #6900:
URL: https://github.com/apache/nifi/pull/6900#issuecomment-1981377674
@AnTu2702 Not sure about this new approach.
I'm not completely against it but I feel like rebuilding only when we get an authentication error would be more porper.
Not trivial as it would require some refactoring but I think it could be worth it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11107 In ConsumeIMAP and ConsumePOP3 added support for OAUTH based authorization. [nifi]
Posted by "AnTu2702 (via GitHub)" <gi...@apache.org>.
AnTu2702 commented on PR #6900:
URL: https://github.com/apache/nifi/pull/6900#issuecomment-1993902221
> @AnTu2702 I think I can add a not too complicated change that only recreates the `messageReciever` when an authentication exception is encountered in a couple of days, hopefully today.
>
> Since there are connections involved and given the fact that onTriggers can be called very frequently, it's better to have it like this right away.
Hm... Not sure about this...
@tpalfy Our access tokens e.g. are valid for 60 minutes only.
Due to my understanding, the StandardOauth2AccessTokenProvider.java is intentionally written such that an Authentication Exception shall never occur at all! It will automatically refresh any token, BEFORE it will expire (using the refresh window) by just asking it for "the access token". It will always be valid this way...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11107 In ConsumeIMAP and ConsumePOP3 added support for OAUTH based authorization. [nifi]
Posted by "tpalfy (via GitHub)" <gi...@apache.org>.
tpalfy commented on PR #6900:
URL: https://github.com/apache/nifi/pull/6900#issuecomment-1988565123
@AnTu2702 I think I can add a not too complicated change that only recreates the `messageReciever` when an authentication exception is encountered in a couple of days, hopefully today.
Since there are connections involved and given the fact that onTriggers can be called very frequently, it's better to have it like this right away.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11107 In ConsumeIMAP and ConsumePOP3 added support for OAUTH based authorization. [nifi]
Posted by "tpalfy (via GitHub)" <gi...@apache.org>.
tpalfy commented on PR #6900:
URL: https://github.com/apache/nifi/pull/6900#issuecomment-1994553665
> Hm... Sorry, but I have doubts about this...
>
> @tpalfy Our access tokens e.g. are valid for 60 minutes only. Due to my understanding, the StandardOauth2AccessTokenProvider.java is intentionally written such that an authentication exception shall never occur at all! It will always automatically refresh any token, BEFORE it will expire (using the refresh window) by just asking it for "the access token" from outside calling `getAccessDetail()`. It will always be valid this way...
>
> So an authentication exception should IMHO never be intended to occur at all...
>
> ...
>
> But simply to have a quickfix für NIFI 1.26 still better than nothing. ;-)
Okay, that's fair. I have a PR up with my proposed changes here: https://github.com/apache/nifi/pull/8494
In it's first state it only recreates the mail receiver if an error is encountered. I'm going to update it by keeping the first commit, adding a revert commit and adding a new one that checks if the token has been refreshed. So both approach can be checked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11107 In ConsumeIMAP and ConsumePOP3 added support for OAUTH based authorization. [nifi]
Posted by "AnTu2702 (via GitHub)" <gi...@apache.org>.
AnTu2702 commented on PR #6900:
URL: https://github.com/apache/nifi/pull/6900#issuecomment-1978198291
> @AnTu2702 The `oauth2AccessTokenProviderOptional` field is used in the `buildUrl` method where we call `oauth2AccessTokenProvider.getAccessDetails().getAccessToken()`. The `getAccessDetail()` should take care of refreshing/reacquiring tokens (this is the same call you suggest doing at the beginning of `onTrigger`).
>
> So according to this, token expiration is handled already. Do we have proof that this doesn't work for some reason?
Hey. The `this.messageReceiver = this.buildMessageReceiver(context)` and thus `buildUrl` method in line 331 is only called once. Reason is line 329: `if (this.messageReceiver == null)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11107 In ConsumeIMAP and ConsumePOP3 added support for OAUTH based authorization. [nifi]
Posted by "AnTu2702 (via GitHub)" <gi...@apache.org>.
AnTu2702 commented on PR #6900:
URL: https://github.com/apache/nifi/pull/6900#issuecomment-1993901612
Hm... Not sure about this...
Our access tokens e.g. are valid for 60 minutes only.
Due to my understanding, the StandardOauth2AccessTokenProvider.java is intentionally written such that an Authentication Exception shall never occur at all! It will automatically refresh any token, BEFORE it will expire (using the refresh window) by just asking it for "the access token". It will always be valid this way...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11107 In ConsumeIMAP and ConsumePOP3 added support for OAUTH based authorization. [nifi]
Posted by "AnTu2702 (via GitHub)" <gi...@apache.org>.
AnTu2702 commented on PR #6900:
URL: https://github.com/apache/nifi/pull/6900#issuecomment-1975971033
I debugged the AbstractEmailProcessor.java and it seems, that AbstractEmailProcessor.java does not handle token expiration at all. The StandardOAuth2AccessTokenProvider.Java does, but it would need the AbstractEmailProcessor.java to call the:
oauth2AccessTokenProvider.getAccessDetails()
in the onTrigger() method instead the onSchedule() method to do so.
The AbstractEmailProcessor.java should be fixed to handle expired OAuth2 access tokens!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11107 In ConsumeIMAP and ConsumePOP3 added support for OAUTH based authorization. [nifi]
Posted by "AnTu2702 (via GitHub)" <gi...@apache.org>.
AnTu2702 commented on PR #6900:
URL: https://github.com/apache/nifi/pull/6900#issuecomment-1976176344
Here would be my suggestion for a fix:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11107 In ConsumeIMAP and ConsumePOP3 added support for OAUTH based authorization. [nifi]
Posted by "tpalfy (via GitHub)" <gi...@apache.org>.
tpalfy commented on PR #6900:
URL: https://github.com/apache/nifi/pull/6900#issuecomment-1978692871
Thanks @AnTu2702 and @RomanOttGmx for the explanaion.
As for your fix @AnTu2702 I'm a bit confused though. If we only refresh the token but don't rebuild the url I don't see how that would work. Yes, we have a fresh token but are we actually using it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11107 In ConsumeIMAP and ConsumePOP3 added support for OAUTH based authorization. [nifi]
Posted by "tpalfy (via GitHub)" <gi...@apache.org>.
tpalfy commented on PR #6900:
URL: https://github.com/apache/nifi/pull/6900#issuecomment-1976510039
@AnTu2702 The `oauth2AccessTokenProviderOptional` field is used exclusively in the `buildUrl` method where we call `oauth2AccessTokenProvider.getAccessDetails().getAccessToken()`. The `getAccessDetail()` should take care of refreshing/reacquiring tokens (this is the same call you suggest doing at the beginning of `onTrigger`).
So according to this token expiration is handled already. Do we have proof that this doesn't work for some reason?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11107 In ConsumeIMAP and ConsumePOP3 added support for OAUTH based authorization. [nifi]
Posted by "AnTu2702 (via GitHub)" <gi...@apache.org>.
AnTu2702 commented on PR #6900:
URL: https://github.com/apache/nifi/pull/6900#issuecomment-1981009647
@tpalfy @RomanOttGmx I tried a different solution that does seem to do the job:
It is a bit ugly, because the `initializeIfNecessary method` will this way reinitialize the `messageRecevier` for OAuth2 based access on every `trigger()` call.
What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11107 In ConsumeIMAP and ConsumePOP3 added support for OAUTH based authorization. [nifi]
Posted by "AnTu2702 (via GitHub)" <gi...@apache.org>.
AnTu2702 commented on PR #6900:
URL: https://github.com/apache/nifi/pull/6900#issuecomment-1978169106
Overview:
ConsumeImap:
StandardOAuth2TokenProvider:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] turcsanyip commented on pull request #6900: NIFI-11107 In ConsumeIMAP and ConsumePOP3 added support for OAUTH based authorization.
Posted by "turcsanyip (via GitHub)" <gi...@apache.org>.
turcsanyip commented on PR #6900:
URL: https://github.com/apache/nifi/pull/6900#issuecomment-1415833885
Thanks for your review and test @nandorsoma!
Merging to main...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] asfgit closed pull request #6900: NIFI-11107 In ConsumeIMAP and ConsumePOP3 added support for OAUTH based authorization.
Posted by "asfgit (via GitHub)" <gi...@apache.org>.
asfgit closed pull request #6900: NIFI-11107 In ConsumeIMAP and ConsumePOP3 added support for OAUTH based authorization.
URL: https://github.com/apache/nifi/pull/6900
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11107 In ConsumeIMAP and ConsumePOP3 added support for OAUTH based authorization. [nifi]
Posted by "RomanOttGmx (via GitHub)" <gi...@apache.org>.
RomanOttGmx commented on PR #6900:
URL: https://github.com/apache/nifi/pull/6900#issuecomment-1977185841
In IMAP Processor buildUrl will be triggered on creation of new receiver.
```
`protected ImapMailReceiver buildMessageReceiver(ProcessContext processContext) {
ImapMailReceiver receiver = new ImapMailReceiver(this.buildUrl(processContext));`
```
And the buildMessageReceiver is requested here in AbstractEmailProcessor
```
private synchronized void initializeIfNecessary(ProcessContext context, ProcessSession processSession) {
if (this.messageReceiver == null) {
this.processSession = processSession;
this.messageReceiver = this.buildMessageReceiver(context);
```
This is called in onTrigger method of AbstractEmailProcessor, but the receiver still exists since start up, so buildUrl will never be requested again after enabling the ConsumeImapProcessor.
```
public void onTrigger(ProcessContext context, ProcessSession processSession) throws ProcessException {
this.initializeIfNecessary(context, processSession);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11107 In ConsumeIMAP and ConsumePOP3 added support for OAUTH based authorization. [nifi]
Posted by "AnTu2702 (via GitHub)" <gi...@apache.org>.
AnTu2702 commented on PR #6900:
URL: https://github.com/apache/nifi/pull/6900#issuecomment-1981577552
How about a quick bugfix using this new approacj for NIFI 1.26 and then refactoring it afterwards?
We really really depend on having that fixed in the next couple of weeks. 😕
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11107 In ConsumeIMAP and ConsumePOP3 added support for OAUTH based authorization. [nifi]
Posted by "RomanOttGmx (via GitHub)" <gi...@apache.org>.
RomanOttGmx commented on PR #6900:
URL: https://github.com/apache/nifi/pull/6900#issuecomment-1977172784
Yes, implement ConsumeImapProcessor with OAUTH2 and an associated oauth2AccessTokenProviderService.
This Provider is only triggered on the onScheduled Method of the AbstractEmailProcessor and oauth2AccessTokenProvider.getAccessDetails() to refresh the token is only requested also on onScheduledMethod.
But in onTriggered Method this will not trigger the oauth2AccessTokenProvider.getAccessDetails()
We startet a local debug with break points. Configured the ConsumeImapProcessor with Schedule "Cron-Based" every 5 minutes.
onScheduled => triggered on enabling the Processor
Every 5 minutes there is the onTriggered Method triggered and not the onScheduled. That is general for all Processors which use the AbstractEmailProcessor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11107 In ConsumeIMAP and ConsumePOP3 added support for OAUTH based authorization. [nifi]
Posted by "AnTu2702 (via GitHub)" <gi...@apache.org>.
AnTu2702 commented on PR #6900:
URL: https://github.com/apache/nifi/pull/6900#issuecomment-1978453332
I was able to build and deploy a local nar file...
...working fine with the patch!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11107 In ConsumeIMAP and ConsumePOP3 added support for OAUTH based authorization. [nifi]
Posted by "AnTu2702 (via GitHub)" <gi...@apache.org>.
AnTu2702 commented on PR #6900:
URL: https://github.com/apache/nifi/pull/6900#issuecomment-1978792542
@tpalfy You may be right. I just tested the refresh mechanism (not did not wait the 3600 s until the first access token really expired) with my intended fix that morning.
But the refreshed token nonetheless needs to be considered by the processor accessing the IMAP server.
And this happens in your `buildurl method`. So the real fix needs to be a bit different.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11107 In ConsumeIMAP and ConsumePOP3 added support for OAUTH based authorization. [nifi]
Posted by "araujf (via GitHub)" <gi...@apache.org>.
araujf commented on PR #6900:
URL: https://github.com/apache/nifi/pull/6900#issuecomment-1753670203
The refresh authentication token is working? When my token is expired, a authenticate error is returned.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11107 In ConsumeIMAP and ConsumePOP3 added support for OAUTH based authorization. [nifi]
Posted by "AnTu2702 (via GitHub)" <gi...@apache.org>.
AnTu2702 commented on PR #6900:
URL: https://github.com/apache/nifi/pull/6900#issuecomment-2106808373
@tpalfy Hey. How are you?
First of all: Thanks for the fix/workaround in NIFI 1.26.
Although things are basically working there, we still don't seem to be through.
The ConsumeIMAP Processor is continuously logging this ERROR...
(although the mailbox can and will be processed successfully using a refreshed token...)
We still have serious doubts to take this into production because of the log being spammed and of course the fact, that we have no clue, that causes this error.
Could you help and double check your fix/workaround?
Thank you in advance...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org