You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by e-...@gmx.fr on 2023/11/08 11:31:19 UTC
Bug : PutGCSObject : Version NIFI 1.23.2
Hello,
I'm trying to user the overwriting storage url parameter in the nifi version
1.23.2.
Do you know what is the correct parameter to set for the "Storage API URL" in
the processor PutGCSObject ?
If I setup "https:// **storage** -euwest1.p.googleapis.com/ **storage** /v1/"
It got this error :
PutGCSObject[id=aea7e07c-018b-1000-ffff] Failure completing upload flowfile=test/minh.txt bucket=xxx key=test/minh.txt reason=404 Not Found
POST https://storage-euwest1.p.googleapis.com/upload/storage/v1/storage/v1/b/xxx/o?name=test/minh.txt&uploadType=resumable
Not Found: com.google.cloud.storage.StorageException: 404 Not Found
If I setup "https:// **storage** -euwest1.p.googleapis.com"
It got this error :
PutGCSObject[id=aea7e07c-018b-1000-ffff] Failure completing upload flowfile=test/minh.txt bucket=xxx key=test/minh.txt reason=www.googleapis.com: com.google.cloud.storage.StorageException: www.googleapis.com
- Caused by: java.net.UnknownHostException: www.googleapis.com
Thanks for help
Minh
Re: Bug : PutGCSObject : Version NIFI 1.23.2
Posted by Joe Witt <jo...@gmail.com>.
What Im focusing on is from your log above
- Caused by: java.net.UnknownHostException: www.googleapis.com
I cant route to that either..
Thanks
On Wed, Nov 8, 2023 at 8:16 AM <e-...@gmx.fr> wrote:
> Hi
>
> I have checked the DNS entry for "*storage*-euwest1.p.googleapis.com
> <https://deref-gmx.fr/mail/client/QqYZdreYCF4/dereferrer/?redirectUrl=http%3A%2F%2Feuwest1.p.googleapis.com%2F>"
> it seems OK
> From the VM nifi, in the terminal, I could reach the port 443 it works
> fine.
>
> I check with the devops google team also.
>
> Thanks
>
> *Envoyé:* mercredi 8 novembre 2023 à 15:40
>
> *De:* "Joe Witt" <jo...@gmail.com>
> *À:* users@nifi.apache.org
> *Objet:* Re: Bug : PutGCSObject : Version NIFI 1.23.2
> Ok sounds good. I do think this is the correct path to be taking then.
>
> I think though there might be a dns configuration needed. Im not quite
> sure but I am aware of users that rely on this value heavily.
>
> Thanks
>
> On Wed, Nov 8, 2023 at 7:34 AM <e-...@gmx.fr> wrote:
>
>> Hello Joe
>>
>> Until now we leave the "storage api url" empty by default, but we use
>> "proxy configuration service" because we are behind the corporate network.
>>
>> Since some month ago we have configured "private connection" with google.
>> So I have setup "storage api url" with the private url and leave empty
>> "proxy configuration service"
>>
>> Regards
>>
>>
>> *Envoyé:* mercredi 8 novembre 2023 à 14:31
>> *De:* "Joe Witt" <jo...@gmail.com>
>> *À:* users@nifi.apache.org
>> *Objet:* Re: Bug : PutGCSObject : Version NIFI 1.23.2
>> Minh
>>
>> Do you need to override that value? Did Google specify a different URL
>> for you or you have a private connection?
>>
>> If you dont set that value what happens?
>>
>> Thanks
>>
>> On Wed, Nov 8, 2023 at 4:57 AM <e-...@gmx.fr> wrote:
>>
>>> Hello,
>>>
>>> I'm trying to user the overwriting storage url parameter in the nifi
>>> version 1.23.2.
>>>
>>> Do you know what is the correct parameter to set for the "Storage API
>>> URL" in the processor PutGCSObject ?
>>>
>>> If I setup "https://*storage*-euwest1.p.googleapis.com/*storage*/v1/"
>>> It got this error :
>>>
>>>
>>> PutGCSObject[id=aea7e07c-018b-1000-ffff] Failure completing upload flowfile=test/minh.txt bucket=xxx key=test/minh.txt reason=404 Not FoundPOST https://storage-euwest1.p.googleapis.com/upload/storage/v1/storage/v1/b/xxx/o?name=test/minh.txt&uploadType=resumableNot Found: com.google.cloud.storage.StorageException: 404 Not Found
>>>
>>>
>>> If I setup "https://*storage*-euwest1.p.googleapis.com"
>>> It got this error :
>>>
>>> PutGCSObject[id=aea7e07c-018b-1000-ffff] Failure completing upload flowfile=test/minh.txt bucket=xxx key=test/minh.txt reason=www.googleapis.com: com.google.cloud.storage.StorageException: www.googleapis.com- Caused by: java.net.UnknownHostException: www.googleapis.com
>>>
>>>
>>> Thanks for help
>>>
>>> Minh
>>>
>>
>>
>>
>
>
>
Re: Bug : PutGCSObject : Version NIFI 1.23.2
Posted by e-...@gmx.fr.
Hello,
Seems not working, I don't have any logs concerning google/putgcsObjects in
the nifi-app.log
Missing something ?
Minh
**Envoy e:** vendredi 10 novembre 2023 a 15:36
**De:** "Pierre Villard" <pi...@gmail.com>
**À :** users@nifi.apache.org
**Objet:** Re: Bug : PutGCSObject : Version NIFI 1.23.2
You'd just need to add
<logger name="com.google.cloud" level="DEBUG" />
in the file, for example next to the one like
<logger name="net.schmizz.sshj" level="WARN" />
And you should see the debug logs in nifi-app.log
Le ven. 10 nov. 2023 a 15:31, <[e-sociaux@gmx.fr](mailto:e-sociaux@gmx.fr)> a
ecrit :
> Hello Pierre,
>
>
>
> I'm not developper, so it is a bit confuse for me.
>
>
>
> I'm trying to temporarily push this config in the logback.xml but it is not
> working.
>
>
>
>
>
> .
>
> .
>
> .
>
> <logger name="com.google.cloud.storage" level="DEBUG" additivity="false">
>
> <appender-ref ref="APP_FILE"/>
>
> </logger>
>
>
>
> <root level="INFO">
>
> <appender-ref ref="APP_FILE"/>
>
> </root>
>
> .
>
> .
>
>
>
> It is missing something ?
>
>
>
> Thanks a lot for your help
>
>
>
> Regards
>
>
>
> Minh
>
> **Envoy e:** mercredi 8 novembre 2023 a 16:52
> **De:** "Pierre Villard"
> <[pierre.villard.fr@gmail.com](mailto:pierre.villard.fr@gmail.com)>
> **À :** [users@nifi.apache.org](mailto:users@nifi.apache.org)
> **Objet:** Re: Bug : PutGCSObject : Version NIFI 1.23.2
>
> I believe that if you temporarily enable debug logs on the GCP client
> library (via logback.xml on com.google.cloud.storage) you'd be able to
> confirm this.
>
>
>
> Le mer. 8 nov. 2023 a 16:43, <[e-sociaux@gmx.fr](mailto:e-sociaux@gmx.fr)> a
> ecrit :
>
>> Hi Pierre,
>>
>>
>>
>> I confirm, it works with Proxy.
>>
>>
>>
>> But how we are sure that the data are transfert from the "private
connection" and not through the proxy...
>>
>>
>>
>> Regards
>>
>>
>>
>>
>>
>> **Envoy e:** mercredi 8 novembre 2023 a 16:36
> **De:** [e-sociaux@gmx.fr](mailto:e-sociaux@gmx.fr)
> **À :** [users@nifi.apache.org](mailto:users@nifi.apache.org)
> **Objet:** Re: Bug : PutGCSObject : Version NIFI 1.23.2
>>
>> Hi Pierre,
>>
>>
>>
>> But it seems weird because when I put the wrong "storage api url : https://
**storage** -[euwest1.p.googleapis.com/](http://euwest1.p.googleapis.com/)
**storage** /v1/"
>>
>>
>>
>> We got the wrong url but has the correct hostname
>>
>>
>>
>> Not Found POST [https://storage-
euwest1.p.googleapis.com/upload/storage/v1/storage/v1/b/xxx/o?name=test/minh.txt&uploadType=resumable](https://storage-
euwest1.p.googleapis.com/upload/storage/v1/storage/v1/b/xxx/o?name=test/minh.txt&uploadType=resumable)
>>
>>
>>
>> **Envoy e:** mercredi 8 novembre 2023 a 16:29
> **De:** "Pierre Villard"
> <[pierre.villard.fr@gmail.com](mailto:pierre.villard.fr@gmail.com)>
> **À :** [users@nifi.apache.org](mailto:users@nifi.apache.org)
> **Objet:** Re: Bug : PutGCSObject : Version NIFI 1.23.2
>>
>> IIRC, but not in a position to confirm right now, the right API to set
would be https:// **storage**
-[euwest1.p.googleapis.com](http://euwest1.p.googleapis.com/) but your host
should still be able to resolve [googleapis.com](http://googleapis.com) (I
believe this can work through proxy as well).
>>
>>
>>
>> Le mer. 8 nov. 2023 a 16:16, <[e-sociaux@gmx.fr](mailto:e-sociaux@gmx.fr)>
a ecrit :
>>
>>> Hi
>>>
>>>
>>>
>>> I have checked the DNS entry for " **storage**
-[euwest1.p.googleapis.com](https://deref-
gmx.fr/mail/client/QqYZdreYCF4/dereferrer/?redirectUrl=http%3A%2F%2Feuwest1.p.googleapis.com%2F)"
it seems OK
>>>
>>> From the VM nifi, in the terminal, I could reach the port 443 it works
fine.
>>>
>>>
>>>
>>> I check with the devops google team also.
>>>
>>>
>>>
>>> Thanks
>>>
>>>
>>>
>>> **Envoy e:** mercredi 8 novembre 2023 a 15:40
> **De:** "Joe Witt" <[joe.witt@gmail.com](mailto:joe.witt@gmail.com)>
> **À :** [users@nifi.apache.org](mailto:users@nifi.apache.org)
> **Objet:** Re: Bug : PutGCSObject : Version NIFI 1.23.2
>>>
>>> Ok sounds good. I do think this is the correct path to be taking then.
>>>
>>>
>>>
>>> I think though there might be a dns configuration needed. Im not quite
sure but I am aware of users that rely on this value heavily.
>>>
>>>
>>>
>>> Thanks
>>>
>>>
>>>
>>> On Wed, Nov 8, 2023 at 7:34 AM
<[e-sociaux@gmx.fr](mailto:e-sociaux@gmx.fr)> wrote:
>>>
>>>> Hello Joe
>>>>
>>>>
>>>>
>>>> Until now we leave the "storage api url" empty by default, but we use
"proxy configuration service" because we are behind the corporate network.
>>>>
>>>>
>>>>
>>>> Since some month ago we have configured "private connection" with google.
>>>>
>>>> So I have setup "storage api url" with the private url and leave empty
"proxy configuration service"
>>>>
>>>>
>>>>
>>>> Regards
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> **Envoy e:** mercredi 8 novembre 2023 a 14:31
> **De:** "Joe Witt" <[joe.witt@gmail.com](mailto:joe.witt@gmail.com)>
> **À :** [users@nifi.apache.org](mailto:users@nifi.apache.org)
> **Objet:** Re: Bug : PutGCSObject : Version NIFI 1.23.2
>>>>
>>>> Minh
>>>>
>>>>
>>>>
>>>> Do you need to override that value? Did Google specify a different URL
for you or you have a private connection?
>>>>
>>>>
>>>>
>>>> If you dont set that value what happens?
>>>>
>>>>
>>>>
>>>> Thanks
>>>>
>>>>
>>>>
>>>> On Wed, Nov 8, 2023 at 4:57 AM
<[e-sociaux@gmx.fr](mailto:e-sociaux@gmx.fr)> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>>
>>>>>
>>>>> I'm trying to user the overwriting storage url parameter in the nifi
version 1.23.2.
>>>>>
>>>>>
>>>>>
>>>>> Do you know what is the correct parameter to set for the "Storage API
URL" in the processor PutGCSObject ?
>>>>>
>>>>>
>>>>>
>>>>> If I setup "https:// **storage**
-[euwest1.p.googleapis.com/](http://euwest1.p.googleapis.com/) **storage**
/v1/"
>>>>>
>>>>> It got this error :
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> PutGCSObject[id=aea7e07c-018b-1000-ffff] Failure completing upload
flowfile=test/minh.txt bucket=xxx key=test/minh.txt reason=404 Not FoundPOST
[https://storage-
euwest1.p.googleapis.com/upload/storage/v1/storage/v1/b/xxx/o?name=test/minh.txt&uploadType=resumable](https://storage-
euwest1.p.googleapis.com/upload/storage/v1/storage/v1/b/xxx/o?name=test/minh.txt&uploadType=resumable)Not
Found: com.google.cloud.storage.StorageException: 404 Not Found
>>>>>
>>>>>
>>>>>
>>>>> If I setup "https:// **storage**
-[euwest1.p.googleapis.com](http://euwest1.p.googleapis.com)"
>>>>>
>>>>> It got this error :
>>>>>
>>>>>
>>>>> PutGCSObject[id=aea7e07c-018b-1000-ffff] Failure completing upload
flowfile=test/minh.txt bucket=xxx key=test/minh.txt
reason=[www.googleapis.com](http://www.googleapis.com):
com.google.cloud.storage.StorageException:
[www.googleapis.com](http://www.googleapis.com)- Caused by:
java.net.UnknownHostException: [www.googleapis.com](http://www.googleapis.com)
>>>>>
>>>>>
>>>>>
>>>>> Thanks for help
>>>>>
>>>>>
>>>>>
>>>>> Minh
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>>
>>
>>
>>
>>
>
>
>
>
Re: Bug : PutGCSObject : Version NIFI 1.23.2
Posted by Pierre Villard <pi...@gmail.com>.
You'd just need to add
<logger name="com.google.cloud" level="DEBUG" />
in the file, for example next to the one like
<logger name="net.schmizz.sshj" level="WARN" />
And you should see the debug logs in nifi-app.log
Le ven. 10 nov. 2023 à 15:31, <e-...@gmx.fr> a écrit :
> Hello Pierre,
>
> I'm not developper, so it is a bit confuse for me.
>
> I'm trying to temporarily push this config in the logback.xml but it is
> not working.
>
>
> .
> .
> .
> <logger name="com.google.cloud.storage" level="DEBUG" additivity="false">
> <appender-ref ref="APP_FILE"/>
> </logger>
>
> <root level="INFO">
> <appender-ref ref="APP_FILE"/>
> </root>
> .
> .
>
> It is missing something ?
>
> Thanks a lot for your help
>
> Regards
>
> Minh
> *Envoyé:* mercredi 8 novembre 2023 à 16:52
> *De:* "Pierre Villard" <pi...@gmail.com>
> *À:* users@nifi.apache.org
> *Objet:* Re: Bug : PutGCSObject : Version NIFI 1.23.2
> I believe that if you temporarily enable debug logs on the GCP client
> library (via logback.xml on com.google.cloud.storage) you'd be able to
> confirm this.
>
> Le mer. 8 nov. 2023 à 16:43, <e-...@gmx.fr> a écrit :
>
>> Hi Pierre,
>>
>> I confirm, it works with Proxy.
>>
>> But how we are sure that the data are transfert from the "private
>> connection" and not through the proxy...
>>
>> Regards
>>
>>
>> *Envoyé:* mercredi 8 novembre 2023 à 16:36
>> *De:* e-sociaux@gmx.fr
>> *À:* users@nifi.apache.org
>> *Objet:* Re: Bug : PutGCSObject : Version NIFI 1.23.2
>> Hi Pierre,
>>
>> But it seems weird because when I put the wrong "storage api url
>> : https://*storage*-euwest1.p.googleapis.com/*storage*/v1/"
>>
>> We got the wrong url but has the correct hostname
>>
>> Not Found POST
>> https://storage-euwest1.p.googleapis.com/upload/storage/v1/storage/v1/b/xxx/o?name=test/minh.txt&uploadType=resumable
>>
>> *Envoyé:* mercredi 8 novembre 2023 à 16:29
>> *De:* "Pierre Villard" <pi...@gmail.com>
>> *À:* users@nifi.apache.org
>> *Objet:* Re: Bug : PutGCSObject : Version NIFI 1.23.2
>> IIRC, but not in a position to confirm right now, the right API to set
>> would be https://*storage*-euwest1.p.googleapis.com but your host should
>> still be able to resolve googleapis.com (I believe this can work through
>> proxy as well).
>>
>> Le mer. 8 nov. 2023 à 16:16, <e-...@gmx.fr> a écrit :
>>
>>> Hi
>>>
>>> I have checked the DNS entry for "*storage*-euwest1.p.googleapis.com
>>> <https://deref-gmx.fr/mail/client/QqYZdreYCF4/dereferrer/?redirectUrl=http%3A%2F%2Feuwest1.p.googleapis.com%2F>"
>>> it seems OK
>>> From the VM nifi, in the terminal, I could reach the port 443 it works
>>> fine.
>>>
>>> I check with the devops google team also.
>>>
>>> Thanks
>>>
>>> *Envoyé:* mercredi 8 novembre 2023 à 15:40
>>> *De:* "Joe Witt" <jo...@gmail.com>
>>> *À:* users@nifi.apache.org
>>> *Objet:* Re: Bug : PutGCSObject : Version NIFI 1.23.2
>>> Ok sounds good. I do think this is the correct path to be taking then.
>>>
>>> I think though there might be a dns configuration needed. Im not quite
>>> sure but I am aware of users that rely on this value heavily.
>>>
>>> Thanks
>>>
>>> On Wed, Nov 8, 2023 at 7:34 AM <e-...@gmx.fr> wrote:
>>>
>>>> Hello Joe
>>>>
>>>> Until now we leave the "storage api url" empty by default, but we use
>>>> "proxy configuration service" because we are behind the corporate network.
>>>>
>>>> Since some month ago we have configured "private connection" with
>>>> google.
>>>> So I have setup "storage api url" with the private url and leave empty
>>>> "proxy configuration service"
>>>>
>>>> Regards
>>>>
>>>>
>>>> *Envoyé:* mercredi 8 novembre 2023 à 14:31
>>>> *De:* "Joe Witt" <jo...@gmail.com>
>>>> *À:* users@nifi.apache.org
>>>> *Objet:* Re: Bug : PutGCSObject : Version NIFI 1.23.2
>>>> Minh
>>>>
>>>> Do you need to override that value? Did Google specify a different URL
>>>> for you or you have a private connection?
>>>>
>>>> If you dont set that value what happens?
>>>>
>>>> Thanks
>>>>
>>>> On Wed, Nov 8, 2023 at 4:57 AM <e-...@gmx.fr> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> I'm trying to user the overwriting storage url parameter in the nifi
>>>>> version 1.23.2.
>>>>>
>>>>> Do you know what is the correct parameter to set for the "Storage API
>>>>> URL" in the processor PutGCSObject ?
>>>>>
>>>>> If I setup "https://*storage*-euwest1.p.googleapis.com/*storage*/v1/"
>>>>> It got this error :
>>>>>
>>>>>
>>>>> PutGCSObject[id=aea7e07c-018b-1000-ffff] Failure completing upload flowfile=test/minh.txt bucket=xxx key=test/minh.txt reason=404 Not FoundPOST https://storage-euwest1.p.googleapis.com/upload/storage/v1/storage/v1/b/xxx/o?name=test/minh.txt&uploadType=resumableNot Found: com.google.cloud.storage.StorageException: 404 Not Found
>>>>>
>>>>>
>>>>> If I setup "https://*storage*-euwest1.p.googleapis.com"
>>>>> It got this error :
>>>>>
>>>>> PutGCSObject[id=aea7e07c-018b-1000-ffff] Failure completing upload flowfile=test/minh.txt bucket=xxx key=test/minh.txt reason=www.googleapis.com: com.google.cloud.storage.StorageException: www.googleapis.com- Caused by: java.net.UnknownHostException: www.googleapis.com
>>>>>
>>>>>
>>>>> Thanks for help
>>>>>
>>>>> Minh
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>
>>
>>
>
>
>
Re: Bug : PutGCSObject : Version NIFI 1.23.2
Posted by e-...@gmx.fr.
Hello Pierre,
I'm not developper, so it is a bit confuse for me.
I'm trying to temporarily push this config in the logback.xml but it is not
working.
.
.
.
<logger name="com.google.cloud.storage" level="DEBUG" additivity="false">
<appender-ref ref="APP_FILE"/>
</logger>
<root level="INFO">
<appender-ref ref="APP_FILE"/>
</root>
.
.
It is missing something ?
Thanks a lot for your help
Regards
Minh
**Envoy e:** mercredi 8 novembre 2023 a 16:52
**De:** "Pierre Villard" <pi...@gmail.com>
**À :** users@nifi.apache.org
**Objet:** Re: Bug : PutGCSObject : Version NIFI 1.23.2
I believe that if you temporarily enable debug logs on the GCP client library
(via logback.xml on com.google.cloud.storage) you'd be able to confirm this.
Le mer. 8 nov. 2023 a 16:43, <[e-sociaux@gmx.fr](mailto:e-sociaux@gmx.fr)> a
ecrit :
> Hi Pierre,
>
>
>
> I confirm, it works with Proxy.
>
>
>
> But how we are sure that the data are transfert from the "private
> connection" and not through the proxy...
>
>
>
> Regards
>
>
>
>
>
> **Envoy e:** mercredi 8 novembre 2023 a 16:36
> **De:** [e-sociaux@gmx.fr](mailto:e-sociaux@gmx.fr)
> **À :** [users@nifi.apache.org](mailto:users@nifi.apache.org)
> **Objet:** Re: Bug : PutGCSObject : Version NIFI 1.23.2
>
> Hi Pierre,
>
>
>
> But it seems weird because when I put the wrong "storage api url : https://
> **storage** -[euwest1.p.googleapis.com/](http://euwest1.p.googleapis.com/)
> **storage** /v1/"
>
>
>
> We got the wrong url but has the correct hostname
>
>
>
> Not Found POST [https://storage-
> euwest1.p.googleapis.com/upload/storage/v1/storage/v1/b/xxx/o?name=test/minh.txt&uploadType=resumable](https://storage-
> euwest1.p.googleapis.com/upload/storage/v1/storage/v1/b/xxx/o?name=test/minh.txt&uploadType=resumable)
>
>
>
> **Envoy e:** mercredi 8 novembre 2023 a 16:29
> **De:** "Pierre Villard"
> <[pierre.villard.fr@gmail.com](mailto:pierre.villard.fr@gmail.com)>
> **À :** [users@nifi.apache.org](mailto:users@nifi.apache.org)
> **Objet:** Re: Bug : PutGCSObject : Version NIFI 1.23.2
>
> IIRC, but not in a position to confirm right now, the right API to set would
> be https:// **storage**
> -[euwest1.p.googleapis.com](http://euwest1.p.googleapis.com/) but your host
> should still be able to resolve [googleapis.com](http://googleapis.com) (I
> believe this can work through proxy as well).
>
>
>
> Le mer. 8 nov. 2023 a 16:16, <[e-sociaux@gmx.fr](mailto:e-sociaux@gmx.fr)> a
> ecrit :
>
>> Hi
>>
>>
>>
>> I have checked the DNS entry for " **storage**
-[euwest1.p.googleapis.com](https://deref-
gmx.fr/mail/client/QqYZdreYCF4/dereferrer/?redirectUrl=http%3A%2F%2Feuwest1.p.googleapis.com%2F)"
it seems OK
>>
>> From the VM nifi, in the terminal, I could reach the port 443 it works
fine.
>>
>>
>>
>> I check with the devops google team also.
>>
>>
>>
>> Thanks
>>
>>
>>
>> **Envoy e:** mercredi 8 novembre 2023 a 15:40
> **De:** "Joe Witt" <[joe.witt@gmail.com](mailto:joe.witt@gmail.com)>
> **À :** [users@nifi.apache.org](mailto:users@nifi.apache.org)
> **Objet:** Re: Bug : PutGCSObject : Version NIFI 1.23.2
>>
>> Ok sounds good. I do think this is the correct path to be taking then.
>>
>>
>>
>> I think though there might be a dns configuration needed. Im not quite
sure but I am aware of users that rely on this value heavily.
>>
>>
>>
>> Thanks
>>
>>
>>
>> On Wed, Nov 8, 2023 at 7:34 AM
<[e-sociaux@gmx.fr](mailto:e-sociaux@gmx.fr)> wrote:
>>
>>> Hello Joe
>>>
>>>
>>>
>>> Until now we leave the "storage api url" empty by default, but we use
"proxy configuration service" because we are behind the corporate network.
>>>
>>>
>>>
>>> Since some month ago we have configured "private connection" with google.
>>>
>>> So I have setup "storage api url" with the private url and leave empty
"proxy configuration service"
>>>
>>>
>>>
>>> Regards
>>>
>>>
>>>
>>>
>>>
>>> **Envoy e:** mercredi 8 novembre 2023 a 14:31
> **De:** "Joe Witt" <[joe.witt@gmail.com](mailto:joe.witt@gmail.com)>
> **À :** [users@nifi.apache.org](mailto:users@nifi.apache.org)
> **Objet:** Re: Bug : PutGCSObject : Version NIFI 1.23.2
>>>
>>> Minh
>>>
>>>
>>>
>>> Do you need to override that value? Did Google specify a different URL
for you or you have a private connection?
>>>
>>>
>>>
>>> If you dont set that value what happens?
>>>
>>>
>>>
>>> Thanks
>>>
>>>
>>>
>>> On Wed, Nov 8, 2023 at 4:57 AM
<[e-sociaux@gmx.fr](mailto:e-sociaux@gmx.fr)> wrote:
>>>
>>>> Hello,
>>>>
>>>>
>>>>
>>>> I'm trying to user the overwriting storage url parameter in the nifi
version 1.23.2.
>>>>
>>>>
>>>>
>>>> Do you know what is the correct parameter to set for the "Storage API
URL" in the processor PutGCSObject ?
>>>>
>>>>
>>>>
>>>> If I setup "https:// **storage**
-[euwest1.p.googleapis.com/](http://euwest1.p.googleapis.com/) **storage**
/v1/"
>>>>
>>>> It got this error :
>>>>
>>>>
>>>>
>>>>
>>>> PutGCSObject[id=aea7e07c-018b-1000-ffff] Failure completing upload
flowfile=test/minh.txt bucket=xxx key=test/minh.txt reason=404 Not FoundPOST
[https://storage-
euwest1.p.googleapis.com/upload/storage/v1/storage/v1/b/xxx/o?name=test/minh.txt&uploadType=resumable](https://storage-
euwest1.p.googleapis.com/upload/storage/v1/storage/v1/b/xxx/o?name=test/minh.txt&uploadType=resumable)Not
Found: com.google.cloud.storage.StorageException: 404 Not Found
>>>>
>>>>
>>>>
>>>> If I setup "https:// **storage**
-[euwest1.p.googleapis.com](http://euwest1.p.googleapis.com)"
>>>>
>>>> It got this error :
>>>>
>>>>
>>>> PutGCSObject[id=aea7e07c-018b-1000-ffff] Failure completing upload
flowfile=test/minh.txt bucket=xxx key=test/minh.txt
reason=[www.googleapis.com](http://www.googleapis.com):
com.google.cloud.storage.StorageException:
[www.googleapis.com](http://www.googleapis.com)- Caused by:
java.net.UnknownHostException: [www.googleapis.com](http://www.googleapis.com)
>>>>
>>>>
>>>>
>>>> Thanks for help
>>>>
>>>>
>>>>
>>>> Minh
>>>
>>>
>>>
>>>
>>
>>
>>
>>
>
>
>
>
Re: Bug : PutGCSObject : Version NIFI 1.23.2
Posted by e-...@gmx.fr.
Hi Pierre,
Thanks for this information.
Minh
**Envoy e:** mercredi 8 novembre 2023 a 19:57
**De:** "Pierre Villard" <pi...@gmail.com>
**À :** users@nifi.apache.org
**Objet:** Re: Bug : PutGCSObject : Version NIFI 1.23.2
So I believe the client library, when using the private endpoint, is making
multiple calls, the first one is on the private endpoint and that's why you
get a 404 when setting
https:// **storage**
-[euwest1.p.googleapis.com/](http://euwest1.p.googleapis.com/) **storage**
/v1/
As you can see, it's making a call to a wrong URL (twice /storage/v1):
[https://storage-
euwest1.p.googleapis.com/upload/storage/v1/storage/v1/b/xxx/o?name=test/minh.txt&uploadType=resumable](https://storage-
euwest1.p.googleapis.com/upload/storage/v1/storage/v1/b/xxx/o?name=test/minh.txt&uploadType=resumable)
This is why you get the 404.
When you set the right endpoint you don't get the 404 but later on the client
library has to make a call to [googleapis.com](http://googleapis.com) (if I
recall correctly this part) and that's why you got the unknown host exception.
Le mer. 8 nov. 2023 a 16:52, Pierre Villard
<[pierre.villard.fr@gmail.com](mailto:pierre.villard.fr@gmail.com)> a ecrit :
> I believe that if you temporarily enable debug logs on the GCP client
> library (via logback.xml on com.google.cloud.storage) you'd be able to
> confirm this.
>
>
>
> Le mer. 8 nov. 2023 a 16:43, <[e-sociaux@gmx.fr](mailto:e-sociaux@gmx.fr)> a
> ecrit :
>
>> Hi Pierre,
>>
>>
>>
>> I confirm, it works with Proxy.
>>
>>
>>
>> But how we are sure that the data are transfert from the "private
connection" and not through the proxy...
>>
>>
>>
>> Regards
>>
>>
>>
>>
>>
>> **Envoy e:** mercredi 8 novembre 2023 a 16:36
> **De:** [e-sociaux@gmx.fr](mailto:e-sociaux@gmx.fr)
> **À :** [users@nifi.apache.org](mailto:users@nifi.apache.org)
> **Objet:** Re: Bug : PutGCSObject : Version NIFI 1.23.2
>>
>> Hi Pierre,
>>
>>
>>
>> But it seems weird because when I put the wrong "storage api url : https://
**storage** -[euwest1.p.googleapis.com/](http://euwest1.p.googleapis.com/)
**storage** /v1/"
>>
>>
>>
>> We got the wrong url but has the correct hostname
>>
>>
>>
>> Not Found POST [https://storage-
euwest1.p.googleapis.com/upload/storage/v1/storage/v1/b/xxx/o?name=test/minh.txt&uploadType=resumable](https://storage-
euwest1.p.googleapis.com/upload/storage/v1/storage/v1/b/xxx/o?name=test/minh.txt&uploadType=resumable)
>>
>>
>>
>> **Envoy e:** mercredi 8 novembre 2023 a 16:29
> **De:** "Pierre Villard"
> <[pierre.villard.fr@gmail.com](mailto:pierre.villard.fr@gmail.com)>
> **À :** [users@nifi.apache.org](mailto:users@nifi.apache.org)
> **Objet:** Re: Bug : PutGCSObject : Version NIFI 1.23.2
>>
>> IIRC, but not in a position to confirm right now, the right API to set
would be https:// **storage**
-[euwest1.p.googleapis.com](http://euwest1.p.googleapis.com/) but your host
should still be able to resolve [googleapis.com](http://googleapis.com) (I
believe this can work through proxy as well).
>>
>>
>>
>> Le mer. 8 nov. 2023 a 16:16, <[e-sociaux@gmx.fr](mailto:e-sociaux@gmx.fr)>
a ecrit :
>>
>>> Hi
>>>
>>>
>>>
>>> I have checked the DNS entry for " **storage**
-[euwest1.p.googleapis.com](https://deref-
gmx.fr/mail/client/QqYZdreYCF4/dereferrer/?redirectUrl=http%3A%2F%2Feuwest1.p.googleapis.com%2F)"
it seems OK
>>>
>>> From the VM nifi, in the terminal, I could reach the port 443 it works
fine.
>>>
>>>
>>>
>>> I check with the devops google team also.
>>>
>>>
>>>
>>> Thanks
>>>
>>>
>>>
>>> **Envoy e:** mercredi 8 novembre 2023 a 15:40
> **De:** "Joe Witt" <[joe.witt@gmail.com](mailto:joe.witt@gmail.com)>
> **À :** [users@nifi.apache.org](mailto:users@nifi.apache.org)
> **Objet:** Re: Bug : PutGCSObject : Version NIFI 1.23.2
>>>
>>> Ok sounds good. I do think this is the correct path to be taking then.
>>>
>>>
>>>
>>> I think though there might be a dns configuration needed. Im not quite
sure but I am aware of users that rely on this value heavily.
>>>
>>>
>>>
>>> Thanks
>>>
>>>
>>>
>>> On Wed, Nov 8, 2023 at 7:34 AM
<[e-sociaux@gmx.fr](mailto:e-sociaux@gmx.fr)> wrote:
>>>
>>>> Hello Joe
>>>>
>>>>
>>>>
>>>> Until now we leave the "storage api url" empty by default, but we use
"proxy configuration service" because we are behind the corporate network.
>>>>
>>>>
>>>>
>>>> Since some month ago we have configured "private connection" with google.
>>>>
>>>> So I have setup "storage api url" with the private url and leave empty
"proxy configuration service"
>>>>
>>>>
>>>>
>>>> Regards
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> **Envoy e:** mercredi 8 novembre 2023 a 14:31
> **De:** "Joe Witt" <[joe.witt@gmail.com](mailto:joe.witt@gmail.com)>
> **À :** [users@nifi.apache.org](mailto:users@nifi.apache.org)
> **Objet:** Re: Bug : PutGCSObject : Version NIFI 1.23.2
>>>>
>>>> Minh
>>>>
>>>>
>>>>
>>>> Do you need to override that value? Did Google specify a different URL
for you or you have a private connection?
>>>>
>>>>
>>>>
>>>> If you dont set that value what happens?
>>>>
>>>>
>>>>
>>>> Thanks
>>>>
>>>>
>>>>
>>>> On Wed, Nov 8, 2023 at 4:57 AM
<[e-sociaux@gmx.fr](mailto:e-sociaux@gmx.fr)> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>>
>>>>>
>>>>> I'm trying to user the overwriting storage url parameter in the nifi
version 1.23.2.
>>>>>
>>>>>
>>>>>
>>>>> Do you know what is the correct parameter to set for the "Storage API
URL" in the processor PutGCSObject ?
>>>>>
>>>>>
>>>>>
>>>>> If I setup "https:// **storage**
-[euwest1.p.googleapis.com/](http://euwest1.p.googleapis.com/) **storage**
/v1/"
>>>>>
>>>>> It got this error :
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> PutGCSObject[id=aea7e07c-018b-1000-ffff] Failure completing upload
flowfile=test/minh.txt bucket=xxx key=test/minh.txt reason=404 Not FoundPOST
[https://storage-
euwest1.p.googleapis.com/upload/storage/v1/storage/v1/b/xxx/o?name=test/minh.txt&uploadType=resumable](https://storage-
euwest1.p.googleapis.com/upload/storage/v1/storage/v1/b/xxx/o?name=test/minh.txt&uploadType=resumable)Not
Found: com.google.cloud.storage.StorageException: 404 Not Found
>>>>>
>>>>>
>>>>>
>>>>> If I setup "https:// **storage**
-[euwest1.p.googleapis.com](http://euwest1.p.googleapis.com)"
>>>>>
>>>>> It got this error :
>>>>>
>>>>>
>>>>> PutGCSObject[id=aea7e07c-018b-1000-ffff] Failure completing upload
flowfile=test/minh.txt bucket=xxx key=test/minh.txt
reason=[www.googleapis.com](http://www.googleapis.com):
com.google.cloud.storage.StorageException:
[www.googleapis.com](http://www.googleapis.com)- Caused by:
java.net.UnknownHostException: [www.googleapis.com](http://www.googleapis.com)
>>>>>
>>>>>
>>>>>
>>>>> Thanks for help
>>>>>
>>>>>
>>>>>
>>>>> Minh
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>>
>>
>>
>>
>>
Re: Bug : PutGCSObject : Version NIFI 1.23.2
Posted by Pierre Villard <pi...@gmail.com>.
So I believe the client library, when using the private endpoint, is making
multiple calls, the first one is on the private endpoint and that's why you
get a 404 when setting
https://*storage*-euwest1.p.googleapis.com/*storage*/v1/
As you can see, it's making a call to a wrong URL (twice /storage/v1):
https://storage-euwest1.p.googleapis.com/upload/storage/v1/storage/v1/b/xxx/o?name=test/minh.txt&uploadType=resumable
This is why you get the 404.
When you set the right endpoint you don't get the 404 but later on the
client library has to make a call to googleapis.com (if I recall correctly
this part) and that's why you got the unknown host exception.
Le mer. 8 nov. 2023 à 16:52, Pierre Villard <pi...@gmail.com> a
écrit :
> I believe that if you temporarily enable debug logs on the GCP client
> library (via logback.xml on com.google.cloud.storage) you'd be able to
> confirm this.
>
> Le mer. 8 nov. 2023 à 16:43, <e-...@gmx.fr> a écrit :
>
>> Hi Pierre,
>>
>> I confirm, it works with Proxy.
>>
>> But how we are sure that the data are transfert from the "private
>> connection" and not through the proxy...
>>
>> Regards
>>
>>
>> *Envoyé:* mercredi 8 novembre 2023 à 16:36
>> *De:* e-sociaux@gmx.fr
>> *À:* users@nifi.apache.org
>> *Objet:* Re: Bug : PutGCSObject : Version NIFI 1.23.2
>> Hi Pierre,
>>
>> But it seems weird because when I put the wrong "storage api url
>> : https://*storage*-euwest1.p.googleapis.com/*storage*/v1/"
>>
>> We got the wrong url but has the correct hostname
>>
>> Not Found POST
>> https://storage-euwest1.p.googleapis.com/upload/storage/v1/storage/v1/b/xxx/o?name=test/minh.txt&uploadType=resumable
>>
>> *Envoyé:* mercredi 8 novembre 2023 à 16:29
>> *De:* "Pierre Villard" <pi...@gmail.com>
>> *À:* users@nifi.apache.org
>> *Objet:* Re: Bug : PutGCSObject : Version NIFI 1.23.2
>> IIRC, but not in a position to confirm right now, the right API to set
>> would be https://*storage*-euwest1.p.googleapis.com but your host should
>> still be able to resolve googleapis.com (I believe this can work through
>> proxy as well).
>>
>> Le mer. 8 nov. 2023 à 16:16, <e-...@gmx.fr> a écrit :
>>
>>> Hi
>>>
>>> I have checked the DNS entry for "*storage*-euwest1.p.googleapis.com
>>> <https://deref-gmx.fr/mail/client/QqYZdreYCF4/dereferrer/?redirectUrl=http%3A%2F%2Feuwest1.p.googleapis.com%2F>"
>>> it seems OK
>>> From the VM nifi, in the terminal, I could reach the port 443 it works
>>> fine.
>>>
>>> I check with the devops google team also.
>>>
>>> Thanks
>>>
>>> *Envoyé:* mercredi 8 novembre 2023 à 15:40
>>> *De:* "Joe Witt" <jo...@gmail.com>
>>> *À:* users@nifi.apache.org
>>> *Objet:* Re: Bug : PutGCSObject : Version NIFI 1.23.2
>>> Ok sounds good. I do think this is the correct path to be taking then.
>>>
>>> I think though there might be a dns configuration needed. Im not quite
>>> sure but I am aware of users that rely on this value heavily.
>>>
>>> Thanks
>>>
>>> On Wed, Nov 8, 2023 at 7:34 AM <e-...@gmx.fr> wrote:
>>>
>>>> Hello Joe
>>>>
>>>> Until now we leave the "storage api url" empty by default, but we use
>>>> "proxy configuration service" because we are behind the corporate network.
>>>>
>>>> Since some month ago we have configured "private connection" with
>>>> google.
>>>> So I have setup "storage api url" with the private url and leave empty
>>>> "proxy configuration service"
>>>>
>>>> Regards
>>>>
>>>>
>>>> *Envoyé:* mercredi 8 novembre 2023 à 14:31
>>>> *De:* "Joe Witt" <jo...@gmail.com>
>>>> *À:* users@nifi.apache.org
>>>> *Objet:* Re: Bug : PutGCSObject : Version NIFI 1.23.2
>>>> Minh
>>>>
>>>> Do you need to override that value? Did Google specify a different URL
>>>> for you or you have a private connection?
>>>>
>>>> If you dont set that value what happens?
>>>>
>>>> Thanks
>>>>
>>>> On Wed, Nov 8, 2023 at 4:57 AM <e-...@gmx.fr> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> I'm trying to user the overwriting storage url parameter in the nifi
>>>>> version 1.23.2.
>>>>>
>>>>> Do you know what is the correct parameter to set for the "Storage API
>>>>> URL" in the processor PutGCSObject ?
>>>>>
>>>>> If I setup "https://*storage*-euwest1.p.googleapis.com/*storage*/v1/"
>>>>> It got this error :
>>>>>
>>>>>
>>>>> PutGCSObject[id=aea7e07c-018b-1000-ffff] Failure completing upload flowfile=test/minh.txt bucket=xxx key=test/minh.txt reason=404 Not FoundPOST https://storage-euwest1.p.googleapis.com/upload/storage/v1/storage/v1/b/xxx/o?name=test/minh.txt&uploadType=resumableNot Found: com.google.cloud.storage.StorageException: 404 Not Found
>>>>>
>>>>>
>>>>> If I setup "https://*storage*-euwest1.p.googleapis.com"
>>>>> It got this error :
>>>>>
>>>>> PutGCSObject[id=aea7e07c-018b-1000-ffff] Failure completing upload flowfile=test/minh.txt bucket=xxx key=test/minh.txt reason=www.googleapis.com: com.google.cloud.storage.StorageException: www.googleapis.com- Caused by: java.net.UnknownHostException: www.googleapis.com
>>>>>
>>>>>
>>>>> Thanks for help
>>>>>
>>>>> Minh
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>
>>
>>
>
Re: Bug : PutGCSObject : Version NIFI 1.23.2
Posted by Pierre Villard <pi...@gmail.com>.
I believe that if you temporarily enable debug logs on the GCP client
library (via logback.xml on com.google.cloud.storage) you'd be able to
confirm this.
Le mer. 8 nov. 2023 à 16:43, <e-...@gmx.fr> a écrit :
> Hi Pierre,
>
> I confirm, it works with Proxy.
>
> But how we are sure that the data are transfert from the "private
> connection" and not through the proxy...
>
> Regards
>
>
> *Envoyé:* mercredi 8 novembre 2023 à 16:36
> *De:* e-sociaux@gmx.fr
> *À:* users@nifi.apache.org
> *Objet:* Re: Bug : PutGCSObject : Version NIFI 1.23.2
> Hi Pierre,
>
> But it seems weird because when I put the wrong "storage api url : https://
> *storage*-euwest1.p.googleapis.com/*storage*/v1/"
>
> We got the wrong url but has the correct hostname
>
> Not Found POST
> https://storage-euwest1.p.googleapis.com/upload/storage/v1/storage/v1/b/xxx/o?name=test/minh.txt&uploadType=resumable
>
> *Envoyé:* mercredi 8 novembre 2023 à 16:29
> *De:* "Pierre Villard" <pi...@gmail.com>
> *À:* users@nifi.apache.org
> *Objet:* Re: Bug : PutGCSObject : Version NIFI 1.23.2
> IIRC, but not in a position to confirm right now, the right API to set
> would be https://*storage*-euwest1.p.googleapis.com but your host should
> still be able to resolve googleapis.com (I believe this can work through
> proxy as well).
>
> Le mer. 8 nov. 2023 à 16:16, <e-...@gmx.fr> a écrit :
>
>> Hi
>>
>> I have checked the DNS entry for "*storage*-euwest1.p.googleapis.com
>> <https://deref-gmx.fr/mail/client/QqYZdreYCF4/dereferrer/?redirectUrl=http%3A%2F%2Feuwest1.p.googleapis.com%2F>"
>> it seems OK
>> From the VM nifi, in the terminal, I could reach the port 443 it works
>> fine.
>>
>> I check with the devops google team also.
>>
>> Thanks
>>
>> *Envoyé:* mercredi 8 novembre 2023 à 15:40
>> *De:* "Joe Witt" <jo...@gmail.com>
>> *À:* users@nifi.apache.org
>> *Objet:* Re: Bug : PutGCSObject : Version NIFI 1.23.2
>> Ok sounds good. I do think this is the correct path to be taking then.
>>
>> I think though there might be a dns configuration needed. Im not quite
>> sure but I am aware of users that rely on this value heavily.
>>
>> Thanks
>>
>> On Wed, Nov 8, 2023 at 7:34 AM <e-...@gmx.fr> wrote:
>>
>>> Hello Joe
>>>
>>> Until now we leave the "storage api url" empty by default, but we use
>>> "proxy configuration service" because we are behind the corporate network.
>>>
>>> Since some month ago we have configured "private connection" with google.
>>> So I have setup "storage api url" with the private url and leave empty
>>> "proxy configuration service"
>>>
>>> Regards
>>>
>>>
>>> *Envoyé:* mercredi 8 novembre 2023 à 14:31
>>> *De:* "Joe Witt" <jo...@gmail.com>
>>> *À:* users@nifi.apache.org
>>> *Objet:* Re: Bug : PutGCSObject : Version NIFI 1.23.2
>>> Minh
>>>
>>> Do you need to override that value? Did Google specify a different URL
>>> for you or you have a private connection?
>>>
>>> If you dont set that value what happens?
>>>
>>> Thanks
>>>
>>> On Wed, Nov 8, 2023 at 4:57 AM <e-...@gmx.fr> wrote:
>>>
>>>> Hello,
>>>>
>>>> I'm trying to user the overwriting storage url parameter in the nifi
>>>> version 1.23.2.
>>>>
>>>> Do you know what is the correct parameter to set for the "Storage API
>>>> URL" in the processor PutGCSObject ?
>>>>
>>>> If I setup "https://*storage*-euwest1.p.googleapis.com/*storage*/v1/"
>>>> It got this error :
>>>>
>>>>
>>>> PutGCSObject[id=aea7e07c-018b-1000-ffff] Failure completing upload flowfile=test/minh.txt bucket=xxx key=test/minh.txt reason=404 Not FoundPOST https://storage-euwest1.p.googleapis.com/upload/storage/v1/storage/v1/b/xxx/o?name=test/minh.txt&uploadType=resumableNot Found: com.google.cloud.storage.StorageException: 404 Not Found
>>>>
>>>>
>>>> If I setup "https://*storage*-euwest1.p.googleapis.com"
>>>> It got this error :
>>>>
>>>> PutGCSObject[id=aea7e07c-018b-1000-ffff] Failure completing upload flowfile=test/minh.txt bucket=xxx key=test/minh.txt reason=www.googleapis.com: com.google.cloud.storage.StorageException: www.googleapis.com- Caused by: java.net.UnknownHostException: www.googleapis.com
>>>>
>>>>
>>>> Thanks for help
>>>>
>>>> Minh
>>>>
>>>
>>>
>>>
>>
>>
>>
>
>
>
Re: Bug : PutGCSObject : Version NIFI 1.23.2
Posted by e-...@gmx.fr.
Hi Pierre,
I confirm, it works with Proxy.
But how we are sure that the data are transfert from the "private connection"
and not through the proxy...
Regards
**Envoy e:** mercredi 8 novembre 2023 a 16:36
**De:** e-sociaux@gmx.fr
**À :** users@nifi.apache.org
**Objet:** Re: Bug : PutGCSObject : Version NIFI 1.23.2
Hi Pierre,
But it seems weird because when I put the wrong "storage api url : https://
**storage** -[euwest1.p.googleapis.com/](http://euwest1.p.googleapis.com/)
**storage** /v1/"
We got the wrong url but has the correct hostname
Not Found POST [https://storage-
euwest1.p.googleapis.com/upload/storage/v1/storage/v1/b/xxx/o?name=test/minh.txt&uploadType=resumable](https://storage-
euwest1.p.googleapis.com/upload/storage/v1/storage/v1/b/xxx/o?name=test/minh.txt&uploadType=resumable)
**Envoy e:** mercredi 8 novembre 2023 a 16:29
**De:** "Pierre Villard" <pi...@gmail.com>
**À :** users@nifi.apache.org
**Objet:** Re: Bug : PutGCSObject : Version NIFI 1.23.2
IIRC, but not in a position to confirm right now, the right API to set would
be https:// **storage**
-[euwest1.p.googleapis.com](http://euwest1.p.googleapis.com/) but your host
should still be able to resolve [googleapis.com](http://googleapis.com) (I
believe this can work through proxy as well).
Le mer. 8 nov. 2023 a 16:16, <[e-sociaux@gmx.fr](mailto:e-sociaux@gmx.fr)> a
ecrit :
> Hi
>
>
>
> I have checked the DNS entry for " **storage**
> -[euwest1.p.googleapis.com](https://deref-
> gmx.fr/mail/client/QqYZdreYCF4/dereferrer/?redirectUrl=http%3A%2F%2Feuwest1.p.googleapis.com%2F)"
> it seems OK
>
> From the VM nifi, in the terminal, I could reach the port 443 it works fine.
>
>
>
> I check with the devops google team also.
>
>
>
> Thanks
>
>
>
> **Envoy e:** mercredi 8 novembre 2023 a 15:40
> **De:** "Joe Witt" <[joe.witt@gmail.com](mailto:joe.witt@gmail.com)>
> **À :** [users@nifi.apache.org](mailto:users@nifi.apache.org)
> **Objet:** Re: Bug : PutGCSObject : Version NIFI 1.23.2
>
> Ok sounds good. I do think this is the correct path to be taking then.
>
>
>
> I think though there might be a dns configuration needed. Im not quite sure
> but I am aware of users that rely on this value heavily.
>
>
>
> Thanks
>
>
>
> On Wed, Nov 8, 2023 at 7:34 AM <[e-sociaux@gmx.fr](mailto:e-sociaux@gmx.fr)>
> wrote:
>
>> Hello Joe
>>
>>
>>
>> Until now we leave the "storage api url" empty by default, but we use
"proxy configuration service" because we are behind the corporate network.
>>
>>
>>
>> Since some month ago we have configured "private connection" with google.
>>
>> So I have setup "storage api url" with the private url and leave empty
"proxy configuration service"
>>
>>
>>
>> Regards
>>
>>
>>
>>
>>
>> **Envoy e:** mercredi 8 novembre 2023 a 14:31
> **De:** "Joe Witt" <[joe.witt@gmail.com](mailto:joe.witt@gmail.com)>
> **À :** [users@nifi.apache.org](mailto:users@nifi.apache.org)
> **Objet:** Re: Bug : PutGCSObject : Version NIFI 1.23.2
>>
>> Minh
>>
>>
>>
>> Do you need to override that value? Did Google specify a different URL for
you or you have a private connection?
>>
>>
>>
>> If you dont set that value what happens?
>>
>>
>>
>> Thanks
>>
>>
>>
>> On Wed, Nov 8, 2023 at 4:57 AM
<[e-sociaux@gmx.fr](mailto:e-sociaux@gmx.fr)> wrote:
>>
>>> Hello,
>>>
>>>
>>>
>>> I'm trying to user the overwriting storage url parameter in the nifi
version 1.23.2.
>>>
>>>
>>>
>>> Do you know what is the correct parameter to set for the "Storage API URL"
in the processor PutGCSObject ?
>>>
>>>
>>>
>>> If I setup "https:// **storage**
-[euwest1.p.googleapis.com/](http://euwest1.p.googleapis.com/) **storage**
/v1/"
>>>
>>> It got this error :
>>>
>>>
>>>
>>>
>>> PutGCSObject[id=aea7e07c-018b-1000-ffff] Failure completing upload
flowfile=test/minh.txt bucket=xxx key=test/minh.txt reason=404 Not FoundPOST
[https://storage-
euwest1.p.googleapis.com/upload/storage/v1/storage/v1/b/xxx/o?name=test/minh.txt&uploadType=resumable](https://storage-
euwest1.p.googleapis.com/upload/storage/v1/storage/v1/b/xxx/o?name=test/minh.txt&uploadType=resumable)Not
Found: com.google.cloud.storage.StorageException: 404 Not Found
>>>
>>>
>>>
>>> If I setup "https:// **storage**
-[euwest1.p.googleapis.com](http://euwest1.p.googleapis.com)"
>>>
>>> It got this error :
>>>
>>>
>>> PutGCSObject[id=aea7e07c-018b-1000-ffff] Failure completing upload
flowfile=test/minh.txt bucket=xxx key=test/minh.txt
reason=[www.googleapis.com](http://www.googleapis.com):
com.google.cloud.storage.StorageException:
[www.googleapis.com](http://www.googleapis.com)- Caused by:
java.net.UnknownHostException: [www.googleapis.com](http://www.googleapis.com)
>>>
>>>
>>>
>>> Thanks for help
>>>
>>>
>>>
>>> Minh
>>
>>
>>
>>
>
>
>
>
Re: Bug : PutGCSObject : Version NIFI 1.23.2
Posted by e-...@gmx.fr.
Hi Pierre,
But it seems weird because when I put the wrong "storage api url : https://
**storage** -[euwest1.p.googleapis.com/](http://euwest1.p.googleapis.com/)
**storage** /v1/"
We got the wrong url but has the correct hostname
Not Found POST [https://storage-
euwest1.p.googleapis.com/upload/storage/v1/storage/v1/b/xxx/o?name=test/minh.txt&uploadType=resumable](https://storage-
euwest1.p.googleapis.com/upload/storage/v1/storage/v1/b/xxx/o?name=test/minh.txt&uploadType=resumable)
**Envoy e:** mercredi 8 novembre 2023 a 16:29
**De:** "Pierre Villard" <pi...@gmail.com>
**À :** users@nifi.apache.org
**Objet:** Re: Bug : PutGCSObject : Version NIFI 1.23.2
IIRC, but not in a position to confirm right now, the right API to set would
be https:// **storage**
-[euwest1.p.googleapis.com](http://euwest1.p.googleapis.com/) but your host
should still be able to resolve [googleapis.com](http://googleapis.com) (I
believe this can work through proxy as well).
Le mer. 8 nov. 2023 a 16:16, <[e-sociaux@gmx.fr](mailto:e-sociaux@gmx.fr)> a
ecrit :
> Hi
>
>
>
> I have checked the DNS entry for " **storage**
> -[euwest1.p.googleapis.com](https://deref-
> gmx.fr/mail/client/QqYZdreYCF4/dereferrer/?redirectUrl=http%3A%2F%2Feuwest1.p.googleapis.com%2F)"
> it seems OK
>
> From the VM nifi, in the terminal, I could reach the port 443 it works fine.
>
>
>
> I check with the devops google team also.
>
>
>
> Thanks
>
>
>
> **Envoy e:** mercredi 8 novembre 2023 a 15:40
> **De:** "Joe Witt" <[joe.witt@gmail.com](mailto:joe.witt@gmail.com)>
> **À :** [users@nifi.apache.org](mailto:users@nifi.apache.org)
> **Objet:** Re: Bug : PutGCSObject : Version NIFI 1.23.2
>
> Ok sounds good. I do think this is the correct path to be taking then.
>
>
>
> I think though there might be a dns configuration needed. Im not quite sure
> but I am aware of users that rely on this value heavily.
>
>
>
> Thanks
>
>
>
> On Wed, Nov 8, 2023 at 7:34 AM <[e-sociaux@gmx.fr](mailto:e-sociaux@gmx.fr)>
> wrote:
>
>> Hello Joe
>>
>>
>>
>> Until now we leave the "storage api url" empty by default, but we use
"proxy configuration service" because we are behind the corporate network.
>>
>>
>>
>> Since some month ago we have configured "private connection" with google.
>>
>> So I have setup "storage api url" with the private url and leave empty
"proxy configuration service"
>>
>>
>>
>> Regards
>>
>>
>>
>>
>>
>> **Envoy e:** mercredi 8 novembre 2023 a 14:31
> **De:** "Joe Witt" <[joe.witt@gmail.com](mailto:joe.witt@gmail.com)>
> **À :** [users@nifi.apache.org](mailto:users@nifi.apache.org)
> **Objet:** Re: Bug : PutGCSObject : Version NIFI 1.23.2
>>
>> Minh
>>
>>
>>
>> Do you need to override that value? Did Google specify a different URL for
you or you have a private connection?
>>
>>
>>
>> If you dont set that value what happens?
>>
>>
>>
>> Thanks
>>
>>
>>
>> On Wed, Nov 8, 2023 at 4:57 AM
<[e-sociaux@gmx.fr](mailto:e-sociaux@gmx.fr)> wrote:
>>
>>> Hello,
>>>
>>>
>>>
>>> I'm trying to user the overwriting storage url parameter in the nifi
version 1.23.2.
>>>
>>>
>>>
>>> Do you know what is the correct parameter to set for the "Storage API URL"
in the processor PutGCSObject ?
>>>
>>>
>>>
>>> If I setup "https:// **storage**
-[euwest1.p.googleapis.com/](http://euwest1.p.googleapis.com/) **storage**
/v1/"
>>>
>>> It got this error :
>>>
>>>
>>>
>>>
>>> PutGCSObject[id=aea7e07c-018b-1000-ffff] Failure completing upload
flowfile=test/minh.txt bucket=xxx key=test/minh.txt reason=404 Not FoundPOST
[https://storage-
euwest1.p.googleapis.com/upload/storage/v1/storage/v1/b/xxx/o?name=test/minh.txt&uploadType=resumable](https://storage-
euwest1.p.googleapis.com/upload/storage/v1/storage/v1/b/xxx/o?name=test/minh.txt&uploadType=resumable)Not
Found: com.google.cloud.storage.StorageException: 404 Not Found
>>>
>>>
>>>
>>> If I setup "https:// **storage**
-[euwest1.p.googleapis.com](http://euwest1.p.googleapis.com)"
>>>
>>> It got this error :
>>>
>>>
>>> PutGCSObject[id=aea7e07c-018b-1000-ffff] Failure completing upload
flowfile=test/minh.txt bucket=xxx key=test/minh.txt
reason=[www.googleapis.com](http://www.googleapis.com):
com.google.cloud.storage.StorageException:
[www.googleapis.com](http://www.googleapis.com)- Caused by:
java.net.UnknownHostException: [www.googleapis.com](http://www.googleapis.com)
>>>
>>>
>>>
>>> Thanks for help
>>>
>>>
>>>
>>> Minh
>>
>>
>>
>>
>
>
>
>
Re: Bug : PutGCSObject : Version NIFI 1.23.2
Posted by Pierre Villard <pi...@gmail.com>.
IIRC, but not in a position to confirm right now, the right API to set
would be https://*storage*-euwest1.p.googleapis.com but your host should
still be able to resolve googleapis.com (I believe this can work through
proxy as well).
Le mer. 8 nov. 2023 à 16:16, <e-...@gmx.fr> a écrit :
> Hi
>
> I have checked the DNS entry for "*storage*-euwest1.p.googleapis.com
> <https://deref-gmx.fr/mail/client/QqYZdreYCF4/dereferrer/?redirectUrl=http%3A%2F%2Feuwest1.p.googleapis.com%2F>"
> it seems OK
> From the VM nifi, in the terminal, I could reach the port 443 it works
> fine.
>
> I check with the devops google team also.
>
> Thanks
>
> *Envoyé:* mercredi 8 novembre 2023 à 15:40
> *De:* "Joe Witt" <jo...@gmail.com>
> *À:* users@nifi.apache.org
> *Objet:* Re: Bug : PutGCSObject : Version NIFI 1.23.2
> Ok sounds good. I do think this is the correct path to be taking then.
>
> I think though there might be a dns configuration needed. Im not quite
> sure but I am aware of users that rely on this value heavily.
>
> Thanks
>
> On Wed, Nov 8, 2023 at 7:34 AM <e-...@gmx.fr> wrote:
>
>> Hello Joe
>>
>> Until now we leave the "storage api url" empty by default, but we use
>> "proxy configuration service" because we are behind the corporate network.
>>
>> Since some month ago we have configured "private connection" with google.
>> So I have setup "storage api url" with the private url and leave empty
>> "proxy configuration service"
>>
>> Regards
>>
>>
>> *Envoyé:* mercredi 8 novembre 2023 à 14:31
>> *De:* "Joe Witt" <jo...@gmail.com>
>> *À:* users@nifi.apache.org
>> *Objet:* Re: Bug : PutGCSObject : Version NIFI 1.23.2
>> Minh
>>
>> Do you need to override that value? Did Google specify a different URL
>> for you or you have a private connection?
>>
>> If you dont set that value what happens?
>>
>> Thanks
>>
>> On Wed, Nov 8, 2023 at 4:57 AM <e-...@gmx.fr> wrote:
>>
>>> Hello,
>>>
>>> I'm trying to user the overwriting storage url parameter in the nifi
>>> version 1.23.2.
>>>
>>> Do you know what is the correct parameter to set for the "Storage API
>>> URL" in the processor PutGCSObject ?
>>>
>>> If I setup "https://*storage*-euwest1.p.googleapis.com/*storage*/v1/"
>>> It got this error :
>>>
>>>
>>> PutGCSObject[id=aea7e07c-018b-1000-ffff] Failure completing upload flowfile=test/minh.txt bucket=xxx key=test/minh.txt reason=404 Not FoundPOST https://storage-euwest1.p.googleapis.com/upload/storage/v1/storage/v1/b/xxx/o?name=test/minh.txt&uploadType=resumableNot Found: com.google.cloud.storage.StorageException: 404 Not Found
>>>
>>>
>>> If I setup "https://*storage*-euwest1.p.googleapis.com"
>>> It got this error :
>>>
>>> PutGCSObject[id=aea7e07c-018b-1000-ffff] Failure completing upload flowfile=test/minh.txt bucket=xxx key=test/minh.txt reason=www.googleapis.com: com.google.cloud.storage.StorageException: www.googleapis.com- Caused by: java.net.UnknownHostException: www.googleapis.com
>>>
>>>
>>> Thanks for help
>>>
>>> Minh
>>>
>>
>>
>>
>
>
>
Re: Bug : PutGCSObject : Version NIFI 1.23.2
Posted by e-...@gmx.fr.
Hi
I have checked the DNS entry for " **storage**
-[euwest1.p.googleapis.com](https://deref-
gmx.fr/mail/client/QqYZdreYCF4/dereferrer/?redirectUrl=http%3A%2F%2Feuwest1.p.googleapis.com%2F)"
it seems OK
From the VM nifi, in the terminal, I could reach the port 443 it works fine.
I check with the devops google team also.
Thanks
**Envoy e:** mercredi 8 novembre 2023 a 15:40
**De:** "Joe Witt" <jo...@gmail.com>
**À :** users@nifi.apache.org
**Objet:** Re: Bug : PutGCSObject : Version NIFI 1.23.2
Ok sounds good. I do think this is the correct path to be taking then.
I think though there might be a dns configuration needed. Im not quite sure
but I am aware of users that rely on this value heavily.
Thanks
On Wed, Nov 8, 2023 at 7:34 AM <[e-sociaux@gmx.fr](mailto:e-sociaux@gmx.fr)>
wrote:
> Hello Joe
>
>
>
> Until now we leave the "storage api url" empty by default, but we use "proxy
> configuration service" because we are behind the corporate network.
>
>
>
> Since some month ago we have configured "private connection" with google.
>
> So I have setup "storage api url" with the private url and leave empty
> "proxy configuration service"
>
>
>
> Regards
>
>
>
>
>
> **Envoy e:** mercredi 8 novembre 2023 a 14:31
> **De:** "Joe Witt" <[joe.witt@gmail.com](mailto:joe.witt@gmail.com)>
> **À :** [users@nifi.apache.org](mailto:users@nifi.apache.org)
> **Objet:** Re: Bug : PutGCSObject : Version NIFI 1.23.2
>
> Minh
>
>
>
> Do you need to override that value? Did Google specify a different URL for
> you or you have a private connection?
>
>
>
> If you dont set that value what happens?
>
>
>
> Thanks
>
>
>
> On Wed, Nov 8, 2023 at 4:57 AM <[e-sociaux@gmx.fr](mailto:e-sociaux@gmx.fr)>
> wrote:
>
>> Hello,
>>
>>
>>
>> I'm trying to user the overwriting storage url parameter in the nifi
version 1.23.2.
>>
>>
>>
>> Do you know what is the correct parameter to set for the "Storage API URL"
in the processor PutGCSObject ?
>>
>>
>>
>> If I setup "https:// **storage**
-[euwest1.p.googleapis.com/](http://euwest1.p.googleapis.com/) **storage**
/v1/"
>>
>> It got this error :
>>
>>
>>
>>
>> PutGCSObject[id=aea7e07c-018b-1000-ffff] Failure completing upload
flowfile=test/minh.txt bucket=xxx key=test/minh.txt reason=404 Not FoundPOST
[https://storage-
euwest1.p.googleapis.com/upload/storage/v1/storage/v1/b/xxx/o?name=test/minh.txt&uploadType=resumable](https://storage-
euwest1.p.googleapis.com/upload/storage/v1/storage/v1/b/xxx/o?name=test/minh.txt&uploadType=resumable)Not
Found: com.google.cloud.storage.StorageException: 404 Not Found
>>
>>
>>
>> If I setup "https:// **storage**
-[euwest1.p.googleapis.com](http://euwest1.p.googleapis.com)"
>>
>> It got this error :
>>
>>
>> PutGCSObject[id=aea7e07c-018b-1000-ffff] Failure completing upload
flowfile=test/minh.txt bucket=xxx key=test/minh.txt
reason=[www.googleapis.com](http://www.googleapis.com):
com.google.cloud.storage.StorageException:
[www.googleapis.com](http://www.googleapis.com)- Caused by:
java.net.UnknownHostException: [www.googleapis.com](http://www.googleapis.com)
>>
>>
>>
>> Thanks for help
>>
>>
>>
>> Minh
>
>
>
>
Re: Bug : PutGCSObject : Version NIFI 1.23.2
Posted by Joe Witt <jo...@gmail.com>.
Ok sounds good. I do think this is the correct path to be taking then.
I think though there might be a dns configuration needed. Im not quite
sure but I am aware of users that rely on this value heavily.
Thanks
On Wed, Nov 8, 2023 at 7:34 AM <e-...@gmx.fr> wrote:
> Hello Joe
>
> Until now we leave the "storage api url" empty by default, but we use
> "proxy configuration service" because we are behind the corporate network.
>
> Since some month ago we have configured "private connection" with google.
> So I have setup "storage api url" with the private url and leave empty
> "proxy configuration service"
>
> Regards
>
>
> *Envoyé:* mercredi 8 novembre 2023 à 14:31
> *De:* "Joe Witt" <jo...@gmail.com>
> *À:* users@nifi.apache.org
> *Objet:* Re: Bug : PutGCSObject : Version NIFI 1.23.2
> Minh
>
> Do you need to override that value? Did Google specify a different URL
> for you or you have a private connection?
>
> If you dont set that value what happens?
>
> Thanks
>
> On Wed, Nov 8, 2023 at 4:57 AM <e-...@gmx.fr> wrote:
>
>> Hello,
>>
>> I'm trying to user the overwriting storage url parameter in the nifi
>> version 1.23.2.
>>
>> Do you know what is the correct parameter to set for the "Storage API
>> URL" in the processor PutGCSObject ?
>>
>> If I setup "https://*storage*-euwest1.p.googleapis.com/*storage*/v1/"
>> It got this error :
>>
>>
>> PutGCSObject[id=aea7e07c-018b-1000-ffff] Failure completing upload flowfile=test/minh.txt bucket=xxx key=test/minh.txt reason=404 Not FoundPOST https://storage-euwest1.p.googleapis.com/upload/storage/v1/storage/v1/b/xxx/o?name=test/minh.txt&uploadType=resumableNot Found: com.google.cloud.storage.StorageException: 404 Not Found
>>
>>
>> If I setup "https://*storage*-euwest1.p.googleapis.com"
>> It got this error :
>>
>> PutGCSObject[id=aea7e07c-018b-1000-ffff] Failure completing upload flowfile=test/minh.txt bucket=xxx key=test/minh.txt reason=www.googleapis.com: com.google.cloud.storage.StorageException: www.googleapis.com- Caused by: java.net.UnknownHostException: www.googleapis.com
>>
>>
>> Thanks for help
>>
>> Minh
>>
>
>
>
Re: Bug : PutGCSObject : Version NIFI 1.23.2
Posted by e-...@gmx.fr.
Hello Joe
Until now we leave the "storage api url" empty by default, but we use "proxy
configuration service" because we are behind the corporate network.
Since some month ago we have configured "private connection" with google.
So I have setup "storage api url" with the private url and leave empty "proxy
configuration service"
Regards
**Envoy e:** mercredi 8 novembre 2023 a 14:31
**De:** "Joe Witt" <jo...@gmail.com>
**À :** users@nifi.apache.org
**Objet:** Re: Bug : PutGCSObject : Version NIFI 1.23.2
Minh
Do you need to override that value? Did Google specify a different URL for
you or you have a private connection?
If you dont set that value what happens?
Thanks
On Wed, Nov 8, 2023 at 4:57 AM <[e-sociaux@gmx.fr](mailto:e-sociaux@gmx.fr)>
wrote:
> Hello,
>
>
>
> I'm trying to user the overwriting storage url parameter in the nifi version
> 1.23.2.
>
>
>
> Do you know what is the correct parameter to set for the "Storage API URL"
> in the processor PutGCSObject ?
>
>
>
> If I setup "https:// **storage**
> -[euwest1.p.googleapis.com/](http://euwest1.p.googleapis.com/) **storage**
> /v1/"
>
> It got this error :
>
>
>
>
> PutGCSObject[id=aea7e07c-018b-1000-ffff] Failure completing upload
> flowfile=test/minh.txt bucket=xxx key=test/minh.txt reason=404 Not FoundPOST
> [https://storage-
> euwest1.p.googleapis.com/upload/storage/v1/storage/v1/b/xxx/o?name=test/minh.txt&uploadType=resumable](https://storage-
> euwest1.p.googleapis.com/upload/storage/v1/storage/v1/b/xxx/o?name=test/minh.txt&uploadType=resumable)Not
> Found: com.google.cloud.storage.StorageException: 404 Not Found
>
>
>
> If I setup "https:// **storage**
> -[euwest1.p.googleapis.com](http://euwest1.p.googleapis.com)"
>
> It got this error :
>
>
> PutGCSObject[id=aea7e07c-018b-1000-ffff] Failure completing upload
> flowfile=test/minh.txt bucket=xxx key=test/minh.txt
> reason=[www.googleapis.com](http://www.googleapis.com):
> com.google.cloud.storage.StorageException:
> [www.googleapis.com](http://www.googleapis.com)- Caused by:
> java.net.UnknownHostException:
> [www.googleapis.com](http://www.googleapis.com)
>
>
>
> Thanks for help
>
>
>
> Minh
Re: Bug : PutGCSObject : Version NIFI 1.23.2
Posted by Joe Witt <jo...@gmail.com>.
Minh
Do you need to override that value? Did Google specify a different URL for
you or you have a private connection?
If you dont set that value what happens?
Thanks
On Wed, Nov 8, 2023 at 4:57 AM <e-...@gmx.fr> wrote:
> Hello,
>
> I'm trying to user the overwriting storage url parameter in the nifi
> version 1.23.2.
>
> Do you know what is the correct parameter to set for the "Storage API URL"
> in the processor PutGCSObject ?
>
> If I setup "https://*storage*-euwest1.p.googleapis.com/*storage*/v1/"
> It got this error :
>
>
> PutGCSObject[id=aea7e07c-018b-1000-ffff] Failure completing upload flowfile=test/minh.txt bucket=xxx key=test/minh.txt reason=404 Not Found
> POST https://storage-euwest1.p.googleapis.com/upload/storage/v1/storage/v1/b/xxx/o?name=test/minh.txt&uploadType=resumable
> Not Found: com.google.cloud.storage.StorageException: 404 Not Found
>
>
>
> If I setup "https://*storage*-euwest1.p.googleapis.com"
> It got this error :
>
> PutGCSObject[id=aea7e07c-018b-1000-ffff] Failure completing upload flowfile=test/minh.txt bucket=xxx key=test/minh.txt reason=www.googleapis.com: com.google.cloud.storage.StorageException: www.googleapis.com
> - Caused by: java.net.UnknownHostException: www.googleapis.com
>
>
> Thanks for help
>
> Minh
>