Re: Broken pipe write failed errors

["Robert R. Bruno" <rb...@gmail.com>]

David,

Quick update for you.  We decided after a bit of troubleshooting with zero
luck to just downgrade the OKHttp to 3.8.1 and the okhttp-digest to
1.18, and no more errors.  Not sure what to say.

Robert

On Mon, May 31, 2021 at 8:41 AM David Handermann <ex...@gmail.com>
wrote:

> Hi Robert,
>
> Thanks for providing the additional details.  It should be possible to
> replace the current version of OkHttp 4.9.1 with an older version to see if
> that makes a difference.  It would also be helpful to know whether the
> remote server supports HTTP/2.  Newer versions of OkHttp have improved
> support for HTTP/2, but it also has different connection characteristics.
> Setting the Disable HTTP/2 property to True in InvokeHTTP would force the
> use of HTTP/1.1.  I would not necessarily expect to see errors on the
> server side, but knowing whether the remote server has a connection or
> write timeout property would be useful.
>
> Regards,
> David Handermann
>
> On Sun, May 30, 2021 at 4:54 AM Robert R. Bruno <rb...@gmail.com> wrote:
>
>> When seeing the error we put our timeouts values in the processor both to
>> 5 mins as a test and still saw the errors and well before 5 minutes.  We
>> also slowed the processor down a lot and still were seeing the error.
>> Failed attempts will often succeed just fine but not always.
>>
>> As an easy test could we just rebuild with the older http client library
>> or did a lot more change with the processor?
>>
>> We do have access to both endpoints and plan to dig deeper there as well,
>> but initial looking did not show errors on server side.
>>
>> On Sat, May 29, 2021, 23:26 David Handermann <ex...@apache.org>
>> wrote:
>>
>>> Hi Robert,
>>>
>>> It would be helpful to know the settings for the Read Timeout and Idle
>>> Timeout properties on the InvokeHTTP processors.  If you have access to the
>>> remote service being called, it would also be interesting to know if there
>>> are timeouts on that side of the connection.  NiFi 1.13.2 includes a much
>>> newer version of the OkHttp client library, which InvokeHTTP uses, so the
>>> internal connection handling has gone through some changes.  In general,
>>> broken pipe errors suggest that the connection is timing out at some point,
>>> which may be related to a number of a variety of factors such as the number
>>> of connections, payload sizes, network latency, or local resource
>>> consumption.
>>>
>>> Regards,
>>> David Handermann
>>>
>>> On Sat, May 29, 2021 at 2:08 PM Joe Witt <jo...@gmail.com> wrote:
>>>
>>>> K. We have seen specific jvm versions causing issues with socket
>>>> handling.  But had not seen it on Java 11 though may be possible.   Is
>>>> there a full stack trace?
>>>>
>>>> On Sat, May 29, 2021 at 12:00 PM Robert R. Bruno <rb...@gmail.com>
>>>> wrote:
>>>>
>>>>> We upgraded to java 11 when we upgrade to 1.13.2 we were on java 8
>>>>> with 1.9.2.
>>>>>
>>>>> On Sat, May 29, 2021, 14:21 Joe Witt <jo...@gmail.com> wrote:
>>>>>
>>>>>> What JVM are you using?
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> On Sat, May 29, 2021 at 11:16 AM Juan Pablo Gardella <
>>>>>> gardellajuanpablo@gmail.com> wrote:
>>>>>>
>>>>>>> Not related to Nifi, but I faced the same type of issue for
>>>>>>> endpoints behind a proxy which takes more than 30 seconds to answer. Fixed
>>>>>>> by replacing Apache Http client by OkHttp. I did not investigate further,
>>>>>>> just simply replaced one library by another and the error was fixed.
>>>>>>>
>>>>>>>
>>>>>>> Juan
>>>>>>>
>>>>>>> On Sat, 29 May 2021 at 15:08, Robert R. Bruno <rb...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I wanted to see if anyone has any ideas on this one.  Since
>>>>>>>> upgrading to 1.13.2 from 1.9.2 we are starting to see broken pipe (write
>>>>>>>> failed) errors from a few invokeHttp processers.
>>>>>>>>
>>>>>>>> It is happening to processors talking to different endpoints, so I
>>>>>>>> am suspecting it is on the nifi side.  We are now using load balanced
>>>>>>>> queues throughout our flow.  Is it possible we are hitting a http
>>>>>>>> connection resource issue or something like that? A total guess I'll admit.
>>>>>>>>
>>>>>>>> If this could be it, does anyone know which parameter(s) to play
>>>>>>>> with in the properties file?  I know there is one setting for jetty threads
>>>>>>>> and another for max concurrent requests, but it isn't quite clear to me of
>>>>>>>> they are at all involved with invokeHttp calls.
>>>>>>>>
>>>>>>>> Thanks in advance!
>>>>>>>>
>>>>>>>> Robert
>>>>>>>>
>>>>>>>

Re: Broken pipe write failed errors

[David Handermann <ex...@gmail.com>]

Robert,

That is helpful, thanks for clarifying the use of plain HTTP, that
certainly eliminates a number of potential issues related to TLS and
HTTPS.  Given the general direction of OkHttp, I do not expect NiFi moving
back to version 3, however, it would be helpful to determine why OkHttp 4
is behaving differently in this scenario.  If there are any other details
you can share about the processor configuration such as the type of HTTP
request, the request and response content type, and the request size, that
would be very helpful.  If it is possible to reproduce the problem in a
test environment, that would help isolate the problem.

Regards,
David Handermann

On Tue, Jun 1, 2021 at 2:42 PM Robert R. Bruno <rb...@gmail.com> wrote:

> We wanted to give you another data point.  We tried 3.14.9 and 1.20 for
> the two libraries and no errors.  Also, our services are http not https.
> Hope that helps.  Thanks for mentioning the OkHttp client change.  Any
> chance a future version of nifi could move back to the 3.x client series?
>
> On Tue, Jun 1, 2021 at 2:50 PM David Handermann <
> exceptionfactory@gmail.com> wrote:
>
>> Robert,
>>
>> Thanks for the update, that's very interesting.  Version 3.8.1 is several
>> years old and is missing a number of updates related to HTTP/2 and TLS.
>> OkHttp 3.12.0 introduced support for TLS 1.3 when running on a supported
>> JVM, so if the connection is occurring over HTTPS, that may be part of the
>> equation.  Java 11 supports TLS 1.3, whereas Java 8 did not support TLS 1.3
>> until more recent updates.  It would be interesting to know if your
>> configuration still works with a more recent version of OkHttp in the 3.x
>> series.  Thanks again for providing the feedback.
>>
>> Regards,
>> David Handermann
>>
>> On Tue, Jun 1, 2021 at 1:31 PM Robert R. Bruno <rb...@gmail.com> wrote:
>>
>>> David,
>>>
>>> Quick update for you.  We decided after a bit of troubleshooting with
>>> zero luck to just downgrade the OKHttp to 3.8.1 and the okhttp-digest to
>>> 1.18, and no more errors.  Not sure what to say.
>>>
>>> Robert
>>>
>>> On Mon, May 31, 2021 at 8:41 AM David Handermann <
>>> exceptionfactory@gmail.com> wrote:
>>>
>>>> Hi Robert,
>>>>
>>>> Thanks for providing the additional details.  It should be possible to
>>>> replace the current version of OkHttp 4.9.1 with an older version to see if
>>>> that makes a difference.  It would also be helpful to know whether the
>>>> remote server supports HTTP/2.  Newer versions of OkHttp have improved
>>>> support for HTTP/2, but it also has different connection characteristics.
>>>> Setting the Disable HTTP/2 property to True in InvokeHTTP would force the
>>>> use of HTTP/1.1.  I would not necessarily expect to see errors on the
>>>> server side, but knowing whether the remote server has a connection or
>>>> write timeout property would be useful.
>>>>
>>>> Regards,
>>>> David Handermann
>>>>
>>>> On Sun, May 30, 2021 at 4:54 AM Robert R. Bruno <rb...@gmail.com>
>>>> wrote:
>>>>
>>>>> When seeing the error we put our timeouts values in the processor both
>>>>> to 5 mins as a test and still saw the errors and well before 5 minutes.  We
>>>>> also slowed the processor down a lot and still were seeing the error.
>>>>> Failed attempts will often succeed just fine but not always.
>>>>>
>>>>> As an easy test could we just rebuild with the older http client
>>>>> library or did a lot more change with the processor?
>>>>>
>>>>> We do have access to both endpoints and plan to dig deeper there as
>>>>> well, but initial looking did not show errors on server side.
>>>>>
>>>>> On Sat, May 29, 2021, 23:26 David Handermann <
>>>>> exceptionfactory@apache.org> wrote:
>>>>>
>>>>>> Hi Robert,
>>>>>>
>>>>>> It would be helpful to know the settings for the Read Timeout and
>>>>>> Idle Timeout properties on the InvokeHTTP processors.  If you have access
>>>>>> to the remote service being called, it would also be interesting to know if
>>>>>> there are timeouts on that side of the connection.  NiFi 1.13.2 includes a
>>>>>> much newer version of the OkHttp client library, which InvokeHTTP uses, so
>>>>>> the internal connection handling has gone through some changes.  In
>>>>>> general, broken pipe errors suggest that the connection is timing out at
>>>>>> some point, which may be related to a number of a variety of factors such
>>>>>> as the number of connections, payload sizes, network latency, or local
>>>>>> resource consumption.
>>>>>>
>>>>>> Regards,
>>>>>> David Handermann
>>>>>>
>>>>>> On Sat, May 29, 2021 at 2:08 PM Joe Witt <jo...@gmail.com> wrote:
>>>>>>
>>>>>>> K. We have seen specific jvm versions causing issues with socket
>>>>>>> handling.  But had not seen it on Java 11 though may be possible.   Is
>>>>>>> there a full stack trace?
>>>>>>>
>>>>>>> On Sat, May 29, 2021 at 12:00 PM Robert R. Bruno <rb...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> We upgraded to java 11 when we upgrade to 1.13.2 we were on java 8
>>>>>>>> with 1.9.2.
>>>>>>>>
>>>>>>>> On Sat, May 29, 2021, 14:21 Joe Witt <jo...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> What JVM are you using?
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>>
>>>>>>>>> On Sat, May 29, 2021 at 11:16 AM Juan Pablo Gardella <
>>>>>>>>> gardellajuanpablo@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Not related to Nifi, but I faced the same type of issue for
>>>>>>>>>> endpoints behind a proxy which takes more than 30 seconds to answer. Fixed
>>>>>>>>>> by replacing Apache Http client by OkHttp. I did not investigate further,
>>>>>>>>>> just simply replaced one library by another and the error was fixed.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Juan
>>>>>>>>>>
>>>>>>>>>> On Sat, 29 May 2021 at 15:08, Robert R. Bruno <rb...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> I wanted to see if anyone has any ideas on this one.  Since
>>>>>>>>>>> upgrading to 1.13.2 from 1.9.2 we are starting to see broken pipe (write
>>>>>>>>>>> failed) errors from a few invokeHttp processers.
>>>>>>>>>>>
>>>>>>>>>>> It is happening to processors talking to different endpoints, so
>>>>>>>>>>> I am suspecting it is on the nifi side.  We are now using load balanced
>>>>>>>>>>> queues throughout our flow.  Is it possible we are hitting a http
>>>>>>>>>>> connection resource issue or something like that? A total guess I'll admit.
>>>>>>>>>>>
>>>>>>>>>>> If this could be it, does anyone know which parameter(s) to play
>>>>>>>>>>> with in the properties file?  I know there is one setting for jetty threads
>>>>>>>>>>> and another for max concurrent requests, but it isn't quite clear to me of
>>>>>>>>>>> they are at all involved with invokeHttp calls.
>>>>>>>>>>>
>>>>>>>>>>> Thanks in advance!
>>>>>>>>>>>
>>>>>>>>>>> Robert
>>>>>>>>>>>
>>>>>>>>>>

Re: Broken pipe write failed errors

["Robert R. Bruno" <rb...@gmail.com>]

We wanted to give you another data point.  We tried 3.14.9 and 1.20 for the
two libraries and no errors.  Also, our services are http not https.  Hope
that helps.  Thanks for mentioning the OkHttp client change.  Any chance a
future version of nifi could move back to the 3.x client series?

On Tue, Jun 1, 2021 at 2:50 PM David Handermann <ex...@gmail.com>
wrote:

> Robert,
>
> Thanks for the update, that's very interesting.  Version 3.8.1 is several
> years old and is missing a number of updates related to HTTP/2 and TLS.
> OkHttp 3.12.0 introduced support for TLS 1.3 when running on a supported
> JVM, so if the connection is occurring over HTTPS, that may be part of the
> equation.  Java 11 supports TLS 1.3, whereas Java 8 did not support TLS 1.3
> until more recent updates.  It would be interesting to know if your
> configuration still works with a more recent version of OkHttp in the 3.x
> series.  Thanks again for providing the feedback.
>
> Regards,
> David Handermann
>
> On Tue, Jun 1, 2021 at 1:31 PM Robert R. Bruno <rb...@gmail.com> wrote:
>
>> David,
>>
>> Quick update for you.  We decided after a bit of troubleshooting with
>> zero luck to just downgrade the OKHttp to 3.8.1 and the okhttp-digest to
>> 1.18, and no more errors.  Not sure what to say.
>>
>> Robert
>>
>> On Mon, May 31, 2021 at 8:41 AM David Handermann <
>> exceptionfactory@gmail.com> wrote:
>>
>>> Hi Robert,
>>>
>>> Thanks for providing the additional details.  It should be possible to
>>> replace the current version of OkHttp 4.9.1 with an older version to see if
>>> that makes a difference.  It would also be helpful to know whether the
>>> remote server supports HTTP/2.  Newer versions of OkHttp have improved
>>> support for HTTP/2, but it also has different connection characteristics.
>>> Setting the Disable HTTP/2 property to True in InvokeHTTP would force the
>>> use of HTTP/1.1.  I would not necessarily expect to see errors on the
>>> server side, but knowing whether the remote server has a connection or
>>> write timeout property would be useful.
>>>
>>> Regards,
>>> David Handermann
>>>
>>> On Sun, May 30, 2021 at 4:54 AM Robert R. Bruno <rb...@gmail.com>
>>> wrote:
>>>
>>>> When seeing the error we put our timeouts values in the processor both
>>>> to 5 mins as a test and still saw the errors and well before 5 minutes.  We
>>>> also slowed the processor down a lot and still were seeing the error.
>>>> Failed attempts will often succeed just fine but not always.
>>>>
>>>> As an easy test could we just rebuild with the older http client
>>>> library or did a lot more change with the processor?
>>>>
>>>> We do have access to both endpoints and plan to dig deeper there as
>>>> well, but initial looking did not show errors on server side.
>>>>
>>>> On Sat, May 29, 2021, 23:26 David Handermann <
>>>> exceptionfactory@apache.org> wrote:
>>>>
>>>>> Hi Robert,
>>>>>
>>>>> It would be helpful to know the settings for the Read Timeout and Idle
>>>>> Timeout properties on the InvokeHTTP processors.  If you have access to the
>>>>> remote service being called, it would also be interesting to know if there
>>>>> are timeouts on that side of the connection.  NiFi 1.13.2 includes a much
>>>>> newer version of the OkHttp client library, which InvokeHTTP uses, so the
>>>>> internal connection handling has gone through some changes.  In general,
>>>>> broken pipe errors suggest that the connection is timing out at some point,
>>>>> which may be related to a number of a variety of factors such as the number
>>>>> of connections, payload sizes, network latency, or local resource
>>>>> consumption.
>>>>>
>>>>> Regards,
>>>>> David Handermann
>>>>>
>>>>> On Sat, May 29, 2021 at 2:08 PM Joe Witt <jo...@gmail.com> wrote:
>>>>>
>>>>>> K. We have seen specific jvm versions causing issues with socket
>>>>>> handling.  But had not seen it on Java 11 though may be possible.   Is
>>>>>> there a full stack trace?
>>>>>>
>>>>>> On Sat, May 29, 2021 at 12:00 PM Robert R. Bruno <rb...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> We upgraded to java 11 when we upgrade to 1.13.2 we were on java 8
>>>>>>> with 1.9.2.
>>>>>>>
>>>>>>> On Sat, May 29, 2021, 14:21 Joe Witt <jo...@gmail.com> wrote:
>>>>>>>
>>>>>>>> What JVM are you using?
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>>
>>>>>>>> On Sat, May 29, 2021 at 11:16 AM Juan Pablo Gardella <
>>>>>>>> gardellajuanpablo@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Not related to Nifi, but I faced the same type of issue for
>>>>>>>>> endpoints behind a proxy which takes more than 30 seconds to answer. Fixed
>>>>>>>>> by replacing Apache Http client by OkHttp. I did not investigate further,
>>>>>>>>> just simply replaced one library by another and the error was fixed.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Juan
>>>>>>>>>
>>>>>>>>> On Sat, 29 May 2021 at 15:08, Robert R. Bruno <rb...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> I wanted to see if anyone has any ideas on this one.  Since
>>>>>>>>>> upgrading to 1.13.2 from 1.9.2 we are starting to see broken pipe (write
>>>>>>>>>> failed) errors from a few invokeHttp processers.
>>>>>>>>>>
>>>>>>>>>> It is happening to processors talking to different endpoints, so
>>>>>>>>>> I am suspecting it is on the nifi side.  We are now using load balanced
>>>>>>>>>> queues throughout our flow.  Is it possible we are hitting a http
>>>>>>>>>> connection resource issue or something like that? A total guess I'll admit.
>>>>>>>>>>
>>>>>>>>>> If this could be it, does anyone know which parameter(s) to play
>>>>>>>>>> with in the properties file?  I know there is one setting for jetty threads
>>>>>>>>>> and another for max concurrent requests, but it isn't quite clear to me of
>>>>>>>>>> they are at all involved with invokeHttp calls.
>>>>>>>>>>
>>>>>>>>>> Thanks in advance!
>>>>>>>>>>
>>>>>>>>>> Robert
>>>>>>>>>>
>>>>>>>>>

Re: Broken pipe write failed errors

[David Handermann <ex...@gmail.com>]

Robert,

Thanks for the update, that's very interesting.  Version 3.8.1 is several
years old and is missing a number of updates related to HTTP/2 and TLS.
OkHttp 3.12.0 introduced support for TLS 1.3 when running on a supported
JVM, so if the connection is occurring over HTTPS, that may be part of the
equation.  Java 11 supports TLS 1.3, whereas Java 8 did not support TLS 1.3
until more recent updates.  It would be interesting to know if your
configuration still works with a more recent version of OkHttp in the 3.x
series.  Thanks again for providing the feedback.

Regards,
David Handermann

On Tue, Jun 1, 2021 at 1:31 PM Robert R. Bruno <rb...@gmail.com> wrote:

> David,
>
> Quick update for you.  We decided after a bit of troubleshooting with zero
> luck to just downgrade the OKHttp to 3.8.1 and the okhttp-digest to
> 1.18, and no more errors.  Not sure what to say.
>
> Robert
>
> On Mon, May 31, 2021 at 8:41 AM David Handermann <
> exceptionfactory@gmail.com> wrote:
>
>> Hi Robert,
>>
>> Thanks for providing the additional details.  It should be possible to
>> replace the current version of OkHttp 4.9.1 with an older version to see if
>> that makes a difference.  It would also be helpful to know whether the
>> remote server supports HTTP/2.  Newer versions of OkHttp have improved
>> support for HTTP/2, but it also has different connection characteristics.
>> Setting the Disable HTTP/2 property to True in InvokeHTTP would force the
>> use of HTTP/1.1.  I would not necessarily expect to see errors on the
>> server side, but knowing whether the remote server has a connection or
>> write timeout property would be useful.
>>
>> Regards,
>> David Handermann
>>
>> On Sun, May 30, 2021 at 4:54 AM Robert R. Bruno <rb...@gmail.com>
>> wrote:
>>
>>> When seeing the error we put our timeouts values in the processor both
>>> to 5 mins as a test and still saw the errors and well before 5 minutes.  We
>>> also slowed the processor down a lot and still were seeing the error.
>>> Failed attempts will often succeed just fine but not always.
>>>
>>> As an easy test could we just rebuild with the older http client library
>>> or did a lot more change with the processor?
>>>
>>> We do have access to both endpoints and plan to dig deeper there as
>>> well, but initial looking did not show errors on server side.
>>>
>>> On Sat, May 29, 2021, 23:26 David Handermann <
>>> exceptionfactory@apache.org> wrote:
>>>
>>>> Hi Robert,
>>>>
>>>> It would be helpful to know the settings for the Read Timeout and Idle
>>>> Timeout properties on the InvokeHTTP processors.  If you have access to the
>>>> remote service being called, it would also be interesting to know if there
>>>> are timeouts on that side of the connection.  NiFi 1.13.2 includes a much
>>>> newer version of the OkHttp client library, which InvokeHTTP uses, so the
>>>> internal connection handling has gone through some changes.  In general,
>>>> broken pipe errors suggest that the connection is timing out at some point,
>>>> which may be related to a number of a variety of factors such as the number
>>>> of connections, payload sizes, network latency, or local resource
>>>> consumption.
>>>>
>>>> Regards,
>>>> David Handermann
>>>>
>>>> On Sat, May 29, 2021 at 2:08 PM Joe Witt <jo...@gmail.com> wrote:
>>>>
>>>>> K. We have seen specific jvm versions causing issues with socket
>>>>> handling.  But had not seen it on Java 11 though may be possible.   Is
>>>>> there a full stack trace?
>>>>>
>>>>> On Sat, May 29, 2021 at 12:00 PM Robert R. Bruno <rb...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> We upgraded to java 11 when we upgrade to 1.13.2 we were on java 8
>>>>>> with 1.9.2.
>>>>>>
>>>>>> On Sat, May 29, 2021, 14:21 Joe Witt <jo...@gmail.com> wrote:
>>>>>>
>>>>>>> What JVM are you using?
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> On Sat, May 29, 2021 at 11:16 AM Juan Pablo Gardella <
>>>>>>> gardellajuanpablo@gmail.com> wrote:
>>>>>>>
>>>>>>>> Not related to Nifi, but I faced the same type of issue for
>>>>>>>> endpoints behind a proxy which takes more than 30 seconds to answer. Fixed
>>>>>>>> by replacing Apache Http client by OkHttp. I did not investigate further,
>>>>>>>> just simply replaced one library by another and the error was fixed.
>>>>>>>>
>>>>>>>>
>>>>>>>> Juan
>>>>>>>>
>>>>>>>> On Sat, 29 May 2021 at 15:08, Robert R. Bruno <rb...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I wanted to see if anyone has any ideas on this one.  Since
>>>>>>>>> upgrading to 1.13.2 from 1.9.2 we are starting to see broken pipe (write
>>>>>>>>> failed) errors from a few invokeHttp processers.
>>>>>>>>>
>>>>>>>>> It is happening to processors talking to different endpoints, so I
>>>>>>>>> am suspecting it is on the nifi side.  We are now using load balanced
>>>>>>>>> queues throughout our flow.  Is it possible we are hitting a http
>>>>>>>>> connection resource issue or something like that? A total guess I'll admit.
>>>>>>>>>
>>>>>>>>> If this could be it, does anyone know which parameter(s) to play
>>>>>>>>> with in the properties file?  I know there is one setting for jetty threads
>>>>>>>>> and another for max concurrent requests, but it isn't quite clear to me of
>>>>>>>>> they are at all involved with invokeHttp calls.
>>>>>>>>>
>>>>>>>>> Thanks in advance!
>>>>>>>>>
>>>>>>>>> Robert
>>>>>>>>>
>>>>>>>>