You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by "Sethuram, Anup" <an...@philips.com> on 2015/04/29 14:29:40 UTC
FW: debugging the flow
Hi,
How do I debug the flow of data in a NiFi workflow. I’m looking for a way in which I could tail the log files and check from the logs as to what is happening, what failed or what is getting queued up ..
I ran a particular workflow and below is the snippet from the output in the nifi-app.log. But during runtime, if I need a log statement to be printed on my console (say when a PutFile runs or when Unzip happens), how do I go about it. Can I tail and grep on specific keywords or should I configure something to obtain these information. What’s the practice being followed?
-----------------------------------------------------------------------------------------------------------------------------------------
2015-04-29 17:40:07,241 INFO [Reporting Task Thread Thread-7] C.Connections Connection Statuses:
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| Connection ID | Source | Connection Name | Destination | Flow Files In | Flow Files Out | FlowFiles Queued |
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| 8edba39b-325c-4680-b5ff-9d5e50abe3cc | PutFile | failure, success | GetFile | 2 / 14.55 MB (+2/+14.55 MB) | 0 / 0 bytes (+0/+0 bytes) | 7 / 57.11 MB (+2/+14.55 MB) |
| bf4fc6ea-def3-4613-8342-77d964512695 | Master Radar | | PutFile | 1 / 1.08 MB (+1/+1.08 MB) | 1 / 1.08 MB (+1/+1.08 MB) | 0 / 0 bytes (+0/+0 bytes) |
| 13b0f7b0-dda7-49b9-8d1b-376a0b7d840f | PutFile | failure, success | Modality | 1 / 1.08 MB (+1/+1.08 MB) | 1 / 1.08 MB (+1/+1.08 MB) | 0 / 0 bytes (+0/+0 bytes) |
| 909f1984-f2b5-3f9a-b028-711ad4ddb39a | RouteOnAttribute | log | PutFile | 1 / 13.47 MB (+1/+13.47 MB) | 1 / 13.47 MB (+1/+13.47 MB) | 0 / 0 bytes (+0/+0 bytes) |
| ef5450a3-5484-4d4a-a6e8-a3782cc7f69e | Modality | | RouteOnAttribute | 1 / 1.08 MB (+1/+1.08 MB) | 1 / 1.08 MB (+1/+1.08 MB) | 0 / 0 bytes (+0/+0 bytes) |
| 89d50091-eaad-3a28-a99f-e8d08fb2ae60 | RouteOnAttribute | zip | UnpackContent | 1 / 1.08 MB (+1/+1.08 MB) | 1 / 1.08 MB (+1/+1.08 MB) | 0 / 0 bytes (+0/+0 bytes) |
| 66b67b5e-ef87-3a91-b857-72e31eea7095 | RouteOnAttribute | unmatched | LogAttribute | 0 / 0 bytes (+0/+0 bytes) | 0 / 0 bytes (+0/+0 bytes) | 0 / 0 bytes (+0/+0 bytes) |
| e178bcd6-da58-327f-827d-62f3f724fd0e | UnpackContent | failure | LogAttribute | 0 / 0 bytes (+0/+0 bytes) | 0 / 0 bytes (+0/+0 bytes) | 0 / 0 bytes (+0/+0 bytes) |
| 094cb4c9-1bd9-3024-8bbd-bc4134de98db | RouteOnAttribute | cdf | PutFile | 0 / 0 bytes (+0/+0 bytes) | 0 / 0 bytes (+0/+0 bytes) | 0 / 0 bytes (+0/+0 bytes) |
| 6cfc5be3-c256-49b1-aca2-5c744aeb226b | GetFile | success | RouteOnAttribute | 1 / 13.47 MB (+1/+13.47 MB) | 1 / 13.47 MB (+1/+13.47 MB) | 0 / 0 bytes (+0/+0 bytes) |
| 03735f78-d8ab-420e-aa5e-f96c3113b46e | PutFile | failure | LogAttribute | 0 / 0 bytes (+0/+0 bytes) | 0 / 0 bytes (+0/+0 bytes) | 0 / 0 bytes (+0/+0 bytes) |
| bb3425b4-1c48-3f9c-9d72-cc73b2b3b7a9 | UnpackContent | original, success | PutFile | 2 / 14.55 MB (+2/+14.55 MB) | 2 / 14.55 MB (+2/+14.55 MB) | 0 / 0 bytes (+0/+0 bytes) |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Regards,
anup
________________________________
The information contained in this message may be confidential and legally protected under applicable law. The message is intended solely for the addressee(s). If you are not the intended recipient, you are hereby notified that any use, forwarding, dissemination, or reproduction of this message is strictly prohibited and may be unlawful. If you are not the intended recipient, please contact the sender by return e-mail and destroy all copies of the original message.
Re: FW: debugging the flow
Posted by Aldrin Piri <al...@gmail.com>.
Anup,
Adam's example is great for getting some quick metrics and monitoring
throughput type items. This is also easily scriptable and you can do a lot
of quick analysis using a command line tool like jq.
If you are looking for a deeper level of specificity, there are a couple of
options that come to mind. The Bulletins provided in NiFi are quite handy
and provide granularity on a per processor basis. You can adjust the
Bulletin level lower to something along the lines of INFO. As an example
you were asking about PutFile. If you change its Bulletin Level to INFO,
you can get a per item message of when a file is placed on the filesystem.
This is then also available via the Bulletin Board which provides filtering
so you could see by the processor ID what it was doing. Additionally, the
Bulletin Board can pop out to its own window and tab if you are interested
in a constant listing.
Second, if you truly prefer using the command line and monitoring logs,
from the logs directory, you can perform a tail -F nifi-app.log | grep
<flags and expression>. Typically, the specific processor IDs make good
candidates to grep on if you are interested in what a given set of
processors is doing. This log is fully configurable and powered by logback
which will allow you to make changes to its configuration and have them
reloaded by logback without having to restart the instance.
On Wed, Apr 29, 2015 at 8:38 AM, Adam Taft <ad...@adamtaft.com> wrote:
> Anup,
>
> Have you considered using the "web api" to extract the statistics you're
> interested in? The web-api is an HTTP/REST endpoint that is used by the
> main GUI. You can easily query it using an HTTP client to determine
> processor state, number of flowfiles in each queue, etc.
>
> Adam
>
>
> On Wed, Apr 29, 2015 at 8:29 AM, Sethuram, Anup <anup.sethuram@philips.com
> >
> wrote:
>
> >
> >
> >
> > Hi,
> > How do I debug the flow of data in a NiFi workflow. I’m looking for a
> > way in which I could tail the log files and check from the logs as to
> what
> > is happening, what failed or what is getting queued up ..
> >
> > I ran a particular workflow and below is the snippet from the output in
> > the nifi-app.log. But during runtime, if I need a log statement to be
> > printed on my console (say when a PutFile runs or when Unzip happens),
> how
> > do I go about it. Can I tail and grep on specific keywords or should I
> > configure something to obtain these information. What’s the practice
> being
> > followed?
> >
> >
> >
> >
> -----------------------------------------------------------------------------------------------------------------------------------------
> >
> > 2015-04-29 17:40:07,241 INFO [Reporting Task Thread Thread-7]
> > C.Connections Connection Statuses:
> >
> >
> >
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> >
> > | Connection ID | Source |
> > Connection Name | Destination |
> > Flow Files In |
> > Flow Files Out | FlowFiles Queued |
> >
> >
> >
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> >
> > | 8edba39b-325c-4680-b5ff-9d5e50abe3cc | PutFile |
> > failure, success | GetFile |
> > 2 / 14.55 MB (+2/+14.55 MB) | 0 / 0 bytes
> > (+0/+0 bytes) | 7 / 57.11 MB (+2/+14.55 MB) |
> >
> > | bf4fc6ea-def3-4613-8342-77d964512695 | Master Radar |
> > | PutFile |
> > 1 / 1.08 MB (+1/+1.08 MB) | 1 / 1.08 MB
> > (+1/+1.08 MB) | 0 / 0 bytes (+0/+0 bytes) |
> >
> > | 13b0f7b0-dda7-49b9-8d1b-376a0b7d840f | PutFile |
> > failure, success | Modality |
> > 1 / 1.08 MB (+1/+1.08 MB) | 1 / 1.08 MB
> > (+1/+1.08 MB) | 0 / 0 bytes (+0/+0 bytes) |
> >
> > | 909f1984-f2b5-3f9a-b028-711ad4ddb39a | RouteOnAttribute |
> > log | PutFile |
> > 1 / 13.47 MB (+1/+13.47 MB) | 1 / 13.47 MB
> > (+1/+13.47 MB) | 0 / 0 bytes (+0/+0 bytes) |
> >
> > | ef5450a3-5484-4d4a-a6e8-a3782cc7f69e | Modality |
> > | RouteOnAttribute |
> > 1 / 1.08 MB (+1/+1.08 MB) | 1 / 1.08 MB
> > (+1/+1.08 MB) | 0 / 0 bytes (+0/+0 bytes) |
> >
> > | 89d50091-eaad-3a28-a99f-e8d08fb2ae60 | RouteOnAttribute |
> > zip | UnpackContent |
> > 1 / 1.08 MB (+1/+1.08 MB) | 1 / 1.08 MB
> > (+1/+1.08 MB) | 0 / 0 bytes (+0/+0 bytes) |
> >
> > | 66b67b5e-ef87-3a91-b857-72e31eea7095 | RouteOnAttribute |
> > unmatched | LogAttribute |
> > 0 / 0 bytes (+0/+0 bytes) | 0 / 0 bytes
> > (+0/+0 bytes) | 0 / 0 bytes (+0/+0 bytes) |
> >
> > | e178bcd6-da58-327f-827d-62f3f724fd0e | UnpackContent |
> > failure | LogAttribute |
> > 0 / 0 bytes (+0/+0 bytes) | 0 / 0 bytes
> > (+0/+0 bytes) | 0 / 0 bytes (+0/+0 bytes) |
> >
> > | 094cb4c9-1bd9-3024-8bbd-bc4134de98db | RouteOnAttribute |
> > cdf | PutFile |
> > 0 / 0 bytes (+0/+0 bytes) | 0 / 0 bytes
> > (+0/+0 bytes) | 0 / 0 bytes (+0/+0 bytes) |
> >
> > | 6cfc5be3-c256-49b1-aca2-5c744aeb226b | GetFile |
> > success | RouteOnAttribute |
> > 1 / 13.47 MB (+1/+13.47 MB) | 1 / 13.47 MB
> > (+1/+13.47 MB) | 0 / 0 bytes (+0/+0 bytes) |
> >
> > | 03735f78-d8ab-420e-aa5e-f96c3113b46e | PutFile |
> > failure | LogAttribute |
> > 0 / 0 bytes (+0/+0 bytes) | 0 / 0 bytes
> > (+0/+0 bytes) | 0 / 0 bytes (+0/+0 bytes) |
> >
> > | bb3425b4-1c48-3f9c-9d72-cc73b2b3b7a9 | UnpackContent |
> > original, success | PutFile |
> > 2 / 14.55 MB (+2/+14.55 MB) | 2 / 14.55 MB
> > (+2/+14.55 MB) | 0 / 0 bytes (+0/+0 bytes) |
> >
> >
> >
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> >
> >
> > Regards,
> > anup
> >
> >
> >
> > ________________________________
> > The information contained in this message may be confidential and legally
> > protected under applicable law. The message is intended solely for the
> > addressee(s). If you are not the intended recipient, you are hereby
> > notified that any use, forwarding, dissemination, or reproduction of this
> > message is strictly prohibited and may be unlawful. If you are not the
> > intended recipient, please contact the sender by return e-mail and
> destroy
> > all copies of the original message.
> >
>
Re: FW: debugging the flow
Posted by Adam Taft <ad...@adamtaft.com>.
Anup,
Have you considered using the "web api" to extract the statistics you're
interested in? The web-api is an HTTP/REST endpoint that is used by the
main GUI. You can easily query it using an HTTP client to determine
processor state, number of flowfiles in each queue, etc.
Adam
On Wed, Apr 29, 2015 at 8:29 AM, Sethuram, Anup <an...@philips.com>
wrote:
>
>
>
> Hi,
> How do I debug the flow of data in a NiFi workflow. I’m looking for a
> way in which I could tail the log files and check from the logs as to what
> is happening, what failed or what is getting queued up ..
>
> I ran a particular workflow and below is the snippet from the output in
> the nifi-app.log. But during runtime, if I need a log statement to be
> printed on my console (say when a PutFile runs or when Unzip happens), how
> do I go about it. Can I tail and grep on specific keywords or should I
> configure something to obtain these information. What’s the practice being
> followed?
>
>
>
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> 2015-04-29 17:40:07,241 INFO [Reporting Task Thread Thread-7]
> C.Connections Connection Statuses:
>
>
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> | Connection ID | Source |
> Connection Name | Destination |
> Flow Files In |
> Flow Files Out | FlowFiles Queued |
>
>
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> | 8edba39b-325c-4680-b5ff-9d5e50abe3cc | PutFile |
> failure, success | GetFile |
> 2 / 14.55 MB (+2/+14.55 MB) | 0 / 0 bytes
> (+0/+0 bytes) | 7 / 57.11 MB (+2/+14.55 MB) |
>
> | bf4fc6ea-def3-4613-8342-77d964512695 | Master Radar |
> | PutFile |
> 1 / 1.08 MB (+1/+1.08 MB) | 1 / 1.08 MB
> (+1/+1.08 MB) | 0 / 0 bytes (+0/+0 bytes) |
>
> | 13b0f7b0-dda7-49b9-8d1b-376a0b7d840f | PutFile |
> failure, success | Modality |
> 1 / 1.08 MB (+1/+1.08 MB) | 1 / 1.08 MB
> (+1/+1.08 MB) | 0 / 0 bytes (+0/+0 bytes) |
>
> | 909f1984-f2b5-3f9a-b028-711ad4ddb39a | RouteOnAttribute |
> log | PutFile |
> 1 / 13.47 MB (+1/+13.47 MB) | 1 / 13.47 MB
> (+1/+13.47 MB) | 0 / 0 bytes (+0/+0 bytes) |
>
> | ef5450a3-5484-4d4a-a6e8-a3782cc7f69e | Modality |
> | RouteOnAttribute |
> 1 / 1.08 MB (+1/+1.08 MB) | 1 / 1.08 MB
> (+1/+1.08 MB) | 0 / 0 bytes (+0/+0 bytes) |
>
> | 89d50091-eaad-3a28-a99f-e8d08fb2ae60 | RouteOnAttribute |
> zip | UnpackContent |
> 1 / 1.08 MB (+1/+1.08 MB) | 1 / 1.08 MB
> (+1/+1.08 MB) | 0 / 0 bytes (+0/+0 bytes) |
>
> | 66b67b5e-ef87-3a91-b857-72e31eea7095 | RouteOnAttribute |
> unmatched | LogAttribute |
> 0 / 0 bytes (+0/+0 bytes) | 0 / 0 bytes
> (+0/+0 bytes) | 0 / 0 bytes (+0/+0 bytes) |
>
> | e178bcd6-da58-327f-827d-62f3f724fd0e | UnpackContent |
> failure | LogAttribute |
> 0 / 0 bytes (+0/+0 bytes) | 0 / 0 bytes
> (+0/+0 bytes) | 0 / 0 bytes (+0/+0 bytes) |
>
> | 094cb4c9-1bd9-3024-8bbd-bc4134de98db | RouteOnAttribute |
> cdf | PutFile |
> 0 / 0 bytes (+0/+0 bytes) | 0 / 0 bytes
> (+0/+0 bytes) | 0 / 0 bytes (+0/+0 bytes) |
>
> | 6cfc5be3-c256-49b1-aca2-5c744aeb226b | GetFile |
> success | RouteOnAttribute |
> 1 / 13.47 MB (+1/+13.47 MB) | 1 / 13.47 MB
> (+1/+13.47 MB) | 0 / 0 bytes (+0/+0 bytes) |
>
> | 03735f78-d8ab-420e-aa5e-f96c3113b46e | PutFile |
> failure | LogAttribute |
> 0 / 0 bytes (+0/+0 bytes) | 0 / 0 bytes
> (+0/+0 bytes) | 0 / 0 bytes (+0/+0 bytes) |
>
> | bb3425b4-1c48-3f9c-9d72-cc73b2b3b7a9 | UnpackContent |
> original, success | PutFile |
> 2 / 14.55 MB (+2/+14.55 MB) | 2 / 14.55 MB
> (+2/+14.55 MB) | 0 / 0 bytes (+0/+0 bytes) |
>
>
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>
> Regards,
> anup
>
>
>
> ________________________________
> The information contained in this message may be confidential and legally
> protected under applicable law. The message is intended solely for the
> addressee(s). If you are not the intended recipient, you are hereby
> notified that any use, forwarding, dissemination, or reproduction of this
> message is strictly prohibited and may be unlawful. If you are not the
> intended recipient, please contact the sender by return e-mail and destroy
> all copies of the original message.
>