You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Aruna Sankaralingam <Ar...@Cormac-Corp.com> on 2017/12/08 20:56:37 UTC
ListS3 Processor Error
I am trying to get a pdf file from S3 and load to Elastic Search. The ListS3 processor is giving me this error. Could someone please let me know where I am going wrong?
20:52:25 UTC
ERROR
37d7226e-0160-1000-6049-d4c489cd32f3
ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process session due to com.amazonaws.SdkClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
20:52:25 UTC
WARNING
37d7226e-0160-1000-6049-d4c489cd32f3
ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] Processor Administratively Yielded for 1 sec due to processing failure
20:52:26 UTC
ERROR
37d7226e-0160-1000-6049-d4c489cd32f3
ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process due to com.amazonaws.SdkClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler; rolling back session: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
20:52:26 UTC
ERROR
37d7226e-0160-1000-6049-d4c489cd32f3
ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process session due to com.amazonaws.SdkClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
20:52:26 UTC
WARNING
37d7226e-0160-1000-6049-d4c489cd32f3
ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] Processor Administratively Yielded for 1 sec due to processing failure
Auto-refresh
[cid:image001.png@01D3703D.1F15E180]
RE: ListS3 Processor Error
Posted by Aruna Sankaralingam <Ar...@Cormac-Corp.com>.
Nifi version 1.4
From: Joe Witt [mailto:joe.witt@gmail.com]
Sent: Friday, December 08, 2017 3:59 PM
To: users@nifi.apache.org
Subject: Re: ListS3 Processor Error
What version of NiFi?
Looks like either a classpath/classloader issue OR the amazon client library cannot parse the response it is getting back...
The logs/nifi-app.log should have the full stack trace. If not you can turn on debug logging for that processor and perhaps then it will.
Thanks
On Fri, Dec 8, 2017 at 3:56 PM, Aruna Sankaralingam <Ar...@cormac-corp.com>> wrote:
I am trying to get a pdf file from S3 and load to Elastic Search. The ListS3 processor is giving me this error. Could someone please let me know where I am going wrong?
20:52:25 UTC
ERROR
37d7226e-0160-1000-6049-d4c489cd32f3
ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process session due to com.amazonaws.SdkClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
20:52:25 UTC
WARNING
37d7226e-0160-1000-6049-d4c489cd32f3
ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] Processor Administratively Yielded for 1 sec due to processing failure
20:52:26 UTC
ERROR
37d7226e-0160-1000-6049-d4c489cd32f3
ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process due to com.amazonaws.SdkClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler; rolling back session: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
20:52:26 UTC
ERROR
37d7226e-0160-1000-6049-d4c489cd32f3
ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process session due to com.amazonaws.SdkClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
20:52:26 UTC
WARNING
37d7226e-0160-1000-6049-d4c489cd32f3
ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] Processor Administratively Yielded for 1 sec due to processing failure
Auto-refresh
[cid:image001.png@01D3703D.8AA3E870]
Re: ListS3 Processor Error
Posted by James Wing <jv...@gmail.com>.
S3 presents an illusion of a folder tree, but the S3 API requires a clear
distinction between the name of the bucket and a key prefix to filter
objects within the bucket. ListS3 tries to mirror that terminology. I
suspect that you are trying to list the Bucket "part-d-prescription-drug",
filtering the objects in the bucket to read only keys within the Prefix
"unstructured". I recommend trying to configure your ListS3 processor
properties as follows:
* Bucket: part-d-prescription-drug
* Prefix: unstructured/
Even better, you can experiment with the AWS CLI to find the right
combination of bucket and prefix that yields the results you want, using
something like this:
aws s3 ls s3://part-d-prescription-drug/unstructured/
Using the AWS CLI can help confirm your bucket and prefix, as well as
provide a contrasting opinion on the security settings.
Thanks,
James
On Wed, Dec 13, 2017 at 11:32 AM, Aruna Sankaralingam <
Aruna.Sankaralingam@cormac-corp.com> wrote:
> James, I am sorry I am not sure if I follow that. Could you please give an
> example?
>
>
>
> *From:* James Wing [mailto:jvwing@gmail.com]
> *Sent:* Wednesday, December 13, 2017 12:55 PM
> *To:* users@nifi.apache.org
> *Subject:* Re: ListS3 Processor Error
>
>
>
> For ListS3, you will want to separate those in the Bucket and Prefix
> properties.
>
>
> On Dec 13, 2017, at 9:34 AM, Aruna Sankaralingam <
> Aruna.Sankaralingam@Cormac-Corp.com> wrote:
>
> James,
>
>
>
> “part-d-prescription-drug” is the main folder in S3 and “unstructured” is
> the sub folder inside the main folder.
>
>
>
> *From:* James Wing [mailto:jvwing@gmail.com <jv...@gmail.com>]
> *Sent:* Wednesday, December 13, 2017 1:34 AM
> *To:* users@nifi.apache.org
> *Subject:* Re: ListS3 Processor Error
>
>
>
> Are you able to list the bucket with the AWS CLI (aws s3 ls)? It can be
> helpful to compare performance between NiFi and the AWS CLI, especially if
> you are able to do so from the same machine, with the same permissions, and
> as similar bucket and prefix settings as you can manage.
>
> In the screenshot above, the bucket is shown as "part-d-prescription-drug/unstructured",
> which looks unusual to me. Is the bucket "part-d-prescription-drug" and
> the prefix "unstructured/"?
>
> Thanks,
>
> James
>
>
>
> On Tue, Dec 12, 2017 at 7:34 AM, Aruna Sankaralingam <
> Aruna.Sankaralingam@cormac-corp.com> wrote:
>
> Joe,
>
>
>
> No, I don’t have anything in between AWS and NiFi.
>
> NiFi is installed in one of the EC2 instance in AWS – N.Virginia Region
>
> S3 is also in N.Virginia Region
>
>
>
> *From:* Joe Witt [mailto:joe.witt@gmail.com]
> *Sent:* Monday, December 11, 2017 1:28 PM
> *To:* users@nifi.apache.org
> *Subject:* Re: ListS3 Processor Error
>
>
>
> The XML response is truncated for some reason as implied by the following.
> Do you have any devices/software/systems/proxies in between your NiFi and
> the amazon service? Are you able to manually issue the request and get the
> response you expect?
>
>
>
> 2017-12-11 18:01:02,875 ERROR [Timer-Driven Process Thread-6]
> org.apache.nifi.processors.aws.s3.ListS3 ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3]
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process session
> due to com.amazonaws.SdkClientException: Failed to parse XML document
> with handler class com.amazonaws.services.s3.model.transform.
> XmlResponsesSaxParser$ListBucketHandler: {}
>
> com.amazonaws.SdkClientException: Failed to parse XML document with
> handler class com.amazonaws.services.s3.model.transform.
> XmlResponsesSaxParser$ListBucketHandler
>
> at com.amazonaws.services.s3.model.transform.
> XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:156)
>
> at com.amazonaws.services.s3.model.transform.
> XmlResponsesSaxParser.parseListBucketObjectsResponse
> (XmlResponsesSaxParser.java:298)
>
> at com.amazonaws.services.s3.model.transform.Unmarshallers$
> ListObjectsUnmarshaller.unmarshall(Unmarshallers.java:70)
>
> at com.amazonaws.services.s3.model.transform.Unmarshallers$
> ListObjectsUnmarshaller.unmarshall(Unmarshallers.java:59)
>
> at com.amazonaws.services.s3.internal.S3XmlResponseHandler.
> handle(S3XmlResponseHandler.java:62)
>
> at com.amazonaws.services.s3.internal.S3XmlResponseHandler.
> handle(S3XmlResponseHandler.java:31)
>
> at com.amazonaws.http.response.AwsResponseHandlerAdapter.
> handle(AwsResponseHandlerAdapter.java:70)
>
> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.
> handleResponse(AmazonHttpClient.java:1444)
>
> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.
> executeOneRequest(AmazonHttpClient.java:1151)
>
> at com.amazonaws.http.AmazonHttpClient$
> RequestExecutor.executeHelper(AmazonHttpClient.java:964)
>
> at com.amazonaws.http.AmazonHttpClient$
> RequestExecutor.doExecute(AmazonHttpClient.java:676)
>
> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.
> executeWithTimer(AmazonHttpClient.java:650)
>
> at com.amazonaws.http.AmazonHttpClient$
> RequestExecutor.execute(AmazonHttpClient.java:633)
>
> at com.amazonaws.http.AmazonHttpClient$
> RequestExecutor.access$300(AmazonHttpClient.java:601)
>
> at com.amazonaws.http.AmazonHttpClient$
> RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:583)
>
> at com.amazonaws.http.AmazonHttpClient.execute(
> AmazonHttpClient.java:447)
>
> at com.amazonaws.services.s3.AmazonS3Client.invoke(
> AmazonS3Client.java:4137)
>
> at com.amazonaws.services.s3.AmazonS3Client.invoke(
> AmazonS3Client.java:4079)
>
> at com.amazonaws.services.s3.AmazonS3Client.listObjects(
> AmazonS3Client.java:819)
>
> at org.apache.nifi.processors.aws.s3.ListS3$
> S3ObjectBucketLister.listVersions(ListS3.java:314)
>
> at org.apache.nifi.processors.aws.s3.ListS3.onTrigger(
> ListS3.java:208)
>
> at org.apache.nifi.processor.AbstractProcessor.onTrigger(
> AbstractProcessor.java:27)
>
> at org.apache.nifi.controller.StandardProcessorNode.onTrigger(
> StandardProcessorNode.java:1119)
>
> at org.apache.nifi.controller.tasks.
> ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
>
> at org.apache.nifi.controller.tasks.
> ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
>
> at org.apache.nifi.controller.scheduling.
> TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
>
> at java.util.concurrent.Executors$RunnableAdapter.
> call(Executors.java:511)
>
> at java.util.concurrent.FutureTask.runAndReset(
> FutureTask.java:308)
>
> at java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>
> at java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1149)
>
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:624)
>
> at java.lang.Thread.run(Thread.java:748)
>
> Caused by: org.xml.sax.SAXParseException: Premature end of file.
>
> at com.sun.org.apache.xerces.internal.util.
> ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:203)
>
> at com.sun.org.apache.xerces.internal.util.
> ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
>
> at com.sun.org.apache.xerces.internal.impl.
> XMLErrorReporter.reportError(XMLErrorReporter.java:400)
>
> at com.sun.org.apache.xerces.internal.impl.
> XMLErrorReporter.reportError(XMLErrorReporter.java:327)
>
> at com.sun.org.apache.xerces.internal.impl.XMLScanner.
> reportFatalError(XMLScanner.java:1472)
>
> at com.sun.org.apache.xerces.internal.impl.
> XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:1014)
>
> at com.sun.org.apache.xerces.internal.impl.
> XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:602)
>
> at com.sun.org.apache.xerces.internal.impl.
> XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:112)
>
> at com.sun.org.apache.xerces.internal.impl.
> XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl
> .java:505)
>
> at com.sun.org.apache.xerces.internal.parsers.
> XML11Configuration.parse(XML11Configuration.java:841)
>
> at com.sun.org.apache.xerces.internal.parsers.
> XML11Configuration.parse(XML11Configuration.java:770)
>
> at com.sun.org.apache.xerces.internal.parsers.XMLParser.
> parse(XMLParser.java:141)
>
> at com.sun.org.apache.xerces.internal.parsers.
> AbstractSAXParser.parse(AbstractSAXParser.java:1213)
>
> at com.amazonaws.services.s3.model.transform.
> XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:142)
>
> ... 32 common frames omitted
>
>
>
>
>
> On Mon, Dec 11, 2017 at 1:07 PM, Aruna Sankaralingam <
> Aruna.Sankaralingam@cormac-corp.com> wrote:
>
> Attached my nifi-app.log. Could you please let me know what went wrong?
>
>
>
> *From:* Joe Witt [mailto:joe.witt@gmail.com]
> *Sent:* Friday, December 08, 2017 4:04 PM
>
>
> *To:* users@nifi.apache.org
> *Subject:* Re: ListS3 Processor Error
>
>
>
> Here is an example I found for another processor
>
>
>
> https://mail-archives.apache.org/mod_mbox/nifi-dev/201509.mbox/%
> 3CCAFddr26AEVqnoQ=mWr7DSNDFVrr9NuYy9GCcXg=4FYyCQAbbuw@mail.gmail.com%3E
>
>
>
> Thanks
>
>
>
> On Fri, Dec 8, 2017 at 4:02 PM, Aruna Sankaralingam <
> Aruna.Sankaralingam@cormac-corp.com> wrote:
>
> Joe,
>
> Could you please let me know how to turn on the debug logging?
>
>
>
> *From:* Joe Witt [mailto:joe.witt@gmail.com]
> *Sent:* Friday, December 08, 2017 3:59 PM
> *To:* users@nifi.apache.org
> *Subject:* Re: ListS3 Processor Error
>
>
>
> What version of NiFi?
>
>
>
> Looks like either a classpath/classloader issue OR the amazon client
> library cannot parse the response it is getting back...
>
>
>
> The logs/nifi-app.log should have the full stack trace. If not you can
> turn on debug logging for that processor and perhaps then it will.
>
>
>
> Thanks
>
>
>
> On Fri, Dec 8, 2017 at 3:56 PM, Aruna Sankaralingam <
> Aruna.Sankaralingam@cormac-corp.com> wrote:
>
> I am trying to get a pdf file from S3 and load to Elastic Search. The
> ListS3 processor is giving me this error. Could someone please let me know
> where I am going wrong?
>
>
>
> *20:52:25 UTC*
>
> *ERROR*
>
> *37d7226e-0160-1000-6049-d4c489cd32f3*
>
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3]
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process session
> due to com.amazonaws.SdkClientException: Failed to parse XML document
> with handler class com.amazonaws.services.s3.model.transform.
> XmlResponsesSaxParser$ListBucketHandler: Failed to parse XML document
> with handler class com.amazonaws.services.s3.model.transform.
> XmlResponsesSaxParser$ListBucketHandler
>
> *20:52:25 UTC*
>
> *WARNING*
>
> *37d7226e-0160-1000-6049-d4c489cd32f3*
>
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] Processor
> Administratively Yielded for 1 sec due to processing failure
>
> *20:52:26 UTC*
>
> *ERROR*
>
> *37d7226e-0160-1000-6049-d4c489cd32f3*
>
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3]
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process due to
> com.amazonaws.SdkClientException: Failed to parse XML document with
> handler class com.amazonaws.services.s3.model.transform.
> XmlResponsesSaxParser$ListBucketHandler; rolling back session: Failed to
> parse XML document with handler class com.amazonaws.services.s3.
> model.transform.XmlResponsesSaxParser$ListBucketHandler
>
> *20:52:26 UTC*
>
> *ERROR*
>
> *37d7226e-0160-1000-6049-d4c489cd32f3*
>
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3]
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process session
> due to com.amazonaws.SdkClientException: Failed to parse XML document
> with handler class com.amazonaws.services.s3.model.transform.
> XmlResponsesSaxParser$ListBucketHandler: Failed to parse XML document
> with handler class com.amazonaws.services.s3.model.transform.
> XmlResponsesSaxParser$ListBucketHandler
>
> *20:52:26 UTC*
>
> *WARNING*
>
> *37d7226e-0160-1000-6049-d4c489cd32f3*
>
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] Processor
> Administratively Yielded for 1 sec due to processing failure
>
> Auto-refresh
>
>
>
> <image001.png>
>
>
>
>
>
>
>
>
>
>
RE: ListS3 Processor Error
Posted by Aruna Sankaralingam <Ar...@Cormac-Corp.com>.
James, I am sorry I am not sure if I follow that. Could you please give an example?
From: James Wing [mailto:jvwing@gmail.com]
Sent: Wednesday, December 13, 2017 12:55 PM
To: users@nifi.apache.org
Subject: Re: ListS3 Processor Error
For ListS3, you will want to separate those in the Bucket and Prefix properties.
On Dec 13, 2017, at 9:34 AM, Aruna Sankaralingam <Ar...@Cormac-Corp.com>> wrote:
James,
“part-d-prescription-drug” is the main folder in S3 and “unstructured” is the sub folder inside the main folder.
From: James Wing [mailto:jvwing@gmail.com]
Sent: Wednesday, December 13, 2017 1:34 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: ListS3 Processor Error
Are you able to list the bucket with the AWS CLI (aws s3 ls)? It can be helpful to compare performance between NiFi and the AWS CLI, especially if you are able to do so from the same machine, with the same permissions, and as similar bucket and prefix settings as you can manage.
In the screenshot above, the bucket is shown as "part-d-prescription-drug/unstructured", which looks unusual to me. Is the bucket "part-d-prescription-drug" and the prefix "unstructured/"?
Thanks,
James
On Tue, Dec 12, 2017 at 7:34 AM, Aruna Sankaralingam <Ar...@cormac-corp.com>> wrote:
Joe,
No, I don’t have anything in between AWS and NiFi.
NiFi is installed in one of the EC2 instance in AWS – N.Virginia Region
S3 is also in N.Virginia Region
From: Joe Witt [mailto:joe.witt@gmail.com<ma...@gmail.com>]
Sent: Monday, December 11, 2017 1:28 PM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: ListS3 Processor Error
The XML response is truncated for some reason as implied by the following. Do you have any devices/software/systems/proxies in between your NiFi and the amazon service? Are you able to manually issue the request and get the response you expect?
2017-12-11 18:01:02,875 ERROR [Timer-Driven Process Thread-6] org.apache.nifi.processors.aws.s3.ListS3 ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process session due to com.amazonaws.SdkClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler: {}
com.amazonaws.SdkClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:156)
at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseListBucketObjectsResponse(XmlResponsesSaxParser.java:298)
at com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsUnmarshaller.unmarshall(Unmarshallers.java:70)
at com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsUnmarshaller.unmarshall(Unmarshallers.java:59)
at com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:62)
at com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:31)
at com.amazonaws.http.response.AwsResponseHandlerAdapter.handle(AwsResponseHandlerAdapter.java:70)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleResponse(AmazonHttpClient.java:1444)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1151)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:964)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:676)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:650)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:633)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:601)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:583)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:447)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4137)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4079)
at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:819)
at org.apache.nifi.processors.aws.s3.ListS3$S3ObjectBucketLister.listVersions(ListS3.java:314)
at org.apache.nifi.processors.aws.s3.ListS3.onTrigger(ListS3.java:208)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1119)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.xml.sax.SAXParseException: Premature end of file.
at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:203)
at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:400)
at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:327)
at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1472)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:1014)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:602)
at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:112)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:505)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:841)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:770)
at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)
at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:142)
... 32 common frames omitted
On Mon, Dec 11, 2017 at 1:07 PM, Aruna Sankaralingam <Ar...@cormac-corp.com>> wrote:
Attached my nifi-app.log. Could you please let me know what went wrong?
From: Joe Witt [mailto:joe.witt@gmail.com<ma...@gmail.com>]
Sent: Friday, December 08, 2017 4:04 PM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: ListS3 Processor Error
Here is an example I found for another processor
https://mail-archives.apache.org/mod_mbox/nifi-dev/201509.mbox/%3CCAFddr26AEVqnoQ=mWr7DSNDFVrr9NuYy9GCcXg=4FYyCQAbbuw@mail.gmail.com%3E
Thanks
On Fri, Dec 8, 2017 at 4:02 PM, Aruna Sankaralingam <Ar...@cormac-corp.com>> wrote:
Joe,
Could you please let me know how to turn on the debug logging?
From: Joe Witt [mailto:joe.witt@gmail.com<ma...@gmail.com>]
Sent: Friday, December 08, 2017 3:59 PM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: ListS3 Processor Error
What version of NiFi?
Looks like either a classpath/classloader issue OR the amazon client library cannot parse the response it is getting back...
The logs/nifi-app.log should have the full stack trace. If not you can turn on debug logging for that processor and perhaps then it will.
Thanks
On Fri, Dec 8, 2017 at 3:56 PM, Aruna Sankaralingam <Ar...@cormac-corp.com>> wrote:
I am trying to get a pdf file from S3 and load to Elastic Search. The ListS3 processor is giving me this error. Could someone please let me know where I am going wrong?
20:52:25 UTC
ERROR
37d7226e-0160-1000-6049-d4c489cd32f3
ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process session due to com.amazonaws.SdkClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
20:52:25 UTC
WARNING
37d7226e-0160-1000-6049-d4c489cd32f3
ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] Processor Administratively Yielded for 1 sec due to processing failure
20:52:26 UTC
ERROR
37d7226e-0160-1000-6049-d4c489cd32f3
ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process due to com.amazonaws.SdkClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler; rolling back session: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
20:52:26 UTC
ERROR
37d7226e-0160-1000-6049-d4c489cd32f3
ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process session due to com.amazonaws.SdkClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
20:52:26 UTC
WARNING
37d7226e-0160-1000-6049-d4c489cd32f3
ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] Processor Administratively Yielded for 1 sec due to processing failure
Auto-refresh
<image001.png>
Re: ListS3 Processor Error
Posted by James Wing <jv...@gmail.com>.
For ListS3, you will want to separate those in the Bucket and Prefix properties.
> On Dec 13, 2017, at 9:34 AM, Aruna Sankaralingam <Ar...@Cormac-Corp.com> wrote:
>
> James,
>
> “part-d-prescription-drug” is the main folder in S3 and “unstructured” is the sub folder inside the main folder.
>
> From: James Wing [mailto:jvwing@gmail.com]
> Sent: Wednesday, December 13, 2017 1:34 AM
> To: users@nifi.apache.org
> Subject: Re: ListS3 Processor Error
>
> Are you able to list the bucket with the AWS CLI (aws s3 ls)? It can be helpful to compare performance between NiFi and the AWS CLI, especially if you are able to do so from the same machine, with the same permissions, and as similar bucket and prefix settings as you can manage.
>
> In the screenshot above, the bucket is shown as "part-d-prescription-drug/unstructured", which looks unusual to me. Is the bucket "part-d-prescription-drug" and the prefix "unstructured/"?
>
> Thanks,
>
> James
>
> On Tue, Dec 12, 2017 at 7:34 AM, Aruna Sankaralingam <Ar...@cormac-corp.com> wrote:
> Joe,
>
> No, I don’t have anything in between AWS and NiFi.
> NiFi is installed in one of the EC2 instance in AWS – N.Virginia Region
> S3 is also in N.Virginia Region
>
> From: Joe Witt [mailto:joe.witt@gmail.com]
> Sent: Monday, December 11, 2017 1:28 PM
> To: users@nifi.apache.org
> Subject: Re: ListS3 Processor Error
>
> The XML response is truncated for some reason as implied by the following. Do you have any devices/software/systems/proxies in between your NiFi and the amazon service? Are you able to manually issue the request and get the response you expect?
>
> 2017-12-11 18:01:02,875 ERROR [Timer-Driven Process Thread-6] org.apache.nifi.processors.aws.s3.ListS3 ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process session due to com.amazonaws.SdkClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler: {}
> com.amazonaws.SdkClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
> at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:156)
> at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseListBucketObjectsResponse(XmlResponsesSaxParser.java:298)
> at com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsUnmarshaller.unmarshall(Unmarshallers.java:70)
> at com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsUnmarshaller.unmarshall(Unmarshallers.java:59)
> at com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:62)
> at com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:31)
> at com.amazonaws.http.response.AwsResponseHandlerAdapter.handle(AwsResponseHandlerAdapter.java:70)
> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleResponse(AmazonHttpClient.java:1444)
> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1151)
> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:964)
> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:676)
> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:650)
> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:633)
> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:601)
> at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:583)
> at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:447)
> at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4137)
> at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4079)
> at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:819)
> at org.apache.nifi.processors.aws.s3.ListS3$S3ObjectBucketLister.listVersions(ListS3.java:314)
> at org.apache.nifi.processors.aws.s3.ListS3.onTrigger(ListS3.java:208)
> at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1119)
> at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
> at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.xml.sax.SAXParseException: Premature end of file.
> at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:203)
> at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
> at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:400)
> at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:327)
> at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1472)
> at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:1014)
> at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:602)
> at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:112)
> at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:505)
> at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:841)
> at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:770)
> at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
> at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)
> at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:142)
> ... 32 common frames omitted
>
>
> On Mon, Dec 11, 2017 at 1:07 PM, Aruna Sankaralingam <Ar...@cormac-corp.com> wrote:
> Attached my nifi-app.log. Could you please let me know what went wrong?
>
> From: Joe Witt [mailto:joe.witt@gmail.com]
> Sent: Friday, December 08, 2017 4:04 PM
>
> To: users@nifi.apache.org
> Subject: Re: ListS3 Processor Error
>
> Here is an example I found for another processor
>
> https://mail-archives.apache.org/mod_mbox/nifi-dev/201509.mbox/%3CCAFddr26AEVqnoQ=mWr7DSNDFVrr9NuYy9GCcXg=4FYyCQAbbuw@mail.gmail.com%3E
>
> Thanks
>
> On Fri, Dec 8, 2017 at 4:02 PM, Aruna Sankaralingam <Ar...@cormac-corp.com> wrote:
> Joe,
> Could you please let me know how to turn on the debug logging?
>
> From: Joe Witt [mailto:joe.witt@gmail.com]
> Sent: Friday, December 08, 2017 3:59 PM
> To: users@nifi.apache.org
> Subject: Re: ListS3 Processor Error
>
> What version of NiFi?
>
> Looks like either a classpath/classloader issue OR the amazon client library cannot parse the response it is getting back...
>
> The logs/nifi-app.log should have the full stack trace. If not you can turn on debug logging for that processor and perhaps then it will.
>
> Thanks
>
> On Fri, Dec 8, 2017 at 3:56 PM, Aruna Sankaralingam <Ar...@cormac-corp.com> wrote:
> I am trying to get a pdf file from S3 and load to Elastic Search. The ListS3 processor is giving me this error. Could someone please let me know where I am going wrong?
>
> 20:52:25 UTC
> ERROR
> 37d7226e-0160-1000-6049-d4c489cd32f3
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process session due to com.amazonaws.SdkClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
> 20:52:25 UTC
> WARNING
> 37d7226e-0160-1000-6049-d4c489cd32f3
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] Processor Administratively Yielded for 1 sec due to processing failure
> 20:52:26 UTC
> ERROR
> 37d7226e-0160-1000-6049-d4c489cd32f3
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process due to com.amazonaws.SdkClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler; rolling back session: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
> 20:52:26 UTC
> ERROR
> 37d7226e-0160-1000-6049-d4c489cd32f3
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process session due to com.amazonaws.SdkClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
> 20:52:26 UTC
> WARNING
> 37d7226e-0160-1000-6049-d4c489cd32f3
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] Processor Administratively Yielded for 1 sec due to processing failure
> Auto-refresh
>
> <image001.png>
>
>
>
>
RE: ListS3 Processor Error
Posted by Aruna Sankaralingam <Ar...@Cormac-Corp.com>.
James,
“part-d-prescription-drug” is the main folder in S3 and “unstructured” is the sub folder inside the main folder.
From: James Wing [mailto:jvwing@gmail.com]
Sent: Wednesday, December 13, 2017 1:34 AM
To: users@nifi.apache.org
Subject: Re: ListS3 Processor Error
Are you able to list the bucket with the AWS CLI (aws s3 ls)? It can be helpful to compare performance between NiFi and the AWS CLI, especially if you are able to do so from the same machine, with the same permissions, and as similar bucket and prefix settings as you can manage.
In the screenshot above, the bucket is shown as "part-d-prescription-drug/unstructured", which looks unusual to me. Is the bucket "part-d-prescription-drug" and the prefix "unstructured/"?
Thanks,
James
On Tue, Dec 12, 2017 at 7:34 AM, Aruna Sankaralingam <Ar...@cormac-corp.com>> wrote:
Joe,
No, I don’t have anything in between AWS and NiFi.
NiFi is installed in one of the EC2 instance in AWS – N.Virginia Region
S3 is also in N.Virginia Region
From: Joe Witt [mailto:joe.witt@gmail.com<ma...@gmail.com>]
Sent: Monday, December 11, 2017 1:28 PM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: ListS3 Processor Error
The XML response is truncated for some reason as implied by the following. Do you have any devices/software/systems/proxies in between your NiFi and the amazon service? Are you able to manually issue the request and get the response you expect?
2017-12-11 18:01:02,875 ERROR [Timer-Driven Process Thread-6] org.apache.nifi.processors.aws.s3.ListS3 ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process session due to com.amazonaws.SdkClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler: {}
com.amazonaws.SdkClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:156)
at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseListBucketObjectsResponse(XmlResponsesSaxParser.java:298)
at com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsUnmarshaller.unmarshall(Unmarshallers.java:70)
at com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsUnmarshaller.unmarshall(Unmarshallers.java:59)
at com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:62)
at com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:31)
at com.amazonaws.http.response.AwsResponseHandlerAdapter.handle(AwsResponseHandlerAdapter.java:70)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleResponse(AmazonHttpClient.java:1444)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1151)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:964)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:676)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:650)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:633)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:601)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:583)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:447)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4137)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4079)
at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:819)
at org.apache.nifi.processors.aws.s3.ListS3$S3ObjectBucketLister.listVersions(ListS3.java:314)
at org.apache.nifi.processors.aws.s3.ListS3.onTrigger(ListS3.java:208)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1119)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.xml.sax.SAXParseException: Premature end of file.
at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:203)
at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:400)
at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:327)
at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1472)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:1014)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:602)
at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:112)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:505)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:841)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:770)
at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)
at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:142)
... 32 common frames omitted
On Mon, Dec 11, 2017 at 1:07 PM, Aruna Sankaralingam <Ar...@cormac-corp.com>> wrote:
Attached my nifi-app.log. Could you please let me know what went wrong?
From: Joe Witt [mailto:joe.witt@gmail.com<ma...@gmail.com>]
Sent: Friday, December 08, 2017 4:04 PM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: ListS3 Processor Error
Here is an example I found for another processor
https://mail-archives.apache.org/mod_mbox/nifi-dev/201509.mbox/%3CCAFddr26AEVqnoQ=mWr7DSNDFVrr9NuYy9GCcXg=4FYyCQAbbuw@mail.gmail.com%3E
Thanks
On Fri, Dec 8, 2017 at 4:02 PM, Aruna Sankaralingam <Ar...@cormac-corp.com>> wrote:
Joe,
Could you please let me know how to turn on the debug logging?
From: Joe Witt [mailto:joe.witt@gmail.com<ma...@gmail.com>]
Sent: Friday, December 08, 2017 3:59 PM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: ListS3 Processor Error
What version of NiFi?
Looks like either a classpath/classloader issue OR the amazon client library cannot parse the response it is getting back...
The logs/nifi-app.log should have the full stack trace. If not you can turn on debug logging for that processor and perhaps then it will.
Thanks
On Fri, Dec 8, 2017 at 3:56 PM, Aruna Sankaralingam <Ar...@cormac-corp.com>> wrote:
I am trying to get a pdf file from S3 and load to Elastic Search. The ListS3 processor is giving me this error. Could someone please let me know where I am going wrong?
20:52:25 UTC
ERROR
37d7226e-0160-1000-6049-d4c489cd32f3
ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process session due to com.amazonaws.SdkClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
20:52:25 UTC
WARNING
37d7226e-0160-1000-6049-d4c489cd32f3
ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] Processor Administratively Yielded for 1 sec due to processing failure
20:52:26 UTC
ERROR
37d7226e-0160-1000-6049-d4c489cd32f3
ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process due to com.amazonaws.SdkClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler; rolling back session: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
20:52:26 UTC
ERROR
37d7226e-0160-1000-6049-d4c489cd32f3
ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process session due to com.amazonaws.SdkClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
20:52:26 UTC
WARNING
37d7226e-0160-1000-6049-d4c489cd32f3
ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] Processor Administratively Yielded for 1 sec due to processing failure
Auto-refresh
[cid:image001.png@01D3740D.753857E0]
Re: ListS3 Processor Error
Posted by James Wing <jv...@gmail.com>.
Are you able to list the bucket with the AWS CLI (aws s3 ls)? It can be
helpful to compare performance between NiFi and the AWS CLI, especially if
you are able to do so from the same machine, with the same permissions, and
as similar bucket and prefix settings as you can manage.
In the screenshot above, the bucket is shown as
"part-d-prescription-drug/unstructured", which looks unusual to me. Is the
bucket "part-d-prescription-drug" and the prefix "unstructured/"?
Thanks,
James
On Tue, Dec 12, 2017 at 7:34 AM, Aruna Sankaralingam <
Aruna.Sankaralingam@cormac-corp.com> wrote:
> Joe,
>
>
>
> No, I don’t have anything in between AWS and NiFi.
>
> NiFi is installed in one of the EC2 instance in AWS – N.Virginia Region
>
> S3 is also in N.Virginia Region
>
>
>
> *From:* Joe Witt [mailto:joe.witt@gmail.com]
> *Sent:* Monday, December 11, 2017 1:28 PM
> *To:* users@nifi.apache.org
> *Subject:* Re: ListS3 Processor Error
>
>
>
> The XML response is truncated for some reason as implied by the following.
> Do you have any devices/software/systems/proxies in between your NiFi and
> the amazon service? Are you able to manually issue the request and get the
> response you expect?
>
>
>
> 2017-12-11 18:01:02,875 ERROR [Timer-Driven Process Thread-6]
> org.apache.nifi.processors.aws.s3.ListS3 ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3]
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process session
> due to com.amazonaws.SdkClientException: Failed to parse XML document
> with handler class com.amazonaws.services.s3.model.transform.
> XmlResponsesSaxParser$ListBucketHandler: {}
>
> com.amazonaws.SdkClientException: Failed to parse XML document with
> handler class com.amazonaws.services.s3.model.transform.
> XmlResponsesSaxParser$ListBucketHandler
>
> at com.amazonaws.services.s3.model.transform.
> XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:156)
>
> at com.amazonaws.services.s3.model.transform.
> XmlResponsesSaxParser.parseListBucketObjectsResponse
> (XmlResponsesSaxParser.java:298)
>
> at com.amazonaws.services.s3.model.transform.Unmarshallers$
> ListObjectsUnmarshaller.unmarshall(Unmarshallers.java:70)
>
> at com.amazonaws.services.s3.model.transform.Unmarshallers$
> ListObjectsUnmarshaller.unmarshall(Unmarshallers.java:59)
>
> at com.amazonaws.services.s3.internal.S3XmlResponseHandler.
> handle(S3XmlResponseHandler.java:62)
>
> at com.amazonaws.services.s3.internal.S3XmlResponseHandler.
> handle(S3XmlResponseHandler.java:31)
>
> at com.amazonaws.http.response.AwsResponseHandlerAdapter.
> handle(AwsResponseHandlerAdapter.java:70)
>
> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.
> handleResponse(AmazonHttpClient.java:1444)
>
> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.
> executeOneRequest(AmazonHttpClient.java:1151)
>
> at com.amazonaws.http.AmazonHttpClient$
> RequestExecutor.executeHelper(AmazonHttpClient.java:964)
>
> at com.amazonaws.http.AmazonHttpClient$
> RequestExecutor.doExecute(AmazonHttpClient.java:676)
>
> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.
> executeWithTimer(AmazonHttpClient.java:650)
>
> at com.amazonaws.http.AmazonHttpClient$
> RequestExecutor.execute(AmazonHttpClient.java:633)
>
> at com.amazonaws.http.AmazonHttpClient$
> RequestExecutor.access$300(AmazonHttpClient.java:601)
>
> at com.amazonaws.http.AmazonHttpClient$
> RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:583)
>
> at com.amazonaws.http.AmazonHttpClient.execute(
> AmazonHttpClient.java:447)
>
> at com.amazonaws.services.s3.AmazonS3Client.invoke(
> AmazonS3Client.java:4137)
>
> at com.amazonaws.services.s3.AmazonS3Client.invoke(
> AmazonS3Client.java:4079)
>
> at com.amazonaws.services.s3.AmazonS3Client.listObjects(
> AmazonS3Client.java:819)
>
> at org.apache.nifi.processors.aws.s3.ListS3$
> S3ObjectBucketLister.listVersions(ListS3.java:314)
>
> at org.apache.nifi.processors.aws.s3.ListS3.onTrigger(
> ListS3.java:208)
>
> at org.apache.nifi.processor.AbstractProcessor.onTrigger(
> AbstractProcessor.java:27)
>
> at org.apache.nifi.controller.StandardProcessorNode.onTrigger(
> StandardProcessorNode.java:1119)
>
> at org.apache.nifi.controller.tasks.
> ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
>
> at org.apache.nifi.controller.tasks.
> ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
>
> at org.apache.nifi.controller.scheduling.
> TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
>
> at java.util.concurrent.Executors$RunnableAdapter.
> call(Executors.java:511)
>
> at java.util.concurrent.FutureTask.runAndReset(
> FutureTask.java:308)
>
> at java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>
> at java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1149)
>
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:624)
>
> at java.lang.Thread.run(Thread.java:748)
>
> Caused by: org.xml.sax.SAXParseException: Premature end of file.
>
> at com.sun.org.apache.xerces.internal.util.
> ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:203)
>
> at com.sun.org.apache.xerces.internal.util.
> ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
>
> at com.sun.org.apache.xerces.internal.impl.
> XMLErrorReporter.reportError(XMLErrorReporter.java:400)
>
> at com.sun.org.apache.xerces.internal.impl.
> XMLErrorReporter.reportError(XMLErrorReporter.java:327)
>
> at com.sun.org.apache.xerces.internal.impl.XMLScanner.
> reportFatalError(XMLScanner.java:1472)
>
> at com.sun.org.apache.xerces.internal.impl.
> XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:1014)
>
> at com.sun.org.apache.xerces.internal.impl.
> XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:602)
>
> at com.sun.org.apache.xerces.internal.impl.
> XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:112)
>
> at com.sun.org.apache.xerces.internal.impl.
> XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl
> .java:505)
>
> at com.sun.org.apache.xerces.internal.parsers.
> XML11Configuration.parse(XML11Configuration.java:841)
>
> at com.sun.org.apache.xerces.internal.parsers.
> XML11Configuration.parse(XML11Configuration.java:770)
>
> at com.sun.org.apache.xerces.internal.parsers.XMLParser.
> parse(XMLParser.java:141)
>
> at com.sun.org.apache.xerces.internal.parsers.
> AbstractSAXParser.parse(AbstractSAXParser.java:1213)
>
> at com.amazonaws.services.s3.model.transform.
> XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:142)
>
> ... 32 common frames omitted
>
>
>
>
>
> On Mon, Dec 11, 2017 at 1:07 PM, Aruna Sankaralingam <
> Aruna.Sankaralingam@cormac-corp.com> wrote:
>
> Attached my nifi-app.log. Could you please let me know what went wrong?
>
>
>
> *From:* Joe Witt [mailto:joe.witt@gmail.com]
> *Sent:* Friday, December 08, 2017 4:04 PM
>
>
> *To:* users@nifi.apache.org
> *Subject:* Re: ListS3 Processor Error
>
>
>
> Here is an example I found for another processor
>
>
>
> https://mail-archives.apache.org/mod_mbox/nifi-dev/201509.mbox/%
> 3CCAFddr26AEVqnoQ=mWr7DSNDFVrr9NuYy9GCcXg=4FYyCQAbbuw@mail.gmail.com%3E
>
>
>
> Thanks
>
>
>
> On Fri, Dec 8, 2017 at 4:02 PM, Aruna Sankaralingam <
> Aruna.Sankaralingam@cormac-corp.com> wrote:
>
> Joe,
>
> Could you please let me know how to turn on the debug logging?
>
>
>
> *From:* Joe Witt [mailto:joe.witt@gmail.com]
> *Sent:* Friday, December 08, 2017 3:59 PM
> *To:* users@nifi.apache.org
> *Subject:* Re: ListS3 Processor Error
>
>
>
> What version of NiFi?
>
>
>
> Looks like either a classpath/classloader issue OR the amazon client
> library cannot parse the response it is getting back...
>
>
>
> The logs/nifi-app.log should have the full stack trace. If not you can
> turn on debug logging for that processor and perhaps then it will.
>
>
>
> Thanks
>
>
>
> On Fri, Dec 8, 2017 at 3:56 PM, Aruna Sankaralingam <
> Aruna.Sankaralingam@cormac-corp.com> wrote:
>
> I am trying to get a pdf file from S3 and load to Elastic Search. The
> ListS3 processor is giving me this error. Could someone please let me know
> where I am going wrong?
>
>
>
> *20:52:25 UTC*
>
> *ERROR*
>
> *37d7226e-0160-1000-6049-d4c489cd32f3*
>
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3]
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process session
> due to com.amazonaws.SdkClientException: Failed to parse XML document
> with handler class com.amazonaws.services.s3.model.transform.
> XmlResponsesSaxParser$ListBucketHandler: Failed to parse XML document
> with handler class com.amazonaws.services.s3.model.transform.
> XmlResponsesSaxParser$ListBucketHandler
>
> *20:52:25 UTC*
>
> *WARNING*
>
> *37d7226e-0160-1000-6049-d4c489cd32f3*
>
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] Processor
> Administratively Yielded for 1 sec due to processing failure
>
> *20:52:26 UTC*
>
> *ERROR*
>
> *37d7226e-0160-1000-6049-d4c489cd32f3*
>
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3]
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process due to
> com.amazonaws.SdkClientException: Failed to parse XML document with
> handler class com.amazonaws.services.s3.model.transform.
> XmlResponsesSaxParser$ListBucketHandler; rolling back session: Failed to
> parse XML document with handler class com.amazonaws.services.s3.
> model.transform.XmlResponsesSaxParser$ListBucketHandler
>
> *20:52:26 UTC*
>
> *ERROR*
>
> *37d7226e-0160-1000-6049-d4c489cd32f3*
>
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3]
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process session
> due to com.amazonaws.SdkClientException: Failed to parse XML document
> with handler class com.amazonaws.services.s3.model.transform.
> XmlResponsesSaxParser$ListBucketHandler: Failed to parse XML document
> with handler class com.amazonaws.services.s3.model.transform.
> XmlResponsesSaxParser$ListBucketHandler
>
> *20:52:26 UTC*
>
> *WARNING*
>
> *37d7226e-0160-1000-6049-d4c489cd32f3*
>
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] Processor
> Administratively Yielded for 1 sec due to processing failure
>
> Auto-refresh
>
>
>
>
>
>
>
>
>
RE: ListS3 Processor Error
Posted by Aruna Sankaralingam <Ar...@Cormac-Corp.com>.
Joe,
No, I don’t have anything in between AWS and NiFi.
NiFi is installed in one of the EC2 instance in AWS – N.Virginia Region
S3 is also in N.Virginia Region
From: Joe Witt [mailto:joe.witt@gmail.com]
Sent: Monday, December 11, 2017 1:28 PM
To: users@nifi.apache.org
Subject: Re: ListS3 Processor Error
The XML response is truncated for some reason as implied by the following. Do you have any devices/software/systems/proxies in between your NiFi and the amazon service? Are you able to manually issue the request and get the response you expect?
2017-12-11 18:01:02,875 ERROR [Timer-Driven Process Thread-6] org.apache.nifi.processors.aws.s3.ListS3 ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process session due to com.amazonaws.SdkClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler: {}
com.amazonaws.SdkClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:156)
at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseListBucketObjectsResponse(XmlResponsesSaxParser.java:298)
at com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsUnmarshaller.unmarshall(Unmarshallers.java:70)
at com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsUnmarshaller.unmarshall(Unmarshallers.java:59)
at com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:62)
at com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:31)
at com.amazonaws.http.response.AwsResponseHandlerAdapter.handle(AwsResponseHandlerAdapter.java:70)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleResponse(AmazonHttpClient.java:1444)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1151)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:964)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:676)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:650)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:633)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:601)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:583)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:447)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4137)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4079)
at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:819)
at org.apache.nifi.processors.aws.s3.ListS3$S3ObjectBucketLister.listVersions(ListS3.java:314)
at org.apache.nifi.processors.aws.s3.ListS3.onTrigger(ListS3.java:208)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1119)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.xml.sax.SAXParseException: Premature end of file.
at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:203)
at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:400)
at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:327)
at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1472)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:1014)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:602)
at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:112)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:505)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:841)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:770)
at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)
at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:142)
... 32 common frames omitted
On Mon, Dec 11, 2017 at 1:07 PM, Aruna Sankaralingam <Ar...@cormac-corp.com>> wrote:
Attached my nifi-app.log. Could you please let me know what went wrong?
From: Joe Witt [mailto:joe.witt@gmail.com<ma...@gmail.com>]
Sent: Friday, December 08, 2017 4:04 PM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: ListS3 Processor Error
Here is an example I found for another processor
https://mail-archives.apache.org/mod_mbox/nifi-dev/201509.mbox/%3CCAFddr26AEVqnoQ=mWr7DSNDFVrr9NuYy9GCcXg=4FYyCQAbbuw@mail.gmail.com%3E
Thanks
On Fri, Dec 8, 2017 at 4:02 PM, Aruna Sankaralingam <Ar...@cormac-corp.com>> wrote:
Joe,
Could you please let me know how to turn on the debug logging?
From: Joe Witt [mailto:joe.witt@gmail.com<ma...@gmail.com>]
Sent: Friday, December 08, 2017 3:59 PM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: ListS3 Processor Error
What version of NiFi?
Looks like either a classpath/classloader issue OR the amazon client library cannot parse the response it is getting back...
The logs/nifi-app.log should have the full stack trace. If not you can turn on debug logging for that processor and perhaps then it will.
Thanks
On Fri, Dec 8, 2017 at 3:56 PM, Aruna Sankaralingam <Ar...@cormac-corp.com>> wrote:
I am trying to get a pdf file from S3 and load to Elastic Search. The ListS3 processor is giving me this error. Could someone please let me know where I am going wrong?
20:52:25 UTC
ERROR
37d7226e-0160-1000-6049-d4c489cd32f3
ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process session due to com.amazonaws.SdkClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
20:52:25 UTC
WARNING
37d7226e-0160-1000-6049-d4c489cd32f3
ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] Processor Administratively Yielded for 1 sec due to processing failure
20:52:26 UTC
ERROR
37d7226e-0160-1000-6049-d4c489cd32f3
ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process due to com.amazonaws.SdkClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler; rolling back session: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
20:52:26 UTC
ERROR
37d7226e-0160-1000-6049-d4c489cd32f3
ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process session due to com.amazonaws.SdkClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
20:52:26 UTC
WARNING
37d7226e-0160-1000-6049-d4c489cd32f3
ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] Processor Administratively Yielded for 1 sec due to processing failure
Auto-refresh
[cid:image001.png@01D37334.B8D8B5A0]
Re: ListS3 Processor Error
Posted by Joe Witt <jo...@gmail.com>.
The XML response is truncated for some reason as implied by the following.
Do you have any devices/software/systems/proxies in between your NiFi and
the amazon service? Are you able to manually issue the request and get the
response you expect?
2017-12-11 18:01:02,875 ERROR [Timer-Driven Process Thread-6]
org.apache.nifi.processors.aws.s3.ListS3
ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3]
ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process session
due to com.amazonaws.SdkClientException: Failed to parse XML document with
handler class
com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler:
{}
com.amazonaws.SdkClientException: Failed to parse XML document with handler
class
com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
at
com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:156)
at
com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseListBucketObjectsResponse(XmlResponsesSaxParser.java:298)
at
com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsUnmarshaller.unmarshall(Unmarshallers.java:70)
at
com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsUnmarshaller.unmarshall(Unmarshallers.java:59)
at
com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:62)
at
com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:31)
at
com.amazonaws.http.response.AwsResponseHandlerAdapter.handle(AwsResponseHandlerAdapter.java:70)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleResponse(AmazonHttpClient.java:1444)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1151)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:964)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:676)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:650)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:633)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:601)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:583)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:447)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4137)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4079)
at
com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:819)
at
org.apache.nifi.processors.aws.s3.ListS3$S3ObjectBucketLister.listVersions(ListS3.java:314)
at org.apache.nifi.processors.aws.s3.ListS3.onTrigger(ListS3.java:208)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1119)
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.xml.sax.SAXParseException: Premature end of file.
at
com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:203)
at
com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
at
com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:400)
at
com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:327)
at
com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1472)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:1014)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:602)
at
com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:112)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:505)
at
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:841)
at
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:770)
at
com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
at
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)
at
com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:142)
... 32 common frames omitted
On Mon, Dec 11, 2017 at 1:07 PM, Aruna Sankaralingam <
Aruna.Sankaralingam@cormac-corp.com> wrote:
> Attached my nifi-app.log. Could you please let me know what went wrong?
>
>
>
> *From:* Joe Witt [mailto:joe.witt@gmail.com]
> *Sent:* Friday, December 08, 2017 4:04 PM
>
> *To:* users@nifi.apache.org
> *Subject:* Re: ListS3 Processor Error
>
>
>
> Here is an example I found for another processor
>
>
>
> https://mail-archives.apache.org/mod_mbox/nifi-dev/201509.mbox/%
> 3CCAFddr26AEVqnoQ=mWr7DSNDFVrr9NuYy9GCcXg=4FYyCQAbbuw@mail.gmail.com%3E
>
>
>
> Thanks
>
>
>
> On Fri, Dec 8, 2017 at 4:02 PM, Aruna Sankaralingam <
> Aruna.Sankaralingam@cormac-corp.com> wrote:
>
> Joe,
>
> Could you please let me know how to turn on the debug logging?
>
>
>
> *From:* Joe Witt [mailto:joe.witt@gmail.com]
> *Sent:* Friday, December 08, 2017 3:59 PM
> *To:* users@nifi.apache.org
> *Subject:* Re: ListS3 Processor Error
>
>
>
> What version of NiFi?
>
>
>
> Looks like either a classpath/classloader issue OR the amazon client
> library cannot parse the response it is getting back...
>
>
>
> The logs/nifi-app.log should have the full stack trace. If not you can
> turn on debug logging for that processor and perhaps then it will.
>
>
>
> Thanks
>
>
>
> On Fri, Dec 8, 2017 at 3:56 PM, Aruna Sankaralingam <
> Aruna.Sankaralingam@cormac-corp.com> wrote:
>
> I am trying to get a pdf file from S3 and load to Elastic Search. The
> ListS3 processor is giving me this error. Could someone please let me know
> where I am going wrong?
>
>
>
> *20:52:25 UTC*
>
> *ERROR*
>
> *37d7226e-0160-1000-6049-d4c489cd32f3*
>
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3]
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process session
> due to com.amazonaws.SdkClientException: Failed to parse XML document
> with handler class com.amazonaws.services.s3.model.transform.
> XmlResponsesSaxParser$ListBucketHandler: Failed to parse XML document
> with handler class com.amazonaws.services.s3.model.transform.
> XmlResponsesSaxParser$ListBucketHandler
>
> *20:52:25 UTC*
>
> *WARNING*
>
> *37d7226e-0160-1000-6049-d4c489cd32f3*
>
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] Processor
> Administratively Yielded for 1 sec due to processing failure
>
> *20:52:26 UTC*
>
> *ERROR*
>
> *37d7226e-0160-1000-6049-d4c489cd32f3*
>
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3]
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process due to
> com.amazonaws.SdkClientException: Failed to parse XML document with
> handler class com.amazonaws.services.s3.model.transform.
> XmlResponsesSaxParser$ListBucketHandler; rolling back session: Failed to
> parse XML document with handler class com.amazonaws.services.s3.
> model.transform.XmlResponsesSaxParser$ListBucketHandler
>
> *20:52:26 UTC*
>
> *ERROR*
>
> *37d7226e-0160-1000-6049-d4c489cd32f3*
>
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3]
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process session
> due to com.amazonaws.SdkClientException: Failed to parse XML document
> with handler class com.amazonaws.services.s3.model.transform.
> XmlResponsesSaxParser$ListBucketHandler: Failed to parse XML document
> with handler class com.amazonaws.services.s3.model.transform.
> XmlResponsesSaxParser$ListBucketHandler
>
> *20:52:26 UTC*
>
> *WARNING*
>
> *37d7226e-0160-1000-6049-d4c489cd32f3*
>
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] Processor
> Administratively Yielded for 1 sec due to processing failure
>
> Auto-refresh
>
>
>
>
>
>
>
RE: ListS3 Processor Error
Posted by Aruna Sankaralingam <Ar...@Cormac-Corp.com>.
Thank you so much, Matt. You have answered my question.
Thanks
Aruna
From: Matt Burgess [mailto:mattyb149@apache.org]
Sent: Monday, December 11, 2017 7:26 PM
To: users@nifi.apache.org
Subject: Re: ListS3 Processor Error
Aruna,
The index and type for Elasticsearch are kinds of partitioning that can help the users organize data, but definitely help in indexing and searching data. Types are not always required, but an index is. Imagine you are trying to store a bunch of tweets from a Twitter feed (or firehose) into Elasticsearch. You could call the index "twitter" and type "tweet" for each tweet that you store in the twitter index. Now say you want to also put Twitter user information into that index. You can reuse "twitter" as the index but then specify "user" as the type. Now you can search the entire index for information in tweets and user data, or you can additionally search by type, perhaps searching only the documents with user type.
In the REST API, the index/type is specified such as GET /twitter/tweet/1 or GET /twitter/user/2 or something like that. The Elasticsearch processors use the index and type information to determine the right call to make to Elasticsearch.
You can certainly choose "pdf" as the type if you like, although depending on the sort of queries you'll be running, you may want to pick an index that incorporates any kind of data you'll be keeping together, and a type that is more domain-specific (such as "customer" if it is a PDF full of customer data). Please let me know if that answers your question, I can provide more information if need be.
Regards,
Matt
On Mon, Dec 11, 2017 at 4:18 PM, Joe Witt <jo...@gmail.com>> wrote:
For that we'll need someone familiar with that processor/Elastic to chime in :)
Thanks
On Mon, Dec 11, 2017 at 4:16 PM, Aruna Sankaralingam <Ar...@cormac-corp.com>> wrote:
Oops I overlooked the question on version that you asked. My apologies. I am using Nifi v1.4.
I moved the pdf file to another folder in the same S3 bucket and Nifi was able to pick up.
Initially it was in
S3 > part-d-prescription-drug/unstructured
I moved to
S3 > Nifi-Pecos-files
I still don’t know what was wrong with the old location. But for now, I am using the one that works.
I am trying to put this pdf file in elasticsearch.
I am not sure what I should give for “Index” and “Type”. Should the type be “PDF” ?
Thanks
Aruna
From: Joe Witt [mailto:joe.witt@gmail.com<ma...@gmail.com>]
Sent: Monday, December 11, 2017 3:32 PM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: ListS3 Processor Error
Aruna,
We'll need to know more about your config/env to help I think. I am not aware of any normal usage situation that should result in truncated responses. It is possible it is a coding bug we can resolve but I think we'll need more details. Did you see the questions in my last reply?
Thanks
On Mon, Dec 11, 2017 at 2:50 PM, Aruna Sankaralingam <Ar...@cormac-corp.com>> wrote:
Could someone please let me know what is wrong with the configuration that it is failing?
From: Aruna Sankaralingam [mailto:Aruna.Sankaralingam@Cormac-Corp.com<ma...@Cormac-Corp.com>]
Sent: Monday, December 11, 2017 1:07 PM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: RE: ListS3 Processor Error
Attached my nifi-app.log. Could you please let me know what went wrong?
From: Joe Witt [mailto:joe.witt@gmail.com]
Sent: Friday, December 08, 2017 4:04 PM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: ListS3 Processor Error
Here is an example I found for another processor
https://mail-archives.apache.org/mod_mbox/nifi-dev/201509.mbox/%3CCAFddr26AEVqnoQ=mWr7DSNDFVrr9NuYy9GCcXg=4FYyCQAbbuw@mail.gmail.com%3E
Thanks
On Fri, Dec 8, 2017 at 4:02 PM, Aruna Sankaralingam <Ar...@cormac-corp.com>> wrote:
Joe,
Could you please let me know how to turn on the debug logging?
From: Joe Witt [mailto:joe.witt@gmail.com<ma...@gmail.com>]
Sent: Friday, December 08, 2017 3:59 PM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: ListS3 Processor Error
What version of NiFi?
Looks like either a classpath/classloader issue OR the amazon client library cannot parse the response it is getting back...
The logs/nifi-app.log should have the full stack trace. If not you can turn on debug logging for that processor and perhaps then it will.
Thanks
On Fri, Dec 8, 2017 at 3:56 PM, Aruna Sankaralingam <Ar...@cormac-corp.com>> wrote:
I am trying to get a pdf file from S3 and load to Elastic Search. The ListS3 processor is giving me this error. Could someone please let me know where I am going wrong?
20:52:25 UTC
ERROR
37d7226e-0160-1000-6049-d4c489cd32f3
ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process session due to com.amazonaws.SdkClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
20:52:25 UTC
WARNING
37d7226e-0160-1000-6049-d4c489cd32f3
ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] Processor Administratively Yielded for 1 sec due to processing failure
20:52:26 UTC
ERROR
37d7226e-0160-1000-6049-d4c489cd32f3
ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process due to com.amazonaws.SdkClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler; rolling back session: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
20:52:26 UTC
ERROR
37d7226e-0160-1000-6049-d4c489cd32f3
ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process session due to com.amazonaws.SdkClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
20:52:26 UTC
WARNING
37d7226e-0160-1000-6049-d4c489cd32f3
ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] Processor Administratively Yielded for 1 sec due to processing failure
Auto-refresh
[cid:image001.png@01D37334.7FDF7180]
Re: ListS3 Processor Error
Posted by Matt Burgess <ma...@apache.org>.
Aruna,
The index and type for Elasticsearch are kinds of partitioning that can
help the users organize data, but definitely help in indexing and searching
data. Types are not always required, but an index is. Imagine you are
trying to store a bunch of tweets from a Twitter feed (or firehose) into
Elasticsearch. You could call the index "twitter" and type "tweet" for each
tweet that you store in the twitter index. Now say you want to also put
Twitter user information into that index. You can reuse "twitter" as the
index but then specify "user" as the type. Now you can search the entire
index for information in tweets and user data, or you can additionally
search by type, perhaps searching only the documents with user type.
In the REST API, the index/type is specified such as GET /twitter/tweet/1
or GET /twitter/user/2 or something like that. The Elasticsearch processors
use the index and type information to determine the right call to make to
Elasticsearch.
You can certainly choose "pdf" as the type if you like, although depending
on the sort of queries you'll be running, you may want to pick an index
that incorporates any kind of data you'll be keeping together, and a type
that is more domain-specific (such as "customer" if it is a PDF full of
customer data). Please let me know if that answers your question, I can
provide more information if need be.
Regards,
Matt
On Mon, Dec 11, 2017 at 4:18 PM, Joe Witt <jo...@gmail.com> wrote:
> For that we'll need someone familiar with that processor/Elastic to chime
> in :)
>
> Thanks
>
> On Mon, Dec 11, 2017 at 4:16 PM, Aruna Sankaralingam <
> Aruna.Sankaralingam@cormac-corp.com> wrote:
>
>> Oops I overlooked the question on version that you asked. My apologies. I
>> am using Nifi v1.4.
>>
>>
>>
>> I moved the pdf file to another folder in the same S3 bucket and Nifi was
>> able to pick up.
>>
>>
>>
>> Initially it was in
>>
>> S3 > part-d-prescription-drug/unstructured
>>
>> I moved to
>>
>> S3 > Nifi-Pecos-files
>>
>>
>>
>> I still don’t know what was wrong with the old location. But for now, I
>> am using the one that works.
>>
>>
>>
>> I am trying to put this pdf file in elasticsearch.
>>
>>
>>
>> I am not sure what I should give for “Index” and “Type”. Should the type
>> be “PDF” ?
>>
>>
>>
>> Thanks
>>
>> Aruna
>>
>>
>>
>> *From:* Joe Witt [mailto:joe.witt@gmail.com]
>> *Sent:* Monday, December 11, 2017 3:32 PM
>>
>> *To:* users@nifi.apache.org
>> *Subject:* Re: ListS3 Processor Error
>>
>>
>>
>> Aruna,
>>
>>
>>
>> We'll need to know more about your config/env to help I think. I am not
>> aware of any normal usage situation that should result in truncated
>> responses. It is possible it is a coding bug we can resolve but I think
>> we'll need more details. Did you see the questions in my last reply?
>>
>>
>>
>> Thanks
>>
>>
>>
>> On Mon, Dec 11, 2017 at 2:50 PM, Aruna Sankaralingam <
>> Aruna.Sankaralingam@cormac-corp.com> wrote:
>>
>> Could someone please let me know what is wrong with the configuration
>> that it is failing?
>>
>>
>>
>> *From:* Aruna Sankaralingam [mailto:Aruna.Sankaralingam@Cormac-Corp.com]
>> *Sent:* Monday, December 11, 2017 1:07 PM
>> *To:* users@nifi.apache.org
>> *Subject:* RE: ListS3 Processor Error
>>
>>
>>
>> Attached my nifi-app.log. Could you please let me know what went wrong?
>>
>>
>>
>> *From:* Joe Witt [mailto:joe.witt@gmail.com <jo...@gmail.com>]
>> *Sent:* Friday, December 08, 2017 4:04 PM
>> *To:* users@nifi.apache.org
>> *Subject:* Re: ListS3 Processor Error
>>
>>
>>
>> Here is an example I found for another processor
>>
>>
>>
>> https://mail-archives.apache.org/mod_mbox/nifi-dev/201509.
>> mbox/%3CCAFddr26AEVqnoQ=mWr7DSNDFVrr9NuYy9GCcXg=4FYyCQAbbuw@
>> mail.gmail.com%3E
>>
>>
>>
>> Thanks
>>
>>
>>
>> On Fri, Dec 8, 2017 at 4:02 PM, Aruna Sankaralingam <
>> Aruna.Sankaralingam@cormac-corp.com> wrote:
>>
>> Joe,
>>
>> Could you please let me know how to turn on the debug logging?
>>
>>
>>
>> *From:* Joe Witt [mailto:joe.witt@gmail.com]
>> *Sent:* Friday, December 08, 2017 3:59 PM
>> *To:* users@nifi.apache.org
>> *Subject:* Re: ListS3 Processor Error
>>
>>
>>
>> What version of NiFi?
>>
>>
>>
>> Looks like either a classpath/classloader issue OR the amazon client
>> library cannot parse the response it is getting back...
>>
>>
>>
>> The logs/nifi-app.log should have the full stack trace. If not you can
>> turn on debug logging for that processor and perhaps then it will.
>>
>>
>>
>> Thanks
>>
>>
>>
>> On Fri, Dec 8, 2017 at 3:56 PM, Aruna Sankaralingam <
>> Aruna.Sankaralingam@cormac-corp.com> wrote:
>>
>> I am trying to get a pdf file from S3 and load to Elastic Search. The
>> ListS3 processor is giving me this error. Could someone please let me know
>> where I am going wrong?
>>
>>
>>
>> *20:52:25 UTC*
>>
>> *ERROR*
>>
>> *37d7226e-0160-1000-6049-d4c489cd32f3*
>>
>> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3]
>> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process
>> session due to com.amazonaws.SdkClientException: Failed to parse XML
>> document with handler class com.amazonaws.services.s3.mode
>> l.transform.XmlResponsesSaxParser$ListBucketHandler: Failed to parse XML
>> document with handler class com.amazonaws.services.s3.mode
>> l.transform.XmlResponsesSaxParser$ListBucketHandler
>>
>> *20:52:25 UTC*
>>
>> *WARNING*
>>
>> *37d7226e-0160-1000-6049-d4c489cd32f3*
>>
>> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] Processor
>> Administratively Yielded for 1 sec due to processing failure
>>
>> *20:52:26 UTC*
>>
>> *ERROR*
>>
>> *37d7226e-0160-1000-6049-d4c489cd32f3*
>>
>> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3]
>> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process due to
>> com.amazonaws.SdkClientException: Failed to parse XML document with
>> handler class com.amazonaws.services.s3.mode
>> l.transform.XmlResponsesSaxParser$ListBucketHandler; rolling back
>> session: Failed to parse XML document with handler class
>> com.amazonaws.services.s3.model.transform.XmlResponsesSaxPar
>> ser$ListBucketHandler
>>
>> *20:52:26 UTC*
>>
>> *ERROR*
>>
>> *37d7226e-0160-1000-6049-d4c489cd32f3*
>>
>> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3]
>> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process
>> session due to com.amazonaws.SdkClientException: Failed to parse XML
>> document with handler class com.amazonaws.services.s3.mode
>> l.transform.XmlResponsesSaxParser$ListBucketHandler: Failed to parse XML
>> document with handler class com.amazonaws.services.s3.mode
>> l.transform.XmlResponsesSaxParser$ListBucketHandler
>>
>> *20:52:26 UTC*
>>
>> *WARNING*
>>
>> *37d7226e-0160-1000-6049-d4c489cd32f3*
>>
>> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] Processor
>> Administratively Yielded for 1 sec due to processing failure
>>
>> Auto-refresh
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
>
Re: ListS3 Processor Error
Posted by Joe Witt <jo...@gmail.com>.
For that we'll need someone familiar with that processor/Elastic to chime
in :)
Thanks
On Mon, Dec 11, 2017 at 4:16 PM, Aruna Sankaralingam <
Aruna.Sankaralingam@cormac-corp.com> wrote:
> Oops I overlooked the question on version that you asked. My apologies. I
> am using Nifi v1.4.
>
>
>
> I moved the pdf file to another folder in the same S3 bucket and Nifi was
> able to pick up.
>
>
>
> Initially it was in
>
> S3 > part-d-prescription-drug/unstructured
>
> I moved to
>
> S3 > Nifi-Pecos-files
>
>
>
> I still don’t know what was wrong with the old location. But for now, I am
> using the one that works.
>
>
>
> I am trying to put this pdf file in elasticsearch.
>
>
>
> I am not sure what I should give for “Index” and “Type”. Should the type
> be “PDF” ?
>
>
>
> Thanks
>
> Aruna
>
>
>
> *From:* Joe Witt [mailto:joe.witt@gmail.com]
> *Sent:* Monday, December 11, 2017 3:32 PM
>
> *To:* users@nifi.apache.org
> *Subject:* Re: ListS3 Processor Error
>
>
>
> Aruna,
>
>
>
> We'll need to know more about your config/env to help I think. I am not
> aware of any normal usage situation that should result in truncated
> responses. It is possible it is a coding bug we can resolve but I think
> we'll need more details. Did you see the questions in my last reply?
>
>
>
> Thanks
>
>
>
> On Mon, Dec 11, 2017 at 2:50 PM, Aruna Sankaralingam <
> Aruna.Sankaralingam@cormac-corp.com> wrote:
>
> Could someone please let me know what is wrong with the configuration that
> it is failing?
>
>
>
> *From:* Aruna Sankaralingam [mailto:Aruna.Sankaralingam@Cormac-Corp.com]
> *Sent:* Monday, December 11, 2017 1:07 PM
> *To:* users@nifi.apache.org
> *Subject:* RE: ListS3 Processor Error
>
>
>
> Attached my nifi-app.log. Could you please let me know what went wrong?
>
>
>
> *From:* Joe Witt [mailto:joe.witt@gmail.com <jo...@gmail.com>]
> *Sent:* Friday, December 08, 2017 4:04 PM
> *To:* users@nifi.apache.org
> *Subject:* Re: ListS3 Processor Error
>
>
>
> Here is an example I found for another processor
>
>
>
> https://mail-archives.apache.org/mod_mbox/nifi-dev/201509.mbox/%
> 3CCAFddr26AEVqnoQ=mWr7DSNDFVrr9NuYy9GCcXg=4FYyCQAbbuw@mail.gmail.com%3E
>
>
>
> Thanks
>
>
>
> On Fri, Dec 8, 2017 at 4:02 PM, Aruna Sankaralingam <
> Aruna.Sankaralingam@cormac-corp.com> wrote:
>
> Joe,
>
> Could you please let me know how to turn on the debug logging?
>
>
>
> *From:* Joe Witt [mailto:joe.witt@gmail.com]
> *Sent:* Friday, December 08, 2017 3:59 PM
> *To:* users@nifi.apache.org
> *Subject:* Re: ListS3 Processor Error
>
>
>
> What version of NiFi?
>
>
>
> Looks like either a classpath/classloader issue OR the amazon client
> library cannot parse the response it is getting back...
>
>
>
> The logs/nifi-app.log should have the full stack trace. If not you can
> turn on debug logging for that processor and perhaps then it will.
>
>
>
> Thanks
>
>
>
> On Fri, Dec 8, 2017 at 3:56 PM, Aruna Sankaralingam <
> Aruna.Sankaralingam@cormac-corp.com> wrote:
>
> I am trying to get a pdf file from S3 and load to Elastic Search. The
> ListS3 processor is giving me this error. Could someone please let me know
> where I am going wrong?
>
>
>
> *20:52:25 UTC*
>
> *ERROR*
>
> *37d7226e-0160-1000-6049-d4c489cd32f3*
>
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3]
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process session
> due to com.amazonaws.SdkClientException: Failed to parse XML document
> with handler class com.amazonaws.services.s3.model.transform.
> XmlResponsesSaxParser$ListBucketHandler: Failed to parse XML document
> with handler class com.amazonaws.services.s3.model.transform.
> XmlResponsesSaxParser$ListBucketHandler
>
> *20:52:25 UTC*
>
> *WARNING*
>
> *37d7226e-0160-1000-6049-d4c489cd32f3*
>
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] Processor
> Administratively Yielded for 1 sec due to processing failure
>
> *20:52:26 UTC*
>
> *ERROR*
>
> *37d7226e-0160-1000-6049-d4c489cd32f3*
>
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3]
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process due to
> com.amazonaws.SdkClientException: Failed to parse XML document with
> handler class com.amazonaws.services.s3.model.transform.
> XmlResponsesSaxParser$ListBucketHandler; rolling back session: Failed to
> parse XML document with handler class com.amazonaws.services.s3.
> model.transform.XmlResponsesSaxParser$ListBucketHandler
>
> *20:52:26 UTC*
>
> *ERROR*
>
> *37d7226e-0160-1000-6049-d4c489cd32f3*
>
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3]
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process session
> due to com.amazonaws.SdkClientException: Failed to parse XML document
> with handler class com.amazonaws.services.s3.model.transform.
> XmlResponsesSaxParser$ListBucketHandler: Failed to parse XML document
> with handler class com.amazonaws.services.s3.model.transform.
> XmlResponsesSaxParser$ListBucketHandler
>
> *20:52:26 UTC*
>
> *WARNING*
>
> *37d7226e-0160-1000-6049-d4c489cd32f3*
>
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] Processor
> Administratively Yielded for 1 sec due to processing failure
>
> Auto-refresh
>
>
>
>
>
>
>
>
>
RE: ListS3 Processor Error
Posted by Aruna Sankaralingam <Ar...@Cormac-Corp.com>.
Oops I overlooked the question on version that you asked. My apologies. I am using Nifi v1.4.
I moved the pdf file to another folder in the same S3 bucket and Nifi was able to pick up.
Initially it was in
S3 > part-d-prescription-drug/unstructured
I moved to
S3 > Nifi-Pecos-files
I still don’t know what was wrong with the old location. But for now, I am using the one that works.
I am trying to put this pdf file in elasticsearch.
I am not sure what I should give for “Index” and “Type”. Should the type be “PDF” ?
Thanks
Aruna
From: Joe Witt [mailto:joe.witt@gmail.com]
Sent: Monday, December 11, 2017 3:32 PM
To: users@nifi.apache.org
Subject: Re: ListS3 Processor Error
Aruna,
We'll need to know more about your config/env to help I think. I am not aware of any normal usage situation that should result in truncated responses. It is possible it is a coding bug we can resolve but I think we'll need more details. Did you see the questions in my last reply?
Thanks
On Mon, Dec 11, 2017 at 2:50 PM, Aruna Sankaralingam <Ar...@cormac-corp.com>> wrote:
Could someone please let me know what is wrong with the configuration that it is failing?
From: Aruna Sankaralingam [mailto:Aruna.Sankaralingam@Cormac-Corp.com<ma...@Cormac-Corp.com>]
Sent: Monday, December 11, 2017 1:07 PM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: RE: ListS3 Processor Error
Attached my nifi-app.log. Could you please let me know what went wrong?
From: Joe Witt [mailto:joe.witt@gmail.com]
Sent: Friday, December 08, 2017 4:04 PM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: ListS3 Processor Error
Here is an example I found for another processor
https://mail-archives.apache.org/mod_mbox/nifi-dev/201509.mbox/%3CCAFddr26AEVqnoQ=mWr7DSNDFVrr9NuYy9GCcXg=4FYyCQAbbuw@mail.gmail.com%3E
Thanks
On Fri, Dec 8, 2017 at 4:02 PM, Aruna Sankaralingam <Ar...@cormac-corp.com>> wrote:
Joe,
Could you please let me know how to turn on the debug logging?
From: Joe Witt [mailto:joe.witt@gmail.com<ma...@gmail.com>]
Sent: Friday, December 08, 2017 3:59 PM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: ListS3 Processor Error
What version of NiFi?
Looks like either a classpath/classloader issue OR the amazon client library cannot parse the response it is getting back...
The logs/nifi-app.log should have the full stack trace. If not you can turn on debug logging for that processor and perhaps then it will.
Thanks
On Fri, Dec 8, 2017 at 3:56 PM, Aruna Sankaralingam <Ar...@cormac-corp.com>> wrote:
I am trying to get a pdf file from S3 and load to Elastic Search. The ListS3 processor is giving me this error. Could someone please let me know where I am going wrong?
20:52:25 UTC
ERROR
37d7226e-0160-1000-6049-d4c489cd32f3
ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process session due to com.amazonaws.SdkClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
20:52:25 UTC
WARNING
37d7226e-0160-1000-6049-d4c489cd32f3
ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] Processor Administratively Yielded for 1 sec due to processing failure
20:52:26 UTC
ERROR
37d7226e-0160-1000-6049-d4c489cd32f3
ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process due to com.amazonaws.SdkClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler; rolling back session: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
20:52:26 UTC
ERROR
37d7226e-0160-1000-6049-d4c489cd32f3
ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process session due to com.amazonaws.SdkClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
20:52:26 UTC
WARNING
37d7226e-0160-1000-6049-d4c489cd32f3
ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] Processor Administratively Yielded for 1 sec due to processing failure
Auto-refresh
[cid:image001.png@01D3729B.645648B0]
Re: ListS3 Processor Error
Posted by Joe Witt <jo...@gmail.com>.
Aruna,
We'll need to know more about your config/env to help I think. I am not
aware of any normal usage situation that should result in truncated
responses. It is possible it is a coding bug we can resolve but I think
we'll need more details. Did you see the questions in my last reply?
Thanks
On Mon, Dec 11, 2017 at 2:50 PM, Aruna Sankaralingam <
Aruna.Sankaralingam@cormac-corp.com> wrote:
> Could someone please let me know what is wrong with the configuration that
> it is failing?
>
>
>
> *From:* Aruna Sankaralingam [mailto:Aruna.Sankaralingam@Cormac-Corp.com]
> *Sent:* Monday, December 11, 2017 1:07 PM
> *To:* users@nifi.apache.org
> *Subject:* RE: ListS3 Processor Error
>
>
>
> Attached my nifi-app.log. Could you please let me know what went wrong?
>
>
>
> *From:* Joe Witt [mailto:joe.witt@gmail.com <jo...@gmail.com>]
> *Sent:* Friday, December 08, 2017 4:04 PM
> *To:* users@nifi.apache.org
> *Subject:* Re: ListS3 Processor Error
>
>
>
> Here is an example I found for another processor
>
>
>
> https://mail-archives.apache.org/mod_mbox/nifi-dev/201509.mbox/%
> 3CCAFddr26AEVqnoQ=mWr7DSNDFVrr9NuYy9GCcXg=4FYyCQAbbuw@mail.gmail.com%3E
>
>
>
> Thanks
>
>
>
> On Fri, Dec 8, 2017 at 4:02 PM, Aruna Sankaralingam <
> Aruna.Sankaralingam@cormac-corp.com> wrote:
>
> Joe,
>
> Could you please let me know how to turn on the debug logging?
>
>
>
> *From:* Joe Witt [mailto:joe.witt@gmail.com]
> *Sent:* Friday, December 08, 2017 3:59 PM
> *To:* users@nifi.apache.org
> *Subject:* Re: ListS3 Processor Error
>
>
>
> What version of NiFi?
>
>
>
> Looks like either a classpath/classloader issue OR the amazon client
> library cannot parse the response it is getting back...
>
>
>
> The logs/nifi-app.log should have the full stack trace. If not you can
> turn on debug logging for that processor and perhaps then it will.
>
>
>
> Thanks
>
>
>
> On Fri, Dec 8, 2017 at 3:56 PM, Aruna Sankaralingam <
> Aruna.Sankaralingam@cormac-corp.com> wrote:
>
> I am trying to get a pdf file from S3 and load to Elastic Search. The
> ListS3 processor is giving me this error. Could someone please let me know
> where I am going wrong?
>
>
>
> *20:52:25 UTC*
>
> *ERROR*
>
> *37d7226e-0160-1000-6049-d4c489cd32f3*
>
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3]
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process session
> due to com.amazonaws.SdkClientException: Failed to parse XML document
> with handler class com.amazonaws.services.s3.model.transform.
> XmlResponsesSaxParser$ListBucketHandler: Failed to parse XML document
> with handler class com.amazonaws.services.s3.model.transform.
> XmlResponsesSaxParser$ListBucketHandler
>
> *20:52:25 UTC*
>
> *WARNING*
>
> *37d7226e-0160-1000-6049-d4c489cd32f3*
>
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] Processor
> Administratively Yielded for 1 sec due to processing failure
>
> *20:52:26 UTC*
>
> *ERROR*
>
> *37d7226e-0160-1000-6049-d4c489cd32f3*
>
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3]
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process due to
> com.amazonaws.SdkClientException: Failed to parse XML document with
> handler class com.amazonaws.services.s3.model.transform.
> XmlResponsesSaxParser$ListBucketHandler; rolling back session: Failed to
> parse XML document with handler class com.amazonaws.services.s3.
> model.transform.XmlResponsesSaxParser$ListBucketHandler
>
> *20:52:26 UTC*
>
> *ERROR*
>
> *37d7226e-0160-1000-6049-d4c489cd32f3*
>
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3]
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process session
> due to com.amazonaws.SdkClientException: Failed to parse XML document
> with handler class com.amazonaws.services.s3.model.transform.
> XmlResponsesSaxParser$ListBucketHandler: Failed to parse XML document
> with handler class com.amazonaws.services.s3.model.transform.
> XmlResponsesSaxParser$ListBucketHandler
>
> *20:52:26 UTC*
>
> *WARNING*
>
> *37d7226e-0160-1000-6049-d4c489cd32f3*
>
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] Processor
> Administratively Yielded for 1 sec due to processing failure
>
> Auto-refresh
>
>
>
>
>
>
>
RE: ListS3 Processor Error
Posted by Aruna Sankaralingam <Ar...@Cormac-Corp.com>.
Could someone please let me know what is wrong with the configuration that it is failing?
From: Aruna Sankaralingam [mailto:Aruna.Sankaralingam@Cormac-Corp.com]
Sent: Monday, December 11, 2017 1:07 PM
To: users@nifi.apache.org
Subject: RE: ListS3 Processor Error
Attached my nifi-app.log. Could you please let me know what went wrong?
From: Joe Witt [mailto:joe.witt@gmail.com]
Sent: Friday, December 08, 2017 4:04 PM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: ListS3 Processor Error
Here is an example I found for another processor
https://mail-archives.apache.org/mod_mbox/nifi-dev/201509.mbox/%3CCAFddr26AEVqnoQ=mWr7DSNDFVrr9NuYy9GCcXg=4FYyCQAbbuw@mail.gmail.com%3E
Thanks
On Fri, Dec 8, 2017 at 4:02 PM, Aruna Sankaralingam <Ar...@cormac-corp.com>> wrote:
Joe,
Could you please let me know how to turn on the debug logging?
From: Joe Witt [mailto:joe.witt@gmail.com<ma...@gmail.com>]
Sent: Friday, December 08, 2017 3:59 PM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: ListS3 Processor Error
What version of NiFi?
Looks like either a classpath/classloader issue OR the amazon client library cannot parse the response it is getting back...
The logs/nifi-app.log should have the full stack trace. If not you can turn on debug logging for that processor and perhaps then it will.
Thanks
On Fri, Dec 8, 2017 at 3:56 PM, Aruna Sankaralingam <Ar...@cormac-corp.com>> wrote:
I am trying to get a pdf file from S3 and load to Elastic Search. The ListS3 processor is giving me this error. Could someone please let me know where I am going wrong?
20:52:25 UTC
ERROR
37d7226e-0160-1000-6049-d4c489cd32f3
ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process session due to com.amazonaws.SdkClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
20:52:25 UTC
WARNING
37d7226e-0160-1000-6049-d4c489cd32f3
ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] Processor Administratively Yielded for 1 sec due to processing failure
20:52:26 UTC
ERROR
37d7226e-0160-1000-6049-d4c489cd32f3
ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process due to com.amazonaws.SdkClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler; rolling back session: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
20:52:26 UTC
ERROR
37d7226e-0160-1000-6049-d4c489cd32f3
ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process session due to com.amazonaws.SdkClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
20:52:26 UTC
WARNING
37d7226e-0160-1000-6049-d4c489cd32f3
ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] Processor Administratively Yielded for 1 sec due to processing failure
Auto-refresh
[cid:image001.png@01D3728F.5E522D50]
RE: ListS3 Processor Error
Posted by Aruna Sankaralingam <Ar...@Cormac-Corp.com>.
Attached my nifi-app.log. Could you please let me know what went wrong?
From: Joe Witt [mailto:joe.witt@gmail.com]
Sent: Friday, December 08, 2017 4:04 PM
To: users@nifi.apache.org
Subject: Re: ListS3 Processor Error
Here is an example I found for another processor
https://mail-archives.apache.org/mod_mbox/nifi-dev/201509.mbox/%3CCAFddr26AEVqnoQ=mWr7DSNDFVrr9NuYy9GCcXg=4FYyCQAbbuw@mail.gmail.com%3E
Thanks
On Fri, Dec 8, 2017 at 4:02 PM, Aruna Sankaralingam <Ar...@cormac-corp.com>> wrote:
Joe,
Could you please let me know how to turn on the debug logging?
From: Joe Witt [mailto:joe.witt@gmail.com<ma...@gmail.com>]
Sent: Friday, December 08, 2017 3:59 PM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: ListS3 Processor Error
What version of NiFi?
Looks like either a classpath/classloader issue OR the amazon client library cannot parse the response it is getting back...
The logs/nifi-app.log should have the full stack trace. If not you can turn on debug logging for that processor and perhaps then it will.
Thanks
On Fri, Dec 8, 2017 at 3:56 PM, Aruna Sankaralingam <Ar...@cormac-corp.com>> wrote:
I am trying to get a pdf file from S3 and load to Elastic Search. The ListS3 processor is giving me this error. Could someone please let me know where I am going wrong?
20:52:25 UTC
ERROR
37d7226e-0160-1000-6049-d4c489cd32f3
ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process session due to com.amazonaws.SdkClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
20:52:25 UTC
WARNING
37d7226e-0160-1000-6049-d4c489cd32f3
ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] Processor Administratively Yielded for 1 sec due to processing failure
20:52:26 UTC
ERROR
37d7226e-0160-1000-6049-d4c489cd32f3
ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process due to com.amazonaws.SdkClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler; rolling back session: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
20:52:26 UTC
ERROR
37d7226e-0160-1000-6049-d4c489cd32f3
ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process session due to com.amazonaws.SdkClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
20:52:26 UTC
WARNING
37d7226e-0160-1000-6049-d4c489cd32f3
ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] Processor Administratively Yielded for 1 sec due to processing failure
Auto-refresh
[cid:image001.png@01D37280.F795B400]
Re: ListS3 Processor Error
Posted by Joe Witt <jo...@gmail.com>.
Here is an example I found for another processor
https://mail-archives.apache.org/mod_mbox/nifi-dev/201509.mbox/%3CCAFddr26AEVqnoQ=mWr7DSNDFVrr9NuYy9GCcXg=4FYyCQAbbuw@mail.gmail.com%3E
Thanks
On Fri, Dec 8, 2017 at 4:02 PM, Aruna Sankaralingam <
Aruna.Sankaralingam@cormac-corp.com> wrote:
> Joe,
>
> Could you please let me know how to turn on the debug logging?
>
>
>
> *From:* Joe Witt [mailto:joe.witt@gmail.com]
> *Sent:* Friday, December 08, 2017 3:59 PM
> *To:* users@nifi.apache.org
> *Subject:* Re: ListS3 Processor Error
>
>
>
> What version of NiFi?
>
>
>
> Looks like either a classpath/classloader issue OR the amazon client
> library cannot parse the response it is getting back...
>
>
>
> The logs/nifi-app.log should have the full stack trace. If not you can
> turn on debug logging for that processor and perhaps then it will.
>
>
>
> Thanks
>
>
>
> On Fri, Dec 8, 2017 at 3:56 PM, Aruna Sankaralingam <
> Aruna.Sankaralingam@cormac-corp.com> wrote:
>
> I am trying to get a pdf file from S3 and load to Elastic Search. The
> ListS3 processor is giving me this error. Could someone please let me know
> where I am going wrong?
>
>
>
> *20:52:25 UTC*
>
> *ERROR*
>
> *37d7226e-0160-1000-6049-d4c489cd32f3*
>
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3]
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process session
> due to com.amazonaws.SdkClientException: Failed to parse XML document
> with handler class com.amazonaws.services.s3.model.transform.
> XmlResponsesSaxParser$ListBucketHandler: Failed to parse XML document
> with handler class com.amazonaws.services.s3.model.transform.
> XmlResponsesSaxParser$ListBucketHandler
>
> *20:52:25 UTC*
>
> *WARNING*
>
> *37d7226e-0160-1000-6049-d4c489cd32f3*
>
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] Processor
> Administratively Yielded for 1 sec due to processing failure
>
> *20:52:26 UTC*
>
> *ERROR*
>
> *37d7226e-0160-1000-6049-d4c489cd32f3*
>
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3]
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process due to
> com.amazonaws.SdkClientException: Failed to parse XML document with
> handler class com.amazonaws.services.s3.model.transform.
> XmlResponsesSaxParser$ListBucketHandler; rolling back session: Failed to
> parse XML document with handler class com.amazonaws.services.s3.
> model.transform.XmlResponsesSaxParser$ListBucketHandler
>
> *20:52:26 UTC*
>
> *ERROR*
>
> *37d7226e-0160-1000-6049-d4c489cd32f3*
>
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3]
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process session
> due to com.amazonaws.SdkClientException: Failed to parse XML document
> with handler class com.amazonaws.services.s3.model.transform.
> XmlResponsesSaxParser$ListBucketHandler: Failed to parse XML document
> with handler class com.amazonaws.services.s3.model.transform.
> XmlResponsesSaxParser$ListBucketHandler
>
> *20:52:26 UTC*
>
> *WARNING*
>
> *37d7226e-0160-1000-6049-d4c489cd32f3*
>
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] Processor
> Administratively Yielded for 1 sec due to processing failure
>
> Auto-refresh
>
>
>
>
>
RE: ListS3 Processor Error
Posted by Aruna Sankaralingam <Ar...@Cormac-Corp.com>.
Joe,
Could you please let me know how to turn on the debug logging?
From: Joe Witt [mailto:joe.witt@gmail.com]
Sent: Friday, December 08, 2017 3:59 PM
To: users@nifi.apache.org
Subject: Re: ListS3 Processor Error
What version of NiFi?
Looks like either a classpath/classloader issue OR the amazon client library cannot parse the response it is getting back...
The logs/nifi-app.log should have the full stack trace. If not you can turn on debug logging for that processor and perhaps then it will.
Thanks
On Fri, Dec 8, 2017 at 3:56 PM, Aruna Sankaralingam <Ar...@cormac-corp.com>> wrote:
I am trying to get a pdf file from S3 and load to Elastic Search. The ListS3 processor is giving me this error. Could someone please let me know where I am going wrong?
20:52:25 UTC
ERROR
37d7226e-0160-1000-6049-d4c489cd32f3
ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process session due to com.amazonaws.SdkClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
20:52:25 UTC
WARNING
37d7226e-0160-1000-6049-d4c489cd32f3
ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] Processor Administratively Yielded for 1 sec due to processing failure
20:52:26 UTC
ERROR
37d7226e-0160-1000-6049-d4c489cd32f3
ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process due to com.amazonaws.SdkClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler; rolling back session: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
20:52:26 UTC
ERROR
37d7226e-0160-1000-6049-d4c489cd32f3
ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process session due to com.amazonaws.SdkClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
20:52:26 UTC
WARNING
37d7226e-0160-1000-6049-d4c489cd32f3
ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] Processor Administratively Yielded for 1 sec due to processing failure
Auto-refresh
[cid:image001.png@01D3703D.EC98A8E0]
Re: ListS3 Processor Error
Posted by Joe Witt <jo...@gmail.com>.
What version of NiFi?
Looks like either a classpath/classloader issue OR the amazon client
library cannot parse the response it is getting back...
The logs/nifi-app.log should have the full stack trace. If not you can
turn on debug logging for that processor and perhaps then it will.
Thanks
On Fri, Dec 8, 2017 at 3:56 PM, Aruna Sankaralingam <
Aruna.Sankaralingam@cormac-corp.com> wrote:
> I am trying to get a pdf file from S3 and load to Elastic Search. The
> ListS3 processor is giving me this error. Could someone please let me know
> where I am going wrong?
>
>
>
> *20:52:25 UTC*
>
> *ERROR*
>
> *37d7226e-0160-1000-6049-d4c489cd32f3*
>
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3]
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process session
> due to com.amazonaws.SdkClientException: Failed to parse XML document
> with handler class com.amazonaws.services.s3.model.transform.
> XmlResponsesSaxParser$ListBucketHandler: Failed to parse XML document
> with handler class com.amazonaws.services.s3.model.transform.
> XmlResponsesSaxParser$ListBucketHandler
>
> *20:52:25 UTC*
>
> *WARNING*
>
> *37d7226e-0160-1000-6049-d4c489cd32f3*
>
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] Processor
> Administratively Yielded for 1 sec due to processing failure
>
> *20:52:26 UTC*
>
> *ERROR*
>
> *37d7226e-0160-1000-6049-d4c489cd32f3*
>
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3]
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process due to
> com.amazonaws.SdkClientException: Failed to parse XML document with
> handler class com.amazonaws.services.s3.model.transform.
> XmlResponsesSaxParser$ListBucketHandler; rolling back session: Failed to
> parse XML document with handler class com.amazonaws.services.s3.
> model.transform.XmlResponsesSaxParser$ListBucketHandler
>
> *20:52:26 UTC*
>
> *ERROR*
>
> *37d7226e-0160-1000-6049-d4c489cd32f3*
>
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3]
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] failed to process session
> due to com.amazonaws.SdkClientException: Failed to parse XML document
> with handler class com.amazonaws.services.s3.model.transform.
> XmlResponsesSaxParser$ListBucketHandler: Failed to parse XML document
> with handler class com.amazonaws.services.s3.model.transform.
> XmlResponsesSaxParser$ListBucketHandler
>
> *20:52:26 UTC*
>
> *WARNING*
>
> *37d7226e-0160-1000-6049-d4c489cd32f3*
>
> ListS3[id=37d7226e-0160-1000-6049-d4c489cd32f3] Processor
> Administratively Yielded for 1 sec due to processing failure
>
> Auto-refresh
>
>
>
>