You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Prannoy <pr...@sigmoidanalytics.com> on 2015/03/12 08:32:45 UTC
Re: Unable to read files In Yarn Mode of Spark Streaming ?
Are the files already present in HDFS before you are starting your
application ?
On Thu, Mar 12, 2015 at 11:11 AM, CH.KMVPRASAD [via Apache Spark User List]
<ml...@n3.nabble.com> wrote:
> Hi am successfully executed sparkPi example on yarn mode but i cant able
> to read files from hdfs in my streaming application using java
> I tried 'textFileStream' and fileStream methods ..........
>
> please help me ...........
> for both methods am getting null .......
>
> please help me ..
> thanks for your help..........
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-spark-user-list.1001560.n3.nabble.com/Unable-to-read-files-In-Yarn-Mode-of-Spark-Streaming-tp22008.html
> To start a new topic under Apache Spark User List, email
> ml-node+s1001560n1h33@n3.nabble.com
> To unsubscribe from Apache Spark User List, click here
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=cHJhbm5veUBzaWdtb2lkYW5hbHl0aWNzLmNvbXwxfC0xNTI2NTg4NjQ2>
> .
> NAML
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Unable-to-read-files-In-Yarn-Mode-of-Spark-Streaming-tp22008p22010.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Unable to read files In Yarn Mode of Spark Streaming ?
Posted by Prannoy <pr...@sigmoidanalytics.com>.
You can put the files from any where it just the streaming application only
picks the file having a timestamp greater than the batch was started.
No you dont need to set any properties for this to work. This is the
default behaviour of the spark streaming.
On Fri, Mar 13, 2015 at 9:38 AM, CH.KMVPRASAD [via Apache Spark User List] <
ml-node+s1001560n22025h77@n3.nabble.com> wrote:
> while running the application we need to put files into directory
> ,correct
> then i can put directly into directory or i need to move from some
> directory to required directory ..
>
> from spark streaming application point of view we need to set any
> properties ,please help me
>
>
> Thanks Prannoy..........
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-spark-user-list.1001560.n3.nabble.com/Unable-to-read-files-In-Yarn-Mode-of-Spark-Streaming-tp22008p22025.html
> To start a new topic under Apache Spark User List, email
> ml-node+s1001560n1h33@n3.nabble.com
> To unsubscribe from Apache Spark User List, click here
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=cHJhbm5veUBzaWdtb2lkYW5hbHl0aWNzLmNvbXwxfC0xNTI2NTg4NjQ2>
> .
> NAML
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Unable-to-read-files-In-Yarn-Mode-of-Spark-Streaming-tp22008p22026.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Unable to read files In Yarn Mode of Spark Streaming ?
Posted by Prannoy <pr...@sigmoidanalytics.com>.
Streaming takes only new files into consideration. Add the file after
starting the job.
On Thu, Mar 12, 2015 at 2:26 PM, CH.KMVPRASAD [via Apache Spark User List] <
ml-node+s1001560n22013h5@n3.nabble.com> wrote:
> yes !
> for testing purpose i defined single file in the specified directory
> ..........
>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-spark-user-list.1001560.n3.nabble.com/Unable-to-read-files-In-Yarn-Mode-of-Spark-Streaming-tp22008p22013.html
> To start a new topic under Apache Spark User List, email
> ml-node+s1001560n1h33@n3.nabble.com
> To unsubscribe from Apache Spark User List, click here
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=cHJhbm5veUBzaWdtb2lkYW5hbHl0aWNzLmNvbXwxfC0xNTI2NTg4NjQ2>
> .
> NAML
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Unable-to-read-files-In-Yarn-Mode-of-Spark-Streaming-tp22008p22015.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.