You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Natu Lauchande <nl...@gmail.com> on 2016/04/02 06:17:55 UTC

Question Spark streaming - S3 textFileStream- How to get the current file name ?

Hi,

I am using spark streamming and using the input strategy of  watching for
files in S3 directories.  Using the textFileStream method in the streamming
context.

The filename contains relevant  for my pipeline manipulation i wonder if
there is a more robust way to get this name other than capturing RDD debug
information and parse the logs.

Thanks,
Natu