You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Christian Homberg (JIRA)" <ji...@apache.org> on 2019/06/07 15:33:01 UTC
[jira] [Comment Edited] (SPARK-27966) input_file_name empty when
listing files in parallel
[ https://issues.apache.org/jira/browse/SPARK-27966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16858739#comment-16858739 ]
Christian Homberg edited comment on SPARK-27966 at 6/7/19 3:32 PM:
-------------------------------------------------------------------
I'm afraid I can't. For one thing I can't share the data, for another even I'm not always able to reproduce the bug. For exactly the same data, code and a clean environment I get filenames and sometimes I don't. All I can provide is logging information and try to debug the issue if anyone can give me pointers.
I can say though, that this has not been an issue so far with a larger spark cluster. Then again, the input data is "only" ~3,000 files, each < 1mb. So I don't get why the original cluster should have any problems regarding size.
was (Author: chr_96er):
I'm afraid I can't. For one thing I can't share the data, for another even I'm not always able to reproduce the bug. For exactly the same data, code and a clean environment I get filenames and sometimes I don't. All I can provide is logging information and try to debug the issue if anyone can give me pointers.
> input_file_name empty when listing files in parallel
> ----------------------------------------------------
>
> Key: SPARK-27966
> URL: https://issues.apache.org/jira/browse/SPARK-27966
> Project: Spark
> Issue Type: Bug
> Components: Input/Output
> Affects Versions: 2.4.0
> Environment: Databricks 5.3 (includes Apache Spark 2.4.0, Scala 2.11)
>
> Worker Type: 14.0 GB Memory, 4 Cores, 0.75 DBU Standard_DS3_v2
> Workers: 3
> Driver Type: 14.0 GB Memory, 4 Cores, 0.75 DBU Standard_DS3_v2
> Reporter: Christian Homberg
> Priority: Minor
> Attachments: input_file_name_bug
>
>
> I ran into an issue similar and probably related to SPARK-26128. The _org.apache.spark.sql.functions.input_file_name_ is sometimes empty.
>
> {code:java}
> df.select(input_file_name()).show(5,false)
> {code}
>
> {code:java}
> +-----------------+
> |input_file_name()|
> +-----------------+
> | |
> | |
> | |
> | |
> | |
> +-----------------+
> {code}
> My environment is databricks and debugging the Log4j output showed me that the issue occurred when the files are being listed in parallel, e.g. when
> {code:java}
> 19/06/06 11:50:47 INFO InMemoryFileIndex: Start listing leaf files and directories. Size of Paths: 127; threshold: 32
> 19/06/06 11:50:47 INFO InMemoryFileIndex: Listing leaf files and directories in parallel under:{code}
>
> Everything's fine as long as
> {code:java}
> 19/06/06 11:54:43 INFO InMemoryFileIndex: Start listing leaf files and directories. Size of Paths: 6; threshold: 32
> 19/06/06 11:54:43 INFO InMemoryFileIndex: Start listing leaf files and directories. Size of Paths: 0; threshold: 32
> 19/06/06 11:54:43 INFO InMemoryFileIndex: Start listing leaf files and directories. Size of Paths: 0; threshold: 32
> 19/06/06 11:54:43 INFO InMemoryFileIndex: Start listing leaf files and directories. Size of Paths: 0; threshold: 32
> 19/06/06 11:54:43 INFO InMemoryFileIndex: Start listing leaf files and directories. Size of Paths: 0; threshold: 32
> 19/06/06 11:54:43 INFO InMemoryFileIndex: Start listing leaf files and directories. Size of Paths: 0; threshold: 32
> 19/06/06 11:54:43 INFO InMemoryFileIndex: Start listing leaf files and directories. Size of Paths: 0; threshold: 32
> {code}
>
> Setting spark.sql.sources.parallelPartitionDiscovery.threshold to 9999 resolves the issue for me.
>
> *edit: the problem is not exclusively linked to listing files in parallel. I've setup a larger cluster for which after parallel file listing the input_file_name did return the correct filename. After inspecting the log4j again, I assume that it's linked to some kind of MetaStore being full. I've attached a section of the log4j output that I think should indicate why it's failing. If you need more, please let me know.*
> **
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org