You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Tianbin Jiang (Jira)" <ji...@apache.org> on 2021/05/18 03:42:00 UTC
[jira] [Commented] (SPARK-14561) History Server does not see new
logs in S3
[ https://issues.apache.org/jira/browse/SPARK-14561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17346563#comment-17346563 ]
Tianbin Jiang commented on SPARK-14561:
---------------------------------------
"If you are going to use s3 as the dest, and you want to see incomplete apps, then you'll need to configure the spark job to have smaller partition size (64? 128? MB)." can anyone explain more on this? Which configuration maps to partition size: https://spark.apache.org/docs/2.4.5/configuration.html
> History Server does not see new logs in S3
> ------------------------------------------
>
> Key: SPARK-14561
> URL: https://issues.apache.org/jira/browse/SPARK-14561
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.6.1
> Reporter: Miles Crawford
> Priority: Major
> Labels: bulk-closed
>
> If you set the Spark history server to use a log directory with an s3a:// url, everything appears to work fine at first, but new log files written by applications are not picked up by the server.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org