You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Ant Kutschera <an...@gmail.com> on 2024/01/05 14:27:54 UTC

[pyspark] how to understand how jobs in UI correlate with code

Hi, I am trying to analyse the jobs created by my program and I am having
difficulty correlating the actual code, with the jobs and stages that I can
see in the UI.

I have found https://issues.apache.org/jira/browse/SPARK-8480 but I am too
much of a noob to understand if it really solves my problem. I don't think
caching is what I am looking for.

Some code:

kdf = kafka_df.selectExpr("CAST(key AS STRING)",
                     "CAST(value AS STRING)",
                     "headers",
                     "CAST(topic AS STRING)",
                     "CAST(partition AS STRING)",
                     "CAST(offset AS STRING)")

kdf = kdf.transform(extract_message, "aUsefulValue")

kdf = kdf.transform(add_unique_key)

kdf = kdf.transform(categorise)

kdf = kdf.transform(iso_date)

output = (kdf
           .writeStream
           .queryName("silver_output")
           .foreachBatch(resolveips_and_saveastable)
           .trigger(availableNow=True)
           .option("checkpointLocation", "s3a://checkpointlocation/")
# very important to be on writeStream!
           .start())

output.awaitTermination()

def resolveips_and_saveastable(bdf: DataFrame, batchid: int) -> None:
    if not bdf.isEmpty():
        bdf = resolveips(bdf, batchid)

        ...


        (bdf
         .select(["iso_date", "pid", "ip", "category", "sessionid",
"username", "lat", "lon", "city", "state",
                  "country", "continent", "message", "unique_key"])  #
must match delta table!
         .write
         .format("delta")
         .saveAsTable(f"{DB_NAME_SILVER}.maillogs", mode="append"))

The following screen shot shows the UI:


[image: image.png]

I have removed a little bit of the code, notably two calls to `count`.
Otherwise, the only thing I can recognise in the screenshot is
"saveAsTable" on job 6.

Question 1: is it possible to supply names to transformations or
groupBy or mapPartitions or withColumn, etc., so that it is easier to
find the code which a job stems from?

Question 2: where can I find detailed info about how Spark decides
what code to run in the driver, and what code to run in the workers?

Question 3: where can I find detailed info about how pyspark works, in
terms of what stuff is executed in the JVM running under the hood, and
what is run in python? I am particularly struggling with how workers
execute python code.

Many thanks,

Ant

Re: [pyspark] how to understand how jobs in UI correlate with code

Posted by Raghavendra Ganesh <ra...@gmail.com>.
Replies inline.
--
Raghavendra


On Fri, Jan 5, 2024 at 10:25 PM Ant Kutschera <an...@gmail.com>
wrote:

> Hi, I am trying to analyse the jobs created by my program and I am having
> difficulty correlating the actual code, with the jobs and stages that I can
> see in the UI.
>
> I have found https://issues.apache.org/jira/browse/SPARK-8480 but I am
> too much of a noob to understand if it really solves my problem. I don't
> think caching is what I am looking for.
>
> Some code:
>
> kdf = kafka_df.selectExpr("CAST(key AS STRING)",
>                      "CAST(value AS STRING)",
>                      "headers",
>                      "CAST(topic AS STRING)",
>                      "CAST(partition AS STRING)",
>                      "CAST(offset AS STRING)")
>
> kdf = kdf.transform(extract_message, "aUsefulValue")
>
> kdf = kdf.transform(add_unique_key)
>
> kdf = kdf.transform(categorise)
>
> kdf = kdf.transform(iso_date)
>
> output = (kdf
>            .writeStream
>            .queryName("silver_output")
>            .foreachBatch(resolveips_and_saveastable)
>            .trigger(availableNow=True)
>            .option("checkpointLocation", "s3a://checkpointlocation/")  # very important to be on writeStream!
>            .start())
>
> output.awaitTermination()
>
> def resolveips_and_saveastable(bdf: DataFrame, batchid: int) -> None:
>     if not bdf.isEmpty():
>         bdf = resolveips(bdf, batchid)
>
>         ...
>
>
>         (bdf
>          .select(["iso_date", "pid", "ip", "category", "sessionid", "username", "lat", "lon", "city", "state",
>                   "country", "continent", "message", "unique_key"])  # must match delta table!
>          .write
>          .format("delta")
>          .saveAsTable(f"{DB_NAME_SILVER}.maillogs", mode="append"))
>
> The following screen shot shows the UI:
>
>
> [image: image.png]
>
> I have removed a little bit of the code, notably two calls to `count`. Otherwise, the only thing I can recognise in the screenshot is "saveAsTable" on job 6.
>
> Question 1: is it possible to supply names to transformations or groupBy or mapPartitions or withColumn, etc., so that it is easier to find the code which a job stems from?
>
> Spark UI is always going to show you Jobs and within jobs you will have
stages and each stage is going to have tasks.(this is w.r.t code that is
running on executors).
In order to understand how your code gets translated, you can use
dataframe.explain().

AFAIK, you can rename at job level to make it more understandable, but not
further down. (pyspark.SparkContext.setJobDescription — PySpark 3.5.0
documentation (apache.org)
<https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.SparkContext.setJobDescription.html>
)

Question 2: where can I find detailed info about how Spark decides
what code to run in the driver, and what code to run in the workers?
>
> Usually anything which is rdd/dataset based would be executor. Other parts
of the code and say if you collect() and do operations will be in driver.

> Question 3: where can I find detailed info about how pyspark works, in terms of what stuff is executed in the JVM running under the hood, and what is run in python? I am particularly struggling with how workers execute python code.
>
> Many thanks,
>
> Ant
>
>
>
>

Re: [pyspark] how to understand how jobs in UI correlate with code

Posted by Ammar <po...@gmail.com>.
unsubscribe

On Sat, Jan 6, 2024 at 5:15 AM Ant Kutschera <an...@gmail.com>
wrote:

> Hi, I am trying to analyse the jobs created by my program and I am having
> difficulty correlating the actual code, with the jobs and stages that I can
> see in the UI.
>
> I have found https://issues.apache.org/jira/browse/SPARK-8480 but I am
> too much of a noob to understand if it really solves my problem. I don't
> think caching is what I am looking for.
>
> Some code:
>
> kdf = kafka_df.selectExpr("CAST(key AS STRING)",
>                      "CAST(value AS STRING)",
>                      "headers",
>                      "CAST(topic AS STRING)",
>                      "CAST(partition AS STRING)",
>                      "CAST(offset AS STRING)")
>
> kdf = kdf.transform(extract_message, "aUsefulValue")
>
> kdf = kdf.transform(add_unique_key)
>
> kdf = kdf.transform(categorise)
>
> kdf = kdf.transform(iso_date)
>
> output = (kdf
>            .writeStream
>            .queryName("silver_output")
>            .foreachBatch(resolveips_and_saveastable)
>            .trigger(availableNow=True)
>            .option("checkpointLocation", "s3a://checkpointlocation/")  # very important to be on writeStream!
>            .start())
>
> output.awaitTermination()
>
> def resolveips_and_saveastable(bdf: DataFrame, batchid: int) -> None:
>     if not bdf.isEmpty():
>         bdf = resolveips(bdf, batchid)
>
>         ...
>
>
>         (bdf
>          .select(["iso_date", "pid", "ip", "category", "sessionid", "username", "lat", "lon", "city", "state",
>                   "country", "continent", "message", "unique_key"])  # must match delta table!
>          .write
>          .format("delta")
>          .saveAsTable(f"{DB_NAME_SILVER}.maillogs", mode="append"))
>
> The following screen shot shows the UI:
>
>
> [image: image.png]
>
> I have removed a little bit of the code, notably two calls to `count`. Otherwise, the only thing I can recognise in the screenshot is "saveAsTable" on job 6.
>
> Question 1: is it possible to supply names to transformations or groupBy or mapPartitions or withColumn, etc., so that it is easier to find the code which a job stems from?
>
> Question 2: where can I find detailed info about how Spark decides what code to run in the driver, and what code to run in the workers?
>
> Question 3: where can I find detailed info about how pyspark works, in terms of what stuff is executed in the JVM running under the hood, and what is run in python? I am particularly struggling with how workers execute python code.
>
> Many thanks,
>
> Ant
>
>
>
>

-- 
​sincerely yours,
 Ammar Poursadegh
telegram: +989124972975

<http://360degree.blogsky.com/>