Memory Leak

[MUHAMMED CAN BAKIR <ca...@trendify.com.tr>]

Hi,

I have an application running with PySpark and I have the problem that RAM is not decrasing.

My application simply involves downloading data from the links of the data received from the user, creating TempTables with the spark.sql function provided by PySpark, creating a final table and saving this table to the Azure VM.

As I added code to the application, I saw that the saving time of the final table increased. While I was doing my tests, after I added TempTables to the application using spark.sql, it became very slow for the application to save my final table. I think the reason for this problem is the increase in RAM usage.

Although I add spark.stop at the end of my code, there is no decrasing in RAM. In addition, I added a DROP TABLES command at the end of the script for the CreateOrReplaceTempTables I created with spark.sql, but still no RAM decrasing.

RAM Decreasing only when I restart the python application.

The Azure VM I am using has 32GB RAM. The configurations I used for this machine are as follows:

"""
self.spark = SparkSession.builder.appName(self.appName).master(self.master)
.config('spark.sql.execution.arrow.pyspark.enabled', True)
.config('spark.sql.session.timeZone', 'UTC')
.config('spark.driver.memory','4G')
.config('spark.ui.showConsoleProgress', True)
.config('spark.sql.repl.eagerEval.enabled', True)
.config("spark.executor.memory", "12g")
.config("spark.driver.maxResultSize", "4096m")
.config("spark.sql.parquet.outputTimestampType", "TIMESTAMP_MICROS")
.config("spark.sql.parquet.datetimeRebaseModeInWrite", "LEGACY")
.config("spark.sql.parquet.datetimeRebaseModeInRead", "LEGACY").getOrCreate()

self.spark.conf.set("spark.sql.parquet.int96RebaseModeInWrite", "LEGACY")
self.spark.conf.set("spark.sql.parquet.int96RebaseModeInRead", "LEGACY")
"""

I want to clear my RAM usage. How can I reduced RAM?

[cid:2150c5be-8e13-4158-9ce4-be45e280d633]