[jira] [Created] (SPARK-58453) Improve show() output to include row count if not truncated

["Jason Altekruse (Jira)" <ji...@apache.org>]

Jason Altekruse created SPARK-58453:
---------------------------------------

             Summary: Improve show() output to include row count if not truncated
                 Key: SPARK-58453
                 URL: https://issues.apache.org/jira/browse/SPARK-58453
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 4.2.0
            Reporter: Jason Altekruse


Very often row counts are a useful integrity check for a query result, or a mental reference for which result set you are thinking about when running a series of exploratory queries.

Many query clients will report row counts for these reasons. When I am querying spark datasets in an environment like a python notebook or the spark shell cli, I only get a report as part of the output of a call to show() if the result set was truncated. Nothing is reported with total row count isn't truncated. Seeing the truncation reported often leads to setting the options in the show() call to something large like n=1000 in pyspark or numRows=1000 in scala environments.

This can get me to the state where it will stop reporting that a truncation occurred, but if I want to know the exact number of rows that were returned I need to run another query, or write an additional line of code to compute the dataframe, print the number of rows and then call show() to see the data. I am currently using spark for a use case where I'm trying to make a SQL interface available to semi-technical users, but trying to limit the amount of python or internal details of spark they need to know, like why they would need to call cache() to resolve the dataframe.

It would be nice if this could just be a one-liner, and/or go as far as to make reporting row count the default behavior of show().
{code:java}
date_error_df = spark.sql("select * from my_table where try_to_date(column) is not null")
print(date_error_df.cache().count())
date_error_df.show(n=100, truncate=False); {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org