[jira] [Created] (SPARK-38806) Unable to initialize the empty pyspark.pandas dataframe

["Prakhar Sandhu (Jira)" <ji...@apache.org>]

Prakhar Sandhu created SPARK-38806:
--------------------------------------

             Summary: Unable to initialize the empty pyspark.pandas dataframe
                 Key: SPARK-38806
                 URL: https://issues.apache.org/jira/browse/SPARK-38806
             Project: Spark
          Issue Type: Bug
          Components: PySpark
    Affects Versions: 3.2.1
            Reporter: Prakhar Sandhu


I am trying to replace pandas library with pyspark.pandas library. But after the replacement the below line of code failed - 
{code:java}
import pyspark.pandas as pd 

self._df = pd.DataFrame()
 {code}
 
It throws the below error : 
 
{code:java}
    self._df = pd.DataFrame()
  File "C:\Users\eapasnr\Anaconda3\envs\oden2\lib\site-packages\pyspark\pandas\frame.py", line 520, in __init__        
    internal = InternalFrame.from_pandas(pdf)
  File "C:\Users\eapasnr\Anaconda3\envs\oden2\lib\site-packages\pyspark\pandas\internal.py", line 1464, in from_pandas 
    sdf = default_session().createDataFrame(pdf, schema=schema)
  File "C:\Users\eapasnr\Anaconda3\envs\oden2\lib\site-packages\pyspark\pandas\utils.py", line 477, in default_session 
    return builder.getOrCreate()
  File "C:\Users\eapasnr\Anaconda3\envs\oden2\lib\site-packages\pyspark\sql\session.py", line 228, in getOrCreate      
    sc = SparkContext.getOrCreate(sparkConf)
  File "C:\Users\eapasnr\Anaconda3\envs\oden2\lib\site-packages\pyspark\context.py", line 392, in getOrCreate
    SparkContext(conf=conf or SparkConf())
  File "C:\Users\eapasnr\Anaconda3\envs\oden2\lib\site-packages\pyspark\context.py", line 144, in __init__
    SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
  File "C:\Users\eapasnr\Anaconda3\envs\oden2\lib\site-packages\pyspark\context.py", line 339, in _ensure_initialized  
    SparkContext._gateway = gateway or launch_gateway(conf)
  File "C:\Users\eapasnr\Anaconda3\envs\oden2\lib\site-packages\pyspark\java_gateway.py", line 101, in launch_gateway  
    proc = Popen(command, **popen_kwargs)
  File "C:\Users\eapasnr\Anaconda3\envs\oden2\lib\subprocess.py", line 800, in __init__
    restore_signals, start_new_session)
  File "C:\Users\eapasnr\Anaconda3\envs\oden2\lib\subprocess.py", line 1207, in _execute_child
    startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified {code}
The code was working fine previously with Pandas
 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org