You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Meena Rajani <me...@gmail.com> on 2024/08/02 01:28:23 UTC
Re: Question about installing Apache Spark [PySpark] computer requirements
Did it work?
Did you check the directory access? In your logs I saw
java.io.IOException: Cannot run program
"C:\Users\mikej\AppData\Local\Programs\Python\Python312":
CreateProcess error=5, Access is denied
Try chmod recursively on the directory and see
Regards,
Meena
On Wed, Jul 31, 2024 at 6:13 AM Bjørn Jørgensen <bj...@gmail.com>
wrote:
> if you have anaconda installed you can type conda create -n py311
> python=3.11 and conda activate py311
>
> ons. 31. juli 2024 kl. 01:33 skrev mike Jadoo <mi...@gmail.com>:
>
>> Hello Bjorn,
>>
>> I am installing Python version 3.11. However, I don't see anaconda
>> python with version 3.11. Where can I find this?
>>
>> Thank you,
>> Mike
>>
>> On Tue, Jul 30, 2024 at 3:37 AM Bjørn Jørgensen <bj...@gmail.com>
>> wrote:
>>
>>> you are using python version 3.12
>>> spark 3.X does not have support for python version 3.12.
>>> Remove python 3.12 and install python version 3.11
>>>
>>> tir. 30. juli 2024 kl. 07:51 skrev mike Jadoo <mi...@gmail.com>:
>>>
>>>> Hello Sadha,
>>>>
>>>> Thank you for the suggestion. I entered the environmental variable,
>>>> however, I am still having trouble. here is a screenshot of the env vars
>>>> and my code.
>>>>
>>>>
>>>> [image: image.png]
>>>>
>>>>
>>>> Here is my code:
>>>>
>>>>
>>>>
>>>> from pyspark.sql import SparkSession
>>>>
>>>> from pyspark import SparkContext
>>>>
>>>>
>>>>
>>>> spark = SparkSession.builder \
>>>>
>>>> .master("local[*]") \
>>>>
>>>> .appName("YourAppName") \
>>>>
>>>> .getOrCreate()
>>>>
>>>>
>>>>
>>>> # Create an RDD from a list of numbers
>>>>
>>>> numbers = [1, 2, 3, 4, 5]
>>>>
>>>> rdd = spark.sparkContext.parallelize(numbers)
>>>>
>>>>
>>>>
>>>> # Perform a transformation on the RDD
>>>>
>>>> squared_rdd = rdd.map(lambda x: x * x)
>>>>
>>>>
>>>>
>>>> # Collect the results into a list
>>>>
>>>> result = squared_rdd.collect()
>>>>
>>>>
>>>>
>>>> # Print the result
>>>>
>>>> print(result)
>>>>
>>>>
>>>>
>>>> error:
>>>>
>>>>
>>>> *---------------------------------------------------------------------------*
>>>>
>>>> *Py4JJavaError* Traceback (most recent
>>>> call last)
>>>>
>>>> Cell *In[5], line 13*
>>>>
>>>> 8 squared_rdd = rdd.map(*lambda* x: x * x)
>>>>
>>>> 10 # Persist the DataFrame in memory
>>>>
>>>> 11 #squared_rdd.persist(StorageLevel.MEMORY_ONLY)
>>>>
>>>> 12 # Collect the results into a list
>>>>
>>>> *---> 13* result = squared_rdd.collect()
>>>>
>>>> 15 # Print the result
>>>>
>>>> 16 print(result)
>>>>
>>>>
>>>>
>>>> File *~\anaconda3\Lib\site-packages\pyspark\rdd.py:1814*, in
>>>> RDD.collect*(self)*
>>>>
>>>> 1812 *with* SCCallSiteSync(self.context):
>>>>
>>>> 1813 *assert* self.ctx._jvm *is* *not* *None*
>>>>
>>>> *-> 1814* sock_info = self.ctx._jvm.PythonRDD.collectAndServe(self.
>>>> _jrdd.rdd())
>>>>
>>>> 1815 *return* list(_load_from_socket(sock_info, self.
>>>> _jrdd_deserializer))
>>>>
>>>>
>>>>
>>>> File *~\anaconda3\Lib\site-packages\py4j\java_gateway.py:1322*, in
>>>> JavaMember.__call__*(self, *args)*
>>>>
>>>> 1316 command = proto.CALL_COMMAND_NAME +\
>>>>
>>>> 1317 self.command_header +\
>>>>
>>>> 1318 args_command +\
>>>>
>>>> 1319 proto.END_COMMAND_PART
>>>>
>>>> 1321 answer = self.gateway_client.send_command(command)
>>>>
>>>> *-> 1322* return_value = get_return_value(
>>>>
>>>> 1323 answer, self.gateway_client, self.target_id, self.name)
>>>>
>>>> 1325 *for* temp_arg *in* temp_args:
>>>>
>>>> 1326 *if* hasattr(temp_arg, "_detach"):
>>>>
>>>>
>>>>
>>>> File
>>>> *~\anaconda3\Lib\site-packages\pyspark\errors\exceptions\captured.py:169*,
>>>> in capture_sql_exception.<locals>.deco*(*a, **kw)*
>>>>
>>>> 167 *def* deco(*a: Any, **kw: Any) -> Any:
>>>>
>>>> 168 *try*:
>>>>
>>>> *--> 169* *return* f(*a, **kw)
>>>>
>>>> 170 *except* Py4JJavaError *as* e:
>>>>
>>>> 171 converted = convert_exception(e.java_exception)
>>>>
>>>>
>>>>
>>>> File *~\anaconda3\Lib\site-packages\py4j\protocol.py:326*, in
>>>> get_return_value*(answer, gateway_client, target_id, name)*
>>>>
>>>> 324 value = OUTPUT_CONVERTER[type](answer[2:], gateway_client)
>>>>
>>>> 325 *if* answer[1] == REFERENCE_TYPE:
>>>>
>>>> *--> 326* *raise* Py4JJavaError(
>>>>
>>>> 327 "An error occurred while calling *{0}{1}{2}*.*\n*".
>>>>
>>>> 328 format(target_id, ".", name), value)
>>>>
>>>> 329 *else*:
>>>>
>>>> 330 *raise* Py4JError(
>>>>
>>>> 331 "An error occurred while calling *{0}{1}{2}*. Trace:
>>>> *\n{3}\n*".
>>>>
>>>> 332 format(target_id, ".", name, value))
>>>>
>>>>
>>>>
>>>> *Py4JJavaError*: An error occurred while calling
>>>> z:org.apache.spark.api.python.PythonRDD.collectAndServe.
>>>>
>>>> : org.apache.spark.SparkException: Job aborted due to stage failure:
>>>> Task 7 in stage 1.0 failed 1 times, most recent failure: Lost task 7.0 in
>>>> stage 1.0 (TID 19) (mjadoo.myfiosgateway.com executor driver):
>>>> org.apache.spark.SparkException: Python worker exited unexpectedly (crashed)
>>>>
>>>> at
>>>> org.apache.spark.api.python.BasePythonRunner$ReaderIterator$$anonfun$1.applyOrElse(PythonRunner.scala:612)
>>>>
>>>> at
>>>> org.apache.spark.api.python.BasePythonRunner$ReaderIterator$$anonfun$1.applyOrElse(PythonRunner.scala:594)
>>>>
>>>> at
>>>> scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:38)
>>>>
>>>> at
>>>> org.apache.spark.api.python.PythonRunner$$anon$3.read(PythonRunner.scala:789)
>>>>
>>>> at
>>>> org.apache.spark.api.python.PythonRunner$$anon$3.read(PythonRunner.scala:766)
>>>>
>>>> at
>>>> org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:525)
>>>>
>>>> at
>>>> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
>>>>
>>>> at
>>>> scala.collection.Iterator.foreach(Iterator.scala:943)
>>>>
>>>> at
>>>> scala.collection.Iterator.foreach$(Iterator.scala:943)
>>>>
>>>> at
>>>> org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28)
>>>>
>>>> at
>>>> scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:62)
>>>>
>>>> at
>>>> scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53)
>>>>
>>>> at
>>>> scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:105)
>>>>
>>>> at
>>>> scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:49)
>>>>
>>>> at scala.collection.TraversableOnce.to
>>>> (TraversableOnce.scala:366)
>>>>
>>>> at scala.collection.TraversableOnce.to
>>>> $(TraversableOnce.scala:364)
>>>>
>>>> at org.apache.spark.InterruptibleIterator.to
>>>> (InterruptibleIterator.scala:28)
>>>>
>>>> at
>>>> scala.collection.TraversableOnce.toBuffer(TraversableOnce.scala:358)
>>>>
>>>> at
>>>> scala.collection.TraversableOnce.toBuffer$(TraversableOnce.scala:358)
>>>>
>>>> at
>>>> org.apache.spark.InterruptibleIterator.toBuffer(InterruptibleIterator.scala:28)
>>>>
>>>> at
>>>> scala.collection.TraversableOnce.toArray(TraversableOnce.scala:345)
>>>>
>>>> at
>>>> scala.collection.TraversableOnce.toArray$(TraversableOnce.scala:339)
>>>>
>>>> at
>>>> org.apache.spark.InterruptibleIterator.toArray(InterruptibleIterator.scala:28)
>>>>
>>>> at
>>>> org.apache.spark.rdd.RDD.$anonfun$collect$2(RDD.scala:1049)
>>>>
>>>> at
>>>> org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2438)
>>>>
>>>> at
>>>> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)
>>>>
>>>> at
>>>> org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166)
>>>>
>>>> at
>>>> org.apache.spark.scheduler.Task.run(Task.scala:141)
>>>>
>>>> at
>>>> org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
>>>>
>>>> at
>>>> org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
>>>>
>>>> at
>>>> org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
>>>>
>>>> at
>>>> org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
>>>>
>>>> at
>>>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
>>>>
>>>> at
>>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>>>
>>>> at
>>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>>>
>>>> at java.base/java.lang.Thread.run(Thread.java:829)
>>>>
>>>> Caused by: java.io.EOFException
>>>>
>>>> at
>>>> java.base/java.io.DataInputStream.readInt(DataInputStream.java:397)
>>>>
>>>> at
>>>> org.apache.spark.api.python.PythonRunner$$anon$3.read(PythonRunner.scala:774)
>>>>
>>>> ... 32 more
>>>>
>>>>
>>>>
>>>> Driver stacktrace:
>>>>
>>>> at
>>>> org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2856)
>>>>
>>>> at
>>>> org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2792)
>>>>
>>>> at
>>>> org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2791)
>>>>
>>>> at
>>>> scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
>>>>
>>>> at
>>>> scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
>>>>
>>>> at
>>>> scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
>>>>
>>>> at
>>>> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2791)
>>>>
>>>> at
>>>> org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1247)
>>>>
>>>> at
>>>> org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1247)
>>>>
>>>> at scala.Option.foreach(Option.scala:407)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Mon, Jul 29, 2024 at 8:19 PM Sadha Chilukoori <
>>>> sage.quotient@gmail.com> wrote:
>>>>
>>>>> Hi Mike,
>>>>>
>>>>> This appears to be an access issue on Windows + Python. Can you try
>>>>> setting up the PYTHON_PATH environment variable as described in this
>>>>> stackoverflow post
>>>>> https://stackoverflow.com/questions/60414394/createprocess-error-5-access-is-denied-pyspark
>>>>>
>>>>> - Sadha
>>>>>
>>>>> On Mon, Jul 29, 2024 at 3:39 PM mike Jadoo <mi...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Thanks. I just downloaded the corretto but I got this error
>>>>>> message, which was the same as before. [It was shared with me that this
>>>>>> saying that I have limited resources, i think]
>>>>>>
>>>>>> ---------------------------------------------------------------------------Py4JJavaError Traceback (most recent call last)
>>>>>> Cell In[3], line 13 8 squared_rdd = rdd.map(lambda x: x * x) 10 # Persist the DataFrame in memory 11 #squared_rdd.persist(StorageLevel.MEMORY_ONLY) 12 # Collect the results into a list---> 13 result = squared_rdd.collect() 15 # Print the result 16 print(result)
>>>>>>
>>>>>> File C:\spark\spark-3.5.1-bin-hadoop3\python\lib\pyspark.zip\pyspark\rdd.py:1833, in RDD.collect(self) 1831 with SCCallSiteSync(self.context): 1832 assert self.ctx._jvm is not None-> 1833 sock_info = self.ctx._jvm.PythonRDD.collectAndServe(self._jrdd.rdd()) 1834 return list(_load_from_socket(sock_info, self._jrdd_deserializer))
>>>>>>
>>>>>> File ~\anaconda3\Lib\site-packages\py4j\java_gateway.py:1322, in JavaMember.__call__(self, *args) 1316 command = proto.CALL_COMMAND_NAME +\ 1317 self.command_header +\ 1318 args_command +\ 1319 proto.END_COMMAND_PART 1321 answer = self.gateway_client.send_command(command)-> 1322 return_value = get_return_value( 1323 answer, self.gateway_client, self.target_id, self.name) 1325 for temp_arg in temp_args: 1326 if hasattr(temp_arg, "_detach"):
>>>>>>
>>>>>> File C:\spark\spark-3.5.1-bin-hadoop3\python\lib\pyspark.zip\pyspark\errors\exceptions\captured.py:179, in capture_sql_exception.<locals>.deco(*a, **kw) 177 def deco(*a: Any, **kw: Any) -> Any: 178 try:--> 179 return f(*a, **kw) 180 except Py4JJavaError as e: 181 converted = convert_exception(e.java_exception)
>>>>>>
>>>>>> File ~\anaconda3\Lib\site-packages\py4j\protocol.py:326, in get_return_value(answer, gateway_client, target_id, name) 324 value = OUTPUT_CONVERTER[type](answer[2:], gateway_client) 325 if answer[1] == REFERENCE_TYPE:--> 326 raise Py4JJavaError( 327 "An error occurred while calling {0}{1}{2}.\n". 328 format(target_id, ".", name), value) 329 else: 330 raise Py4JError( 331 "An error occurred while calling {0}{1}{2}. Trace:\n{3}\n". 332 format(target_id, ".", name, value))
>>>>>> Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
>>>>>> : org.apache.spark.SparkException: Job aborted due to stage failure: Task 7 in stage 0.0 failed 1 times, most recent failure: Lost task 7.0 in stage 0.0 (TID 7) (mjadoo.myfiosgateway.com executor driver): java.io.IOException: Cannot run program "C:\Users\mikej\AppData\Local\Programs\Python\Python312": CreateProcess error=5, Access is denied
>>>>>> at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1128)
>>>>>> at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1071)
>>>>>> at org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:181)
>>>>>> at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:109)
>>>>>> at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:124)
>>>>>> at org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:174)
>>>>>> at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:67)
>>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:367)
>>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:331)
>>>>>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)
>>>>>> at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166)
>>>>>> at org.apache.spark.scheduler.Task.run(Task.scala:141)
>>>>>> at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
>>>>>> at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
>>>>>> at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
>>>>>> at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
>>>>>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
>>>>>> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>>>>> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>>>>> at java.base/java.lang.Thread.run(Thread.java:829)
>>>>>> Caused by: java.io.IOException: CreateProcess error=5, Access is denied
>>>>>> at java.base/java.lang.ProcessImpl.create(Native Method)
>>>>>> at java.base/java.lang.ProcessImpl.<init>(ProcessImpl.java:492)
>>>>>> at java.base/java.lang.ProcessImpl.start(ProcessImpl.java:153)
>>>>>> at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1107)
>>>>>> ... 19 more
>>>>>>
>>>>>> Driver stacktrace:
>>>>>> at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2856)
>>>>>> at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2792)
>>>>>> at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2791)
>>>>>> at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
>>>>>> at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
>>>>>> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
>>>>>> at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2791)
>>>>>> at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1247)
>>>>>> at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1247)
>>>>>> at scala.Option.foreach(Option.scala:407)
>>>>>> at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1247)
>>>>>> at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:3060)
>>>>>> at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2994)
>>>>>> at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2983)
>>>>>> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
>>>>>> at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:989)
>>>>>> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2398)
>>>>>> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2419)
>>>>>> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2438)
>>>>>> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2463)
>>>>>> at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1049)
>>>>>> at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
>>>>>> at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
>>>>>> at org.apache.spark.rdd.RDD.withScope(RDD.scala:410)
>>>>>> at org.apache.spark.rdd.RDD.collect(RDD.scala:1048)
>>>>>> at org.apache.spark.api.python.PythonRDD$.collectAndServe(PythonRDD.scala:195)
>>>>>> at org.apache.spark.api.python.PythonRDD.collectAndServe(PythonRDD.scala)
>>>>>> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>>>> at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>>>>>> at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
>>>>>> at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
>>>>>> at py4j.Gateway.invoke(Gateway.java:282)
>>>>>> at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
>>>>>> at py4j.commands.CallCommand.execute(CallCommand.java:79)
>>>>>> at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
>>>>>> at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
>>>>>> at java.base/java.lang.Thread.run(Thread.java:829)
>>>>>> Caused by: java.io.IOException: Cannot run program "C:\Users\mikej\AppData\Local\Programs\Python\Python312": CreateProcess error=5, Access is denied
>>>>>> at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1128)
>>>>>> at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1071)
>>>>>> at org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:181)
>>>>>> at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:109)
>>>>>> at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:124)
>>>>>> at org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:174)
>>>>>> at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:67)
>>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:367)
>>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:331)
>>>>>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)
>>>>>> at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166)
>>>>>> at org.apache.spark.scheduler.Task.run(Task.scala:141)
>>>>>> at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
>>>>>> at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
>>>>>> at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
>>>>>> at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
>>>>>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
>>>>>> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>>>>> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>>>>> ... 1 more
>>>>>>
>>>>>>
>>>>>> On Mon, Jul 29, 2024 at 4:34 PM Sadha Chilukoori <
>>>>>> sage.quotient@gmail.com> wrote:
>>>>>>
>>>>>>> Hi Mike,
>>>>>>>
>>>>>>> I'm not sure about the minimum requirements of a machine for running
>>>>>>> Spark. But to run some Pyspark scripts (and Jupiter notbebooks) on a local
>>>>>>> machine, I found the following steps are the easiest.
>>>>>>>
>>>>>>>
>>>>>>> I installed Amazon corretto and updated the java_home variable as
>>>>>>> instructed here
>>>>>>> https://docs.aws.amazon.com/corretto/latest/corretto-11-ug/downloads-list.html
>>>>>>> (Any other java works too, I'm used to corretto from work).
>>>>>>>
>>>>>>> Then installed the Pyspark module using pip, which enabled me run
>>>>>>> Pyspark on my machine.
>>>>>>>
>>>>>>> -Sadha
>>>>>>>
>>>>>>> On Mon, Jul 29, 2024, 12:51 PM mike Jadoo <mi...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> I am trying to run Pyspark on my computer without success. I
>>>>>>>> follow several different directions from online sources and it appears that
>>>>>>>> I need to get a faster computer.
>>>>>>>>
>>>>>>>> I wanted to ask what are some recommendations for computer
>>>>>>>> specifications to run PySpark (Apache Spark).
>>>>>>>>
>>>>>>>> Any help would be greatly appreciated.
>>>>>>>>
>>>>>>>> Thank you,
>>>>>>>>
>>>>>>>> Mike
>>>>>>>>
>>>>>>>
>>>
>>> --
>>> Bjørn Jørgensen
>>> Vestre Aspehaug 4, 6010 Ålesund
>>> Norge
>>>
>>> +47 480 94 297
>>>
>>
>
> --
> Bjørn Jørgensen
> Vestre Aspehaug 4, 6010 Ålesund
> Norge
>
> +47 480 94 297
>
Re: Question about installing Apache Spark [PySpark] computer requirements
Posted by mike Jadoo <mi...@gmail.com>.
Hello Angel
I updated my environment variables to what you shared.
[image: image.png]
Thanks.
Question for the group, where can I find Java 8 JDK windows x64. One
can't download it from oracle website anymore. The staff of oracle is not
helpful. can someone share it with me?
Thanks
On Thu, Jan 16, 2025 at 11:50 AM Ángel <an...@gmail.com>
wrote:
>
> - *PYSPARK_HOME*: Should point to the root directory of your Spark
> installation (e.g., /path/to/spark), not the Python executable.
> - *PYSPARK_PYTHON*: Should point to the Python executable (e.g.,
> /path/to/python or python.exe).
>
>
> El jue, 16 ene 2025 a las 16:55, mike Jadoo (<mi...@gmail.com>)
> escribió:
>
>> Hello Angel,
>>
>> I dont have a PYSPARK_PYTHON environment variable . I do have a
>> PYSPARK_HOME and here is the folder:
>> C:\Users\mikej\AppData\Local\Programs\Python\Python313\python.exe.
>>
>> Here is a snapshot of my environment variables:
>>
>> [image: image.png]
>>
>> I currently have Python 3.12
>> PySpark spark-3.5.3
>> hadoop- 3.3.1
>> Java- 17.0.12
>>
>> Mike
>>
>> On Thu, Jan 16, 2025 at 9:26 AM Ángel <an...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> It seems like the Spark JVM is unable to start the Python processes for
>>> the executors. Have you tried setting the PYSPARK_PYTHON environment
>>> variable to your Python installation path?
>>>
>>> I usually avoid doing that because I don't want to set the environment
>>> variable globally on my Windows system or configure it every time I start a
>>> PySpark application. Instead, I add the following code to the
>>> venv\Lib\site-packages\pyspark\java_gateway.py file, inside the
>>> launch_gateway method, just after the docstring:
>>>
>>> import sys
>>> os.environ['PYSPARK_PYTHON'] = sys.executable
>>>
>>> This code sets the PYSPARK_PYTHON environment variable to the same
>>> Python executable that started the initial PySpark process.
>>>
>>> Hope this helps!
>>>
>>>
>>> El jue, 16 ene 2025 a las 14:02, mike Jadoo (<mi...@gmail.com>)
>>> escribió:
>>>
>>>> Hello everyone,
>>>>
>>>> Its been a long time and I wish everyone a happy New Year. I just
>>>> wanted to share I got a new laptop and tried installing Pyspark again and
>>>> it worked. However, I followed the instructions on setting
>>>> environmental variables from this youtube video:
>>>> https://www.youtube.com/watch?v=Qvke9S_ma7A&t=740s.
>>>>
>>>> I am also using Jupyter lab to run Pyspark. I would love to show anyone
>>>> here via Zoom call about my experience.
>>>>
>>>> Let me know what you think. Thanks for taking the time to help me
>>>> out before. This was a difficult experience and I appreciate all of you.
>>>>
>>>> Thanks,
>>>> Mike
>>>>
>>>> On Thu, Aug 1, 2024 at 9:28 PM Meena Rajani <me...@gmail.com>
>>>> wrote:
>>>>
>>>>> Did it work?
>>>>>
>>>>> Did you check the directory access? In your logs I saw
>>>>>
>>>>> java.io.IOException: Cannot run program "C:\Users\mikej\AppData\Local\Programs\Python\Python312": CreateProcess error=5, Access is denied
>>>>>
>>>>>
>>>>> Try chmod recursively on the directory and see
>>>>>
>>>>> Regards,
>>>>> Meena
>>>>>
>>>>> On Wed, Jul 31, 2024 at 6:13 AM Bjørn Jørgensen <
>>>>> bjornjorgensen@gmail.com> wrote:
>>>>>
>>>>>> if you have anaconda installed you can type conda create -n py311
>>>>>> python=3.11 and conda activate py311
>>>>>>
>>>>>> ons. 31. juli 2024 kl. 01:33 skrev mike Jadoo <mi...@gmail.com>:
>>>>>>
>>>>>>> Hello Bjorn,
>>>>>>>
>>>>>>> I am installing Python version 3.11. However, I don't see
>>>>>>> anaconda python with version 3.11. Where can I find this?
>>>>>>>
>>>>>>> Thank you,
>>>>>>> Mike
>>>>>>>
>>>>>>> On Tue, Jul 30, 2024 at 3:37 AM Bjørn Jørgensen <
>>>>>>> bjornjorgensen@gmail.com> wrote:
>>>>>>>
>>>>>>>> you are using python version 3.12
>>>>>>>> spark 3.X does not have support for python version 3.12.
>>>>>>>> Remove python 3.12 and install python version 3.11
>>>>>>>>
>>>>>>>> tir. 30. juli 2024 kl. 07:51 skrev mike Jadoo <mikejadoow@gmail.com
>>>>>>>> >:
>>>>>>>>
>>>>>>>>> Hello Sadha,
>>>>>>>>>
>>>>>>>>> Thank you for the suggestion. I entered the environmental
>>>>>>>>> variable, however, I am still having trouble. here is a screenshot of the
>>>>>>>>> env vars and my code.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [image: image.png]
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Here is my code:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> from pyspark.sql import SparkSession
>>>>>>>>>
>>>>>>>>> from pyspark import SparkContext
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> spark = SparkSession.builder \
>>>>>>>>>
>>>>>>>>> .master("local[*]") \
>>>>>>>>>
>>>>>>>>> .appName("YourAppName") \
>>>>>>>>>
>>>>>>>>> .getOrCreate()
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> # Create an RDD from a list of numbers
>>>>>>>>>
>>>>>>>>> numbers = [1, 2, 3, 4, 5]
>>>>>>>>>
>>>>>>>>> rdd = spark.sparkContext.parallelize(numbers)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> # Perform a transformation on the RDD
>>>>>>>>>
>>>>>>>>> squared_rdd = rdd.map(lambda x: x * x)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> # Collect the results into a list
>>>>>>>>>
>>>>>>>>> result = squared_rdd.collect()
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> # Print the result
>>>>>>>>>
>>>>>>>>> print(result)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> error:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *---------------------------------------------------------------------------*
>>>>>>>>>
>>>>>>>>> *Py4JJavaError* Traceback (most
>>>>>>>>> recent call last)
>>>>>>>>>
>>>>>>>>> Cell *In[5], line 13*
>>>>>>>>>
>>>>>>>>> 8 squared_rdd = rdd.map(*lambda* x: x * x)
>>>>>>>>>
>>>>>>>>> 10 # Persist the DataFrame in memory
>>>>>>>>>
>>>>>>>>> 11 #squared_rdd.persist(StorageLevel.MEMORY_ONLY)
>>>>>>>>>
>>>>>>>>> 12 # Collect the results into a list
>>>>>>>>>
>>>>>>>>> *---> 13* result = squared_rdd.collect()
>>>>>>>>>
>>>>>>>>> 15 # Print the result
>>>>>>>>>
>>>>>>>>> 16 print(result)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> File *~\anaconda3\Lib\site-packages\pyspark\rdd.py:1814*, in
>>>>>>>>> RDD.collect*(self)*
>>>>>>>>>
>>>>>>>>> 1812 *with* SCCallSiteSync(self.context):
>>>>>>>>>
>>>>>>>>> 1813 *assert* self.ctx._jvm *is* *not* *None*
>>>>>>>>>
>>>>>>>>> *-> 1814* sock_info = self.ctx._jvm.PythonRDD.collectAndServe(
>>>>>>>>> self._jrdd.rdd())
>>>>>>>>>
>>>>>>>>> 1815 *return* list(_load_from_socket(sock_info, self.
>>>>>>>>> _jrdd_deserializer))
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> File *~\anaconda3\Lib\site-packages\py4j\java_gateway.py:1322*,
>>>>>>>>> in JavaMember.__call__*(self, *args)*
>>>>>>>>>
>>>>>>>>> 1316 command = proto.CALL_COMMAND_NAME +\
>>>>>>>>>
>>>>>>>>> 1317 self.command_header +\
>>>>>>>>>
>>>>>>>>> 1318 args_command +\
>>>>>>>>>
>>>>>>>>> 1319 proto.END_COMMAND_PART
>>>>>>>>>
>>>>>>>>> 1321 answer = self.gateway_client.send_command(command)
>>>>>>>>>
>>>>>>>>> *-> 1322* return_value = get_return_value(
>>>>>>>>>
>>>>>>>>> 1323 answer, self.gateway_client, self.target_id, self.
>>>>>>>>> name)
>>>>>>>>>
>>>>>>>>> 1325 *for* temp_arg *in* temp_args:
>>>>>>>>>
>>>>>>>>> 1326 *if* hasattr(temp_arg, "_detach"):
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> *~\anaconda3\Lib\site-packages\pyspark\errors\exceptions\captured.py:169*,
>>>>>>>>> in capture_sql_exception.<locals>.deco*(*a, **kw)*
>>>>>>>>>
>>>>>>>>> 167 *def* deco(*a: Any, **kw: Any) -> Any:
>>>>>>>>>
>>>>>>>>> 168 *try*:
>>>>>>>>>
>>>>>>>>> *--> 169* *return* f(*a, **kw)
>>>>>>>>>
>>>>>>>>> 170 *except* Py4JJavaError *as* e:
>>>>>>>>>
>>>>>>>>> 171 converted = convert_exception(e.java_exception)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> File *~\anaconda3\Lib\site-packages\py4j\protocol.py:326*, in
>>>>>>>>> get_return_value*(answer, gateway_client, target_id, name)*
>>>>>>>>>
>>>>>>>>> 324 value = OUTPUT_CONVERTER[type](answer[2:], gateway_client)
>>>>>>>>>
>>>>>>>>> 325 *if* answer[1] == REFERENCE_TYPE:
>>>>>>>>>
>>>>>>>>> *--> 326* *raise* Py4JJavaError(
>>>>>>>>>
>>>>>>>>> 327 "An error occurred while calling *{0}{1}{2}*.*\n*"
>>>>>>>>> .
>>>>>>>>>
>>>>>>>>> 328 format(target_id, ".", name), value)
>>>>>>>>>
>>>>>>>>> 329 *else*:
>>>>>>>>>
>>>>>>>>> 330 *raise* Py4JError(
>>>>>>>>>
>>>>>>>>> 331 "An error occurred while calling *{0}{1}{2}*.
>>>>>>>>> Trace:*\n{3}\n*".
>>>>>>>>>
>>>>>>>>> 332 format(target_id, ".", name, value))
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *Py4JJavaError*: An error occurred while calling
>>>>>>>>> z:org.apache.spark.api.python.PythonRDD.collectAndServe.
>>>>>>>>>
>>>>>>>>> : org.apache.spark.SparkException: Job aborted due to stage
>>>>>>>>> failure: Task 7 in stage 1.0 failed 1 times, most recent failure: Lost task
>>>>>>>>> 7.0 in stage 1.0 (TID 19) (mjadoo.myfiosgateway.com executor
>>>>>>>>> driver): org.apache.spark.SparkException: Python worker exited unexpectedly
>>>>>>>>> (crashed)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.spark.api.python.BasePythonRunner$ReaderIterator$$anonfun$1.applyOrElse(PythonRunner.scala:612)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.spark.api.python.BasePythonRunner$ReaderIterator$$anonfun$1.applyOrElse(PythonRunner.scala:594)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:38)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.spark.api.python.PythonRunner$$anon$3.read(PythonRunner.scala:789)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.spark.api.python.PythonRunner$$anon$3.read(PythonRunner.scala:766)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:525)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> scala.collection.Iterator.foreach(Iterator.scala:943)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> scala.collection.Iterator.foreach$(Iterator.scala:943)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:62)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:105)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:49)
>>>>>>>>>
>>>>>>>>> at scala.collection.TraversableOnce.to
>>>>>>>>> (TraversableOnce.scala:366)
>>>>>>>>>
>>>>>>>>> at scala.collection.TraversableOnce.to
>>>>>>>>> $(TraversableOnce.scala:364)
>>>>>>>>>
>>>>>>>>> at org.apache.spark.InterruptibleIterator.to
>>>>>>>>> (InterruptibleIterator.scala:28)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> scala.collection.TraversableOnce.toBuffer(TraversableOnce.scala:358)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> scala.collection.TraversableOnce.toBuffer$(TraversableOnce.scala:358)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.spark.InterruptibleIterator.toBuffer(InterruptibleIterator.scala:28)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> scala.collection.TraversableOnce.toArray(TraversableOnce.scala:345)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> scala.collection.TraversableOnce.toArray$(TraversableOnce.scala:339)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.spark.InterruptibleIterator.toArray(InterruptibleIterator.scala:28)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.spark.rdd.RDD.$anonfun$collect$2(RDD.scala:1049)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2438)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.spark.scheduler.Task.run(Task.scala:141)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> java.base/java.lang.Thread.run(Thread.java:829)
>>>>>>>>>
>>>>>>>>> Caused by: java.io.EOFException
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> java.base/java.io.DataInputStream.readInt(DataInputStream.java:397)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.spark.api.python.PythonRunner$$anon$3.read(PythonRunner.scala:774)
>>>>>>>>>
>>>>>>>>> ... 32 more
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Driver stacktrace:
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2856)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2792)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2791)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2791)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1247)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1247)
>>>>>>>>>
>>>>>>>>> at scala.Option.foreach(Option.scala:407)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Jul 29, 2024 at 8:19 PM Sadha Chilukoori <
>>>>>>>>> sage.quotient@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Mike,
>>>>>>>>>>
>>>>>>>>>> This appears to be an access issue on Windows + Python. Can you
>>>>>>>>>> try setting up the PYTHON_PATH environment variable as described in this
>>>>>>>>>> stackoverflow post
>>>>>>>>>> https://stackoverflow.com/questions/60414394/createprocess-error-5-access-is-denied-pyspark
>>>>>>>>>>
>>>>>>>>>> - Sadha
>>>>>>>>>>
>>>>>>>>>> On Mon, Jul 29, 2024 at 3:39 PM mike Jadoo <mi...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Thanks. I just downloaded the corretto but I got this error
>>>>>>>>>>> message, which was the same as before. [It was shared with me that this
>>>>>>>>>>> saying that I have limited resources, i think]
>>>>>>>>>>>
>>>>>>>>>>> ---------------------------------------------------------------------------Py4JJavaError Traceback (most recent call last)
>>>>>>>>>>> Cell In[3], line 13 8 squared_rdd = rdd.map(lambda x: x * x) 10 # Persist the DataFrame in memory 11 #squared_rdd.persist(StorageLevel.MEMORY_ONLY) 12 # Collect the results into a list---> 13 result = squared_rdd.collect() 15 # Print the result 16 print(result)
>>>>>>>>>>>
>>>>>>>>>>> File C:\spark\spark-3.5.1-bin-hadoop3\python\lib\pyspark.zip\pyspark\rdd.py:1833, in RDD.collect(self) 1831 with SCCallSiteSync(self.context): 1832 assert self.ctx._jvm is not None-> 1833 sock_info = self.ctx._jvm.PythonRDD.collectAndServe(self._jrdd.rdd()) 1834 return list(_load_from_socket(sock_info, self._jrdd_deserializer))
>>>>>>>>>>>
>>>>>>>>>>> File ~\anaconda3\Lib\site-packages\py4j\java_gateway.py:1322, in JavaMember.__call__(self, *args) 1316 command = proto.CALL_COMMAND_NAME +\ 1317 self.command_header +\ 1318 args_command +\ 1319 proto.END_COMMAND_PART 1321 answer = self.gateway_client.send_command(command)-> 1322 return_value = get_return_value( 1323 answer, self.gateway_client, self.target_id, self.name) 1325 for temp_arg in temp_args: 1326 if hasattr(temp_arg, "_detach"):
>>>>>>>>>>>
>>>>>>>>>>> File C:\spark\spark-3.5.1-bin-hadoop3\python\lib\pyspark.zip\pyspark\errors\exceptions\captured.py:179, in capture_sql_exception.<locals>.deco(*a, **kw) 177 def deco(*a: Any, **kw: Any) -> Any: 178 try:--> 179 return f(*a, **kw) 180 except Py4JJavaError as e: 181 converted = convert_exception(e.java_exception)
>>>>>>>>>>>
>>>>>>>>>>> File ~\anaconda3\Lib\site-packages\py4j\protocol.py:326, in get_return_value(answer, gateway_client, target_id, name) 324 value = OUTPUT_CONVERTER[type](answer[2:], gateway_client) 325 if answer[1] == REFERENCE_TYPE:--> 326 raise Py4JJavaError( 327 "An error occurred while calling {0}{1}{2}.\n". 328 format(target_id, ".", name), value) 329 else: 330 raise Py4JError( 331 "An error occurred while calling {0}{1}{2}. Trace:\n{3}\n". 332 format(target_id, ".", name, value))
>>>>>>>>>>> Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
>>>>>>>>>>> : org.apache.spark.SparkException: Job aborted due to stage failure: Task 7 in stage 0.0 failed 1 times, most recent failure: Lost task 7.0 in stage 0.0 (TID 7) (mjadoo.myfiosgateway.com executor driver): java.io.IOException: Cannot run program "C:\Users\mikej\AppData\Local\Programs\Python\Python312": CreateProcess error=5, Access is denied
>>>>>>>>>>> at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1128)
>>>>>>>>>>> at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1071)
>>>>>>>>>>> at org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:181)
>>>>>>>>>>> at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:109)
>>>>>>>>>>> at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:124)
>>>>>>>>>>> at org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:174)
>>>>>>>>>>> at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:67)
>>>>>>>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:367)
>>>>>>>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:331)
>>>>>>>>>>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)
>>>>>>>>>>> at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166)
>>>>>>>>>>> at org.apache.spark.scheduler.Task.run(Task.scala:141)
>>>>>>>>>>> at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
>>>>>>>>>>> at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
>>>>>>>>>>> at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
>>>>>>>>>>> at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
>>>>>>>>>>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
>>>>>>>>>>> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>>>>>>>>>> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>>>>>>>>>> at java.base/java.lang.Thread.run(Thread.java:829)
>>>>>>>>>>> Caused by: java.io.IOException: CreateProcess error=5, Access is denied
>>>>>>>>>>> at java.base/java.lang.ProcessImpl.create(Native Method)
>>>>>>>>>>> at java.base/java.lang.ProcessImpl.<init>(ProcessImpl.java:492)
>>>>>>>>>>> at java.base/java.lang.ProcessImpl.start(ProcessImpl.java:153)
>>>>>>>>>>> at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1107)
>>>>>>>>>>> ... 19 more
>>>>>>>>>>>
>>>>>>>>>>> Driver stacktrace:
>>>>>>>>>>> at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2856)
>>>>>>>>>>> at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2792)
>>>>>>>>>>> at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2791)
>>>>>>>>>>> at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
>>>>>>>>>>> at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
>>>>>>>>>>> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
>>>>>>>>>>> at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2791)
>>>>>>>>>>> at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1247)
>>>>>>>>>>> at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1247)
>>>>>>>>>>> at scala.Option.foreach(Option.scala:407)
>>>>>>>>>>> at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1247)
>>>>>>>>>>> at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:3060)
>>>>>>>>>>> at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2994)
>>>>>>>>>>> at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2983)
>>>>>>>>>>> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
>>>>>>>>>>> at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:989)
>>>>>>>>>>> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2398)
>>>>>>>>>>> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2419)
>>>>>>>>>>> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2438)
>>>>>>>>>>> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2463)
>>>>>>>>>>> at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1049)
>>>>>>>>>>> at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
>>>>>>>>>>> at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
>>>>>>>>>>> at org.apache.spark.rdd.RDD.withScope(RDD.scala:410)
>>>>>>>>>>> at org.apache.spark.rdd.RDD.collect(RDD.scala:1048)
>>>>>>>>>>> at org.apache.spark.api.python.PythonRDD$.collectAndServe(PythonRDD.scala:195)
>>>>>>>>>>> at org.apache.spark.api.python.PythonRDD.collectAndServe(PythonRDD.scala)
>>>>>>>>>>> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>>>>>>> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>>>>>>>>> at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>>>>>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>>>>>>>>>>> at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
>>>>>>>>>>> at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
>>>>>>>>>>> at py4j.Gateway.invoke(Gateway.java:282)
>>>>>>>>>>> at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
>>>>>>>>>>> at py4j.commands.CallCommand.execute(CallCommand.java:79)
>>>>>>>>>>> at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
>>>>>>>>>>> at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
>>>>>>>>>>> at java.base/java.lang.Thread.run(Thread.java:829)
>>>>>>>>>>> Caused by: java.io.IOException: Cannot run program "C:\Users\mikej\AppData\Local\Programs\Python\Python312": CreateProcess error=5, Access is denied
>>>>>>>>>>> at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1128)
>>>>>>>>>>> at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1071)
>>>>>>>>>>> at org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:181)
>>>>>>>>>>> at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:109)
>>>>>>>>>>> at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:124)
>>>>>>>>>>> at org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:174)
>>>>>>>>>>> at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:67)
>>>>>>>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:367)
>>>>>>>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:331)
>>>>>>>>>>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)
>>>>>>>>>>> at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166)
>>>>>>>>>>> at org.apache.spark.scheduler.Task.run(Task.scala:141)
>>>>>>>>>>> at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
>>>>>>>>>>> at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
>>>>>>>>>>> at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
>>>>>>>>>>> at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
>>>>>>>>>>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
>>>>>>>>>>> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>>>>>>>>>> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>>>>>>>>>> ... 1 more
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Jul 29, 2024 at 4:34 PM Sadha Chilukoori <
>>>>>>>>>>> sage.quotient@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Mike,
>>>>>>>>>>>>
>>>>>>>>>>>> I'm not sure about the minimum requirements of a machine for
>>>>>>>>>>>> running Spark. But to run some Pyspark scripts (and Jupiter notbebooks) on
>>>>>>>>>>>> a local machine, I found the following steps are the easiest.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I installed Amazon corretto and updated the java_home variable
>>>>>>>>>>>> as instructed here
>>>>>>>>>>>> https://docs.aws.amazon.com/corretto/latest/corretto-11-ug/downloads-list.html
>>>>>>>>>>>> (Any other java works too, I'm used to corretto from work).
>>>>>>>>>>>>
>>>>>>>>>>>> Then installed the Pyspark module using pip, which enabled me
>>>>>>>>>>>> run Pyspark on my machine.
>>>>>>>>>>>>
>>>>>>>>>>>> -Sadha
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Jul 29, 2024, 12:51 PM mike Jadoo <mi...@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I am trying to run Pyspark on my computer without success. I
>>>>>>>>>>>>> follow several different directions from online sources and it appears that
>>>>>>>>>>>>> I need to get a faster computer.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I wanted to ask what are some recommendations for computer
>>>>>>>>>>>>> specifications to run PySpark (Apache Spark).
>>>>>>>>>>>>>
>>>>>>>>>>>>> Any help would be greatly appreciated.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thank you,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Mike
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Bjørn Jørgensen
>>>>>>>> Vestre Aspehaug 4, 6010 Ålesund
>>>>>>>> Norge
>>>>>>>>
>>>>>>>> +47 480 94 297
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Bjørn Jørgensen
>>>>>> Vestre Aspehaug 4, 6010 Ålesund
>>>>>> Norge
>>>>>>
>>>>>> +47 480 94 297
>>>>>>
>>>>>
Re: Question about installing Apache Spark [PySpark] computer requirements
Posted by Ángel <an...@gmail.com>.
- *PYSPARK_HOME*: Should point to the root directory of your Spark
installation (e.g., /path/to/spark), not the Python executable.
- *PYSPARK_PYTHON*: Should point to the Python executable (e.g.,
/path/to/python or python.exe).
El jue, 16 ene 2025 a las 16:55, mike Jadoo (<mi...@gmail.com>)
escribió:
> Hello Angel,
>
> I dont have a PYSPARK_PYTHON environment variable . I do have a
> PYSPARK_HOME and here is the folder:
> C:\Users\mikej\AppData\Local\Programs\Python\Python313\python.exe.
>
> Here is a snapshot of my environment variables:
>
> [image: image.png]
>
> I currently have Python 3.12
> PySpark spark-3.5.3
> hadoop- 3.3.1
> Java- 17.0.12
>
> Mike
>
> On Thu, Jan 16, 2025 at 9:26 AM Ángel <an...@gmail.com>
> wrote:
>
>> Hi,
>>
>> It seems like the Spark JVM is unable to start the Python processes for
>> the executors. Have you tried setting the PYSPARK_PYTHON environment
>> variable to your Python installation path?
>>
>> I usually avoid doing that because I don't want to set the environment
>> variable globally on my Windows system or configure it every time I start a
>> PySpark application. Instead, I add the following code to the
>> venv\Lib\site-packages\pyspark\java_gateway.py file, inside the
>> launch_gateway method, just after the docstring:
>>
>> import sys
>> os.environ['PYSPARK_PYTHON'] = sys.executable
>>
>> This code sets the PYSPARK_PYTHON environment variable to the same
>> Python executable that started the initial PySpark process.
>>
>> Hope this helps!
>>
>>
>> El jue, 16 ene 2025 a las 14:02, mike Jadoo (<mi...@gmail.com>)
>> escribió:
>>
>>> Hello everyone,
>>>
>>> Its been a long time and I wish everyone a happy New Year. I just
>>> wanted to share I got a new laptop and tried installing Pyspark again and
>>> it worked. However, I followed the instructions on setting
>>> environmental variables from this youtube video:
>>> https://www.youtube.com/watch?v=Qvke9S_ma7A&t=740s.
>>>
>>> I am also using Jupyter lab to run Pyspark. I would love to show anyone
>>> here via Zoom call about my experience.
>>>
>>> Let me know what you think. Thanks for taking the time to help me out
>>> before. This was a difficult experience and I appreciate all of you.
>>>
>>> Thanks,
>>> Mike
>>>
>>> On Thu, Aug 1, 2024 at 9:28 PM Meena Rajani <me...@gmail.com>
>>> wrote:
>>>
>>>> Did it work?
>>>>
>>>> Did you check the directory access? In your logs I saw
>>>>
>>>> java.io.IOException: Cannot run program "C:\Users\mikej\AppData\Local\Programs\Python\Python312": CreateProcess error=5, Access is denied
>>>>
>>>>
>>>> Try chmod recursively on the directory and see
>>>>
>>>> Regards,
>>>> Meena
>>>>
>>>> On Wed, Jul 31, 2024 at 6:13 AM Bjørn Jørgensen <
>>>> bjornjorgensen@gmail.com> wrote:
>>>>
>>>>> if you have anaconda installed you can type conda create -n py311
>>>>> python=3.11 and conda activate py311
>>>>>
>>>>> ons. 31. juli 2024 kl. 01:33 skrev mike Jadoo <mi...@gmail.com>:
>>>>>
>>>>>> Hello Bjorn,
>>>>>>
>>>>>> I am installing Python version 3.11. However, I don't see
>>>>>> anaconda python with version 3.11. Where can I find this?
>>>>>>
>>>>>> Thank you,
>>>>>> Mike
>>>>>>
>>>>>> On Tue, Jul 30, 2024 at 3:37 AM Bjørn Jørgensen <
>>>>>> bjornjorgensen@gmail.com> wrote:
>>>>>>
>>>>>>> you are using python version 3.12
>>>>>>> spark 3.X does not have support for python version 3.12.
>>>>>>> Remove python 3.12 and install python version 3.11
>>>>>>>
>>>>>>> tir. 30. juli 2024 kl. 07:51 skrev mike Jadoo <mikejadoow@gmail.com
>>>>>>> >:
>>>>>>>
>>>>>>>> Hello Sadha,
>>>>>>>>
>>>>>>>> Thank you for the suggestion. I entered the environmental
>>>>>>>> variable, however, I am still having trouble. here is a screenshot of the
>>>>>>>> env vars and my code.
>>>>>>>>
>>>>>>>>
>>>>>>>> [image: image.png]
>>>>>>>>
>>>>>>>>
>>>>>>>> Here is my code:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> from pyspark.sql import SparkSession
>>>>>>>>
>>>>>>>> from pyspark import SparkContext
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> spark = SparkSession.builder \
>>>>>>>>
>>>>>>>> .master("local[*]") \
>>>>>>>>
>>>>>>>> .appName("YourAppName") \
>>>>>>>>
>>>>>>>> .getOrCreate()
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> # Create an RDD from a list of numbers
>>>>>>>>
>>>>>>>> numbers = [1, 2, 3, 4, 5]
>>>>>>>>
>>>>>>>> rdd = spark.sparkContext.parallelize(numbers)
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> # Perform a transformation on the RDD
>>>>>>>>
>>>>>>>> squared_rdd = rdd.map(lambda x: x * x)
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> # Collect the results into a list
>>>>>>>>
>>>>>>>> result = squared_rdd.collect()
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> # Print the result
>>>>>>>>
>>>>>>>> print(result)
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> error:
>>>>>>>>
>>>>>>>>
>>>>>>>> *---------------------------------------------------------------------------*
>>>>>>>>
>>>>>>>> *Py4JJavaError* Traceback (most recent
>>>>>>>> call last)
>>>>>>>>
>>>>>>>> Cell *In[5], line 13*
>>>>>>>>
>>>>>>>> 8 squared_rdd = rdd.map(*lambda* x: x * x)
>>>>>>>>
>>>>>>>> 10 # Persist the DataFrame in memory
>>>>>>>>
>>>>>>>> 11 #squared_rdd.persist(StorageLevel.MEMORY_ONLY)
>>>>>>>>
>>>>>>>> 12 # Collect the results into a list
>>>>>>>>
>>>>>>>> *---> 13* result = squared_rdd.collect()
>>>>>>>>
>>>>>>>> 15 # Print the result
>>>>>>>>
>>>>>>>> 16 print(result)
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> File *~\anaconda3\Lib\site-packages\pyspark\rdd.py:1814*, in
>>>>>>>> RDD.collect*(self)*
>>>>>>>>
>>>>>>>> 1812 *with* SCCallSiteSync(self.context):
>>>>>>>>
>>>>>>>> 1813 *assert* self.ctx._jvm *is* *not* *None*
>>>>>>>>
>>>>>>>> *-> 1814* sock_info = self.ctx._jvm.PythonRDD.collectAndServe(
>>>>>>>> self._jrdd.rdd())
>>>>>>>>
>>>>>>>> 1815 *return* list(_load_from_socket(sock_info, self.
>>>>>>>> _jrdd_deserializer))
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> File *~\anaconda3\Lib\site-packages\py4j\java_gateway.py:1322*, in
>>>>>>>> JavaMember.__call__*(self, *args)*
>>>>>>>>
>>>>>>>> 1316 command = proto.CALL_COMMAND_NAME +\
>>>>>>>>
>>>>>>>> 1317 self.command_header +\
>>>>>>>>
>>>>>>>> 1318 args_command +\
>>>>>>>>
>>>>>>>> 1319 proto.END_COMMAND_PART
>>>>>>>>
>>>>>>>> 1321 answer = self.gateway_client.send_command(command)
>>>>>>>>
>>>>>>>> *-> 1322* return_value = get_return_value(
>>>>>>>>
>>>>>>>> 1323 answer, self.gateway_client, self.target_id, self.name)
>>>>>>>>
>>>>>>>> 1325 *for* temp_arg *in* temp_args:
>>>>>>>>
>>>>>>>> 1326 *if* hasattr(temp_arg, "_detach"):
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> File
>>>>>>>> *~\anaconda3\Lib\site-packages\pyspark\errors\exceptions\captured.py:169*,
>>>>>>>> in capture_sql_exception.<locals>.deco*(*a, **kw)*
>>>>>>>>
>>>>>>>> 167 *def* deco(*a: Any, **kw: Any) -> Any:
>>>>>>>>
>>>>>>>> 168 *try*:
>>>>>>>>
>>>>>>>> *--> 169* *return* f(*a, **kw)
>>>>>>>>
>>>>>>>> 170 *except* Py4JJavaError *as* e:
>>>>>>>>
>>>>>>>> 171 converted = convert_exception(e.java_exception)
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> File *~\anaconda3\Lib\site-packages\py4j\protocol.py:326*, in
>>>>>>>> get_return_value*(answer, gateway_client, target_id, name)*
>>>>>>>>
>>>>>>>> 324 value = OUTPUT_CONVERTER[type](answer[2:], gateway_client)
>>>>>>>>
>>>>>>>> 325 *if* answer[1] == REFERENCE_TYPE:
>>>>>>>>
>>>>>>>> *--> 326* *raise* Py4JJavaError(
>>>>>>>>
>>>>>>>> 327 "An error occurred while calling *{0}{1}{2}*.*\n*".
>>>>>>>>
>>>>>>>> 328 format(target_id, ".", name), value)
>>>>>>>>
>>>>>>>> 329 *else*:
>>>>>>>>
>>>>>>>> 330 *raise* Py4JError(
>>>>>>>>
>>>>>>>> 331 "An error occurred while calling *{0}{1}{2}*.
>>>>>>>> Trace:*\n{3}\n*".
>>>>>>>>
>>>>>>>> 332 format(target_id, ".", name, value))
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> *Py4JJavaError*: An error occurred while calling
>>>>>>>> z:org.apache.spark.api.python.PythonRDD.collectAndServe.
>>>>>>>>
>>>>>>>> : org.apache.spark.SparkException: Job aborted due to stage
>>>>>>>> failure: Task 7 in stage 1.0 failed 1 times, most recent failure: Lost task
>>>>>>>> 7.0 in stage 1.0 (TID 19) (mjadoo.myfiosgateway.com executor
>>>>>>>> driver): org.apache.spark.SparkException: Python worker exited unexpectedly
>>>>>>>> (crashed)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.spark.api.python.BasePythonRunner$ReaderIterator$$anonfun$1.applyOrElse(PythonRunner.scala:612)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.spark.api.python.BasePythonRunner$ReaderIterator$$anonfun$1.applyOrElse(PythonRunner.scala:594)
>>>>>>>>
>>>>>>>> at
>>>>>>>> scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:38)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.spark.api.python.PythonRunner$$anon$3.read(PythonRunner.scala:789)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.spark.api.python.PythonRunner$$anon$3.read(PythonRunner.scala:766)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:525)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
>>>>>>>>
>>>>>>>> at
>>>>>>>> scala.collection.Iterator.foreach(Iterator.scala:943)
>>>>>>>>
>>>>>>>> at
>>>>>>>> scala.collection.Iterator.foreach$(Iterator.scala:943)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28)
>>>>>>>>
>>>>>>>> at
>>>>>>>> scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:62)
>>>>>>>>
>>>>>>>> at
>>>>>>>> scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53)
>>>>>>>>
>>>>>>>> at
>>>>>>>> scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:105)
>>>>>>>>
>>>>>>>> at
>>>>>>>> scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:49)
>>>>>>>>
>>>>>>>> at scala.collection.TraversableOnce.to
>>>>>>>> (TraversableOnce.scala:366)
>>>>>>>>
>>>>>>>> at scala.collection.TraversableOnce.to
>>>>>>>> $(TraversableOnce.scala:364)
>>>>>>>>
>>>>>>>> at org.apache.spark.InterruptibleIterator.to
>>>>>>>> (InterruptibleIterator.scala:28)
>>>>>>>>
>>>>>>>> at
>>>>>>>> scala.collection.TraversableOnce.toBuffer(TraversableOnce.scala:358)
>>>>>>>>
>>>>>>>> at
>>>>>>>> scala.collection.TraversableOnce.toBuffer$(TraversableOnce.scala:358)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.spark.InterruptibleIterator.toBuffer(InterruptibleIterator.scala:28)
>>>>>>>>
>>>>>>>> at
>>>>>>>> scala.collection.TraversableOnce.toArray(TraversableOnce.scala:345)
>>>>>>>>
>>>>>>>> at
>>>>>>>> scala.collection.TraversableOnce.toArray$(TraversableOnce.scala:339)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.spark.InterruptibleIterator.toArray(InterruptibleIterator.scala:28)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.spark.rdd.RDD.$anonfun$collect$2(RDD.scala:1049)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2438)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.spark.scheduler.Task.run(Task.scala:141)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
>>>>>>>>
>>>>>>>> at
>>>>>>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>>>>>>>
>>>>>>>> at
>>>>>>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>>>>>>>
>>>>>>>> at
>>>>>>>> java.base/java.lang.Thread.run(Thread.java:829)
>>>>>>>>
>>>>>>>> Caused by: java.io.EOFException
>>>>>>>>
>>>>>>>> at
>>>>>>>> java.base/java.io.DataInputStream.readInt(DataInputStream.java:397)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.spark.api.python.PythonRunner$$anon$3.read(PythonRunner.scala:774)
>>>>>>>>
>>>>>>>> ... 32 more
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Driver stacktrace:
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2856)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2792)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2791)
>>>>>>>>
>>>>>>>> at
>>>>>>>> scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
>>>>>>>>
>>>>>>>> at
>>>>>>>> scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
>>>>>>>>
>>>>>>>> at
>>>>>>>> scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2791)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1247)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1247)
>>>>>>>>
>>>>>>>> at scala.Option.foreach(Option.scala:407)
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Jul 29, 2024 at 8:19 PM Sadha Chilukoori <
>>>>>>>> sage.quotient@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Mike,
>>>>>>>>>
>>>>>>>>> This appears to be an access issue on Windows + Python. Can you
>>>>>>>>> try setting up the PYTHON_PATH environment variable as described in this
>>>>>>>>> stackoverflow post
>>>>>>>>> https://stackoverflow.com/questions/60414394/createprocess-error-5-access-is-denied-pyspark
>>>>>>>>>
>>>>>>>>> - Sadha
>>>>>>>>>
>>>>>>>>> On Mon, Jul 29, 2024 at 3:39 PM mike Jadoo <mi...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Thanks. I just downloaded the corretto but I got this error
>>>>>>>>>> message, which was the same as before. [It was shared with me that this
>>>>>>>>>> saying that I have limited resources, i think]
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------------Py4JJavaError Traceback (most recent call last)
>>>>>>>>>> Cell In[3], line 13 8 squared_rdd = rdd.map(lambda x: x * x) 10 # Persist the DataFrame in memory 11 #squared_rdd.persist(StorageLevel.MEMORY_ONLY) 12 # Collect the results into a list---> 13 result = squared_rdd.collect() 15 # Print the result 16 print(result)
>>>>>>>>>>
>>>>>>>>>> File C:\spark\spark-3.5.1-bin-hadoop3\python\lib\pyspark.zip\pyspark\rdd.py:1833, in RDD.collect(self) 1831 with SCCallSiteSync(self.context): 1832 assert self.ctx._jvm is not None-> 1833 sock_info = self.ctx._jvm.PythonRDD.collectAndServe(self._jrdd.rdd()) 1834 return list(_load_from_socket(sock_info, self._jrdd_deserializer))
>>>>>>>>>>
>>>>>>>>>> File ~\anaconda3\Lib\site-packages\py4j\java_gateway.py:1322, in JavaMember.__call__(self, *args) 1316 command = proto.CALL_COMMAND_NAME +\ 1317 self.command_header +\ 1318 args_command +\ 1319 proto.END_COMMAND_PART 1321 answer = self.gateway_client.send_command(command)-> 1322 return_value = get_return_value( 1323 answer, self.gateway_client, self.target_id, self.name) 1325 for temp_arg in temp_args: 1326 if hasattr(temp_arg, "_detach"):
>>>>>>>>>>
>>>>>>>>>> File C:\spark\spark-3.5.1-bin-hadoop3\python\lib\pyspark.zip\pyspark\errors\exceptions\captured.py:179, in capture_sql_exception.<locals>.deco(*a, **kw) 177 def deco(*a: Any, **kw: Any) -> Any: 178 try:--> 179 return f(*a, **kw) 180 except Py4JJavaError as e: 181 converted = convert_exception(e.java_exception)
>>>>>>>>>>
>>>>>>>>>> File ~\anaconda3\Lib\site-packages\py4j\protocol.py:326, in get_return_value(answer, gateway_client, target_id, name) 324 value = OUTPUT_CONVERTER[type](answer[2:], gateway_client) 325 if answer[1] == REFERENCE_TYPE:--> 326 raise Py4JJavaError( 327 "An error occurred while calling {0}{1}{2}.\n". 328 format(target_id, ".", name), value) 329 else: 330 raise Py4JError( 331 "An error occurred while calling {0}{1}{2}. Trace:\n{3}\n". 332 format(target_id, ".", name, value))
>>>>>>>>>> Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
>>>>>>>>>> : org.apache.spark.SparkException: Job aborted due to stage failure: Task 7 in stage 0.0 failed 1 times, most recent failure: Lost task 7.0 in stage 0.0 (TID 7) (mjadoo.myfiosgateway.com executor driver): java.io.IOException: Cannot run program "C:\Users\mikej\AppData\Local\Programs\Python\Python312": CreateProcess error=5, Access is denied
>>>>>>>>>> at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1128)
>>>>>>>>>> at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1071)
>>>>>>>>>> at org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:181)
>>>>>>>>>> at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:109)
>>>>>>>>>> at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:124)
>>>>>>>>>> at org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:174)
>>>>>>>>>> at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:67)
>>>>>>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:367)
>>>>>>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:331)
>>>>>>>>>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)
>>>>>>>>>> at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166)
>>>>>>>>>> at org.apache.spark.scheduler.Task.run(Task.scala:141)
>>>>>>>>>> at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
>>>>>>>>>> at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
>>>>>>>>>> at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
>>>>>>>>>> at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
>>>>>>>>>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
>>>>>>>>>> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>>>>>>>>> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>>>>>>>>> at java.base/java.lang.Thread.run(Thread.java:829)
>>>>>>>>>> Caused by: java.io.IOException: CreateProcess error=5, Access is denied
>>>>>>>>>> at java.base/java.lang.ProcessImpl.create(Native Method)
>>>>>>>>>> at java.base/java.lang.ProcessImpl.<init>(ProcessImpl.java:492)
>>>>>>>>>> at java.base/java.lang.ProcessImpl.start(ProcessImpl.java:153)
>>>>>>>>>> at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1107)
>>>>>>>>>> ... 19 more
>>>>>>>>>>
>>>>>>>>>> Driver stacktrace:
>>>>>>>>>> at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2856)
>>>>>>>>>> at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2792)
>>>>>>>>>> at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2791)
>>>>>>>>>> at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
>>>>>>>>>> at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
>>>>>>>>>> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
>>>>>>>>>> at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2791)
>>>>>>>>>> at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1247)
>>>>>>>>>> at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1247)
>>>>>>>>>> at scala.Option.foreach(Option.scala:407)
>>>>>>>>>> at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1247)
>>>>>>>>>> at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:3060)
>>>>>>>>>> at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2994)
>>>>>>>>>> at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2983)
>>>>>>>>>> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
>>>>>>>>>> at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:989)
>>>>>>>>>> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2398)
>>>>>>>>>> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2419)
>>>>>>>>>> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2438)
>>>>>>>>>> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2463)
>>>>>>>>>> at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1049)
>>>>>>>>>> at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
>>>>>>>>>> at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
>>>>>>>>>> at org.apache.spark.rdd.RDD.withScope(RDD.scala:410)
>>>>>>>>>> at org.apache.spark.rdd.RDD.collect(RDD.scala:1048)
>>>>>>>>>> at org.apache.spark.api.python.PythonRDD$.collectAndServe(PythonRDD.scala:195)
>>>>>>>>>> at org.apache.spark.api.python.PythonRDD.collectAndServe(PythonRDD.scala)
>>>>>>>>>> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>>>>>> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>>>>>>>> at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>>>>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>>>>>>>>>> at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
>>>>>>>>>> at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
>>>>>>>>>> at py4j.Gateway.invoke(Gateway.java:282)
>>>>>>>>>> at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
>>>>>>>>>> at py4j.commands.CallCommand.execute(CallCommand.java:79)
>>>>>>>>>> at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
>>>>>>>>>> at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
>>>>>>>>>> at java.base/java.lang.Thread.run(Thread.java:829)
>>>>>>>>>> Caused by: java.io.IOException: Cannot run program "C:\Users\mikej\AppData\Local\Programs\Python\Python312": CreateProcess error=5, Access is denied
>>>>>>>>>> at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1128)
>>>>>>>>>> at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1071)
>>>>>>>>>> at org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:181)
>>>>>>>>>> at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:109)
>>>>>>>>>> at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:124)
>>>>>>>>>> at org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:174)
>>>>>>>>>> at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:67)
>>>>>>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:367)
>>>>>>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:331)
>>>>>>>>>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)
>>>>>>>>>> at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166)
>>>>>>>>>> at org.apache.spark.scheduler.Task.run(Task.scala:141)
>>>>>>>>>> at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
>>>>>>>>>> at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
>>>>>>>>>> at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
>>>>>>>>>> at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
>>>>>>>>>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
>>>>>>>>>> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>>>>>>>>> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>>>>>>>>> ... 1 more
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Jul 29, 2024 at 4:34 PM Sadha Chilukoori <
>>>>>>>>>> sage.quotient@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Mike,
>>>>>>>>>>>
>>>>>>>>>>> I'm not sure about the minimum requirements of a machine for
>>>>>>>>>>> running Spark. But to run some Pyspark scripts (and Jupiter notbebooks) on
>>>>>>>>>>> a local machine, I found the following steps are the easiest.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I installed Amazon corretto and updated the java_home variable
>>>>>>>>>>> as instructed here
>>>>>>>>>>> https://docs.aws.amazon.com/corretto/latest/corretto-11-ug/downloads-list.html
>>>>>>>>>>> (Any other java works too, I'm used to corretto from work).
>>>>>>>>>>>
>>>>>>>>>>> Then installed the Pyspark module using pip, which enabled me
>>>>>>>>>>> run Pyspark on my machine.
>>>>>>>>>>>
>>>>>>>>>>> -Sadha
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Jul 29, 2024, 12:51 PM mike Jadoo <mi...@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hello,
>>>>>>>>>>>>
>>>>>>>>>>>> I am trying to run Pyspark on my computer without success. I
>>>>>>>>>>>> follow several different directions from online sources and it appears that
>>>>>>>>>>>> I need to get a faster computer.
>>>>>>>>>>>>
>>>>>>>>>>>> I wanted to ask what are some recommendations for computer
>>>>>>>>>>>> specifications to run PySpark (Apache Spark).
>>>>>>>>>>>>
>>>>>>>>>>>> Any help would be greatly appreciated.
>>>>>>>>>>>>
>>>>>>>>>>>> Thank you,
>>>>>>>>>>>>
>>>>>>>>>>>> Mike
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Bjørn Jørgensen
>>>>>>> Vestre Aspehaug 4, 6010 Ålesund
>>>>>>> Norge
>>>>>>>
>>>>>>> +47 480 94 297
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Bjørn Jørgensen
>>>>> Vestre Aspehaug 4, 6010 Ålesund
>>>>> Norge
>>>>>
>>>>> +47 480 94 297
>>>>>
>>>>
Re: Question about installing Apache Spark [PySpark] computer requirements
Posted by mike Jadoo <mi...@gmail.com>.
Hello Angel,
I dont have a PYSPARK_PYTHON environment variable . I do have a
PYSPARK_HOME and here is the folder:
C:\Users\mikej\AppData\Local\Programs\Python\Python313\python.exe.
Here is a snapshot of my environment variables:
[image: image.png]
I currently have Python 3.12
PySpark spark-3.5.3
hadoop- 3.3.1
Java- 17.0.12
Mike
On Thu, Jan 16, 2025 at 9:26 AM Ángel <an...@gmail.com>
wrote:
> Hi,
>
> It seems like the Spark JVM is unable to start the Python processes for
> the executors. Have you tried setting the PYSPARK_PYTHON environment
> variable to your Python installation path?
>
> I usually avoid doing that because I don't want to set the environment
> variable globally on my Windows system or configure it every time I start a
> PySpark application. Instead, I add the following code to the
> venv\Lib\site-packages\pyspark\java_gateway.py file, inside the
> launch_gateway method, just after the docstring:
>
> import sys
> os.environ['PYSPARK_PYTHON'] = sys.executable
>
> This code sets the PYSPARK_PYTHON environment variable to the same Python
> executable that started the initial PySpark process.
>
> Hope this helps!
>
>
> El jue, 16 ene 2025 a las 14:02, mike Jadoo (<mi...@gmail.com>)
> escribió:
>
>> Hello everyone,
>>
>> Its been a long time and I wish everyone a happy New Year. I just
>> wanted to share I got a new laptop and tried installing Pyspark again and
>> it worked. However, I followed the instructions on setting
>> environmental variables from this youtube video:
>> https://www.youtube.com/watch?v=Qvke9S_ma7A&t=740s.
>>
>> I am also using Jupyter lab to run Pyspark. I would love to show anyone
>> here via Zoom call about my experience.
>>
>> Let me know what you think. Thanks for taking the time to help me out
>> before. This was a difficult experience and I appreciate all of you.
>>
>> Thanks,
>> Mike
>>
>> On Thu, Aug 1, 2024 at 9:28 PM Meena Rajani <me...@gmail.com>
>> wrote:
>>
>>> Did it work?
>>>
>>> Did you check the directory access? In your logs I saw
>>>
>>> java.io.IOException: Cannot run program "C:\Users\mikej\AppData\Local\Programs\Python\Python312": CreateProcess error=5, Access is denied
>>>
>>>
>>> Try chmod recursively on the directory and see
>>>
>>> Regards,
>>> Meena
>>>
>>> On Wed, Jul 31, 2024 at 6:13 AM Bjørn Jørgensen <
>>> bjornjorgensen@gmail.com> wrote:
>>>
>>>> if you have anaconda installed you can type conda create -n py311
>>>> python=3.11 and conda activate py311
>>>>
>>>> ons. 31. juli 2024 kl. 01:33 skrev mike Jadoo <mi...@gmail.com>:
>>>>
>>>>> Hello Bjorn,
>>>>>
>>>>> I am installing Python version 3.11. However, I don't see anaconda
>>>>> python with version 3.11. Where can I find this?
>>>>>
>>>>> Thank you,
>>>>> Mike
>>>>>
>>>>> On Tue, Jul 30, 2024 at 3:37 AM Bjørn Jørgensen <
>>>>> bjornjorgensen@gmail.com> wrote:
>>>>>
>>>>>> you are using python version 3.12
>>>>>> spark 3.X does not have support for python version 3.12.
>>>>>> Remove python 3.12 and install python version 3.11
>>>>>>
>>>>>> tir. 30. juli 2024 kl. 07:51 skrev mike Jadoo <mi...@gmail.com>:
>>>>>>
>>>>>>> Hello Sadha,
>>>>>>>
>>>>>>> Thank you for the suggestion. I entered the environmental variable,
>>>>>>> however, I am still having trouble. here is a screenshot of the env vars
>>>>>>> and my code.
>>>>>>>
>>>>>>>
>>>>>>> [image: image.png]
>>>>>>>
>>>>>>>
>>>>>>> Here is my code:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> from pyspark.sql import SparkSession
>>>>>>>
>>>>>>> from pyspark import SparkContext
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> spark = SparkSession.builder \
>>>>>>>
>>>>>>> .master("local[*]") \
>>>>>>>
>>>>>>> .appName("YourAppName") \
>>>>>>>
>>>>>>> .getOrCreate()
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> # Create an RDD from a list of numbers
>>>>>>>
>>>>>>> numbers = [1, 2, 3, 4, 5]
>>>>>>>
>>>>>>> rdd = spark.sparkContext.parallelize(numbers)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> # Perform a transformation on the RDD
>>>>>>>
>>>>>>> squared_rdd = rdd.map(lambda x: x * x)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> # Collect the results into a list
>>>>>>>
>>>>>>> result = squared_rdd.collect()
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> # Print the result
>>>>>>>
>>>>>>> print(result)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> error:
>>>>>>>
>>>>>>>
>>>>>>> *---------------------------------------------------------------------------*
>>>>>>>
>>>>>>> *Py4JJavaError* Traceback (most recent
>>>>>>> call last)
>>>>>>>
>>>>>>> Cell *In[5], line 13*
>>>>>>>
>>>>>>> 8 squared_rdd = rdd.map(*lambda* x: x * x)
>>>>>>>
>>>>>>> 10 # Persist the DataFrame in memory
>>>>>>>
>>>>>>> 11 #squared_rdd.persist(StorageLevel.MEMORY_ONLY)
>>>>>>>
>>>>>>> 12 # Collect the results into a list
>>>>>>>
>>>>>>> *---> 13* result = squared_rdd.collect()
>>>>>>>
>>>>>>> 15 # Print the result
>>>>>>>
>>>>>>> 16 print(result)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> File *~\anaconda3\Lib\site-packages\pyspark\rdd.py:1814*, in
>>>>>>> RDD.collect*(self)*
>>>>>>>
>>>>>>> 1812 *with* SCCallSiteSync(self.context):
>>>>>>>
>>>>>>> 1813 *assert* self.ctx._jvm *is* *not* *None*
>>>>>>>
>>>>>>> *-> 1814* sock_info = self.ctx._jvm.PythonRDD.collectAndServe(
>>>>>>> self._jrdd.rdd())
>>>>>>>
>>>>>>> 1815 *return* list(_load_from_socket(sock_info, self.
>>>>>>> _jrdd_deserializer))
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> File *~\anaconda3\Lib\site-packages\py4j\java_gateway.py:1322*, in
>>>>>>> JavaMember.__call__*(self, *args)*
>>>>>>>
>>>>>>> 1316 command = proto.CALL_COMMAND_NAME +\
>>>>>>>
>>>>>>> 1317 self.command_header +\
>>>>>>>
>>>>>>> 1318 args_command +\
>>>>>>>
>>>>>>> 1319 proto.END_COMMAND_PART
>>>>>>>
>>>>>>> 1321 answer = self.gateway_client.send_command(command)
>>>>>>>
>>>>>>> *-> 1322* return_value = get_return_value(
>>>>>>>
>>>>>>> 1323 answer, self.gateway_client, self.target_id, self.name)
>>>>>>>
>>>>>>> 1325 *for* temp_arg *in* temp_args:
>>>>>>>
>>>>>>> 1326 *if* hasattr(temp_arg, "_detach"):
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> File
>>>>>>> *~\anaconda3\Lib\site-packages\pyspark\errors\exceptions\captured.py:169*,
>>>>>>> in capture_sql_exception.<locals>.deco*(*a, **kw)*
>>>>>>>
>>>>>>> 167 *def* deco(*a: Any, **kw: Any) -> Any:
>>>>>>>
>>>>>>> 168 *try*:
>>>>>>>
>>>>>>> *--> 169* *return* f(*a, **kw)
>>>>>>>
>>>>>>> 170 *except* Py4JJavaError *as* e:
>>>>>>>
>>>>>>> 171 converted = convert_exception(e.java_exception)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> File *~\anaconda3\Lib\site-packages\py4j\protocol.py:326*, in
>>>>>>> get_return_value*(answer, gateway_client, target_id, name)*
>>>>>>>
>>>>>>> 324 value = OUTPUT_CONVERTER[type](answer[2:], gateway_client)
>>>>>>>
>>>>>>> 325 *if* answer[1] == REFERENCE_TYPE:
>>>>>>>
>>>>>>> *--> 326* *raise* Py4JJavaError(
>>>>>>>
>>>>>>> 327 "An error occurred while calling *{0}{1}{2}*.*\n*".
>>>>>>>
>>>>>>> 328 format(target_id, ".", name), value)
>>>>>>>
>>>>>>> 329 *else*:
>>>>>>>
>>>>>>> 330 *raise* Py4JError(
>>>>>>>
>>>>>>> 331 "An error occurred while calling *{0}{1}{2}*. Trace:
>>>>>>> *\n{3}\n*".
>>>>>>>
>>>>>>> 332 format(target_id, ".", name, value))
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> *Py4JJavaError*: An error occurred while calling
>>>>>>> z:org.apache.spark.api.python.PythonRDD.collectAndServe.
>>>>>>>
>>>>>>> : org.apache.spark.SparkException: Job aborted due to stage failure:
>>>>>>> Task 7 in stage 1.0 failed 1 times, most recent failure: Lost task 7.0 in
>>>>>>> stage 1.0 (TID 19) (mjadoo.myfiosgateway.com executor driver):
>>>>>>> org.apache.spark.SparkException: Python worker exited unexpectedly (crashed)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.spark.api.python.BasePythonRunner$ReaderIterator$$anonfun$1.applyOrElse(PythonRunner.scala:612)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.spark.api.python.BasePythonRunner$ReaderIterator$$anonfun$1.applyOrElse(PythonRunner.scala:594)
>>>>>>>
>>>>>>> at
>>>>>>> scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:38)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.spark.api.python.PythonRunner$$anon$3.read(PythonRunner.scala:789)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.spark.api.python.PythonRunner$$anon$3.read(PythonRunner.scala:766)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:525)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
>>>>>>>
>>>>>>> at
>>>>>>> scala.collection.Iterator.foreach(Iterator.scala:943)
>>>>>>>
>>>>>>> at
>>>>>>> scala.collection.Iterator.foreach$(Iterator.scala:943)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28)
>>>>>>>
>>>>>>> at
>>>>>>> scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:62)
>>>>>>>
>>>>>>> at
>>>>>>> scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53)
>>>>>>>
>>>>>>> at
>>>>>>> scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:105)
>>>>>>>
>>>>>>> at
>>>>>>> scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:49)
>>>>>>>
>>>>>>> at scala.collection.TraversableOnce.to
>>>>>>> (TraversableOnce.scala:366)
>>>>>>>
>>>>>>> at scala.collection.TraversableOnce.to
>>>>>>> $(TraversableOnce.scala:364)
>>>>>>>
>>>>>>> at org.apache.spark.InterruptibleIterator.to
>>>>>>> (InterruptibleIterator.scala:28)
>>>>>>>
>>>>>>> at
>>>>>>> scala.collection.TraversableOnce.toBuffer(TraversableOnce.scala:358)
>>>>>>>
>>>>>>> at
>>>>>>> scala.collection.TraversableOnce.toBuffer$(TraversableOnce.scala:358)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.spark.InterruptibleIterator.toBuffer(InterruptibleIterator.scala:28)
>>>>>>>
>>>>>>> at
>>>>>>> scala.collection.TraversableOnce.toArray(TraversableOnce.scala:345)
>>>>>>>
>>>>>>> at
>>>>>>> scala.collection.TraversableOnce.toArray$(TraversableOnce.scala:339)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.spark.InterruptibleIterator.toArray(InterruptibleIterator.scala:28)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.spark.rdd.RDD.$anonfun$collect$2(RDD.scala:1049)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2438)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.spark.scheduler.Task.run(Task.scala:141)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
>>>>>>>
>>>>>>> at
>>>>>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>>>>>>
>>>>>>> at
>>>>>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>>>>>>
>>>>>>> at java.base/java.lang.Thread.run(Thread.java:829)
>>>>>>>
>>>>>>> Caused by: java.io.EOFException
>>>>>>>
>>>>>>> at
>>>>>>> java.base/java.io.DataInputStream.readInt(DataInputStream.java:397)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.spark.api.python.PythonRunner$$anon$3.read(PythonRunner.scala:774)
>>>>>>>
>>>>>>> ... 32 more
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Driver stacktrace:
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2856)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2792)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2791)
>>>>>>>
>>>>>>> at
>>>>>>> scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
>>>>>>>
>>>>>>> at
>>>>>>> scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
>>>>>>>
>>>>>>> at
>>>>>>> scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2791)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1247)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1247)
>>>>>>>
>>>>>>> at scala.Option.foreach(Option.scala:407)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Jul 29, 2024 at 8:19 PM Sadha Chilukoori <
>>>>>>> sage.quotient@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi Mike,
>>>>>>>>
>>>>>>>> This appears to be an access issue on Windows + Python. Can you try
>>>>>>>> setting up the PYTHON_PATH environment variable as described in this
>>>>>>>> stackoverflow post
>>>>>>>> https://stackoverflow.com/questions/60414394/createprocess-error-5-access-is-denied-pyspark
>>>>>>>>
>>>>>>>> - Sadha
>>>>>>>>
>>>>>>>> On Mon, Jul 29, 2024 at 3:39 PM mike Jadoo <mi...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Thanks. I just downloaded the corretto but I got this error
>>>>>>>>> message, which was the same as before. [It was shared with me that this
>>>>>>>>> saying that I have limited resources, i think]
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------------Py4JJavaError Traceback (most recent call last)
>>>>>>>>> Cell In[3], line 13 8 squared_rdd = rdd.map(lambda x: x * x) 10 # Persist the DataFrame in memory 11 #squared_rdd.persist(StorageLevel.MEMORY_ONLY) 12 # Collect the results into a list---> 13 result = squared_rdd.collect() 15 # Print the result 16 print(result)
>>>>>>>>>
>>>>>>>>> File C:\spark\spark-3.5.1-bin-hadoop3\python\lib\pyspark.zip\pyspark\rdd.py:1833, in RDD.collect(self) 1831 with SCCallSiteSync(self.context): 1832 assert self.ctx._jvm is not None-> 1833 sock_info = self.ctx._jvm.PythonRDD.collectAndServe(self._jrdd.rdd()) 1834 return list(_load_from_socket(sock_info, self._jrdd_deserializer))
>>>>>>>>>
>>>>>>>>> File ~\anaconda3\Lib\site-packages\py4j\java_gateway.py:1322, in JavaMember.__call__(self, *args) 1316 command = proto.CALL_COMMAND_NAME +\ 1317 self.command_header +\ 1318 args_command +\ 1319 proto.END_COMMAND_PART 1321 answer = self.gateway_client.send_command(command)-> 1322 return_value = get_return_value( 1323 answer, self.gateway_client, self.target_id, self.name) 1325 for temp_arg in temp_args: 1326 if hasattr(temp_arg, "_detach"):
>>>>>>>>>
>>>>>>>>> File C:\spark\spark-3.5.1-bin-hadoop3\python\lib\pyspark.zip\pyspark\errors\exceptions\captured.py:179, in capture_sql_exception.<locals>.deco(*a, **kw) 177 def deco(*a: Any, **kw: Any) -> Any: 178 try:--> 179 return f(*a, **kw) 180 except Py4JJavaError as e: 181 converted = convert_exception(e.java_exception)
>>>>>>>>>
>>>>>>>>> File ~\anaconda3\Lib\site-packages\py4j\protocol.py:326, in get_return_value(answer, gateway_client, target_id, name) 324 value = OUTPUT_CONVERTER[type](answer[2:], gateway_client) 325 if answer[1] == REFERENCE_TYPE:--> 326 raise Py4JJavaError( 327 "An error occurred while calling {0}{1}{2}.\n". 328 format(target_id, ".", name), value) 329 else: 330 raise Py4JError( 331 "An error occurred while calling {0}{1}{2}. Trace:\n{3}\n". 332 format(target_id, ".", name, value))
>>>>>>>>> Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
>>>>>>>>> : org.apache.spark.SparkException: Job aborted due to stage failure: Task 7 in stage 0.0 failed 1 times, most recent failure: Lost task 7.0 in stage 0.0 (TID 7) (mjadoo.myfiosgateway.com executor driver): java.io.IOException: Cannot run program "C:\Users\mikej\AppData\Local\Programs\Python\Python312": CreateProcess error=5, Access is denied
>>>>>>>>> at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1128)
>>>>>>>>> at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1071)
>>>>>>>>> at org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:181)
>>>>>>>>> at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:109)
>>>>>>>>> at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:124)
>>>>>>>>> at org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:174)
>>>>>>>>> at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:67)
>>>>>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:367)
>>>>>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:331)
>>>>>>>>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)
>>>>>>>>> at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166)
>>>>>>>>> at org.apache.spark.scheduler.Task.run(Task.scala:141)
>>>>>>>>> at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
>>>>>>>>> at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
>>>>>>>>> at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
>>>>>>>>> at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
>>>>>>>>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
>>>>>>>>> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>>>>>>>> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>>>>>>>> at java.base/java.lang.Thread.run(Thread.java:829)
>>>>>>>>> Caused by: java.io.IOException: CreateProcess error=5, Access is denied
>>>>>>>>> at java.base/java.lang.ProcessImpl.create(Native Method)
>>>>>>>>> at java.base/java.lang.ProcessImpl.<init>(ProcessImpl.java:492)
>>>>>>>>> at java.base/java.lang.ProcessImpl.start(ProcessImpl.java:153)
>>>>>>>>> at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1107)
>>>>>>>>> ... 19 more
>>>>>>>>>
>>>>>>>>> Driver stacktrace:
>>>>>>>>> at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2856)
>>>>>>>>> at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2792)
>>>>>>>>> at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2791)
>>>>>>>>> at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
>>>>>>>>> at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
>>>>>>>>> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
>>>>>>>>> at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2791)
>>>>>>>>> at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1247)
>>>>>>>>> at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1247)
>>>>>>>>> at scala.Option.foreach(Option.scala:407)
>>>>>>>>> at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1247)
>>>>>>>>> at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:3060)
>>>>>>>>> at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2994)
>>>>>>>>> at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2983)
>>>>>>>>> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
>>>>>>>>> at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:989)
>>>>>>>>> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2398)
>>>>>>>>> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2419)
>>>>>>>>> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2438)
>>>>>>>>> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2463)
>>>>>>>>> at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1049)
>>>>>>>>> at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
>>>>>>>>> at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
>>>>>>>>> at org.apache.spark.rdd.RDD.withScope(RDD.scala:410)
>>>>>>>>> at org.apache.spark.rdd.RDD.collect(RDD.scala:1048)
>>>>>>>>> at org.apache.spark.api.python.PythonRDD$.collectAndServe(PythonRDD.scala:195)
>>>>>>>>> at org.apache.spark.api.python.PythonRDD.collectAndServe(PythonRDD.scala)
>>>>>>>>> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>>>>> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>>>>>>> at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>>>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>>>>>>>>> at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
>>>>>>>>> at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
>>>>>>>>> at py4j.Gateway.invoke(Gateway.java:282)
>>>>>>>>> at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
>>>>>>>>> at py4j.commands.CallCommand.execute(CallCommand.java:79)
>>>>>>>>> at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
>>>>>>>>> at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
>>>>>>>>> at java.base/java.lang.Thread.run(Thread.java:829)
>>>>>>>>> Caused by: java.io.IOException: Cannot run program "C:\Users\mikej\AppData\Local\Programs\Python\Python312": CreateProcess error=5, Access is denied
>>>>>>>>> at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1128)
>>>>>>>>> at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1071)
>>>>>>>>> at org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:181)
>>>>>>>>> at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:109)
>>>>>>>>> at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:124)
>>>>>>>>> at org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:174)
>>>>>>>>> at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:67)
>>>>>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:367)
>>>>>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:331)
>>>>>>>>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)
>>>>>>>>> at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166)
>>>>>>>>> at org.apache.spark.scheduler.Task.run(Task.scala:141)
>>>>>>>>> at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
>>>>>>>>> at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
>>>>>>>>> at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
>>>>>>>>> at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
>>>>>>>>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
>>>>>>>>> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>>>>>>>> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>>>>>>>> ... 1 more
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Jul 29, 2024 at 4:34 PM Sadha Chilukoori <
>>>>>>>>> sage.quotient@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Mike,
>>>>>>>>>>
>>>>>>>>>> I'm not sure about the minimum requirements of a machine for
>>>>>>>>>> running Spark. But to run some Pyspark scripts (and Jupiter notbebooks) on
>>>>>>>>>> a local machine, I found the following steps are the easiest.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I installed Amazon corretto and updated the java_home variable as
>>>>>>>>>> instructed here
>>>>>>>>>> https://docs.aws.amazon.com/corretto/latest/corretto-11-ug/downloads-list.html
>>>>>>>>>> (Any other java works too, I'm used to corretto from work).
>>>>>>>>>>
>>>>>>>>>> Then installed the Pyspark module using pip, which enabled me run
>>>>>>>>>> Pyspark on my machine.
>>>>>>>>>>
>>>>>>>>>> -Sadha
>>>>>>>>>>
>>>>>>>>>> On Mon, Jul 29, 2024, 12:51 PM mike Jadoo <mi...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hello,
>>>>>>>>>>>
>>>>>>>>>>> I am trying to run Pyspark on my computer without success. I
>>>>>>>>>>> follow several different directions from online sources and it appears that
>>>>>>>>>>> I need to get a faster computer.
>>>>>>>>>>>
>>>>>>>>>>> I wanted to ask what are some recommendations for computer
>>>>>>>>>>> specifications to run PySpark (Apache Spark).
>>>>>>>>>>>
>>>>>>>>>>> Any help would be greatly appreciated.
>>>>>>>>>>>
>>>>>>>>>>> Thank you,
>>>>>>>>>>>
>>>>>>>>>>> Mike
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Bjørn Jørgensen
>>>>>> Vestre Aspehaug 4, 6010 Ålesund
>>>>>> Norge
>>>>>>
>>>>>> +47 480 94 297
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Bjørn Jørgensen
>>>> Vestre Aspehaug 4, 6010 Ålesund
>>>> Norge
>>>>
>>>> +47 480 94 297
>>>>
>>>
Re: Question about installing Apache Spark [PySpark] computer requirements
Posted by Ángel <an...@gmail.com>.
Hi,
It seems like the Spark JVM is unable to start the Python processes for the
executors. Have you tried setting the PYSPARK_PYTHON environment variable
to your Python installation path?
I usually avoid doing that because I don't want to set the environment
variable globally on my Windows system or configure it every time I start a
PySpark application. Instead, I add the following code to the
venv\Lib\site-packages\pyspark\java_gateway.py file, inside the
launch_gateway method, just after the docstring:
import sys
os.environ['PYSPARK_PYTHON'] = sys.executable
This code sets the PYSPARK_PYTHON environment variable to the same Python
executable that started the initial PySpark process.
Hope this helps!
El jue, 16 ene 2025 a las 14:02, mike Jadoo (<mi...@gmail.com>)
escribió:
> Hello everyone,
>
> Its been a long time and I wish everyone a happy New Year. I just wanted
> to share I got a new laptop and tried installing Pyspark again and it
> worked. However, I followed the instructions on setting
> environmental variables from this youtube video:
> https://www.youtube.com/watch?v=Qvke9S_ma7A&t=740s.
>
> I am also using Jupyter lab to run Pyspark. I would love to show anyone
> here via Zoom call about my experience.
>
> Let me know what you think. Thanks for taking the time to help me out
> before. This was a difficult experience and I appreciate all of you.
>
> Thanks,
> Mike
>
> On Thu, Aug 1, 2024 at 9:28 PM Meena Rajani <me...@gmail.com>
> wrote:
>
>> Did it work?
>>
>> Did you check the directory access? In your logs I saw
>>
>> java.io.IOException: Cannot run program "C:\Users\mikej\AppData\Local\Programs\Python\Python312": CreateProcess error=5, Access is denied
>>
>>
>> Try chmod recursively on the directory and see
>>
>> Regards,
>> Meena
>>
>> On Wed, Jul 31, 2024 at 6:13 AM Bjørn Jørgensen <bj...@gmail.com>
>> wrote:
>>
>>> if you have anaconda installed you can type conda create -n py311
>>> python=3.11 and conda activate py311
>>>
>>> ons. 31. juli 2024 kl. 01:33 skrev mike Jadoo <mi...@gmail.com>:
>>>
>>>> Hello Bjorn,
>>>>
>>>> I am installing Python version 3.11. However, I don't see anaconda
>>>> python with version 3.11. Where can I find this?
>>>>
>>>> Thank you,
>>>> Mike
>>>>
>>>> On Tue, Jul 30, 2024 at 3:37 AM Bjørn Jørgensen <
>>>> bjornjorgensen@gmail.com> wrote:
>>>>
>>>>> you are using python version 3.12
>>>>> spark 3.X does not have support for python version 3.12.
>>>>> Remove python 3.12 and install python version 3.11
>>>>>
>>>>> tir. 30. juli 2024 kl. 07:51 skrev mike Jadoo <mi...@gmail.com>:
>>>>>
>>>>>> Hello Sadha,
>>>>>>
>>>>>> Thank you for the suggestion. I entered the environmental variable,
>>>>>> however, I am still having trouble. here is a screenshot of the env vars
>>>>>> and my code.
>>>>>>
>>>>>>
>>>>>> [image: image.png]
>>>>>>
>>>>>>
>>>>>> Here is my code:
>>>>>>
>>>>>>
>>>>>>
>>>>>> from pyspark.sql import SparkSession
>>>>>>
>>>>>> from pyspark import SparkContext
>>>>>>
>>>>>>
>>>>>>
>>>>>> spark = SparkSession.builder \
>>>>>>
>>>>>> .master("local[*]") \
>>>>>>
>>>>>> .appName("YourAppName") \
>>>>>>
>>>>>> .getOrCreate()
>>>>>>
>>>>>>
>>>>>>
>>>>>> # Create an RDD from a list of numbers
>>>>>>
>>>>>> numbers = [1, 2, 3, 4, 5]
>>>>>>
>>>>>> rdd = spark.sparkContext.parallelize(numbers)
>>>>>>
>>>>>>
>>>>>>
>>>>>> # Perform a transformation on the RDD
>>>>>>
>>>>>> squared_rdd = rdd.map(lambda x: x * x)
>>>>>>
>>>>>>
>>>>>>
>>>>>> # Collect the results into a list
>>>>>>
>>>>>> result = squared_rdd.collect()
>>>>>>
>>>>>>
>>>>>>
>>>>>> # Print the result
>>>>>>
>>>>>> print(result)
>>>>>>
>>>>>>
>>>>>>
>>>>>> error:
>>>>>>
>>>>>>
>>>>>> *---------------------------------------------------------------------------*
>>>>>>
>>>>>> *Py4JJavaError* Traceback (most recent
>>>>>> call last)
>>>>>>
>>>>>> Cell *In[5], line 13*
>>>>>>
>>>>>> 8 squared_rdd = rdd.map(*lambda* x: x * x)
>>>>>>
>>>>>> 10 # Persist the DataFrame in memory
>>>>>>
>>>>>> 11 #squared_rdd.persist(StorageLevel.MEMORY_ONLY)
>>>>>>
>>>>>> 12 # Collect the results into a list
>>>>>>
>>>>>> *---> 13* result = squared_rdd.collect()
>>>>>>
>>>>>> 15 # Print the result
>>>>>>
>>>>>> 16 print(result)
>>>>>>
>>>>>>
>>>>>>
>>>>>> File *~\anaconda3\Lib\site-packages\pyspark\rdd.py:1814*, in
>>>>>> RDD.collect*(self)*
>>>>>>
>>>>>> 1812 *with* SCCallSiteSync(self.context):
>>>>>>
>>>>>> 1813 *assert* self.ctx._jvm *is* *not* *None*
>>>>>>
>>>>>> *-> 1814* sock_info = self.ctx._jvm.PythonRDD.collectAndServe(
>>>>>> self._jrdd.rdd())
>>>>>>
>>>>>> 1815 *return* list(_load_from_socket(sock_info, self.
>>>>>> _jrdd_deserializer))
>>>>>>
>>>>>>
>>>>>>
>>>>>> File *~\anaconda3\Lib\site-packages\py4j\java_gateway.py:1322*, in
>>>>>> JavaMember.__call__*(self, *args)*
>>>>>>
>>>>>> 1316 command = proto.CALL_COMMAND_NAME +\
>>>>>>
>>>>>> 1317 self.command_header +\
>>>>>>
>>>>>> 1318 args_command +\
>>>>>>
>>>>>> 1319 proto.END_COMMAND_PART
>>>>>>
>>>>>> 1321 answer = self.gateway_client.send_command(command)
>>>>>>
>>>>>> *-> 1322* return_value = get_return_value(
>>>>>>
>>>>>> 1323 answer, self.gateway_client, self.target_id, self.name)
>>>>>>
>>>>>> 1325 *for* temp_arg *in* temp_args:
>>>>>>
>>>>>> 1326 *if* hasattr(temp_arg, "_detach"):
>>>>>>
>>>>>>
>>>>>>
>>>>>> File
>>>>>> *~\anaconda3\Lib\site-packages\pyspark\errors\exceptions\captured.py:169*,
>>>>>> in capture_sql_exception.<locals>.deco*(*a, **kw)*
>>>>>>
>>>>>> 167 *def* deco(*a: Any, **kw: Any) -> Any:
>>>>>>
>>>>>> 168 *try*:
>>>>>>
>>>>>> *--> 169* *return* f(*a, **kw)
>>>>>>
>>>>>> 170 *except* Py4JJavaError *as* e:
>>>>>>
>>>>>> 171 converted = convert_exception(e.java_exception)
>>>>>>
>>>>>>
>>>>>>
>>>>>> File *~\anaconda3\Lib\site-packages\py4j\protocol.py:326*, in
>>>>>> get_return_value*(answer, gateway_client, target_id, name)*
>>>>>>
>>>>>> 324 value = OUTPUT_CONVERTER[type](answer[2:], gateway_client)
>>>>>>
>>>>>> 325 *if* answer[1] == REFERENCE_TYPE:
>>>>>>
>>>>>> *--> 326* *raise* Py4JJavaError(
>>>>>>
>>>>>> 327 "An error occurred while calling *{0}{1}{2}*.*\n*".
>>>>>>
>>>>>> 328 format(target_id, ".", name), value)
>>>>>>
>>>>>> 329 *else*:
>>>>>>
>>>>>> 330 *raise* Py4JError(
>>>>>>
>>>>>> 331 "An error occurred while calling *{0}{1}{2}*. Trace:
>>>>>> *\n{3}\n*".
>>>>>>
>>>>>> 332 format(target_id, ".", name, value))
>>>>>>
>>>>>>
>>>>>>
>>>>>> *Py4JJavaError*: An error occurred while calling
>>>>>> z:org.apache.spark.api.python.PythonRDD.collectAndServe.
>>>>>>
>>>>>> : org.apache.spark.SparkException: Job aborted due to stage failure:
>>>>>> Task 7 in stage 1.0 failed 1 times, most recent failure: Lost task 7.0 in
>>>>>> stage 1.0 (TID 19) (mjadoo.myfiosgateway.com executor driver):
>>>>>> org.apache.spark.SparkException: Python worker exited unexpectedly (crashed)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.api.python.BasePythonRunner$ReaderIterator$$anonfun$1.applyOrElse(PythonRunner.scala:612)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.api.python.BasePythonRunner$ReaderIterator$$anonfun$1.applyOrElse(PythonRunner.scala:594)
>>>>>>
>>>>>> at
>>>>>> scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:38)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.api.python.PythonRunner$$anon$3.read(PythonRunner.scala:789)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.api.python.PythonRunner$$anon$3.read(PythonRunner.scala:766)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:525)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
>>>>>>
>>>>>> at
>>>>>> scala.collection.Iterator.foreach(Iterator.scala:943)
>>>>>>
>>>>>> at
>>>>>> scala.collection.Iterator.foreach$(Iterator.scala:943)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28)
>>>>>>
>>>>>> at
>>>>>> scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:62)
>>>>>>
>>>>>> at
>>>>>> scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53)
>>>>>>
>>>>>> at
>>>>>> scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:105)
>>>>>>
>>>>>> at
>>>>>> scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:49)
>>>>>>
>>>>>> at scala.collection.TraversableOnce.to
>>>>>> (TraversableOnce.scala:366)
>>>>>>
>>>>>> at scala.collection.TraversableOnce.to
>>>>>> $(TraversableOnce.scala:364)
>>>>>>
>>>>>> at org.apache.spark.InterruptibleIterator.to
>>>>>> (InterruptibleIterator.scala:28)
>>>>>>
>>>>>> at
>>>>>> scala.collection.TraversableOnce.toBuffer(TraversableOnce.scala:358)
>>>>>>
>>>>>> at
>>>>>> scala.collection.TraversableOnce.toBuffer$(TraversableOnce.scala:358)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.InterruptibleIterator.toBuffer(InterruptibleIterator.scala:28)
>>>>>>
>>>>>> at
>>>>>> scala.collection.TraversableOnce.toArray(TraversableOnce.scala:345)
>>>>>>
>>>>>> at
>>>>>> scala.collection.TraversableOnce.toArray$(TraversableOnce.scala:339)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.InterruptibleIterator.toArray(InterruptibleIterator.scala:28)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.rdd.RDD.$anonfun$collect$2(RDD.scala:1049)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2438)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.scheduler.Task.run(Task.scala:141)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
>>>>>>
>>>>>> at
>>>>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>>>>>
>>>>>> at
>>>>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>>>>>
>>>>>> at java.base/java.lang.Thread.run(Thread.java:829)
>>>>>>
>>>>>> Caused by: java.io.EOFException
>>>>>>
>>>>>> at
>>>>>> java.base/java.io.DataInputStream.readInt(DataInputStream.java:397)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.api.python.PythonRunner$$anon$3.read(PythonRunner.scala:774)
>>>>>>
>>>>>> ... 32 more
>>>>>>
>>>>>>
>>>>>>
>>>>>> Driver stacktrace:
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2856)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2792)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2791)
>>>>>>
>>>>>> at
>>>>>> scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
>>>>>>
>>>>>> at
>>>>>> scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
>>>>>>
>>>>>> at
>>>>>> scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2791)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1247)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1247)
>>>>>>
>>>>>> at scala.Option.foreach(Option.scala:407)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Jul 29, 2024 at 8:19 PM Sadha Chilukoori <
>>>>>> sage.quotient@gmail.com> wrote:
>>>>>>
>>>>>>> Hi Mike,
>>>>>>>
>>>>>>> This appears to be an access issue on Windows + Python. Can you try
>>>>>>> setting up the PYTHON_PATH environment variable as described in this
>>>>>>> stackoverflow post
>>>>>>> https://stackoverflow.com/questions/60414394/createprocess-error-5-access-is-denied-pyspark
>>>>>>>
>>>>>>> - Sadha
>>>>>>>
>>>>>>> On Mon, Jul 29, 2024 at 3:39 PM mike Jadoo <mi...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Thanks. I just downloaded the corretto but I got this error
>>>>>>>> message, which was the same as before. [It was shared with me that this
>>>>>>>> saying that I have limited resources, i think]
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------------Py4JJavaError Traceback (most recent call last)
>>>>>>>> Cell In[3], line 13 8 squared_rdd = rdd.map(lambda x: x * x) 10 # Persist the DataFrame in memory 11 #squared_rdd.persist(StorageLevel.MEMORY_ONLY) 12 # Collect the results into a list---> 13 result = squared_rdd.collect() 15 # Print the result 16 print(result)
>>>>>>>>
>>>>>>>> File C:\spark\spark-3.5.1-bin-hadoop3\python\lib\pyspark.zip\pyspark\rdd.py:1833, in RDD.collect(self) 1831 with SCCallSiteSync(self.context): 1832 assert self.ctx._jvm is not None-> 1833 sock_info = self.ctx._jvm.PythonRDD.collectAndServe(self._jrdd.rdd()) 1834 return list(_load_from_socket(sock_info, self._jrdd_deserializer))
>>>>>>>>
>>>>>>>> File ~\anaconda3\Lib\site-packages\py4j\java_gateway.py:1322, in JavaMember.__call__(self, *args) 1316 command = proto.CALL_COMMAND_NAME +\ 1317 self.command_header +\ 1318 args_command +\ 1319 proto.END_COMMAND_PART 1321 answer = self.gateway_client.send_command(command)-> 1322 return_value = get_return_value( 1323 answer, self.gateway_client, self.target_id, self.name) 1325 for temp_arg in temp_args: 1326 if hasattr(temp_arg, "_detach"):
>>>>>>>>
>>>>>>>> File C:\spark\spark-3.5.1-bin-hadoop3\python\lib\pyspark.zip\pyspark\errors\exceptions\captured.py:179, in capture_sql_exception.<locals>.deco(*a, **kw) 177 def deco(*a: Any, **kw: Any) -> Any: 178 try:--> 179 return f(*a, **kw) 180 except Py4JJavaError as e: 181 converted = convert_exception(e.java_exception)
>>>>>>>>
>>>>>>>> File ~\anaconda3\Lib\site-packages\py4j\protocol.py:326, in get_return_value(answer, gateway_client, target_id, name) 324 value = OUTPUT_CONVERTER[type](answer[2:], gateway_client) 325 if answer[1] == REFERENCE_TYPE:--> 326 raise Py4JJavaError( 327 "An error occurred while calling {0}{1}{2}.\n". 328 format(target_id, ".", name), value) 329 else: 330 raise Py4JError( 331 "An error occurred while calling {0}{1}{2}. Trace:\n{3}\n". 332 format(target_id, ".", name, value))
>>>>>>>> Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
>>>>>>>> : org.apache.spark.SparkException: Job aborted due to stage failure: Task 7 in stage 0.0 failed 1 times, most recent failure: Lost task 7.0 in stage 0.0 (TID 7) (mjadoo.myfiosgateway.com executor driver): java.io.IOException: Cannot run program "C:\Users\mikej\AppData\Local\Programs\Python\Python312": CreateProcess error=5, Access is denied
>>>>>>>> at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1128)
>>>>>>>> at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1071)
>>>>>>>> at org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:181)
>>>>>>>> at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:109)
>>>>>>>> at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:124)
>>>>>>>> at org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:174)
>>>>>>>> at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:67)
>>>>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:367)
>>>>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:331)
>>>>>>>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)
>>>>>>>> at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166)
>>>>>>>> at org.apache.spark.scheduler.Task.run(Task.scala:141)
>>>>>>>> at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
>>>>>>>> at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
>>>>>>>> at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
>>>>>>>> at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
>>>>>>>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
>>>>>>>> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>>>>>>> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>>>>>>> at java.base/java.lang.Thread.run(Thread.java:829)
>>>>>>>> Caused by: java.io.IOException: CreateProcess error=5, Access is denied
>>>>>>>> at java.base/java.lang.ProcessImpl.create(Native Method)
>>>>>>>> at java.base/java.lang.ProcessImpl.<init>(ProcessImpl.java:492)
>>>>>>>> at java.base/java.lang.ProcessImpl.start(ProcessImpl.java:153)
>>>>>>>> at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1107)
>>>>>>>> ... 19 more
>>>>>>>>
>>>>>>>> Driver stacktrace:
>>>>>>>> at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2856)
>>>>>>>> at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2792)
>>>>>>>> at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2791)
>>>>>>>> at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
>>>>>>>> at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
>>>>>>>> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
>>>>>>>> at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2791)
>>>>>>>> at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1247)
>>>>>>>> at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1247)
>>>>>>>> at scala.Option.foreach(Option.scala:407)
>>>>>>>> at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1247)
>>>>>>>> at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:3060)
>>>>>>>> at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2994)
>>>>>>>> at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2983)
>>>>>>>> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
>>>>>>>> at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:989)
>>>>>>>> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2398)
>>>>>>>> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2419)
>>>>>>>> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2438)
>>>>>>>> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2463)
>>>>>>>> at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1049)
>>>>>>>> at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
>>>>>>>> at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
>>>>>>>> at org.apache.spark.rdd.RDD.withScope(RDD.scala:410)
>>>>>>>> at org.apache.spark.rdd.RDD.collect(RDD.scala:1048)
>>>>>>>> at org.apache.spark.api.python.PythonRDD$.collectAndServe(PythonRDD.scala:195)
>>>>>>>> at org.apache.spark.api.python.PythonRDD.collectAndServe(PythonRDD.scala)
>>>>>>>> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>>>> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>>>>>> at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>>>>>>>> at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
>>>>>>>> at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
>>>>>>>> at py4j.Gateway.invoke(Gateway.java:282)
>>>>>>>> at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
>>>>>>>> at py4j.commands.CallCommand.execute(CallCommand.java:79)
>>>>>>>> at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
>>>>>>>> at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
>>>>>>>> at java.base/java.lang.Thread.run(Thread.java:829)
>>>>>>>> Caused by: java.io.IOException: Cannot run program "C:\Users\mikej\AppData\Local\Programs\Python\Python312": CreateProcess error=5, Access is denied
>>>>>>>> at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1128)
>>>>>>>> at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1071)
>>>>>>>> at org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:181)
>>>>>>>> at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:109)
>>>>>>>> at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:124)
>>>>>>>> at org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:174)
>>>>>>>> at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:67)
>>>>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:367)
>>>>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:331)
>>>>>>>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)
>>>>>>>> at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166)
>>>>>>>> at org.apache.spark.scheduler.Task.run(Task.scala:141)
>>>>>>>> at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
>>>>>>>> at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
>>>>>>>> at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
>>>>>>>> at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
>>>>>>>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
>>>>>>>> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>>>>>>> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>>>>>>> ... 1 more
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Jul 29, 2024 at 4:34 PM Sadha Chilukoori <
>>>>>>>> sage.quotient@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Mike,
>>>>>>>>>
>>>>>>>>> I'm not sure about the minimum requirements of a machine for
>>>>>>>>> running Spark. But to run some Pyspark scripts (and Jupiter notbebooks) on
>>>>>>>>> a local machine, I found the following steps are the easiest.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I installed Amazon corretto and updated the java_home variable as
>>>>>>>>> instructed here
>>>>>>>>> https://docs.aws.amazon.com/corretto/latest/corretto-11-ug/downloads-list.html
>>>>>>>>> (Any other java works too, I'm used to corretto from work).
>>>>>>>>>
>>>>>>>>> Then installed the Pyspark module using pip, which enabled me run
>>>>>>>>> Pyspark on my machine.
>>>>>>>>>
>>>>>>>>> -Sadha
>>>>>>>>>
>>>>>>>>> On Mon, Jul 29, 2024, 12:51 PM mike Jadoo <mi...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hello,
>>>>>>>>>>
>>>>>>>>>> I am trying to run Pyspark on my computer without success. I
>>>>>>>>>> follow several different directions from online sources and it appears that
>>>>>>>>>> I need to get a faster computer.
>>>>>>>>>>
>>>>>>>>>> I wanted to ask what are some recommendations for computer
>>>>>>>>>> specifications to run PySpark (Apache Spark).
>>>>>>>>>>
>>>>>>>>>> Any help would be greatly appreciated.
>>>>>>>>>>
>>>>>>>>>> Thank you,
>>>>>>>>>>
>>>>>>>>>> Mike
>>>>>>>>>>
>>>>>>>>>
>>>>>
>>>>> --
>>>>> Bjørn Jørgensen
>>>>> Vestre Aspehaug 4, 6010 Ålesund
>>>>> Norge
>>>>>
>>>>> +47 480 94 297
>>>>>
>>>>
>>>
>>> --
>>> Bjørn Jørgensen
>>> Vestre Aspehaug 4, 6010 Ålesund
>>> Norge
>>>
>>> +47 480 94 297
>>>
>>
Re: Question about installing Apache Spark [PySpark] computer requirements
Posted by mike Jadoo <mi...@gmail.com>.
Hello everyone,
Its been a long time and I wish everyone a happy New Year. I just wanted
to share I got a new laptop and tried installing Pyspark again and it
worked. However, I followed the instructions on setting
environmental variables from this youtube video:
https://www.youtube.com/watch?v=Qvke9S_ma7A&t=740s.
I am also using Jupyter lab to run Pyspark. I would love to show anyone
here via Zoom call about my experience.
Let me know what you think. Thanks for taking the time to help me out
before. This was a difficult experience and I appreciate all of you.
Thanks,
Mike
On Thu, Aug 1, 2024 at 9:28 PM Meena Rajani <me...@gmail.com> wrote:
> Did it work?
>
> Did you check the directory access? In your logs I saw
>
> java.io.IOException: Cannot run program "C:\Users\mikej\AppData\Local\Programs\Python\Python312": CreateProcess error=5, Access is denied
>
>
> Try chmod recursively on the directory and see
>
> Regards,
> Meena
>
> On Wed, Jul 31, 2024 at 6:13 AM Bjørn Jørgensen <bj...@gmail.com>
> wrote:
>
>> if you have anaconda installed you can type conda create -n py311
>> python=3.11 and conda activate py311
>>
>> ons. 31. juli 2024 kl. 01:33 skrev mike Jadoo <mi...@gmail.com>:
>>
>>> Hello Bjorn,
>>>
>>> I am installing Python version 3.11. However, I don't see anaconda
>>> python with version 3.11. Where can I find this?
>>>
>>> Thank you,
>>> Mike
>>>
>>> On Tue, Jul 30, 2024 at 3:37 AM Bjørn Jørgensen <
>>> bjornjorgensen@gmail.com> wrote:
>>>
>>>> you are using python version 3.12
>>>> spark 3.X does not have support for python version 3.12.
>>>> Remove python 3.12 and install python version 3.11
>>>>
>>>> tir. 30. juli 2024 kl. 07:51 skrev mike Jadoo <mi...@gmail.com>:
>>>>
>>>>> Hello Sadha,
>>>>>
>>>>> Thank you for the suggestion. I entered the environmental variable,
>>>>> however, I am still having trouble. here is a screenshot of the env vars
>>>>> and my code.
>>>>>
>>>>>
>>>>> [image: image.png]
>>>>>
>>>>>
>>>>> Here is my code:
>>>>>
>>>>>
>>>>>
>>>>> from pyspark.sql import SparkSession
>>>>>
>>>>> from pyspark import SparkContext
>>>>>
>>>>>
>>>>>
>>>>> spark = SparkSession.builder \
>>>>>
>>>>> .master("local[*]") \
>>>>>
>>>>> .appName("YourAppName") \
>>>>>
>>>>> .getOrCreate()
>>>>>
>>>>>
>>>>>
>>>>> # Create an RDD from a list of numbers
>>>>>
>>>>> numbers = [1, 2, 3, 4, 5]
>>>>>
>>>>> rdd = spark.sparkContext.parallelize(numbers)
>>>>>
>>>>>
>>>>>
>>>>> # Perform a transformation on the RDD
>>>>>
>>>>> squared_rdd = rdd.map(lambda x: x * x)
>>>>>
>>>>>
>>>>>
>>>>> # Collect the results into a list
>>>>>
>>>>> result = squared_rdd.collect()
>>>>>
>>>>>
>>>>>
>>>>> # Print the result
>>>>>
>>>>> print(result)
>>>>>
>>>>>
>>>>>
>>>>> error:
>>>>>
>>>>>
>>>>> *---------------------------------------------------------------------------*
>>>>>
>>>>> *Py4JJavaError* Traceback (most recent
>>>>> call last)
>>>>>
>>>>> Cell *In[5], line 13*
>>>>>
>>>>> 8 squared_rdd = rdd.map(*lambda* x: x * x)
>>>>>
>>>>> 10 # Persist the DataFrame in memory
>>>>>
>>>>> 11 #squared_rdd.persist(StorageLevel.MEMORY_ONLY)
>>>>>
>>>>> 12 # Collect the results into a list
>>>>>
>>>>> *---> 13* result = squared_rdd.collect()
>>>>>
>>>>> 15 # Print the result
>>>>>
>>>>> 16 print(result)
>>>>>
>>>>>
>>>>>
>>>>> File *~\anaconda3\Lib\site-packages\pyspark\rdd.py:1814*, in
>>>>> RDD.collect*(self)*
>>>>>
>>>>> 1812 *with* SCCallSiteSync(self.context):
>>>>>
>>>>> 1813 *assert* self.ctx._jvm *is* *not* *None*
>>>>>
>>>>> *-> 1814* sock_info = self.ctx._jvm.PythonRDD.collectAndServe(self
>>>>> ._jrdd.rdd())
>>>>>
>>>>> 1815 *return* list(_load_from_socket(sock_info, self.
>>>>> _jrdd_deserializer))
>>>>>
>>>>>
>>>>>
>>>>> File *~\anaconda3\Lib\site-packages\py4j\java_gateway.py:1322*, in
>>>>> JavaMember.__call__*(self, *args)*
>>>>>
>>>>> 1316 command = proto.CALL_COMMAND_NAME +\
>>>>>
>>>>> 1317 self.command_header +\
>>>>>
>>>>> 1318 args_command +\
>>>>>
>>>>> 1319 proto.END_COMMAND_PART
>>>>>
>>>>> 1321 answer = self.gateway_client.send_command(command)
>>>>>
>>>>> *-> 1322* return_value = get_return_value(
>>>>>
>>>>> 1323 answer, self.gateway_client, self.target_id, self.name)
>>>>>
>>>>> 1325 *for* temp_arg *in* temp_args:
>>>>>
>>>>> 1326 *if* hasattr(temp_arg, "_detach"):
>>>>>
>>>>>
>>>>>
>>>>> File
>>>>> *~\anaconda3\Lib\site-packages\pyspark\errors\exceptions\captured.py:169*,
>>>>> in capture_sql_exception.<locals>.deco*(*a, **kw)*
>>>>>
>>>>> 167 *def* deco(*a: Any, **kw: Any) -> Any:
>>>>>
>>>>> 168 *try*:
>>>>>
>>>>> *--> 169* *return* f(*a, **kw)
>>>>>
>>>>> 170 *except* Py4JJavaError *as* e:
>>>>>
>>>>> 171 converted = convert_exception(e.java_exception)
>>>>>
>>>>>
>>>>>
>>>>> File *~\anaconda3\Lib\site-packages\py4j\protocol.py:326*, in
>>>>> get_return_value*(answer, gateway_client, target_id, name)*
>>>>>
>>>>> 324 value = OUTPUT_CONVERTER[type](answer[2:], gateway_client)
>>>>>
>>>>> 325 *if* answer[1] == REFERENCE_TYPE:
>>>>>
>>>>> *--> 326* *raise* Py4JJavaError(
>>>>>
>>>>> 327 "An error occurred while calling *{0}{1}{2}*.*\n*".
>>>>>
>>>>> 328 format(target_id, ".", name), value)
>>>>>
>>>>> 329 *else*:
>>>>>
>>>>> 330 *raise* Py4JError(
>>>>>
>>>>> 331 "An error occurred while calling *{0}{1}{2}*. Trace:
>>>>> *\n{3}\n*".
>>>>>
>>>>> 332 format(target_id, ".", name, value))
>>>>>
>>>>>
>>>>>
>>>>> *Py4JJavaError*: An error occurred while calling
>>>>> z:org.apache.spark.api.python.PythonRDD.collectAndServe.
>>>>>
>>>>> : org.apache.spark.SparkException: Job aborted due to stage failure:
>>>>> Task 7 in stage 1.0 failed 1 times, most recent failure: Lost task 7.0 in
>>>>> stage 1.0 (TID 19) (mjadoo.myfiosgateway.com executor driver):
>>>>> org.apache.spark.SparkException: Python worker exited unexpectedly (crashed)
>>>>>
>>>>> at
>>>>> org.apache.spark.api.python.BasePythonRunner$ReaderIterator$$anonfun$1.applyOrElse(PythonRunner.scala:612)
>>>>>
>>>>> at
>>>>> org.apache.spark.api.python.BasePythonRunner$ReaderIterator$$anonfun$1.applyOrElse(PythonRunner.scala:594)
>>>>>
>>>>> at
>>>>> scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:38)
>>>>>
>>>>> at
>>>>> org.apache.spark.api.python.PythonRunner$$anon$3.read(PythonRunner.scala:789)
>>>>>
>>>>> at
>>>>> org.apache.spark.api.python.PythonRunner$$anon$3.read(PythonRunner.scala:766)
>>>>>
>>>>> at
>>>>> org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:525)
>>>>>
>>>>> at
>>>>> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
>>>>>
>>>>> at
>>>>> scala.collection.Iterator.foreach(Iterator.scala:943)
>>>>>
>>>>> at
>>>>> scala.collection.Iterator.foreach$(Iterator.scala:943)
>>>>>
>>>>> at
>>>>> org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28)
>>>>>
>>>>> at
>>>>> scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:62)
>>>>>
>>>>> at
>>>>> scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53)
>>>>>
>>>>> at
>>>>> scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:105)
>>>>>
>>>>> at
>>>>> scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:49)
>>>>>
>>>>> at scala.collection.TraversableOnce.to
>>>>> (TraversableOnce.scala:366)
>>>>>
>>>>> at scala.collection.TraversableOnce.to
>>>>> $(TraversableOnce.scala:364)
>>>>>
>>>>> at org.apache.spark.InterruptibleIterator.to
>>>>> (InterruptibleIterator.scala:28)
>>>>>
>>>>> at
>>>>> scala.collection.TraversableOnce.toBuffer(TraversableOnce.scala:358)
>>>>>
>>>>> at
>>>>> scala.collection.TraversableOnce.toBuffer$(TraversableOnce.scala:358)
>>>>>
>>>>> at
>>>>> org.apache.spark.InterruptibleIterator.toBuffer(InterruptibleIterator.scala:28)
>>>>>
>>>>> at
>>>>> scala.collection.TraversableOnce.toArray(TraversableOnce.scala:345)
>>>>>
>>>>> at
>>>>> scala.collection.TraversableOnce.toArray$(TraversableOnce.scala:339)
>>>>>
>>>>> at
>>>>> org.apache.spark.InterruptibleIterator.toArray(InterruptibleIterator.scala:28)
>>>>>
>>>>> at
>>>>> org.apache.spark.rdd.RDD.$anonfun$collect$2(RDD.scala:1049)
>>>>>
>>>>> at
>>>>> org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2438)
>>>>>
>>>>> at
>>>>> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)
>>>>>
>>>>> at
>>>>> org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166)
>>>>>
>>>>> at
>>>>> org.apache.spark.scheduler.Task.run(Task.scala:141)
>>>>>
>>>>> at
>>>>> org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
>>>>>
>>>>> at
>>>>> org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
>>>>>
>>>>> at
>>>>> org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
>>>>>
>>>>> at
>>>>> org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
>>>>>
>>>>> at
>>>>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
>>>>>
>>>>> at
>>>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>>>>
>>>>> at
>>>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>>>>
>>>>> at java.base/java.lang.Thread.run(Thread.java:829)
>>>>>
>>>>> Caused by: java.io.EOFException
>>>>>
>>>>> at
>>>>> java.base/java.io.DataInputStream.readInt(DataInputStream.java:397)
>>>>>
>>>>> at
>>>>> org.apache.spark.api.python.PythonRunner$$anon$3.read(PythonRunner.scala:774)
>>>>>
>>>>> ... 32 more
>>>>>
>>>>>
>>>>>
>>>>> Driver stacktrace:
>>>>>
>>>>> at
>>>>> org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2856)
>>>>>
>>>>> at
>>>>> org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2792)
>>>>>
>>>>> at
>>>>> org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2791)
>>>>>
>>>>> at
>>>>> scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
>>>>>
>>>>> at
>>>>> scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
>>>>>
>>>>> at
>>>>> scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
>>>>>
>>>>> at
>>>>> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2791)
>>>>>
>>>>> at
>>>>> org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1247)
>>>>>
>>>>> at
>>>>> org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1247)
>>>>>
>>>>> at scala.Option.foreach(Option.scala:407)
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Jul 29, 2024 at 8:19 PM Sadha Chilukoori <
>>>>> sage.quotient@gmail.com> wrote:
>>>>>
>>>>>> Hi Mike,
>>>>>>
>>>>>> This appears to be an access issue on Windows + Python. Can you try
>>>>>> setting up the PYTHON_PATH environment variable as described in this
>>>>>> stackoverflow post
>>>>>> https://stackoverflow.com/questions/60414394/createprocess-error-5-access-is-denied-pyspark
>>>>>>
>>>>>> - Sadha
>>>>>>
>>>>>> On Mon, Jul 29, 2024 at 3:39 PM mike Jadoo <mi...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks. I just downloaded the corretto but I got this error
>>>>>>> message, which was the same as before. [It was shared with me that this
>>>>>>> saying that I have limited resources, i think]
>>>>>>>
>>>>>>> ---------------------------------------------------------------------------Py4JJavaError Traceback (most recent call last)
>>>>>>> Cell In[3], line 13 8 squared_rdd = rdd.map(lambda x: x * x) 10 # Persist the DataFrame in memory 11 #squared_rdd.persist(StorageLevel.MEMORY_ONLY) 12 # Collect the results into a list---> 13 result = squared_rdd.collect() 15 # Print the result 16 print(result)
>>>>>>>
>>>>>>> File C:\spark\spark-3.5.1-bin-hadoop3\python\lib\pyspark.zip\pyspark\rdd.py:1833, in RDD.collect(self) 1831 with SCCallSiteSync(self.context): 1832 assert self.ctx._jvm is not None-> 1833 sock_info = self.ctx._jvm.PythonRDD.collectAndServe(self._jrdd.rdd()) 1834 return list(_load_from_socket(sock_info, self._jrdd_deserializer))
>>>>>>>
>>>>>>> File ~\anaconda3\Lib\site-packages\py4j\java_gateway.py:1322, in JavaMember.__call__(self, *args) 1316 command = proto.CALL_COMMAND_NAME +\ 1317 self.command_header +\ 1318 args_command +\ 1319 proto.END_COMMAND_PART 1321 answer = self.gateway_client.send_command(command)-> 1322 return_value = get_return_value( 1323 answer, self.gateway_client, self.target_id, self.name) 1325 for temp_arg in temp_args: 1326 if hasattr(temp_arg, "_detach"):
>>>>>>>
>>>>>>> File C:\spark\spark-3.5.1-bin-hadoop3\python\lib\pyspark.zip\pyspark\errors\exceptions\captured.py:179, in capture_sql_exception.<locals>.deco(*a, **kw) 177 def deco(*a: Any, **kw: Any) -> Any: 178 try:--> 179 return f(*a, **kw) 180 except Py4JJavaError as e: 181 converted = convert_exception(e.java_exception)
>>>>>>>
>>>>>>> File ~\anaconda3\Lib\site-packages\py4j\protocol.py:326, in get_return_value(answer, gateway_client, target_id, name) 324 value = OUTPUT_CONVERTER[type](answer[2:], gateway_client) 325 if answer[1] == REFERENCE_TYPE:--> 326 raise Py4JJavaError( 327 "An error occurred while calling {0}{1}{2}.\n". 328 format(target_id, ".", name), value) 329 else: 330 raise Py4JError( 331 "An error occurred while calling {0}{1}{2}. Trace:\n{3}\n". 332 format(target_id, ".", name, value))
>>>>>>> Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
>>>>>>> : org.apache.spark.SparkException: Job aborted due to stage failure: Task 7 in stage 0.0 failed 1 times, most recent failure: Lost task 7.0 in stage 0.0 (TID 7) (mjadoo.myfiosgateway.com executor driver): java.io.IOException: Cannot run program "C:\Users\mikej\AppData\Local\Programs\Python\Python312": CreateProcess error=5, Access is denied
>>>>>>> at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1128)
>>>>>>> at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1071)
>>>>>>> at org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:181)
>>>>>>> at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:109)
>>>>>>> at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:124)
>>>>>>> at org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:174)
>>>>>>> at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:67)
>>>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:367)
>>>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:331)
>>>>>>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)
>>>>>>> at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166)
>>>>>>> at org.apache.spark.scheduler.Task.run(Task.scala:141)
>>>>>>> at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
>>>>>>> at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
>>>>>>> at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
>>>>>>> at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
>>>>>>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
>>>>>>> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>>>>>> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>>>>>> at java.base/java.lang.Thread.run(Thread.java:829)
>>>>>>> Caused by: java.io.IOException: CreateProcess error=5, Access is denied
>>>>>>> at java.base/java.lang.ProcessImpl.create(Native Method)
>>>>>>> at java.base/java.lang.ProcessImpl.<init>(ProcessImpl.java:492)
>>>>>>> at java.base/java.lang.ProcessImpl.start(ProcessImpl.java:153)
>>>>>>> at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1107)
>>>>>>> ... 19 more
>>>>>>>
>>>>>>> Driver stacktrace:
>>>>>>> at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2856)
>>>>>>> at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2792)
>>>>>>> at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2791)
>>>>>>> at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
>>>>>>> at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
>>>>>>> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
>>>>>>> at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2791)
>>>>>>> at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1247)
>>>>>>> at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1247)
>>>>>>> at scala.Option.foreach(Option.scala:407)
>>>>>>> at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1247)
>>>>>>> at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:3060)
>>>>>>> at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2994)
>>>>>>> at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2983)
>>>>>>> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
>>>>>>> at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:989)
>>>>>>> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2398)
>>>>>>> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2419)
>>>>>>> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2438)
>>>>>>> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2463)
>>>>>>> at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1049)
>>>>>>> at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
>>>>>>> at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
>>>>>>> at org.apache.spark.rdd.RDD.withScope(RDD.scala:410)
>>>>>>> at org.apache.spark.rdd.RDD.collect(RDD.scala:1048)
>>>>>>> at org.apache.spark.api.python.PythonRDD$.collectAndServe(PythonRDD.scala:195)
>>>>>>> at org.apache.spark.api.python.PythonRDD.collectAndServe(PythonRDD.scala)
>>>>>>> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>>> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>>>>> at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>>>>>>> at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
>>>>>>> at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
>>>>>>> at py4j.Gateway.invoke(Gateway.java:282)
>>>>>>> at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
>>>>>>> at py4j.commands.CallCommand.execute(CallCommand.java:79)
>>>>>>> at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
>>>>>>> at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
>>>>>>> at java.base/java.lang.Thread.run(Thread.java:829)
>>>>>>> Caused by: java.io.IOException: Cannot run program "C:\Users\mikej\AppData\Local\Programs\Python\Python312": CreateProcess error=5, Access is denied
>>>>>>> at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1128)
>>>>>>> at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1071)
>>>>>>> at org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:181)
>>>>>>> at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:109)
>>>>>>> at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:124)
>>>>>>> at org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:174)
>>>>>>> at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:67)
>>>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:367)
>>>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:331)
>>>>>>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)
>>>>>>> at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166)
>>>>>>> at org.apache.spark.scheduler.Task.run(Task.scala:141)
>>>>>>> at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
>>>>>>> at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
>>>>>>> at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
>>>>>>> at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
>>>>>>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
>>>>>>> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>>>>>> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>>>>>> ... 1 more
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Jul 29, 2024 at 4:34 PM Sadha Chilukoori <
>>>>>>> sage.quotient@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi Mike,
>>>>>>>>
>>>>>>>> I'm not sure about the minimum requirements of a machine for
>>>>>>>> running Spark. But to run some Pyspark scripts (and Jupiter notbebooks) on
>>>>>>>> a local machine, I found the following steps are the easiest.
>>>>>>>>
>>>>>>>>
>>>>>>>> I installed Amazon corretto and updated the java_home variable as
>>>>>>>> instructed here
>>>>>>>> https://docs.aws.amazon.com/corretto/latest/corretto-11-ug/downloads-list.html
>>>>>>>> (Any other java works too, I'm used to corretto from work).
>>>>>>>>
>>>>>>>> Then installed the Pyspark module using pip, which enabled me run
>>>>>>>> Pyspark on my machine.
>>>>>>>>
>>>>>>>> -Sadha
>>>>>>>>
>>>>>>>> On Mon, Jul 29, 2024, 12:51 PM mike Jadoo <mi...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> I am trying to run Pyspark on my computer without success. I
>>>>>>>>> follow several different directions from online sources and it appears that
>>>>>>>>> I need to get a faster computer.
>>>>>>>>>
>>>>>>>>> I wanted to ask what are some recommendations for computer
>>>>>>>>> specifications to run PySpark (Apache Spark).
>>>>>>>>>
>>>>>>>>> Any help would be greatly appreciated.
>>>>>>>>>
>>>>>>>>> Thank you,
>>>>>>>>>
>>>>>>>>> Mike
>>>>>>>>>
>>>>>>>>
>>>>
>>>> --
>>>> Bjørn Jørgensen
>>>> Vestre Aspehaug 4, 6010 Ålesund
>>>> Norge
>>>>
>>>> +47 480 94 297
>>>>
>>>
>>
>> --
>> Bjørn Jørgensen
>> Vestre Aspehaug 4, 6010 Ålesund
>> Norge
>>
>> +47 480 94 297
>>
>