You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "HyukjinKwon (via GitHub)" <gi...@apache.org> on 2024/04/04 03:17:23 UTC
[PR] [DO-NOT-MERGE][SPARK-47725][INFRA] Set up the CI for pyspark-connect package [spark]
HyukjinKwon opened a new pull request, #45870:
URL: https://github.com/apache/spark/pull/45870
### What changes were proposed in this pull request?
TBD
### Why are the changes needed?
TBD
### Does this PR introduce _any_ user-facing change?
TBD
### How was this patch tested?
TBD
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [DO-NOT-MERGE][SPARK-47725][INFRA] Set up the CI for pyspark-connect package [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #45870:
URL: https://github.com/apache/spark/pull/45870#issuecomment-2036206491
https://github.com/HyukjinKwon/spark/actions/runs/8549624298
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [DO-NOT-MERGE][SPARK-47725][INFRA] Set up the CI for pyspark-connect package [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #45870:
URL: https://github.com/apache/spark/pull/45870#discussion_r1554825945
##########
python/pyspark/sql/tests/connect/test_connect_session.py:
##########
@@ -248,7 +250,7 @@ def setUp(self) -> None:
.config("integer", 1)
.config("boolean", False)
.appName(self.__class__.__name__)
- .remote("local[4]")
+ .remote(os.environ.get("SPARK_CONNECT_TESTING_REMOTE", "local[4]"))
Review Comment:
```suggestion
.remote("local[4]")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [DO-NOT-MERGE][SPARK-47725][INFRA] Set up the CI for pyspark-connect package [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #45870:
URL: https://github.com/apache/spark/pull/45870#discussion_r1554957126
##########
python/pyspark/sql/tests/test_catalog.py:
##########
@@ -73,6 +73,9 @@ def test_list_tables(self):
spark.sql("CREATE DATABASE some_db")
with self.table("tab1", "some_db.tab2", "tab3_via_catalog"):
with self.tempView("temp_tab"):
+ # Make sure tables are all dropped.
Review Comment:
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [DO-NOT-MERGE][SPARK-47725][INFRA] Set up the CI for pyspark-connect package [spark]
Posted by "09306677806 (via GitHub)" <gi...@apache.org>.
09306677806 commented on PR #45870:
URL: https://github.com/apache/spark/pull/45870#issuecomment-2041245617
Withdrawing only my physical body in my branch and my bank, my physical
body and my ego, if it is through the real system, it will be ok, and all
the fingerprints should be physically real, and wash your hands with a
strong cleaner before handing over to the main bank.
ShahrzadMahro
در تاریخ پنجشنبه ۴ آوریل ۲۰۲۴، ۰۶:۴۹ Hyukjin Kwon ***@***.***>
نوشت:
> What changes were proposed in this pull request?
>
> TBD
> Why are the changes needed?
>
> TBD
> Does this PR introduce *any* user-facing change?
>
> TBD
> How was this patch tested?
>
> TBD
> Was this patch authored or co-authored using generative AI tooling?
>
> No.
> ------------------------------
> You can view, comment on, or merge this pull request online at:
>
> https://github.com/apache/spark/pull/45870
> Commit Summary
>
> - 5b077f4
> <https://github.com/apache/spark/pull/45870/commits/5b077f451566502eeee9db1c181ce4b2fada3424>
> test
>
> File Changes
>
> (2 files <https://github.com/apache/spark/pull/45870/files>)
>
> - *M* .github/workflows/build_main.yml
> <https://github.com/apache/spark/pull/45870/files#diff-a57a6c53c37bb148a07f1122f0d4c8cf4b746509bfcb4bcf6c95e0764aaea87d>
> (2)
> - *A* .github/workflows/build_python_connect.yml
> <https://github.com/apache/spark/pull/45870/files#diff-cab8d808b530042ed863202e2d9d2419b942fa9fe18edc725df2a4f3206cf9b0>
> (101)
>
> Patch Links:
>
> - https://github.com/apache/spark/pull/45870.patch
> - https://github.com/apache/spark/pull/45870.diff
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/spark/pull/45870>, or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ANEZZRH3KD2UMEVWNWE6O33Y3TBETAVCNFSM6AAAAABFWOLGN6VHI2DSMVQWIX3LMV43ASLTON2WKOZSGIZDIMZWGM4TSNA>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [DO-NOT-MERGE][SPARK-47725][INFRA] Set up the CI for pyspark-connect package [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #45870:
URL: https://github.com/apache/spark/pull/45870#issuecomment-2036100626
https://github.com/HyukjinKwon/spark/actions/runs/8548784419
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [DO-NOT-MERGE][SPARK-47725][INFRA] Set up the CI for pyspark-connect package [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #45870:
URL: https://github.com/apache/spark/pull/45870#discussion_r1554802432
##########
.github/workflows/build_python_connect.yml:
##########
@@ -0,0 +1,96 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: Build / Spark Connect Python-only (master, Python 3.11)
+
+on:
+ push:
+ branches:
+ - '**'
+
+jobs:
+ # Build: build Spark and run the tests for specified modules using SBT
+ build:
+ name: "Build modules: pyspark-connect"
+ runs-on: ubuntu-latest
+ timeout-minutes: 300
+ steps:
+ - name: Checkout Spark repository
+ uses: actions/checkout@v4
+ - name: Cache Scala, SBT and Maven
+ uses: actions/cache@v4
+ with:
+ path: |
+ build/apache-maven-*
+ build/scala-*
+ build/*.jar
+ ~/.sbt
+ key: build-spark-connect-python-only-${{ hashFiles('**/pom.xml', 'project/build.properties', 'build/mvn', 'build/sbt', 'build/sbt-launch-lib.bash', 'build/spark-build-info') }}
+ restore-keys: |
+ build-spark-connect-python-only-
+ - name: Cache Coursier local repository
+ uses: actions/cache@v4
+ with:
+ path: ~/.cache/coursier
+ key: coursier-build-spark-connect-python-only-${{ hashFiles('**/pom.xml') }}
+ restore-keys: |
+ coursier-build-spark-connect-python-only-
+ - name: Install Java 17
+ uses: actions/setup-java@v4
+ with:
+ distribution: zulu
+ java-version: 17
+ - name: Install Python 3.11
+ uses: actions/setup-python@v5
+ with:
+ python-version: '3.11'
+ architecture: x64
+ - name: Build Spark
+ run: |
+ ./build/sbt -Phive test:package
+ - name: Install pure Python package (pyspark-connect)
+ env:
+ SPARK_TESTING: 1
+ run: |
+ cd python
+ python packaging/connect/setup.py sdist
+ cd dist
+ pip install pyspark-connect-*.tar.gz
+ - name: Run tests
+ env:
+ SPARK_CONNECT_TESTING_REMOTE: sc://localhost
Review Comment:
```suggestion
SPARK_CONNECT_TESTING_REMOTE: sc://localhost
SPARK_TESTING: 1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [DO-NOT-MERGE][SPARK-47725][INFRA] Set up the CI for pyspark-connect package [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #45870:
URL: https://github.com/apache/spark/pull/45870#issuecomment-2036151110
https://github.com/HyukjinKwon/spark/actions/runs/8549163391
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [DO-NOT-MERGE][SPARK-47725][INFRA] Set up the CI for pyspark-connect package [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #45870:
URL: https://github.com/apache/spark/pull/45870#discussion_r1554729072
##########
python/pyspark/core/context.py:
##########
@@ -145,7 +145,7 @@ class SparkContext:
Examples
--------
- >>> from pyspark.core.context import SparkContext
+ >>> from pyspark.context import SparkContext
Review Comment:
```suggestion
>>> from pyspark.core.context import SparkContext
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [DO-NOT-MERGE][SPARK-47725][INFRA] Set up the CI for pyspark-connect package [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #45870:
URL: https://github.com/apache/spark/pull/45870#discussion_r1554802133
##########
.github/workflows/build_python_connect.yml:
##########
@@ -0,0 +1,96 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: Build / Spark Connect Python-only (master, Python 3.11)
+
+on:
+ push:
+ branches:
+ - '**'
+
+jobs:
+ # Build: build Spark and run the tests for specified modules using maven
Review Comment:
```suggestion
# Build: build Spark and run the tests for specified modules using SBT
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [DO-NOT-MERGE][SPARK-47725][INFRA] Set up the CI for pyspark-connect package [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #45870:
URL: https://github.com/apache/spark/pull/45870#discussion_r1554957268
##########
python/pyspark/sql/tests/test_catalog.py:
##########
@@ -73,6 +73,9 @@ def test_list_tables(self):
spark.sql("CREATE DATABASE some_db")
with self.table("tab1", "some_db.tab2", "tab3_via_catalog"):
with self.tempView("temp_tab"):
+ # Make sure tables are all dropped.
+ for t in spark.catalog.listTables():
+ self.spark.sql("DROP TABLE IF EXISTS %s" % t.name)
Review Comment:
```suggestion
```
##########
python/pyspark/sql/tests/test_catalog.py:
##########
@@ -73,6 +73,9 @@ def test_list_tables(self):
spark.sql("CREATE DATABASE some_db")
with self.table("tab1", "some_db.tab2", "tab3_via_catalog"):
with self.tempView("temp_tab"):
+ # Make sure tables are all dropped.
+ for t in spark.catalog.listTables():
Review Comment:
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [DO-NOT-MERGE][SPARK-47725][INFRA] Set up the CI for pyspark-connect package [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #45870:
URL: https://github.com/apache/spark/pull/45870#issuecomment-2036181138
https://github.com/HyukjinKwon/spark/actions/runs/8549387447
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [DO-NOT-MERGE][SPARK-47725][INFRA] Set up the CI for pyspark-connect package [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #45870:
URL: https://github.com/apache/spark/pull/45870#discussion_r1554802432
##########
.github/workflows/build_python_connect.yml:
##########
@@ -0,0 +1,96 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: Build / Spark Connect Python-only (master, Python 3.11)
+
+on:
+ push:
+ branches:
+ - '**'
+
+jobs:
+ # Build: build Spark and run the tests for specified modules using SBT
+ build:
+ name: "Build modules: pyspark-connect"
+ runs-on: ubuntu-latest
+ timeout-minutes: 300
+ steps:
+ - name: Checkout Spark repository
+ uses: actions/checkout@v4
+ - name: Cache Scala, SBT and Maven
+ uses: actions/cache@v4
+ with:
+ path: |
+ build/apache-maven-*
+ build/scala-*
+ build/*.jar
+ ~/.sbt
+ key: build-spark-connect-python-only-${{ hashFiles('**/pom.xml', 'project/build.properties', 'build/mvn', 'build/sbt', 'build/sbt-launch-lib.bash', 'build/spark-build-info') }}
+ restore-keys: |
+ build-spark-connect-python-only-
+ - name: Cache Coursier local repository
+ uses: actions/cache@v4
+ with:
+ path: ~/.cache/coursier
+ key: coursier-build-spark-connect-python-only-${{ hashFiles('**/pom.xml') }}
+ restore-keys: |
+ coursier-build-spark-connect-python-only-
+ - name: Install Java 17
+ uses: actions/setup-java@v4
+ with:
+ distribution: zulu
+ java-version: 17
+ - name: Install Python 3.11
+ uses: actions/setup-python@v5
+ with:
+ python-version: '3.11'
+ architecture: x64
+ - name: Build Spark
+ run: |
+ ./build/sbt -Phive test:package
+ - name: Install pure Python package (pyspark-connect)
+ env:
+ SPARK_TESTING: 1
+ run: |
+ cd python
+ python packaging/connect/setup.py sdist
+ cd dist
+ pip install pyspark-connect-*.tar.gz
+ - name: Run tests
+ env:
+ SPARK_CONNECT_TESTING_REMOTE: sc://localhost
Review Comment:
```suggestion
SPARK_CONNECT_TESTING_REMOTE: sc://localhost
SPARK_TESTING: 1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [DO-NOT-MERGE][SPARK-47725][INFRA] Set up the CI for pyspark-connect package [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #45870:
URL: https://github.com/apache/spark/pull/45870#issuecomment-2036103963
https://github.com/HyukjinKwon/spark/actions/runs/8548832973/job/23423174130
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [DO-NOT-MERGE][SPARK-47725][INFRA] Set up the CI for pyspark-connect package [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #45870:
URL: https://github.com/apache/spark/pull/45870#discussion_r1554802133
##########
.github/workflows/build_python_connect.yml:
##########
@@ -0,0 +1,96 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: Build / Spark Connect Python-only (master, Python 3.11)
+
+on:
+ push:
+ branches:
+ - '**'
+
+jobs:
+ # Build: build Spark and run the tests for specified modules using maven
Review Comment:
```suggestion
# Build: build Spark and run the tests for specified modules using SBT
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [DO-NOT-MERGE][SPARK-47725][INFRA] Set up the CI for pyspark-connect package [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #45870:
URL: https://github.com/apache/spark/pull/45870#discussion_r1554957196
##########
python/pyspark/sql/tests/test_catalog.py:
##########
@@ -73,6 +73,9 @@ def test_list_tables(self):
spark.sql("CREATE DATABASE some_db")
with self.table("tab1", "some_db.tab2", "tab3_via_catalog"):
with self.tempView("temp_tab"):
+ # Make sure tables are all dropped.
+ for t in spark.catalog.listTables():
Review Comment:
```suggestion
```
##########
python/pyspark/sql/tests/test_catalog.py:
##########
@@ -73,6 +73,9 @@ def test_list_tables(self):
spark.sql("CREATE DATABASE some_db")
with self.table("tab1", "some_db.tab2", "tab3_via_catalog"):
with self.tempView("temp_tab"):
+ # Make sure tables are all dropped.
Review Comment:
```suggestion
```
##########
python/pyspark/sql/tests/test_catalog.py:
##########
@@ -73,6 +73,9 @@ def test_list_tables(self):
spark.sql("CREATE DATABASE some_db")
with self.table("tab1", "some_db.tab2", "tab3_via_catalog"):
with self.tempView("temp_tab"):
+ # Make sure tables are all dropped.
+ for t in spark.catalog.listTables():
+ self.spark.sql("DROP TABLE IF EXISTS %s" % t.name)
Review Comment:
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47725][INFRA] Set up the CI for pyspark-connect package [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #45870:
URL: https://github.com/apache/spark/pull/45870#issuecomment-2043203281
Let me merge this to start from this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [DO-NOT-MERGE][SPARK-47725][INFRA] Set up the CI for pyspark-connect package [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #45870:
URL: https://github.com/apache/spark/pull/45870#issuecomment-2036147769
https://github.com/HyukjinKwon/spark/actions/runs/8549142728
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47725][INFRA] Set up the CI for pyspark-connect package [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #45870: [SPARK-47725][INFRA] Set up the CI for pyspark-connect package
URL: https://github.com/apache/spark/pull/45870
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [DO-NOT-MERGE][SPARK-47725][INFRA] Set up the CI for pyspark-connect package [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #45870:
URL: https://github.com/apache/spark/pull/45870#issuecomment-2036105177
https://github.com/HyukjinKwon/spark/actions/runs/8548849051
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [DO-NOT-MERGE][SPARK-47725][INFRA] Set up the CI for pyspark-connect package [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #45870:
URL: https://github.com/apache/spark/pull/45870#discussion_r1554729072
##########
python/pyspark/core/context.py:
##########
@@ -145,7 +145,7 @@ class SparkContext:
Examples
--------
- >>> from pyspark.core.context import SparkContext
+ >>> from pyspark.context import SparkContext
Review Comment:
```suggestion
>>> from pyspark.core.context import SparkContext
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47725][INFRA] Set up the CI for pyspark-connect package [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #45870:
URL: https://github.com/apache/spark/pull/45870#issuecomment-2042534288
Test results: https://github.com/HyukjinKwon/spark/actions/runs/8598881063
Should be ready to go, cc @zhengruifeng @dongjoon-hyun @ueshin
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [DO-NOT-MERGE][SPARK-47725][INFRA] Set up the CI for pyspark-connect package [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #45870:
URL: https://github.com/apache/spark/pull/45870#discussion_r1554825945
##########
python/pyspark/sql/tests/connect/test_connect_session.py:
##########
@@ -248,7 +250,7 @@ def setUp(self) -> None:
.config("integer", 1)
.config("boolean", False)
.appName(self.__class__.__name__)
- .remote("local[4]")
+ .remote(os.environ.get("SPARK_CONNECT_TESTING_REMOTE", "local[4]"))
Review Comment:
```suggestion
.remote("local[4]")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org