You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "zhengruifeng (via GitHub)" <gi...@apache.org> on 2023/03/23 08:34:37 UTC
[GitHub] [spark] zhengruifeng opened a new pull request, #40535: [SPARK-42907][CONNECT][PYTHON] Implement Avro functions
zhengruifeng opened a new pull request, #40535:
URL: https://github.com/apache/spark/pull/40535
### What changes were proposed in this pull request?
Implement Avro functions
### Why are the changes needed?
For function parity
### Does this PR introduce _any_ user-facing change?
yes, new APIs
### How was this patch tested?
added doctest and manually check
```
(spark_dev) ➜ spark git:(connect_avro_functions) ✗ bin/pyspark --remote "local[*]" --jars connector/avro/target/scala-2.12/spark-avro_2.12-3.5.0-SNAPSHOT.jar
Python 3.9.16 (main, Mar 8 2023, 04:29:24)
Type 'copyright', 'credits' or 'license' for more information
IPython 8.11.0 -- An enhanced Interactive Python. Type '?' for help.
23/03/23 16:28:50 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/__ / .__/\_,_/_/ /_/\_\ version 3.5.0.dev0
/_/
Using Python version 3.9.16 (main, Mar 8 2023 04:29:24)
Client connected to the Spark Connect server at localhost
SparkSession available as 'spark'.
In [1]: >>> from pyspark.sql import Row
...: >>> from pyspark.sql.avro.functions import from_avro, to_avro
...: >>> data = [(1, Row(age=2, name='Alice'))]
...: >>> df = spark.createDataFrame(data, ("key", "value"))
...: >>> avroDf = df.select(to_avro(df.value).alias("avro"))
In [2]: avroDf.collect()
Out[2]: [Row(avro=bytearray(b'\x00\x00\x04\x00\nAlice'))]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #40535: [SPARK-42907][CONNECT][PYTHON] Implement Avro functions
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #40535:
URL: https://github.com/apache/spark/pull/40535#discussion_r1147474920
##########
python/pyspark/sql/connect/avro/functions.py:
##########
@@ -0,0 +1,108 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+A collections of builtin avro functions
+"""
+
+from pyspark.sql.connect.utils import check_dependencies
+
+check_dependencies(__name__)
+
+from typing import Dict, Optional, TYPE_CHECKING
+
+from pyspark.sql.avro import functions as PyAvroFunctions
+
+from pyspark.sql.connect.column import Column
+from pyspark.sql.connect.functions import _invoke_function, _to_col, _options_to_col, lit
+
+if TYPE_CHECKING:
+ from pyspark.sql.connect._typing import ColumnOrName
+
+
+def from_avro(
+ data: "ColumnOrName", jsonFormatSchema: str, options: Optional[Dict[str, str]] = None
+) -> Column:
+ if options is None:
+ return _invoke_function("from_avro", _to_col(data), lit(jsonFormatSchema))
+ else:
+ return _invoke_function(
+ "from_avro", _to_col(data), lit(jsonFormatSchema), _options_to_col(options)
+ )
+
+
+from_avro.__doc__ = PyAvroFunctions.from_avro.__doc__
+
+
+def to_avro(data: "ColumnOrName", jsonFormatSchema: str = "") -> Column:
+ if jsonFormatSchema == "":
+ return _invoke_function("to_avro", _to_col(data))
+ else:
+ return _invoke_function("to_avro", _to_col(data), lit(jsonFormatSchema))
+
+
+to_avro.__doc__ = PyAvroFunctions.to_avro.__doc__
+
+
+def _test() -> None:
+ import os
+ import sys
+ from pyspark.testing.utils import search_jar
+
+ avro_jar = search_jar("connector/avro", "spark-avro", "spark-avro")
+
+ if avro_jar is None:
+ print(
+ "Skipping all Avro Python tests as the optional Avro project was "
+ "not compiled into a JAR. To run these tests, "
+ "you need to build Spark with 'build/sbt -Pavro package' or "
+ "'build/mvn -Pavro package' before running this test."
+ )
+ sys.exit(0)
+ else:
+ existing_args = os.environ.get("PYSPARK_SUBMIT_ARGS", "pyspark-shell")
+ jars_args = "--jars %s" % avro_jar
+ os.environ["PYSPARK_SUBMIT_ARGS"] = " ".join([jars_args, existing_args])
Review Comment:
Hm, yeah seems correct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a diff in pull request #40535: [SPARK-42907][CONNECT][PYTHON] Implement Avro functions
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #40535:
URL: https://github.com/apache/spark/pull/40535#discussion_r1146040349
##########
python/pyspark/sql/connect/avro/functions.py:
##########
@@ -0,0 +1,108 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+A collections of builtin avro functions
+"""
+
+from pyspark.sql.connect.utils import check_dependencies
+
+check_dependencies(__name__)
+
+from typing import Dict, Optional, TYPE_CHECKING
+
+from pyspark.sql.avro import functions as PyAvroFunctions
+
+from pyspark.sql.connect.column import Column
+from pyspark.sql.connect.functions import _invoke_function, _to_col, _options_to_col, lit
+
+if TYPE_CHECKING:
+ from pyspark.sql.connect._typing import ColumnOrName
+
+
+def from_avro(
+ data: "ColumnOrName", jsonFormatSchema: str, options: Optional[Dict[str, str]] = None
+) -> Column:
+ if options is None:
+ return _invoke_function("from_avro", _to_col(data), lit(jsonFormatSchema))
+ else:
+ return _invoke_function(
+ "from_avro", _to_col(data), lit(jsonFormatSchema), _options_to_col(options)
+ )
+
+
+from_avro.__doc__ = PyAvroFunctions.from_avro.__doc__
+
+
+def to_avro(data: "ColumnOrName", jsonFormatSchema: str = "") -> Column:
+ if jsonFormatSchema == "":
+ return _invoke_function("to_avro", _to_col(data))
+ else:
+ return _invoke_function("to_avro", _to_col(data), lit(jsonFormatSchema))
+
+
+to_avro.__doc__ = PyAvroFunctions.to_avro.__doc__
+
+
+def _test() -> None:
+ import os
+ import sys
+ from pyspark.testing.utils import search_jar
+
+ avro_jar = search_jar("connector/avro", "spark-avro", "spark-avro")
+
+ if avro_jar is None:
+ print(
+ "Skipping all Avro Python tests as the optional Avro project was "
+ "not compiled into a JAR. To run these tests, "
+ "you need to build Spark with 'build/sbt -Pavro package' or "
+ "'build/mvn -Pavro package' before running this test."
+ )
+ sys.exit(0)
+ else:
+ existing_args = os.environ.get("PYSPARK_SUBMIT_ARGS", "pyspark-shell")
+ jars_args = "--jars %s" % avro_jar
+ os.environ["PYSPARK_SUBMIT_ARGS"] = " ".join([jars_args, existing_args])
Review Comment:
@HyukjinKwon is it the proper way to add jars in Python Client?
this doctest pass in my local, but it seems the jar is not found in CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a diff in pull request #40535: [SPARK-42907][CONNECT][PYTHON] Implement Avro functions
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #40535:
URL: https://github.com/apache/spark/pull/40535#discussion_r1150299249
##########
assembly/pom.xml:
##########
@@ -160,6 +160,12 @@
<artifactId>spark-connect_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>
+ <dependency>
Review Comment:
I found that I have to add `avro` as a provided dependency in profile `connect`.
otherwise, after `build/sbt -Pconnect clean package` it can not correctly find the avro class.
cc @HyukjinKwon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a diff in pull request #40535: [SPARK-42907][CONNECT][PYTHON] Implement Avro functions
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #40535:
URL: https://github.com/apache/spark/pull/40535#discussion_r1146040349
##########
python/pyspark/sql/connect/avro/functions.py:
##########
@@ -0,0 +1,108 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+A collections of builtin avro functions
+"""
+
+from pyspark.sql.connect.utils import check_dependencies
+
+check_dependencies(__name__)
+
+from typing import Dict, Optional, TYPE_CHECKING
+
+from pyspark.sql.avro import functions as PyAvroFunctions
+
+from pyspark.sql.connect.column import Column
+from pyspark.sql.connect.functions import _invoke_function, _to_col, _options_to_col, lit
+
+if TYPE_CHECKING:
+ from pyspark.sql.connect._typing import ColumnOrName
+
+
+def from_avro(
+ data: "ColumnOrName", jsonFormatSchema: str, options: Optional[Dict[str, str]] = None
+) -> Column:
+ if options is None:
+ return _invoke_function("from_avro", _to_col(data), lit(jsonFormatSchema))
+ else:
+ return _invoke_function(
+ "from_avro", _to_col(data), lit(jsonFormatSchema), _options_to_col(options)
+ )
+
+
+from_avro.__doc__ = PyAvroFunctions.from_avro.__doc__
+
+
+def to_avro(data: "ColumnOrName", jsonFormatSchema: str = "") -> Column:
+ if jsonFormatSchema == "":
+ return _invoke_function("to_avro", _to_col(data))
+ else:
+ return _invoke_function("to_avro", _to_col(data), lit(jsonFormatSchema))
+
+
+to_avro.__doc__ = PyAvroFunctions.to_avro.__doc__
+
+
+def _test() -> None:
+ import os
+ import sys
+ from pyspark.testing.utils import search_jar
+
+ avro_jar = search_jar("connector/avro", "spark-avro", "spark-avro")
+
+ if avro_jar is None:
+ print(
+ "Skipping all Avro Python tests as the optional Avro project was "
+ "not compiled into a JAR. To run these tests, "
+ "you need to build Spark with 'build/sbt -Pavro package' or "
+ "'build/mvn -Pavro package' before running this test."
+ )
+ sys.exit(0)
+ else:
+ existing_args = os.environ.get("PYSPARK_SUBMIT_ARGS", "pyspark-shell")
+ jars_args = "--jars %s" % avro_jar
+ os.environ["PYSPARK_SUBMIT_ARGS"] = " ".join([jars_args, existing_args])
Review Comment:
@HyukjinKwon is it the proper way to add jars in Python Client?
this doctest pass in my local, but it seems the jar is not correctly added in CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #40535: [SPARK-42907][CONNECT][PYTHON] Implement Avro functions
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #40535:
URL: https://github.com/apache/spark/pull/40535#discussion_r1147472671
##########
dev/sparktestsupport/modules.py:
##########
@@ -747,6 +747,7 @@ def __hash__(self):
"pyspark.sql.connect.readwriter",
Review Comment:
oh sorry. Yes, you're correct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #40535: [SPARK-42907][CONNECT][PYTHON] Implement Avro functions
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #40535:
URL: https://github.com/apache/spark/pull/40535#issuecomment-1487796203
It has a conflict with branch-3.4. mind creating a backport please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #40535: [SPARK-42907][CONNECT][PYTHON] Implement Avro functions
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #40535:
URL: https://github.com/apache/spark/pull/40535#issuecomment-1487842406
Sure that's fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on pull request #40535: [SPARK-42907][CONNECT][PYTHON] Implement Avro functions
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #40535:
URL: https://github.com/apache/spark/pull/40535#issuecomment-1487819994
@HyukjinKwon Thanks for reviews. What about only adding it to master? since it changes the dependency, so I want to be a bit conservative :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #40535: [SPARK-42907][CONNECT][PYTHON] Implement Avro functions
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #40535: [SPARK-42907][CONNECT][PYTHON] Implement Avro functions
URL: https://github.com/apache/spark/pull/40535
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #40535: [SPARK-42907][CONNECT][PYTHON] Implement Avro functions
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #40535:
URL: https://github.com/apache/spark/pull/40535#issuecomment-1487795766
Merged to master and branch-3.4.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #40535: [SPARK-42907][CONNECT][PYTHON] Implement Avro functions
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #40535:
URL: https://github.com/apache/spark/pull/40535#discussion_r1147472671
##########
dev/sparktestsupport/modules.py:
##########
@@ -747,6 +747,7 @@ def __hash__(self):
"pyspark.sql.connect.readwriter",
Review Comment:
oh sorry. Yes, you're correct - we don't need to fix the dependnecy
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on pull request #40535: [SPARK-42907][CONNECT][PYTHON] Implement Avro functions
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #40535:
URL: https://github.com/apache/spark/pull/40535#issuecomment-1480782795
It doesn't have UT in vanilla PySpark, so I only add the doctest
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #40535: [SPARK-42907][CONNECT][PYTHON] Implement Avro functions
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #40535:
URL: https://github.com/apache/spark/pull/40535#discussion_r1146139737
##########
dev/sparktestsupport/modules.py:
##########
@@ -747,6 +747,7 @@ def __hash__(self):
"pyspark.sql.connect.readwriter",
Review Comment:
I think we should add `avro` into `dependencies` above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a diff in pull request #40535: [SPARK-42907][CONNECT][PYTHON] Implement Avro functions
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #40535:
URL: https://github.com/apache/spark/pull/40535#discussion_r1147254339
##########
dev/sparktestsupport/modules.py:
##########
@@ -747,6 +747,7 @@ def __hash__(self):
"pyspark.sql.connect.readwriter",
Review Comment:
thanks, let me have a try
I thought `pyspark_sql` already depends on avro, so didn't touch it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org