You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/09/08 20:23:50 UTC

[GitHub] [spark] ueshin commented on a change in pull request #29686: [SPARK-32312][SQL][PYTHON] Upgrade Apache Arrow to version 1.0.1

ueshin commented on a change in pull request #29686:
URL: https://github.com/apache/spark/pull/29686#discussion_r485174089



##########
File path: pom.xml
##########
@@ -2329,19 +2329,26 @@
           </exclusion>
           <exclusion>
             <groupId>com.fasterxml.jackson.core</groupId>
-            <artifactId>jackson-databind</artifactId>
+            <artifactId>jackson-core</artifactId>
           </exclusion>
           <exclusion>
             <groupId>io.netty</groupId>
-            <artifactId>netty-buffer</artifactId>
+            <artifactId>netty-common</artifactId>
           </exclusion>
+        </exclusions>
+      </dependency>
+      <dependency>
+        <groupId>org.apache.arrow</groupId>
+        <artifactId>arrow-memory-netty</artifactId>

Review comment:
       Don't we need any configs except for adding the dependency to indicate which allocator should be used?

##########
File path: python/pyspark/sql/pandas/utils.py
##########
@@ -40,7 +40,7 @@ def require_minimum_pyarrow_version():
     """ Raise ImportError if minimum version of pyarrow is not installed
     """
     # TODO(HyukjinKwon): Relocate and deduplicate the version specification.
-    minimum_pyarrow_version = "0.15.1"
+    minimum_pyarrow_version = "1.0.0"

Review comment:
       +1

##########
File path: python/setup.py
##########
@@ -96,11 +96,11 @@ def _supports_symlinks():
               file=sys.stderr)
         sys.exit(-1)
 
-# If you are changing the versions here, please also change ./python/pyspark/sql/utils.py
+# If you are changing the versions here, please also change ./python/pyspark/sql/pandas/utils.py

Review comment:
       👍 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org