You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "ShuDaoNan (Jira)" <ji...@apache.org> on 2021/04/27 09:57:00 UTC
[jira] [Resolved] (SPARK-35217)
com.google.protobuf.Parser.parseFrom() method Can't use in spark
[ https://issues.apache.org/jira/browse/SPARK-35217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ShuDaoNan resolved SPARK-35217.
-------------------------------
Resolution: Incomplete
Amazon emr自带的protobuf-java-version.jar 版本过低导致
> com.google.protobuf.Parser.parseFrom() method Can't use in spark
> ------------------------------------------------------------------
>
> Key: SPARK-35217
> URL: https://issues.apache.org/jira/browse/SPARK-35217
> Project: Spark
> Issue Type: Question
> Components: Deploy, ML
> Affects Versions: 2.4.7
> Environment: *platform：*
> Linux ip-172-17-1-1 4.14.219-164.354.amzn2.x86_64 [#1|https://github.com/apache/spark/pull/1] SMP Mon Feb 22 21:18:39 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
> Hive 2.3.7, ZooKeeper 3.4.14, Spark 2.4.7, TensorFlow 2.4.1
> (Windows10 do not have this problem.)
> Reporter: ShuDaoNan
> Priority: Trivial
>
> *platform：*
> Linux ip-172-17-1-1 4.14.219-164.354.amzn2.x86_64 [#1|https://github.com/apache/spark/pull/1] SMP Mon Feb 22 21:18:39 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
> Hive 2.3.7, ZooKeeper 3.4.14, Spark 2.4.7, TensorFlow 2.4.1
> (Windows10 do not have this problem.)
> *problem：*
> Can not load TensorFlow model(Java or scala):
> {code:java}
> [hadoop@ip-172-17-1-1 ~]$ spark-shell --master local[2] --jars s3://jars/jasypt-1.9.2.jar,s3://jars/commons-pool2-2.0.jar,s3://jars/tensorflow-core-api-0.3.1.jar,s3://jars/tensorflow-core-api-0.3.1-linux-x86_64-mkl.jar,s3://jars/ndarray-0.3.1.jar,s3://jars/javacpp-1.5.4.jar,s3://jars/tensorflow-core-platform-0.3.1.jar,s3://jars/protobuf-java-3.8.0.jar
> Spark session available as 'spark'.
> Welcome to
> ____ __
> / __/__ ___ _____/ /__
> _\ \/ _ \/ _ `/ __/ '_/
> /___/ .__/\_,_/_/ /_/\_\ version 2.4.7-amzn-1
> /_/
>
> Using Scala version 2.11.12 (OpenJDK 64-Bit Server VM, Java 1.8.0_282)
> Type in expressions to have them evaluated.
> Type :help for more information.
> scala> import org.{tensorflow => tf}
> import org.{tensorflow=>tf}
> scala> val bundle = tf.SavedModelBundle.load("/home/hadoop/xDeepFM","serve")
> 2021-04-23 07:32:56.223881: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:32] Reading SavedModel from: /home/hadoop/xDeepFM
> 2021-04-23 07:32:56.266424: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:55] Reading meta graph with tags { serve }
> 2021-04-23 07:32:56.266468: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:93] Reading SavedModel debug info (if present) from: /home/hadoop/xDeepFM
> 2021-04-23 07:32:56.346757: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:206] Restoring SavedModel bundle.
> 2021-04-23 07:32:56.873838: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:190] Running initialization op on SavedModel bundle at path: /home/hadoop/xDeepFM
> 2021-04-23 07:32:56.928656: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:277] SavedModel load for tags { serve }; Status: success: OK. Took 704788 microseconds.
> java.lang.NoSuchMethodError: com.google.protobuf.Parser.parseFrom(Ljava/nio/ByteBuffer;)Ljava/lang/Object;
> at org.tensorflow.proto.framework.MetaGraphDef.parseFrom(MetaGraphDef.java:3067)
> at org.tensorflow.SavedModelBundle.load(SavedModelBundle.java:422)
> at org.tensorflow.SavedModelBundle.access$000(SavedModelBundle.java:59)
> at org.tensorflow.SavedModelBundle$Loader.load(SavedModelBundle.java:68)
> at org.tensorflow.SavedModelBundle.load(SavedModelBundle.java:242)
> ... 49 elided
> scala>
> {code}
> *But I can load the model in a single 'Testtf.java' like that:*
> {code:java}
> [hadoop@ip-172-17-1-1 ~]$ vi Testtf.java
> import org.tensorflow.SavedModelBundle;
> public class Testtf {
> public static void main(String[] args) {
> System.out.println("test load...");
> SavedModelBundle bundle = org.tensorflow.SavedModelBundle.load("/home/hadoop/xDeepFM","serve");
> System.out.println("loaded bundle...");
> System.out.println(bundle);
> }
> }
> [hadoop@ip-172-17-1-1 ~]$ javac -cp javacpp-1.5.4-linux-x86_64.jar:tensorflow-core-api-0.3.1-linux-x86_64.jar:tensorflow-core-api-0.3.1.jar:ndarray-0.3.1.jar:javacpp-1.5.4.jar:tensorflow-core-platform-0.3.1.jar:tensorflow-core-platform-mkl-0.3.1.jar:protobuf-java-3.8.0.jar Testtf.java
> [hadoop@ip-172-17-1-1 ~]$ java -cp javacpp-1.5.4-linux-x86_64.jar:tensorflow-core-api-0.3.1-linux-x86_64.jar:tensorflow-core-api-0.3.1.jar:ndarray-0.3.1.jar:javacpp-1.5.4.jar:tensorflow-core-platform-0.3.1.jar:tensorflow-core-platform-mkl-0.3.1.jar:protobuf-java-3.8.0.jar:. Testtf
> test load...
> 2021-04-25 02:33:47.247120: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:32] Reading SavedModel from: /home/hadoop/xDeepFM
> 2021-04-25 02:33:47.275349: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:55] Reading meta graph with tags { serve }
> 2021-04-25 02:33:47.275405: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:93] Reading SavedModel debug info (if present) from: /home/hadoop/xDeepFM
> 2021-04-25 02:33:47.275480: I external/org_tensorflow/tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
> To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
> 2021-04-25 02:33:47.370208: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:206] Restoring SavedModel bundle.
> 2021-04-25 02:33:47.381656: I external/org_tensorflow/tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2300065000 Hz
> 2021-04-25 02:33:47.667941: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:190] Running initialization op on SavedModel bundle at path: /home/hadoop/xDeepFM
> 2021-04-25 02:33:47.725458: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:277] SavedModel load for tags { serve }; Status: success: OK. Took 478354 microseconds.
> loaded bundle...
> org.tensorflow.SavedModelBundle@80503
> [hadoop@ip-172-17-1-1 ~]$ {code}
> *Thanks!*
> refer: [tensorflow/java#298|https://github.com/tensorflow/java/issues/298]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org