You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "ShuDaoNan (Jira)" <ji...@apache.org> on 2021/04/25 08:09:00 UTC
[jira] [Created] (SPARK-35217)
com.google.protobuf.Parser.parseFrom() method Can't use in spark
ShuDaoNan created SPARK-35217:
---------------------------------
Summary: com.google.protobuf.Parser.parseFrom() method Can't use in spark
Key: SPARK-35217
URL: https://issues.apache.org/jira/browse/SPARK-35217
Project: Spark
Issue Type: Bug
Components: Deploy, ML
Affects Versions: 2.4.7
Environment: *platform：*
Linux ip-172-17-1-1 4.14.219-164.354.amzn2.x86_64 [#1|https://github.com/apache/spark/pull/1] SMP Mon Feb 22 21:18:39 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Hive 2.3.7, ZooKeeper 3.4.14, Spark 2.4.7, TensorFlow 2.4.1
(Windows10 do not have this problem.)
Reporter: ShuDaoNan
*platform：*
Linux ip-172-17-1-1 4.14.219-164.354.amzn2.x86_64 [#1|https://github.com/apache/spark/pull/1] SMP Mon Feb 22 21:18:39 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Hive 2.3.7, ZooKeeper 3.4.14, Spark 2.4.7, TensorFlow 2.4.1
(Windows10 do not have this problem.)
*problem：*
Can not load TensorFlow model(Java or scala):
{code:java}
[hadoop@ip-172-17-1-1 ~]$ spark-shell --master local[2] --jars s3://jars/jasypt-1.9.2.jar,s3://jars/commons-pool2-2.0.jar,s3://jars/tensorflow-core-api-0.3.1.jar,s3://jars/tensorflow-core-api-0.3.1-linux-x86_64-mkl.jar,s3://jars/ndarray-0.3.1.jar,s3://jars/javacpp-1.5.4.jar,s3://jars/tensorflow-core-platform-0.3.1.jar,s3://jars/protobuf-java-3.8.0.jar Spark session available as 'spark'. Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 2.4.7-amzn-1 /_/ Using Scala version 2.11.12 (OpenJDK 64-Bit Server VM, Java 1.8.0_282) Type in expressions to have them evaluated. Type :help for more information. scala> import org.{tensorflow => tf} import org.{tensorflow=>tf} scala> val bundle = tf.SavedModelBundle.load("/home/hadoop/xDeepFM","serve") 2021-04-23 07:32:56.223881: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:32] Reading SavedModel from: /home/hadoop/xDeepFM 2021-04-23 07:32:56.266424: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:55] Reading meta graph with tags { serve } 2021-04-23 07:32:56.266468: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:93] Reading SavedModel debug info (if present) from: /home/hadoop/xDeepFM 2021-04-23 07:32:56.346757: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:206] Restoring SavedModel bundle. 2021-04-23 07:32:56.873838: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:190] Running initialization op on SavedModel bundle at path: /home/hadoop/xDeepFM 2021-04-23 07:32:56.928656: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:277] SavedModel load for tags { serve }; Status: success: OK. Took 704788 microseconds. java.lang.NoSuchMethodError: com.google.protobuf.Parser.parseFrom(Ljava/nio/ByteBuffer;)Ljava/lang/Object; at org.tensorflow.proto.framework.MetaGraphDef.parseFrom(MetaGraphDef.java:3067) at org.tensorflow.SavedModelBundle.load(SavedModelBundle.java:422) at org.tensorflow.SavedModelBundle.access$000(SavedModelBundle.java:59) at org.tensorflow.SavedModelBundle$Loader.load(SavedModelBundle.java:68) at org.tensorflow.SavedModelBundle.load(SavedModelBundle.java:242) ... 49 elided
scala>
{code}
*But I can load the model in a single 'Testtf.java' like that:*
*{{}}*
{code:java}
[hadoop@ip-172-17-1-1 ~]$ vi Testtf.java
import org.tensorflow.SavedModelBundle;
public class Testtf {
public static void main(String[] args) {
System.out.println("test load...");
SavedModelBundle bundle = org.tensorflow.SavedModelBundle.load("/home/hadoop/xDeepFM","serve");
System.out.println("loaded bundle...");
System.out.println(bundle);
}
}
[hadoop@ip-172-17-1-1 ~]$ javac -cp javacpp-1.5.4-linux-x86_64.jar:tensorflow-core-api-0.3.1-linux-x86_64.jar:tensorflow-core-api-0.3.1.jar:ndarray-0.3.1.jar:javacpp-1.5.4.jar:tensorflow-core-platform-0.3.1.jar:tensorflow-core-platform-mkl-0.3.1.jar:protobuf-java-3.8.0.jar Testtf.java
[hadoop@ip-172-17-1-1 ~]$ java -cp javacpp-1.5.4-linux-x86_64.jar:tensorflow-core-api-0.3.1-linux-x86_64.jar:tensorflow-core-api-0.3.1.jar:ndarray-0.3.1.jar:javacpp-1.5.4.jar:tensorflow-core-platform-0.3.1.jar:tensorflow-core-platform-mkl-0.3.1.jar:protobuf-java-3.8.0.jar:. Testtf
test load...
2021-04-25 02:33:47.247120: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:32] Reading SavedModel from: /home/hadoop/xDeepFM
2021-04-25 02:33:47.275349: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:55] Reading meta graph with tags { serve }
2021-04-25 02:33:47.275405: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:93] Reading SavedModel debug info (if present) from: /home/hadoop/xDeepFM
2021-04-25 02:33:47.275480: I external/org_tensorflow/tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-04-25 02:33:47.370208: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:206] Restoring SavedModel bundle.
2021-04-25 02:33:47.381656: I external/org_tensorflow/tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2300065000 Hz
2021-04-25 02:33:47.667941: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:190] Running initialization op on SavedModel bundle at path: /home/hadoop/xDeepFM
2021-04-25 02:33:47.725458: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:277] SavedModel load for tags { serve }; Status: success: OK. Took 478354 microseconds.
loaded bundle...
org.tensorflow.SavedModelBundle@80503
[hadoop@ip-172-17-1-1 ~]$ {code}
*{{}}*
*Thanks!*
refer: [tensorflow/java#298|https://github.com/tensorflow/java/issues/298]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org