You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/04/25 06:38:35 UTC
[GitHub] [spark] JWenBin opened a new pull request #32329: com.google.protobuf.Parser.parseFrom() method Can't use in spark
JWenBin opened a new pull request #32329:
URL: https://github.com/apache/spark/pull/32329
**platform：**
Linux ip-172-17-1-1 4.14.219-164.354.amzn2.x86_64 #1 SMP Mon Feb 22 21:18:39 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Hive 2.3.7, ZooKeeper 3.4.14, Spark 2.4.7, TensorFlow 2.4.1
(Windows10 do not have this problem.)
problem：
Can not load TensorFlow model:
```
[hadoop@ip-172-17-1-1 ~]$ spark-shell --master local[2] --jars s3://jars/jasypt-1.9.2.jar,s3://jars/commons-pool2-2.0.jar,s3://jars/tensorflow-core-api-0.3.1.jar,s3://jars/tensorflow-core-api-0.3.1-linux-x86_64-mkl.jar,s3://jars/ndarray-0.3.1.jar,s3://jars/javacpp-1.5.4.jar,s3://jars/tensorflow-core-platform-0.3.1.jar,s3://jars/protobuf-java-3.8.0.jar
Spark session available as 'spark'.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.4.7-amzn-1
/_/
Using Scala version 2.11.12 (OpenJDK 64-Bit Server VM, Java 1.8.0_282)
Type in expressions to have them evaluated.
Type :help for more information.
scala> import org.{tensorflow => tf}
import org.{tensorflow=>tf}
scala> val bundle = tf.SavedModelBundle.load("/home/hadoop/xDeepFM","serve")
2021-04-23 07:32:56.223881: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:32] Reading SavedModel from: /home/hadoop/xDeepFM
2021-04-23 07:32:56.266424: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:55] Reading meta graph with tags { serve }
2021-04-23 07:32:56.266468: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:93] Reading SavedModel debug info (if present) from: /home/hadoop/xDeepFM
2021-04-23 07:32:56.346757: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:206] Restoring SavedModel bundle.
2021-04-23 07:32:56.873838: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:190] Running initialization op on SavedModel bundle at path: /home/hadoop/xDeepFM
2021-04-23 07:32:56.928656: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:277] SavedModel load for tags { serve }; Status: success: OK. Took 704788 microseconds.
java.lang.NoSuchMethodError: com.google.protobuf.Parser.parseFrom(Ljava/nio/ByteBuffer;)Ljava/lang/Object;
at org.tensorflow.proto.framework.MetaGraphDef.parseFrom(MetaGraphDef.java:3067)
at org.tensorflow.SavedModelBundle.load(SavedModelBundle.java:422)
at org.tensorflow.SavedModelBundle.access$000(SavedModelBundle.java:59)
at org.tensorflow.SavedModelBundle$Loader.load(SavedModelBundle.java:68)
at org.tensorflow.SavedModelBundle.load(SavedModelBundle.java:242)
... 49 elided
scala>
```
But I can load the model in a single 'Testtf.java' like that:
```
vi Testtf.java
import org.tensorflow.SavedModelBundle;
public class Testtf {
public static void main(String[] args) {
System.out.println("test load...");
SavedModelBundle bundle = org.tensorflow.SavedModelBundle.load("/home/hadoop/xDeepFM","serve");
System.out.println("loaded bundle...");
System.out.println(bundle);
}
}
[hadoop@ip-172-17-1-1 ~]$ javac -cp javacpp-1.5.4-linux-x86_64.jar:tensorflow-core-api-0.3.1-linux-x86_64.jar:tensorflow-core-api-0.3.1.jar:ndarray-0.3.1.jar:javacpp-1.5.4.jar:tensorflow-core-platform-0.3.1.jar:tensorflow-core-platform-mkl-0.3.1.jar:protobuf-java-3.8.0.jar Testtf.java
[hadoop@ip-172-17-1-1 ~]$ java -cp javacpp-1.5.4-linux-x86_64.jar:tensorflow-core-api-0.3.1-linux-x86_64.jar:tensorflow-core-api-0.3.1.jar:ndarray-0.3.1.jar:javacpp-1.5.4.jar:tensorflow-core-platform-0.3.1.jar:tensorflow-core-platform-mkl-0.3.1.jar:protobuf-java-3.8.0.jar:. Testtf
test load...
2021-04-25 02:33:47.247120: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:32] Reading SavedModel from: /home/hadoop/xDeepFM
2021-04-25 02:33:47.275349: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:55] Reading meta graph with tags { serve }
2021-04-25 02:33:47.275405: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:93] Reading SavedModel debug info (if present) from: /home/hadoop/xDeepFM
2021-04-25 02:33:47.275480: I external/org_tensorflow/tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-04-25 02:33:47.370208: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:206] Restoring SavedModel bundle.
2021-04-25 02:33:47.381656: I external/org_tensorflow/tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2300065000 Hz
2021-04-25 02:33:47.667941: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:190] Running initialization op on SavedModel bundle at path: /home/hadoop/xDeepFM
2021-04-25 02:33:47.725458: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:277] SavedModel load for tags { serve }; Status: success: OK. Took 478354 microseconds.
loaded bundle...
org.tensorflow.SavedModelBundle@80503
[hadoop@ip-172-17-1-1 ~]$
```
Thanks!
refer: https://github.com/tensorflow/java/issues/298
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] JWenBin commented on pull request #32329: com.google.protobuf.Parser.parseFrom() method Can't use in spark
Posted by GitBox <gi...@apache.org>.
JWenBin commented on pull request #32329:
URL: https://github.com/apache/spark/pull/32329#issuecomment-826275834
>
>
> Please use JIRA to file an issue: https://issues.apache.org/jira/projects/SPARK
thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] JWenBin commented on pull request #32329: com.google.protobuf.Parser.parseFrom() method Can't use in spark
Posted by GitBox <gi...@apache.org>.
JWenBin commented on pull request #32329:
URL: https://github.com/apache/spark/pull/32329#issuecomment-826275834
>
>
> Please use JIRA to file an issue: https://issues.apache.org/jira/projects/SPARK
thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32329: com.google.protobuf.Parser.parseFrom() method Can't use in spark
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32329:
URL: https://github.com/apache/spark/pull/32329#issuecomment-826272906
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #32329: com.google.protobuf.Parser.parseFrom() method Can't use in spark
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #32329:
URL: https://github.com/apache/spark/pull/32329#issuecomment-826275418
Please use JIRA to file an issue: https://issues.apache.org/jira/projects/SPARK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #32329: com.google.protobuf.Parser.parseFrom() method Can't use in spark
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #32329:
URL: https://github.com/apache/spark/pull/32329
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32329: com.google.protobuf.Parser.parseFrom() method Can't use in spark
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32329:
URL: https://github.com/apache/spark/pull/32329#issuecomment-826272906
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #32329: com.google.protobuf.Parser.parseFrom() method Can't use in spark
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #32329:
URL: https://github.com/apache/spark/pull/32329#issuecomment-826275418
Please use JIRA to file an issue: https://issues.apache.org/jira/projects/SPARK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #32329: com.google.protobuf.Parser.parseFrom() method Can't use in spark
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #32329:
URL: https://github.com/apache/spark/pull/32329
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org