[jira] [Comment Edited] (TIKA-3119) General upgrades for 1.25

["Thamme Gowda (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TIKA-3119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17140962#comment-17140962 ] 

Thamme Gowda edited comment on TIKA-3119 at 6/20/20, 4:29 AM:
--------------------------------------------------------------

[~tallison]
 The test fails because of this Exception:
{code:java}
org.deeplearning4j.exception.DL4JInvalidInputException: Cannot do forward pass in Convolution layer (layer name = conv2d_1, layer index = 1): input array channels does not match CNN layer configuration (data format = NHWC, data input channels = 299, [minibatch, height, width, channels]=[1, 3, 299, 299]; expected input channels = 3) (layer name: conv2d_1, layer index: 1, layer type: ConvolutionLayer)
Note: Convolution layers can be configured for either NCHW (channels first) or NHWC (channels last) format for input images and activations.
Layers can be configured using .dataFormat(CNN2DFormat.NCHW/NHWC) when constructing the layer, or for the entire net using .setInputType(InputType.convolutional(height,  width, depth, CNN2DForman.NCHW/NHWC)).
ImageRecordReader and NativeImageLoader can also be configured to load image data in either NCHW or NHWC format which must match the network

{code}
!Screen Shot 2020-06-19 at 8.56.05 PM.png!

Originally the tensoir dims were of NCHW, but now the default seems to be NHWC (where N=batch, C=channels=3, H=height=299, W=Width=299)

So we are giving input as [1, 3, 299, 299] but the model in the latest API is asking [1, 299, 299, 3]

I tried transposing my input from  NCHW -> NHWC as a quick fix, but the JVM crashes. 
 That is because, (I am guessing) the some of the weights of model's layers are still NCHW, so it didn't work.

The newer API supposedly has a method to explicitly indicate NCHW input type, as informed by Exception
{{.setInputType(InputType.convolutional(height, width, depth, CNN2DForman.NCHW/NHWC))}}

which we should call with these args:
 {{.setInputType(InputType.convolutional(299, 299, 1, CNN2DForman.NCHW));}}

Here is where it has to go:
 [https://github.com/apache/tika/blob/233a72d45de515a7ecb5988802559690cd91f57a/tika-dl/src/main/java/org/apache/tika/dl/imagerec/DL4JInceptionV3Net.java#L261-L264]

But the KerasModelBuilder class we are using doesn't have that method (Not does KerasModel or the class returned by model.getComputationGraph(); So I dont know where it should go (I think, the KerasModelBuilder class should have had it; but somehow DL4J folks missed it?)

 

We need help from DL4J team to fix this. I don't know whom to tag or CC.


was (Author: thammegowda):
[~tallison]
 The test fails because of this Exception:
{code:java}
org.deeplearning4j.exception.DL4JInvalidInputException: Cannot do forward pass in Convolution layer (layer name = conv2d_1, layer index = 1): input array channels does not match CNN layer configuration (data format = NHWC, data input channels = 299, [minibatch, height, width, channels]=[1, 3, 299, 299]; expected input channels = 3) (layer name: conv2d_1, layer index: 1, layer type: ConvolutionLayer)
Note: Convolution layers can be configured for either NCHW (channels first) or NHWC (channels last) format for input images and activations.
Layers can be configured using .dataFormat(CNN2DFormat.NCHW/NHWC) when constructing the layer, or for the entire net using .setInputType(InputType.convolutional(height,  width, depth, CNN2DForman.NCHW/NHWC)).
ImageRecordReader and NativeImageLoader can also be configured to load image data in either NCHW or NHWC format which must match the network

{code}
!Screen Shot 2020-06-19 at 8.56.05 PM.png!

Originally the tensoir dims were of NCHW, but now the default seems to be NHWC (where N=batch, C=channels=3, H=height=299, W=Width=299)

So we are giving input as [1, 3, 299, 299] but the model in the latest API is asking [1, 299, 299, 3]

I tried transposing my input from  NCHW -> NHWC as a quick fix, but the JVM crashes. 
 That is because, (I am guessing) the some of the weights of model's layers are still NCHW, so it didn't work.


The newer API supposedly has a method to explicitly indicate NCHW input type, as informed by Exception
{{ .setInputType(InputType.convolutional(height, width, depth, CNN2DForman.NCHW/NHWC))}}

which we should call with these args:
{{.setInputType(InputType.convolutional(299, 299, 1, CNN2DForman.NCHW));}}{{}}

Here is where it has to go:
 [https://github.com/apache/tika/blob/233a72d45de515a7ecb5988802559690cd91f57a/tika-dl/src/main/java/org/apache/tika/dl/imagerec/DL4JInceptionV3Net.java#L261-L264]

But the KerasModelBuilder class we are using doesn't have that method (Not does KerasModel or the class returned by model.getComputationGraph(); So I dont know where it should go (I think, the KerasModelBuilder class should have had it; but somehow DL4J folks missed it?)

 

We need help from DL4J team to fix this. I don't know whom to tag or CC.

> General upgrades for 1.25
> -------------------------
>
>                 Key: TIKA-3119
>                 URL: https://issues.apache.org/jira/browse/TIKA-3119
>             Project: Tika
>          Issue Type: Task
>            Reporter: Tim Allison
>            Priority: Major
>         Attachments: Screen Shot 2020-06-19 at 8.56.05 PM.png, Screenshot from 2020-06-19 14-50-21.png
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)