You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Mich Talebzadeh <mi...@gmail.com> on 2024/10/21 16:32:41 UTC
Issue with VAE Model - KerasTensor Incompatibility with TensorFlow Functions
Spark version 3.4.0
Python 3.9.16
tensorflow 2.17.0
Hi,
I encountered an issue while building a VAE (Variational Autoencoder
<https://en.wikipedia.org/wiki/Variational_autoencoder>) model using the
following configuration: I am doing this work as part of imputation of
fraud data
- Input dimension: 250
- Latent dimension: 32
- Method name: build_vae_model
This error occurred when calling build_vae_model within the impute_data_vae
module, leading to a failure with the following error description:
*Error:*
A KerasTensor cannot be used as input to a TensorFlow function. A
KerasTensor is a symbolic placeholder for a shape and dtype, used when
constructing Keras Functional models or Keras Functions. You can only use
it as input to a Keras layer or a Keras operation (from the namespaces
`keras.layers` and `keras.operations`). You are likely doing something like:
```
x = Input(...)
...
tf_fn(x) # Invalid.
```
What you should do instead is wrap `tf_fn` in a layer:
```
class MyLayer(Layer):
def call(self, x):
return tf_fn(x)
x = MyLayer()(x)
```
As a next step, I will be adjusting the build_vae_model method to wrap the
TensorFlow function(s) inside appropriate Keras layers. It is becoming very
time consuming. If anyone has faced a similar issue or has recommendations
on the best practices for handling, I will appreciate it.
Thanks
Mich Talebzadeh,
Architect | Data Engineer | Data Science | Financial Crime
PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial College
London <https://en.wikipedia.org/wiki/Imperial_College_London>
London, United Kingdom
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
https://en.everybodywiki.com/Mich_Talebzadeh
*Disclaimer:* The information provided is correct to the best of my
knowledge but of course cannot be guaranteed . It is essential to note
that, as with any advice, quote "one test result is worth one-thousand
expert opinions (Werner <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von
Braun <https://en.wikipedia.org/wiki/Wernher_von_Braun>)".
Re: Issue with VAE Model - KerasTensor Incompatibility with TensorFlow Functions
Posted by Mich Talebzadeh <mi...@gmail.com>.
?
Mich Talebzadeh,
Architect | Data Engineer | Data Science | Financial Crime
PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial College
London <https://en.wikipedia.org/wiki/Imperial_College_London>
London, United Kingdom
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
https://en.everybodywiki.com/Mich_Talebzadeh
*Disclaimer:* The information provided is correct to the best of my
knowledge but of course cannot be guaranteed . It is essential to note
that, as with any advice, quote "one test result is worth one-thousand
expert opinions (Werner <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von
Braun <https://en.wikipedia.org/wiki/Wernher_von_Braun>)".
On Mon, 21 Oct 2024 at 18:08, Jason Tan <jb...@gmail.com> wrote:
> Jjjb
>
> On Mon, 21 Oct 2024, 17:59 Mich Talebzadeh, <mi...@gmail.com>
> wrote:
>
>>
>> Spark version 3.4.0
>> Python 3.9.16
>> tensorflow 2.17.0
>>
>> Hi,
>>
>> I encountered an issue while building a VAE (Variational Autoencoder
>> <https://en.wikipedia.org/wiki/Variational_autoencoder>) model using the
>> following configuration: I am doing this work as part of imputation of
>> fraud data
>>
>> - Input dimension: 250
>> - Latent dimension: 32
>> - Method name: build_vae_model
>>
>> This error occurred when calling build_vae_model within the
>> impute_data_vae module, leading to a failure with the following error
>> description:
>>
>> *Error:*
>> A KerasTensor cannot be used as input to a TensorFlow function. A
>> KerasTensor is a symbolic placeholder for a shape and dtype, used when
>> constructing Keras Functional models or Keras Functions. You can only use
>> it as input to a Keras layer or a Keras operation (from the namespaces
>> `keras.layers` and `keras.operations`). You are likely doing something like:
>>
>> ```
>>
>> x = Input(...)
>>
>> ...
>>
>> tf_fn(x) # Invalid.
>>
>> ```
>>
>> What you should do instead is wrap `tf_fn` in a layer:
>>
>> ```
>> class MyLayer(Layer):
>>
>> def call(self, x):
>>
>> return tf_fn(x)
>> x = MyLayer()(x)
>> ```
>>
>> As a next step, I will be adjusting the build_vae_model method to wrap
>> the TensorFlow function(s) inside appropriate Keras layers. It is becoming
>> very time consuming. If anyone has faced a similar issue or has
>> recommendations on the best practices for handling, I will appreciate it.
>>
>> Thanks
>>
>> Mich Talebzadeh,
>> Architect | Data Engineer | Data Science | Financial Crime
>> PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial
>> College London <https://en.wikipedia.org/wiki/Imperial_College_London>
>> London, United Kingdom
>>
>>
>> view my Linkedin profile
>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>
>>
>> https://en.everybodywiki.com/Mich_Talebzadeh
>>
>>
>>
>> *Disclaimer:* The information provided is correct to the best of my
>> knowledge but of course cannot be guaranteed . It is essential to note
>> that, as with any advice, quote "one test result is worth one-thousand
>> expert opinions (Werner
>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun
>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>)".
>>
>
Re: Issue with VAE Model - KerasTensor Incompatibility with TensorFlow Functions
Posted by Jason Tan <jb...@gmail.com>.
Jjjb
On Mon, 21 Oct 2024, 17:59 Mich Talebzadeh, <mi...@gmail.com>
wrote:
>
> Spark version 3.4.0
> Python 3.9.16
> tensorflow 2.17.0
>
> Hi,
>
> I encountered an issue while building a VAE (Variational Autoencoder
> <https://en.wikipedia.org/wiki/Variational_autoencoder>) model using the
> following configuration: I am doing this work as part of imputation of
> fraud data
>
> - Input dimension: 250
> - Latent dimension: 32
> - Method name: build_vae_model
>
> This error occurred when calling build_vae_model within the
> impute_data_vae module, leading to a failure with the following error
> description:
>
> *Error:*
> A KerasTensor cannot be used as input to a TensorFlow function. A
> KerasTensor is a symbolic placeholder for a shape and dtype, used when
> constructing Keras Functional models or Keras Functions. You can only use
> it as input to a Keras layer or a Keras operation (from the namespaces
> `keras.layers` and `keras.operations`). You are likely doing something like:
>
> ```
>
> x = Input(...)
>
> ...
>
> tf_fn(x) # Invalid.
>
> ```
>
> What you should do instead is wrap `tf_fn` in a layer:
>
> ```
> class MyLayer(Layer):
>
> def call(self, x):
>
> return tf_fn(x)
> x = MyLayer()(x)
> ```
>
> As a next step, I will be adjusting the build_vae_model method to wrap
> the TensorFlow function(s) inside appropriate Keras layers. It is becoming
> very time consuming. If anyone has faced a similar issue or has
> recommendations on the best practices for handling, I will appreciate it.
>
> Thanks
>
> Mich Talebzadeh,
> Architect | Data Engineer | Data Science | Financial Crime
> PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial College
> London <https://en.wikipedia.org/wiki/Imperial_College_London>
> London, United Kingdom
>
>
> view my Linkedin profile
> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>
>
> https://en.everybodywiki.com/Mich_Talebzadeh
>
>
>
> *Disclaimer:* The information provided is correct to the best of my
> knowledge but of course cannot be guaranteed . It is essential to note
> that, as with any advice, quote "one test result is worth one-thousand
> expert opinions (Werner <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von
> Braun <https://en.wikipedia.org/wiki/Wernher_von_Braun>)".
>