Resizing Image with Scrimage in Spark

[Adline Dsilva <ad...@mimos.my>]

Hi All,

   I have a Hive Table which contains around 500 million photos(Profile picture of Users) stored as hex string and total size of the table is 5TB. I'm trying to make a solution where images can be retrieved in real-time.

Current Solution,  Resize the images, index it along the user profile to solr. For Resizing, Im using a scala library called scrimage<https://github.com/sksamuel/scrimage>

While running the udf function im getting below error.
Serialization stack:
    - object not serializable (class: com.sksamuel.scrimage.Image, value: Image [width=767, height=1024, type=2])
    - field (class: $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC, name: imgR, type: class com.sksamuel.scrimage.Image)

Can anyone suggest method to overcome the above error.

Regards,
Adline

________________________________
DISCLAIMER:


This e-mail (including any attachments) is for the addressee(s) only and may be confidential, especially as regards personal data. If you are not the intended recipient, please note that any dealing, review, distribution, printing, copying or use of this e-mail is strictly prohibited. If you have received this email in error, please notify the sender immediately and delete the original message (including any attachments).

MIMOS Berhad is a research and development institution under the purview of the Malaysian Ministry of Science, Technology and Innovation. Opinions, conclusions and other information in this e-mail that do not relate to the official business of MIMOS Berhad and/or its subsidiaries shall be understood as neither given nor endorsed by MIMOS Berhad and/or its subsidiaries and neither MIMOS Berhad nor its subsidiaries accepts responsibility for the same. All liability arising from or in connection with computer viruses and/or corrupted e-mails is excluded to the fullest extent permitted by law.

Re: Resizing Image with Scrimage in Spark

[Sean Owen <so...@cloudera.com>]

It pretty much means what it says. Objects you send across machines must be
serializable, and the object from the library is not.
You can write a wrapper object that is serializable and knows how to
serialize it. Or ask the library dev to consider making this object
serializable.

On Mon, Oct 17, 2016 at 8:04 AM Adline Dsilva <ad...@mimos.my>
wrote:

> Hi All,
>
>    I have a Hive Table which contains around 500 million photos(Profile
> picture of Users) stored as hex string and total size of the table is 5TB.
> I'm trying to make a solution where images can be retrieved in real-time.
>
> Current Solution,  Resize the images, index it along the user profile to
> solr. For Resizing, Im using a scala library called scrimage
> <https://github.com/sksamuel/scrimage>
>
> While running the udf function im getting below error.
> Serialization stack:
>     - *object not serializable* (class: com.sksamuel.scrimage.Image,
> value: Image [width=767, height=1024, type=2])
>     - field (class: $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC, name: imgR,
> type: class com.sksamuel.scrimage.Image)
>
> Can anyone suggest method to overcome the above error.
>
> Regards,
> Adline
>
> ------------------------------
> *DISCLAIMER:*
>
> This e-mail (including any attachments) is for the addressee(s) only and
> may be confidential, especially as regards personal data. If you are not
> the intended recipient, please note that any dealing, review, distribution,
> printing, copying or use of this e-mail is strictly prohibited. If you have
> received this email in error, please notify the sender immediately and
> delete the original message (including any attachments).
>
> MIMOS Berhad is a research and development institution under the purview
> of the Malaysian Ministry of Science, Technology and Innovation. Opinions,
> conclusions and other information in this e-mail that do not relate to the
> official business of MIMOS Berhad and/or its subsidiaries shall be
> understood as neither given nor endorsed by MIMOS Berhad and/or its
> subsidiaries and neither MIMOS Berhad nor its subsidiaries accepts
> responsibility for the same. All liability arising from or in connection
> with computer viruses and/or corrupted e-mails is excluded to the fullest
> extent permitted by law.
>