You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "zhouyuan (via GitHub)" <gi...@apache.org> on 2024/03/13 01:33:30 UTC
[I] [VL] remove dynamic packaging and use static packaing [incubator-gluten]
zhouyuan opened a new issue, #4942:
URL: https://github.com/apache/incubator-gluten/issues/4942
### Description
The dynamic packaging will make a jar with all necessary dependencies for shared libs. On gluten start it will try to extract the libs and then load into JVM.
This method is introduced in early stage of Gluten. The production env is quite different: Cent7, Cent8, Ubuntu 20, Debian, ArchLinux and some customized Linux releases. For different OS the dependencies are different, especially on the library path and the library versions. The code is quite difficult to maintain as Velox is bumping the dependencies.
For better packaging releasing/maintaining, gluten may should pick the static packaging(vcpkg based) so all dependencies are all statically linked and can work on all OS.
CC: @weiting-chen
Thanks,
-yuan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] remove dynamic packaging and use static packaing [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #4942:
URL: https://github.com/apache/incubator-gluten/issues/4942#issuecomment-1995250218
We may mention this in dev maillist.
static build makes the deployment much easier. The only issue may be the jar size but now even with debug info, it's just ~88MB.
For developer with current build script, all static library will only be built once. Or build again after vcpkg script changed. velox will be built when commit id changed. So not an issue to development either. It may increase the link time somehow.
See if anyone have concerns about this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] remove dynamic packaging and use static packaing [incubator-gluten]
Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.
zhouyuan commented on issue #4942:
URL: https://github.com/apache/incubator-gluten/issues/4942#issuecomment-2005706221
https://github.com/apache/incubator-gluten/pull/5013
> > > We may mention this in dev maillist.
> >
> >
> > Hey @zhouyuan @FelixYBW Since we're in Apache incubator community now, if we decide to use dev maillist, better update
> > https://github.com/apache/incubator-gluten/blob/62746c40e6e725f1c20ad06251ef976350d2bfc3/README.md?plain=1#L113
> >
> > to reflect email address whereas others can join it.
>
> Hi Rex, yes, will add the maillist to the doc, thanks!
>
> Thanks, -yuan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] remove dynamic packaging and use static packaing [incubator-gluten]
Posted by "boneanxs (via GitHub)" <gi...@apache.org>.
boneanxs commented on issue #4942:
URL: https://github.com/apache/incubator-gluten/issues/4942#issuecomment-1999310507
> We may mention this in dev maillist.
Hey @zhouyuan @FelixYBW Since we're in Apache incubator community now, if we decide to use dev maillist, better update https://github.com/apache/incubator-gluten/blob/62746c40e6e725f1c20ad06251ef976350d2bfc3/README.md?plain=1#L113 to reflect email address whereas others can join it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] remove dynamic packaging and use static packaing [incubator-gluten]
Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you commented on issue #4942:
URL: https://github.com/apache/incubator-gluten/issues/4942#issuecomment-2017033485
I have an another issue about libarrow.so
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] remove dynamic packaging and use static packaing [incubator-gluten]
Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.
zhouyuan commented on issue #4942:
URL: https://github.com/apache/incubator-gluten/issues/4942#issuecomment-2017035345
> I have an another issue about libarrow.so. It seems similar with @PHILO-HE <img alt="image" width="691" src="https://private-user-images.githubusercontent.com/12025282/316358021-6a30bd38-aabd-4098-b9c3-b478331946d5.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTEzMjg5MDQsIm5iZiI6MTcxMTMyODYwNCwicGF0aCI6Ii8xMjAyNTI4Mi8zMTYzNTgwMjEtNmEzMGJkMzgtYWFiZC00MDk4LWI5YzMtYjQ3ODMzMTk0NmQ1LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDAzMjUlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwMzI1VDAxMDMyNFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWRmZmIzYjY5NzIwMjI4YTIyNmZkMGQ3MTMyZmFiYjM0MTE4ZmZlYzg3MTczZTdkZTk2MWI1MzllMTM4NmFhYWMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.-FHtzrd9THxR7JlMph9BRgHLrQkvonA9bs6duxDd6K4">
@ulysses-you the issue should be fixed by using static building libarrow.so/libparquet.so, will try to fix this in
https://github.com/apache/incubator-gluten/pull/5033
Thanks,
-yuan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] remove dynamic packaging and use static packaing [incubator-gluten]
Posted by "PHILO-HE (via GitHub)" <gi...@apache.org>.
PHILO-HE commented on issue #4942:
URL: https://github.com/apache/incubator-gluten/issues/4942#issuecomment-2002987247
FYI.
Just tried using static link on centos-8, then running test on centos-7. An issue was reported at runtime:
```
java.lang.UnsatisfiedLinkError:
version `GLIBC_2.27' not found (required by /tmp/gluten-de464be8-5c4e-45eb-8fda-8749954938c4/jni/6730a5e2-3e06-4112-b509-6941ceb4126f/gluten-6403925142698728788/libarrow.so.1400.1.0)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] remove dynamic packaging and use static packaing [incubator-gluten]
Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.
zhouyuan commented on issue #4942:
URL: https://github.com/apache/incubator-gluten/issues/4942#issuecomment-1999568915
> > We may mention this in dev maillist.
>
> Hey @zhouyuan @FelixYBW Since we're in Apache incubator community now, if we decide to use dev maillist, better update
>
> https://github.com/apache/incubator-gluten/blob/62746c40e6e725f1c20ad06251ef976350d2bfc3/README.md?plain=1#L113
>
> to reflect email address whereas others can join it.
Hi Rex, yes, will add the maillist to the doc, thanks!
Thanks,
-yuan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] remove dynamic packaging and use static packaing [incubator-gluten]
Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you commented on issue #4942:
URL: https://github.com/apache/incubator-gluten/issues/4942#issuecomment-2017035886
thank you @zhouyuan !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] remove dynamic packaging and use static packaing [incubator-gluten]
Posted by "Yohahaha (via GitHub)" <gi...@apache.org>.
Yohahaha commented on issue #4942:
URL: https://github.com/apache/incubator-gluten/issues/4942#issuecomment-2024590312
We can provide options to choose building static or dynamic artifacts, and enable static by default. Just not force ban dynamic packaging directly, thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org