You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ctakes.apache.org by Javi Roman <jr...@gmail.com> on 2021/05/03 08:13:05 UTC
Re: svn or github [EXTERNAL]
A way to properly work with large files and GitHub is to use the Git Large
File Storage (LFS) plugin created by GitHub. The following is a session
using this feature:
$ ctakes-testbed.git(main)]$ find -size +50M
./rest-api/healthnlp-examples/ctakes-temporal-demo/src/main/resources/org/apache/ctakes/dictionary/lookup/fast/sno_rx_16ab/sno_rx_16ab.script
./rest-api/healthnlp-examples/ctakes-temporal-demo/src/main/resources/org/apache/ctakes/dictionary/lookup/fast/ctakessnorx/ctakessnorx.script
./rest-api/healthnlp-examples/ctakes-web-client/src/main/resources/org/apache/ctakes/dictionary/lookup/fast/sno_rx_16ab/sno_rx_16ab.script
./rest-api/healthnlp-examples/ctakes-web-client/src/main/resources/org/apache/ctakes/dictionary/lookup/fast/ctakessnorx/ctakessnorx.script
$ git lfs track ctakessnorx.script
$ git lfs track ctakessnorx.script
$ git lfs track
Listing tracked patterns
ctakessnorx.script (.gitattributes)
ctakessnorx.script (.gitattributes)
$ git add .gitattributes
$ git add .
$ git commit -m "....."
$ git push origin main
That enables the git version control system to track huge binary blobs. It
does so by creating a text-based reference to the blob, then tracking and
storing the blob in a location external to the git repository itself, in
this case hosted by GitHub.
This is just an idea.
--
Javi Roman
Twitter: @javiromanrh
GitHub: github.com/javiroman
Linkedin: es.linkedin.com/in/javiroman
Big Data Blog: dataintensive.info
On Tue, Apr 27, 2021 at 8:16 PM Finan, Sean <
Sean.Finan@childrens.harvard.edu> wrote:
> Hi Javi,
>
> https://github.com/apache/ctakes
> is / was an attempt at a mirror of the svn trunk repository. There is
> nothing more complicated than that.
>
> Sean
>
> ________________________________________
> From: Javi Roman <jr...@gmail.com>
> Sent: Tuesday, April 27, 2021 12:58 PM
> To: dev@ctakes.apache.org
> Subject: Re: svn or github [EXTERNAL]
>
> * External Email - Caution *
>
>
> Many thanks Sean.
>
> Any documentation about the repositories organization in Subversion? If I
> understand correctly, the mirror in Github is only the trunk folder in
> Subversion.
>
> --
> Javi Roman
>
> Twitter: @javiromanrh
> GitHub: github.com/javiroman
> Linkedin: es.linkedin.com/in/javiroman
> Big Data Blog: dataintensive.info
>
>
> On Tue, Apr 27, 2021 at 5:15 PM Finan, Sean <
> Sean.Finan@childrens.harvard.edu> wrote:
>
> > Hi Javi,
> >
> > I too would like to get more developers and activity with source
> available
> > on Github. Hopefully you can help us do it.
> >
> > One problem that we had in the past concerning use of Github is caused by
> > large machine learning models in ctakes. Github has file size limits for
> > repositories and some of our models surpassed these limits, which caused
> a
> > corruption of the original migration attempt and errors with subsequent
> > auto-merge checkins. ctakes had to be removed from the svn : github
> > "mirroring".
> >
> > While large files (models, etc.) can be hosted as "release" binaries in
> > github, modifying ctakes' github use in such a way breaks mirroring that
> > would keep both the apache svn and github repositories synchronized.
> > Removing the large model files from the svn area could require further
> > customization of not only that layout but also getting things published
> in
> > maven central.
> >
> > There might be a simple way to reorganize files, simply maintain version
> > control on large files, keep repository mirroring and publication
> automated
> > and document the whole paradigm so that a community can support it.
> > Unfortunately, when this topic was last visited nobody authored or
> > implemented such a solution.
> >
> > It has been many years since this topic was discussed, maybe some fresh
> > perspectives or modernizations can get ctakes on github.
> >
> > Thanks,
> > Sean
> >
> > ________________________________________
> > From: Javi Roman <jr...@gmail.com>
> > Sent: Tuesday, April 27, 2021 10:26 AM
> > To: dev@ctakes.apache.org
> > Subject: Re: svn or github [EXTERNAL]
> >
> > * External Email - Caution *
> >
> >
> > I've just seen the development is based on subversion.
> >
> > It looks like some movement for migrating the subversion to GitHub (most
> of
> > ASF projects migrated to github) in this issue [1], however the issue was
> > created at 19/Nov/17 (it's in progress) and there aren't updates.
> >
> > I would like to open this discussion (fully migration to git) in order to
> > get more developers and activity with an easier interface.
> >
> > Many thanks.
> >
> >
> > [1]
> >
> https://urldefense.com/v3/__https://issues.apache.org/jira/browse/CTAKES-482__;!!NZvER7FxgEiBAiR_!4oyqgJknp0p2BR3zyrRLt-jYvzQkbeztpZ3Dx0lSJIsYxv97mbcSdUFU3W1H4BGE76GgEL4G-58$
> > --
> > Javi Roman
> >
> > Twitter: @javiromanrh
> > GitHub: github.com/javiroman
> > Linkedin: es.linkedin.com/in/javiroman
> > Big Data Blog: dataintensive.info
> >
> >
> > On Tue, Apr 27, 2021 at 3:53 PM Javi Roman <jr...@gmail.com>
> > wrote:
> >
> > > Hi community!
> > >
> > > Is cTakes development currently done in github or subversion?
> > >
> > > --
> > > Javi Roman
> > >
> > > Twitter: @javiromanrh
> > > GitHub: github.com/javiroman
> > > Linkedin: es.linkedin.com/in/javiroman
> > > Big Data Blog: dataintensive.info
> > >
> >
>
Re: svn or github [EXTERNAL]
Posted by "Finan, Sean" <Se...@childrens.harvard.edu>.
Thanks Javi,
I am aware of github's lfs. It is a good idea, but I am not sure how well it would work for a project with a community as large as ctakes. Using the lfs tool seems like it is an inhibitor to easy adoption - which is not where we want to go.
Just to be clear, I am not talking about a migration to github. The desire is to have a mirror of the svn repo on github. The last I spoke with apache infra on this, the lfs was not a viable solution to the problem because it didn't fit into the mirroring technique. The details on that were all behind a door that I never opened, so that is where my knowledge of the matter ends.
Thanks, and keep the ideas rolling,
Sean
________________________________________
From: Javi Roman <jr...@gmail.com>
Sent: Monday, May 3, 2021 4:13 AM
To: dev@ctakes.apache.org
Subject: Re: svn or github [EXTERNAL]
* External Email - Caution *
A way to properly work with large files and GitHub is to use the Git Large
File Storage (LFS) plugin created by GitHub. The following is a session
using this feature:
$ ctakes-testbed.git(main)]$ find -size +50M
./rest-api/healthnlp-examples/ctakes-temporal-demo/src/main/resources/org/apache/ctakes/dictionary/lookup/fast/sno_rx_16ab/sno_rx_16ab.script
./rest-api/healthnlp-examples/ctakes-temporal-demo/src/main/resources/org/apache/ctakes/dictionary/lookup/fast/ctakessnorx/ctakessnorx.script
./rest-api/healthnlp-examples/ctakes-web-client/src/main/resources/org/apache/ctakes/dictionary/lookup/fast/sno_rx_16ab/sno_rx_16ab.script
./rest-api/healthnlp-examples/ctakes-web-client/src/main/resources/org/apache/ctakes/dictionary/lookup/fast/ctakessnorx/ctakessnorx.script
$ git lfs track ctakessnorx.script
$ git lfs track ctakessnorx.script
$ git lfs track
Listing tracked patterns
ctakessnorx.script (.gitattributes)
ctakessnorx.script (.gitattributes)
$ git add .gitattributes
$ git add .
$ git commit -m "....."
$ git push origin main
That enables the git version control system to track huge binary blobs. It
does so by creating a text-based reference to the blob, then tracking and
storing the blob in a location external to the git repository itself, in
this case hosted by GitHub.
This is just an idea.
--
Javi Roman
Twitter: @javiromanrh
GitHub: github.com/javiroman
Linkedin: es.linkedin.com/in/javiroman
Big Data Blog: dataintensive.info
On Tue, Apr 27, 2021 at 8:16 PM Finan, Sean <
Sean.Finan@childrens.harvard.edu> wrote:
> Hi Javi,
>
> https://urldefense.com/v3/__https://github.com/apache/ctakes__;!!NZvER7FxgEiBAiR_!8HtG2DkQcka-Vf_LqzSiO-3nGu0P_X_2bTHtEX0UqdhZaxMS0c5Tp0FG7ONqFNniFlzhAI3QDKI$
> is / was an attempt at a mirror of the svn trunk repository. There is
> nothing more complicated than that.
>
> Sean
>
> ________________________________________
> From: Javi Roman <jr...@gmail.com>
> Sent: Tuesday, April 27, 2021 12:58 PM
> To: dev@ctakes.apache.org
> Subject: Re: svn or github [EXTERNAL]
>
> * External Email - Caution *
>
>
> Many thanks Sean.
>
> Any documentation about the repositories organization in Subversion? If I
> understand correctly, the mirror in Github is only the trunk folder in
> Subversion.
>
> --
> Javi Roman
>
> Twitter: @javiromanrh
> GitHub: github.com/javiroman
> Linkedin: es.linkedin.com/in/javiroman
> Big Data Blog: dataintensive.info
>
>
> On Tue, Apr 27, 2021 at 5:15 PM Finan, Sean <
> Sean.Finan@childrens.harvard.edu> wrote:
>
> > Hi Javi,
> >
> > I too would like to get more developers and activity with source
> available
> > on Github. Hopefully you can help us do it.
> >
> > One problem that we had in the past concerning use of Github is caused by
> > large machine learning models in ctakes. Github has file size limits for
> > repositories and some of our models surpassed these limits, which caused
> a
> > corruption of the original migration attempt and errors with subsequent
> > auto-merge checkins. ctakes had to be removed from the svn : github
> > "mirroring".
> >
> > While large files (models, etc.) can be hosted as "release" binaries in
> > github, modifying ctakes' github use in such a way breaks mirroring that
> > would keep both the apache svn and github repositories synchronized.
> > Removing the large model files from the svn area could require further
> > customization of not only that layout but also getting things published
> in
> > maven central.
> >
> > There might be a simple way to reorganize files, simply maintain version
> > control on large files, keep repository mirroring and publication
> automated
> > and document the whole paradigm so that a community can support it.
> > Unfortunately, when this topic was last visited nobody authored or
> > implemented such a solution.
> >
> > It has been many years since this topic was discussed, maybe some fresh
> > perspectives or modernizations can get ctakes on github.
> >
> > Thanks,
> > Sean
> >
> > ________________________________________
> > From: Javi Roman <jr...@gmail.com>
> > Sent: Tuesday, April 27, 2021 10:26 AM
> > To: dev@ctakes.apache.org
> > Subject: Re: svn or github [EXTERNAL]
> >
> > * External Email - Caution *
> >
> >
> > I've just seen the development is based on subversion.
> >
> > It looks like some movement for migrating the subversion to GitHub (most
> of
> > ASF projects migrated to github) in this issue [1], however the issue was
> > created at 19/Nov/17 (it's in progress) and there aren't updates.
> >
> > I would like to open this discussion (fully migration to git) in order to
> > get more developers and activity with an easier interface.
> >
> > Many thanks.
> >
> >
> > [1]
> >
> https://urldefense.com/v3/__https://issues.apache.org/jira/browse/CTAKES-482__;!!NZvER7FxgEiBAiR_!4oyqgJknp0p2BR3zyrRLt-jYvzQkbeztpZ3Dx0lSJIsYxv97mbcSdUFU3W1H4BGE76GgEL4G-58$
> > --
> > Javi Roman
> >
> > Twitter: @javiromanrh
> > GitHub: github.com/javiroman
> > Linkedin: es.linkedin.com/in/javiroman
> > Big Data Blog: dataintensive.info
> >
> >
> > On Tue, Apr 27, 2021 at 3:53 PM Javi Roman <jr...@gmail.com>
> > wrote:
> >
> > > Hi community!
> > >
> > > Is cTakes development currently done in github or subversion?
> > >
> > > --
> > > Javi Roman
> > >
> > > Twitter: @javiromanrh
> > > GitHub: github.com/javiroman
> > > Linkedin: es.linkedin.com/in/javiroman
> > > Big Data Blog: dataintensive.info
> > >
> >
>
Re: Healthcare Division for Apache
Posted by Javi Roman <jr...@gmail.com>.
Hi Boson,
at the moment there is not much individual interest in collaborating,
possibly because they are not very clear about the idea I have proposed. In
any case, we are discussing the creation of a space for it [1].
For my part I am going to create a test page that will serve as a
demonstration of what I would like to do, as soon as I have it I will share
it so that anyone can give their opinion. I'm hoping that by seeing the
structure of the page there will be more interest.
[1]
https://lists.apache.org/thread.html/r75eb3930f4d730d586a94f3492cde2a9e6ac9df5e10d323c5f6edcac%40%3Cdev.community.apache.org%3E
--
Javi Roman
On Fri, May 7, 2021 at 3:00 PM Bonson Mampilli <bo...@yahoo.com.invalid>
wrote:
> Hello Javi and Peter,
> I shall be very happy to collaborate and help in this.
>
> Regards,Bonson Sebastian MampilliMy LinkedIn
> My Google Scholar
>
>
> On Monday, 3 May, 2021, 11:28:56 pm IST, Javi Roman <
> jroman.espinar@gmail.com> wrote:
>
> On Mon, May 3, 2021 at 6:37 PM Peter Abramowitsch <
> pabramowitsch@gmail.com>
> wrote:
>
> > Hi Javi,
> >
> > I'm a long term ctakes user and contributor who has worked in various
> > professional, incubator, and standards-creating environments for
> > healthcare. I like your idea in theory, but I see an on-the-ground issue
> > with a specific health oriented section of Apache - and that is a lack of
> > content. Unless I have missed something, cTakes is the only mature
> Apache
> > project in this space - it is not as if there are a set of domain-related
> > projects in this space waiting to be promoted.
> >
>
> With lesser software there are foundations that are putting the focus on
> health [1] , to promote the association and join forces in something as
> laudable as health. My original idea is based on public movements in Europe
> for the promotion of Big Data and health [2]. If we take into account that
> innovation in Big Data is in the open source and also the main organization
> in Big Data is the ASF, this together with a really important project
> focused on health (Apache cTAKES + Apache UIMA + Apache OpenNLP).
>
> [1] https://www.lfph.io/
> [2] https://www.bigmedilytics.eu/
>
>
> > There would also be something materially different about a Healthcare
> > specific component of Apache - which is not to say it isn't a valid idea,
> > but it is worth discussion. The difference is that most all of the
> > projects in Apache are embedded in the information technology domain and
> > its challenges - storage, search, management, presentation etc - but tend
> > to be agnostic with regard to the information content over which the
> Apache
> > technology is implemented.
> >
> > Health, on the other hand is its own specific domain, so the tasks,
> > problems, and solutions here are not just about creation, transport and
> > presentation of health related information, but what the information
> > actually is, how it is used, and by whom. Here we're talking about
> > clinical informatics and a host of other information spaces that require
> > very high levels of experience and understanding that are tangential to
> the
> > problems of system infrastructure.
> >
>
> I agree that healthcare is a very specialized field with issues beyond pure
> infrastructure technology. But infrastructure is an enabler for
> researchers, scientists and users. One of the ideas I propose (and I'm not
> saying it has to be done, but I think it's good to discuss it) is how a set
> of ASF technologies can give complete solutions focused on health. It's not
> just medical records processing, there is a lot in healthcare today that
> can be modernized (at least in the healthcare system I know).
>
>
> >
> > If I could invent an analogy... A lot of Apache's software is probably
> > being used in the logistics domain, warehousing, supply chain
> > optimization, package deliveries and such, but would that be a reason to
> > create an Apache shipping and delivery software presence?
> >
>
> This could be the beginning of similar initiatives, what's wrong with that?
> It is a way to promote ASF projects, with a different approach: How
> different technologies can be integrated to provide solutions to societal
> needs, how open source breaks new ground and catalyzes innovation in all
> sectors.
>
> >
> > So this is my hesitation about endorsing a Health initiative at the
> > moment. As I've said, it would be a wonderful idea, but the cart is
> > somewhat before the horse in my opinion. More localized and feasible
> > might be a cTakes user group that goes beyond just this email address.
> >
>
> Thank you very much for your comments Peter, I think this is an exciting
> topic, and the principle can be a place to discuss these ideas, by way of
> exploration. It may come to nothing in the end, or something interesting
> may come out of it, but who knows, I would try something new.
>
> >
> > Regards
> > Peter
> >
>
Re: Healthcare Division for Apache
Posted by Bonson Mampilli <bo...@yahoo.com.INVALID>.
Hello Javi and Peter,
I shall be very happy to collaborate and help in this.
Regards,Bonson Sebastian MampilliMy LinkedIn
My Google Scholar
On Monday, 3 May, 2021, 11:28:56 pm IST, Javi Roman <jr...@gmail.com> wrote:
On Mon, May 3, 2021 at 6:37 PM Peter Abramowitsch <pa...@gmail.com>
wrote:
> Hi Javi,
>
> I'm a long term ctakes user and contributor who has worked in various
> professional, incubator, and standards-creating environments for
> healthcare. I like your idea in theory, but I see an on-the-ground issue
> with a specific health oriented section of Apache - and that is a lack of
> content. Unless I have missed something, cTakes is the only mature Apache
> project in this space - it is not as if there are a set of domain-related
> projects in this space waiting to be promoted.
>
With lesser software there are foundations that are putting the focus on
health [1] , to promote the association and join forces in something as
laudable as health. My original idea is based on public movements in Europe
for the promotion of Big Data and health [2]. If we take into account that
innovation in Big Data is in the open source and also the main organization
in Big Data is the ASF, this together with a really important project
focused on health (Apache cTAKES + Apache UIMA + Apache OpenNLP).
[1] https://www.lfph.io/
[2] https://www.bigmedilytics.eu/
> There would also be something materially different about a Healthcare
> specific component of Apache - which is not to say it isn't a valid idea,
> but it is worth discussion. The difference is that most all of the
> projects in Apache are embedded in the information technology domain and
> its challenges - storage, search, management, presentation etc - but tend
> to be agnostic with regard to the information content over which the Apache
> technology is implemented.
>
> Health, on the other hand is its own specific domain, so the tasks,
> problems, and solutions here are not just about creation, transport and
> presentation of health related information, but what the information
> actually is, how it is used, and by whom. Here we're talking about
> clinical informatics and a host of other information spaces that require
> very high levels of experience and understanding that are tangential to the
> problems of system infrastructure.
>
I agree that healthcare is a very specialized field with issues beyond pure
infrastructure technology. But infrastructure is an enabler for
researchers, scientists and users. One of the ideas I propose (and I'm not
saying it has to be done, but I think it's good to discuss it) is how a set
of ASF technologies can give complete solutions focused on health. It's not
just medical records processing, there is a lot in healthcare today that
can be modernized (at least in the healthcare system I know).
>
> If I could invent an analogy... A lot of Apache's software is probably
> being used in the logistics domain, warehousing, supply chain
> optimization, package deliveries and such, but would that be a reason to
> create an Apache shipping and delivery software presence?
>
This could be the beginning of similar initiatives, what's wrong with that?
It is a way to promote ASF projects, with a different approach: How
different technologies can be integrated to provide solutions to societal
needs, how open source breaks new ground and catalyzes innovation in all
sectors.
>
> So this is my hesitation about endorsing a Health initiative at the
> moment. As I've said, it would be a wonderful idea, but the cart is
> somewhat before the horse in my opinion. More localized and feasible
> might be a cTakes user group that goes beyond just this email address.
>
Thank you very much for your comments Peter, I think this is an exciting
topic, and the principle can be a place to discuss these ideas, by way of
exploration. It may come to nothing in the end, or something interesting
may come out of it, but who knows, I would try something new.
>
> Regards
> Peter
>
Re: Healthcare Division for Apache
Posted by Javi Roman <jr...@gmail.com>.
On Mon, May 3, 2021 at 6:37 PM Peter Abramowitsch <pa...@gmail.com>
wrote:
> Hi Javi,
>
> I'm a long term ctakes user and contributor who has worked in various
> professional, incubator, and standards-creating environments for
> healthcare. I like your idea in theory, but I see an on-the-ground issue
> with a specific health oriented section of Apache - and that is a lack of
> content. Unless I have missed something, cTakes is the only mature Apache
> project in this space - it is not as if there are a set of domain-related
> projects in this space waiting to be promoted.
>
With lesser software there are foundations that are putting the focus on
health [1] , to promote the association and join forces in something as
laudable as health. My original idea is based on public movements in Europe
for the promotion of Big Data and health [2]. If we take into account that
innovation in Big Data is in the open source and also the main organization
in Big Data is the ASF, this together with a really important project
focused on health (Apache cTAKES + Apache UIMA + Apache OpenNLP).
[1] https://www.lfph.io/
[2] https://www.bigmedilytics.eu/
> There would also be something materially different about a Healthcare
> specific component of Apache - which is not to say it isn't a valid idea,
> but it is worth discussion. The difference is that most all of the
> projects in Apache are embedded in the information technology domain and
> its challenges - storage, search, management, presentation etc - but tend
> to be agnostic with regard to the information content over which the Apache
> technology is implemented.
>
> Health, on the other hand is its own specific domain, so the tasks,
> problems, and solutions here are not just about creation, transport and
> presentation of health related information, but what the information
> actually is, how it is used, and by whom. Here we're talking about
> clinical informatics and a host of other information spaces that require
> very high levels of experience and understanding that are tangential to the
> problems of system infrastructure.
>
I agree that healthcare is a very specialized field with issues beyond pure
infrastructure technology. But infrastructure is an enabler for
researchers, scientists and users. One of the ideas I propose (and I'm not
saying it has to be done, but I think it's good to discuss it) is how a set
of ASF technologies can give complete solutions focused on health. It's not
just medical records processing, there is a lot in healthcare today that
can be modernized (at least in the healthcare system I know).
>
> If I could invent an analogy... A lot of Apache's software is probably
> being used in the logistics domain, warehousing, supply chain
> optimization, package deliveries and such, but would that be a reason to
> create an Apache shipping and delivery software presence?
>
This could be the beginning of similar initiatives, what's wrong with that?
It is a way to promote ASF projects, with a different approach: How
different technologies can be integrated to provide solutions to societal
needs, how open source breaks new ground and catalyzes innovation in all
sectors.
>
> So this is my hesitation about endorsing a Health initiative at the
> moment. As I've said, it would be a wonderful idea, but the cart is
> somewhat before the horse in my opinion. More localized and feasible
> might be a cTakes user group that goes beyond just this email address.
>
Thank you very much for your comments Peter, I think this is an exciting
topic, and the principle can be a place to discuss these ideas, by way of
exploration. It may come to nothing in the end, or something interesting
may come out of it, but who knows, I would try something new.
>
> Regards
> Peter
>
Re: Healthcare Division for Apache
Posted by Peter Abramowitsch <pa...@gmail.com>.
Hi Javi,
I'm a long term ctakes user and contributor who has worked in various
professional, incubator, and standards-creating environments for
healthcare. I like your idea in theory, but I see an on-the-ground issue
with a specific health oriented section of Apache - and that is a lack of
content. Unless I have missed something, cTakes is the only mature Apache
project in this space - it is not as if there are a set of domain-related
projects in this space waiting to be promoted.
There would also be something materially different about a Healthcare
specific component of Apache - which is not to say it isn't a valid idea,
but it is worth discussion. The difference is that most all of the
projects in Apache are embedded in the information technology domain and
its challenges - storage, search, management, presentation etc - but tend
to be agnostic with regard to the information content over which the Apache
technology is implemented.
Health, on the other hand is its own specific domain, so the tasks,
problems, and solutions here are not just about creation, transport and
presentation of health related information, but what the information
actually is, how it is used, and by whom. Here we're talking about
clinical informatics and a host of other information spaces that require
very high levels of experience and understanding that are tangential to the
problems of system infrastructure.
If I could invent an analogy... A lot of Apache's software is probably
being used in the logistics domain, warehousing, supply chain
optimization, package deliveries and such, but would that be a reason to
create an Apache shipping and delivery software presence?
So this is my hesitation about endorsing a Health initiative at the
moment. As I've said, it would be a wonderful idea, but the cart is
somewhat before the horse in my opinion. More localized and feasible
might be a cTakes user group that goes beyond just this email address.
Regards
Peter