You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@chemistry.apache.org by Mike IT Expert <mi...@gmail.com> on 2018/05/25 10:19:41 UTC

Does cmislib python support multi-threading?

Dear All,

Apologies in advance for sending this message to dev mailing list (I wish
there was userlist also I wasn't sure it is bug or not.)

I have started working with cmislib since past week using Alfresco as the
local server.

My goal is to upload documents provided by json rcord list in a  for loop
as you see below *which is part of run-overiden method of
threading.Thread.run method for python 2.7*.

The main code is attached just in case.



The above code works find using ONE thread only. If I use more than one
thread I get the below exception for some records which is odd.


File "module_proj_2_parts_1_and_2.py", line 82, in run
    newDoc = targetFolder.createDocument( docName, contentFile = inFile )
  File
"/home/mike/Desktop/hassan_proj2/.pyenv2/local/lib/python2.7/site-packages/cmislib/atompub/binding.py",
line 2788, in createDocument
    contentEncoding)
  File
"/home/mike/Desktop/hassan_proj2/.pyenv2/local/lib/python2.7/site-packages/cmislib/atompub/binding.py",
line 1807, in createDocument
    ATOM_XML_ENTRY_TYPE)
  File
"/home/mike/Desktop/hassan_proj2/.pyenv2/local/lib/python2.7/site-packages/cmislib/atompub/binding.py",
line 188, in post
    self._processCommonErrors(resp, url)
  File
"/home/mike/Desktop/hassan_proj2/.pyenv2/local/lib/python2.7/site-packages/cmislib/cmis_services.py",
line 63, in _processCommonErrors
    raise UpdateConflictException(error[\'status\'],
url)\nUpdateConflictException: Error 409 at
http://127.0.0.1:8080/alfresco/api/-default-/public/cmis/versions/1.1/atom/children?id=4baddda1-87eb-45eb-8d0b-9958b8942c4f\n


I doubt if it is bug ... but if it is please let me know where I can start
to fix this.

Also, my pip freeze package versions are as below:

certifi==2018.4.16
chardet==3.0.4
cmislib==0.6.0
httplib2==0.11.3
idna==2.6
iso8601==0.1.12
requests==2.18.4
urllib3==1.22

I appreciate any clue/comment.

Best Regards,
Mike

Re: Does cmislib python support multi-threading?

Posted by Jeff Potts <je...@gmail.com>.

What I am seeing using cmislib 0.6.0 and Python 2.7.12 against Alfresco
201707-GA...

If you switch to the browser binding, and you comment out the
newDoc.updateProperties(props) call, your code runs successfully.

If you run with the browser binding and leave in the
newDoc.updateProperties(props) call you get a different problem.

If you run with the atompub binding you will get an update conflict.

If you add a delay with time.sleep(1) just before the createDocument call
the update conflict will go away.

Side note: You do not need to re-init client or repo. Those calls are
redundant. However, removing them makes no difference on the error
condition.

Jeff

On Tue, May 29, 2018 at 6:25 AM, Mignon, Laurent <la...@acsone.eu>
wrote:

> HI
>
> "from cmislib.browser import BrowserBinding" works in all my projects.
>
>
> On Mon, May 28, 2018 at 8:25 PM, Mike IT Expert <mi...@gmail.com>
> wrote:
>
> > I tried
> >
> > client =  cmislib.CmisClient(SERVER_URL, USER, PASSWD,
> > binding=cmislib.browser.BrowserBinding())
> >
> > But apparently it couldn't be found :
> >
> > Exception in thread Thread-15:
> > Traceback (most recent call last):
> >   File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
> >     self.run()
> >   File "cmislib_multi_threading_test_.py", line 40, in run
> >     client =  cmislib.CmisClient(SERVER_URL, USER, PASSWD,
> > binding=cmislib.browser.BrowserBinding())
> > AttributeError: module 'cmislib' has no attribute 'browser'
> >
> > Where is BrowserBinding?
> >
> >
> >
> >
> >
> > On Mon, May 28, 2018 at 5:04 PM, Mignon, Laurent <
> laurent.mignon@acsone.eu
> > > wrote:
> >
> >> By default the cmislib uses the 'atompub' binding. Can you try with the
> >> 'browser' binding.
> >>
> >> from cmislib.browser.binding import BrowserBinding
> >> client  = CmisClient(.., .., binding=BrowserBinding())
> >>
> >>
> >>
> >>
> >> On Mon, May 28, 2018 at 2:26 PM, Mike IT Expert <mikeitexpert@gmail.com
> >
> >> wrote:
> >>
> >> > Thanks you so much for quick resposne.
> >> >
> >> > Still the same no difference. (files from one thread goes to folder
> for
> >> > another thread)
> >> >
> >> > I checked out the code like below:
> >> >
> >> > git clone
> >> > git fetch origin pull/12/head
> >> > git checkout 1073eb533fc3478515f4af3a6f779b63c1b651b2
> >> >
> >> >
> >> > I used Python 3.6.
> >> >
> >> > I don't think it is related to Alfresco, I used the same logic for
> >> > multi-threading using REST API it works just fine.
> >> >
> >> > Thanks,
> >> > Mike
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On Mon, May 28, 2018 at 12:26 PM, Mignon, Laurent <
> >> > laurent.mignon@acsone.eu>
> >> > wrote:
> >> >
> >> > > Hi Mike,
> >> > >
> >> > > I'm sorry but I've no time to investigate if the problem is on the
> >> > cmislib
> >> > > side or on the alfresco side. Nevertheless, I recently completely
> >> > reworked
> >> > > the cmislib library to make it py2 py3 compatible and at the same
> >> time I
> >> > > replaced the dependency on httplib2 by requests. (
> >> > > https://github.com/apache/chemistry-cmislib/pull/12)
> >> > > It could be interesting to make a try with this new version of the
> >> > library
> >> > > and see if the problem is also present into this new one. If the
> >> problem
> >> > > comes from cmislib, personally I would prefer to correct it in this
> >> new
> >> > > version and speed up thus its release.
> >> > >
> >> > > Regards,
> >> > >
> >> > > Laurent Mignon
> >> > >
> >> > >
> >> > > On Sat, May 26, 2018 at 9:42 PM, Mike IT Expert <
> >> mikeitexpert@gmail.com>
> >> > > wrote:
> >> > >
> >> > > > Hi Laurent,
> >> > > >
> >> > > > Please let me know if there has been update on this. I have added
> my
> >> > > > employer just to see there is anything we can do on our side to
> make
> >> > the
> >> > > > cmislib thread safe.
> >> > > >
> >> > > > Regards,
> >> > > > M
> >> > > >
> >> > > > On Fri, May 25, 2018 at 7:12 PM, Mike IT Expert <
> >> > mikeitexpert@gmail.com>
> >> > > > wrote:
> >> > > >
> >> > > > >
> >> > > > > Apparently the exception doesn't occur if I use separate parent
> >> > folder
> >> > > > for
> >> > > > > each thread, but the new documents SOMETIMES are created under
> >> > > incorrect
> >> > > > > folder (created by other threads).
> >> > > > >
> >> > > > > Attached is the source code but I cleaned up the code a little
> and
> >> > took
> >> > > > > out unrelated stuff of mine.
> >> > > > >
> >> > > > > Please let me know the outcome ....
> >> > > > >
> >> > > > > mike
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > On Fri, May 25, 2018 at 5:59 PM, Mignon, Laurent <
> >> > > > laurent.mignon@acsone.eu
> >> > > > > > wrote:
> >> > > > >
> >> > > > >> Not sure if it's the root cause but new files are added at same
> >> time
> >> > > > into
> >> > > > >> the same parent folder. That means that the parent folder is
> >> updated
> >> > > at
> >> > > > >> same time.
> >> > > > >> Do you have the same problem if each thread create files in its
> >> own
> >> > > > parent
> >> > > > >> folder?
> >> > > > >>
> >> > > > >> lmignon
> >> > > > >>
> >> > > > >> On Fri, May 25, 2018 at 1:52 PM, Mike IT Expert <
> >> > > mikeitexpert@gmail.com
> >> > > > >
> >> > > > >> wrote:
> >> > > > >>
> >> > > > >> > You are right but my script does't update the same
> >> > resource/document
> >> > > > >> > simultaneously because each thread uploads new files under
> new
> >> > > unique
> >> > > > >> names
> >> > > > >> > on the server, in that each thread
> >> > > > >> >
> >> > > > >> > *1) creates its own instance of the repository*
> >> > > > >> >
> >> > > > >> > * # 2) upload files and update properties*
> >> > > > >> > * # connect to the server *
> >> > > > >> > * client =  cmislib.CmisClient(SERVER_URL, USER, PASSWD)*
> >> > > > >> >
> >> > > > >> > * # get default repository*
> >> > > > >> > * repo = client.defaultRepository*
> >> > > > >> >
> >> > > > >> > *2) retrieves the target folder's node id*
> >> > > > >> >
> >> > > > >> > * # get the target folder to upload new documents into*
> >> > > > >> > * results = repo.query("select * from cmis:folder where
> >> cmis:name
> >> > =
> >> > > > >> '%s'"%(
> >> > > > >> > TARGET_ALFRESCO_FOLDER, ))*
> >> > > > >> > * targetFolder = results[0]*
> >> > > > >> > * targetFolder = repo.getObject( targetFolder.id )*
> >> > > > >> >
> >> > > > >> > *3) for each file (eg. file_i ) in a loop python statement*
> >> > > > >> >
> >> > > > >> >
> >> > > > >> >
> >> > > > >> > * newDoc = targetFolder.createDocument( file_i['docName'],
> >> > > > contentFile =
> >> > > > >> > inFile ) *
> >> > > > >> >
> >> > > > >> > * props = {*
> >> > > > >> > * u'cm:title': file_i[ HDR_TITLE ],*
> >> > > > >> > * u'cm:description': file_i[ HDR_DESCR ],*
> >> > > > >> > * }*
> >> > > > >> >
> >> > > > >> >
> >> > > > >> > * newDoc.updateProperties(props)*
> >> > > > >> >
> >> > > > >> >
> >> > > > >> > So what I am doing is that I access the same folder and
> upload
> >> new
> >> > > > >> > documents using unique names. I am 100% sure the document
> names
> >> > (ie.
> >> > > > >> > * file_i['docName']*) are unique under the same folder.
> >> > > > >> >
> >> > > > >> > It just doesn't seem to me to be any concurrency problem with
> >> > above
> >> > > > >> code.
> >> > > > >> >
> >> > > > >> >
> >> > > > >> > Many thanks anyways,
> >> > > > >> > M
> >> > > > >> >
> >> > > > >> >
> >> > > > >> > On Fri, May 25, 2018 at 3:57 PM, Mignon, Laurent <
> >> > > > >> laurent.mignon@acsone.eu
> >> > > > >> > >
> >> > > > >> > wrote:
> >> > > > >> >
> >> > > > >> > > Hi Mike,
> >> > > > >> > >
> >> > > > >> > > I'm not aware of a design issue that could prevent to use
> >> > cmislib
> >> > > > >> python
> >> > > > >> > > into multi threads.
> >> > > > >> > > The problem you describe is a normal behaviour when two
> >> actors
> >> > > > (human
> >> > > > >> or
> >> > > > >> > > prog) update the same resource at same time into alfresco.
> >> > > > >> > >
> >> > > > >> > > Regards,
> >> > > > >> > >
> >> > > > >> > > lmignon
> >> > > > >> > >
> >> > > > >> > > On Fri, May 25, 2018 at 12:19 PM, Mike IT Expert <
> >> > > > >> mikeitexpert@gmail.com
> >> > > > >> > >
> >> > > > >> > > wrote:
> >> > > > >> > >
> >> > > > >> > >> Dear All,
> >> > > > >> > >>
> >> > > > >> > >> Apologies in advance for sending this message to dev
> mailing
> >> > list
> >> > > > (I
> >> > > > >> > wish
> >> > > > >> > >> there was userlist also I wasn't sure it is bug or not.)
> >> > > > >> > >>
> >> > > > >> > >> I have started working with cmislib since past week using
> >> > > Alfresco
> >> > > > as
> >> > > > >> > the
> >> > > > >> > >> local server.
> >> > > > >> > >>
> >> > > > >> > >> My goal is to upload documents provided by json rcord list
> >> in a
> >> > > > for
> >> > > > >> > loop
> >> > > > >> > >> as you see below *which is part of run-overiden method of
> >> > > > >> > >> threading.Thread.run method for python 2.7*.
> >> > > > >> > >>
> >> > > > >> > >> The main code is attached just in case.
> >> > > > >> > >>
> >> > > > >> > >>
> >> > > > >> > >>
> >> > > > >> > >> The above code works find using ONE thread only. If I use
> >> more
> >> > > than
> >> > > > >> one
> >> > > > >> > >> thread I get the below exception for some records which is
> >> odd.
> >> > > > >> > >>
> >> > > > >> > >>
> >> > > > >> > >> File "module_proj_2_parts_1_and_2.py", line 82, in run
> >> > > > >> > >>     newDoc = targetFolder.createDocument( docName,
> >> contentFile
> >> > =
> >> > > > >> inFile
> >> > > > >> > )
> >> > > > >> > >>   File "/home/mike/Desktop/hassan_
> proj2/.pyenv2/local/lib/
> >> > > python2.
> >> > > > >> > >> 7/site-packages/cmislib/atompub/binding.py", line 2788,
> in
> >> > > > >> > createDocument
> >> > > > >> > >>     contentEncoding)
> >> > > > >> > >>   File "/home/mike/Desktop/hassan_
> proj2/.pyenv2/local/lib/
> >> > > python2.
> >> > > > >> > >> 7/site-packages/cmislib/atompub/binding.py", line 1807,
> in
> >> > > > >> > createDocument
> >> > > > >> > >>     ATOM_XML_ENTRY_TYPE)
> >> > > > >> > >>   File "/home/mike/Desktop/hassan_
> proj2/.pyenv2/local/lib/
> >> > > python2.
> >> > > > >> > >> 7/site-packages/cmislib/atompub/binding.py", line 188, in
> >> post
> >> > > > >> > >>     self._processCommonErrors(resp, url)
> >> > > > >> > >>   File "/home/mike/Desktop/hassan_
> proj2/.pyenv2/local/lib/
> >> > > python2.
> >> > > > >> > >> 7/site-packages/cmislib/cmis_services.py", line 63, in
> >> > > > >> > >> _processCommonErrors
> >> > > > >> > >>     raise UpdateConflictException(error[\'status\'],
> >> > > > >> > >> url)\nUpdateConflictException: Error 409 at
> >> > > > >> > >> http://127.0.0.1:8080/alfresco/api/-default-/public/cmis/
> >> > > > >> > >> versions/1.1/atom/children?id=4baddda1-87eb-45eb-8d0b-
> >> > > > 9958b8942c4f\n
> >> > > > >> > >>
> >> > > > >> > >>
> >> > > > >> > >> I doubt if it is bug ... but if it is please let me know
> >> where
> >> > I
> >> > > > can
> >> > > > >> > >> start to fix this.
> >> > > > >> > >>
> >> > > > >> > >> Also, my pip freeze package versions are as below:
> >> > > > >> > >>
> >> > > > >> > >> certifi==2018.4.16
> >> > > > >> > >> chardet==3.0.4
> >> > > > >> > >> cmislib==0.6.0
> >> > > > >> > >> httplib2==0.11.3
> >> > > > >> > >> idna==2.6
> >> > > > >> > >> iso8601==0.1.12
> >> > > > >> > >> requests==2.18.4
> >> > > > >> > >> urllib3==1.22
> >> > > > >> > >>
> >> > > > >> > >> I appreciate any clue/comment.
> >> > > > >> > >>
> >> > > > >> > >> Best Regards,
> >> > > > >> > >> Mike
> >> > > > >> > >>
> >> > > > >> > >>
> >> > > > >> > >>
> >> > > > >> > >>
> >> > > > >> > >>
> >> > > > >> > >
> >> > > > >> > >
> >> > > > >> > > --
> >> > > > >> > > *Laurent Mignon*
> >> > > > >> > > Senior software engineer
> >> > > > >> > >
> >> > > > >> > > Tel: +352 20 21 10 20 32
> >> > > > >> > > Fax: +352 20 21 10 21
> >> > > > >> > > Email: laurent.mignon@acsone.eu
> >> > > > >> > >
> >> > > > >> > > Acsone SA, Succursale de Luxembourg
> >> > > > >> > > 22, Zone Industrielle
> >> > > > >> > >  
> >> > > > >> > > | L-8287 Kehlen, Luxembourg
> >> > > > >> > > TVA LU24733605
> >> > > > >> > >  | RCS B160400
> >> > > > >> > >
> >> > > > >> > > 
> >> > > > >> > > Acsone sa/nv
> >> > > > >> > > Waterloo Atrium, Drève Richelle 167
> >> <https://maps.google.com/?q=Dr%C3%A8ve+Richelle+167&entry=
> gmail&source=g>
> >> > > > >> > > <https://maps.google.com/?q=Dr%C3%A8ve+Richelle+167&entry=
> gm
> >> > > > >> ail&source=g
> >> > > > >> > >
> >> > > > >> > >  | B-1410 Waterloo
> >> > > > >> > > RPM Bruxelles 0835.207.216 RPR Brussel
> >> > > > >> > >
> >> > > > >> > > www.acsone.eu
> >> > > > >> > >
> >> > > > >> > >
> >> > > > >> > >
> >> > > > >> > >
> >> > > > >> >
> >> > > > >>
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> > >
> >> > >
> >> > > --
> >> > > *Laurent Mignon*
> >> > > Senior software engineer
> >> > >
> >> > > Tel: +352 20 21 10 20 32
> >> > > Fax: +352 20 21 10 21
> >> > > Email: laurent.mignon@acsone.eu
> >> > >
> >> > > Acsone SA, Succursale de Luxembourg
> >> > > 22, Zone Industrielle
> >> > >  
> >> > > | L-8287 Kehlen, Luxembourg
> >> > > TVA LU24733605
> >> > >  | RCS B160400
> >> > >
> >> > > 
> >> > > Acsone sa/nv
> >> > > Waterloo Atrium, Drève Richelle 167
> >> <https://maps.google.com/?q=Dr%C3%A8ve+Richelle+167+%0D%
> 0A+%C2%A0+%7C+B-1410+Waterloo&entry=gmail&source=g>
> >> > >  | B-1410 Waterloo
> >> <https://maps.google.com/?q=Dr%C3%A8ve+Richelle+167+%0D%
> 0A+%C2%A0+%7C+B-1410+Waterloo&entry=gmail&source=g>
> >> > > RPM Bruxelles 0835.207.216 RPR Brussel
> >> > >
> >> > > www.acsone.eu
> >> > >
> >> >
> >>
> >>
> >>
> >> --
> >> *Laurent Mignon*
> >> Senior software engineer
> >>
> >> Tel: +352 20 21 10 20 32
> >> Fax: +352 20 21 10 21
> >> Email: laurent.mignon@acsone.eu
> >>
> >> Acsone SA, Succursale de Luxembourg
> >> 22, Zone Industrielle
> >>  
> >> | L-8287 Kehlen, Luxembourg
> >> TVA LU24733605
> >>  | RCS B160400
> >>
> >> 
> >> Acsone sa/nv
> >> Waterloo Atrium, Drève Richelle 167
> >> <https://maps.google.com/?q=Dr%C3%A8ve+Richelle+167+%0D%
> 0A%C2%A0%7C+B-1410+Waterloo&entry=gmail&source=g>
> >>  | B-1410 Waterloo
> >> RPM Bruxelles 0835.207.216 RPR Brussel
> >>
> >> www.acsone.eu
> >>
> >
> >
>
>
> --
> *Laurent Mignon*
> Senior software engineer
>
> Tel: +352 20 21 10 20 32
> Fax: +352 20 21 10 21
> Email: laurent.mignon@acsone.eu
>
> Acsone SA, Succursale de Luxembourg
> 22, Zone Industrielle
>  
> | L-8287 Kehlen, Luxembourg
> TVA LU24733605
>  | RCS B160400
>
> 
> Acsone sa/nv
> Waterloo Atrium, Drève Richelle 167
>  | B-1410 Waterloo
> RPM Bruxelles 0835.207.216 RPR Brussel
>
> www.acsone.eu
>