You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@chemistry.apache.org by "Sethi, Rajendra" <Ra...@bchydro.com> on 2014/09/20 03:22:34 UTC
DotCMIS query
Hi there,
I am working on SharePoint 2010 (CMIS client) to FileNet P8 5.1 (CMIS Server 1.0) content transfer using CMIS and here we are implementing DotCMIS library in SharePoint to initiate file transfer requests targeted for FileNet. We have large files transfer scenarios and while doing some performance tests, memory consumption on SharePoint (the client) is too high. I would like your help to suggest how could I improve these transfers.
The implementation is using AtomPub binding and streaming the file content from a document available in SharePoint library. During the SetContentStream operation, Modified Physical Memory equivalent to the size of the file being transferred becomes Not Available (for example, for 2 gb file, about 2 gb memory becomes Not Available) and hence the Free Memory on the server is reduced by that file size. Once the transfer operation is completed, this physical memory becomes available once again.
We are trying to achieve a high performance solution and may consider multiple threads to transfer a few files simultaneously. If each process is blocking a large chunk of memory, we will be limited to 1 or max 2 threads at a time. Is there any code tuning or correction required in the connection/session object to achieve better results? Please consider the code provided below.
This is the code from CMIS client (SharePoint ) ---
Dictionary<string, string> parameters = new Dictionary<string, string>();
// RESTful
parameters[DotCMIS.SessionParameter.BindingType] = BindingType.AtomPub;
parameters[DotCMIS.SessionParameter.AtomPubUrl] = "<AtomPubUrl>";
parameters[DotCMIS.SessionParameter.RepositoryId] = "<RepositoryID>";
parameters[DotCMIS.SessionParameter.User] = "<username>";
parameters[DotCMIS.SessionParameter.Password] = "<password>";
parameters[DotCMIS.SessionParameter.Compression] = "true"; // turn on HTTP response compression
parameters[DotCMIS.SessionParameter.ConnectTimeout] = "600000"; // 10 minutes // HTTP connect timeout in milliseconds; -1 for infinite
parameters[DotCMIS.SessionParameter.ReadTimeout] = "600000"; // 10 minutes // -1 for infinite
SessionFactory factory = SessionFactory.NewInstance();
session = factory.CreateSession(parameters);
IFolder cmisFolder = (IFolder)session.GetObjectByPath("/cmis test");
SPFile file = SPContext.Current.Web.GetFile("http://<server>/sites/TS/Shared%20Documents/bigfile.txt");
Dictionary<string, object> properties = new Dictionary<string, object>()
{
{"cmis:name", file.Title },
{"cmis:objectTypeId", "cmis:document"},
};
Stream stream = file.OpenBinaryStream(SPOpenBinaryOptions.SkipVirusScan);
IDocument doc = cmisFolder.CreateDocument(properties, null, VersioningState.CheckedOut);
IObjectFactory objectFactory = session.ObjectFactory;
ContentStream contentStream = (ContentStream)objectFactory.CreateContentStream(file.Title, file.Length, "text/plain", stream);
doc.SetContentStream(contentStream, true, false);
doc.CheckIn(true, null, null, "optional check in comment");
Thanks
Rajendra Sethi
________________________________
This email and its attachments are intended solely for the personal use of the individual or entity named above. Any use of this communication by an unintended recipient is strictly prohibited. If you have received this email in error, any publication, use, reproduction, disclosure or dissemination of its contents is strictly prohibited. Please immediately delete this message and its attachments from your computer and servers. We would also appreciate if you would contact us by a collect call or return email to notify us of this error. Thank you for your cooperation.
-BCHydroDisclaimerID5.2.8.1541
Re: DotCMIS query
Posted by Sascha Homeier <sh...@meyle-mueller.de>.
Hi Rajendra,
This sounds like the effect Florian told me about when using non chunked Transfer encoding (due to the required length the document is prepared in Memory at whole).
Maybe a Transfer in chunks will help you. I do not know about DotCmis but Maybe there is an Option to create a partial content Stream or sth. In that way.
Cheers
Sascha
> Am 20.09.2014 um 10:58 schrieb "Sethi, Rajendra" <Ra...@bchydro.com>:
>
> Hi there,
>
> I am working on SharePoint 2010 (CMIS client) to FileNet P8 5.1 (CMIS Server 1.0) content transfer using CMIS and here we are implementing DotCMIS library in SharePoint to initiate file transfer requests targeted for FileNet. We have large files transfer scenarios and while doing some performance tests, memory consumption on SharePoint (the client) is too high. I would like your help to suggest how could I improve these transfers.
>
> The implementation is using AtomPub binding and streaming the file content from a document available in SharePoint library. During the SetContentStream operation, Modified Physical Memory equivalent to the size of the file being transferred becomes Not Available (for example, for 2 gb file, about 2 gb memory becomes Not Available) and hence the Free Memory on the server is reduced by that file size. Once the transfer operation is completed, this physical memory becomes available once again.
>
> We are trying to achieve a high performance solution and may consider multiple threads to transfer a few files simultaneously. If each process is blocking a large chunk of memory, we will be limited to 1 or max 2 threads at a time. Is there any code tuning or correction required in the connection/session object to achieve better results? Please consider the code provided below.
>
> This is the code from CMIS client (SharePoint ) ---
>
> Dictionary<string, string> parameters = new Dictionary<string, string>();
>
> // RESTful
> parameters[DotCMIS.SessionParameter.BindingType] = BindingType.AtomPub;
> parameters[DotCMIS.SessionParameter.AtomPubUrl] = "<AtomPubUrl>";
>
> parameters[DotCMIS.SessionParameter.RepositoryId] = "<RepositoryID>";
> parameters[DotCMIS.SessionParameter.User] = "<username>";
> parameters[DotCMIS.SessionParameter.Password] = "<password>";
>
> parameters[DotCMIS.SessionParameter.Compression] = "true"; // turn on HTTP response compression
> parameters[DotCMIS.SessionParameter.ConnectTimeout] = "600000"; // 10 minutes // HTTP connect timeout in milliseconds; -1 for infinite
> parameters[DotCMIS.SessionParameter.ReadTimeout] = "600000"; // 10 minutes // -1 for infinite
>
> SessionFactory factory = SessionFactory.NewInstance();
> session = factory.CreateSession(parameters);
>
>
> IFolder cmisFolder = (IFolder)session.GetObjectByPath("/cmis test");
>
> SPFile file = SPContext.Current.Web.GetFile("http://<server>/sites/TS/Shared%20Documents/bigfile.txt");
> Dictionary<string, object> properties = new Dictionary<string, object>()
> {
> {"cmis:name", file.Title },
> {"cmis:objectTypeId", "cmis:document"},
> };
>
> Stream stream = file.OpenBinaryStream(SPOpenBinaryOptions.SkipVirusScan);
>
> IDocument doc = cmisFolder.CreateDocument(properties, null, VersioningState.CheckedOut);
>
> IObjectFactory objectFactory = session.ObjectFactory;
> ContentStream contentStream = (ContentStream)objectFactory.CreateContentStream(file.Title, file.Length, "text/plain", stream);
>
> doc.SetContentStream(contentStream, true, false);
> doc.CheckIn(true, null, null, "optional check in comment");
>
>
> Thanks
> Rajendra Sethi
> ________________________________
> This email and its attachments are intended solely for the personal use of the individual or entity named above. Any use of this communication by an unintended recipient is strictly prohibited. If you have received this email in error, any publication, use, reproduction, disclosure or dissemination of its contents is strictly prohibited. Please immediately delete this message and its attachments from your computer and servers. We would also appreciate if you would contact us by a collect call or return email to notify us of this error. Thank you for your cooperation.
> -BCHydroDisclaimerID5.2.8.1541
Re: DotCMIS query
Posted by Florian Müller <fm...@apache.org>.
Hi Rajendra,
Your code looks good.
DotCMIS doesn't buffer the content. It reads 4k blocks from the stream
and sends them. HTTP chunking is turned on by default. That is, also the
HTTP client shouldn't (!) buffer the content.
Does SPFile.OpenBinaryStream buffer the content under some
circumstances?
- Florian
> Hi there,
>
> I am working on SharePoint 2010 (CMIS client) to FileNet P8 5.1 (CMIS
> Server 1.0) content transfer using CMIS and here we are implementing
> DotCMIS library in SharePoint to initiate file transfer requests
> targeted for FileNet. We have large files transfer scenarios and while
> doing some performance tests, memory consumption on SharePoint (the
> client) is too high. I would like your help to suggest how could I
> improve these transfers.
>
> The implementation is using AtomPub binding and streaming the file
> content from a document available in SharePoint library. During the
> SetContentStream operation, Modified Physical Memory equivalent to the
> size of the file being transferred becomes Not Available (for example,
> for 2 gb file, about 2 gb memory becomes Not Available) and hence the
> Free Memory on the server is reduced by that file size. Once the
> transfer operation is completed, this physical memory becomes
> available once again.
>
> We are trying to achieve a high performance solution and may consider
> multiple threads to transfer a few files simultaneously. If each
> process is blocking a large chunk of memory, we will be limited to 1
> or max 2 threads at a time. Is there any code tuning or correction
> required in the connection/session object to achieve better results?
> Please consider the code provided below.
>
> This is the code from CMIS client (SharePoint ) ---
>
> Dictionary<string, string> parameters = new Dictionary<string,
> string>();
>
> // RESTful
> parameters[DotCMIS.SessionParameter.BindingType] = BindingType.AtomPub;
> parameters[DotCMIS.SessionParameter.AtomPubUrl] = "<AtomPubUrl>";
>
> parameters[DotCMIS.SessionParameter.RepositoryId] = "<RepositoryID>";
> parameters[DotCMIS.SessionParameter.User] = "<username>";
> parameters[DotCMIS.SessionParameter.Password] = "<password>";
>
> parameters[DotCMIS.SessionParameter.Compression] = "true"; // turn on
> HTTP response compression
> parameters[DotCMIS.SessionParameter.ConnectTimeout] = "600000"; // 10
> minutes // HTTP connect timeout in milliseconds; -1 for infinite
> parameters[DotCMIS.SessionParameter.ReadTimeout] = "600000"; // 10
> minutes // -1 for infinite
>
> SessionFactory factory = SessionFactory.NewInstance();
> session = factory.CreateSession(parameters);
>
>
> IFolder cmisFolder = (IFolder)session.GetObjectByPath("/cmis test");
>
> SPFile file =
> SPContext.Current.Web.GetFile("http://<server>/sites/TS/Shared%20Documents/bigfile.txt");
> Dictionary<string, object> properties = new Dictionary<string,
> object>()
> {
> {"cmis:name", file.Title },
> {"cmis:objectTypeId", "cmis:document"},
> };
>
> Stream stream =
> file.OpenBinaryStream(SPOpenBinaryOptions.SkipVirusScan);
>
> IDocument doc = cmisFolder.CreateDocument(properties, null,
> VersioningState.CheckedOut);
>
> IObjectFactory objectFactory = session.ObjectFactory;
> ContentStream contentStream =
> (ContentStream)objectFactory.CreateContentStream(file.Title,
> file.Length, "text/plain", stream);
>
> doc.SetContentStream(contentStream, true, false);
> doc.CheckIn(true, null, null, "optional check in comment");
>
>
> Thanks
> Rajendra Sethi
> ________________________________
> This email and its attachments are intended solely for the personal
> use of the individual or entity named above. Any use of this
> communication by an unintended recipient is strictly prohibited. If
> you have received this email in error, any publication, use,
> reproduction, disclosure or dissemination of its contents is strictly
> prohibited. Please immediately delete this message and its attachments
> from your computer and servers. We would also appreciate if you would
> contact us by a collect call or return email to notify us of this
> error. Thank you for your cooperation.
> -BCHydroDisclaimerID5.2.8.1541