You are viewing a plain text version of this content. The canonical link for it is here.
Posted to nmaven-dev@incubator.apache.org by James Carpenter <jc...@yahoo.com> on 2008/09/23 16:19:28 UTC
Research into Symbol and Source Servers
Overview:
I have been doing some reading/research to better understand symbol servers and source servers. I realize some readers may already be familiar with the topic, but for those that are not the information may prove useful.
To provide the ability to easily step down into source code of referenced libraries within Visual Studio, NMaven will need to integrate with Microsoft's Source and Symbol server functionality.
A high level overview can be found using the following links:
http://msdn.microsoft.com/en-us/magazine/cc301459.aspx
http://msdn.microsoft.com/en-us/magazine/cc163563.aspx
===================================
Additional Useful Information:
The source server directory in the "Debugging Tools for Windows" distribution contains a particularly useful MS Word document which provides a great deal of the details missing from the articles above. The section on "HTTP Sites and UNC Shares" is particularly interesting as it provides an alternate way to easily support source server functionality. Assuming you have installed "Debugging Tools for Windows" in the standard location the MS Word document can be found at:
C:\Program Files\Debugging Tools for Windows (x86)\srcsrv\srcsrv.doc
Note: You will need to install "Debugging Tools for Windows".
----------------------------------------
The MS Help files installed along with "Debugging Tools for Windows" as well as most other documentation don't really cover the use of multiple symbol servers well. The omission can lead one to the false conclusion only a single remote symbol server is supported. From Chapter 4 (Managing Symbol and Source Files) of Advanced Windows Debugging by Daniel Pravat reveals one can include as many remote symbol servers in the sympath as desired. These can also be URLs not just UNC paths. (I may have found this in the MS Word document mentioned above.)
Title: Advanced Windows Debugging
by: Mario Hewardt; Daniel Pravat
Publisher: Addison Wesley Professional
Pub Date: October 29, 2007
Print ISBN-10: 0-321-37446-0
Print ISBN-13: 978-0-321-37446-2
(Available on Safari.)
---------------------------------------
When you store a pdb in the symbol server
e.g.: prompt>symstore add /r /f C:\junk2\bin\*.* /s C:\symbolstore\ /t "MyApp" /v "Build 632"
the symstore will create a directory structure which contains a GUID extracted from the PDB. As it turns out a PDB is given a GUID when it is created. One way to read the GUID is to use the dbh tool.
e.g.: C:\junk2\bin>dbh -v srvind MySymbol.pdb
Details on the dbh tool can be found in the MS Help files shipped with the "Debugging Tools for Windows".
Note: Examples assume C:\Program Files\Debugging Tools for Windows (x86) is on your path.
===============================================
Supporting Symbol/Source Servers in NMaven
To properly support symbol and source server functionality in NMaven there appear to be two good choices I can see. These are:
--------------
Option 1:
a) Create a plugin(s) which downloads the pdb files of every dependency (saved in the mvn repo as attached artifacts) and executes symstore to save the pdb files in a local symbol store.
b) Create a plugin(s) which downloads the source archives of every dependency (saved in the mvn repo as attached artifacts) and expands them into a directory structure matching that of a UNC store. (See "HTTP Sites and UNC Shares" section of srcsrv.doc.)
c) Ensure the pdb files are properly processed to include the necessary source stream during or before the deploy phase. It is important SNAPSHOT artifacts include this information so one can't wait for the release plugin to do the job.
--------------
Option 2:
a) Same as Option 1.a
b) Process the pdb files during/before the deploy phase to include the information necessary to extract a single source file. Under the covers the command line call formed when VS processed the source stream from the pdb file will need to resolve the relevant source archive and extract the relevant file from the source archive. The best way to do this is likely to work out a one line mvn call which will return the relevant source file, creating a plugin or two as necessary. The command line call formed when processing the source stream in the pdb file will therefore simply be a call into maven.
-----------------
Note:
Careful examination of the perl scripts included in C:\Program Files\Debugging Tools for Windows (x86)\srcsrv\ and documented in the associated MS Word document should be enough to understand how the pdb processing works in both Option 1 and 2.
-----------------
Effort Estimate:
Option 1 may be slightly easier to implement, but option 2 will probably be a tad more elegant. It wouldn't be that much more effort to support both. I would guess either approach will probably take a week or two of work to complete, including testing.
Re: Research into Symbol and Source Servers
Posted by James Carpenter <jc...@yahoo.com>.
I've done a little more work with adding source stream information to
pdb files. As it turns out visual studio only uses the source server
functionality during debugging. It is too stupid to use it outside of
the debugger. So unlike most Java IDE's which can navigate into the
source code of referenced libraries (as long as the source jars are
provided), Visual Studio can only step into the source code of a
referenced library when debugging.
Assuming the hooks are there, a VS addin could probably add this
functionality.
On Sep 24, 2008, at 5:27 PM, James Carpenter wrote:
> Creation of the source stream to insert into the pdb files requires
> an understanding of which source files are associated with a given
> class/type in the associated assembly. Unlike Java, C# does not
> require the directory/filename structure to match the namespace/
> classname. Is this information available as output from csc? Is
> there a utility in nmaven which already understands this association?
>
> Although I haven't looked closely, I believe the perl files which
> come with srcsrv are doing a simple search of the source files.
> Generically speaking it would be much better to use information
> output by the relevant .NET compiler.
Sincerely,
James Carpenter
cell: 832-677-7247
email: jcarpenter621@yahoo.com
Re: Research into Symbol and Source Servers
Posted by James Carpenter <jc...@yahoo.com>.
Creation of the source stream to insert into the pdb files requires an understanding of which source files are associated with a given class/type in the associated assembly. Unlike Java, C# does not require the directory/filename structure to match the namespace/classname. Is this information available as output from csc? Is there a utility in nmaven which already understands this association?
Although I haven't looked closely, I believe the perl files which come with srcsrv are doing a simple search of the source files. Generically speaking it would be much better to use information output by the relevant .NET compiler.
Re: Research into Symbol and Source Servers
Posted by James Carpenter <jc...@yahoo.com>.
Option 2b and option 1c both result in very little extra space in the PDB. In neither case does the PDB contain the source.
I haven't gotten my hands dirty yet, but basically there is just a little bit of meta-data inserted into the PDB for source server support in either case. This meta-data is just enough to teach the debugger (Visual Studio/etc.) how to acquire the source.
I recommend the following approach to getting your head around this.
1) Install "Debugging Tools for Windows". Add the binaries to your PATH.
2) Insert a few random pdb files into the symbol server. (See example command from my original post.) Take a look at the structure created.
3) Skim the two online articles:
http://msdn.microsoft.com/en-us/magazine/cc301459.aspx http://msdn.microsoft.com/en-us/magazine/cc163563.aspx
4) Carefully read the MS Word document, skimming the contents of the srcsrv directory and the associated perl modules as you do so.
C:\Program Files\Debugging Tools for Windows (x86)\srcsrv\srcsrv.doc
5) Skim the relevant chapter in the book I mentioned if you have easy/inexpensive access to it. Otherwise, just skip it.
6) Briefly look at the symbol/source server section of the MS help which ships with "Debugging Tools for Windows".
>Would the client end of this be easier to achieve if it were managed from Visual Studio integration itself?
I don't think so. Apparently, symbol and source server support as described by the referenced documents is the standard Microsoft approach to supporting this sort of functionality. Microsoft supplies several debuggers (command line and GUI) which all appear to be integrated with the symbol and source server mechanism described. Using another approach would be swimming upstream. I think you need the enterprise license of VS to get source server integration, but such is life.
The only functionality a VS plugin might add would be the ability to easily browse sources associated with a project resource. Under the covers the plugin would still be using the symbol/source server support to get at the files. This sort of browsing is where option 1b might be useful. As I mentioned orginally, option 1 and 2 can probably peacefully coexist. The MS Word document sort of implies they could.
My personal opinion is that Microsoft should enhance the resource reference mechanism in Visual Studio and msbuild to support a proper artifact repository. If MS supplied tools did for .NET what maven and maven IDE plugins do for Java, there would be no compelling reason to leave the MS camp. Companies would be best served by simply paying for Team Server and getting on with life. Currently, MS hasn't done any of this so integrating with the symbol and source server support as it exists today is the way to go. If MS does enhance things, I suspect they will do so by enhancing the existing symbol server support, source server support and VS integration with the same.
--- On Tue, 9/23/08, Brett Porter <br...@apache.org> wrote:
> From: Brett Porter <br...@apache.org>
> Subject: Re: Research into Symbol and Source Servers
> To: nmaven-dev@incubator.apache.org
> Date: Tuesday, September 23, 2008, 6:43 PM
> Thanks for this James. I have some questions...
>
> On 24/09/2008, at 2:19 AM, James Carpenter wrote:
>
> > c) Ensure the pdb files are properly processed to
> include the
> > necessary source stream during or before the deploy
> phase. It is
> > important SNAPSHOT artifacts include this information
> so one can't
> > wait for the release plugin to do the job.
> >
> > --------------
> > Option 2:
> >
> > a) Same as Option 1.a
> >
> > b) Process the pdb files during/before the deploy
> phase to include
> > the information necessary to extract a single source
> file. Under
> > the covers the command line call formed when VS
> processed the source
> > stream from the pdb file will need to resolve the
> relevant source
> > archive and extract the relevant file from the source
> archive. The
> > best way to do this is likely to work out a one line
> mvn call which
> > will return the relevant source file, creating a
> plugin or two as
> > necessary. The command line call formed when
> processing the source
> > stream in the pdb file will therefore simply be a call
> into maven.
>
> Does option 2 (b) result in a larger PDB file than 1 (c)?
> IIUC, you
> are basically including the entire source in there, vs a
> reference to
> a source which would be stored elsewhere.
>
> If so, option 1 seems slightly more efficient from the
> repository's
> perspective (as having a source archive could make sense
> regardless).
> But it depends on how much complexity this adds on the
> client side.
>
> Would the client end of this be easier to achieve if it
> were managed
> from Visual Studio integration itself?
>
> Thanks,
> Brett
>
> --
> Brett Porter
> brett@apache.org
> http://blogs.exist.com/bporter/
Re: Research into Symbol and Source Servers
Posted by Brett Porter <br...@apache.org>.
Thanks for this James. I have some questions...
On 24/09/2008, at 2:19 AM, James Carpenter wrote:
> c) Ensure the pdb files are properly processed to include the
> necessary source stream during or before the deploy phase. It is
> important SNAPSHOT artifacts include this information so one can't
> wait for the release plugin to do the job.
>
> --------------
> Option 2:
>
> a) Same as Option 1.a
>
> b) Process the pdb files during/before the deploy phase to include
> the information necessary to extract a single source file. Under
> the covers the command line call formed when VS processed the source
> stream from the pdb file will need to resolve the relevant source
> archive and extract the relevant file from the source archive. The
> best way to do this is likely to work out a one line mvn call which
> will return the relevant source file, creating a plugin or two as
> necessary. The command line call formed when processing the source
> stream in the pdb file will therefore simply be a call into maven.
Does option 2 (b) result in a larger PDB file than 1 (c)? IIUC, you
are basically including the entire source in there, vs a reference to
a source which would be stored elsewhere.
If so, option 1 seems slightly more efficient from the repository's
perspective (as having a source archive could make sense regardless).
But it depends on how much complexity this adds on the client side.
Would the client end of this be easier to achieve if it were managed
from Visual Studio integration itself?
Thanks,
Brett
--
Brett Porter
brett@apache.org
http://blogs.exist.com/bporter/
Re: Research into Symbol and Source Servers
Posted by James Carpenter <jc...@yahoo.com>.
Oops, I meant to say I created NMAVEN-213 to track symbol and source server support.
--- On Tue, 9/23/08, James Carpenter <jc...@yahoo.com> wrote:
> From: James Carpenter <jc...@yahoo.com>
> Subject: Re: Research into Symbol and Source Servers
> To: nmaven-dev@incubator.apache.org
> Date: Tuesday, September 23, 2008, 2:46 PM
> I have created JIRA issue NMAVEN-120 to address
> symbol/source server support.
>
>
> --- On Tue, 9/23/08, James Carpenter
> <jc...@yahoo.com> wrote:
>
> > From: James Carpenter <jc...@yahoo.com>
> > Subject: Research into Symbol and Source Servers
> > To: nmaven-dev@incubator.apache.org
> > Date: Tuesday, September 23, 2008, 11:19 AM
> > Overview:
> >
> > I have been doing some reading/research to better
> > understand symbol servers and source servers. I
> realize
> > some readers may already be familiar with the topic,
> but for
> > those that are not the information may prove useful.
> >
> > To provide the ability to easily step down into source
> code
> > of referenced libraries within Visual Studio, NMaven
> will
> > need to integrate with Microsoft's Source and
> Symbol
> > server functionality.
> >
> > A high level overview can be found using the following
> > links:
> > http://msdn.microsoft.com/en-us/magazine/cc301459.aspx
> > http://msdn.microsoft.com/en-us/magazine/cc163563.aspx
> >
> > ===================================
> > Additional Useful Information:
> >
> > The source server directory in the "Debugging
> Tools
> > for Windows" distribution contains a particularly
> > useful MS Word document which provides a great deal of
> the
> > details missing from the articles above. The section
> on
> > "HTTP Sites and UNC Shares" is particularly
> > interesting as it provides an alternate way to easily
> > support source server functionality. Assuming you
> have
> > installed "Debugging Tools for Windows" in
> the
> > standard location the MS Word document can be found
> at:
> > C:\Program Files\Debugging Tools for Windows
> > (x86)\srcsrv\srcsrv.doc
> >
> > Note: You will need to install "Debugging Tools
> for
> > Windows".
> >
> > ----------------------------------------
> > The MS Help files installed along with "Debugging
> > Tools for Windows" as well as most other
> documentation
> > don't really cover the use of multiple symbol
> servers
> > well. The omission can lead one to the false
> conclusion
> > only a single remote symbol server is supported.
> From
> > Chapter 4 (Managing Symbol and Source Files) of
> Advanced
> > Windows Debugging by Daniel Pravat reveals one can
> include
> > as many remote symbol servers in the sympath as
> desired.
> > These can also be URLs not just UNC paths. (I may
> have
> > found this in the MS Word document mentioned above.)
> >
> > Title: Advanced Windows Debugging
> > by: Mario Hewardt; Daniel Pravat
> > Publisher: Addison Wesley Professional
> > Pub Date: October 29, 2007
> > Print ISBN-10: 0-321-37446-0
> > Print ISBN-13: 978-0-321-37446-2
> > (Available on Safari.)
> >
> > ---------------------------------------
> > When you store a pdb in the symbol server
> >
> > e.g.: prompt>symstore add /r /f
> > C:\junk2\bin\*.* /s
> C:\symbolstore\ /t
> > "MyApp" /v "Build 632"
> >
> > the symstore will create a directory structure which
> > contains a GUID extracted from the PDB. As it turns
> out a
> > PDB is given a GUID when it is created. One way to
> read
> > the GUID is to use the dbh tool.
> >
> > e.g.: C:\junk2\bin>dbh -v srvind
> MySymbol.pdb
> >
> > Details on the dbh tool can be found in the MS Help
> files
> > shipped with the "Debugging Tools for
> Windows".
> >
> > Note: Examples assume C:\Program
> Files\Debugging
> > Tools for Windows (x86) is on your path.
> >
> > ===============================================
> > Supporting Symbol/Source Servers in NMaven
> >
> > To properly support symbol and source server
> functionality
> > in NMaven there appear to be two good choices I can
> see.
> > These are:
> >
> > --------------
> > Option 1:
> >
> > a) Create a plugin(s) which downloads the pdb files of
> > every dependency (saved in the mvn repo as attached
> > artifacts) and executes symstore to save the pdb files
> in a
> > local symbol store.
> >
> > b) Create a plugin(s) which downloads the source
> archives
> > of every dependency (saved in the mvn repo as attached
> > artifacts) and expands them into a directory structure
> > matching that of a UNC store. (See "HTTP Sites
> and
> > UNC Shares" section of srcsrv.doc.)
> >
> > c) Ensure the pdb files are properly processed to
> include
> > the necessary source stream during or before the
> deploy
> > phase. It is important SNAPSHOT artifacts include
> this
> > information so one can't wait for the release
> plugin to
> > do the job.
> >
> > --------------
> > Option 2:
> >
> > a) Same as Option 1.a
> >
> > b) Process the pdb files during/before the deploy
> phase to
> > include the information necessary to extract a single
> > source file. Under the covers the command line call
> formed
> > when VS processed the source stream from the pdb file
> will
> > need to resolve the relevant source archive and
> extract the
> > relevant file from the source archive. The best way
> to do
> > this is likely to work out a one line mvn call which
> will
> > return the relevant source file, creating a plugin or
> two as
> > necessary. The command line call formed when
> processing
> > the source stream in the pdb file will therefore
> simply be a
> > call into maven.
> >
> > -----------------
> > Note:
> >
> > Careful examination of the perl scripts included in
> > C:\Program Files\Debugging Tools for Windows
> > (x86)\srcsrv\ and documented in the associated
> MS
> > Word document should be enough to understand how the
> pdb
> > processing works in both Option 1 and 2.
> >
> > -----------------
> > Effort Estimate:
> >
> > Option 1 may be slightly easier to implement, but
> option 2
> > will probably be a tad more elegant. It wouldn't
> be
> > that much more effort to support both. I would guess
> > either approach will probably take a week or two of
> work
> > to complete, including testing.
> >
> >
Re: Research into Symbol and Source Servers
Posted by James Carpenter <jc...@yahoo.com>.
I have created JIRA issue NMAVEN-120 to address symbol/source server support.
--- On Tue, 9/23/08, James Carpenter <jc...@yahoo.com> wrote:
> From: James Carpenter <jc...@yahoo.com>
> Subject: Research into Symbol and Source Servers
> To: nmaven-dev@incubator.apache.org
> Date: Tuesday, September 23, 2008, 11:19 AM
> Overview:
>
> I have been doing some reading/research to better
> understand symbol servers and source servers. I realize
> some readers may already be familiar with the topic, but for
> those that are not the information may prove useful.
>
> To provide the ability to easily step down into source code
> of referenced libraries within Visual Studio, NMaven will
> need to integrate with Microsoft's Source and Symbol
> server functionality.
>
> A high level overview can be found using the following
> links:
> http://msdn.microsoft.com/en-us/magazine/cc301459.aspx
> http://msdn.microsoft.com/en-us/magazine/cc163563.aspx
>
> ===================================
> Additional Useful Information:
>
> The source server directory in the "Debugging Tools
> for Windows" distribution contains a particularly
> useful MS Word document which provides a great deal of the
> details missing from the articles above. The section on
> "HTTP Sites and UNC Shares" is particularly
> interesting as it provides an alternate way to easily
> support source server functionality. Assuming you have
> installed "Debugging Tools for Windows" in the
> standard location the MS Word document can be found at:
> C:\Program Files\Debugging Tools for Windows
> (x86)\srcsrv\srcsrv.doc
>
> Note: You will need to install "Debugging Tools for
> Windows".
>
> ----------------------------------------
> The MS Help files installed along with "Debugging
> Tools for Windows" as well as most other documentation
> don't really cover the use of multiple symbol servers
> well. The omission can lead one to the false conclusion
> only a single remote symbol server is supported. From
> Chapter 4 (Managing Symbol and Source Files) of Advanced
> Windows Debugging by Daniel Pravat reveals one can include
> as many remote symbol servers in the sympath as desired.
> These can also be URLs not just UNC paths. (I may have
> found this in the MS Word document mentioned above.)
>
> Title: Advanced Windows Debugging
> by: Mario Hewardt; Daniel Pravat
> Publisher: Addison Wesley Professional
> Pub Date: October 29, 2007
> Print ISBN-10: 0-321-37446-0
> Print ISBN-13: 978-0-321-37446-2
> (Available on Safari.)
>
> ---------------------------------------
> When you store a pdb in the symbol server
>
> e.g.: prompt>symstore add /r /f
> C:\junk2\bin\*.* /s C:\symbolstore\ /t
> "MyApp" /v "Build 632"
>
> the symstore will create a directory structure which
> contains a GUID extracted from the PDB. As it turns out a
> PDB is given a GUID when it is created. One way to read
> the GUID is to use the dbh tool.
>
> e.g.: C:\junk2\bin>dbh -v srvind MySymbol.pdb
>
> Details on the dbh tool can be found in the MS Help files
> shipped with the "Debugging Tools for Windows".
>
> Note: Examples assume C:\Program Files\Debugging
> Tools for Windows (x86) is on your path.
>
> ===============================================
> Supporting Symbol/Source Servers in NMaven
>
> To properly support symbol and source server functionality
> in NMaven there appear to be two good choices I can see.
> These are:
>
> --------------
> Option 1:
>
> a) Create a plugin(s) which downloads the pdb files of
> every dependency (saved in the mvn repo as attached
> artifacts) and executes symstore to save the pdb files in a
> local symbol store.
>
> b) Create a plugin(s) which downloads the source archives
> of every dependency (saved in the mvn repo as attached
> artifacts) and expands them into a directory structure
> matching that of a UNC store. (See "HTTP Sites and
> UNC Shares" section of srcsrv.doc.)
>
> c) Ensure the pdb files are properly processed to include
> the necessary source stream during or before the deploy
> phase. It is important SNAPSHOT artifacts include this
> information so one can't wait for the release plugin to
> do the job.
>
> --------------
> Option 2:
>
> a) Same as Option 1.a
>
> b) Process the pdb files during/before the deploy phase to
> include the information necessary to extract a single
> source file. Under the covers the command line call formed
> when VS processed the source stream from the pdb file will
> need to resolve the relevant source archive and extract the
> relevant file from the source archive. The best way to do
> this is likely to work out a one line mvn call which will
> return the relevant source file, creating a plugin or two as
> necessary. The command line call formed when processing
> the source stream in the pdb file will therefore simply be a
> call into maven.
>
> -----------------
> Note:
>
> Careful examination of the perl scripts included in
> C:\Program Files\Debugging Tools for Windows
> (x86)\srcsrv\ and documented in the associated MS
> Word document should be enough to understand how the pdb
> processing works in both Option 1 and 2.
>
> -----------------
> Effort Estimate:
>
> Option 1 may be slightly easier to implement, but option 2
> will probably be a tad more elegant. It wouldn't be
> that much more effort to support both. I would guess
> either approach will probably take a week or two of work
> to complete, including testing.
>
>