You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@manifoldcf.apache.org by Karl Wright <da...@gmail.com> on 2011/12/06 08:25:24 UTC
Re: Problem crawling windows share
About your capture - Michael Allen says the following:
"Actually this has nothing to do with DFS. JCIFS does not get to the
point where it does DFS anything. The capture shows a vanilla
STATUS_LOGON_FAILURE when GLOBAL\swapna.vuppala tries to auth with
l-carx01.global.arup.com. So the possible causes for this are 1) the
account name is not valid 2) the supplied password is incorrect 3)
some security policy is deliberately blocking that user or particular
type of auth or 4) some server configuration is incompatible with
JCIFS. I only mention this last option because I noticed the target
server has security signatures disabled. That's strange. If they're
messing around with things like that, who knows what their clients are
expected to do.
Try a Windows client that uses NTLM instead of Kerberos. Meaning try a
machine that is not joined to the domain so that when you try to
access the target it asks you for credentials at which point you can
test with GLOBAL\swapna.vuppala. Then it will use NTLM and you can
actually compare captures. If the operator doesn't have a laptop or
something not joined to the domain, it might be sufficient to log into
a workstation using machine credentials and not domain credentials.
Also when testing JCIFS you should use a simple stand-alone program
like examples/ListFiles.java."
In other words:
(a) Since JCIFS does not use Kerberos for authentication, you need to
try to log into the recalcitrant server via Windows without using
Kerberos to be able to do a side-by-side comparison. Michael has some
ways of doing that, above.
(b) You may find that it doesn't work, in which case JCIFS is not
going to work either.
(c) If it *does* work, then try to generate your side-by-side
comparisons using a simpler example rather than ManifoldCF en toto;
you can see how at jcifs.samba.org, or I can help you further.
He also mentions that there is some bizarreness on the response that
indicates that the server is configured in a way that he's never seen
before. And believe me, Michael has seen a *lot* of strange
configurations...
Hope this helps.
Karl
On Mon, Nov 28, 2011 at 4:12 AM, Karl Wright <da...@gmail.com> wrote:
> That should read "properties.xml", not "properties.ini". It looks
> like this page needs updating.
>
> The debug property in the XML form is:
>
> <property name="org.apache.manifoldcf.connectors" value="DEBUG"/>
>
> I don't think it will provide you with any additional information that
> is useful for debugging your authentication issue, however, if that is
> why you are looking at it. There may be some jcifs.jar debugging
> switches that might be of more help, but in the end I suspect you will
> need a packet capture of both a successful connection (via Windows)
> and an unsuccessful one (via MCF). The guy you will need to talk with
> after that is the jcifs author Michael Allen; I can give you his email
> address if you get that far.
>
> Karl
>
>
> On Mon, Nov 28, 2011 at 1:30 AM, Swapna Vuppala
> <sw...@gmail.com> wrote:
>> Hi Karl,
>>
>> I was planning to debug jCIFS repository connection using WireShark and I
>> came across this
>> https://cwiki.apache.org/CONNECTORS/debugging-connections.html
>> Here, I see something as add "org.apache.manifoldcf.connectors=DEBUG" to the
>> properties.ini file. Is it the properties.xml file that is being referred
>> here ? If not, where do I find properties.ini file ?
>>
>> Thanks and Regards,
>> Swapna.
>>
>> On Thu, Nov 17, 2011 at 1:31 PM, Karl Wright <da...@gmail.com> wrote:
>>>
>>> See http://jcifs.samba.org/src/docs/api/overview-summary.html#scp.
>>> The properties jcifs.smb.lmCompatibility and
>>> jcifs.smb.client.useExtendedSecurity are the ones you may want to
>>> change. These two properties go together so certain combinations make
>>> sense and others don't, so there's really only combinations you need
>>> but I'll need to look at what they are and get back to you later
>>> today.
>>>
>>> As far as setting the switches are concerned, if you are using the
>>> Quick Start you do this trivially by:
>>>
>>> <java> -Dxxx -Dyyy -jar start.jar
>>>
>>> If you are using the multi-process configuration, that is what the
>>> "defines" directory is for; you only need to create files in that
>>> directory with the names "jcifs.smb.lmCompatibility" and
>>> "jcifs.smb.client.useExtendedSecurity" containing the values you want
>>> to set.
>>>
>>> Karl
>>>
>>>
>>> On Thu, Nov 17, 2011 at 1:11 AM, Swapna Vuppala
>>> <sw...@gmail.com> wrote:
>>> > Hi Karl,
>>> >
>>> > Am able to access the folders on the problem server through windows
>>> > explorer, (\\server3\Folder1). I tried couple of things with the
>>> > credentials
>>> > form, changing username, domain etc.. but I keep getting the same error
>>> > "Couldn't connect to server: Logon failure: unknown user name or bad
>>> > password"
>>> >
>>> > Can you tell me more about the -D switch you were talking of ?
>>> >
>>> > Thanks and Regards,
>>> > Swapna.
>>> >
>>> > On Tue, Nov 15, 2011 at 12:40 PM, Karl Wright <da...@gmail.com>
>>> > wrote:
>>> >>
>>> >> Glad you chased it down this far.
>>> >>
>>> >> First thing to try is whether you can get into the problem server
>>> >> using Windows Explorer. Obviously ManifoldCF is not going to be able
>>> >> to do it if Windows can't. If you *can* get in, then just playing
>>> >> with the form of the credentials in the MCF connection might do the
>>> >> trick. Some Windows or net appliance servers are picky about this.
>>> >> Try various things like leaving the domain blank and specifying the
>>> >> user as "abc@domain.com", for instance. There's also a different NTLM
>>> >> mode you can operation jcifs in that some servers may be configured to
>>> >> require; this would need you to set a -D switch on the command line to
>>> >> enable.
>>> >>
>>> >> Karl
>>> >>
>>> >> On Tue, Nov 15, 2011 at 12:10 AM, Swapna Vuppala
>>> >> <sw...@gmail.com> wrote:
>>> >> > Hi Karl,
>>> >> >
>>> >> > Thanks for the input. It looks like my problem is related to the
>>> >> > second
>>> >> > one
>>> >> > that you specified. One of the directories in the path am trying to
>>> >> > index is
>>> >> > actually redirecting to a different server. And when I specify this
>>> >> > new
>>> >> > server in defining the repository connection, with my credentials,
>>> >> > the
>>> >> > connection fails with the message: "Couldn't connect to server:
>>> >> > Logon
>>> >> > failure: unknown user name or bad password"
>>> >> >
>>> >> > I'll look into why am not able to connect to this server.
>>> >> >
>>> >> > Thanks and Regards,
>>> >> > Swapna.
>>> >> >
>>> >> > On Mon, Nov 14, 2011 at 4:56 PM, Karl Wright <da...@gmail.com>
>>> >> > wrote:
>>> >> >>
>>> >> >> There's two kinds of problem you might be having. The first is
>>> >> >> intermittent, and the second is not intermittent but would have
>>> >> >> something to do with specific directories.
>>> >> >>
>>> >> >> Intermittent problems might include a domain controller that is not
>>> >> >> always accessible. In such cases, the crawl will proceed but will
>>> >> >> tend to fail unpredictably. On the other hand, if you have a
>>> >> >> directory that is handled by a DFS redirection, it is possible that
>>> >> >> the redirection is indicating a new server (lets call it server3)
>>> >> >> which may not like the precise form of your login credentials. Can
>>> >> >> you determine which scenario you are seeing?
>>> >> >>
>>> >> >> Karl
>>> >> >>
>>> >> >> On Mon, Nov 14, 2011 at 3:11 AM, Swapna Vuppala
>>> >> >> <sw...@gmail.com> wrote:
>>> >> >> > Hi,
>>> >> >> >
>>> >> >> > I have been using windows share repository connection to crawl and
>>> >> >> > get
>>> >> >> > data
>>> >> >> > from a particular server (server 1). Its working perfectly fine.
>>> >> >> > However, am
>>> >> >> > having trouble when I try with data from another server (server
>>> >> >> > 2).
>>> >> >> >
>>> >> >> > When I define a repository connection of type windows share and
>>> >> >> > specify
>>> >> >> > the
>>> >> >> > server name (server 2) with my credentials, the connection status
>>> >> >> > shows
>>> >> >> > "Connection working". But when I run a job to use this repository
>>> >> >> > connection
>>> >> >> > and index data from a location on this server 2, I keep getting
>>> >> >> > the
>>> >> >> > exception below:
>>> >> >> >
>>> >> >> > JCIFS: Possibly transient exception detected on attempt 3 while
>>> >> >> > checking
>>> >> >> > if
>>> >> >> > file exists: Logon failure: unknown user name or bad password.
>>> >> >> > jcifs.smb.SmbAuthException: Logon failure: unknown user name or
>>> >> >> > bad
>>> >> >> > password.
>>> >> >> > at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:544)
>>> >> >> > at jcifs.smb.SmbTransport.send(SmbTransport.java:661)
>>> >> >> > at jcifs.smb.SmbSession.sessionSetup(SmbSession.java:390)
>>> >> >> > at jcifs.smb.SmbSession.send(SmbSession.java:218)
>>> >> >> > at jcifs.smb.SmbTree.treeConnect(SmbTree.java:176)
>>> >> >> > at jcifs.smb.SmbFile.doConnect(SmbFile.java:911)
>>> >> >> > at jcifs.smb.SmbFile.connect(SmbFile.java:954)
>>> >> >> > at jcifs.smb.SmbFile.connect0(SmbFile.java:880)
>>> >> >> > at jcifs.smb.SmbFile.queryPath(SmbFile.java:1335)
>>> >> >> > at jcifs.smb.SmbFile.exists(SmbFile.java:1417)
>>> >> >> > at
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> > org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.fileExists(SharedDriveConnector.java:2064)
>>> >> >> > at
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> > org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getDocumentVersions(SharedDriveConnector.java:521)
>>> >> >> > at
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> > org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:318)
>>> >> >> >
>>> >> >> > Am able to access this location from windows explorer. What else
>>> >> >> > should
>>> >> >> > I be
>>> >> >> > checking or what could be the reasons/factors causing this to fail
>>> >> >> > ?
>>> >> >> >
>>> >> >> > Thanks and Regards,
>>> >> >> > Swapna.
>>> >> >> >
>>> >> >
>>> >> >
>>> >
>>> >
>>
>>
Re: Problem crawling windows share
Posted by Swapna Vuppala <sw...@gmail.com>.
Hi Karl,
Sorry for responding late.
I just opened a ticket
CONNECTORS-507<https://issues.apache.org/jira/browse/CONNECTORS-507>and
submitted a patch. Please let me know if there's any issue with the
patch.
Thanks and Regards,
Swapna.
On Mon, Jul 30, 2012 at 5:44 PM, Karl Wright <da...@gmail.com> wrote:
> Thanks, Swapna, for letting us know what the problem/solution was.
>
> If you think the documentation for the JCIFS connector could be
> improved, I encourage you to open a ticket and submit a patch. This
> would help others who have similar problems. At the very least we
> should include this in the FAQ document.
>
> Karl
>
>
> On Mon, Jul 30, 2012 at 6:14 AM, Swapna Vuppala
> <sw...@gmail.com> wrote:
> >
> > Hi Karl,
> >
> > This is with respect to a quite old issue that we had discussed for some
> > time and left.
> >
> > The issue got solved for us by setting the property
> > "org.apache.manifoldcf.crawler.connectors.jcifs.usentlmv1" to "true" in
> > properties.xml. Just wanted to update you on this.
> > With this property set, we are able to crawl the windows share
> (mentioned in
> > previous mails) successfully and index them into Solr.
> >
> > Thanks for all the help.
> >
> > Thanks and Regards,
> > Swapna.
> >
> >
> > On Tue, Dec 13, 2011 at 11:15 AM, Swapna Vuppala
> > <sw...@gmail.com> wrote:
> >>
> >> Thanks Karl for getting this far.
> >>
> >> I understand that the chances of jCIFS working with this particular
> server
> >> are very less. We'll leave it here for now and I'll let you know if I
> have
> >> any more details on the server.
> >>
> >> Thanks and Regards,
> >> Swapna.
> >>
> >>
> >> On Tue, Dec 13, 2011 at 7:24 AM, Karl Wright <da...@gmail.com>
> wrote:
> >>>
> >>> His response:
> >>>
> >>> "Actually from looking at the capture again, the successful operations
> >>> in it were ultimately performed by a session that used Kerberos. But
> >>> there are two other successful NTLM authentications that use the
> >>> Anonymous credential. It so happens that neither of these went on to
> >>> do anything successful. Meaning your capture does not contain a
> >>> successful operation using NTLM and thus there is nothing to compare
> >>> JCIFS too. It could be that the environment actually does not support
> >>> NTLM properly. There are a lot of errors and oddities in the non-JCIFS
> >>> communications with numerous login failures, disabled signatures,
> >>> access denied conditions and unsupported versions of Windows (Windows
> >>> Server 2002 SP2 build 2600). The environment is not optimal.
> >>>
> >>> To continue I would need proper comparison captures. Clearly the
> >>> NtlmPasswordAuthentication("","","") does not work. But
> >>> NtlmPasswordAuthentication.ANONYMOUS obviously does work to some
> >>> degree but there's something else wrong with the session setup
> >>> request. That something could be the fact that the hostname used is
> >>> actually l-carx01.global.arup.com and not lon-data-arx.global.arup.com
> >>> and then the tree connect goes on to mount
> >>> //lon-data-arx.global.arup.com/stg-jobs$ so clearly there is some
> >>> hostname discontinuity going on as well."
> >>>
> >>> I think this is pretty much self explanatory. Whatever it is that you
> >>> are trying to talk to seems to be one truly crazy machine, certainly
> >>> not really Windows, and it is almost an accident that it works with
> >>> Windows at all. I think this is rapidly reaching the point of
> >>> diminishing returns in getting jCIFS to work with it.
> >>>
> >>> Karl
> >>>
> >>>
> >>> On Mon, Dec 12, 2011 at 4:24 AM, Karl Wright <da...@gmail.com>
> wrote:
> >>> > I forwarded this research again to Mr. Allen for further suggestions.
> >>> > Thanks!
> >>> >
> >>> > Karl
> >>> >
> >>> > On Mon, Dec 12, 2011 at 1:58 AM, Swapna Vuppala
> >>> > <sw...@gmail.com> wrote:
> >>> >> Hi Karl,
> >>> >>
> >>> >> I tried modifying the test code as you suggested.
> >>> >>
> >>> >> When I tried with empty credentials
> >>> >>
> >>> >> NtlmPasswordAuthentication pa = new
> >>> >> NtlmPasswordAuthentication("","","");
> >>> >> SmbFile smbconnection = new SmbFile("smb://" +
> >>> >> "lon-data-arx.global.arup.com" + "/" + "stg-jobs$",pa);
> >>> >>
> >>> >> I got the same exception as before
> >>> >>
> >>> >> Exception in thread "main" jcifs.smb.SmbAuthException: Logon
> failure:
> >>> >> unknown us
> >>> >>
> >>> >> er name or bad password.
> >>> >> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:546)
> >>> >> at jcifs.smb.SmbTransport.send(SmbTransport.java:663)
> >>> >>
> >>> >> at jcifs.smb.SmbSession.sessionSetup(SmbSession.java:390)
> >>> >> at jcifs.smb.SmbSession.send(SmbSession.java:218)
> >>> >> at jcifs.smb.SmbTree.treeConnect(SmbTree.java:176)
> >>> >> at jcifs.smb.SmbFile.doConnect(SmbFile.java:911)
> >>> >>
> >>> >> ......
> >>> >>
> >>> >> When I tried with NtlmPasswordAuthentication.ANONYMOUS
> >>> >>
> >>> >> SmbFile smbconnection = new SmbFile("smb://" +
> >>> >> "lon-data-arx.global.arup.com" + "/" +
> >>> >> "stg-jobs$",NtlmPasswordAuthentication.ANONYMOUS);
> >>> >>
> >>> >> I got a different exception
> >>> >>
> >>> >> Exception in thread "main" jcifs.smb.SmbException: Incorrect
> function.
> >>> >> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:563)
> >>> >> at jcifs.smb.SmbTransport.send(SmbTransport.java:663)
> >>> >> at jcifs.smb.SmbSession.sessionSetup(SmbSession.java:316)
> >>> >>
> >>> >> at jcifs.smb.SmbSession.send(SmbSession.java:218)
> >>> >> at jcifs.smb.SmbTree.treeConnect(SmbTree.java:176)
> >>> >> at jcifs.smb.SmbFile.doConnect(SmbFile.java:911)
> >>> >> .....
> >>> >>
> >>> >> Thanks and Regards,
> >>> >> Swapna.
> >>> >> On Fri, Dec 9, 2011 at 11:46 PM, Karl Wright <da...@gmail.com>
> >>> >> wrote:
> >>> >>>
> >>> >>> Michael Allen's response to your capture:
> >>> >>>
> >>> >>> "It looks like the non-JCIFS communication is successful because
> the
> >>> >>> anonymous credential is used.
> >>> >>>
> >>> >>> Note that there is still no sign of DFS going on anywhere. Actually
> >>> >>> JCIFS tries to do a referral for the DFS root and gets an error of
> >>> >>> STATUS_FS_DRIVER_REQUIRED so it looks like DFS may not even be
> setup
> >>> >>> at all in the target environment. At least not domain based DFS.
> >>> >>>
> >>> >>> So if you want to emulate Windows behavior, I guess you need to
> retry
> >>> >>> the operation with credential NtlmPasswordAuthentication.ANONYMOUS
> >>> >>> (or
> >>> >>> all empty strings should work too I think like ";:")."
> >>> >>>
> >>> >>> In other words, the issue is that the server in question does not
> >>> >>> actually recognize the credentials given at all, and gives an
> error.
> >>> >>> But while Windows then retries with no credentials whatsoever,
> >>> >>> nothing
> >>> >>> in JCIFS does that retry.
> >>> >>>
> >>> >>> You can confirm that this is what is happening by using an empty
> user
> >>> >>> name, password, and domain in the JCIFS test code you wrote to
> obtain
> >>> >>> the captures. If, then, the test code works against that server,
> >>> >>> I'll
> >>> >>> create a ticket for downgrading to ANONYMOUS whenever there is an
> >>> >>> authentication error in the JCIFS connector.
> >>> >>>
> >>> >>> Thanks,
> >>> >>> Karl
> >>> >>>
> >>> >>> On Wed, Dec 7, 2011 at 6:41 AM, Karl Wright <da...@gmail.com>
> >>> >>> wrote:
> >>> >>> > The next step is to obtain two packet captures using WireShark.
> >>> >>> > These
> >>> >>> > should both be done from the machine that is not joined to the
> >>> >>> > domain.
> >>> >>> > The first capture should be the interaction that takes place
> when
> >>> >>> > you
> >>> >>> > try to go to the foreign directory, and you supply the
> credentials,
> >>> >>> > and then you list the folders. The second capture will require
> you
> >>> >>> > to
> >>> >>> > do the same using JCIFS. For that capture, you will need
> jcifs.jar
> >>> >>> > and a simple example class that provides jcifs with credentials
> and
> >>> >>> > tries to list the directory. A simple class should be enough to
> do
> >>> >>> > this, including code such as:
> >>> >>> >
> >>> >>> > import jcifs.smb.*;
> >>> >>> >
> >>> >>> > NtlmPasswordAuthentication pa = new
> >>> >>> > NtlmPasswordAuthentication(domain,username,password);
> >>> >>> > SmbFile smbconnection = new SmbFile("smb://" + server + "/" +
> >>> >>> > share,pa);
> >>> >>> > SmbFile[] files = smbconnection.listFiles();
> >>> >>> >
> >>> >>> > for (int i = 0 ; i < files.length ; i++)
> >>> >>> > {
> >>> >>> > System.out.println(file.toString());
> >>> >>> > }
> >>> >>> >
> >>> >>> > I suggest you put together your test class first and make sure it
> >>> >>> > works, and then go after the captures.
> >>> >>> >
> >>> >>> > Let me know how it goes.
> >>> >>> >
> >>> >>> > Karl
> >>> >>> >
> >>> >>> > On Wed, Dec 7, 2011 at 6:10 AM, Swapna Vuppala
> >>> >>> > <sw...@gmail.com> wrote:
> >>> >>> >> Hi Karl,
> >>> >>> >>
> >>> >>> >> Thanks for the detailed reply.
> >>> >>> >>
> >>> >>> >> As per your suggestion, I tried connecting to the problematic
> >>> >>> >> server
> >>> >>> >> via
> >>> >>> >> windows using a machine that is not in the domain. On supplying
> >>> >>> >> the
> >>> >>> >> credentials via windows, I could connect to the server, although
> >>> >>> >> it did
> >>> >>> >> not
> >>> >>> >> list any folders. But when I tried accessing the folder of my
> >>> >>> >> interest
> >>> >>> >> via
> >>> >>> >> \\server\folder$, I could see all the sub folders. All this is
> >>> >>> >> when I
> >>> >>> >> connected via windows from a machine that's not joined to the
> >>> >>> >> domain.
> >>> >>> >>
> >>> >>> >> The behavior is same when I connect to the server via windows
> from
> >>> >>> >> a
> >>> >>> >> machine
> >>> >>> >> that's joined to the domain.
> >>> >>> >>
> >>> >>> >> So I assume this means that through both Kerberos and NTLM, I
> see
> >>> >>> >> the
> >>> >>> >> same
> >>> >>> >> behavior. Whereas ManifoldCF throws an exception when trying to
> >>> >>> >> define
> >>> >>> >> the
> >>> >>> >> repository connection.
> >>> >>> >>
> >>> >>> >> Please advise me on what I should be doing next to resolve this.
> >>> >>> >>
> >>> >>> >> Thanks and Regards,
> >>> >>> >> Swapna.
> >>> >>> >>
> >>> >>> >>
> >>> >>> >>
> >>> >>> >>
> >>> >>> >> On Tue, Dec 6, 2011 at 12:55 PM, Karl Wright <
> daddywri@gmail.com>
> >>> >>> >> wrote:
> >>> >>> >>>
> >>> >>> >>> About your capture - Michael Allen says the following:
> >>> >>> >>>
> >>> >>> >>> "Actually this has nothing to do with DFS. JCIFS does not get
> to
> >>> >>> >>> the
> >>> >>> >>> point where it does DFS anything. The capture shows a vanilla
> >>> >>> >>> STATUS_LOGON_FAILURE when GLOBAL\swapna.vuppala tries to auth
> >>> >>> >>> with
> >>> >>> >>> l-carx01.global.arup.com. So the possible causes for this are
> 1)
> >>> >>> >>> the
> >>> >>> >>> account name is not valid 2) the supplied password is incorrect
> >>> >>> >>> 3)
> >>> >>> >>> some security policy is deliberately blocking that user or
> >>> >>> >>> particular
> >>> >>> >>> type of auth or 4) some server configuration is incompatible
> with
> >>> >>> >>> JCIFS. I only mention this last option because I noticed the
> >>> >>> >>> target
> >>> >>> >>> server has security signatures disabled. That's strange. If
> >>> >>> >>> they're
> >>> >>> >>> messing around with things like that, who knows what their
> >>> >>> >>> clients are
> >>> >>> >>> expected to do.
> >>> >>> >>>
> >>> >>> >>> Try a Windows client that uses NTLM instead of Kerberos.
> Meaning
> >>> >>> >>> try a
> >>> >>> >>> machine that is not joined to the domain so that when you try
> to
> >>> >>> >>> access the target it asks you for credentials at which point
> you
> >>> >>> >>> can
> >>> >>> >>> test with GLOBAL\swapna.vuppala. Then it will use NTLM and you
> >>> >>> >>> can
> >>> >>> >>> actually compare captures. If the operator doesn't have a
> laptop
> >>> >>> >>> or
> >>> >>> >>> something not joined to the domain, it might be sufficient to
> log
> >>> >>> >>> into
> >>> >>> >>> a workstation using machine credentials and not domain
> >>> >>> >>> credentials.
> >>> >>> >>>
> >>> >>> >>> Also when testing JCIFS you should use a simple stand-alone
> >>> >>> >>> program
> >>> >>> >>> like examples/ListFiles.java."
> >>> >>> >>>
> >>> >>> >>> In other words:
> >>> >>> >>> (a) Since JCIFS does not use Kerberos for authentication, you
> >>> >>> >>> need to
> >>> >>> >>> try to log into the recalcitrant server via Windows without
> using
> >>> >>> >>> Kerberos to be able to do a side-by-side comparison. Michael
> has
> >>> >>> >>> some
> >>> >>> >>> ways of doing that, above.
> >>> >>> >>> (b) You may find that it doesn't work, in which case JCIFS is
> not
> >>> >>> >>> going to work either.
> >>> >>> >>> (c) If it *does* work, then try to generate your side-by-side
> >>> >>> >>> comparisons using a simpler example rather than ManifoldCF en
> >>> >>> >>> toto;
> >>> >>> >>> you can see how at jcifs.samba.org, or I can help you further.
> >>> >>> >>>
> >>> >>> >>> He also mentions that there is some bizarreness on the response
> >>> >>> >>> that
> >>> >>> >>> indicates that the server is configured in a way that he's
> never
> >>> >>> >>> seen
> >>> >>> >>> before. And believe me, Michael has seen a *lot* of strange
> >>> >>> >>> configurations...
> >>> >>> >>>
> >>> >>> >>> Hope this helps.
> >>> >>> >>> Karl
> >>> >>> >>>
> >>> >>> >>> On Mon, Nov 28, 2011 at 4:12 AM, Karl Wright <
> daddywri@gmail.com>
> >>> >>> >>> wrote:
> >>> >>> >>> > That should read "properties.xml", not "properties.ini". It
> >>> >>> >>> > looks
> >>> >>> >>> > like this page needs updating.
> >>> >>> >>> >
> >>> >>> >>> > The debug property in the XML form is:
> >>> >>> >>> >
> >>> >>> >>> > <property name="org.apache.manifoldcf.connectors"
> >>> >>> >>> > value="DEBUG"/>
> >>> >>> >>> >
> >>> >>> >>> > I don't think it will provide you with any additional
> >>> >>> >>> > information
> >>> >>> >>> > that
> >>> >>> >>> > is useful for debugging your authentication issue, however,
> if
> >>> >>> >>> > that
> >>> >>> >>> > is
> >>> >>> >>> > why you are looking at it. There may be some jcifs.jar
> >>> >>> >>> > debugging
> >>> >>> >>> > switches that might be of more help, but in the end I suspect
> >>> >>> >>> > you
> >>> >>> >>> > will
> >>> >>> >>> > need a packet capture of both a successful connection (via
> >>> >>> >>> > Windows)
> >>> >>> >>> > and an unsuccessful one (via MCF). The guy you will need to
> >>> >>> >>> > talk
> >>> >>> >>> > with
> >>> >>> >>> > after that is the jcifs author Michael Allen; I can give you
> >>> >>> >>> > his
> >>> >>> >>> > email
> >>> >>> >>> > address if you get that far.
> >>> >>> >>> >
> >>> >>> >>> > Karl
> >>> >>> >>> >
> >>> >>> >>> >
> >>> >>> >>> > On Mon, Nov 28, 2011 at 1:30 AM, Swapna Vuppala
> >>> >>> >>> > <sw...@gmail.com> wrote:
> >>> >>> >>> >> Hi Karl,
> >>> >>> >>> >>
> >>> >>> >>> >> I was planning to debug jCIFS repository connection using
> >>> >>> >>> >> WireShark
> >>> >>> >>> >> and
> >>> >>> >>> >> I
> >>> >>> >>> >> came across this
> >>> >>> >>> >>
> https://cwiki.apache.org/CONNECTORS/debugging-connections.html
> >>> >>> >>> >> Here, I see something as add
> >>> >>> >>> >> "org.apache.manifoldcf.connectors=DEBUG"
> >>> >>> >>> >> to the
> >>> >>> >>> >> properties.ini file. Is it the properties.xml file that is
> >>> >>> >>> >> being
> >>> >>> >>> >> referred
> >>> >>> >>> >> here ? If not, where do I find properties.ini file ?
> >>> >>> >>> >>
> >>> >>> >>> >> Thanks and Regards,
> >>> >>> >>> >> Swapna.
> >>> >>> >>> >>
> >>> >>> >>> >> On Thu, Nov 17, 2011 at 1:31 PM, Karl Wright
> >>> >>> >>> >> <da...@gmail.com>
> >>> >>> >>> >> wrote:
> >>> >>> >>> >>>
> >>> >>> >>> >>> See
> >>> >>> >>> >>>
> http://jcifs.samba.org/src/docs/api/overview-summary.html#scp.
> >>> >>> >>> >>> The properties jcifs.smb.lmCompatibility and
> >>> >>> >>> >>> jcifs.smb.client.useExtendedSecurity are the ones you may
> >>> >>> >>> >>> want to
> >>> >>> >>> >>> change. These two properties go together so certain
> >>> >>> >>> >>> combinations
> >>> >>> >>> >>> make
> >>> >>> >>> >>> sense and others don't, so there's really only combinations
> >>> >>> >>> >>> you
> >>> >>> >>> >>> need
> >>> >>> >>> >>> but I'll need to look at what they are and get back to you
> >>> >>> >>> >>> later
> >>> >>> >>> >>> today.
> >>> >>> >>> >>>
> >>> >>> >>> >>> As far as setting the switches are concerned, if you are
> >>> >>> >>> >>> using the
> >>> >>> >>> >>> Quick Start you do this trivially by:
> >>> >>> >>> >>>
> >>> >>> >>> >>> <java> -Dxxx -Dyyy -jar start.jar
> >>> >>> >>> >>>
> >>> >>> >>> >>> If you are using the multi-process configuration, that is
> >>> >>> >>> >>> what the
> >>> >>> >>> >>> "defines" directory is for; you only need to create files
> in
> >>> >>> >>> >>> that
> >>> >>> >>> >>> directory with the names "jcifs.smb.lmCompatibility" and
> >>> >>> >>> >>> "jcifs.smb.client.useExtendedSecurity" containing the
> values
> >>> >>> >>> >>> you
> >>> >>> >>> >>> want
> >>> >>> >>> >>> to set.
> >>> >>> >>> >>>
> >>> >>> >>> >>> Karl
> >>> >>> >>> >>>
> >>> >>> >>> >>>
> >>> >>> >>> >>> On Thu, Nov 17, 2011 at 1:11 AM, Swapna Vuppala
> >>> >>> >>> >>> <sw...@gmail.com> wrote:
> >>> >>> >>> >>> > Hi Karl,
> >>> >>> >>> >>> >
> >>> >>> >>> >>> > Am able to access the folders on the problem server
> through
> >>> >>> >>> >>> > windows
> >>> >>> >>> >>> > explorer, (\\server3\Folder1). I tried couple of things
> >>> >>> >>> >>> > with the
> >>> >>> >>> >>> > credentials
> >>> >>> >>> >>> > form, changing username, domain etc.. but I keep getting
> >>> >>> >>> >>> > the
> >>> >>> >>> >>> > same
> >>> >>> >>> >>> > error
> >>> >>> >>> >>> > "Couldn't connect to server: Logon failure: unknown user
> >>> >>> >>> >>> > name or
> >>> >>> >>> >>> > bad
> >>> >>> >>> >>> > password"
> >>> >>> >>> >>> >
> >>> >>> >>> >>> > Can you tell me more about the -D switch you were talking
> >>> >>> >>> >>> > of ?
> >>> >>> >>> >>> >
> >>> >>> >>> >>> > Thanks and Regards,
> >>> >>> >>> >>> > Swapna.
> >>> >>> >>> >>> >
> >>> >>> >>> >>> > On Tue, Nov 15, 2011 at 12:40 PM, Karl Wright
> >>> >>> >>> >>> > <da...@gmail.com>
> >>> >>> >>> >>> > wrote:
> >>> >>> >>> >>> >>
> >>> >>> >>> >>> >> Glad you chased it down this far.
> >>> >>> >>> >>> >>
> >>> >>> >>> >>> >> First thing to try is whether you can get into the
> problem
> >>> >>> >>> >>> >> server
> >>> >>> >>> >>> >> using Windows Explorer. Obviously ManifoldCF is not
> going
> >>> >>> >>> >>> >> to
> >>> >>> >>> >>> >> be
> >>> >>> >>> >>> >> able
> >>> >>> >>> >>> >> to do it if Windows can't. If you *can* get in, then
> just
> >>> >>> >>> >>> >> playing
> >>> >>> >>> >>> >> with the form of the credentials in the MCF connection
> >>> >>> >>> >>> >> might do
> >>> >>> >>> >>> >> the
> >>> >>> >>> >>> >> trick. Some Windows or net appliance servers are picky
> >>> >>> >>> >>> >> about
> >>> >>> >>> >>> >> this.
> >>> >>> >>> >>> >> Try various things like leaving the domain blank and
> >>> >>> >>> >>> >> specifying
> >>> >>> >>> >>> >> the
> >>> >>> >>> >>> >> user as "abc@domain.com", for instance. There's also a
> >>> >>> >>> >>> >> different
> >>> >>> >>> >>> >> NTLM
> >>> >>> >>> >>> >> mode you can operation jcifs in that some servers may be
> >>> >>> >>> >>> >> configured
> >>> >>> >>> >>> >> to
> >>> >>> >>> >>> >> require; this would need you to set a -D switch on the
> >>> >>> >>> >>> >> command
> >>> >>> >>> >>> >> line
> >>> >>> >>> >>> >> to
> >>> >>> >>> >>> >> enable.
> >>> >>> >>> >>> >>
> >>> >>> >>> >>> >> Karl
> >>> >>> >>> >>> >>
> >>> >>> >>> >>> >> On Tue, Nov 15, 2011 at 12:10 AM, Swapna Vuppala
> >>> >>> >>> >>> >> <sw...@gmail.com> wrote:
> >>> >>> >>> >>> >> > Hi Karl,
> >>> >>> >>> >>> >> >
> >>> >>> >>> >>> >> > Thanks for the input. It looks like my problem is
> >>> >>> >>> >>> >> > related to
> >>> >>> >>> >>> >> > the
> >>> >>> >>> >>> >> > second
> >>> >>> >>> >>> >> > one
> >>> >>> >>> >>> >> > that you specified. One of the directories in the path
> >>> >>> >>> >>> >> > am
> >>> >>> >>> >>> >> > trying
> >>> >>> >>> >>> >> > to
> >>> >>> >>> >>> >> > index is
> >>> >>> >>> >>> >> > actually redirecting to a different server. And when I
> >>> >>> >>> >>> >> > specify
> >>> >>> >>> >>> >> > this
> >>> >>> >>> >>> >> > new
> >>> >>> >>> >>> >> > server in defining the repository connection, with my
> >>> >>> >>> >>> >> > credentials,
> >>> >>> >>> >>> >> > the
> >>> >>> >>> >>> >> > connection fails with the message: "Couldn't connect
> to
> >>> >>> >>> >>> >> > server:
> >>> >>> >>> >>> >> > Logon
> >>> >>> >>> >>> >> > failure: unknown user name or bad password"
> >>> >>> >>> >>> >> >
> >>> >>> >>> >>> >> > I'll look into why am not able to connect to this
> >>> >>> >>> >>> >> > server.
> >>> >>> >>> >>> >> >
> >>> >>> >>> >>> >> > Thanks and Regards,
> >>> >>> >>> >>> >> > Swapna.
> >>> >>> >>> >>> >> >
> >>> >>> >>> >>> >> > On Mon, Nov 14, 2011 at 4:56 PM, Karl Wright
> >>> >>> >>> >>> >> > <da...@gmail.com>
> >>> >>> >>> >>> >> > wrote:
> >>> >>> >>> >>> >> >>
> >>> >>> >>> >>> >> >> There's two kinds of problem you might be having.
> The
> >>> >>> >>> >>> >> >> first
> >>> >>> >>> >>> >> >> is
> >>> >>> >>> >>> >> >> intermittent, and the second is not intermittent but
> >>> >>> >>> >>> >> >> would
> >>> >>> >>> >>> >> >> have
> >>> >>> >>> >>> >> >> something to do with specific directories.
> >>> >>> >>> >>> >> >>
> >>> >>> >>> >>> >> >> Intermittent problems might include a domain
> controller
> >>> >>> >>> >>> >> >> that
> >>> >>> >>> >>> >> >> is
> >>> >>> >>> >>> >> >> not
> >>> >>> >>> >>> >> >> always accessible. In such cases, the crawl will
> >>> >>> >>> >>> >> >> proceed
> >>> >>> >>> >>> >> >> but
> >>> >>> >>> >>> >> >> will
> >>> >>> >>> >>> >> >> tend to fail unpredictably. On the other hand, if
> you
> >>> >>> >>> >>> >> >> have
> >>> >>> >>> >>> >> >> a
> >>> >>> >>> >>> >> >> directory that is handled by a DFS redirection, it is
> >>> >>> >>> >>> >> >> possible
> >>> >>> >>> >>> >> >> that
> >>> >>> >>> >>> >> >> the redirection is indicating a new server (lets call
> >>> >>> >>> >>> >> >> it
> >>> >>> >>> >>> >> >> server3)
> >>> >>> >>> >>> >> >> which may not like the precise form of your login
> >>> >>> >>> >>> >> >> credentials.
> >>> >>> >>> >>> >> >> Can
> >>> >>> >>> >>> >> >> you determine which scenario you are seeing?
> >>> >>> >>> >>> >> >>
> >>> >>> >>> >>> >> >> Karl
> >>> >>> >>> >>> >> >>
> >>> >>> >>> >>> >> >> On Mon, Nov 14, 2011 at 3:11 AM, Swapna Vuppala
> >>> >>> >>> >>> >> >> <sw...@gmail.com> wrote:
> >>> >>> >>> >>> >> >> > Hi,
> >>> >>> >>> >>> >> >> >
> >>> >>> >>> >>> >> >> > I have been using windows share repository
> connection
> >>> >>> >>> >>> >> >> > to
> >>> >>> >>> >>> >> >> > crawl
> >>> >>> >>> >>> >> >> > and
> >>> >>> >>> >>> >> >> > get
> >>> >>> >>> >>> >> >> > data
> >>> >>> >>> >>> >> >> > from a particular server (server 1). Its working
> >>> >>> >>> >>> >> >> > perfectly
> >>> >>> >>> >>> >> >> > fine.
> >>> >>> >>> >>> >> >> > However, am
> >>> >>> >>> >>> >> >> > having trouble when I try with data from another
> >>> >>> >>> >>> >> >> > server
> >>> >>> >>> >>> >> >> > (server
> >>> >>> >>> >>> >> >> > 2).
> >>> >>> >>> >>> >> >> >
> >>> >>> >>> >>> >> >> > When I define a repository connection of type
> windows
> >>> >>> >>> >>> >> >> > share
> >>> >>> >>> >>> >> >> > and
> >>> >>> >>> >>> >> >> > specify
> >>> >>> >>> >>> >> >> > the
> >>> >>> >>> >>> >> >> > server name (server 2) with my credentials, the
> >>> >>> >>> >>> >> >> > connection
> >>> >>> >>> >>> >> >> > status
> >>> >>> >>> >>> >> >> > shows
> >>> >>> >>> >>> >> >> > "Connection working". But when I run a job to use
> >>> >>> >>> >>> >> >> > this
> >>> >>> >>> >>> >> >> > repository
> >>> >>> >>> >>> >> >> > connection
> >>> >>> >>> >>> >> >> > and index data from a location on this server 2, I
> >>> >>> >>> >>> >> >> > keep
> >>> >>> >>> >>> >> >> > getting
> >>> >>> >>> >>> >> >> > the
> >>> >>> >>> >>> >> >> > exception below:
> >>> >>> >>> >>> >> >> >
> >>> >>> >>> >>> >> >> > JCIFS: Possibly transient exception detected on
> >>> >>> >>> >>> >> >> > attempt 3
> >>> >>> >>> >>> >> >> > while
> >>> >>> >>> >>> >> >> > checking
> >>> >>> >>> >>> >> >> > if
> >>> >>> >>> >>> >> >> > file exists: Logon failure: unknown user name or
> bad
> >>> >>> >>> >>> >> >> > password.
> >>> >>> >>> >>> >> >> > jcifs.smb.SmbAuthException: Logon failure: unknown
> >>> >>> >>> >>> >> >> > user
> >>> >>> >>> >>> >> >> > name
> >>> >>> >>> >>> >> >> > or
> >>> >>> >>> >>> >> >> > bad
> >>> >>> >>> >>> >> >> > password.
> >>> >>> >>> >>> >> >> > at
> >>> >>> >>> >>> >> >> >
> >>> >>> >>> >>> >> >> >
> jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:544)
> >>> >>> >>> >>> >> >> > at
> >>> >>> >>> >>> >> >> > jcifs.smb.SmbTransport.send(SmbTransport.java:661)
> >>> >>> >>> >>> >> >> > at
> >>> >>> >>> >>> >> >> >
> >>> >>> >>> >>> >> >> >
> jcifs.smb.SmbSession.sessionSetup(SmbSession.java:390)
> >>> >>> >>> >>> >> >> > at
> jcifs.smb.SmbSession.send(SmbSession.java:218)
> >>> >>> >>> >>> >> >> > at
> >>> >>> >>> >>> >> >> > jcifs.smb.SmbTree.treeConnect(SmbTree.java:176)
> >>> >>> >>> >>> >> >> > at
> jcifs.smb.SmbFile.doConnect(SmbFile.java:911)
> >>> >>> >>> >>> >> >> > at jcifs.smb.SmbFile.connect(SmbFile.java:954)
> >>> >>> >>> >>> >> >> > at jcifs.smb.SmbFile.connect0(SmbFile.java:880)
> >>> >>> >>> >>> >> >> > at
> jcifs.smb.SmbFile.queryPath(SmbFile.java:1335)
> >>> >>> >>> >>> >> >> > at jcifs.smb.SmbFile.exists(SmbFile.java:1417)
> >>> >>> >>> >>> >> >> > at
> >>> >>> >>> >>> >> >> >
> >>> >>> >>> >>> >> >> >
> >>> >>> >>> >>> >> >> >
> >>> >>> >>> >>> >> >> >
> >>> >>> >>> >>> >> >> >
> >>> >>> >>> >>> >> >> >
> >>> >>> >>> >>> >> >> >
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.fileExists(SharedDriveConnector.java:2064)
> >>> >>> >>> >>> >> >> > at
> >>> >>> >>> >>> >> >> >
> >>> >>> >>> >>> >> >> >
> >>> >>> >>> >>> >> >> >
> >>> >>> >>> >>> >> >> >
> >>> >>> >>> >>> >> >> >
> >>> >>> >>> >>> >> >> >
> >>> >>> >>> >>> >> >> >
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getDocumentVersions(SharedDriveConnector.java:521)
> >>> >>> >>> >>> >> >> > at
> >>> >>> >>> >>> >> >> >
> >>> >>> >>> >>> >> >> >
> >>> >>> >>> >>> >> >> >
> >>> >>> >>> >>> >> >> >
> >>> >>> >>> >>> >> >> >
> >>> >>> >>> >>> >> >> >
> >>> >>> >>> >>> >> >> >
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:318)
> >>> >>> >>> >>> >> >> >
> >>> >>> >>> >>> >> >> > Am able to access this location from windows
> >>> >>> >>> >>> >> >> > explorer.
> >>> >>> >>> >>> >> >> > What
> >>> >>> >>> >>> >> >> > else
> >>> >>> >>> >>> >> >> > should
> >>> >>> >>> >>> >> >> > I be
> >>> >>> >>> >>> >> >> > checking or what could be the reasons/factors
> causing
> >>> >>> >>> >>> >> >> > this
> >>> >>> >>> >>> >> >> > to
> >>> >>> >>> >>> >> >> > fail
> >>> >>> >>> >>> >> >> > ?
> >>> >>> >>> >>> >> >> >
> >>> >>> >>> >>> >> >> > Thanks and Regards,
> >>> >>> >>> >>> >> >> > Swapna.
> >>> >>> >>> >>> >> >> >
> >>> >>> >>> >>> >> >
> >>> >>> >>> >>> >> >
> >>> >>> >>> >>> >
> >>> >>> >>> >>> >
> >>> >>> >>> >>
> >>> >>> >>> >>
> >>> >>> >>
> >>> >>> >>
> >>> >>
> >>> >>
> >>
> >>
> >
> >
>
Re: Problem crawling windows share
Posted by Karl Wright <da...@gmail.com>.
Thanks, Swapna, for letting us know what the problem/solution was.
If you think the documentation for the JCIFS connector could be
improved, I encourage you to open a ticket and submit a patch. This
would help others who have similar problems. At the very least we
should include this in the FAQ document.
Karl
On Mon, Jul 30, 2012 at 6:14 AM, Swapna Vuppala
<sw...@gmail.com> wrote:
>
> Hi Karl,
>
> This is with respect to a quite old issue that we had discussed for some
> time and left.
>
> The issue got solved for us by setting the property
> "org.apache.manifoldcf.crawler.connectors.jcifs.usentlmv1" to "true" in
> properties.xml. Just wanted to update you on this.
> With this property set, we are able to crawl the windows share (mentioned in
> previous mails) successfully and index them into Solr.
>
> Thanks for all the help.
>
> Thanks and Regards,
> Swapna.
>
>
> On Tue, Dec 13, 2011 at 11:15 AM, Swapna Vuppala
> <sw...@gmail.com> wrote:
>>
>> Thanks Karl for getting this far.
>>
>> I understand that the chances of jCIFS working with this particular server
>> are very less. We'll leave it here for now and I'll let you know if I have
>> any more details on the server.
>>
>> Thanks and Regards,
>> Swapna.
>>
>>
>> On Tue, Dec 13, 2011 at 7:24 AM, Karl Wright <da...@gmail.com> wrote:
>>>
>>> His response:
>>>
>>> "Actually from looking at the capture again, the successful operations
>>> in it were ultimately performed by a session that used Kerberos. But
>>> there are two other successful NTLM authentications that use the
>>> Anonymous credential. It so happens that neither of these went on to
>>> do anything successful. Meaning your capture does not contain a
>>> successful operation using NTLM and thus there is nothing to compare
>>> JCIFS too. It could be that the environment actually does not support
>>> NTLM properly. There are a lot of errors and oddities in the non-JCIFS
>>> communications with numerous login failures, disabled signatures,
>>> access denied conditions and unsupported versions of Windows (Windows
>>> Server 2002 SP2 build 2600). The environment is not optimal.
>>>
>>> To continue I would need proper comparison captures. Clearly the
>>> NtlmPasswordAuthentication("","","") does not work. But
>>> NtlmPasswordAuthentication.ANONYMOUS obviously does work to some
>>> degree but there's something else wrong with the session setup
>>> request. That something could be the fact that the hostname used is
>>> actually l-carx01.global.arup.com and not lon-data-arx.global.arup.com
>>> and then the tree connect goes on to mount
>>> //lon-data-arx.global.arup.com/stg-jobs$ so clearly there is some
>>> hostname discontinuity going on as well."
>>>
>>> I think this is pretty much self explanatory. Whatever it is that you
>>> are trying to talk to seems to be one truly crazy machine, certainly
>>> not really Windows, and it is almost an accident that it works with
>>> Windows at all. I think this is rapidly reaching the point of
>>> diminishing returns in getting jCIFS to work with it.
>>>
>>> Karl
>>>
>>>
>>> On Mon, Dec 12, 2011 at 4:24 AM, Karl Wright <da...@gmail.com> wrote:
>>> > I forwarded this research again to Mr. Allen for further suggestions.
>>> > Thanks!
>>> >
>>> > Karl
>>> >
>>> > On Mon, Dec 12, 2011 at 1:58 AM, Swapna Vuppala
>>> > <sw...@gmail.com> wrote:
>>> >> Hi Karl,
>>> >>
>>> >> I tried modifying the test code as you suggested.
>>> >>
>>> >> When I tried with empty credentials
>>> >>
>>> >> NtlmPasswordAuthentication pa = new
>>> >> NtlmPasswordAuthentication("","","");
>>> >> SmbFile smbconnection = new SmbFile("smb://" +
>>> >> "lon-data-arx.global.arup.com" + "/" + "stg-jobs$",pa);
>>> >>
>>> >> I got the same exception as before
>>> >>
>>> >> Exception in thread "main" jcifs.smb.SmbAuthException: Logon failure:
>>> >> unknown us
>>> >>
>>> >> er name or bad password.
>>> >> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:546)
>>> >> at jcifs.smb.SmbTransport.send(SmbTransport.java:663)
>>> >>
>>> >> at jcifs.smb.SmbSession.sessionSetup(SmbSession.java:390)
>>> >> at jcifs.smb.SmbSession.send(SmbSession.java:218)
>>> >> at jcifs.smb.SmbTree.treeConnect(SmbTree.java:176)
>>> >> at jcifs.smb.SmbFile.doConnect(SmbFile.java:911)
>>> >>
>>> >> ......
>>> >>
>>> >> When I tried with NtlmPasswordAuthentication.ANONYMOUS
>>> >>
>>> >> SmbFile smbconnection = new SmbFile("smb://" +
>>> >> "lon-data-arx.global.arup.com" + "/" +
>>> >> "stg-jobs$",NtlmPasswordAuthentication.ANONYMOUS);
>>> >>
>>> >> I got a different exception
>>> >>
>>> >> Exception in thread "main" jcifs.smb.SmbException: Incorrect function.
>>> >> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:563)
>>> >> at jcifs.smb.SmbTransport.send(SmbTransport.java:663)
>>> >> at jcifs.smb.SmbSession.sessionSetup(SmbSession.java:316)
>>> >>
>>> >> at jcifs.smb.SmbSession.send(SmbSession.java:218)
>>> >> at jcifs.smb.SmbTree.treeConnect(SmbTree.java:176)
>>> >> at jcifs.smb.SmbFile.doConnect(SmbFile.java:911)
>>> >> .....
>>> >>
>>> >> Thanks and Regards,
>>> >> Swapna.
>>> >> On Fri, Dec 9, 2011 at 11:46 PM, Karl Wright <da...@gmail.com>
>>> >> wrote:
>>> >>>
>>> >>> Michael Allen's response to your capture:
>>> >>>
>>> >>> "It looks like the non-JCIFS communication is successful because the
>>> >>> anonymous credential is used.
>>> >>>
>>> >>> Note that there is still no sign of DFS going on anywhere. Actually
>>> >>> JCIFS tries to do a referral for the DFS root and gets an error of
>>> >>> STATUS_FS_DRIVER_REQUIRED so it looks like DFS may not even be setup
>>> >>> at all in the target environment. At least not domain based DFS.
>>> >>>
>>> >>> So if you want to emulate Windows behavior, I guess you need to retry
>>> >>> the operation with credential NtlmPasswordAuthentication.ANONYMOUS
>>> >>> (or
>>> >>> all empty strings should work too I think like ";:")."
>>> >>>
>>> >>> In other words, the issue is that the server in question does not
>>> >>> actually recognize the credentials given at all, and gives an error.
>>> >>> But while Windows then retries with no credentials whatsoever,
>>> >>> nothing
>>> >>> in JCIFS does that retry.
>>> >>>
>>> >>> You can confirm that this is what is happening by using an empty user
>>> >>> name, password, and domain in the JCIFS test code you wrote to obtain
>>> >>> the captures. If, then, the test code works against that server,
>>> >>> I'll
>>> >>> create a ticket for downgrading to ANONYMOUS whenever there is an
>>> >>> authentication error in the JCIFS connector.
>>> >>>
>>> >>> Thanks,
>>> >>> Karl
>>> >>>
>>> >>> On Wed, Dec 7, 2011 at 6:41 AM, Karl Wright <da...@gmail.com>
>>> >>> wrote:
>>> >>> > The next step is to obtain two packet captures using WireShark.
>>> >>> > These
>>> >>> > should both be done from the machine that is not joined to the
>>> >>> > domain.
>>> >>> > The first capture should be the interaction that takes place when
>>> >>> > you
>>> >>> > try to go to the foreign directory, and you supply the credentials,
>>> >>> > and then you list the folders. The second capture will require you
>>> >>> > to
>>> >>> > do the same using JCIFS. For that capture, you will need jcifs.jar
>>> >>> > and a simple example class that provides jcifs with credentials and
>>> >>> > tries to list the directory. A simple class should be enough to do
>>> >>> > this, including code such as:
>>> >>> >
>>> >>> > import jcifs.smb.*;
>>> >>> >
>>> >>> > NtlmPasswordAuthentication pa = new
>>> >>> > NtlmPasswordAuthentication(domain,username,password);
>>> >>> > SmbFile smbconnection = new SmbFile("smb://" + server + "/" +
>>> >>> > share,pa);
>>> >>> > SmbFile[] files = smbconnection.listFiles();
>>> >>> >
>>> >>> > for (int i = 0 ; i < files.length ; i++)
>>> >>> > {
>>> >>> > System.out.println(file.toString());
>>> >>> > }
>>> >>> >
>>> >>> > I suggest you put together your test class first and make sure it
>>> >>> > works, and then go after the captures.
>>> >>> >
>>> >>> > Let me know how it goes.
>>> >>> >
>>> >>> > Karl
>>> >>> >
>>> >>> > On Wed, Dec 7, 2011 at 6:10 AM, Swapna Vuppala
>>> >>> > <sw...@gmail.com> wrote:
>>> >>> >> Hi Karl,
>>> >>> >>
>>> >>> >> Thanks for the detailed reply.
>>> >>> >>
>>> >>> >> As per your suggestion, I tried connecting to the problematic
>>> >>> >> server
>>> >>> >> via
>>> >>> >> windows using a machine that is not in the domain. On supplying
>>> >>> >> the
>>> >>> >> credentials via windows, I could connect to the server, although
>>> >>> >> it did
>>> >>> >> not
>>> >>> >> list any folders. But when I tried accessing the folder of my
>>> >>> >> interest
>>> >>> >> via
>>> >>> >> \\server\folder$, I could see all the sub folders. All this is
>>> >>> >> when I
>>> >>> >> connected via windows from a machine that's not joined to the
>>> >>> >> domain.
>>> >>> >>
>>> >>> >> The behavior is same when I connect to the server via windows from
>>> >>> >> a
>>> >>> >> machine
>>> >>> >> that's joined to the domain.
>>> >>> >>
>>> >>> >> So I assume this means that through both Kerberos and NTLM, I see
>>> >>> >> the
>>> >>> >> same
>>> >>> >> behavior. Whereas ManifoldCF throws an exception when trying to
>>> >>> >> define
>>> >>> >> the
>>> >>> >> repository connection.
>>> >>> >>
>>> >>> >> Please advise me on what I should be doing next to resolve this.
>>> >>> >>
>>> >>> >> Thanks and Regards,
>>> >>> >> Swapna.
>>> >>> >>
>>> >>> >>
>>> >>> >>
>>> >>> >>
>>> >>> >> On Tue, Dec 6, 2011 at 12:55 PM, Karl Wright <da...@gmail.com>
>>> >>> >> wrote:
>>> >>> >>>
>>> >>> >>> About your capture - Michael Allen says the following:
>>> >>> >>>
>>> >>> >>> "Actually this has nothing to do with DFS. JCIFS does not get to
>>> >>> >>> the
>>> >>> >>> point where it does DFS anything. The capture shows a vanilla
>>> >>> >>> STATUS_LOGON_FAILURE when GLOBAL\swapna.vuppala tries to auth
>>> >>> >>> with
>>> >>> >>> l-carx01.global.arup.com. So the possible causes for this are 1)
>>> >>> >>> the
>>> >>> >>> account name is not valid 2) the supplied password is incorrect
>>> >>> >>> 3)
>>> >>> >>> some security policy is deliberately blocking that user or
>>> >>> >>> particular
>>> >>> >>> type of auth or 4) some server configuration is incompatible with
>>> >>> >>> JCIFS. I only mention this last option because I noticed the
>>> >>> >>> target
>>> >>> >>> server has security signatures disabled. That's strange. If
>>> >>> >>> they're
>>> >>> >>> messing around with things like that, who knows what their
>>> >>> >>> clients are
>>> >>> >>> expected to do.
>>> >>> >>>
>>> >>> >>> Try a Windows client that uses NTLM instead of Kerberos. Meaning
>>> >>> >>> try a
>>> >>> >>> machine that is not joined to the domain so that when you try to
>>> >>> >>> access the target it asks you for credentials at which point you
>>> >>> >>> can
>>> >>> >>> test with GLOBAL\swapna.vuppala. Then it will use NTLM and you
>>> >>> >>> can
>>> >>> >>> actually compare captures. If the operator doesn't have a laptop
>>> >>> >>> or
>>> >>> >>> something not joined to the domain, it might be sufficient to log
>>> >>> >>> into
>>> >>> >>> a workstation using machine credentials and not domain
>>> >>> >>> credentials.
>>> >>> >>>
>>> >>> >>> Also when testing JCIFS you should use a simple stand-alone
>>> >>> >>> program
>>> >>> >>> like examples/ListFiles.java."
>>> >>> >>>
>>> >>> >>> In other words:
>>> >>> >>> (a) Since JCIFS does not use Kerberos for authentication, you
>>> >>> >>> need to
>>> >>> >>> try to log into the recalcitrant server via Windows without using
>>> >>> >>> Kerberos to be able to do a side-by-side comparison. Michael has
>>> >>> >>> some
>>> >>> >>> ways of doing that, above.
>>> >>> >>> (b) You may find that it doesn't work, in which case JCIFS is not
>>> >>> >>> going to work either.
>>> >>> >>> (c) If it *does* work, then try to generate your side-by-side
>>> >>> >>> comparisons using a simpler example rather than ManifoldCF en
>>> >>> >>> toto;
>>> >>> >>> you can see how at jcifs.samba.org, or I can help you further.
>>> >>> >>>
>>> >>> >>> He also mentions that there is some bizarreness on the response
>>> >>> >>> that
>>> >>> >>> indicates that the server is configured in a way that he's never
>>> >>> >>> seen
>>> >>> >>> before. And believe me, Michael has seen a *lot* of strange
>>> >>> >>> configurations...
>>> >>> >>>
>>> >>> >>> Hope this helps.
>>> >>> >>> Karl
>>> >>> >>>
>>> >>> >>> On Mon, Nov 28, 2011 at 4:12 AM, Karl Wright <da...@gmail.com>
>>> >>> >>> wrote:
>>> >>> >>> > That should read "properties.xml", not "properties.ini". It
>>> >>> >>> > looks
>>> >>> >>> > like this page needs updating.
>>> >>> >>> >
>>> >>> >>> > The debug property in the XML form is:
>>> >>> >>> >
>>> >>> >>> > <property name="org.apache.manifoldcf.connectors"
>>> >>> >>> > value="DEBUG"/>
>>> >>> >>> >
>>> >>> >>> > I don't think it will provide you with any additional
>>> >>> >>> > information
>>> >>> >>> > that
>>> >>> >>> > is useful for debugging your authentication issue, however, if
>>> >>> >>> > that
>>> >>> >>> > is
>>> >>> >>> > why you are looking at it. There may be some jcifs.jar
>>> >>> >>> > debugging
>>> >>> >>> > switches that might be of more help, but in the end I suspect
>>> >>> >>> > you
>>> >>> >>> > will
>>> >>> >>> > need a packet capture of both a successful connection (via
>>> >>> >>> > Windows)
>>> >>> >>> > and an unsuccessful one (via MCF). The guy you will need to
>>> >>> >>> > talk
>>> >>> >>> > with
>>> >>> >>> > after that is the jcifs author Michael Allen; I can give you
>>> >>> >>> > his
>>> >>> >>> > email
>>> >>> >>> > address if you get that far.
>>> >>> >>> >
>>> >>> >>> > Karl
>>> >>> >>> >
>>> >>> >>> >
>>> >>> >>> > On Mon, Nov 28, 2011 at 1:30 AM, Swapna Vuppala
>>> >>> >>> > <sw...@gmail.com> wrote:
>>> >>> >>> >> Hi Karl,
>>> >>> >>> >>
>>> >>> >>> >> I was planning to debug jCIFS repository connection using
>>> >>> >>> >> WireShark
>>> >>> >>> >> and
>>> >>> >>> >> I
>>> >>> >>> >> came across this
>>> >>> >>> >> https://cwiki.apache.org/CONNECTORS/debugging-connections.html
>>> >>> >>> >> Here, I see something as add
>>> >>> >>> >> "org.apache.manifoldcf.connectors=DEBUG"
>>> >>> >>> >> to the
>>> >>> >>> >> properties.ini file. Is it the properties.xml file that is
>>> >>> >>> >> being
>>> >>> >>> >> referred
>>> >>> >>> >> here ? If not, where do I find properties.ini file ?
>>> >>> >>> >>
>>> >>> >>> >> Thanks and Regards,
>>> >>> >>> >> Swapna.
>>> >>> >>> >>
>>> >>> >>> >> On Thu, Nov 17, 2011 at 1:31 PM, Karl Wright
>>> >>> >>> >> <da...@gmail.com>
>>> >>> >>> >> wrote:
>>> >>> >>> >>>
>>> >>> >>> >>> See
>>> >>> >>> >>> http://jcifs.samba.org/src/docs/api/overview-summary.html#scp.
>>> >>> >>> >>> The properties jcifs.smb.lmCompatibility and
>>> >>> >>> >>> jcifs.smb.client.useExtendedSecurity are the ones you may
>>> >>> >>> >>> want to
>>> >>> >>> >>> change. These two properties go together so certain
>>> >>> >>> >>> combinations
>>> >>> >>> >>> make
>>> >>> >>> >>> sense and others don't, so there's really only combinations
>>> >>> >>> >>> you
>>> >>> >>> >>> need
>>> >>> >>> >>> but I'll need to look at what they are and get back to you
>>> >>> >>> >>> later
>>> >>> >>> >>> today.
>>> >>> >>> >>>
>>> >>> >>> >>> As far as setting the switches are concerned, if you are
>>> >>> >>> >>> using the
>>> >>> >>> >>> Quick Start you do this trivially by:
>>> >>> >>> >>>
>>> >>> >>> >>> <java> -Dxxx -Dyyy -jar start.jar
>>> >>> >>> >>>
>>> >>> >>> >>> If you are using the multi-process configuration, that is
>>> >>> >>> >>> what the
>>> >>> >>> >>> "defines" directory is for; you only need to create files in
>>> >>> >>> >>> that
>>> >>> >>> >>> directory with the names "jcifs.smb.lmCompatibility" and
>>> >>> >>> >>> "jcifs.smb.client.useExtendedSecurity" containing the values
>>> >>> >>> >>> you
>>> >>> >>> >>> want
>>> >>> >>> >>> to set.
>>> >>> >>> >>>
>>> >>> >>> >>> Karl
>>> >>> >>> >>>
>>> >>> >>> >>>
>>> >>> >>> >>> On Thu, Nov 17, 2011 at 1:11 AM, Swapna Vuppala
>>> >>> >>> >>> <sw...@gmail.com> wrote:
>>> >>> >>> >>> > Hi Karl,
>>> >>> >>> >>> >
>>> >>> >>> >>> > Am able to access the folders on the problem server through
>>> >>> >>> >>> > windows
>>> >>> >>> >>> > explorer, (\\server3\Folder1). I tried couple of things
>>> >>> >>> >>> > with the
>>> >>> >>> >>> > credentials
>>> >>> >>> >>> > form, changing username, domain etc.. but I keep getting
>>> >>> >>> >>> > the
>>> >>> >>> >>> > same
>>> >>> >>> >>> > error
>>> >>> >>> >>> > "Couldn't connect to server: Logon failure: unknown user
>>> >>> >>> >>> > name or
>>> >>> >>> >>> > bad
>>> >>> >>> >>> > password"
>>> >>> >>> >>> >
>>> >>> >>> >>> > Can you tell me more about the -D switch you were talking
>>> >>> >>> >>> > of ?
>>> >>> >>> >>> >
>>> >>> >>> >>> > Thanks and Regards,
>>> >>> >>> >>> > Swapna.
>>> >>> >>> >>> >
>>> >>> >>> >>> > On Tue, Nov 15, 2011 at 12:40 PM, Karl Wright
>>> >>> >>> >>> > <da...@gmail.com>
>>> >>> >>> >>> > wrote:
>>> >>> >>> >>> >>
>>> >>> >>> >>> >> Glad you chased it down this far.
>>> >>> >>> >>> >>
>>> >>> >>> >>> >> First thing to try is whether you can get into the problem
>>> >>> >>> >>> >> server
>>> >>> >>> >>> >> using Windows Explorer. Obviously ManifoldCF is not going
>>> >>> >>> >>> >> to
>>> >>> >>> >>> >> be
>>> >>> >>> >>> >> able
>>> >>> >>> >>> >> to do it if Windows can't. If you *can* get in, then just
>>> >>> >>> >>> >> playing
>>> >>> >>> >>> >> with the form of the credentials in the MCF connection
>>> >>> >>> >>> >> might do
>>> >>> >>> >>> >> the
>>> >>> >>> >>> >> trick. Some Windows or net appliance servers are picky
>>> >>> >>> >>> >> about
>>> >>> >>> >>> >> this.
>>> >>> >>> >>> >> Try various things like leaving the domain blank and
>>> >>> >>> >>> >> specifying
>>> >>> >>> >>> >> the
>>> >>> >>> >>> >> user as "abc@domain.com", for instance. There's also a
>>> >>> >>> >>> >> different
>>> >>> >>> >>> >> NTLM
>>> >>> >>> >>> >> mode you can operation jcifs in that some servers may be
>>> >>> >>> >>> >> configured
>>> >>> >>> >>> >> to
>>> >>> >>> >>> >> require; this would need you to set a -D switch on the
>>> >>> >>> >>> >> command
>>> >>> >>> >>> >> line
>>> >>> >>> >>> >> to
>>> >>> >>> >>> >> enable.
>>> >>> >>> >>> >>
>>> >>> >>> >>> >> Karl
>>> >>> >>> >>> >>
>>> >>> >>> >>> >> On Tue, Nov 15, 2011 at 12:10 AM, Swapna Vuppala
>>> >>> >>> >>> >> <sw...@gmail.com> wrote:
>>> >>> >>> >>> >> > Hi Karl,
>>> >>> >>> >>> >> >
>>> >>> >>> >>> >> > Thanks for the input. It looks like my problem is
>>> >>> >>> >>> >> > related to
>>> >>> >>> >>> >> > the
>>> >>> >>> >>> >> > second
>>> >>> >>> >>> >> > one
>>> >>> >>> >>> >> > that you specified. One of the directories in the path
>>> >>> >>> >>> >> > am
>>> >>> >>> >>> >> > trying
>>> >>> >>> >>> >> > to
>>> >>> >>> >>> >> > index is
>>> >>> >>> >>> >> > actually redirecting to a different server. And when I
>>> >>> >>> >>> >> > specify
>>> >>> >>> >>> >> > this
>>> >>> >>> >>> >> > new
>>> >>> >>> >>> >> > server in defining the repository connection, with my
>>> >>> >>> >>> >> > credentials,
>>> >>> >>> >>> >> > the
>>> >>> >>> >>> >> > connection fails with the message: "Couldn't connect to
>>> >>> >>> >>> >> > server:
>>> >>> >>> >>> >> > Logon
>>> >>> >>> >>> >> > failure: unknown user name or bad password"
>>> >>> >>> >>> >> >
>>> >>> >>> >>> >> > I'll look into why am not able to connect to this
>>> >>> >>> >>> >> > server.
>>> >>> >>> >>> >> >
>>> >>> >>> >>> >> > Thanks and Regards,
>>> >>> >>> >>> >> > Swapna.
>>> >>> >>> >>> >> >
>>> >>> >>> >>> >> > On Mon, Nov 14, 2011 at 4:56 PM, Karl Wright
>>> >>> >>> >>> >> > <da...@gmail.com>
>>> >>> >>> >>> >> > wrote:
>>> >>> >>> >>> >> >>
>>> >>> >>> >>> >> >> There's two kinds of problem you might be having. The
>>> >>> >>> >>> >> >> first
>>> >>> >>> >>> >> >> is
>>> >>> >>> >>> >> >> intermittent, and the second is not intermittent but
>>> >>> >>> >>> >> >> would
>>> >>> >>> >>> >> >> have
>>> >>> >>> >>> >> >> something to do with specific directories.
>>> >>> >>> >>> >> >>
>>> >>> >>> >>> >> >> Intermittent problems might include a domain controller
>>> >>> >>> >>> >> >> that
>>> >>> >>> >>> >> >> is
>>> >>> >>> >>> >> >> not
>>> >>> >>> >>> >> >> always accessible. In such cases, the crawl will
>>> >>> >>> >>> >> >> proceed
>>> >>> >>> >>> >> >> but
>>> >>> >>> >>> >> >> will
>>> >>> >>> >>> >> >> tend to fail unpredictably. On the other hand, if you
>>> >>> >>> >>> >> >> have
>>> >>> >>> >>> >> >> a
>>> >>> >>> >>> >> >> directory that is handled by a DFS redirection, it is
>>> >>> >>> >>> >> >> possible
>>> >>> >>> >>> >> >> that
>>> >>> >>> >>> >> >> the redirection is indicating a new server (lets call
>>> >>> >>> >>> >> >> it
>>> >>> >>> >>> >> >> server3)
>>> >>> >>> >>> >> >> which may not like the precise form of your login
>>> >>> >>> >>> >> >> credentials.
>>> >>> >>> >>> >> >> Can
>>> >>> >>> >>> >> >> you determine which scenario you are seeing?
>>> >>> >>> >>> >> >>
>>> >>> >>> >>> >> >> Karl
>>> >>> >>> >>> >> >>
>>> >>> >>> >>> >> >> On Mon, Nov 14, 2011 at 3:11 AM, Swapna Vuppala
>>> >>> >>> >>> >> >> <sw...@gmail.com> wrote:
>>> >>> >>> >>> >> >> > Hi,
>>> >>> >>> >>> >> >> >
>>> >>> >>> >>> >> >> > I have been using windows share repository connection
>>> >>> >>> >>> >> >> > to
>>> >>> >>> >>> >> >> > crawl
>>> >>> >>> >>> >> >> > and
>>> >>> >>> >>> >> >> > get
>>> >>> >>> >>> >> >> > data
>>> >>> >>> >>> >> >> > from a particular server (server 1). Its working
>>> >>> >>> >>> >> >> > perfectly
>>> >>> >>> >>> >> >> > fine.
>>> >>> >>> >>> >> >> > However, am
>>> >>> >>> >>> >> >> > having trouble when I try with data from another
>>> >>> >>> >>> >> >> > server
>>> >>> >>> >>> >> >> > (server
>>> >>> >>> >>> >> >> > 2).
>>> >>> >>> >>> >> >> >
>>> >>> >>> >>> >> >> > When I define a repository connection of type windows
>>> >>> >>> >>> >> >> > share
>>> >>> >>> >>> >> >> > and
>>> >>> >>> >>> >> >> > specify
>>> >>> >>> >>> >> >> > the
>>> >>> >>> >>> >> >> > server name (server 2) with my credentials, the
>>> >>> >>> >>> >> >> > connection
>>> >>> >>> >>> >> >> > status
>>> >>> >>> >>> >> >> > shows
>>> >>> >>> >>> >> >> > "Connection working". But when I run a job to use
>>> >>> >>> >>> >> >> > this
>>> >>> >>> >>> >> >> > repository
>>> >>> >>> >>> >> >> > connection
>>> >>> >>> >>> >> >> > and index data from a location on this server 2, I
>>> >>> >>> >>> >> >> > keep
>>> >>> >>> >>> >> >> > getting
>>> >>> >>> >>> >> >> > the
>>> >>> >>> >>> >> >> > exception below:
>>> >>> >>> >>> >> >> >
>>> >>> >>> >>> >> >> > JCIFS: Possibly transient exception detected on
>>> >>> >>> >>> >> >> > attempt 3
>>> >>> >>> >>> >> >> > while
>>> >>> >>> >>> >> >> > checking
>>> >>> >>> >>> >> >> > if
>>> >>> >>> >>> >> >> > file exists: Logon failure: unknown user name or bad
>>> >>> >>> >>> >> >> > password.
>>> >>> >>> >>> >> >> > jcifs.smb.SmbAuthException: Logon failure: unknown
>>> >>> >>> >>> >> >> > user
>>> >>> >>> >>> >> >> > name
>>> >>> >>> >>> >> >> > or
>>> >>> >>> >>> >> >> > bad
>>> >>> >>> >>> >> >> > password.
>>> >>> >>> >>> >> >> > at
>>> >>> >>> >>> >> >> >
>>> >>> >>> >>> >> >> > jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:544)
>>> >>> >>> >>> >> >> > at
>>> >>> >>> >>> >> >> > jcifs.smb.SmbTransport.send(SmbTransport.java:661)
>>> >>> >>> >>> >> >> > at
>>> >>> >>> >>> >> >> >
>>> >>> >>> >>> >> >> > jcifs.smb.SmbSession.sessionSetup(SmbSession.java:390)
>>> >>> >>> >>> >> >> > at jcifs.smb.SmbSession.send(SmbSession.java:218)
>>> >>> >>> >>> >> >> > at
>>> >>> >>> >>> >> >> > jcifs.smb.SmbTree.treeConnect(SmbTree.java:176)
>>> >>> >>> >>> >> >> > at jcifs.smb.SmbFile.doConnect(SmbFile.java:911)
>>> >>> >>> >>> >> >> > at jcifs.smb.SmbFile.connect(SmbFile.java:954)
>>> >>> >>> >>> >> >> > at jcifs.smb.SmbFile.connect0(SmbFile.java:880)
>>> >>> >>> >>> >> >> > at jcifs.smb.SmbFile.queryPath(SmbFile.java:1335)
>>> >>> >>> >>> >> >> > at jcifs.smb.SmbFile.exists(SmbFile.java:1417)
>>> >>> >>> >>> >> >> > at
>>> >>> >>> >>> >> >> >
>>> >>> >>> >>> >> >> >
>>> >>> >>> >>> >> >> >
>>> >>> >>> >>> >> >> >
>>> >>> >>> >>> >> >> >
>>> >>> >>> >>> >> >> >
>>> >>> >>> >>> >> >> > org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.fileExists(SharedDriveConnector.java:2064)
>>> >>> >>> >>> >> >> > at
>>> >>> >>> >>> >> >> >
>>> >>> >>> >>> >> >> >
>>> >>> >>> >>> >> >> >
>>> >>> >>> >>> >> >> >
>>> >>> >>> >>> >> >> >
>>> >>> >>> >>> >> >> >
>>> >>> >>> >>> >> >> > org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getDocumentVersions(SharedDriveConnector.java:521)
>>> >>> >>> >>> >> >> > at
>>> >>> >>> >>> >> >> >
>>> >>> >>> >>> >> >> >
>>> >>> >>> >>> >> >> >
>>> >>> >>> >>> >> >> >
>>> >>> >>> >>> >> >> >
>>> >>> >>> >>> >> >> >
>>> >>> >>> >>> >> >> > org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:318)
>>> >>> >>> >>> >> >> >
>>> >>> >>> >>> >> >> > Am able to access this location from windows
>>> >>> >>> >>> >> >> > explorer.
>>> >>> >>> >>> >> >> > What
>>> >>> >>> >>> >> >> > else
>>> >>> >>> >>> >> >> > should
>>> >>> >>> >>> >> >> > I be
>>> >>> >>> >>> >> >> > checking or what could be the reasons/factors causing
>>> >>> >>> >>> >> >> > this
>>> >>> >>> >>> >> >> > to
>>> >>> >>> >>> >> >> > fail
>>> >>> >>> >>> >> >> > ?
>>> >>> >>> >>> >> >> >
>>> >>> >>> >>> >> >> > Thanks and Regards,
>>> >>> >>> >>> >> >> > Swapna.
>>> >>> >>> >>> >> >> >
>>> >>> >>> >>> >> >
>>> >>> >>> >>> >> >
>>> >>> >>> >>> >
>>> >>> >>> >>> >
>>> >>> >>> >>
>>> >>> >>> >>
>>> >>> >>
>>> >>> >>
>>> >>
>>> >>
>>
>>
>
>
Re: Problem crawling windows share
Posted by Swapna Vuppala <sw...@gmail.com>.
Hi Karl,
This is with respect to a quite old issue that we had discussed for some
time and left.
The issue got solved for us by setting the property
"org.apache.manifoldcf.crawler.connectors.jcifs.usentlmv1" to "true" in
properties.xml. Just wanted to update you on this.
With this property set, we are able to crawl the windows share (mentioned
in previous mails) successfully and index them into Solr.
Thanks for all the help.
Thanks and Regards,
Swapna.
On Tue, Dec 13, 2011 at 11:15 AM, Swapna Vuppala <swapna.kollipara@gmail.com
> wrote:
> Thanks Karl for getting this far.
>
> I understand that the chances of jCIFS working with this particular server
> are very less. We'll leave it here for now and I'll let you know if I have
> any more details on the server.
>
> Thanks and Regards,
> Swapna.
>
>
> On Tue, Dec 13, 2011 at 7:24 AM, Karl Wright <da...@gmail.com> wrote:
>
>> His response:
>>
>> "Actually from looking at the capture again, the successful operations
>> in it were ultimately performed by a session that used Kerberos. But
>> there are two other successful NTLM authentications that use the
>> Anonymous credential. It so happens that neither of these went on to
>> do anything successful. Meaning your capture does not contain a
>> successful operation using NTLM and thus there is nothing to compare
>> JCIFS too. It could be that the environment actually does not support
>> NTLM properly. There are a lot of errors and oddities in the non-JCIFS
>> communications with numerous login failures, disabled signatures,
>> access denied conditions and unsupported versions of Windows (Windows
>> Server 2002 SP2 build 2600). The environment is not optimal.
>>
>> To continue I would need proper comparison captures. Clearly the
>> NtlmPasswordAuthentication("","","") does not work. But
>> NtlmPasswordAuthentication.ANONYMOUS obviously does work to some
>> degree but there's something else wrong with the session setup
>> request. That something could be the fact that the hostname used is
>> actually l-carx01.global.arup.com and not lon-data-arx.global.arup.com
>> and then the tree connect goes on to mount
>> //lon-data-arx.global.arup.com/stg-jobs$ so clearly there is some
>> hostname discontinuity going on as well."
>>
>> I think this is pretty much self explanatory. Whatever it is that you
>> are trying to talk to seems to be one truly crazy machine, certainly
>> not really Windows, and it is almost an accident that it works with
>> Windows at all. I think this is rapidly reaching the point of
>> diminishing returns in getting jCIFS to work with it.
>>
>> Karl
>>
>>
>> On Mon, Dec 12, 2011 at 4:24 AM, Karl Wright <da...@gmail.com> wrote:
>> > I forwarded this research again to Mr. Allen for further suggestions.
>> Thanks!
>> >
>> > Karl
>> >
>> > On Mon, Dec 12, 2011 at 1:58 AM, Swapna Vuppala
>> > <sw...@gmail.com> wrote:
>> >> Hi Karl,
>> >>
>> >> I tried modifying the test code as you suggested.
>> >>
>> >> When I tried with empty credentials
>> >>
>> >> NtlmPasswordAuthentication pa = new
>> NtlmPasswordAuthentication("","","");
>> >> SmbFile smbconnection = new SmbFile("smb://" +
>> >> "lon-data-arx.global.arup.com" + "/" + "stg-jobs$",pa);
>> >>
>> >> I got the same exception as before
>> >>
>> >> Exception in thread "main" jcifs.smb.SmbAuthException: Logon failure:
>> >> unknown us
>> >>
>> >> er name or bad password.
>> >> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:546)
>> >> at jcifs.smb.SmbTransport.send(SmbTransport.java:663)
>> >>
>> >> at jcifs.smb.SmbSession.sessionSetup(SmbSession.java:390)
>> >> at jcifs.smb.SmbSession.send(SmbSession.java:218)
>> >> at jcifs.smb.SmbTree.treeConnect(SmbTree.java:176)
>> >> at jcifs.smb.SmbFile.doConnect(SmbFile.java:911)
>> >>
>> >> ......
>> >>
>> >> When I tried with NtlmPasswordAuthentication.ANONYMOUS
>> >>
>> >> SmbFile smbconnection = new SmbFile("smb://" +
>> >> "lon-data-arx.global.arup.com" + "/" +
>> >> "stg-jobs$",NtlmPasswordAuthentication.ANONYMOUS);
>> >>
>> >> I got a different exception
>> >>
>> >> Exception in thread "main" jcifs.smb.SmbException: Incorrect function.
>> >> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:563)
>> >> at jcifs.smb.SmbTransport.send(SmbTransport.java:663)
>> >> at jcifs.smb.SmbSession.sessionSetup(SmbSession.java:316)
>> >>
>> >> at jcifs.smb.SmbSession.send(SmbSession.java:218)
>> >> at jcifs.smb.SmbTree.treeConnect(SmbTree.java:176)
>> >> at jcifs.smb.SmbFile.doConnect(SmbFile.java:911)
>> >> .....
>> >>
>> >> Thanks and Regards,
>> >> Swapna.
>> >> On Fri, Dec 9, 2011 at 11:46 PM, Karl Wright <da...@gmail.com>
>> wrote:
>> >>>
>> >>> Michael Allen's response to your capture:
>> >>>
>> >>> "It looks like the non-JCIFS communication is successful because the
>> >>> anonymous credential is used.
>> >>>
>> >>> Note that there is still no sign of DFS going on anywhere. Actually
>> >>> JCIFS tries to do a referral for the DFS root and gets an error of
>> >>> STATUS_FS_DRIVER_REQUIRED so it looks like DFS may not even be setup
>> >>> at all in the target environment. At least not domain based DFS.
>> >>>
>> >>> So if you want to emulate Windows behavior, I guess you need to retry
>> >>> the operation with credential NtlmPasswordAuthentication.ANONYMOUS (or
>> >>> all empty strings should work too I think like ";:")."
>> >>>
>> >>> In other words, the issue is that the server in question does not
>> >>> actually recognize the credentials given at all, and gives an error.
>> >>> But while Windows then retries with no credentials whatsoever, nothing
>> >>> in JCIFS does that retry.
>> >>>
>> >>> You can confirm that this is what is happening by using an empty user
>> >>> name, password, and domain in the JCIFS test code you wrote to obtain
>> >>> the captures. If, then, the test code works against that server, I'll
>> >>> create a ticket for downgrading to ANONYMOUS whenever there is an
>> >>> authentication error in the JCIFS connector.
>> >>>
>> >>> Thanks,
>> >>> Karl
>> >>>
>> >>> On Wed, Dec 7, 2011 at 6:41 AM, Karl Wright <da...@gmail.com>
>> wrote:
>> >>> > The next step is to obtain two packet captures using WireShark.
>> These
>> >>> > should both be done from the machine that is not joined to the
>> domain.
>> >>> > The first capture should be the interaction that takes place when
>> you
>> >>> > try to go to the foreign directory, and you supply the credentials,
>> >>> > and then you list the folders. The second capture will require you
>> to
>> >>> > do the same using JCIFS. For that capture, you will need jcifs.jar
>> >>> > and a simple example class that provides jcifs with credentials and
>> >>> > tries to list the directory. A simple class should be enough to do
>> >>> > this, including code such as:
>> >>> >
>> >>> > import jcifs.smb.*;
>> >>> >
>> >>> > NtlmPasswordAuthentication pa = new
>> >>> > NtlmPasswordAuthentication(domain,username,password);
>> >>> > SmbFile smbconnection = new SmbFile("smb://" + server + "/" +
>> share,pa);
>> >>> > SmbFile[] files = smbconnection.listFiles();
>> >>> >
>> >>> > for (int i = 0 ; i < files.length ; i++)
>> >>> > {
>> >>> > System.out.println(file.toString());
>> >>> > }
>> >>> >
>> >>> > I suggest you put together your test class first and make sure it
>> >>> > works, and then go after the captures.
>> >>> >
>> >>> > Let me know how it goes.
>> >>> >
>> >>> > Karl
>> >>> >
>> >>> > On Wed, Dec 7, 2011 at 6:10 AM, Swapna Vuppala
>> >>> > <sw...@gmail.com> wrote:
>> >>> >> Hi Karl,
>> >>> >>
>> >>> >> Thanks for the detailed reply.
>> >>> >>
>> >>> >> As per your suggestion, I tried connecting to the problematic
>> server
>> >>> >> via
>> >>> >> windows using a machine that is not in the domain. On supplying the
>> >>> >> credentials via windows, I could connect to the server, although
>> it did
>> >>> >> not
>> >>> >> list any folders. But when I tried accessing the folder of my
>> interest
>> >>> >> via
>> >>> >> \\server\folder$, I could see all the sub folders. All this is
>> when I
>> >>> >> connected via windows from a machine that's not joined to the
>> domain.
>> >>> >>
>> >>> >> The behavior is same when I connect to the server via windows from
>> a
>> >>> >> machine
>> >>> >> that's joined to the domain.
>> >>> >>
>> >>> >> So I assume this means that through both Kerberos and NTLM, I see
>> the
>> >>> >> same
>> >>> >> behavior. Whereas ManifoldCF throws an exception when trying to
>> define
>> >>> >> the
>> >>> >> repository connection.
>> >>> >>
>> >>> >> Please advise me on what I should be doing next to resolve this.
>> >>> >>
>> >>> >> Thanks and Regards,
>> >>> >> Swapna.
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >> On Tue, Dec 6, 2011 at 12:55 PM, Karl Wright <da...@gmail.com>
>> >>> >> wrote:
>> >>> >>>
>> >>> >>> About your capture - Michael Allen says the following:
>> >>> >>>
>> >>> >>> "Actually this has nothing to do with DFS. JCIFS does not get to
>> the
>> >>> >>> point where it does DFS anything. The capture shows a vanilla
>> >>> >>> STATUS_LOGON_FAILURE when GLOBAL\swapna.vuppala tries to auth with
>> >>> >>> l-carx01.global.arup.com. So the possible causes for this are 1)
>> the
>> >>> >>> account name is not valid 2) the supplied password is incorrect 3)
>> >>> >>> some security policy is deliberately blocking that user or
>> particular
>> >>> >>> type of auth or 4) some server configuration is incompatible with
>> >>> >>> JCIFS. I only mention this last option because I noticed the
>> target
>> >>> >>> server has security signatures disabled. That's strange. If
>> they're
>> >>> >>> messing around with things like that, who knows what their
>> clients are
>> >>> >>> expected to do.
>> >>> >>>
>> >>> >>> Try a Windows client that uses NTLM instead of Kerberos. Meaning
>> try a
>> >>> >>> machine that is not joined to the domain so that when you try to
>> >>> >>> access the target it asks you for credentials at which point you
>> can
>> >>> >>> test with GLOBAL\swapna.vuppala. Then it will use NTLM and you can
>> >>> >>> actually compare captures. If the operator doesn't have a laptop
>> or
>> >>> >>> something not joined to the domain, it might be sufficient to log
>> into
>> >>> >>> a workstation using machine credentials and not domain
>> credentials.
>> >>> >>>
>> >>> >>> Also when testing JCIFS you should use a simple stand-alone
>> program
>> >>> >>> like examples/ListFiles.java."
>> >>> >>>
>> >>> >>> In other words:
>> >>> >>> (a) Since JCIFS does not use Kerberos for authentication, you
>> need to
>> >>> >>> try to log into the recalcitrant server via Windows without using
>> >>> >>> Kerberos to be able to do a side-by-side comparison. Michael has
>> some
>> >>> >>> ways of doing that, above.
>> >>> >>> (b) You may find that it doesn't work, in which case JCIFS is not
>> >>> >>> going to work either.
>> >>> >>> (c) If it *does* work, then try to generate your side-by-side
>> >>> >>> comparisons using a simpler example rather than ManifoldCF en
>> toto;
>> >>> >>> you can see how at jcifs.samba.org, or I can help you further.
>> >>> >>>
>> >>> >>> He also mentions that there is some bizarreness on the response
>> that
>> >>> >>> indicates that the server is configured in a way that he's never
>> seen
>> >>> >>> before. And believe me, Michael has seen a *lot* of strange
>> >>> >>> configurations...
>> >>> >>>
>> >>> >>> Hope this helps.
>> >>> >>> Karl
>> >>> >>>
>> >>> >>> On Mon, Nov 28, 2011 at 4:12 AM, Karl Wright <da...@gmail.com>
>> >>> >>> wrote:
>> >>> >>> > That should read "properties.xml", not "properties.ini". It
>> looks
>> >>> >>> > like this page needs updating.
>> >>> >>> >
>> >>> >>> > The debug property in the XML form is:
>> >>> >>> >
>> >>> >>> > <property name="org.apache.manifoldcf.connectors"
>> value="DEBUG"/>
>> >>> >>> >
>> >>> >>> > I don't think it will provide you with any additional
>> information
>> >>> >>> > that
>> >>> >>> > is useful for debugging your authentication issue, however, if
>> that
>> >>> >>> > is
>> >>> >>> > why you are looking at it. There may be some jcifs.jar
>> debugging
>> >>> >>> > switches that might be of more help, but in the end I suspect
>> you
>> >>> >>> > will
>> >>> >>> > need a packet capture of both a successful connection (via
>> Windows)
>> >>> >>> > and an unsuccessful one (via MCF). The guy you will need to
>> talk
>> >>> >>> > with
>> >>> >>> > after that is the jcifs author Michael Allen; I can give you his
>> >>> >>> > email
>> >>> >>> > address if you get that far.
>> >>> >>> >
>> >>> >>> > Karl
>> >>> >>> >
>> >>> >>> >
>> >>> >>> > On Mon, Nov 28, 2011 at 1:30 AM, Swapna Vuppala
>> >>> >>> > <sw...@gmail.com> wrote:
>> >>> >>> >> Hi Karl,
>> >>> >>> >>
>> >>> >>> >> I was planning to debug jCIFS repository connection using
>> WireShark
>> >>> >>> >> and
>> >>> >>> >> I
>> >>> >>> >> came across this
>> >>> >>> >> https://cwiki.apache.org/CONNECTORS/debugging-connections.html
>> >>> >>> >> Here, I see something as add
>> >>> >>> >> "org.apache.manifoldcf.connectors=DEBUG"
>> >>> >>> >> to the
>> >>> >>> >> properties.ini file. Is it the properties.xml file that is
>> being
>> >>> >>> >> referred
>> >>> >>> >> here ? If not, where do I find properties.ini file ?
>> >>> >>> >>
>> >>> >>> >> Thanks and Regards,
>> >>> >>> >> Swapna.
>> >>> >>> >>
>> >>> >>> >> On Thu, Nov 17, 2011 at 1:31 PM, Karl Wright <
>> daddywri@gmail.com>
>> >>> >>> >> wrote:
>> >>> >>> >>>
>> >>> >>> >>> See
>> http://jcifs.samba.org/src/docs/api/overview-summary.html#scp.
>> >>> >>> >>> The properties jcifs.smb.lmCompatibility and
>> >>> >>> >>> jcifs.smb.client.useExtendedSecurity are the ones you may
>> want to
>> >>> >>> >>> change. These two properties go together so certain
>> combinations
>> >>> >>> >>> make
>> >>> >>> >>> sense and others don't, so there's really only combinations
>> you
>> >>> >>> >>> need
>> >>> >>> >>> but I'll need to look at what they are and get back to you
>> later
>> >>> >>> >>> today.
>> >>> >>> >>>
>> >>> >>> >>> As far as setting the switches are concerned, if you are
>> using the
>> >>> >>> >>> Quick Start you do this trivially by:
>> >>> >>> >>>
>> >>> >>> >>> <java> -Dxxx -Dyyy -jar start.jar
>> >>> >>> >>>
>> >>> >>> >>> If you are using the multi-process configuration, that is
>> what the
>> >>> >>> >>> "defines" directory is for; you only need to create files in
>> that
>> >>> >>> >>> directory with the names "jcifs.smb.lmCompatibility" and
>> >>> >>> >>> "jcifs.smb.client.useExtendedSecurity" containing the values
>> you
>> >>> >>> >>> want
>> >>> >>> >>> to set.
>> >>> >>> >>>
>> >>> >>> >>> Karl
>> >>> >>> >>>
>> >>> >>> >>>
>> >>> >>> >>> On Thu, Nov 17, 2011 at 1:11 AM, Swapna Vuppala
>> >>> >>> >>> <sw...@gmail.com> wrote:
>> >>> >>> >>> > Hi Karl,
>> >>> >>> >>> >
>> >>> >>> >>> > Am able to access the folders on the problem server through
>> >>> >>> >>> > windows
>> >>> >>> >>> > explorer, (\\server3\Folder1). I tried couple of things
>> with the
>> >>> >>> >>> > credentials
>> >>> >>> >>> > form, changing username, domain etc.. but I keep getting the
>> >>> >>> >>> > same
>> >>> >>> >>> > error
>> >>> >>> >>> > "Couldn't connect to server: Logon failure: unknown user
>> name or
>> >>> >>> >>> > bad
>> >>> >>> >>> > password"
>> >>> >>> >>> >
>> >>> >>> >>> > Can you tell me more about the -D switch you were talking
>> of ?
>> >>> >>> >>> >
>> >>> >>> >>> > Thanks and Regards,
>> >>> >>> >>> > Swapna.
>> >>> >>> >>> >
>> >>> >>> >>> > On Tue, Nov 15, 2011 at 12:40 PM, Karl Wright
>> >>> >>> >>> > <da...@gmail.com>
>> >>> >>> >>> > wrote:
>> >>> >>> >>> >>
>> >>> >>> >>> >> Glad you chased it down this far.
>> >>> >>> >>> >>
>> >>> >>> >>> >> First thing to try is whether you can get into the problem
>> >>> >>> >>> >> server
>> >>> >>> >>> >> using Windows Explorer. Obviously ManifoldCF is not going
>> to
>> >>> >>> >>> >> be
>> >>> >>> >>> >> able
>> >>> >>> >>> >> to do it if Windows can't. If you *can* get in, then just
>> >>> >>> >>> >> playing
>> >>> >>> >>> >> with the form of the credentials in the MCF connection
>> might do
>> >>> >>> >>> >> the
>> >>> >>> >>> >> trick. Some Windows or net appliance servers are picky
>> about
>> >>> >>> >>> >> this.
>> >>> >>> >>> >> Try various things like leaving the domain blank and
>> specifying
>> >>> >>> >>> >> the
>> >>> >>> >>> >> user as "abc@domain.com", for instance. There's also a
>> >>> >>> >>> >> different
>> >>> >>> >>> >> NTLM
>> >>> >>> >>> >> mode you can operation jcifs in that some servers may be
>> >>> >>> >>> >> configured
>> >>> >>> >>> >> to
>> >>> >>> >>> >> require; this would need you to set a -D switch on the
>> command
>> >>> >>> >>> >> line
>> >>> >>> >>> >> to
>> >>> >>> >>> >> enable.
>> >>> >>> >>> >>
>> >>> >>> >>> >> Karl
>> >>> >>> >>> >>
>> >>> >>> >>> >> On Tue, Nov 15, 2011 at 12:10 AM, Swapna Vuppala
>> >>> >>> >>> >> <sw...@gmail.com> wrote:
>> >>> >>> >>> >> > Hi Karl,
>> >>> >>> >>> >> >
>> >>> >>> >>> >> > Thanks for the input. It looks like my problem is
>> related to
>> >>> >>> >>> >> > the
>> >>> >>> >>> >> > second
>> >>> >>> >>> >> > one
>> >>> >>> >>> >> > that you specified. One of the directories in the path am
>> >>> >>> >>> >> > trying
>> >>> >>> >>> >> > to
>> >>> >>> >>> >> > index is
>> >>> >>> >>> >> > actually redirecting to a different server. And when I
>> >>> >>> >>> >> > specify
>> >>> >>> >>> >> > this
>> >>> >>> >>> >> > new
>> >>> >>> >>> >> > server in defining the repository connection, with my
>> >>> >>> >>> >> > credentials,
>> >>> >>> >>> >> > the
>> >>> >>> >>> >> > connection fails with the message: "Couldn't connect to
>> >>> >>> >>> >> > server:
>> >>> >>> >>> >> > Logon
>> >>> >>> >>> >> > failure: unknown user name or bad password"
>> >>> >>> >>> >> >
>> >>> >>> >>> >> > I'll look into why am not able to connect to this server.
>> >>> >>> >>> >> >
>> >>> >>> >>> >> > Thanks and Regards,
>> >>> >>> >>> >> > Swapna.
>> >>> >>> >>> >> >
>> >>> >>> >>> >> > On Mon, Nov 14, 2011 at 4:56 PM, Karl Wright
>> >>> >>> >>> >> > <da...@gmail.com>
>> >>> >>> >>> >> > wrote:
>> >>> >>> >>> >> >>
>> >>> >>> >>> >> >> There's two kinds of problem you might be having. The
>> first
>> >>> >>> >>> >> >> is
>> >>> >>> >>> >> >> intermittent, and the second is not intermittent but
>> would
>> >>> >>> >>> >> >> have
>> >>> >>> >>> >> >> something to do with specific directories.
>> >>> >>> >>> >> >>
>> >>> >>> >>> >> >> Intermittent problems might include a domain controller
>> that
>> >>> >>> >>> >> >> is
>> >>> >>> >>> >> >> not
>> >>> >>> >>> >> >> always accessible. In such cases, the crawl will
>> proceed
>> >>> >>> >>> >> >> but
>> >>> >>> >>> >> >> will
>> >>> >>> >>> >> >> tend to fail unpredictably. On the other hand, if you
>> have
>> >>> >>> >>> >> >> a
>> >>> >>> >>> >> >> directory that is handled by a DFS redirection, it is
>> >>> >>> >>> >> >> possible
>> >>> >>> >>> >> >> that
>> >>> >>> >>> >> >> the redirection is indicating a new server (lets call it
>> >>> >>> >>> >> >> server3)
>> >>> >>> >>> >> >> which may not like the precise form of your login
>> >>> >>> >>> >> >> credentials.
>> >>> >>> >>> >> >> Can
>> >>> >>> >>> >> >> you determine which scenario you are seeing?
>> >>> >>> >>> >> >>
>> >>> >>> >>> >> >> Karl
>> >>> >>> >>> >> >>
>> >>> >>> >>> >> >> On Mon, Nov 14, 2011 at 3:11 AM, Swapna Vuppala
>> >>> >>> >>> >> >> <sw...@gmail.com> wrote:
>> >>> >>> >>> >> >> > Hi,
>> >>> >>> >>> >> >> >
>> >>> >>> >>> >> >> > I have been using windows share repository connection
>> to
>> >>> >>> >>> >> >> > crawl
>> >>> >>> >>> >> >> > and
>> >>> >>> >>> >> >> > get
>> >>> >>> >>> >> >> > data
>> >>> >>> >>> >> >> > from a particular server (server 1). Its working
>> perfectly
>> >>> >>> >>> >> >> > fine.
>> >>> >>> >>> >> >> > However, am
>> >>> >>> >>> >> >> > having trouble when I try with data from another
>> server
>> >>> >>> >>> >> >> > (server
>> >>> >>> >>> >> >> > 2).
>> >>> >>> >>> >> >> >
>> >>> >>> >>> >> >> > When I define a repository connection of type windows
>> >>> >>> >>> >> >> > share
>> >>> >>> >>> >> >> > and
>> >>> >>> >>> >> >> > specify
>> >>> >>> >>> >> >> > the
>> >>> >>> >>> >> >> > server name (server 2) with my credentials, the
>> connection
>> >>> >>> >>> >> >> > status
>> >>> >>> >>> >> >> > shows
>> >>> >>> >>> >> >> > "Connection working". But when I run a job to use this
>> >>> >>> >>> >> >> > repository
>> >>> >>> >>> >> >> > connection
>> >>> >>> >>> >> >> > and index data from a location on this server 2, I
>> keep
>> >>> >>> >>> >> >> > getting
>> >>> >>> >>> >> >> > the
>> >>> >>> >>> >> >> > exception below:
>> >>> >>> >>> >> >> >
>> >>> >>> >>> >> >> > JCIFS: Possibly transient exception detected on
>> attempt 3
>> >>> >>> >>> >> >> > while
>> >>> >>> >>> >> >> > checking
>> >>> >>> >>> >> >> > if
>> >>> >>> >>> >> >> > file exists: Logon failure: unknown user name or bad
>> >>> >>> >>> >> >> > password.
>> >>> >>> >>> >> >> > jcifs.smb.SmbAuthException: Logon failure: unknown
>> user
>> >>> >>> >>> >> >> > name
>> >>> >>> >>> >> >> > or
>> >>> >>> >>> >> >> > bad
>> >>> >>> >>> >> >> > password.
>> >>> >>> >>> >> >> > at
>> >>> >>> >>> >> >> >
>> jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:544)
>> >>> >>> >>> >> >> > at
>> jcifs.smb.SmbTransport.send(SmbTransport.java:661)
>> >>> >>> >>> >> >> > at
>> >>> >>> >>> >> >> > jcifs.smb.SmbSession.sessionSetup(SmbSession.java:390)
>> >>> >>> >>> >> >> > at jcifs.smb.SmbSession.send(SmbSession.java:218)
>> >>> >>> >>> >> >> > at jcifs.smb.SmbTree.treeConnect(SmbTree.java:176)
>> >>> >>> >>> >> >> > at jcifs.smb.SmbFile.doConnect(SmbFile.java:911)
>> >>> >>> >>> >> >> > at jcifs.smb.SmbFile.connect(SmbFile.java:954)
>> >>> >>> >>> >> >> > at jcifs.smb.SmbFile.connect0(SmbFile.java:880)
>> >>> >>> >>> >> >> > at jcifs.smb.SmbFile.queryPath(SmbFile.java:1335)
>> >>> >>> >>> >> >> > at jcifs.smb.SmbFile.exists(SmbFile.java:1417)
>> >>> >>> >>> >> >> > at
>> >>> >>> >>> >> >> >
>> >>> >>> >>> >> >> >
>> >>> >>> >>> >> >> >
>> >>> >>> >>> >> >> >
>> >>> >>> >>> >> >> >
>> >>> >>> >>> >> >> >
>> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.fileExists(SharedDriveConnector.java:2064)
>> >>> >>> >>> >> >> > at
>> >>> >>> >>> >> >> >
>> >>> >>> >>> >> >> >
>> >>> >>> >>> >> >> >
>> >>> >>> >>> >> >> >
>> >>> >>> >>> >> >> >
>> >>> >>> >>> >> >> >
>> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getDocumentVersions(SharedDriveConnector.java:521)
>> >>> >>> >>> >> >> > at
>> >>> >>> >>> >> >> >
>> >>> >>> >>> >> >> >
>> >>> >>> >>> >> >> >
>> >>> >>> >>> >> >> >
>> >>> >>> >>> >> >> >
>> >>> >>> >>> >> >> >
>> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:318)
>> >>> >>> >>> >> >> >
>> >>> >>> >>> >> >> > Am able to access this location from windows explorer.
>> >>> >>> >>> >> >> > What
>> >>> >>> >>> >> >> > else
>> >>> >>> >>> >> >> > should
>> >>> >>> >>> >> >> > I be
>> >>> >>> >>> >> >> > checking or what could be the reasons/factors causing
>> this
>> >>> >>> >>> >> >> > to
>> >>> >>> >>> >> >> > fail
>> >>> >>> >>> >> >> > ?
>> >>> >>> >>> >> >> >
>> >>> >>> >>> >> >> > Thanks and Regards,
>> >>> >>> >>> >> >> > Swapna.
>> >>> >>> >>> >> >> >
>> >>> >>> >>> >> >
>> >>> >>> >>> >> >
>> >>> >>> >>> >
>> >>> >>> >>> >
>> >>> >>> >>
>> >>> >>> >>
>> >>> >>
>> >>> >>
>> >>
>> >>
>>
>
>
Re: Problem crawling windows share
Posted by Swapna Vuppala <sw...@gmail.com>.
Hi Karl,
This is with respect to a quite old issue that we had discussed for some
time and left.
The issue got solved for us by setting the property
"org.apache.manifoldcf.crawler.connectors.jcifs.usentlmv1" to "true" in
properties.xml. Just wanted to update you on this.
With this property set, we are able to crawl the windows share (mentioned
in previous mails) successfully and index them into Solr.
Thanks for all the help.
Thanks and Regards,
Swapna.
On Tue, Dec 13, 2011 at 11:15 AM, Swapna Vuppala <swapna.kollipara@gmail.com
> wrote:
> Thanks Karl for getting this far.
>
> I understand that the chances of jCIFS working with this particular server
> are very less. We'll leave it here for now and I'll let you know if I have
> any more details on the server.
>
> Thanks and Regards,
> Swapna.
>
>
> On Tue, Dec 13, 2011 at 7:24 AM, Karl Wright <da...@gmail.com> wrote:
>
>> His response:
>>
>> "Actually from looking at the capture again, the successful operations
>> in it were ultimately performed by a session that used Kerberos. But
>> there are two other successful NTLM authentications that use the
>> Anonymous credential. It so happens that neither of these went on to
>> do anything successful. Meaning your capture does not contain a
>> successful operation using NTLM and thus there is nothing to compare
>> JCIFS too. It could be that the environment actually does not support
>> NTLM properly. There are a lot of errors and oddities in the non-JCIFS
>> communications with numerous login failures, disabled signatures,
>> access denied conditions and unsupported versions of Windows (Windows
>> Server 2002 SP2 build 2600). The environment is not optimal.
>>
>> To continue I would need proper comparison captures. Clearly the
>> NtlmPasswordAuthentication("","","") does not work. But
>> NtlmPasswordAuthentication.ANONYMOUS obviously does work to some
>> degree but there's something else wrong with the session setup
>> request. That something could be the fact that the hostname used is
>> actually l-carx01.global.arup.com and not lon-data-arx.global.arup.com
>> and then the tree connect goes on to mount
>> //lon-data-arx.global.arup.com/stg-jobs$ so clearly there is some
>> hostname discontinuity going on as well."
>>
>> I think this is pretty much self explanatory. Whatever it is that you
>> are trying to talk to seems to be one truly crazy machine, certainly
>> not really Windows, and it is almost an accident that it works with
>> Windows at all. I think this is rapidly reaching the point of
>> diminishing returns in getting jCIFS to work with it.
>>
>> Karl
>>
>>
>> On Mon, Dec 12, 2011 at 4:24 AM, Karl Wright <da...@gmail.com> wrote:
>> > I forwarded this research again to Mr. Allen for further suggestions.
>> Thanks!
>> >
>> > Karl
>> >
>> > On Mon, Dec 12, 2011 at 1:58 AM, Swapna Vuppala
>> > <sw...@gmail.com> wrote:
>> >> Hi Karl,
>> >>
>> >> I tried modifying the test code as you suggested.
>> >>
>> >> When I tried with empty credentials
>> >>
>> >> NtlmPasswordAuthentication pa = new
>> NtlmPasswordAuthentication("","","");
>> >> SmbFile smbconnection = new SmbFile("smb://" +
>> >> "lon-data-arx.global.arup.com" + "/" + "stg-jobs$",pa);
>> >>
>> >> I got the same exception as before
>> >>
>> >> Exception in thread "main" jcifs.smb.SmbAuthException: Logon failure:
>> >> unknown us
>> >>
>> >> er name or bad password.
>> >> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:546)
>> >> at jcifs.smb.SmbTransport.send(SmbTransport.java:663)
>> >>
>> >> at jcifs.smb.SmbSession.sessionSetup(SmbSession.java:390)
>> >> at jcifs.smb.SmbSession.send(SmbSession.java:218)
>> >> at jcifs.smb.SmbTree.treeConnect(SmbTree.java:176)
>> >> at jcifs.smb.SmbFile.doConnect(SmbFile.java:911)
>> >>
>> >> ......
>> >>
>> >> When I tried with NtlmPasswordAuthentication.ANONYMOUS
>> >>
>> >> SmbFile smbconnection = new SmbFile("smb://" +
>> >> "lon-data-arx.global.arup.com" + "/" +
>> >> "stg-jobs$",NtlmPasswordAuthentication.ANONYMOUS);
>> >>
>> >> I got a different exception
>> >>
>> >> Exception in thread "main" jcifs.smb.SmbException: Incorrect function.
>> >> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:563)
>> >> at jcifs.smb.SmbTransport.send(SmbTransport.java:663)
>> >> at jcifs.smb.SmbSession.sessionSetup(SmbSession.java:316)
>> >>
>> >> at jcifs.smb.SmbSession.send(SmbSession.java:218)
>> >> at jcifs.smb.SmbTree.treeConnect(SmbTree.java:176)
>> >> at jcifs.smb.SmbFile.doConnect(SmbFile.java:911)
>> >> .....
>> >>
>> >> Thanks and Regards,
>> >> Swapna.
>> >> On Fri, Dec 9, 2011 at 11:46 PM, Karl Wright <da...@gmail.com>
>> wrote:
>> >>>
>> >>> Michael Allen's response to your capture:
>> >>>
>> >>> "It looks like the non-JCIFS communication is successful because the
>> >>> anonymous credential is used.
>> >>>
>> >>> Note that there is still no sign of DFS going on anywhere. Actually
>> >>> JCIFS tries to do a referral for the DFS root and gets an error of
>> >>> STATUS_FS_DRIVER_REQUIRED so it looks like DFS may not even be setup
>> >>> at all in the target environment. At least not domain based DFS.
>> >>>
>> >>> So if you want to emulate Windows behavior, I guess you need to retry
>> >>> the operation with credential NtlmPasswordAuthentication.ANONYMOUS (or
>> >>> all empty strings should work too I think like ";:")."
>> >>>
>> >>> In other words, the issue is that the server in question does not
>> >>> actually recognize the credentials given at all, and gives an error.
>> >>> But while Windows then retries with no credentials whatsoever, nothing
>> >>> in JCIFS does that retry.
>> >>>
>> >>> You can confirm that this is what is happening by using an empty user
>> >>> name, password, and domain in the JCIFS test code you wrote to obtain
>> >>> the captures. If, then, the test code works against that server, I'll
>> >>> create a ticket for downgrading to ANONYMOUS whenever there is an
>> >>> authentication error in the JCIFS connector.
>> >>>
>> >>> Thanks,
>> >>> Karl
>> >>>
>> >>> On Wed, Dec 7, 2011 at 6:41 AM, Karl Wright <da...@gmail.com>
>> wrote:
>> >>> > The next step is to obtain two packet captures using WireShark.
>> These
>> >>> > should both be done from the machine that is not joined to the
>> domain.
>> >>> > The first capture should be the interaction that takes place when
>> you
>> >>> > try to go to the foreign directory, and you supply the credentials,
>> >>> > and then you list the folders. The second capture will require you
>> to
>> >>> > do the same using JCIFS. For that capture, you will need jcifs.jar
>> >>> > and a simple example class that provides jcifs with credentials and
>> >>> > tries to list the directory. A simple class should be enough to do
>> >>> > this, including code such as:
>> >>> >
>> >>> > import jcifs.smb.*;
>> >>> >
>> >>> > NtlmPasswordAuthentication pa = new
>> >>> > NtlmPasswordAuthentication(domain,username,password);
>> >>> > SmbFile smbconnection = new SmbFile("smb://" + server + "/" +
>> share,pa);
>> >>> > SmbFile[] files = smbconnection.listFiles();
>> >>> >
>> >>> > for (int i = 0 ; i < files.length ; i++)
>> >>> > {
>> >>> > System.out.println(file.toString());
>> >>> > }
>> >>> >
>> >>> > I suggest you put together your test class first and make sure it
>> >>> > works, and then go after the captures.
>> >>> >
>> >>> > Let me know how it goes.
>> >>> >
>> >>> > Karl
>> >>> >
>> >>> > On Wed, Dec 7, 2011 at 6:10 AM, Swapna Vuppala
>> >>> > <sw...@gmail.com> wrote:
>> >>> >> Hi Karl,
>> >>> >>
>> >>> >> Thanks for the detailed reply.
>> >>> >>
>> >>> >> As per your suggestion, I tried connecting to the problematic
>> server
>> >>> >> via
>> >>> >> windows using a machine that is not in the domain. On supplying the
>> >>> >> credentials via windows, I could connect to the server, although
>> it did
>> >>> >> not
>> >>> >> list any folders. But when I tried accessing the folder of my
>> interest
>> >>> >> via
>> >>> >> \\server\folder$, I could see all the sub folders. All this is
>> when I
>> >>> >> connected via windows from a machine that's not joined to the
>> domain.
>> >>> >>
>> >>> >> The behavior is same when I connect to the server via windows from
>> a
>> >>> >> machine
>> >>> >> that's joined to the domain.
>> >>> >>
>> >>> >> So I assume this means that through both Kerberos and NTLM, I see
>> the
>> >>> >> same
>> >>> >> behavior. Whereas ManifoldCF throws an exception when trying to
>> define
>> >>> >> the
>> >>> >> repository connection.
>> >>> >>
>> >>> >> Please advise me on what I should be doing next to resolve this.
>> >>> >>
>> >>> >> Thanks and Regards,
>> >>> >> Swapna.
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >> On Tue, Dec 6, 2011 at 12:55 PM, Karl Wright <da...@gmail.com>
>> >>> >> wrote:
>> >>> >>>
>> >>> >>> About your capture - Michael Allen says the following:
>> >>> >>>
>> >>> >>> "Actually this has nothing to do with DFS. JCIFS does not get to
>> the
>> >>> >>> point where it does DFS anything. The capture shows a vanilla
>> >>> >>> STATUS_LOGON_FAILURE when GLOBAL\swapna.vuppala tries to auth with
>> >>> >>> l-carx01.global.arup.com. So the possible causes for this are 1)
>> the
>> >>> >>> account name is not valid 2) the supplied password is incorrect 3)
>> >>> >>> some security policy is deliberately blocking that user or
>> particular
>> >>> >>> type of auth or 4) some server configuration is incompatible with
>> >>> >>> JCIFS. I only mention this last option because I noticed the
>> target
>> >>> >>> server has security signatures disabled. That's strange. If
>> they're
>> >>> >>> messing around with things like that, who knows what their
>> clients are
>> >>> >>> expected to do.
>> >>> >>>
>> >>> >>> Try a Windows client that uses NTLM instead of Kerberos. Meaning
>> try a
>> >>> >>> machine that is not joined to the domain so that when you try to
>> >>> >>> access the target it asks you for credentials at which point you
>> can
>> >>> >>> test with GLOBAL\swapna.vuppala. Then it will use NTLM and you can
>> >>> >>> actually compare captures. If the operator doesn't have a laptop
>> or
>> >>> >>> something not joined to the domain, it might be sufficient to log
>> into
>> >>> >>> a workstation using machine credentials and not domain
>> credentials.
>> >>> >>>
>> >>> >>> Also when testing JCIFS you should use a simple stand-alone
>> program
>> >>> >>> like examples/ListFiles.java."
>> >>> >>>
>> >>> >>> In other words:
>> >>> >>> (a) Since JCIFS does not use Kerberos for authentication, you
>> need to
>> >>> >>> try to log into the recalcitrant server via Windows without using
>> >>> >>> Kerberos to be able to do a side-by-side comparison. Michael has
>> some
>> >>> >>> ways of doing that, above.
>> >>> >>> (b) You may find that it doesn't work, in which case JCIFS is not
>> >>> >>> going to work either.
>> >>> >>> (c) If it *does* work, then try to generate your side-by-side
>> >>> >>> comparisons using a simpler example rather than ManifoldCF en
>> toto;
>> >>> >>> you can see how at jcifs.samba.org, or I can help you further.
>> >>> >>>
>> >>> >>> He also mentions that there is some bizarreness on the response
>> that
>> >>> >>> indicates that the server is configured in a way that he's never
>> seen
>> >>> >>> before. And believe me, Michael has seen a *lot* of strange
>> >>> >>> configurations...
>> >>> >>>
>> >>> >>> Hope this helps.
>> >>> >>> Karl
>> >>> >>>
>> >>> >>> On Mon, Nov 28, 2011 at 4:12 AM, Karl Wright <da...@gmail.com>
>> >>> >>> wrote:
>> >>> >>> > That should read "properties.xml", not "properties.ini". It
>> looks
>> >>> >>> > like this page needs updating.
>> >>> >>> >
>> >>> >>> > The debug property in the XML form is:
>> >>> >>> >
>> >>> >>> > <property name="org.apache.manifoldcf.connectors"
>> value="DEBUG"/>
>> >>> >>> >
>> >>> >>> > I don't think it will provide you with any additional
>> information
>> >>> >>> > that
>> >>> >>> > is useful for debugging your authentication issue, however, if
>> that
>> >>> >>> > is
>> >>> >>> > why you are looking at it. There may be some jcifs.jar
>> debugging
>> >>> >>> > switches that might be of more help, but in the end I suspect
>> you
>> >>> >>> > will
>> >>> >>> > need a packet capture of both a successful connection (via
>> Windows)
>> >>> >>> > and an unsuccessful one (via MCF). The guy you will need to
>> talk
>> >>> >>> > with
>> >>> >>> > after that is the jcifs author Michael Allen; I can give you his
>> >>> >>> > email
>> >>> >>> > address if you get that far.
>> >>> >>> >
>> >>> >>> > Karl
>> >>> >>> >
>> >>> >>> >
>> >>> >>> > On Mon, Nov 28, 2011 at 1:30 AM, Swapna Vuppala
>> >>> >>> > <sw...@gmail.com> wrote:
>> >>> >>> >> Hi Karl,
>> >>> >>> >>
>> >>> >>> >> I was planning to debug jCIFS repository connection using
>> WireShark
>> >>> >>> >> and
>> >>> >>> >> I
>> >>> >>> >> came across this
>> >>> >>> >> https://cwiki.apache.org/CONNECTORS/debugging-connections.html
>> >>> >>> >> Here, I see something as add
>> >>> >>> >> "org.apache.manifoldcf.connectors=DEBUG"
>> >>> >>> >> to the
>> >>> >>> >> properties.ini file. Is it the properties.xml file that is
>> being
>> >>> >>> >> referred
>> >>> >>> >> here ? If not, where do I find properties.ini file ?
>> >>> >>> >>
>> >>> >>> >> Thanks and Regards,
>> >>> >>> >> Swapna.
>> >>> >>> >>
>> >>> >>> >> On Thu, Nov 17, 2011 at 1:31 PM, Karl Wright <
>> daddywri@gmail.com>
>> >>> >>> >> wrote:
>> >>> >>> >>>
>> >>> >>> >>> See
>> http://jcifs.samba.org/src/docs/api/overview-summary.html#scp.
>> >>> >>> >>> The properties jcifs.smb.lmCompatibility and
>> >>> >>> >>> jcifs.smb.client.useExtendedSecurity are the ones you may
>> want to
>> >>> >>> >>> change. These two properties go together so certain
>> combinations
>> >>> >>> >>> make
>> >>> >>> >>> sense and others don't, so there's really only combinations
>> you
>> >>> >>> >>> need
>> >>> >>> >>> but I'll need to look at what they are and get back to you
>> later
>> >>> >>> >>> today.
>> >>> >>> >>>
>> >>> >>> >>> As far as setting the switches are concerned, if you are
>> using the
>> >>> >>> >>> Quick Start you do this trivially by:
>> >>> >>> >>>
>> >>> >>> >>> <java> -Dxxx -Dyyy -jar start.jar
>> >>> >>> >>>
>> >>> >>> >>> If you are using the multi-process configuration, that is
>> what the
>> >>> >>> >>> "defines" directory is for; you only need to create files in
>> that
>> >>> >>> >>> directory with the names "jcifs.smb.lmCompatibility" and
>> >>> >>> >>> "jcifs.smb.client.useExtendedSecurity" containing the values
>> you
>> >>> >>> >>> want
>> >>> >>> >>> to set.
>> >>> >>> >>>
>> >>> >>> >>> Karl
>> >>> >>> >>>
>> >>> >>> >>>
>> >>> >>> >>> On Thu, Nov 17, 2011 at 1:11 AM, Swapna Vuppala
>> >>> >>> >>> <sw...@gmail.com> wrote:
>> >>> >>> >>> > Hi Karl,
>> >>> >>> >>> >
>> >>> >>> >>> > Am able to access the folders on the problem server through
>> >>> >>> >>> > windows
>> >>> >>> >>> > explorer, (\\server3\Folder1). I tried couple of things
>> with the
>> >>> >>> >>> > credentials
>> >>> >>> >>> > form, changing username, domain etc.. but I keep getting the
>> >>> >>> >>> > same
>> >>> >>> >>> > error
>> >>> >>> >>> > "Couldn't connect to server: Logon failure: unknown user
>> name or
>> >>> >>> >>> > bad
>> >>> >>> >>> > password"
>> >>> >>> >>> >
>> >>> >>> >>> > Can you tell me more about the -D switch you were talking
>> of ?
>> >>> >>> >>> >
>> >>> >>> >>> > Thanks and Regards,
>> >>> >>> >>> > Swapna.
>> >>> >>> >>> >
>> >>> >>> >>> > On Tue, Nov 15, 2011 at 12:40 PM, Karl Wright
>> >>> >>> >>> > <da...@gmail.com>
>> >>> >>> >>> > wrote:
>> >>> >>> >>> >>
>> >>> >>> >>> >> Glad you chased it down this far.
>> >>> >>> >>> >>
>> >>> >>> >>> >> First thing to try is whether you can get into the problem
>> >>> >>> >>> >> server
>> >>> >>> >>> >> using Windows Explorer. Obviously ManifoldCF is not going
>> to
>> >>> >>> >>> >> be
>> >>> >>> >>> >> able
>> >>> >>> >>> >> to do it if Windows can't. If you *can* get in, then just
>> >>> >>> >>> >> playing
>> >>> >>> >>> >> with the form of the credentials in the MCF connection
>> might do
>> >>> >>> >>> >> the
>> >>> >>> >>> >> trick. Some Windows or net appliance servers are picky
>> about
>> >>> >>> >>> >> this.
>> >>> >>> >>> >> Try various things like leaving the domain blank and
>> specifying
>> >>> >>> >>> >> the
>> >>> >>> >>> >> user as "abc@domain.com", for instance. There's also a
>> >>> >>> >>> >> different
>> >>> >>> >>> >> NTLM
>> >>> >>> >>> >> mode you can operation jcifs in that some servers may be
>> >>> >>> >>> >> configured
>> >>> >>> >>> >> to
>> >>> >>> >>> >> require; this would need you to set a -D switch on the
>> command
>> >>> >>> >>> >> line
>> >>> >>> >>> >> to
>> >>> >>> >>> >> enable.
>> >>> >>> >>> >>
>> >>> >>> >>> >> Karl
>> >>> >>> >>> >>
>> >>> >>> >>> >> On Tue, Nov 15, 2011 at 12:10 AM, Swapna Vuppala
>> >>> >>> >>> >> <sw...@gmail.com> wrote:
>> >>> >>> >>> >> > Hi Karl,
>> >>> >>> >>> >> >
>> >>> >>> >>> >> > Thanks for the input. It looks like my problem is
>> related to
>> >>> >>> >>> >> > the
>> >>> >>> >>> >> > second
>> >>> >>> >>> >> > one
>> >>> >>> >>> >> > that you specified. One of the directories in the path am
>> >>> >>> >>> >> > trying
>> >>> >>> >>> >> > to
>> >>> >>> >>> >> > index is
>> >>> >>> >>> >> > actually redirecting to a different server. And when I
>> >>> >>> >>> >> > specify
>> >>> >>> >>> >> > this
>> >>> >>> >>> >> > new
>> >>> >>> >>> >> > server in defining the repository connection, with my
>> >>> >>> >>> >> > credentials,
>> >>> >>> >>> >> > the
>> >>> >>> >>> >> > connection fails with the message: "Couldn't connect to
>> >>> >>> >>> >> > server:
>> >>> >>> >>> >> > Logon
>> >>> >>> >>> >> > failure: unknown user name or bad password"
>> >>> >>> >>> >> >
>> >>> >>> >>> >> > I'll look into why am not able to connect to this server.
>> >>> >>> >>> >> >
>> >>> >>> >>> >> > Thanks and Regards,
>> >>> >>> >>> >> > Swapna.
>> >>> >>> >>> >> >
>> >>> >>> >>> >> > On Mon, Nov 14, 2011 at 4:56 PM, Karl Wright
>> >>> >>> >>> >> > <da...@gmail.com>
>> >>> >>> >>> >> > wrote:
>> >>> >>> >>> >> >>
>> >>> >>> >>> >> >> There's two kinds of problem you might be having. The
>> first
>> >>> >>> >>> >> >> is
>> >>> >>> >>> >> >> intermittent, and the second is not intermittent but
>> would
>> >>> >>> >>> >> >> have
>> >>> >>> >>> >> >> something to do with specific directories.
>> >>> >>> >>> >> >>
>> >>> >>> >>> >> >> Intermittent problems might include a domain controller
>> that
>> >>> >>> >>> >> >> is
>> >>> >>> >>> >> >> not
>> >>> >>> >>> >> >> always accessible. In such cases, the crawl will
>> proceed
>> >>> >>> >>> >> >> but
>> >>> >>> >>> >> >> will
>> >>> >>> >>> >> >> tend to fail unpredictably. On the other hand, if you
>> have
>> >>> >>> >>> >> >> a
>> >>> >>> >>> >> >> directory that is handled by a DFS redirection, it is
>> >>> >>> >>> >> >> possible
>> >>> >>> >>> >> >> that
>> >>> >>> >>> >> >> the redirection is indicating a new server (lets call it
>> >>> >>> >>> >> >> server3)
>> >>> >>> >>> >> >> which may not like the precise form of your login
>> >>> >>> >>> >> >> credentials.
>> >>> >>> >>> >> >> Can
>> >>> >>> >>> >> >> you determine which scenario you are seeing?
>> >>> >>> >>> >> >>
>> >>> >>> >>> >> >> Karl
>> >>> >>> >>> >> >>
>> >>> >>> >>> >> >> On Mon, Nov 14, 2011 at 3:11 AM, Swapna Vuppala
>> >>> >>> >>> >> >> <sw...@gmail.com> wrote:
>> >>> >>> >>> >> >> > Hi,
>> >>> >>> >>> >> >> >
>> >>> >>> >>> >> >> > I have been using windows share repository connection
>> to
>> >>> >>> >>> >> >> > crawl
>> >>> >>> >>> >> >> > and
>> >>> >>> >>> >> >> > get
>> >>> >>> >>> >> >> > data
>> >>> >>> >>> >> >> > from a particular server (server 1). Its working
>> perfectly
>> >>> >>> >>> >> >> > fine.
>> >>> >>> >>> >> >> > However, am
>> >>> >>> >>> >> >> > having trouble when I try with data from another
>> server
>> >>> >>> >>> >> >> > (server
>> >>> >>> >>> >> >> > 2).
>> >>> >>> >>> >> >> >
>> >>> >>> >>> >> >> > When I define a repository connection of type windows
>> >>> >>> >>> >> >> > share
>> >>> >>> >>> >> >> > and
>> >>> >>> >>> >> >> > specify
>> >>> >>> >>> >> >> > the
>> >>> >>> >>> >> >> > server name (server 2) with my credentials, the
>> connection
>> >>> >>> >>> >> >> > status
>> >>> >>> >>> >> >> > shows
>> >>> >>> >>> >> >> > "Connection working". But when I run a job to use this
>> >>> >>> >>> >> >> > repository
>> >>> >>> >>> >> >> > connection
>> >>> >>> >>> >> >> > and index data from a location on this server 2, I
>> keep
>> >>> >>> >>> >> >> > getting
>> >>> >>> >>> >> >> > the
>> >>> >>> >>> >> >> > exception below:
>> >>> >>> >>> >> >> >
>> >>> >>> >>> >> >> > JCIFS: Possibly transient exception detected on
>> attempt 3
>> >>> >>> >>> >> >> > while
>> >>> >>> >>> >> >> > checking
>> >>> >>> >>> >> >> > if
>> >>> >>> >>> >> >> > file exists: Logon failure: unknown user name or bad
>> >>> >>> >>> >> >> > password.
>> >>> >>> >>> >> >> > jcifs.smb.SmbAuthException: Logon failure: unknown
>> user
>> >>> >>> >>> >> >> > name
>> >>> >>> >>> >> >> > or
>> >>> >>> >>> >> >> > bad
>> >>> >>> >>> >> >> > password.
>> >>> >>> >>> >> >> > at
>> >>> >>> >>> >> >> >
>> jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:544)
>> >>> >>> >>> >> >> > at
>> jcifs.smb.SmbTransport.send(SmbTransport.java:661)
>> >>> >>> >>> >> >> > at
>> >>> >>> >>> >> >> > jcifs.smb.SmbSession.sessionSetup(SmbSession.java:390)
>> >>> >>> >>> >> >> > at jcifs.smb.SmbSession.send(SmbSession.java:218)
>> >>> >>> >>> >> >> > at jcifs.smb.SmbTree.treeConnect(SmbTree.java:176)
>> >>> >>> >>> >> >> > at jcifs.smb.SmbFile.doConnect(SmbFile.java:911)
>> >>> >>> >>> >> >> > at jcifs.smb.SmbFile.connect(SmbFile.java:954)
>> >>> >>> >>> >> >> > at jcifs.smb.SmbFile.connect0(SmbFile.java:880)
>> >>> >>> >>> >> >> > at jcifs.smb.SmbFile.queryPath(SmbFile.java:1335)
>> >>> >>> >>> >> >> > at jcifs.smb.SmbFile.exists(SmbFile.java:1417)
>> >>> >>> >>> >> >> > at
>> >>> >>> >>> >> >> >
>> >>> >>> >>> >> >> >
>> >>> >>> >>> >> >> >
>> >>> >>> >>> >> >> >
>> >>> >>> >>> >> >> >
>> >>> >>> >>> >> >> >
>> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.fileExists(SharedDriveConnector.java:2064)
>> >>> >>> >>> >> >> > at
>> >>> >>> >>> >> >> >
>> >>> >>> >>> >> >> >
>> >>> >>> >>> >> >> >
>> >>> >>> >>> >> >> >
>> >>> >>> >>> >> >> >
>> >>> >>> >>> >> >> >
>> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getDocumentVersions(SharedDriveConnector.java:521)
>> >>> >>> >>> >> >> > at
>> >>> >>> >>> >> >> >
>> >>> >>> >>> >> >> >
>> >>> >>> >>> >> >> >
>> >>> >>> >>> >> >> >
>> >>> >>> >>> >> >> >
>> >>> >>> >>> >> >> >
>> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:318)
>> >>> >>> >>> >> >> >
>> >>> >>> >>> >> >> > Am able to access this location from windows explorer.
>> >>> >>> >>> >> >> > What
>> >>> >>> >>> >> >> > else
>> >>> >>> >>> >> >> > should
>> >>> >>> >>> >> >> > I be
>> >>> >>> >>> >> >> > checking or what could be the reasons/factors causing
>> this
>> >>> >>> >>> >> >> > to
>> >>> >>> >>> >> >> > fail
>> >>> >>> >>> >> >> > ?
>> >>> >>> >>> >> >> >
>> >>> >>> >>> >> >> > Thanks and Regards,
>> >>> >>> >>> >> >> > Swapna.
>> >>> >>> >>> >> >> >
>> >>> >>> >>> >> >
>> >>> >>> >>> >> >
>> >>> >>> >>> >
>> >>> >>> >>> >
>> >>> >>> >>
>> >>> >>> >>
>> >>> >>
>> >>> >>
>> >>
>> >>
>>
>
>
Re: Problem crawling windows share
Posted by Swapna Vuppala <sw...@gmail.com>.
Thanks Karl for getting this far.
I understand that the chances of jCIFS working with this particular server
are very less. We'll leave it here for now and I'll let you know if I have
any more details on the server.
Thanks and Regards,
Swapna.
On Tue, Dec 13, 2011 at 7:24 AM, Karl Wright <da...@gmail.com> wrote:
> His response:
>
> "Actually from looking at the capture again, the successful operations
> in it were ultimately performed by a session that used Kerberos. But
> there are two other successful NTLM authentications that use the
> Anonymous credential. It so happens that neither of these went on to
> do anything successful. Meaning your capture does not contain a
> successful operation using NTLM and thus there is nothing to compare
> JCIFS too. It could be that the environment actually does not support
> NTLM properly. There are a lot of errors and oddities in the non-JCIFS
> communications with numerous login failures, disabled signatures,
> access denied conditions and unsupported versions of Windows (Windows
> Server 2002 SP2 build 2600). The environment is not optimal.
>
> To continue I would need proper comparison captures. Clearly the
> NtlmPasswordAuthentication("","","") does not work. But
> NtlmPasswordAuthentication.ANONYMOUS obviously does work to some
> degree but there's something else wrong with the session setup
> request. That something could be the fact that the hostname used is
> actually l-carx01.global.arup.com and not lon-data-arx.global.arup.com
> and then the tree connect goes on to mount
> //lon-data-arx.global.arup.com/stg-jobs$ so clearly there is some
> hostname discontinuity going on as well."
>
> I think this is pretty much self explanatory. Whatever it is that you
> are trying to talk to seems to be one truly crazy machine, certainly
> not really Windows, and it is almost an accident that it works with
> Windows at all. I think this is rapidly reaching the point of
> diminishing returns in getting jCIFS to work with it.
>
> Karl
>
>
> On Mon, Dec 12, 2011 at 4:24 AM, Karl Wright <da...@gmail.com> wrote:
> > I forwarded this research again to Mr. Allen for further suggestions.
> Thanks!
> >
> > Karl
> >
> > On Mon, Dec 12, 2011 at 1:58 AM, Swapna Vuppala
> > <sw...@gmail.com> wrote:
> >> Hi Karl,
> >>
> >> I tried modifying the test code as you suggested.
> >>
> >> When I tried with empty credentials
> >>
> >> NtlmPasswordAuthentication pa = new
> NtlmPasswordAuthentication("","","");
> >> SmbFile smbconnection = new SmbFile("smb://" +
> >> "lon-data-arx.global.arup.com" + "/" + "stg-jobs$",pa);
> >>
> >> I got the same exception as before
> >>
> >> Exception in thread "main" jcifs.smb.SmbAuthException: Logon failure:
> >> unknown us
> >>
> >> er name or bad password.
> >> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:546)
> >> at jcifs.smb.SmbTransport.send(SmbTransport.java:663)
> >>
> >> at jcifs.smb.SmbSession.sessionSetup(SmbSession.java:390)
> >> at jcifs.smb.SmbSession.send(SmbSession.java:218)
> >> at jcifs.smb.SmbTree.treeConnect(SmbTree.java:176)
> >> at jcifs.smb.SmbFile.doConnect(SmbFile.java:911)
> >>
> >> ......
> >>
> >> When I tried with NtlmPasswordAuthentication.ANONYMOUS
> >>
> >> SmbFile smbconnection = new SmbFile("smb://" +
> >> "lon-data-arx.global.arup.com" + "/" +
> >> "stg-jobs$",NtlmPasswordAuthentication.ANONYMOUS);
> >>
> >> I got a different exception
> >>
> >> Exception in thread "main" jcifs.smb.SmbException: Incorrect function.
> >> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:563)
> >> at jcifs.smb.SmbTransport.send(SmbTransport.java:663)
> >> at jcifs.smb.SmbSession.sessionSetup(SmbSession.java:316)
> >>
> >> at jcifs.smb.SmbSession.send(SmbSession.java:218)
> >> at jcifs.smb.SmbTree.treeConnect(SmbTree.java:176)
> >> at jcifs.smb.SmbFile.doConnect(SmbFile.java:911)
> >> .....
> >>
> >> Thanks and Regards,
> >> Swapna.
> >> On Fri, Dec 9, 2011 at 11:46 PM, Karl Wright <da...@gmail.com>
> wrote:
> >>>
> >>> Michael Allen's response to your capture:
> >>>
> >>> "It looks like the non-JCIFS communication is successful because the
> >>> anonymous credential is used.
> >>>
> >>> Note that there is still no sign of DFS going on anywhere. Actually
> >>> JCIFS tries to do a referral for the DFS root and gets an error of
> >>> STATUS_FS_DRIVER_REQUIRED so it looks like DFS may not even be setup
> >>> at all in the target environment. At least not domain based DFS.
> >>>
> >>> So if you want to emulate Windows behavior, I guess you need to retry
> >>> the operation with credential NtlmPasswordAuthentication.ANONYMOUS (or
> >>> all empty strings should work too I think like ";:")."
> >>>
> >>> In other words, the issue is that the server in question does not
> >>> actually recognize the credentials given at all, and gives an error.
> >>> But while Windows then retries with no credentials whatsoever, nothing
> >>> in JCIFS does that retry.
> >>>
> >>> You can confirm that this is what is happening by using an empty user
> >>> name, password, and domain in the JCIFS test code you wrote to obtain
> >>> the captures. If, then, the test code works against that server, I'll
> >>> create a ticket for downgrading to ANONYMOUS whenever there is an
> >>> authentication error in the JCIFS connector.
> >>>
> >>> Thanks,
> >>> Karl
> >>>
> >>> On Wed, Dec 7, 2011 at 6:41 AM, Karl Wright <da...@gmail.com>
> wrote:
> >>> > The next step is to obtain two packet captures using WireShark.
> These
> >>> > should both be done from the machine that is not joined to the
> domain.
> >>> > The first capture should be the interaction that takes place when
> you
> >>> > try to go to the foreign directory, and you supply the credentials,
> >>> > and then you list the folders. The second capture will require you
> to
> >>> > do the same using JCIFS. For that capture, you will need jcifs.jar
> >>> > and a simple example class that provides jcifs with credentials and
> >>> > tries to list the directory. A simple class should be enough to do
> >>> > this, including code such as:
> >>> >
> >>> > import jcifs.smb.*;
> >>> >
> >>> > NtlmPasswordAuthentication pa = new
> >>> > NtlmPasswordAuthentication(domain,username,password);
> >>> > SmbFile smbconnection = new SmbFile("smb://" + server + "/" +
> share,pa);
> >>> > SmbFile[] files = smbconnection.listFiles();
> >>> >
> >>> > for (int i = 0 ; i < files.length ; i++)
> >>> > {
> >>> > System.out.println(file.toString());
> >>> > }
> >>> >
> >>> > I suggest you put together your test class first and make sure it
> >>> > works, and then go after the captures.
> >>> >
> >>> > Let me know how it goes.
> >>> >
> >>> > Karl
> >>> >
> >>> > On Wed, Dec 7, 2011 at 6:10 AM, Swapna Vuppala
> >>> > <sw...@gmail.com> wrote:
> >>> >> Hi Karl,
> >>> >>
> >>> >> Thanks for the detailed reply.
> >>> >>
> >>> >> As per your suggestion, I tried connecting to the problematic server
> >>> >> via
> >>> >> windows using a machine that is not in the domain. On supplying the
> >>> >> credentials via windows, I could connect to the server, although it
> did
> >>> >> not
> >>> >> list any folders. But when I tried accessing the folder of my
> interest
> >>> >> via
> >>> >> \\server\folder$, I could see all the sub folders. All this is when
> I
> >>> >> connected via windows from a machine that's not joined to the
> domain.
> >>> >>
> >>> >> The behavior is same when I connect to the server via windows from a
> >>> >> machine
> >>> >> that's joined to the domain.
> >>> >>
> >>> >> So I assume this means that through both Kerberos and NTLM, I see
> the
> >>> >> same
> >>> >> behavior. Whereas ManifoldCF throws an exception when trying to
> define
> >>> >> the
> >>> >> repository connection.
> >>> >>
> >>> >> Please advise me on what I should be doing next to resolve this.
> >>> >>
> >>> >> Thanks and Regards,
> >>> >> Swapna.
> >>> >>
> >>> >>
> >>> >>
> >>> >>
> >>> >> On Tue, Dec 6, 2011 at 12:55 PM, Karl Wright <da...@gmail.com>
> >>> >> wrote:
> >>> >>>
> >>> >>> About your capture - Michael Allen says the following:
> >>> >>>
> >>> >>> "Actually this has nothing to do with DFS. JCIFS does not get to
> the
> >>> >>> point where it does DFS anything. The capture shows a vanilla
> >>> >>> STATUS_LOGON_FAILURE when GLOBAL\swapna.vuppala tries to auth with
> >>> >>> l-carx01.global.arup.com. So the possible causes for this are 1)
> the
> >>> >>> account name is not valid 2) the supplied password is incorrect 3)
> >>> >>> some security policy is deliberately blocking that user or
> particular
> >>> >>> type of auth or 4) some server configuration is incompatible with
> >>> >>> JCIFS. I only mention this last option because I noticed the target
> >>> >>> server has security signatures disabled. That's strange. If they're
> >>> >>> messing around with things like that, who knows what their clients
> are
> >>> >>> expected to do.
> >>> >>>
> >>> >>> Try a Windows client that uses NTLM instead of Kerberos. Meaning
> try a
> >>> >>> machine that is not joined to the domain so that when you try to
> >>> >>> access the target it asks you for credentials at which point you
> can
> >>> >>> test with GLOBAL\swapna.vuppala. Then it will use NTLM and you can
> >>> >>> actually compare captures. If the operator doesn't have a laptop or
> >>> >>> something not joined to the domain, it might be sufficient to log
> into
> >>> >>> a workstation using machine credentials and not domain credentials.
> >>> >>>
> >>> >>> Also when testing JCIFS you should use a simple stand-alone program
> >>> >>> like examples/ListFiles.java."
> >>> >>>
> >>> >>> In other words:
> >>> >>> (a) Since JCIFS does not use Kerberos for authentication, you need
> to
> >>> >>> try to log into the recalcitrant server via Windows without using
> >>> >>> Kerberos to be able to do a side-by-side comparison. Michael has
> some
> >>> >>> ways of doing that, above.
> >>> >>> (b) You may find that it doesn't work, in which case JCIFS is not
> >>> >>> going to work either.
> >>> >>> (c) If it *does* work, then try to generate your side-by-side
> >>> >>> comparisons using a simpler example rather than ManifoldCF en toto;
> >>> >>> you can see how at jcifs.samba.org, or I can help you further.
> >>> >>>
> >>> >>> He also mentions that there is some bizarreness on the response
> that
> >>> >>> indicates that the server is configured in a way that he's never
> seen
> >>> >>> before. And believe me, Michael has seen a *lot* of strange
> >>> >>> configurations...
> >>> >>>
> >>> >>> Hope this helps.
> >>> >>> Karl
> >>> >>>
> >>> >>> On Mon, Nov 28, 2011 at 4:12 AM, Karl Wright <da...@gmail.com>
> >>> >>> wrote:
> >>> >>> > That should read "properties.xml", not "properties.ini". It
> looks
> >>> >>> > like this page needs updating.
> >>> >>> >
> >>> >>> > The debug property in the XML form is:
> >>> >>> >
> >>> >>> > <property name="org.apache.manifoldcf.connectors" value="DEBUG"/>
> >>> >>> >
> >>> >>> > I don't think it will provide you with any additional information
> >>> >>> > that
> >>> >>> > is useful for debugging your authentication issue, however, if
> that
> >>> >>> > is
> >>> >>> > why you are looking at it. There may be some jcifs.jar debugging
> >>> >>> > switches that might be of more help, but in the end I suspect you
> >>> >>> > will
> >>> >>> > need a packet capture of both a successful connection (via
> Windows)
> >>> >>> > and an unsuccessful one (via MCF). The guy you will need to talk
> >>> >>> > with
> >>> >>> > after that is the jcifs author Michael Allen; I can give you his
> >>> >>> > email
> >>> >>> > address if you get that far.
> >>> >>> >
> >>> >>> > Karl
> >>> >>> >
> >>> >>> >
> >>> >>> > On Mon, Nov 28, 2011 at 1:30 AM, Swapna Vuppala
> >>> >>> > <sw...@gmail.com> wrote:
> >>> >>> >> Hi Karl,
> >>> >>> >>
> >>> >>> >> I was planning to debug jCIFS repository connection using
> WireShark
> >>> >>> >> and
> >>> >>> >> I
> >>> >>> >> came across this
> >>> >>> >> https://cwiki.apache.org/CONNECTORS/debugging-connections.html
> >>> >>> >> Here, I see something as add
> >>> >>> >> "org.apache.manifoldcf.connectors=DEBUG"
> >>> >>> >> to the
> >>> >>> >> properties.ini file. Is it the properties.xml file that is being
> >>> >>> >> referred
> >>> >>> >> here ? If not, where do I find properties.ini file ?
> >>> >>> >>
> >>> >>> >> Thanks and Regards,
> >>> >>> >> Swapna.
> >>> >>> >>
> >>> >>> >> On Thu, Nov 17, 2011 at 1:31 PM, Karl Wright <
> daddywri@gmail.com>
> >>> >>> >> wrote:
> >>> >>> >>>
> >>> >>> >>> See
> http://jcifs.samba.org/src/docs/api/overview-summary.html#scp.
> >>> >>> >>> The properties jcifs.smb.lmCompatibility and
> >>> >>> >>> jcifs.smb.client.useExtendedSecurity are the ones you may want
> to
> >>> >>> >>> change. These two properties go together so certain
> combinations
> >>> >>> >>> make
> >>> >>> >>> sense and others don't, so there's really only combinations you
> >>> >>> >>> need
> >>> >>> >>> but I'll need to look at what they are and get back to you
> later
> >>> >>> >>> today.
> >>> >>> >>>
> >>> >>> >>> As far as setting the switches are concerned, if you are using
> the
> >>> >>> >>> Quick Start you do this trivially by:
> >>> >>> >>>
> >>> >>> >>> <java> -Dxxx -Dyyy -jar start.jar
> >>> >>> >>>
> >>> >>> >>> If you are using the multi-process configuration, that is what
> the
> >>> >>> >>> "defines" directory is for; you only need to create files in
> that
> >>> >>> >>> directory with the names "jcifs.smb.lmCompatibility" and
> >>> >>> >>> "jcifs.smb.client.useExtendedSecurity" containing the values
> you
> >>> >>> >>> want
> >>> >>> >>> to set.
> >>> >>> >>>
> >>> >>> >>> Karl
> >>> >>> >>>
> >>> >>> >>>
> >>> >>> >>> On Thu, Nov 17, 2011 at 1:11 AM, Swapna Vuppala
> >>> >>> >>> <sw...@gmail.com> wrote:
> >>> >>> >>> > Hi Karl,
> >>> >>> >>> >
> >>> >>> >>> > Am able to access the folders on the problem server through
> >>> >>> >>> > windows
> >>> >>> >>> > explorer, (\\server3\Folder1). I tried couple of things with
> the
> >>> >>> >>> > credentials
> >>> >>> >>> > form, changing username, domain etc.. but I keep getting the
> >>> >>> >>> > same
> >>> >>> >>> > error
> >>> >>> >>> > "Couldn't connect to server: Logon failure: unknown user
> name or
> >>> >>> >>> > bad
> >>> >>> >>> > password"
> >>> >>> >>> >
> >>> >>> >>> > Can you tell me more about the -D switch you were talking of
> ?
> >>> >>> >>> >
> >>> >>> >>> > Thanks and Regards,
> >>> >>> >>> > Swapna.
> >>> >>> >>> >
> >>> >>> >>> > On Tue, Nov 15, 2011 at 12:40 PM, Karl Wright
> >>> >>> >>> > <da...@gmail.com>
> >>> >>> >>> > wrote:
> >>> >>> >>> >>
> >>> >>> >>> >> Glad you chased it down this far.
> >>> >>> >>> >>
> >>> >>> >>> >> First thing to try is whether you can get into the problem
> >>> >>> >>> >> server
> >>> >>> >>> >> using Windows Explorer. Obviously ManifoldCF is not going
> to
> >>> >>> >>> >> be
> >>> >>> >>> >> able
> >>> >>> >>> >> to do it if Windows can't. If you *can* get in, then just
> >>> >>> >>> >> playing
> >>> >>> >>> >> with the form of the credentials in the MCF connection
> might do
> >>> >>> >>> >> the
> >>> >>> >>> >> trick. Some Windows or net appliance servers are picky
> about
> >>> >>> >>> >> this.
> >>> >>> >>> >> Try various things like leaving the domain blank and
> specifying
> >>> >>> >>> >> the
> >>> >>> >>> >> user as "abc@domain.com", for instance. There's also a
> >>> >>> >>> >> different
> >>> >>> >>> >> NTLM
> >>> >>> >>> >> mode you can operation jcifs in that some servers may be
> >>> >>> >>> >> configured
> >>> >>> >>> >> to
> >>> >>> >>> >> require; this would need you to set a -D switch on the
> command
> >>> >>> >>> >> line
> >>> >>> >>> >> to
> >>> >>> >>> >> enable.
> >>> >>> >>> >>
> >>> >>> >>> >> Karl
> >>> >>> >>> >>
> >>> >>> >>> >> On Tue, Nov 15, 2011 at 12:10 AM, Swapna Vuppala
> >>> >>> >>> >> <sw...@gmail.com> wrote:
> >>> >>> >>> >> > Hi Karl,
> >>> >>> >>> >> >
> >>> >>> >>> >> > Thanks for the input. It looks like my problem is related
> to
> >>> >>> >>> >> > the
> >>> >>> >>> >> > second
> >>> >>> >>> >> > one
> >>> >>> >>> >> > that you specified. One of the directories in the path am
> >>> >>> >>> >> > trying
> >>> >>> >>> >> > to
> >>> >>> >>> >> > index is
> >>> >>> >>> >> > actually redirecting to a different server. And when I
> >>> >>> >>> >> > specify
> >>> >>> >>> >> > this
> >>> >>> >>> >> > new
> >>> >>> >>> >> > server in defining the repository connection, with my
> >>> >>> >>> >> > credentials,
> >>> >>> >>> >> > the
> >>> >>> >>> >> > connection fails with the message: "Couldn't connect to
> >>> >>> >>> >> > server:
> >>> >>> >>> >> > Logon
> >>> >>> >>> >> > failure: unknown user name or bad password"
> >>> >>> >>> >> >
> >>> >>> >>> >> > I'll look into why am not able to connect to this server.
> >>> >>> >>> >> >
> >>> >>> >>> >> > Thanks and Regards,
> >>> >>> >>> >> > Swapna.
> >>> >>> >>> >> >
> >>> >>> >>> >> > On Mon, Nov 14, 2011 at 4:56 PM, Karl Wright
> >>> >>> >>> >> > <da...@gmail.com>
> >>> >>> >>> >> > wrote:
> >>> >>> >>> >> >>
> >>> >>> >>> >> >> There's two kinds of problem you might be having. The
> first
> >>> >>> >>> >> >> is
> >>> >>> >>> >> >> intermittent, and the second is not intermittent but
> would
> >>> >>> >>> >> >> have
> >>> >>> >>> >> >> something to do with specific directories.
> >>> >>> >>> >> >>
> >>> >>> >>> >> >> Intermittent problems might include a domain controller
> that
> >>> >>> >>> >> >> is
> >>> >>> >>> >> >> not
> >>> >>> >>> >> >> always accessible. In such cases, the crawl will proceed
> >>> >>> >>> >> >> but
> >>> >>> >>> >> >> will
> >>> >>> >>> >> >> tend to fail unpredictably. On the other hand, if you
> have
> >>> >>> >>> >> >> a
> >>> >>> >>> >> >> directory that is handled by a DFS redirection, it is
> >>> >>> >>> >> >> possible
> >>> >>> >>> >> >> that
> >>> >>> >>> >> >> the redirection is indicating a new server (lets call it
> >>> >>> >>> >> >> server3)
> >>> >>> >>> >> >> which may not like the precise form of your login
> >>> >>> >>> >> >> credentials.
> >>> >>> >>> >> >> Can
> >>> >>> >>> >> >> you determine which scenario you are seeing?
> >>> >>> >>> >> >>
> >>> >>> >>> >> >> Karl
> >>> >>> >>> >> >>
> >>> >>> >>> >> >> On Mon, Nov 14, 2011 at 3:11 AM, Swapna Vuppala
> >>> >>> >>> >> >> <sw...@gmail.com> wrote:
> >>> >>> >>> >> >> > Hi,
> >>> >>> >>> >> >> >
> >>> >>> >>> >> >> > I have been using windows share repository connection
> to
> >>> >>> >>> >> >> > crawl
> >>> >>> >>> >> >> > and
> >>> >>> >>> >> >> > get
> >>> >>> >>> >> >> > data
> >>> >>> >>> >> >> > from a particular server (server 1). Its working
> perfectly
> >>> >>> >>> >> >> > fine.
> >>> >>> >>> >> >> > However, am
> >>> >>> >>> >> >> > having trouble when I try with data from another server
> >>> >>> >>> >> >> > (server
> >>> >>> >>> >> >> > 2).
> >>> >>> >>> >> >> >
> >>> >>> >>> >> >> > When I define a repository connection of type windows
> >>> >>> >>> >> >> > share
> >>> >>> >>> >> >> > and
> >>> >>> >>> >> >> > specify
> >>> >>> >>> >> >> > the
> >>> >>> >>> >> >> > server name (server 2) with my credentials, the
> connection
> >>> >>> >>> >> >> > status
> >>> >>> >>> >> >> > shows
> >>> >>> >>> >> >> > "Connection working". But when I run a job to use this
> >>> >>> >>> >> >> > repository
> >>> >>> >>> >> >> > connection
> >>> >>> >>> >> >> > and index data from a location on this server 2, I keep
> >>> >>> >>> >> >> > getting
> >>> >>> >>> >> >> > the
> >>> >>> >>> >> >> > exception below:
> >>> >>> >>> >> >> >
> >>> >>> >>> >> >> > JCIFS: Possibly transient exception detected on
> attempt 3
> >>> >>> >>> >> >> > while
> >>> >>> >>> >> >> > checking
> >>> >>> >>> >> >> > if
> >>> >>> >>> >> >> > file exists: Logon failure: unknown user name or bad
> >>> >>> >>> >> >> > password.
> >>> >>> >>> >> >> > jcifs.smb.SmbAuthException: Logon failure: unknown user
> >>> >>> >>> >> >> > name
> >>> >>> >>> >> >> > or
> >>> >>> >>> >> >> > bad
> >>> >>> >>> >> >> > password.
> >>> >>> >>> >> >> > at
> >>> >>> >>> >> >> >
> jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:544)
> >>> >>> >>> >> >> > at
> jcifs.smb.SmbTransport.send(SmbTransport.java:661)
> >>> >>> >>> >> >> > at
> >>> >>> >>> >> >> > jcifs.smb.SmbSession.sessionSetup(SmbSession.java:390)
> >>> >>> >>> >> >> > at jcifs.smb.SmbSession.send(SmbSession.java:218)
> >>> >>> >>> >> >> > at jcifs.smb.SmbTree.treeConnect(SmbTree.java:176)
> >>> >>> >>> >> >> > at jcifs.smb.SmbFile.doConnect(SmbFile.java:911)
> >>> >>> >>> >> >> > at jcifs.smb.SmbFile.connect(SmbFile.java:954)
> >>> >>> >>> >> >> > at jcifs.smb.SmbFile.connect0(SmbFile.java:880)
> >>> >>> >>> >> >> > at jcifs.smb.SmbFile.queryPath(SmbFile.java:1335)
> >>> >>> >>> >> >> > at jcifs.smb.SmbFile.exists(SmbFile.java:1417)
> >>> >>> >>> >> >> > at
> >>> >>> >>> >> >> >
> >>> >>> >>> >> >> >
> >>> >>> >>> >> >> >
> >>> >>> >>> >> >> >
> >>> >>> >>> >> >> >
> >>> >>> >>> >> >> >
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.fileExists(SharedDriveConnector.java:2064)
> >>> >>> >>> >> >> > at
> >>> >>> >>> >> >> >
> >>> >>> >>> >> >> >
> >>> >>> >>> >> >> >
> >>> >>> >>> >> >> >
> >>> >>> >>> >> >> >
> >>> >>> >>> >> >> >
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getDocumentVersions(SharedDriveConnector.java:521)
> >>> >>> >>> >> >> > at
> >>> >>> >>> >> >> >
> >>> >>> >>> >> >> >
> >>> >>> >>> >> >> >
> >>> >>> >>> >> >> >
> >>> >>> >>> >> >> >
> >>> >>> >>> >> >> >
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:318)
> >>> >>> >>> >> >> >
> >>> >>> >>> >> >> > Am able to access this location from windows explorer.
> >>> >>> >>> >> >> > What
> >>> >>> >>> >> >> > else
> >>> >>> >>> >> >> > should
> >>> >>> >>> >> >> > I be
> >>> >>> >>> >> >> > checking or what could be the reasons/factors causing
> this
> >>> >>> >>> >> >> > to
> >>> >>> >>> >> >> > fail
> >>> >>> >>> >> >> > ?
> >>> >>> >>> >> >> >
> >>> >>> >>> >> >> > Thanks and Regards,
> >>> >>> >>> >> >> > Swapna.
> >>> >>> >>> >> >> >
> >>> >>> >>> >> >
> >>> >>> >>> >> >
> >>> >>> >>> >
> >>> >>> >>> >
> >>> >>> >>
> >>> >>> >>
> >>> >>
> >>> >>
> >>
> >>
>
Re: Problem crawling windows share
Posted by Karl Wright <da...@gmail.com>.
His response:
"Actually from looking at the capture again, the successful operations
in it were ultimately performed by a session that used Kerberos. But
there are two other successful NTLM authentications that use the
Anonymous credential. It so happens that neither of these went on to
do anything successful. Meaning your capture does not contain a
successful operation using NTLM and thus there is nothing to compare
JCIFS too. It could be that the environment actually does not support
NTLM properly. There are a lot of errors and oddities in the non-JCIFS
communications with numerous login failures, disabled signatures,
access denied conditions and unsupported versions of Windows (Windows
Server 2002 SP2 build 2600). The environment is not optimal.
To continue I would need proper comparison captures. Clearly the
NtlmPasswordAuthentication("","","") does not work. But
NtlmPasswordAuthentication.ANONYMOUS obviously does work to some
degree but there's something else wrong with the session setup
request. That something could be the fact that the hostname used is
actually l-carx01.global.arup.com and not lon-data-arx.global.arup.com
and then the tree connect goes on to mount
//lon-data-arx.global.arup.com/stg-jobs$ so clearly there is some
hostname discontinuity going on as well."
I think this is pretty much self explanatory. Whatever it is that you
are trying to talk to seems to be one truly crazy machine, certainly
not really Windows, and it is almost an accident that it works with
Windows at all. I think this is rapidly reaching the point of
diminishing returns in getting jCIFS to work with it.
Karl
On Mon, Dec 12, 2011 at 4:24 AM, Karl Wright <da...@gmail.com> wrote:
> I forwarded this research again to Mr. Allen for further suggestions. Thanks!
>
> Karl
>
> On Mon, Dec 12, 2011 at 1:58 AM, Swapna Vuppala
> <sw...@gmail.com> wrote:
>> Hi Karl,
>>
>> I tried modifying the test code as you suggested.
>>
>> When I tried with empty credentials
>>
>> NtlmPasswordAuthentication pa = new NtlmPasswordAuthentication("","","");
>> SmbFile smbconnection = new SmbFile("smb://" +
>> "lon-data-arx.global.arup.com" + "/" + "stg-jobs$",pa);
>>
>> I got the same exception as before
>>
>> Exception in thread "main" jcifs.smb.SmbAuthException: Logon failure:
>> unknown us
>>
>> er name or bad password.
>> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:546)
>> at jcifs.smb.SmbTransport.send(SmbTransport.java:663)
>>
>> at jcifs.smb.SmbSession.sessionSetup(SmbSession.java:390)
>> at jcifs.smb.SmbSession.send(SmbSession.java:218)
>> at jcifs.smb.SmbTree.treeConnect(SmbTree.java:176)
>> at jcifs.smb.SmbFile.doConnect(SmbFile.java:911)
>>
>> ......
>>
>> When I tried with NtlmPasswordAuthentication.ANONYMOUS
>>
>> SmbFile smbconnection = new SmbFile("smb://" +
>> "lon-data-arx.global.arup.com" + "/" +
>> "stg-jobs$",NtlmPasswordAuthentication.ANONYMOUS);
>>
>> I got a different exception
>>
>> Exception in thread "main" jcifs.smb.SmbException: Incorrect function.
>> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:563)
>> at jcifs.smb.SmbTransport.send(SmbTransport.java:663)
>> at jcifs.smb.SmbSession.sessionSetup(SmbSession.java:316)
>>
>> at jcifs.smb.SmbSession.send(SmbSession.java:218)
>> at jcifs.smb.SmbTree.treeConnect(SmbTree.java:176)
>> at jcifs.smb.SmbFile.doConnect(SmbFile.java:911)
>> .....
>>
>> Thanks and Regards,
>> Swapna.
>> On Fri, Dec 9, 2011 at 11:46 PM, Karl Wright <da...@gmail.com> wrote:
>>>
>>> Michael Allen's response to your capture:
>>>
>>> "It looks like the non-JCIFS communication is successful because the
>>> anonymous credential is used.
>>>
>>> Note that there is still no sign of DFS going on anywhere. Actually
>>> JCIFS tries to do a referral for the DFS root and gets an error of
>>> STATUS_FS_DRIVER_REQUIRED so it looks like DFS may not even be setup
>>> at all in the target environment. At least not domain based DFS.
>>>
>>> So if you want to emulate Windows behavior, I guess you need to retry
>>> the operation with credential NtlmPasswordAuthentication.ANONYMOUS (or
>>> all empty strings should work too I think like ";:")."
>>>
>>> In other words, the issue is that the server in question does not
>>> actually recognize the credentials given at all, and gives an error.
>>> But while Windows then retries with no credentials whatsoever, nothing
>>> in JCIFS does that retry.
>>>
>>> You can confirm that this is what is happening by using an empty user
>>> name, password, and domain in the JCIFS test code you wrote to obtain
>>> the captures. If, then, the test code works against that server, I'll
>>> create a ticket for downgrading to ANONYMOUS whenever there is an
>>> authentication error in the JCIFS connector.
>>>
>>> Thanks,
>>> Karl
>>>
>>> On Wed, Dec 7, 2011 at 6:41 AM, Karl Wright <da...@gmail.com> wrote:
>>> > The next step is to obtain two packet captures using WireShark. These
>>> > should both be done from the machine that is not joined to the domain.
>>> > The first capture should be the interaction that takes place when you
>>> > try to go to the foreign directory, and you supply the credentials,
>>> > and then you list the folders. The second capture will require you to
>>> > do the same using JCIFS. For that capture, you will need jcifs.jar
>>> > and a simple example class that provides jcifs with credentials and
>>> > tries to list the directory. A simple class should be enough to do
>>> > this, including code such as:
>>> >
>>> > import jcifs.smb.*;
>>> >
>>> > NtlmPasswordAuthentication pa = new
>>> > NtlmPasswordAuthentication(domain,username,password);
>>> > SmbFile smbconnection = new SmbFile("smb://" + server + "/" + share,pa);
>>> > SmbFile[] files = smbconnection.listFiles();
>>> >
>>> > for (int i = 0 ; i < files.length ; i++)
>>> > {
>>> > System.out.println(file.toString());
>>> > }
>>> >
>>> > I suggest you put together your test class first and make sure it
>>> > works, and then go after the captures.
>>> >
>>> > Let me know how it goes.
>>> >
>>> > Karl
>>> >
>>> > On Wed, Dec 7, 2011 at 6:10 AM, Swapna Vuppala
>>> > <sw...@gmail.com> wrote:
>>> >> Hi Karl,
>>> >>
>>> >> Thanks for the detailed reply.
>>> >>
>>> >> As per your suggestion, I tried connecting to the problematic server
>>> >> via
>>> >> windows using a machine that is not in the domain. On supplying the
>>> >> credentials via windows, I could connect to the server, although it did
>>> >> not
>>> >> list any folders. But when I tried accessing the folder of my interest
>>> >> via
>>> >> \\server\folder$, I could see all the sub folders. All this is when I
>>> >> connected via windows from a machine that's not joined to the domain.
>>> >>
>>> >> The behavior is same when I connect to the server via windows from a
>>> >> machine
>>> >> that's joined to the domain.
>>> >>
>>> >> So I assume this means that through both Kerberos and NTLM, I see the
>>> >> same
>>> >> behavior. Whereas ManifoldCF throws an exception when trying to define
>>> >> the
>>> >> repository connection.
>>> >>
>>> >> Please advise me on what I should be doing next to resolve this.
>>> >>
>>> >> Thanks and Regards,
>>> >> Swapna.
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> On Tue, Dec 6, 2011 at 12:55 PM, Karl Wright <da...@gmail.com>
>>> >> wrote:
>>> >>>
>>> >>> About your capture - Michael Allen says the following:
>>> >>>
>>> >>> "Actually this has nothing to do with DFS. JCIFS does not get to the
>>> >>> point where it does DFS anything. The capture shows a vanilla
>>> >>> STATUS_LOGON_FAILURE when GLOBAL\swapna.vuppala tries to auth with
>>> >>> l-carx01.global.arup.com. So the possible causes for this are 1) the
>>> >>> account name is not valid 2) the supplied password is incorrect 3)
>>> >>> some security policy is deliberately blocking that user or particular
>>> >>> type of auth or 4) some server configuration is incompatible with
>>> >>> JCIFS. I only mention this last option because I noticed the target
>>> >>> server has security signatures disabled. That's strange. If they're
>>> >>> messing around with things like that, who knows what their clients are
>>> >>> expected to do.
>>> >>>
>>> >>> Try a Windows client that uses NTLM instead of Kerberos. Meaning try a
>>> >>> machine that is not joined to the domain so that when you try to
>>> >>> access the target it asks you for credentials at which point you can
>>> >>> test with GLOBAL\swapna.vuppala. Then it will use NTLM and you can
>>> >>> actually compare captures. If the operator doesn't have a laptop or
>>> >>> something not joined to the domain, it might be sufficient to log into
>>> >>> a workstation using machine credentials and not domain credentials.
>>> >>>
>>> >>> Also when testing JCIFS you should use a simple stand-alone program
>>> >>> like examples/ListFiles.java."
>>> >>>
>>> >>> In other words:
>>> >>> (a) Since JCIFS does not use Kerberos for authentication, you need to
>>> >>> try to log into the recalcitrant server via Windows without using
>>> >>> Kerberos to be able to do a side-by-side comparison. Michael has some
>>> >>> ways of doing that, above.
>>> >>> (b) You may find that it doesn't work, in which case JCIFS is not
>>> >>> going to work either.
>>> >>> (c) If it *does* work, then try to generate your side-by-side
>>> >>> comparisons using a simpler example rather than ManifoldCF en toto;
>>> >>> you can see how at jcifs.samba.org, or I can help you further.
>>> >>>
>>> >>> He also mentions that there is some bizarreness on the response that
>>> >>> indicates that the server is configured in a way that he's never seen
>>> >>> before. And believe me, Michael has seen a *lot* of strange
>>> >>> configurations...
>>> >>>
>>> >>> Hope this helps.
>>> >>> Karl
>>> >>>
>>> >>> On Mon, Nov 28, 2011 at 4:12 AM, Karl Wright <da...@gmail.com>
>>> >>> wrote:
>>> >>> > That should read "properties.xml", not "properties.ini". It looks
>>> >>> > like this page needs updating.
>>> >>> >
>>> >>> > The debug property in the XML form is:
>>> >>> >
>>> >>> > <property name="org.apache.manifoldcf.connectors" value="DEBUG"/>
>>> >>> >
>>> >>> > I don't think it will provide you with any additional information
>>> >>> > that
>>> >>> > is useful for debugging your authentication issue, however, if that
>>> >>> > is
>>> >>> > why you are looking at it. There may be some jcifs.jar debugging
>>> >>> > switches that might be of more help, but in the end I suspect you
>>> >>> > will
>>> >>> > need a packet capture of both a successful connection (via Windows)
>>> >>> > and an unsuccessful one (via MCF). The guy you will need to talk
>>> >>> > with
>>> >>> > after that is the jcifs author Michael Allen; I can give you his
>>> >>> > email
>>> >>> > address if you get that far.
>>> >>> >
>>> >>> > Karl
>>> >>> >
>>> >>> >
>>> >>> > On Mon, Nov 28, 2011 at 1:30 AM, Swapna Vuppala
>>> >>> > <sw...@gmail.com> wrote:
>>> >>> >> Hi Karl,
>>> >>> >>
>>> >>> >> I was planning to debug jCIFS repository connection using WireShark
>>> >>> >> and
>>> >>> >> I
>>> >>> >> came across this
>>> >>> >> https://cwiki.apache.org/CONNECTORS/debugging-connections.html
>>> >>> >> Here, I see something as add
>>> >>> >> "org.apache.manifoldcf.connectors=DEBUG"
>>> >>> >> to the
>>> >>> >> properties.ini file. Is it the properties.xml file that is being
>>> >>> >> referred
>>> >>> >> here ? If not, where do I find properties.ini file ?
>>> >>> >>
>>> >>> >> Thanks and Regards,
>>> >>> >> Swapna.
>>> >>> >>
>>> >>> >> On Thu, Nov 17, 2011 at 1:31 PM, Karl Wright <da...@gmail.com>
>>> >>> >> wrote:
>>> >>> >>>
>>> >>> >>> See http://jcifs.samba.org/src/docs/api/overview-summary.html#scp.
>>> >>> >>> The properties jcifs.smb.lmCompatibility and
>>> >>> >>> jcifs.smb.client.useExtendedSecurity are the ones you may want to
>>> >>> >>> change. These two properties go together so certain combinations
>>> >>> >>> make
>>> >>> >>> sense and others don't, so there's really only combinations you
>>> >>> >>> need
>>> >>> >>> but I'll need to look at what they are and get back to you later
>>> >>> >>> today.
>>> >>> >>>
>>> >>> >>> As far as setting the switches are concerned, if you are using the
>>> >>> >>> Quick Start you do this trivially by:
>>> >>> >>>
>>> >>> >>> <java> -Dxxx -Dyyy -jar start.jar
>>> >>> >>>
>>> >>> >>> If you are using the multi-process configuration, that is what the
>>> >>> >>> "defines" directory is for; you only need to create files in that
>>> >>> >>> directory with the names "jcifs.smb.lmCompatibility" and
>>> >>> >>> "jcifs.smb.client.useExtendedSecurity" containing the values you
>>> >>> >>> want
>>> >>> >>> to set.
>>> >>> >>>
>>> >>> >>> Karl
>>> >>> >>>
>>> >>> >>>
>>> >>> >>> On Thu, Nov 17, 2011 at 1:11 AM, Swapna Vuppala
>>> >>> >>> <sw...@gmail.com> wrote:
>>> >>> >>> > Hi Karl,
>>> >>> >>> >
>>> >>> >>> > Am able to access the folders on the problem server through
>>> >>> >>> > windows
>>> >>> >>> > explorer, (\\server3\Folder1). I tried couple of things with the
>>> >>> >>> > credentials
>>> >>> >>> > form, changing username, domain etc.. but I keep getting the
>>> >>> >>> > same
>>> >>> >>> > error
>>> >>> >>> > "Couldn't connect to server: Logon failure: unknown user name or
>>> >>> >>> > bad
>>> >>> >>> > password"
>>> >>> >>> >
>>> >>> >>> > Can you tell me more about the -D switch you were talking of ?
>>> >>> >>> >
>>> >>> >>> > Thanks and Regards,
>>> >>> >>> > Swapna.
>>> >>> >>> >
>>> >>> >>> > On Tue, Nov 15, 2011 at 12:40 PM, Karl Wright
>>> >>> >>> > <da...@gmail.com>
>>> >>> >>> > wrote:
>>> >>> >>> >>
>>> >>> >>> >> Glad you chased it down this far.
>>> >>> >>> >>
>>> >>> >>> >> First thing to try is whether you can get into the problem
>>> >>> >>> >> server
>>> >>> >>> >> using Windows Explorer. Obviously ManifoldCF is not going to
>>> >>> >>> >> be
>>> >>> >>> >> able
>>> >>> >>> >> to do it if Windows can't. If you *can* get in, then just
>>> >>> >>> >> playing
>>> >>> >>> >> with the form of the credentials in the MCF connection might do
>>> >>> >>> >> the
>>> >>> >>> >> trick. Some Windows or net appliance servers are picky about
>>> >>> >>> >> this.
>>> >>> >>> >> Try various things like leaving the domain blank and specifying
>>> >>> >>> >> the
>>> >>> >>> >> user as "abc@domain.com", for instance. There's also a
>>> >>> >>> >> different
>>> >>> >>> >> NTLM
>>> >>> >>> >> mode you can operation jcifs in that some servers may be
>>> >>> >>> >> configured
>>> >>> >>> >> to
>>> >>> >>> >> require; this would need you to set a -D switch on the command
>>> >>> >>> >> line
>>> >>> >>> >> to
>>> >>> >>> >> enable.
>>> >>> >>> >>
>>> >>> >>> >> Karl
>>> >>> >>> >>
>>> >>> >>> >> On Tue, Nov 15, 2011 at 12:10 AM, Swapna Vuppala
>>> >>> >>> >> <sw...@gmail.com> wrote:
>>> >>> >>> >> > Hi Karl,
>>> >>> >>> >> >
>>> >>> >>> >> > Thanks for the input. It looks like my problem is related to
>>> >>> >>> >> > the
>>> >>> >>> >> > second
>>> >>> >>> >> > one
>>> >>> >>> >> > that you specified. One of the directories in the path am
>>> >>> >>> >> > trying
>>> >>> >>> >> > to
>>> >>> >>> >> > index is
>>> >>> >>> >> > actually redirecting to a different server. And when I
>>> >>> >>> >> > specify
>>> >>> >>> >> > this
>>> >>> >>> >> > new
>>> >>> >>> >> > server in defining the repository connection, with my
>>> >>> >>> >> > credentials,
>>> >>> >>> >> > the
>>> >>> >>> >> > connection fails with the message: "Couldn't connect to
>>> >>> >>> >> > server:
>>> >>> >>> >> > Logon
>>> >>> >>> >> > failure: unknown user name or bad password"
>>> >>> >>> >> >
>>> >>> >>> >> > I'll look into why am not able to connect to this server.
>>> >>> >>> >> >
>>> >>> >>> >> > Thanks and Regards,
>>> >>> >>> >> > Swapna.
>>> >>> >>> >> >
>>> >>> >>> >> > On Mon, Nov 14, 2011 at 4:56 PM, Karl Wright
>>> >>> >>> >> > <da...@gmail.com>
>>> >>> >>> >> > wrote:
>>> >>> >>> >> >>
>>> >>> >>> >> >> There's two kinds of problem you might be having. The first
>>> >>> >>> >> >> is
>>> >>> >>> >> >> intermittent, and the second is not intermittent but would
>>> >>> >>> >> >> have
>>> >>> >>> >> >> something to do with specific directories.
>>> >>> >>> >> >>
>>> >>> >>> >> >> Intermittent problems might include a domain controller that
>>> >>> >>> >> >> is
>>> >>> >>> >> >> not
>>> >>> >>> >> >> always accessible. In such cases, the crawl will proceed
>>> >>> >>> >> >> but
>>> >>> >>> >> >> will
>>> >>> >>> >> >> tend to fail unpredictably. On the other hand, if you have
>>> >>> >>> >> >> a
>>> >>> >>> >> >> directory that is handled by a DFS redirection, it is
>>> >>> >>> >> >> possible
>>> >>> >>> >> >> that
>>> >>> >>> >> >> the redirection is indicating a new server (lets call it
>>> >>> >>> >> >> server3)
>>> >>> >>> >> >> which may not like the precise form of your login
>>> >>> >>> >> >> credentials.
>>> >>> >>> >> >> Can
>>> >>> >>> >> >> you determine which scenario you are seeing?
>>> >>> >>> >> >>
>>> >>> >>> >> >> Karl
>>> >>> >>> >> >>
>>> >>> >>> >> >> On Mon, Nov 14, 2011 at 3:11 AM, Swapna Vuppala
>>> >>> >>> >> >> <sw...@gmail.com> wrote:
>>> >>> >>> >> >> > Hi,
>>> >>> >>> >> >> >
>>> >>> >>> >> >> > I have been using windows share repository connection to
>>> >>> >>> >> >> > crawl
>>> >>> >>> >> >> > and
>>> >>> >>> >> >> > get
>>> >>> >>> >> >> > data
>>> >>> >>> >> >> > from a particular server (server 1). Its working perfectly
>>> >>> >>> >> >> > fine.
>>> >>> >>> >> >> > However, am
>>> >>> >>> >> >> > having trouble when I try with data from another server
>>> >>> >>> >> >> > (server
>>> >>> >>> >> >> > 2).
>>> >>> >>> >> >> >
>>> >>> >>> >> >> > When I define a repository connection of type windows
>>> >>> >>> >> >> > share
>>> >>> >>> >> >> > and
>>> >>> >>> >> >> > specify
>>> >>> >>> >> >> > the
>>> >>> >>> >> >> > server name (server 2) with my credentials, the connection
>>> >>> >>> >> >> > status
>>> >>> >>> >> >> > shows
>>> >>> >>> >> >> > "Connection working". But when I run a job to use this
>>> >>> >>> >> >> > repository
>>> >>> >>> >> >> > connection
>>> >>> >>> >> >> > and index data from a location on this server 2, I keep
>>> >>> >>> >> >> > getting
>>> >>> >>> >> >> > the
>>> >>> >>> >> >> > exception below:
>>> >>> >>> >> >> >
>>> >>> >>> >> >> > JCIFS: Possibly transient exception detected on attempt 3
>>> >>> >>> >> >> > while
>>> >>> >>> >> >> > checking
>>> >>> >>> >> >> > if
>>> >>> >>> >> >> > file exists: Logon failure: unknown user name or bad
>>> >>> >>> >> >> > password.
>>> >>> >>> >> >> > jcifs.smb.SmbAuthException: Logon failure: unknown user
>>> >>> >>> >> >> > name
>>> >>> >>> >> >> > or
>>> >>> >>> >> >> > bad
>>> >>> >>> >> >> > password.
>>> >>> >>> >> >> > at
>>> >>> >>> >> >> > jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:544)
>>> >>> >>> >> >> > at jcifs.smb.SmbTransport.send(SmbTransport.java:661)
>>> >>> >>> >> >> > at
>>> >>> >>> >> >> > jcifs.smb.SmbSession.sessionSetup(SmbSession.java:390)
>>> >>> >>> >> >> > at jcifs.smb.SmbSession.send(SmbSession.java:218)
>>> >>> >>> >> >> > at jcifs.smb.SmbTree.treeConnect(SmbTree.java:176)
>>> >>> >>> >> >> > at jcifs.smb.SmbFile.doConnect(SmbFile.java:911)
>>> >>> >>> >> >> > at jcifs.smb.SmbFile.connect(SmbFile.java:954)
>>> >>> >>> >> >> > at jcifs.smb.SmbFile.connect0(SmbFile.java:880)
>>> >>> >>> >> >> > at jcifs.smb.SmbFile.queryPath(SmbFile.java:1335)
>>> >>> >>> >> >> > at jcifs.smb.SmbFile.exists(SmbFile.java:1417)
>>> >>> >>> >> >> > at
>>> >>> >>> >> >> >
>>> >>> >>> >> >> >
>>> >>> >>> >> >> >
>>> >>> >>> >> >> >
>>> >>> >>> >> >> >
>>> >>> >>> >> >> > org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.fileExists(SharedDriveConnector.java:2064)
>>> >>> >>> >> >> > at
>>> >>> >>> >> >> >
>>> >>> >>> >> >> >
>>> >>> >>> >> >> >
>>> >>> >>> >> >> >
>>> >>> >>> >> >> >
>>> >>> >>> >> >> > org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getDocumentVersions(SharedDriveConnector.java:521)
>>> >>> >>> >> >> > at
>>> >>> >>> >> >> >
>>> >>> >>> >> >> >
>>> >>> >>> >> >> >
>>> >>> >>> >> >> >
>>> >>> >>> >> >> >
>>> >>> >>> >> >> > org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:318)
>>> >>> >>> >> >> >
>>> >>> >>> >> >> > Am able to access this location from windows explorer.
>>> >>> >>> >> >> > What
>>> >>> >>> >> >> > else
>>> >>> >>> >> >> > should
>>> >>> >>> >> >> > I be
>>> >>> >>> >> >> > checking or what could be the reasons/factors causing this
>>> >>> >>> >> >> > to
>>> >>> >>> >> >> > fail
>>> >>> >>> >> >> > ?
>>> >>> >>> >> >> >
>>> >>> >>> >> >> > Thanks and Regards,
>>> >>> >>> >> >> > Swapna.
>>> >>> >>> >> >> >
>>> >>> >>> >> >
>>> >>> >>> >> >
>>> >>> >>> >
>>> >>> >>> >
>>> >>> >>
>>> >>> >>
>>> >>
>>> >>
>>
>>
Re: Problem crawling windows share
Posted by Karl Wright <da...@gmail.com>.
I forwarded this research again to Mr. Allen for further suggestions. Thanks!
Karl
On Mon, Dec 12, 2011 at 1:58 AM, Swapna Vuppala
<sw...@gmail.com> wrote:
> Hi Karl,
>
> I tried modifying the test code as you suggested.
>
> When I tried with empty credentials
>
> NtlmPasswordAuthentication pa = new NtlmPasswordAuthentication("","","");
> SmbFile smbconnection = new SmbFile("smb://" +
> "lon-data-arx.global.arup.com" + "/" + "stg-jobs$",pa);
>
> I got the same exception as before
>
> Exception in thread "main" jcifs.smb.SmbAuthException: Logon failure:
> unknown us
>
> er name or bad password.
> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:546)
> at jcifs.smb.SmbTransport.send(SmbTransport.java:663)
>
> at jcifs.smb.SmbSession.sessionSetup(SmbSession.java:390)
> at jcifs.smb.SmbSession.send(SmbSession.java:218)
> at jcifs.smb.SmbTree.treeConnect(SmbTree.java:176)
> at jcifs.smb.SmbFile.doConnect(SmbFile.java:911)
>
> ......
>
> When I tried with NtlmPasswordAuthentication.ANONYMOUS
>
> SmbFile smbconnection = new SmbFile("smb://" +
> "lon-data-arx.global.arup.com" + "/" +
> "stg-jobs$",NtlmPasswordAuthentication.ANONYMOUS);
>
> I got a different exception
>
> Exception in thread "main" jcifs.smb.SmbException: Incorrect function.
> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:563)
> at jcifs.smb.SmbTransport.send(SmbTransport.java:663)
> at jcifs.smb.SmbSession.sessionSetup(SmbSession.java:316)
>
> at jcifs.smb.SmbSession.send(SmbSession.java:218)
> at jcifs.smb.SmbTree.treeConnect(SmbTree.java:176)
> at jcifs.smb.SmbFile.doConnect(SmbFile.java:911)
> .....
>
> Thanks and Regards,
> Swapna.
> On Fri, Dec 9, 2011 at 11:46 PM, Karl Wright <da...@gmail.com> wrote:
>>
>> Michael Allen's response to your capture:
>>
>> "It looks like the non-JCIFS communication is successful because the
>> anonymous credential is used.
>>
>> Note that there is still no sign of DFS going on anywhere. Actually
>> JCIFS tries to do a referral for the DFS root and gets an error of
>> STATUS_FS_DRIVER_REQUIRED so it looks like DFS may not even be setup
>> at all in the target environment. At least not domain based DFS.
>>
>> So if you want to emulate Windows behavior, I guess you need to retry
>> the operation with credential NtlmPasswordAuthentication.ANONYMOUS (or
>> all empty strings should work too I think like ";:")."
>>
>> In other words, the issue is that the server in question does not
>> actually recognize the credentials given at all, and gives an error.
>> But while Windows then retries with no credentials whatsoever, nothing
>> in JCIFS does that retry.
>>
>> You can confirm that this is what is happening by using an empty user
>> name, password, and domain in the JCIFS test code you wrote to obtain
>> the captures. If, then, the test code works against that server, I'll
>> create a ticket for downgrading to ANONYMOUS whenever there is an
>> authentication error in the JCIFS connector.
>>
>> Thanks,
>> Karl
>>
>> On Wed, Dec 7, 2011 at 6:41 AM, Karl Wright <da...@gmail.com> wrote:
>> > The next step is to obtain two packet captures using WireShark. These
>> > should both be done from the machine that is not joined to the domain.
>> > The first capture should be the interaction that takes place when you
>> > try to go to the foreign directory, and you supply the credentials,
>> > and then you list the folders. The second capture will require you to
>> > do the same using JCIFS. For that capture, you will need jcifs.jar
>> > and a simple example class that provides jcifs with credentials and
>> > tries to list the directory. A simple class should be enough to do
>> > this, including code such as:
>> >
>> > import jcifs.smb.*;
>> >
>> > NtlmPasswordAuthentication pa = new
>> > NtlmPasswordAuthentication(domain,username,password);
>> > SmbFile smbconnection = new SmbFile("smb://" + server + "/" + share,pa);
>> > SmbFile[] files = smbconnection.listFiles();
>> >
>> > for (int i = 0 ; i < files.length ; i++)
>> > {
>> > System.out.println(file.toString());
>> > }
>> >
>> > I suggest you put together your test class first and make sure it
>> > works, and then go after the captures.
>> >
>> > Let me know how it goes.
>> >
>> > Karl
>> >
>> > On Wed, Dec 7, 2011 at 6:10 AM, Swapna Vuppala
>> > <sw...@gmail.com> wrote:
>> >> Hi Karl,
>> >>
>> >> Thanks for the detailed reply.
>> >>
>> >> As per your suggestion, I tried connecting to the problematic server
>> >> via
>> >> windows using a machine that is not in the domain. On supplying the
>> >> credentials via windows, I could connect to the server, although it did
>> >> not
>> >> list any folders. But when I tried accessing the folder of my interest
>> >> via
>> >> \\server\folder$, I could see all the sub folders. All this is when I
>> >> connected via windows from a machine that's not joined to the domain.
>> >>
>> >> The behavior is same when I connect to the server via windows from a
>> >> machine
>> >> that's joined to the domain.
>> >>
>> >> So I assume this means that through both Kerberos and NTLM, I see the
>> >> same
>> >> behavior. Whereas ManifoldCF throws an exception when trying to define
>> >> the
>> >> repository connection.
>> >>
>> >> Please advise me on what I should be doing next to resolve this.
>> >>
>> >> Thanks and Regards,
>> >> Swapna.
>> >>
>> >>
>> >>
>> >>
>> >> On Tue, Dec 6, 2011 at 12:55 PM, Karl Wright <da...@gmail.com>
>> >> wrote:
>> >>>
>> >>> About your capture - Michael Allen says the following:
>> >>>
>> >>> "Actually this has nothing to do with DFS. JCIFS does not get to the
>> >>> point where it does DFS anything. The capture shows a vanilla
>> >>> STATUS_LOGON_FAILURE when GLOBAL\swapna.vuppala tries to auth with
>> >>> l-carx01.global.arup.com. So the possible causes for this are 1) the
>> >>> account name is not valid 2) the supplied password is incorrect 3)
>> >>> some security policy is deliberately blocking that user or particular
>> >>> type of auth or 4) some server configuration is incompatible with
>> >>> JCIFS. I only mention this last option because I noticed the target
>> >>> server has security signatures disabled. That's strange. If they're
>> >>> messing around with things like that, who knows what their clients are
>> >>> expected to do.
>> >>>
>> >>> Try a Windows client that uses NTLM instead of Kerberos. Meaning try a
>> >>> machine that is not joined to the domain so that when you try to
>> >>> access the target it asks you for credentials at which point you can
>> >>> test with GLOBAL\swapna.vuppala. Then it will use NTLM and you can
>> >>> actually compare captures. If the operator doesn't have a laptop or
>> >>> something not joined to the domain, it might be sufficient to log into
>> >>> a workstation using machine credentials and not domain credentials.
>> >>>
>> >>> Also when testing JCIFS you should use a simple stand-alone program
>> >>> like examples/ListFiles.java."
>> >>>
>> >>> In other words:
>> >>> (a) Since JCIFS does not use Kerberos for authentication, you need to
>> >>> try to log into the recalcitrant server via Windows without using
>> >>> Kerberos to be able to do a side-by-side comparison. Michael has some
>> >>> ways of doing that, above.
>> >>> (b) You may find that it doesn't work, in which case JCIFS is not
>> >>> going to work either.
>> >>> (c) If it *does* work, then try to generate your side-by-side
>> >>> comparisons using a simpler example rather than ManifoldCF en toto;
>> >>> you can see how at jcifs.samba.org, or I can help you further.
>> >>>
>> >>> He also mentions that there is some bizarreness on the response that
>> >>> indicates that the server is configured in a way that he's never seen
>> >>> before. And believe me, Michael has seen a *lot* of strange
>> >>> configurations...
>> >>>
>> >>> Hope this helps.
>> >>> Karl
>> >>>
>> >>> On Mon, Nov 28, 2011 at 4:12 AM, Karl Wright <da...@gmail.com>
>> >>> wrote:
>> >>> > That should read "properties.xml", not "properties.ini". It looks
>> >>> > like this page needs updating.
>> >>> >
>> >>> > The debug property in the XML form is:
>> >>> >
>> >>> > <property name="org.apache.manifoldcf.connectors" value="DEBUG"/>
>> >>> >
>> >>> > I don't think it will provide you with any additional information
>> >>> > that
>> >>> > is useful for debugging your authentication issue, however, if that
>> >>> > is
>> >>> > why you are looking at it. There may be some jcifs.jar debugging
>> >>> > switches that might be of more help, but in the end I suspect you
>> >>> > will
>> >>> > need a packet capture of both a successful connection (via Windows)
>> >>> > and an unsuccessful one (via MCF). The guy you will need to talk
>> >>> > with
>> >>> > after that is the jcifs author Michael Allen; I can give you his
>> >>> > email
>> >>> > address if you get that far.
>> >>> >
>> >>> > Karl
>> >>> >
>> >>> >
>> >>> > On Mon, Nov 28, 2011 at 1:30 AM, Swapna Vuppala
>> >>> > <sw...@gmail.com> wrote:
>> >>> >> Hi Karl,
>> >>> >>
>> >>> >> I was planning to debug jCIFS repository connection using WireShark
>> >>> >> and
>> >>> >> I
>> >>> >> came across this
>> >>> >> https://cwiki.apache.org/CONNECTORS/debugging-connections.html
>> >>> >> Here, I see something as add
>> >>> >> "org.apache.manifoldcf.connectors=DEBUG"
>> >>> >> to the
>> >>> >> properties.ini file. Is it the properties.xml file that is being
>> >>> >> referred
>> >>> >> here ? If not, where do I find properties.ini file ?
>> >>> >>
>> >>> >> Thanks and Regards,
>> >>> >> Swapna.
>> >>> >>
>> >>> >> On Thu, Nov 17, 2011 at 1:31 PM, Karl Wright <da...@gmail.com>
>> >>> >> wrote:
>> >>> >>>
>> >>> >>> See http://jcifs.samba.org/src/docs/api/overview-summary.html#scp.
>> >>> >>> The properties jcifs.smb.lmCompatibility and
>> >>> >>> jcifs.smb.client.useExtendedSecurity are the ones you may want to
>> >>> >>> change. These two properties go together so certain combinations
>> >>> >>> make
>> >>> >>> sense and others don't, so there's really only combinations you
>> >>> >>> need
>> >>> >>> but I'll need to look at what they are and get back to you later
>> >>> >>> today.
>> >>> >>>
>> >>> >>> As far as setting the switches are concerned, if you are using the
>> >>> >>> Quick Start you do this trivially by:
>> >>> >>>
>> >>> >>> <java> -Dxxx -Dyyy -jar start.jar
>> >>> >>>
>> >>> >>> If you are using the multi-process configuration, that is what the
>> >>> >>> "defines" directory is for; you only need to create files in that
>> >>> >>> directory with the names "jcifs.smb.lmCompatibility" and
>> >>> >>> "jcifs.smb.client.useExtendedSecurity" containing the values you
>> >>> >>> want
>> >>> >>> to set.
>> >>> >>>
>> >>> >>> Karl
>> >>> >>>
>> >>> >>>
>> >>> >>> On Thu, Nov 17, 2011 at 1:11 AM, Swapna Vuppala
>> >>> >>> <sw...@gmail.com> wrote:
>> >>> >>> > Hi Karl,
>> >>> >>> >
>> >>> >>> > Am able to access the folders on the problem server through
>> >>> >>> > windows
>> >>> >>> > explorer, (\\server3\Folder1). I tried couple of things with the
>> >>> >>> > credentials
>> >>> >>> > form, changing username, domain etc.. but I keep getting the
>> >>> >>> > same
>> >>> >>> > error
>> >>> >>> > "Couldn't connect to server: Logon failure: unknown user name or
>> >>> >>> > bad
>> >>> >>> > password"
>> >>> >>> >
>> >>> >>> > Can you tell me more about the -D switch you were talking of ?
>> >>> >>> >
>> >>> >>> > Thanks and Regards,
>> >>> >>> > Swapna.
>> >>> >>> >
>> >>> >>> > On Tue, Nov 15, 2011 at 12:40 PM, Karl Wright
>> >>> >>> > <da...@gmail.com>
>> >>> >>> > wrote:
>> >>> >>> >>
>> >>> >>> >> Glad you chased it down this far.
>> >>> >>> >>
>> >>> >>> >> First thing to try is whether you can get into the problem
>> >>> >>> >> server
>> >>> >>> >> using Windows Explorer. Obviously ManifoldCF is not going to
>> >>> >>> >> be
>> >>> >>> >> able
>> >>> >>> >> to do it if Windows can't. If you *can* get in, then just
>> >>> >>> >> playing
>> >>> >>> >> with the form of the credentials in the MCF connection might do
>> >>> >>> >> the
>> >>> >>> >> trick. Some Windows or net appliance servers are picky about
>> >>> >>> >> this.
>> >>> >>> >> Try various things like leaving the domain blank and specifying
>> >>> >>> >> the
>> >>> >>> >> user as "abc@domain.com", for instance. There's also a
>> >>> >>> >> different
>> >>> >>> >> NTLM
>> >>> >>> >> mode you can operation jcifs in that some servers may be
>> >>> >>> >> configured
>> >>> >>> >> to
>> >>> >>> >> require; this would need you to set a -D switch on the command
>> >>> >>> >> line
>> >>> >>> >> to
>> >>> >>> >> enable.
>> >>> >>> >>
>> >>> >>> >> Karl
>> >>> >>> >>
>> >>> >>> >> On Tue, Nov 15, 2011 at 12:10 AM, Swapna Vuppala
>> >>> >>> >> <sw...@gmail.com> wrote:
>> >>> >>> >> > Hi Karl,
>> >>> >>> >> >
>> >>> >>> >> > Thanks for the input. It looks like my problem is related to
>> >>> >>> >> > the
>> >>> >>> >> > second
>> >>> >>> >> > one
>> >>> >>> >> > that you specified. One of the directories in the path am
>> >>> >>> >> > trying
>> >>> >>> >> > to
>> >>> >>> >> > index is
>> >>> >>> >> > actually redirecting to a different server. And when I
>> >>> >>> >> > specify
>> >>> >>> >> > this
>> >>> >>> >> > new
>> >>> >>> >> > server in defining the repository connection, with my
>> >>> >>> >> > credentials,
>> >>> >>> >> > the
>> >>> >>> >> > connection fails with the message: "Couldn't connect to
>> >>> >>> >> > server:
>> >>> >>> >> > Logon
>> >>> >>> >> > failure: unknown user name or bad password"
>> >>> >>> >> >
>> >>> >>> >> > I'll look into why am not able to connect to this server.
>> >>> >>> >> >
>> >>> >>> >> > Thanks and Regards,
>> >>> >>> >> > Swapna.
>> >>> >>> >> >
>> >>> >>> >> > On Mon, Nov 14, 2011 at 4:56 PM, Karl Wright
>> >>> >>> >> > <da...@gmail.com>
>> >>> >>> >> > wrote:
>> >>> >>> >> >>
>> >>> >>> >> >> There's two kinds of problem you might be having. The first
>> >>> >>> >> >> is
>> >>> >>> >> >> intermittent, and the second is not intermittent but would
>> >>> >>> >> >> have
>> >>> >>> >> >> something to do with specific directories.
>> >>> >>> >> >>
>> >>> >>> >> >> Intermittent problems might include a domain controller that
>> >>> >>> >> >> is
>> >>> >>> >> >> not
>> >>> >>> >> >> always accessible. In such cases, the crawl will proceed
>> >>> >>> >> >> but
>> >>> >>> >> >> will
>> >>> >>> >> >> tend to fail unpredictably. On the other hand, if you have
>> >>> >>> >> >> a
>> >>> >>> >> >> directory that is handled by a DFS redirection, it is
>> >>> >>> >> >> possible
>> >>> >>> >> >> that
>> >>> >>> >> >> the redirection is indicating a new server (lets call it
>> >>> >>> >> >> server3)
>> >>> >>> >> >> which may not like the precise form of your login
>> >>> >>> >> >> credentials.
>> >>> >>> >> >> Can
>> >>> >>> >> >> you determine which scenario you are seeing?
>> >>> >>> >> >>
>> >>> >>> >> >> Karl
>> >>> >>> >> >>
>> >>> >>> >> >> On Mon, Nov 14, 2011 at 3:11 AM, Swapna Vuppala
>> >>> >>> >> >> <sw...@gmail.com> wrote:
>> >>> >>> >> >> > Hi,
>> >>> >>> >> >> >
>> >>> >>> >> >> > I have been using windows share repository connection to
>> >>> >>> >> >> > crawl
>> >>> >>> >> >> > and
>> >>> >>> >> >> > get
>> >>> >>> >> >> > data
>> >>> >>> >> >> > from a particular server (server 1). Its working perfectly
>> >>> >>> >> >> > fine.
>> >>> >>> >> >> > However, am
>> >>> >>> >> >> > having trouble when I try with data from another server
>> >>> >>> >> >> > (server
>> >>> >>> >> >> > 2).
>> >>> >>> >> >> >
>> >>> >>> >> >> > When I define a repository connection of type windows
>> >>> >>> >> >> > share
>> >>> >>> >> >> > and
>> >>> >>> >> >> > specify
>> >>> >>> >> >> > the
>> >>> >>> >> >> > server name (server 2) with my credentials, the connection
>> >>> >>> >> >> > status
>> >>> >>> >> >> > shows
>> >>> >>> >> >> > "Connection working". But when I run a job to use this
>> >>> >>> >> >> > repository
>> >>> >>> >> >> > connection
>> >>> >>> >> >> > and index data from a location on this server 2, I keep
>> >>> >>> >> >> > getting
>> >>> >>> >> >> > the
>> >>> >>> >> >> > exception below:
>> >>> >>> >> >> >
>> >>> >>> >> >> > JCIFS: Possibly transient exception detected on attempt 3
>> >>> >>> >> >> > while
>> >>> >>> >> >> > checking
>> >>> >>> >> >> > if
>> >>> >>> >> >> > file exists: Logon failure: unknown user name or bad
>> >>> >>> >> >> > password.
>> >>> >>> >> >> > jcifs.smb.SmbAuthException: Logon failure: unknown user
>> >>> >>> >> >> > name
>> >>> >>> >> >> > or
>> >>> >>> >> >> > bad
>> >>> >>> >> >> > password.
>> >>> >>> >> >> > at
>> >>> >>> >> >> > jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:544)
>> >>> >>> >> >> > at jcifs.smb.SmbTransport.send(SmbTransport.java:661)
>> >>> >>> >> >> > at
>> >>> >>> >> >> > jcifs.smb.SmbSession.sessionSetup(SmbSession.java:390)
>> >>> >>> >> >> > at jcifs.smb.SmbSession.send(SmbSession.java:218)
>> >>> >>> >> >> > at jcifs.smb.SmbTree.treeConnect(SmbTree.java:176)
>> >>> >>> >> >> > at jcifs.smb.SmbFile.doConnect(SmbFile.java:911)
>> >>> >>> >> >> > at jcifs.smb.SmbFile.connect(SmbFile.java:954)
>> >>> >>> >> >> > at jcifs.smb.SmbFile.connect0(SmbFile.java:880)
>> >>> >>> >> >> > at jcifs.smb.SmbFile.queryPath(SmbFile.java:1335)
>> >>> >>> >> >> > at jcifs.smb.SmbFile.exists(SmbFile.java:1417)
>> >>> >>> >> >> > at
>> >>> >>> >> >> >
>> >>> >>> >> >> >
>> >>> >>> >> >> >
>> >>> >>> >> >> >
>> >>> >>> >> >> >
>> >>> >>> >> >> > org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.fileExists(SharedDriveConnector.java:2064)
>> >>> >>> >> >> > at
>> >>> >>> >> >> >
>> >>> >>> >> >> >
>> >>> >>> >> >> >
>> >>> >>> >> >> >
>> >>> >>> >> >> >
>> >>> >>> >> >> > org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getDocumentVersions(SharedDriveConnector.java:521)
>> >>> >>> >> >> > at
>> >>> >>> >> >> >
>> >>> >>> >> >> >
>> >>> >>> >> >> >
>> >>> >>> >> >> >
>> >>> >>> >> >> >
>> >>> >>> >> >> > org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:318)
>> >>> >>> >> >> >
>> >>> >>> >> >> > Am able to access this location from windows explorer.
>> >>> >>> >> >> > What
>> >>> >>> >> >> > else
>> >>> >>> >> >> > should
>> >>> >>> >> >> > I be
>> >>> >>> >> >> > checking or what could be the reasons/factors causing this
>> >>> >>> >> >> > to
>> >>> >>> >> >> > fail
>> >>> >>> >> >> > ?
>> >>> >>> >> >> >
>> >>> >>> >> >> > Thanks and Regards,
>> >>> >>> >> >> > Swapna.
>> >>> >>> >> >> >
>> >>> >>> >> >
>> >>> >>> >> >
>> >>> >>> >
>> >>> >>> >
>> >>> >>
>> >>> >>
>> >>
>> >>
>
>
Re: Problem crawling windows share
Posted by Swapna Vuppala <sw...@gmail.com>.
Hi Karl,
I tried modifying the test code as you suggested.
When I tried with empty credentials
NtlmPasswordAuthentication pa = new NtlmPasswordAuthentication("","","");
SmbFile smbconnection = new SmbFile("smb://" + "lon-data-arx.global.arup.com"
+ "/" + "stg-jobs$",pa);
I got the same exception as before
*Exception in thread "main" jcifs.smb.SmbAuthException: Logon failure:
unknown us
er name or bad password.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:546)
at jcifs.smb.SmbTransport.send(SmbTransport.java:663)
at jcifs.smb.SmbSession.sessionSetup(SmbSession.java:390)
at jcifs.smb.SmbSession.send(SmbSession.java:218)
at jcifs.smb.SmbTree.treeConnect(SmbTree.java:176)
at jcifs.smb.SmbFile.doConnect(SmbFile.java:911)*
......
When I tried with NtlmPasswordAuthentication.ANONYMOUS
SmbFile smbconnection = new SmbFile("smb://" + "lon-data-arx.global.arup.com"
+ "/" + "stg-jobs$",NtlmPasswordAuthentication.ANONYMOUS);
I got a different exception
*Exception in thread "main" jcifs.smb.SmbException: Incorrect function.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:563)
at jcifs.smb.SmbTransport.send(SmbTransport.java:663)
at jcifs.smb.SmbSession.sessionSetup(SmbSession.java:316)
at jcifs.smb.SmbSession.send(SmbSession.java:218)
at jcifs.smb.SmbTree.treeConnect(SmbTree.java:176)
at jcifs.smb.SmbFile.doConnect(SmbFile.java:911)
.....
*Thanks and Regards,
Swapna.* *
On Fri, Dec 9, 2011 at 11:46 PM, Karl Wright <da...@gmail.com> wrote:
> Michael Allen's response to your capture:
>
> "It looks like the non-JCIFS communication is successful because the
> anonymous credential is used.
>
> Note that there is still no sign of DFS going on anywhere. Actually
> JCIFS tries to do a referral for the DFS root and gets an error of
> STATUS_FS_DRIVER_REQUIRED so it looks like DFS may not even be setup
> at all in the target environment. At least not domain based DFS.
>
> So if you want to emulate Windows behavior, I guess you need to retry
> the operation with credential NtlmPasswordAuthentication.ANONYMOUS (or
> all empty strings should work too I think like ";:")."
>
> In other words, the issue is that the server in question does not
> actually recognize the credentials given at all, and gives an error.
> But while Windows then retries with no credentials whatsoever, nothing
> in JCIFS does that retry.
>
> You can confirm that this is what is happening by using an empty user
> name, password, and domain in the JCIFS test code you wrote to obtain
> the captures. If, then, the test code works against that server, I'll
> create a ticket for downgrading to ANONYMOUS whenever there is an
> authentication error in the JCIFS connector.
>
> Thanks,
> Karl
>
> On Wed, Dec 7, 2011 at 6:41 AM, Karl Wright <da...@gmail.com> wrote:
> > The next step is to obtain two packet captures using WireShark. These
> > should both be done from the machine that is not joined to the domain.
> > The first capture should be the interaction that takes place when you
> > try to go to the foreign directory, and you supply the credentials,
> > and then you list the folders. The second capture will require you to
> > do the same using JCIFS. For that capture, you will need jcifs.jar
> > and a simple example class that provides jcifs with credentials and
> > tries to list the directory. A simple class should be enough to do
> > this, including code such as:
> >
> > import jcifs.smb.*;
> >
> > NtlmPasswordAuthentication pa = new
> > NtlmPasswordAuthentication(domain,username,password);
> > SmbFile smbconnection = new SmbFile("smb://" + server + "/" + share,pa);
> > SmbFile[] files = smbconnection.listFiles();
> >
> > for (int i = 0 ; i < files.length ; i++)
> > {
> > System.out.println(file.toString());
> > }
> >
> > I suggest you put together your test class first and make sure it
> > works, and then go after the captures.
> >
> > Let me know how it goes.
> >
> > Karl
> >
> > On Wed, Dec 7, 2011 at 6:10 AM, Swapna Vuppala
> > <sw...@gmail.com> wrote:
> >> Hi Karl,
> >>
> >> Thanks for the detailed reply.
> >>
> >> As per your suggestion, I tried connecting to the problematic server via
> >> windows using a machine that is not in the domain. On supplying the
> >> credentials via windows, I could connect to the server, although it did
> not
> >> list any folders. But when I tried accessing the folder of my interest
> via
> >> \\server\folder$, I could see all the sub folders. All this is when I
> >> connected via windows from a machine that's not joined to the domain.
> >>
> >> The behavior is same when I connect to the server via windows from a
> machine
> >> that's joined to the domain.
> >>
> >> So I assume this means that through both Kerberos and NTLM, I see the
> same
> >> behavior. Whereas ManifoldCF throws an exception when trying to define
> the
> >> repository connection.
> >>
> >> Please advise me on what I should be doing next to resolve this.
> >>
> >> Thanks and Regards,
> >> Swapna.
> >>
> >>
> >>
> >>
> >> On Tue, Dec 6, 2011 at 12:55 PM, Karl Wright <da...@gmail.com>
> wrote:
> >>>
> >>> About your capture - Michael Allen says the following:
> >>>
> >>> "Actually this has nothing to do with DFS. JCIFS does not get to the
> >>> point where it does DFS anything. The capture shows a vanilla
> >>> STATUS_LOGON_FAILURE when GLOBAL\swapna.vuppala tries to auth with
> >>> l-carx01.global.arup.com. So the possible causes for this are 1) the
> >>> account name is not valid 2) the supplied password is incorrect 3)
> >>> some security policy is deliberately blocking that user or particular
> >>> type of auth or 4) some server configuration is incompatible with
> >>> JCIFS. I only mention this last option because I noticed the target
> >>> server has security signatures disabled. That's strange. If they're
> >>> messing around with things like that, who knows what their clients are
> >>> expected to do.
> >>>
> >>> Try a Windows client that uses NTLM instead of Kerberos. Meaning try a
> >>> machine that is not joined to the domain so that when you try to
> >>> access the target it asks you for credentials at which point you can
> >>> test with GLOBAL\swapna.vuppala. Then it will use NTLM and you can
> >>> actually compare captures. If the operator doesn't have a laptop or
> >>> something not joined to the domain, it might be sufficient to log into
> >>> a workstation using machine credentials and not domain credentials.
> >>>
> >>> Also when testing JCIFS you should use a simple stand-alone program
> >>> like examples/ListFiles.java."
> >>>
> >>> In other words:
> >>> (a) Since JCIFS does not use Kerberos for authentication, you need to
> >>> try to log into the recalcitrant server via Windows without using
> >>> Kerberos to be able to do a side-by-side comparison. Michael has some
> >>> ways of doing that, above.
> >>> (b) You may find that it doesn't work, in which case JCIFS is not
> >>> going to work either.
> >>> (c) If it *does* work, then try to generate your side-by-side
> >>> comparisons using a simpler example rather than ManifoldCF en toto;
> >>> you can see how at jcifs.samba.org, or I can help you further.
> >>>
> >>> He also mentions that there is some bizarreness on the response that
> >>> indicates that the server is configured in a way that he's never seen
> >>> before. And believe me, Michael has seen a *lot* of strange
> >>> configurations...
> >>>
> >>> Hope this helps.
> >>> Karl
> >>>
> >>> On Mon, Nov 28, 2011 at 4:12 AM, Karl Wright <da...@gmail.com>
> wrote:
> >>> > That should read "properties.xml", not "properties.ini". It looks
> >>> > like this page needs updating.
> >>> >
> >>> > The debug property in the XML form is:
> >>> >
> >>> > <property name="org.apache.manifoldcf.connectors" value="DEBUG"/>
> >>> >
> >>> > I don't think it will provide you with any additional information
> that
> >>> > is useful for debugging your authentication issue, however, if that
> is
> >>> > why you are looking at it. There may be some jcifs.jar debugging
> >>> > switches that might be of more help, but in the end I suspect you
> will
> >>> > need a packet capture of both a successful connection (via Windows)
> >>> > and an unsuccessful one (via MCF). The guy you will need to talk
> with
> >>> > after that is the jcifs author Michael Allen; I can give you his
> email
> >>> > address if you get that far.
> >>> >
> >>> > Karl
> >>> >
> >>> >
> >>> > On Mon, Nov 28, 2011 at 1:30 AM, Swapna Vuppala
> >>> > <sw...@gmail.com> wrote:
> >>> >> Hi Karl,
> >>> >>
> >>> >> I was planning to debug jCIFS repository connection using WireShark
> and
> >>> >> I
> >>> >> came across this
> >>> >> https://cwiki.apache.org/CONNECTORS/debugging-connections.html
> >>> >> Here, I see something as add
> "org.apache.manifoldcf.connectors=DEBUG"
> >>> >> to the
> >>> >> properties.ini file. Is it the properties.xml file that is being
> >>> >> referred
> >>> >> here ? If not, where do I find properties.ini file ?
> >>> >>
> >>> >> Thanks and Regards,
> >>> >> Swapna.
> >>> >>
> >>> >> On Thu, Nov 17, 2011 at 1:31 PM, Karl Wright <da...@gmail.com>
> >>> >> wrote:
> >>> >>>
> >>> >>> See http://jcifs.samba.org/src/docs/api/overview-summary.html#scp.
> >>> >>> The properties jcifs.smb.lmCompatibility and
> >>> >>> jcifs.smb.client.useExtendedSecurity are the ones you may want to
> >>> >>> change. These two properties go together so certain combinations
> make
> >>> >>> sense and others don't, so there's really only combinations you
> need
> >>> >>> but I'll need to look at what they are and get back to you later
> >>> >>> today.
> >>> >>>
> >>> >>> As far as setting the switches are concerned, if you are using the
> >>> >>> Quick Start you do this trivially by:
> >>> >>>
> >>> >>> <java> -Dxxx -Dyyy -jar start.jar
> >>> >>>
> >>> >>> If you are using the multi-process configuration, that is what the
> >>> >>> "defines" directory is for; you only need to create files in that
> >>> >>> directory with the names "jcifs.smb.lmCompatibility" and
> >>> >>> "jcifs.smb.client.useExtendedSecurity" containing the values you
> want
> >>> >>> to set.
> >>> >>>
> >>> >>> Karl
> >>> >>>
> >>> >>>
> >>> >>> On Thu, Nov 17, 2011 at 1:11 AM, Swapna Vuppala
> >>> >>> <sw...@gmail.com> wrote:
> >>> >>> > Hi Karl,
> >>> >>> >
> >>> >>> > Am able to access the folders on the problem server through
> windows
> >>> >>> > explorer, (\\server3\Folder1). I tried couple of things with the
> >>> >>> > credentials
> >>> >>> > form, changing username, domain etc.. but I keep getting the same
> >>> >>> > error
> >>> >>> > "Couldn't connect to server: Logon failure: unknown user name or
> bad
> >>> >>> > password"
> >>> >>> >
> >>> >>> > Can you tell me more about the -D switch you were talking of ?
> >>> >>> >
> >>> >>> > Thanks and Regards,
> >>> >>> > Swapna.
> >>> >>> >
> >>> >>> > On Tue, Nov 15, 2011 at 12:40 PM, Karl Wright <
> daddywri@gmail.com>
> >>> >>> > wrote:
> >>> >>> >>
> >>> >>> >> Glad you chased it down this far.
> >>> >>> >>
> >>> >>> >> First thing to try is whether you can get into the problem
> server
> >>> >>> >> using Windows Explorer. Obviously ManifoldCF is not going to be
> >>> >>> >> able
> >>> >>> >> to do it if Windows can't. If you *can* get in, then just
> playing
> >>> >>> >> with the form of the credentials in the MCF connection might do
> the
> >>> >>> >> trick. Some Windows or net appliance servers are picky about
> this.
> >>> >>> >> Try various things like leaving the domain blank and specifying
> the
> >>> >>> >> user as "abc@domain.com", for instance. There's also a
> different
> >>> >>> >> NTLM
> >>> >>> >> mode you can operation jcifs in that some servers may be
> configured
> >>> >>> >> to
> >>> >>> >> require; this would need you to set a -D switch on the command
> line
> >>> >>> >> to
> >>> >>> >> enable.
> >>> >>> >>
> >>> >>> >> Karl
> >>> >>> >>
> >>> >>> >> On Tue, Nov 15, 2011 at 12:10 AM, Swapna Vuppala
> >>> >>> >> <sw...@gmail.com> wrote:
> >>> >>> >> > Hi Karl,
> >>> >>> >> >
> >>> >>> >> > Thanks for the input. It looks like my problem is related to
> the
> >>> >>> >> > second
> >>> >>> >> > one
> >>> >>> >> > that you specified. One of the directories in the path am
> trying
> >>> >>> >> > to
> >>> >>> >> > index is
> >>> >>> >> > actually redirecting to a different server. And when I specify
> >>> >>> >> > this
> >>> >>> >> > new
> >>> >>> >> > server in defining the repository connection, with my
> >>> >>> >> > credentials,
> >>> >>> >> > the
> >>> >>> >> > connection fails with the message: "Couldn't connect to
> server:
> >>> >>> >> > Logon
> >>> >>> >> > failure: unknown user name or bad password"
> >>> >>> >> >
> >>> >>> >> > I'll look into why am not able to connect to this server.
> >>> >>> >> >
> >>> >>> >> > Thanks and Regards,
> >>> >>> >> > Swapna.
> >>> >>> >> >
> >>> >>> >> > On Mon, Nov 14, 2011 at 4:56 PM, Karl Wright <
> daddywri@gmail.com>
> >>> >>> >> > wrote:
> >>> >>> >> >>
> >>> >>> >> >> There's two kinds of problem you might be having. The first
> is
> >>> >>> >> >> intermittent, and the second is not intermittent but would
> have
> >>> >>> >> >> something to do with specific directories.
> >>> >>> >> >>
> >>> >>> >> >> Intermittent problems might include a domain controller that
> is
> >>> >>> >> >> not
> >>> >>> >> >> always accessible. In such cases, the crawl will proceed but
> >>> >>> >> >> will
> >>> >>> >> >> tend to fail unpredictably. On the other hand, if you have a
> >>> >>> >> >> directory that is handled by a DFS redirection, it is
> possible
> >>> >>> >> >> that
> >>> >>> >> >> the redirection is indicating a new server (lets call it
> >>> >>> >> >> server3)
> >>> >>> >> >> which may not like the precise form of your login
> credentials.
> >>> >>> >> >> Can
> >>> >>> >> >> you determine which scenario you are seeing?
> >>> >>> >> >>
> >>> >>> >> >> Karl
> >>> >>> >> >>
> >>> >>> >> >> On Mon, Nov 14, 2011 at 3:11 AM, Swapna Vuppala
> >>> >>> >> >> <sw...@gmail.com> wrote:
> >>> >>> >> >> > Hi,
> >>> >>> >> >> >
> >>> >>> >> >> > I have been using windows share repository connection to
> crawl
> >>> >>> >> >> > and
> >>> >>> >> >> > get
> >>> >>> >> >> > data
> >>> >>> >> >> > from a particular server (server 1). Its working perfectly
> >>> >>> >> >> > fine.
> >>> >>> >> >> > However, am
> >>> >>> >> >> > having trouble when I try with data from another server
> >>> >>> >> >> > (server
> >>> >>> >> >> > 2).
> >>> >>> >> >> >
> >>> >>> >> >> > When I define a repository connection of type windows share
> >>> >>> >> >> > and
> >>> >>> >> >> > specify
> >>> >>> >> >> > the
> >>> >>> >> >> > server name (server 2) with my credentials, the connection
> >>> >>> >> >> > status
> >>> >>> >> >> > shows
> >>> >>> >> >> > "Connection working". But when I run a job to use this
> >>> >>> >> >> > repository
> >>> >>> >> >> > connection
> >>> >>> >> >> > and index data from a location on this server 2, I keep
> >>> >>> >> >> > getting
> >>> >>> >> >> > the
> >>> >>> >> >> > exception below:
> >>> >>> >> >> >
> >>> >>> >> >> > JCIFS: Possibly transient exception detected on attempt 3
> >>> >>> >> >> > while
> >>> >>> >> >> > checking
> >>> >>> >> >> > if
> >>> >>> >> >> > file exists: Logon failure: unknown user name or bad
> password.
> >>> >>> >> >> > jcifs.smb.SmbAuthException: Logon failure: unknown user
> name
> >>> >>> >> >> > or
> >>> >>> >> >> > bad
> >>> >>> >> >> > password.
> >>> >>> >> >> > at
> >>> >>> >> >> > jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:544)
> >>> >>> >> >> > at jcifs.smb.SmbTransport.send(SmbTransport.java:661)
> >>> >>> >> >> > at
> jcifs.smb.SmbSession.sessionSetup(SmbSession.java:390)
> >>> >>> >> >> > at jcifs.smb.SmbSession.send(SmbSession.java:218)
> >>> >>> >> >> > at jcifs.smb.SmbTree.treeConnect(SmbTree.java:176)
> >>> >>> >> >> > at jcifs.smb.SmbFile.doConnect(SmbFile.java:911)
> >>> >>> >> >> > at jcifs.smb.SmbFile.connect(SmbFile.java:954)
> >>> >>> >> >> > at jcifs.smb.SmbFile.connect0(SmbFile.java:880)
> >>> >>> >> >> > at jcifs.smb.SmbFile.queryPath(SmbFile.java:1335)
> >>> >>> >> >> > at jcifs.smb.SmbFile.exists(SmbFile.java:1417)
> >>> >>> >> >> > at
> >>> >>> >> >> >
> >>> >>> >> >> >
> >>> >>> >> >> >
> >>> >>> >> >> >
> >>> >>> >> >> >
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.fileExists(SharedDriveConnector.java:2064)
> >>> >>> >> >> > at
> >>> >>> >> >> >
> >>> >>> >> >> >
> >>> >>> >> >> >
> >>> >>> >> >> >
> >>> >>> >> >> >
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getDocumentVersions(SharedDriveConnector.java:521)
> >>> >>> >> >> > at
> >>> >>> >> >> >
> >>> >>> >> >> >
> >>> >>> >> >> >
> >>> >>> >> >> >
> >>> >>> >> >> >
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:318)
> >>> >>> >> >> >
> >>> >>> >> >> > Am able to access this location from windows explorer. What
> >>> >>> >> >> > else
> >>> >>> >> >> > should
> >>> >>> >> >> > I be
> >>> >>> >> >> > checking or what could be the reasons/factors causing this
> to
> >>> >>> >> >> > fail
> >>> >>> >> >> > ?
> >>> >>> >> >> >
> >>> >>> >> >> > Thanks and Regards,
> >>> >>> >> >> > Swapna.
> >>> >>> >> >> >
> >>> >>> >> >
> >>> >>> >> >
> >>> >>> >
> >>> >>> >
> >>> >>
> >>> >>
> >>
> >>
>
Re: Problem crawling windows share
Posted by Karl Wright <da...@gmail.com>.
Michael Allen's response to your capture:
"It looks like the non-JCIFS communication is successful because the
anonymous credential is used.
Note that there is still no sign of DFS going on anywhere. Actually
JCIFS tries to do a referral for the DFS root and gets an error of
STATUS_FS_DRIVER_REQUIRED so it looks like DFS may not even be setup
at all in the target environment. At least not domain based DFS.
So if you want to emulate Windows behavior, I guess you need to retry
the operation with credential NtlmPasswordAuthentication.ANONYMOUS (or
all empty strings should work too I think like ";:")."
In other words, the issue is that the server in question does not
actually recognize the credentials given at all, and gives an error.
But while Windows then retries with no credentials whatsoever, nothing
in JCIFS does that retry.
You can confirm that this is what is happening by using an empty user
name, password, and domain in the JCIFS test code you wrote to obtain
the captures. If, then, the test code works against that server, I'll
create a ticket for downgrading to ANONYMOUS whenever there is an
authentication error in the JCIFS connector.
Thanks,
Karl
On Wed, Dec 7, 2011 at 6:41 AM, Karl Wright <da...@gmail.com> wrote:
> The next step is to obtain two packet captures using WireShark. These
> should both be done from the machine that is not joined to the domain.
> The first capture should be the interaction that takes place when you
> try to go to the foreign directory, and you supply the credentials,
> and then you list the folders. The second capture will require you to
> do the same using JCIFS. For that capture, you will need jcifs.jar
> and a simple example class that provides jcifs with credentials and
> tries to list the directory. A simple class should be enough to do
> this, including code such as:
>
> import jcifs.smb.*;
>
> NtlmPasswordAuthentication pa = new
> NtlmPasswordAuthentication(domain,username,password);
> SmbFile smbconnection = new SmbFile("smb://" + server + "/" + share,pa);
> SmbFile[] files = smbconnection.listFiles();
>
> for (int i = 0 ; i < files.length ; i++)
> {
> System.out.println(file.toString());
> }
>
> I suggest you put together your test class first and make sure it
> works, and then go after the captures.
>
> Let me know how it goes.
>
> Karl
>
> On Wed, Dec 7, 2011 at 6:10 AM, Swapna Vuppala
> <sw...@gmail.com> wrote:
>> Hi Karl,
>>
>> Thanks for the detailed reply.
>>
>> As per your suggestion, I tried connecting to the problematic server via
>> windows using a machine that is not in the domain. On supplying the
>> credentials via windows, I could connect to the server, although it did not
>> list any folders. But when I tried accessing the folder of my interest via
>> \\server\folder$, I could see all the sub folders. All this is when I
>> connected via windows from a machine that's not joined to the domain.
>>
>> The behavior is same when I connect to the server via windows from a machine
>> that's joined to the domain.
>>
>> So I assume this means that through both Kerberos and NTLM, I see the same
>> behavior. Whereas ManifoldCF throws an exception when trying to define the
>> repository connection.
>>
>> Please advise me on what I should be doing next to resolve this.
>>
>> Thanks and Regards,
>> Swapna.
>>
>>
>>
>>
>> On Tue, Dec 6, 2011 at 12:55 PM, Karl Wright <da...@gmail.com> wrote:
>>>
>>> About your capture - Michael Allen says the following:
>>>
>>> "Actually this has nothing to do with DFS. JCIFS does not get to the
>>> point where it does DFS anything. The capture shows a vanilla
>>> STATUS_LOGON_FAILURE when GLOBAL\swapna.vuppala tries to auth with
>>> l-carx01.global.arup.com. So the possible causes for this are 1) the
>>> account name is not valid 2) the supplied password is incorrect 3)
>>> some security policy is deliberately blocking that user or particular
>>> type of auth or 4) some server configuration is incompatible with
>>> JCIFS. I only mention this last option because I noticed the target
>>> server has security signatures disabled. That's strange. If they're
>>> messing around with things like that, who knows what their clients are
>>> expected to do.
>>>
>>> Try a Windows client that uses NTLM instead of Kerberos. Meaning try a
>>> machine that is not joined to the domain so that when you try to
>>> access the target it asks you for credentials at which point you can
>>> test with GLOBAL\swapna.vuppala. Then it will use NTLM and you can
>>> actually compare captures. If the operator doesn't have a laptop or
>>> something not joined to the domain, it might be sufficient to log into
>>> a workstation using machine credentials and not domain credentials.
>>>
>>> Also when testing JCIFS you should use a simple stand-alone program
>>> like examples/ListFiles.java."
>>>
>>> In other words:
>>> (a) Since JCIFS does not use Kerberos for authentication, you need to
>>> try to log into the recalcitrant server via Windows without using
>>> Kerberos to be able to do a side-by-side comparison. Michael has some
>>> ways of doing that, above.
>>> (b) You may find that it doesn't work, in which case JCIFS is not
>>> going to work either.
>>> (c) If it *does* work, then try to generate your side-by-side
>>> comparisons using a simpler example rather than ManifoldCF en toto;
>>> you can see how at jcifs.samba.org, or I can help you further.
>>>
>>> He also mentions that there is some bizarreness on the response that
>>> indicates that the server is configured in a way that he's never seen
>>> before. And believe me, Michael has seen a *lot* of strange
>>> configurations...
>>>
>>> Hope this helps.
>>> Karl
>>>
>>> On Mon, Nov 28, 2011 at 4:12 AM, Karl Wright <da...@gmail.com> wrote:
>>> > That should read "properties.xml", not "properties.ini". It looks
>>> > like this page needs updating.
>>> >
>>> > The debug property in the XML form is:
>>> >
>>> > <property name="org.apache.manifoldcf.connectors" value="DEBUG"/>
>>> >
>>> > I don't think it will provide you with any additional information that
>>> > is useful for debugging your authentication issue, however, if that is
>>> > why you are looking at it. There may be some jcifs.jar debugging
>>> > switches that might be of more help, but in the end I suspect you will
>>> > need a packet capture of both a successful connection (via Windows)
>>> > and an unsuccessful one (via MCF). The guy you will need to talk with
>>> > after that is the jcifs author Michael Allen; I can give you his email
>>> > address if you get that far.
>>> >
>>> > Karl
>>> >
>>> >
>>> > On Mon, Nov 28, 2011 at 1:30 AM, Swapna Vuppala
>>> > <sw...@gmail.com> wrote:
>>> >> Hi Karl,
>>> >>
>>> >> I was planning to debug jCIFS repository connection using WireShark and
>>> >> I
>>> >> came across this
>>> >> https://cwiki.apache.org/CONNECTORS/debugging-connections.html
>>> >> Here, I see something as add "org.apache.manifoldcf.connectors=DEBUG"
>>> >> to the
>>> >> properties.ini file. Is it the properties.xml file that is being
>>> >> referred
>>> >> here ? If not, where do I find properties.ini file ?
>>> >>
>>> >> Thanks and Regards,
>>> >> Swapna.
>>> >>
>>> >> On Thu, Nov 17, 2011 at 1:31 PM, Karl Wright <da...@gmail.com>
>>> >> wrote:
>>> >>>
>>> >>> See http://jcifs.samba.org/src/docs/api/overview-summary.html#scp.
>>> >>> The properties jcifs.smb.lmCompatibility and
>>> >>> jcifs.smb.client.useExtendedSecurity are the ones you may want to
>>> >>> change. These two properties go together so certain combinations make
>>> >>> sense and others don't, so there's really only combinations you need
>>> >>> but I'll need to look at what they are and get back to you later
>>> >>> today.
>>> >>>
>>> >>> As far as setting the switches are concerned, if you are using the
>>> >>> Quick Start you do this trivially by:
>>> >>>
>>> >>> <java> -Dxxx -Dyyy -jar start.jar
>>> >>>
>>> >>> If you are using the multi-process configuration, that is what the
>>> >>> "defines" directory is for; you only need to create files in that
>>> >>> directory with the names "jcifs.smb.lmCompatibility" and
>>> >>> "jcifs.smb.client.useExtendedSecurity" containing the values you want
>>> >>> to set.
>>> >>>
>>> >>> Karl
>>> >>>
>>> >>>
>>> >>> On Thu, Nov 17, 2011 at 1:11 AM, Swapna Vuppala
>>> >>> <sw...@gmail.com> wrote:
>>> >>> > Hi Karl,
>>> >>> >
>>> >>> > Am able to access the folders on the problem server through windows
>>> >>> > explorer, (\\server3\Folder1). I tried couple of things with the
>>> >>> > credentials
>>> >>> > form, changing username, domain etc.. but I keep getting the same
>>> >>> > error
>>> >>> > "Couldn't connect to server: Logon failure: unknown user name or bad
>>> >>> > password"
>>> >>> >
>>> >>> > Can you tell me more about the -D switch you were talking of ?
>>> >>> >
>>> >>> > Thanks and Regards,
>>> >>> > Swapna.
>>> >>> >
>>> >>> > On Tue, Nov 15, 2011 at 12:40 PM, Karl Wright <da...@gmail.com>
>>> >>> > wrote:
>>> >>> >>
>>> >>> >> Glad you chased it down this far.
>>> >>> >>
>>> >>> >> First thing to try is whether you can get into the problem server
>>> >>> >> using Windows Explorer. Obviously ManifoldCF is not going to be
>>> >>> >> able
>>> >>> >> to do it if Windows can't. If you *can* get in, then just playing
>>> >>> >> with the form of the credentials in the MCF connection might do the
>>> >>> >> trick. Some Windows or net appliance servers are picky about this.
>>> >>> >> Try various things like leaving the domain blank and specifying the
>>> >>> >> user as "abc@domain.com", for instance. There's also a different
>>> >>> >> NTLM
>>> >>> >> mode you can operation jcifs in that some servers may be configured
>>> >>> >> to
>>> >>> >> require; this would need you to set a -D switch on the command line
>>> >>> >> to
>>> >>> >> enable.
>>> >>> >>
>>> >>> >> Karl
>>> >>> >>
>>> >>> >> On Tue, Nov 15, 2011 at 12:10 AM, Swapna Vuppala
>>> >>> >> <sw...@gmail.com> wrote:
>>> >>> >> > Hi Karl,
>>> >>> >> >
>>> >>> >> > Thanks for the input. It looks like my problem is related to the
>>> >>> >> > second
>>> >>> >> > one
>>> >>> >> > that you specified. One of the directories in the path am trying
>>> >>> >> > to
>>> >>> >> > index is
>>> >>> >> > actually redirecting to a different server. And when I specify
>>> >>> >> > this
>>> >>> >> > new
>>> >>> >> > server in defining the repository connection, with my
>>> >>> >> > credentials,
>>> >>> >> > the
>>> >>> >> > connection fails with the message: "Couldn't connect to server:
>>> >>> >> > Logon
>>> >>> >> > failure: unknown user name or bad password"
>>> >>> >> >
>>> >>> >> > I'll look into why am not able to connect to this server.
>>> >>> >> >
>>> >>> >> > Thanks and Regards,
>>> >>> >> > Swapna.
>>> >>> >> >
>>> >>> >> > On Mon, Nov 14, 2011 at 4:56 PM, Karl Wright <da...@gmail.com>
>>> >>> >> > wrote:
>>> >>> >> >>
>>> >>> >> >> There's two kinds of problem you might be having. The first is
>>> >>> >> >> intermittent, and the second is not intermittent but would have
>>> >>> >> >> something to do with specific directories.
>>> >>> >> >>
>>> >>> >> >> Intermittent problems might include a domain controller that is
>>> >>> >> >> not
>>> >>> >> >> always accessible. In such cases, the crawl will proceed but
>>> >>> >> >> will
>>> >>> >> >> tend to fail unpredictably. On the other hand, if you have a
>>> >>> >> >> directory that is handled by a DFS redirection, it is possible
>>> >>> >> >> that
>>> >>> >> >> the redirection is indicating a new server (lets call it
>>> >>> >> >> server3)
>>> >>> >> >> which may not like the precise form of your login credentials.
>>> >>> >> >> Can
>>> >>> >> >> you determine which scenario you are seeing?
>>> >>> >> >>
>>> >>> >> >> Karl
>>> >>> >> >>
>>> >>> >> >> On Mon, Nov 14, 2011 at 3:11 AM, Swapna Vuppala
>>> >>> >> >> <sw...@gmail.com> wrote:
>>> >>> >> >> > Hi,
>>> >>> >> >> >
>>> >>> >> >> > I have been using windows share repository connection to crawl
>>> >>> >> >> > and
>>> >>> >> >> > get
>>> >>> >> >> > data
>>> >>> >> >> > from a particular server (server 1). Its working perfectly
>>> >>> >> >> > fine.
>>> >>> >> >> > However, am
>>> >>> >> >> > having trouble when I try with data from another server
>>> >>> >> >> > (server
>>> >>> >> >> > 2).
>>> >>> >> >> >
>>> >>> >> >> > When I define a repository connection of type windows share
>>> >>> >> >> > and
>>> >>> >> >> > specify
>>> >>> >> >> > the
>>> >>> >> >> > server name (server 2) with my credentials, the connection
>>> >>> >> >> > status
>>> >>> >> >> > shows
>>> >>> >> >> > "Connection working". But when I run a job to use this
>>> >>> >> >> > repository
>>> >>> >> >> > connection
>>> >>> >> >> > and index data from a location on this server 2, I keep
>>> >>> >> >> > getting
>>> >>> >> >> > the
>>> >>> >> >> > exception below:
>>> >>> >> >> >
>>> >>> >> >> > JCIFS: Possibly transient exception detected on attempt 3
>>> >>> >> >> > while
>>> >>> >> >> > checking
>>> >>> >> >> > if
>>> >>> >> >> > file exists: Logon failure: unknown user name or bad password.
>>> >>> >> >> > jcifs.smb.SmbAuthException: Logon failure: unknown user name
>>> >>> >> >> > or
>>> >>> >> >> > bad
>>> >>> >> >> > password.
>>> >>> >> >> > at
>>> >>> >> >> > jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:544)
>>> >>> >> >> > at jcifs.smb.SmbTransport.send(SmbTransport.java:661)
>>> >>> >> >> > at jcifs.smb.SmbSession.sessionSetup(SmbSession.java:390)
>>> >>> >> >> > at jcifs.smb.SmbSession.send(SmbSession.java:218)
>>> >>> >> >> > at jcifs.smb.SmbTree.treeConnect(SmbTree.java:176)
>>> >>> >> >> > at jcifs.smb.SmbFile.doConnect(SmbFile.java:911)
>>> >>> >> >> > at jcifs.smb.SmbFile.connect(SmbFile.java:954)
>>> >>> >> >> > at jcifs.smb.SmbFile.connect0(SmbFile.java:880)
>>> >>> >> >> > at jcifs.smb.SmbFile.queryPath(SmbFile.java:1335)
>>> >>> >> >> > at jcifs.smb.SmbFile.exists(SmbFile.java:1417)
>>> >>> >> >> > at
>>> >>> >> >> >
>>> >>> >> >> >
>>> >>> >> >> >
>>> >>> >> >> >
>>> >>> >> >> > org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.fileExists(SharedDriveConnector.java:2064)
>>> >>> >> >> > at
>>> >>> >> >> >
>>> >>> >> >> >
>>> >>> >> >> >
>>> >>> >> >> >
>>> >>> >> >> > org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getDocumentVersions(SharedDriveConnector.java:521)
>>> >>> >> >> > at
>>> >>> >> >> >
>>> >>> >> >> >
>>> >>> >> >> >
>>> >>> >> >> >
>>> >>> >> >> > org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:318)
>>> >>> >> >> >
>>> >>> >> >> > Am able to access this location from windows explorer. What
>>> >>> >> >> > else
>>> >>> >> >> > should
>>> >>> >> >> > I be
>>> >>> >> >> > checking or what could be the reasons/factors causing this to
>>> >>> >> >> > fail
>>> >>> >> >> > ?
>>> >>> >> >> >
>>> >>> >> >> > Thanks and Regards,
>>> >>> >> >> > Swapna.
>>> >>> >> >> >
>>> >>> >> >
>>> >>> >> >
>>> >>> >
>>> >>> >
>>> >>
>>> >>
>>
>>
Re: Problem crawling windows share
Posted by Karl Wright <da...@gmail.com>.
The next step is to obtain two packet captures using WireShark. These
should both be done from the machine that is not joined to the domain.
The first capture should be the interaction that takes place when you
try to go to the foreign directory, and you supply the credentials,
and then you list the folders. The second capture will require you to
do the same using JCIFS. For that capture, you will need jcifs.jar
and a simple example class that provides jcifs with credentials and
tries to list the directory. A simple class should be enough to do
this, including code such as:
import jcifs.smb.*;
NtlmPasswordAuthentication pa = new
NtlmPasswordAuthentication(domain,username,password);
SmbFile smbconnection = new SmbFile("smb://" + server + "/" + share,pa);
SmbFile[] files = smbconnection.listFiles();
for (int i = 0 ; i < files.length ; i++)
{
System.out.println(file.toString());
}
I suggest you put together your test class first and make sure it
works, and then go after the captures.
Let me know how it goes.
Karl
On Wed, Dec 7, 2011 at 6:10 AM, Swapna Vuppala
<sw...@gmail.com> wrote:
> Hi Karl,
>
> Thanks for the detailed reply.
>
> As per your suggestion, I tried connecting to the problematic server via
> windows using a machine that is not in the domain. On supplying the
> credentials via windows, I could connect to the server, although it did not
> list any folders. But when I tried accessing the folder of my interest via
> \\server\folder$, I could see all the sub folders. All this is when I
> connected via windows from a machine that's not joined to the domain.
>
> The behavior is same when I connect to the server via windows from a machine
> that's joined to the domain.
>
> So I assume this means that through both Kerberos and NTLM, I see the same
> behavior. Whereas ManifoldCF throws an exception when trying to define the
> repository connection.
>
> Please advise me on what I should be doing next to resolve this.
>
> Thanks and Regards,
> Swapna.
>
>
>
>
> On Tue, Dec 6, 2011 at 12:55 PM, Karl Wright <da...@gmail.com> wrote:
>>
>> About your capture - Michael Allen says the following:
>>
>> "Actually this has nothing to do with DFS. JCIFS does not get to the
>> point where it does DFS anything. The capture shows a vanilla
>> STATUS_LOGON_FAILURE when GLOBAL\swapna.vuppala tries to auth with
>> l-carx01.global.arup.com. So the possible causes for this are 1) the
>> account name is not valid 2) the supplied password is incorrect 3)
>> some security policy is deliberately blocking that user or particular
>> type of auth or 4) some server configuration is incompatible with
>> JCIFS. I only mention this last option because I noticed the target
>> server has security signatures disabled. That's strange. If they're
>> messing around with things like that, who knows what their clients are
>> expected to do.
>>
>> Try a Windows client that uses NTLM instead of Kerberos. Meaning try a
>> machine that is not joined to the domain so that when you try to
>> access the target it asks you for credentials at which point you can
>> test with GLOBAL\swapna.vuppala. Then it will use NTLM and you can
>> actually compare captures. If the operator doesn't have a laptop or
>> something not joined to the domain, it might be sufficient to log into
>> a workstation using machine credentials and not domain credentials.
>>
>> Also when testing JCIFS you should use a simple stand-alone program
>> like examples/ListFiles.java."
>>
>> In other words:
>> (a) Since JCIFS does not use Kerberos for authentication, you need to
>> try to log into the recalcitrant server via Windows without using
>> Kerberos to be able to do a side-by-side comparison. Michael has some
>> ways of doing that, above.
>> (b) You may find that it doesn't work, in which case JCIFS is not
>> going to work either.
>> (c) If it *does* work, then try to generate your side-by-side
>> comparisons using a simpler example rather than ManifoldCF en toto;
>> you can see how at jcifs.samba.org, or I can help you further.
>>
>> He also mentions that there is some bizarreness on the response that
>> indicates that the server is configured in a way that he's never seen
>> before. And believe me, Michael has seen a *lot* of strange
>> configurations...
>>
>> Hope this helps.
>> Karl
>>
>> On Mon, Nov 28, 2011 at 4:12 AM, Karl Wright <da...@gmail.com> wrote:
>> > That should read "properties.xml", not "properties.ini". It looks
>> > like this page needs updating.
>> >
>> > The debug property in the XML form is:
>> >
>> > <property name="org.apache.manifoldcf.connectors" value="DEBUG"/>
>> >
>> > I don't think it will provide you with any additional information that
>> > is useful for debugging your authentication issue, however, if that is
>> > why you are looking at it. There may be some jcifs.jar debugging
>> > switches that might be of more help, but in the end I suspect you will
>> > need a packet capture of both a successful connection (via Windows)
>> > and an unsuccessful one (via MCF). The guy you will need to talk with
>> > after that is the jcifs author Michael Allen; I can give you his email
>> > address if you get that far.
>> >
>> > Karl
>> >
>> >
>> > On Mon, Nov 28, 2011 at 1:30 AM, Swapna Vuppala
>> > <sw...@gmail.com> wrote:
>> >> Hi Karl,
>> >>
>> >> I was planning to debug jCIFS repository connection using WireShark and
>> >> I
>> >> came across this
>> >> https://cwiki.apache.org/CONNECTORS/debugging-connections.html
>> >> Here, I see something as add "org.apache.manifoldcf.connectors=DEBUG"
>> >> to the
>> >> properties.ini file. Is it the properties.xml file that is being
>> >> referred
>> >> here ? If not, where do I find properties.ini file ?
>> >>
>> >> Thanks and Regards,
>> >> Swapna.
>> >>
>> >> On Thu, Nov 17, 2011 at 1:31 PM, Karl Wright <da...@gmail.com>
>> >> wrote:
>> >>>
>> >>> See http://jcifs.samba.org/src/docs/api/overview-summary.html#scp.
>> >>> The properties jcifs.smb.lmCompatibility and
>> >>> jcifs.smb.client.useExtendedSecurity are the ones you may want to
>> >>> change. These two properties go together so certain combinations make
>> >>> sense and others don't, so there's really only combinations you need
>> >>> but I'll need to look at what they are and get back to you later
>> >>> today.
>> >>>
>> >>> As far as setting the switches are concerned, if you are using the
>> >>> Quick Start you do this trivially by:
>> >>>
>> >>> <java> -Dxxx -Dyyy -jar start.jar
>> >>>
>> >>> If you are using the multi-process configuration, that is what the
>> >>> "defines" directory is for; you only need to create files in that
>> >>> directory with the names "jcifs.smb.lmCompatibility" and
>> >>> "jcifs.smb.client.useExtendedSecurity" containing the values you want
>> >>> to set.
>> >>>
>> >>> Karl
>> >>>
>> >>>
>> >>> On Thu, Nov 17, 2011 at 1:11 AM, Swapna Vuppala
>> >>> <sw...@gmail.com> wrote:
>> >>> > Hi Karl,
>> >>> >
>> >>> > Am able to access the folders on the problem server through windows
>> >>> > explorer, (\\server3\Folder1). I tried couple of things with the
>> >>> > credentials
>> >>> > form, changing username, domain etc.. but I keep getting the same
>> >>> > error
>> >>> > "Couldn't connect to server: Logon failure: unknown user name or bad
>> >>> > password"
>> >>> >
>> >>> > Can you tell me more about the -D switch you were talking of ?
>> >>> >
>> >>> > Thanks and Regards,
>> >>> > Swapna.
>> >>> >
>> >>> > On Tue, Nov 15, 2011 at 12:40 PM, Karl Wright <da...@gmail.com>
>> >>> > wrote:
>> >>> >>
>> >>> >> Glad you chased it down this far.
>> >>> >>
>> >>> >> First thing to try is whether you can get into the problem server
>> >>> >> using Windows Explorer. Obviously ManifoldCF is not going to be
>> >>> >> able
>> >>> >> to do it if Windows can't. If you *can* get in, then just playing
>> >>> >> with the form of the credentials in the MCF connection might do the
>> >>> >> trick. Some Windows or net appliance servers are picky about this.
>> >>> >> Try various things like leaving the domain blank and specifying the
>> >>> >> user as "abc@domain.com", for instance. There's also a different
>> >>> >> NTLM
>> >>> >> mode you can operation jcifs in that some servers may be configured
>> >>> >> to
>> >>> >> require; this would need you to set a -D switch on the command line
>> >>> >> to
>> >>> >> enable.
>> >>> >>
>> >>> >> Karl
>> >>> >>
>> >>> >> On Tue, Nov 15, 2011 at 12:10 AM, Swapna Vuppala
>> >>> >> <sw...@gmail.com> wrote:
>> >>> >> > Hi Karl,
>> >>> >> >
>> >>> >> > Thanks for the input. It looks like my problem is related to the
>> >>> >> > second
>> >>> >> > one
>> >>> >> > that you specified. One of the directories in the path am trying
>> >>> >> > to
>> >>> >> > index is
>> >>> >> > actually redirecting to a different server. And when I specify
>> >>> >> > this
>> >>> >> > new
>> >>> >> > server in defining the repository connection, with my
>> >>> >> > credentials,
>> >>> >> > the
>> >>> >> > connection fails with the message: "Couldn't connect to server:
>> >>> >> > Logon
>> >>> >> > failure: unknown user name or bad password"
>> >>> >> >
>> >>> >> > I'll look into why am not able to connect to this server.
>> >>> >> >
>> >>> >> > Thanks and Regards,
>> >>> >> > Swapna.
>> >>> >> >
>> >>> >> > On Mon, Nov 14, 2011 at 4:56 PM, Karl Wright <da...@gmail.com>
>> >>> >> > wrote:
>> >>> >> >>
>> >>> >> >> There's two kinds of problem you might be having. The first is
>> >>> >> >> intermittent, and the second is not intermittent but would have
>> >>> >> >> something to do with specific directories.
>> >>> >> >>
>> >>> >> >> Intermittent problems might include a domain controller that is
>> >>> >> >> not
>> >>> >> >> always accessible. In such cases, the crawl will proceed but
>> >>> >> >> will
>> >>> >> >> tend to fail unpredictably. On the other hand, if you have a
>> >>> >> >> directory that is handled by a DFS redirection, it is possible
>> >>> >> >> that
>> >>> >> >> the redirection is indicating a new server (lets call it
>> >>> >> >> server3)
>> >>> >> >> which may not like the precise form of your login credentials.
>> >>> >> >> Can
>> >>> >> >> you determine which scenario you are seeing?
>> >>> >> >>
>> >>> >> >> Karl
>> >>> >> >>
>> >>> >> >> On Mon, Nov 14, 2011 at 3:11 AM, Swapna Vuppala
>> >>> >> >> <sw...@gmail.com> wrote:
>> >>> >> >> > Hi,
>> >>> >> >> >
>> >>> >> >> > I have been using windows share repository connection to crawl
>> >>> >> >> > and
>> >>> >> >> > get
>> >>> >> >> > data
>> >>> >> >> > from a particular server (server 1). Its working perfectly
>> >>> >> >> > fine.
>> >>> >> >> > However, am
>> >>> >> >> > having trouble when I try with data from another server
>> >>> >> >> > (server
>> >>> >> >> > 2).
>> >>> >> >> >
>> >>> >> >> > When I define a repository connection of type windows share
>> >>> >> >> > and
>> >>> >> >> > specify
>> >>> >> >> > the
>> >>> >> >> > server name (server 2) with my credentials, the connection
>> >>> >> >> > status
>> >>> >> >> > shows
>> >>> >> >> > "Connection working". But when I run a job to use this
>> >>> >> >> > repository
>> >>> >> >> > connection
>> >>> >> >> > and index data from a location on this server 2, I keep
>> >>> >> >> > getting
>> >>> >> >> > the
>> >>> >> >> > exception below:
>> >>> >> >> >
>> >>> >> >> > JCIFS: Possibly transient exception detected on attempt 3
>> >>> >> >> > while
>> >>> >> >> > checking
>> >>> >> >> > if
>> >>> >> >> > file exists: Logon failure: unknown user name or bad password.
>> >>> >> >> > jcifs.smb.SmbAuthException: Logon failure: unknown user name
>> >>> >> >> > or
>> >>> >> >> > bad
>> >>> >> >> > password.
>> >>> >> >> > at
>> >>> >> >> > jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:544)
>> >>> >> >> > at jcifs.smb.SmbTransport.send(SmbTransport.java:661)
>> >>> >> >> > at jcifs.smb.SmbSession.sessionSetup(SmbSession.java:390)
>> >>> >> >> > at jcifs.smb.SmbSession.send(SmbSession.java:218)
>> >>> >> >> > at jcifs.smb.SmbTree.treeConnect(SmbTree.java:176)
>> >>> >> >> > at jcifs.smb.SmbFile.doConnect(SmbFile.java:911)
>> >>> >> >> > at jcifs.smb.SmbFile.connect(SmbFile.java:954)
>> >>> >> >> > at jcifs.smb.SmbFile.connect0(SmbFile.java:880)
>> >>> >> >> > at jcifs.smb.SmbFile.queryPath(SmbFile.java:1335)
>> >>> >> >> > at jcifs.smb.SmbFile.exists(SmbFile.java:1417)
>> >>> >> >> > at
>> >>> >> >> >
>> >>> >> >> >
>> >>> >> >> >
>> >>> >> >> >
>> >>> >> >> > org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.fileExists(SharedDriveConnector.java:2064)
>> >>> >> >> > at
>> >>> >> >> >
>> >>> >> >> >
>> >>> >> >> >
>> >>> >> >> >
>> >>> >> >> > org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getDocumentVersions(SharedDriveConnector.java:521)
>> >>> >> >> > at
>> >>> >> >> >
>> >>> >> >> >
>> >>> >> >> >
>> >>> >> >> >
>> >>> >> >> > org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:318)
>> >>> >> >> >
>> >>> >> >> > Am able to access this location from windows explorer. What
>> >>> >> >> > else
>> >>> >> >> > should
>> >>> >> >> > I be
>> >>> >> >> > checking or what could be the reasons/factors causing this to
>> >>> >> >> > fail
>> >>> >> >> > ?
>> >>> >> >> >
>> >>> >> >> > Thanks and Regards,
>> >>> >> >> > Swapna.
>> >>> >> >> >
>> >>> >> >
>> >>> >> >
>> >>> >
>> >>> >
>> >>
>> >>
>
>
Re: Problem crawling windows share
Posted by Swapna Vuppala <sw...@gmail.com>.
Hi Karl,
Thanks for the detailed reply.
As per your suggestion, I tried connecting to the problematic server via
windows using a machine that is not in the domain. On supplying the
credentials via windows, I could connect to the server, although it did not
list any folders. But when I tried accessing the folder of my interest via
\\server\folder$, I could see all the sub folders. All this is when I
connected via windows from a machine that's not joined to the domain.
The behavior is same when I connect to the server via windows from a
machine that's joined to the domain.
So I assume this means that through both Kerberos and NTLM, I see the same
behavior. Whereas ManifoldCF throws an exception when trying to define the
repository connection.
Please advise me on what I should be doing next to resolve this.
Thanks and Regards,
Swapna.
On Tue, Dec 6, 2011 at 12:55 PM, Karl Wright <da...@gmail.com> wrote:
> About your capture - Michael Allen says the following:
>
> "Actually this has nothing to do with DFS. JCIFS does not get to the
> point where it does DFS anything. The capture shows a vanilla
> STATUS_LOGON_FAILURE when GLOBAL\swapna.vuppala tries to auth with
> l-carx01.global.arup.com. So the possible causes for this are 1) the
> account name is not valid 2) the supplied password is incorrect 3)
> some security policy is deliberately blocking that user or particular
> type of auth or 4) some server configuration is incompatible with
> JCIFS. I only mention this last option because I noticed the target
> server has security signatures disabled. That's strange. If they're
> messing around with things like that, who knows what their clients are
> expected to do.
>
> Try a Windows client that uses NTLM instead of Kerberos. Meaning try a
> machine that is not joined to the domain so that when you try to
> access the target it asks you for credentials at which point you can
> test with GLOBAL\swapna.vuppala. Then it will use NTLM and you can
> actually compare captures. If the operator doesn't have a laptop or
> something not joined to the domain, it might be sufficient to log into
> a workstation using machine credentials and not domain credentials.
>
> Also when testing JCIFS you should use a simple stand-alone program
> like examples/ListFiles.java."
>
> In other words:
> (a) Since JCIFS does not use Kerberos for authentication, you need to
> try to log into the recalcitrant server via Windows without using
> Kerberos to be able to do a side-by-side comparison. Michael has some
> ways of doing that, above.
> (b) You may find that it doesn't work, in which case JCIFS is not
> going to work either.
> (c) If it *does* work, then try to generate your side-by-side
> comparisons using a simpler example rather than ManifoldCF en toto;
> you can see how at jcifs.samba.org, or I can help you further.
>
> He also mentions that there is some bizarreness on the response that
> indicates that the server is configured in a way that he's never seen
> before. And believe me, Michael has seen a *lot* of strange
> configurations...
>
> Hope this helps.
> Karl
>
> On Mon, Nov 28, 2011 at 4:12 AM, Karl Wright <da...@gmail.com> wrote:
> > That should read "properties.xml", not "properties.ini". It looks
> > like this page needs updating.
> >
> > The debug property in the XML form is:
> >
> > <property name="org.apache.manifoldcf.connectors" value="DEBUG"/>
> >
> > I don't think it will provide you with any additional information that
> > is useful for debugging your authentication issue, however, if that is
> > why you are looking at it. There may be some jcifs.jar debugging
> > switches that might be of more help, but in the end I suspect you will
> > need a packet capture of both a successful connection (via Windows)
> > and an unsuccessful one (via MCF). The guy you will need to talk with
> > after that is the jcifs author Michael Allen; I can give you his email
> > address if you get that far.
> >
> > Karl
> >
> >
> > On Mon, Nov 28, 2011 at 1:30 AM, Swapna Vuppala
> > <sw...@gmail.com> wrote:
> >> Hi Karl,
> >>
> >> I was planning to debug jCIFS repository connection using WireShark and
> I
> >> came across this
> >> https://cwiki.apache.org/CONNECTORS/debugging-connections.html
> >> Here, I see something as add "org.apache.manifoldcf.connectors=DEBUG"
> to the
> >> properties.ini file. Is it the properties.xml file that is being
> referred
> >> here ? If not, where do I find properties.ini file ?
> >>
> >> Thanks and Regards,
> >> Swapna.
> >>
> >> On Thu, Nov 17, 2011 at 1:31 PM, Karl Wright <da...@gmail.com>
> wrote:
> >>>
> >>> See http://jcifs.samba.org/src/docs/api/overview-summary.html#scp.
> >>> The properties jcifs.smb.lmCompatibility and
> >>> jcifs.smb.client.useExtendedSecurity are the ones you may want to
> >>> change. These two properties go together so certain combinations make
> >>> sense and others don't, so there's really only combinations you need
> >>> but I'll need to look at what they are and get back to you later
> >>> today.
> >>>
> >>> As far as setting the switches are concerned, if you are using the
> >>> Quick Start you do this trivially by:
> >>>
> >>> <java> -Dxxx -Dyyy -jar start.jar
> >>>
> >>> If you are using the multi-process configuration, that is what the
> >>> "defines" directory is for; you only need to create files in that
> >>> directory with the names "jcifs.smb.lmCompatibility" and
> >>> "jcifs.smb.client.useExtendedSecurity" containing the values you want
> >>> to set.
> >>>
> >>> Karl
> >>>
> >>>
> >>> On Thu, Nov 17, 2011 at 1:11 AM, Swapna Vuppala
> >>> <sw...@gmail.com> wrote:
> >>> > Hi Karl,
> >>> >
> >>> > Am able to access the folders on the problem server through windows
> >>> > explorer, (\\server3\Folder1). I tried couple of things with the
> >>> > credentials
> >>> > form, changing username, domain etc.. but I keep getting the same
> error
> >>> > "Couldn't connect to server: Logon failure: unknown user name or bad
> >>> > password"
> >>> >
> >>> > Can you tell me more about the -D switch you were talking of ?
> >>> >
> >>> > Thanks and Regards,
> >>> > Swapna.
> >>> >
> >>> > On Tue, Nov 15, 2011 at 12:40 PM, Karl Wright <da...@gmail.com>
> >>> > wrote:
> >>> >>
> >>> >> Glad you chased it down this far.
> >>> >>
> >>> >> First thing to try is whether you can get into the problem server
> >>> >> using Windows Explorer. Obviously ManifoldCF is not going to be
> able
> >>> >> to do it if Windows can't. If you *can* get in, then just playing
> >>> >> with the form of the credentials in the MCF connection might do the
> >>> >> trick. Some Windows or net appliance servers are picky about this.
> >>> >> Try various things like leaving the domain blank and specifying the
> >>> >> user as "abc@domain.com", for instance. There's also a different
> NTLM
> >>> >> mode you can operation jcifs in that some servers may be configured
> to
> >>> >> require; this would need you to set a -D switch on the command line
> to
> >>> >> enable.
> >>> >>
> >>> >> Karl
> >>> >>
> >>> >> On Tue, Nov 15, 2011 at 12:10 AM, Swapna Vuppala
> >>> >> <sw...@gmail.com> wrote:
> >>> >> > Hi Karl,
> >>> >> >
> >>> >> > Thanks for the input. It looks like my problem is related to the
> >>> >> > second
> >>> >> > one
> >>> >> > that you specified. One of the directories in the path am trying
> to
> >>> >> > index is
> >>> >> > actually redirecting to a different server. And when I specify
> this
> >>> >> > new
> >>> >> > server in defining the repository connection, with my credentials,
> >>> >> > the
> >>> >> > connection fails with the message: "Couldn't connect to server:
> >>> >> > Logon
> >>> >> > failure: unknown user name or bad password"
> >>> >> >
> >>> >> > I'll look into why am not able to connect to this server.
> >>> >> >
> >>> >> > Thanks and Regards,
> >>> >> > Swapna.
> >>> >> >
> >>> >> > On Mon, Nov 14, 2011 at 4:56 PM, Karl Wright <da...@gmail.com>
> >>> >> > wrote:
> >>> >> >>
> >>> >> >> There's two kinds of problem you might be having. The first is
> >>> >> >> intermittent, and the second is not intermittent but would have
> >>> >> >> something to do with specific directories.
> >>> >> >>
> >>> >> >> Intermittent problems might include a domain controller that is
> not
> >>> >> >> always accessible. In such cases, the crawl will proceed but
> will
> >>> >> >> tend to fail unpredictably. On the other hand, if you have a
> >>> >> >> directory that is handled by a DFS redirection, it is possible
> that
> >>> >> >> the redirection is indicating a new server (lets call it server3)
> >>> >> >> which may not like the precise form of your login credentials.
> Can
> >>> >> >> you determine which scenario you are seeing?
> >>> >> >>
> >>> >> >> Karl
> >>> >> >>
> >>> >> >> On Mon, Nov 14, 2011 at 3:11 AM, Swapna Vuppala
> >>> >> >> <sw...@gmail.com> wrote:
> >>> >> >> > Hi,
> >>> >> >> >
> >>> >> >> > I have been using windows share repository connection to crawl
> and
> >>> >> >> > get
> >>> >> >> > data
> >>> >> >> > from a particular server (server 1). Its working perfectly
> fine.
> >>> >> >> > However, am
> >>> >> >> > having trouble when I try with data from another server (server
> >>> >> >> > 2).
> >>> >> >> >
> >>> >> >> > When I define a repository connection of type windows share and
> >>> >> >> > specify
> >>> >> >> > the
> >>> >> >> > server name (server 2) with my credentials, the connection
> status
> >>> >> >> > shows
> >>> >> >> > "Connection working". But when I run a job to use this
> repository
> >>> >> >> > connection
> >>> >> >> > and index data from a location on this server 2, I keep getting
> >>> >> >> > the
> >>> >> >> > exception below:
> >>> >> >> >
> >>> >> >> > JCIFS: Possibly transient exception detected on attempt 3 while
> >>> >> >> > checking
> >>> >> >> > if
> >>> >> >> > file exists: Logon failure: unknown user name or bad password.
> >>> >> >> > jcifs.smb.SmbAuthException: Logon failure: unknown user name or
> >>> >> >> > bad
> >>> >> >> > password.
> >>> >> >> > at
> jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:544)
> >>> >> >> > at jcifs.smb.SmbTransport.send(SmbTransport.java:661)
> >>> >> >> > at jcifs.smb.SmbSession.sessionSetup(SmbSession.java:390)
> >>> >> >> > at jcifs.smb.SmbSession.send(SmbSession.java:218)
> >>> >> >> > at jcifs.smb.SmbTree.treeConnect(SmbTree.java:176)
> >>> >> >> > at jcifs.smb.SmbFile.doConnect(SmbFile.java:911)
> >>> >> >> > at jcifs.smb.SmbFile.connect(SmbFile.java:954)
> >>> >> >> > at jcifs.smb.SmbFile.connect0(SmbFile.java:880)
> >>> >> >> > at jcifs.smb.SmbFile.queryPath(SmbFile.java:1335)
> >>> >> >> > at jcifs.smb.SmbFile.exists(SmbFile.java:1417)
> >>> >> >> > at
> >>> >> >> >
> >>> >> >> >
> >>> >> >> >
> >>> >> >> >
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.fileExists(SharedDriveConnector.java:2064)
> >>> >> >> > at
> >>> >> >> >
> >>> >> >> >
> >>> >> >> >
> >>> >> >> >
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getDocumentVersions(SharedDriveConnector.java:521)
> >>> >> >> > at
> >>> >> >> >
> >>> >> >> >
> >>> >> >> >
> >>> >> >> >
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:318)
> >>> >> >> >
> >>> >> >> > Am able to access this location from windows explorer. What
> else
> >>> >> >> > should
> >>> >> >> > I be
> >>> >> >> > checking or what could be the reasons/factors causing this to
> fail
> >>> >> >> > ?
> >>> >> >> >
> >>> >> >> > Thanks and Regards,
> >>> >> >> > Swapna.
> >>> >> >> >
> >>> >> >
> >>> >> >
> >>> >
> >>> >
> >>
> >>
>