I come back to this problem every so often, but never work on it for very long because it's so frustrating. Finally, I've isolated the problem to NTLMv2 authentication in Mac OS X's version of Samba. (Round 1, Round 2)
After updating to 10.3.6, I decided to try connecting again, since the notice claimed "improved file sharing for … PC (SMB/CIFS)". From the Windows side, everything looked the same: Windows tries passing domain authentication (a few dozen times, according to the logs). After giving up, it presents a password dialog. You enter the username and password of the Mac OS X account, which are rejected.
Maybe I missed it in the logs last time, but this whole process spit out a pretty informative message (at log level = 1) this time around:
[2004/11/08 09:48:22, 1] auth_ods.c:opendirectory_ntlmv2_auth_user(312)
User "dwc" failed to authenticate with "dsAuthMethodStandard:dsAuthNodeNTLMv2" (-14091) :(
There are server options to disable the weaker LANMAN and NTLM authentication methods, but not the NTLMv2 method (please, someone correct me if I'm wrong). You can force NTLMv2 authentication on the server side by saying:
lanman auth = no
ntlm auth = no
When I restarted smbd with these options, all clients were denied access, even clients that could connect before, with the same error in the logs. So the Windows clients just happened to expose a problem with NTLMv2 authentication (due to their LMCompatibilityLevel, NTLMv2 is preferred). I either missed this fact because I wasn't reading the logs closely enough, or it wasn't being logged in any useful manner until the 10.3.6 update.
Once again, I'm sort of stuck. At this point, I know the problem lies in the opendirectory_ntlmv2_auth_user method, but I'm not sure I know how to fix it. I'd like to finally close this issue, but I still have some research and testing ahead.
If you're having similar problems, you might want to read a paper by one of the Samba developers from a USENIX conference in 2000. It discusses some of the registry values which control the authentication; if you have access to regedit (I don't), you may be able to solve the problem fairly easily. (The server NTLMv2 option referenced in that paper does not work on Mac OS X's version of Samba.)
All my world in one grain of sand
And I've blown it