dwc

The Most Annoying Appliance Ever

Joel on Software got a free Mini appliance from Google. Within an hour or two of using it, he discovered the same behavior I find so infuriating about ours: The upgrade process.

Upgrading requires two steps: one for the "system version" and another for the "software version". No problem, right? Well, you have to reboot the appliance in between these two steps. Which requires physical access.

Yeah, instead of programming a "reboot" button into the Web interface, Google decided to give you a "shutdown" button and instructions to pull the plug after waiting "several minutes for the system to shutdown". (For your reference, Google, all the reboot button has to do is system("/sbin/shutdown", "-r", "now"); instead of system("/sbin/shutdown", "-h", "now");.)

Sigh.

More on Basecamp

(These work entries should probably go in my work blog…)

So after thinking about how we use Basecamp a little more, and revisiting some of my issues with it, I've realized a few things:

1. Creating really small milestones is not a bad thing. In fact, it's probably a good practice. This forces you to break a project down into little pieces that are much easier to finish.
2. Predicting all your milestones ahead of time is hard. But that's okay, because you can always change them.

Taking a more active approach is important with any sort of project management - unfortunately we weren't really doing that. ;-)

Basecamp: "Everyone's favorite"?

At work we've been using Basecamp to track projects. In short, it's not bad and I'm glad that we are keeping a better record of things.

The long story is a little more complicated. Originally the programming team (me and one student employee, who has since graduated and found a full-time job elsewhere) used CVS, Bugzilla, and a whiteboard to keep track of things. (We also tried a wiki, but it never stuck.) It was painful at times, but generally worked for us. (If Trac had been around then, we almost certainly would have used that instead.)

Unfortunately my coworkers couldn't get past Bugzilla's interface. It's a common complaint about the product, even from programmers. Admittedly I found it confusing at first, but eventually learned to ignore most of the fields. ;-)

The end result was a pretty serious communication gap between the programmers, designers, and our manager. Occasionally we had all-hands meetings, but these were lacking in focus (sure, let's talk in detail about ALL of our projects at once) and waaaaaaaay too long (hello, I have NADD).

Things were pretty bad for about a year. Projects weren't getting done but somehow we kept taking on new responsibility. Ugh.

We've been using Basecamp for about 3 months now, and everyone seems pretty happy with the product. Usage seems a bit light, but it's better than nothing.

If you've used 37signals' other products, you've pretty much used Basecamp. The only extra features Basecamp provides, as far as I can tell, are:

1. An overview of all your projects (the "dashboard")
2. Milestones
3. Messages

I like the dashboard. Milestones are nice, although we don't use them much. Messages … well, I prefer mailing lists or newsgroups over lightweight forums.

Parts of it feel a little, well, cobbled together. For instance, the writeboards (which are the same thing as 37signals' standalone product of same name) have an interstitial redirect page which BREAKS MY BACK BUTTON. Uh, guys? Didn't we solve this problem like 10 years ago?

You can't comment on a to-do item. You can associate a message with a milestone, but that's not specific enough for me. Yes, Bugzilla swings too far in the other direction for 99% of users, but a little bit of focused discussion is nice.

Only writeboards come with history. Comparing two versions of a writeboard sometimes yields strange results. I don't know if they thought giving people a diff was too confusing, but whatever they're using ends up confusing me more than helping me.

Even more frustrating is the lack of integration with our version control. Granted, 37signals isn't targeting people with this problem. However, there's an API for Backpack. Not for Basecamp. All I want to be able to do is add messages and close to-dos from an svk ci.

I understand that my needs as a programmer are different than those of desgners and managers, but I think it could be balanced better.

Everything looks perfect from far away

No

I was listening to The Downward Spiral at work.

Boss: Is this new Nine Inch Nails?

Finally.

We finally released the new UF News site today. It's technically in beta for now, but I'm just excited to be getting feedback from a wider audience.

The site runs WordPress and about 20 plugins. More information is available in my development blog.

Taylor did a great job on the design of the site. I'm sure he'll be blogging about it at some point. :-)

My Office

It has a door.

When this door is closed, I do not want to be bothered.

Especially if all you have to say is "SUMMER B PUSSY OH YEAH".

Legal Stuff

I should probably note that the previous entry about accessing the UF home page from handheld devices was a personal opinion, and that anything I write here is not endorsed by my employer.

(No, they didn't ask me to say this.)

HOWTO: Ensure No One Will Develop for Your Platform

Recently we received (via a public mailing list) a report that our site did not work on Palm devices. We all found this a little strange, since our site is developed with standards in mind and contains pretty basic HTML.

I knew that Palm made an emulator available, and was happy to find that an ebuild was available. However, the ebuild didn't so much work, so I made an attempt to fix it. Once it was installed, I booted up one of the ROMs to find that no Web browser came installed. I gave up after searching for a while.

It turns out the person testing our site was using a Tungsten C device, with PalmSource Web Browser 2.0. Okay, so I search around the PalmSource site some more and find that I should be using a device running either "Garnet" or "Cobalt".

"Simulators" exist for Garnet and Cobalt, but they're for Windows only. Well, this is why I have VMware.

Neither simulator included a Web browser, so I started looking around for PalmSource Web Browser 2.0. I think it's now Palm Web Browser Pro or maybe Palm Web Pro 3.0, but who knows? The "Buy Now" links those pages didn't go anywhere useful, so even if I wanted to give Palm my money, I probably wouldn't get much in return.

Let this be a lesson to platform developers: if you don't make it easy to test Web sites on your device, Web developers will eventually give up out of frustration. I apologize to the three people trying to access our site from a handheld, but you're SOL.

Mac OS X vs. Samba: It Does Work, But…

I upgraded to Tiger on the storage server at work, not really expecting my problems with Samba to be fixed.

To recap, the Windows XP workstations in our office are controlled by a separate group. Their domain policy forces NTLMv2 authentication for Windows shares. Mac OS X did not support NTLMv2 authentication, which made our file shares inaccessible except via SFTP. We eventually set up WebDrive to make it easier to access the file share.

Amazingly, NTLMv2 authentication is supported in Mac OS X 10.4. Once I solved a small issue with NetBIOS (connecting to \\the.servers.fqdn\share didn't work, because the.servers.fqdn was too long for NetBIOS or some shit), the Windows clients were able to connect. w00t!

There's still some weirdness in OS X's Samba support, namely that it restricts access to specific accounts. This isn't really a bad thing. However, when you enable access for an account, you are warned that doing so makes the account's password less secure. In other words, it rehashes the user's password (and thus has to prompt you for it), using a weaker algorithm. Yay!

I guess I have another reason to look into LDAP authentication or maybe Active Directory integration, which would theoretically let us use GatorLink instead of OS X's crap. I would hope OS X's Samba is smart enough to use Open Directory properly, but I don't have much hope.

Minor Improvements

This evening, I quietly published a new version of the UF home page with two minor improvements:

1. Replace some instances of "..." with the HTML entity … (horizontal ellipsis). We're waiting on a few other changes to replace the other instances.
2. Add a document relationship to the University of Florida News feed. We've been using the feed internally for almost 3 years, but for a number of reasons never got around to promoting it publicly. The whole text/xml vs. application/xml vs. application/rss+xml debate was troubling, so I sampled Movable Type and WordPress blogs and decided on application/rss+xml. It's still an open debate, and I'm not convinced I made the right choice.

Sometimes the little victories are rewarding.

And this is only the beginning. I'm planning on converting the feed Real Soon Now to RSS 0.92 (oops, I forgot), and eventually creating RSS 1.0, RSS 2.0, and Atom feeds. Taylor and Mark have been busy cleaning up the CSS and markup. Soon things will be beautiful.

The Problem With Being Webmaster…

…is that your email address is easy to guess. After our week-long vacation, I found 1500 emails that SpamBayes correctly identified as spam and 500 returned emails that were obviously spammers using our email address to send their crap.

Mac OS X vs. Samba: Redux

Karl Espe from the University of Minnesota emailed me with one solution to the problem with Windows XP clients being unable to connect to Mac OS X's Samba server.

Karl says that you can change the LAN Manager authentication level on Windows XP by going to Start » Control Panel » Administrative Tools » Local Security Policy, then navigating to Security Settings » Local Policies » Security Options. You should see an option, Network security: LAN Manager authentication level. If this is set to "Send NTLMv2 response only\refuse LM & NTLM", your clients will almost certainly have problems connecting to Mac OS X via Windows Sharing. You can change this to "Send LM & NTLM responses", which should fix the problem. (If you're interested in the various options in that menu, see Microsoft's support entry.)

Unfortunately in my case, that option is disabled by our domain administrators. I have a feeling they force NTLMv2 because LM and NTLM are less secure (as Karl mentions in the email). But in some cases (e.g. on a private LAN), it might not matter.

Thanks to Karl for emailing me this information.

Mac OS X vs. Samba: Defeat

Apple closed my bug report (Problem ID: 3873724), saying that the Open Directory plugin does not support NTLMv2 authentication. (See Round 1, Round 2, Round 3, Round 4, Frustration, and opendirectory_ntlmv2_auth_user.)

I'm sorry to say that I'm pretty disappointed with the outcome. Obviously someone at Apple made an attempt to implement NTLMv2 authentication for Open Directory (as evidenced by the opendirectory_ntlmv2_auth_user method in auth_ods/auth_ods.c). I guess it's not finished, and as a result, is not supported by Apple.

Apple Engineering suggests that I manually configure a different passdb backend (which I tried without success in Round 2). They also claim that "NTLMv2 is supported by users with ShadowHash and Password Server authentication authorities for Panther", which I'll need to look into (read: I don't understand what that means).

Argh.

Half of the time we're gone but we don't know where

Well

I received my "emergency paycheck" today. It turns out that it might be for the wrong pay period, and that they also took out money for a retirement plan that I haven't yet enrolled in. Yeah, I think I'll be holding onto this one until we can be sure everything's kosher.

I'm not supposed to feel

Content Management

D. Keith Robinson recently wrote an article about the process of managing content.

At work, this seems to be a pretty common issue. On the one hand, we've been very successful in the past with giving personal attention to our customers and their sites. On the other hand, some of our clients want more control over their sites (which really means more frequent and involved updates), but don't have the knowledge or time to do so. Oh, did I mention we're understaffed right now?

Paychecks in the 21st Century

When I was hired to a full-time position on November 5, I expected there to be some delay before myUFL figured it out.

Two weeks later, myUFL still lists me as an OPS employee. Apparently, my notice of appointment was "lost in the system" somwhere. No one really knows if I'm getting a paycheck next week.

Seriously.

opendirectory_ntlmv2_auth_user

Yep, when NTLMv2 authentication is forced, no clients are able to connect. Not even smbclient on Linux, which can successfully connect via NTLM authentication. Fun!

Exotic fish will never be able to walk
As long as we live

Frustration

Apple can't reproduce the problem, so I have more debugging to do. I'm going to try tracking down another license of Windows XP Professional to do a clean install (I love VMware, by the way) that is not part of our Windows domain. It shouldn't matter, though, because I thought I tried connecting to the Mac using smbclient with NTLMv2 authentication and having the same problem.

I really don't want it to come to this, but the source IS open. I could always try debugging the opendirectory_ntlmv2_auth_user method. And as a last resort, there's always Ethereal and tcpdump.

In other work-related news, I'm going to new employee orientation tomorrow from 8:30 to noon. I've been a little overwhelmed by the process of signing up for benefits - I feel like I'm lacking the information necessary to make the best choices - so hopefully this training will fill in some of the gaps.

What kind of fool am I

Mac OS X vs. Samba, Round 4

I finally got sick of the whole issue, and filed a bug report with Apple. I'm not really sure what to expect from them. I've debugged the problem and isolated it to a specific function (and further verified that a stock Samba installation does not exhibit the problem when NTLMv2 authentication is forced), so I hope the report will be taken seriously.

Now that I think about it, I should add an UPSTREAM resolution to my Bugzilla. :-)

Mac OS X vs. Samba, Round 3

I come back to this problem every so often, but never work on it for very long because it's so frustrating. Finally, I've isolated the problem to NTLMv2 authentication in Mac OS X's version of Samba. (Round 1, Round 2)

After updating to 10.3.6, I decided to try connecting again, since the notice claimed "improved file sharing for … PC (SMB/CIFS)". From the Windows side, everything looked the same: Windows tries passing domain authentication (a few dozen times, according to the logs). After giving up, it presents a password dialog. You enter the username and password of the Mac OS X account, which are rejected.

Maybe I missed it in the logs last time, but this whole process spit out a pretty informative message (at log level = 1) this time around:

[2004/11/08 09:48:22, 1] auth_ods.c:opendirectory_ntlmv2_auth_user(312)
  User "dwc" failed to authenticate with "dsAuthMethodStandard:dsAuthNodeNTLMv2" (-14091) :(

There are server options to disable the weaker LANMAN and NTLM authentication methods, but not the NTLMv2 method (please, someone correct me if I'm wrong). You can force NTLMv2 authentication on the server side by saying:

lanman auth = no
ntlm auth = no

When I restarted smbd with these options, all clients were denied access, even clients that could connect before, with the same error in the logs. So the Windows clients just happened to expose a problem with NTLMv2 authentication (due to their LMCompatibilityLevel, NTLMv2 is preferred). I either missed this fact because I wasn't reading the logs closely enough, or it wasn't being logged in any useful manner until the 10.3.6 update.

Once again, I'm sort of stuck. At this point, I know the problem lies in the opendirectory_ntlmv2_auth_user method, but I'm not sure I know how to fix it. I'd like to finally close this issue, but I still have some research and testing ahead.

If you're having similar problems, you might want to read a paper by one of the Samba developers from a USENIX conference in 2000. It discusses some of the registry values which control the authentication; if you have access to regedit (I don't), you may be able to solve the problem fairly easily. (The server NTLMv2 option referenced in that paper does not work on Mac OS X's version of Samba.)

All my world in one grain of sand
And I've blown it

Is this Yon Hall?

We get a lot of people asking how to find Yon 4C, or where to pay for CPR classes, or where some random person's office is. The thing is, we have absolutely no clue. We just run the UF Home Page.

We've tried to reduce the number of questions by posting signs. It started with a simple "100s = first floor, 200s = second floor", which we all thought was obvious. We later added a "4C is downstairs", since that was a pretty common question.

It wasn't enough, apparently. We tried a big STAIRS sign generated with FIGlet, which didn't work very well. My latest attempt:

I printed it as a 4-ft banner (which is a lot of trouble when all you have is letter-size paper). It's probably the first thing you see if your head isn't shoved up your butt, so hopefully this will reduce the number of questions....

Mac OS X vs. Samba, Round 2

A few months ago, I ran into some problems with Mac OS X's implementation of Samba. After spending a little more time with it, I'm pretty convinced that Apple indeed screwed it up.

I started debugging the problem again by verifying that my Windows XP installation could connect to a Linux Samba server. The server was configured using the defaults for authentication options: encrypt passwords = yes, auth methods = (use security setting), passdb backend = smbpasswd, username map = (no username map), and security = user. I added a new SMB user using smbpasswd. In Windows Explorer, I typed \\linuxserver\share, and after a brief delay (during which the client seemed to be sending the username and password used on our domain) was prompted for a username and password. The share showed up fine after entering them.

This definitely suggested that there was some issue with the Mac OS X version of Samba. I opened up the Mac's smb.conf, and started setting the authentication options to be as similar as those on the Linux box. I used the following configuration:

[global]
  workgroup = WEBADMIN
  netbios name = server
  os level = 0
  lm announce = no
  preferred master = no
  local master = no
  domain master = no
  guest account = unknown
  invalid users = root
  encrypt passwords = yes
;;; Default Mac configuration
;  auth methods = guest opendirectory
;  passdb backend = opendirectorysam guest
;;; -END- Default Mac configuration
;;; Default Samba configuration
  auth methods =
  passdb backend = smbpasswd:/etc/smbpasswd
  username map =
  security = user
;;; -END- Default Samba configuration
  printable = no
;  printer admin = @admin, @staff
  server string = Samba %v on %h (Mac OS X)
  unix charset = UTF-8-MAC
  display charset = UTF-8-MAC
  dos charset = 437
  use spnego = no
  client ntlmv2 auth = no
  unix extensions = no
  log level = 1

Save smb.conf, restart Windows Sharing. I then tried smbpasswd -a dwc, and it bombs out with the same error as last time:

[15:10:16 dwc@server ~]$ sudo smbpasswd -a dwc
New SMB password:
Retype new SMB password:
could not find new user/computer dwc in passdb.
Failed to initialise SAM_ACCOUNT for user dwc.
Failed to modify password entry for user dwc


When using the default Mac configuration, smbpasswd dies with an even worse error:

server:/etc root# smbpasswd -a dwc
New SMB password:
Retype new SMB password:
dyld: smbpasswd Undefined symbols:
/etc/pdb/opendirectorysam.so undefined reference to _get_opendirectory_authenticator expected to be defined in the executable
Trace/BPT trap

ARGH. I didn't really expect this to work, but it would have been interesting to see the results.

I understand that the Mac OS X user accounts are stored in NetInfo, so maybe Samba isn't able to see them when it's using the smbpasswd backend. Windows clients can connect to Linux servers using this backend, but can't connect to Mac servers using the default backend. So what the hell am I supposed to do?

Installing Samba from source might solve the problem, but I wonder how it will interface with NetInfo (if at all). Setting up a Linux server to act as a proxy of sorts would almost certainly work, but I shouldn't have to resort to that...

Samba Permissions

As I previously mentioned Mac OS X's implementation of Samba has been giving me a lot of trouble. I fixed one issue today, thanks to Google. On my Linux workstation, I was having issues with getting smbmount to set the file ownership according to the uid and gid parameters in /etc/fstab. The solution is to turn off the new UNIX extensions to the SMB/CIFS protocol on the Samba server. I'm guessing this is a bug in Samba, but I'm not sure.

That's one of the 30928423098423098 things I have to do right now.

More Portal Fun

From Bridges Advisory #18 - July 20, 2004:

DIRECT DEPOSIT INFORMATION REMOVED FROM myUFL. Due to campus security concerns, the university has removed self service access in the portal for viewing of direct deposit information.

:-D

Mac OS X vs. Samba

Mac OS X's implementation of Samba is a load of crap.

Taking a trick out of Microsoft's book, Apple has embraced and extended Samba to use Open Directory for user authentication. I won't speak on the merits of Open Directory, since I don't have much experience with it (other than using NetInfo Manager to manage groups), but something went wrong with the marriage of Samba and Open Directory.

Before ranting, I should probably note that I've used Samba on Linux for years now. I'm used to getting it running with a fairly simple configuration, using smbpasswd as my passdb backend. It's a simple way to get Samba running for one or two people. It hasn't failed me for any reason other than my own stupidity, but it's certainly not something I'd want to use in an "enterprise".

Anyway, I've spent the better part of my week trying to figure out why our Windows XP clients (which I don't administer) could not connect to our Mac OS X file server (which I manage). The clients refused to authenticate to any account on the Mac via Windows file sharing, regardless of what I tried - specifying the domain, leaving it off, etc. I tried creating new accounts on the Mac, but those didn't work. This kind of problem gets to be very frustrating when your coworkers need to access their files to do their jobs, and you have no idea why.

Did I mention that Mac OS X and Linux clients had no problems connecting (except for a strange permissions issue on my Linux workstation)?

When a Windows XP client would attempt to connect, I saw the following over and over in smbd.log:

[2004/07/16 14:19:42, 0] /SourceCache/samba/samba-56/samba/source/smbd/server.c:main(747)
  smbd version 3.0.2 started.
  Copyright Andrew Tridgell and the Samba Team 1992-2004

Weird, huh? Sometimes I would see an entry like:

[2004/07/16 14:19:42, 1] pdb_ods.c:odssam_getsampwrid(1831)
  odssam_getsampwrid: rid<501> rid str<501>

This was neither informative nor helpful. Actually, it didn't make any sense. The rid matched my account's uid, but not the uid of the account which was attempting to connect via Samba. I spent some time reading the code from Apple's implementation of Samba, but I know next to nothing about Samba's codebase to begin with, let alone Open Directory. I was desperate, but not that desperate. (If anyone from Apple reads this: why the fuck isn't that code documented?)

Oh, did I mention I'm debugging "production" services during business hours, when people who can connect are actively working from the file server? "Sorry, guys, I have to restart Samba again."

I pretty much figured that our Windows XP clients were passing authentication information from the domain which they are a part of. Sure, if your domain authentication fails, Windows prompts you for a username and password. But apparently ignores what you enter. I can't be sure, since I don't know where Windows logs this kind of information, even if it does.

The "Network Guys" (our Windows domain administrators) suggested that I give the Mac a NetBIOS name, set its workgroup to match the name of the Windows domain, and create accounts with the same username as the domain accounts. Didn't work, but I'll admit I might have missed something.

So now you're probably wondering why I'm blaming Mac OS X for these problems. Just for fun, I set up a Samba share on Linux. The Windows clients were able to connect with no problems. So something on Mac OS X was failing, and the log entries suggested Open Directory. (Truthfully, I should verify this again - I tested this on my laptop when it was still able (magically) to connect to the Mac.)

"Okay," I thought, "let's try using smbpasswd instead of opendirectorysam." I change smb.conf, restart the service, and try adding a Samba password:

[14:39:23 dwc@server ~]$ sudo smbpasswd -a dwc
New SMB password:
Retype new SMB password:
could not find new user/computer dwc in passdb.
Failed to initialise SAM_ACCOUNT for user dwc.
Failed to modify password entry for user dwc


So it seems that smbpasswd doesn't understand Mac OS X's user accounts (through Open Directory) when you use smbpasswd as the passdb backend. Great. I wasn't about to go mucking about in /etc/passwd when I didn't know how it would affect Open Directory and Mac OS X as a whole.

I have found these links - but they are fairly old and haven't been very helpful so far.

• Mac OS X and SMB HOWTO (I don't have access to regedit.exe, by the way, so I can't enable plain text passwords - nor would I want to)
• Using MacOS X 10.2.X Windows File Sharing
• Mac OS X Server Cross-platform Issues

As a result of all these headaches, I've been pushing for moving the file share back to the Network Guys. Once you're on a domain, it really seems like an all or nothing kind of deal.

There are really two issues here:

1. People not listening to me. A few months ago: "Let's move some of our services away from the Network Guys, since us Mac users can't connect to the shares they provide." Fine, as long as the Windows computers can connect. I didn't like the idea because I was still a student, and didn't want an important server to go down when I wasn't around. I made this pretty clear to everyone involved, but it was decided that we still wanted the server. Now we have the same problem on a wider scale.
2. Commercial software getting in my way. Time and time again, I've been disappointed by commercial software because it gets in my way. It pushes my buttons. The free software I've chosen to use doesn't hinder me in this way. It works consistently, tells me when I'm being stupid (usually with an informative message), and is generally better documented. So why do people use commercial software? Oh, becuase pressing a button and hoping it works is easier than modifying a text file and knowing it will work.

It's not up to you
It never really was

Scratch That

Nevermind. We aren't moving to Subversion. I guess it was fun playing with it for a few days - I learned quite a bit about Apache 2.

*sigh*, I don't think I've ever done anything important.

cvs [server aborted]: received broken pipe signal

Today is the day that we begin the process of moving from CVS to Subversion at work. I spent about a week evaluating CVS for use by all employees in the office - including the Web designers. After running into a problems with automatic deploying of files to our Web servers, I decided that we could not use CVS and expect any sort of reliable behavior.

No matter what I tried, I would receive the following error when hooking a script into the end of a commit using loginfo:

cvs [server aborted]: received broken pipe signal

I searched and searched for solutions to this problem, but mostly found people having the same problem and no resolution. The CVS manual has this to say:

This message seems to be caused by a hard-to-track-down bug in CVS or the systems it runs on (we don't know--we haven't tracked it down yet!). It seems to happen only after a CVS command has completed, and you should be able to just ignore the message. However, if you have discovered information concerning its cause, please let us know as described in Dealing with bugs in CVS or this manual.

Unfortunately, the broken pipe kills the cvs server process on the remote host, meaning commits accross directories stop, usually after the first directory. This leaves the CVS repository in an inconsistent state, which is never good.

So, we're moving to Subversion, which means I will probably make the move at home also for my personal and school files. I'll probably wait until the end of the semester (my last as an undergraduate!), since there's only a month left and I've already trained a few group members to use CVS.