Opening a file with an unknown extension

Friday 18 March 2005This is more than 18 years old. Be careful.

I was trying to fix a bug at work yesterday that had bothered me for a while. Our product launches document files, and the bug was that if the file has no file association, it wouldn’t do what most Windows programs do. If you attach a file called foo.xyzzy to an email, and double-click it, you get a dialog that says “Windows cannot open this file”, and it lets you choose an application to open it with.

In our product, the user got a message box that said “There is no application associated with this file”, and nothing she could do about it. The original developer of the feature was the type to try for a while, then claim it couldn’t be done. He had marked the bug as fixed because he put up the alert telling the user they were screwed.

I remembered seeing tips on how to get the whole choreography right, so I started digging. Here’s a bit about the solution, and how I found it.

We’re writing in C#, using the .NET method System.Diagnostic.Process.Start to start a new process for the document. Reading the documentation makes clear that this method is a wrapper for Win32’s ShellExecute, which was a good start because ShellExecute was what I thought you needed to use for launching documents. Also, there’s a lot more chatter on the web about ShellExecute.

So I start searching with Google. This is one of those tricky searches (“Windows cannot open the file ShellExecute” or “ShellExecute unknown”, etc) where there aren’t enough unusual words to give you good results. I found Raymond Chen’s entry Simple things you can do with the ShellExecuteEx function, which looked good, and one commenter there even mentions my problem, but without a solution. The best thing about that post was the sample program. I built it and confirmed that a single call to ShellExecute does just what our Process.Start was doing. I had feared that I needed an unwrapped feature of ShellExecute, and would have to hack past the .NET wrapper. Now it looked like I didn’t have to.

Poking around some more, I found the name of the error returned from ShellExecute when there’s no association (SE_ERR_NOASSOC). That at least was another unusual word to search on. There’s a lot of code out there that calls ShellExecute and defines this symbol, but then just prints error messages, which didn’t help me. My code already did that.

When you want to find code these days, there’s only one place to go: Koders, the source code search engine. It’s fabulous for finding a chunk of code that uses a particular method or symbol. Searching there for ShellExecute SE_ERR_NOASSOC produced lots more code that calls ShellExecute and doesn’t do anything insightful with the error code.

In fact, there was nothing useful. But Raymond had been talking about ShellExecuteEx rather than plain-old ShellExecute. I don’t understand how they differ (they take the same structure as an argument), but what the heck, search on that. This produced a similar page of results, but the very last result had the answer. LaunchHandlers.cpp from the CvsGui project shows how to do it.

The answer turns out to be calling ShellExecute twice. The first time, call it with a verb of “open”, and tell it not to show any UI. If it returns SE_ERR_NOASSOC, then call it again with a verb of “openas”, and let it show a UI. You’ll get the familiar Windows interface.

Coding up this technique in my app produced: nothing different. It turns out I still had to wrestle with error code translation. The .NET method returns ERROR_NO_ASSOCIATION (1155) in NativeErrorCode rather than SE_ERR_NOASSOC (31). Whatever. Once I adapted to that, everything worked perfectly!

I went into the problem believing there was a solution where a previous developer had thrown up his hands. It was very satisfying to find the answer and to be able to make the app do just what I wanted it to do, rather than have to settle for a lame compromise.

Ah, west and wewaxation at wast!


Although my coding skills are a far cry from yours, I also find that believing that there is a solution -and the corollary, that Google can often find it- goes a long way towards a satisfactory resolution.
You're right about that. If I hadn't seen information about it in the dim past, I might have given up on this too.
See, this is the problem I have with you computer nerd types....

"I went into the problem believing there was a solution where a previous developer had thrown up his hands. It was very satisfying to find the answer and to be able to make the app do just what I wanted it to do, rather than have to settle for a lame compromise."

Congratulations, you made the app do just what you wanted it do. (Maybe Kevin can pat you on the back, if you are having trouble reaching around...).

How about what the user wanted it to do?

In the first case, the user is told:

I don't know which application to launch to open this file. It's up to you to figure it out.

In the second case, the user is told:

I don't know which application to launch to open this file. It's up to you to figure it out. By the way, here are a list of applications to try. One of them most likely will understand and open the file.No guarantees though. Give it a shot, and good luck. Oh, and if you pick the wrong one, not telling what the heck might happen...

Looks to me like you took a lame compromise and made it at best, marginally less lame. You also presented the user the opportunity to shoot herself in the foot, which in my experience, is almost never a good thing.

How about trying to figure out what the file is and launch the right app? I can point you to some links if you really want to solve the problem.... ;-)


P.S. Sorry if I'm cranky. I just read Damien's blog and he swore at me!!!
Ned, Kudla totally smote you.
Kudla, I can understand your point of view in general, but in this case, you're misguided. If there is no file association for the file, how on earth is my software going to "guess" what to open it with? An unassociated file is a difficult thing for the user to deal with, but trying to innovate in that area is just a waste of my time. My product will not succeed or fail based on whether I have solved this corner case better or worse than my competitor.

My goal here was to make sure the user wasn't stuck: if they encountered an unknown file extension, they'd be presented with an unsurprising UI that would let them find and launch an application. I managed that, and now Kubi behaves the same as Outlook in the same situation. Success.
I think kudla's confusion arises from a fundamental misunderstanding of what it is programmers do: we take stuff that sucks, and we rejigger it so it sucks less.

That's just how it works.
I think Kudla's right on, even if he did sort of come out of left field to pounce you. But then he's always coming out of left field.

The point is that the standard ui is still a shitty ui. Most home PCs have so much crap installed on them that application list is going to be a mile long. If I pick the wrong application, is it going to mangle the file, format my hard drive, or shave "kick me" into my cat? Engineers don't feel anxious when presented that ui, but lots of users get really panicked about stuff like that. And just because Word isn't the registered viewer of Wordperfect files files it doesn't mean my software shouldn't know it can view those files.
Ned, me, misguided!? Balding, overweight and somewhat less than handsome, maybe, but misguided? Now that hurts.

Of course there are things your application can do to try and understand unassociated files. Damien mentions one. That said, your assertion that it is not worth the effort to try and innovate in this space becase the potential benefit to your application in the marketplace simply isn't there, is dead on.

Your goal was to be as good as Outlook and you have certainly succeeded. It makes sense, for a Windows application, to present the same UI, when confronted with the same problem. My only point was that the "solution" still stinks. Damien understands this, because where he and I come from, being as good as Outlook ain't nearly good enough :-)

Ned, I think you may have taken my post too seriously. My main intent was simply to zing you in good fun. This doesn't always come across in print, as you can't see my boyish smile or my mischievous wink. Again, only Damien seems to truly understand me (scary that).

I have noticed more of your work related posts dealing with UI issues. All I can say is, hurrah for you Ned. My career has gone in the opposite direction. I now find myself confronted with scalability, performance and thread-safety issues I always assumed Russ would take care of. I guess it's good for a couple old war horses like us. You rock! Keep up the good work.

P.S. Hey Jeff, thanks for clearing up my confusion. I never understood what it is you programmers do. I always thought it involved slide rules, silicon chips and soldering irons (nice alliteration there, eh?)

P.S. Damien, thanks for having my back. You are the man!
Kudla, believe me, I know you well enough to take everything with a grain of salt. Perhaps my fault was in not responding graciously enough. Keep it coming! We old war horses have to stick together but also keep each other on our toes.
Found this solution (seemingly more elegant) on a chinese blog site:
ProcessStartInfo info = new ProcessStartInfo ();
info.FileName = "";
info.ErrorDialog = true;
Process.Start( info );

And it worked on my testing without double-launching, checking error codes, etc.
It actually works with Jared's (or rather the chinese guy's) solution. Seems to be the ProcessStartInfo.ErrorDialog showing the error instead of throwing an exception.
Thanks. It saved my hours.

Add a comment:

Ignore this:
Leave this empty:
Name is required. Either email or web are required. Email won't be displayed and I won't spam you. Your web site won't be indexed by search engines.
Don't put anything here:
Leave this empty:
Comment text is Markdown.