Rethinking Sup

It's been clear to me for a while now that Sup has been trying to be two very different things at once, thus pleasing no one and irritating everyone. There's Sup the email client, which is kind of the standard view of things. And then there's Sup the service: a threaded, fielded, searchable, labelable view into your email.

Sup the email client is lacking in many ways, as many people have been very quick to point out to me. The most obvious of these is that it refuses to actually, you know, actually write back any state to your mailstore. Specifically, read/unread state is never written anywhere except its internal index. Furthermore, mailstore rescans of most any type are incredibly slow. These two features make using it in conjunction with other clients near impossible, which pretty much breaks one of the primary principles of tool design: don't break other tools. (Then there's also the problem of IMAP connections being terrifically slow and prone to crashes, but I lay most of that blame on IMAP being a crappy protocol and the Ruby IMAP libraries leaving a lot to be desired.)

Sup the service, on the other hand, suffers from the rather obvious flaw of not being exposed in any manner other than through Sup itself (and irb, I suppose).

I think the reason for this bizarre situation stems from my goal of fusing two very different things together: mutt and Gmail. Mutt is a client; Gmail is a service; Sup cherry-picks functionality, and lack of functionality, from both. Examples: I refused to have Sup write back to mailstores because Gmail didn't have to export to your local Maildir or mbox file, so why should I? (Well technically, I said I would accept patches that did that, but that I wouldn't be working on that feature myself. A fine distinction!) At the same time, I pooh-poohed the notion of a Sup server because mutt didn't have a server, and so why should Sup? And so on.

For Sup to evolve into something more useful than it is, and that appeals to a broader audience than it currently does, I believe it has to go down one of these routes completely. And I believe I know which one, and I believe this can be done without compromising the basic user experience, which I would be very reluctant to do because it has been lovingly tweaked over the years to be William's Ideal Email Experience.

The first option is to make Sup more of a client. In order to be a real email client, Sup must be able to interoperate with other clients. This means it has to write back all its state to the mailstores: read/unread status in whatever manner the mailstore supports, and probably something like all labels in a special header. It must also be able to do a full rescan in a fast manner, so that changes by other clients are reflected.

Right off the bat, that seems impossible, redundant with other software, and not that interesting. As I wrote in a sup-talk thread from a few months ago:
Sup is never going to be able to compete with programs like Mutt in terms of operations like "open up a mailstore of some format X, and mark a bunch of messages as read, and move a bunch of messages to this other mailstore." That's a tremendous amount of work to get right, get safe and get fast, and Mutt's already done it well, and I sure don't want to have to reimplement it.
Competing with mutt on grounds of speed, stability, and breadth of Mailstore usage is a recipe for fail. Ruby sure as shit ain't gonna come close to C for speed (at least until Rubinius gets LLVM working), and mutt's already hammered out all the quirkinesses with Exchange, etc.

But not only would it be impossible, it wouldn't be interesting. The things that make Sup valuable are the UI, the indexing and the flags, and those simple don't translate to external mailstores. Furthermore, Sup is aimed at the mailstores of the future (my present mailstores), which are so big that mutt can't handle them anyways.

So that leaves Sup as a service. And that's where things get interesting. But I'll save that for a later post.

10 comments:

John said...

"I believe this can be done without compromising the basic user experience, which I would be very reluctant to do because it has been lovingly tweaked over the years to be William's Ideal Email Experience."

I love sup and I'm not bothered by any of the problems you describe so this whole thread just seems philosophical and worrisome to me. But I'm reassured that you think you can do whatever without breaking the Ideal Email Experience. Thanks again for sup!

Anonymous said...

I'm not sure not being able to move mails between stores and setting state is such a big deal. But what is a showstopper, is that sup probably doesn't save the mails I send anywhere?

William said...

Anonymous: Sup does save a copy of your send emails. Look in ~/.sup/sent.mbox.

Moving mails around between mailstores and setting state is a frequent request, because people (rightly) expect Sup to behave like a regular MUA like mutt.

Anonymous said...

"Furthermore, Sup is aimed at the mailstores of the future (my present mailstores), which are so big that mutt can't handle them anyways."

Out of curiosity, what are your mailstores? I ask because I've had to abandon sup and go back to mutt, because anything over 50k messages, spread throughout several Maildir mailstores, caused a LOT of instability and weird behavior in sup, while mutt seems to handle everything without a hiccup.

Of course, I'm now growing ever-farther behind on keeping up with my mail, since mutt makes it too easy to ignore things, but I had to make the trade-off of "works" vs "global view".

William said...

"Out of curiosity, what are your mailstores? I ask because I've had to abandon sup and go back to mutt, because anything over 50k messages, spread throughout several Maildir mailstores, caused a LOT of instability and weird behavior in sup, while mutt seems to handle everything without a hiccup."

I've always used mbox files. The only instability I've seen is Ferret's inevitable index corruption.

I don't use Maildir, but I know a lot of people do. I'm surprised that you had so much trouble because AFAIK they are all happy with it.

At any rate, I am in the midst of writing "Sup the Service" which should fix these issues in that *I* will control how things are stored on disk. :)

ryan said...

This makes a lot of sense. s the follow-up to this post on the way? Have you made any concrete decisions about sup's future?

I ask because I've been meaning to try sup, and possibly switch over, for a while now. This post stopped me in my tracks, though, since it implies that sup's future probably isn't its current incarnation as a mail client.

Switching mail clients is a big thing. It's at least a little traumatic for most people, including me. If I'm going to do it, I'd like to know that I'm not going to get stranded and have to switch again later.

William said...

"This makes a lot of sense. s the follow-up to this post on the way? Have you made any concrete decisions about sup's future?"

The followup post is on the way, and Sup's future is pretty concrete. I went on vacation last week and spent a lot of time hacking on the foundation, so progress is progressing.

"Switching mail clients is a big thing. It's at least a little traumatic for most people, including me. If I'm going to do it, I'd like to know that I'm not going to get stranded and have to switch again later."

I understand completely. FWIW, there will be an easy upgrade path to the new Sup, and the interface will be the same, and it will be possible to get your email back out of Sup if you decide it ain't for you.

Adrian Quark said...

I like sup's UI and focus on indexing/tagging over folders, but the idea of sup storing my mail in some opaque format is a huge turn-off. I've had to move my mail between clients too many times to trust it to anything but a dead-simple plaintext format like Maildir or mbox. If I wanted to be tied to a specific platform, I'd just use GMail.

William said...

"I like sup's UI and focus on indexing/tagging over folders, but the idea of sup storing my mail in some opaque format is a huge turn-off."

Sadly, that's the only way I can reasonably support the kinds of things
that I want Sup to do. Sup has never really supported mbox, Maildir, etc. very well, because it just wasn't able to handle the concurrent modifications files in these formats undergo when there's more than one client involved.
With the addition of non-email document types, a custom format kinda makes sense.

If it makes you feel any better, there will be tools to extract content from Sup into standard formats.

madduck said...

I sincerely hope that you will rewrite sup so that tags and all the other metadata you can attach to messages will be IMAP-synchronisable. Many of us use two or more computers, and it would be useless if we had to maintain tags independently on two machines. In fact, it would send us right back into POP2/POP3 land.

Blog Archive