Making multiple Mbox files look like one virtual folder
The problem with standard email storage methods is that they don't scale well. Mbox, the default in Thunderbird, creates large files, and if you backup changed files, every backup requires recopying a multi-gigabyte file, because it has changed when even one message was added or deleted. Maildir, the alternative, puts each message in a separate file. But file systems have a lot of overhead and their own limits. When you have hundreds of thousands of messages, especially on a PC, that clutters the computer's file system, and between the file cluster size and directory overhead, it wastes a lot of space and slows down other operations. Two bad options. And let's not be too snarky about what that company from Redmond does in its awful mail system, with a giant proprietary mail database file that sometimes gets corrupted...
It is obviously a lot to ask to create a new mail file format that scales better. It would not be standard and could break some things anyway, though in the long run it might be a good idea. But I have an idea that might work for Thunderbird. As it stands now, it is possible to create archive files by month or year. Then new mail doesn't require those to be re-copied. WHAT IF a "virtual mailbox" could be created that, instead of consisting of one Mbox file or one maildir directory, consisted of a set of Mbox files treated by the UI as one? Quick Filter would filter them all as one, unlike with archives, and you could scroll through them all and sort them and see threads, but they'd be stored as monthly or annual (or whatever) files, the way archives are. Sending and receiving would be unchanged, using the current folders, but periodically the folder could have the oldest ones moved out and still made part of a virtual folder.
There are some modest virtual folder capabilities now, but I don't see how to do this. Does it seem like a possibility with what's there, or a modest enhancement, or is that the kind of thing that would be a huge effort?
All Replies (4)
re :the default in Thunderbird, creates large files,
As an example: If you choose to keep all your incoming emails only in the Inbox folder then you will eventually have an enormous Inbox mbox file which is rather risky and is like putting all your emails in a single basket. After all it is just a single text file. Hence it is advised you create additional folder or subfolders to organise and store emails, then the mbox files may be many but they are smaller in size.
Also, whenever you move or delete any emails from eg: Inbox folder you will find Thunderbird marks them as deleted and they are hidden from view, but in reality there is still the original copy in the 'Inbox' mbox file. You need to periodically 'compact' the Inbox to completely remove old data and this also reduces the file size. Note: compacting is not the same as compressing.
No, Toad, I already sort the mail into folders. But those get big. The problem is that if I split a subject folder in time (archiving) in order to make the current one smaller, I lose visibility of the older mail -- the stuff in the not-latest folder -- when I quick filter or expression search (that's nice add-on) the latest mail folder, or try to follow a topic that goes back in time a ways. I don't know if the regular "search" function works on archived messages, but it's such a terrible function (1994 called and wants its search back) that it is not a useful substitute for quick filter. Hence I would like a way to have those smaller folder files but make several of them (per topic, not the whole collection) look like one bigger one.
The closest analogy I can think of now -- not relevant but as an example -- is unified inbox, where more than one account's inbox is shown as one. I don't use that because I use POP and filter on various parameters into the folder layout I want but I think it's in T'bird (and in my phone's mail program). Here I'm asking about unifying a folder (not just "inbox") out of multiple (older mail) sub-folders. Perhaps the unified inbox code could contribute to that?
Sounds like you might find the 'Edit' > 'Find' > 'Search Messages' option as the good option. That option is on the 'Menu Bar' toolbar to enable 'Menu app icon' > 'View' > Toolbars' select 'Menu Bar'
or use the keyboard shortcut keys : 'Ctrl+Shift+F'
- Search for messages in :
click on drop down to see mail account name - hover over and it will show a drop down of folders etc and at the top it will say the mail account name. If you want to search only on let's say two specific folders, then select on one of them at this point (you can add another later) or select the actual mail account name to search on all folders in the account.
- Select check 'Search subfolders'
- Select 'Match any of the following'
If you want to search specifically for emails on a known Subject then select:
- Subject, contains and type text
If you want to include more than one Subject then clickon small + sign to create a new line
- Subject, contains and type text
Maybe you want to search on a couple of email addresses, then do the above but using:
- From, contains and type email address
If you are looking for a specific email, but only one specific topic from that person:
- Select 'Match all of the following', so both conditions must be true
- From, contains and type email address
- click on small + sign to create a new line
- Subject, contains and type text
Click on 'Search'
Results will appear below.
- Then click on the button 'Save as Search Folder'
- Name: Give it a suitable name.
- Create as subfolder of: Choose where folder should be - either select the mail name to set as a top level folder or select a folder name to set as a subfolder.
At this point - if previously you had just selected one folder then now is the time when you can add more folders to the search.
- Click on 'Choose...' button
- Select folders by selecting checkboxes
- click on 'OK'
- Click on 'Create'
Folder will now display in Folder PAne as a Search folder - it's a type of virtual folder and should get auto updated. If you ever need to alter anything in that search folder : Right click on search folder in Folder Pane and select 'Properties' It will get you back to where you can select different folders maybe to add another fodler to the search or remove one or maybe modifiy the search parameters.
I find this option really useful and I do not have to delete that search folder just to maniplate the search.
What you've pointed to is interesting, since on the one hand it demonstrates that there is code somewhere in T'bird that creates and displays virtual files, but on the other hand it is tied to one of the search functions. And that search function, while powerful, is very clumsy -- it takes a lot of precise mouse movement to go down those menus and find the folder. Contrast to, say, Quick Filter, where you just type into the box that's already there.
Now I realize that those virtual search folders are more permanent, which is good, which is why I think the virtual folder mechanism is probably hiding in plain sight. But because it's hidden behind search, it is only there after you've gone through the folders you want to include in the virtual folder, one by one, and add them. Yes, you can search subfolders -- but that doesn't work for the purpose I'm talking about, because the mechanism for keeping folders small, Archiving, puts them below Archive at the top level of Local Folders (I keep everything in local folders, as I don't own my own server) rather than below the parent (new incoming filtered mail) folder. And that's good, because if I had to waste space on the left side of the screen with folder-its archive rather than just folder with archives below the list, the folder list wouldn't fit! And I don't want to have to scroll the folder list.
I wonder if some clever programmer could come up with an add-on that created a set of virtual folders, if they could be displayed contiguously, by automating that search of each desired parent folder and its collection of archives. I mean this could also be an enhancement that MZLA did, but they seem to have other priorities.