Dan Wood: The Eponymous Weblog (Archives)

I recently added an on-disk data cache in order to speeed up a slow aspect of my application. I've come up with caching solutions before, but I thought I'd take a fresh look at the problem.

If you look in the contents of ~/Library/Caches/ you will see that a lot of applications have some sort of cache in there. It turns out that many of them are actually just generated by NSURLRequest.

Figuring that Apple probably has given some thought to cache file organization, I took a close look at the structure of a typical application's cache folder.

A typical cache file sub-path is something like this:

13/02/4281199315-1410279211.cache

It appears that the first two directories are given a name of decimal 00 through 15; the actual file name is comprised of two decimal thirty-two-bit unsigned integers.

Therefore, each cache entry is indexed by, essentially, 2^72 or a nine-byte number. A hash function that reduces to 72 bits is obviously going to have a fair number of collisions, so the cache file needs to store the source URL in the file itself. I'm guessing that a cache file of this type stores multiple data for each hash value, rather than just punting if a hash doesn't match, but I haven't tried digging into the binary format to confirm this.

What I did for my needs was to start out similarly to Apple's approach, creating two levels of directories based on the first eight bits of the hash value. I'm not that familiar with the intracacies of file systems, but I am guessing that Apple used this technique for a reason. (If any reader has any further insight on this, I'd be curious to learn more about it in the comments.)

For the hashing, I decided to use an SHA1 hash on the source URL, which hashes to 20 bytes (160 bits), way less likely to have a collision than Apple's 72-bit hash. (So much so, in fact, that I just didn't deal with the possibility of collision; it's not that important. I just store the actual binary data in the cache file.) The first byte of the hash determines the two enclosing folders, the last nineteen bytes get converted to hexadecimal for the long file name. So a typical path looks like this:

05/07/f1e53115eb37b4e4df9e1879bc530daa65666e.tiff

I'm not married to this particular approach; in fact it's not being used in any production code yet. I'm curious if any readers have come up with similar or different approaches, or even found (or written) a nice API around the notion of cache files. I'm happy to throw away this technique if somebody else has come up with a better solution!

Update: Just to clarify, the reason I'm coming up with my own caching scheme is that the data that I want to cache do not come from URL requests. They are generated thumbnails of images and quicktime movies. It would be great if apple exposed something that would allow me to use their system but with my own data.

I'm a big fan of Core Image; I blogged (1, 2) about quite a while ago. And I like NSImage for its simplicity. But only recently have I been introduced to Core Graphics. I wish I had been aware of its coolness much earlier!

This came up recently when I met Dan of Stunt Software recently at Macworld Expo. He has an application that shows thumbnails of images. He mentioned that he had tried Epeg, which Sandvox has been using to quickly generate thumbnails from JPEG images, but found that Core Graphics has a much faster technique: specifically the function CGImageSourceCreateThumbnailAtIndex.

I started to look into this, and came across a blog post by Wil Shipley that mentioned how fast CG was at doing thumbnails. I had actually read this, but didn't pay too much attention back then! I probably avoided CG because it doesn't have a nice Cocoa-like interface to it. My loss!

I bit the bullet and started to convert some of our code, starting with the thumbnailing in the media browser, to use this technique. Wow, what a speedup! And I actually can go further: all I'm doing now is building the thumbnail and then converting the CGImageRef to an NSImage for MUPhotoView to draw. But I can go further if I modify MUPhotoView to natively deal with the CGImageRef instead of NSImage. That will be my next step, but it will be a bit more work.

Here are some snippets of code (Tiger-only). The first is to read from a file and create a thumbnail.

NSURL *url = [NSURL fileURLWithPath:imagePath];
CGImageSourceRef source = CGImageSourceCreateWithURL((CFURLRef)url,NULL);
if (source)
{
  NSDictionary* thumbOpts = [NSDictionary dictionaryWithObjectsAndKeys:
  (id)kCFBooleanTrue, (id)kCGImageSourceCreateThumbnailWithTransform,
  (id)kCFBooleanTrue, (id)kCGImageSourceCreateThumbnailFromImageIfAbsent,
  [NSNumber numberWithInt:MAX_THUMB_SIZE],
        (id)kCGImageSourceThumbnailMaxPixelSize, 
    nil];
  
  CGImageRef theCGImage = CGImageSourceCreateThumbnailAtIndex(
    source, 0, (CFDictionaryRef)thumbOpts);
   ...
    CFRelease(theCGImage);
  }
  CFRelease(source);
}

To convert the CGImage to an NSImage, I used this code. I had to flip the image, in my situation, to make the image draw properly.

NSRect imageRect = NSZeroRect;
CGContextRef imageContext = nil;

imageRect.size.height = CGImageGetHeight(theCGImage);
imageRect.size.width = CGImageGetWidth(theCGImage);

img = [[[NSImage alloc] initWithSize:imageRect.size] autorelease];
[img setFlipped:YES];
[img lockFocus];
imageContext = (CGContextRef)
    [[NSGraphicsContext currentContext] graphicsPort];
CGContextDrawImage(imageContext, *(CGRect*)&imageRect, theCGImage);
[img unlockFocus];

I hadn't heard of this Screen Mimic before today, but the timing is perfect — I was just wishing for an application like this yesterday. I am often in the need to make up a quick little screencast (of course not as polished as the Sandvox demo) with live voice recording, so I can explain what I'm doing on my Mac.

This is a really slick, useful program. I can record a fixed area or have the window follow my mouse. My clicks and my drags can be superimposed (with a little circle) so it's clearer what I'm doing. (Maybe it needs the ability to also indicate that with sound effects!)

What's really cool is that its author, Lee Falin, was inspired to become an indie Mac developer after hearing a panel that I put together at an O'Reilly Mac OS X conference. (Oliver Breidenbach mentioned this today over on MacDevCenter.)

I like to work down at a nearby café one day a week or so, just for a change of pace, and to try to emulate the behavior of famous software companies. :-) Recently I started chatting with another patron there when I noticed that he stopped bringing in a Windows laptop and was now using a Mac. I learned his first name, and over the last couple of months, we've chatted at bit when we run into each other there.

Meanwhile, I've been corresponding with a fellow Alameda resident over the last couple of months; he has started up a local political group that I've been hoping to get involved with — but haven't yet.

Yesterday I discovered that those two people — the guy at the café and the guy I've been exchanging emails with — are the same person!

What a strange thing to happen, to have to now merge two "people" (in my mental model of the world) into one!

I recently had the occasion to create a number of cocoa classes. These ones were not using Core Data, but I thought it would be cool to make use of XCode's modeling tools and build up my objects as a Core Data model.

To accomplish this, I first ran across Jonathan 'Wolf' Rentzsch's mogenerator (installing by following this excellent tutorial).

mogenerator is little program that analyzes your data model and spits out classes (in two halves, a machine-generated part and a developer-modifiable part). But the classes that are generated are intended to be NSManagedObject subclasses, which I didn't want.

So I rebuilt the template files (which you install into ~/Library/Application Support/mogenerator/) to build instance variables, more standard accessors, an init and dealloc method, and so forth.

These files are available for download here. If you use it, you might want to modify these further, to change the accessor styles, replace NSMutableSet with NSMutableArray, and so forth. But it might be a useful start for your project.

OK, just kidding. This post is actually a recap of last week's Macworld Expo activity.

The headline is a joke that I came up with at the "Mac Small Business" gathering last Wednesday. Fellow developers and the few Apple employees there appreciated the idea that maybe if Karelia software announced a developer's kit, then Apple would be quick to get one released. :-)

The iPhone-lacking-a-developer's kit issue has become a big issue on the internets; I'm glad I'm not alone in wishing that one could develop software for it. That's what I do for a living, and the iPhone is a droolworthy mini-micro-computer that is just begging for third-party add-ons.

In any case, the iPhone may have been big news, but it only a tiny part of my week. I arrived at the Expo on Wednesday; immediately before entering I bumped into StuFF mc of Pomcast; he had interviewed us a while back about Sandvox. I then spent most of the day wandering around the booths. One of the best activities on the show floor is running into old friends, many from my days working with BMUG in the early days of the Macintosh. I checked out the booths run by developers whom I know; many like MemoryMiner were at small "pod" booths and a few, like Plasq, had a larger setup. (I find that the large booths run by the big companies are usually much less interesting than to the smaller ones run by small companies!)

Wednesday evening, there were three events I wanted to go to. I was scheduled to do a demonstration of Sandvox at a BoF (Birds-of-a-Feather) gathering hosted by Dori Smith. Alas, the video projection equipment had been taken away. So I headed over to the aforementioned Mac Small Business gathering, which was a great opportunity to connect and re-connect with fellow developers who also wear the business-owner hat. One of my more interesting conversations was with Paul of Rogue Amoeba, who had organized the very successful MacSanta event a few weeks ago. After that petered out, I went literally around the corner to another restaurant to spend some time with some other friends who had a big gathering there.

Friday, I headed back into San Francisco for more fun. I had a lunch meeting scheduled with a well-known User Interface Guru from Apple to pick his brain and get ideas for the UI of the next generation of our application. We got a bit sidetracked, and instead he picked my brain about how can Apple's developer program be improved, including its website and developer conference. It was fun to provide constructive feedback and brainstorm some ideas. We ran out of time before we could talk about User Interface, so we'll have to set up another meeting down the road.

After lunch, I wandered around some of the "pods" that I had missed on Wednesday. On Brent's recommendation, I visited Dan of Stunt Software. Some very nice stuff from a small shop! I also stumbled upon a cool-looking offsite backup utility called CrashPlan when Tim beckoned me over. This is an application that looks like it is going somewhere (and will probably double its usefulness when Leopard, with its "Time Machine" capabilities, is available).

Overall, it was a good expo, and even though the SteveNote didn't have any Macintosh content to it, it's clear from the expo floor that the Macintosh software industry is alive and well.

Yes, that's right. YAAAWWWWWNNN to the iPhone.

Sure, it has nice eye candy, and you can do a lot with it. But it isn't that different from a lot of the other over-functional cellphone/PDA/Camera combinations out there as far as what it can do.

What would set it apart would be if one could develop software for it in Cocoa. But apparently that's not in Apple's plans. From Macintouch:

The only two iPhones at the show were under glass, and Apple representatives said it is a "closed platform", refusing even to identify the specific processor it uses, and there's apparently no developer kit for it, though "developers who want to do applications [for the iPhone] are welcome to contact Apple developer relations."

If they are really going to make this a closed-platform appliance, then I really do understand why they are not longer "Apple Computer, Inc." It would be the nail on the coffin for Mac OS X as a computer platform. If they are not going to open up the iPhone for development, then they might as well take the next logical step and release their next set of iMacs to also be appliances that one cannot develop software for. With iLife and Dashboard pre-installed, who needs anything else? Though this is a bit of a hyperbole, that seems to be the attitude.

Me? I'm going to be contacting Apple developer relations and giving them my perspective. Perhaps I'm not the only developer who would like to come up with some cool software for this little gadget, and if enough people express their interest in writing software for the device, maybe Apple will change their tune and we'll be learning all about it at WWDC 2007.

Update: Wolf suggests we file a bug report. Good plan.

In a few places in Sandvox, we want the user to be able to specify his/her email address. So we make this easy by collecting a number of possible email addresses they might use and presenting them in a combo box. The user can then pick one of those, or hand-enter a different address if desired.

For today's episode, I thought I'd share the logic we use for collecting those addresses. We collect the possible addresses from three places, adding the addresses to a mutable set. We also try to determine the primary address, meaning the address most likely to be the one the user will choose.

NSMutableSet *emails = [NSMutableSet set];
NSString *primaryAddress = nil;

First off, we look in the address book's "me" card. The first address listed there will probably be the primary address.

ABMultiValue *meAddresses = [[[ABAddressBook sharedAddressBook] me]
    valueForProperty:kABEmailProperty];
int i;
for (i = 0 ; i < [meAddresses count] ; i++)
{
    NSString *thisAddress = [meAddresses valueAtIndex:i];
    [emails addObject:thisAddress];
    if (nil == primaryAddress)
    {
        primaryAddress = thisAddress;
    }
}

Next, we look in Apple Mail for some more addresses you might use, and merge them into the set. We look in all the active accounts and get the address or addresses associated with each. If we don't have a primary address yet, set it from the first address you find here.

NSMutableArray* mailAccounts
    = (NSMutableArray*) CFPreferencesCopyAppValue(
        (CFStringRef) @"MailAccounts",
        (CFStringRef) @"com.apple.mail");
[mailAccounts autorelease];
NSEnumerator *theEnum = [mailAccounts objectEnumerator];
NSDictionary *accountDict;

while (nil != (accountDict = [theEnum nextObject]) )
{
    BOOL isActive = YES;
    NSString *isActiveValue = [accountDict objectForKey:@"IsActive"];
    if (nil != isActiveValue)
    {
        isActive = [isActiveValue intValue];
    }
    if (isActive)
    {
        NSArray *emailAddresses
            = [accountDict objectForKey:@"EmailAddresses"];
        if ([emailAddresses count])
        {
            [emails addObjectsFromArray:emailAddresses];
            if (nil == primaryAddress)
            {
                primaryAddress = [emailAddresses objectAtIndex:0];
            }
        }
    }
}

The last place to look is your .mac account. If you have a .mac account, then you have a mac.com email address.

NSString *iToolsMember = [[NSUserDefaults standardUserDefaults]
    objectForKey:@"iToolsMember"];
if (nil != iToolsMember)
{
    NSString *iToolsAddr = [NSString stringWithFormat:
        @"%@@mac.com",iToolsMember];
    [emails addObject:iToolsAddr];
    if (nil == primaryAddress)
    {
        primaryAddress = iToolsAddr;
    }
}

That's all that you need. I suppose if I wanted I could also look in account lists for Entourage, Eudora, etc. but I don't have that information, and it might not be worth the trouble.

OK, it doesn't involve any hard-drive swapping or deal-making with Apple. Here's how I've upgraded to a 40 GB iPod -- I bought a new iPod, and I'm selling my 30GB iPod on eBay.

Actually, what I'm selling is almost completely new -- I had taken in my 30Gigger into the Apple Store because it was having a few problems (e.g. it was narcoleptic, falling asleep while playing music, even while docked). They swapped it for a brand new unit. But since it was so new, I thought this was a good time to "upgrade" to 40 GB, so I could sell the almost-new 30-gigger.

And it still doesn't hold my entire music collection. (I was an eMusic member for a while; it's amazing how much you can accumulate!)

If anybody wants to buy this iPod, which has all the accessories still wrapped up, since I just swapped them with the new stuff that came with the 40-gigger, I'll throw in a license to Watson to the lucky winner!

I've had wonderful success so far with using this blog as a way to reach out to the Mac community to look for contractors and partners I can work with on Karelia projects.

So as my first post for the new year, I wanted to put out some feelers to see if there is somebody who would be interesting in help out Karelia by doing first-line technical support for Sandvox: reading incoming inquiries, responding if needed, helping users troubleshoot or find the answers in the online help, assigning as bug reports if needed, and so forth. This would be a part-time task, with compensation structure negotiable.

Obviously, somebody who is familiar with Sandvox, and the Mac in general, is going to be best qualified, also good people skills (over email, possibly chat) is important as well.

If interested, please email a mac-focused résumé and cover email to me. The best way to get past the spam filters is to send it to dan____ (where the blanks are replaced by the current year -- happy new year, of course) at the domain of karelia.com ....