PatronusCharm Techblog

Current development: Memcache, Tagging

I, as a developer, have an insatiable urge to optimize, simplify, and speed up my code. There are developers out there that would have no qualms running a query like this every time somebody hit their site.

SELECT
(SELECT COUNT(userid) FROM users) AS users,
(SELECT COUNT(id) FROM reviews) AS reviews,
(SELECT COUNT(chapterid) FROM chapters) AS chapters,
COUNT(storyid) AS stories, COUNT(DISTINCT author) AS authors,
SUM(wordcount) AS total_words
FROM stories

As you may or may not know, this is the MySQL query PatronusCharm uses to collect statistics on the site as seen on the devsite index page. Considering our new hardware from the great server upgrade, this isn’t much, but as I said. I like to optimize. What I’ve done is I cache the template (with the Smarty Template Engine) for 15 minutes, and the first person to hit the index page after it expires regenerates it. I see little need to update it more frequently. Thus, I use memcache.

You might be asking, so what is Memcache? From the memcache site.

memcached is a high-performance, distributed memory object caching system, generic in nature, but intended for use in speeding up dynamic web applications by alleviating database load.

Danga Interactive developed memcached to enhance the speed of LiveJournal.com, a site which was already doing 20 million+ dynamic page views per day for 1 million users with a bunch of webservers and a bunch of database servers. memcached dropped the database load to almost nothing, yielding faster page load times for users, better resource utilization, and faster access to the databases on a memcache miss.

Memcache allows me to cache data in the RAM of the server (Which, is hardly used. We have 4GB worth in the server.) and get rid of expensive or useless queries. For example, the query which loads all of a user’s information including settings, hashed password, and other details used to be called each time a user logged into the site. Now it’s called once, and invalidated when a setting is changed or a few hours goes by.

Since beginning to use memcache, I’ve moved many queries over to memcache, especially queries which require further processing after collection, such as the taglist which is only updated every 15 minutes. After the tags are collected from the database, split apart, frequency-sorted, and duplicates removed it’s cached with memcache, and further hits to the page only generate font-weights, as the depth option dynamically filters this frequency-sorted array of tags.

The Smarty templating engine also uses memcache. While by default it uses the filesystem, I decided I’d like to consolidate my cache with memcache and I wrote a custom memcache handler for it. It’s now in use at the devsite. Whenever you see an entry such as “read:index_latest.tpl:noslash:00aa2bfa1d5e7500e48603468f9ab490″ that’s Smarty using the cache handler. The debug information is used to help me work out the bugs, which have in fact been worked out over the course of a few hours and judicious edits to the cache handler.

As memcache implementation, at least in the first initial stage to remove unnecessary load to the server, comes to an end, I begin to work on new features.

Tags aren’t a new feature. They came to be in 1.0.1, but there were flaws. The current implementation doesn’t allow for spaces in tags, making “Stargate Atlantis” into “Stargate”, “Atlantis”. This has been corrected in 1.0.2, but I’ve also added some interesting new features.

Common Tags (click to expand)

As seen above (older screenshot), the common tags feature allows one to add and remove tags to a story’s tags. My hope is that by giving users a list of 20-25 common tags, they’ll be less inclined to use strange tags which end up cluttering the tag cloud taglist. The common list finds and crosses off already used tags, which clicking again will remove. I’m hoping this will go a long way towards helping users categorize their own stories, rather than being bound by my two-category system. This feature can be currently found in a broken state on the devsite if you wish to explore it. It’ll add tags, but not remove, and incorrectly finds tags (such as crossing off Harry Potter when Harry was the only tag used).

More development notes to come. In the next post I’ll talk about the new moderation system which directly ties into the event log and other neat features soon to come.