PatronusCharm Techblog

PatronusCharm 1.1 Beta Release - Midnight CST

December 31, 2009 – 12:53 am by Lord Ravenclaw

Long overdue, I am finally releasing PatronusCharm 1.1 Beta. This will soon be followed up by 1.1.1 with miscellaneous improvements and changes I felt were unimportant for the 1.1 release. The changes and new features are numerous, and I may forget plenty but here are some highlights.

Sphinx Search

Sphinx, as you may have seen from my post Sphinx Search: MySQL Fulltext Search Engine Replacement is a replacement for the search engine from PatronusCharm 1.0.1. It’s faster, finds more, and has a powerful language to allow for advanced searching. Most importantly, it allows for searching queries such as “HP”, or “SGA”. The search window has been completely redesigned as well as collapsible. Further improvements will be made here later.

Tagging

Tags have been improved in beta 1.1. Tags can now be multiple words. Improved browsing by tags has been added with a Tag Cloud. The tag cloud allows filtering by usage to get to the more commonly used tags. Clicking on any tag will bring up a list of stories that use that tag.

In addition to the tag cloud, a list of the 20 most common tags is built into the story edit/add dialogs, allowing easy toggling of some of the more common tags.

Story Pairings

At long last, stories can have their pairing(s) displayed. They will soon be searchable, as well as the ability to use custom names for crossovers and the like outside of the large database of canon Harry Potter characters.

And the story bit…
Characters2

The “main” (first) pairing is shown in the story bit, while the others are shown with a hover-over menu.

Document Multi-Uploader

Multiple file boxes have been added to the Document uploader to assist with fast uploads of new (or old) stories. Currently only 5 document uploads at a time can be done but will be expanded to allow as many as needed.
Documents

Google Gears Caching

New with PatronusCharm 1.1 Beta is the ability to utilize certain features of Google Gears. More information on Google Gears can be found in my post Google Gears: Offline Applications. Google Gears is used to cache the 80 or so JavaScript/CSS files and serve them up locally from your computer, ensuring a snappy user experience with JavaScript-based parts of the site.

Display Width

A much requested feature, and made before FanFiction.Net came up with theirs, display widths. By clicking the appropriate link, the width of the story is modified to some percent of the size of the screen for easier, more relaxed reading.

PDF Exports: They’re Back!

PDF exports are back and better than ever. After fixing resource issues (as well as restricting to currently logged in users), users can now export PDFs of a story or chapter. These PDFs are well formatted, much more so than in the past. PDF bookmarking of each chapter allows readers to find each chapter more easily. New chapters start on new pages, allowing for a much more e-book like format. Abuse of PDF exports via automated downloaders will result in banning.

Reviews

Paging

Reviews are now properly paginated in the User Control Panel as well as when displaying reviews from a story. Stories with many reviews will benefit from this

AJAX Reviews

The “Quick Review” at the bottom of each story is now driven by JavaScript AJAX, which is a much more fluid way of submitting reviews. Which leads into a new feature, review drafts.

Review Drafts

Review drafts allow a user to save a review for later. Typing into the box will automatically trigger PatronusCharm to save a copy of your current review, especially good for those who review as they go through the chapter, to prevent lost work due to browser crashes or the need to leave on short notice.

Styling Code

PatronusCharm now supports use of “bbcode” styling tags. These styling tags can be used in reviews. Full Review windows have a bbcode supported editor, as well as user biographies. BBcode tags can be used in Quick Review as well, manually. Supported tags include [url=http://...][/url] for links, [email][/email] for email addresses, [b][/b] for bold, [i][/i] for italics, and [color=#FFFFFF] [/color] for colors.

Member List

A list of members can now be seen by clicking “Member List”, it’s a sorted list of members which links to the profiles of each member. In a future release, this may be sortable by authors/not authors.

Contact Information

User profiles now have support for showing other contact information such as AIM, MSN, ICQ, Skype, and other forms of contact besides email.
Display Width

Other Improvements

Latest Updates

Latest Updates on the home page has been improved and sorted differently. Mousing over a story shows the details of the newest chapter.

FAQ

A more comprehensive FAQ has been added.

SSL

The ability for users to “force” use of SSL has been added. This can be changed in CP -> Settings -> Force SSL. All non-SSL PatronusCharm links will redirect to a SSL secured version of the site while logged in.

Atom Feed Improvements

Story feeds have the contents of chapters now, allowing one to read a story from the chapter feed.
Feeds are now more compliant, returning proper HTTP status codes to optimize use in RSS/Atom readers and allow for proper caching.

News Archive

A new improved news archive has been designed which pulls from the PatronusCharm blog.

Improved Registration

Registration has been improved and the JavaScript powering the backend username/email checks has been rewritten. Also, the CAPTCHA image verification has been changed to use RECAPTCHA.

Improved Performance

And last, but far from least, much of the backend of the site has been rewritten for even greater performance. Caching has been implemented on rarely changing parts of the site, and memcached has been used to improve performance even more on unavoidably slow parts of the site.

Posted in Development, New Features | No Comments »

1.0.2 Beta Out Soon(TM)

March 7, 2009 – 6:40 pm by Lord Ravenclaw

Well, my prediction of 1.0.2 Beta coming out months ago obviously didn’t stick.

Obviously.

I’ve been quite busy with life, but due to extra free time available, it’s possible and likely that 1.0.2 Beta will be rolled out within the week, though perhaps lacking some features will be a worthy addition of features and fixes before 1.0.3 Beta.

Stay tuned, PatronusCharm is coming back.

Posted in Commentary, Status | No Comments »

Google Gears: Offline Applications

July 28, 2008 – 10:30 am by Lord Ravenclaw

What is Google Gears?

As defined by the Gears website…

Gears is an open source browser extension that lets developers create web applications that can run offline. Gears provides three key features:

A local server, to cache and serve application resources (HTML, JavaScript, images, etc.) without needing to contact a server

A database, to store and access data from within the browser

A worker thread pool, to make web applications more responsive by performing expensive operations in the background

I recently discovered Google Gears from Wordpress. Wordpress mentioned that version 2.6 included a caching mechanism for the Dashboard by using the Gears browser plugin with the Javascript API. I was intrigued. After some short research, I realized the power of Gears and within a few hours implemented a working cache system to cache the some ~75 static files that PatronusCharm uses around the site including images, CSS, and Javascript. Gears works by caching these files locally and intercepting HTTP/HTTPS requests for these files and serving them. After implementing it I noticed it was noticeably snappier than before. But this is only stage one, and what will be rolled out with 1.0.2 due next weekend.

In 1.0.3, I’ll be rolling out stage two. Stage two will be a lightweight version of PatronusCharm. This version of PatronusCharm will cache the entire site with Gears. It will be especially designed to allow users to store and read stories offline using the familar interface of PatronusCharm. If all goes well, users will also be able to review each chapter while offline and have PatronusCharm Offline synchronize the reviews with to the server when connectivity is restored. At this time, it will also check for updates on each story.

Google Gears is pretty exciting, as while like every technology is blogged about for ages, this is one of the few I feel could really take off. I’ll for sure be taking advantage of its features. It’s pretty genius to not only provide a cache which intercepts HTTP/HTTPS requests, but access to SQLite databases, threaded javascript, threaded timers, threaded AJAX, and desktop icon helper code. It’s a step towards bringing web applications into the world of offline applications. I’m also quite excited to get this working on PatronusCharm, reading stories offline has long been a goal of mine. I never quite thought it’d be this easy to allow users to click a button and read stories offline however.

We’re not quite there yet, there’s still plenty to do, however 1.0.3 will be an exciting release, and I look forward to working on PatronusCharm Offline.

Posted in Development, New Features | No Comments »

Sphinx Search: MySQL Fulltext Search Engine Replacement

July 21, 2008 – 12:07 am by Lord Ravenclaw

For PatronusCharm 1.0.0 and 1.0.1, we’ve used the MySQL fulltext search engine. That’s a mouthful, right? MySQL is the database software PatronusCharm utilizes to store all of the data used in PatronusCharm. It stores users, stories, news articles, FAQ items, everything. As you well know, a search engine is needed to pile through this data. As PatronusCharm finds more and more stories to host on its pages, a good search engine is important to continue to allow users to search for the stories they’d like to read.

MySQL has functions to do this. Using what’s known as a fulltext index, it searches through content such as titles, summaries, and tags to return matches to the query. Unfortunately, MySQL fulltext searches also suck. Limited by features, and woefully undeveloped facilities, MySQL fulltext searches are not the best. I’ve attempted to make them better, developing my own search query off examples and tips from other sites, but I find it inadequate. As do you all, I’m sure. In order to search for words below 4 letters long, one must change it for ALL databases on the server, which leads to a ballooning of search index sizes and performance reduction. My query is complicated, and taxes the server. For those interested, it looks like this (including PHP):

// Our query is massive enough to be multiline.
$list_query = $this-&gt;database-&gt;sql_query("
SELECT SQL_CALC_FOUND_ROWS s.*, u.name,
( (1.3 * (MATCH(summary) AGAINST ('" . $search . "' " . $boolean . ")))
+ (1.1 * (MATCH(tags) AGAINST ('" . $search . "' " . $boolean . ")))
+ (0.6 * (MATCH(title) AGAINST ('" . $search . "' " . $boolean . ")))
) AS relevance
FROM stories s
LEFT JOIN users u USING(userid)
WHERE ( MATCH(summary, title, tags) AGAINST ('" . $search . "' " . $boolean . ") )
" . $cwhere . "
HAVING relevance &gt; 0
ORDER BY relevance DESC, updated DESC
LIMIT " . (($curpage - 1) * $perpage) . "," . ($curpage * $perpage)
);

I found a different solution. After searching, I found the Sphinx Fulltext Search Engine. It works as a replacement to the MySQL Fulltext Search Engine. Complete with a PHP API class, I designed a new search engine based around Sphinx, which gave me the features and flexibility to do whatever necessary to create a good search engine. What it does is crawl through each record in the database to be indexed and creates its own indexer files, eliminating the need for MySQL fulltext indexes (and thus increased performance). Then a daemon is run which the PHP API connects to and searches with, receiving results back from Sphinx.

After weeks of work a few months ago, I’m quite proud of it. It’s faster, lighter, does more, and finds more. One can search for “HP” or “SGA” and return results. “HP/RR” returns my story as the one with the greatest relevance. In all, I’m quite pleased with how it’s turned out.

I’ve enabled it on the devsite, if you’re curious to give it a go. I find it a much more elegant search, and far more reliable. Please, try it and comment back here with feedback. Be sure to tell me if you find any bugs…you all are far better at breaking it than I.

Posted in Development, New Features, Under the Hood | 6 Comments »

New Feature: Document Multi-Uploader

July 13, 2008 – 4:07 pm by Lord Ravenclaw

In the past, when uploading stories I’ve noticed how long it takes to upload stories. It’s even worse for authors who are posting previously written stories. Thus I’ve developed (to Shezza’s delight), a way to upload multiple documents. Due to the funky way Javascript reads the Document Object Model, which is the hierarchy of HTML tags that Javascript manipulates one cannot add/subtract the number of documents per upload like one can sorting types in the new search engine. When these tags are loaded into a page say by AJAX, it doesn’t reload the DOM and makes it difficult to manipulate, requiring many hacks. So, currently, it’s limited to five documents per page.

My plan was to have a single upload box which users could add to as required. When I figure out a better way to load pages in the control panel, it will change to a system of variable uploads to allow for easier uploads of older stories.

Posted in Development, New Features | 1 Comment »

XML Syndication

July 5, 2008 – 3:22 pm by Lord Ravenclaw

Syndication

XML Syndication is a means of distributing content to other websites and users in a quick and easy format. It can be used to syndicate summaries of content, all content, updates, anything the creator needs to be passed out in a simple to use format. These are commonly known as ‘feeds’. Most of you use them in some form or another. I’m sure most of you either have a feed reader, or use Firefox addons to do so, or even use iGoogle to load content from your favorite websites. PatronusCharm also has feeds. We use the Atom feed specification (as opposed to RSS) currently. Atom is a superior design, though not quite as widely used as RSS is. Most readers can handle both quite easily.

PatronusCharm uses feeds for a number of different aspects of the site.

Latest updates. Each time a story is updated, this feed is updated with the last ten updates.
Latest news. My news posts trigger a feed update.
Story feeds. Each time a specific story is updated a feed will be updated.
User feeds. When a user updates it updated their feed containing all their stories.

These are the four currently accessible types of feeds in use at PatronusCharm today. However, upcoming in 1.0.2 two new feeds will be appearing. First, a variation on user feeds which posts chapters (and a decent length excerpt) for use by XML feed readers. Second, a category feed, which will alert to updates on a specific category or a combination of a primary and secondary category, e.g. Action/Adventure, or Action/Adventure/Timetravel.

The four current feeds can be accessed as follows:

Latest update: http://www.patronuscharm.net/x/latest/
Latest news: http://www.patronuscharm.net/x/news/
Story feed: http://www.patronuscharm.net/x/story/<storyid>/
User feed: http://www.patronuscharm.net/x/user/<userid>/

I’m currently reworking how feeds work however, and user feeds are the first up. Firstly they will become more friendly, they will allow syntax such as /x/user/LordRavenclaw/ in 1.0.2. I’m also working on a way to ensure that XML feed readers see each successive update as a new entry, rather than a reordering. This way XML feed readers will alert one each time they update. This also goes for latest updates, as items having already appeared in the feed are considering updated, rather than new.

In general, feeds are being reworked. I wrote the class which generates the Atom feeds well over three years ago and it needs a great deal of restructuring. That’s my current project for PatronusCharm.

Posted in Development, New Features, Under the Hood | 2 Comments »

SSL and DLP: Little Known Secret

July 1, 2008 – 1:00 am by Lord Ravenclaw

Sniffing

I’m not sure about all of you, but I hate using public WiFi access points. Why? They’re quite insecure. You might ask yourself, “What does he mean by that?” Public WiFi access points are a hotspot (pun intended) for people looking to prey on others through packet sniffing. As you surf the internet, you’ll likely enter many passwords, either by web form, instant messenger application, email client, or any other number of applications. This data can be easily intercepted by other WiFi users by putting their WiFi cards into a mode known as promiscuous mode. In this mode the wireless card captures and logs all the packets it receives and is generally loaded into any number of programs for Windows, Mac, and Linux. Frequently this is used to break secured WiFi passwords by capturing parts of password packets sent to the base station. However, it’s more often used to capture data by other users on a WiFi network.

As a side note, this can be done by any network card wired or wireless, though generally wired networks use switches rather than hubs, which route data to the proper network port using a MAC address lookup table. If you’ve ever tried to get 100mbit out of your 10/100mbit hub between PCs, you notice that you can’t get the full speed from it as it frantically sends all of the data moving through every port to every other port (this is aside from the harddrive being unable to keep up) It’s a dumb repeater, whereas a switch will intelligently route the data and prevent data collisions.

Back to packet sniffing. I myself have done this in the past. Sitting at the airport for hours lends one to be quite bored. I’d frequently pull out my laptop and start capturing packets. Within minutes I’d have a couple email passwords, AIM passwords, and any number of other authentication tokens. Many times I’d find other security vulnerabilities, and even open shares.

What can you do to avoid this? Personally, what I do is use the DLP server as a proxy, and use iptables to block off all incoming traffic to my laptop. By using SSH (secured shell) in a certain mode, it acts as a SOCKS 5 proxy listening only on localhost. Those who know what SSH is know that SSH is generally used as a remote terminal, however in SOCKS 5 proxy mode it acts as a completely (or, as completely as one can be) secure proxy. This is my favorite method of securing my browsing from prying eyes around me. But frequently, one doesn’t have SSH access, or it’s not feasible. Maybe you just don’t want people prying in on what you’re browsing. However, this is somewhat slow, and I rarely feel like completely setting it up when all I plan to do is surf DLP.

I’m sure all of you who read this blog are going: “Get to the point, Raven.”

The Point

DLP, and PatronusCharm both have SSL URLs. It’s not a fact I’ve advertised greatly, but since I find it so useful, perhaps others will too. The following sites can be reached via SSL, which is an encrypted form of transporting data between a browser and webserver.

Your browser will most likely complain about the SSL certificate, which is expected. The SSL certificate is self-signed. That doesn’t mean that it’s any less secure, SSL is SSL. What determines the security is your browser and the server, generally Firefox will use 256-bit AES encryption with SSLv3 and TLSv1. IE7 also offers the same encryption, though IE6 only has 128-bit encryption. For DLP and PatronusCharm, this hardly matters. The point being is that no matter who the certificate comes from, it’s still encrypted. What SSL certificate brokers offer is a guarantee that they’ve verified who is behind the site. Depending on the certificate class, with 1 being the lowest and 3 being the highest (plus extended verification, which includes background checks), more verification of the entity behind the certificate and generally offer a certain amount of guarantee financially if the encryption fails or the entity is false.

Well, this started as a notification of DLP/PC both having SSL urls and it turned into a lesson on packet sniffing, SSL, and SSL certificates. In conclusion, use those URLs to secure your transactions with DLP. I know I’ll be using them while surfing from college this upcoming term.

Posted in Commentary, Security, Under the Hood | 1 Comment »

Current development: Memcache, Tagging

June 29, 2008 – 4:56 pm by Lord Ravenclaw

I, as a developer, have an insatiable urge to optimize, simplify, and speed up my code. There are developers out there that would have no qualms running a query like this every time somebody hit their site.

SELECT
(SELECT COUNT(userid) FROM users) AS users,
(SELECT COUNT(id) FROM reviews) AS reviews,
(SELECT COUNT(chapterid) FROM chapters) AS chapters,
COUNT(storyid) AS stories, COUNT(DISTINCT author) AS authors,
SUM(wordcount) AS total_words
FROM stories

As you may or may not know, this is the MySQL query PatronusCharm uses to collect statistics on the site as seen on the devsite index page. Considering our new hardware from the great server upgrade, this isn’t much, but as I said. I like to optimize. What I’ve done is I cache the template (with the Smarty Template Engine) for 15 minutes, and the first person to hit the index page after it expires regenerates it. I see little need to update it more frequently. Thus, I use memcache.

You might be asking, so what is Memcache? From the memcache site.

memcached is a high-performance, distributed memory object caching system, generic in nature, but intended for use in speeding up dynamic web applications by alleviating database load.

Danga Interactive developed memcached to enhance the speed of LiveJournal.com, a site which was already doing 20 million+ dynamic page views per day for 1 million users with a bunch of webservers and a bunch of database servers. memcached dropped the database load to almost nothing, yielding faster page load times for users, better resource utilization, and faster access to the databases on a memcache miss.

Memcache allows me to cache data in the RAM of the server (Which, is hardly used. We have 4GB worth in the server.) and get rid of expensive or useless queries. For example, the query which loads all of a user’s information including settings, hashed password, and other details used to be called each time a user logged into the site. Now it’s called once, and invalidated when a setting is changed or a few hours goes by.

Since beginning to use memcache, I’ve moved many queries over to memcache, especially queries which require further processing after collection, such as the taglist which is only updated every 15 minutes. After the tags are collected from the database, split apart, frequency-sorted, and duplicates removed it’s cached with memcache, and further hits to the page only generate font-weights, as the depth option dynamically filters this frequency-sorted array of tags.

The Smarty templating engine also uses memcache. While by default it uses the filesystem, I decided I’d like to consolidate my cache with memcache and I wrote a custom memcache handler for it. It’s now in use at the devsite. Whenever you see an entry such as “read:index_latest.tpl:noslash:00aa2bfa1d5e7500e48603468f9ab490″ that’s Smarty using the cache handler. The debug information is used to help me work out the bugs, which have in fact been worked out over the course of a few hours and judicious edits to the cache handler.

As memcache implementation, at least in the first initial stage to remove unnecessary load to the server, comes to an end, I begin to work on new features.

Tags aren’t a new feature. They came to be in 1.0.1, but there were flaws. The current implementation doesn’t allow for spaces in tags, making “Stargate Atlantis” into “Stargate”, “Atlantis”. This has been corrected in 1.0.2, but I’ve also added some interesting new features.

Common Tags (click to expand)

As seen above (older screenshot), the common tags feature allows one to add and remove tags to a story’s tags. My hope is that by giving users a list of 20-25 common tags, they’ll be less inclined to use strange tags which end up cluttering the tag cloud taglist. The common list finds and crosses off already used tags, which clicking again will remove. I’m hoping this will go a long way towards helping users categorize their own stories, rather than being bound by my two-category system. This feature can be currently found in a broken state on the devsite if you wish to explore it. It’ll add tags, but not remove, and incorrectly finds tags (such as crossing off Harry Potter when Harry was the only tag used).

More development notes to come. In the next post I’ll talk about the new moderation system which directly ties into the event log and other neat features soon to come.

Posted in Development, New Features, Under the Hood | 4 Comments »