How The Farrago Works

Recent changes

Table of contents

Links to this page

FRONT PAGE / INDEX

Subscribe!

My latest posts can be found here:

Colins Blog

Previous blog posts:

Additionally, some earlier writings:

Some time ago I was asked about the way this website works, and as I started to explain, I realised just how much there was going on and how complicated it seemed. But they seemed to think it was neat and elegant, so I thought I'd write it up.

The world's very first Wiki Wiki was written by Ward Cunningham and went live on his website in March 1995. I stumbled across it not too long after that, and was intrigued. The idea of a website that could be edited through the site itself was simply brilliant, and I wanted to have something similar.

Where we start
Problem was, at the time it wasn't possible for me, with the hosting I had, to run scripts of any type on my web site. So how could I do it?

It's easy enough to start with the concept of a definitive text version. That has to live somewhere, so let's have it (to start with) on my local machine.

Converting to HTML
Next, we need a script that converts the plain text to HTML. I wrote a quick and simple version to do that, making up the rules as I went along. How do I indicate italics, bold, underline, and other formatting? Markdown wouldn't exist for 8 years yet, so inventing a "mark-up language" was essential. It's not a particularly good one, but it works.

The text in a form
So we have a definitive text version, and we auto-generate the site from it, including links both internal and external. But if we're going to allow the site to be edited through the site - somehow - we need the plain text to be "exposed" in some way. So what we do is put it in a form. In this way each HTML page has an associated form containing the plain text.

There are also fields in the form for a visitor to provide credentials. So now we have a way a visitor, a potential editor, can offer a change.

An efficient upload
All this, so far, is local, so now we have to upload it to the server. But for any change made, most of the site won't need uploading because most will remain unchanged. So we make sure we keep a local copy of the current live site, and run a script to determine the differences between that and our newly generated site. Then based on that list of diffs we upload the minimum number of pages, and update out local copy of the live site.

It's worth noting at this point that because pages point to each other, editing one page might, in an obscure manner, cause a change to another. That means that we can't just upload the page we changed, others may also need updating.

So now we have a setup that lets us have a plain text version of our website in some sort of markup language that we can auto-convert into the HTML version, and upload it to the server.

How do we make changes?

Suggesting a change
Each live HTML page has its plain text in an associated web form, and also has a link to that form. So when a reader wants to make a change they can click on the link to bring up the form, change the plain text in the form, provide their credentials, and click "SUBMIT". So the modified plain text is now sitting on the mail server, waiting to be used.

Back on the local machine, there is already some automated processing of emails. In particular, I perform my own spam filtering, so every 20 minutes the auto-processor runs a script to fetch emails via POP3, check for spam, then anything designated as spam is deleted from the server. That means that whenever I access my email from anywhere, the mailbox has been "washed".

But as a side effect, any emails of the right format to be web update emails get copied to a separate directory. At that point the site processing software is triggered.

Extracting the change
The site processing scripts check the credentials on the emails. If a suggested change doesn't have the appropriate credentials it's put into a queue for me to check by hand, but if it does have the appropriate credentials, the modified text is extracted from the form contents and copied to the "Definitive Text Version" of the site.

The generation and upload scripts are run, and the cycle is complete.

And so we have the final design. There are significant short-comings, agreed, but none that are critical. The system works, doesn't require any scripts to be run on the server, the resulting web site is static and hence fast to serve, and is resistant to most forms of attack.

For 1996 technology, I think it's survived rather well.