What is Tale Anyway?

So, I've been working on this project that's currently called Tale for nearly a decade now. The notion is to build an interesting, fun, humorous, immersive, narrative virtual reality. If you're into such things, it's more succinctly called a distributed, web-based MUD. I don't intend to go into detail about how Tale is like a MUD and unlike a MUD. This article is intended to explore some of the novel aspects of Tale.

Gameplay

Tale's landing page, tale.im, is the game. You click play and the game begins. There won't be a lengthy sign up and character creation process; everyone starts as a ghost in Limbo or gets straight back to playing from wherever they left off. To start playing, you just click or type "start" and you're consciousness is installed in a standard-issue resident of Dya, the Tale world. From there you can start doing whatever it is you like doing most: socializing, exploring, fighting, building, or getting into character.

Playing Tale is like many things you're probably already familiar with. It's like chatting with friends with an Instant Message client, or IRC, or opening a Terminal and issuing commands. MUDs in general combine the social aspects of chatting with friends, with a virtual world simulator and its own narrative and commands. Some MUDs lean in favor of socialization and others on gameplay. Tale favors neither, allowing you to gracefully shift between mostly chatting and mostly playing. Since the game is in a web-browser instead of a specialized chat application or old-fashioned command terminal, Tale can alternate among coherent interactive modes like entering quotes to say aloud like "hello, world!", entering action commands like "go north", typing quick commands with single keys for mini-games like "p" for "parry", "r" for riposte, "f" for feint if you're fencing, or browsing through menus to discover commands visually.

Already in Instant Message and IRC clients, you might be familiar with commands like "/me laughs" (that you can use in AIM) or "/nick cowbert" (that you can use to chose a name in IRC). If you use Vim, you might already be familiar with colon commands like ":w". If you use FireFox's incremental search feature, you know that you can use single-quote and slash to start searching for text as you type, alternating between single-key command mode into a full-line command mode.

Tale's input field will accept similar commands depending on whether you're mostly chatting, mostly playing, or performing quick, expert actions. If you're mostly chatting, you'll be able to issue commands with the slash "/" prefix like /go north, and when you're mostly playing you'll be able to chat with a quote prefix like "hi. If you're in an expert mode where you're, say, navigating a yacht on the high-sea with discrete arrow key and letter key commands, you can start entering a chat or command with the same keys.

Narration

Each player in the Tale world has a narrator. This is the charming fellow who listens to your commands and tells you how the story unfolds in response. The narrator is a program that keeps track of what you already know and what you're likely to be interested in and interprets the events occurring around your character like, "Joe says 'hi'.", into just that kind of textual prose, as well as taking your commands and plugging them into the simulator in the selfsame event objects. The ideas of "events" and "things" are central to how Tale works behind the scenes, but it suffices to say that if you are "Elbereth" and you're talking to "Glorfiniel", the narrator would translate events like:

Elbereth Say "Hello" to Gorlfiniel
Glorfiniel Say "Hi" to Glorfiniel

…to story-like narrative accounting for various linguistic and cognitive assumptions that you regularly make — "You say, 'Hello', to Glorfiniel and she says, 'Hi', back.". There is a lot of room for making the narrator more and more interesting as Tale development continues, but there's code in Tale already that creates reasonable paragraphs from a stream of events, accounting for pronouns and such, but still sounds fairly robotic.

Having a complex narrator serves many purposes in Tale. First, it makes the game immersive. It also relieves the burden of creating lots of static content, which is one of the limiting factors that ultimately ends with most games feeling rather shallow. A complex narrator can also abstract the tired MUD practice of BATTLE SPAM, where lines upon lines of discrete hits and jabs rapidly scroll up the chat buffer.

The most subtle benefit of the narrator is that it can introduce a significant burden for programmers seeking to automate their characters, often called "botting". "botting" tends to diminish the quality of gameplay for non-programmer players. With the narrator in our arsenal, we can engage in an arms race with "botters", continuously evolving the narrator to deter botting without banning the practice outright. This incentive structure could be a lot of fun for everyone.

Beyond the Narrative

The narrative is the centerpiece of Tale gameplay. However, unlike a Telnet or Teletype MUD, Tale is not strictly text-based. Since the game takes place in a web browser, we intend to take advantage of the opportunity to add icons, maps, inventory visualizations, sounds, and atmosphere and lighting changes to punctuate the narrative.

Among other things, we're planning to use "parametric" graphics to underscore the narrative. For example, if you're in a sailing or flying vessel, or riding a rabbit around the Tale world, your ride will have a parametric icon. We're using SVG graphic manipulation on the server-side to render permutations of layers from "macro" graphics to produce customized versions of each ride and vessel. The parameters include things like the genetic domain of your steed (Is it descended from an elephant? Did the Narwhal bless it with a tusk?), and the installed rigging on your Zeppelin. The two Ride macro graphics have around 100 layers each.

Inspired by Josh Lewis's old blog site, we intend to mix up the style sheet colors on the page slowly and subtly to reflect the time of day and lighting of your surroundings as you explore the world. Each face of the world has its own color scheme and different parts of the world are lit by the 20-sided Sun and various celestial dice, at times filtered through dappled shade, or illuminated by magma and mana. Using alpha-translucent PNG renderings of SVG graphics, and subtly tuning the foreground and background colors of the entire page, we'll gradually modify the mood of the narration.

And there's Music! The musical selection you experience as you roam the world will vary based on what face of Dya you're travelling on and the tension level of the narrative: safety, adventure, battle, and peril. Chris Pasillas has composed six brilliant themes and variations for Tale and is currently rendering them. The game uses a variation of Scott Shiller's SoundManager library that's been ported to operate as a Chiron JavaScript module. (The largest distinctions are that the Flash bridge code has been factored into its own module, and the API has been simplified. It's a complete rewrite using SM2 as an MTASC reference.)

Engine

The Tale engine is a real-time clock-based event-driven simulation. Every event can trace its cause to a tick of the world clock, which for the time being is imagined to be once per second. The difference between a player character and a non-player character (NPC or MOB in the MUD parlance) is that a non-player character generates a progression of commands on demand using a Python generator that yields Event or Verb object trees in response to observations about their environment and their current state, and a player character has a queue of the same kind of Event or Verb trees. This means that invoking a command in Tale does not render an immediate physical response from the world's engine. Rather, a command is translated by the narrator into an Event and it's enqueued to be invoked when the world clock ticks and visits everything that's not busy or waiting, in the whole world. This levels the playing field between people with nearly instantaneous Internet connections to the Tale server with those who have up to one clock tick worth of lag. It also, in a sense, regulates a certain "conservation of energy" law throughout the world, permitting game mechanics to be tuned more accurately and fairly.

World

Feature Number One in Tale is the data structure we've chosen to model the world. This data-structure underpins it's potential consistency, algorithmic subtlety, performance, scalability, and distributability. This data-structure conveniently models a world of arbitrary size, scope, and detail and provides the scaffolding for engine effort minimization for event propagation, abstraction, and synthesis.

So, what is it? Well, nothing new really: it's a tree. Depending on the scope and particular region of application, its children have 0, 4, 6, or 9 branches on each node. Most of the world is simply a quadtree. Each node has four children, one for each quadrant. The top of the quadtree has a bounding box that encompasses exactly all of contained, square region, and for purposes of Tale, conceptually encompasses all of the air-space above that region. A room is a leaf of the quadtree: a node through which players travel with a unit width and height. Each node is an abstraction of all of its contained rooms, containing the sum of its rooms contents. Roaming the quadtree, each player can divine it surroundings by analyzing the contents and events occurring in its own room, and then those events and contents of each of the more abstract rooms that contain it all the way up to the root of the world tree. Each room is directly connected to its parent and children, and traversing linearly across the plane of the world uses tree traversal algorithms where you zoom in and out of the quadtree to find your destination.

Among the algorithmic possibilities that the quadtree provides a scaffold for, we can provide multi-scale gameplay, the same inspiration behind the recently released, Spore. The most mundane way this might be applied would be to have larger scale creature roam the world in abstract bounding boxes, like a Fire Drake that would move from 16x16 room to 16x16 room. Room abstractions could also be used for flight or sea navigation, since one travels higher and faster with that medium, somewhat negating the mundane obstacles of the world floor. As an eagle, one might move about the world by zooming in and out instead of traversing in cardinal directions. Also, more abstract forms of gameplay might take place entirely on room abstractions. For example, melee would occur in 1x1 rooms, ranged attack (missiles and magic) in 8x8, tactics at 32x32 with 4x4 tiles, strategy at 256x256 with region-sized tiles, and politics at larger yet. Taking a moment to step out of a first-person player experience, garnering increasingly abstract political support from larger social organizations, gameplay could scale from thief to Queen).

Another algorithmic possibility is maintenance of the world's physical invariants, like global distribution and availability of various resources. Using abstractions to redistribute resources absolutely throughout the world would make it possible to keep the weight of gold throughout the world constant. This would prevent the most common physical inflation problems that most MUDs suffer.

The other major implication is that the quadtree permits entire branches of the world to run on independent servers with minimal cross-talk. During my last year at Cal Poly, Ryan Witt engaged his distributed computing team at Caltech to implement a Distributed Hash Table (Chord) for Tale and we extensively discussed the implications of the quadtree for purposes of distribution for scale and redundancy. Ryan and I are now roommates working at FastSoft. The topic of distributed computing comes up from time to time, even though we still need to get a vertical gameplay stack working on a single system before we begin implementing distribution.

Events

One problem Shawn Tice (who needs his own home page, because he's too awesome not to have one at this juncture) and I struggled with when we were coding up the second Python rendition of the quadtree was event management. The general notion has always been that events would escalate up the quadtree to a scope that contains all objects that can notice the event, then each object would have an opportunity to respond to the event by visiting each node in their room's "scope chain". The borrowed notion of a scope chain, in this case, means the containing room and the transitive closure† on its parents. Events would each have an "impact" attribute and each object would have a "sense" attribute. So, each time the world ticks, a bottom up visitation of the world would calculate the minimum impact that an event would need to have to be noticed by the one thing in that branch of the world with the highest sensitivity. This would be performed for all of the senses in parallel: visual, aural, olfactory, thermal, thaumaturgical, social, and such. All objects that can have effect events would have senses. For example, a wall can hear a sound so that it can echo it. Most events would also be directed, so they can only propagate in one of the cardinal directions: north, south, east, or west.

However, as you may have already had an inkling of suspicion, this quadtree-based event propagation model isn't actually realistic, and adding realism would nullify the performance and scalability benefits of the system. For example, with the described event propagation system, if you were standing in a room off-center of a quadtree, as at (2, 2) of a 4x4 quadtree, an sound that is loud enough to carry to room (3, 3) might not be loud enough to carry to room (1, 1). The sound would have to be loud enough to carry to (0, 0) before (1, 1) would hear it at all. So, to guarantee that events radiate the same distance in every direction, you need to propagate events laterally, not merely along the quadtree abstraction scope. However, radiating events laterally defeats the performance benefits of restricting perceptible events to those along the scope chain.

So, there was a trade off between performance and realism. Then we remembered we were writing a game. Thinking back to the original Zelda, we observed that each tile of the screen was analogous to a room in Tale, and that every screen in Zelda was analogous to one of the nodes in its abstract node scope chain. That is, there was some parent room that contains about 30 of the rooms Link walked around in. When he got to the edge of the screen, he was no less privy to what was going on in the neighboring room than he was on the opposite side of the screen. So, we decided that the same abstraction would be sufficient for Tale, except that in Tale, the opportunity for zooming in and out around your character was much more profound; we could provide zoom perspectives for every binary order of magnitude.

Frameworks

The current underlying technologies include Python, Twisted, AJAX, Comet, and Lighttpd. In the past, I dabbled with C++ and Boost, including an iostream system that decoupled input and output, had reasonable variable names, and supported VT100 stream functors, but that code branch has been abandoned since 2002. In addition, the project has inspired the creation of frameworks that I'm calling Chiron (a JavaScript module system), and Python on Planes (a system for persistent HTTP route and responder trees).

Parting Words

So, Tale is a pretty massive undertaking. When I started this project, I had no idea that good ideas took so much more time to implement than paltry ones, but that lesson's learned. To meet at least one of my New Year resolutions (that is, one of the milestones for Tale), I'm cutting back scope in the short term in order to get the project up and running by January. A lot of this design, if not all of its powerful implication, I plan to realize by then. This weekend, I've been working on sewing the quadtree, the Dyan world design, and the Narrator into a single piece. As always, I'm looking for people who are willing to volunteer to set themselves up with commit access to Tale so that, in the absence of time to spend on the project (which I know none of us have), a sufficient body of people can randomly wake up in a cold sweat with a bad idea and have nothing between them and implementing it for Tale!

---

† The excessively precise term transitive closure, in this case, means the room's parent, grandparent, great-grand-parent, ad nauseam. Ad nauseam, in this case, means, "until you retch from nausea or meet Adam and Eve: whichever comes first."

War of the Worlds 2.0 Update

It turns out I write a lot of words and say very little. Here's what you need to know about the upcoming alien invasion:

• Follow @wotw2
• On Halloween, tweet in earnest about the unfolding alien invasion. Make your own story. Keep in touch with your friends like you would in any other disaster situation. Try to imagine what you would tweet as the aliens land, emerge, and lay waste to every metropolis around the world.
• Email wotw2@cixar.com if you want to help plan the story or provide technical assistance.

War of the Worlds 2.0

Last week, Josh Lewis; friend, former Apple coworker, and Lex Luthor hairstyle enthusiast; got me thinking about fiction on Twitter. Then, Ryan Paul pointed out that he had written an article about Twitter fiction already. Here's the idea. Let's (and by "us", I mean everyone) reenact The War of the Worlds, on the Internet, for Halloween!

CBS ran the Orsen Welles War of the Worlds radio program in the tense times leading up to the second World War. The aforementioned Wikipedia article claims that Adolf Hitler denounced the program as a sign of democratic decadence. Without commercial breaks and interpolated among real news, the program was broadcast in earnest and public panic ensued. War of the Worlds was the Ultimate Prank, never to be reproduced. As an homage, let's take Halloween to tweet, share bookmarks and status messages, and blog in earnest about the alien invasion in progress. Everyone, tell your story.

The War of the Worlds has several phases that I believe have strong analogs in the tense times leading to World Web II (point oh).

1. a noteworthy astronomer speculates on the possibility of an alien invasion. This would be a good time to talk about the Drake equation in your blog, especially if astronomy is your hobby. Send an email to wotw@cixar.com with your blog so we can proliferate it on the @wotw2 Twitter account.
2. The alien invasion occurs. Follow @wotw2 to keep in sync with the progress of the invasion. This Twitter feed will automatically update, in general terms, the unfolding of the alien invasion like clockwork throughout the world. Coordinate with Tweeters in your area to tell local stories.
1. cylinders fall from the sky. Tweet about where you are. Ask your friends where they are. Form posses. Skip town or take a closer look.
2. tripods emerge. Flee, get stuck in traffic, or take refuge and tell us what you see.
3. Martians begin obliterating every Terran metropolitan area with heat rays. Don't call them heat-rays; that would be a dead giveaway. Describe what they do and come up with your own name! Do you work in a public service like hospitals or fire? What's your job and what do you do? Do you organize your coworkers and flee? Do you head for the hills with your go-bag?
4. Military, local militia, and national guard units get organized and attack the alien invaders. Do you serve in the military? This is your last chance to tell us where you're headed. Do you have family in a militia? Try to keep in touch and let us know how and where they valiantly fought and lost.
5. The invasion spreads from cities to countryside.
6. Tripods begin to shut down an malfunction. Are you near one? Do you take a closer look?
3. After the threat dies down, people begin to blog and speculate about what happened, and every topic near and dear to them.
4. The curtain rises. Blog, link, and tweet about the experience.

The "War of the Worlds 2.0" event will be synchronized, on Halloween Friday, with the http://twitter.com/wotw2 account and we're writing automation with Ryan Paul's Gwibber tool to automatically post Tweets to various services on that day. If you would like to make additional accounts to synchronize local events, please send an email to wotw2@cixar.com with a Twitter account name and password and we'll hook you up with edit privileges for the tweet plan on Google Docs. Tweets (rows) will be deployed for each column (account).

Let's tell a story!

JSON Basics

JSON is a strict subset of JavaScript, particularly a subset of its object-literal notation, discovered and specified by Doug Crockford. The subset is sufficient to make the creation of parsers and formatters in various languages nearly trivial, while completely trivializing the process of parsing the notation in a web browser.

Object-literals in JavaScript are very similar to their analogs in many other languages, like Perl, Python, and even AppleScript. The notation provisions text strings, numbers, booleans, and the literal null for all elemental (scalar) data. Then the notation provides Arrays for ordered values and Objects for unordered, unique, string-to-anything key-value-pair mappings. With these types, you can express most hierarchical data easily.

10
3.1415
"a"
true
false
null
[1, 2, 3]
{"a": 10, "b": 20}
[{"a": 10}, {"a": 20}]

JSON makes a couple simplifications even on JavaScript's object literal grammar. These make it even easier to write parsers and formatters.

1. a JSON expression contains no new-line characters.
2. all strings, including keys in objects, are enquoted between double quotes.

JSON joins the ranks of many other markup languages that we can use to easily transfer data among web servers, but its primary function is as a common data interchange language between web browser clients and web servers hosted in the numerous languages of the web. JSON is well adapted for this space because it can take advantage, through various channels, of every web browser's fast, built-in JavaScript interpreter.

There are two flavors of JSON services with subtle differences for both clients and servers. One is JSON proper, which I will call XHR JSON because it uses an XML HTTP Request. The other is called JSONP or "JSON with Padding". You would use one, the other, or neither based on security and performance concerns.

XHR JSON

XHR JSON uses a feature that exists in one form or another in all modern web-browsers, called an XML HTTP Request, and a function in JavaScript that lets you evaluate arbitrary text strings as JavaScript programs, called eval. With an XML HTTP Request, the client can make an HTTP Request to any URL on the same domain as the hosting page. This constraint is called the "Same Origin Policy". Unfortunately, the policy is in many cases not sufficient to isolate vulnerability to paths of a particular domain, nor to prevent cross site scripts from using the data (more on that later). Whether the security mechanism is effective or not is a longer and later discussion. The point being that this mechanism is the crux of your choice between XHR JSON and JSONP.

With an XHR, you request plain text from a URL on the same domain as your page, then you evaluate it as a JavaScript program. Performing a cross-browser compatible XML HTTP Request isn't trivial in itself, so let's assume that I'm using a library that provides an xhr function that synchronously (blocks) until an HTTP request is complete and returns the content of the HTTP response as a String. There are other variations on xhr for asynchronous requests and grabbing XML.

var text = xhr("/some.json");
var json = eval(text);

In this case, the "/some.json" contains unadulterated JSON. However, because your JSON is likely to be an Object like {"a": 10}, we need to make a special arrangement. A JavaScript program that is merely an Object literal would be mis-parsed initially as a code block. For this reason, we must force the JavaScript parser into expression context instead of statement context. For this reason, we wrap the JSON string in parentheses or assign it to a variable.

var text = xhr("/some.json");
var json = eval("(" + text + ")");

I prefer parentheses because I haven't, in my fallible memory, encountered a browser that supported XHR but wasn't standards compliant enough for the eval function to return the value of the last evaluated expression.

When you use an XHR to grab JSON, you are vulnerable to the host, depending on it to not give you an alternate JavaScript program that insidiously evaluates to the same data. For that reason, a lot of JSON libraries engage in the slower and dubious attempt to validate the JSON before evaluating it with a regular expression. The bottom line is that you're vulnerable to your server when you use XHR and eval.

That being said, it's better to get an exception early than an untraceable error later. For that reason, attempting to validate JSON is a good idea. There's another one: variable laundering. The eval function inherits the calling context's scope chain. This means that the server can read and alter any variables on your scope chain. I use an evalGlobal routine to launder my scope chain. This doesn't give security all by itself, but it could help lead to a secure system down the way and can turn a bunch of silent name resolution errors or data leaks into thrown NameErrors.

(function (evalGlobal) {
	var text = xhr("/some.json");
	validateJson(text);
	var json = evalGlobal("(" + text + ")");
	...
})(function () {
	return eval(arguments[0]);
})

I'll give a detailed explanation of evalGlobal in another article. For now, suffice it to say, this is much more elegant than using a variable to capture the JSON value, although it is possible:

var text = xhr("/some.json");
var json;
eval("json = " + text);

JSONP

JSONP is different than XHR JSON in both the way the server hosts the data and in the way that the client consumes it. On the client side, the user adds a <script> tag to their own page. The source of the script is the URL of a server side program with the name of a callback function that the client places in global scope sent as a query argument.

<script src="http://other.com/some.jsonp?callback=foo">

The client script arranges to receive parsed and evaluated JSON data asynchronously by adding a foo method to the global scope, the window object.

window.foo = function (json) {
	...
	delete window.foo;
};
var script = document.createElement('script');
script.src = "http://other.com/some.jsonp?callback=foo";
document.getElementsByTagName('head').appendChild(script);

When you add a script tag to your document, browsers are kind enough to notice and automatically send an HTTP request to fetch the request JavaScript program. In this case, your page trusts the script on other.com to return a JavaScript program that will call your "foo" function and pass it some JSON data.

foo({"a": 10});

With JSONP, the client is vulnerable to the remote server because it can opt to write an arbitrary JavaScript program before calling foo, if at all. So, you should only use JSONP to request data from domains that you trust.

Also JSONP is slower and less responsive to errors than XHR JSON can be. The only signal that you receive as to the progress or status of a JSONP request is whether your callback gets called in a timely fashion. XHR JSON is preferable in all situations where it is possible and it may even make sense to create a proxy to the other domain from your same domain server. Using a proxy also gives you an opportunity to validate the JSON on the server-side, where computation time is cheap.

Same Origin XHR JSON

There's another trick with XHR JSON, and I'm not sure what scenarios require it. Some people can use the JSONP technique to fetch JSON from a foreign domain. These folks can't send HTTP headers or cookies, and they never get an opportunity to alter the text of the HTTP request before it's evaluated as JavaScript in their browser. I would think that it would be sufficient to prevent an other domain client from intercepting sensitive data for the server to require an authenticated token in the HTTP request, as can be provided by an XML HTTP Request but not a JSONP request. However, some crackers can attempt to get data from your service by using JSON data in a cross site script, much like JSONP. However, instead of using a callback, these crackers arrange to override the Array or Object constructors and thus can monitor and transmit snippets of JSON constructed by simply evaluating JavaScript object literal notation. This technique can be prevented by padding your XHR JSON service with an infinite loop.

while (1);
{"a": 10}

In this case, your client conspires with the server, and since it does get a chance to intercept and modify the text of the JSON, it strips the known number of characters for the while loop before evaluating it.

var text = xhr("/some.json");
var json = evalGlobal("(" + text.substring("while (1);".length) + ")");

What baffles me is that, unless you've secured your JSON with an authentication token, any server capable of making HTTP requests and parsing JSON could do the same thing. Malicious clients are not required to use web browsers. The only situation where this might occur would be if the cracker had compromised your client already, had your authentication token, and needed your browser to make a cross domain JSONP request on its behalf. That could not be the case since the server would have no reason to give an XHR JSON authentication token to a client that could only fetch data with JSONP. That point is even moot since, going back, the cracker has already compromised your client and might as well use XHR JSON with all the same rights as you on even the same origin.

That being said, I noticed that GMail used this technique, so I assume there's substance to it. Perhaps someone will clarify in the comments.

Designing Django's Object-Relational-Model - The Python Saga - Part 6

Django is a web application framework in the Python language. One of the advantages that Django has over other libraries is that it was written and designed by Python experts. That is to say, they knew about variadic arguments, properties, and metaclasses. Furthermore, they knew how to cleverly use these ideas to sweep a lot of complexity under the hood so that common developers, or uncommonly good developers who want to think about other things most of the time, can gracefully suspend disbelief that anything complicated is going on when they design their database in pure Python. This article will illustrate how Django uses metaclasses and properties to present an abstraction layer where you can specify a database schema with Python classes. For simplicity, the "database" backend will be plain Python primitive objects—tables will be dictionaries of dictionaries.

In the end, we want to be able to write code that looks a whole lot like it's using Django's Object-Relational-Model:

class Cow(Model):
    id = PrimaryKey()
    name = ModelProperty()

cow = Cow(id = 0, name = 'Moolius')
cow.save()

cow = Cow.objects.get(0)

The easiest part (for the purpose of this exercise) is Django's concept of an "object manager". In Django, every model has an object manager that provides a query API and, depending on the backend, might cache instances of Model objects. Conveniently, a very narrow subset of the object manager API is almost exactly the same as a dictionary. Conceptually, the object manager boils down to a dictionary proxy for the database where you can use the get function to retrieve records from the database. For simplicity, our ObjectManager is just going to be a dictionary.

class ObjectManager(dict):
    pass

Beyond the scope of this article, the ObjectManager should be handy for grabbing lots of objects from the database at once. Django provides a very thorough and relatively well-optimized lazy query system with its object managers. The ObjectManager has get, and filter methods which, instead of simply accepting the primary key, accept keyword arguments that translate to predicate logic rules. In particular, the filter function is lazy, so you can chain filter commands to construct complex queries and Django only goes to the database once.

While it would be super-cool to model all of this with native Python, it actually is a lot of code, so that's a topic for maybe later. We'll just use the built in dict.get.

We'll also need all of the code from Part 5 since models will be another application of the ordered property pattern. This is how Django creates SQL tables with fields in the same order as the Python properties.

from ordered_class import \
    OrderedMetaclass,\
    OrderedClass,\
    OrderedProperty

We use the OrderedMetaclass to make a ModelMetaclass. The model metaclass will have all the same responsibilities as our StructMetaclass, including "dubbing" the properties so that they know their own names. The model metaclass will also create an ObjectManager for the class. This isn't the complete ModelMetaclass; we'll come back to it.

class ModelMetaclass(OrderedMetaclass):
    def __init__(self, name, bases, attys):
        super(ModelMetaclass, self).__init__(name, bases, attys)
        if '_abstract' not in attys:
            self.objects = ObjectManager()
            for name, property in self._ordered_properties:
                property.dub(name, self)

The next step is to create a ModelProperty base class. This class will be an OrderedProperty so it's sortable. It will also need to implement the dub method so it can figure out its name. Other than that, it'll be just like the StructProperty from the previous section: it will get and set its corresponding item in the given object.

class ModelProperty(OrderedProperty):
    def __get__(self, objekt, klass):
        return objekt[self.item_name]
    def __set__(self, objekt, value):
        objekt[self.item_name] = value
    def dub(self, name):
        self.item_name = name
        return self

There is a distinction in the refinement of ModelProperty from StructProperty: ModelProperty objects will eventually need to distinguish the value stored in the dictionary from the value returned when you access an attribute. In the primitive case, they're the same, but for ForeignKey objects, down the road, you'll store the primary key for the foreign model instead of the actual object. This is the same as the behavior in an underlying database backend.

class ModelProperty(OrderedProperty):
    def __get__(self, objekt, klass):
        return objekt[self.item_name]
    def __set__(self, objekt, value):
        objekt[self.item_name] = value
    def dub(self, name):
        self.attr_name = name
        self.item_name = name
        return self

Let's consider a PrimaryKey ModelProperty. The purpose of a PrimaryKey is to designate a property of a model that will be used as the index in its object manager dictionary. In Django, this can be an implicit id field at the beginning of the table. For simplicity in this exercise, we'll require every model to explicitly declare a PrimaryKey. The ModelMetaclass will identify which of its ordered properties is the primary key by observing its type. Other than their type, a primary key's behavior is the same as a normal ModelProperty, so it's a really easy declaration:

class PrimaryKey(ModelProperty):
    pass

Now we can go back to our ModelMetaclass and add the code we need for every class to know the name of its primary key. I create a list of PrimaryKey objects from my _ordered_properties and pop off the last one, leaving error checking as an exercise for a more rigorous implementation. There should be only one primary key.

class ModelMetaclass(OrderedMetaclass):
    def __init__(self, name, bases, attys):
        super(ModelMetaclass, self).__init__(name, bases, attys)
            if '_abstract' not in attys:
                self.objects = ObjectManager()
            for name, property in self._ordered_properties:
                property.dub(name)
            self._pk_name = [
                name
                for name, property in self._ordered_properties
                if isinstance(property, PrimaryKey)
            ].pop()

Now all we need is a Model base class. The model base class will just be a dictionary with the model metaclass and a note that it's abstract: that is, it does not have properties so the metaclass better not treat it as a normal model.

class Model(OrderedClass, dict):
    __metaclass__ = ModelMetaclass
    _abstract = True

The model will also have a special pk attribute for accessing the primary key and a save method for committing a model to the ObjectManager.

class Model(OrderedClass, dict):
    __metaclass__ = ModelMetaclass
    _abstract = True

    def save(self):
        self.objects[self.pk] = self

    @property
    def pk(self):
        return getattr(self, self._pk_name)

Now we have all the pieces we need to begin using the API. Let's look at that cow model.

class Cow(Model):
    id = PrimaryKey()
    name = ModelProperty()

cow = Cow(id = 0, name = 'Moolius')
cow.save()

cow = Cow.objects.get(0)

All of this works now. You make a cow model; that invokes the model metaclass that sets up Cow._pk_name to be "id" and tacks on a Cow.objects object manager. Then we make a cow and put it in Cow.objects with the save method. This is analogous to committing it to the database backend. From that point, we can use the object manager to retrieve it again.

We can refine the Model base class to take advantage of the fact that it's not just a dictionary anymore: it's an ordered dictionary. We create a better __init__ method that will let us assign the attributes of our Cow either positionally or with keywords. That makes our cow more like a hybrid of a list and a dictionary. Also, since our model instances aren't merely dictionaries, we create a new __repr__ method that will note that cows are cows and moose are moooose. The new __repr__ method also takes the liberty to write the items in the order in which their properties were declared.

class Model(OrderedClass, dict):

    …

    def __init__(self, *values, **kws):
        super(Model, self).__init__()
        found = set()
        for (name, property), value in zip(
            self._ordered_properties,
            values,
        ):
            setattr(self, name, value)
            found.add(name)
        for name, value in kws.items():
            if name in found:
                raise TypeError("Multiple values for argument %s." % repr(name))
            setattr(self, name, value)

    …

    def __repr__(self):
        return '<%s %s>' % (
            self.__class__.__name__,
            " ".join(
                "%s:%s" % (
                    property.item_name,
                    repr(self[property.item_name])
                )
                for name, property in self._ordered_properties
            )
        )

Now we can make a cow model with positional and keyword arguments, and print it out nice and fancy-like:

>>> Cow(0, name = 'Moolius')
<Cow id:0 name:"Moolius">

The next step is to introduce ForeignKey model properties. These are properties that will refer, via a relation on a primary key, to an object in another model. So, the ForeignKey class will accept a Model for the foreign model. Its dub method will override the item_name (preserving the attr_name) provided by it's super-class's dub method. The new item_name with be the attr_name and the name of the primary key from the foreign table, delimited by an underbar. This will let the foreign key property hide the fact that it does not contain a direct reference to the foreign object; it just keeps the foreign object's primary key. However, if you access the foreign key property on a model instance, it will go off and diligently fetch the corresponding model instance. If you assign to the foreign key property, it'll tolerate either a primary key or an actual instance.

class ForeignKey(ModelProperty):
    def __init__(self, model, *args, **kws):
        super(ForeignKey, self).__init__(*args, **kws)
        self.foreign_model = model
    def __get__(self, objekt, klass):
        return self.foreign_model.objects.get(objekt[self.item_name])
    def __set__(self, objekt, value):
        if isinstance(value, self.foreign_model):
            objekt[self.item_name] = value.pk
        else:
            objekt[self.item_name] = value
    def dub(self, name):
        super(ForeignKey, self).dub(name)
        self.item_name = '%s_%s' % (
            name,
            self.foreign_model._pk_name,
        )

Now we can write code with more than one model using relationships. Let's give our cow a bell.

class Bell(Model):
    id = PrimaryKey()

class Cow(Model):
    id = PrimaryKey()
    name = ModelProperty()
    bell = ForeignKey(Bell)

bell = Bell(0)
bell.save()
cow = Cow(0, 'Moolius', bell)
cow.save()

Note that you must save the bell so that when you construct the cow, it can fetch the bell from Bell.objects.

There's more to Django's ORM, of course. This article doesn't cover parsing and validation, which are both assisted by the ORM. Nor does it cover queries, query sets, the related_name for ForeignKey properties on foreign models, Django's ability to use strings for forward references to models that have not yet been declared, or many of the other really neat features.

What this article does cover though, is that you can create a powerful abstraction of a proxied database with pure-Python in less than 200 lines of code. This means that you could create a light-weight proxy over HTTP to a Django database that exposes itself with a REST API. You could also create an abstraction layer that would allow you to pump Django ORM duck-types back into Django to use pure Python objects in addition to or in stead of a database backend.

But, if this article does nothing else, I hope it communicates that Django is cool. I have read a whole lot of code from every dark corner of the web and I have liked very little of it; people I've worked with will testify that I've regularly "hated on" every library or framework I've ever seen. I've never met Adrian Halovalty, Malcolm Tredinnick, or Simon Willson and the growing developer community around Django. However, I've read their code and now I can tell you, over the course of several articles, that they're really smart and you should use their code.

Ordered Properties - The Python Saga - Part 5

In C and SQL, structs and records have properties that appear in a particular order. In Python, objects use hash-tables to store their attributes, so the order is not deterministic nor relevant. That's no consolation if you're trying to model C structs or SQL tables in Python though. Reading though the Django code, I discovered that those wily coders had synthesized a technique that combines the virtues of properties and metaclasses that we can generally model the order in which fields of a struct or SQL table appear.

The trick is to provide an API that allows your users to write code like:

from time import gmtime
from cstruct import Struct, IntegerProperty

class EpochTime(Struct):
	seconds = IntegerProperty()
	microseconds = IntegerProperty()

epoch_time = EpochTime()
timestruct = gmtime(epoch_time.seconds)

In this case, the Struct type cares about the size, order, and disposition of the C structure it models so that it can unpack a byte buffer presumably retrieved from a C-type library.

There are two components to the general solution. First, there's an OrderedProperty base type. Ordered properties track the order in which they were initialized. For this purpose, ordered properties use a global counter. The absolute value of a property's creation counter is irrelevant. The only requirement is that they monotonically increase as each property is declared, and that classes declared concurrently do not interfere.

from itertools import count
next_counter = count().next

I learned about itertools from Brett Cannon who was preparing his thesis defense when I was at Cal Poly. In another piece of code, which I would cite if I could recall, I learned the trick of using the itertools.count function to atomize a global counter. It takes advantage of Python's dubious global interpreter lock.

Then the OrderedProperty base type just stores a creation counter upon initialization. Keep in mind that all derived types must trickle their __init__ call—your base types are not always what they seem and there are use cases you can not foresee where you might be required to receive and pass your initialization arguments to an unknown super-type. That's a side-effect of working in a language with mix-ins, and it's a "good thing".

class OrderedProperty(object):
	def __init__(self, *args, **kws):
		self._creation_counter = next_counter()
		super(OrderedProperty, self).__init__(*args, **kws)

The next part of the puzzle is inspecting your property order. We use a metaclass to note when new types are created and we give them an _ordered_properties attribute with the ordered-property items in their attributes. Keep in mind that base types might have properties too. __mro__ is an attribute of all classes that is a linearized list of a classes inheritance hierarchy. It's effectively the result of a dynamic topological sort algorithm for the closure of your base types. We traverse it backwards so that if you create a dictionary via dict(_ordered_properties), name collisions are resolved chronologically.

class OrderedMetaclass(type):
	def __init__(self, name, bases, attys):
		super(OrderedMetaclass, self).__init__(name, bases, attys)
		self._ordered_properties = sorted(
			(
				(name, value)
				for base in reversed(self.__mro__)
				for name, value in base.__dict__.items()
				if isinstance(value, OrderedProperty)
			),
			key = lambda (name, property): property._creation_counter,
		)

Then, we create our OrderedClass that we can inherit to get free _ordered_properties attributes.

class OrderedClass(object):
	__metaclass__ = OrderedMetaclass

Let's see it in action:

class Foo(OrderedClass):
	bar = OrderedProperty()
	baz = OrderedProperty()
Foo._ordered_properties == [
	('bar', <Ordered Property instance somewhere>),
	('baz', <Ordered Property instance somewhere>),
]

With a small modification, we can track ordered inner classes too.

class OrderedMetaclass(type):
	def __init__(self, name, bases, attys):
		super(OrderedMetaclass, self).__init__(name, bases, attys)
		self._creation_counter = next_counter()
		self._ordered_properties = sorted(
			(
				(name, value)
				for base in reversed(self.__mro__)
				for name, value in base.__dict__.items()
				if isinstance(value, OrderedProperty)
				or isinstance(value, OrderedMetaclass)
			),
			key = lambda (name, property): property._creation_counter,
		)

So, we put all of those order classes in our generic ordered properties module. Now we can delve into our particular Struct example.

First, we need field types. We will need a base type and derivative types for all of the field types we want to model from C, like IntegerField. Bear in mind that OrderedProperty is just a mix-in; it doesn't actually define __get__ and its ilk because those are all specific to the derivative implementation. All struct fields are going to store their actual values in a special _value dictionary on their corresponding Struct instance.

class StructProperty(OrderedProperty):
	def __get__(self, objekt, klass):
		return objekt._values[self.name]
	def __set__(self, objekt, value):
		objekt._values[self.name] = value
	def dub(self, name):
		self.name = name
		return self

You'll notice that instances of StructProperty need to know their name, the key in their corresponding instance's _values dictionary. The struct property instances don't get this from their initializer. Instead, each property will get "dubbed", given a name, when the StructMetaclass visits its _ordered_properties.

Let's just look at an integer.

class IntegerProperty(StructProperty):
	_format = 'i'

The only thing special about an integer is that it's format specifier for the pack routine is "i".

We're going to want to use a metaclass again, this time inheriting from our OrderedMetaclass. The purpose of this metaclass will be to analyze the ordered properties, construct an aggregate format specifier for pack and unpack methods, and to dub each of its properties with their name.

class StructMetaclass(OrderedMetaclass):
	def __init__(self, name, bases, attys):
		super(StructMetaclass, self).__init__(name, bases, attys)
		for name, property in self._ordered_properties:
			property.dub(name)
		self._format = "".join(
			property._format
			for name, property in self._ordered_properties
		)

All that remains is to create our API base class, Struct. This class initializes its values and declares its metaclass.

from struct import unpack

class Struct(OrderedClass):
	__metaclass__ = StructMetaclass

	def __init__(self, *args, **kws):
		super(Struct, self).__init__(*args, **kws)
		self._values = {}

	def unpack(self, value, prefix = None):
		if prefix is None: prefix = ""
		for (name, property), value in zip(
			self._ordered_properties,
			unpack(prefix + self._format, value)
		):
			self._values[name] = value

So, now our epoch time example is possible using Struct and IntegerProperty, completely oblivious to the machinations behind the scenes.

from time import gmtime
from datetime import datetime

class EpochTime(Struct):
	seconds = IntegerProperty()
	microseconds = IntegerProperty()
epoch_time = EpochTime()

timestruct = gmtime(epoch_time.seconds)
datestruct = (timestruct[:6] + (epoch_time.microseconds,))
print datetime(*datestruct)