tag:blogger.com,1999:blog-36187808045447927272024-03-12T21:36:20.346-07:00Ask a Wizard<a href="mailto:askawizard@cixar.com">askawizard@cixar.com</a>Kris Kowalhttp://www.blogger.com/profile/01443956999129365941noreply@blogger.comBlogger132125tag:blogger.com,1999:blog-3618780804544792727.post-25839349360163503162011-07-28T15:30:00.000-07:002011-07-28T16:06:25.508-07:00TearsI try to cry when I’m moved to cry. I decided to do this when I was very young, for a few reasons; I’ve only seen my father cry once—when our dog Sandi went to sleep one last time. He has only ever admitted to crying one other time—at his mother’s funeral when he was 16 years old. But also, I wrestled with gender expectations when I was a kid, and for the better and the worse, I decided to be a person before being a man, mostly because I figured that women and men could live more freely with common social expectations. Part of being a person is crying. (Part of being a person is also accepting your own gender, and I certainly suffered and struggled with becoming a man, but that is a different story.)
<p>I’ve cried every time I’ve read <em>The Lord of the Rings</em>. The interesting thing is that each time I read it, I’m moved to tears at a different point, and looking back to see what parts of the story resonated the most for me at different parts of my life.
<p>The first time, I cried when Gandalf fell in Moria. I was just out of High School. Gandalf was immortal in my eyes. He was powerful both in capability and personality. I wanted to be like Gandalf, so watching Gandalf fall presumably to his death was a great hurt to me. I was barely able to read on and all through Lothlórien and beyond, I keenly felt the fellowship’s loss.
<p>The second time, I cried as Frodo was carrying the ring through the tumbled plains of Mordor. I was at Cal Poly at the time. Forget my airs and pretense for a moment; graduating from college was very difficult and even in the last week of it, any of three things could have protracted it to the point I would not have had the will to carry on. I was in the shadow of heartbreak for two years and it took a total of seven for me to complete junior college and then university.
<p>The third time, I cried at the very end, when Sam came home and said, “Well, I’m back.”. I had finally put my feet in the shoes of a supporter instead of a leader.
<p>I just finished reading the series a fourth time, moments ago. It had been some time since I had last watched the movies. I attempted to distance my mind from the memories and pictures from the film, so that I could experience the books again from my own imagination, and with the added insights into Elvish and the maps from working on <a href="http://3rin.gs/">3rin.gs</a>. It seems that I’ve gone back to Frodo. This time, tears came to my eyes as Frodo is spirited away to Valinor. The words here of Frodo’s experience approaching the spiritual realm accessible only to the Elves are the same words that Gandalf uses to describe heaven to Pippin on the walls of Minas Tirith in the film.
<blockquote>“The grey rain-curtain turned all to silver glass and was rolled back, and he beheld white shores and beyond them a far green country under a swift sunrise.”</blockquote>
<p>Frodo takes Arwen’s place on the ship that bears her father, Galadriel, and Gandalf back to the land of the Valar “who have been called gods by men”, thus Heaven in a sense, because the burden of carrying the ring has left him with wounds that do not heal. It is then that Sam returns home, back to his mortal burdens and mortal love.Kris Kowalhttp://www.blogger.com/profile/01443956999129365941noreply@blogger.com1tag:blogger.com,1999:blog-3618780804544792727.post-68447577517038023792010-12-14T11:38:00.000-08:002010-12-14T12:18:20.382-08:00jsconf.eu 2010<embed src="http://blip.tv/play/hq0Kgof7TQI" type="application/x-shockwave-flash" width="480" height="300" allowscriptaccess="always" allowfullscreen="true"></embed>
<p>This year at <a href="http://jsconf.eu/">jsconf.eu</a> in Berlin, I delivered this presentation about the CommonJS effort and how to use the Q API for asynchronous promises. The part about promises is about 15 minutes in.</p>
<p>For the time being, my code examples and figures for the promise API are available out of my Dropbox folder: <a href="http://dl.dropbox.com/u/1688099/jsconf-eu-2010.html">Promises</a>.</p>Kris Kowalhttp://www.blogger.com/profile/01443956999129365941noreply@blogger.com4tag:blogger.com,1999:blog-3618780804544792727.post-19174924477776133032009-12-06T13:00:00.000-08:002009-12-06T13:58:18.776-08:00Object: Container or Type
<p>JavaScript, in a vain attempt to make the language simpler, conflates the concerns of the Object system and the lookup-table. "objects" in JavaScript serve as both instances of types and as key to value mappings, but only for string keys. Object-literals can be used as lookup tables, instances of types, or both at the same time. The "Object" constructor itself serves as both a lookup table base-type, and the base-type of all types.</p>
<p>The principle advantage of using objects as types and as lookups is a reduction in syntax. Many languages have two separate notations for dealing with properties and keys. Keys get the brackets; properties get the dot notation. But dot notation does not provide a facility for parameterized properties names. The easy solution was to just use brackets for both properties and keys.</p>
<pre>object[propertyName]</pre>
<p>Many other languages separate these concerns. The principle advantage of separating these concerns is that a lookup-table object needs to have two key domains, that of its type and that of its contents. When these domains are conflated, neither can express the full range of potential keys. Their key-spaces collide.</p>
<p>Furthermore, the domain of object properties <em>should</em> be more restricted than that of a lookup table. In the former you want all keys to be valid symbols. In the latter, you want keys to be any reference and any string. Instead, Objects-used-as-lookups can only use <em>some</em> Strings: those that do not collide with methods, unless you're <em>really</em> careful.</p>
<p>Which brings me to my thesis. My point is not that JavaScript should be fixed; there is no technically viable solution to that problem, and using another language isn't always a solution. My point is that we have to be <em>really</em> careful. Objects can be safely used as lookup tables for the full range of at least Strings. In order to do so, you have to avoid using them as instances. That means you can't call their function properties (member functions). To do so would be to assume that the member function name is an invalid entry in the lookup key-space. You cannot enforce that restriction without peril.</p>
<p>So, to use an Object as a lookup table, you must only use the "owned" properties of the "Object". By convention, any function in its prototype chain must be treated as a method of the type, not contents of the lookup table. This distinction is useful in determining whether a property is a member function or contents of a lookup type.</p>
<p><strong>has</strong></p>
<pre>return Object.prototype.hasOwnProperty.call(mapping, key)</pre>
<p><strong>get</strong></p>
<pre>
if (Object.prototype.hasOwnProperty.call(mapping, key)
return mapping[key];</pre>
<p><strong>set</strong></p>
<pre>mapping[key] = value;</pre>
<p><strong>getset</strong></p>
<pre>
if (!Object.prototype.hasOwnProperty.call(mapping, key)
mapping[key] = value;
return mapping[key];</pre>
<p><strong>del</strong></p>
<pre>delete mapping[key];</pre>
<p>The complete and hideous <code>Object.prototype.hasOwnProperty.call(mapping, key)</code> instead of the polymorphic <code>mapping.hasOwnProperty(mapping, key)</code> is draconian but enables "hasOwnProperty" to be a key in the container space. Some would argue that this particular value is not worth the effort, and that a polymorphic "hasOwnProperty" is useful in creating Object-as-lookup-and-as-subtypes. If you can validate your key space, it might be an optimization you can use. However, if you are writing generic code to operate on objects that may have been crafted by suspect users, this is not a luxury you can afford. If you want polymorphic types, use a polymorphic type.</p>
<p>To that end, I propose that you make or find a polymorphic collection type. These are easy to define. We do not have the luxury of creating hash tables in JavaScript since there is no good hashing solution for arbitrary objects, but we do have "toString". We can use "toString" as a hash function and arrays as collision buckets. Then, we can wrap the "internal" Object of Arrays with polymorphic "get", "set", "has", "getset", "del", "put", and "len" property functions in the type name space. <a href="http://github.com/kriskowal/chiron/blob/master/lib/chiron/set.js#L82-96">Chiron</a> defines sets and dictionaries in this fashion.</p>
<p>Narwhal has a <a href="http://github.com/280north/narwhal/blob/master/lib/util.js#L334-399"><tt>util</tt></a> module that exports top-level functions by those names that will operate, via their first argument, on <em>either</em> objects-as-mappings or objects-as-instances generically. It distinguishes name-as-key from name-as-method by checking whether it is an owned property. So, an object literal that happens to be tracking whether it has encountered the "get" method name in a collection of instances would own a "get" property, but an instance that has a "get" method that mediates some crazy internal storage mechanism would not own its "get" property, it would be in the prototype chain.</p>
<p>To this end, I also propose that any Crockford-style constructor that returns an object-as-instance should use the new ECMAScript 5 "Object.create(self)" <a href="http://github.com/280north/narwhal/blob/1f10254fa2df9f2736ecf922b79eab450eefca38/engines/rhino/lib/io-engine.js#L194-196">idiom</a> so that its member functions can be distinguished from object-literal contents.</p>
Kris Kowalhttp://www.blogger.com/profile/01443956999129365941noreply@blogger.com3tag:blogger.com,1999:blog-3618780804544792727.post-80853909812226166852009-03-28T14:36:00.000-07:002009-03-28T14:36:02.343-07:00~/bin
<p><!-- :vim:lbr:wrap:co=80 --></p>
<p>I've started a project on <a href="http://github.com">github</a> for my collection of general-purpose shell scripts: the ones I keep in <a href="http://github.com/kriskowal/xbin/tree/master">~/bin</a> on each of my shell accounts. If you have any general purpose utilities, don't hesitate to fork the project; I'm sure we could collectively build a fantastic set of power tools.</p>
<hr>
<p>I wrote a new one this week, called <tt>xip</tt>, that is a shell analog for the <tt>zip</tt> function in many languages (the name <tt>zip</tt> is naturally reserved for the <tt>pkzip</tt> utility). I created this script to join the ranks of <tt>diff</tt> and <tt>comm</tt>, all functions that benefit from multiple input streams. This comes on the heels of discovering at <a href="http://commandlinefu.com">commandlinefu.com</a> that there's a syntax for subshell fifo replacement. That is, you can supply a subshell as an argument to a command, and it will be replaced with the file name of a named pipe. Let's take the the canonical example:</p>
<pre>
<b>$ cat a</b>
a
b
c
<b>$ cat b</b>
b
a
<b>$ diff <(sort a) <(sort b)</b>
3d2
< c</pre>
<p>To peer under the hood, I used <tt>echo</tt>.</p>
<pre>
<b>$ echo <(echo)</b>
/dev/fd/63</pre>
<p>Ahah! The stream gets passed as an argument!</p>
<p>So, this opens up a world of possibilities. Normally you can only work with linear pipelines because the functions or programs only have one input and one output stream, and this limitation has created a dearth of standard utilities for working with multiple input streams. Before discovering this feature, the command line was like a programming language where functions only accepted one argument (and no implicit partial application, smarty-pants). Now I feel like I've discovered <tt>bash</tt>'s secret cow level.</p>
<p>So, to remedy the lack of multi-parameter functions in shell, I started by making <tt>xip</tt>. It takes any number of file names as arguments and interlaces the lines of their output until one of the streams closes.</p>
<pre>
<b>$ xip <(echo 1; echo 2) <(echo a; echo b)</b>
1
a
2
b</pre>
<p>You can then pipe that to a while read loop, or <tt>xargs -n 2</tt> loop, to create a table. This example enumerates the lines of a file (<tt>jot</tt> for BSD, <tt>seq</tt> for Linux).</p>
<pre>
<b>$ xip <(seq `cat a | wc -l`) a | xargs -n 2</b>
1 a
2 b
3 c</pre>
<p>I suppose the next fun trick is producing multiple output streams, with something like <tt>tee</tt> and <tt>mkfifo</tt>. I leave this as an exercise for the reader.</p>
<hr>
<p>I've also included some of my older scripts from back in the days when I was working exclusively on Linux and used <tt>mpg123</tt> to play my music. <tt>mpg123</tt> is a command line music player, and it doesn't really have a playlist system built in (for that there are alternatives, but I digress). So, I used a pipeline to generate my playlist stream. <tt>cycle</tt>, <tt>shuffle</tt>, and <tt>enquote</tt> are in the github <a href="http://github.com/kriskowal/xbin/tree/master">~/bin</a> project.</p>
<pre>
<b>$ find . -name '*.mp3' \
| cycle \
| shuffle `find . -name '*.mp3' | wc -l` \
| enquote \
| xargs -n 1 mpg123</b></pre>
Kris Kowalhttp://www.blogger.com/profile/01443956999129365941noreply@blogger.com2tag:blogger.com,1999:blog-3618780804544792727.post-90696077065363072392009-03-21T15:09:00.000-07:002009-03-21T15:13:04.897-07:00Interoperable JavaScript Modules<!-- vim:co=80:wrap:lbr: -->
<p>This year has begun with a combination of wondrous events in the JavaScript theatre. I've been struggling to promote the idea of module system in JavaScript for several years now. There has been a sudden explosion of progress.</p>
<p>It started with a series of prototypes for module loaders for <a href="http://cixar.com/tale/wiki">Tale</a> in college, which eventually developed into <a href="http://modulesjs.com">modules.js</a>, that over the last few years was refined by the development of the <a href="http://chironjs.googlecode.com">Chiron</a> module library. I presented Chiron at BarCampLA last year, only succeeding to put <a href="http://en.wikipedia.org/wiki/Dan_Kaminsky">Dan Kaminsky</a> to sleep. Before I left Apple and the bay area, I introduced myself to Mark Miller from the <a href="http://code.google.com/p/google-caja/">Google Caja</a> team including <a href="http://code.google.com/u/ihab.awad/">Ihab Awad</a> and <a href="http://code.google.com/u/mikesamuel/">Mike Samuel</a> and discussed modules over lunch in Mountain View. In May last year, <a href="http://peter.michaux.ca/">Peter Michaux</a> and I started <a href="http://groups.google.com/group/xjs-talk/browse_thread/thread/289ef7a423cbc1a5/dbcf40c099d9237c">discussing</a> converging on a common module standard so that our Chiron and <a href="http://groups.google.com/group/xjs-talk">XJS</a> libraries could be interoperable, but that effort floundered. However, Peter introduced me to the <a href="http://dev.helma.org/ng/">Helma NG</a> project and <a href="http://groups.google.com/groups/profile?enc_user=T_Vq-BEAAAAt9d9v6A8wjP0doSFNOVajkdEasx1kiYTQavV7mdW13Q">Hannes Wallnoefer</a> which has a compatible notion about <a href="http://dev.helma.org/ng/Modules+and+Scopes/">modules</a>. We <a href="http://groups.google.com/group/helma-ng/browse_thread/thread/6d002cb42a47ae42">converged</a> partially toward a <a href="http://askawizard.blogspot.com/2008/09/javascript-module-standard_04.html">standard</a> in August. Peter was also kind enough to notify me when Ihab Awad started a <a href="https://mail.mozilla.org/pipermail/es-discuss/2008-August/006915.html">discussion</a> about modules on the ECMAScript standard discussion list. I met up with the Caja team again in October for a full day to specifically design a module system that was both usable and securable. We discovered a way to make a module system that looked just like any other, but also reused inert module factories for multiple sandboxes, addressing the need for dependency injection.</p>
<p>January this year, Mark Miller sent word to Ihab and me that we were on the agenda for the next ECMAScript committee meeting later that month to present a proposal for adding modules to a future version of JavaScript. After extensive discussion, we nailed down a <a href="http://docs.google.com/Doc?id=dfgxb7gk_34gpk37z9v&hl=en">proposal</a> and Ihab flew down to LA to work on a <a href="http://docs.google.com/Presentation?docid=dcd8d5dk_0cs639jg8&hl=en">presentation</a> with me before the meeting. We presented to the committee on the second day and it was received well. The conversation focused on what additional requirements we would need to nail down to actually make the modules secure.</p>
<p>At about the same time, Kevin Dangoor from the <a href="https://bespin.mozilla.com/">Bespin</a> team at Mozilla <a href="http://www.blueskyonmars.com/2009/01/29/what-server-side-javascript-needs/">prompted</a> a massive <a href="http://groups.google.com/group/serverjs">discussion</a> that attracted a flash crowd of developers around the world who were interested in sharing code among JavaScript implementations outside the browser. <a href="http://www.blueskyonmars.com/2009/02/05/serverjs-one-week-into-building-a-better-javascript/">One week later</a>, with 224 members, and 653 messages posted, we knew Kevin had <a href="http://peter.michaux.ca/articles/a-bright-future-for-standardized-server-side-javascript">struck a nerve</a>.</p>
<p>The group founded the <a href="https://wiki.mozilla.org/ServerJS">ServerJS</a> project, and among the first common efforts was to converge on a <a href="https://wiki.mozilla.org/ServerJS/Modules">module system</a>. Ihab and I camped out on the list promoting, receiving feedback, and refining a <a href="https://wiki.mozilla.org/ServerJS/Modules/SecurableModules">securable module</a> proposal. There are now several efforts to create compliant module loaders for various platforms including <a href="http://jackjs.org">Jack</a> (which works on <a href="http://www.mozilla.org/rhino/">Rhino</a> with <a href="http://www.mortbay.org/">Jetty</a> and <a href="http://sjs.sourceforge.net/">Simple</a>, and eventually <a href="http://code.google.com/p/v8cgi/">v8cgi</a> among others), a project called JSEng or GPSE to be released eventually by Wes Garland at PageMail, <a href="http://www.xucia.com/">Kris Zyp's</a> <a href="http://code.google.com/p/persevere-framework/">Persevere</a>, and of course <a href="http://code.google.com/p/chironjs/source/browse/branch/safe/src/modules.js">Chiron</a>. We're working on getting the various platforms passing <a href="http://code.google.com/p/interoperablejs/source/browse/trunk/README">unit tests</a> and sharing code. I've got about 11KLOCs of Chiron ported to the standard.</p>
<p>Meanwhile, Kevin has <a href="http://groups.google.com/group/serverjs/browse_thread/thread/ed8fb8a9a67ce292">hinted</a> that Bespin may eventually have a JavaScript backend running on Jack, which would be an impressive foothold for the eventual JavaScript standard module library.</p>
<p>So, if last year was the year of JavaScript module struggles, this year looks like it will be the year of JavaScript module success.</p>
<hr>
<p>The technical details are on the <a href="https://wiki.mozilla.org/ServerJS/Modules/SecurableModules">Securable Modules</a> wiki page. The general idea is that modules receive a "require" function for getting other modules with both absolute and relative identifiers, an "exports" object which the module shares with other modules, and an "environment" object for modules that use dependency injection, those things that ultimately provide IO in secured sandboxes.</p>
<p>A module would look like:</p>
<pre>
var file = require('file');
exports.foo = function (bar) {
return file.File(bar, 'r');
};</pre>
<p>Secure module loaders would prevent tampering with the primordials and the global scope by creating module factory functions that receive those three variables under a hermetic bell. A sandbox would be a group of secured singleton modules produced by calling the module factory functions, and sandboxes can create smaller sandboxes and share loaders to improve performance without "leaking" capabilities. If you're used to dependency injection modules, the difference is that the only security boundary is at the sandbox interface, and instead of instantiating modules with an explicit list of its required modules, you inject capabilities in the environment and all modules in that environment are loaded on demand and have access to those capabilities. The hermetic bell is a special evaluator to be provided by the JavaScript engine that runs programs in an alternate transitively frozen global scope.</p>
<p>Enjoy!</p>Kris Kowalhttp://www.blogger.com/profile/01443956999129365941noreply@blogger.com0tag:blogger.com,1999:blog-3618780804544792727.post-72683475406272946172009-03-08T16:56:00.000-07:002009-03-09T17:28:21.970-07:00Dict versus dict<p>Chiron's <tt>base</tt> module provides both a <tt>dict</tt> operator and a <tt>Dict</tt> factory method, as well as <tt>List</tt> and <tt>list</tt>, and analogously <tt>Set</tt> and <tt>unique</tt>. <tt>Dict</tt> and <tt>dict</tt> both accept the same basic types.</p>
<pre>
dict({a: 10}) =
Dict({a: 10}) =
Dict([["a", 10]])</pre>
<pre>
dict("abc") =
Dict("abc") =
Dict([[0, "a"], [1, "b"], [2, "c"]]) =
dict(["a", "b", "c"]) =
Dict(["a", "b", "c"]) =
Dict(iter("abc"))</pre>
<p>In code, the difference is that <tt>Dict</tt> is a <tt>type</tt> and <tt>dict</tt> is an <tt>operator</tt>. The difference in practice is that <tt>dict</tt> first checks whether the first argument is a subtype of <tt>Base</tt> (which includes all objects using the <tt>type</tt> system), and whether it implements a <tt>dict</tt> method. If so, it defers to that polymorphic <tt>dict</tt> method. Otherwise, it defers to <tt>Dict</tt>.</p>
<pre>
var base = require('./base');
var test = require('./test');
exports.Foo = base.type(function (self, supr) {
self.dict = function () {
return base.Dict({'a': 10});
};
});
var foo = exports.Foo();
test.assertEq(
base.dict(foo),
base.Dict({'a': 10}),
'dict behaves as a polymorphic operator'
);</pre>
<p>The same difference applies to the polymorphic operators <tt>unique</tt> (that defers to <tt>Set</tt> if <tt>unique</tt> is not a member of the type), or <tt>list</tt> (that defers to <tt>List</tt> if <tt>list</tt> is not a member of the type). <tt>object</tt> and <tt>array</tt> are also polymorphic operators that work as copy constructors for <tt>Object</tt> and <tt>Array</tt>, but also defer to polymorphic <tt>object</tt> and <tt>array</tt> members. The default behaviors of <tt>object</tt> and <tt>array</tt> are to copy or coerce the argument to an <tt>Object</tt> or <tt>Array</tt>, since <tt>Object</tt> and <tt>Array</tt> cannot be used as copy constructors themselves. A complete variety of coercions are possible, extending well into the bizarre and insane.</p>
<pre>
array("abc") =
["a", "b", "c"]</pre>
<pre>
object([1, 2, 3]) =
{'0': 1, '1': 2, '2': 3}</pre>Kris Kowalhttp://www.blogger.com/profile/01443956999129365941noreply@blogger.com0tag:blogger.com,1999:blog-3618780804544792727.post-51373453122634152832008-12-12T20:41:00.000-08:002008-12-12T21:38:08.781-08:00Holiday Bash
<p>I'll be holding a little party at my terminal for the holidays. Please join me by running the following command in your shell.</p>
<pre>((while true; do echo -en "\e[31m." >&2; echo; done) | (while read line; do echo -en "\e[32m."; done))</pre>
Kris Kowalhttp://www.blogger.com/profile/01443956999129365941noreply@blogger.com0tag:blogger.com,1999:blog-3618780804544792727.post-26423296792616829602008-11-27T17:00:00.000-08:002008-11-29T00:29:48.904-08:00On Faith<p>I grew up in two religions—I promise to tell you which ones later. Contemplating the agreements and arguments between these two faiths helped me realize something very important.</p>
<p>I am wrong.</p>
<p>I'm wrong for a variety of reasons. But, by a twist of logic that's hard to accept until you've been alive for a few weeks, the best reasons for me to believe that I'm wrong are the ones I don't know yet.</p>
<ul>
<li>If I change my mind about a proposition, I either <em>am</em> wrong, or <em>was</em> wrong.</li>
<li>I have changed my mind many times.</li>
<li>I have either been wrong, or become wrong many times.</li>
<li>I will probably change my mind many more times.</li>
<li>I'm probably wrong.</li>
</ul>
<p>On several occasions, I've realized that I was so wrong that I had to become a completely different person to become right. While I was in school, this occurred so frequently and dramatically, that by the time I was in junior high school, I had become the kind of person that I would have <em>despised</em> when I was in early elementary school. And now, I pity the boy I was in junior high for his bitterness. I'm furthermore haunted by the persisting memories and following cognizance of the people I have failed or hurt along the way.</p>
<p>The reason I was bitter in elementary school was that no-one wanted to be my friend. In hindsight, I've realized that I had alienated my peers and thus brought this ill fate upon myself. I was, however, kind. I was not condescending, like similar kids in my situation. I did not pick fights, nor did I take what I did not fairly deserve. If anything, I retained few pleasures for myself for fear of taking them from others. For ample, to this day, I have never ridden a shiny-silver tricycle like the ones that were so popular in my kindergarten playground, and I always waited for everyone to lose interest before I tried the swings.</p>
<p>My fault was more subtle than any of the unkind things that children can cause upon each other. I refused to play their game, a game that you and I play together every day. I refused to play a role. It was my opinion that every person should always wear their true feelings on their faces, and say only what they believed. I believed that it was dishonest to do anything else and could only lead to deferred hurt. I couldn't bear to stand around with a bunch of people and have trite conversations about immaterial matters or play fake dramas among friends. It would have violated my sense of honesty.</p>
<p>This is why I was a wholly uninteresting, boring, stick-in-the-mud person for the rest of that decade. I did eventually learn how to play a role, to play power-games, and to act out dramas about how I certainly "must" feel for the amusement of the situation. I got over myself, a little bit, and now I enjoy the more important truth. As I came to accept wholly the moment Dr. Janice Daurio uttered it over coffee in my final year of junior college, life is about building relationships with other people.</p>
<p>So, yes. I was wrong. I'm likely still wrong about many things. But, more importantly, even if I do find the ultimate truth, the general theory of everything, I will have no reason to ever be completely certain about it. No matter how small the universe is, it's larger than me, and it's larger than I can perceive. It is finer than the least perceptible granularity. There is no vantage in the universe from which I can see its entirety. Furthermore, since I am part of this universe, the last impediment to my full understanding of the universe, is the ability for me to fully understand myself. I take it to be a paradox for a box to contain itself completely, and no more do I believe it possible for my mind to contain a complete and detailed model of itself.</p>
<p>I used to think that the word "abstraction" meant to find a better, more general, and reusable theory. When I was in Clark Turner's class on professional responsibilities for software engineers, he called out the real definition of abstraction: to omit detail. The price of a bigger idea is less detail. That's the limit of our mind, and it's the limit of to our understanding of the universe. We can only claw at the shapes around us with our minds, bending our minds to better fit the model, but never fully contain it.</p>
<p>For these reasons, I believe that if there is a sin, it is certainty.</p>
<hr>
<p>However, apart from never being sure about anything, there are many things worth believing. There are three good reasons to believe a proposition: correspondence, coherence, and pragmatism.</p>
<ul>
<li>Correspondence is a theory that there is a universe. A thing is true because it corresponds to that universe.</li>
<li>Coherence is the theory that all truths must be logically consistent. There are bodies and combinations of premises that are consistent among themselves. If a premise contradicts an accepted truth, it may not be a member of that body of accepted truths. It, however, may be the basis for a new body of truths.</li>
<li>Pragmatism is the harsh realization that neither of the other two theories gives you a place to start believing things. You need a basis, a first truth, and you need to assume that it's inherent to the universe. The problem with coherence is that there are many bodies of coherent truths, not all of which necessarily correspond to the universe. To be pragmatic, or practical, we have to make a guess. With regard to correspondence, the question is, "Which universe?", and the pragmatic answer is, "This one!".</li>
</ul>
<p>Based on these three mechanisms, consider this minimal guide to finding the truth:</p>
<ul>
<li>Guess.</li>
<li>Always remember that the guess might not correspond to the universe. Test it through observation.</li>
<li>Always remember that the coherent body of truths based on the guess might be wrong too. Test it with logic.</li>
<li>With gathered knowledge and experience, always try to make better guesses such that over time you can asymptotically approach certainty.</li>
</ul>
<p>Correspondence, coherence, and pragmatism come out of the discipline of philosophy. The ideas are innate, but it's in philosophy classes and text books that I heard them thus succinctly articulated. The guide to finding the truth is the empirical method: science. Oddly enough, both ultimately boil down to faith and uncertainty, traditionally the purview of religion and agnosticism respectively. The discord between religion and science, under this light, looks more like a popular myth.</p>
<p>Becoming more right requires a lot of observation and thought, but there are some pretty good tools for contemplating truth. For one, coherence is probably the most solid theory of the three. If you focus on culling incoherent premises, or at least organizing them into piles of propositions to be evaluated as a whole, you might not find the truth, but you can mitigate a lot of probable falsehoods. Coherence is math, a concept that's free of the messy details of the physical world, a metaphysical archetype. As such, its employment is the most satisfying of the three reasons to believe.</p>
<p>The next best behavior is observation. Keep sense open. Gather and mull premises in hope that between perception and abstraction, some mote of truth about the universe to which those observations correspond leaks into your mental model of the universe.</p>
<p>That brings us to pragmatism. Of the three reasons to believe a thing, pragmatism is the most tenuous. Pragmatism is about faith, specifically making leaps of faith. In math and logic, these leaps are called postulates, certain suppositions that you make so that you have some basis for evaluating the coherence of other ideas. The trouble with a populate is that, by necessity, it must be the foundation for any certainty that follows. An article of faith is further beyond question or doubt than any conclusion arrived at through the trial of observation and coherence. For that reason, it's best to use pragmatism as seldom as possible.</p>
<p>In mathematics, we presume that there's something called incrementation. There's no proof that incrementation exists, and if we found one such proof, behind it would be terms that would have to be assumed instead. We accept incrementation, that there is a void called zero and a unit away from it in some direction called one. From there, we have all the material we need to converse about all the various kinds of arithmetic and algebra that are possible in the metaphysical verse in which there's incrementation.</p>
<p>Likewise, to engage in meaningful discourse about our universe, we have to make certain leaps of faith.</p>
<ul>
<li>The universe exists (otherwise, correspondence theory would be out).</li>
<li>The universe is consistent (otherwise, coherence theory would be a wash too).</li>
<li>You exist (<i>cogito ergo sum</i>, except that there's a paradox inherent to the assumption that if you think you must be. Let's just stick to the assumption, "I am.").</li>
<li>You are actually you (this one really gives me the shivers once in a while).</li>
<li>The universe is significant.</li>
</ul>
<p>That's a sample of good things to believe in the absence of proof or even it's lesser cousin, evidence. In particular, if you're going to talk about the universe with anyone, it's pretty safe to assume that these are your common ground. If they're not, one of you is either yanking the other person's chain, or the conversation is pointless. Either way, it's time to move on.</p>
<hr>
<p>The trouble with leaps of faith is that, the more of them that you make, the more likely they are to be inconsistent. Coherence and correspondence are much more reliable allies. For that reason, it's best to minimize leaps of faith. It takes especial care to frame explanations for observations without making careless leaps of faith to which you haven't already committed. They tend to sneak in for less wholesome reasons than the search for truth, and tend to get in the way when you try to reconcile your theory with the next observation.</p>
<p>Occam's Razor does not mean that the truth is simple. The quest for a General Theory of the Universe, wherein all fields (electric, magnetic, gravitational) are explained in the terms of a simple formula like Einstein's relation between energy and matter, is predicated on the faith that the universe does have a simple mathematical root. However, this is not an application of Occam's Razor. Occam's Razor states that if you have a simple explanation for a phenomenon and an elaborate explanation for the same phenomenon, it's best to assume that the former is true. That is, chose the explanation that requires the least faith.</p>
<p>My father is fond of recanting a story from my childhood wherein, to my chagrin, I am not the protagonist. I learned how to write my name with crayons roughly when I was four years old. About a day later, my parents discovered my name written on every surface that could retain a crayon's impression, including my sister's back. Producing Kathy's marked verso as evidence, my parents accused me of the crime. In my defense I claimed that she had done it herself. Bear in mind, Kathy was half my age.</p>
<hr>
<p>Apart from unlikely stories for easily explicable phenomena, a person who believes nothing that they do not perceive, induce, or deduce would be a sad and empty shell.</p>
<p>One problem for atheists, agnostics, and skeptics is that there is no natural source of Hope. In fact, you don't have to learn much physics to get to the part where the universe appears to be spiraling down toward a low-energy drain, the "Heat Death of the Universe". At some point, you really do have to make a leap of faith. So, it's important to add another precept to the catalog of postulates.</p>
<ul>
<li>There is hope.</li>
</ul>
<p>In Yoda's words, "Fear leads to anger, anger leads to hate, hate leads to suffering.". While fear often comes from rational distrust and can drive us to excel, it can also mire us in depression. Hope is the cure for fear. Hope is our reason to try to make the future better than the past, and without it we are lost. Since there is no definitive evidence that the future will be better than the past, and the case is piled high that it will ultimately be worse or end in a zero sum, we must postulate that hope exists, that we can craft a better future.</p>
<p>Sometimes, when I am almost completely broken, I lay awake and let that truth fill me to the brim. I let the warmth flow from my core to my pores and to the bottoms of my feet. Hope, the critical endogenous morphine, eventually fills my dreams with blurry but bright visions of solace. I banish my doubts with a crucial lie and my strength returns.</p>
<p>Consider that power though. Can't blind belief drive us to do wicked things? Perhaps it's better to let hope stand alone, without attaching it to a litany of prerequisites or consequences. Everything you attach to a blind supposition, increasing its complexity, risks motivating you to do cruel things without good reasons. It's better to let hope stand alone, a single pure island of beauty on the horizon, or otherwise risk contaminating that vision with zeal for something less perfect.</p>
<p>There are, after all, many reasons to believe a proposition apart from need, logic, and observation. These include wont, fear, and pressure to conform with peers. Despite my disdain for the vapid writings of Terry Goodkind in the <i>Sword of Truth</i> novels, I find myself quoting <i>The Wizards' First Rule</i>, "People…will believe a lie because they want to believe it's true, or because they are afraid it might be true.". Terry wrote this in the spirit of condescent and bitterness, but it serves a greater purpose as an admonition. There are certain conditions where it's important to elevate skepticism:</p>
<ul>
<li>you want to believe</li>
<li>someone else wants you to believe</li>
<li>you're afraid not to believe</li>
</ul>
<p>One of the greatest quotes said in service of hope and truth was that all we had to fear was fear, and one of the worst offenses against those same ideals was the creation of a rainbow of codes for how afraid we must be. It takes a powerful will to see coercion through fear, particularly communal fear, and these are the precise conditions when a healthy skepticism and faith in hope against what another would have you believe, what you are afraid might be true, and what you most desperately want to believe, might serve you best.</p>
<p>There are many good reasons to build communities, but getting together to publicly profess your agreement with one another, or to be fed the truth without argument, smothers that mote of latent uncertainty that helps people listen each other's ruminations, doubts, and observations. It repels people who disagree, the people who need your fellowship most. For those who grow up in such communities, they either grow with a hindered analytical sense, or buried revulsion and fear of rejection or guilt. If you gather to bolster your faith through the comfort of your peers agreement, your community submerges the relevance and welcomeness of pragmatism, coherence, and correspondence. A community should feel free to express their actual beliefs, argue, and converge through trial and support.</p>
<p>An article on <em>Slate</em> called, <a href="http://www.slate.com/id/2203614/pagenum/all">Does Religion Make you Nice?</a>, explores the possibility that religion and atheism are orthogonal to how friendly a person grows to be. American atheists tend to be less nice than European atheists. The dependent factor in how well adjusted a person is in a society is acceptance in a community. We need to have communities that welcome people without regard to the details of their faith. Using tenets of faith to build walls around your church ultimately alienates people with the potential to be good.</p>
<p>If communities gather to ponder difficult choices on the fringes of desire and righteousness, they might discover that they actually hold very different beliefs. It's unhealthy to delude yourself into believing that every member of a community subscribes to the same faith. Our differences in experience, values, and chemistry, <em>from birth</em>, bring us to very different conclusions. We need to value each-other's experience more than our collective synthesis.</p>
<hr>
<p>Regarding the philosophy of behavior, ethics, there is an ancient test for whether to do or not do a thing. You first generalize the act to a rule. For example, if I'm pondering whether to wear a silly hat in public, I would generalize that thought into a rule, like, "People should be welcome to wear silly hats in public.". Then you ask yourself whether society would function if everyone were to follow that rule. What would happen if everyone were welcome to wear silly hats in public?</p>
<p>Emanuel Kant called it the Categorical Imperative, but with subtle variations and derivations, it's a timeless strategy embodied by the Golden Rule. "Do unto others as you would have them do unto you.". Or, via my father the Cowboy Western fanatic, "Before judging a person, walk a mile in their moccasins.". Or, the corollary, "Do for others as you would have them do for you.".</p>
<p>"Categorical imperative" literally means, "the most important rule.". It's the recipe for peace and stability. However, like all mortal truths, it is far from perfect. Often to find a balance between reason and love, you must consider why every situation is special and carefully weigh how specific your situation is and how far away from your path you would, should, or can go for another soul. It's often best to err on the side of altruism. That is, defy your mortal laziness and do everything in your power to help others, and expect nothing from them.</p>
<p>That leads us to a flaw in the imperative. The categorical imperative also tends to imply that you can expect others to treat you with the same degree of cordiality with which you would treat them.</p>
<p>To address that flaw, consider a special rule. In order for the categorical imperative to bring about stability in a community, we have to create a bias, a pair of poles within which the imperfect universe can safely vacillate and gradually approach peace. The categorical imperative implies that your behavior must exactly match the expectations of all people in society. That cannot be the case. So we consider two rules.</p>
<ul>
<li>In so far as you can carry your mortal husk, be strict about your own behavior.</li>
<li>In so far as you can defy your mortal indignation, be tolerant of the behavior of others. For those acts you cannot bring yourself to tolerate, try to bring yourself to forgive.</li>
</ul>
<p>The beauty of these rules is that they fit within the mechanics of the categorical imperative. If every person in the world were to follow these two rules perfectly, we could eventually have peace. It also establishes the reason and necessity of forgiveness, and a solid basis for discussing the lossy abstraction of "human rights".</p>
<p>The last remaining flaw of these rules is that they can promote self-neglect. As I learned from the Star Trek episode, <a href="http://memory-alpha.org/en/wiki/Evolution_%28episode%29">Evolution</a>, self-neglect serves no-one, so there's a third important rule from Guinan: always take for yourself what you can fairly assume for your own health and happiness.</p>
<hr>
<p>I've used the term, "universe", extensively. I've expounded upon the roles of uncertainty, empiricism, and hope in the quest for truth and peace. Throughout, it's important to have a solid definition of "The Universe". Many unfortunate arguments can be rendered moot by consensus on the meaning of "universe" and a thorough reevaluation of dependent ideas in terms of that meaning.</p>
<p>Let's consider an inductive definition of the universe.</p>
<ul>
<li>basis: I am in the universe.</li>
<li>recursive step: anything that has impact on a part of the universe is also in the universe.</li>
</ul>
<p>Apart from discussions about causal-domain-sheer, this definition is concise, simple, and readily applicable. We could consider much more detailed and complicated definitions with intricacies to suit our particular belief systems or definitions of other words we know only descriptively. For the sake of argument, let us take this definition as a postulate and let the definitions of other word and ideas move freely around it.</p>
<hr>
<p>Imagine for a moment that you have achieved a vantage from which you can see the entire universe. This vantage, by necessity, is outside the universe and your perception pierces its occlusive barriers. You truly see the universe in its entirety.</p>
<p>Zoom out.</p>
<p>The universe is surrounded by void: a conceptual region in which the universe does not exist. No matter how many transendences you make to get to the vantage of the universe, there will still be an eternal nothing beyond it, at least one transcendently infinite scope larger than the universe itself.</p>
<p>Zoom out. Zoom out until you have taken in enough of the void into your perspective that the entire universe's size divided into the size of the infinite void is merely a point. There is only void.</p>
<p>From this vantage, there is an infinity of nothing filled with infinitesimal points, one of which is our entire universe. This does not diminish the significance of our universe, but it frees you to consider all of those other points. Each one is a universe, apart from our own, some perhaps sharing attributes but each distinct.</p>
<p>Mathematical concepts are special. There are uncountably infinite mathematical concepts. Each one is its own metaphysical realm whose complexity emerges from simple rules. This flushes well with the assertions that we all make whether consciously or not:</p>
<ul>
<li>A point is true because it corresponds to the universe.</li>
<li>The universe is perfectly coherent.</li>
</ul>
<p>One nice thing about this model of the universe is that it clears up that existential question about creation. Almost all explanations about how our universe came to be rely on intervention outside our universe. Consider this sequence of propositions:</p>
<ul>
<li>A exists. B does not exist.</li>
<li>A creates B.</li>
<li>B does exist.</li>
</ul>
<p>The trouble is that A clearly has an impact on B and thus is part of the universe, B. That would necessitate A and B to both exist initially, which is a contradiction. What if A and B simply exist? Mathematical concepts do not require creation events to be internally consistent. This liberates us of the contradiction of creation, but also leaves room for the possibility that A and B existed initially, and for A to exist before B within the confines of the real universe, C. This opens the possibilities of towers of tortoises, big explosions, or sub-universal progenitors.</p>
<p>From a programmer's perspective, among fractals, programming languages, and Conway's Game of Life, there's compelling evidence that suggests that extraordinarily complex patterns can emerge from simple rules.</p>
<ul>
<li>Define a collection of simple rules.</li>
<li>Apply those rules with mathematical vigor.</li>
<li>Observe that complexity emerges.</li>
</ul>
<p>Thus, it's possible to become comfortable with the idea that from a simple set of rules, a universe of great complexity can emerge. It does not exist any more than it does not exist. Since we're in it, it's significance to us, more so than any other metaphysical construct that my small mind can begin to fathom, is relevant. Furthermore, a universe arrived at by math, is unassailable perfect.</p>
<hr>
<p>Let's define "nature" as the set of rules inherent to the "universe". Consider the notion of "super-natural" things or events. If we refer back to our definition of the universe, the only things that can be apart from the universe are those that have no impact on the universe. From there, let's consider the notion of "miracles", events in the universe that defy our conventional understanding of how the universe works, or "nature". One confusion that I think we can avoid is the bundling of the ideas "super-natural", "para-normal", and "miracle". By the definition of the universe, for a miracle to actually be a part of the universe, it must abide by "nature". Thus, for a miracle to exist, it must be "para-normal", not "super-natural".</p>
<hr>
<p>Various tomes at various times talk about "God". Particularly, I've heard the following attributed to God:</p>
<ul>
<li>perfection</li>
<li>anthropomorphism</li>
<li>creation of the universe</li>
<li>omniscience</li>
<li>omnipotence</li>
<li>super-natural</li>
<li>performance of miracles</li>
</ul>
<p>Let's take a step back. God can only be one of the following:</p>
<ul>
<li>beyond and outside the universe,</li>
<li>within the universe, or</li>
<li>be the universe itself.</li>
</ul>
<p>If we take the universe to be perfect in whole or part, God could be any one of these things, but the term perfect suffers the diminution of its meaning. Let's say that perfection cannot be broken, thus a whole may be perfect and any part of it falls from perfection in so much as it is not complete. If we adhere to the notion that the universe is perfect, by this reasoning, the god would have to be either the universe or beyond it.</p>
<p>Omniscience denotes the ability to see the entire universe. That would imply that God is beyond the universe, or is the universe as well.</p>
<p>Omnipotence denotes the ability to do anything. If the universe is defined by natural rules, to break those rules could only occur in a universe with different rules. I would contend for God to be omnipotent, omnipotent would have to mean "capable of performing miracles", rather than "capable of anything, and thus capable of super-natural acts". There's quite a bit of room for the unexpected within the realm of possibility. I would prefer to pick a different word than omnipotence rather than conjure a new meaning from a word that clearly means "all-capable". How about, "super-powerful", or "ultra-potent".</p>
<p>Referring to the universe as a mathematical construct based on rules, the universe needs no creator any more than a story needs a story-teller in order to exist. Creation means to make something from nothing. I would contend that this definition is unsuitable for any real act. For example, creating a building is really construction from baser materials. If we substitute the term "creation" for "construction", the term could easily apply to the so-called "creations" of people, and also to the construction of a region of the universe by some super-powerful being. In order for this to be the case, God would have to be a part of the universe, and while not the creator of the universe, the constructor of a world.</p>
<hr>
<p>The meaning of the word "death" is "end of life". To rationalize the notion of life after death, it is necessary to distinguish corporeal life from ethereal life. In order for either to be meaningful, they both most be real, that is to say, part of the universe. Ethereal life, therefore is either real, or part of another universe. Either possibility is plausible. For example, ethereal life could be in a distinct universe, where it can have no impact on our universe. However, there are infinite verses apart from our universe abiding infinitely diverse rules, none of which are relevant to ours. This notion of ethereal life does not have any meaning in the context of our corporeal lives. For example, such a life would not be bounded by consequences of our mortal lives, but rather bounded by the infinite possibilities of patterns of thought. The other possibility is that ethereal life exists within our universe, is bounded by the same rules, and thus the actions of our mortal lives could have an impact on them. If this is the case, ethereal life is a matter of a mysterious science, not math. In either case, ethereal life is probably not even similar to anything we would want to believe of it.</p>
<p>There are a lot of reasons to want to believe in life after death. For some people, it counters their fear of death. For others, it fills the gap in justice that the universe does not appear to serve during life. Another reason to believe in life after death is to lend an additional purpose to the immortal soul: if one's soul is separate from their body, living an ethereal life instead of a corporeal life, it's easy to reconcile determinism and free choice.</p>
<p>If we take the universe to be a mathematical construct, bounded by rules, the universe is deterministic. That is, if one had complete knowledge of one state of the universe, perfect knowledge of the rules of the universe, and <a href="http://xkcd.com/505/">infinite computational power</a>, one could extrapolate the entire universe. Naturally, this is beyond our means as mortal components of the universe, but the notion permits us to use inductive reasoning, the notion that, for a statistically representative sample of the universe, if a rule appears be in action, it can be expected to occur in other samples. Inductive reasoning is imperfect in that our sample, and our perceptions of the rules are imperfect, but the idea is predicated on a predictable universe, the kind we can reason about. It's only rational to make this assumption.</p>
<p>So, the argument occasionally comes up that the universe can only be either deterministic or individuals have the ability to make choices. The notion is that, for a choice to be freely made, it must not be constrained by the rules of the universe. If the universe "made our choices for us", we would not be accountable for our actions. It's furthermore complicated because some people believe that God broke the world so that we could make choices, thus giving our choices meaning.</p>
<p>One way to reconcile determinism and free choice is to assert that a part of the mind, the ethereal soul, is not bound by the rules of the universe. This would contradict the definition of the universe, wherein all things that impact another thing in the universe are part of the universe.</p>
<p>There's another potential resolution. There are wheels within wheels in the universe, and our mind is one of them. In our minds, we are capable of formulating abstractions of the universe, metaphysical verses wherein we make many different choices. These metaphysical verses are in the realm of mathematical possibility because they are imperfect and incomplete renditions of the universe. These are not choices. They are options, and they are no more real than their perceived consequences. Also, the simple fact that we only ever choose <em>one</em> option reinforces the notion that the universe is deterministic. Making a choice is bounded by physical laws, sure, but that does not mean that we are any less responsible for them. It's a necessary assumption that all people are responsible for their actions and that gives our choices meaning.</p>
<p>Neither does this diminish the truth underlying the story of how God broke the world to give us choice. Consider a perfect universe, where we define perfection in this case to be one where good things happen by the rules of the verse. In this realm, choices do not render actions good or bad, but the law of the universe. For choices to be meaningful, good and evil must be independent of the laws of the verse. This explains why bad things can happen in a perfect universe. The story does not, however, address the premise that the rules of the universe are independent of an act of creation.</p>
<p>There's also consolation in relegating one's soul to oblivion after mortal death. If anything, mortality heightens the importance of life, this one chance to make good choices. We are custodians of the universe and for each other. We will have a legacy forever, and from a vantage outside our universe, our impact appears to be timeless.</p>
<hr>
<p>I mentioned that I was raised in two religions. I'm an unconfirmed, non-practicing Roman Catholic. I also watched Star Trek. Laugh, but episode after episode, it's a series of complex metaphysical and moral thought experiments: a recipe for premature awakening for a child.</p>
<p>When I was in elementary school, I attended a Catholic program called CCD, or Confraternity of Christian Doctrine. At that age in public school, teachers and administrators appealed to our sense of justice and fairness. It was presumed that people my age only understood <em>quid pro quo</em>, a philosophy of balance and revenge, not forgiveness and stability. In CCD, teachers and administrators appealed to our desire to be loved. But, having watched more than one morality play on Star Trek, I was already an idealist, both of justice and reason. I wanted altruism in school, and I wanted philosophy and logic in church.</p>
<p>I never managed to escape school. As soon as I learned modesty, I deduced that my mother could not take me to school if I refused to get dressed in the morning.</p>
<p>I was wrong.</p>
<p>My attendance was impeccable for the rest of my academic career.</p>
<p>However, after a couple years of coloring in pastoral scenes with Jesus and being told that I should believe a boatload of facts mostly relating to how I should behave, <em>because he loved me so completely</em>, I told my mother that I no longer wanted to go to CCD. My mother is a well-intentioned person with a heart that fills every part of her. She does not fathom why a person would question the doctrine of the church. She also had no idea how to convince me that I needed to go to CCD. So, she telephoned the director of religious education, an intimidating pear-shaped woman with a smoker's voice, and put me on the line. She asked, "Why do you not want to come to CCD." I was mortified. I really had no idea how to articulate my thoughts. I feared offending her. I feared the loss of love. I had nothing to say. I also never went to CCD again. It wasn't until junior college that I met Catholics who believed that there were children for whom CCD was not appropriate.</p>
<hr>
<p>Reading through this article, you have probably noted points that you agreed with or disagreed with. In the past, I've accidentally alienated people with a small number of my thoughts, perhaps not because they disliked me, but because they did not want to be exposed to my point of view and mar their conviction to their beliefs. It's a hard thing to have your faith questioned. It's my hope and care while writing this article to keep you all, my friends and other readers, in my fellowship, my community. It is inevitable that any two of us will disagree on any number of points, but if we hold in our hearts a mote of uncertainty and a well of hope, we can asymptotically approach the truth through discourse together.</p>Kris Kowalhttp://www.blogger.com/profile/01443956999129365941noreply@blogger.com2tag:blogger.com,1999:blog-3618780804544792727.post-58987128208723961552008-11-24T17:27:00.000-08:002008-11-24T18:27:38.750-08:00Google Eyes 2
<p>Ryan Paul <a href="http://askawizard.blogspot.com/2006/05/google-eyes.html">finally</a> found a use for my Google Eyes graphic in his article, <a href="http://arstechnica.com/news.ars/post/20080912-security-expert-google-anonymization-not-anonymous-enough.html">Security Expert: Google not anonymous enough</a>, which was posted a couple months ago.</p>
<p><img src="http://cixar.com/~kris.kowal/bdoc/program/google-eyes.png" alt=""></p>
Kris Kowalhttp://www.blogger.com/profile/01443956999129365941noreply@blogger.com0tag:blogger.com,1999:blog-3618780804544792727.post-49388794472017280862008-11-08T22:11:00.001-08:002008-11-08T23:24:30.331-08:00Blogosphere LinksMy apologies once again: the previous post (which I again hope you haven't read yet), was scraped by Google before I could fix various links that broke in the transition from my Subversion staging area to GData. I wrote a migration script for my old blog that had a hack to fix various Cixar-relative URL's. Unfortunately, when I started fully qualifying all of my links, these hacks started over-correcting my blog posts. I've now removed these hacks and fixed the broken references, but again, please view the updated version on Ask a Wizard if you need to follow links.
Sincerely, The Management.Kris Kowalhttp://www.blogger.com/profile/01443956999129365941noreply@blogger.com0tag:blogger.com,1999:blog-3618780804544792727.post-77286697924117215452008-11-08T20:56:00.000-08:002008-11-08T22:05:49.429-08:00Blogosphere<p>When I moved my blog over to <a href="http://askawizard.blogspot.com">askawizard.blogspot.com</a>, the quickest and easiest way to give my page a unique look was to make my own banner graphic. I decided to make a depiction of the "blogosphere" as I envisioned it based on the mimetically virulent <a href="http://xkcd.com/239/">xkcd 239</a>. Randal Munroe stakes claim to a literal interpretation of the popular term "blogosphere": a layer of the atmosphere where free speech enthusiasts convene aboard various flying platforms and hawk their ideas in a sort of aerial or ethereal bazaar.</p>
<p>According to <tt>xkcd</tt>, <a href="http://craphound.com/">Cory Doctorow</a> from the <a href="http://www.eff.org/">EFF</a> is the only blogger who actually wears a cape and goggles. I caught a certain implication that he's the only person crazy enough to dress up when they passionately jotcast. This is not true. I put on my robe and wizard hat. Don't take that the wrong way.</p>
<p>So, I drew a <a href="http://cixar.com/~kris.kowal/ask-a-wizard.gif">depiction of the blogosphere</a>. Over the last few weeks, I've taken some time to characterize more of my web neighborhood.</p>
<ul>
<li><a href="http://onecreativeblog.com">Ryan</a> and I are in the foreground in the guises we took for our web comic, <a href="http://punnished.com">Punnished</a>.</li>
<li>Cory Doctorow occupies his <a href="http://xkcd.com/497/">ironclad</a> balloon on the opposite side of the scene. <tt>(?:T)?ron</tt> Paul lurks behind in his blimp.</li>
<li>Like the wizards and witches of Harry Potter, in this metaphysical representation of the web, microblogging is facilitated by sending flocks of <a href="http://twitter.com">Twitter</a> bird messengers from balloon to balloon. I called out <a href="http://blog.joshlewis.org">Josh Lewis</a>, a prolific blogger in my network, by putting him and his family in a balloon the same <i>initial</i> color as his old blog, emanating a halo of tweety birds.</li>
<li><a href="http://arstechnica.com">Ars Technica</a> is heralded as one of the largest "blogs" on the Internet. My long-time MUD project partner and friend, <a href="http://arstechnica.com/authors.ars/segphault">Ryan Paul</a> edits the Ars <i>Open Ended</i> journal. While the blogosphere is not quite high enough to qualify as "orbital" in the sense of Ars's purported "Orbital HQ", I decided to put Ryan in a nearby balloon emblazoned with a likeness of the Ars logo. Ryan is wearing a cloak of Roman imperial purple.</li>
<li>One of the earliest bloggers, <a href="http://www.quirksmode.org/">Peter-Paul Koch</a> flies in the dark-blue Quirksmode balloon, from which he provides a steady hail of browser compatibility tables.</li>
<li>Zeppelin enthusiast, <a href="http://simonwillison.net/">Simon Willson</a>, flies a <a href="http://www.djangoproject.com/">Django</a> themed zeppelin. He is trailed by the <a href="http://code.google.com/p/django-pony/">mimetically</a> <a href="http://hackety.org/2008/09/15/documentsRevealDjangoPonyTailOfLies.html">unstoppable</a> <a href="http://avalonstar.com/blog/2008/sep/9/web-framework-ponies/">Django Pony</a>.</li>
<li>A man I believe saved JavaScript from horrible doom, <a href="http://www.crockford.com/">Doug Crockford</a> occupies a balloon in his blog's colors.</li>
<li>As a nod to all the fabulous people who participated in <a href="http://twitter.com/wotw2">#wotw2</a>, a Martian Unpowered Inter-planetary Attack Cannister falls through the scene on the left.</li>
</ul>
<p>I've posted an <a href="http://cixar.com/~kris.kowal/ask-a-wizard-who.png">annotated version</a> of the current banner as a larger image, including a version in the original <a href="http://cixar.com/~kris.kowal/ask-a-wizard.svg">Scalable Vector Graphics format</a>, wherein the annotations are a hidden layer.</p>
<p>I plan to add to the scene every once in a while.</p>Kris Kowalhttp://www.blogger.com/profile/01443956999129365941noreply@blogger.com1tag:blogger.com,1999:blog-3618780804544792727.post-34308501898880942602008-11-08T14:25:00.000-08:002008-11-08T14:26:54.598-08:00iTunes Playlist ImagesGoogle's feed scraper is a bit overzealous about my blog these days. When I uploaded my previous entry (which I hope you haven't ready yet), the image URL's were incorrect. They have been rectified. Please consider following the link to the original article if you find broken images.
Thanks,
The Management.Kris Kowalhttp://www.blogger.com/profile/01443956999129365941noreply@blogger.com0tag:blogger.com,1999:blog-3618780804544792727.post-16344321059640508602008-11-08T12:56:00.000-08:002008-11-08T17:17:46.975-08:00My iTunes Playlist<p>I have almost 9,000 songs, comprising a month of continuous play and 60GB of storage. I've actually listened to maybe half and rated about 20% of the library. Random shuffle doesn't work for me anymore.</p>
<p>When I was in college, I noticed that trying to do homework with lyrical music hindered my productivity, presumably because it engaged my otherwise occupied language brain-matter. I'm also a musician, so I've definitely collected a trove of anecdotal evidence supporting the assertion that exposure and repetition to particular pieces (and broadly but with weaker correlation, certain genres) expands appreciation. Also, overexposure diminishes the effect, presumably because the amygdala begins tuning out the pattern. Randomly traversing my library usually leads to irritation. Also, I thirst for new, rare, serendipitous musical experiences at the expense of occasionally hearing a piece I'm not ready for.</p>
<p>So, on account of what I presume to be common psychological phenomena, my ideal musical experience would consist of:</p>
<ul>
<li>mostly music I know and like</li>
<li>some music I don't know occasionally</li>
<li>no music that I definitely don't like</li>
<li>some constant minimum interval between playback of a given piece.</li>
<li>intervals between hearing a given song inversely proportional to how well I like the piece</li>
<li>over time, my playlist should improve as I provide feedback to the system</li>
</ul>
<p>A few months ago, I found a way to accomplish this with iTunes smart playlists. I construct "pools" of songs. Each pool contains only songs that have a particular rating and haven't been played recently. If there are lot of songs with a particular rating, I generally require a longer interval between plays. So, I created a pool for each rating, and maximum number of songs in each pool to tune the probability of a a song being chosen from each category. Then, I created a master smart playlist that incorporates all of the rating pools.</p>
<p><img src="http://cixar.com/~kris.kowal/bdoc/program/itunes/mixes.png" alt="Playlists"></p>
<h2>
5 stars
</h2>
<p>I have 76 tunes in this category. Almost all of them can make it to the master playlist. Since they play frequently, the dominant factor is when they were last played. The number of songs that haven't been played or skipped in the last 5 days hovers around 20. As a special factor for these, I broke the songs that are longer than 10 minutes into their own category so that only two of them contribute to the master mix at any time. I like Mahler's second symphony a <b>WHOLE LOT</b>, but a half an hour is a long commitment and I prefer to save it for maybe once every other week. I also have about ten different versions of Beethoven's Symphony 7 movement 2, so I might have to make a new category for it so only one of them comes up at a time.</p>
<p><img src="http://cixar.com/~kris.kowal/bdoc/program/itunes/mix-5.png" alt="5 Star Mix Predicates"></p>
<h2>
4 stars
</h2>
<p>200 tunes total. 200 tune limit. 20 days between playing. Defer 10 days if skipped. Stable around 20 tunes contributed to the master playlist, so minimum interval is the dominant factor.</p>
<h2>
3 stars
</h2>
<p>800 tunes total. 100 tune limit, sampled from least recently played. Defer 1 week if skipped. Stable at 100 tunes contributed to the master playlist, so the maximum sample size is the dominant factor.</p>
<p><img src="http://cixar.com/~kris.kowal/bdoc/program/itunes/mix-3.png" alt="3 Star Mix Predicates"></p>
<h2>
2 stars
</h2>
<p>800 tunes total. 20 tune limit, sampled randomly. Defer 4 months when played or skipped. stable at 20 contributed to the master playlist, so the maximum sample size is the dominant factor.</p>
<h2>
1 star
</h2>
<p>I reserve this rating for songs I don't want to hear in a random shuffle, but don't want to delete either. This includes Christmas tunes, apart from the soundtrack to <i>The Nightmare Before Christmas</i>. There are about 400 of these. I should probably use this rating for songs I would like to very rarely hear and use the checkbox to exorcise a song from my random shuffle.</p>
<h2>
0 stars
</h2>
<p>These are songs I've not rated. 6500 tunes. 30 tune limit, sampled randomly. Playing or skipping defers the next chance to play for six months. There are 30 tunes in this playlist, so the dominant factor is the max tune limit.</p>
<h2>
Mix
</h2>
<p>Then I create my main mix playlist from these rating pool lists. The trick is to create an "any" predicate and include all of the smaller playlists with playlist predicate rules.</p>
<p><img src="http://cixar.com/~kris.kowal/bdoc/program/itunes/mix.png" alt="Master Mix Predicates"></p>
<hr>
<p>So, believe it or not, this minimizes my need to fiddle with iTunes and keeps me focused on work. If you've got a ridiculous music library too, I highly recommend this technique.</p>
<p>Also, if you work on iTunes, I highly recommend providing a smart playlist abstraction with equalizer knobs for each rating to automate similar processes; not everyone's a programmer and I bet this problem is just beginning to surface for most folks. It would be nice to see how probable a particular song is to be played based on the size of its pool, the effective sample size of its pool, and the total of all sample sizes. I also want to be able to sort and filter and bulk set "checked" and "unchecked".</p>Kris Kowalhttp://www.blogger.com/profile/01443956999129365941noreply@blogger.com4tag:blogger.com,1999:blog-3618780804544792727.post-46112845668229805562008-11-01T14:35:00.000-07:002008-11-08T00:28:36.589-08:00War of the Worlds 2.0 - The Post Mortem<p>Stuff's just beginning to trickle in reflecting on our Halloween reenactment of The War of the Worlds on Twitter. Here are some links to what we've found so far.</p>
<ul>
<li><a href="http://askawizard.blogspot.com/2008/10/war-of-worlds-20.html">The premise</a></li>
<li>The invasion status feed, <a href="http://twitter.com/wotw2">@wotw2</a></li>
<li>Search for participants using the <a href="http://search.twitter.com/search?q=wotw2">#wotw2</a> hashtag.</li>
<li><a href="http://www.cnet.com/8300-11455_1-10.html">The Buzz Out Loud CNET podcast</a> that Josh Lewis called into to advertise the event</li>
<li><a href="http://arstechnica.com/news.ars/post/20081031-the-call-of-cthubuntu.html">The Call of Cthubuntu</a>, wherein Ryan Paul plugged the event on Arstechnica.</li>
<li><a href="http://blog.wired.com/underwire/2008/10/twitterers-stag.html">Coverage from Wired's Underwire Blog</a></li>
<li><a href="http://digg.com/tech_news/Twitterers_Stage_Mock_Martian_Invasion_a_la_War_of_the_World">Diggs</a> for the Wired article.</li>
<li><a href="http://theparrot.starnewsonline.com/default.asp?item=2280040">Coverage from Ashville, NC</a></li>
<li><a href="http://www.presse-citron.net/twitter-un-truc-dextra-terrestres">French coverage from Presse Citron</a></li>
<li><a href="http://www.youtube.com/watch?v=mJIbgwjdJ7g">Josh Lewis's video</a></li>
<li><a href="http://twitter.com/RhythmHippy">@RhythmHippy</a>'s <a href="http://twitpic.com/photos/rhythmhippy">doctored photos</a></li>
<li><a href="http://www.flickr.com/search/?q=wotw2&ss=2&s=rec">Flickr photos</a></li>
<li><a href="http://172.16.213.128/config/management_network/">@biz</a> Stone's (Twitter co-founder's) tweet.</li>
<li><a href="http://twitter.com/MackReed">@MackReed</a>'s <a href="http://heavylittleobjects.com/?p=990">blog entry</a></li>
<li><a href="http://twitter.com/sea_dot">@sea_dot</a>'s <a href="http://www.quartzfilms.com/blog/?p=395">blog entry</a></li>
<li>Jimmy Roger's <a href="http://www.thetechlife.org/did-you-survive-the-war-of-the-worlds-20-wotw2">post</a> on The Tech Life</li>
<li><a href="http://twitter.com/MackReed">@MackReed</a>'s <a href="http://heavylittleobjects.com/?p=991">retrospective blog entry</a></li>
<li><a href="http://twitter.com/sea_dot">@sea_dot</a>'s <a href="http://www.quartzfilms.com/blog/">retrospective blog entry</a></li>
<li><a href="http://blog.joshlewis.org/2008/11/08/war-of-the-worlds-20-on-halloween-a-retrospective/">Josh Lewis's</a> retrospective blog</li>
<li><a href="http://www.kevindhendricks.com/2008/11/01/war-of-the-worlds-on-twitter/">Kevin Hendriks's</a> retrospective blog</li>
<li><a href="http://cixar.com/svn/wotw2">Archive</a> of everything we could scrape when the event concluded and we realized we were going to want an archive to build cool visualizations from. The archive includes source code you can use to perform your own automated tweets in conjunction with <a href="http://twitter.com/segphault">@segphault</a>'s <a href="https://launchpad.net/gwibber">Gwibber</a> microblog Python library.</li>
<li><a href="http://spreadsheets.google.com/ccc?key=peXwPeIqxoPDBGWH-mHNhUw">The tweet plan spreadsheet</a></li>
</ul>
<p>Each participant averaged 2.6 tweets total. The most active participants posted 60 to 100 tweets. There were around 600 people following the invasion progress. That's about 1500 tweets for all participants over the course of the event. I think it's safe to assume that about 10,000 people were touched by these apocalyptic tidings.</p>
<p>Hopefully, we'll have a participation histogram (total tweet count for each participant) soon. I'd like to see a timeline integrating all of the media we produced.</p>
<p>That was a lot of fun. We'll have to do something similar again some time.</p>Kris Kowalhttp://www.blogger.com/profile/01443956999129365941noreply@blogger.com3tag:blogger.com,1999:blog-3618780804544792727.post-12573261208642040412008-10-19T19:07:00.000-07:002008-10-19T19:24:11.983-07:00What is Tale Anyway?<p>So, I've been working on this project that's currently called Tale for nearly a decade now. The notion is to build an interesting, fun, humorous, immersive, narrative virtual reality. If you're into such things, it's more succinctly called a distributed, web-based <a href="http://en.wikipedia.org/wiki/MUD">MUD</a>. I don't intend to go into <a href="http://en.wikipedia.org/wiki/MUD_trees">detail</a> about how Tale is like a <a href="http://www.brandeis.edu/pubs/jove/HTML/v2/keegan.html">MUD</a> and unlike a MUD. This article is intended to explore some of the novel aspects of Tale.</p>
<h2>
Gameplay
</h2>
<p>Tale's landing page, <a href="http://tale.im/">tale.im</a>, <em>is</em> the game. You click play and the game begins. There won't be a lengthy sign up and character creation process; everyone starts as a ghost in Limbo or gets straight back to playing from wherever they left off. To start playing, you just click or type "start" and you're consciousness is installed in a standard-issue resident of Dya, the Tale world. From there you can start doing whatever it is you like doing most: socializing, exploring, fighting, building, or getting into character.</p>
<p>Playing Tale is like many things you're probably already familiar with. It's like chatting with friends with an Instant Message client, or IRC, or opening a Terminal and issuing commands. MUDs in general combine the social aspects of chatting with friends, with a virtual world simulator and its own narrative and commands. Some MUDs lean in favor of socialization and others on gameplay. Tale favors neither, allowing you to gracefully shift between mostly chatting and mostly playing. Since the game is in a web-browser instead of a specialized chat application or old-fashioned command terminal, Tale can alternate among coherent interactive modes like entering quotes to say aloud like "hello, world!", entering action commands like "go north", typing quick commands with single keys for mini-games like "p" for "parry", "r" for riposte, "f" for feint if you're fencing, or browsing through menus to discover commands visually.</p>
<p>Already in Instant Message and IRC clients, you might be familiar with commands like "/me laughs" (that you can use in AIM) or "/nick cowbert" (that you can use to chose a name in IRC). If you use Vim, you might already be familiar with colon commands like ":w". If you use FireFox's incremental search feature, you know that you can use single-quote and slash to start searching for text as you type, alternating between single-key command mode into a full-line command mode.</p>
<p>Tale's input field will accept similar commands depending on whether you're mostly chatting, mostly playing, or performing quick, expert actions. If you're mostly chatting, you'll be able to issue commands with the slash "/" prefix like <tt>/go north</tt>, and when you're mostly playing you'll be able to chat with a quote prefix like <tt>"hi</tt>. If you're in an expert mode where you're, say, navigating a yacht on the high-sea with discrete arrow key and letter key commands, you can start entering a chat or command with the same keys.</p>
<h2>
Narration
</h2>
<p>Each player in the Tale world has a narrator. This is the charming fellow who listens to your commands and tells you how the story unfolds in response. The narrator is a program that keeps track of what you already know and what you're likely to be interested in and interprets the events occurring around your character like, "Joe says 'hi'.", into just that kind of textual prose, as well as taking your commands and plugging them into the simulator in the selfsame event objects. The ideas of "events" and "things" are central to how Tale works behind the scenes, but it suffices to say that if you are "Elbereth" and you're talking to "Glorfiniel", the narrator would translate events like:</p>
<pre>
Elbereth Say "Hello" to Gorlfiniel
Glorfiniel Say "Hi" to Glorfiniel</pre>
<p>…to story-like narrative accounting for various linguistic and cognitive assumptions that you regularly make — <i>"You say, 'Hello', to Glorfiniel and she says, 'Hi', back."</i>. There is a lot of room for making the narrator more and more interesting as Tale development continues, but there's code in Tale already that creates reasonable paragraphs from a stream of events, accounting for pronouns and such, but still sounds fairly robotic.</p>
<p>Having a complex narrator serves many purposes in Tale. First, it makes the game immersive. It also relieves the burden of creating lots of static content, which is one of the limiting factors that ultimately ends with most games feeling rather shallow. A complex narrator can also abstract the tired MUD practice of BATTLE SPAM, where lines upon lines of discrete hits and jabs rapidly scroll up the chat buffer.</p>
<p>The most subtle benefit of the narrator is that it can introduce a significant burden for programmers seeking to automate their characters, often called "botting". "botting" tends to diminish the quality of gameplay for non-programmer players. With the narrator in our arsenal, we can engage in an arms race with "botters", continuously evolving the narrator to deter botting without banning the practice outright. This incentive structure could be a lot of fun for everyone.</p>
<h2>
Beyond the Narrative
</h2>
<p>The narrative is the centerpiece of Tale gameplay. However, unlike a Telnet or Teletype MUD, Tale is not strictly text-based. Since the game takes place in a web browser, we intend to take advantage of the opportunity to add icons, maps, inventory visualizations, sounds, and atmosphere and lighting changes to punctuate the narrative.</p>
<p>Among other things, we're planning to use "parametric" graphics to underscore the narrative. For example, if you're in a <a href="https://cixar.com/tale/wiki/MallardListLarge">sailing or flying vessel</a>, or riding a <a href="https://cixar.com/tale/wiki/MountListLarge">rabbit</a> around the Tale world, your ride will have a parametric icon. We're using SVG graphic manipulation on the server-side to render permutations of layers from "macro" graphics to produce customized versions of each ride and vessel. The parameters include things like the genetic domain of your steed (Is it descended from an elephant? Did the Narwhal bless it with a tusk?), and the installed rigging on your Zeppelin. The two <a href="https://cixar.com/tale/wiki/Ride">Ride</a> macro graphics have around 100 layers each.</p>
<p>Inspired by <a href="http://blog.joshlewis.org/">Josh Lewis's</a> old blog site, we intend to mix up the style sheet <a href="https://cixar.com/tracs/javascript/browser/trunk/src/color.js">colors</a> on the page slowly and subtly to reflect the time of day and lighting of your surroundings as you explore the world. Each face of the world has its own <a href="https://cixar.com/tale/wiki/HexForce">color scheme</a> and different parts of the world are lit by the 20-sided Sun and various celestial dice, at times filtered through dappled shade, or illuminated by magma and mana. Using alpha-translucent PNG renderings of SVG graphics, and subtly tuning the foreground and background colors of the entire page, we'll gradually modify the mood of the narration.</p>
<p>And there's Music! The musical selection you experience as you roam the world will vary based on what face of <a href="https://cixar.com/tale/wiki/Dya">Dya</a> you're travelling on and the tension level of the narrative: safety, adventure, battle, and peril. Chris Pasillas has composed <a href="http://cixar.com/tale/music/pasillas/">six brilliant themes and variations</a> for Tale and is currently rendering them. The game uses a <a href="https://cixar.com/tracs/javascript/browser/trunk/src/sound.js">variation</a> of Scott Shiller's <a href="http://www.schillmania.com/projects/soundmanager2/">SoundManager</a> library that's been ported to operate as a Chiron JavaScript module. (The largest distinctions are that the Flash bridge code has been factored into its own module, and the API has been simplified. It's a complete rewrite using SM2 as an <a href="http://osflash.org/mtasc">MTASC</a> reference.)</p>
<h2>
Engine
</h2>
<p>The Tale engine is a real-time clock-based event-driven simulation. Every event can trace its cause to a tick of the world clock, which for the time being is imagined to be once per second. The difference between a player character and a non-player character (NPC or MOB in the MUD parlance) is that a non-player character generates a progression of commands on demand using a Python generator that yields Event or Verb object trees in response to observations about their environment and their current state, and a player character has a queue of the same kind of Event or Verb trees. This means that invoking a command in Tale does not render an immediate physical response from the world's engine. Rather, a command is translated by the narrator into an Event and it's enqueued to be invoked when the world clock ticks and visits everything that's not busy or waiting, in the whole world. This levels the playing field between people with nearly instantaneous Internet connections to the Tale server with those who have up to one clock tick worth of lag. It also, in a sense, regulates a certain "conservation of energy" law throughout the world, permitting game mechanics to be tuned more accurately and fairly.</p>
<h2>
World
</h2>
<p>Feature Number One in Tale is the data structure we've chosen to model the world. This data-structure underpins it's potential consistency, algorithmic subtlety, performance, scalability, and distributability. This data-structure conveniently models a world of arbitrary size, scope, and detail and provides the scaffolding for engine effort minimization for event propagation, abstraction, and synthesis.</p>
<p>So, what is it? Well, nothing new really: it's a tree. Depending on the scope and particular region of application, its children have 0, 4, 6, or 9 branches on each node. Most of the world is simply a <a href="http://en.wikipedia.org/wiki/Quadtree">quadtree</a>. Each node has four children, one for each quadrant. The top of the quadtree has a bounding box that encompasses exactly all of contained, square region, and for purposes of Tale, conceptually encompasses all of the air-space above that region. A room is a leaf of the quadtree: a node through which players travel with a unit width and height. Each node is an abstraction of all of its contained rooms, containing the sum of its rooms contents. Roaming the quadtree, each player can divine it surroundings by analyzing the contents and events occurring in its own room, and then those events and contents of each of the more abstract rooms that contain it all the way up to the root of the world tree. Each room is directly connected to its parent and children, and traversing linearly across the plane of the world uses tree traversal algorithms where you zoom in and out of the quadtree to find your destination.</p>
<p>Among the algorithmic possibilities that the quadtree provides a scaffold for, we can provide multi-scale gameplay, the same inspiration behind the recently released, <a href="http://en.wikipedia.org/wiki/Spore_(game)">Spore</a>. The most mundane way this might be applied would be to have larger scale creature roam the world in abstract bounding boxes, like a Fire Drake that would move from 16x16 room to 16x16 room. Room abstractions could also be used for flight or sea navigation, since one travels higher and faster with that medium, somewhat negating the mundane obstacles of the world floor. As an eagle, one might move about the world by zooming in and out instead of traversing in cardinal directions. Also, more abstract forms of gameplay might take place entirely on room abstractions. For example, melee would occur in 1x1 rooms, ranged attack (missiles and magic) in 8x8, tactics at 32x32 with 4x4 tiles, strategy at 256x256 with region-sized tiles, and politics at larger yet. Taking a moment to step out of a first-person player experience, garnering increasingly abstract political support from larger social organizations, gameplay could scale from thief to Queen).</p>
<p>Another algorithmic possibility is maintenance of the world's physical invariants, like global distribution and availability of various resources. Using abstractions to redistribute resources absolutely throughout the world would make it possible to keep the weight of gold throughout the world constant. This would prevent the most common physical inflation problems that most MUDs suffer.</p>
<p>The other major implication is that the quadtree permits entire branches of the world to run on independent servers with minimal cross-talk. During my last year at <a href="http://calpoly.edu/">Cal Poly</a>, <a href="http://www.linkedin.com/pub/1/964/bb9">Ryan Witt</a> engaged his distributed computing team at <a href="http://www.caltech.edu/">Caltech</a> to implement a <a href="http://en.wikipedia.org/wiki/Distributed_hash_table">Distributed Hash Table</a> (<a href="http://en.wikipedia.org/wiki/Chord_project">Chord</a>) for Tale and we extensively discussed the implications of the quadtree for purposes of distribution for scale and redundancy. Ryan and I are now roommates working at <a href="http://www.fastsoft.com/">FastSoft</a>. The topic of distributed computing comes up from time to time, even though we still need to get a vertical gameplay stack working on a single system before we begin implementing distribution.</p>
<h2>
Events
</h2>
<p>One problem <a href="http://www.ifixit.com/">Shawn Tice</a> (who needs his own home page, because he's too awesome not to have one at this juncture) and I struggled with when we were coding up the second Python rendition of the quadtree was event management. The general notion has always been that events would escalate up the quadtree to a scope that contains all objects that can notice the event, then each object would have an opportunity to respond to the event by visiting each node in their room's "scope chain". The borrowed notion of a scope chain, in this case, means the containing room and the <a name="dagger-1-src"></a><a href="http://www.blogger.com/post-edit.g?blogID=3618780804544792727&postID=1257326120864204041#dagger-1">transitive closure†</a> on its parents. Events would each have an "impact" attribute and each object would have a "sense" attribute. So, each time the world ticks, a bottom up visitation of the world would calculate the minimum impact that an event would need to have to be noticed by the one thing in that branch of the world with the highest sensitivity. This would be performed for all of the senses in parallel: visual, aural, olfactory, thermal, thaumaturgical, social, and such. All objects that can have effect events would have senses. For example, a wall can hear a sound so that it can echo it. Most events would also be directed, so they can only propagate in one of the cardinal directions: north, south, east, or west.</p>
<p>However, as you may have already had an inkling of suspicion, this quadtree-based event propagation model isn't actually realistic, and adding realism would nullify the performance and scalability benefits of the system. For example, with the described event propagation system, if you were standing in a room off-center of a quadtree, as at (2, 2) of a 4x4 quadtree, an sound that is loud enough to carry to room (3, 3) might not be loud enough to carry to room (1, 1). The sound would have to be loud enough to carry to (0, 0) before (1, 1) would hear it at all. So, to guarantee that events radiate the same distance in every direction, you need to propagate events laterally, not merely along the quadtree abstraction scope. However, radiating events laterally defeats the performance benefits of restricting perceptible events to those along the scope chain.</p>
<p>So, there was a trade off between performance and realism. Then we remembered we were writing a game. Thinking back to the original <a href="http://en.wikipedia.org/wiki/The_Legend_of_Zelda">Zelda</a>, we observed that each tile of the screen was analogous to a room in Tale, and that every screen in Zelda was analogous to one of the nodes in its abstract node scope chain. That is, there was some parent room that contains about 30 of the rooms Link walked around in. When he got to the edge of the screen, he was no less privy to what was going on in the neighboring room than he was on the opposite side of the screen. So, we decided that the same abstraction would be sufficient for Tale, except that in Tale, the opportunity for zooming in and out around your character was much more profound; we could provide zoom perspectives for every binary order of magnitude.</p>
<h2>
Frameworks
</h2>
<p>The current underlying technologies include <a href="http://python.org/">Python</a>, <a href="http://twistedmatrix.com/trac/">Twisted</a>, <a href="http://en.wikipedia.org/wiki/AJAX">AJAX</a>, <a href="http://cometdaily.com/2007/12/11/the-future-of-comet-part-1-comet-today/">Comet</a>, and <a href="http://www.lighttpd.net/">Lighttpd</a>. In the past, I dabbled with C++ and <a href="http://www.boost.org/">Boost</a>, including an <tt>iostream</tt> system that decoupled input and output, had reasonable variable names, and supported VT100 stream functors, but that code branch has been abandoned since 2002. In addition, the project has inspired the creation of frameworks that I'm calling <a href="http://chironjs.org/">Chiron</a> (a JavaScript module system), and <a href="https://cixar.com/tracs/planes/browser/trunk/src">Python on Planes</a> (a system for persistent HTTP route and responder trees).</p>
<h2>
Parting Words
</h2>
<p>So, Tale is a pretty massive undertaking. When I started this project, I had no idea that good ideas took so much more time to implement than paltry ones, but that lesson's learned. To meet at least one of my New Year resolutions (that is, one of the milestones for Tale), I'm cutting back scope in the short term in order to get the project up and running by January. A lot of this design, if not all of its powerful implication, I plan to realize by then. This weekend, I've been working on sewing the quadtree, the Dyan world design, and the Narrator into a single piece. As always, I'm looking for people who are willing to volunteer to set themselves up with commit access to Tale so that, in the absence of time to spend on the project (which I know none of us have), a sufficient body of people can randomly wake up in a cold sweat with a bad idea and have nothing between them and implementing it for Tale!</p>
<hr />
<p><a name="dagger-1"></a><a href="http://www.blogger.com/post-edit.g?blogID=3618780804544792727&postID=1257326120864204041#dagger-1-src">†</a> The excessively precise term <a href="http://en.wikipedia.org/wiki/Transitive_closure">transitive closure</a>, in this case, means the room's parent, grandparent, great-grand-parent, <i>ad nauseam</i>. <i>Ad nauseam</i>, in this case, means, "until you retch from nausea or meet Adam and Eve: whichever comes first."</p>Kris Kowalhttp://www.blogger.com/profile/01443956999129365941noreply@blogger.com0tag:blogger.com,1999:blog-3618780804544792727.post-22595711363277381632008-10-05T23:21:00.000-07:002008-10-05T23:21:00.503-07:00War of the Worlds 2.0 Update
<p>It turns out I write a lot of words and say very little. Here's what you need to know about the upcoming alien invasion:</p>
<ul>
<li>Follow <a href="http://twitter.com/wotw2">@wotw2</a></li>
<li>On Halloween, tweet in earnest about the unfolding alien invasion. Make your own story. Keep in touch with your friends like you would in any other disaster situation. Try to imagine what you would tweet as the aliens land, emerge, and lay waste to every metropolis around the world.</li>
<li>Email <a href="mailto:wotw2@cixar.com">wotw2@cixar.com</a> if you want to help plan the story or provide technical assistance.</li>
</ul>
Kris Kowalhttp://www.blogger.com/profile/01443956999129365941noreply@blogger.com1tag:blogger.com,1999:blog-3618780804544792727.post-45801781160175150702008-10-05T21:51:00.000-07:002008-10-05T21:51:00.116-07:00War of the Worlds 2.0
<p>Last week, <a href="http://blog.joshlewis.org/">Josh Lewis</a>; friend, former Apple coworker, and <a href="http://www.friendlystegosaurus.com/2007/10/09/052-pastabrain-was-already-taken/">Lex Luthor hairstyle enthusiast</a>; got me thinking about <a href="http://blog.joshlewis.org/">fiction on Twitter</a>. Then, <a href="http://arstechnica.com/authors.ars/segphault">Ryan Paul</a> pointed out that he had written <a href="http://arstechnica.com/news.ars/post/20080901-byte-sized-stories-twittering-a-tiny-tale.html">an article about Twitter fiction</a> already. Here's the idea. Let's (and by "us", I mean everyone) reenact <a href="http://en.wikipedia.org/wiki/The_War_of_the_Worlds_(radio)">The War of the Worlds</a>, on the Internet, for Halloween!</p>
<p>CBS ran the Orsen Welles War of the Worlds radio program in the tense times leading up to the second World War. The aforementioned Wikipedia article claims that Adolf Hitler denounced the program as a sign of democratic decadence. Without commercial breaks and interpolated among real news, the program was broadcast in earnest and public panic ensued. War of the Worlds was the Ultimate Prank, never to be reproduced. As an homage, let's take Halloween to tweet, share bookmarks and status messages, and blog in earnest about the alien invasion in progress. Everyone, tell <i>your</i> story.</p>
<p>The War of the Worlds has several phases that I believe have strong analogs in the tense times leading to World Web II (point oh).</p>
<ol type=1>
<li>a noteworthy astronomer speculates on the possibility of an alien invasion. This would be a good time to talk about the <a href="http://en.wikipedia.org/wiki/Drake_equation">Drake equation</a> in your blog, especially if astronomy is your hobby. Send an email to <a href="mailto:wotw@cixar.com">wotw@cixar.com</a> with your blog so we can proliferate it on the <a href="http://twitter.com/wotw2">@wotw2</a> Twitter account.</li>
<li>The alien invasion occurs. Follow <a href="http://twitter.com/wotw2">@wotw2</a> to keep in sync with the progress of the invasion. This Twitter feed will automatically update, in general terms, the unfolding of the alien invasion like clockwork throughout the world. Coordinate with Tweeters in your area to tell local stories.</li>
<ol type=1>
<li>cylinders fall from the sky. Tweet about where you are. Ask your friends where they are. Form posses. Skip town or take a closer look.</li>
<li>tripods emerge. Flee, get stuck in traffic, or take refuge and tell us what you see.</li>
<li>Martians begin obliterating every Terran metropolitan area with heat rays. Don't call them heat-rays; that would be a dead giveaway. Describe what they do and come up with your own name! Do you work in a public service like hospitals or fire? What's your job and what do you do? Do you organize your coworkers and flee? Do you head for the hills with your go-bag?</li>
<li>Military, local militia, and national guard units get organized and attack the alien invaders. Do you serve in the military? This is your last chance to tell us where you're headed. Do you have family in a militia? Try to keep in touch and let us know how and where they valiantly fought and lost.</li>
<li>The invasion spreads from cities to countryside.</li>
<li>Tripods begin to shut down an malfunction. Are you near one? Do you take a closer look?</li>
</ol>
<li>After the threat dies down, people begin to blog and speculate about what happened, and every topic near and dear to them.</li>
<li>The curtain rises. Blog, link, and tweet about the experience.</li>
</ol>
<p>The "War of the Worlds 2.0" event will be synchronized, on Halloween Friday, with the <a href="http://twitter.com/wotw2">http://twitter.com/wotw2</a> account and we're writing automation with Ryan Paul's <a href="https://launchpad.net/gwibber">Gwibber</a> tool to automatically post Tweets to various services on that day. If you would like to make additional accounts to synchronize local events, please send an email to <a href="mailto:wotw2@cixar.com">wotw2@cixar.com</a> with a Twitter account name and password and we'll hook you up with edit privileges for the <a href="http://spreadsheets.google.com/pub?key=peXwPeIqxoPDBGWH-mHNhUw">tweet plan</a> on Google Docs. Tweets (rows) will be deployed for each column (account).</p>
<p>Let's tell a story!</p>
Kris Kowalhttp://www.blogger.com/profile/01443956999129365941noreply@blogger.com8tag:blogger.com,1999:blog-3618780804544792727.post-63606311313971814322008-10-05T00:38:00.000-07:002008-10-05T00:38:00.578-07:00JSON Basics
<p><a href="http://json.org">JSON</a> is a strict subset of JavaScript, particularly a subset of its object-literal notation, discovered and specified by <a href="http://www.crockford.com/">Doug Crockford</a>. The subset is sufficient to make the creation of parsers and formatters in various languages nearly trivial, while completely trivializing the process of parsing the notation in a web browser.</p>
<p>Object-literals in JavaScript are very similar to their analogs in many other languages, like Perl, Python, and even AppleScript. The notation provisions text strings, numbers, booleans, and the literal <tt>null</tt> for all elemental (scalar) data. Then the notation provides Arrays for ordered values and Objects for unordered, unique, string-to-anything key-value-pair mappings. With these types, you can express most hierarchical data easily.</p>
<pre>
10
3.1415
"a"
true
false
null
[1, 2, 3]
{"a": 10, "b": 20}
[{"a": 10}, {"a": 20}]</pre>
<p>JSON makes a couple simplifications even on JavaScript's object literal grammar. These make it even easier to write parsers and formatters.</p>
<ol type=1>
<li>a JSON expression contains no new-line characters.</li>
<li>all strings, including keys in objects, are enquoted between double quotes.</li>
</ol>
<p>JSON joins the ranks of many other markup languages that we can use <abbr title="split infinitives are cool">to easily transfer</abbr> data among web servers, but its primary function is as a common data interchange language between web browser clients and web servers hosted in the numerous languages of the web. JSON is well adapted for this space because it can take advantage, through various channels, of every web browser's fast, built-in JavaScript interpreter.</p>
<p>There are two flavors of JSON services with subtle differences for both clients and servers. One is JSON proper, which I will call XHR JSON because it uses an XML HTTP Request. The other is called JSONP or "JSON with Padding". You would use one, the other, or neither based on security and performance concerns.</p>
<h2>
XHR JSON
</h2>
<p>XHR JSON uses a feature that exists in one form or another in all modern web-browsers, called an XML HTTP Request, and a function in JavaScript that lets you evaluate arbitrary text strings as JavaScript programs, called <tt>eval</tt>. With an XML HTTP Request, the client can make an HTTP Request to any URL on the same domain as the hosting page. This constraint is called the "Same Origin Policy". Unfortunately, the policy is in many cases not sufficient to isolate vulnerability to paths of a particular domain, nor to prevent cross site scripts from using the data (more on that later). Whether the security mechanism is effective or not is a longer and later discussion. The point being that this mechanism is the crux of your choice between XHR JSON and JSONP.</p>
<p>With an XHR, you request plain text from a URL on the same domain as your page, then you evaluate it as a JavaScript program. Performing a cross-browser compatible XML HTTP Request isn't trivial in itself, so let's assume that I'm using a library that provides an <tt>xhr</tt> function that synchronously (blocks) until an HTTP request is complete and returns the content of the HTTP response as a String. There are other variations on <tt>xhr</tt> for asynchronous requests and grabbing XML.</p>
<pre>
var text = xhr("/some.json");
var json = eval(text);</pre>
<p>In this case, the "/some.json" contains unadulterated JSON. However, because your JSON is likely to be an Object like <tt>{"a": 10}</tt>, we need to make a special arrangement. A JavaScript program that is merely an <tt>Object</tt> literal would be mis-parsed initially as a code block. For this reason, we must force the JavaScript parser into expression context instead of statement context. For this reason, we wrap the JSON string in parentheses or assign it to a variable.</p>
<pre>
var text = xhr("/some.json");
var json = eval("(" + text + ")");</pre>
<p>I prefer parentheses because I haven't, in my fallible memory, encountered a browser that supported XHR but wasn't standards compliant enough for the <tt>eval</tt> function to return the value of the last evaluated expression.</p>
<p>When you use an XHR to grab JSON, you are vulnerable to the host, depending on it to not give you an alternate JavaScript program that insidiously evaluates to the same data. For that reason, a lot of JSON libraries engage in the slower and dubious attempt to validate the JSON before evaluating it with a regular expression. The bottom line is that you're vulnerable to your server when you use <tt>XHR</tt> and <tt>eval</tt>.</p>
<p>That being said, it's better to get an exception early than an untraceable error later. For that reason, attempting to validate JSON is a good idea. There's another one: variable laundering. The <tt>eval</tt> function inherits the calling context's scope chain. This means that the server can read and alter any variables on your scope chain. I use an <tt>evalGlobal</tt> routine to launder my scope chain. This doesn't give security all by itself, but it could help lead to a secure system down the way and can turn a bunch of silent name resolution errors or data leaks into thrown <tt>NameError</tt>s.</p>
<pre>
(function (evalGlobal) {
var text = xhr("/some.json");
validateJson(text);
var json = evalGlobal("(" + text + ")");
...
})(function () {
return eval(arguments[0]);
})</pre>
<p>I'll give a detailed explanation of <tt>evalGlobal</tt> in another article. For now, suffice it to say, this is much more elegant than using a variable to capture the JSON value, although it is possible:</p>
<pre>
var text = xhr("/some.json");
var json;
eval("json = " + text);</pre>
<h2>
JSONP
</h2>
<p>JSONP is different than XHR JSON in both the way the server hosts the data and in the way that the client consumes it. On the client side, the user adds a <tt><script></tt> tag to their own page. The source of the script is the URL of a server side program with the name of a callback function that the client places in global scope sent as a query argument.</p>
<pre><script src="http://other.com/some.jsonp?callback=foo"></pre>
<p>The client script arranges to receive parsed and evaluated JSON data asynchronously by adding a <tt>foo</tt> method to the global scope, the <tt>window</tt> object.</p>
<pre>
window.foo = function (json) {
...
delete window.foo;
};
var script = document.createElement('script');
script.src = "http://other.com/some.jsonp?callback=foo";
document.getElementsByTagName('head').appendChild(script);</pre>
<p>When you add a <tt>script</tt> tag to your document, browsers are kind enough to notice and automatically send an HTTP request to fetch the request JavaScript program. In this case, your page trusts the script on <tt>other.com</tt> to return a JavaScript program that will call your "foo" function and pass it some JSON data.</p>
<pre>foo({"a": 10});</pre>
<p>With JSONP, the client is vulnerable to the remote server because it can opt to write an arbitrary JavaScript program before calling <tt>foo</tt>, if at all. So, you should only use JSONP to request data from domains that you trust.</p>
<p>Also JSONP is slower and less responsive to errors than XHR JSON can be. The only signal that you receive as to the progress or status of a JSONP request is whether your callback gets called in a timely fashion. XHR JSON is preferable in all situations where it is possible and it may even make sense to create a proxy to the other domain from your same domain server. Using a proxy also gives you an opportunity to validate the JSON on the server-side, where computation time is cheap.</p>
<h2>
Same Origin XHR JSON
</h2>
<p>There's another trick with XHR JSON, and I'm not sure what scenarios require it. Some people can use the JSONP technique to fetch JSON from a foreign domain. These folks can't send HTTP headers or cookies, and they never get an opportunity to alter the text of the HTTP request before it's evaluated as JavaScript in their browser. I would think that it would be sufficient to prevent an other domain client from intercepting sensitive data for the server to require an authenticated token in the HTTP request, as can be provided by an XML HTTP Request but not a JSONP request. However, some crackers can attempt to get data from your service by using JSON data in a cross site script, much like JSONP. However, instead of using a callback, these crackers arrange to override the <tt>Array</tt> or <tt>Object</tt> constructors and thus can monitor and transmit snippets of JSON constructed by simply evaluating JavaScript object literal notation. This technique can be prevented by padding your XHR JSON service with an infinite loop.</p>
<pre>
while (1);
{"a": 10}</pre>
<p>In this case, your client conspires with the server, and since it does get a chance to intercept and modify the text of the JSON, it strips the known number of characters for the while loop before evaluating it.</p>
<pre>
var text = xhr("/some.json");
var json = evalGlobal("(" + text.substring("while (1);".length) + ")");</pre>
<p>What baffles me is that, unless you've secured your JSON with an authentication token, any server capable of making HTTP requests and parsing JSON could do the same thing. Malicious clients are not required to use web browsers. The only situation where this might occur would be if the cracker had compromised your client already, had your authentication token, and needed your browser to make a cross domain JSONP request on its behalf. That could not be the case since the server would have no reason to give an XHR JSON authentication token to a client that could only fetch data with JSONP. That point is even moot since, going back, the cracker has already compromised your client and might as well use XHR JSON with all the same rights as you on even the same origin.</p>
<p>That being said, I noticed that GMail used this technique, so I assume there's substance to it. Perhaps someone will clarify in the comments.</p>
Kris Kowalhttp://www.blogger.com/profile/01443956999129365941noreply@blogger.com0tag:blogger.com,1999:blog-3618780804544792727.post-83547895940882721712008-10-03T15:13:00.000-07:002009-03-18T14:48:36.083-07:00Designing Django's Object-Relational-Model - The Python Saga - Part 6<p>Django is a web application framework in the Python language. One of the advantages that Django has over other libraries is that it was written and designed by Python experts. That is to say, they knew about variadic arguments, properties, and metaclasses. Furthermore, they knew how to cleverly use these ideas to sweep a lot of complexity under the hood so that common developers, or uncommonly good developers who want to think about other things most of the time, can gracefully suspend disbelief that anything complicated is going on when they design their database in pure Python. This article will illustrate how Django uses metaclasses and properties to present an abstraction layer where you can specify a database schema with Python classes. For simplicity, the "database" backend will be plain Python primitive objects—tables will be dictionaries of dictionaries.</p>
<p>In the end, we want to be able to write code that looks a whole lot like it's using Django's Object-Relational-Model:</p>
<pre>
class Cow(Model):
id = PrimaryKey()
name = ModelProperty()
cow = Cow(id = 0, name = 'Moolius')
cow.save()
cow = Cow.objects.get(0)</pre>
<p>The easiest part (for the purpose of this exercise) is Django's concept of an "object manager". In Django, every model has an object manager that provides a query API and, depending on the backend, might cache instances of Model objects. Conveniently, a very narrow subset of the object manager API is almost exactly the same as a dictionary. Conceptually, the object manager boils down to a dictionary proxy for the database where you can use the <tt>get</tt> function to retrieve records from the database. For simplicity, our <tt>ObjectManager</tt> is just going to be a dictionary.</p>
<pre>
class ObjectManager(dict):
pass</pre>
<p>Beyond the scope of this article, the <tt>ObjectManager</tt> should be handy for grabbing lots of objects from the database at once. Django provides a very thorough and relatively well-optimized lazy query system with its object managers. The <tt>ObjectManager</tt> has <tt>get</tt>, and <tt>filter</tt> methods which, instead of simply accepting the primary key, accept keyword arguments that translate to predicate logic rules. In particular, the <tt>filter</tt> function is lazy, so you can chain filter commands to construct complex queries and Django only goes to the database once.</p>
<p>While it would be super-cool to model all of this with native Python, it actually is a lot of code, so that's a topic for maybe later. We'll just use the built in <tt>dict.get</tt>.</p>
<p>We'll also need all of the code from <a href="http://askawizard.blogspot.com/2008/10/ordered-properties-python-saga-part-5.html">Part 5</a> since models will be another application of the ordered property pattern. This is how Django creates SQL tables with fields in the same order as the Python properties.</p>
<pre>
from ordered_class import \
OrderedMetaclass,\
OrderedClass,\
OrderedProperty</pre>
<p>We use the <tt>OrderedMetaclass</tt> to make a <tt>ModelMetaclass</tt>. The model metaclass will have all the same responsibilities as our <tt>StructMetaclass</tt>, including "dubbing" the properties so that they know their own names. The model metaclass will also create an <tt>ObjectManager</tt> for the class. This isn't the complete <tt>ModelMetaclass</tt>; we'll come back to it.</p>
<pre>
class ModelMetaclass(OrderedMetaclass):
def __init__(self, name, bases, attys):
super(ModelMetaclass, self).__init__(name, bases, attys)
if '_abstract' not in attys:
self.objects = ObjectManager()
for name, property in self._ordered_properties:
property.dub(name, self)</pre>
<p>The next step is to create a <tt>ModelProperty</tt> base class. This class will be an <tt>OrderedProperty</tt> so it's sortable. It will also need to implement the <tt>dub</tt> method so it can figure out its name. Other than that, it'll be just like the <tt>StructProperty</tt> from the previous section: it will get and set its corresponding item in the given object.</p>
<pre>
class ModelProperty(OrderedProperty):
def __get__(self, objekt, klass):
return objekt[self.item_name]
def __set__(self, objekt, value):
objekt[self.item_name] = value
def dub(self, name):
self.item_name = name
return self</pre>
<p>There is a distinction in the refinement of <tt>ModelProperty</tt> from <tt>StructProperty</tt>: <tt>ModelProperty</tt> objects will eventually need to distinguish the value stored in the dictionary from the value returned when you access an attribute. In the primitive case, they're the same, but for <tt>ForeignKey</tt> objects, down the road, you'll store the primary key for the foreign model instead of the actual object. This is the same as the behavior in an underlying database backend.</p>
<pre>
class ModelProperty(OrderedProperty):
def __get__(self, objekt, klass):
return objekt[self.item_name]
def __set__(self, objekt, value):
objekt[self.item_name] = value
def dub(self, name):
self.attr_name = name
self.item_name = name
return self</pre>
<p>Let's consider a <tt>PrimaryKey</tt> <tt>ModelProperty</tt>. The purpose of a <tt>PrimaryKey</tt> is to designate a property of a model that will be used as the index in its object manager dictionary. In Django, this can be an implicit <tt>id</tt> field at the beginning of the table. For simplicity in this exercise, we'll require every model to explicitly declare a <tt>PrimaryKey</tt>. The <tt>ModelMetaclass</tt> will identify which of its ordered properties is <i>the</i> primary key by observing its type. Other than their type, a primary key's behavior is the same as a normal <tt>ModelProperty</tt>, so it's a really easy declaration:</p>
<pre>
class PrimaryKey(ModelProperty):
pass</pre>
<p>Now we can go back to our <tt>ModelMetaclass</tt> and add the code we need for every class to know the name of its primary key. I create a list of <tt>PrimaryKey</tt> objects from my <tt>_ordered_properties</tt> and pop off the last one, leaving error checking as an exercise for a more rigorous implementation. There should be only one primary key.</p>
<pre>
class ModelMetaclass(OrderedMetaclass):
def __init__(self, name, bases, attys):
super(ModelMetaclass, self).__init__(name, bases, attys)
if '_abstract' not in attys:
self.objects = ObjectManager()
for name, property in self._ordered_properties:
property.dub(name)
self._pk_name = [
name
for name, property in self._ordered_properties
if isinstance(property, PrimaryKey)
].pop()</pre>
<p>Now all we need is a <tt>Model</tt> base class. The model base class will just be a dictionary with the model metaclass and a note that it's abstract: that is, it does not have properties so the metaclass better not treat it as a normal model.</p>
<pre>
class Model(OrderedClass, dict):
__metaclass__ = ModelMetaclass
_abstract = True</pre>
<p>The model will also have a special <tt>pk</tt> attribute for accessing the primary key and a <tt>save</tt> method for committing a model to the <tt>ObjectManager</tt>.</p>
<pre>
class Model(OrderedClass, dict):
__metaclass__ = ModelMetaclass
_abstract = True
def save(self):
self.objects[self.pk] = self
@property
def pk(self):
return getattr(self, self._pk_name)</pre>
<p>Now we have all the pieces we need to begin using the API. Let's look at that cow model.</p>
<pre>
class Cow(Model):
id = PrimaryKey()
name = ModelProperty()
cow = Cow(id = 0, name = 'Moolius')
cow.save()
cow = Cow.objects.get(0)</pre>
<p>All of this works now. You make a cow model; that invokes the model metaclass that sets up <tt>Cow._pk_name</tt> to be <tt>"id"</tt> and tacks on a <tt>Cow.objects</tt> object manager. Then we make a cow and put it in <tt>Cow.objects</tt> with the save method. This is analogous to committing it to the database backend. From that point, we can use the object manager to retrieve it again.</p>
<p>We can refine the <tt>Model</tt> base class to take advantage of the fact that it's not just a dictionary anymore: it's an <i>ordered</i> dictionary. We create a better <tt>__init__</tt> method that will let us assign the attributes of our <tt>Cow</tt> either positionally or with keywords. That makes our cow more like a hybrid of a list and a dictionary. Also, since our model instances aren't merely dictionaries, we create a new <tt>__repr__</tt> method that will note that cows are cows and moose are moooose. The new <tt>__repr__</tt> method also takes the liberty to write the items in the order in which their properties were declared.</p>
<pre>
class Model(OrderedClass, dict):
…
def __init__(self, *values, **kws):
super(Model, self).__init__()
found = set()
for (name, property), value in zip(
self._ordered_properties,
values,
):
setattr(self, name, value)
found.add(name)
for name, value in kws.items():
if name in found:
raise TypeError("Multiple values for argument %s." % repr(name))
setattr(self, name, value)
…
def __repr__(self):
return '<%s %s>' % (
self.__class__.__name__,
" ".join(
"%s:%s" % (
property.item_name,
repr(self[property.item_name])
)
for name, property in self._ordered_properties
)
)</pre>
<p>Now we can make a cow model with positional and keyword arguments, and print it out nice and fancy-like:</p>
<pre>
>>> Cow(0, name = 'Moolius')
<Cow id:0 name:"Moolius"></pre>
<p>The next step is to introduce <tt>ForeignKey</tt> model properties. These are properties that will refer, via a relation on a primary key, to an object in another model. So, the <tt>ForeignKey</tt> class will accept a <tt>Model</tt> for the foreign model. Its dub method will override the <tt>item_name</tt> (preserving the <tt>attr_name</tt>) provided by it's super-class's <tt>dub</tt> method. The new <tt>item_name</tt> with be the <tt>attr_name</tt> and the name of the primary key from the foreign table, delimited by an underbar. This will let the foreign key property hide the fact that it does not contain a direct reference to the foreign object; it just keeps the foreign object's primary key. However, if you access the foreign key property on a model instance, it will go off and diligently fetch the corresponding model instance. If you assign to the foreign key property, it'll tolerate either a primary key or an actual instance.</p>
<pre>
class ForeignKey(ModelProperty):
def __init__(self, model, *args, **kws):
super(ForeignKey, self).__init__(*args, **kws)
self.foreign_model = model
def __get__(self, objekt, klass):
return self.foreign_model.objects.get(objekt[self.item_name])
def __set__(self, objekt, value):
if isinstance(value, self.foreign_model):
objekt[self.item_name] = value.pk
else:
objekt[self.item_name] = value
def dub(self, name):
super(ForeignKey, self).dub(name)
self.item_name = '%s_%s' % (
name,
self.foreign_model._pk_name,
)</pre>
<p>Now we can write code with more than one model using relationships. Let's give our cow a bell.</p>
<pre>
class Bell(Model):
id = PrimaryKey()
class Cow(Model):
id = PrimaryKey()
name = ModelProperty()
bell = ForeignKey(Bell)
bell = Bell(0)
bell.save()
cow = Cow(0, 'Moolius', bell)
cow.save()</pre>
<p>Note that you must <tt>save</tt> the bell so that when you construct the cow, it can fetch the bell from <tt>Bell.objects</tt>.</p>
<p>There's more to Django's ORM, of course. This article doesn't cover parsing and validation, which are both assisted by the ORM. Nor does it cover queries, query sets, the <tt>related_name</tt> for <tt>ForeignKey</tt> properties on foreign models, Django's ability to use strings for forward references to models that have not yet been declared, or many of the other really neat features.</p>
<p>What this article does cover though, is that you can create a powerful abstraction of a proxied database with pure-Python in less than 200 lines of code. This means that you could create a light-weight proxy over HTTP to a Django database that exposes itself with a REST API. You could also create an abstraction layer that would allow you to pump Django ORM duck-types back into Django to use pure Python objects in addition to or in stead of a database backend.</p>
<p>But, if this article does nothing else, I hope it communicates that Django is cool. I have read a whole lot of code from every dark corner of the web and I have liked very little of it; people I've worked with will testify that I've regularly "hated on" every library or framework I've ever seen. I've never met <a href="http://holovaty.com/">Adrian Halovalty</a>, <a href="http://www.pointy-stick.com/blog/">Malcolm Tredinnick</a>, or <a href="http://simonwillison.net/">Simon Willson</a> and the growing developer community around Django. However, I've read their code and now I can tell you, over the course of several articles, that they're really smart and you should use their <a href="http://www.djangoproject.com/">code</a>.</p>Kris Kowalhttp://www.blogger.com/profile/01443956999129365941noreply@blogger.com3tag:blogger.com,1999:blog-3618780804544792727.post-11782264542563997092008-10-01T20:38:00.000-07:002008-10-01T20:38:00.810-07:00Ordered Properties - The Python Saga - Part 5
<p>In C and SQL, structs and records have properties that appear in a particular order. In Python, objects use hash-tables to store their attributes, so the order is not deterministic nor relevant. That's no consolation if you're trying to model C structs or SQL tables in Python though. Reading though the Django code, I discovered that those wily coders had synthesized a technique that combines the virtues of properties and metaclasses that we can generally model the order in which fields of a struct or SQL table appear.</p>
<p>The trick is to provide an API that allows your users to write code like:</p>
<pre>
from time import gmtime
from cstruct import Struct, IntegerProperty
class EpochTime(Struct):
seconds = IntegerProperty()
microseconds = IntegerProperty()
epoch_time = EpochTime()
timestruct = gmtime(epoch_time.seconds)</pre>
<p>In this case, the <tt>Struct</tt> type cares about the size, order, and disposition of the C structure it models so that it can unpack a byte buffer presumably retrieved from a C-type library.</p>
<p>There are two components to the general solution. First, there's an <tt>OrderedProperty</tt> base type. Ordered properties track the order in which they were initialized. For this purpose, ordered properties use a global counter. The absolute value of a property's creation counter is irrelevant. The only requirement is that they monotonically increase as each property is declared, and that classes declared concurrently do not interfere.</p>
<pre>
from itertools import count
next_counter = count().next</pre>
<p>I learned about <tt>itertools</tt> from <a href="http://sayspy.blogspot.com/">Brett Cannon</a> who was preparing his thesis defense when I was at <a href="http://http://www.calpoly.edu/">Cal Poly</a>. In another piece of code, which I would cite if I could recall, I learned the trick of using the <tt>itertools.count</tt> function to atomize a global counter. It takes advantage of Python's dubious global interpreter lock.</p>
<p>Then the <tt>OrderedProperty</tt> base type just stores a creation counter upon initialization. Keep in mind that all derived types must trickle their <tt>__init__</tt> call—your base types are not always what they seem and there are use cases you can not foresee where you might be required to receive and pass your initialization arguments to an unknown super-type. That's a side-effect of working in a language with mix-ins, and it's a "good thing".</p>
<pre>
class OrderedProperty(object):
def __init__(self, *args, **kws):
self._creation_counter = next_counter()
super(OrderedProperty, self).__init__(*args, **kws)</pre>
<p>The next part of the puzzle is inspecting your property order. We use a metaclass to note when new types are created and we give them an <tt>_ordered_properties</tt> attribute with the ordered-property items in their attributes. Keep in mind that base types might have properties too. <tt>__mro__</tt> is an attribute of all classes that is a linearized list of a classes inheritance hierarchy. It's effectively the result of a dynamic topological sort algorithm for the closure of your base types. We traverse it backwards so that if you create a dictionary via <tt>dict(_ordered_properties)</tt>, name collisions are resolved chronologically.</p>
<pre>
class OrderedMetaclass(type):
def __init__(self, name, bases, attys):
super(OrderedMetaclass, self).__init__(name, bases, attys)
self._ordered_properties = sorted(
(
(name, value)
for base in reversed(self.__mro__)
for name, value in base.__dict__.items()
if isinstance(value, OrderedProperty)
),
key = lambda (name, property): property._creation_counter,
)</pre>
<p>Then, we create our <tt>OrderedClass</tt> that we can inherit to get free <tt>_ordered_properties</tt> attributes.</p>
<pre>
class OrderedClass(object):
__metaclass__ = OrderedMetaclass</pre>
<p>Let's see it in action:</p>
<pre>
class Foo(OrderedClass):
bar = OrderedProperty()
baz = OrderedProperty()
Foo._ordered_properties == [
('bar', <Ordered Property instance somewhere>),
('baz', <Ordered Property instance somewhere>),
]</pre>
<p>With a small modification, we can track ordered inner classes too.</p>
<pre>
class OrderedMetaclass(type):
def __init__(self, name, bases, attys):
super(OrderedMetaclass, self).__init__(name, bases, attys)
self._creation_counter = next_counter()
self._ordered_properties = sorted(
(
(name, value)
for base in reversed(self.__mro__)
for name, value in base.__dict__.items()
if isinstance(value, OrderedProperty)
or isinstance(value, OrderedMetaclass)
),
key = lambda (name, property): property._creation_counter,
)</pre>
<p>So, we put all of those order classes in our generic ordered properties module. Now we can delve into our particular <tt>Struct</tt> example.</p>
<p>First, we need field types. We will need a base type and derivative types for all of the field types we want to model from C, like <tt>IntegerField</tt>. Bear in mind that <tt>OrderedProperty</tt> is just a mix-in; it doesn't actually define <tt>__get__</tt> and its ilk because those are all specific to the derivative implementation. All struct fields are going to store their actual values in a special <tt>_value</tt> dictionary on their corresponding <tt>Struct</tt> instance.</p>
<pre>
class StructProperty(OrderedProperty):
def __get__(self, objekt, klass):
return objekt._values[self.name]
def __set__(self, objekt, value):
objekt._values[self.name] = value
def dub(self, name):
self.name = name
return self</pre>
<p>You'll notice that instances of <tt>StructProperty</tt> need to know their <tt>name</tt>, the key in their corresponding instance's <tt>_values</tt> dictionary. The struct property instances don't get this from their initializer. Instead, each property will get "dubbed", given a name, when the <tt>StructMetaclass</tt> visits its <tt>_ordered_properties</tt>.</p>
<p>Let's just look at an integer.</p>
<pre>
class IntegerProperty(StructProperty):
_format = 'i'</pre>
<p>The only thing special about an integer is that it's format specifier for the <tt>pack</tt> routine is "i".</p>
<p>We're going to want to use a metaclass again, this time inheriting from our <tt>OrderedMetaclass</tt>. The purpose of this metaclass will be to analyze the ordered properties, construct an aggregate format specifier for <tt>pack</tt> and <tt>unpack</tt> methods, and to <tt>dub</tt> each of its properties with their name.</p>
<pre>
class StructMetaclass(OrderedMetaclass):
def __init__(self, name, bases, attys):
super(StructMetaclass, self).__init__(name, bases, attys)
for name, property in self._ordered_properties:
property.dub(name)
self._format = "".join(
property._format
for name, property in self._ordered_properties
)</pre>
<p>All that remains is to create our API base class, <tt>Struct</tt>. This class initializes its values and declares its metaclass.</p>
<pre>
from struct import unpack
class Struct(OrderedClass):
__metaclass__ = StructMetaclass
def __init__(self, *args, **kws):
super(Struct, self).__init__(*args, **kws)
self._values = {}
def unpack(self, value, prefix = None):
if prefix is None: prefix = ""
for (name, property), value in zip(
self._ordered_properties,
unpack(prefix + self._format, value)
):
self._values[name] = value</pre>
<p>So, now our epoch time example is possible using <tt>Struct</tt> and <tt>IntegerProperty</tt>, completely oblivious to the machinations behind the scenes.</p>
<pre>
from time import gmtime
from datetime import datetime
class EpochTime(Struct):
seconds = IntegerProperty()
microseconds = IntegerProperty()
epoch_time = EpochTime()
timestruct = gmtime(epoch_time.seconds)
datestruct = (timestruct[:6] + (epoch_time.microseconds,))
print datetime(*datestruct)</pre>
Kris Kowalhttp://www.blogger.com/profile/01443956999129365941noreply@blogger.com0tag:blogger.com,1999:blog-3618780804544792727.post-15348985708536672642008-09-30T22:45:00.000-07:002008-09-30T22:45:00.753-07:00Metaclasses - The Python Saga - Part 4
<p>The original <tt>type</tt> function, whose behavior is preserved in modern Python 2.5, accepts an object and returns the class, albeit the type, that would emit it. It's like the <tt>typeof</tt> operator in JavaScript that returns the String name of the primitive type of an object, or the C++ function that returns a pointer to an object's virtual function table. They're all sufficient for comparing apples to oranges, but all of them are also insufficient for the more interesting comparison of apples to the idea of a Fiji apple: the question, "Does your type inherit from this?", that can be accomplished with Python's <tt>isinstance</tt>, JavaScript's <tt>instanceof</tt>, or C++'s infernal <tt>dynamic_cast</tt>. So, <tt>type</tt>'s single argument behavior is effectively retired.</p>
<p>At some transcendental moment, somebody deeply involved in the Python project must have been thinking, "Well, if functions and classes return objects, what returns a class? Could a class, like a property, be syntactic sugar for some deeply metaphysical latent behavior in pure Python?". I figure this is how the <tt>type</tt> function grew its new wings.</p>
<p>So consider a class declaration:</p>
<pre>
class Foo(object):
bar = 10
def __init__(self, bar = None):
if bar is not None:
self.bar = bar</pre>
<p>This is what is actually happening behind the curtains:</p>
<pre>
name = 'Foo'
bases = (object,)
def __init__(self, bar = None):
if bar is not None:
self.bar = bar
attys = {'bar': bar, '__init__': __init__}
Foo = type(name, bases, attys)</pre>
<p>That is to say, there is no magic in the syntax. Ultimately all of the magic happens when you call <tt>type</tt>. By "magic" I mean functionality that cannot be replicated in pure Python without the interpreter's intervention.</p>
<p>The <tt>type</tt> function returns a type: a function that returns new instances. It's also called a "metaclass". <tt>type</tt> just happens to also be the implied metaclass of <tt>object</tt>. That is to say, you can create your own metaclasses.</p>
<p>The big question about metaclasses is, "Why on earth would you want to define a metaclass?". David Mertz from IBM wrote that you would simply know when you needed them. Since I read that article, I've wracked my mind for a reason to use metaclasses to no avail. At some point, I was reading Django's ORM code and it occurred to me that the reason you would want to define a metaclass is to provide a class in your API that, when subclassed by unsuspecting users, would invoke certain preparations without their knowledge or consent. Here's how:</p>
<p>Define a metaclass. The best way to define a metaclass is to inherit <tt>type</tt> and override its <tt>__init__</tt> method.</p>
<pre>
class FooType(type):
def __init__(self, name, bases, attys):
super(FooType, self).__init__(name, bases, attys)
print '%s was declared!' % name</pre>
<p>Define a base class for your API. The trick here is that you can override its metaclass. Let's look at this one in an interactive interpreter:</p>
<pre>
>>> class Foo(object):
... __metaclass__ = FooType
...
Foo was declared!
>>></pre>
<p>Whoa! You didn't call anything. Not true. Here's what actually happened:</p>
<pre>
name = 'Foo'
bases = (object,)
attys = {}
attys['__metaclass__'] = FooType
Foo = attys.get('__metaclass__', type)(name, bases, attys)</pre>
<p>Python checks your attributes for a metaclass before defaulting to <tt>type</tt>.</p>
<p>That means that your <tt>FooType.__init__</tt> got called. Hot damn. I wonder what happens if you create a subclass.</p>
<pre>
>>> class Bar(Foo):
... pass
...
Bar was declared!
>>></pre>
<p>Whoa! I totally inherited a metaclass.</p>
<p>So, the reason for writing a metaclass is that metaclasses give you an opportunity to get and manipulate your derived class objects before anyone instantiates them. You get to do this once, right after the class dictionary is fully populated. You can take this opportunity to monitor class declarations, to prepare additional attributes, or to interpolate additional base types.</p>
<hr>
<p>Keep in mind that metaclasses are jealous. If you create a metaclass for a type that inherits from base classes in someone else's API, your metaclass must inherit from their metaclass. I suspect that it's best not to assume that your base types use a particular metaclass. Thankfully, you can use an expression for your base type.</p>
<pre>
class FooType(getattr(Bar, '__metaclass__', type)):
pass
class Foo(Bar):
__metaclass__ = FooType</pre>
<p>This takes advantage of the Python idiom of accessor methods like <tt>dict.get</tt> and <tt>getattr</tt> that accept a default-if-none-exists argument. Unfortunately, Python's <tt>object</tt> doesn't explicitly state that <tt>type</tt> is its metaclass. Otherwise, you could safely say:</p>
<pre>
class FooType(Bar.__metaclass__):
pass</pre>
<p>Such things are to be looked for in Python 3. I find that the Python developers have either, after considerable review and debate, already accepted or rejected most of my ideas before I even consider them, so I'm not even going to check for a PEP on this one.</p>
Kris Kowalhttp://www.blogger.com/profile/01443956999129365941noreply@blogger.com0tag:blogger.com,1999:blog-3618780804544792727.post-38970290572902194842008-09-29T23:34:00.002-07:002008-09-30T13:56:19.633-07:00Properties - The Python Saga - Part 3
<p>Properties come out of a tired programming language genesis. In the beginning, there were structs. The trouble with structs was that an opaque data structure could not programmatically monitor or intercept access and mutation of its member data.</p>
<p>So that's not a big deal; we could solve the problem with classes. The best practice to avoid programming yourself into a corner was to <i>never</i> expose a datum; you would write accessor and mutator functions, whether you needed them at the moment or not. Thus, as your design grew, you <i>could</i> eventually do nice things like validation, observation, or proxying. The trouble with this approach was that you had to write six times as much code on the off chance you'd need to extend it some day. But it was worth it.</p>
<p>The idea of managed properties came along eventually in various languages (Python, C#, some implementations of JavaScript, and recent versions of [C]). The notion is that you would initially write all of your classes like structs with member data camped in public view. You would encourage your API consumers to interact with those members directly. Then, as need arose, you would subvert the member variables with property objects. These objects would intercept accesses and mutations with functions that you could write at any time of your design process.</p>
<p>Lets observe this design shift in Python. Here's a class with unmanaged data:</p>
<pre>
class Foo(object):
def __init__(self);
self.bar = 10</pre>
<p>Here's some other fellow's code that uses your class:</p>
<pre>
foo = Foo()
foo.bar = 20
print foo.bar
del foo.bar</pre>
<p>So there you have it. Just to keep on the same page, the idea at this point is to add a feature to Python that permits both of those code samples to work and, in-fact, be perfectly cromulent. However, we also want to eventually add features to <tt>Foo</tt> such that its <tt>bar</tt> attribute can be managed, validated, proxied, secured, or outright lied about. Enter <tt>property</tt>. <tt>property</tt> is a function that accepts an accessor function and optional mutator and deleter functions. The property must be a class attribute to work. Here's how you would use a property:</p>
<pre>
class Foo(object):
def __init__(self):
self.bar = 10
def get_bar(self, objekt, klass):
return self.baz / 2
def set_bar(self, objekt, value):
self.baz = value * 2
def del_bar(self, objekt):
del self.baz
bar = property(get_bar, set_bar, del_bar)</pre>
<p>Now we have a Foo class that transparently maintains the invariant that "bar" will always be half of "baz".</p>
<p>Sometimes you don't need to have a setter for a property, and you almost never need a deleter. For the common case, you can use the <tt>property</tt> function as a decorator.</p>
<pre>
class Foo(object):
def __init__(self):
self.baz = 20
@property
def bar(self):
return self.baz / 2</pre>
<h2>
Creating the <tt>property</tt> function.
</h2>
<p>So, it's easy to assume that the <tt>property</tt> function does all the magic behind the scenes, setting up traps in your class's accessor and mutator paths. There's actually another layer of code that can be done entirely in Python. That is, we can implement the <tt>property</tt> function in pure Python. The trick is that the <tt>property</tt> function is actually a type or factory method (who cares which) that returns a Python duck-type: a property object. A property object is any object that implements <tt>__get__</tt>, <tt>__set__</tt>, or <tt>__del__</tt>. These are special magic Python functions that intercept access, mutation, and deletion on members. All you have to do is install an object on a class with one of methods defined, with the name of the member you want to manage. The <tt>property</tt> function just handles the common cases. Let's redefine the property function in Python, as the <tt>Property</tt> class.</p>
<pre>
class Property(object):
def __init__(self, fget):
self.fget = fget
def __get__(self, objekt, klass):
return self.fget(objekt)</pre>
<p>This defines enough of the <tt>Property</tt> object to decorate an accessor function.</p>
<pre>
class Foo(object):
def __init__(self):
self.baz = 20
@Property
def bar(self):
return self.baz / 2</pre>
<p>Here's a full implementation of <tt>Property</tt>. You will note that, in order to exactly emulate the <tt>property</tt> object, the <tt>__init__</tt> method has the same argument names as the internal <tt>property</tt> so that code that uses keyword arguments will function in perfect ambivalence.</p>
<pre>
class Property(object):
def __init__(
self,
fget,
fset = None,
fdel = None,
doc = None,
):
self.fget = fget
self.fset = fset
self.fdel = fdel
self.__doc__ = doc
def __get__(self, objekt, klass):
return self.fget(objekt)
def __set__(self, objekt, value):
self.fset(objekt, value)
def __del__(self, objekt):
self.fdel(objekt)</pre>
Kris Kowalhttp://www.blogger.com/profile/01443956999129365941noreply@blogger.com0tag:blogger.com,1999:blog-3618780804544792727.post-4525804917563619382008-09-28T21:42:00.001-07:002008-09-30T13:56:18.801-07:00Decorators - The Python Saga - Part 2
<p>Python introduced a short-hand for the adapter pattern on functions. You can "decorate" a function with another function. This is a neat tool you can use to factor out some common code from a bunch of functions. You can fiddle with the arguments, return values, or intercept exceptions thrown by any function you decorate.</p>
<p>The canonical example is a memoize decorator. The idea is to generalize the notion of memoization so you can simply subscribe to it in any function you want to memoize.</p>
<pre>
def factorial(n):
if n == 1: return 1
return n * factorial(n - 1)
factorial = memoize(factorial)</pre>
<p>You accomplish this by writing the <tt>memoize</tt> decorator. A decorator is a function that accepts a function and returns another. Python virtuously provides a shorthand for taking the function, decorating it, and assigning it to a variable with the same name.</p>
<pre>
@memoize
def factorial(n):
if n == 1: return 1
return n * factorial(n - 1)</pre>
<p>In the imagined normal case of decorators, the returned function accepts the same arguments and returns the same kinds of values as the accepted function. However, a decorator does have the liberty of extending or restricting that interface, like accepting additional arguments or raising an exception if the arguments are of the wrong type. It might also perform some common computation on the original arguments and pass the result to the original function as an additional argument. In any case, you can use some closures to create a decorator:</p>
<pre>
def memoize(function):
cache = {}
def decorated(*args):
if args not in cache:
cache[args] = function(*args)
return cache[args]
return decorated</pre>
<p>Of course, that's too simple. A lot of things you put after the "@" symbol are just functions that return decorators so that they can be configured with arguments. For example, you probably want to make a <tt>memoize</tt> decorator that lets you specify your own <tt>cache</tt> object. So, you need another layer of deference.</p>
<pre>
def memoize(cache = None):
if cache is None: cache = {}
def decorator(function):
def decorated(*args):
if args not in cache:
cache[args] = function(*args)
return cache[args]
return decorated
return decorator
@memoize({})
def factorial(n):
if n == 1: return 1
return n * factorial(n - 1)</pre>
<p>Since, in Python, functions, objects, and types are indistinguishable to the casual observer, you can do the exact same thing with a class, although I shudder to think that you might want to forgo the simplicity and elegance of closures. After the transform, the previous code might look like this:</p>
<pre>
class memoize(object):
def __init__(self, cache = None):
self.cache = cache
def __call__(self, function):
return Memoized(function, self.cache)
class Memoized(object):
def __init__(self, function, cache = None):
if cache is None: cache = {}
self.function = function
self.cache = cache
def __call__(self, *args):
if args not in self.cache:
self.cache[args] = self.function(*args)
return self.cache[args]
@memoize()
def factorial(n):
if n == 1: return 1
return n * factorial(n - 1)</pre>
<p>So now you can use a Least Recently Used Cache, assuming it is a dictionary-like-object (a duck-dict, if you will):</p>
<pre>
from lru_cache import LruCache
@memoize(LruCache(max_size = 100, cull = .25))
def factorial(n):
if n == 1: return 1
return n * factorial(n - 1)</pre>
<p>Download <a href="http://cixar.com/~kris.kowal/bdoc/program/python/decorators.zip">decorators.zip</a>.</p>
Kris Kowalhttp://www.blogger.com/profile/01443956999129365941noreply@blogger.com3tag:blogger.com,1999:blog-3618780804544792727.post-42934036095581118242008-09-28T00:09:00.001-07:002008-09-30T13:56:17.433-07:00Variadic Positional and Keyword Arguments - The Python Saga - Part 1
<p>Python supports "variadic" arguments. Variadic arguments are the man behind the curtain for C's <tt>printf</tt> function. The idea is that a function can accept a variable number of positional arguments, the values to put in your format string. In C this is accomplished with an ellipsis, <tt>...</tt>, and some <tt>VA</tt> macro-linked-list-stuff that I always have to look up. Python goes a couple steps further with variadic arguments and the results are stunning, orthogonal, and <i>actually useful</i> almost every day. With Python, you get both "positional" arguments, like C, and keyword arguments: those arguments that conceptually map, in any order, to the names of the arguments in your function's declaration. The magic symbols are "*" and "**" for positional and keyword arguments respectively. With one "*", you can declare a function that accepts any number of arguments as the declared list object:</p>
<pre>
def foo(*args):
return args
assert foo(1, 2, 3) == [1, 2, 3]</pre>
<p>You can also pass an array of positional arguments to a function with very similar syntax:</p>
<pre>
def foo(a, b, c):
print [a, b, c]
assert foo(*[1, 2, 3]) == [1, 2, 3]</pre>
<p>And you can do the same thing with keyword arguments except you use dictionaries:</p>
<pre>
def foo(**kwargs):
return kwargs
assert foo(a = 10, b = 20, c = 30) == {'a': 10, 'b': 20, 'c': 30}</pre>
<p>Likewise, you can pass keyword arguments:</p>
<pre>
def foo(a, b, c):
print [a, b, c]
assert foo(**{'a': 10, 'b': 20, 'c': 30}) == [10, 20, 30]</pre>
<p>You can use them in combination, along with default arguments to provide beautiful, orthogonal, reusable abstractions:</p>
<pre>
def foo(a, b = None, c = None, d = None):
return [a, b, c, d]
assert foo(*[1, 2], **{'c': 3}) == [1, 2, 3, None]
def bar(a, b, c, *args, **kws):
return [a, b, c], args, kws
assert bar(1, 2, 3, 4, 5, f = 6) == ([1, 2, 3], [4], {'f': 5})</pre>
Kris Kowalhttp://www.blogger.com/profile/01443956999129365941noreply@blogger.com3tag:blogger.com,1999:blog-3618780804544792727.post-9472572637244365952008-09-27T11:37:00.000-07:002008-09-30T13:59:28.459-07:00It Shtarts with a Bloody Esh.
<p>The light saber application for the iPhone was recently "upgraded" to a naggy advertisement application for a game that I will not give the benefit of calling out by name. If, like me, you need to "downgrade" your iPhone application, fear not, there is a way.</p>
<p>If you upgraded the application directly on the phone, the old version may be backed up on your main iTunes library. If you have docked since you got the application originally, and have not docked since you upgraded, just delete the application from your phone then dock and sync. The old version will be restored.</p>
<p>If you have already docked and synced your iPhone with your iTunes library, it will appear that the new application is both on your phone and your computer with no sign of the old one. Check the trash—it might still be there. Copy the <tt>.ipa</tt> (<u>iP</u>one <u>A</u>pp) file for your application to your <tt>iTunes/Mobile Applications</tt> directory and open it in iTunes. It will prompt you for whether you would like to replace the newer version. Just say, "YES!".</p>
Kris Kowalhttp://www.blogger.com/profile/01443956999129365941noreply@blogger.com0