Unicode

What I've learned today—

In order for a program or library to operate in a Unicode compatible fashion, all strings must be in Unicode. All input strings must be brought into Unicode, and all output strings must be sent out of Unicode at the very last possible moment. This is because, outside of Unicode strings, encoding is not a function of type and type information does not generally cross API boundaries accurately, plus regular expressions don't play well against mixed-length characters.

Django works entirely in Unicode and drops a string to UTF-8 at the last moment. I had need for Python's Textile library to transform text, but it only dealt with byte strings. Anyhow, It turned out to be quick work to change all of its strings to unicode and not bother with encoding and decoding.

Integrating Simile

Not to name names, but I've been working on integrating code from Simile from MIT into Chiron. Refactoring an existing JavaScript project highlights all the things you get for free in Chiron.

Simile has it's own XHR engine, DOM event wrappers, DOM layout and style functions, PNG transparency solution, and a SortedArray type that provides binary search functions. Here are some of my observations.

• Simile's layout getSize was better than mine. I will rectify this.
• Not having a module system makes us reinvent the wheel: frequently.
• It's hard to write a good XHR engine. There are a lot of XHR modules out there, most of them have some issue or another: missing browser, doesn't report OK status on local files, doesn't unify browser caching inconsistencies, doesn't support timeouts, doesn't expose XML (the X in AJAX) in IE, or so on. If you're going to make a new one, you should use these as references and do some serious research, development, and testing. Otherwise, you should copy or use the best of them (jQuery, in my opinion). Also, it needs to support asynchronous (the A in AJAX) requests, and you need to use them as often as possible.
• Not having a solid, modular library makes us lazy. The inconvenience of name-spacing makes us lazy. This causes us to write sloppy code. For example, we should always use an enquote function when we're string interpolating HTML attributes and an inoculate function when we're interpolating HTML, or we should use DOM functions or a DOM wrapper API to generate our HTML.
• As I integrate code from other libraries, a pattern emerges. In my first pass, I collapse the name-spaces. Every module is a name-space, so all the manual creating of hierarchies like Simile.DOM (Simile = {} presumed, then Simile.DOM = {}, then endless repetition of Simile.DOM to augment or use its contents) is unnecessary and undesirable.

Referencing URL's of resources, like other scripts and images, relative to the URL of the script you're currently in, is hard. Starting from scratch, this usually means you're going to have a global URL constant. This means domain-coupling. Maybe you make the URL relative to the root. This means domain-coupling. Maybe you provide it as a configuration variable. This means site-coupling. Maybe you scrape the script tags on the page for the URL of your script then resolve the URL relative to your own URL. This means you're going to write a lot of slow code for what you perceive to be little value. In Chiron, you can get a function called resolve from http.js that resolves a URL relative to a base URL. Chiron also provides your modules with a moduleUrl variable that is the URL of the script you're in. resolve also implicitly uses this variable as your base URL if you don't provide a second argument(include('http.js'); resolve('images/blah.gif')).

Chiron grabs the script tag href of modules.js and removes the script object from the DOM (so other scripts can't sniff it) exactly once, since it needs that URL to resolve other module URL's. From there, Chiron keeps track of where all of your modules are relative to it and provides that information to each module.

• About SortedArray:
• A collection type should create an empty instance if you pass no arguments in.
• A collection type should populate itself from the values of another collection if you pass one in as its first argument. This should always be the first argument, even if you frequently create empty collections with overrides on later arguments. Force your user to pass in a null or undefined.
• Try to accept null and undefined as equivalent unless the distinction is meaningful.
• Try to distinguish null and undefined from 0 in all meaningful cases.
• Invariants like "sorted" are a promise. Guarantee your invariants across all function calls, including construction. If this means an unacceptable performance degradation, permit the user to suppress whatever code you need to verify the invariant if they are willing to provide treated data.
• If there is a reasonable default, it should always be implicit. I should not have to explicitly send the global compare function into a SortedArray if I want a SortedArray of types supported by compare.
• Not having a system of base types makes for noisy API's where names from different organizations have different meanings. For example, find functions should always accept the same kinds of arguments and return the same kinds of values. Simile's name choices are very close to mine, to the effect that they could almost be used as partially implemented duck types for mine, but some of the names would have to be realigned. find in Simile accepts a comparator and returns an acceptable index to insert or remove a particular element. find in Chiron returns an index or key at which an item can be inserted or removed, and guarantees that it will be the first occurrence of a given value (not a comparator). It was very easy to refactor SortedArray to subscribe to the strict model. Also, removeAll needed to be clear, length and getCount both needed to be getLength, getIterator needed to be iter, next needed to throw StopIteration once in a while, among others.

I'm looking forward to having a semblance of Simile Timeplot and Timechart in the Chiron family.

Polymorphic repr

I'm not about to debate whether debugging is a valuable exercise in JavaScript, nor whether introspection or reflection tools are useful, nor whether they would be especially helpful in a dynamic language. Ruby has inspect. Python has repr. The Chiron JavaScript library has repr too.

Notionally, repr is the inverse of eval for a reasonable subset of JavaScript. There are a lot of object hierarchies that cannot be reconstructed from a repr serialization, but as a debugging tool, repr is indispensable.

repr is a polymorphic function you can import from base.js. If you pass repr an object that implements a repr member function, repr will defer to your overridable repr. Otherwise, repr returns reasonable defaults for other types. For example, repr provides defaults for Array, Object, String, Number, Boolean, and Date. repr also recursively represents members of arrays and objects, but provides circular reference protection by tracking visited objects in a memo Set.

Chiron's debugger uses repr to convert the value of an expression on the command line to a human-readable string.

j$ 1
1
j$ repr(1)
"1"
j$ "hi"
"hi"
j$ repr("hi")
"\"hi\""
j$ true
true
j$ {a: 10}
{"a": 10}
j$ [1, 2, 3]
[1, 2, 3]
j$ [{a: 10}]
[{"a": 10}]
j$ var a = {}; a.a = a; a
{"a": <cycle>}
j$ type()()
<instance run.html#0 0>
j$ type({'repr': function () {return "x"}})()
x

Tale Mailing List

Please subscribe to Tale's mailing list if you are interested in joining the invitation alpha in July or the beta in November.

Some Handy Commands

I've learned a few handy commands this year, so here's the wealth.

Sets have always been a big hole in my command line toolkit. I've gone to such lengths as to implement commands for basic set intersection and union in Perl or Python. Go no farther! While looking over Lasermacaroni's shoulder about a year ago, I noticed that he used the comm command. comm operates on sorted unique streams of lines and produces a three column output of which elements were in column A only, column B only, or both. Of course, three columns are nearly useless, so you can tell comm to suppress the output of certain columns. Yes, this is stupid, but here's your mnemonic: ask not what columns comm can display for you, rather ask what columns you can suppress from comm. Go!

A & B: comm -12 A B
A & !B: comm -23 A B
!A & B: comm -13 A B
A | B: cat A B | sort | uniq

Of course, I like to use comm like grep or grep -v to find the elements from an input stream that either are or are not in a given file. Mind that your file needs to be sorted and uniqued.

also in A:  ... | sort | uniq | comm -12 - A
not in A:   ... | sort | uniq | comm -23 - A

I've known about the seq command in Linux-land for a while. It creates lists of numbers in a given range.

seq last
seq first last
seq first stride last

For examples:

$ seq 3
1
2
3
$ seq 3 5
3
4
5
$ seq 0 2 4
0
2
4

I was quite disappointed not to find seq on Mac OS X. Turns out the BSD folks have a pretty bad case of NIH. Instead of the seq function, you may, having the good fortune of working around brilliant people every day, notice a friend, coworker, or other friendly mammal use the jot command to produce their streams of numbers.

jot [reps [begin [end [stride]]]]

Here are some occlusive examples:

$ jot 3
1
2
3
$ jot 3 5
5
6
7
% jot 3 0 6
0
3
6
% jot 3 1 1
1
1
1
jot -b+ -s- 40
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

fin

modules.js

We've posted modulesjs.com this evening to proliferate what could be the most important piece of code I've ever written, modules.js. This JavaScript is the core of my Chiron JavaScript library project for the Tale game user interface, but could stand alone as the core of any number of JavaScript library projects. Every popular JavaScript library could benefit from a module system that handles dependencies and isolates name spaces like Python or Ruby. Read the tutorial and look over the code. Find me on an instant messenger or my face in the big blue box or send me an email and tell me what you think.

Thanks go out to Ryan Witt for managing the server and domain, my sister Kathleen Kowal for the graphic design, Ryan Paul, Ryan Ernst, and Mike Stone for proofreading and particularly to Ryan Paul for letting me expound at him nigh daily for the last decade.

New Year's Resolutions

I don't generally make New Years resolutions. But, since I'm a software engineer, and I've got some milestones to plot, I've decided to post a release schedule for my personal software projects. This year, I'm going to release Tale.

This involves continued development on Chiron JavaScript for the front-end, Python on Planes for the back-end, and lots of new musical, graphical, and programmatic content.

I'm of course still looking for volunteers (occasionally paid help, if I must and it's worth it) to develop various parts. Chris Pasillas has been working on composing music for the game; we've got a lot of content and had a meeting [mp3] last week to develop creative direction. However, if someone's equipped to render orchestrated versions of the music from the MIDI's (I've got Jonathan DeKlotz in mind for this, but he's a busy fellow too), that would be a super-awesome contribution. We could also use some help from smart, creative, driven (perhaps by angst, anger, or escapism; whatever works for you) programmers who know or are eager to learn Python to develop the actual game engine logic and "mob procs"; this is a lot of work and the bulk of player experience, but I must admit that I'm only marginally interested in developing this aspect of the game, plus I've got a lot of work ahead of me to re-integrate PoP and Chiron. I've had some help from Scott, and Josh "Jynx" Jenkins to develop the game math, plus some ideas of my own in the works, but someone who can dedicate programming time to make ideas real could have a big impact. There are also a lot of Python and JavaScript mini-projects that I may or may not have time to develop myself with the remaining year of schedule, many of which would be very interesting, like intelligent narrative generation from event description trees, SVG sprite composition and rendering (partially implemented), much more 2d vector art for content, game state logic (making lots of use of Python generators if you want to learn about that), and much more. Most of all, Tale needs volunteers who are willing to give a part of their soul, as I have, to the project. Here's the development schedule.

January 2008
Sun	Mon	Tue	Wed	Thu	Fri	Sat
		1	2	3	4	5
6	7	8	9	10	11	12
13	14	15	16	17	18	19
20	21	22	23	24	25	26
27	28	29	30	31
1	Post Chiron 0.1 (done)
February 2008
Sun	Mon	Tue	Wed	Thu	Fri	Sat
					1	2
3	4	5	6	7	8	9
10	11	12	13	14	15	16
17	18	19	20	21	22	23
24	25	26	27	28
2	Deploy Chiron Website
May 2008
Sun	Mon	Tue	Wed	Thu	Fri	Sat
				1	2	3
4	5	6	7	8	9	10
11	12	13	14	15	16	17
18	19	20	21	22	23	24
25	26	27	28	29	30	31
3	Post Python on Planes 0.1
June 2008
Sun	Mon	Tue	Wed	Thu	Fri	Sat
1	2	3	4	5	6	7
8	9	10	11	12	13	14
15	16	17	18	19	20	21
22	23	24	25	26	27	28
29	30
1	Release Chiron 1.0
1	Begin Tale Invitation Beta
December 2008
Sun	Mon	Tue	Wed	Thu	Fri	Sat
		1	2	3	4	5
6	7	8	9	10	11	12
13	14	15	16	17	18	19
20	21	22	23	24	25	26
27	28	29	30	31
5	Begin Tale Public Beta
January 2009
Sun	Mon	Tue	Wed	Thu	Fri	Sat
				1	2	3
4	5	6	7	8	9	10
11	12	13	14	15	16	17
18	19	20	21	22	23	24
25	26	27	28	29	30	31
1	Deploy Tale 1.0

This year will also mark another major step in the collision of my hobby-life and work-life. I've never mentioned this in my personal blog, but I've been working at Apple for the last year and a half, but I'll be moving on to work for a start-up spun off of a networking lab at Caltech called FastSoft where I will be developing the web user-interface for their Aria, a wide-area network performance optimizer. This means I'll be much busier, and I'll be able to work on open-source projects like Chiron as part of my job.

Looking back at all of my personal projects for the last eight years, a pattern emerges. I want my code to help as many people as possible. As for my motivations, as I realized over lunch with Ryan Witt and Christine Ortega over lunch last week, I'm an artist — I'm performing for an audience and am gratified for appreciation of my creations. I'm a fan of Reusability and Repurposability with big R's. This has implications for the nature of the projects I find interesting. I prefer to write libraries and languages: crystalline, orthogonal, refined, applicable, general, powerful, hosting usefulness through emergent patterns: uses unforeseen. Reusability is also best served by proliferation, and what better way to proliferate a thing by making it valuable and free. This isn't economics where value, cost, supply, and demand have interesting but vague relationships; I mean value as usefulness minus cost. Also, what better way to make a thing valuable than to permit your users to exercise and refine it themselves and give them every reason in the world to share their contributions. On a selfish note, I prefer to write software I can use anywhere, even if I'm working on something proprietary. What I'm getting at here is Open Source, with a free, non-viral license. I've chosen other licenses in the past for various purposes, withholding a piece of value for myself for money or the cause of freedom, but for the raw purposes of Reusability, Repurposability, proliferation, and the advancement of the state of the art, I've chosen to use a permissive, free, non-viral license. So, my contributions to Chiron and PoP, at work and at home, will be available with the MIT license.

Happy New Year; let's make.