MetaCard Corporation

Part 6: Performance Benchmarks

Some simple measures of performance were taken for each of the languages. The scripts perform the same operations, but were written using the most common idioms for each language. See and compare the scripts used for each language: Tcl, Ksh, MetaTalk, and Perl.

Note that these scripts have been converted to HTML to be readable in WWW browsers. If you need the scripts in ready-to-run form, get

Test # of loops Tcl Ksh MetaTalk Perl5
Simple loop 1000000 111 103 4 5
Iterative factorial 10000 260 88 9 8
Iterative factorial with 'if' 10000 345 117 14 10
Recursive factorial 10000 403 1921 59 63
String operations (stems) 10 202 221 22 16
File write and read 2000 63 104 50 22
Run Subprocess 1000 9 21 23 24
Set widget state 5000 30 52 16 30
Set with locked screen 5000 1 5 2 1
Start up to simple window 1 1 1 1 2
Start up to complex window 1 3 5 2 6

The test machine was a 90 MHz Pentium with 32 MB RAM running UnixWare 2.0. Versions used were Tcl 7.4 pl3 with Tk 4.0 pl3, dtksh for UnixWare 2.0 (from, ksh version M-12/28/93, dtksh version 1 August 1995), MetaCard 2.0, and Perl 5.002 with Tk-b11.02 (based on Tk 4.0 pl3). All times are in seconds. Script execution times were taken using each language's built-in time functions, but were verified using the UNIX time command. Times were averaged over at least two runs.

The first test measured the performance of a simple loop operation, adding 1 to a numeric variable within the loop. The second test adds a multiply for computing factorials. The third added an if-then statement to the factorial loop. The fourth called a factorial subroutine recursively instead of using a repeat loop.

Two things stand out in these numeric-operation benchmarks: Tcl has relatively poor performance in numeric operations, and dtksh has huge overhead when calling subroutines that return values (due to the fact that ksh procedures can only return values between 0 and 255 without forcing a special redirection-of-output operation).

The string operation benchmark extracts words from the Linux dictionary file /usr/dict/words (350K/38470 words) that meet a certain criteria: they must be 5 letter words that are not plurals, don't contain apostrophes, and at least one other 5 letter non-plural must share the same first 3 letters. While this may seem like an unusual task, it is a real-world example of a type of task that all of these languages were designed to solve easily. In this case the words were used as stimuli in a psychological experiment that measured performance in a word-completion task (the subject saw three letters followed by two underscores and asked to report the first 5-letter word they thought of).

The ksh implementation of this benchmark is not directly comparable to the other two since ksh lacks the important "sort" operator needed to sort the associative array keys. The scripts should also have converted the words to lower-case, but this was also omitted from this benchmark: Tcl and MetaTalk have operators for performing this conversion built in, but ksh does not and the steps required to implement it would drastically reduce its already poor performance.

As in the factorial benchmarks, there is a striking difference between performance of the two languages that use conventional interpreters (Tcl and Ksh) and the two that use "virtual compiler" technology (MetaTalk and Perl), the latter being at least 9 times as fast as the former.

The next two benchmarks show that there isn't as much difference between the languages for file and process intensive operations, presumably because the OS and hardware are the limiting factors here.

The last four tests measure the GUI performance of the tools. To improve measurement of the start-up tests, the GUI measurements were made running the applications across an unloaded network to an HP 9000/700 display. This additional overhead slowed down start-up times allowing more precise stop-watch measurements.

The update test measures performance getting a value out of a widget, incrementing it, and then putting it back. All three tools allow suppression of updates, and so performance was also measured when suppressing updates (the comments in the scripts show how this was done). Even without this using feature it is clear that the tools have plenty of display updating performance. Given the performance bottlenecks of malloc and the client-server communication path, writing an application in C probably offers little performance advantage in display update intensive applications. Indeed, the much shorter times for Tcl/Tk and MetaCard relative to dtksh on the update test compared with the first two tests is an indication that update-intensive applications written in C with Xt/Motif are likely to be even slower than if they had been written in Tcl/Tk or MetaCard.

The "start up to simple window" benchmark shows that starting up an application with only a small number of controls is very fast with all three tools. On the other hand, adding 256 buttons to the window as was done for the "startup to complex window" benchmark shows that there is considerably more overhead for managing widgets in the Xt/Motif-based Desktop KornShell than in the other two tools. The Perl/Tk benchmark shows that there is considerable overhead in the "glue" routines required to connect Perl to the Tk library.

Benchmark conclusion

The general conclusion that can be drawn is that any of the languages should be plenty fast enough to keep up with a human being. MetaCard's considerable performance advantage over the other two make it the best tool to use for developing larger applications, especially for applications that do a lot of numeric processing.

Another characteristic of the languages that isn't reflected in the benchmarks was the relative stability of the tools. While many programming errors were made developing these tests in all three languages, MetaCard and Tcl never crashed during development. dtksh, on the other hand dumped core dozens of times during development of these simple tests. Strange as it may sound, the most "mature" of these tools was also the least reliable.

The performance differences between Perl5 and MetaCard are significant for some benchmarks, but not for others. The raw-statement execution speeds of the two languages are almost identical, but Perl has several "shortcuts" that allow faster execution of some operations. For example, requiring different operators for comparing numbers and strings gives Perl a considerable boost in the factorial with 'if' and string operations benchmarks. MetaTalk lacks this requirement and so has to check to see if the operands are, or can be converted to, numbers in each comparison. This has some performance penalty, but simplifies the process of writing scripts.

In general, the MetaTalk language design philosophy has been to simplify syntax whenever possible, even if it means minor costs in performance. For example, of the 17 "Common Goofs for Novices" listed in "Programming Perl" (Larry Wall and Randal L. Shwartz, 1990, O'Reilly & Associates, p. 361-364), 16 can't occur with MetaTalk because the syntax is so much simpler. Particularly noteworthy is goof #3 "Using == instead of eq and != instead of ne" for comparisons, which is the aforementioned performance-enhancing Perl feature. Even so, unlike the difference between the performance of MetaTalk vs ksh or Tcl, MetaTalk performance is not qualitatively different from that of Perl5.

Some of benchmarks for Tcl have improved with the release of the new byte-code compiler for Tcl 8, but preliminary results indicate that the speedup is far short of what is needed to bring Tcl up to parity with MetaTalk and Perl. The primary reason for this lack of improvement is that the design of the Tcl language is such that it is often impossible to preprocess arguments to function calls as is required to precompile them. Consider the following example:

set x 7; set y 9
puts stdout "$x + $y = [expr $x + $y]"
=> 7 + 9 = 16
The problem here is that the [] around the expr function is inside quotes, which means it cannot be evaluated until after the puts command is executed, which means it can't be precompiled into byte code. This technique is commonly used by Tcl developers, and there is no built-in support in the language for distinguishing what can be compiled and what can't be. Developers will have to spend considerable time developing the skill to determine where execution occurs, and therefore whether or not optimization can be performed. In this sense, MetaTalk represents a difference in capability over Tcl that is comparable to that made by structured languages like Pascal over GOTO-dependent languages like BASIC: by separating argument processing from string handling, MetaTalk provides language-level support for ensuring high-performance applications.

On to Application deployment, documentation, and support

Back to Sample Application

Up to Table of Contents