Part 6: Performance Benchmarks

Note that these scripts have been converted to HTML to be readable in WWW browsers. If you need the scripts in ready-to-run form, get ftp://ftp.metacard.com/MetaCard/contrib/bench.tar.gz.

Test

# of loops

Tcl

Ksh

MetaTalk

Perl5

Simple loop	1000000	111	103	4	5
Iterative factorial	10000	260	88	9	8
Iterative factorial with 'if'	10000	345	117	14	10
Recursive factorial	10000	403	1921	59	63
String operations (stems)	10	202	221	22	16
File write and read	2000	63	104	50	22
Run Subprocess	1000	9	21	23	24
Set widget state	5000	30	52	16	30
Set with locked screen	5000	1	5	2	1
Start up to simple window	1	1	1	1	2
Start up to complex window	1	3	5	2	6

The test machine was a 90 MHz Pentium with 32 MB RAM running UnixWare 2.0. Versions used were Tcl 7.4 pl3 with Tk 4.0 pl3, dtksh for UnixWare 2.0 (from ftp.novell.de, ksh version M-12/28/93, dtksh version 1 August 1995), MetaCard 2.0, and Perl 5.002 with Tk-b11.02 (based on Tk 4.0 pl3). All times are in seconds. Script execution times were taken using each language's built-in time functions, but were verified using the UNIX time command. Times were averaged over at least two runs.

The first test measured the performance of a simple loop operation, adding 1 to a numeric variable within the loop. The second test adds a multiply for computing factorials. The third added an if-then statement to the factorial loop. The fourth called a factorial subroutine recursively instead of using a repeat loop.

Two things stand out in these numeric-operation benchmarks: Tcl has relatively poor performance in numeric operations, and dtksh has huge overhead when calling subroutines that return values (due to the fact that ksh procedures can only return values between 0 and 255 without forcing a special redirection-of-output operation).

The string operation benchmark extracts words from the Linux dictionary file /usr/dict/words (350K/38470 words) that meet a certain criteria: they must be 5 letter words that are not plurals, don't contain apostrophes, and at least one other 5 letter non-plural must share the same first 3 letters. While this may seem like an unusual task, it is a real-world example of a type of task that all of these languages were designed to solve easily. In this case the words were used as stimuli in a psychological experiment that measured performance in a word-completion task (the subject saw three letters followed by two underscores and asked to report the first 5-letter word they thought of).

The ksh implementation of this benchmark is not directly comparable to the other two since ksh lacks the important "sort" operator needed to sort the associative array keys. The scripts should also have converted the words to lower-case, but this was also omitted from this benchmark: Tcl and MetaTalk have operators for performing this conversion built in, but ksh does not and the steps required to implement it would drastically reduce its already poor performance.

As in the factorial benchmarks, there is a striking difference between performance of the two languages that use conventional interpreters (Tcl and Ksh) and the two that use "virtual compiler" technology (MetaTalk and Perl), the latter being at least 9 times as fast as the former.

The next two benchmarks show that there isn't as much difference between the languages for file and process intensive operations, presumably because the OS and hardware are the limiting factors here.

The last four tests measure the GUI performance of the tools. To improve measurement of the start-up tests, the GUI measurements were made running the applications across an unloaded network to an HP 9000/700 display. This additional overhead slowed down start-up times allowing more precise stop-watch measurements.

The update test measures performance getting a value out of a widget, incrementing it, and then putting it back. All three tools allow suppression of updates, and so performance was also measured when suppressing updates (the comments in the scripts show how this was done). Even without this using feature it is clear that the tools have plenty of display updating performance. Given the performance bottlenecks of malloc and the client-server communication path, writing an application in C probably offers little performance advantage in display update intensive applications. Indeed, the much shorter times for Tcl/Tk and MetaCard relative to dtksh on the update test compared with the first two tests is an indication that update-intensive applications written in C with Xt/Motif are likely to be even slower than if they had been written in Tcl/Tk or MetaCard.

The "start up to simple window" benchmark shows that starting up an application with only a small number of controls is very fast with all three tools. On the other hand, adding 256 buttons to the window as was done for the "startup to complex window" benchmark shows that there is considerably more overhead for managing widgets in the Xt/Motif-based Desktop KornShell than in the other two tools. The Perl/Tk benchmark shows that there is considerable overhead in the "glue" routines required to connect Perl to the Tk library.

Benchmark conclusion

Another characteristic of the languages that isn't reflected in the benchmarks was the relative stability of the tools. While many programming errors were made developing these tests in all three languages, MetaCard and Tcl never crashed during development. dtksh, on the other hand dumped core dozens of times during development of these simple tests. Strange as it may sound, the most "mature" of these tools was also the least reliable.

The performance differences between Perl5 and MetaCard are significant for some benchmarks, but not for others. The raw-statement execution speeds of the two languages are almost identical, but Perl has several "shortcuts" that allow faster execution of some operations. For example, requiring different operators for comparing numbers and strings gives Perl a considerable boost in the factorial with 'if' and string operations benchmarks. MetaTalk lacks this requirement and so has to check to see if the operands are, or can be converted to, numbers in each comparison. This has some performance penalty, but simplifies the process of writing scripts.

In general, the MetaTalk language design philosophy has been to simplify syntax whenever possible, even if it means minor costs in performance. For example, of the 17 "Common Goofs for Novices" listed in "Programming Perl" (Larry Wall and Randal L. Shwartz, 1990, O'Reilly & Associates, p. 361-364), 16 can't occur with MetaTalk because the syntax is so much simpler. Particularly noteworthy is goof #3 "Using == instead of eq and != instead of ne" for comparisons, which is the aforementioned performance-enhancing Perl feature. Even so, unlike the difference between the performance of MetaTalk vs ksh or Tcl, MetaTalk performance is not qualitatively different from that of Perl5.

Some of benchmarks for Tcl have improved with the release of the new byte-code compiler for Tcl 8, but preliminary results indicate that the speedup is far short of what is needed to bring Tcl up to parity with MetaTalk and Perl. The primary reason for this lack of improvement is that the design of the Tcl language is such that it is often impossible to preprocess arguments to function calls as is required to precompile them. Consider the following example:

set x 7; set y 9
puts stdout "$x + $y = [expr $x + $y]"
=> 7 + 9 = 16

On to Application deployment, documentation, and support

Back to Sample Application

Up to Table of Contents