until nil {

Recursive Fibonacci Benchmark Sequel

07 Oct 2010

As a sequel to the previous benchmark, but now with an improved comparisons scheme taking total execution time into account. The bench rides again, adding some zizany to the established order by welcoming in some unusual suspects as well as some references languages I hadn’t included before.

With Node.js awesomeness factor in perpetual escalation, it seemed imperative to now include Javascript on v8, as well as its meta-sibling CoffeeScript. Then to add perspective to the interpreted camp, two old timers, Perl and Lua, have now also been challenged. Lua, having the non-official title of fastest scripting language, with bonus tail recursion skills, surely has potential in this benchmark. Falcon, a new multi-paradigm interpreted language, unknown to most programmers, seemed like an interesting guest to invite and so is now also part of the comparison.

One more Objective-C based language, F-Script, will allow some runtime implementation comparison with the other Objective-C languages, Nu and MacRuby. Finally, some more rubyist attention was put in by also including jRuby, as well as upgrading MacRuby to its newest 0.7 iteration and adding an Ahead-Of-Time compiled variant of MacRuby.

Todays contenders are:

As stated in the previous instalment: “All the benchmarks were implemented in a recursive fashion, and so highlight the languages runtime ability to deal with deep recursion. And as a side-show, when fibonacci is called on smaller numbers, the benchmark provide us with a comparison on runtime initialization delay.”, so while you evaluate the results, remember this benchmark doesn’t aim at absolute language power comparison.

Benching:

Case value: 5
                     ruby: | 0 failed |     0.007s overall | score:     1.000 |
                     objc: | 0 failed |     0.009s overall | score:     1.356 |
                     perl: | 0 failed |     0.009s overall | score:     1.390 |
                      lua: | 0 failed |     0.022s overall | score:     3.383 |
                nu w/objc: | 0 failed |     0.064s overall | score:     9.800 |
            macruby w/aot: | 0 failed |     0.082s overall | score:    12.674 |
          f-script w/objc: | 0 failed |     0.116s overall | score:    17.833 |
                       nu: | 0 failed |     0.122s overall | score:    18.810 |
                  macruby: | 0 failed |     0.134s overall | score:    20.541 |
                 f-script: | 0 failed |     0.139s overall | score:    21.433 |
            sbcl compiled: | 0 failed |     0.144s overall | score:    22.152 |
                   falcon: | 0 failed |     0.147s overall | score:    22.587 |
                 js on v8: | 0 failed |     0.147s overall | score:    22.605 |
                   coffee: | 0 failed |     0.275s overall | score:    42.268 |
                   python: | 0 failed |     0.276s overall | score:    42.481 |
           macruby w/objc: | 0 failed |     0.533s overall | score:    81.943 |
                    jruby: | 0 failed |     1.195s overall | score:   183.746 |
                     sbcl: | 0 failed |     2.735s overall | score:   420.637 |

Case value: 10
                      lua: | 0 failed |     0.004s overall | score:     1.000 |
                     ruby: | 0 failed |     0.006s overall | score:     1.370 |
                     objc: | 0 failed |     0.009s overall | score:     2.082 |
                     perl: | 0 failed |     0.009s overall | score:     2.168 |
                   falcon: | 0 failed |     0.013s overall | score:     3.175 |
                   python: | 0 failed |     0.031s overall | score:     7.650 |
                 js on v8: | 0 failed |     0.053s overall | score:    12.985 |
                       nu: | 0 failed |     0.063s overall | score:    15.402 |
                nu w/objc: | 0 failed |     0.064s overall | score:    15.538 |
            macruby w/aot: | 0 failed |     0.082s overall | score:    19.835 |
            sbcl compiled: | 0 failed |     0.090s overall | score:    21.916 |
          f-script w/objc: | 0 failed |     0.097s overall | score:    23.574 |
                 f-script: | 0 failed |     0.115s overall | score:    28.060 |
                     sbcl: | 0 failed |     0.117s overall | score:    28.502 |
                  macruby: | 0 failed |     0.125s overall | score:    30.357 |
                   coffee: | 0 failed |     0.160s overall | score:    38.912 |
           macruby w/objc: | 0 failed |     0.219s overall | score:    53.206 |
                    jruby: | 0 failed |     0.703s overall | score:   170.890 |

Case value: 15
                      lua: | 0 failed |     0.004s overall | score:     1.000 |
                     ruby: | 0 failed |     0.007s overall | score:     1.702 |
                     objc: | 0 failed |     0.009s overall | score:     2.087 |
                     perl: | 0 failed |     0.011s overall | score:     2.656 |
                   falcon: | 0 failed |     0.014s overall | score:     3.310 |
                   python: | 0 failed |     0.032s overall | score:     7.737 |
                 js on v8: | 0 failed |     0.054s overall | score:    12.874 |
                nu w/objc: | 0 failed |     0.065s overall | score:    15.735 |
            macruby w/aot: | 0 failed |     0.082s overall | score:    19.820 |
            sbcl compiled: | 0 failed |     0.090s overall | score:    21.743 |
                       nu: | 0 failed |     0.090s overall | score:    21.748 |
          f-script w/objc: | 0 failed |     0.095s overall | score:    22.805 |
                     sbcl: | 0 failed |     0.117s overall | score:    28.131 |
                  macruby: | 0 failed |     0.125s overall | score:    29.987 |
                   coffee: | 0 failed |     0.159s overall | score:    38.296 |
           macruby w/objc: | 0 failed |     0.217s overall | score:    52.272 |
                 f-script: | 0 failed |     0.299s overall | score:    71.919 |
                    jruby: | 0 failed |     0.701s overall | score:   168.601 |

Case value: 20
                      lua: | 0 failed |     0.008s overall | score:     1.000 |
                     objc: | 0 failed |     0.009s overall | score:     1.165 |
                     ruby: | 0 failed |     0.023s overall | score:     3.003 |
                   falcon: | 0 failed |     0.027s overall | score:     3.545 |
                     perl: | 0 failed |     0.033s overall | score:     4.274 |
                   python: | 0 failed |     0.047s overall | score:     6.164 |
                 js on v8: | 0 failed |     0.054s overall | score:     7.036 |
                nu w/objc: | 0 failed |     0.064s overall | score:     8.450 |
            macruby w/aot: | 0 failed |     0.082s overall | score:    10.828 |
            sbcl compiled: | 0 failed |     0.091s overall | score:    11.984 |
          f-script w/objc: | 0 failed |     0.093s overall | score:    12.244 |
                     sbcl: | 0 failed |     0.116s overall | score:    15.272 |
                  macruby: | 0 failed |     0.128s overall | score:    16.763 |
                   coffee: | 0 failed |     0.160s overall | score:    21.015 |
           macruby w/objc: | 0 failed |     0.219s overall | score:    28.771 |
                       nu: | 0 failed |     0.404s overall | score:    53.040 |
                    jruby: | 0 failed |     0.853s overall | score:   112.039 |
                 f-script: | 0 failed |     2.275s overall | score:   298.831 |

Case value: 25
                     objc: | 0 failed |     0.011s overall | score:     1.000 |
                      lua: | 0 failed |     0.046s overall | score:     4.210 |
                 js on v8: | 0 failed |     0.058s overall | score:     5.358 |
                nu w/objc: | 0 failed |     0.066s overall | score:     6.058 |
            macruby w/aot: | 0 failed |     0.089s overall | score:     8.127 |
            sbcl compiled: | 0 failed |     0.096s overall | score:     8.825 |
          f-script w/objc: | 0 failed |     0.100s overall | score:     9.183 |
                     sbcl: | 0 failed |     0.124s overall | score:    11.411 |
                   falcon: | 0 failed |     0.135s overall | score:    12.333 |
                  macruby: | 0 failed |     0.136s overall | score:    12.461 |
                   python: | 0 failed |     0.136s overall | score:    12.487 |
                   coffee: | 0 failed |     0.166s overall | score:    15.182 |
                     ruby: | 0 failed |     0.199s overall | score:    18.269 |
           macruby w/objc: | 0 failed |     0.219s overall | score:    20.065 |
                     perl: | 0 failed |     0.275s overall | score:    25.164 |
                    jruby: | 0 failed |     0.854s overall | score:    78.260 |
                       nu: | 0 failed |     3.824s overall | score:   350.476 |
                 f-script: | 0 failed |    24.284s overall | score:  2225.848 |

Case value: 30
                     objc: | 0 failed |     0.032s overall | score:     1.000 |
                nu w/objc: | 0 failed |     0.090s overall | score:     2.866 |
                 js on v8: | 0 failed |     0.092s overall | score:     2.927 |
          f-script w/objc: | 0 failed |     0.134s overall | score:     4.260 |
            macruby w/aot: | 0 failed |     0.146s overall | score:     4.624 |
            sbcl compiled: | 0 failed |     0.156s overall | score:     4.945 |
                     sbcl: | 0 failed |     0.184s overall | score:     5.845 |
                   coffee: | 0 failed |     0.225s overall | score:     7.153 |
                  macruby: | 0 failed |     0.234s overall | score:     7.441 |
           macruby w/objc: | 0 failed |     0.241s overall | score:     7.654 |
                      lua: | 0 failed |     0.465s overall | score:    14.769 |
                   falcon: | 0 failed |     1.071s overall | score:    33.993 |
                    jruby: | 0 failed |     1.103s overall | score:    35.010 |
                   python: | 0 failed |     1.107s overall | score:    35.132 |
                     ruby: | 0 failed |     2.125s overall | score:    67.452 |
                     perl: | 0 failed |     2.952s overall | score:    93.696 |
                       nu: | 0 failed |    42.562s overall | score:  1351.047 |
                 f-script: | 0 failed |   274.925s overall | score:  8726.948 |

Case value: 35
                     objc: | 0 failed |     0.257s overall | score:     1.000 |
                nu w/objc: | 0 failed |     0.318s overall | score:     1.235 |
          f-script w/objc: | 0 failed |     0.350s overall | score:     1.361 |
                 js on v8: | 0 failed |     0.454s overall | score:     1.763 |
           macruby w/objc: | 0 failed |     0.463s overall | score:     1.800 |
            macruby w/aot: | 0 failed |     0.784s overall | score:     3.046 |
            sbcl compiled: | 0 failed |     0.818s overall | score:     3.178 |
                     sbcl: | 0 failed |     0.842s overall | score:     3.274 |
                   coffee: | 0 failed |     0.853s overall | score:     3.314 |
                  macruby: | 0 failed |     1.380s overall | score:     5.364 |
                    jruby: | 0 failed |     3.956s overall | score:    15.372 |
                      lua: | 0 failed |     5.125s overall | score:    19.919 |
                   falcon: | 0 failed |    11.598s overall | score:    45.074 |
                   python: | 0 failed |    11.875s overall | score:    46.150 |
                     ruby: | 0 failed |    23.344s overall | score:    90.725 |
                     perl: | 0 failed |    32.544s overall | score:   126.477 |
                       nu: | 0 failed |   473.608s overall | score:  1840.603 |
                 f-script: | 0 failed |  3330.532s overall | score: 12943.599 |

Case value: 40
                     objc: | 0 failed |     2.785s overall | score:     1.000 |
           macruby w/objc: | 0 failed |     2.909s overall | score:     1.045 |
          f-script w/objc: | 0 failed |     3.217s overall | score:     1.155 |
                nu w/objc: | 0 failed |     4.319s overall | score:     1.551 |
                 js on v8: | 0 failed |     4.660s overall | score:     1.673 |
            macruby w/aot: | 0 failed |     7.818s overall | score:     2.807 |
                   coffee: | 0 failed |     8.102s overall | score:     2.909 |
            sbcl compiled: | 0 failed |     8.202s overall | score:     2.945 |
                     sbcl: | 0 failed |    10.506s overall | score:     3.772 |
                  macruby: | 0 failed |    13.866s overall | score:     4.979 |
                    jruby: | 0 failed |    35.971s overall | score:    12.916 |
                      lua: | 0 failed |    56.933s overall | score:    20.442 |
                   falcon: | 0 failed |   125.456s overall | score:    45.046 |
                   python: | 0 failed |   131.910s overall | score:    47.364 |
                     ruby: | 0 failed |   259.228s overall | score:    93.078 |
                     perl: | 0 failed |   360.102s overall | score:   129.298 |
                       nu: | 0 failed |  6375.526s overall | score:  2289.197 |
                 f-script: | 0 failed | 38450.093s overall | score: 13805.894 |
                 
Ranking by Total Times:
                     objc: |     3.120 seconds overall |
          f-script w/objc: |     4.202 seconds overall |
           macruby w/objc: |     5.021 seconds overall |
                nu w/objc: |     5.051 seconds overall |
                 js on v8: |     5.572 seconds overall |
            macruby w/aot: |     9.165 seconds overall |
            sbcl compiled: |     9.687 seconds overall |
                   coffee: |    10.100 seconds overall |
                     sbcl: |    14.743 seconds overall |
                  macruby: |    16.128 seconds overall |
                    jruby: |    45.336 seconds overall |
                      lua: |    62.607 seconds overall |
                   falcon: |   138.460 seconds overall |
                   python: |   145.415 seconds overall |
                     ruby: |   284.939 seconds overall |
                     perl: |   395.934 seconds overall |
                       nu: |  6896.199 seconds overall |
                 f-script: | 42082.663 seconds overall |



Ranking by Total Scores:
                     objc: | score:    10.691 |
                nu w/objc: | score:    61.231 |
                      lua: | score:    65.724 |
                 js on v8: | score:    67.221 |
            macruby w/aot: | score:    81.761 |
          f-script w/objc: | score:    92.415 |
            sbcl compiled: | score:    97.688 |
                  macruby: | score:   127.895 |
                   coffee: | score:   169.050 |
                   falcon: | score:   169.064 |
                   python: | score:   205.164 |
           macruby w/objc: | score:   246.756 |
                     ruby: | score:   276.599 |
                     perl: | score:   385.124 |
                     sbcl: | score:   516.843 |
                    jruby: | score:   776.835 |
                       nu: | score:  5940.324 |
                 f-script: | score: 38122.532 |

Combined Ranking:

           objc: |  2 |     3.120 seconds overall | score:    10.691 |
f-script w/objc: |  8 |     4.202 seconds overall | score:    92.415 |
       js on v8: |  9 |     5.572 seconds overall | score:    67.221 |
      nu w/objc: | 10 |     5.051 seconds overall | score:    61.231 |
  macruby w/aot: | 11 |     9.165 seconds overall | score:    81.761 |
  sbcl compiled: | 14 |     9.687 seconds overall | score:    97.688 |
            lua: | 15 |    62.607 seconds overall | score:    65.724 |
 macruby w/objc: | 15 |     5.021 seconds overall | score:   246.756 |
         coffee: | 17 |    10.100 seconds overall | score:   169.050 |
        macruby: | 18 |    16.128 seconds overall | score:   127.895 | 
         falcon: | 23 |   138.460 seconds overall | score:   169.064 |
           sbcl: | 24 |    14.743 seconds overall | score:   516.843 |
         python: | 25 |   145.415 seconds overall | score:   205.164 |
          jruby: | 27 |    45.336 seconds overall | score:   776.835 |
           ruby: | 28 |   284.939 seconds overall | score:   276.599 |
           perl: | 30 |   395.934 seconds overall | score:   385.124 |
             nu: | 34 |  6896.199 seconds overall | score:  5940.324 |
       f-script: | 36 | 42082.663 seconds overall | score: 38122.532 |

Interpretation:

What strikes first, is how Lua is as fast as can be when loading, while staying just above the waterline under which Perl, Ruby, Python and Falcon are crushed when recursion gets too deep, adding a remarkable note to its performance reputation. Objective-C really needed to stay first the list, while F-Script came to be worst than pure Nu at the very low bottom, surpassing it when combined with Objective-C functions at the top of the rankings.

From the new players list, we can emit a couple of novel statements:

  • v8 is awesome, endures deep recursion almost as fast as the Objective-C based routines, but does not load up as fast as some of its mates.
  • CoffeeScript features proportional awesomeness to pure v8, 2 times slower but more than 2 times funnier, plus it gets its speed back when pre-compiled to JavaScript.
  • jRuby really has a tough wake up but stays upright under the weight of deep recursion.
  • MacRuby is ruby impressive, no need for any extra Objective-C help, a big kid with its own tricks, like AOT compilation providing super easy, efficient optimization potential.
  • Falcon really stands high, scoring the highest in the fully loaded interpreted languages group composed of Perl, Python and Ruby. I voluntarily exclude Lua from this group because while its is an interpreted language, it is not fully loaded, or Batteries included as one might say and this fact surely accounts for its higher score. Falcon on the other end, almost overflows with base functionalities.
  • F-Script is really not meant for standalone scripting. One thing the bench does not show, is its light memory footprint: unlike Nu all you can eat appetite, F-Script will constrain itself to a very acceptable maximum. Making it an ideal choice as a scripting extension inside some Cocoa application.

In the end, no benchmark can show a true picture of what would be a real world use of these language technologies. This one truly has its limitations, but within them we were still able to get a rough feeling of how these languages matches up and which compromises they would imply.

blog comments powered by Disqus

}

Older Posts... Blog powered by Jekyll.
Built using Liquid, RedCloth, Pygments and Blueprint.

Copyright © 2008-2010 Louis-Philippe Perron