until nil {

AppEngine's Python Templates Benchmark Conclusion

27 Aug 2010

After the incremental improvements made to the benchmarks in the 3 last posts (1, 2, 3), what more can we do? First, not all possible case combination of Frameworks versus Engines have been tested. Django Templates, SimpleTemplate and Templator were still tested only in their default configuration. Implementing those new combinations will give us symmetry of templates and frameworks, useful later for cross-comparisons benchmarks. We now have:

5 web application frameworks:

Crossed with 7 templating engine:

For a total of 35 possible combinations.

This renewed benchmark will compare these 35 combinations, with a new focus on classifying the frameworks and the engines separately. The same test algorithm will be used in this benchmark, implementing a short controller action coupled with a long view compilation. This type of implementation allow us to pinpoint:

  • in web application frameworks:
    • a good figure confirms the controller to be unobtrusive
    • a bad figure reveals a controller’s heavy processing overhead
  • in templating engines:
    • a good result shows the templates have tolerance for pressure
    • a bad result let us believe the engine can’t withstand heavy load

The last comparison step will involve a simple classification of default configurations, as many developers will more likely choose an integrated batteries included solution, rather than a DIY. This comparison will highlight the best all in one in terms of performance. These default configurations are:

  • Bottle with SimpleTemplate
  • Flask with Jinja
  • tipfy with Jinja
  • webapp with Django
  • web.py with Templator

Lets again proceed with the benchmarks. This time, I will skip pasting all new code implementations and will instead redirect you to the benchmark repository, where you can find all about the implementations details. The execution was also dropped from 1000 loop counts to 250, as it seems to be enough to represent the average performance of the systems.

Low Overhead:

       webapp w/Templator: | 250x, 0 failed |   7.316s overall, 0.029s each | score:  1.000 |
  webapp w/SimpleTemplate: | 250x, 0 failed |   7.386s overall, 0.030s each | score:  1.010 |
            tipfy w/Jinja: | 250x, 0 failed |   7.386s overall, 0.030s each | score:  1.010 |
          bottle w/Tenjin: | 250x, 0 failed |   7.388s overall, 0.030s each | score:  1.010 |
          webapp w/Tenjin: | 250x, 0 failed |   7.391s overall, 0.030s each | score:  1.010 |
        tipfy w/Templator: | 250x, 0 failed |   7.392s overall, 0.030s each | score:  1.010 |
          web.py w/Tenjin: | 250x, 0 failed |   7.401s overall, 0.030s each | score:  1.012 |
       web.py w/Templator: | 250x, 0 failed |   7.413s overall, 0.030s each | score:  1.013 |
            flask w/Jinja: | 250x, 0 failed |   7.430s overall, 0.030s each | score:  1.016 |
  bottle w/SimpleTemplate: | 250x, 0 failed |   7.435s overall, 0.030s each | score:  1.016 |
  web.py w/SimpleTemplate: | 250x, 0 failed |   7.450s overall, 0.030s each | score:  1.018 |
        flask w/Templator: | 250x, 0 failed |   7.460s overall, 0.030s each | score:  1.020 |
           tipfy w/Tenjin: | 250x, 0 failed |   7.467s overall, 0.030s each | score:  1.021 |
   tipfy w/SimpleTemplate: | 250x, 0 failed |   7.474s overall, 0.030s each | score:  1.022 |
   flask w/SimpleTemplate: | 250x, 0 failed |   7.489s overall, 0.030s each | score:  1.024 |
       bottle w/Templator: | 250x, 0 failed |   7.500s overall, 0.030s each | score:  1.025 |
          webapp w/Django: | 250x, 0 failed |   7.533s overall, 0.030s each | score:  1.030 |
          bottle w/Django: | 250x, 0 failed |   7.570s overall, 0.030s each | score:  1.035 |
           flask w/Django: | 250x, 0 failed |   7.588s overall, 0.030s each | score:  1.037 |
          web.py w/Django: | 250x, 0 failed |   7.606s overall, 0.030s each | score:  1.040 |
            flask w/Breve: | 250x, 0 failed |   7.616s overall, 0.030s each | score:  1.041 |
           tipfy w/Django: | 250x, 0 failed |   7.620s overall, 0.030s each | score:  1.041 |
           bottle w/Breve: | 250x, 0 failed |   7.708s overall, 0.031s each | score:  1.054 |
           flask w/Tenjin: | 250x, 0 failed |   7.814s overall, 0.031s each | score:  1.068 |
           webapp w/Breve: | 250x, 0 failed |   7.841s overall, 0.031s each | score:  1.072 |
           webapp w/Jinja: | 250x, 0 failed |   8.137s overall, 0.033s each | score:  1.112 |
            webapp w/Mako: | 250x, 0 failed |   8.156s overall, 0.033s each | score:  1.115 |
            tipfy w/Breve: | 250x, 0 failed |   8.177s overall, 0.033s each | score:  1.118 |
           bottle w/Jinja: | 250x, 0 failed |   8.187s overall, 0.033s each | score:  1.119 |
             tipfy w/Mako: | 250x, 0 failed |   8.227s overall, 0.033s each | score:  1.124 |
            bottle w/Mako: | 250x, 0 failed |   8.256s overall, 0.033s each | score:  1.128 |
            web.py w/Mako: | 250x, 0 failed |   8.290s overall, 0.033s each | score:  1.133 |
           web.py w/Jinja: | 250x, 0 failed |   8.375s overall, 0.033s each | score:  1.145 |
           web.py w/Breve: | 250x, 0 failed |   8.477s overall, 0.034s each | score:  1.159 |
             flask w/Mako: | 250x, 0 failed |   8.679s overall, 0.035s each | score:  1.186 |

Medium Overhead

       webapp w/Templator: | 250x, 0 failed |   7.508s overall, 0.030s each | score:  1.000 |
          webapp w/Tenjin: | 250x, 0 failed |   7.546s overall, 0.030s each | score:  1.005 |
        tipfy w/Templator: | 250x, 0 failed |   7.603s overall, 0.030s each | score:  1.013 |
          bottle w/Tenjin: | 250x, 0 failed |   7.642s overall, 0.031s each | score:  1.018 |
       bottle w/Templator: | 250x, 0 failed |   7.649s overall, 0.031s each | score:  1.019 |
           flask w/Tenjin: | 250x, 0 failed |   7.673s overall, 0.031s each | score:  1.022 |
        flask w/Templator: | 250x, 0 failed |   7.676s overall, 0.031s each | score:  1.022 |
            tipfy w/Jinja: | 250x, 0 failed |   7.727s overall, 0.031s each | score:  1.029 |
          web.py w/Tenjin: | 250x, 0 failed |   7.728s overall, 0.031s each | score:  1.029 |
           tipfy w/Tenjin: | 250x, 0 failed |   7.736s overall, 0.031s each | score:  1.030 |
       web.py w/Templator: | 250x, 0 failed |   7.861s overall, 0.031s each | score:  1.047 |
  webapp w/SimpleTemplate: | 250x, 0 failed |   7.905s overall, 0.032s each | score:  1.053 |
            flask w/Jinja: | 250x, 0 failed |   7.928s overall, 0.032s each | score:  1.056 |
   flask w/SimpleTemplate: | 250x, 0 failed |   7.990s overall, 0.032s each | score:  1.064 |
  web.py w/SimpleTemplate: | 250x, 0 failed |   7.997s overall, 0.032s each | score:  1.065 |
   tipfy w/SimpleTemplate: | 250x, 0 failed |   8.023s overall, 0.032s each | score:  1.069 |
           web.py w/Jinja: | 250x, 0 failed |   8.253s overall, 0.033s each | score:  1.099 |
  bottle w/SimpleTemplate: | 250x, 0 failed |   8.273s overall, 0.033s each | score:  1.102 |
             flask w/Mako: | 250x, 0 failed |   8.296s overall, 0.033s each | score:  1.105 |
            webapp w/Mako: | 250x, 0 failed |   8.299s overall, 0.033s each | score:  1.105 |
          webapp w/Django: | 250x, 0 failed |   8.310s overall, 0.033s each | score:  1.107 |
           bottle w/Jinja: | 250x, 0 failed |   8.314s overall, 0.033s each | score:  1.107 |
             tipfy w/Mako: | 250x, 0 failed |   8.322s overall, 0.033s each | score:  1.108 |
           webapp w/Jinja: | 250x, 0 failed |   8.322s overall, 0.033s each | score:  1.108 |
            web.py w/Mako: | 250x, 0 failed |   8.341s overall, 0.033s each | score:  1.111 |
            bottle w/Mako: | 250x, 0 failed |   8.358s overall, 0.033s each | score:  1.113 |
          bottle w/Django: | 250x, 0 failed |   8.450s overall, 0.034s each | score:  1.125 |
           tipfy w/Django: | 250x, 0 failed |   8.471s overall, 0.034s each | score:  1.128 |
          web.py w/Django: | 250x, 0 failed |   8.572s overall, 0.034s each | score:  1.142 |
           flask w/Django: | 250x, 0 failed |   8.603s overall, 0.034s each | score:  1.146 |
            tipfy w/Breve: | 250x, 0 failed |   9.475s overall, 0.038s each | score:  1.262 |
           webapp w/Breve: | 250x, 0 failed |   9.687s overall, 0.039s each | score:  1.290 |
           bottle w/Breve: | 250x, 0 failed |   9.714s overall, 0.039s each | score:  1.294 |
            flask w/Breve: | 250x, 0 failed |   9.729s overall, 0.039s each | score:  1.296 |
           web.py w/Breve: | 250x, 0 failed |  10.434s overall, 0.042s each | score:  1.390 |

High Overhead

            tipfy w/Jinja: | 250x, 0 failed |   8.306s overall, 0.033s each | score:  1.000 |
             flask w/Mako: | 250x, 0 failed |   9.181s overall, 0.037s each | score:  1.105 |
            bottle w/Mako: | 250x, 0 failed |   9.213s overall, 0.037s each | score:  1.109 |
            webapp w/Mako: | 250x, 0 failed |   9.216s overall, 0.037s each | score:  1.109 |
             tipfy w/Mako: | 250x, 0 failed |   9.277s overall, 0.037s each | score:  1.117 |
           web.py w/Jinja: | 250x, 0 failed |   9.393s overall, 0.038s each | score:  1.131 |
            web.py w/Mako: | 250x, 0 failed |   9.478s overall, 0.038s each | score:  1.141 |
           bottle w/Jinja: | 250x, 0 failed |   9.505s overall, 0.038s each | score:  1.144 |
       webapp w/Templator: | 250x, 0 failed |   9.513s overall, 0.038s each | score:  1.145 |
       bottle w/Templator: | 250x, 0 failed |   9.562s overall, 0.038s each | score:  1.151 |
        flask w/Templator: | 250x, 0 failed |   9.691s overall, 0.039s each | score:  1.167 |
          webapp w/Tenjin: | 250x, 0 failed |   9.871s overall, 0.039s each | score:  1.188 |
        tipfy w/Templator: | 250x, 0 failed |   9.886s overall, 0.040s each | score:  1.190 |
          bottle w/Tenjin: | 250x, 0 failed |   9.930s overall, 0.040s each | score:  1.196 |
       web.py w/Templator: | 250x, 0 failed |   9.997s overall, 0.040s each | score:  1.204 |
          web.py w/Tenjin: | 250x, 0 failed |  10.015s overall, 0.040s each | score:  1.206 |
           tipfy w/Tenjin: | 250x, 0 failed |  10.018s overall, 0.040s each | score:  1.206 |
           flask w/Tenjin: | 250x, 0 failed |  10.229s overall, 0.041s each | score:  1.231 |
           webapp w/Jinja: | 250x, 0 failed |  10.254s overall, 0.041s each | score:  1.234 |
            flask w/Jinja: | 250x, 0 failed |  12.856s overall, 0.051s each | score:  1.548 |
  webapp w/SimpleTemplate: | 250x, 0 failed |  13.352s overall, 0.053s each | score:  1.607 |
  bottle w/SimpleTemplate: | 250x, 0 failed |  13.426s overall, 0.054s each | score:  1.616 |
   flask w/SimpleTemplate: | 250x, 0 failed |  13.434s overall, 0.054s each | score:  1.617 |
   tipfy w/SimpleTemplate: | 250x, 0 failed |  13.454s overall, 0.054s each | score:  1.620 |
  web.py w/SimpleTemplate: | 250x, 0 failed |  13.959s overall, 0.056s each | score:  1.680 |
          bottle w/Django: | 250x, 0 failed |  16.898s overall, 0.068s each | score:  2.034 |
          webapp w/Django: | 250x, 0 failed |  16.973s overall, 0.068s each | score:  2.043 |
           tipfy w/Django: | 250x, 0 failed |  17.392s overall, 0.070s each | score:  2.094 |
          web.py w/Django: | 250x, 0 failed |  18.585s overall, 0.074s each | score:  2.237 |
           flask w/Django: | 250x, 0 failed |  18.731s overall, 0.075s each | score:  2.255 |
           webapp w/Breve: | 250x, 0 failed |  30.264s overall, 0.121s each | score:  3.644 |
           web.py w/Breve: | 250x, 0 failed |  30.776s overall, 0.123s each | score:  3.705 |
            tipfy w/Breve: | 250x, 0 failed |  31.302s overall, 0.125s each | score:  3.768 |
            flask w/Breve: | 250x, 0 failed |  31.461s overall, 0.126s each | score:  3.788 |
           bottle w/Breve: | 250x, 0 failed |  31.860s overall, 0.127s each | score:  3.836 |

Very High Overhead

            tipfy w/Jinja: | 250x, 0 failed |  16.239s overall, 0.065s each | score:  1.000 |
           webapp w/Jinja: | 250x, 0 failed |  17.114s overall, 0.068s each | score:  1.054 |
           web.py w/Jinja: | 250x, 0 failed |  17.162s overall, 0.069s each | score:  1.057 |
           bottle w/Jinja: | 250x, 0 failed |  17.356s overall, 0.069s each | score:  1.069 |
            bottle w/Mako: | 250x, 0 failed |  18.707s overall, 0.075s each | score:  1.152 |
            web.py w/Mako: | 250x, 0 failed |  18.845s overall, 0.075s each | score:  1.160 |
             tipfy w/Mako: | 250x, 0 failed |  19.193s overall, 0.077s each | score:  1.182 |
            webapp w/Mako: | 250x, 0 failed |  19.392s overall, 0.078s each | score:  1.194 |
             flask w/Mako: | 250x, 0 failed |  19.393s overall, 0.078s each | score:  1.194 |
       bottle w/Templator: | 250x, 0 failed |  29.569s overall, 0.118s each | score:  1.821 |
       webapp w/Templator: | 250x, 0 failed |  29.582s overall, 0.118s each | score:  1.822 |
        tipfy w/Templator: | 250x, 0 failed |  29.795s overall, 0.119s each | score:  1.835 |
        flask w/Templator: | 250x, 0 failed |  30.181s overall, 0.121s each | score:  1.859 |
       web.py w/Templator: | 250x, 0 failed |  30.306s overall, 0.121s each | score:  1.866 |
          bottle w/Tenjin: | 250x, 0 failed |  32.968s overall, 0.132s each | score:  2.030 |
          webapp w/Tenjin: | 250x, 0 failed |  33.843s overall, 0.135s each | score:  2.084 |
           tipfy w/Tenjin: | 250x, 0 failed |  33.910s overall, 0.136s each | score:  2.088 |
          web.py w/Tenjin: | 250x, 0 failed |  34.397s overall, 0.138s each | score:  2.118 |
           flask w/Tenjin: | 250x, 0 failed |  34.558s overall, 0.138s each | score:  2.128 |
            flask w/Jinja: | 250x, 0 failed |  62.129s overall, 0.249s each | score:  3.826 |
  webapp w/SimpleTemplate: | 250x, 0 failed |  67.637s overall, 0.271s each | score:  4.165 |
  web.py w/SimpleTemplate: | 250x, 0 failed |  67.807s overall, 0.271s each | score:  4.176 |
   flask w/SimpleTemplate: | 250x, 0 failed |  67.834s overall, 0.271s each | score:  4.177 |
   tipfy w/SimpleTemplate: | 250x, 0 failed |  68.286s overall, 0.273s each | score:  4.205 |
  bottle w/SimpleTemplate: | 250x, 0 failed |  68.810s overall, 0.275s each | score:  4.237 |
          bottle w/Django: | 250x, 0 failed | 102.666s overall, 0.411s each | score:  6.322 |
          webapp w/Django: | 250x, 0 failed | 103.002s overall, 0.412s each | score:  6.343 |
           tipfy w/Django: | 250x, 0 failed | 103.221s overall, 0.413s each | score:  6.356 |
          web.py w/Django: | 250x, 0 failed | 119.719s overall, 0.479s each | score:  7.372 |
           flask w/Django: | 250x, 0 failed | 119.746s overall, 0.479s each | score:  7.374 |
            tipfy w/Breve: | 250x, 0 failed | 253.913s overall, 1.016s each | score: 15.636 |
           bottle w/Breve: | 250x, 0 failed | 254.588s overall, 1.018s each | score: 15.678 |
           web.py w/Breve: | 250x, 0 failed | 254.943s overall, 1.020s each | score: 15.700 |
           webapp w/Breve: | 250x, 0 failed | 255.114s overall, 1.020s each | score: 15.710 |
            flask w/Breve: | 250x, 0 failed | 256.383s overall, 1.026s each | score: 15.788 |

Overall classification based on cumulative scores:

            tipfy w/Jinja: |  39.658 seconds overall | score:   4.039 |
           web.py w/Jinja: |  43.183 seconds overall | score:   4.432 |
           bottle w/Jinja: |  43.362 seconds overall | score:   4.439 |
            bottle w/Mako: |  44.533 seconds overall | score:   4.503 |
           webapp w/Jinja: |  43.827 seconds overall | score:   4.509 |
            webapp w/Mako: |  45.063 seconds overall | score:   4.524 |
             tipfy w/Mako: |  45.018 seconds overall | score:   4.532 |
            web.py w/Mako: |  44.954 seconds overall | score:   4.546 |
             flask w/Mako: |  45.549 seconds overall | score:   4.591 |
       webapp w/Templator: |  53.918 seconds overall | score:   4.967 |
       bottle w/Templator: |  54.280 seconds overall | score:   5.016 |
        tipfy w/Templator: |  54.676 seconds overall | score:   5.048 |
        flask w/Templator: |  55.007 seconds overall | score:   5.067 |
       web.py w/Templator: |  55.577 seconds overall | score:   5.130 |
          bottle w/Tenjin: |  57.928 seconds overall | score:   5.253 |
          webapp w/Tenjin: |  58.651 seconds overall | score:   5.288 |
           tipfy w/Tenjin: |  59.130 seconds overall | score:   5.345 |
          web.py w/Tenjin: |  59.540 seconds overall | score:   5.365 |
           flask w/Tenjin: |  60.274 seconds overall | score:   5.450 |
            flask w/Jinja: |  90.343 seconds overall | score:   7.445 |
  webapp w/SimpleTemplate: |  96.280 seconds overall | score:   7.835 |
   flask w/SimpleTemplate: |  96.746 seconds overall | score:   7.882 |
   tipfy w/SimpleTemplate: |  97.237 seconds overall | score:   7.915 |
  web.py w/SimpleTemplate: |  97.213 seconds overall | score:   7.940 |
  bottle w/SimpleTemplate: |  97.945 seconds overall | score:   7.972 |
          bottle w/Django: | 135.583 seconds overall | score:  10.517 |
          webapp w/Django: | 135.819 seconds overall | score:  10.523 |
           tipfy w/Django: | 136.704 seconds overall | score:  10.620 |
          web.py w/Django: | 154.482 seconds overall | score:  11.791 |
           flask w/Django: | 154.667 seconds overall | score:  11.812 |
           webapp w/Breve: | 302.906 seconds overall | score:  21.716 |
            tipfy w/Breve: | 302.867 seconds overall | score:  21.784 |
           bottle w/Breve: | 303.870 seconds overall | score:  21.861 |
            flask w/Breve: | 305.190 seconds overall | score:  21.913 |
           web.py w/Breve: | 304.630 seconds overall | score:  21.953 |

Templating Engines classification:

           Mako: |  225.118s overall | score:   1.000 |
          Jinja: |  260.373s overall | score:   1.157 |
      Templator: |  273.459s overall | score:   1.215 |
         Tenjin: |  295.523s overall | score:   1.313 |
 SimpleTemplate: |  485.420s overall | score:   2.156 |
         Django: |  717.255s overall | score:   3.186 |
          Breve: | 1519.463s overall | score:   6.750 |

Web Application Frameworks classification:

          tipfy: | 735.291s overall | score:   1.000 |
         webapp: | 736.464s overall | score:   1.002 |
         Bottle: | 737.502s overall | score:   1.003 |
         wep.py: | 759.578s overall | score:   1.033 |
          Flask: | 807.776s overall | score:   1.099 |

Batteries Included Frameworks comparison:

            tipfy w/Jinja: |  39.658s overall | score:   1.000 |
       web.py w/Templator: |  55.577s overall | score:   1.401 |
            flask w/Jinja: |  90.343s overall | score:   2.278 |
  bottle w/SimpleTemplate: |  97.945s overall | score:   2.470 |
          webapp w/Django: | 135.819s overall | score:   3.425 |

What can we see in there? First, tipfy w/Jinja rocks! It is made for AppEngine, and has the tweeks to make a difference. If you look at it a bit more, you can see that tipfy’s performance in general wins big. Second, as a Templating engine in general, mako rocks! Its speed ranking validates its philosophical position ‘…your templates can handle it !’. Templator and Tenjin clearly stay better than others all along and surpass all, even Mako and Jinja, in simple scenarios. Stepping outside the views, webapp, pre-installed on the AppEngine, shows us it knows its ways around and step second, preceding Bottle which by many standards, might be considered the real connoisseur framework. In term of default configurations, the benchmarks shows tipfy and web.py to have some kick, but the rest would be better coupled with one of the other engines, as their built-in engines seems to be disadvantageous after all.

All in all, we can say every framework and engine positively behaved and proved to be apt at delivering the job at hand. In other words, these are all Class A projects deserving attention and respect. Your end verdict shouldn’t be so much influenced by these performance figures after all. You need to also like them! As inspiration is likely to be the primary tool bringing your projects to a successful conclusion.

blog comments powered by Disqus

}

Older Posts... Blog powered by Jekyll.
Built using Liquid, RedCloth, Pygments and Blueprint.

Copyright © 2008-2010 Louis-Philippe Perron