until nil {

Fast Python templates on AppEngine

23 Aug 2010

Faced with the task of building a fast xml api server with Google’s AppEngine I dived into the cloud service runtime options for web application construction. I generally have a bad feeling about interfacing with Java technologies, and benchmarks are there to prove that the engine’s Python API are at least on par with Java in term of speed and features, so I opted to follow the Pythonic way for the first time in a web application project.

Lets first make a small detour on my first impressions with Python as a server side scripting language. What stroked me first, is that Python mostly always stay Python, as opposed to Ruby’s capacity to bend into different DSL focussed on the application composition tasks. If I had done the job in Ruby, I would have used a light framework like Sinatra instead of Ruby On Rails. So it felt normal to look for some Sinatra equivalent in the Python world. Django, Turbogears, web2py and Pylons were all ditched out in the first round, I might have been wrong but they all seamed nice but overkill for my project. Bottle, web.py and webapp were all chosen for exactly the opposite reasons. Bottle being the the closest to Sinatra, it won a slot at the top of my list. I also hooked with Mako when looking for templating system options. I liked the way it seamed to put no limits into its templating power and its figures in page rendering benchmarks were impressives1.

Before diving deep into learning anyone of those frameworks, I decided to get a basic impression of them by building a small benchmark comparison. Three frameworks, each with their default templating module, plus these three same in combination with Mako, making for six comparison cases:

The benchmark was run on AppEngine local development server, because otherways the benchmark would have measured my internet bandwidth and possibly also run into internet bottleneck inconsistencies. I implemented the same simple api using the six systems, testing them alternatively, instead of successively like most benchmarks do. The server algorithms basically serve templates with calculations embedded. In only one instance of those I could not implement the calculation in the templating engine2. I know that most of the readers might think it is good not to be doing those things in the views anyway, I totally agree, but for the sake of evaluating the raw performance of the views, it’s worth it. Here comes the code for the different servers implementations:

webapp with Django Templates:

import os
from google.appengine.ext import webapp
from google.appengine.ext.webapp import util
from google.appengine.ext.webapp import template

class MainHandler(webapp.RequestHandler):
    def get(self,numA,numB):
        nums = []
        for i in range(int(numA)):
          rmult = i * int(numB) 
          nums.append(str(rmult))
        templateValues = {
          "nums" : nums,
          "numB" : numB
        }
        self.response.out.write(template.render(os.path.join(
          os.path.dirname(__file__),"gnativeTestTemplate.html"),templateValues))

def main():
    application = webapp.WSGIApplication([(r'/gnative/(.*)/(.*)', MainHandler)],debug=False)
    util.run_wsgi_app(application)

if __name__ == '__main__':
    main()
<html>
<body>
    <table>
        {% for i in nums %}
            <tr><td>{{ forloop.counter }}</td><td>x {{ numB }}</td><td>= {{ i }}</td></tr>
        {% endfor %}
    </table>
</body>
</html>

webapp with Mako:

import os
from google.appengine.ext import webapp
from google.appengine.ext.webapp import util
from mako.template import Template

class MainHandler(webapp.RequestHandler):
    def get(self,numA,numB):
        self.response.out.write(Template(filename=os.path.join(
          os.path.dirname(__file__),"gTestMakoTemplate.html")).render(numA=numA,numB=numB))

def main():
    application = webapp.WSGIApplication([(r'/gmako/(.*)/(.*)', MainHandler)],debug=False)
    util.run_wsgi_app(application)

if __name__ == '__main__':
    main()
<html>
<body>
    <table>
        % for i in range(int(numA)):
            <tr><td>${i}</td><td>x ${numB}</td><td>= ${int(i) * int(numB)}</td></tr>
        % endfor
    </table>
</body>
</html>

web.py with Templator:

from google.appengine.ext.webapp import util
import web

web.config.debug = False
render = web.template.render('app/webpy/templates')

class index:
  def GET(self,numA,numB):
    return render.index(numA,numB)
    
def main():
  urls = (
  	'/webpy/(.*)/(.*)',	'index'
  )
  app = web.application(urls, globals())
  util.run_wsgi_app(app.wsgifunc())

if __name__ == "__main__":
  main()
$def with (numA,numB)
<html>
<body>
    <table>
        $for i in range(int(numA)):
            <tr><td>$i</td><td>x $numB</td><td>= ${ i * int(numB) }</td></tr>
    </table>
</body>
</html>

web.py with Mako:

import os, web
from google.appengine.ext.webapp import util
from mako.template import Template

web.config.debug = False

class index:
  def GET(self,numA,numB):
    return Template(filename=os.path.join(
      os.path.dirname(__file__),"webpyTestMakoTemplate.html")).render(numA=numA,numB=numB)
    
def main():
  urls = (
  	'/webpymako/(.*)/(.*)',	'index'
  )
  app = web.application(urls, globals())
  util.run_wsgi_app(app.wsgifunc())

if __name__ == "__main__":
  main()
<html>
<body>
    <table>
        % for i in range(int(numA)):
            <tr><td>${i}</td><td>x ${numB}</td><td>= ${int(i) * int(numB)}</td></tr>
        % endfor
    </table>
</body>
</html>

Bottle with SimpleTemplate:

import os, bottle
from bottle import get, template
from google.appengine.ext.webapp import util

bottle.debug(False)

@get('/bottle/:numA/:numB')
def index(numA,numB):
  return template(os.path.join(
    os.path.dirname(__file__),'bottleTestTemplate.tpl'),numA=numA,numB=numB)

def main():
  app = bottle.default_app()
  util.run_wsgi_app(app)

if __name__ == "__main__":
  main()
<html>
<body>
    <table>
    %for i in range(int(numA)):
        <tr><td></td><td>x </td><td>= </td></tr>
    %end
    </table>
</body>
</html>

Bottle with Mako:

import os, bottle
from bottle import get
from google.appengine.ext.webapp import util
from mako.template import Template

bottle.debug(False)

@get('/bottlemako/:numA/:numB')
def index(numA,numB):
  return Template(filename=os.path.join(
    os.path.dirname(__file__),"bottleTestMakoTemplate.html")).render(numA=numA,numB=numB)

def main():
  app = bottle.default_app()
  util.run_wsgi_app(app)

if __name__ == "__main__":
  main()
<html>
<body>
    <table>
        % for i in range(int(numA)):
            <tr><td>${i}</td><td>x ${numB}</td><td>= ${int(i) * int(numB)}</td></tr>
        % endfor
    </table>
</body>
</html>

The benchmark is implemented in the following Ruby script:

#!/usr/bin/env ruby

require 'benchmark'
require 'net/http'
require 'uri'

baseURI = 'localhost:8082'
results = {
  'bottle' => {:time => 0.0, :success => 0, :failed => 0, :name => "bottle"},
  'bottlemako' => {:time => 0.0, :success => 0, :failed => 0, :name => "bottle w/Mako"},
  'gmako' => {:time => 0.0, :success => 0, :failed => 0, :name => "webapp w/Mako"},
  'gnative' => {:time => 0.0, :success => 0, :failed => 0, :name => "webapp w/Django"},
  'webpy' => {:time => 0.0, :success => 0, :failed => 0, :name => "web.py"},
  'webpymako' => {:time => 0.0, :success => 0, :failed => 0, :name => "web.py w/Mako"}
}

def callURI(baseurl,name,rows,index)
  uri = sprintf("http://%s/%s/%d/%d",baseurl,name,rows,index)
  url = URI.parse(uri)
  req = Net::HTTP::Get.new(url.path)
  res = Net::HTTP.start(url.host,url.port) do |http|
    http.request(req)
  end
  return res.kind_of? Net::HTTPOK
end

n = ARGV[0].to_i
rows = ARGV[1].to_i

1.upto(n) do |index|
  results.each_pair do |k,v|
    success = nil
    rez = Benchmark.realtime { success = callURI(baseURI,k,rows,index) }
    if success
      results[k][:time] += rez
      results[k][:success] += 1 
    else
      results[k][:failed] += 1
    end
    results[k][:average] = results[k][:time]/results[k][:success]
  end
end


best = 1
results.each_value.sort_by { |v| v[:average] }.each_with_index do |v,i|
  rank = 0
  if i == 0
    rank = 1
    best = v[:average]
  else
    rank = v[:average] / best
  end
  printf(
  "%16s: | %d total, %d OK, %d failed | %.3f secs overall, %.3f secs each | relative score: %.3f |\n",
  v[:name], v[:success]+v[:failed], v[:success], v[:failed], v[:time], v[:average], rank)
end

To further the comparison cases, the script is run with command line arguments assigned to the number of test iterations and the processing overhead in each cycles. After a bit of experimentation, I proceeded to benchmark with a high number of iterations, for better averaging, successively incrementing the processing overhead parameter to evaluate different rendering scenarios. Here is what comes out when testing.

no overhead:

ruby testScript.rb 1000 1
          bottle: | 1000 total, 1000 OK, 0 failed | 23.362s overall, 0.023s each | score: 1.000 |
   bottle w/Mako: | 1000 total, 1000 OK, 0 failed | 27.434s overall, 0.027s each | score: 1.174 |
   webapp w/Mako: | 1000 total, 1000 OK, 0 failed | 28.670s overall, 0.029s each | score: 1.227 |
          web.py: | 1000 total, 1000 OK, 0 failed | 79.013s overall, 0.079s each | score: 3.382 |
 webapp w/Django: | 1000 total, 1000 OK, 0 failed | 82.196s overall, 0.082s each | score: 3.518 |
   web.py w/Mako: | 1000 total, 1000 OK, 0 failed | 98.712s overall, 0.099s each | score: 4.225 |

low overhead:

ruby testScript.rb 1000 10
          bottle: | 1000 total, 1000 OK, 0 failed | 24.038s overall, 0.024s each | score: 1.000 |
   bottle w/Mako: | 1000 total, 1000 OK, 0 failed | 28.012s overall, 0.028s each | score: 1.165 |
   webapp w/Mako: | 1000 total, 1000 OK, 0 failed | 28.851s overall, 0.029s each | score: 1.200 |
 webapp w/Django: | 1000 total, 1000 OK, 0 failed | 89.095s overall, 0.089s each | score: 3.706 |
   web.py w/Mako: | 1000 total, 1000 OK, 0 failed | 94.399s overall, 0.094s each | score: 3.927 |
          web.py: | 1000 total, 1000 OK, 0 failed | 95.307s overall, 0.095s each | score: 3.965 |

medium overhead:

ruby testScript.rb 1000 100
          bottle: | 1000 total, 1000 OK, 0 failed | 26.813s overall, 0.027s each | score: 1.000 |
   bottle w/Mako: | 1000 total, 1000 OK, 0 failed | 27.378s overall, 0.027s each | score: 1.021 |
   webapp w/Mako: | 1000 total, 1000 OK, 0 failed | 28.920s overall, 0.029s each | score: 1.079 |
          web.py: | 1000 total, 1000 OK, 0 failed | 30.882s overall, 0.031s each | score: 1.152 |
 webapp w/Django: | 1000 total, 1000 OK, 0 failed | 34.118s overall, 0.034s each | score: 1.272 |
   web.py w/Mako: | 1000 total, 1000 OK, 0 failed | 224.659s overall, 0.225s each | score: 8.379 |

high overhead:

ruby testScript.rb 1000 1000
          web.py: | 1000 total, 1000 OK, 0 failed | 33.115s overall, 0.033s each | score: 1.000 |
   web.py w/Mako: | 1000 total, 1000 OK, 0 failed | 45.490s overall, 0.045s each | score: 1.374 |
 webapp w/Django: | 1000 total, 1000 OK, 0 failed | 62.941s overall, 0.063s each | score: 1.901 |
   bottle w/Mako: | 1000 total, 1000 OK, 0 failed | 74.669s overall, 0.075s each | score: 2.255 |
   webapp w/Mako: | 1000 total, 1000 OK, 0 failed | 81.663s overall, 0.082s each | score: 2.466 |
          bottle: | 1000 total, 1000 OK, 0 failed | 94.596s overall, 0.095s each | score: 2.857 |

very high overhead:

ruby testScript.rb 1000 10000
   bottle w/Mako: | 1000 total, 1000 OK, 0 failed | 78.331s overall, 0.078s each | score: 1.000 |
   webapp w/Mako: | 1000 total, 1000 OK, 0 failed | 80.489s overall, 0.080s each | score: 1.028 |
   web.py w/Mako: | 1000 total, 1000 OK, 0 failed | 97.224s overall, 0.097s each | score: 1.241 |
          web.py: | 1000 total, 1000 OK, 0 failed | 121.340s overall, 0.121s each | score: 1.549 |
          bottle: | 1000 total, 1000 OK, 0 failed | 288.903s overall, 0.289s each | score: 3.688 |
 webapp w/Django: | 1000 total, 1000 OK, 0 failed | 416.255s overall, 0.416s each | score: 5.314 |

As much as these benchmarks seems full of inexplicable corner cases, they can easily be reproduced with the scored average staying in a consistent range. No framework stayed on top of the list in all cases, adding the benchmark totals still reveals an average winner.

totals:

  bottle w/Mako: | 1.174 + 1.165 + 1.021 + 2.255 + 1.000 | total score: 6.615
  webapp w/Mako: | 1.227 + 1.200 + 1.079 + 2.466 + 1.028 | total score: 7.000
         bottle: | 1.000 + 1.000 + 1.000 + 2.857 + 3.688 | total score: 9.545
         web.py: | 3.382 + 3.965 + 1.152 + 1.000 + 1.549 | total score: 11.048
webapp w/Django: | 3.518 + 3.706 + 1.272 + 1.901 + 5.314 | total score: 15.711
  web.py w/Mako: | 4.225 + 3.927 + 8.379 + 1.374 + 1.241 | total score: 19.146

Finding absolutes insights through benchmarking frameworks performance, isn’t really possible. One can still use the results as a general guideline, knowing what the odds may look like, to make a more educated choice. As it turns out, the benchmark proves my choice of configuration has good speed potential. It also shows that both Bottle and web.py, with their built-in templating engine, could show better performance than AppEngine’s generic webapp with Django templates. None of those systems will surpass the others in every situations, so it is imperative not to rely only on performance figures when choosing companions components for your project. Aspects worth considering also includes features, stability, design, documentation, community and your own gut feelings about the component.

UPDATE!! (2010-08-25) Flask and Jinja are also added in the next benchmark

1 According to data from Genshi’s bench.

2 Django Templates (AppEngine’s built-in)

blog comments powered by Disqus

}

Older Posts... Blog powered by Jekyll.
Built using Liquid, RedCloth, Pygments and Blueprint.

Copyright © 2008-2010 Louis-Philippe Perron