Garden Threads Patch
Posted by lp
Poetic Infancy Paradigm
This month was very much single threaded. A fantastic single Ruby thread. Ruby is wonderful, its ecosystem is is very rich and the flower it grows even comes to perfume other unscented languages. A benefactor creative influence on the overall programming sphere. Many say it's not yet matured, but no one is pleased thinking of Ruby as young, because in fact its inception is as old as Java. While Java was raised on corporate growth hormone injections, Ruby was rice fed in a secluded zen master monastery. As some of you may know, life in a monastery is very different than the normal human life. Ruby's life is now starting to grow past its monk education into a very fertile adolescence. It wants to differentiate from its roots and evacuate its unmanageable mating desire. jRuby! Rubinius! YARV! MacRuby! IronRuby! Anyone? So, while Ruby love making style turns out to be pretty ecstatic and polygamous, its monastery roots are still deadly single threaded. Ruby's got its gifts from ancient teachings and will honor its masters always. Its new funky lifestyle makes for a paragraph in its epitaph, but one cannot escape its own destiny, and Ruby's destiny is one of Masters.
thread of the multi-threads
I had to face two deep truths this month, first, my total ignorance about code threads (damn! so absurd!! but my Perl antecedents never thought me about this TINY DETAIL of programming), second, the fact that Ruby threads are not much more than a really basic way to execute code in parallel. By really basic I mean, not really up to any serious concurrent task execution, this is because of its Global Interpreter Lock, a kind of thread memory protection mechanism, also implemented in other Dynamic languages like Python and Perl, also making them ineffective at handling threads. MRI (Matz Ruby Interpreter or Monastery Ruby Inception) in its 1.8 and 1.9 realities, surely had a thought for threading, but the model is akin to sprint relay, where multiple process exchange priority on one row of execution. It's a thread implementation which doesn't scale to multiple processors, given the only row available, for which blocking is likely to interfere at some point in execution.
Ten Thousands Ways
Certainly not the first one to face this problem, solution is already turned to many. The first obvious one being to use jRuby instead of MRI... If I could I would have done it, but the project I am working on does not allow it because of specific dependencies, and I always had trouble with Java technologies, I must have bad JVM karma! Many other very good production grade tools are available, DRb, Starling and Beanstalkd are amongst them, all having their good and bad sides, none of them still would fit my plan...had to resign to write my own tool for the job.
KISS my Thread
needs:
- unobtrusive to program flow
- built from Ruby standard library
- simple to use
require 'abundance' gardener = Abundance.gardener( :block_size => 8192, :rows => 2, :init_timeout => 2) do # your garden process pre-run initialization comes here Abundance.grow do |seed| # block execute forever, growing the sent commands command = seed.sprout # gets your seed in results = some_magic_function(command) # do something with command seed.crop( true, results) # send the crop out, ready to be harvested end end id1 = gardener.seed('command1') id2 = gardener.seed('command2') result1 = gardener.harvest(id1) result2 = gardener.harvest(id2) # with many more seeds over here gardener.close
It's surprising that even on my single core machine, I get a speed gain from parallelizing certain processes with Abundance, and I can run what would be thread blocking processes in a non-blocking fashion . On multicore boxes I expect it to truly crop gold. In its current early implementation, it's really meant for batch processing, like throwing a bunch of paths in for parallel file parsing. You can get it from GitHub: http://github.com/lp/abundance/tree/master. An installation page is on the way...
Update!!! abundance is now stable, you can learn more about the subject abundance@rubyforge
