require 'principle_of_least_surprise'

Posted by lp Thu, 26 Feb 2009 03:21:00 GMT

It is the karma of simple subjects to go unnoticed.

Starting a ruby program, requiring files, is one of those so simple subject that no one dare to question if better ways of doing exist. (Mostly)-Everybody on the user side just do the usual require 'rubygems', statement. (Mostly)-Everybody on the rubygems developer side just require all files in the gem, in a way allowing tests to pass, package everything, done... This works. Sometimes it doesn't, so the user go to the mailing, or the forum, doing whatever hack he can find until the require finally happens as intended. Why does it needs to be so? Why can't the gem developer package its jewell in a way which always works?

Ways to require

To make it always work, you first need to know what are the possible usage scenarios, this is usually where the developer understanding fails. Most think only of one, else fails miserably. The default rubygems usage is to install them as part of your local gem repository, then require 'rubygems' and require 'the-installed-gem'. For this to fail, you need to have a misconfigured gem load path, the developer can't do it for you, its your fault. The second rubygems usage usually still involves the local repository installation, but then instead of doing the require from there, you unpack it somewhere inside your program's directory and require it directly. This is not a hack, its a gem feature. It allows standalone programs to include their dependencies, and facilitates the deployment of ruby webapps in shared hosting environments. So rubygems developers MUST also support this flavor of require.

Where does it breaks?

The problem which can arise when unpacking a gem and requiring it directly is that the gem's folder is usually not included in the gem load path. The symptom gives you an error message where some deep folded file inside the gem is unfound. This happens when your top gem file (the one which is required), requires an other file with a path relative to the root of the gem folder. This path is nowhere to be found in the requiring program, so it fails.

Many ways, one solution

How do you get around this problem? If you are the user, you can either patch all the require inside the gem or add the gem's directory to the load path, then forget to tell the developer something went wrong, the world keep-on turning. If you are the developer, you also have 2 options. Either you add your gem's folder to the load path from your top gem file, which could be considered rude by many, because it impose yet an other load path on the user's gem environment. The second way, is the pure, always work way, of the absolute path. A bit prudish, but you can never be too careful to get your path right, when what you want is the user experience to go as much along the principle of least surprise as possible.

The reference example:

Lets say you have a gem organised like this:

/mygem/mygem.rb

/mygem/lib/helpers.rb

/mygem/lib/helpers/precious.rb

then the require relationship should look like this:

# file /mygem/mygem.rb
require File.join( File.dirname( File.expand_path(__FILE__)), 'lib', 'helpers')

# file /mygem/lib/helpers.rb
require File.join( File.dirname( File.expand_path(__FILE__)), 'lib', 'helpers', 'precious')

This approach is platform agnostic, because of File.join which will generate the proper slash/backslash, it is also immune to problematic loading of the same required file multiple times, because of the absolute paths obtained by File.expand_path. With absolute paths the require engine cannot be mistaken to load the same file twice. And finally, it is respectful of the user clean gem load path and choice of require flavor.

Posted in best practice | Tags absolute, bestpractice, loadpath, path, ruby, rubygems | no comments | no trackbacks

YAML::dump != Marshal.dump

Posted by lp Sun, 04 Jan 2009 21:08:00 GMT

People! Using Yaml for serialising and quick and dirty DB projects is fun... but not optimal.

If you can serialise with Marshal.dump instead of YAML::dump, then go for it!

A > 30x dump and > 2x load, is not what I would call a marginal difference.

See for yourself by running this test script: (git://gist.github.com/43177.git )

#!/usr/bin/env ruby

require 'yaml'

marshal_file = 'marshal_dump.marshal'
yaml_file = 'yaml_dump.yml'

def line; puts "\n"; end; line

# building a immense array of hashes
print "building the bully data object..."; $stdout.flush
bully = Array.new
indexes = 5000; keys = 100; rand_value = keys * 100; entry_count = 0
indexes.times do |index|
  bully[index] = Hash.new
  keys.times do
    bully[index][rand(rand_value)] = rand(rand_value)
  end
  entry_count += bully[index].size
  print '.' if index % 10 == 0; $stdout.flush
end
line
puts "bully data structure containing #{entry_count} entry built"
line

# marshall test

puts "Testing Marshal dump..."
start = Time.now

f = File.new(marshal_file, 'w')
f.puts( Marshal.dump(bully) )
f.close

marshal_dump = Time.now - start
puts "marshall dump time: #{marshal_dump} sec"; line

puts "Testing Marshal load..."
start = Time.now

bully_marshaled = Marshal.load(File.open(marshal_file))

marshal_load = Time.now - start
puts "marshall load time: #{marshal_load} sec"
marshal_size = File.size(marshal_file)
puts "for #{marshal_size} bytes"; line

if bully.first == bully_marshaled.first &&
  bully.last == bully_marshaled.last &&
  bully.length == bully_marshaled.length then
  puts "and both bully match!!!"
else
  puts "and they don't match!!!"
end
line
# yaml test

puts "Testing YAML dump..."
start = Time.now

f = File.new(yaml_file, 'w')
f.puts( YAML::dump(bully) )
f.close

yaml_dump = Time.now - start
puts "yaml dump time: #{yaml_dump} sec"; line


puts "Testing YAML load..."
start = Time.now

bully_yaml = YAML::load(File.open(yaml_file))

yaml_load = Time.now - start
puts "yaml load time: #{yaml_load} sec"
yaml_size = File.size(yaml_file)
puts "for #{yaml_size} bytes"; line

if bully.first == bully_yaml.first &&
  bully.last == bully_yaml.last &&
  bully.length == bully_yaml.length then
  puts "and both bully match!!!"
else
  puts "and they don't match!!!"
end
line
# Final Stats:

dump_stat = marshal_dump / yaml_dump
load_stat = marshal_load / yaml_load
size_stat = marshal_size.to_f / yaml_size

printf "STATS: Marshal dump finishes in %.4f of YAML time\n", dump_stat
printf "STATS: Marshal load finishes in %.4f of YAML time\n", load_stat
printf "STATS: Marshal file is %.4f of YAML file size\n", size_stat
line

Posted in benchmark | Tags marshal, ruby, serialisation, yaml | no comments | no trackbacks

Forward Thinking

Posted by lp Wed, 31 Dec 2008 23:55:00 GMT

Let me end the year on a pompous tone, making 3 bold statements about next year tech world transformations.

The Browsers Year

2009 will be the year where people which are still using Microsoft Internet Explorer, of any version, will wake up to the fact that you can actually choose your internet browser. The times of the Netscape vs IE monopoly are far behind, we now get the benefits of competition for browser features and collaboration for rendering and scripting engines. One of the fastest improving ground in the programming world today seems to be the JavaScript engine of all browsers but IE. When Google, Apple and the Mozilla Foundation all get on a speed race developing a better JavaScript engine for their browsers, one cannot be really far away from reality thinking JavaScript is the next big programming thing.

Make that iPhone bigger, let me throw away my mouse

What do you think is the next insanely great product Apple will launch? No kidding, the iPhone and iPodTouch are killer products, and its likely along this line of success that Apple will plan their future: the Mac elegant interface coupled with multi-touch screen wonders. Multi-touch screen interfaces have been prototiped for at least 10 years, but they never made it through to the consumer market, mainly because of the unavailability of any willing desktop OS company, ready to develop multi-touch pointer devices support for their systems. Now that the market has been seduced by the idea, Apple has all the cards in its game to launch that same beautiful iPhone interface paradigm for its desktop computers. Do you think they'll miss that chance? Many will say Microsoft already did it when they tried to launch their own, bulky, Surface computer. But again, who wants this 10k$ coffee table to grab dust in their living room. No mind blowing "Only this device allow me such and such" has been reclaimed for the Surface, at least nothing compared to how the iPhone has hit the market. The Surface is ready made to be thrown to the weird technologies junkyard. Consumers want useful, unobtrusive, invisible technologies and this is exactly what Apple will provide, they control their hardware and software platforms in a way no other manufacturers can, and will use this unique power to leverage the next mainstream household computer technology.

All for One

In the current failing world economy, taking into consideration a possible, maybe inevitable, society paradigm shift, solidarity is key. Joint Open Source component development is an already widespread mechanism of technology adoption, and will become the de facto approach embraced by every trendsetting company. The tendency will be "Do it or miss it". Changing for the good, the way competition and collaboration operates on innovation. Entrepreneurial strategy will evolve from being the one owning a technology to being the best using a technology. You'll need to be fast, intelligent and creative to excel at these new business games, and hopefully these tendencies will allow for a quick, positive restructuration, of the now failing world economy.

Posted in opinions | Tags 2009, predictions, technology, trends | no comments | no trackbacks

ruby abundance 32 core ready

Posted by lp Sun, 30 Nov 2008 07:28:00 GMT

Posted in screenshots | Tags concurrency, ruby, smp | no comments | no trackbacks

Garden Threads Patch

Posted by lp Sat, 29 Nov 2008 04:21:00 GMT

Poetic Infancy Paradigm

This month was very much single threaded. A fantastic single Ruby thread. Ruby is wonderful, its ecosystem is is very rich and the flower it grows even comes to perfume other unscented languages. A benefactor creative influence on the overall programming sphere. Many say it's not yet matured, but no one is pleased thinking of Ruby as young, because in fact its inception is as old as Java. While Java was raised on corporate growth hormone injections, Ruby was rice fed in a secluded zen master monastery. As some of you may know, life in a monastery is very different than the normal human life. Ruby's life is now starting to grow past its monk education into a very fertile adolescence. It wants to differentiate from its roots and evacuate its unmanageable mating desire. jRuby! Rubinius! YARV! MacRuby! IronRuby! Anyone? So, while Ruby love making style turns out to be pretty ecstatic and polygamous, its monastery roots are still deadly single threaded. Ruby's got its gifts from ancient teachings and will honor its masters always. Its new funky lifestyle makes for a paragraph in its epitaph, but one cannot escape its own destiny, and Ruby's destiny is one of Masters.

thread of the multi-threads

I had to face two deep truths this month, first, my total ignorance about code threads (damn! so absurd!! but my Perl antecedents never thought me about this TINY DETAIL of programming), second, the fact that Ruby threads are not much more than a really basic way to execute code in parallel. By really basic I mean, not really up to any serious concurrent task execution, this is because of its Global Interpreter Lock, a kind of thread memory protection mechanism, also implemented in other Dynamic languages like Python and Perl, also making them ineffective at handling threads. MRI (Matz Ruby Interpreter or Monastery Ruby Inception) in its 1.8 and 1.9 realities, surely had a thought for threading, but the model is akin to sprint relay, where multiple process exchange priority on one row of execution. It's a thread implementation which doesn't scale to multiple processors, given the only row available, for which blocking is likely to interfere at some point in execution.

Ten Thousands Ways

Certainly not the first one to face this problem, solution is already turned to many. The first obvious one being to use jRuby instead of MRI... If I could I would have done it, but the project I am working on does not allow it because of specific dependencies, and I always had trouble with Java technologies, I must have bad JVM karma! Many other very good production grade tools are available, DRb, Starling and Beanstalkd are amongst them, all having their good and bad sides, none of them still would fit my plan...had to resign to write my own tool for the job.

KISS my Thread

needs:

• unobtrusive to program flow
• built from Ruby standard library
• simple to use

require 'abundance'

gardener = Abundance.gardener( :block_size => 8192, :rows => 2, :init_timeout => 2) do
  
  # your garden process pre-run initialization comes here
  
  Abundance.grow do |seed|
    # block execute forever, growing the sent commands
    
    command = seed.sprout  # gets your seed in

    results = some_magic_function(command) # do something with command

    seed.crop( true, results)  # send the crop out, ready to be harvested 
    
  end
 
end

id1 = gardener.seed('command1')
id2 = gardener.seed('command2')

result1 = gardener.harvest(id1)
result2 = gardener.harvest(id2)

# with many more seeds over here

gardener.close

It's surprising that even on my single core machine, I get a speed gain from parallelizing certain processes with Abundance, and I can run what would be thread blocking processes in a non-blocking fashion . On multicore boxes I expect it to truly crop gold. In its current early implementation, it's really meant for batch processing, like throwing a bunch of paths in for parallel file parsing. You can get it from GitHub: http://github.com/lp/abundance/tree/master. An installation page is on the way...

Update!!! abundance is now stable, you can learn more about the subject abundance@rubyforge

Posted in introductions | Tags concurrency, library, ruby, threading | no comments | no trackbacks

oh, so you can also do that!!!

Posted by lp Sat, 08 Nov 2008 03:50:00 GMT

and maybe you should...

Posted in screenshots | Tags programming, ruby, threading | no comments | no trackbacks

SkypeIn For The Rest of Us

Posted by lp Thu, 30 Oct 2008 03:09:00 GMT

Since Skype launched its SkypeIn service, somewhere in 2005, I keep going back to their website, always in dismay, facing the absence of a Canadian flag on the supported countries page. Last time I went though, I kept looking a bit more, thats when I found about the DIDs...

A much better hack

DID, stands for Direct Inward Dialing. It can be seen as a way to buy yourself a phone number, which doesn't come attached to any telco with their usual lock-in contracts. DID providers usually offers this service at very low monthly fees, and they can even supplement it with many flexible routing services. The one I choosed, DIDww, charges you around 5$ a month for the number, and of course it offers support for Skype!

But wait!!! the Skype service its only beta...

I would have liked it to work at first sight... but I'll have to wait a little! I must tell though its really no good reason to stop, because there is lots of possibilities in there! Versatility!! It works with most VOIP systems, with MSN and GTalk, with traditional PSTN lines, and also allows you to mix and match them to your taste. With all those possibilities there is no way to get locked in there like in the old-telecom-days!

My cute little bug

At present, there seem to be a couple of bugs with the Skype mapping in there. The first one I noticed, gives the symptom of only sometimes sending the caller a ringtone... a long blank pause at the other end of the line. Then it gets to Skype, but cuts the line when voicemail jumps in.... ARGGHHH!!!! I take the deep breath Skype has told me to... then mmm, what can I do? I really want to stick with Skype, having a dedicated Belkin Skypephone, a beautifull wi-fi pet I like to bring everywhere with me.

What I DID

Conviced that it would work, I decided to try a few things before totally losing faith... Grouping a few mappings together seemed like the best strategy. I tried with GTalk, the voicemail did work, but then GTalk seems a bit too beta to me, and most of all only offerered a Windows client, shame on you Google! I looked at a couple of VOIP providers, most made me feel like going into yet another wanabe telecom, and the rests makes you doubt they will survive in the long run. There was also this alien, but very creative and with lots of potential PhoneGnome, very very very interesting, but not at all what I need... and Gizmo, this humble project feels very much like Skype, has clients for the 3 OSes and guess what, the voicemail not only works, it works good! You are allowed to record your own welcome message, it sends the received message as wave files to the email of your choice, all for free!!

A good starting point

What's fun getting into this DID based setup is that it will allow me to evolve my communications assets as technology grows, all while keeping the same phone number, or even to keep the same assets while moving to a different area code. The recipe which works for me now is to chain Gizmo after Skype with deactivated voicemail. What it does is it tries to call me on Skype for 15 seconds (sometimes with no rings, ARGHH!! but people get used to it), if I don't answer then it jumps to Gizmo, for which a client is not open anywhere, giving the caller a voicemail service straight away. Cheep phone service that serves you better than those from the all-encompassing you-will-never-get-away-from-us Telecoms!

Posted in tech hacks | Tags did, gizmo, skype, skypein, voip | no comments | no trackbacks

Stand Up For Haml!

Posted by lp Sun, 05 Oct 2008 05:23:00 GMT

  <tag id="idname" class="classname1 classname2">  
    This is my
    Content
  </tag>

  %tag#idname.classname1.classname2 
    This is my
    Content

1 : <
2 : p
3 : >
4 : <
5 : /
6 : p
7 : >

1 : %
2 : p

Posted in opinions | Tags evolution, haml, language, programming, templating | no comments | no trackbacks

CodeRay textfilter for Typo

Posted by lp Thu, 18 Sep 2008 00:11:00 GMT

How could I build a tech blog without proper code block highlighting?

Thats what I realised when I saw Typo coming a bit short in term of syntax highlighting. Not that there isn't one, but the Syntax textfilter included with the app is missing some languages definition in its arsenal to make it a decent choice, unless someone would only write Ruby, XML and YAML blocks in its blog. There is alway the prospect of writing a language module for Syntax but lets not rely on this for now.

The Odds:

I did a bit of research, and found three other syntax highlighting systems that suited Typo, with no coding or a minimum of coding required:

• tmcode, already built into Typo, assure the loading of the css necessary to display code html generated by the wonderfull Textmate OSX text editor.
  Pros: very esthetic result, fast load time, support lots of languages
  Cons: you need a Mac with Textmate, not a true "Markup and Publish" solution.
• Ultraviolet, syntax highlighting library built on top of Textpow, which is built on top of Oniguruma-Ruby, which is built on top of Oniguruma. You get the option of using either Radiograph or tm_syntax_hightlighting as your Rails interface for Ultraviolet.
  Pros: uses Textmate highlighting making it an esthetic choice, decent language support
  Cons: dependency chain has many obstacles, not very fast,
• CodeRay, syntax highlighting gem built entirely in Ruby
  Pros: easy gem installation, esthetically ok
  Cons: language support not that extensive.

The Test:

I tried hard to make Ultraviolet work... no problem on my development machine... but the hosted server never got to compile the first dependency, Oniguruma, adequately. So I jumped off the UV boat a bit depressed after a couple of days of hard labor. It was my first choice mainly for the large language palette, ignoring the speed issue which could have backlashed after regular usage.

So I turned to tmcode and Coderay. tmcode wasn't a bad option after all, giving the speed gain allowed by not having to process anything on the server at display time... and I happen to be both a Mac lover and a Textmate lover so there wouldn't be much pain in using this solution. But still, I wanted something I could use from any computer, so I embarked the CodeRay installation anyway... my language support of choice is not there at the moment, but the developper has a new version coming late september, which could change that consideration, and at worst, I could end up coding the languages support I need myself.

CodeRay has hit the place!

Installation is straightforward:

gem install coderay

Then what?
Typo has an ideal mechanism to integrate with CodeRay, called textfilters.
You can code one yourself by reading the Typo documentation on the subject: http://scottstuff.net/blog/articles/2005/08/23/introduction-to-typo-filters

which I did...
so you also have the option of using my implementation of CodeRay as a textfilter called CodeView.
Here it is: http://software.spiralix.org/typo/textfilter/typo_textfilter_codeview.zip
place this whole folder, unzipped, inside the vendor/plugins/ directory of your Typo installation and you are ready to go.
To get usage information, loggin to your Typo admin panel, under Settings->TextFilters you will find your CodeView macro with its help link to the far right of the page.

Posted in tutorials | Tags coderay, highlighting, oniguruma, rails, textfilter, tmcode, typo, ultraviolet | 1 comment | no trackbacks

Typo Blog on Dreamhost

Posted by lp Wed, 17 Sep 2008 01:03:00 GMT

Typo, is a blogging system built with Ruby on Rails. I found this gem at a moment where I was thinking about building my own blog engine, and as it turned out, I liked it enough to use it to build this blog.

The overall Typo on Dreamhost installation is pretty straightforward. Dreamhost does a very good job at running Rails web application, mostly since the arrival of mod_passenger, the Apache module for running Ruby web applications. Think about it, from all web application languages you can run on Dreamhost, only Ruby has its own module for running on the web server. Everything else gets to run on CGI or Fast-CGI (yes, even PHP run as CGI on Dreamhost as on many other shared hosting, mainly for security purposes) giving Ruby a noticeable performance advantage. For this reason I expect to see a rise in the number of Ruby CMS, which are almost inexistent at this point compared to the plethora of PHP CMS out there. Also in the favour of this Ruby uprise, we could mention the sexyness of the language and the ease of development of the Rails framework as well as other Ruby frameworks.

One thing to keep in mind when running Ruby webapps on Dreamhost under Passenger, is that you don’t have acces to the Apache .conf file. This is really to protect you from messing with critical web server settings, a subject you probably don’t want to delve into considering the amount of possibility/flexybility/complexity it offers, especially if you want some free time to blog on your own afterward. This lack of access to the .conf file won’t stop you from running a Ruby blog on Dreamhost though, will it? It could… the main consequence being that the Ruby environment (Ruby and its gems) that Passenger will use for serving your apps is not configurable. (Goshh!!) But, there is a but! You can still install your own Ruby environment under your own Dreamhost user account, some generator gems will be usefull right away, like the Typo gem, which creates the code to run the app and then let Passenger serve is as a normal Ruby webapp, and for most of the other gems there is a neat packaging gem, called gemsonrails, that allows to pack gems inside your Rails apps with no more external-custom-installed gem dependencies left.

Preparation:

First, you must configure your Dreamhost user environment to allow local installations to happen. You will do this by setting a couple of environment variables to point to a local install path. I recommend doing all these operations using SSH/SFTP for security considerations, but you may also choose to use telnet/ftp as it is somewhat easier to setup and in most occasions faster.

Start by creating directories for your installation on top of your user’s home folder:

cd $HOME
mkdir .gem bin include lib src

Edit .bash_profile, adding this to the end of the file:

. .bashrc

Then edit .bashrc, putting those lines in it:

export PATH="$HOME/bin:$HOME/.gems/bin:$PATH"
export RUBYLIB="$HOME/lib:$RUBYLIB"
export GEM_HOME="$HOME/.gem"
export GEM_PATH="/usr/lib/ruby/gems/1.8:$GEM_HOME"
alias gem="nice -n19 ~/bin/gem"

Then comes the installations:

Proceed to the download the sources for Ruby and Rubygems, place them inside the src directory you created earlier in this tutorial. I choosed Ruby Enterprise Edition from Phusion, the creators of mod_passenger, mainly because of its ease of installation and in hope I could benefit from its improved memory management (garbage collection), but unfotunately at this point since we cannot modify the Apache .conf file we are stuck with the Dreamhost installed Ruby… but we will be ready for the day we get that configuration option!

You can download Ruby from: www.rubyenterpriseedition.com/

And Rubygem from: http://rubyforge.org/frs/download.php/45905/rubygems-1.3.1.tgz

The Rubygem and Ruby installations goes as follow:

cd ~/src
tar xzvf rubygems-1.3.1.tgz
cd rubygems-1.3.1
ruby setup.rb --prefix=$HOME

cd ~/src
tar xzvf ruby-enterprise-X.X.X.tar.gz
./ruby-enterprise-X.X.X/installer

When asked by the Ruby Enterprise installer where the installation should take place, answer with the full path to your user home directory. And by fullpath I don’t only mean /home/YOURUSERNAME/, what I really mean is something like /home/.HIDDENFOLDERNAME/YOURUSERNAME/ . If you don’t know that path, try connecting to your users directory with a SFTP client and the information will leak for you somewhere in there.

If you also choosed to install Ruby Enterprise Edition , then you will already have Rails and friends installed, if not then go for it, install them! So the last thing left to install is the Typo gem itself, and that is an easy one:

gem install typo

and thats all!
To get started with Typo refer to typosphere’s documentation:
http://typosphere.org/wiki/typo

wait! wait! wait!

One last detail… If you follow the typosphere documentation exactly you’ll get into troubles with the Dreamhost Police, because the installer defaults to using Mongrel… and starts it automatically at the end of the installation process, which goes against the host policy on permanently running processes. So don’t forget to include a “web-server=external” in your install options… and if its already too late… they usually let you correct your mistake without sending you to the Dreamhost Jail, so you can use the “config” typo terminal command to finally set your options properly.

for more info on Typo’s options visit:
http://scottstuff.net/blog/articles/2006/07/23/typo-installer

Posted in tutorials | Tags dreamhost, gem, installation, rails, tutorial, typo | no comments | no trackbacks