Rake Fundamentals: FileLists

In the last post, we created a rakefile that will copy a discreet set of *.bin files to a package directory for distribution using a single “publish” task.  But it’s pretty verbose.  We can refactor a bit to make the rakefile smaller, and make it handle an arbitrary list of *.bin files to be published to the /package directory using a FileList.  A FileList is pretty simple – it’s just an array of file names.  We’ll gather up all the *.bin files in the source/bin directory, and iterate through them to dynamically create file tasks for each of them and append each of those file tasks to the publish task’s prerequisite list.  Here we go:

require 'rake'

BIN_DIR     = "source/bin"
PACKAGE_DIR = "package"

directory PACKAGE_DIR

BUILD_PRODUCTS = FileList[File.join(BIN_DIR, "**/*.bin")]
BUILD_PRODUCTS.each do |product|
packaged = product.sub(BIN_DIR, PACKAGE_DIR)
file packaged => product do
cp product, packaged, :verbose => true
end
task :publish => [PACKAGE_DIR, packaged]  
end

desc "publish *.bin files to #{PACKAGE_DIR}"
task :publish

We cut out a bunch of stuff we don’t really need, and all we’re left with on the command line is the publish task:

We don’t really care about all the individual file tasks, or the PACKAGE_DIR directory task, we just want our development team to care about the publish task.  So we set BUILD_PRODUCTS to a list of all the *.bin files in the BIN_DIR, and stuffed them all into the publish task that appears in our task list.  And to make it appear in the task list, we just give it a description after we get out of the .each do…end loop.  The completed publish task works just like it did before, except that now it can handle a completely arbitrary list of *.bin files.  Try it out – run “rake publish”, add a file called “file4.bin” to /source/bin, and then run “rake publish” again and see what happens.

Rake Fundamentals: directory tasks

A build system is about building software.  And on a purely mechanical level, the work of building software is largely about creating files and moving them around.  The task keyword is the general workhorse for doing all kinds of things in Rake.  But you can create specialized tasks focused on automating particular things.  Rake comes with a couple of special kinds of tasks for managing directories and files. 

The directory task

Remember how the keyword “task” is really just a call to a method named “task()”?  Well the “directory” keyword is very similar – it’s really just a call to a method named “directory()”.  “directory” is a special kind of task that has one main purpose: it creates the directory it names if that directory doesn’t exist.  You don’t need to provide a do…end block parameter to a directory task, just a directory name.  Here’s a little rakefile to illustrate what I mean:

require 'rake'

BIN_DIR       = "source/bin"
PACKAGE_DIR   = "package"

desc "create a #{BIN_DIR} directory"
directory BIN_DIR

desc "create a #{PACKAGE_DIR} directory"
directory PACKAGE_DIR

There are a couple of new concepts here:  1) BIN_DIR and PACKAGE_DIR are Ruby constants.  By convention, Ruby constants are expressed in all caps with underscores.  2) The #{…} stuff is Ruby string replacement syntax.  So here’s what we see when we look at the task list for this rakefile:

We have two tasks: one for creating a directory named “package”, and another for creating a directory named “source/bin”.  Since we typically work with compiled languages in the .NET world, there will usually be a compile step in our build process that will take care of creating a /bin directory for us.  But since we’re just pretending right now, I threw it in just for funsies.  You can call these tasks from the command line, and since the directories they name don’t initially exist, rake creates those directories:

Making a directory is pretty simple – you just make the directory, and then you have the directory you wanted.  That’s why you don’t have to pass a do…end block to the directory task – the directory task already knows how to create a directory.  Ok, I hear what you’re saying: big deal.  And you’re right, calling “rake package” from the command line is not particularly useful when you could just call “md package” on the command line.  But directory tasks are very useful when you use them as prerequisites for file tasks.

Next up: file tasks

Rake Fundamentals: file tasks

The file task

The “file” task keyword is a call to a method called, you guessed it, “file()”.  Like the directory task, it’s purpose is to create the thing that it names.  But making a file is a bit more involved than making a directory – you have to know what goes into the file when you make it.  What’s the name and extension of the file?  What’s the source for the data in the file?  Should there be any kind of transformation, reformatting, or translation of the source data to the data to be saved in the file?  What directory should you save the file in?  You can provide all that info to the file task when you define it.  Here’s some pseudo-code for a file task:

desc "copy source file to target file"
file target_file => source_file do
cp source_file, target_file
end 

This isn’t a real file task, don’t try to run it.  But what’s going on here?  The name of this task is “target_file”, and it has a prerequisite called “source_file”.  And if the file task decides that it’s supposed to execute the do…end block, rake uses the “cp” command to copy the file named “source_file” to a file named “target_file”.  How does the file task decide that the do…end block needs to be executed?  Two things factor into that decision: 1) if target_file doesn’t exist, then it obviously needs to be created, so the do…end block gets executed.  2) If the source_file is newer than the target_file, that means that the source_file has changed since the last time that the target_file was created, and the do…end block gets executed again so that the target_file will get refreshed with the changes in source_file.

Ok, let’s pretend.  Let’s go back to our little rakefile we used to create directories.  And let’s pretend that we have a source/bin directory that contains some binary files that are the result of compilation – we’ll call them “file1.bin”, “file2.bin”, and “file3.bin”.  And furthermore, let’s pretend that we want to package those files up for distribution.  We can write some file tasks to make that happen:

require 'rake'

BIN_DIR = "source/bin"
PACKAGE_DIR = "package"

desc "create a #{BIN_DIR} directory"
directory BIN_DIR

desc "create a #{PACKAGE_DIR} directory"
directory PACKAGE_DIR


desc "copy file1.bin to package"
file "package/file1.bin" => [PACKAGE_DIR, "source/bin/file1.bin"] do |t|
cp t.prerequisites[1], t.name, :verbose => true
end

desc "copy file2.bin to package"
file "package/file2.bin" => [PACKAGE_DIR, File.join(BIN_DIR, "file2.bin")] do |t|
cp t.prerequisites[1], t.name, :verbose => true
end

desc "copy file3.bin to package"
file "package/file3.bin" => [PACKAGE_DIR, File.join(BIN_DIR, "file3.bin")] do |t|
cp t.prerequisites[1], t.name, :verbose => true
end

And here’s our task list now:

In our file “package/file1.bin” task, there are two prerequisites: PACKAGE_DIR and “source/bin/file1.bin”.  If we execute “rake package/file1.bin”, the first thing that will happen is the package directory will get created since it doesn’t already exist.  And since package/file1.bin doesn’t exist either, our do…end block will get executed.  That do…end block looks a bit different than anything we’ve seen so far, but it’s really not that complicated.  Just like a lambda in c#, you can pass parameters to a do…end block in Ruby – in this case |t| represents the file task, and we’re passing it into our do…end block so that we can have access to some info about the task, particularly the prerequisites list and the name of the task.  “cp t.prerequisites[1], t.name, :verbose => true” is very similar to the “cp source_file, target_file” line from the pseudo-code file task above.  In this case, the source_file is “t.prerequisites[1]” – in other words, the source_file is “source/bin/file1.bin”, which is the value in element 1 of the file task’s prerequisites array.  The target_file is “t.name” – in other words, the target_file is “package/file1.bin”, the name of the file task.  And the “:verbose => true” thing just means “tell me what you’re doing when you copy this file.”  If we run this task, here’s what we get:

Before we run the task, there’s no package directory.  After we run the task, there’s a package directory and it contains file1.bin.  Cool.

The file “package/file2.bin” task looks a little bit different. Instead of a source_file prerequisite called “source/bin/file2.bin”, we’ve got this:

File.join(BIN_DIR, “file2.bin”)

File.join() is a convenient Ruby method for formatting string input into properly structured path+filenames.  “source/bin” and “file2.bin” are appended together and separated by a “/” character, and the result is “source/bin/file2.bin”.

I see you rolling your eyes again: whooptie-do, we can make files from the command line.  Yep, I agree, the way we’ve set up our rakefile so far doesn’t really give us a compelling reason to use file and directory tasks.  Calling each file task from the command line seems really cumbersome.  What if we had a “publish” task that published all three of our *.bin files to the /package directory in one fell swoop?  Let’s add this to the end of our rakefile:

desc "copy file1.bin to package"
task :publish => "package/file1.bin"

desc "copy file2.bin to package"
task :publish => "package/file2.bin"

desc "copy file3.bin to package"
task :publish => "package/file3.bin"

It looks like we’re adding three “publish” tasks, but we’re really only adding one:

Ruby allows you to do some pretty fancy dynamic stuff, and rake takes advantage of that by allowing you to “re-open” a task that has already been defined and add some more stuff to it.  In our case, we’re re-opening the “publish” task and adding a new prerequisite for each of our *.bin files.  The result is a single publish task that copies all three *.bin files to the /package directory:

Ok, now we’re getting somewhere.  Now if only one of our source/bin/*.bin files changes, only the changed file will get copied to the /package folder.  Open up source/bin/file2.bin and put some random text in it, and then run “rake publish” again:

We’re making progress, but our rakefile is still way too verbose.  We need to find a way to reduce the amount of code it takes to copy these files out to /package.  And we should also be able to copy an arbitrary list of files, not just a discreet list we have to keep updating.

Next up: FileLists

Rake Fundamentals: tasks with prerequisites

Rake tasks can have other tasks as prerequisites.  That means that a tasks prerequisites are executed before the task itself when the task is called from the command line.  Just like Nant, if you have a lot of small tasks you often run together, it’s convenient to roll them up into one task with prerequisites.  Let’s use rake to rock out:

require 'rake'

desc "open guitar case and get guitar"
task :get_axe do
puts "opening guitar case"
puts "grabbing telecaster"
end

desc "plug guitar into amp"
task :crank_up do
puts "plugging into 1959 fender tremolux"
puts "turning up volume"
end

desc "play A chord"
task :play_a do
puts "playing A chord"
end

desc "play D chord"
task :play_d do
puts "playing D chord"
end

desc "play E chord"
task :play_e do
puts "playing E chord"
end

desc "bask in the glory of adulation"
task :take_bow do
puts "taking a bow"
puts "winking at the girl in the front row"
end

desc "give the people what they want"
task :rock_the_house => [:get_axe, :crank_up, :play_a, :play_d,
:play_a, :play_e, :take_bow]

This rakefile defines these tasks:

And you could call each of these tasks individually if you wanted to.  But we want to rock the house, right?  So all we need to do is call “rake rock_the_house”.  According to the prerequisite list in the :rock_the_house task, rocking the house should consist of getting out our axe, cranking up, jamming some chords in this order: ADAEA, and bowing to a standing ovation.  Let’s give it a try and see what happens:

Our rock_the_house task ran all the prerequisites, except it didn’t play all the chords we told it to play – we only played ADE, not ADAEA.  LAME!  That’s because rake keeps track of tasks it has executed, and by default will only execute them once.  That might be cool for some tasks, but we want to play the blues, so we have to re-enable our “play_a” task every time we execute it so that we can execute it again.  Let’s fix our rakefile by creating a “play_the_blues” task that knows how to play our chords correctly:

require 'rake'

desc "open guitar case and get guitar"
task :get_axe do
puts "opening guitar case"
puts "grabbing telecaster"
end

desc "plug guitar into amp"
task :crank_up do
puts "plugging into 1959 fender tremolux"
puts "turning up volume"
end

desc "play A chord"
task :play_a do
puts "playing A chord"
end

desc "play D chord"
task :play_d do
puts "playing D chord"
end

desc "play E chord"
task :play_e do
puts "playing E chord"
end

desc "bask in the glory of adulation"
task :take_bow do
puts "taking a bow"
puts "winking at the girl in the front row"
end

task :play_the_blues do
Rake::Task[:play_a].invoke
Rake::Task[:play_a].reenable

Rake::Task[:play_d].invoke

Rake::Task[:play_a].invoke
Rake::Task[:play_a].reenable

Rake::Task[:play_e].invoke

Rake::Task[:play_a].invoke
end

desc "give the people what they want"
task :rock_the_house => [:get_axe, :crank_up, :play_the_blues, :take_bow]

Sometimes instead of calling a task as a prerequisite, you want to invoke it directly from within another task.  Calling the invoke() method on Rake::Task[] is how you do that.  Once you’ve called a task, you can reenable it to be called again by using Rake::Task[].reenable().  We’ve modified our “rock_the_house” task by calling our new “play_the_blues” task as a prerequisite.  Notice that since we didn’t give our “play_the_blues” task a description, it doesn’t show up on the task list when you type “rake –T”.  But it’s still available for use.  Also, notice that we didn’t provide a do…end block to our “rock_the_house” task.  Since we defined everything we need for rocking the house in our other tasks, we only needed to provide the “rock_the_house” task with prerequisites.  Let’s see what happens now:

Perfect!  All the chords get played in the right order.  We’ve got a single task to call from the command line that can do all the things we want it to do. 

Next up: directory and file tasks

Rake Fundamentals: rakefiles and tasks

Ok, what is Rake again? 

Rake is a collection of Ruby scripts that live deep within the bowels of your Ruby installation.  Since you made sure you checked the box that adds the Ruby stuff to your PATH variable as you stepped through the Ruby installer wizard, you now have easy access to your Ruby installation and all of the special scripts it contains right from the command line.  To make use of those Ruby scripts, you do a couple of things:

1) Create a rakefile, which is itself a ruby script containing named tasks you can run from the command line.

2) Run one or more of those tasks from the command line like so:

C:\sandbox\rakefun\> rake mytask

And what’s a rakefile? 

That command you just issued told rake to look for a file called c:\sandbox\rakefun\rakefile.rb and execute a task called ‘mytask’. You can call the rakefile something else if you want to, but if you name it “rakefile.rb”, you don’t have to include the “-f my_special_rakefile_name_because_im_a_unique_delicate_flower.rb” switch on the rake command line call – just less to type if you stick to the convention. It’s easy to create a rakefile: open up notepad, type in your tasks, save it as “rakefile.rb”, and you’re done. No compiling. No real need for fancy IDE’s, just any old text editor will do. I like Notepad2, and Komodo Edit is another nice free one with Ruby syntax highlighting. So, what exactly is in a rakefile?

tasks

The fundamental building block of a rakefile is the “task”.   Here’s a hello world rakefile that has one task in it called “:say_hi”.

require 'rake'

desc "My first rake task"
task :say_hi =>[] do
puts "Hello, dude!"
end

Ok, I fudged a little bit: when you open your editor to create your rakefile, you should actually require ‘rake’ at the beginning of the script.  This tells Ruby to find the rake.rb ruby script and pull it into your rakefile when Ruby parses it.  Ruby parses your rakefile any time you make a call to it with the rake keyword on the command line.

Rake provides a keyword called “task”.  The task keyword is actually just a call to a method named “task()” that lives somewhere in the rake.rb ruby script.  But through the magic of Ruby, the task() method call in your rakefile is formatted as you see above.  Takes a little getting used to.  Every time you call a task in your rakefile, Ruby parses the entire file, and all the task() method calls get executed.  But don’t worry – as each task() method call is executed, Ruby is just looking through the rakefile to gather up the name of each task, and the code that Ruby should run when that task name is called from the command line.  So if we had “:task1” and “:task2” in our rakefile, and we called “rake task1” from the command line, Ruby would know that we have both :task1 and :task2 available, but it would only run the code associated with task1.

There are two arguments being passed to task(), and if you’re not familiar with Ruby, they’re a little hard to distinguish:

1) The first argument is “:say_hi =>[]” – This is a Ruby hash object, which is just a collection of key/value pairs.  In this case the hash contains only one key/value pair.  The key is the symbol “:say_hi”, which by convention becomes the name of the task.  The value is “[]”, which is Ruby parlance for an empty array.  In more complex tasks, this array would contain the names of other tasks which should be prerequisites to this task.  Ruby is pretty forgiving, and rake takes advantage of that, so if you have no prerequisites you don’t actually have to include the “=>[]” part; and when you have one or more prerequisites, you can exclude the brackets and just supply a comma-separated list of task names.

2) The second argument is this highlighted block of code starting with “do” and ending with “end”:
task :say_hi =>[] do
  puts "Hello, dude!"
end
A code block is a special Ruby construct that you can think of like an anonymous method or a lambda.  When you call a rake task, rake first executes all the prerequisite tasks you listed in the array in the first argument, and then executes the code in the do…end code block.  If you only want to run prerequisite tasks without executing any other code, just don’t include a do…end code block.

The :say_hi task is called from the command line like this:

Get a list of tasks with rake -T

Tasks decorated with a “desc” are shown when you call “rake –T”, “rake –tasks”, or “rake –D” on the command line: 

Tasks with no “desc” are not shown in the task list, but can still be called from the command line, or used as prerequisites for other tasks.

Next time: Tasks with prerequisites.

Rake. Huh?

Lately I’ve been using rake at my office to build .NET solutions with Derick Bailey’s wonderful albacore gem.  But I’ve had lots of questions from others on my team about rake:
1) What the heck is rake?
The official line on rake: it is a build system written in Ruby inspired by the make build system from the world of c.  If that doesn’t really mean anything to you, just think of rake this way: it’s just a task runner.  A rake script is a convenient place to script out small tasks that you have to do every day – tasks that don’t take long individually, but that you’re performing multiple times per hour while you are writing code.  Automating those little rote tasks will save you time, and will cut down on the number of silly errors you get performing those tasks by hand.
2) A build system?  Why do I need some fancy build system when I can just click the “Build Solution” menu option in Visual Studio?
Well first of all, having a build system outside of Visual Studio doesn’t keep you from using the Visual Studio build tools.  A good build system should work alongside the IDE, not against it.  But more importantly, when you have a complex solution with one or more deployable apps, some redistributable assemblies, and multiple test projects, you’ve got a lot of stuff to manage; a good build system enables you to manage all that stuff with a measure of convenience that Visual Studio just isn’t designed to provide.
3)  Why rake instead of NAnt?  Are you just trying to be cool?
NAnt is a perfectly useful build system, and I’ve used it for years.  And with UppercuT, NAnt is certainly easier to use than it used to be.  But it’s still just XML.  Same thing goes for MSBuild.  Ultimately you’re talking about filling out an XML configuration file. If I want a custom target type that does something NAnt (or MSBuild) doesn’t do out of the box, I have to use a real programming language to implement the guts of the target; so now I’ve got yet another project to manage.  I’m sure there are people who love it and use it very successfully, but it’s just not my cup of tea.  Not only is it not my cup of tea: it makes me hate tea.  I’m a programmer, not a form-filler-outer.  I want a programming language, not an XML configuration file. 
4) We’re building .NET apps on Windows machines.  Why automate with rake instead of PowerShell?
I’ve seen some very compelling demos of psake, another make-inspired build system, written in PowerShell.  It’s certainly more to my liking than an XML-based build system.  But PowerShell is vendorscript.  I would rather spend time using a language that is a more transferrable skill, like Ruby. 
Those are the “why” questions I get about rake, and my lame attempts to legitimize my desire to mess around with Ruby at work.  I’m off and running on rake now, and the rest of the questions I get usually start with “how”.  I’m hoping to write a few posts answering those questions, and I’ll update this post with links to other posts in the series as I go.

Installing Ruby and Rake

Before we talk about rake, rakefiles, and tasks, we’ll need a couple of things:
1) Ruby – rake is really just a set of Ruby scripts. And what do you need to run Ruby scripts? Ruby! The easiest way to get up and running with Ruby on a Windows box download the latest Ruby One-click Installer (version 1.9.2 as of this blog post), and just run it. Important: Make sure you check the box that adds the Ruby stuff to your PATH variable as you step through the installer wizard. 

Once you’ve gotten through the wizard, open up a command prompt and type “ruby –v” to see the version number.  You should see something like this:

If you see a message saying something along the lines of “ruby is not a recognized command”, just reboot.  Windows just needs to deal with the PATH variable.
2) Rake – now that you’ve got Ruby installed, open up a command prompt and type

gem install rake

and hit enter.  The “gem” keyword hooks you into Ruby’s package management system – you’re telling Ruby to grab the “rake” package from the RubyGems repository, and install it in your local Ruby installation.  A bunch of stuff that won’t make a lot of sense will scroll by in the command window.  But at the end of it, once you get back to a command prompt, you’ll be ready to run rake scripts. 
3) A text editor – ok, three things.  Notepad works just fine.  If you want something fancier, there’s Komodo Edit, Notepad2, Notepad++ , etc.  And if you wanna pay for a nice Ruby editor, JetBrains (the people behind the wonderful ReSharper) have a good one called RubyMine.  I like RubyMine a lot, and I got lucky and picked up a copy for $29.  Every once in a while they put it on sale, you just gotta check the website for it.

Next up: rakefiles and tasks