Testing Multiple Browsers with Selenium and Cucumber

Testing web applications can be a difficult task with the many complexities involved with current web technologies. At one time it was possible to test web applications without needing to rely on javascript at all. For this you could leverage java's htmlunit or even httpunit to write unit tests based on the source of a webpage. Javascript support in htmlunit was not the least bit usable on the web applications that I tested at my previous job, so it was completely off the table for the current project I'm responsible for testing. This application is fully ajax based, and thus we've had to use something with better support for javascript.
Our solution originally was to use a combination of Watir and FireWatir so that we could test both IE and Firefox. I had some issues getting it to work correctly though, the api between the two is not 100% consistent yet (last I checked anyways).

I then realized that the smart folks working on the Selenium project had added a new tool called Selenium Remote Control to the mix. With the Selenium remote control you no longer have the prerequisite of having the selenium javascript installed on the server application. I wanted our testing environment to not rely on server side extras being installed unless absolutely necessary, so the obvious solution was to leverage Selenium Remote instead.

Selenium remote creates a proxy server which the client connects to, and uses to intercept javascript calls made by the client browser. This provides a workaround for javascript's same origin policy.

To use selenium you must start up the selenium remote control server first.
If you have installed the Selenium rubygem you can do this by simply typing

$ selenium

in a shell window

Install the gem using the following: (note the case sensitive Selenium)

$ gem install Selenium

Note that the last time I checked the selenium jar file bundled with the Selenium gem is crap and will not work correctly. Instead use the latest build from the selenium site.

Download the latest jar here:
http://seleniumhq.org/download/
(I actually downloaded a more recent nightly release and it seemed to be more stable than the one listed on the home page -- current home page lists 1.0 beta2)
Anyways, once you've done this you need to take that jar file and copy it to your where your current Selenium gem has the selenium-rc jar file. On my system it was in this directory:
C:\ruby\lib\ruby\gems\1.8\gems\Selenium-1.1.14\lib\selenium\openqa
and the filename is selenium-server.jar.txt
Supposedly there was an issue with a file name ending in .jar in ruby so they gave the file a .txt extension (who knows?). Anyways, you can make a backup of that one and copy the newest version there and give it the same name as the old one. From there you can just type selenium at the command prompt and the server will start up for you. One option I usually run with is -singleWindow when I'm trying to see what is going on (its very hard to see when the windows are real small).

Anyways, on my system I also have the selenium-server.jar file installed at:
c:\selenium-rc\selenium-server.jar
which is where I run it from within my custom rake tasks.

If you use the rakefile I've posted below to run your tests, you won't need to bother starting selenium first. In fact, if you start selenium before you run the rake tasks, the selenium server you start up will be immediately shut down by the rake tasks. The rake tasks starts up its own server so that you don't have to remember to start up selenium first. Its nice to be able to run cucumber tests one a time when you are developing though, and for those times you'll want to start up selenium without messing with rake.

Here is the rakefile i'm currently using which will run your cucumber tests through each browser (if the test uses a browser).

require 'cucumber/rake/task'
require 'selenium/rake/tasks'
#require 'rubygems' #add this if you don't have rubygems added automatically


# Change these to specify which browsers on which 
# platforms you want to cover
@win_browsers = ["*iexploreproxy","*firefox"]
@osx_browsers = ["*safari", "*firefox"]
@linux_browsers = ["*firefox"]

# Supported platforms
@platforms = {"win"=>"Windows", "osx"=>"OS X", "linux"=>"Linux"}

if RUBY_PLATFORM.downcase.include?("darwin")
  @platform = 'osx'
elsif RUBY_PLATFORM.downcase.include?("mswin")
  @platform = 'win'
elsif RUBY_PLATFORM.downcase.include?("linux")
  @platform = 'linux'
end

if ENV["SELENIUM_RC_JAR"]
  # User override
  SELENIUM_RC_JAR = ENV["SELENIUM_RC_JAR"] 
elsif File.exists?(File.dirname(__FILE__) + \
 "/vendor/selenium-server/selenium-server.jar")
  # Bundled version included with project
  SELENIUM_RC_JAR = File.dirname(__FILE__) + \
"/vendor/selenium-server/selenium-server.jar"
else
  SELENIUM_RC_JAR  = case @platform
  when 'osx' then "/var/lib/selenium/selenium-server.jar"
  when 'linux' then "/var/lib/selenium/selenium-server.jar"
  when 'win' then "C:/selenium-rc/selenium-server.jar"
  else raise 'Unsupported Operating System'
  end 
end
if not File.exists?(SELENIUM_RC_JAR)
  raise "File Not Found -- SELENIUM_RC_JAR"
end

desc "Invoke behaviours on all browsers on specified platform"
task :test do
  @browsers = case @platform
  when 'win' then @win_browsers
  when 'osx' then @osx_browsers
  when 'linux' then @linux_browsers
  end

  puts "Running tests for the #{@platforms[@platform]} platform"
  
  Rake::Task[:"selenium:rc:stop"].execute [] rescue nil
  begin
     Rake::Task[:"selenium:rc:start"].execute []
    @browsers.each do |browser|
      puts "executing on browser" + browser.to_s + "\n"
      @current_browser = browser
      ENV['SELENIUM_BROWSER'] = browser
      year,month,day = Date.today.strftime("%Y,%m,%d").split(",")
      dir = "reports/#{year}/#{month}"
      FileUtils::mkdir_p(dir)
      filename = "#{dir}/#{day}-#{browser.delete "*"}.html"
      ENV['CUCUMBER_OPTS'] = "--format progress --format html \
--out=#{filename} features"
      begin
        Rake::Task[ :run_browser_tests ].execute() 
      rescue RuntimeError
        puts "Error while running task"
      end
    end    
  ensure
    Rake::Task[:"selenium:rc:stop"].execute []
  end
end


Cucumber::Rake::Task.new(:'run_browser_tests') do |t| 
  #t.cucumber_opts = "--format progress --format html \
--out=#{filename} features"
end

Selenium::Rake::RemoteControlStartTask.new do |rc|
  rc.port = 4444
  rc.timeout_in_seconds = 30
  rc.background = true
  rc.wait_until_up_and_running = true
  rc.jar_file = SELENIUM_RC_JAR
  rc.additional_args << "-singleWindow"
end
  
Selenium::Rake::RemoteControlStopTask.new do |rc|
  rc.host = "localhost"
  rc.port = 4444
  rc.timeout_in_seconds = 30
  rc.wait_until_stopped = true
end

desc "Restart Selenium Remote Control"
task :'selenium:rc:restart' do
  Rake::Task[:"selenium:rc:stop"].execute [] rescue nil
  Rake::Task[:"selenium:rc:start"].execute []
end


task :default => [:test]

Then in your env.rb file do this:

Before do
  profile = ENV['PROFILE'] || 'default'
	c = YAML.load_file('selenium.yml')[profile]
  browser_type = ENV['SELENIUM_BROWSER'] || "*firefox"
  
  @browser = Selenium::Client::Driver.new(c['server_host'], \
c['server_port'], browser_type, c['root_url'], c['timeout'])
  @browser.start  
end

Which means you'll need a selenium.yml file like this (mine is in the same directory as the rakefile):

default: 
  server_host: localhost  
  root_url: http://server-ip-address
  server_port: 4444
  timeout: 28

And that should be it. Now just put your feature tests in a features directory and then put all your step definitions in files in a step_definitions directory..
Your tree should end up looking like this:

path/to/project/feature_tests/Rakefile.rb
path/to/project/feature_tests/selenium.yml
path/to/project/feature_tests/features/my_feature.rb
path/to/project/feature_tests/features/step_definitions/my_feature_steps.rb
path/to/project/feature_tests/features/support/env.rb
path/to/project/feature_tests/features/lib/SomeClassToAbstractFeature.rb

Thats how I have my tests structured and it seems to be working out good.

Thanks

Thanks for taking the time to post this. Is there any way you might be able to comment the code so we have an understanding of what is going on? I have a slightly different set-up and it would be really helpful to see your train of thought as you developed the script...

Cheers