Pearware Blog

I’m writing an application that downloads access logs from our production servers and runs the AWStats package against them to create the statistics web pages. This process is setup as a Rake task that uses the Net::SFTP library used by Capistrano, written by Jamis Buck. There is also a front-end Rails application to manage each of the applications to be retrieved. Everything was working great until I tried to grab a 550MB file from one of our servers. Net::SFTP chocked as it ran out of memory.

It turns out that the command:

    sftp.get_file log_file, local_file

ends up putting the whole file into memory, which is fine for small files, but not the large one that I was trying to download. Luckily it wasn’t too bad to refactor my class. Here’s the new code to achieve the same effect as the above sftp.get_file command.

        stat = sftp.stat( log_file )
        offset = 0
        file_length = stat.size
        length = 64 * 1024 * 1024
        File.open(local_file, File::CREAT|File::TRUNC|File::RDWR, 0644) do |f|
          while (offset < file_length)
            sftp.open_handle(log_file) do |handle|
              data = sftp.read(handle, :length => length, :offset => offset)
              f.write(data)
              offset += data.length
            end
          end
        end

This downloads the file in 64MB increments, using only that much memory at any time.

RadiantOnRails is a Radiant extension I created to allow a Rails application to co-exist with Radiant, giving the developer the best of both (dynamic and static) worlds. You can now visit the new project page on RubyForge.

This extension will be a major piece for the website I’m currently working on, RealIdaho.com. Most of the pages for the site just display static content about cities, but there are other portions of the site that will be fully-dynamic Rails pages. This extension allows me to combine both portions of the site into one application so we can develop the Rails pages for displaying the data-driven pages and still leverage the wonderful user interface created by the Radiant team. This will allow the realtor to make content changes without me having to make code changes and redeploy.

RadiantOnRails currently allows Radiant snippets to be inserted into Rails views and the next step is to allow the Rails views to use the Radiant layouts so that the look ‘n’ feel of the site is consistent, while keeping the views DRY. I’ll also be working with Loren Johnson to make Radiant available as a plugin which will make integrating Radiant with Rails even easier.

I just switched some Ruby on Rails apps I’m running from Pound to Nginx based on the results from some articles I’ve read online. The two biggest advantages of Nginx are 1. It’s raw performance, and 2. It can serve up static files, which is great for running Capistrano’s disable_web command to show a maintenance page when redeploying an application.

At work, we’re working on a standard Ruby on Rails setup and are currently investigating two options. The first is a Mongrel cluster running behind Nginx, as I described above. The second option is fronting the Mongrel cluster with Lighttpd. Since the 1.4.x series of Lighttpd is known to have some issues with it’s mod_proxy implementation, we would use Pen until a stable 1.5 version is released.

I’ll post another article once we have finished our evaluation and chosen which option we’re going to deploy at work. Stay tuned…

I wrote an article awhile back about using the Ruby/LDAP library to handle LDAP authentication in Ruby on Rails. I just finished swapping out the LDAP client library in that application from Ruby/LDAP to ruby-net-ldap. The problems with Ruby/LDAP are that it isn’t a GEM, so installation is a bit more difficult, and it relies on a common LDAP library, like OpenLDAP, to already be installed on the system. The ruby-net-ldap library is written in pure Ruby, so no other library needs to be installed on the system.

Here is the new code that performs the authentication:

require "net/ldap" 

class User < ActiveRecord::Base
  def self.authenticate(login, password, host, port)
    if login.to_s.length > 0 and password.to_s.length > 0
      ldap = Net::LDAP.new
      ldap.host = host
      ldap.port = port
      ldap.auth = "cn=#{login},cn=users,o=xyz...", password
      if ldap.bin
        return find(:first, :conditions => ['username=?', login])
      else
        return false
      end
    end
  end
end

The site I am working on, RealIdaho.com, is hosted by site5. There are some forms that, when filled out by the user, send confirmation emails to the user, to us internally, and to the realtor’s cellphone. We started getting some reports that the confirmation page wouldn’t come up, and users were resubmitting the form many times. I discovered a few things while investigating that may be useful to others.

As part of my Application migration project, I need to pre-populate the new database with zip code data. The Rails Recipes book (very useful) has a nice recipe on extracting fixtures from live data. So, I’ve created a zips.yml file that contains all the zip code data that I can insert into the new database. However, when I try to load the fixture using this very cool rake task from Technoweenie, the YAML library throws the following exception: SystemStackError: stack level too deep.

I’m working on a project for a client to convert a website from PHP to Ruby on Rails. The database is also changing from Microsoft SQL Server to MySQL, along with several schema redesigns.

One thing I wanted to do to our new db, was change a foreign key column from NOT NULL DEFAULT 0 to allow nulls and remove the default. Currently, the organizations table has a record with ID = 0 to signify a blank organization. Users that do not belong to an organization, have a foreign key to that Zero-record. Well, a better way to represent that is for the foreign key column in users to simply be NULL and ditch the Zero-record in the organizations table.

I have created a Migration class to handle the schema change. However, it looks like I have to drop to SQL as there is not Migration way of doing this. Here’s what it looks like:

    execute 'ALTER TABLE users ALTER organization_id DROP DEFAULT'

I wanted to be able to handle this in the Migration-syntax, but I don’t think that’s possible. It would be nice to do something like:

    change_column :users, :organization_id, :integer, :default => :null, :null => true

I just redeployed a rails site for a client of mine, Real Idaho, using Capistrano and it didn’t work properly. I was able to resolve the issue, and here are the details in case this ever happens to you…

$ rake deploy

...
transaction: commit
  * executing task restart
  * executing "/home/*****/apps/*****/current/script/process/reaper -d 'dispatch.fcgi'" 
    servers: ["216.118.83.207"]
    [216.118.83.207] executing command
 ** [out :: 216.118.83.207] bash: /home/*****/apps/*****/current/script/process/reaper: Permission denied
    command finished
rake aborted!
command "/home/******/apps/******/current/script/process/reaper -d 'dispatch.fcgi'" failed on 216.***.***.***

-rw-r--r--

$ svn propset svn:executable

$ svn propset svn:executable "*" `find script -type f | grep -v '.svn'` public/dispatch.*

$ svn commit