Pearware Blog

I’m writing an application that downloads access logs from our production servers and runs the AWStats package against them to create the statistics web pages. This process is setup as a Rake task that uses the Net::SFTP library used by Capistrano, written by Jamis Buck. There is also a front-end Rails application to manage each of the applications to be retrieved. Everything was working great until I tried to grab a 550MB file from one of our servers. Net::SFTP chocked as it ran out of memory.

It turns out that the command:

    sftp.get_file log_file, local_file

ends up putting the whole file into memory, which is fine for small files, but not the large one that I was trying to download. Luckily it wasn’t too bad to refactor my class. Here’s the new code to achieve the same effect as the above sftp.get_file command.

        stat = sftp.stat( log_file )
        offset = 0
        file_length = stat.size
        length = 64 * 1024 * 1024
        File.open(local_file, File::CREAT|File::TRUNC|File::RDWR, 0644) do |f|
          while (offset < file_length)
            sftp.open_handle(log_file) do |handle|
              data = sftp.read(handle, :length => length, :offset => offset)
              f.write(data)
              offset += data.length
            end
          end
        end

This downloads the file in 64MB increments, using only that much memory at any time.

I just switched some Ruby on Rails apps I’m running from Pound to Nginx based on the results from some articles I’ve read online. The two biggest advantages of Nginx are 1. It’s raw performance, and 2. It can serve up static files, which is great for running Capistrano’s disable_web command to show a maintenance page when redeploying an application.

At work, we’re working on a standard Ruby on Rails setup and are currently investigating two options. The first is a Mongrel cluster running behind Nginx, as I described above. The second option is fronting the Mongrel cluster with Lighttpd. Since the 1.4.x series of Lighttpd is known to have some issues with it’s mod_proxy implementation, we would use Pen until a stable 1.5 version is released.

I’ll post another article once we have finished our evaluation and chosen which option we’re going to deploy at work. Stay tuned…

Now you can run your website through a handy-dandy Web 2.0 validator, to see if it really is Web 2.0. This site was put together by 30 Second Rule, a website design and development company in Phoenix that is active in the Ruby community. My blog’s score should be upped by writing this article! :)

Go to the Web 2.0 Validator