Reworking Net::SFTP to handle large file downloads

Tue, 26 Jun 2007 14:13:00 -0500

I’m writing an application that downloads access logs from our production servers and runs the AWStats package against them to create the statistics web pages. This process is setup as a Rake task that uses the Net::SFTP library used by Capistrano, written by Jamis Buck. There is also a front-end Rails application to manage each of the applications to be retrieved. Everything was working great until I tried to grab a 550MB file from one of our servers. Net::SFTP chocked as it ran out of memory.

It turns out that the command:

    sftp.get_file log_file, local_file

ends up putting the whole file into memory, which is fine for small files, but not the large one that I was trying to download. Luckily it wasn’t too bad to refactor my class. Here’s the new code to achieve the same effect as the above sftp.get_file command.

        stat = sftp.stat( log_file )
        offset = 0
        file_length = stat.size
        length = 64 * 1024 * 1024
        File.open(local_file, File::CREAT|File::TRUNC|File::RDWR, 0644) do |f|
          while (offset &lt; file_length)
            sftp.open_handle(log_file) do |handle|
              data = sftp.read(handle, :length =&gt; length, :offset =&gt; offset)
              f.write(data)
              offset += data.length
            end
          end
        end

This downloads the file in 64MB increments, using only that much memory at any time.

Pearware Blog : Archives for June 2007

Reworking Net::SFTP to handle large file downloads