How you zip larger files in ruby

Question

Easy way that compress files in ruby is using zip lib.

Zip::File.open('./compress.zip', create: true) do |zip|
  zip.get_output_stream("./stuff_to_ziped.pdf") do |f|
    f.puts File.read("./stuff_to_zip.pdf")
   end
end

When you need to zip larger files or when you need to perform any task that requires opening many files, good practice is to avoid loading all files into memory. To achieve this, you can use stream operations available in IO classes.

Let's look at the code above again but with a thread that monitory the memory for us.

Thread.new do
  loop do
    print "\r memory: %d MB" % "#{`ps -o rss= -p #{Process.pid}`.to_i/1024}\t"
  end
end

Zip::File.open('./compress.zip', create: true) do |zip|
  zip.get_output_stream("./stuff_to_ziped.pdf") do |f|
    f.puts File.read("./stuff_to_zip.pdf")
   end
end

Now you can see how much memory the program consumis until done. The method File.read load all bytes of file in memory. Lets see other implementation.

Thread.new do
  loop do
    print "\r memory: %d MB" % "#{`ps -o rss= -p #{Process.pid}`.to_i/1024}\t"
  end
end


Zip::File.open('./compactados.zip', create: true) do |zip|
	10.times do |i|
		zip.get_output_stream("./copy_#{i}.pdf") do |f| 
			input = File.new("./copy_#{i}.pdf")
			while !input.eof?
				f.puts(input.gets)
			end
		end
	end
end