Latest Entries »

ActiveRecord is great when you need to quickly access and manipulate a few rows within a database. Loading records into a database is just as easy… Seed files and custom rake tasks make inserting records a breeze. But what happens when you need to import lots of rows. To clarify, when I say lots of rows, I’m talking about millions of rows from a delimited text file.

So many rows, so little time

My first instinct was to read in the text file and do a Model.create() on each row. This took a long, long time. (I actually gave up on it)

  # This is going to be awhile!!!
  CSV.parse(File.read('largefile.txt'), headers: true) do |row|
    SomeModel.create(row)
  end

Next, I tried batching the rows to an array in an effort limit the number of database calls. I then imported each batch using the activerecord-import gem. Batching helps if your recordset is a couple thousand rows, but doesn’t scale efficiently to a million+ rows.

  # Faster, but still not quite there...
  batch = []
  batch_size = 1000
  
  CSV.parse(File.read('largefile.txt'), headers: true) do |row|
    batch << SomeModel.new(row)
    if batch.size >= batch_size
      SomeModel.import batch
      batch = []
    end
  end
  SomeModel.import batch

COPY to the rescue

In order to get in the millions, I needed to take Rails out of the equation as much as possible. This meant using an ActiveRecord raw_connection and the COPY command from postresql.

Due to permission issues, you most likely will not be able to use the filename option of the COPY command. This actually turns out to not be a big deal since you can still use STDIN to pass data to the command. Confused? Let’s look at the example

  # Setup raw connection
  conn = ActiveRecord::Base.connection
  rc = conn.raw_connection
  rc.exec("COPY large_table (col1, col2, etc) FROM STDIN WITH CSV")

  file = File.open('largefile.txt', 'r')
  while !file.eof?
    # Add row to copy data
    rc.put_copy_data(file.readline)
  end

  # We are done adding copy data
  rc.put_copy_end
  
  # Display any error messages
  while res = rc.get_result
    if e_message = res.error_message
      p e_message
    end
  end

It’s actually pretty simple. I execute the COPY command on our large_table with data from STDIN. Then I read in the file and put the data into STDIN for each line. When the file is finished, I issue an end copy instruction.

Conclusion

With the copy technique above, I was able to import over 2.4 million rows in less than 4 min. Not too shabby. I would be interested in hearing what strategies you all have used for large imports.

Cutting circles in your HTML with CSS3

Masking images on the web has typically been a job reserved for Adobe Flash or some equivalent photo manipulation tool as a static image. While Flash was prevalent on the web, it became really easy to mask an image with a vector shape. Now that Flash content has been largely replaced with a combination of HTML, CSS3 and Javascript, it is time to dig back into the toolbox and find new solutions for old problems.

Screen-Shot-2013-10-08-at-2.47.29-PM

Masking Techniques

Making a mask for an image requires some sort of shape to use as the “mask”. This will determine what portions of your image will “show through” or be excluded. Typically a vector shape is used to make a mask. On the web, SVG (Scaleable Vector Graphics) is used to create these vector graphics. All the different ways to mask an image are outside of the scope of this article, however one technique that is illustrated in the next section does not require SVG at all. For more detailed information on the different ways to mask content with SVG check out this article

Masking with Radial Gradients

After a slight detour, it is time to get back to the task at hand. Cutting a transparent circle in a DOM element. As we have seen above, we can cut out all sorts of shapes using SVG. By layering a radial gradient over the top of a particular element we can achieve a similar effect using a circle. Check out the fiddle below for more details.

http://jsfiddle.net/HAKuS/1/

To create the illusion of a cutout circle we use the color-stops property of radial gradients. In the background-image style property the first color-stop is a 100px transparent color. This creates our “hole” in the white overlay. The next color-stop creates a 5px inner shadow of semi-transparent black. The final color-stop creates the background color for the rest of the DOM element. And volia, there is a masked hole in your DOM element with no SVG!

Screen-Shot-2013-10-10-at-9.17.01-AM

iOS 7, PhoneGap and the Status Bar

For some folks that just tried out their PhoneGap apps for the first time on iOS 7, you may have seen something like this…

ios7 Status bar

If you are like me, now is the time where you google for awhile to find out how to get rid of something that you have already figured out with iOS 6 :( Thanks for the update Apple.

Removing the Status Bar

It turns out that there is yet another new plist setting that will fix this issue. If you are using PhoneGap make sure to update Cordova to the latest release. (3.1)

  npm install -g cordova
  cd my_project_directory
  cordova platform update ios

Next, open your plist file (Resources/{ProjectName}-Info.plist) and add the key “View controller-based status bar appearance” and set it to “NO”. Also, for good measure make sure that “Status bar is initially hidden” is set to “YES”

Plist file