Laurel Fan - Home

Mt Baker, Easton Glacier, June 14-15 2008

2008-06-18T00:36:00Z

The trip started on Saturday, plodding up from Schreiber’s Meadow to a campsite on the Easton Glacier. All 20 or so of us straggled in, started digging, and we soon had our multicolored tent village. The sun was warm, the wind was light, and the mood was relaxed as we lazily set up tents, ate freeze dried dinners, and lounged around the rocks. We tried to sleep under the still hot afternoon sun to prepare for getting up at midnight the “next day”.

The first hours of the ascent were like nothing else. The glow of headlamps strung out along the route, neighboring peaks floating above pink clouds at dawn, seeing the moon set and sun rise. Though I was literally tied to four other people, my task as second on the rope was simple and constrained: keep walking, don’t fall, watch the rope in front of me. In the darkness and silence, separated by 10 meters or so of rope from anyone else, I knew nothing of their private struggles with cold, nausea, dying batteries, uncertainty, impatience, and they knew nothing of mine. I didn’t know if anyone else saw the same thin clouds drifting in front of the red setting moon, or the same drape of icicles off the edge of a crevasse.

(more of my pictures)

The climb was uneventful (which was a good thing – nobody wants an event). We reached the summit 5 or 6 hours after we started, spent some time taking pictures, and descended to camp before noon. All that was left was a mildly unpleasant walk through soupy slush and on melting gravel roads and we were in the car heading back to sea level and back to the city. We had glimpses of the mountain all the way home, but we were already in a different world of traffic, laundry, work, and hot showers.

(other pictures from Kristin, John, Daniel, Cara, and Ben)

Official trip report on the WAC site

0 Surfboards

2008-05-22T18:57:00Z

The web checkin form for Hawaiian Airlines asked me how many surfboards I will be bringing:

Unfortunately I had to click 0.

Bugs and Features

2008-04-21T17:04:00Z

Too bad this only works in unit tests:

Playing with graphs

2008-04-08T00:33:00Z

I’ve always liked playing with graphs and representing data, so when we we decided to implement a version of Buster’s Morale-O-Meter for the new 43 Things profile page

I started with gruff, but I wanted something a little more stylized and less techy looking, so I took out all of the axes and hacked it to use numbers as the points. I kind of like how it turned out:

(This isn’t the final version, and the feature isn’t live on 43 Things yet, but it will be “soon”...)

Refactoring SQL Strings

2008-02-28T01:23:00Z

Our apps have a lot of custom sql. A lot of the time it’s easier just to write exactly the sql we want instead of messing around with ActiveRecord. But any time two programming languages run in to each other things can get out of hand, so there are lots of opportunities for refactoring.

Before:

    NUM_STARTED_OR_COMPLETED_ITEMS_DURING_RANGE = <<-SQL.prettify_sql
        SELECT count(*) FROM listed_items 
        WHERE active = 1 
        AND person_id = ?
        AND ( (completed_on IS NULL 
        AND complete = 0 
        AND give_up = 0 
        AND posted_date > ? 
        AND posted_date < ?)
        OR (completed_on > FROM_UNIXTIME(?) 
        AND completed_on < FROM_UNIXTIME(?) ) )
    SQL

    STARTED_OR_COMPLETED_ITEMS_DURING_RANGE = <<-SQL.prettify_sql
         SELECT * FROM listed_items
         WHERE active = 1
         AND person_id = ?
         AND ( (completed_on IS NULL
         AND complete = 0
         AND give_up = 0 
         AND posted_date > ?
         AND posted_date < ?)
         OR (completed_on > FROM_UNIXTIME(?)
         AND completed_on < FROM_UNIXTIME(?) ) )
         ORDER BY updated_date DESC
         LIMIT 0, 100
    SQL

    def num_completed_or_started_items_during_range(start_date, stop_date)
        ListedItem.count_by_sql [NUM_STARTED_OR_COMPLETED_ITEMS_DURING_RANGE, self.id, start_date, stop_date, start_date, stop_date]
    end

    def completed_or_started_items_during_range(start_date, stop_date, offset=0, limit=20)
        items = ListedItem.find_by_sql [STARTED_OR_COMPLETED_ITEMS_DURING_RANGE, self.id, start_date, stop_date, start_date, stop_date]    
        item_to_sort = Hash.new
        items.each do |item| 
            item_to_sort[item.id] = item.completed_on ? Time.at(item.completed_on).to_i : item.posted_date
        end
        items.sort{|a,b| item_to_sort[a.id] <=> item_to_sort[b.id]}.reverse[offset .. (offset + limit - 1)]
    end

What’s wrong with this?

First of all, there are no tests. So first step is to write some. If I were just refactoring, I’d write tests that pass, but the reason I’m in this code is that there’s a bug (long story short, the bug is caused by the fact that a listed_item can have a non-null completed_on but have completed = 0). So I write a failing one that catches the bug too.

The STARTED_OR_COMPLETED_ITEMS_DURING_RANGE and NUM_STARTED_OR_COMPLETED_ITEMS_DURING_RANGE string constants are almost identical. So when I start changing the strings I realize that I’m going to mess things up if I leave them duplicated like that (even if I do the right thing, change both strings, it’ll be wrong for the moment where I’m moving the cursor 10 lines down), so I stop and factor out the where clause.

The next thing is that I don’t like the way the parameters for the SQL string work. Parameters are repeated: if you look at completed_or_started_items_during_range, the start_date and stop_date parameters are both in there twice. Also, it’s annoying and error prone to keep paging between the string and the methods (I pasted the excerpt together, so it doesn’t look like it here, but this file is arranged so all of the constants are together at the top of the class definition, so there are several pages separating these constants and methods) to remember what the parameters mean. Rails lets me use named bind variables so I can give them names like a real programming language.

Ok, now that I have > :start and < :end instead of > ? and < ?, I’m more comfortable changing that to use the SQL between operator, and it looks a lot nicer (and one line shorter). I realize that between doesn’t quite have the semantics of < and >, but that’s fine with me in this case.

It’s a good thing it’s shorter because I’m going to take a break from refactoring and fix the bug, which involves adding a line. Ok, finally the tests pass.

We’re doing sorting and limiting with ruby. Sometimes it’s faster to do the sorting in ruby because the database is doing something dumb, so I’ll have to keep track of the timings. And sometimes it looks nicer in Ruby if it’s complicated code. But in this case the Ruby code is 5 lines of really dense code (use of the ternary operator and the compact {} form of blocks makes it look like you’re trying to hide something). After staring at it for a while, it’s just doing a coalesce and a normal limit with offset, so doing it in SQL looks nicer to me.

After:

    WHERE_STARTED_OR_COMPLETED_ITEMS_DURING_RANGE = <<-SQL
        WHERE active = 1 
        AND person_id = :person_id
        AND ( (completed_on IS NULL 
               AND complete = 0 
               AND give_up = 0 
               AND posted_date BETWEEN :start AND :end)
           OR (completed_on BETWEEN FROM_UNIXTIME(:start) AND FROM_UNIXTIME(:end)
               AND complete = 1 ) )
    SQL

    NUM_STARTED_OR_COMPLETED_ITEMS_DURING_RANGE = <<-SQL.prettify_sql
        SELECT count(*) FROM listed_items
        #{WHERE_STARTED_OR_COMPLETED_ITEMS_DURING_RANGE}
    SQL

    STARTED_OR_COMPLETED_ITEMS_DURING_RANGE = <<-SQL.prettify_sql
        SELECT * FROM listed_items
        #{WHERE_STARTED_OR_COMPLETED_ITEMS_DURING_RANGE}
        ORDER BY COALESCE(completed_on, FROM_UNIXTIME(posted_date)) DESC
        LIMIT :offset, :limit
    SQL

    def num_completed_or_started_items_during_range(start_date, stop_date)
        ListedItem.count_by_sql [NUM_STARTED_OR_COMPLETED_ITEMS_DURING_RANGE,
                                 {:person_id => self.id,
                                     :start => start_date,
                                     :end => stop_date}]
    end

    def completed_or_started_items_during_range(start_date, stop_date, offset=0, limit=20)
        ListedItem.find_by_sql [STARTED_OR_COMPLETED_ITEMS_DURING_RANGE,
                                {:person_id => self.id,
                                    :start => start_date,
                                    :end => stop_date,
                                    :offset => offset,
                                    :limit => limit}]
    end

Getting the weekday name and month name from a Time

2008-02-27T01:50:00Z

Pop quiz: how do you get the month (as a name, not a number) of a Time in Ruby (without looking at the Time#strftime documentation)?

All I remembered was that it was some meaningless looking format string. Here’s my first guess:

12	irb(main):001:0> Time.now.strftime("%a")=> "Tue"

Oops, it’s actually:

12	irb(main):002:0> Time.now.strftime("%b")=> "Feb"

That doesn’t look very ruby-like. Aren’t % format strings for C programmers (whoever wrote the code I’m currently refactoring is probably a recovering C programmer. They even use sprintf…)?

How about:

12	irb(main):003:0> Time.now.short_month_name=> "Feb"

I’m sure others have done this already, but I couldn’t find it so I made my own Rails plugin, and stuck it here: http://svn.laurelfan.com/decorated_time/, mostly to see if I could set up an svn repository on dreamhost. The stuff that actually does the work is a total of about 4 lines of code since Ruby nicely lets me reopen the Time class at will.

(speaking of the day of week, I noticed that 1.9 added monday?, tuesday?, etc methods)

Fun with Emacs

2008-02-19T19:36:00Z

One good part of doing a big refactoring is that I get to have fun with emacs.

I found a great step by step tutorial with all the details. But basically, first you get in to dired (directory editing), then mark the files you want to search through, and then do dired-do-query-replace-regexp.

So for example, to replace @params with params in all controllers:

M-x find-dired
- Run find in directory: app/controllers
- Run find (with args): -name '*.rb'
%m
- Mark files (regexp): .
M-x dired-do-query-replace-regexp
- Query replace in marked files (regexp): @params
- Query replace @params by: params

Dired is fun stuff like emacs’s other “everything is a buffer!!” things (just like unix’s “everything is a file!!”). I’m not quite emacs-hacker enough to use it as a shell for extended periods of time though.

Reraising Exceptions in Ruby

2008-02-14T22:04:00Z

Don’t do this:

123	rescue Exception => e # other stuff raise "#{e.class}: #{e.message}"

Do this:

123	rescue Exception => e # other stuff raise

raise without any arguments will reraise the current exception, complete with class, message, and stack trace.

See also Programming Ruby on exceptions.

Stupid SEO Tricks

2008-01-17T23:36:00Z

We found a site that looked like it was stealing our content. Then we go there, and it’s exactly the same site! What are they doing? Are they crawling the site and sucking up all of our pages? Proxying us?

But wait:

 > dig d****list.com
d****list.com.          1736    IN      A       209.61.175.237

That IP address looks familiar!

 > dig 43things.com  
43things.com.           86400   IN      A       209.61.175.237

Why would someone do that? Is it some SEO trick to steal our google rank for their domain?

Josh noticed that the domain was registered to someone in China (we don’t think this is a good person trying to get us around the Great Firewall—a test tool we tried showed that 43things.com isn’t blocked). So we decided to cause them some trouble:

    RewriteCond %{HTTP_HOST} .*d****list.com$
    RewriteRule ^/(.*) http://en.wikipedia.org/wiki/Tiananmen_Square_protests_of_1989 [R]

Ruby Memory Usage

2008-01-15T19:50:00Z

Tracking down a memory leak? Here’s a way to find the memory usage of the current process on Ruby:

memory_usage = `ps -o rss= -p #{Process.pid}`.to_i # in kilobytes

-o rss= asks ps to print only the RSS (Resident Set Size, or physical memory used). You could also use vsz/vsize (virtual memory). The hanging = sign sets the header text to a blank string so you don’t have to filter out the header line.

-p #{Process.pid} limits the ps to only show the current process.

The backticks are kind of hacky, but this cuts down on the piping and grepping. It works on all of the unixes I’ve tried (Linux, FreeBSD, OSX), but of course ps is notoriously nonstandardized.

MySQL Performance Tuning Links

2008-01-10T22:29:00Z

Here are some links to things mentioned at the MySQL Performance Tuning class that I’ve been attending this past week.

Day 1 Morning: MySQL website, basics, upsell for more training classes and certifications

Documentation pages for all options and variables and the system variables page (which indicates which ones are dynamic).

Day 1 Afternoon: architecture overview

Documentation for the innodb transaction model
- multiple versions of uncommitted rows are stored on disk
SHOW TABLE STATUS shows statistics like table size, number of rows (it’s actually an approximation)

Day 2 Morning: data types, benchmarking, logs, admin tools, indexes

slow query log and mysqldumpslow (which summarizes the information in the slow query log)
PROCEDURE ANALYSE, which tries to figure out the optimal data type for the columns of a table
Benchmarking tools: mysqlslap, which comes with 5.1 and might not work with 4.1 and Super Smack
mysqlreport, a more human readable version of SHOW STATUS
mytop, a top-view of SHOW PROCESSLIST
innotop monitors all kinds of innodb stuff
Maatkit contains essential command-line tools for MySQL, such as table checksums, a query profiler, and a visual EXPLAIN tool. It provides missing features such as checking whether slaves have the same data as the master.

Day 2 Afternoon: statement tuning, query cache

EXPLAIN EXTENDED and then SHOW WARNINGS will show you the actual SQL being executed (which can be different if the optimizer wants to rewrite it).
Query cache documentation

Day 3 Morning: server configuration

table_cache setting (aka table_definition_cache and table_open_cache in 5.1) how many file descriptors are used to open table files.
max_connections
open_files_limit
thread_cache_size

Day 3 Afternoon: MyISAM

MyISAM storage engine
spatial index for efficiently searching geographical data
MyISAM key cache for caching MyISAM indexes
MERGE storage engine concatenates (like UNION ALL not UNION) more than one MyISAM table with the same schema

Day 4 Morning: InnoDB, Transactions, Locking

InnoDB storage engine
Configuring InnoDB to use a separate file per table (instead of the default of one shared ibdata file for all tables)
Partitioning (new in 5.1), partitions a table over multiple files.
Advisory locks are semaphores implemented in MySQL (not InnoDB specific). These don’t affect table locks.
Next key/gap locks means that when a locking SELECT on a range of indexed values is done, the table is also locked for INSERTs within that range (and the gap before that range!).
doublewrite setting
innodb_flush_log_at_trx_commit
Disk I/O
InnoDB status and monitors that print information about the InnoDB internal state
More about SHOW INNODB STATUS

Day 4 Afternoon: Other Storage Engines: Memory, Blackhole, CSV, Falcon

Memory is a table that’s stored completely in memory
Blackhole is like /dev/null
CSV yes, Comma Separated Value file
Falcon the 6.0 storage engine that will solve all of our perf problems
Example for writing your own storage engine

Singular and Plural in Rails

2008-01-08T06:01:00Z

One powerful and confusing thing in rails is how it wires up different things (database tables, models, controllers, urls…) with the “same” name. Rails uses both singular and plural, and is opinionated about grammar. One clue about what Rails wants is what happens when you generate a resource.

> ./script/generate resource pancake

      create  db/migrate/001_create_pancakes.rb # plural 
      create  app/models/pancake.rb # singular
      create  test/unit/pancake_test.rb # singular
      create  test/fixtures/pancakes.yml # plural
      create  app/controllers/pancakes_controller.rb # plural
      create  test/functional/pancakes_controller_test.rb # plural
      create  app/views/pancakes # plural
      create  app/helpers/pancakes_helper.rb # plural
      route  map.resources :pancakes # plural

The database table is plural.

db/migrate/001_create_pancakes.rb

123	class CreatePancakes < ActiveRecord::Migration def self.up create_table :pancakes do \|t\|

The model is singular, and associations are pluralization sensitive (the associations should be easy).

app/models/pancake.rb

123	class Pancake < ActiveRecord::Base belongs_to :eater # singular has_many :ingredients # plural

This is probably the confusing part: resource-based controllers should be plural (which means the directory for the views and the helper is also plural).

app/controllers/pancake_controller.rb

1	class PancakesController < ApplicationController

The route name and the urls it generates/recognizes are plural.

config/routes.rb

12	# /pancakes/:id => PancakesController#show, etc map.resources :pancakes

The rake routes command (also a good reminder of what to call your controller actions) will tell you what routes are generated from this. Here are a few of the pancake routes:

> rake routes

# named route	HTTP verb	path	generated parameters
pancakes	GET	/pancakes	{:controller=>”pancakes”, :action=>”index”}
	POST	/pancakes	{:controller=>”pancakes”, :action=>”create”}
pancake	GET	/pancakes/:id	{:controller=>”pancakes”, :action=>”show”}

(Is this right? Or am I still confused?)

Anorexia Causes Inability to Spell

2007-12-26T19:48:00Z

From our logs:

Dec 26 00:00:00 43things: Parameters: {“action”=>”query”, “q”=>”hw to be aorexic”, “controller”=>”search”}

Instant Web 2.0

2007-12-18T18:14:00Z

2.0-ize the entire web!

Mocking Facebook

2007-12-11T01:34:00Z

Any sort of interaction with external services is a serious hassle when trying to do TDD or even Development with Some Tests In It. External services could include anything from web services like facebook to our own drb ferret service (the real purists would include your database in this, but one step at a time…). The worst thing to do is to actually interact with the service in the test—it makes tests slow, dependent on a network connection, potentially messes up production data, etc.

The solution is mock objects. For a long time I did mock objects the wrong way, by inheriting from the real object and essentially reimplementing the functionality. This is pretty painful, so I didn’t do it much. Fortunately, really good mocking frameworks are everywhere now. Ruby even has two of them—mocha and flexmock.


    def test_facebook_publish_complete_goal
      @mock_fbsession = valid_facebook_session
      
      person = people(:rob_cooper)
      person.facebook_session = @mock_fbsession

      @mock_team_member = flexmock
      @mock_team_member.should_receive("goal.name").and_return("write a facebook app")
      @mock_team_member.should_receive("goal_is_complete?").and_return(true)
      @mock_team_member.should_receive("give_up?").and_return(false)

      @mock_fbsession.should_receive(:feed_publishActionOfUser).
        with(:title => 'has completed the goal: write a facebook app').once
      
      person.facebook_publish_goal_activity(@mock_team_member)
    end

First some preliminaries

First I need to set up a mock FacebookSession (I use a pseudomock wrapped around a real FacebookWebSession because I’m lazy and don’t want to mock the session_id and session_key accessors). The RFacebook library obviously wasn’t implemented TDD, so there are a few ways that it’s a bit messy to test.

Hide the description


    def valid_facebook_session
      mock_session =  flexmock(RFacebook::FacebookWebSession.new("test", "test"))
      mock_session.should_receive(:is_valid?).and_return(true)
      mock_session.should_receive(:is_ready?).and_return(true)

      return mock_session
    end

Next, I’m making a mock team member (a TeamMember is a model that represents a Person doing a Goal), but this time I use a plain mock that doesn’t reference the “real” object at all. And check out this line:

12	@mock_team_member.should_receive("goal.name").and_return("write a facebook app")

This call is actually making a chain of mock objects, so I can call @mock_team_member.goal.name without having to explicitly create a mock object for the goal.

Here’s the guts

The first line sets up the expectation—kind of like an assert.


      @mock_fbsession.should_receive(:feed_publishActionOfUser).
        with(:title => 'has completed the goal: write a facebook app').once
      
      person.facebook_publish_goal_activity(@mock_team_member)

The second line calls the method we wants to test. The expectation (that the method feed_publishActionOfUser will be called once with the specified arguments) will magically be evaluated when the test is over.