Pink Pucker.net

Yikes, it’s 2008. I’ve been working full-time on Ruby and Rails for about a year now. It’s been awesome. I’ve made a ton of mistakes and learned a ton from them. :D

Random thoughts:

Version Control

Version controlling your source code is really important. About a month ago, I switched over to git. No more subversion. It’s been an interesting transition. Git’s documentation is pretty bad, imo. I still don’t know what the hell I’m doing half the time. But time will fix that. For me, the big benefits of git over svn:

• Branching and merging are sooo much easier. You don’t need to be hooked up to the internet to make a branch. It’s completely painless to make branches and merge in between them. It was always painful to do that in svn. Every developer can have their own branch, and merge between them easily.
• gitk, the graphical source browser, is pretty nifty.
• git bisect. That’s a tool that lets you go back in time to quickly identify the source of a bug. Super awesome.
• Really, you can do pretty much everything in git offline. I’ve been offline for weeks at a time over the christmas and new years break, and I was so productive. And then when I got back online, it’s a simple ‘git push’ to push everything to the remote server.

Books

I’ve really liked The Rails Way. It’s a fantastic reference and guide to Rails.

Software

BackgroundRB got a much needed overhaul and recently hit version 1.0. I’ve been using it in place of cron and it’s been rock solid.

More to come later.

I moved Tanga over to Amazon S3. That was quite the adventure.

I hope it works well.

I recently switched Tanga’s DNS over to DNS Made Easy. I highly recommend them.

I love making DNS records. Don’t ask me why.

What’s cool about them is that you can set up http redirections easily—so that http://stats.tanga.com takes you to our Google Analytics page.

Tanga.com is now using Spread for logging. Right now, only mongrel / rails and lighttpd are logging to spread. It’s cool because:

1. The database and application servers don’t need to waste time logging files to disk, they just make one function call that mutlicasts the log data to spread and that’s it.
2. One or more dedicated (or semi-dedicated) machines can listen to the spread network and capture the log data and write it to the log files.
3. Unified logging mechanism.
4. Should scale up very well.

One thing I’m curious about—it would be neat to somehow extend syslog to be able to write to the spread network.

I had to write two small pieces of software to get the logging to spread.

1. For lighttpd, here’s a patch against 1.4.16
2. For mongrel / rails, below is a small Logger class that’s a dropin replacement for the standard logger. It’s largely stolen from Eric Hodel’s SyslogLogger. You’ll need to have the rb_spread library as well.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107

require 'logger'
require 'spread'

class SpreadLogger

  LOGGER_MAP = {
    :unknown => :alert,
    :fatal   => :err,
    :error   => :warning,
    :warn    => :notice,
    :info    => :info,
    :debug   => :debug,
  }


  LOGGER_LEVEL_MAP = {}

  LOGGER_MAP.each_key do |key|
    LOGGER_LEVEL_MAP[key] = Logger.const_get key.to_s.upcase
  end

  LEVEL_LOGGER_MAP = {}

  LOGGER_LEVEL_MAP.invert.each do |level, severity|
    LEVEL_LOGGER_MAP[level] = LOGGER_MAP[severity]
  end


  def self.make_methods(meth)
    eval <<-EOM, nil, __FILE__, __LINE__ + 1
      def #{meth}(message = nil)
    begin
        return true if #{LOGGER_LEVEL_MAP[meth]} < 1
        log_message(#{LOGGER_LEVEL_MAP[meth]}, message)
    rescue Exception => e
    end
        return true
      end

      def #{meth}?
        @level <= Logger::#{meth.to_s.upcase}
      end
    EOM
  end

  LOGGER_MAP.each_key do |level|
    make_methods level
  end

  attr_accessor :level

  def log_message severity, msg
    begin
      initialize_spread
      msg = clean(msg)
      msg = "#{Time.now.strftime("%b %d %H:%M:%S")} #{Socket.gethostname.split('.').first} rails[#{$PID}]: #{msg.gsub(/\n/, '').lstrip} \n"
      @spread.multicast(msg, @group, 1)  if @spread
    rescue Exception => e
      @spread = nil
    end
  end



  def initialize_spread 
    @spread ||= Spread.new @server, @my_name 
  rescue Exception => e
    puts 'couldnt initialize spread'
    puts e.inspect
  end

  def initialize(server, group)
    @level = Logger::DEBUG
    @my_name = `hostname`.chomp
    @group = group
    @server = server
    initialize_spread
  end

  def add(severity, message = nil, progname = nil, &block)
    severity ||= Logger::DEBUG
    @progname = progname || "tanga-mongrel"
    return true if severity < @level
    return true
  end


  def silence(temporary_level = Logger::ERROR)
    old_logger_level = @level
    @level = temporary_level
    yield
  ensure
    @level = old_logger_level
  end

  private


  def clean(message)
    message = message.to_s.dup
    message.strip!
    message.gsub!(/%/, '%%') 
    message.gsub!(/\e\[[^m]*m/, '') # remove useless ansi color codes
    return message
  end

end

If you want a paper version of the excellent Postgresql online manual, check out lulu.com.

$56.75 shipped for the entire thing. Not a bad deal.

All about the technical things at Tanga:

We use Joyent Accelerators.

• asset_packager It packages up all your javascript and css files into one file each. Also does minimizing and reduces whitespace. If running on edge rails and are using the asset_host functionality, you’ll need to remove some code from the plugin. I should talk to the guy about that.
• exception notifier Tells me when stuff breaks.
• attachment_fu Handles file uploads. I had to modify it a lot to do SWF uploads. I used it with image_science. image_science doesn’t deal nicely with certain gifts, so I rename gifs to pngs.
• ActiveMerchant is a great Ruby credit card processing library. No complaints.
• RSpec is fantastic for testing. All my new tests are doing using RSpec.

• I’ve got three accelerators running mongrels. Mongrels do the dynamic Rails page generation. Static files are served by lighttpd. Load balancing of the mongrels is done by a Big-IP. The Big-IP also does hardware SSL.
• Rough benchmark shows that I can do a few hundred dynamic Rails pages per second. It’s fast enough for me. Should scale up to a couple thousand per second just by adding new accelerators.
• Subversion is used for source code version control.
• I do (or used to do, hasn’t worked for a while) continuous integration. So, on every subversion check in, all the tests should run and it should tell the developers if stuff breaks.
• Nagios is running on a friend’s server. It monitors Tanga—watches stuff like disk space, tries to log in to the website, make sure the front page loads ok, makes sure that the media servers serve stuff ok, checks to make sure the ssl certificate for tanga is valid, stuff like that. It’s great! I love how flexible it is. It’s really easy to customize and extend.
• Backups are sent to Amazon’s S3 using a modified copy of S3 Rake.
• I’m using postgresql 8.2.4. Works great! Worried about replication though.

Stats

note: I have a lot more tests—for some reason, only the RSpec tests are included in this, the rest of the old-school tests are not included in the total.

+----------------------+-------+-------+---------+---------+-----+-------+
| Name                 | Lines |   LOC | Classes | Methods | M/C | LOC/M |
+----------------------+-------+-------+---------+---------+-----+-------+
| Libraries            |   606 |   482 |      21 |      74 |   3 |     4 |
| Models               |  3618 |  2887 |      64 |     454 |   7 |     4 |
| Helpers              |   646 |   553 |       0 |      67 |   0 |     6 |
| Components           |     0 |     0 |       0 |       0 |   0 |     0 |
| Controllers          |  2844 |  2400 |      53 |     329 |   6 |     5 |
| APIs                 |     0 |     0 |       0 |       0 |   0 |     0 |
| Model specs          |   817 |   679 |       0 |       2 |   0 |   337 |
| View specs           |   284 |   241 |       0 |       1 |   0 |   239 |
| Controller specs     |   415 |   335 |       0 |       1 |   0 |   333 |
| Helper specs         |   284 |   241 |       0 |       1 |   0 |   239 |
+----------------------+-------+-------+---------+---------+-----+-------+
| Total                |  9514 |  7818 |     138 |     929 |   6 |     6 |
+----------------------+-------+-------+---------+---------+-----+-------+
  Code LOC: 6322     Test LOC: 1496     Code to Test Ratio: 1:0.2

Books and websites

Other useful software

Owies

We’ve had some downtime:

• Every so often, our credit card provider (authorize.net) acts up and gives us an empty response back when we try to authorize a credit card. I’m rewriting our credit card processing software so I’m curious if the problem happens after the rewrite.
• Joyent is investigating a problem with SSL and the Big-IP load balancer. I’m not sure if they’ve resolved it yet. The problem manifests itself by not letting people connect to https://secure.tanga.com. Which people need to login and buy stuff from us. Highly important! And unfortunately, their monitoring system wasn’t set up to catch this sort of problem. It’s really unfortunate when I go to bed and then wake up to see eight hours of past downtime.

Nagios is great for letting me know about problems when they occur—I just need to be awake.

Other misc

We use AppRiver for email (Exchange) hosting.

I’ve got a BlackBerry Pearl that keeps me in the email loop.

I got Nagios installed and configured. It’s pretty slick, although it was a sort of a pain to get everything working.

First off, you want to get this book: Building a Monitoring Infrastructure with Nagios. It’ll get you up to speed on nagios.

Then, find a Linux box. You can use a non-linux machine, but the above book recommends linux. Install nagios. Installing it via a package is easiest, but the book recommends installing from source. I installed from source.

Then you gotta install NRPE on all the systems that you want to monitor. I installed NRPE via package—didn’t build it from source.

• I used the blastwave packages (pkg-get -i nrpe)
• Added the SMF manifest (svccfg import /opt/csw/share/doc/nrpe/nrpe.xml)
• Made appropriate changes to nrpe.cfg (you’ll learn what the changes are from reading the above book) and copied /opt/csw/share/doc/nrpe/nrpe.cfg to /opt/csw/etc
• Copied the SMF script from /opt/csw/share/doc/nrpe/svc-nrpe to /opt/csw/lib/svc/methods
• Enabled NRPE (svcadm enable cswnrpe)

In the nagios hosts.cfg file, I added the level 3 router (209.247.11.146) as the parent of all the accelerators.