Author Archive

The word cloud has to be the most abused term in the history of technology buzzwords. The concept of a computing cloud varies depending on who you ask, typically I define it as a number of general purpose servers that can perform a range of tasks. Traditionally you might have had separate sets of servers doing mail, servers doing web, servers doing calendaring. With a cloud you’d have computers that did all of those tasks thanks to virtualization, partitioning or clever job/configuration management that moved work around as needed. Where does the line between cluster and cloud exist? I’d say a cluster is generally single purpose and a cloud is never in the next room, it’s remote.

Cloud hosting sprung up. Typically it meant that your website wasn’t hosted on a single machine or even a pair of machines, it was spread across many machines. Historically we’d have probably called this cluster hosting but the term worked fine. Then your data was in the cloud. Cloud backup. Cloud storage. Historically we would have called this remote storage, remote backup, maybe even remote cluster hosting but hey it works. Using remote web-based mail client instead of local mail application on your PC? You’re using a cloud app. It’s a cooler word I guess and marketing loves cool words. I’m down with the cloud. I do love Amazon Web Services after all.

Along comes Dear Adobe who have launched the Creative Cloud. (I’ll stop the bold italics now, I promise). The Creative Cloud is a new licensing scheme for Adobe Creative Suite. Instead of spending $2200 on Adobe Creative Suite Master Collection you can now pay $50/month. I think the name is awful given that there is an actual push towards remotely hosted applications like Office 365 and Google Docs. These are ‘Cloud’ apps by some definition, they live in someones cloud. The Creative Cloud is really just a licensing / DRM scheme. Unfortunately Adobe no longer plans to sell Creative Suite, it will be a rental only. If you were the kind of customer who bought a new version of CS Master Collection very 2 years then the $1200 will be a lot better than $2200. Unfortunately many (most?) customers do not buy every version, and even fewer likely buy the entire Master Collection. If you are interested in just Photoshop or Illustrator, Adobe will rent them to you at $20/mo each. Amazon is currently selling the latest and greatest Photoshop CS6 for $637 (Mac, PC is slightly cheaper), boxed with media and documentation.

At $637, all you have to do is keep Photoshop for more than 31 months before owning it has paid off. It may seem like a long time but the reality is most Photoshop users I know (who aren’t just using pirated versions) do indeed keep their software for 3 years or more. In fact CS5 which is just the previous version, was released over 3 years ago. Heck CS4 from 5 years ago is still plenty good and I know plenty of folks still using CS3. Photographers treat their software like their gear: you don’t need to upgrade every time a new version comes out. If it delivers the results you want, why spend more money? Anyone who bought CS5 within 6 months of its release still using it today would be getting a bad deal under the Creative Cloud. In fact the Creative Cloud seems to encourage piracy as it defeats the notion that you actually own anything. (Owning Photoshop was always a point of pride for me)

Adobe promises to support the last boxed retail CS6 software for the next version of MacOS and Windows (OSX 10.9 and Windows 9? I assume they don’t mean 8.1 but you never know…) so you can always snatch up a copy of CS6 and use it until Adobe comes to their senses or a competitive product comes around. And hey each Creative Cloud license includes 20GB of cloud* storage because studios really want to upload potentially license/NDA encumbered client work to the public cloud and personal users really want to remotely save giant intermediary images over their slow home broadband. Or they could always use the Windows SkyDrive, Google Drive, Apple iDrive, DropBox or 1TB free storage from Flickr.

Ugh, that’s three negative blog posts in a row, and I’m actually in a good mood today. Next one will be positive, I promise!

* –sorry

(Header photo was from Burlington’s waterfront on a day when a storm was rolling in)

Back in the early 90s, I split my gaming between the computer and consoles pretty evenly. PCs had complex RPGs and consoles had compelling action games. As time went on I stopped caring about upgrading my PC, debugging DirectX  drivers, messing with DLL versions and migrated entirely to console games. While PC gaming certainly stabilized in the Windows XP era it also began to become more and more DRM encumbered. Even the gamers beloved Steam is really a pretty rigorous form of DRM, it just happens to be the best implemented. (I admit I’ve happily purchased a bunch of cheap stuff from Steam) When I’d buy a console game I actually owned something. You just pop the disc in and play. 10 even 20 years later you can pop the disc/cartridge in and play. Legend of Zelda: A Link to the Past at the New Years Eve 2013 party? No problem.

The original Xbox and it’s successor the 360 kept this sane tradition going. If you happen to have a working 360 in 20 years you’re still good to fire up a single player game of Call of Duty 47. (Xbox-live will surely be turned off, so understandably no multiplayer) This weeks ‘Xbox-one’ unveil changes everything and not for the better. Games will include some sort of CD key and the console will go online to verify it is legit. Worse yet, it’s unsure how often it’ll need to be online to check. A snipit from TheVerge’s fairly comprehensive roundup:

What that means is in 20 years your discs will be bricks. There is absolutely no way to play these games, I just do not believe Microsoft will keep the auth servers up past the end of life for the console. Microsoft shut down Xbox Live services for the original Xbox in 2010, 8 years after its original release. Any games that required Xbox Live (i.e. online only games) no longer worked. Thing is I understand that for multiplayer games. Sure the ability to host local games would be great, but most games these days are tailored to large hosted environments. Having this requirement for single player games is nuts. You aren’t buying anything for your $60 (maybe $75 for the Xbox-one?), just a time limited license to run software. The very notion of having to pay a fee to ‘reactivate’ used games or lend your games to a friend is insane. I personally haven’t rented a video game since the Super Nintendo days but the industry still exists, and it still is a way many people try-before-buy. Good-bye rental industry.

What is Microsoft doing? This trend will certainly be the downfall of the console industry. It’s so blatantly anti-consumer and pro-publisher. Even the generally pro-Xbox US market is going to likely see the ‘no used games’ thing as a pretty big win for the Playstation 4. Heck, it’s probably a win for tablets and the casual-focused Android set top boxes whose games cost so little it doesn’t matter. I certainly know I won’t buy an Xbox-one if games require online activation, not even if there is a killer title for me. It’s an absolute non starter.

Also their unveil touted Kinect and voice controls heavily. Maybe it’s just me and I’m old fashioned but Kinect doesn’t appeal to me aside from the previously mentioned New Years Eve party setup. Personal taste I guess. The concept of the Xbox-one running in a low power standby mode constantly recording the audio for voice commands is mildly creepy too. I was more than slightly amused that 360 Kinect users watching the stream from their Xbox found themselves repeatedly booted out of the livestream thanks to the Kinect commands. The future trolling potential is excellent.

Unrelated but related, the quality of Google’s search results has really gone down hill. A couple years ago it would *always* ask if you meant Y when you typed X or at least tell you it was assuming you typed Y. Now it doesn’t say anything. I searched xbox one AMD trying to find out if the Xbox One has an AMD CPU and Google matched xbox one and. (I find it a little boring that both the next gen consoles have 8GB RAM, 8-core AMD x86 chips, etc. Where is the neat hard-to-dev-for exotic hardware :P)

Yet another lapse in blogging, and I haven’t really even had a good excuse this week. I’d broken in-bound links a couple weeks ago while adding a second blog to this WordPress instance, wondered why traffic fell off for a couple days… Sorry blog!

So Yahoo had a bunch of big news in the last week. First was the acquisition of Tumblr for over a billion dollars. This strikes me as more than slightly strange as Tumblr made $13 million last year with a cost of $25 million. For Tumblr to even make $100m in revenue they would need to monetize a great deal of their traffic. Considering the bulk of their traffic is very young users who aren’t purchasers their CPMs aren’t going to be that great. Also estimates put Tumblr’s traffic to be 15-20% adult content which is virtually un-monetizable. I’m not entirely sure how Yahoo plans to extract value without ‘ruining’ Tumblr frankly. (Marissa made a promise not to ruin it after all) It’s a lot of money to spend on cool-factor alone.

Tumblr has never really been of interest to me though, both the unpleasant infinite scrolling (which given enough animated GIFs chugs Chrome on an i7 Macbook) and the fact that it most of its content creators aren’t targeting my age group. However, I have been an avid supporter of Yahoo’s other acquisitionary property Flickr.

Flickr is one of a countless number of image hosting services. There are so many, but a few that are important to me: Imgur is the defacto host for Reddit, mlkshk is more of a community than an image host and Flickr was for photographers. What made Flickr for photographers? It showed camera details from the metadata beside the image in some views. Also Flickr built relationships with services to ‘woo’ professional photographers including Getty who would buy stock photos and a bevy of printing and delivery services. What resulted was a community of photographers swapping tips and discussing technique, equipment and locations. $25/year netted you unlimited space (instead of very limited space), finer grained controls over who could download what, and a little pro badge next to your name. While Flickr lapsed technology wise its dedicated community flourished. If I needed to find a lab in an obscure part of Japan? There was already a thread for that. Needed advice on scanning techniques? Thread for that. Interested in discussing a new lens release? Discussion ready to jump into.

May 20th Yahoo unveiled ‘the new’ Flickr. The dashboard now featured a vertical scroll of very large photos instead of a series of small thumbnails. That was annoying as I’m just not a fan of this trend towards less efficient content consumption. Users “photostreams” however were fairly tastefully redone. What was once a vertical stream of photos now resembled Tumblr’s archive and in fact allowed for faster content consumption. Decent enough, though the addition of a semi-infite scroll sucked (thankfully it really only loaded a couple pages at a time). OK I can learn to live with this I said, and probably even like the modern photostream view.

However Yahoo also overhauled the pricing structure. Gone was the fairly limited base account and $25 unlimited account, instead replaced with a free ad-supported 1TB account and $50 ad-free 1TB account and a $500 2TB account. There are a few problems with this though. Firstly, many users upload giant images from the 20+ megapixel cameras, 1TB is actually not that many photos, quite a step down from ‘unlimited’. The $500 2TB ‘Doublr’ account is just obscene. Yahoos are quick to point out buying 2TB of cloud storage will cost you more than $500, but there are plenty of photo hosting businesses that offer unlimited plans competitive to Flickr’s old Pro account. On top of that it isn’t clear the $500 Doublr is actually Ad-free…

Now Flickr has allowed Pro users to continue to renew at their $25 rate and keep their Unlimited storage. I’ve got a couple problems with that though. Terms will change, I just don’t believe they’ll allow them to exist forever and I’m unclear of the new ad policy. If I’m a paying user, do non paying users see ads on my photostream? If so that is mildly offensive but given that I couldn’t get a concrete answer about this I’m going to say that it’s on the table. Considering Yahoo has made the ‘Ad free’ value proposition considerably worse than the old Pro account it would appear they’d rather have ad impressions than paying users. Thankfully Yahoo CEO Marissa Mayer seems to have cleared up any confusion as to Flickr’s desire for a professional community:

Ooops. Better not tell the thousands of local wedding photographers, the product folk, the model photogs, oh and those guys who work in war zones for the press. Oh and hey, Flickr actually let me monetize my photos for stock purposes… decently no less. But there are no professional photographers. Frankly it’s this attitude + the overhaul that makes me believe I’m not going to like what will come in the future from the Flickr folk. Thankfully there are lots of companies that have sprung up to fill Flickr’s shoes. For me it was between Smugmug and 500px, and it looks like I’ll be migrating to Smugmug. Flickr will refund my year and a bit of remaining ‘Pro’ after 4 previous renewals. I can’t imagine I’m alone, and I just can’t imagine the world of non-pro photographers are really going to flock from Facebook’s readily available photo hosting, the built in Picassa hosting at Google+, Imgur, anywhere else. 1TB of space when all the others basically offer unlimited, it’s meaningless. The only people who care about resolution are pros and serious amateurs: people who likely don’t want ads on their portfolio. Where are those epic ad revenues going to come from Marissa?

Flickr you were the one social platform I actually liked… The real shame was I was a big fan of Yahoo in the mid 90s. I’d sorta hoped Marissa could turn it around and make a serious Google competitor somehow, we need more big competing internet properties. I just don’t see these changes turning Flickr into a social juggernaut, there just isn’t enough there to pull users from elsewhere. Don’t even get me started on Yahoo’s recent policy banning telecommuting…

I was going over some car pictures on the weekend and thought I really should just throw them all together in one place. Without further ado, here are my favourite pictures of the M Coupe….

There are quite a few choices when it comes to Ruby app servers. In my experience Unicorn is the best performing (by a modest 2-3% for our app) and is one of the most flexible. While Unicorn memory usage can be on the high side (it stores a copy of the code base in memory), RAM has become incredibly cheap. Most importantly, unlike Passenger you don’t need to buy ‘Passenger Enterprise‘ to perform hot or ‘rolling restarts’.

Hot restarts are critical for doing no downtime deployments. Tradition software deployments require you to throw up a maintenance page while your backend app servers restart (which can take upwards of minutes) which severely limits what time of day you can push new code or hot fixes. An alternative strategy is to take a server out of the cluster, update it, put it back in, update the next; but even automated this is very time consuming and can result in users experiencing multiple versions of your app within the same session as they go between different nodes. Plus there is, in my opinion, a lot more than can go wrong during such a deployment. Far more moving pieces.

Hot restarts solve this problem by spinning up an identical app server in background and seamlessly cut over between serving requests. Downside being you need a bunch of free memory, but upside is that switching over to new code takes about 30 seconds without any user facing outage. Unicorn does this beautifully, just not quite out of the box.

Once you have unicorn setup you’ll need to write a unicorn.rb to configure it. The default file has a bunch of important stuff commented out, and is entirely missing one entire critical section if you’re rolling off old releases from the disk.

Let’s start writing our config, first the easy stuff

# this should probably be between CPU threads and CPU threads * 2
worker_processes 8

# this is your current deployed code symlink
working_directory "/path/to/app/current" 

# don't use TCP to talk to Nginx
listen "/tmp/.sock", :backlog => 64;

# how long is it ok for your workers to hang
timeout 30 

pid "/path/to/app/shared/pids/unicorn.pid"
stderr_path "/path/to/app/shared/log/unicorn.stderr.log"
stdout_path "/path/to/app/shared/log/unicorn.stdout.log"

Next up we want to tell Unicorn to preload our app into memory before forking. This is pretty critical because otherwise users are going to get very slow responses as the app servers cache everything after a deploy. Very very slow.

preload_app true
GC.respond_to?(:copy_on_write_friendly=) and
GC.copy_on_write_friendly = true

Next up is really critical and not in the example. If you roll off old code from your app servers (i.e. the way chef, capistrano, basically anyone does it) you need to make sure Unicorn is not looking for the Gemfile it was originally started with. This also leads me to believe that without the following, you may be serving old gems, but I’ve never specifically tested. Either way, without this Unicorn will fail to hot restart after 5 or 10 deployments, whenever its original deployment is removed from your server.

before_exec do |server|
   ENV['BUNDLE_GEMFILE'] = "/path/to/app/current/Gemfile"
end

Next up is the meat and potatoes (or tofu and kale?) of hot restarts. We tell Unicorn to kill off it’s old self incrementally, gracefully. Also there are a few directives to disconnect the master from the DB under Rails, that bit is likely optional for a lot of people.

before_fork do |server, worker|
   defined?(ActiveRecord::Base) and
      ActiveRecord::Base.connection.disconnect!

   old_pid = "#{server.config[:pid]}.oldbin"
   if old_pid != server.pid
      begin
         sig = (worker.nr + 1) >= server.worker_processes ? :QUIT : :TTOU
         Process.kill(sig, File.read(old_pid).to_i)
      rescue Errno::ENOENT, Errno::ESRCH
      end
   end
end

after_fork do |server, worker|
   defined?(ActiveRecord::Base) and
      ActiveRecord::Base.establish_connection
end

With that done you just need a script to perform the hot restart. While you can just add a kill -USR2 `cat /path/to/app/shared/pids/unicorn.pid` into your deployment post symlink cutover, you’ll never know a) when the hot restart finishes or b) if the hot restart fails (does the new unicorn fail to start? does the old unicorn fail to quit?) So I wrote a little script to just do some sanity checks:

#!/bin/bash
# unicorn hot restart script --nick@crunched.com

PID="/path/to/app/shared/pids/unicorn.pid"
OLDPID="$PID.oldbin"

# how long should we let the hot restart go before we call it a fail
TIMEOUT=240 # seconds
# command to kick start unicorn if things arent working out
UNICORN_START="" 
# monit start unicorn? /etc/init.d/unicorn start? whatever you want

###################################
# check if unicorn isn't running
if [ ! -f $PID ]; then
  echo "[ERROR] No Unicorn PID found, attempting to start"
  eval $UNICORN_START
  exit 1
fi

# make sure a hot restart isnt in progress
if [ -f $OLDPID ]; then
  echo "[ERROR] A Unicorn hot restart appears to already be in progress, exiting"
  exit 1
fi

 get the PID of the current unicorn
CURRUNI=`cat $PID`

# verify the PID is actually running Unicorn
if ! grep --quiet -a "^unicorn master" /proc/$CURRUNI/cmdline; then
  echo "[ERROR] Unicorn does not appear to be running, attempting to start"
  eval $UNICORN_START
  exit 1
fi

# send USR2
echo "Kicking off a Unicorn hot restart for pid $CURRUNI"
kill -USR2 $CURRUNI

n=0

while (("$n" < "$TIMEOUT")); do
  echo -n '.'
  sleep 1 # always sleep on first run, we're too fast
  n=$(($n + 1))
  NEWUNI=`cat $PID 2>/dev/null` # if it hasnt started yet, no biggie

 # restart hasn't occured yet, no biggie
  if [ "$NEWUNI" = "$CURRUNI" ]; then
    continue
  fi

  # but what if the new PID is different....
  # is our old unicorn still alive?
  if grep --quiet -a "^unicorn master" /proc/$CURRUNI/cmdline 2>/dev/null; then
    continue
  fi

  # it's not! ok make sure its a unicorn master
  if ! grep --quiet -a "^unicorn master" /proc/$NEWUNI/cmdline; then
    # it's still starting, give it some time
    continue
  fi

  # woo hoo we've hot restarted!
  echo
  echo "Hot restart finished in $n seconds, new Unicorn is pid $NEWUNI!"
  exit 0
done

# uhoh we've reached here, the hot restart has failed
echo
echo "[ERROR] Timeout of $TIMEOUT seconds reached! Hot restart failed!"
# optional >>>>>>
echo "Attempting hard restart of Unicorn!"
kill -QUIT `cat $PID`
eval $UNICORN_START
# <<<<<< optional
exit 1

The last bit is totally optional. You may *not* want to hard kick a running but failed hot starting Unicorn in prod, while you might want to in a development environment. Hopefully this will all be helpful to someone attempting to add some really safe hot restarts to their Ruby app deployments. Happy Unicorning!