Gary Allison's Leadership Blog

November 2011

Cloud Computing and Effective Software Projects and Leadership and Teams27 Nov 2011 10:28 pm

Last week, one of my team members forwarded me a link to this blog by Savio Rodrigues, entitled Why devops is no silver bullet for developers.  It’s a well written blog and Savio makes some good points, namely that environments that the Devops team hopes to build on need to be standardized. He comes so close to hitting some important topics right on the head, and then just misses the mark slightly, IMHO.

Savio nails it when he points out

“One thing I’ve come to understand is that these two groups tend to think differently, even if they are using the same words and nodding in agreement.”

Bingo Savio.  He goes on to say,

“It’s no surprise developers want to adopt tools and processes that allow them to become more efficient in delivering new applications and continuous updates to existing applications. Today, these two tasks are hindered to a degree by the operations teams that are responsible for production environments”

But then, he misses an opportunity to drive the point home and starts a discussion about standards. I agree standards are important, but what needs to be reckoned with are the very different culture, goals, and reward systems between the two disciplines of Engineering/Development and IT/Operations.

How are these teams measured and rewarded? The answers to these questions tell you many things about the team’s culture. A Development team is typically measured and rewarded by amount of innovation, quality of their deliverables, timeliness of delivery, and responsiveness to market.  An IT team is measured and rewarded typically by uptime, stability, security, and control.  (Note rewarded can mean “not punished due to failure” as well as more expected definitions of reward).

All of the above seem like good things! We want uptime, innovation, quality, stability, etc!  Right? I envision one could draw a Venn diagram for the Dev culture and the IT culture and there would be overlap, but there would be just as much outside the intersection.  Innovation is often at odds with stability.  Responsiveness to market can be at odds with uptime, etc.

We’ve had the good fortune of having a few opportunities to implement a new Devops model.  When everyone is rowing together the boat certainly moves faster in the desired direction. But it is difficult. It requires continual investment in the Devops team because at the core, these two very different cultures aren’t going away anytime soon.  Savio sees it too when he says, “This isn’t a technical issue. It’s a cultural issue.” I’d suggest we spend as much time looking at the measurements and rewards as we do thinking about standardizing platforms.

Cloud Computing and Everyday Tech27 Nov 2011 08:38 am

Recently, a tragedy hit one of the members on my team where he lost his home.  Our team has rallied around him and his family and have done what we can to help – that’s just the kind of people I’m fortunate enough to work with.  In talking to him, one of his regrets is that he didn’t have his photos backed up offsite.  He said he looked into it, but then just didn’t get around to it.  That was inspiration to get me moving…

I investigated a number of commercial solutions first, and the best I found was Carbonite.  One yearly fee to backup all your documents, music and photos (no movies), $59. That is hard to beat for those with a significant amount of photos or music.  (With the ever improving CCD imaging of digital cameras, everytime you buy a new camera, the photo files are larger. Is it a plot between the hard drive makers and the camera manufacturers? LOL) Sounds like a great deal, right?

The Carbonite app installed smoothly and ran well. It seems one key to their business model is to control bandwidth. Or, perhaps the service is very popular. They warn you that the initial backup could take several days.  Well, after more that a week, mine was still less that 50% complete.  About that time my trial period and patience both expired.  If you don’t mind leaving your computer on for a month, this still looks like a very good option.  They also have a switch in their UI where you can use less bandwidth on the upload.  This will make the backup take even longer, but will allow the kids to still watch YouTube while you are taking your backups. Lastly, they have a web UI where your can explore your backed up files from anywhere. It’s a viable solution IMHO.

I started looking at other Cloud Backup solutions for my Mac (not the kind of cloud backup they get in Kentucky where it just seems to rain all the time). Amazon S3 seemed like the natural next choice to investigate, but what is it going to cost? Looking at S3 pricing, currently it runs:

Standard Storage Reduced Redundancy Storage
First 1 TB / month $0.140 per GB $0.093 per GB
Next 49 TB / month $0.125 per GB $0.083 per GB
Next 450 TB / month $0.110 per GB $0.073 per GB
Next 500 TB / month $0.095 per GB $0.063 per GB
Next 4000 TB / month $0.080 per GB $0.053 per GB
Over 5000 TB / month $0.055 per GB $0.037 per GB


So, this is more than Carbonite at my data volume, but more reasonable at the “reduced redundancy” pricing.  Reduced redundancy is perfect for my use case since I backup all my files to an external hard drive already and this really is a disaster recovery scenario.  So for me, this will run around $84 dollars a year.  Still expensive, but S3 prices also go down at least twice a year historically.  We’ll see how it works out. At the very least, it’s cool.

Another option worth considering is Amazon’s new “Cloud Drive“.  The prices are lower than S3, with 5Gb free and other tiers at $10/Gb per year.  The tools are a little clunky right now as it is really aimed at working with music.  If you are mostly worried about backing up music, Cloud Drive makes it completely simple with their music upload and streaming tools.  For other file types its a little more manual.  But, the price it right.

Back to exploring S3.  First, we need to check out tools available for managing S3.  At this point, I was feeling very cheap since the storage costs are a little more than I wanted in the first place.  There are some good tools out there like jungle disk that would likely make this much easier, but I was looking for cheap as opposed to easy.  With jungle disk, you could take the complexity of the rest of this solution down considerably.

First step is to go to Amazon and create an Amazon Web Services account.  You probably already have an account and you can use the same login.  Then login to Amazon Web Services and create an S3 bucket.

For syncing files to S3, I found an attractive free option in s3sync, a Ruby gem that gives us a command line way to sync between my Mac and S3.  Here’s a great blog entry on the Ruby gem installation and config, so I won’t repeat that part.  Then, to backup your files, use a command similar to this:

s3sync -r -v /Users/YOURUSERNAME/Pictures/iPhoto\ Library/Originals/2011 yourbucket:iPhotoBackup/Originals

The above will copy the photos out of iPhoto on your Mac that were taken this year (2011) into your bucket in the folder Originals. You’ll need to create the folder structure iPhotoBackup/Originals before executing this command.  You could also leave off the “/2011″ and the /”Originals” like this to back up your entire iPhoto library, but this is going to take a very long time to upload to S3:

s3sync -r -v /Users/YOURUSERNAME/Pictures/iPhoto\ Library/Originals yourbucket:iPhotoBackup

With the -v option you see each file listed as it is uploaded.  Like Carbonite, this will also take quite a while, and during the upload, a lot of your internet bandwidth will be consumed such that Netflix on demand, web browsing, etc will be slow for everyone in the house. Not surprising, just thought I’d throw that out there.  This is a good reason to do it directory by directory perhaps overnight until you have it all complete.

The net step is very important to save you $$$s.  You need to go to your Amazon Webservices Console, explore your S3 bucket, right click on the folder you just uploaded, and select Properties (or select Properties button at top right).  From there, you need to select “Reduced Redundancy” and Save.  This will then iterate through all the items in the bucket and mark them for reduced redundancy.  There is no way to select this as the default for all files uploaded to a bucket.  Hmmmmm, I wonder why?  Greedy a bit Amazon?

If you are a Windows user, you may want to check out Cloudberry Explorer.  They have a nice S3 interface that supposedly can mark each file for reduced redundancy after uploading for you.  Looks like an interesting option.

There is quite a bit more to know about S3 than contained in this blog.  For example, you can make selected files or folders public and hand out URLs, etc.  Also, Amazon doesn’t charge you for transfer bandwidth on the upload, but does on the download.  There are many other considerations to think through in choosing a cloud backup solution that is right for you, but hopefully you find this informative and useful.


Leadership and Teams24 Nov 2011 08:46 am

This is unquestionably one of my favorite times of the year – surrounded by family and tradition, all the memories of Thanksgivings past flood back over me.  I am so Thankful for my amazing family, that I have been able to celebrate another year full of joy and challenges, seeing my daughters grow up and change so much, and still being able to share these things with my Mom and Dad. I’m Thankful for my good friends of various and assorted funny nicknames and for them helping me smile when I am overly stressed.   I’m very Thankful for my amazing and supportive wife who gets the pleasure of seeing all that stress as well, handling it with grace.  Finally, I am so very Thankful for the talented, brilliant, and dedicated people I have the privilege of working with every day.  To every engineer, support tech, IT engineer, sales person, account manager, web developer, product manager, marketing professional, trainer, and fellow exec, you really are changing the world and making it a better place.  Thank you for letting me be a part of it.