Gary Allison's Cloud Blog

Last week, one of my team members forwarded me a link to this blog by Savio Rodrigues, entitled Why devops is no silver bullet for developers.  It’s a well written blog and Savio makes some good points, namely that environments that the Devops team hopes to build on need to be standardized. He comes so close to hitting some important topics right on the head, and then just misses the mark slightly, IMHO.

Savio nails it when he points out

“One thing I’ve come to understand is that these two groups tend to think differently, even if they are using the same words and nodding in agreement.”

Bingo Savio.  He goes on to say,

“It’s no surprise developers want to adopt tools and processes that allow them to become more efficient in delivering new applications and continuous updates to existing applications. Today, these two tasks are hindered to a degree by the operations teams that are responsible for production environments”

But then, he misses an opportunity to drive the point home and starts a discussion about standards. I agree standards are important, but what needs to be reckoned with are the very different culture, goals, and reward systems between the two disciplines of Engineering/Development and IT/Operations.

How are these teams measured and rewarded? The answers to these questions tell you many things about the team’s culture. A Development team is typically measured and rewarded by amount of innovation, quality of their deliverables, timeliness of delivery, and responsiveness to market.  An IT team is measured and rewarded typically by uptime, stability, security, and control.  (Note rewarded can mean “not punished due to failure” as well as more expected definitions of reward).

All of the above seem like good things! We want uptime, innovation, quality, stability, etc!  Right? I envision one could draw a Venn diagram for the Dev culture and the IT culture and there would be overlap, but there would be just as much outside the intersection.  Innovation is often at odds with stability.  Responsiveness to market can be at odds with uptime, etc.

We’ve had the good fortune of having a few opportunities to implement a new Devops model.  When everyone is rowing together the boat certainly moves faster in the desired direction. But it is difficult. It requires continual investment in the Devops team because at the core, these two very different cultures aren’t going away anytime soon.  Savio sees it too when he says, “This isn’t a technical issue. It’s a cultural issue.” I’d suggest we spend as much time looking at the measurements and rewards as we do thinking about standardizing platforms.

Recently, a tragedy hit one of the members on my team where he lost his home.  Our team has rallied around him and his family and have done what we can to help – that’s just the kind of people I’m fortunate enough to work with.  In talking to him, one of his regrets is that he didn’t have his photos backed up offsite.  He said he looked into it, but then just didn’t get around to it.  That was inspiration to get me moving…

I investigated a number of commercial solutions first, and the best I found was Carbonite.  One yearly fee to backup all your documents, music and photos (no movies), $59. That is hard to beat for those with a significant amount of photos or music.  (With the ever improving CCD imaging of digital cameras, everytime you buy a new camera, the photo files are larger. Is it a plot between the hard drive makers and the camera manufacturers? LOL) Sounds like a great deal, right?

The Carbonite app installed smoothly and ran well. It seems one key to their business model is to control bandwidth. Or, perhaps the service is very popular. They warn you that the initial backup could take several days.  Well, after more that a week, mine was still less that 50% complete.  About that time my trial period and patience both expired.  If you don’t mind leaving your computer on for a month, this still looks like a very good option.  They also have a switch in their UI where you can use less bandwidth on the upload.  This will make the backup take even longer, but will allow the kids to still watch YouTube while you are taking your backups. Lastly, they have a web UI where your can explore your backed up files from anywhere. It’s a viable solution IMHO.

I started looking at other Cloud Backup solutions for my Mac (not the kind of cloud backup they get in Kentucky where it just seems to rain all the time). Amazon S3 seemed like the natural next choice to investigate, but what is it going to cost? Looking at S3 pricing, currently it runs:

So, this is more than Carbonite at my data volume, but more reasonable at the “reduced redundancy” pricing.  Reduced redundancy is perfect for my use case since I backup all my files to an external hard drive already and this really is a disaster recovery scenario.  So for me, this will run around $84 dollars a year.  Still expensive, but S3 prices also go down at least twice a year historically.  We’ll see how it works out. At the very least, it’s cool.

Another option worth considering is Amazon’s new “Cloud Drive“.  The prices are lower than S3, with 5Gb free and other tiers at $10/Gb per year.  The tools are a little clunky right now as it is really aimed at working with music.  If you are mostly worried about backing up music, Cloud Drive makes it completely simple with their music upload and streaming tools.  For other file types its a little more manual.  But, the price it right.

Back to exploring S3.  First, we need to check out tools available for managing S3.  At this point, I was feeling very cheap since the storage costs are a little more than I wanted in the first place.  There are some good tools out there like jungle disk that would likely make this much easier, but I was looking for cheap as opposed to easy.  With jungle disk, you could take the complexity of the rest of this solution down considerably.

First step is to go to Amazon and create an Amazon Web Services account.  You probably already have an amazon.com account and you can use the same login.  Then login to Amazon Web Services and create an S3 bucket.

For syncing files to S3, I found an attractive free option in s3sync, a Ruby gem that gives us a command line way to sync between my Mac and S3.  Here’s a great blog entry on the Ruby gem installation and config, so I won’t repeat that part.  Then, to backup your files, use a command similar to this:

s3sync -r -v /Users/YOURUSERNAME/Pictures/iPhoto\ Library/Originals/2011 yourbucket:iPhotoBackup/Originals

The above will copy the photos out of iPhoto on your Mac that were taken this year (2011) into your bucket in the folder Originals. You’ll need to create the folder structure iPhotoBackup/Originals before executing this command.  You could also leave off the “/2011″ and the /”Originals” like this to back up your entire iPhoto library, but this is going to take a very long time to upload to S3:

s3sync -r -v /Users/YOURUSERNAME/Pictures/iPhoto\ Library/Originals yourbucket:iPhotoBackup

With the -v option you see each file listed as it is uploaded.  Like Carbonite, this will also take quite a while, and during the upload, a lot of your internet bandwidth will be consumed such that Netflix on demand, web browsing, etc will be slow for everyone in the house. Not surprising, just thought I’d throw that out there.  This is a good reason to do it directory by directory perhaps overnight until you have it all complete.

The net step is very important to save you $$$s.  You need to go to your Amazon Webservices Console, explore your S3 bucket, right click on the folder you just uploaded, and select Properties (or select Properties button at top right).  From there, you need to select “Reduced Redundancy” and Save.  This will then iterate through all the items in the bucket and mark them for reduced redundancy.  There is no way to select this as the default for all files uploaded to a bucket.  Hmmmmm, I wonder why?  Greedy a bit Amazon?

If you are a Windows user, you may want to check out Cloudberry Explorer.  They have a nice S3 interface that supposedly can mark each file for reduced redundancy after uploading for you.  Looks like an interesting option.

There is quite a bit more to know about S3 than contained in this blog.  For example, you can make selected files or folders public and hand out URLs, etc.  Also, Amazon doesn’t charge you for transfer bandwidth on the upload, but does on the download.  There are many other considerations to think through in choosing a cloud backup solution that is right for you, but hopefully you find this informative and useful.

Reaching the Summit

Today was an especially good day.  Few things can compare to seeing the combined efforts of a large team who has worked so hard together towards a shared vision finally reach that goal.  Like hikers on a long trek finally reaching the summit, today we took time to survey the terrain we conquered, thought a little about the road ahead, but still enjoyed the moment.  Building SaaS based products with Agile process can result in a relentless pace, so these moments are special.

I’ve had the pleasure of leading terrific teams of very talented people in delivery of software projects, and this day, I have the privilege of leading the best of those.  The scale of innovation, integration, and imagination that as a team they delivered is a tribute to their commitment to our company and our clients.  For me, it is a huge thrill to see what months ago was a set of ideas turned into a quality solution.

Today, in our team meeting, I tried to convey the importance of this accomplishment.  Ten years from now, each of us may look back on this day, proud and maybe a little amazed of what we did together.  There’s a very good chance many of us will never deliver so much innovation in one day again on this scale.  I’m proud of our team, grateful for their efforts, and hopeful many non-profits will benefit from their hard work.

9:20 Marc Benioff has just taken the stage – quite a showman he is. Nice video overview of day 1. Benioff now talking about Microsoft as the evil empire – says they are trying to stop chatter, the sales cloud, and his new socks. Talking now about the Microsoft lawsuit – and Micosoft “protest” outside yesterday. (MS had seqway drivers out front with “Don get Forced” slogans.)

Benioff asks the crowd to help him get this customer back and ask Bernard to come on out…  Its the guy in the MS advertisements – on the trucks and seqways.  “Bernard I’m so sorry…. we want to apologize to you”  “We don’t want you to go back to software, the constant upgrades, waiting for new features…”  Crowd applauds and Bernard says he will come back.  Classic Benioff.

9:38 Benioff now talking about his donation of $100M for a children’s hospital at UCSF. Describing all the investment in a new campus at Mission Bay.  Today is UCSF day at Dreamforce. Working to raise $1.5B for the new hospital. Still waiting for the force.com announcements.

9:52 here we go with the meat of the keynote – “your platform is too proprietary” Benioff wants to open the platform further. Adding Ruby on Rails support. “Ruby is the true language of cloud 2″ (talking about speed and agility of using the language). Salesforce buying Heroku!  Wow.

Heroku will be salesforce’s seventh cloud.  Will keep Heroku as an independent team. Heroku founded in 2007 to enable fast deployment/updgrade/delivery of Ruby apps. Fancy d- emo, but no details on how this will integrate with force.com yet.

Eighth cloud announcement: BMC Software – CEO Bob Beauchamp on stage now. Announcing Remdyforce now available on salesforce.com, IT configuration management.

Platform team announcements – “force.com 2″ -going to build a killer product for each market:

appforce – departmental and collaborative applications. looks like a marketing repackaging of what force.com already provides. don’t see anything new here.  Improvements to sharing – new sharing model to be delivered in the Spring. Configurable Visualforce pages, Reduce governor limits by 70% in Spring.

siteforce – integrated cms, point and click editor, prebuilt components, social and mobile built in, 24×7 availability. Killer UI on the new CMS for content creation and publishing.

vmforce – accenture on stage talking about vmforce. Accenture is investing in vmforce – I don’t see anything really new here and the whole thing feels like a little bit of a false start.  With the new acquisition of Heroku, I would be dubious of a large investment in vmforce.

isvforce – packaged apps on the appexchange – more marketecture.  Didn’t see anything new here.

Now have CEO and CIOs on stage from Blackboard, Belkin, Avon, Kelly Services, and Deloitte talking about how the platform has helped them.

that’s a wrap.

Marc Benioff speaking now – 14,000 in attendance at the keynote, 30K registered for conference.  Emphasizing platform and database in the cloud.  Waiting to see where he is going with this, what new announcements he has in his pocket…

Salesforce serves “100,000 customers, running of 1,500 Dell PCs” Marc says. That’s a gross over simplification.  His point is that cloud computing is green because of more efficient use of resources 90% more efficient than traditional hosting.

Talking about salesforce foundation now – asked all non-profits attending to stand up and be recognized – applause.

Broad change in internet usage.  Social networking users surpassed email users last year.  Significant growth in usage via smartphones. Cloud 2 is the shift from easy fast and low cost (Cloud 1) to social and mobile (Cloud 2). Interesting that Marc positions Amazon, Google, and eBay as Cloud 1.

If half a billion people are on facebook, why aren’t we building software that looks like facebook?  Why isn’t enterprise software like facebook?  Focus on 6 clouds: chatter, jigsaw data could, force.com, database.com, along with sales cloud2, service cloud 2.

Major changes in salesforce.com will be annouced tomorow, and appforce to build apps, and site force for sites.

Demoing jigsaw – clean and update your contacts from jigsaw database, can also search for contacts at a opportunity.

CEO of Symantec on stage now. Arguing chatter is more efficient than email, claiming productivity going up. I privately wonder how this can be measured.  Is it just more information overload and more surface level communication without depth? I do get that it is useful to communicate very brief bits of info quickly.

“How do I get my whole company on chatter?” New product announcement – ChatterFree.  Whole company can be on chatter.  Great strategy – obviously trying to be the facebook of business. Admins have to enable it in the organization. Employees can be provisioned automatically or can be invited to join.  Includes chatter on mobile device.  Also, Chatter.com coming by end of year – free for everyone, generally available public site.

Demo of service cloud 2 – screen pop integrated with telephone system, integrated with knowledge base and a call script. Also demoed the availability of the KB via google search. Agent to web visitor chat is included in service cloud 2.  Demoed twitter integration showing how service cloud can monitor tweets and create cases, including photos and location information, and send answer via tweet.

force.com huge announcements coming tomorrow – vision of leading platform for cloud 2.0 applications – faster cloud 2 apps, including java. 185K force.com apps today.  Big emphasis on Open – make it open.

Database.com announcement: claim to have the most scalable cloud database – 25B db xactions through Q311, 200B DB records, 12B custom tables, response times falling to around 275ms.  When they released chatter,  they extended the DB model to include social data model (follow any entity) and recently published a mobile / REST APIs.

Announcing today database.com – “first enterprise database for cloud 2″ open to any language / platform.  Full relational db, full text search index, user mgmt, row level security, triggers and stored procs, authentication, APIs.  Elastic, auto-upgrade, auto-backup, auto-disaster recovery. Trusted and secure, SAS 70 Type II certified.

Database.com demo: view of db instances after login. Create an db, graphical schema editor in nice UI, showing console running a select to show the db is online.  Showing VMForce java code connecting to db.com. demoing facebook app querying jobs database and uploading a resume to a open position. facebook app is a php app running on EC2. Demoing android app now pulling job data from db.com. Finally, showing recruiting app running on ipad.  Sharing and security of db.com restricts visibility of job applications to relevant departments. Showing collaboration in interview process attached to job application using chatter like social sharing.

Database.com will be available next year. first 100K records are free. $10/month per 100K records. Benioff promises even more announcements tomorrow. Will I Am and Stevie Wonder tonight.  Gnereal Powell tomorrow?  Wow.

Here waiting for keynote to start…. It is at least double the conference of 2008. I estimate this room can hold 10,000 people. Will be interesting to hear what Benioff announces today.

It’s been a busy week as usual this time of year, but right in the middle of the week appeared my very own shiny Droid phone.  I’m still just  a little in awe that Google has offered a new Android based phone to everyone at the conference.  Wow, what an impressive display of financial clout.  I feel it is a smart move – to get the people that obviously care most about what Google is doing to get interested in building Android apps.

I’m very interested in any announcements that might be made at the conference.  I feel relatively confident that Google is working on a tablet to compete with iPad as predicted in this blog in January. (Oh, that was my last post.  Like I said it is very busy this time of year!)

Will they follow Apple’s move and push a table based on Android?  Seems like a plausible move.  With the wood they have behind Android application development, this seems the logical course.

I can’t wait to build an app for this phone.  Yep, tech bribery works.  Good job Google.

Today, Amazon announced their Amazon Elastic MapReduce cloud web service.  A natural extension to their EC2 cloud services on one hand, and a somewhat startling event on the other.  In a recent post, I spoke of the global implications of the readily available low cost cloud computing infrastructure.  Now, it seems this service has entered the realms of massively parallel computing.

Not clear at this point what the limits of this service are, but the possibilities are staggering.  Not enough computing power in Pyongyang?  No problem, run your nuke simulations right here Mr Kim Jong-il. An extreme case, and perhaps too complex / compute intensive for this offering, but the point is never have resources of this scale been so readily available.  Amazing.

A better fit for tasks like SETI@Home, these distributed networks are useful in solving very large data intensive types of problems like web indexing.  Google even provides a nice tutorial on mapreduce.  This is also a nice class lecture on mapreduce from Cal Berkeley.  Enjoy.