Gary’s Tech Thoughts Blog

Lots of discussion lately about measuring productivity has had me spending time I should be sleeping thinking about the same.  I love accountants and finance folks.  I find them very bright and love the way they typically approach any business discussion from the point of logic, but they can be an intractable lot as well.  They’d love to measure software engineering efforts like a consultancy – hours in, output out, utilization metrics pop right out the other side of the equation.  More utilization of the team means more productivity!  Wonderful!  Its so simple and we should have figured this out so long ago.  All that time wasted counting KLOCS and function points….

Of course it doesn’t really work. You can count hours, or days, or whatever to your hearts content but you are only measuring effort.  And, measuring effort of a software development group is an exceptionally tricky (and potentially dangerous) thing.  Its not the effort that matters, but rather the results.  So how do we measure the results?  Ahhh there’s the rub.

What we need to measure is the business value of the stories the team is being asked to build.  For the consultant this is very simple – you are paid by the hour for the consulting performed.  Thus, hours billed X hourly rate = business value.  The business value of a software going into a product is not so easily measured.  But, lets assume this is a solvable problem.  It gets even more interesting in the planning phase when you are making product choices.  For a proposed feature, what is the busniess value?  Now suddenly, this is not a software engineering question at all.

Last week, the Austin Agile development user group, AgileAustin, published a online poll of tools that teams in the local area favor for agile planning.  In response to the email announcing the poll, a couple of members emailed the list saying their favorite tool is no tool at all.  Some said index cards.  I shook my head a bit and emailed a good friend of mine who also leads development teams for his thoughts.  We came to the conclusion that it must be nice to work in a project so small you need no tools to help the team plan a sprint.

Actually, I don’t think it would be that nice.  I’d submit that if your plans are so simple that you need nothing to track them, or index cards suffice, you’re probably not doing much interesting.  For my team, Rally Software’s Rally Enterprise has been very successful in helping us plan very complex agile projects.  I recommend it without hesitation.

In many cases, our product management team enters user stories (sometimes we use epics for very large user stories) while the development, QA, and docs team breaks these down into stories and tasks.  We plan sprints as a fully integrated feature team across all these disciplines.  It’s not perfect, but it works and works well.

You can keep your index cards, thank you.

As you can tell by the frequency of my posts, things have been very busy at work – no complaints though, I know of friends and associates that would like to have such a problem. I wanted to follow up on one of the thoughts in the prior post regarding the global ramifications of effect of freely / very affordable cloud computing services.

With services like google app engine and amazon’s elastic compute cloud offering free and low cost resources that would have previously required investment of at least six zeros, the bar is substantially lowered to take a good idea to market.  To build out an idea, you now really just need a handful of expertise and time – not to mention a lot less time that you once did.

This opens competition up on a global scale never seen before – certainly there are many excellent and bright developers in India, China, Eastern Europe, Russia, Brazil, you get the picture.  The value increases for innovative ideas, domain knowledge, and the ability to market the solution.

I firmly believe America remains the cradle of innovation; it is in the very core of our society and our DNA.  I’ve had the privilege over my career to work with some of the best and brightest, and feel very blessed that I still do every day.  It is heartening to see the innovation coming out of Apple and my own company.  Still it is going to be a very different world when my daughters enter the workforce.   Change is coming and it is coming even faster than we can imagine.

Tomorrow, I have the opportunity to speak at an Rally customer success tour at the Renaissance.  Tonight, I am reflecting on the leaders I have a privilege to serve with on the panel, Israel Gat, Jack Yang, Torsten Weirch, and Eric Huddleson.  These gentlemen are all significant leaders in the Austin Community and I’m looking forward to tomorrow.

I’ve chosen the topic of “Agile in the Large” since the real power of Agile is evident when you turn loose the teams to run at their full speed in a self directed agile process.  Agile gets really interesting with development teams over 50, and also holds its most promise.  We believe that the best way to scale a team and continue to deliver successful software projects is to build it as a set of Agile teams.

Here’s few things we’ve learned along the way.  Interestingly enough, in comparing notes with the some of the development leaders at salesforce.com, it seems they have learned many of the same things.

• There’s no replacement for a great technical leader in the Agile team – effective team leads are priceless.  You should always be growing and recruiting for leaders.
• Great Product Owners are hard to find.  I have the pleasure to work with some of the best, but to scale the teams, you need more.
• An Agile team must be composed of all the skills they need to produce a shippable result.  In our case, it is the product owner or manager, usability, dev team, QA team, and tech writers.
• You run into problems when you depend on other parts of the organization that may not be Agile.  Just for an example, if you have a supporting organization that is still working on the Hero principle, or the waterfall process, there is going to be trouble.
• Tool support is critical and everyone must adopt. For large teams, roll up views and quick reporting is absolutely essential.  Rally gives us this capability.
• Test automation is absolutely key to keeping the product shippable.  As each sprint there is significant new functionality being delivered, the QA team must at the same time deliver automated tests so that doing regression testing onthe whole product is a swift process at the end of the release.  There’s just not time to regression test the whole product.

We have even been able to extend Agile to expose our feature teams to end clients through the Rally Agile tool.

Veteran’s Day was this week and I received an email from my Dad, the Colonel, that I wanted to share with everyone.  No, it has nothing to do with technology, but it has everything to do with being an American, and being grateful.  Things that I believe are in short supply these days.

“Each year on Vets day and/or Memorial day, I normally send out a greeting to my fellow veterans and loyal American friends.  For various reasons this year, I have not been in the mood and have not been very moved.  I cannot pinpoint precisely why.  I have not changed in my firm belief in this great nation and all that it stands for. Having personally spent over thirty years in support of and defense of a set of values, I am not likely to change my mindset.

What is finally getting through to me is that there are a measure if folk who claim citizenship to this nation that I dearly love who do not feel nearly as committed to it as do I.  Most of these citizens have never done anything in support of this country but rather have gone to great effort to verbally and by their actions demeaned and disgraced it.  While I have always been aware that such folk were there, I had them fixed on the fringe and that we loyal and patriotic citizens were in the mainstream.  By nature I have always been optimistic, yet for the first time, I now harbor some lingering doubts.

In awful places all over this earth good and patriotic servicemen and women stand in support of this wonderful nation, putting themselves in grave danger by their own choice.  This knowledge gives me reason for hope.  Yet, I see large segments of our population who either are ignorant of this concept of service or who chose to ignore any responsibility to defend our nation by word or deed.  When we reach a point where there are few willing to support our values but many who are willing to damn them, we fail to function as a nation and are reprehensible as a people.  As Vets, we must insure that we do all that we can to keep those who love our nation in the forefront.  We must teach well our children and grandchildren.

When the next catastrophic attack occurs, let us pray that another generation of warriors are out there like a sleeping tiger ready to come forward.  It is my belief that we will likely need those young warriors and very soon if our leadership wavers or shows weakness.  Amid the gloom, there is always cause for hope.  I wish all of my former comrades in arms a wonderful Vets day and hope that my remarks did not throw cold water on an other wise wonderful day.  I feel that we must remain very vigilant in the next few months/years to insure we do not loose our moral fabric and thus our way as a nation.”

Thank you Dad.  Thanks for the many times you risked your life for our country.  I am so proud of you.

We just returned from Smith Point Texas where we witnessed feats of ordinary heroism that warrant recording here.  Lets start with Fred, president of the volunteer fire department, and his wife Jennifer, who the next day after the storm return to Ike’s bullseye to assess the damage, and took pictures of everyone’s home and posted it on the web so that all the neighborhood could see the condition of their home.  And Fred, heeding the calls of a desperate voice far off in a quagmire of mud dredge up from the bay and deposited all over Smith’s Point by Ike, found and rescued a man who floated on a tank 11 miles from Port Bolivar.  Fred and Jennifer are a force at the Fire station every day, handing out meals to those returning to dig out.  Did I mention, Fred and Jennifer’s home is a total loss?

And Louis, tirelessly helping neighbors rework broken pipes so that they can get water flowing again in their homes.  And Ben, fixing, hauling, and building everything in sight, for anyone, even though he doesn’t have a home there.  And the countless trucks of linesmen from Virginia, North Carolina, and Michigan, who had new lines up and power flowing after a week back to the remote community of Smith Point.  And the stories go on, and on, of neighbor helping neighbor to dig out of 8 inches of mud in their homes, hauling out water heaters, washer, dryers out of yards, cutting up fallen trees, hauling away dead cattle, and searching for lost memories among the flotsam.

The American Red Cross has won everyone’s heart in Smith Point.  Their providing of meals, MREs (boy have these improved since I was a child in the Air Force), and clean water.  I am a donor for life at http://www.redcross.org.  In our little neighborhood, probably 1/3 of the houses are simply gone without a trace, boats are everywhere, the woods are full of homes demolished in Crystal Beach or Port Bolivar, no one escaped unscathed.

More than anything, the story of Smith Point is a story of a community drawn closer by tragedy and adversity.  A story of human determination and the quiet, resolve of Texans picking up the pieces, helping their neighbors and starting over.  It makes you proud to be an American.

Recently, we evaluated several Agile Project Management tools.  Having used XPlanner for years, it was time for a change.  XPlanner is great in some ways – fairly lightweight and easy to use for developers.  It really falls apart though when you start to consider multiple agile teams working together to deliver a release.  It is just really problematic to get a group view of where you are, what progress is being made by the team, and where the hot spots might be.  To accomplish this, you need to drill into the details of every scrum team and study the metrics / charts.  I even went so far as to change the source code to build a dashboard – that’s when we started to approach diminishing returns.

The other shortcoming of XPlanner is the management of the product backlog and release planning.  Yes, you can work around this, but intrinsically, the tool does not support building a backlog and then moving stories into a sprint.  Yes, this can be done, but it is arduous.  The interface also is stuck in Web 1.0 land, making data entry into a form submit after form submit affair.

So then what?  Surveying the market and talking to many of my longtime friends developing software with agile process, we quickly build the short list to replace XPlanner. We looked in detail at Rally, VersionOne, and FogBugz.  Though FogBugz had some very interesting capabilities around predicting the accuracy of estimates, it didn’t really seem to support agile planning methodologies and the scrum process.  Also, though the predictive capabilities are interesting, this really isn’t a huge benefit in my opinion if agile is really used and you know your people.

So, it was down to VersionOne vs Rally.   Both companies did extensive demos for our leadership team and key stakeholders.  Both tools intrinsically are built around the scrum agile process.  Both were priced around the same level with VersionOne being just a little less per seat, per month, but Rally matched and beat this price point in our negotiations.  The huge gaping hole in VersionOne for us was that it really didn’t assist with resource planning at all.  That is, they don’t enable you to enter the amount of available resources in terms of hours, days, etc, and then in the planning cycle show you where you are in using those hours as you take stories from the backlog and add them to the sprint.  Both tools track burndown during the sprint of course, but only Rally lets you know if you are planning too many stories in the sprint.  Even XPlanner supports this, so it is a big miss for VersionOne.  We can only assume they are working to add this capability.

Also, the rollup reporting for an entire release is more powerful and flexible in Rally.  This was a big plus for us.  To be sure, Rally isn’t super sophisticated in resource planning.  It doesn’t allow the individual team members enter their availability and then sum it up for the sprint.  (I would like this feature – I need to add this to the Rally Community.)  Rather, it just allows you to add the total number of hours available for a sprint at the beginning of the planning cycle.  How you figure this out is up to you.  After you add the total number of hours available, it shows you hours remaining as you add stories.

In coming blogs, we’ll talk more about the pros and cons of Rally as a Agile management tool.

Have you heard of reverse outsourcing? Indian IT firms that built incredible profit margins on the outsourcing boom in the West are themselves headed offshore, from Malaysia to Mexico, to escape the double sting of surging salaries and a rising rupee. Tata Consultancy, Infosys, Wipro, Satyam and smaller companies are stepping up acquisitions and opening more facilities closer to US and European clients to cut costs — the reason why work was farmed out to India in the first place.

Salaries of software professionals rose 18.7 percent in 2007, according to a survey, while the rupee has gained almost 10 percent this year to near 10-year highs against the dollar. That’s eroding the cost advantage once enjoyed by the 50 billion dollar information technology industry, which bills two-thirds of sales in dollars but whose expenses are almost all incurred in rupees.

Hyderabad-based Satyam has hired 300 mostly-Malaysian IT engineers to man the facility, whose workforce will rise to 2,000 in four years to cater to clients such as GlaxoSmithKline, one of its top 10 customers. Malaysia was chosen because of its “competitive cost environment”. The company is distributing work to locations where “it makes the most business sense.”

Mumbai-based Tata Consultancy, India’s top software maker, opened a centre in the Mexican city of Guadalajara with 500 employees and said it will employ “thousands more” in the next five years.

The problem here is that reverse outsourcing erodes many of the core, intrinsic values of outsourcing to India in the first place: a common language, British fundamentals of law, and to some extent the economic driver.

Wow. Things have changed. I’ve been working with Indian Outsourcers now for over 13 years dating back to those early projects at IBM. In the last two years, I’ve seen costs drastically increase and the availability of skills dramatically diminish. This is now clearly a trend from my chair and worth some serious examination in this blog. Too much to cover in just one post, this needs a bit deeper treatment.

In our world, software outsourcing has become an integral part of the IT industry. It has lived, grown and matured to its full potential. Companies that wish to cut costs look upon software outsourcing as a boon. One of the most important factors that attract Fortune 500 companies to outsourcing is the element of savings attached to an outsourced project. Outsourcers claim an average reported 40-60% increase in net savings.

Among the destinations chosen for software outsourcing, India stands as the most preferred, apart from China. Ireland, Romania ad Philippines also come into play. Clearly, the British colonization of India in the 17th and 18th century set off a cascade of domino that created the current IT outsourcing bonanza in this country. By introducing the English language and a western style legal system, the building blocks were laid to make the current attractive environment possible.

There are a number of major benefits of software outsourcing in India, including cost efficiency, availability of skilled workers, adoption of quality practices, and ease of doing business through the aforementioned common language and legal practices (mostly).

The winds of change are now blowing however. The first two of these advantages of oursourcing are fast disappearing. We’ll examine these in greater depth here in subsequent posts, but I’ve seen over the last year, a rapid deterioration of the cost advantage and skill availability in India. In addition to the supply / demand economics driving up costs, the rapid swings in the exchange rate are taking their toll.

For example, during two months last year, the value of the Rupee increased an astounding 8.5% in just 40 working days against the US Dollar. Attribute this to economic growth of an incredible 8 to 9%, a rate only exceeded by China.

This incredible economic success has had the result of making some Indian entrepreneurs incredibly wealthy, and deservedly so. But it has also had the effect of dramatically increasing costs and increased competition for skilled and experienced software developers. I’ve seen rates for well-qualified candidates now approach within 20% of a US hire. The days of the 3 for 1 cost advantage are no longer. We’ll dive into more detail in upcoming posts.

Ok, so your want to startup or install self-directed teams.(see a previous post on change). As with any change, this is a big challenge and there is no guarantee that it will be accepted completely by the team. Some will be uncomfortable with accepting accountability, a strong Scrum master will be needed, and some just don’t like having to fit within a structured process. In our adoption of Scrum, a few things have helped us gain acceptance:

Training We are fortunate to have a certified Scrum Master on board who has trained the whole team. A full day of training was invested in every member of the team. Additional training is needed for Scrum Masters because their execution needs to be effortless for the process to flow.Executive Buy-in and Support All levels of management must support the process. The integrity of Sprint content must be held in tact for the process to work. During the sprint, business needs sometimes require us to trade Sprint tasks for backlog items, but the integrity is maintained if the team signs up to the change.Strong Leads that are Willing to take on Scrum Master role this process of self directed team still requires someone on the team to drive the process. In my experience, a strong and effective lead developer is one of the most rare and valuable commodities in the Software Engineering world.In short, the only mistake we have made with Scrum is not more quickly adopting the process everywhere in the team. At the end of the day, team success is the greatest motivator for this or any other change.

About 18 months ago, we adopted a self-directed process called Scrum in part of our development team – the part that focuses on end user applications.  Most of the team was a little apprehensive, but they are solid folks and they put it in action. 

Recently, we made the move to transition the whole team to Scrum, something I should have done long ago.  Initially, I believed that complex pieces of server-side software had longer life cycles, longer testing cycles, and therefore perhaps the Scrum process wasn’t a good fit.  I was wrong.  What we could have seen are the same benefits on the server-side that we’ve seen on the client.  So, now we are rolling it out team-wide.  So what are some of those benefits?

The most important benefit is the placing of ownership where it needs to be – at the team level and ultimately at the individual level.  Whether they are taking a turn at Scrum Master, or just part of the team, each person owns reporting on their progress, the whole team owns the sprint backlog delivery, and has the power to say no to things not in the Sprint.

Team planning and ownership of the deliverable by the team – each Sprint (we use a month), the team signs up to a number of developer days, test days, and Tech writing days available.  We estimate the line items in the backlog and allocate the available days.  Its all team driven, and as such, the team owns the plan for the Sprint delivery.  In addition, the team demonstrates the deliverable from last Sprint – a powerful way for them to show accomplishments.

Peer-to-peer accountability – another significant benefit.  With Scrum, there is no place to hide from missed deliverables, poor quality, and lack of skill.  Any team member can take any item of the backlog and work on any piece of the system. As such, Scrum breaks down artificial barriers.  And, with daily scrum meetings on what was accomplished and planned for each day, it is pretty easy to tell if someone isn’t pulling their weight.

Forcing the business to define priorities – Scrum forces a booked plan, reconciled to business priorities at the beginning of each Sprint, monthly for us.  It provides a systematic way to pull in all the business inputs (sales, marketing, support, et al) into the planning process – and enforces that this is there only chance to get input into the development cycle until next Sprint. (Not really of course, but it does force the conversation.)

 next Blog…. Success Criteria for Adoption of Scrum

The Second Law of Software Dynamics:  shipping software is a commitment, not than an event.

Customers generally expect once they have adopted your software or service that you are going to continue to develop it, service it, respond to their requests for tweaks, and keep up with competitive forces.  In short, they made a commitment (time, dollars, training, or perhaps a key part of their business) to you, and they are expecting the same.

Yes, this applies to Web 2.0, SAS, and whatever the new paradigm of the day is. It is all just software in different cloth. It may be delivered in a different way, or have a new business model attached, but the commitment remains the same.

(What is the First Law?  Given any non-trivial system, as long as you test software, you continue to find bugs. I need to blog on when is enough, enough someday soon)

Do you watch Dirty Jobs on Discovery Channel? I have to admit, I think I’m hooked on this show. It might be the deadpan delivery of Mike Rowe, or could it be somewhat deeper? As a leader in software development, sometimes you can feel like the star in the PigWrangler episode (and no developers are not pigs). Bear with me as I take metaphorical license… Sometimes you have an uncooperative environment, and after the product goes to market, well, theres just a lot of poo to clean up. And, boy is there poo! Well, you’d just have to see the episode.

 In fact, the poo theme seems to run through many of the Dirty Jobs episodes. At the end of the day, in the midst of all the poo, there is a successful business. Art imitates life again.  In software development, there is more than a fair share of poo – changing requirements,customer promises of features that don’t exist, missed sales targets, layoffs, inept people at all levels (yes executive management too), pass-the-buckers, slackers, hackers, you name it. Face it – on any given day, we are all capable of poo. Still, out of all this comes innovation and products that people use every day. For many of us, this makes the job worthwhile, even when it’s a Dirty Job.

Lately, I’ve been interviewing to fill an open management position. Clearly these are positions of the utmost importance in that a great hire will push the team farther and higher, while a bad hire can be disastrous.

I’ve put together some question (but not all in case you’re next) I’ve used, and hope they are of help to you.What does a manger do? This will generate lots of conversation. Looking beyond the technical to an understanding of the business environment willingness to get up close with a customer and truly understand their need ability to ask the hard questions, whether with marketing/sales, or with software engineers. Above all someone who embodies characteristics of servant leadership. How do you distinguish between top performers and above average performers? What are some examples of how you have rewarded these top performers? Looking for bonuses, creative recognition, and monetary recognition that clearly separates the top performers.What has been your highest performing team? What made them that way? Looking for people who give credit rather than take credit.What has been the most difficult situation you’ve encountered in mgmt? What would you do different about it if you relived it today? Amazing what you can learn here. If they can’t think of one, they are not being honest or don’t have any real world experience!Have you ever needed to manage an employee out of the business for performance reasons? Describe this process.What gets you excited?What would your goals be for the first week on the job? (I know it is such a stock question, but its still a good one).What agile development processes have you used? Where were they successful and where were there problems?What qualities do you look for when you hire? (what I’d like to hear is: willingness to go the extra mile (commitment), examples of driving issues/opportunities all the way through closure, even outside of normal channels (ownership), demonstrated ability to innovate, someone that works well within a team and demonstrates a fair amount of personal leadership.So, how long have you had that wart? (just seeing if you are paying attention)

It just amazes me the creative and inventive ways people use to avoid goal setting.    Believe me, I’ve seen it all.  When it comes right down to it, most people would just rather not be held accountable.  And, to some degree, this is understandable.  Life is full of the unexpected – and software is a great example.  There are new technologies, new toolkits, new platforms, and just plain new stuff all the time!  Who can say in any degree of certainty, how long something will take.  If you are truly certain, then you’re probably not working on anything very interesting.

But, beware of the one who continually avoids commitment.  Enough said.

About measurement. … A goal is only worthwhile if you measure, reward, and learn from it.  If you are setting goals using a process like SCRUM, goal measurement is fairly clear.  I love this process because it provides daily feedback and peer-to-peer accountability of the type not possible to deliver hierarchically from management.  It just means so much more coming from your comrades in arms. At the organizational level, I’m a huge proponent of setting high level vision goals at a functional level each quarter of things that need to get done, and then validating these with each team.  Once each team develops the detailed goals supporting the vision goals, we can validate they’re achievable, refine them, and make them more specific. This reminds me, sometime I really need to blog on what it is like to develop software for Japanese customers.  I wonder… can my career survive that much honesty….