2013 Goals

Personal Health:
- Exercise 120 Minutes per Week (this and following item has been merged)
- Bicycling to train stations 2 days a week
- Bicycle in average 4 hours a week
- Go to bed before 12:00PM

Family Health:
- Towards more health diets
- Schedule family traveling to increase communication and family mentality sync
   . A trip South Australia
   . Schedule a return trip back to China
   . Schedule a trip to country side

Personal Growth:
- Atthend Melbourne Uni MBA Classes (Remove from List)
- Join 2-3 Conference at least one focus on Lean and another one focus on trending of technology
   . Joined Lean/Agile/System Thinking Camp
- A blog post a month around Agile/Lean (need some boost for this)
   . Collecting topics
   . Trying to catch up to make up 1 in month in average

- Attend lean or scrum meetup once a month (too busy to keep it)
- Attend 2-3 Conference and start spread network (need more work)

Professional Development: 
- Start to know one colleague a month out of product/support/engineering team (Started Coffee Club, it works in some way; Need to solve the problemt hat uncertainy commitment from sales teams)
- Help defining Agile culture in other departments and senior managers (Spread agile culture through the Coffee Club at the moment)
- Apply system thinking for organization changes (need more reading and study for this)


Story Points, Velocity and How we treat it

People keep asking me about story points, velocity. Can we use it for performance measurement or so? One and half year ago, I would definitly say yes. And today, I will definitly say no. Then why we need story points  and velocity. Most people will ask.

Story points will tell us the complexity of problem. Not actually time that we gona spend to finish it. And complexity of problem can depend on solutions team gona to implement. But please note, we won't be able to give "accurate" estimate because:

1. Story gona to have a change or more which will impact time consumed
2. There will be other distraction can make time consumed one or two times longer
3. We didn't spend time to find every detail of "requirement". there will be 40-70% variance there

Then hey what is planning session for? Story planning's purpose is to

1. Encourage discussion between team members. When there are difference in story points, they are encouraged to have more discussion. (But I normally skip one or 0.5 point difference, if you have different opnion, please let me know)
2. Ensure team to have enough stories to work in next 1 sprint (I normally groom and estimated 1.5 x average velocity load) - that is major reason that we should have velocity, not performance measurement

And in the same time, velocity has been used by me for estimation of project/product delivery with release burndown. I constantly find out that the result is in-constant and not really useful at beginning of project. If you have different experience, please let me know how you have done that.

What bring me to agile world

What bring me to agile world and what keep me doing these? After listening Mike Lee's talk at YOW 2012, it is true inspirint topic. The question keep poping up at my head. I feel myself lost recently with Agile.

What bring me to Agile?

First thing that I was into agile because it will bring maximum benefits to developers and cut lots of waste. At that time, I don't scrum yet. All I do is to let developer to be as everything from projet manager, architect, software engineer and tester. I started trying the concepts at my own business around 2003. I find out that developer meet customers will deliver product into customer hand in much shorter cycle time and much  higher quality (I mean mostly customer won't send any complain about product delivered).  Customer is very satified. Developers feel achievement too. And every developers in my business can do that at that time. 

Second thing is that bring maximum value to customer. When developer communicate directly with customer,   putting what and how together can fit solutions into budget and quickly deliver solutions. And mostly importantly, mentality of customers and developers are always in the same line. 

Third thing is minimizing waste. I personally believe different roles in software development have bring lots of waste into the industry. Everything changed needs to communicate. My experiements before prove that we can deliver a project a minimum time expected if we can leave all decision to developers. 

I find myself lost in words recently like continuous deployment, velocity, cycle time and so on. As complexity of system, using that to measure performance is non-sense and counter productive. 

I believe that I will continue my support of agile and lean and shall never forget about why I love agile. 


Separate Deployment and Release for Web Applications


Most organization has treated deployment as release. Of course, there are many advantages doing that. Like, it is easier to maintain and manage. And it will be very efficient for small team when one feature delivery requires time. So when new version get deployed into server, everyone will see the changes.

But when team grows, it becomes biggest problem if we still want to delivery features to customers as early as possible. There are few problems as my discovery:

  1. Feature deployed into server, but product team and sales team are not ready for end users to use yet.
  2. Some features will only bring value into certain group of customers and not bring any value to other group of customers
  3.  Some features are designed for working with certain user group but not for others yet. Release into everyone will cause
  4. Frequent change of user inteface are note acceptable to most customer although it will be liked by most of customers

What we do

Separate Release from Deployment

First way that I can thought of will be separate release from deployment. There are few pratices out there:

  1. Feature Flipping/Toggling
  2. Early Access Program
  3. Any other you are using? 
Do you have any ideas how to make it better? There must be some very good idea out there. Please let me know so that I can compare features.

Feature Flipping/Toggling

With feature flipping/toggling, you will single version of software but be able to allow certain user group to see new features like evanglist, trial users, users from certain region or country using GeoIP, users with certain behaviors.

It is easy to manage source code. But it is very hard to keep all software engineers to use feature flipping properly. If not, the code grows into a big mess. 

Early Access Program

There will be different version of software installed on server. Some users can be configured to access to newest version/some version of software.  It is hard to manage bugs found or some urgent feature needs to be in production.

The advantage of this is that you don't need to worry too much problems may be caused by feature flipping (architecture design and clean up of flags)

Why we will separate release from deployment

With separating release from deployment, we can
  • Minimize impact while doing deployment 
  • Minimize risk for deployment and release
  • Support continuous deployment without degrade too much of user experience because of frequent change
  • Customer can get feature as early as possible
  • A very good communication channel between customers and engineers
  • We still deliver much better product to most end user after some iterations with user feedback

Side Note

In both solution that I have found,  my feel is that most important part is to keep communication channel wide open between engineers and customers. The communication channel should be easily accessible for both engineers and customers.  Only with that, your team can be Agile.


Why you won't do continuous deployment


Continue deployment is very popular around world at the moment. It has been seen as one corner stone of agile practices. But I do believe it takes very long time for organization to be able deploy often. 
My view about continue deployment is that you should NOT do continuous deployment unless you have a safe net. 

Safe net means differently from business to business. But there are several things that are essential across different business types. 


Monitoring becomes more and more important for IT businesses. Without any data supporting your theory, you may make wrong decision. The decision may cost your company as highly competition from anywhere around the world with low entry requirement for an IT business. 

There are tons of article talking about monitoring. But I do believe that every business should encourage monitoring across all layers:

  • Operation/System Services including databases
  • Application Layer error
  • User Interactions
Everything has its important in helping us making right decision and fix problems as quick as possible. I believe that nothing can be 100% right unless we put them in production. With fast growing amount of data, it is not possible for teams to test most systems with scale which is same as production. Expecting fail and have a safe net to catch failure as early as possible and design a system to make problem to be fixed as soon as possible. 

If you don't have monitoring in place, do NOT do continuous deployment. 

Build Pipeline

Environments always should be first problem you solve if you want team to be agile. I have been experience in lots of companies that they are experience as following order:

1. Use product as development sandbox
2. Have staging environment and staging environment db box is used for development
3. Have continuous build and automatic acceptance testing environments
4. Then have dedicated dev, testing and staging environment
5. Then have dedicated preview sites
6. Have gateway function to select users for preview features (Not done in any company yet, only read in paper now)

Without achieving (4),  there are not enough testing before going to production. In that case, please do NOT use continuous deployment unless your system in VERY small. If you have done continuous deployment without reach (4), please let me know.I would love to know that how can you minimize risk while not having such environments. 

Testing and Quality Control

Unit testing, functional testing/integration testing and acceptance tests are essential for your continuous deployment. 

Some people out there are fans of unit testing or functional testing. I don't believe one type of test is enough for continuous deployment.  

Functional testing and Acceptance Tests will tell us whether system did what we expect. As every software engineer knows, there are many cases that you can't put into functional test cases. It tests from user point of view. But it never tests from software engineer point of view. There are lots of place that we can miss with functional testing. I have see that some one promote functional testing and they promote technical stories in the same time. It is kind of making sense because they want to cover in more granular level. 

Unit testing is essential as most agree. It give us confident in a granular level so that we will be very confident as software engineer. And another advantage is that we have static about how much coverage of testing is at the moment. It is very important thing for me. Someone says 80% is ideal line that you can have enough confidence. I would say sometimes 70% maybe enough too. And progressively working towards 80% is best way to do. But you will find that after 65%, it normally becomes very hard to do. And it is worth to write test coverage as requirement in stories from that time.  

With unit testing only, we can assure that system does what user or product team wants. And we haven't tested in an integrated environment yet. With those reasons, functional and acceptance tests are very important too.

Feature Flipping

I have not done any feature flipping in any organization yet. But I believe that feature flipping is important for continuous deployment. There are frequent requests about hide certain features for different kinds of reasons. Normally reaction at the moment, we just delay deployments. But it will prevent other important features to be released too. When team grows, the problems will occur more and more. And I am pushing feature flipping at the moment. But it may result in messy code and it needs frequent clean up are always concerns. 


Feature Team or/and Component Team for Agile Development


Question arised when senior manager want to put out a component based team structure for engineering/development team. I can see some benefits from team structure. But in the same time, I feel there is something wrong with my gut feeling. What is wrong and what would be better format? 

After a careful thought, I believe that mixed team structure will be best for current business that I work at. 

My Thought

My thought the mixed team structure will be best fit for the business. There will be three product/feature teams and 2 component teams. 

My reason is as following: 

  • There is a component requires high availability, scalability and stability which requires different skill set than just software engineering and needs to understand infrastructure. 
  • There is another component similar to a component above which requires specific attention and skill sets. They will create a platform to make entry skills much lower for product teams
  • But for fast delivery, to minimize handover waste, product teams will be formed to delivery end to end development except first one above. 
  • Also using product/feature team to deliver features will minimize complexity of dependency management
  • Product team will make handling defects easier 
  • Product team will make integration testing easier and earlier in stages as features developed in one team instead of across different teams
But there are difficulties:
  • For some business, building cross functional team is very hard
  • Discipline and architecture guide to access codes concurrently
  • A shift to share responsibility of design (I don't remember where I read an article. But consult community with member from different product teams and component teams may be an answer. Have no experience with that)
  • Developing and co-ordinating common functions
  • For other business but not mine
    • Infrastructure reuse (I do think there is innovation solution for this most of time)
    • How to build a platform to make learning of some skills easier for product/feature team as some of frequent used skills are hard to learn
I believe there are someone out there that have more experience than I do and do have different opinion on this. Please comment on this and I do believe there will be a very good debate and discussion. 


"Bureaucratic Hierarchy" or"Flat" for Start-Ups


I have been wondering for what fits better for start-ups. Will that be "Bureaucratic Hierarchy" or "Flat"? It has always impression that "flat" organization will fit more for start-ups. But at this time, which one is better in general?

As agile coach and SCRUM master for start-ups, I have been thinking the problem in a scope for Agile and Start-Ups. It is worth for me to share some thought with you guys.

For you reference, it is interesting to look at organization charts.

Bureaucratic Hierarchy

I have been experiencing "Bureaucratic Hierarchy" since I have been in IT industry since 1997.  As a software Engineer and Project manager working in China, it has been the mindset that you either move to management or you get fired after 40. The mindset was embedded in everyone who were working in IT industry at that time. From my person opinion, it has been caused by

. No one want to work more than 12 hours a day from day to day
. Software Engineer doesn't produce any value other than code
. Organization runs like traditional enterprises

.com bubble partially explain that the organization structure can't support needs of startups which includes

. Innovation
. Motivation
. New skills
. Competition of resources
. Slow response to market

Flat Organization

Every company start with small and very flat organization. But most of small companies that I have worked had  very flat structure. Every senior engineers have worked as project manager, support team members, and infrastructure team members. With new technologies, all those things are quite easy to achieve. That has caused that turn around of a feature is very quick and feature delivery is mostly quite close to what is required. 

Scale up has been concerns across different start-ups. Most of organizations has been trying to get managers from big company which were transforming organization into very "Bureaucratic Hierarchy" ones. 

But if you look out, you will find that several organization continue road with "Flat" are quite successful, like Google, Valve, Apple and others.  

In the same time, there are bad sides about "flat organization" like innovation costs are very high esp for Google. 

Problems with Startups

Any solution should start with problems. What are most frequent problems seen in Start-Ups these days? From my personal experience, they should be as following:
  1. Urge to make money or prove in good track to  make money
  2. Stay survive in such hard economics time
  3. Respond to market quickly to fullfil customer request
  4. Keep competition strength over new comers
  5. Competition about human resources

"Flat" or "Bureaucratic Hierarchy"

I am personally in favor of "Flat" organization with my strong Engineering background. And I totally believe Engineering should be ones driving company directions instead of marketing and sales because Engineering make things impossible as possible and provide unique opportunities for start-ups.

But in practice, I believe different organization may have their own version of organization structure to fit their own purpose.  My preference is put up here for debate. 

Let's have1 and 2 problems as last to discuss because the difficulty and too many variable from company to company. 

Responding to market quickly to fullfil customer request. As a start-up, making existing customer happy and keep advantage at existing market are super important at this time. Bureaucratic Hierarchy will make decision making harder and there is larger chance to miss opportunities. 

Keeping competition strength over new comers is very hard thing to do because entry requirement for IT start-ups are very low at the moment and it is very easy for new start-ups. Innovation is key strength as technology start-up to keep their strength over new comers. From my point of Bureaucratic Hierarchy will kill innovation because as my experience, there is no motivation from engineering team to do any innovation in that case.  

Competition about human resources are across world for start-ups. What to solve? Solution 1 will be making more money and have more people either by buying company or recruiting new employee. Solution 2 will be keeping team small and employee up to date. With such financial hard time, I believe more in second option as it will bring more innovation into team at the same time and it will keep cost of running very low. But for solution 1, it has been preference for companies which want to grow quickly. I don't believe in such solution because that fast growing won't bring best out of people. Communication will shut-down whatever you try with my experience. Component teams seem to be best solution. But cost in problem solving and development will grow at least in double size because hand-over required and problem investigation needs to involve different teams. In summary, I believe "flat" is better choice for solving this kind of problem. 

Let's think about "Staying Survive in such hard time". It is highly related to "making more money or prove to be able to make more money". So how can we achieve these two. My first thought will be "innovation" and "keeping focus on both product and customer".  "Flat" supports better for innovation. "keeping focus on both product and customer" is strictly fine with any organization structure. But communication failure is mostly associated with "bureaucratic hierarchy". It will be hard to keep company focused with "bureaucratic hierarchy" because of communication failures. From my point of view, costs for communication will be much higher for "Bureaucratic Hierarchy" organizations too.

So I believe "flat" organization structure is much better at this time for start-ups. 


Problems with bindec 64bits

I have experience some problem using bindec and sprintf for 64bits binary string. The only cause that I can think of that is float number issues in php. I haven't tried any solution. But I do think bcmath may help in the case.

for($j = 0; $j < 15; $j++) {
   $binary_string = str_repeat('1', 64);
   for ($i = 0; $i < $j; $i++) {
      $binary_string[$i] = '0';
   $ret = bindec($binary_string);
   $dec = sprintf("%.0f", $ret);
   echo "\n{$binary_string}\t{$ret}\t{$dec}";

   for ($i=63; $i>$j; $i--) {
      $binary_string[$i] = '0';
      $ret = bindec($binary_string);
      $dec = sprintf("%.0f", $ret);
      echo "\n{$binary_string}\t{$ret}\t{$dec}";
   echo "\n";

This piece of code will generate some problematic outcome. If you look at some set of outcome at top of each set for first 10 sets. Although binary string are different. their bindec and sprintf outcome are different. Now we are temporary changed to use 32 bits. But is there anyone out there who can explain what has happened; The outcome is as following:

1111111111111111111111111111111111111111111111111111111111111111 1.844674407371E+19 18446744073709551616
1111111111111111111111111111111111111111111111111111111111111110 1.844674407371E+19 18446744073709551616
1111111111111111111111111111111111111111111111111111111111111100 1.844674407371E+19 18446744073709551616
1111111111111111111111111111111111111111111111111111111111111000 1.844674407371E+19 18446744073709551616
1111111111111111111111111111111111111111111111111111111111110000 1.844674407371E+19 18446744073709551616
1111111111111111111111111111111111111111111111111111111111100000 1.844674407371E+19 18446744073709551616
1111111111111111111111111111111111111111111111111111111111000000 1.844674407371E+19 18446744073709551616
1111111111111111111111111111111111111111111111111111111110000000 1.844674407371E+19 18446744073709551616
1111111111111111111111111111111111111111111111111111111100000000 1.844674407371E+19 18446744073709551616
1111111111111111111111111111111111111111111111111111111000000000 1.844674407371E+19 18446744073709551616
1111111111111111111111111111111111111111111111111111110000000000 1.844674407371E+19 18446744073709551616
1111111111111111111111111111111111111111111111111111100000000000 1.844674407371E+19 18446744073709549568
1111111111111111111111111111111111111111111111111111000000000000 1.844674407371E+19 18446744073709547520


Tips for handling changes in one sprint

I have been working across several development houses. Changes have been forever topics for both SCRUM teams and no-SCRUM  teams. As a SCRUM Master, I have tried to stopped any changes into sprint. But attempts failed and it caused business loss too.Hopefully, my experience will help you guys and in the same time, please post comment if you have any suggestion or think my ideas are shit. So how have I been handling change in one sprint?

First of all, as SCRUM masters and product owners who works in startups and small development house,  we should "expect change" in all manners instead of resisting it as most of customers are very important to business growth.

Now we are expecting change, what we should do next? Should we just let changes in? Of course not,  team can't finish anything if you let all changes into team directly.
1. Let Product Owner know,  change will slow down projects and team productivity will be lower comparing with teams without changes.  But team will expect changes if it impact business values. Some items need to be exchanged from sprint backlog to backlog.
2. Working with team to teach them about proper feeling of urgency in fixing bugs
3. Have a tool and even specific tags/categories to track changes in one sprint
My personal feeling is most of bugs should be fixed straight away if it impacts customer and bugs which don't impact customers can be fixed in 48 hours. And tracking changes in one sprint will help you finding proper

I have found that for small startups, there are often requests of features or changes directly to development team. It must be stopped. And they should send requests to product owner instead of team members.

Now changes come and product owner agree to put that in current sprint for business reason. I have few tricks, you may want to use:
  • Ask requestor and product owner standing in front of backlog and move other stories out before move "changes" in
  • Ask requestor and product owner send email to business about there are some stories to be postponed because of new "changes"
After two or three sprints, normally I will have a feel how big of "changes" will be with tracking tools (even just with Excel). I have tried three ways after that:
  1. Leaving space for bugs and feature requests
  2. Leaving space for bugs and bug exchange feature request for stores in sprint backlog
  3. Still as before must exchange stories in sprint backlog for changes like bugs and feature requests
For different environment, the outcome are different. 2 & 3 are better than 1 from my personal feeling because with 1, some team may not have enough stories to work with some time because that there is no bugs and much fewer bugs.

 Now, we are here. Will we let every change in? Definitly not. Good product owners will block most of change request doesn't fit product strategy and business value. From my personal experience as project manager, I am intending to block most of changes which are not related with current sprint stories if they are in a big size. How big is big? Every team will have its own definition.

But finally, tracking tools will help you team decide what to do and what to try. Some companies with legacy software and experiences lots of bugs, first important to solve "changes" in one sprint is to increasing test coverage and focus on best engineering practice like code review, pair programming and others. Rewriting part of systems may be good option. Every team will have their own way to handle that because problems are mostly different from one team to another. And team with lots of changes as feature requests, shorting sprint length and splitting stories are good options. And of course. SCRUM Master needs to discuss with team members and product owner to find solution for that.


SCRUM/Agile/Lean Team Building for Fast Growing Startups


Fast growing startups face unique difficulties for building development and operation teams. Although I haven't had fully successful experience, I will share my experience across few different startups. There are successes and failures.


There are unique difficulties for building agile/SCRUM/Lean team for fast growing startup. The ones are on top of my list are:

(1) Uncertain about how big team will be in 3 months, 6 months, 12 months
(2) Lots of pressure from business to catch opportunities
(3) There are not enough senior team members which faciliate growth of business
(4) It is hard for business to find right candidates
(5) What is strategy for team growth, splitting team or build a new team
(6) System developed is a "Legacy" system although it just has been developed from 1-2 years ago

Of course, there are lots of others. If you want to share your experience, feel free to post here.

My Solutions

I will post my solutions and ideas here for broad discussion. And hopefully, with that, we can work together to find few best solution fits for several scenarios. Feel free to give critic and feedback. I love to hear them.

Agile Way to Find right candidate

For whatever reason it is, my personal experience with few startups are that team member growth and budget are uncertain. Most of time, recruitment will be managed by top level management who has technology exposures like General Management of Technology, CTO or others.

We have personally tried or experienced following ways:

- Start with contract and then moved as full time
- Have a graduate who gona graduate in next 6 months, and have him onboard fulltime after his/her graduation
- Interview someone and have he/she onboard if he/she is right candidate
- Buy a company for its development team members

Every way has its own advantage and dis-advantage and it fits for different purpose:

(1) Start with a contractor give business an opportunity having a look at candidate's skill set for a high pressure. It fits that there are spike for one or two major projects which have hard due date and current resource can't handle the project anymore.
(2) Have a fresh graduate or someone who gona graduate is big fun and interesting. Most of them love technology and work very hard with my experience. Most of them are willing to stay if company can offer reasonable salary package. It fits for projects which require unique skills. The advantage of this way will be low cost and outcomes are normally very good. The dis-advantage is that there are risks that developers may leave in short time and it takes time educate. If there are enough senior team members, this way will be best going forward. I have successfully mentor most fresh in past 3-4 years.
(3) Have someone full time after interview. I always am keen to recruit right employee instead an employee. For fast growing company, it should never stop looking for a best candidate.
(4) Buy a company which has unique skills that we require. There are many companies who have big budgets are doing these. The advantage is that new team members can pickup skills very quickly. Disadvantage is that costs are too much. Google even just throw product away and have team members to do other things.

The "Agile" way that I want to prompt is to try different choices and add team members one/two at once. My personal preference is (2) because most of time, the outcome is fantastic.

Cope with uncertain of team size

Uncertain of team size has been a big problem for me all the time. But it has been natural for most of startups.

There is a rule that I always follow "always solve current resource shortage" instead of "trying too hard to solve future problem".

I personally find out the "cumulative" diagram are extremely helpful to decide whether current resources can cope with workload that we have.

Senior VS Junior Candidate

For any company, it is good to have as many senior members as possible. But the reality, it is hard to rightful senior candidate for a startup working in niche market. Most of time, senior team members are familiar with other technologies which are different from the technologies you gona to use. At the moment, I am more towards to team members with 0-2 years experience which can be trained to use technologies which you are familiar with.

With my experience, a candidate without experience and lots of passion on technologies and business are far better than a candidate with more than 3 years experience and has not progressed well in career.

But the skills that you need to look out for are:

(1) Communication
(2) Over-Engineering (lots of candidate are intended to do over-engineering. It is hard to change the habit.)
(3) Be able and willing to talk with users directly
(4) Strong skills and knowledge background

Team Growth, Splitting or Build a new team

There are arguments about splitting or build a new team while team grows very quickly. I am more towards splitting team, the following are my points:

(1) The most important part for development to spread is "domain knowledge". Without splitting team, it is very hard to do so esp for startups
(2) The most concerning part for splitting team is bond and connection. If we can keep most of one team unchanged, the impact is small.

My solution for agile team growth are:
(1) Let a team grow to 8-10;
(2) Split 2-3 team members out as base for new team
(3) Try to train new team leader / SCRUM Master while team growth period

Legacy System

The startups that I have worked with have the same problem, a legacy system. There are general two ways to get test coverage up:
(1) Get test written for new jobs
(2) Write a new system from ground up and make sure that at least 95% test coverage

Difficulty of first way is the difficulty in calculate test coverage. But I am strongly towards first way as risk on re-write a system from ground up. But an alternate solution would be enforcing test cases on new jobs on old systems and write test cases on new systems to make sure test coverage if there are architecture problem in old system.

I am personally find that it is very easy to use the first method if you are able to get test coverage for existing system. Ironically, sometimes, I am able to easily calculate test coverage for one system but it is very hard to calculate for another. I just find it is very hard for most PHP systems.

If you are starting new projects and want to choose a framework, you need to choose a framework with test written.