Sunday, December 13, 2009

When to Scrum

I often hear companies claiming that they are much better in software engineering since they follow Scrum. How valid is that claim? Rather than challenging further, let me share some of the experience on when can an organization be effective with Scrum.

First of all, it is important to understand what problems Scrum is actually trying to solve. Scrum is mainly about making things more visible and measurable. It makes requirements more visible, progress more measurable, impedances more visible, etc... It encourages decisions to be made by the right people keeping everyone involved and interested. 

Wait... isn't this we always hear and know about Scrum? Yes... but is that the complete picture of Scrum? Unfortunately NOT! It is equally important to understand what Scrum is not! Scrum is a management framework but does not define engineering practices. What happens if your Scrum team lacks engineering practices? It will be like trying to win 'Formula 1' race, by employing Michael Schumacher to drive a bullock cart. No matter how well Michael Schumacher can drive, bullock cart is not going to perform. In short, it is not possible to drive, if engine is not up to the expectations. Scrum is NOT about the engine, but about driving it. The engine is the engineering practices followed such as how the coding is done, reviews are made, documentation is done, testing is done, resources are planned, infrastructure is managed, and this list can go on and on.... 

It is important to know that none of the above is addressed by Scrum. Many get confused by trying to understand Scrum by comparing to XP (Extreme Programming) which has defined some good engineering practices such as unit testing, continuous integration, etc. Scrum is a different concept altogether. 

For a company with proper engineering practices Scrum can make a lot of sense. But when there is no proper engineering process exist, it is quite easy to get your incapabilities hindered under the word 'agile'. Unless you have addressed software engineering processes in your company, adapting Scrum will probably have negative results. 

Do not use Scrum as a means to fix the engineering process. It is impossible. Instead you may admire the engineering value added by standards such as CMMi, ISO, etc.

[tag: 99xp ]

Saturday, November 28, 2009

Spring - the silver bullet

Last week I was interviewing couple of Java Techleads/architects from the industry. The fact that almost everybody has used Spring in most of their projects didn't come as a surprise. I remember my life as an J2EE developer many years ago. I remember we spending half the time fighting container complexities over coding business functions (Here me expressing my gratitude to EJB3 efforts back in 2005).
Still were all faults were with J2EE spec?  I agree that the EJB spec had enough pitfalls itself, but the core of the problem was more to do with the way it was used. Any technology has a place and EJB2 was no exception. It was really useful in certain challenges such as dealing with distributed databases, component distribution or in message driven applications. But unfortunately it was accepted and promoted as the de-facto blueprint for all enterprise Java applications by the giants at that time. For that reason many of the developers selected EJB to write every JEE application without much rationalizing. 
When I read Rod Johnson's second book 'J2EE development without EJB - 2004' five years ago, Spring looked appealing with the simplicity it offered. Although I never considered Spring as an alternative to EJB, it was definitely a better choice for people who just "depends" on blueprints for architecting. I agree that Spring way is better than EJB way, but does that mean it is the way forward? I used Spring for some work some years ago but I was not completely happy about it. This does not mean the principles behind Spring were bad but Spring philosophy of being a one-stop-shop for JEE distracted me a lot. Spring started to become a complex day-by-day and now it can no more be considered a light weight framework. 
Recently I revisited Rod's original Spring book "J2EE Design and Development - 2003". Here are some picks from the 'conclusion' chapter. Interestingly, the arguments Rod brings against EJB appears to be very relevant to Spring use today.
Rod: A good application is the simplest possible application
>> Don't many of us use Spring quite blindly? I have seen many techies promoting all important objects to be Spring beans and the injections to be via XML configurations. Who can really consider this to be the simplest possible application with your business logic divided in to a thousand line XML file?
Rod: Don't use EJB unless it delivers real value
>> Very same applies for Spring as well. Back on the interviews... I asked the interviewees to explain me when to use Spring beans and when to use plain Java bindings. I Hardly got an answer from any of them. Just because your framework allows you to bind at runtime, should you do that unless needed? 
My last Java project was to build a configurable search engine. I decided not to use Spring although project had much to share. I made my POJOs testable with plain setter or constructor injection patterns. It cost me writing additional constructors and setter injectors for mocking but ultimately proven to be simpler to Spring configurations. There were some instances it was really necessary to inject dependencies at runtime where I used simple reflection based plugin mechanism to inject dependencies (but this is only in handful of situations). So Don't use 'Spring' unless it delivers real value.
Rod: Try to achieve portability between J2EE application servers if
>> Spring books demote using Spring at its lowest denominator but promote making use of as much as framework features to 'empower' your application. Soon developers may find them building application tightly coupled to Spring and following an architecture laid by somebody else having no idea of your requirements. When time come to revert, Spring might have consumed you beyond rectification. So Try to achieve portability between any framework you use including Spring. 
Rod: Use J2EE: don't let it use you
>> My message is that be conscious of what you do even with Spring. Do not just follow blueprints unless your business requirements justify them. Use Spring: don't let it use you. 

Please do not interpret this post as that I do not like Spring. I love the innovations Spring folks bring in with every major release. I love using Spring as a platform when appropriate and its appropriate more often than some other platforms. It is just that I'm disturbed to see J2EE mistakes are being repeated again with Spring. 
Everything works at its best when used, not when overused!

[tag: 99xt ]

Friday, September 18, 2009

3 month projects - the real challenge for architects

We have just successfully completed an challenging "enterprise training tool" on Flex and PHP. This surface computing based training tool is expected to train 40,000 employees of one of the largest European companies. Interestingly, the product had to be completed at production quality in 3 months with 3 major subsystems namely the 'training server', 'admin console' and 'training application'.

Nowadays, is quite often the situation for most of the projects. Delivering projects in 3 months becoming increasingly common whereas considered impossible a year ago. For these projects, there is absolutely no room for architectural mistakes. Single mistake can cost the total easily.

When there is no time to waste, it is really important for architects to have a clear view on the process of architecture. In this short time, what to be performed and to which extent should be a conscious decision rather than ad-hoc.

In this post, let me share my experience on how different pieces get connected in the design phase of a project. Following diagram depicts my view of the software design and architectural process.

For the moment let's assume you have successfully completed the requirement analysis and elicitation. Once you have well defined requirements, it is important to identify the requirements with architectural significance to the project. You may read this post for get to know more on this.

Further analysis on these architecturally significant requirements will tell you the architectural qualities that need to be present to satisfy the requirement in concern. Ensure that you do not just write down a list of general qualities such as performance, reliability, etc. But make your qualities specific to the system you build and measurable too.

Once the required qualities are identified, you need to look for possible ways of realizing those. Each of the quality may be achieved through multiple different tactics. Patterns, blueprints will definitely help you in this exercise.

When multiple tactics present, you will have to compare alternatives and decide the  best suitable tactic for your requirements. Following some decision analysis and resolution method will be useful in structuring and rationalizing the decisions.

Often this selection means compromising.  Therefore each of the selection will possibly raise an architectural risk in the overall architecture. It is important that these risks are properly monitored and managed throughout the project life cycle.

Above we have looked in the process of deriving architecture (high level design) for a system. Where does the low level design (LLD) fits in this picture? Architecture is the base structure of the mini structures (LLDs). LLDs should  elaborate how the functional requirements are realized but should be placed within the rules defined in the architecture. For example, LLD for 'User Login' usecase will have to properly follow the layering demanded by the architecture.

[tag: 99xt ]

Tuesday, January 27, 2009

From requirements to architecture

Learning software architecture never flows smoothly as reading a good novel. Often novices get lost within a jungle of technical jargon such as architectural views, frameworks, styles, patterns, etc. When asked, I have seen many of them struggling just to get started due to this complexity mostly created for marketing purposes. With this post, I thought of sharing some of my experience on how to get started with architecting process when you are asked to do so.

Following is one of the definitions for the word 'architecture' by Grady Booch.
"All architecture is design, but not all design is architecture. Architecture represents the significant design decisions that shape a system, where significant is measured by cost of change." - Grady Booch

I'm selecting the above as it is much relevant to our discussion. Correctly stated by Booch, it is the high 'cost-of-change' decisions that should be agreed in the system architecture. But the question is that 'from where and how' do we derive these decisions. Keep in mind that you 'MUST' start with requirements, not with a blueprint or a whitepaper that solves another set of problems (blueprints, etc has enormous value but that is not the correct place to start with!). Look for functional/non-functional requirements to identify areas where explicit architecture/design definitions are required (I will brief you on this process in my next post). Then sort the findings in the order of  'significance'. Interestingly it is preferable to omit lower 'significance' decisions from being pre-specified for few reasons.

 - problem understanding increases over the project life cycle, hence decisions taken later are generally more accurate
 - allowing development team to take some design decisions increases the architecture buy-in and provide good design exposure
 - having lesser concerns in the specification makes it less complex but focused

A good understanding of the product road map (explicitly defined) helps to make the architecture less brittle as time passes. It is important not to reinvent every single design decision by your self. Always look around to see relevant existing patterns and blueprints. Learning from someones experience is much cheaper than learning from your own for most of the cases!

I will share more of my experiences on the matter in a future post. Interesting reader can have a look at one of my previous posts on the subject here.

[tag: 99xt ]

Thursday, January 22, 2009

Personal Firewall Development

I have been looking in to "Personal Firewall" development techniques during last couple of days for my interest. Despite the less documentation on the subject, there are several methods available for implementing personal firewalls. But unfortunately what I found was that, only a very few methods are actually suitable. Most of the articles and samples available are not really meant for serious product development.

First of all I should say that I'm not an expert on  "Windows network architecture". But let me try to explain the subject in simple terms as it is really interesting to study. Following is a diagram summarizes the layers (pale blue) and extension points (dark blue) of the Windows network stack.

One of the primary functions of a firewall is to control the network traffic by packet filtering (block ports, etc). In order to do this, firewall has to intercept the  network stack at some point. Important question is "what is the best place to do this?".

 - Any intercepts at user mode (e.g. Winsock LSP) is useless because malware can easily bypass user mode to operate at kernel mode. Windows 2000 packet filtering API is another user mode alternative but with same limitations (sample implementation).

 - In kernel mode, a popular model is to plug in to TDI layer but this is still too high in the stack. Malware can bypass TCP layer to access NDIS layer if it wishes. Also with this approach, your TCP layer is open to a hacker coming from outside. Despite these facts, several commercial firewall products seems to use this method for their packet filtering.

 - Windows TCP layer provide an extension called 'Firewall hook driver'. But according to MSDN documentation 'firewall hook driver' has some severe limitations. But apparently, Microsoft has gone against their own recommendations to use it for their own Windows Firewall. You can find a sample implementation here.

 - Another extension point of TCP layer is to use 'Filter hook driver'. Filter hook drivers are also not recommended by Microsoft due to the limitation of only a single application can use this extension on a machine. Sample implementation is found here.

With all above considerations, NDIS layer stays as the only sustainable alternative for a commercial grade firewalls. One model is to develop a NDIS intermediate driver which is the recommended method by Microsoft. But due to various compatibility and stability issues, most of the vendors have considered a different approach for their products. Rather developing an intermediate driver, they have overridden some of the NDIS functions to point at your custom functions. This approach is called NDIS hooking but mostly undocumented. Despite less documentation, NDIS hooking seems to remain as the most favored model for developing commercial grade firewalls. As no Microsoft provided API available this method is sensitive to the OS changes. Good discussion thread comparing the two methods can be found here.

What we have discussed up to now is implementation details for just one feature (packet filtering) of a firewall. There are many other features that cannot be done by tapping to NDIS but require upper level tapping as well.

One thing I have noted is that there is very little information available on the net and the available information is hard to find on this subject. I hope you have learnt something by reading this post!

Some Interesting Reads:
Windows Network Architecture
NDIS hooking sample
Article on firewall Development
Firewall-hook driver
Alternative model in Vista
Design of a ideal firewall

[tag: 99xt ]

Saturday, January 17, 2009

What makes a good software architecture?

I remember reading dozens of 'must have' attributes for a good software architecture in several design books. Sometime back I even made a blog post with a list of such attributes. Last Friday I got to rethink when I was in a argument with two of my friendly techies Sanjaya and Samudra. When asked what makes a good architecture, I stated that an architecture is good if it has two simple qualities:
You shouldn't need an architect to understand it: Just because you are an architect, you don't need to use all complex design patterns in the books. Developing a simple workable architecture is always harder than building a complex one. When a developer reads your specification, if he asks "why the hell we needed an architect to design this?", the mission accoumplished! I love these two quotes taken from Kala's blog!
- Simplicity before generality, use before reuse
- If you design it, you should be able to code it

It solves the problem in hand but nothing more: As an architect what is expected from you is to solve the problem in hand but not to build a crystal ball that solves any other problem in the world. For example, if you are asked to build a custom LOB application, you are not expected to build a 'highly extensible framework for LOB products' from you. When expected if the former and you try to build the later, often the result is a system that is too complex. Design only for what is required; if scalability beyond 100k nodes isn't a requirement, don't design for it, unlimited scalability has unlimited cost! (Dont take me wrong, if requirement is for 100k nodes, rollup your sleeves and do it!)

Solving real world problem with software is naturally a complex task; don't make things unnecessarily complex with your architecture!

[tag: 99xt ]