Each company has a pre-defined process for handling its BI requirements. But at the same time almost all of these companies try to avoid using it. This is the key reason why entropy arises in data-objects and reports and leads to multi-million dollar losses. Why does this happen?
Every company has a set process for handling the requirements of business intelligence outputs. When a business user needs a new report or a change made to an existing one, there's usually a process laid out that defines how to achieve such a goal. The same with new data-structures in the company’s DWH or when a change is required to the old ones. These process's guidelines usually contain information about where to find and put the documentation for the implemented changes and/or the new data-objects / reports. So here's a couple of simple questions: Why do these processes work so poorly in so many companies? And why do so many companies accept living with such a mess of undocumented reports and data-structures?
The requirement process in a nutshell
Let’s look at the process first - usually it consists of these steps:
a business user needs to create or change a report or data-structure,
the business user will draw up a business requirement concerning this need and submit it,
the requirement usually goes to the "business partner" or "business demand manager", the individual within the company responsible for the translation of business requirements into formalised BI requests,
at the BI end, an analyst takes the output from the business demand manager and elaborates what data will be impacted and (usually other analysts) prepares an assignment for a developer,
the developer implements & a tester tests,
the business user who submitted the request accepts the work and,
the product or change is deployed.
The first problem with the process I've just described is its speed. Usually it takes days or weeks for the business requirement manager to prepare a request for BI. Then, very likely there won't be any analysts available with free capacity at that moment, so there's a further delay in the BI department while they wait.
It's true, there are usually processes for the handling of priority requests to speed up their particular delivery, but they require approval from higher management and are often difficult to use.
A more serious problem is the fact that BI often modifies the initial request: the business user requires a new data-mart, dimension or report, but a BI / DWH architect can come along and point out that there's already something similar that should satisfy the business need.
In these cases the business user usually ends up waiting a long time only to be disappointed when his request is finally answered and he gets something different to what he asked for. I’m not criticising the attitude of the architect here. I fully understand he wants to avoid creating some kind of spaghetti system with lots of similar almost redundant data-objects or reports. I'm just trying to summarise how the situation is, seen from the perspective of the requestor.
The reality of BI requirements handling
When a BI department is too slow or the architectural supervision too tight, business users usually find their own way to do something. They find an external contractor and by-pass the defined requirement process.
The business requestor gets the output faster and without awkward modifications.
The contractor gets more money for his work, because he's not only doing the specified tasks he'd normally do but also the parts of the job which would ordinarily be done internally by BI.
The entropy in data-objects and reports grows. The contractor has no motivation to respect the company's architectural guidelines and when the budget is tight the documentation that details and defines the work carried out is the first to be cut.
In general, there are two possible solutions:
enforce discipline: the company adopts and implements strict rules about what must be provided by each contractor delivering data-objects or reports
make the discipline natural: the requirements are handled in a collaborative system where the analytical work and contractors' contributions can take place alongside each other and the request process.
Attempts to enforce discipline aren't usually very successful. Business users will often find ways to do things on their own anyhow e.g, by creating the data-marts they need on their own DBs outside of the DWH or by creating reports in MS Excel / Access. Often it's even difficult to find political support for such a unification of the project delivery process.
Natural discipline seems more promising. When all data-structures and reports are documented in a collaborative platform, where business requirements can be entered and elaborated with the active participation of contractors, then such an environment can create the vital documentation as a by-product of the work that's being done. It's true that even in this environment it can be difficult to find a satisfactory compromise between the requestor's practical view point and the need for architectural purity. But at least all the sloppiness can be described fully and be easily searchable. Such an approach is very close to Semanta's mindset and I'll write about it in more detail in the future.