This is because I am nearing renewal soon and am wondering if it makes sense to have your own domain or not.

Will keep you updated.

But please bookmark http://biexplorer.wordpress.com too.

Ok, what is a stage area?
It is that part of a data warehouse where data is stored physically on disks or files (or whatever)… but as an intermediate step before loading the data warehouse / data marts. It is where activities like cleansing, de-duplication, etc take place. It is like a pit stop for a racing car before reaching the destination.

Some characteristics of the staging area are

1. accessible to and owned by ETL / DW team
2. OLAP / reporting teams do not have access to it
3. indexed very little
4. ETL developers are usually free to create / drop tables, controlled though (by the architect or modeling team)

Types of staging areas:

1. Persistent staging - stage data is not deleted, if you want to maintain history.
2. Transient staging - stage data is deleted after each ETL load

Most data warehouses have one or more staging areas, the types being either persistent or transient or both.

But should you really have a stage area? Can’t you do without it? After all these days, ETL tools are more capable of handling more data in memory fully.

Is staging necessary or is it an overhead?

I think it is very critical to have a staging area, especially if you are moving a lot of data from multiple sources. Here are a few reasons why :

1. Intermediate processing: Often you need to perform transformations, cleansing and other processing on huge chunks of data from multiple sources. It is not feasible to do all of this in memory purely (extraction from source, processing and direct load to target). We need intermediate steps where data can be stored, the ETL tool can take a breather and start again. This also helps in several other ways discussed below.
2. Auditing: Having a stage area provides a means of knowing the condition of data at certain points during the ETL cycle. If your ETL job fails mid-way, you can look at the stage tables and figure out the condition of data as it passes through the various steps. Thus it provides for easy auditing.
3. Recovery: Suppose if your ETL job fails midway, you would need to fetch the data once again from the source and repeat the entire process all over again, if you do not maintain a stage area. Having stage areas could mean that the job can be restarted from the point of failure.
4. Backup: If your source data gets overwritten, the original data is lost.. probably forever. And there is no way in which you can perform a reconciliation with the DW data against the source. But if you had a persistent stage, your data would be backed up in these intermittent databases.
5. Cleansing, synchronizing, de-duping data, etc: If you have multiple sources, stage areas are a neat method to synchronize your data, at the same time, to cleanse and conform them. (For eg, if your customer data is from 3 different sources, it makes sense to pull them all into a stage area and have then synched. This is especially true if the different sources are not available at the same time)
6. Helps reduce contention on busy source (OLTP) systems: It helps bring down the contention on the source systems. Once data is pulled, you can work on it in the stage area and get it transformed to your needs. If you were to apply these transformations on source data directly without a stage, you would be engaging the connections to the source for much longer. Thsi would impact OLTP systems adversely.
7. Increases availability of data: If one of your source systems is temporarily down, you still will have access to the data which is staged.
8. Joins are easier and faster: It is much easier and faster if the tables are in the same database schema as against different disparate databases.

Source: Several, but notably, Ralph Kimball, The data warehouse ETL toolkit.

There are several ways of estimating the effort needed to complete an ETL job. Work Breakdown Structure (WBS) is popular. And so is Function Point Analysis (FPA).

But the most widely used is the one that factors in complexity based on the understanding of things like source, target, resources on project, etc.

Though I haven’t really seen anyone use this method to perfection, it is a good place to start with. Some people argue against this method, but I see this as a complementary option to whatever method you have.

So, here is a list of points that I think would be useful when you do any ETL effort estimation. I have grouped it under 5 heads: Source, target, transformations, resources, other.

Source based:

1. No of different sources & types
2. Incremental extraction needs
3. Profiling of data sources
4. Cleansing / de-duplication dirty data sources
5. Availability of documentation / transition of knowledge of source data
6. Access control & management, if needed
7. Data volumes for unit testing

————————————————————
Stage / Target based:

1. Proper database design available (primary keys defined, right columns indexed, etc)
2. Familiarity of the team with the various source, staging and target systems
3. No of different target types
4. Truncate / append / merge options
5. Fact / dimension / stage load

————————————————————
Transformation / features / mechanisms based:

1. No of transformations and their complexity
2. Usage of reusability feature
3. Usage of features like CDC, SCD, etc
4. Need for error / failure handling and recovery mechanisms
5. Auditing and validation needs of ETL
6. Complex functions / calculations for measures
7. Parallelism / DOP / etc

————————————————————
Resource based :

1. Which is the tool used? What is the skill level of the resources (fresh, medium skilled, expert)?
2. Availability of the resources (part time, full time, etc)

————————————————————
Other :

1. Are there any people, task or decision dependencies?
2. Unit / system & integration testing needs
3. Migration / deployment needs
4. Performance and tuning needs
5. Project management / status reporting needs
6. How clear is the requirement? Is scope properly defined?
7. How well is the project being managed?
8. Have you factored in time for rework?

————————————————————

Few interesting links on this topic :
1. Vincent McBurney’s article on estimating ETL development time
2. FPA
3. FPA based estimation for a Data warehouse

What do you think? Is open source database a viable solution?

The link says (as quoted by Forrester) that the market share for

• Open source database is at $850 million
• Commercial databases is $ 16 billion

Key take aways from this article:

1. Many vendors are now open to the idea of a open source database. I personally feel small and medium sized enterprises are the ones who would be keen on it.
2. Forrester says the open source databases available today are capable of handling 80% of applications today. That to me is a surprisingly high number.

In my opinion, this is an interesting area for the future. Today we have a few reliable open source databases like

1. Postgre SQL
2. MySQL

But open source has its own advantages and disadvantages. Below are a few that I can think of.

Advantages:

1. Cheap / free to use and deploy.
2. Is almost capable of handling all small enterprise applications.

Disadvantages:

1. Very difficult to migrate from a commercial database to open source as of today. Can change soon.
2. Though database itself is cheap, sometimes the cost of maintaining it is high. And finding resources capable of working on an open source database is a challenge.
3. Post implementation support can be a challenge. But it is improving.
4. Lack of proper case studies and published implementations make enterprises skeptical of open source.
5. Awareness and confidence is poor.

As a parting thought, the article also rightly says that companies should try the open source way for small to medium sized applications that are NON-MISSION-CRITICAL and evaluate the database first. If found satisfactory, they should migrate to an open source database for their other applications.

http://optimizermagic.blogspot.com/2008/02/displaying-and-reading-execution-plans.html

Karen explains why having min and max in the same query impact performance and how to tackle the same.

I mailed someone who interacts closely with the product group in SAP - Business Objects.

As of today, BOBJ PG says, DECFLOAT datatypes are not supported by both BO Data Integrator and BO Data Services. They have taken it as a enhancement request. Probably will not be as high a priority to see them in the next release itself.

Until then, avoid DECFLOAT datatypes in DB2.

Select upper (al_lang.NAME) as jobname,
upper (al_sched_info.sched_name) as schedule_name,
al_sched_info.start_time as start_time,
al_sched_info.host_name host_server
from di_edw.al_lang al_lang full outer join di_edw.al_sched_info al_sched_info on al_lang.guid = al_sched_info.job_guid
where active = 1
and al_lang.object_type = 0
and TYPE = 0
and al_lang.object_key =
(SELECT MAX (object_key)
FROM di_edw.al_lang l
WHERE l.NAME = al_lang.NAME AND l.object_type = 0 AND l.TYPE = 0)
ORDER BY 1, 2

And here is another that might help you.