Showing posts with label dwh-bi. Show all posts
Showing posts with label dwh-bi. Show all posts

Wednesday, December 1, 2010

Rejecting records in Fact table loads - Informatica

In some development environments, you dont have all the required dimension data and as a result, your fact loading mapping's test runs go for a toss. The mapping wont be able to load anything (reject everything) since some or other foreign key would be missing for each row.

In other words, only those records would be loaded for which ALL the foreign key constraints would be satisfied. However, in Production environment, this would almost never happen. Or even if its the case, we'd actually want those rows to be rejected.

This can work out to be a serious impediment to development/unit testing. It prevents the developer from seeing whether or not his his mapping is behaving appropriately for the happy flow functionality.

One way of achieving this can be to work using a Mapping Variable indicating the Environment. The developer can run using a value like 'D' or something, indicating a different environment than Production, with 'P' (production) being the default value.

Now in the mapping, just about where you decide to reject a record based on different type of conditions, you could put an AND condition involving this mapping variable, e.g. ....AND $$MAPP_ENV = 'P'

Now, the expression would return true only in the Production environment, and therefore would work as expected. In Dev though, this expression would return false and would not reject that row.  

Now, to be able to satisfy the db constraints for the fact table so that the row is actually inserted, you'd need to use some placeholder convention. One of the approaches can be to use an outlier value as the foreign key value.  For Example, for customer id, keep a -1 in dimension table, meaning "Undefined".  And, in all such dev cases, send -1 to the fact. 

It would serve both the purposes, tell your fact table that there is something diff about that row, and still inserting a row in there, so that the testing for the rest of the columns is not stopped because of one foreign key missing out.

Thursday, March 18, 2010

Explain plan of Infobright

We all knw that Infobright is based on MySql. However, when I started with Infobright, I was equally unaware of both.

Today I got to know about Infobright's explain plan collection technique, and realized how different it is from Oracle's.

A typical explain plan from oracle talks about the path it follows to retrieve the data, and somehow makes common sense to read a plan.

On the other hand, an explain plan from Infobright looks nothing like the Oracle's plan. Its a set of cryptic rows put together. An example of such a plan -

2010-03-18 01:31:01 [4] T:-1 = TABLE_ALIAS(T:0,"myTableName")
T:-2 = TMP_TABLE(T:-1)
VC:-2.0 = CREATE_VC(T:-2,PHYS_COL(T:-1,
A:20))
A:-1 = T:-2.ADD_COLUMN(VC:-2.0,AVG,"
avg(col1)","ALL")
A:-2 = T:-2.ADD_COLUMN(VC:-2.0,MAX,"
max(col1)","ALL")
A:-3 = T:-2.ADD_COLUMN(VC:-2.0,MIN,"
min(col1)","ALL")
A:-4 = T:-2.ADD_COLUMN(,COUNT,"count(
1)","ALL")
VC:-2.1 = CREATE_VC(T:-2,PHYS_COL(T:-1,
A:2))
A:-5 = T:-2.ADD_COLUMN(VC:-2.1,GROUP_
BY,"col2","ALL")
VC:-2.2 = CREATE_VC(T:-2,PHYS_COL(T:-1,
A:5))
A:-6 = T:-2.ADD_COLUMN(VC:-2.2,GROUP_
BY,"col3","ALL")
VC:-2.3 = CREATE_VC(T:-2,PHYS_COL(T:-1,
A:7))
A:-7 = T:-2.ADD_COLUMN(VC:-2.3,GROUP_
BY,"col4","ALL")
VC:-2.4 = CREATE_VC(T:-2,PHYS_COL(T:-1,
A:11))
A:-8 = T:-2.ADD_COLUMN(VC:-2.4,GROUP_
BY,"col5","ALL")
VC:-2.5 = CREATE_VC(T:-2,PHYS_COL(T:-1,
A:12))
A:-9 = T:-2.ADD_COLUMN(VC:-2.5,GROUP_
BY,"col6","ALL")
VC:-2.6 = CREATE_VC(T:-2,PHYS_COL(T:-1,
A:13))
A:-10 = T:-2.ADD_COLUMN(VC:-2.6,GROUP_
BY,"col7","ALL")
VC:-2.7 = CREATE_VC(T:-2,PHYS_COL(T:-2,
A:-1))
T:-2.ADD_ORDER(VC:-2.7,DESC)
RESULT(T:-2)


And, guess what, to generate a plan, you have to change a setting in the initialization parameters file, and then bounce the db. I wonder if there have been any instances when someone had to look into production database's performance and therefore needed to look at some plan.

Being as new a system as it is, I cant really complain, since such things of maturity will slowly creep in. The focus right now is probably elsewhere, building the functionality, and probably performance.

in reference to: Infobright.org Forums | Enable MySQL Logging (view on Google Sidewiki)

Thursday, January 21, 2010

availability of native drivers !!!

As I mentioned earlier, I was busy experimenting with a columnar database, InfoBright.

The experience till now is mixed, in some scenarios, the data loading is very good, as much as 40k rows per second, whereas through other channels its a poorer 500 rows per second. When I go with their built in loader, its lightening fast, but when I try from Pentaho or Informatica, its measly.

Apparantly, the drivers and the compatibility of the third party tools do play a role in attaining the performance.

The fact that they dont have any native driver published, is a huge bottleneck. Informatica dont even have a native connector for MySql, the more known cousin of InfoBright (being open source, the core engine of InfoBright is based on MySql only).

One thing came out of this experience for sure, the liking for open source got better and better. During the hunt to see the reason for slow performance from Pentaho, I even tried and managed to get the source code of the plugin (transformation) that pentaho uses for Infobright. Its pure java and it was a very powerful and awakening feeling to see the code of the item. I felt as if I have the power, the choice to make a difference, make it better. :)


I am currently exploring other options, one of them includes creating a dump file (read : csv) of the data, and then launching the command line tool to load the data into target db. I dont like it, but lets see if there is any better (read:faster) way around...

Tuesday, May 26, 2009

Tuesday, July 22, 2008

Data Warehousing on a Shoestring Budget : TDWI

Another interesting set of articles around low budget data warehousing...

Data Warehousing on a Shoestring Budget

Blogged with the Flock Browser

Ideas on DWH Testing...

Recently I was asked about strategy on data warehouse testing.  Realizing how rarely we talk about this, I went to google for this and spend about half an hour searching.

Found the following links, which still need to be researched/analyzed further.

DM Review : Where are the Articles on Data Warehouse Testing and Validation Strategy?

A Wordpress blog : Strategies for Testing Data Warehouse Applications « Business Intelligence and Datawarehousing

DM Review : Strategies for Testing Data Warehouse Applications

Blogged with the Flock Browser