Data Warehousing on a Shoestring Budget : TDWI

Another interesting set of articles around low budget data warehousing...

Data Warehousing on a Shoestring Budget

Blogged with the Flock Browser

Ideas on DWH Testing...

Recently I was asked about strategy on data warehouse testing.  Realizing how rarely we talk about this, I went to google for this and spend about half an hour searching.

Found the following links, which still need to be researched/analyzed further.

DM Review : Where are the Articles on Data Warehouse Testing and Validation Strategy?

A Wordpress blog : Strategies for Testing Data Warehouse Applications « Business Intelligence and Datawarehousing

DM Review : Strategies for Testing Data Warehouse Applications

Blogged with the Flock Browser

Lookup Transformations in Informatica

Lookup is a transformation to look up the values from a relational table/view or a flat file. There are two types of Lookups in Powercenter, namely;

1. Connected Lookup 
2. Unconnected Lookup

Caching is an important facet of lookup transformation planning. You need to know what kind of data you are dealing with, how frequently do you call the lookup, how frequently does the data change, what is the size of your lookup table etc. among other things. Once you use cache, the trip to database can be avoided, thereby enhancing performance.

Different types of caches can be used with lookup like static, dynamic, persistent, and shared(The dynamic cache cannot be used while creating an un-connected lookup). Each of these has its own identification. For more details, refer to Informatica Transformation Guide.

Lookup is a passive transformation, and can be used either connected or unconnected.  Typically, connected lookup is used when you want to do the lookup for all rows. When you have selective lookup requirements, its normally better to go for unconnected lookup. Unconnected Lookup can be used as if its a function call.

To read more, here is a good article -
What is lookup transformation in informatica? - IT Community

Getting Started with ActiveScaffold

Recently I returned to my rails env to do some more experiments (which might be used for a future project)

To start with, I have seen a bit of active scaffold stuff and was very impressed with it originally already.  In the meantime, rails 2 have arrived.  Quite a few things have changed and I was thinking that it might have an effect on the overall picture of active scaffold also.

However, checking up on railsforum etc brought out that not much effect has gone down to active scaffold kind of plugins from rails 2.

Starting up active scaffold is really easy as they say on its website also...

try it here... www.activescaffold.com It turned out to be really simple to initialize yourself with it. 

Searching a bit more I came across a few tutorials -

Tutorial from Active Scaffold guys

Another one from someone like me, experimenting with activeScaffold...

AkillesBlog » Blog Archive » Ruby on Rails: Experimenting with ActiveScaffold   

Blogged with the Flock Browser

Open Solaris build May 2008 ( 2008.05 ) released

Today I came across a few news items and articles here and there that talked about the release of OpenSolaris projects' May 2008 release.

Though its available for download from the open solaris home page, Sun is offering Free CD shipments for the latest release here : https://www2.sun.de/dct/forms/reg_us_2307_228_0.jsp

I already ordered my copies :)

Also, there is a nice article about bit of unix OS history and open solaris 2008.05 release review here : http://blogs.zdnet.com/BTL/?p=8703

Installing Oracle 11g on Ubuntu server 64bit on vmware server 2 beta

I have this wish to install a hardy server version on my system. Since I did not want to add another partition, so I chose to do that inside a virtual machine (using vmware). vmware supports 64bit Guest Operating system, as opposed to virtual box which does not. :(

So, I have been able to install the server edition inside the vmware machine. I have never used nfs before, but the tutorial I was following was suggesting to use it.

http://www.pythian.com/blogs/654/installing-oracle-11g-on-ubuntu-linux-710-gutsy-gibbon

However, I am having problems with -

1. Trying to expand the virtual hard disk inside vmware. I tried it, but havent succeeded yet.
2. The Oracle installer exits with a strange error message.

I have posted my issues on the ubuntu forums, and havent got any solution yet. Lets see how it goes...

Using Base64 encoding with Oracle / Perl

For basic encryption purposes, base 64 encoding can be used. In fact its so basic that it would probably be incorrect to call it encryption. Its more encoding, basically preventing the subject text from appearing in plain text.

For example, if in some tool, all the parameters are stored in repository tables, and therefore, its not very nice if the passwords are stored there in plain text. They should at least be made illegible, so that they dont appear in plain text and there is some effort spent in decoding it.

Base64 encoding is an industry standard and almost all programming languages, development toolkits provide plugins for this. For Oracle, there are pre-existing packages providing the functionality.

We can use base 64 encoding on oracle side using Oracle supplied package UTL_ENCODE. On the perl side, we could use cpan package MIME:Base64. Other languages provide different kinds of plug-ins. For example, ruby on rails has a plug in for this, Javascript also allows this to be done relatively easily.

The idea is that, such sensitive data to be entered by customers, in encoded format. So, we also dont see the plain text version. And then the program uses the encoded version, right to the point before actual data needs to be used.

For achieving this, following way of working can be used :

1. On Oracle side, the sensitive data is encoded using a program call equivalent to the following will return an encoded/decoded string for a given input string.

FUNCTION fnc_encrypt(lv_str IN VARCHAR2)
RETURN VARCHAR2
AS
BEGIN
   RETURN utl_raw.cast_to_varchar2(UTL_ENCODE.BASE64_ENCODE(utl_raw.cast_to_raw(lv_str)));
END;

FUNCTION fnc_decrypt(lv_str IN VARCHAR2)
RETURN VARCHAR2
AS
BEGIN
    RETURN utl_raw.cast_to_varchar2(UTL_ENCODE.BASE64_deCODE(utl_raw.cast_to_raw(lv_str)));
END;

Its a design decision whether to put this code in a procedure/package somewhere.

2. On the other end of the processing chain, reverse operation would have to be performed. For example, in Perl, the implementation relies on the cpan package MIME:Base64

use MIME::Base64 ();

and then use the decode/encode function as per the need...

MIME::Base64::decode($PI_CAT_PWD);
MIME::Base64::encode($PI_CAT_PWD);

vmware 2 beta on hardy

I installed vmware 2 beta on my hardy. It was a surprise and a pleasant one at that, the installation procedure did not require any tweaks at all. It worked out of the box.

Further, when I tried to connect to the web based management interface, I encountered my first problem. vmware requires a log-in with root user, and as any ubuntu user would tell you, the root user is well-hidden in ubuntu. Almost never used, my knowledge on how to set a password to root user was zero.

Again, communities came to help. This thread on vmware communities helped me through that.

http://communities.vmware.com/message/901283#901283

Its a small tip, but none-the-less important. Just run "sudo gnome-terminal" and it would open a root terminal for you. Once there, type "passwd" to change the password for root user and you are done.

now, in the browser, log in to vmware web mgmt interface with user as root, and password you chose for root user.

As a precaution, I gave my normal user admin privileges so that I dont have to login as root all the time.

Oracle wrap utility...

Oracle's wrap utility helps to hide the application logic from being visible. However, I recently discovered that it doesnt hide string literals.

Hard to grasp though, since my understanding was a bit different. I cant really think of a reason why would they keep strings etc in plain text. Perhaps to force people (like me) not to store their encryption key in a package and wrap it to achieve another level of security...

Here's the standard statement from Oracle about the limitations of wrap utility -

Limitations of the Wrap Utility

String literals, number literals, and names of variables, tables, and columns remain in plain text within the wrapped file. Wrapping a procedure helps to hide the algorithm and prevent reverse-engineering, but it is not a way to hide passwords or table names that you want to be secret.

From : http://download.oracle.com/docs/cd/B10501_01/appdev.920/a96624/c_wrap.htm#LNPLS016

I wonder if there are any tricks to achieve something like that... my investigations are still on.. will report...

xgl - System monitor bug in hardy...

On Ubuntu's latest release, Hardy Heron, if you are using xserver-xgl package for graphics, you are in for a surprise.

The system monitor carries on a bug in xgl package wrt incorrect reporting of memory usage, and shows processes using zillion bytes of memory.

I started using xserver-xgl from the times in Gutsy when Compiz would not run on my hardware without that. However, I didnt quite notice this until I involved myself with beta release of hardy.

As it happened, I raised this issue on the Ubuntu forums (more than once actually, link-1, link-2), and finally after a few discussions/posts, it was discovered that

• xserver-xgl has a few bugs
• no more active development is happening on this stream of packages
• Compiz should work without needing this package as well.

As a result, I tried removing xgl from my laptop, and voila, memory consumption numbers are from earth again.. the martian numbers have left... :)

Attaching two screenshots here to give the idea of the Martian numbers...

acid3 and webkit on hardy

I have been fascinated with the rendering of web pages by Safari. Always liked it, it has a certain edge to it when talking about the look and feel. When I read more, it turned out that the rendering engine is similar to Konqueror (somewhere I read that its actually based on open source libraries from Konqueror).

I dont have a mac so I cant really experience it first hand, but the windows release of Safari made me get my first experience.

This weekend, I was surfing net when I came across "webkit". Reading a bit more about it informed me -

WebKit is an open source web browser engine. WebKit is also the name of the Mac OS X system framework version of the engine that's used by Safari, Dashboard, Mail, and many other OS X applications. WebKit's HTML and JavaScript code began as a branch of the KHTML and KJS libraries from KDE. This website is also the home of S60's S60 WebKit development.

This made me think, if its actually open source to start with, the code must be available and may be I can compile it for my system.

Well, I didnt had to look much further... search produced this link...
http://blog.kagou.fr/post/2008/04/21/Test-Webkit-on-Ubuntu-Gutsy-and-Hardy

However, my first attempt at compilation failed, due to lack of one of the packages. libxt-dev

So, now the list of pacakges that should be installed on hardy to compile webkit would look like this :

sudo apt-get install autoconf automake libtool bison flex gperf libicu-dev libxslt-dev libcurl4-openssl-dev libsqlite3-dev libjpeg62-dev libpng12-dev libglibmm-2.4-dev libgtk2.0-dev build-essential libxt-dev

The sample browser, gives me 98 on acid3 (acid3.acidtests.org). Though very minimalistic in capabilities, this is the best score for acid3 that I could get on my system.

Here are my scores
Firefox 3 beta 5 : 71
Flock : 51
Opera 9.5 beta 2 : 78
Midori (webkit based) : 64
Webkit minimal browser (nightly build r32862 ) 98

On my windows partition as well, none of the browsers went beyong 70's. Best being opera at 71.

Linux Kernel sites...

Found this interesting site. If you want to start knowing or playing around with linux kernel, this site could be a interesting start.

Linux Kernel Newbies

Also interesting would be to watch out for this site... www.kernel.org

Blogged with the Flock Browser

vmware infrastructure web access on vista...

For those who try to use vmware server on windows vista (unfortunately I am trying these days), the vmware server 2 beta is using tomcat engine to provide web based environment.

The access control is weird to say the least. It doesnt matter if ur user has Administrtor rights on the vista box. You have to have the "Administrator" user and log in as that only. So, you got to make a new user called Administrator and then log in to Vmware Infrastructure Web access.

I found the help on vmware forums. reproducing here for benefit of myself (lest I forget) and others who might come across.

1. Logon to Vista using your usual account.
2. Launch the cmd prompt - Make sure you select, 'Run as administrator'
3. Net user administrator yourFavPassword
4. Net user administrator /active:yes
5. Switch User, or logoff
6. Logon as Administrator Password yourFavPassword
   (Your password may be different!)

And then the VI web access works... :)

making vmware work on hardy..

Had trouble originally making vmware work on my new hardy 64 bit system (behaves almost same on desktop and server variants).

Careful looking up found out two things,

• the new beta version (2) of vmware works from web based console, and could take quite some time making it work
• vmware on hardy 64 bit requires some tweaking to do before you can really get up and running with it.

Finally managed to get through with help from ubuntuforums.org. Nice fellow has posted his experience with 64 bit version, and it turned out to work exactly the same way for me...

Loads of thanks :) to http://ubuntuforums.org/showpost.php?p=4357442&postcount=10

25 reasons to use Ubuntu... :)

Well, its a very interesting collection of reasons to use Ubuntu...

Some might not be as 100% agreeable as other might be, but still most are completely agreeable...
Well done...

Found it on this blog-... here u read 25 reasons for using Ubuntu

running a 64bit guest os in vmware

I tried to install a 64bit guest os and that took much longer to realize the problem.

What I found out (may be you guys already know that) the hard way today, is that, you cant keep your virtual disks on a non-ext3 partition (I was trying to keep them on my ext hard disk - all ntfs) :(

Also, you need to switch on the VT feature from your bios to be able to actually use 64 bit guest os.. My Core2Duo supported that, so luckily I had it already with my hardware. But if you want to do that, make sure ur cpu supports it.

converting .img to .iso and mounting iso

.img files are CloneCD files created as a result of dumping contents of a Cd into a file. Very similar to .iso files which are more popular for containing cd images.

Many tools, like virtual box allow mounting .iso files as a virtual cd/dvd. Ubuntu repositories provide a tool called ccd2iso for converting the .img to .iso.

just install ccd2iso from the repositories using

sudo apt-get install ccd2iso

Once installed, just launch it for converting

ccd2iso file.img file.iso

This will create the iso equivalent of the img file.

To mount the iso file normally in the linux environment, just type

mount -o loop -t iso9660 filename.iso /mnt/iso

Experiments With Hardy...

Ubuntu is by far the more stable of the OS's.. nothing new about that.

Last friday the release candidate of the new Hardy came out. It was beta for sometime now, and I had it for a few days in beta phase too..

It was my first time playing around with an OS in its beta stage and sure.. it feels good to be involved..

I found a few things, suggested on the forums, put out bug reports and it all works so nicely. Everybody involved is so open and helping... it was actually very nice exp. A good learning curve as well for a few things... :)

Currently there are a few things still going on with the "Hardy Release Candidate" with differing opinions, but all in all it looks like a good package to me. On my laptop, it works nicely, no issues as such...

In my view, the Ubuntu series has had its share of issues with hardware compatibility. With different people using different set of hardware, it turns out that not everything is 100% picked up by the installer as it should be.

I tried using virtualbox on hardy to install another copy of hardy 64 bit to experiment something, however, virtualbox wont allow that !! People on the ubuntu forums suggest that it could be a "feature" of vbox not to support 64bit cpu as of now... That would be a shame, actually... :(

Checking out vmware has been another shock. Earlier it used to be "FREE" for Linux, however now its only 30day evaluation version. Sounds like just another corporate, not a company supporting open source...

On the 64 bit point, I am really thinking of going 64 bit completely... all out... I have tried it for a few weeks now, and for me its been no problems at all... and the support for packages etc is fine for my needs... Yeah, some more programs need to be compiled by hand, but thats ok for me... :)

Ubuntu directory structure...

In the *nix operating systems, the tree style of file systems are there from time immemorial. In fact the idea of a inverted tree directory system or all filesystems originated with Unix.

The root of the tree is called "root directory" here and is the top level directory, and all its subdirectories make up the directory hierarchy. This is different as compared to Windows wherein there are drives. Especially diff since each diff hardware device has its own dir tree with its own root directory. In Unix there is only one root dir. Everything else, falls under that tree...

Here's a summary of the standard directories found on a typical Unix/Linux system. Current reference being Ubuntu.

• /bin - binary applications (most of your executable files)

• /boot - files required to boot (such as the kernel, etc)

• /dev - your devices (everything from drives to displays)

• /etc - just about every configuration file for your system

• /etc/profile.d - contains scripts that are run by /etc/profile upon login.

• /etc/rc.d - contains a number of shell scripts that are run on bootup at different run levels. There is also typically an rc.inet1 script to set up networking (in Slackwar), an rc.modules script to load modular device drivers, and an rc.local script that can be edited to run commands desired by the administrator, along the lines of autoexec.bat in DOS.

• /etc/rc.d/init.d - contains most of the initialization scripts themselves on an rpm-based system.

• /etc/rc.d/rc*.d - where “*'’ is a number corresponding to the default run level. Contains files for services to be started and stopped at that run level. On rpm-based systems, these files are symbolic links to the initialization scripts themselves, which are in /etc/rc.d/init.d.

• /etc/skel - directory containing several example or skeleton initialization shells. Often contains subdirectories and files used to populate a new user’s home directory.

• /etc/X11 - configuration files for the X Window system

• /home - locally stored user files and folders

• /lib - system libraries (similar to Program Files)

• /lost+found - lost and found for lost files

• /media - mounted (or loaded) devices such as cdroms, digital cameras, etc.

• /mnt - mounted file systems

• /opt - location for “optionally” installed programs

• /proc - dynamic directory including information about and listing of processes

• /root - “home” folder for the root user

• /sbin - system-only binaries (see /bin)

• /sys - contains information about the system

• /tmp - temporary files

• /usr - applications mainly for regular users

• /var - mainly logs, databases, etc.

• /usr/local/bin - the place to put your own programs. They will not be overwritten with upgrades.

• /usr/share/doc - documentation.

Some content shamelessly borrowed from this page :Linux or ubuntu Directory structure

Get rid of the Ubuntu splash screen during boot

If you want to see Ubuntu booting like the good old text based linux systems, here are some ideas on removing the splash screen from the booting process....

Quickzi: Get rid of the Ubuntu splash screen during boot

Blogged with the Flock Browser