my technology playground

And now.. a ubuntu laptop concept that works on Solar energy .. wow...

2012-02-03T04:08:00.002+05:30

Nick Rutledge has conceptualized a thin laptop that runs Ubuntu, is beautiful and hopes to run on Solar power... isnt that a killer combination...

check out his concept here...

http://nrutledge.blogspot.com/p/ubuntu-laptop-concept.html

Data Lineage.. what is that ?

2011-10-15T01:55:00.002+05:30

It is one of those buzzwords, that keep doing the circuit every once in a while. Almost every enterprise wants to do the analysis regarding this, and is almost always hard to find people with knowledge/experience doing this kind of analysis.

For the unaware, Data Lineage is basically (really in very short words) a study of the data from its source to its eventual target, similar to what we'd do for our generation tree, we analyze the generation analysis of the data we are dealing with.

Starting from the source of the data, it travels through different subsystems, sometimes going through transformations, and thus possibly changing shape too...

Informatica had a very interesting blog post around this (already in 2007), which can turn out to be fairly informative.

http://blogs.informatica.com/perspectives/index.php/2007/05/21/data-lineage-where-did-that-data-come-from/

Informatica & hadoop... solutions for future ?

2011-09-28T02:11:00.000+05:30

Distributed computing using hadoop has taken the IT industry by a whirlwind in the last few years. After getting almost "adopted" by yahoo, hadoop has progressed quite fast, and is now maturing slowly but steadily.

More and more enterprise solution providers are annoucing their support for the hadoop platform, hoping to get a pie of the big Data business chunk. Its possibly a fair thing to expect that the leader in Data Integration business solutions space, Informatica has also announced a tie up with Cloudera, for porting Informatica platform to hadoop.

Though the exact details are yet to come out, the possibilities are endless. With hadoop (and its inherent distributed computing based on map/reduce technology), informatica can actually think of processing big data in sustainable time frames.

For one my customers, I deal with about 200 million rows of data per day in one job. Besides the issues with oracle in tuning the query etc, the informatica component itself consumes times in terms of hours. With map reduce in place, I hope to get that in minutes, oracle issues notwithstanding.

Although word about hadoop is spreading quite fast, its adoption (from buzzword to actual usage in enterprise) is not as fast. To aid their cause, Informatica and cloudera have started an interesting series of webinars, termed as "hadoop tuesdays". Its free to join, and they get experts to talk about various related issues around hadoop and big data and informatica. Its been very useful and informative so far.

Switching defaults in Ubuntu

2011-07-25T20:43:00.003+05:30

Ubuntu allows you to have multiple alternatives installed for many software.. for example, java.
You can have the default open jdk installed, and then you can actually have the Sun version installed.

For example, to see what alternatives are installed for your software, try going to /etc/alternatives. Here you'd see many pieces of software with alternatives listed.

With these software installed, you would need to point your system to use one of them as the default, this is important especially after installing a newer version of the software.

In such a case, to switch the alternatives, you need to use this

sudo update-alternatives

If you do a man on update-alternatives, there is a plethora of options to use.

For our example, to configure the default for java, use this

sudo update-alternatives --config java

writing for cloudtimes.org these days

2011-05-12T22:00:00.000+05:30

Read some of my content at http://cloudtimes.org/author/raghav/

HTML 5

2011-04-20T22:55:00.002+05:30

I attended the .WEB day of GIDS (The Great Indian Developer Summit) 2011 edition.

Among many talks, there were two focusing on HTML5. One by Scott Davis (of http://www.thirstyhead.com/) and Venkat Subramaniam (of http://www.agiledeveloper.com) . Scott's talk was more on the conceptual and capability side of HTML 5. Venkat focused more on the implementation and initiating newbies to HTML 5 coding.

Before these discussions, I would not have been able to say much on the capabilities of HTML5. It was more of a buzzword before, however, now its more of another technology holding lot of promise. I think that should say a lot for the two speakers, that within two sessions, they have been able to lift the standard of know how around a cutting edge technology from buzzword to daily use.

Both these talks, put together were able to provide a rather complete picture. Enumerate the benefits, major improvements, new tags which are bringing in so much functionality to native HTML without need of any third party libraries, plugins etc.

Of course its cutting edge today, since not all browsers support all of the HTML 5 specification. The specification is huge in itself anyway. As one of the speakers put it, the HTML 5 spec is a combination of HTML plus all of CSS 3 plus a lot of RIA functioanlities based on JavaScript libraries. One can say that html5 is rather heavy from browser engine side, however, it intends to provide all the features across the browsers (eventually). Since its a huge spec, not all browsers implement it ** completely and ** uniformly.

There would be a time when all the browsers (at least the leading ones) would implement it completely (or almost all of it), but till then, the developers would have to live with polyfill (polimorphically backfill) the html5 functionality for non supporting browsers. A javascript library at www.modernizer.com is a big help in implementing this transparently.

As Scott very aptly put it, "We'd program for the faster animal in the herd, and allow the rest of the slower ones to polyfill. As and when they catch up with the fastest one, need for polyfill will automatically go away".

From what I see in html5 spec (what ever part that I have come to know), it looks very very interesting and powerful. Lots of current functionality that is implemented today with the help of third party libraries/plugins is going to be implemented natively.

And, let me not forget to mention the one single most important innovation that is coming through with html5, semantic web. Its not really a set of tags or something similar, rather a concept. There are tags available in spec, which actually indicate the semantics (meaning) of the content. For example, there is a tag called

convert Infrmatica Session Logs to text/xml format for out of tool readability

2011-03-13T17:56:00.001+05:30

There was a situation recently when the infa repository was not able to point us to the old session logs.
However, the file system still had those files.

But the session log files are in binary format by default. If you didnt ask for backward compatible session log files, you'd get a binary format of session log file on file system. This is done to allow better importability of the session log files for infa support guys.

So, I had a few log files in binary format, and needed to analyze them.
Informatica provides a subcommand for infacmd to achieve that conversion.

The convertLogFile subcommand takes three parameters. Syntax is as follows -

convertLogFile <-InputFile|-in> input_file_name

[<-Format|-fm> format_TEXT_XML]

[<-OutputFile|-lo> output_file_name]

so, when you launch this, you can specify the input file to be converted, the format as TEXT or XML and the output file that you'd want as a result of conversion.

An example call would look like this - (expecting server on unix)

infacmd.sh convertLogFile -in /path/to/binary/format/sesslog/file -fm TEXT -lo /my/home/text/format/sesslog/file

Just found out about this amazing thing...

2011-02-22T06:15:00.000+05:30

A research initiative at Stanford University, Data Wrangler.. Wonderfully helping for analysts.

Try a demo video here -

And read more about it on http://vis.stanford.edu/wrangler

Wrangler Demo Video from Stanford Visualization Group on Vimeo.

Exadata - is it really worth the hype

2011-02-20T00:18:00.000+05:30

Well, I am not going to try to answer that, rather, more on the question side...
Recently one of my projects moved to exadata device. There was so much talk around that, the queries and db processes need not be looked into, exadata will take care of them already.

However, two things happened.. first, there was a technical talk on the device's configuration. The device turned out to be a mammoth piece of hardware. In nutshell, its a 8 node cluster, each node having 4 cpu's. Each CPU has around 2-4 GB of RAM. Then there is this high speed secondary storage which can hold a lot of cache.

The nodes are interconnected using a special switch which can transfer data faster than Gigabit networks.

With such hardware configuration, any software can claim the kind of performance gain they claim. Not to undermine the performance gains, I just want to say that the hype around the out of the world performance gains, is actually the result of better hardware, not really revolutionary software.

I, personally was expecting something of that type from Oracle, since they lack in that area. Except Teradata, there is almost no player who delivers that kind of Data Warehouse architecture and performance. and I was hoping that Oracle would do something around there and bring out something.

And, the second thing, one of the processes tried to load data to an exadata instance using informatica. Initially we left things at default so that exadata can tune it itself and we should not force anything. However, there too, exadata failed big time and couldnt put in any perf gain. At the end, all the tuning had to be done by us only.

So, the other claim of exadata regarding intelligence to pick up processes and fix them on its own also went down for us.

Though I agree that its rather new for its own evolution, i believe oracle marketing should be doing a better job.:)

Rejecting records in Fact table loads - Informatica

2010-12-02T06:37:00.000+05:30

In some development environments, you dont have all the required dimension data and as a result, your fact loading mapping's test runs go for a toss. The mapping wont be able to load anything (reject everything) since some or other foreign key would be missing for each row.

In other words, only those records would be loaded for which ALL the foreign key constraints would be satisfied. However, in Production environment, this would almost never happen. Or even if its the case, we'd actually want those rows to be rejected.

This can work out to be a serious impediment to development/unit testing. It prevents the developer from seeing whether or not his his mapping is behaving appropriately for the happy flow functionality.

One way of achieving this can be to work using a Mapping Variable indicating the Environment. The developer can run using a value like 'D' or something, indicating a different environment than Production, with 'P' (production) being the default value.

Now in the mapping, just about where you decide to reject a record based on different type of conditions, you could put an AND condition involving this mapping variable, e.g. ....AND $$MAPP_ENV = 'P'

Now, the expression would return true only in the Production environment, and therefore would work as expected. In Dev though, this expression would return false and would not reject that row.

Now, to be able to satisfy the db constraints for the fact table so that the row is actually inserted, you'd need to use some placeholder convention. One of the approaches can be to use an outlier value as the foreign key value. For Example, for customer id, keep a -1 in dimension table, meaning "Undefined". And, in all such dev cases, send -1 to the fact.

It would serve both the purposes, tell your fact table that there is something diff about that row, and still inserting a row in there, so that the testing for the rest of the columns is not stopped because of one foreign key missing out.

Setting a useful command prompt in Unix

2010-11-18T01:05:00.000+05:30

I just came across a unix system at my workplace that had a static prompt set. Basically, the prompt was just the shell executable's name and version, more like

bash-3.2 $

Well, this kind of prompt has many drawbacks, some of them i'd list here -

1. You never know (just like that) where you are in the file system. When you are dealing with multi-directory situations, you might want to stop typing pwd to figure out ur current location.

2. You never know by what user you are logged in (again, just by looking at the prompt). You'd have to run a whoami to figure that out.

3. More importantly, if you are dealing with multiple systems, this one's the most killer. You never know to what system you are logged in right now. you'd have to issue a hostname command.

Well, there might be, and for sure there are many other, consequences of having such a cryptic command prompt. And therefore, my favourite, to have a command prompt, that displays at least these three things, always, dynamically...

Something like,

raghav@deskubuntu:/homes/raghav/rails $ 

wherein, I am always aware of the three things mentioned earlier. This is very very useful when you are dealing with multiple systems and you have multiple users who are configured to run different types of processes. For example, an oracle user who is supposed to be owner of oracle processes, and an informatica user which is supposed to own everything linked to informatica, and then a connect direct user which owns the CD processes, which receives files coming in from some other system.

With this kind of system, and your own user id to log in to the system, you'd better be careful which processes you are looking at / launching and by what user. Its really very very important.

When and if you are dealing with a multiple system scenario, like dev / test / acceptance / production, you'd be better advised to use something like this only.

the magic command to do that is by setting appropriate flags and text in a environment variable called PS1.

Just set PS1 to your .profile or .bashrc (depending on your environment) file and you are set .

The example prompt that I mentioned can be achieved by saying -

export PS1="\\u@\\h:\\w \\$ "

There are many more possibilities that go with special meanings for PS1 variable. Read some of them here -

    * \$ : if the effective UID is 0, a #, otherwise a $
    * \[ : begin a sequence of non-printing characters, which could be used to embed a terminal control sequence into the prompt
    * \\ : a backslash
    * \] : end a sequence of non-printing characters
    * \a : an ASCII bell character (07)
    * \@ : the current time in 12-hour am/pm format

    * \A : the current time in 24-hour HH:MM format
    * \d : the date in "Weekday Month Date" format (e.g., "Tue May 26")
    * \D{format} : the format is passed to strftime(3) and the result is inserted into the prompt string; an empty format results in a locale-specific time representation. The braces are required
    * \e : an ASCII escape character (033)
    * \H : the hostname
    * \h : the hostname up to the first '.'
    * \j : the number of jobs currently managed by the shell
    * \l : the basename of the shell’s terminal device name
    * \n : newline
    * \nnn : the character corresponding to the octal number nnn
    * \r : carriage return
    * \T : the current time in 12-hour HH:MM:SS format
    * \t : the current time in 24-hour HH:MM:SS format
    * \u : the username of the current user
    * \s : the name of the shell, the basename of $0 (the portion following the final slash)

    * \V : the release of bash, version + patch level (e.g., 2.00.0)
    * \v : the version of bash (e.g., 2.00)
    * \W : the basename of the current working directory
    * \w : the current working directory

scripts and hash bang ( #! )

2010-11-16T00:30:00.000+05:30

More often than not, people have to tell the unix shell / perl scripts or other programs where lies their interpretor, e.g. write their command line calls as

perl SomeScript.pl

ruby ARubyProgram.rb

sh SomeShellScript.sh

this is because the system may not be aware of the location of the the executable interpreter of the exact type that needs to be used for the corresponding script. Well, for this purpose, windows has the file extension association concept, but we are dealing with Unix like systems not windows, so that option is not really available to us (besides, there are ill effects of that convention too, but lets not go in that discussion).

So, to tell a unix program where to find its interpreter, besides launching the script along with it on command line, there is another way, and rather beautiful at that.

Just put the exact path of your interpreter executable at the very first line of your script preceded by these two magic characters, a hash and an exclamation (#!) also called as hash-bang or shebang. Now, once your script is marked as executable (see chmod), you are good to go, no need of putting explicit calls to the interpreter to run your code.

Basically, your code should now look like this -

#!/usr/local/bin/perl5

print "testing hashbang with raghav"

Save this short script as aa.pl (assuming that your system has perl 5 interpreter installed in the location I used). Make the script executable (chmod) and you can just launch the script, like ./aa.pl instead of earlier example perl aa.pl

A word of caution though, these magic characters have to be absolutely the first and second character of the file, no exceptions to that. Else, the system cant make out the special meaning of this and the purpose is lost.

Pretty neat.. hunh...

A Note to New Consultants

2010-11-01T03:35:00.000+05:30

I am about to start a new role, that of a consultant at a new customer site soon. In order to prepare for that mentally, I was looking around for inspiration and advice. In the process, I stumbled across this gem from the founder of Boston Consulting Group. I have picked up the text from this webpage(http://www.careers-in-business.com/consulting/hendnote.htm), and then tried to see through it from my own eyes. All credits with the original webpage owners.

Written by Bruce Henderson, Founder of Boston Consulting Group in the 1970s

In a sense the consultant's role is a paradox. He gives advice to people of equal intelligence who have vastly superior and extensive experience and knowledge of the problem. Yet he is not necessarily an expert in anything. What is the justification for his value?

Need of Consultant

The consultant can function as a specialist or expert, In this role he must be more knowledgeable than the client. This implies a very narrow field of specialization, otherwise the client with his greater continuity of experience would be equally expert.

The consultant can function as a counselor or advisor on the process of decision making. This implies an expertise of a special kind, that of the psychotherapist. This is merely a particular kind of expertise in a particular field.

The most typical role for a consultant is that of auxiliary staff. This does not preclude any of the other roles mentioned before, but it does require a quite different emphasis.

All companies have staff capabilities of their own. Some of this staff is very good. Yet no company can afford to have standby staff adequate for any and all problems. This is why there is an opportunity for consultants. They fill the staff role that cannot be filled internally.

By definition this means that consultants are most useful on the unusual, the non-recurring, the unfamiliar problem. Outside consultants are also most useful where the problem is poorly defined and politically sensitive, but the correct decision is extremely important. Outside consultants get the tough, the important and the sensitive problems.

The natural function of a consultant is to reduce anxiety and uncertainty. Those are the conditions under which anxiety and uncertainty are greatest and where consultants are most likely to be hired.

Problem Definition
If this point of view is our starting point, then problem definition becomes extremely important.
- If the problem is incorrectly defined, then even its complete solution may not satisfy the client's perceived needs.
- If the problem is improperly defined, it may be beyond our ability to solve.
Problem definition is a major test of professional ability. Outside consultants can frequently define problems in a more satisfactory fashion than internal staff, primarily because they are unencumbered with the historical perspective of the client and the resulting "house" definition.

A consultant's problem definition is the end of the assignment if the problem is not researchable. If the problem is not researchable, then the consultant is either a specialist-expert or a psychotherapist. Neither of these roles are suitable for the use of the resources of an organization such as The Boston Consulting Group.

A researchable problem is usually a problem that should be dealt with by a group approach. Data gathering and analysis requires differing skills and different levels of experience that can best be provided by a group. The insights into complex problems are usually best developed by verbal discussion and testing of alternate hypotheses.

Good research is far more than the application of intellect and common sense. It must start with a set of hypotheses to be explored. Otherwise, the mass of available data is chaotic and cannot be referenced to anything. Such starting hypotheses are often rejected and new ones substituted. This, however, does not change the process sequence of hypothesize / data gathering / analysis / validation / rehypothesize.

Great skill in interviewing and listening is required to do this. Our client starts his own analysis from some hypothesis or concept. We must understand this thoroughly and be able to play it back to him in detail or he does not feel that we understand the situation. Furthermore, we must be sure that we do not exclude any relevant data that may be volunteered. Yet we must formulate our own hypothesis.

Finally, we must be able to take our client through the steps required for him to translate his own perspective into the perspective we achieve as a result of our research. This requires a high order of personal empathy as well as developed teaching skills.
The end result of a successful consulting assignment is not a single product. It is a new insight on the part of the client. It is also a commitment to take the required action to implement the new insights. Equally important, it is an acute awareness of the new problems and opportunities that are revealed by the new insights.

We fail if we do not get the client to act on his new insights. The client must implement the insights or we failed. It is our professional responsibility to see that there is implementation whether we do it or the client does it.

Much of the performance of a consultant depends upon the development of concepts that extend beyond the client's perception of the world. This is not expertise and specialization. It is the exact opposite. It is an appreciation of how a wide variety of interacting factors are related. This appreciation must be more than an awareness. It must be an ability to quantify the interaction sufficiently to predict the consequences of altering the relationships.

Consultants have a unique opportunity to develop concepts since they are exposed to a wide range of situations in which they deal with relationships instead of techniques. This mastery of concepts is probably the most essential characteristic for true professional excellence.

A successful consultant is first of all a perceptive and sensitive analyst. He must be in order to define a complex problem in the client's terms with inadequate data. This requires highly developed interpersonal intuitions even before the analysis begins.

His analytical thinking must be rigorous and logical, or he will commit himself to the undoable or the unuseful assignment. Whatever his other strengths, he must be the effective and respected organizer of group activities which are both complex and difficult to coordinate. Failure in this is to fall into the restricted role of the specialist.

[raghav] The first time I have read that a specialist role can be restrictive, and honestly, when you think about it again, it does come back as a correct statement, specially in the wider world of other opportunities. Specially for a management consultant.

In defining the problem, the effective consultant must have the courage and the initiative to state his convictions and press the client for acceptance and resolution of the problem as defined. The client expects the consultant to have the strength of his convictions if he is to be dependent upon him. Consultants who are unskilled at this are often liked and respected but employed only as counselors, not as true management consultants.

The successful professional inevitably must be both self-disciplined and rigorous in his data gathering as well as highly cooperative as a member of a case team.

The continuing client relationship requires a sustained and highly developed empathy with the client representative. Inability to do this is disqualifying for the more significant roles in management consulting.

In other words, the successful consultant:

Identifies his client's significant problems;
Persuades his client to act on the problems by researching them;
Organizes a diversified task force of his own firm and coordinates its activity;
Fully utilizes the insights and staff work available in his client's organization;
Uses the full conceptual power of his own project team;
Successfully transmits his findings to the client and sees that they are implemented;
Identifies the succeeding problems and maintains the client relationship;
Fully satisfies the client expectations that he raised;
Does all these things within a framework of the time and cost constraints imposed by himself or the client.

my experiments with solr :)

2010-10-01T08:30:00.000+05:30

Its a catchy title, but yes.. thats what I am going to talk about...

I came across hadoop, when I was looking for a new solution for one of our in-house projects. The need was quite clear, however, the solution had to be dramatically different.

The one statement we received from business was, "We need an exceptionally fast search interface". And for that fast interface to search upon they had more than a hundred million rows worth of data in a popular RDBMS.

So, when I sat about thinking, how to make a fast search application, the first thing that came to my mind was, Google. Actually, whenever we talk about speed or performance of web sites, Google is invariably the first name that comes across.

Further, Google has a plus point that there is always some activity at the back end to generate the page or results that we see, its never static content. And, then, another point, Google has a few trillion pieces of information to store/index/search whereas our system was going to have significantly lower volume of data to manage.   So, going with that, Google looked like a very good benchmark for this fast search application.

Then I started to look for "How Google generates that kind of performance". There are quite a few pages on the web talking about just that.   But, probably none of them has the definitive/authoritative view on Google's technology or for that matter the insider's view on how it actually does what it does so fast.

Some pages pointed towards their storage technology, some talked about their indexing technology, some about their access to huge volumes of high performance hardware and what not...

For me, some of them turned out to be genuinely interesting, one of them was the indexing technology. There has to be a decent indexing mechanism to which the crawler's would feed and the search algorithms hit. The storage efficiency is probably the next thing to come in the play. How fast can they access the corresponding item ?

Another of my observation is that, the search results (the page mentioning page titles and stuff) comes real fast, mostly less than 0.25 seconds, but the click on the links does take some time. So, I think it has to be their indexing methodology that plays the bigger role.

With that in mind, I sat about finding what can do similar things and how much of Google's behaviour they can simulate/implement.

Then I found Hadoop project on apache (http://hadoop.apache.org/) which to a large extent reflects the way Google kind of system would work. It provides distributed computing(hadoop core), it provides a bigTable kind of database (hbase), provides map/reduce layer, and more. Reading into it more, I figured out that this system is nice for a batch processing kind for mechanism, but not for our need of real time search.

Then I found solr(http://lucene.apache.org/solr/), a full text search engine under Apache Lucene. It is a java written, xml indexing based genuinely fast search engine. It provides many features that we normally wish for in more commercial applications, an being from apache, I would like to think of it as much more reliable and stable than compared to many others.

When we sat about doing a Proof of Concept with it, I figured out a few things –

•    It supports only one schema, as in, rdbms tables – only one. So, basically you would have to denormalize all your content to fit into this one flat structure.
•    It supports interactions with the server interface only through http methods be it the standard methods get/put etc or be it REST like interfaces.
•    It allows you loading data in varying formats, through xml documents, through delimited formats and through db interactions as well.
•    It has support for clustering as well. Either you can host it on top of something like hadoop or you can just configure it to do it within solr as well.
•    It supports things like expression and function based searches
•    It supports faceting
•    Extensive caching and “partitioning” features.

Besides other features, the kind of performance without any specific tuning efforts made me think of it as a viable solution.

In a nutshell, I loaded around 50 million rows on a “old” Pentium-D powered desktop box with 3 GB RAM running ubutnu 10.04 server edition (64 bit) with two local hard disks configured over a logical volume manager.

The loading performance was not quite great. Though its not that bad either. I was able to load a few million rows (in a file that was sized about 6 GB) in about 45 minutes when the file was on the same file system.

In return, it gave me query performances in the range of 2-4 seconds for the first query. For subsequent re-runs of the same query (within a span of an hour or so), it came back in approx 1-2 milliseconds. I would like to think that its pretty great performance given the kind of hardware I was running upon, and the kind of tuning effort I put in (basically none – zero, I just ran the default configuration).

Given that, I wont say that I have found the equivalent or replacement of Google’s search for our system, but yeah, we should be doing pretty good with this.

Although there is more testing and experimentation that is required to be able to judge solr better, the initial tests look pretty good.. pretty much in line with the experiences of others who are using it.

Business & Open Source - How both can benefit

2010-09-13T07:53:00.000+05:30

I had the opportunity to scout for a new technology/solution for one of our in-house projects. Quite a few of the options that I looked for were from open source arena. And amazingly, the products were far more capable from our expectations, just that we'd have to pitch in with some effort to get it working for us.

I have always felt that for open source projects/products to become commercially viable for a business enterprise, the enterprise has to come up and spend some resources to it to get the actual value out of it.

In other words, if an organization wants to use an open source product, which has an equivalent competitive commercial product available in market, they should be open enough to have their own in-house people who can take ownership of the installation. The organization shouldn't completely rely on the support available from the community forums and such.

I have seen more than one manager complain about the lack of support on the open source products. Had there been proper support system for each of the open source products, we'd see a lot of stories similar to mysql's model or pentaho model.

What I would like to see perhaps is that the organizations' becoming mature enough in their adaptation of the open source products. By that, I expect them to have a open vision, have people who understand and like and own the product, and at the same time tweak and tune the product to suit the organization's business needs.

In the process, the organization should contribute to the product's development cycle. This could happen in many ways, bug fixes, contribution of new features, the employees could contribute on community forums and such. Using the terminology from peer to peer sharing, only leechers dont help a torrent, people need to seed to it as well. Same way, unless organizations contribute to an open source product, they would stand to become only leechers.

Only after we have a decent balance of organizations using and contributing to the open source products, we'd see the ecosystem flourishing...

Tips for brainstorming...

2010-09-09T18:31:00.001+05:30

Interesting read, from both positive and negative viewpoints -

1. Use brainstorming to combine and extend ideas, not just to harvest ideas.

2. Don't bother if people live in fear.

3. Do individual brainstorming before and after group sessions.

4. Brainstorming sessions are worthless unless they are woven with other work practices.

5. Brainstorming requires skill and experience both to do and, especially, to facilitate.

6. A good brainstorming session is competitive—in the right way.

7. Use brainstorming sessions for more than just generating good ideas.

8. Follow the rules, or don't call it a brainstorm.

Read more here - http://www.businessweek.com/innovate/content/jul2006/id20060726_517774.htm?chan=innovation_innovation+++design_innovation+and+design+lead

in reference to:

"8. Follow the rules, or don't call it a brainstorm."
- Eight Tips for Better Brainstorming (view on Google Sidewiki)

Big help...

2010-09-08T17:29:00.001+05:30

I wanted to get my table sizes in infobright, and this page came to my help...

SELECT table_schema,table_name,engine, table_rows, avg_row_length,
(data_length+index_length)/1024/1024 as total_mb,(data_length)/1024/1024 as data_mb,
(index_length)/1024/1024 as index_mb, CURDATE() AS today
FROM information_schema.tables
WHERE table_schema='mySchemaName'
ORDER BY 7 DESC

Thanks Ron...
in reference to: Calculating your database size | MySQL Expert | MySQL Performance | MySQL Consulting (view on Google Sidewiki)

I also feel like saying, 1984...

2010-08-25T09:45:00.001+05:30

This story appeared in Economic Times, wherein Apple claims to have developed (or is busy doing that) sensitive info about an iphone user. Subsequently, Apple intends to hold/halt usage of the iphone device from the "unauthorized" user, this unauthorized reportedly includes -

1. an iphone that has been hacked to work outside the contract with which it was sold, read "jailbroken"

2. an iphone that is perhaps being used by someone other than the person who registered the first heartbeat or facial recognition info..

Apple intends to capture the phone location using GPS/other tech and perhaps control the device remotely if they feel its being used "unauthorized"..

i agree with people who remember 1984 after reading apple's intentions...ha.. time does come back...George Orwell.. were u too right ??
in reference to: Apple to make iPhone theft-proof - Hardware - Infotech - The Economic Times (view on Google Sidewiki)

Country General Mood using Tweets

2010-08-02T14:23:00.001+05:30

Well, it sure is pretty fascinating to do that kind of study and come back with results as commonsensical as we see here...

http://www.iq.harvard.edu/blog/netgov/2010/07/mood_twitter_and_the_new_shape.html

I quote - (with all credits where its due, none to me...)

A group of researchers from Northeastern and Harvard universities have gathered enough data from Twitter to give us all a snapshot of how U.S. residents feel throughout a typical day or week.

Not only did they analyze the sentiments we collectively expressed in 300 million tweets over three years against a scholarly word list, these researchers also mashed up that data with information from the U.S. Census Bureau, the Google Maps API and more. What they ended up with was a fascinating visualization showing the pulse of our nation, our very moods as they fluctuate over time.

The researchers have put this information into density-preserving cartograms, maps that take the volume of tweets into account when representing land area. In other words, in areas where there are more tweets, those spots on the map will appear larger than they do in real life.

A apparantly public domain result of the analysis is available here -
http://cdn.mashable.com/wp-content/uploads/2010/07/twitter-moods.jpg

Oracle Count(1) vs Count(*)

2010-07-28T11:21:00.001+05:30

Well, it might have been an everlasting discussion about which one of these to use, count(1) or count(*).

I guess, this article of Thomas Kyte already clarified the situation long long ago (well, for IT industry 2005 is long ago anyway, especially given the speed at which we are moving.)

Essentially, what askTom says that, count(*) is better than count(1) since count(1) translates to count(*) internally anyway. I wonder then, why would someone want to use count(1) anyway.

There is at least one more step involved in getting to the actual result. And there is another possible tweak, count(1) has to evaluate an expression as well, "count(1) where 1 is not null". Though its a tautology equivalent, it has to be evaluated nonetheless.

Further, there was some misconception about how the result is returned, whether its read from the data dictionary, this view or table or something like that. I dont think so. The result is calculated at the exact run time,when the query is run, and it actually goes ahead and counts the records in the table.

Should set the record straight...

in reference to: Ask Tom "COUNT(*) Vs COUNT(1) on tables with CLOB..." (view on Google Sidewiki)

Developing a Rails application using an existing database

2010-07-24T00:46:00.000+05:30

This is the latest challenge for me. A database exists with real data in there, and I have to develop a rails application around that.

Initially we needed the basic CRUD screens for some tables. Being lazy (i m really proud of that), I set out finding if there a solution that generates the forms (read views) for the existing tables/models.

I have already managed to generate models/schema.rb using another gem. This is called magic_model. Read more about that here

Then google helped me find this another gem called scaffold_form_generator which generates the necessary views/forms for a given model. However, there need to be some improvements required on that (I think). perhaps I would contribute something (if I find out enough on how to do that)

Well, for the moment, I am struggling with handling of the missing special meaning column from the legacy tables. Will continue writing on this...

Making rails talk to SQL Server Express Edition

2010-07-21T16:30:00.000+05:30

I had zero exposure to sqlserver before today. With the help of a dba and google :) I finally achieved the communication. Here's what I did to achieve this -

Make an odbc connection - pointing to the db server, preferably make a sql server authentication user

Install the following gems

activerecord-odbc-adapter (2.0)
activerecord-sqlserver-adapter (2.3.8)
dbd-odbc (0.2.5)
dbi (0.4.5)
deprecated (2.0.1)

Further, I copied odbc.so and odbc_utf8.so files from http://www.ch-werner.de/rubyodbc/i386-msvcrt-ruby-odbc.zip to .../ruby/1.8/i386-msvcrt even though exact directory was not found in the path. So, the files were eventually copied to ...\Ruby187\lib\ruby\1.8\i386-mingw32

My database.yml file looks like this -

development:
adapter: sqlserver
mode: ODBC
dsn:
username: myUserName
password: myPassword

winscp vs cuteftp

2010-07-16T10:54:00.001+05:30

I recently had an experience of transferring relatively larger files from a windows box to linux one.

I had some files which were ranging from 2.1 GB to 4.x GB. The company traditionally uses cuteftp to transfer (read ftp) files across servers. So I started with that anyway.

However, all my transfer attempts were failing. After transferring 2 GB of data, cuteftp would end the transfer and give away some or other error message. I figured that this was happening only to files which were larger in size. And this transfer consumed more than hour or so before it failed.

Google couldnt help a lot, so, finally i got down to winscp. And, it worked out so well. Not only the transfers didnt fail, but they finished within minutes, instead of the hours and hours spent by cuteftp.

Perhaps its some setting somewhere in cuteftp that I couldnt find, but the whole experience has left me more inclined towards the open source community.

Thanks a ton guys .. :)

in reference to: Download WinSCP from SourceForge.net (view on Google Sidewiki)

Parallel Querying using Oracle

2010-05-13T12:45:00.001+05:30

Perhaps a long awaited feature, this one is clearly the one that I like most.

Using pre-built packages its now possible to break a single (relatively heavy) operation into multiple small operations, which run in parallel. The benefits are obvious, and cant be ignored.

I am yet to ascertain the usability of such a feature with tools like informatica or pentaho, but I am quite sure a lot can be achieved, especially in the direction of updates to huge tables in data warehouses.

And looking at the implementation, its genuinely simple and straightforward.

Would be nice to see more applications utilizing the benefits of such features..

Read more on Oracle magazine, article written by Steven Feuerstein...

in reference to: PL/SQL Practices: On Working in Parallel (view on Google Sidewiki)

Ubuntu 10.04 is here.... Upgrade today...

2010-05-06T09:58:00.001+05:30

Its only been a few hours since I have had the pleasure to upgrade to the latest version of Ubuntu.

And without any doubt, or any detailed analysis/investigation I can say this much for sure -

- The upgrade process was super easy. No hassles whatsoever, you just need to have a decent internet connection, few times you need to confirm the suggested decision, and done...

- The new system looks cleaner, much much cleaner. Its neat, to some extent beautiful. Its not about the background or themes or anything like that, but the overall look and feel is genuinely cool. Perhaps its to do with one thing that I have done on my part. My system font is "Lucida Grande" all across. And, for some reason the shapes and looks on this font are far better than anyone else I have encountered.

- The boot time has come down a good notch.

Well, I guess this should be a good starting point for my review on the new version, lets see how it comes across further.

in reference to: my tech playground (view on Google Sidewiki)

Implementing Cartesian Product in Informatica Mapping

2010-05-01T02:02:00.000+05:30

As against pentaho, Informatica doesnt provide a ready made transformation for implementing cartesian product in a mapping. Although, most of us would agree that its not often that we tend to go for cartesian product joins. [the instinct generally is to do enough to avoid a cartesian product, because its a performance killer in general]

However, when your requirements need this, there is no direct way to do it in informatica joiner transformation. Either you do it in the db side, by overriding your source qualifier sql statement and building it in there.

However, I have seen that some designers dont like to override sql statements, in such cases you'd have to implement it inside the mapping only. Here's a workaround for achieving that. Here goes -

Read both the sources using their own source qualifiers, normally.
For both of them, put in an Expression Transformation after the source
In both the expression transformations, create an output port with a constant value. For Example, call it dummy1 for stream 1 and assign it a value -1. Similarly, a port would be created in the second pipeline, lets call it dummy2 and assign it a value -1.
Now create a joiner transformation. Link ports [including the one that we created with a constant value] to the joiner from both the expressions.
In the join condition, choose to compare the dummy columns.
The rest of the joiner configuration would have to be like any other joiner. Nothing specific.

You might want to keep the smaller source as the master in the joiner, since it would save on the caching.

Before implementing the above solution, be sure to go back and check if its actually required for your application to have cartesian product !!!!

Hard vs Soft Parse in Oracle...

2010-04-29T13:42:00.001+05:30

Was reading on asktom, came across this response from Tom, that seemed so direct and clear...

So, shamelessly copying.. :)

[From AskTom]
Parsing

This is the first step in the processing of any statement in Oracle. Parsing is the act of breaking the submitted statement down into its component parts ? determining what type of statement it is (query, DML, DDL) and performing various checks on it.

The parsing process performs two main functions:

o Syntax Check: is the statement a valid one? Does it make sense given the SQL grammar documented in the SQL Reference Manual. Does it follow all of the rules for SQL.

o Semantic Analysis: Going beyond the syntax ? is the statement valid in light of the objects in the database (do the tables and columns referenced exist). Do you have access to the objects ? are the proper privileges in place? Are there ambiguities in the statement ?

For example if there are two tables T1 and T2 and both have a column X, the
query ?select X from T1, T2 where ?? is ambiguous, we don?t know which table to get X from. And so on.

So, you can think of parsing as basically a two step process, that of a syntax check to
check the validity of the statement and that of a semantic check ? to ensure the
statement can execute properly. The difference between the two types of checks are hard for you to see ? Oracle does not come back and say "it failed the syntax check", rather it returns the statement with a error code and message. So for example, this statement
fails with a syntax error:

SQL> select from where 2;

select from where 2

*

ERROR at line 1:

ORA-00936: missing expression

While this statement failed with a semantic error ? if the table NOT_A_TABLE existed and we had permission to access it, this statement would have succeeded:

SQL> select * from not_a_table;

select * from not_a_table

*

ERROR at line 1:

ORA-00942: table or view does not exist

That is the only way to really tell the difference between a semantic and syntactic error ? if the statement COULD have executed given the proper objects and privileges, you had a semantic error, otherwise if the statement could not execute under any circumstances, you
have a syntax error. Regardless ? Oracle will not execute the statement for you!

The next step in the parse operation is to see if the statement we are currently parsing has already in fact been processed by some other session. If it has ? we may be in luck here, we can skip the next two steps in the process, that of optimization and row source generation. If we can skip these next two steps in the process, we have done what is known as a Soft Parse ? a shorter process to getting our query going. If we cannot, if we must do all of the steps, we are performing what is known as a Hard Parse ? we must parse, optimize, generate the plan for the query. This distinction is very important. When developing our applications we want a very high percentage of our queries to be Soft Parsed ? to be able to skip the optimize/generate phases ? as they
are very CPU intensive as well as a point of contention (serialization). If we have to
Hard Parse a large percentage of our queries, our system will function slowly and in some cases ? not at all.

The way this sharing of SQL in Oracle is accomplished is via the shared pool, a piece of memory in the SGA maintained by Oracle. After Oracle parses the query and it passes the syntax and semantic checks ? it will look in the shared pool component of the SGA to see if that same exact query has already been processed by another session.

Since it has performed the semantic check it has already figured out:

o Exactly what tables are involved
o That we have access to the tables (the proper privileges are there)

And so on. Now, it can look at all of the queries in the shared pool that have already been parsed/optimized and generated to see if the work has already been done.

[Raghav]

As has been discussed here, the soft parsing of queries makes our application faster, since it doesn't have to do everything all over again (from parsing point of view). Bind variables come to help in this case.

For example, a query like the following -

select a,b from tbl1 where c = 2333;

would be parsed and stored under a different identifier than

select a,b from tbl1 where c = 58763;

whereas, essentially they are same queries only, the literal value being different for the where condition.

In this case, the second query would go for hard parse, instead of our expectation of a soft parse. So, what could be done to tell oracle to think that these are actually same queries and there is no need to do the hard parse again. Well, you can do that by using a bind variable, in the sense, removing the only differentiating part of query by a runtime replacement.

Look at the following query -

select a,b from tbl1 where c = :bb;

now, this query can be used for both the examples above, at the runtime (after parsing), the variable :bb would be replaced by the respective literal value, and you have your answer. However, this gains a lot of performance, since, the second run goes for a soft parse, instead of a hard parse, so, every subsequent run, after the first one, is re-using the parsing information collected the first time.

So, simple recommendation, use bind variables wherever you see a query being reused with different literal values, and use run time substitution of the literals.

AWS Application Demo

2010-04-29T01:02:00.000+05:30

Amazon started with S3 (Simple Storage Service) - purely storage service.
For Computing purposes, Amazon started EC2
EC2 allows hosting your application on virtual servers operated by Amazon, widely known as AWS.
For calculating the availability as a resource, 1 compute unit at Amazon EC2 is roughly equivalent to a 1.2 GHz Xeon server.

All computing resources are virtualized, none is physical. No physical details are ever published. However, the user has to manage his own file system. Based on need/request, disk space is mounted and is made available to the server. Further, it has to be managed by the application.
There can be three types of instances,

Small
Medium
Large instances

Behavior on instances

like a normal web server
has a public web address

Java Software available to convert command line instructions into SOAP wrapped API calls to AWS
Need to download API tools from AWS
Key pairs are tied to regions

The benefit of this is that the servers are replicated across data centers located physically separately

Autoscaling service

Allows scaling of servers based on need
Consumes about 60-90 seconds to scale up based on the need (e.g. load on the server). If configured the appropriate way, the system will replicate the application code and launch new server instances within seconds to handle the extra load.
Handles scaling up as well as down both at the same time. Automaticaly, the extra/unused instances will be shut down and released from the application deployment when there is no load to require their service.
very useful in cases of spikes, specially high-peaks. New feature launches on sites, or sudden outbreak of news etc, cause spikes in server loads. At such times, AWS works perfectly to scale up the required computing power. The application owners are saved from buying extra servers for that 1 day load... :)
Its possibel to say how many servers to add
Such a scaling up/down can be configured. For example, it can be specifiied that request for new instances when current system's load exceeds 85% or scale down, when the usage goes below 20% used

OSs supported

Linux
Windows
openSolaris

Buzzwords

AMI = Amazon Machine Images
EBS = Elastic Block Device
Elastic IP = static ips assigned to instances
Amazon EC2 = Amazon Elastic Computing Cloud

Web Application Debugging Tools

2010-04-27T00:16:00.000+05:30

A Talk by Matthew McCullogh

Some tools he talked about and demonstrated

1. tcpdump
      Captures tcp-ip packets during traffic
      Available as an open source project at http://www.tcpdump.org/
      Allows you to save the captured packet information into a file on disk, thereby allowing offline analysis.

tcpdump -i en1 -s0 -n -A

2. WireShark - visual interpretation of tcpdump captured data
   Once you capture packet information using tcpdump, you can get a better view of it, more readable using this tool. Again an open source project, available from http://www.wireshark.org/

3. netstat
An all time *nix flavors favourite. Every system admin's first choice. Allows port information.
      Available by defautl with almost all port of all OSs. Sometimes differences are found in implementations pertaining to syntaxes, or switch names/usage.

4. curl - Another all time favorite.
      1. Allows calling/initiating the http request from command line, saves a lot of time since no program or skeleton is needed to be created for testing http calls.

5. jMeter - designed to load test functional behavior and measure performance

6. soapUI => mocking a server.
   Available from http://www.soapui.org/

7. JavaScript
      FireBug - for firefox - Javascript functions/elements are easily visible and manageable. Debugging on the fly is allowed too. Available as an addon for firefox.
      FireBug - Firebug implementation for Safari.

8. FireFinder- find item in DOM. Easily allow an item in the DOM of a given web page.

More details coming on the following -

9. Visual Event

10. FireFocus

11. JASH - CLI for test javascript

buzzwords
- bpf - berkeley packet filter

workshop day GIDS 2010 - 3 - Functional Programming

2010-04-25T02:23:00.000+05:30

Well, for the third session of the day, my choice was clear, cloud computing discussion with Simone, from AWS. I had already had the taste of google cloud solution (app engine), and now I wanted to see the Amazon flavour. Quite horribly unfortunately, the internet link gave way and Simone's presentation plans went awry.

The guy was quite angry and it was so visible, so very visible. He tried to talk people through, plain talk talk talk it never settled in. I see people literally dozing off, and I guess he realized that too, since on more than one occasion, he instructed to keep the hall lights switched on. A bit of bad thing on part of Saltmarch/Airtel...

As a result, midway during the session, I left off, and thought of joining in the Flex-php workshop from adobe guys. However, somehow, I ended up entering the hall where Venkat was talking about functional programming.

And did I regret that decision, no chance. I guess, I have never enjoyed a programming language discussion so much, probably because I didnt get a chance to speak to such a learned speaker, and so many of keen and interested students (yeah, blame my college, I know, I do that myself.. all the time :)

It was a packed house, and fantastic atmosphere. For one thing, Venkat knew what he was talking about, and for other, I found that discussion took me back about 15 years to my college days. I had a strong feeling, why didnt I had an educational experience as nice as this. The teacher, the students, the atmosphere, whole place was like a wonderful experience. Though I had no clue what scala is before I entered the place, it never felt like a problem.

The way he talked about issues, the benefits, the arguments he presented, and I think more importantly, the way he engaged the audience in discussions, it was just wonderful. You might think that I am over reacting, but i dont care, I just loved this guy, this session.

He talked about programming paradigms, the way procedural languages were king one day, and the way they are looked at as if ancient history is being talked about today. And similarly, for today's generation OOPS is becoming more of a thing of the past, and new paradigms are coming over.

The way he propounded functional programming, the very idea of not allowing immutability, the way of thinking in terms of having to think of a solution without that.. amazing...

At more than one point in time, the people in the audience tried to contest the idea/need of another language, when we already have so many, and the need of learning another one. There were thoughts on the toughness of a programming language vs the other, and the way he answered.. fantastic... His point in this case was, its as good as a Karnataka person going to Punjab and saying, "These people are crazy, their language is so touch, I cant even read that". And it was a good laugh..

At more than one point in time, he compared different programming languages, their features and pros and cons of each of them. The kind of knowledge that he had, to be able to compare them, was amazing...

I have to say, without any doubt, that this session was my best experience at GIDS 2010. I loved this, probably due to my personal interest in the theory of computer science as such, but also because there probably wasnt any other speaker with that kind of personality and flair for talking.

Hats off to Saltmarch for inviting people like him too...

workshop day gids 2010 - 2 - paypal x

2010-04-24T19:10:00.000+05:30

It went even better afterwards, Paypal has opened doors to their Payment methodology to developers. They call it PayPal X. They have published a complete SDK which allows developers to write independent programs that allow integration of paypal as a payment gateway in their e commerce applications.

Their idea is to popularize paypal and use its simplicity and capability in spreading its business reach. The whole team of PayPal was present at the workshop and the atmosphere was pretty nice. Khurram Khan started off by talking a bit on the background of financial transactions, further, Rangarajan took over and talked in detail.

Frankly, the session was interactive, because perhaps people are motivated by money, the chances of earning it, and the basic idea of linking real e-commerce to their websites by such low effort and simplicity. My personal interests in that area are purely out of curiosity, in the sense that I am more keen to know as to whats going on, then get down to the code details and start implementing it.

One thing that stood out for me is that paypal proposes to review the code of the developer before actually allowing to move it to production (live), but has no control thereafter. So, in essence, someone who intends to do something funny (read : wrong/illegal etc...) might pass the initial test by presenting a genuine case, and then once he;s live, he can go back and change the code. I feel that by allowing updations like this, paypal is losing a bit of control on its API usage by the people.

I would probably have built somekind of a dynamic filter or a pair matching kind of mechanism (or a checksum for that matter) which would change the moment the developer changes his code on production site. Every call to the paypal API should check this checksum/authentication token and should go through only if this checks also holds good. Well, its just a thought, and probably paypal has its own reasons not to enforce any such check, but, if I were Khurram, I would probably start like that, perhaps remove it later or something like that.

When I posed this to Khurram, he was saying that, the responsibility of doing anything illegal or wrong lies anyway with the developer or the site owner, so, paypal doenst really want to get in their way. They would be apprehanded sometime anyway, sooner or later. As much as I agree with his argument, I still think that paypal could probably play a role and possibly stop "wrong" things from happening to some extent, after all its happening through their infrastructure, although the ownership lies somewhere else.

Other than, this particular point, I think the SDK is pretty nice and ok. They allow Java, no special downloads etc required to start developing, besides the SDK of course, and that the API supports almost all kinds of operations.

I specially liked their idea of generalizing the paypal kind of services, where it comes to trust building. During one discussion a statement was said, I dont remember which one, but a paypal employee said that, "I am sure that when it comes to trusting someone over internet, you can trust paypal a lot more with your financial information than a relatively unknown website that you are using for the first time." That I agree with, and perhaps the Indian jinx of not using web for payments can be broken with a trusted guy on the net, paypal.

You trust one person, paypal, and the rest is handled by paypal. As far as this statement goes, there actually is a greater risk, what if paypal goes rogue ? It would then have all my information, all of it... well then you would have to trust someone.. right ?? or not ???

The workshop day @GIDS 2010 - I - cloud computing using Google App engine

2010-04-24T17:14:00.000+05:30

Well, A developer day cant get better than that... And, whatever issues I mentioned regarding scheduling of the summit were gone up in air.

I had the choice to go attend the Cloud Computing/hosting workshop with Mathhew McCullogh. It was based on java, which I dont care about anyway. Not that I have any kind of disrespect for the language, but I strongly think that we ought to think beyond languages.

He perfectly showcased how easy it is to build and host you own web application on the google app engine cloud platform. Its hard to imagine how easy it has become with eclipse plugin for google app engine and web toolkit, to develop an application for the cloud and deploy it. With those tools installed and configured (its surprisingly easy task to do it anyway), its a button click to deploy your application to the cloud.

I am not a big fan of java, or of huge complex configuration files for that matter, but the way this app engine thing is coming pre-configured, it reminds me of ruby on rails. At least to start with. Its surprisingly easy to start developing with GWT (Google Web Toolkit) and Google App Engine.

I tried to replicate what he was doing/showing, and, remember I am no big shots in java, that last professional java work I did was about 8 years back, It was surprisingly easy. Just a few clicks and it was done. I have to say, it was infectious, as if I felt like delving more into java and go back to experimenting with it...

Is that a workable idea ?

2010-04-22T12:37:00.001+05:30

Actually, we are waiting for a new server to be installed and any new physical server installation takes some time, we all know that. The project would request it, there would be financials and then the order would be approved internally.
It would be placed with the vendor by purchasing, the vendor will ship it, and then a few days/weeks after, technicians will come over to install the stuff.

Then the local admins will get to work and configure it to be used by the respective teams. Another few days/weeks...

Just wondering, any given enterprise has hundreds ir not thousands computers, all classes, laptops, desktop development PCs, server class PCs and others...

How about taking off 10% of all these PCs' computing power to create a cloud computing environment (of course within the company environment only). An application can be installed which will make sure that a given percentage of computing power will go to only a particular resource.

A lot like how BOINC works. You install a client/manager app on ur PC and then u receive a piece of computation for your part of computing power.

That kind of computing environment can easily replace an average server's computing power.
The storage can easily be hooked up to a SAN.

I'd encourage a discussion on this... see what others think around the idea...

in reference to: Distributed computing - Wikipedia, the free encyclopedia (view on Google Sidewiki)

Fantastic Effort by SaltMarch

2010-04-22T11:06:00.001+05:30

Well, i have to say it like that, The one day that I attended at the GIDS - 2010 was amazing. Probably I am reacting like that since its my first such summit, nonetheless the quality of speakers and the content they delivered was top of the ladder.

Amazing truely.

To get people of such caliber at one platform is a job in itself which saltmarch have done to perfection. I chose to be there for .Web day, and the kind of knowledge that was flowing around, technologies being talked about and the manner in which they were addressed, simply great.

We know some things as simple buzzwords, but we ought to realize that the buzzwords actually have lot more behind them. The way Simone (from AWS) put it, "Everybody in the confrerence hall would know what is Cloud computing, but I am sure that when I ask you what is it, you'd all come back with different definitions, and perhaps they are all correct as well". That statement sort of sums up the whole experience, we all know to some extent, or varying degree of perception what something is about, but unless we know it from the source, as they say, from the horse's mouth, its always a bit farther from the real reality.

I like that way the speaker and sessions were compiled, though a few of my favorite sessions were canceled (blame that volcano in Iceland for canceling all the flights), I still enjoyed a lot of it.

Personally me, interested in Ajax, came a lot closure to experimenting with it, and perhaps using it in professional environment as well. Thanks to people at Adobe booth there, I have a chance to try out the Adobe Flex Builder IDE, and check out the potential first hand.

Tomorrow, there is a workshop on php and Flex delivering RIA, and I cant wait to be there :)

However, as is the custom for people like me, to comment and find fault with something or other, after all we are humans, we are bound to make mistakes and other humans are bound to find them and report them publicly, like on this blog... :)

I believe the scheduling of some of the sessions could have been better. For example, I missed out on the PayPal X presentation, because I wanted to attend Marty Hall's session on evaluation of Ajax/JavaScript Libraries too. Now, I'd have to contend only with the video of that presentation, hoping that it would be made available. :)

I like the scheduling of the workshop day a lot better. It gives people like me options to attend sessions diff technologies ...

Looking forward to another exciting day of fun filled experiments that enhance know how...

in reference to: Great Indian Developer Summit :: Conference, Expo, and Awards on Java, .NET, Rich Web :: Saltmarch Media Summits (view on Google Sidewiki)

Web Changing the face of the world : Ramesh

2010-04-21T10:40:00.000+05:30

Ramesh, a senior technologist/scientist from Adobe, talked about how and what has web 2.0 done to change the world, especially the way the information is treated/used/processed etc...

Here are some of his points -

Real Time Web

kind of content that is coming around
no more old docs
social networking
live collaboration

experiencing the web - how the web is experienced, the way its accessed, used etc.

Some tools are coming up in the VR world, which allows
augmented reality -> web apps capturing reality and using the inputs

proliferating devices

Already more web access from non PC devices
More devices are coming through which access web differently

democratization of data

Some govts have already started putting their data in public domain.
Lots of application development opportunities based on that data.
Cloud computing also coming around to provide processing power and applications for the data.

Buzzwords

ActionScript
ECMAScript
Flex

Quite interesting talk :) Learned person, Ramesh.

Marty Hall : Why Ajax : Using Ajax in ur web apps

2010-04-21T09:34:00.000+05:30

Why WebApps - Ajax

Asynchronous JavaScript and XML => doesnt hold good anymore
allow the browser to utilize the time when the main page is gone tofetch the result of your initial app
Add/modify content on the webpage at runtime
trends -> more jobs on Ajax than any other technology - php, vb, asp etc...

Ajaxifying an Application

libraries - jQuery, ext-ja, dojo, yui, prototype, google closure, mootols
Browser allow calls to only the server from where the original code came in, due to firewall issues.

Situation 2 - Hybrid
Libraries to use for new apps - JSF 2.0, Struts 2.0, Spring MVC 3.
JSF is better because
integrated in main code - jquery needs u to write javascript
no javascript to write - a simple tag

Situation 3 - Hardcore Ajax
Google web toolkit
Use java everywhere
Write java code at front end -> compiles to javascript at run time
Write java code at back end-> provides facility of communication in terms of java objects

Buzzwords

Jboss Seam <-> JSF 2.0

@GIDS 2010 Bangalore

2010-04-21T08:34:00.000+05:30

I am going to be there at the "Great Indian Developer Summit" in Bangalore April 20th to 23rd 2010.

Its hosted at the lush green IISc campus, in the J N Tata Auditorium. Various pieces of info are available from the organizer website -

www.saltmarch.com

http://www.developermarch.com/developersummit/

It looks like an amazing place to be, full of tech discussions, and hopefully knowledge too...

Apple and India...

2010-03-31T11:54:00.001+05:30

Well, this is sad, particularly for someone in India, who avoids getting cracked iPhones...

Only yesterday Apple and Airtel announced availability of iPhone 3GS in India and today I get to read this, that Apple is releasing a new/better hardware for iPhone coming summer.

So, basically, India as an iPhone market is going to lag behind the rest of world by at least one model this time around as well.

Sometimes, I dont understand Apple's marketing strategy, specially in Indian context. As an Economy, India is probably the largest in the emerging market economies closely behind China. That statement should alone prove to be a "lucrative market" for any company which aims for growth.

This is specially interesting when we look at the number of iPhones already used in India with cracked OS's.

This doenst help a company's revenues if the product is consumed in a market where the company doesn't officially sell the product.

Also, the number of iPhones used in India, should give Apple an indication about the marketability of their product in this market. Personally, I think as a product from Apple's stable, iPhone has a lot of glamour linked to it.
And therefore, huge possibilities in India for Apple.

The new iPhone's rumoured specs look stunning, 5 MP camera, better display resolution, more processing power, possible multi tasking, video calling camera and few other things. After reading that, my plans to order an iPhone 3GS have taken a back seat. I want to see this product coming to market, further to see, when does it come to India.

Apple seems to listening to customers.. the kind of improvements rumoured is possibly the exact list of things someone might say are downsides of iPhone...

Good luck Apple, and India, hoping for a better collaboration this time around...

in reference to: More on next-gen iPhone and Verizon iPhone dreams (view on Google Sidewiki)

Explain plan of Infobright

2010-03-18T17:20:00.001+05:30

We all knw that Infobright is based on MySql. However, when I started with Infobright, I was equally unaware of both.

Today I got to know about Infobright's explain plan collection technique, and realized how different it is from Oracle's.

A typical explain plan from oracle talks about the path it follows to retrieve the data, and somehow makes common sense to read a plan.

On the other hand, an explain plan from Infobright looks nothing like the Oracle's plan. Its a set of cryptic rows put together. An example of such a plan -

2010-03-18 01:31:01 [4] T:-1 = TABLE_ALIAS(T:0,"myTableName")
T:-2 = TMP_TABLE(T:-1)
VC:-2.0 = CREATE_VC(T:-2,PHYS_COL(T:-1,A:20))
A:-1 = T:-2.ADD_COLUMN(VC:-2.0,AVG,"avg(col1)","ALL")
A:-2 = T:-2.ADD_COLUMN(VC:-2.0,MAX,"max(col1)","ALL")
A:-3 = T:-2.ADD_COLUMN(VC:-2.0,MIN,"min(col1)","ALL")
A:-4 = T:-2.ADD_COLUMN(,COUNT,"count(1)","ALL")
VC:-2.1 = CREATE_VC(T:-2,PHYS_COL(T:-1,A:2))
A:-5 = T:-2.ADD_COLUMN(VC:-2.1,GROUP_BY,"col2","ALL")
VC:-2.2 = CREATE_VC(T:-2,PHYS_COL(T:-1,A:5))
A:-6 = T:-2.ADD_COLUMN(VC:-2.2,GROUP_BY,"col3","ALL")
VC:-2.3 = CREATE_VC(T:-2,PHYS_COL(T:-1,A:7))
A:-7 = T:-2.ADD_COLUMN(VC:-2.3,GROUP_BY,"col4","ALL")
VC:-2.4 = CREATE_VC(T:-2,PHYS_COL(T:-1,A:11))
A:-8 = T:-2.ADD_COLUMN(VC:-2.4,GROUP_BY,"col5","ALL")
VC:-2.5 = CREATE_VC(T:-2,PHYS_COL(T:-1,A:12))
A:-9 = T:-2.ADD_COLUMN(VC:-2.5,GROUP_BY,"col6","ALL")
VC:-2.6 = CREATE_VC(T:-2,PHYS_COL(T:-1,A:13))
A:-10 = T:-2.ADD_COLUMN(VC:-2.6,GROUP_BY,"col7","ALL")
VC:-2.7 = CREATE_VC(T:-2,PHYS_COL(T:-2,A:-1))
T:-2.ADD_ORDER(VC:-2.7,DESC)
RESULT(T:-2)

And, guess what, to generate a plan, you have to change a setting in the initialization parameters file, and then bounce the db. I wonder if there have been any instances when someone had to look into production database's performance and therefore needed to look at some plan.

Being as new a system as it is, I cant really complain, since such things of maturity will slowly creep in. The focus right now is probably elsewhere, building the functionality, and probably performance.

in reference to: Infobright.org Forums | Enable MySQL Logging (view on Google Sidewiki)

Discovered BOINC 5 years ago, joined today :)

2010-02-19T16:01:00.001+05:30

It was George, a colleague at Axa Tech in Switzerland who first introduced me to this idea. I was thrilled alright, but didnt join right away.

Today I did.

Its a nice feeling, and a feeling of giving back something to science.

http://en.wikipedia.org/wiki/Berkeley_Open_Infrastructure_for_Network_Computing

http://boinc.berkeley.edu/

I would not go into details how it works and all that. However, I would have to say, for whatever time your laptop/workstation remains idle, you can donate the processing power to a better cause. Absolutely fantastic idea, genuinely noble.

And then, look a the stats, people like me are contributing to the processing power generated to the extent of 4 pFLOPS, as against the fastest supercomputer system at mere 1.79 pFLOPS.

As I said, feels great. :)

I hope to inspire others to join in...

in reference to: BOINC (view on Google Sidewiki)

Connecting to an Oracle Server...which way is faster...

2010-02-17T07:20:00.000+05:30

I've never came across this question as such, because its sort of given that you connect to a server any-which way the administrators define the policy. The tuning perspectives came in only afterwards.

However, in a recent situation, I had to connect to a remote server from my client system and was wondering whether I should connect using tnsnames.ora or should I use the basic authentication mechanism, using the complete connect string.

I thought that the tnsnames.ora would incur some latency in connections. On the other hand, connecting directly without using any such files is like doing a telnet to the server on a given port.

Also, I have seen that if you attempt to connect using the complete connect string from the tnsnames.ora file, it also works.

Since I was not clear myself, I did post this question on oracle forums. And a gentleman, Billy Verreynne took out some time to help explain the concept. Many thanks to him.

As it stands, the basic methodology of connection is same in both cases. However, if you happen to have a huge tnsnames.ora file, it would be an overhead to parse the file and get the connect string for the service name you attempted to connect to. But it would only make a difference, if your client connection process go into hundreds then it might make any noticeable degradation. Basically, client would have to parse the huge tnsnames file few hundred times per second.. then it might make a difference.

Billy also suggested that if you switch the server mode from dedicated to shared, it would help.

So, for my conclusions, its the same if you connect through tnsnames or through a native connect string.

availability of native drivers !!!

2010-01-22T07:01:00.000+05:30

As I mentioned earlier, I was busy experimenting with a columnar database, InfoBright.

The experience till now is mixed, in some scenarios, the data loading is very good, as much as 40k rows per second, whereas through other channels its a poorer 500 rows per second. When I go with their built in loader, its lightening fast, but when I try from Pentaho or Informatica, its measly.

Apparantly, the drivers and the compatibility of the third party tools do play a role in attaining the performance.

The fact that they dont have any native driver published, is a huge bottleneck. Informatica dont even have a native connector for MySql, the more known cousin of InfoBright (being open source, the core engine of InfoBright is based on MySql only).

One thing came out of this experience for sure, the liking for open source got better and better. During the hunt to see the reason for slow performance from Pentaho, I even tried and managed to get the source code of the plugin (transformation) that pentaho uses for Infobright. Its pure java and it was a very powerful and awakening feeling to see the code of the item. I felt as if I have the power, the choice to make a difference, make it better. :)

I am currently exploring other options, one of them includes creating a dump file (read : csv) of the data, and then launching the command line tool to load the data into target db. I dont like it, but lets see if there is any better (read:faster) way around...

Its been a while...

2009-12-18T01:50:00.000+05:30

Its been quite some time since I wrote anything to this blog, let alone anything valuable.

However, last few months have been quite eventful in my personal and professional life. I have landed a job with a company in Bangalore now, which works a lot more with open source systems/applications than my previous employer. I can easily say that this was one of the points in my decision making.

Currently, I am opening up to working on Ubuntu and probably virtualization, mysql, postgresql and most importantly, Informatica Powercenter and Pentaho. The suite of tools is just too exciting for me to find right words to express my feelings.

Recently I have picked up a project to do a Proof of Concept for using an open source columnar database and compare its behavior with oracle for a given application. Of course it involves data loading, and analytical queries, so there would be some interesting stats collection and comparison. Attempts to break a system are always more interesting than to make it J

Lets see how it goes…

Also, I would try to be more regular on the blog now onwards…

Compile all packages in a schema

2009-07-02T09:29:00.000+05:30

BEGIN
FOR cur IN
(
SELECT OBJECT_NAME,
OBJECT_TYPE ,
owner
FROM all_objects
WHERE object_type IN('PACKAGE', 'PACKAGE BODY')
AND owner = ':OWNER
AND status = 'INVALID'
)
LOOP
BEGIN
IF cur.OBJECT_TYPE = 'PACKAGE BODY' THEN
EXECUTE IMMEDIATE 'alter package "' || cur.owner || '"."' || cur.OBJECT_NAME ||
'" compile body';
ELSE
EXECUTE IMMEDIATE 'alter ' || cur.OBJECT_TYPE || ' "' || cur.owner || '"."' ||
cur.OBJECT_NAME || '" compile';
END IF;
EXCEPTION
WHEN OTHERS THEN
DBMS_OUTPUT.PUT_LINE('Errors compiling - ' || cur.owner ||'.'||cur.object_name);
END;
END LOOP;
END;

MDX Tutorial

2009-05-28T16:43:00.001+05:30

MDX Tutorial

Blogged with the Flock Browser

Interesting Demos from Informatica

2009-05-26T16:55:00.001+05:30

Pretty Basic Demo by Infa guys on their DW offering...
Informatica Enterprise Data Warehousing Demo

Another interesting one on Data Migration...
Informatica Data Migration

tweaks around Oracle 11g on Hardy 64bit

2009-05-01T02:05:00.000+05:30

I did an oracle 11g 64bit install on my 64bit core 2 quad using the resources available on internet and some past experience.

The tutorial here is great :

http://www.pythian.com/blogs/968/installing-oracle-11g-on-ubuntu-804-lts-hardy-heron

However, some things still show up after the initial setup is done. Here are bits of my experience -

1. Starting Enterprise manager after a reboot -
Try emctl status agent
The last line of the output should tell you whats happening. If its not running, start it using the following -
emctl start dbconsole
This thanks to http://www.linuxquestions.org/questions/linux-general-1/linux-5-oracle-11g-enterprise-manager-question-616532/

2. The ORACLE_SID variable must be declared in either /etc/profile (for a single database on the system) or on user's local profile.

Earning money using cloud Computing ??

2009-04-22T08:58:00.000+05:30

I dont know much more about cloud computing, but it sure sounds like an interesting idea. The way it allows computing resources to be shared and connected through internet and all that. The benefit to the average developer is huge, getting computing power so cheap..

However, I think the reverse should also be true and could turn out to be pretty interesting. The way we (developers) use resources from the cloud for our computing purposes, we could also share our spare resources to the same cloud so that someone else might use them and allow us to earn some money in exchange. There would be some preconditions and requirements and what not, but essentially that should be a possibility...

I dont know for sure whether this concept already exists, but i would sure like to see that happening...

Blank Subject Reminder in Outlook...

2009-04-17T16:23:00.000+05:30

1. Open your outlook.

2. Press Alt+F11. This opens the Visual Basic editor and then Press Ctrl+R which in turn open Project-Project 1 (left side).

3. On the Left Pane, one can see "Microsoft Outlook Objects" or "Project1", expand this. Now one can see the "ThisOutLookSession".

4. Double click on "ThisOutLookSession". It will open up a code pane.

5. Copy and Paste the attached code in the right pane. (Code Pane) and save it.

Private Sub Application_ItemSend(ByVal Item As Object, Cancel As Boolean)

Dim strSubject As String
strSubject = Item.Subject
If Len(Trim(strSubject)) = 0 Then
Prompt$ = "Subject is Empty. Are you sure you want to send the Mail?"
If MsgBox(Prompt$, vbYesNo + vbQuestion + vbMsgBoxSetForeground, "Check for Subject") = vbNo Then
Cancel = True
End If
End If
End Sub

For this you need to create a digital certificate. Go to http://office.microsoft.com/en-us/help/HP052495581033.aspx to see how to do it. Name your certificate whatever you like e.g No_Subject.

Once that’s done go back to the code window, click Tools, Digital Signature, Choose - and choose that signature you just created. Hit ok, save this project, close it. Close Outlook completely too.

Start up Outlook, you get the security box; click always trust running this macro thing. And you’re done! Try sending a message without body, you’ll get an alert. Now even if you restart outlook it’ll still work.

Courtesy a colleague, Mukesh.

All about Dual Table : AskTom

2009-04-08T09:51:00.001+05:30

A very very interesting article from AskTom on dual table, its purpose and usage and everything else about it...

Here's some excerpts from it, leaving the rest for reading...

"DUAL is owned by SYS.  SYS owns the data dictionary, 
therefore DUAL is part of the data dictionary.  You are not to modify the data dictionary 
via SQL ever -- wierd things can and will happen"

the optimizer understands dual is a magic, special 1 row table.  It stopped on the 
select * because there is to be one row in there.  Its just the way it works.

Read the rest of article here...

Ask Tom "All about the DUAL table "

Blogged with the Flock Browser

Oracle SQL tuning - Burleson Consulting

2009-04-07T17:02:00.001+05:30

Burleson Consulting website has come to rescue in many situations. For many questions their site provides great pointers and ideas.

Here's a series of papers/articles on Oracle SQL Tuning -

Oracle SQL tuning parameters

And, here's an article on tuning of individual SQL Statements -

Tuning Individual SQL Statements

Blogged with the Flock Browser

Beware of Question Authorities « H.Tonguç Yılmaz - Oracle Blog

2009-04-07T15:49:00.003+05:30

Beware of Question Authorities

Following suggestions are from Mr.Lewis Blog ;
If it’s not dated - don’t assume it’s true*
If its date is more than about 18 months old - don’t assume it’s (still) true*
If there’s no version number - don’t assume it’s true*
If it’s not your exact version number - don’t assume it’s (still) true
For ‘technical implementation’ details, if there’s no platform mentioned - don’t assume it’s true
For ‘technical implementation’ details, if the platform’s not the same as yours - don’t assume it’s true
If there’s no rational justification supplied, and no repeatable test case - don’t assume it’s true.
And even when all the details are perfect and there is a repeatable test case - and even after the repeatable test case produces the same results - ask yourself this question
Could there be a different explanation for the same set of results - and if so, how badly could this advice damage my system, and how hard would it be to test my alternative hypothesis ?
Once you’ve got through that lot - then you might be safe trying the advice on a development system.

Mr.Kyte also mentions this problem under the ‘Question Authority.’ terminology;

There are lots of ‘experts’ out there;
Make them prove everything
Statements that should raise your eyebrows:
    It is my opinion…
    I claim…
    I think…
    I feel…

Everything can (and should) be proven - TKPROF goes a long way hereStatspack is great‘Runstats’ is a tool I use as well (search asktom for runstats)
Things change, expect that

Remember a test becomes a proof when;
it has a specification- the results are reproducible
alternative explanations have been eliminated
it is published- it survives peer-group review
Reference
“The Burden of Proof” presentation by Jonathan LewisDon’t take any “guru’s” word, test it and make sure you are convinced of the results..

Beware of Question Authorities « H.Tonguç Yılmaz - Oracle Blog

Blogged with the Flock Browser

SQL - Difference Between Hash Join & Merge Join

2009-04-07T15:49:00.001+05:30

Difference Between Hash Join & Merge Join

Merge Join :Oracle performs a join between two sets of row data using the mergejoin algorithm. The inputs are two separate sets of row data. Output isthe results of the join. Oracle reads rows from both inputs in analternating fashion and merges together matching rows in order togenerate output. The two inputs are sorted on join column.

Hash Join :Oracle performs a join between two sets of row data using hash joinalgorithm. Input and Output same as Merge Join. Oracle reads all rowsfrom the second input and builds a hash structure (like has table injava), before reading each row from the first input one at a time. Foreach row from the first input, the hash structure is probed and matchingrows generate output.

SQL - Difference Between Hash Join & Merge Join

Blogged with the Flock Browser

An Extract from AskTom

2009-04-02T12:57:00.000+05:30

Efficient SQL

This was probably the hardest part of the book to write - this chapter.  That is not 
because the material is all that complex, rather because I know what people want - and I 
know what can be delivered.  What people want:  The 10 step process by which you can tune 
any query.  What can be delivered:  Knowledge about how queries are processed, knowledge 
you can use and apply day to day as you develop them.

Think about it for a moment.  If there were a 10 step or even 1,000,000 step process by 
which any query can be tuned (or even X% of queries for that matter), we would write a 
program to do it.  Oh don't get me wrong, there are many programs that actually try to do 
this - Oracle Enterprise Manager with its tuning pack, SQL Navigator and others.  What 
they do is primarily recommend indexing schemes to tune a query, suggest materialized 
views, offer to add hints to the query to try other access plans.  They show you 
different query plans for the same statement and allow you to pick one.  They offer 
"rules of thumb" (what I generally call ROT since the acronym and the word is maps to are 
so appropriate for each other) SQL optimizations - which if they were universally 
applicable - the optimizer would do it as a matter of fact.  In fact, the cost based 
optimizer does that already - it rewrites our queries all of the time.  These tuning 
tools use a very limited set of rules that sometimes can suggest that index or set of 
indexes you really should have thought of during your design.  

I'll close this idea out with this thought - if there were an N step process to tuning a 
query, to writing efficient SQL - the optimizer would incorporate it all and we would not 
be having a discussion about this topic at all.  It is like the search for the holy grail 
- maybe someday the software will be sophisticated enough to be perfect in this regards, 
it will be able to take our SQL, understand the question being asked and process the 
question - rather then syntax.  

To me - writing efficient SQL requires a couple of things:

o    Knowledge of the physical organization of what I'm asked to query against.  That is 
- the schema.  Knowledge that the physical organization was actually designed in order to 
help me answer my frequently asked questions (refer back to the chapter on designing an 
efficient schema for advice in that arena)

o    Knowledge of what the database is capable of doing.  If I did not know about "skip 
scan indexes" and what they did (we'll cover them below) - I might look at a schema and 
say "ah hah, we are missing an index" when in fact we are not.

o    Knowledge of all of the intricacies of SQL - from the lowly "WHERE" clause on up to 
analytics and psuedo columns.  Knowledge of what using a particular construct will do to 
my runtime processing.

o    And most importantly of all - a solid understanding of the goal, of what the 
question is.  Tuning a query or process is really hard (impossible I would say) - unless 
you understand the question in the first place.  I cannot tell you how many times I've 
not been able to tune a query until I had the question in hand.  Certainly you can derive 
a question from a query - however, many times that derived question is much more 
confining then the real question being asked.  For example, many people use outer joins 
in all queries - they are "afraid" of losing a row (perhaps they got "burned" in some 
past experience and now use outer joins everywhere).  If the objects are related in a one 
to one mandatory fashion - we don't need an outer join at all.  The question derived from 
the query is much more confining then reality.

That last topic or point is so important, I'll close out this section with it. In this 
chapter we'll cover the topics of what the database is capable of doing in general - 
looking at many of the access paths and join operations available to us.  We'll look at 
what SQL is capable of doing - not by discussing the entire language, that in itself is a 
book.  Rather, we'll look at a couple of things that will whet you appetite - show you 
how powerful this language can be, how much more than just "SELECT" "FROM" "WHERE" and  
"ORDER BY" there is.  Then we'll close up with a look at that most important topic - why 
understanding the question is more important then having a query at hand to tune.

So, this section will not provide you with the N steps you need to follow in order to 
tune a query or write the best queries in the world.  For every rule of thumb out there 
anyone has ever shown me regarding writing "efficient SQL", I've been able to come up 
with a slew of common (not esoteric) counter cases to prove that rule of thumb is wrong 
in as many cases as it is right.  I've talked to people who swear "NOT IN" is fatal, 
never use it - always use NOT EXISTS.  Then I show them NOT IN running a query 10 times 
faster then NOT EXISTS.   I talk with people who feel NOT EXISTS is the worst construct 
on the planet - you must use IN.  Then I do the same - showing them how NOT EXISTS can 
run many times faster then IN.

On SQL Tuning

2009-04-02T12:03:00.000+05:30

Found some interesting stuff... (Collection from many sources over internet)

1. Basic Tipcs from dba-oracle.com - http://www.dba-oracle.com/art_sql_tune.htm
2. a more light hearted approach - http://philip.greenspun.com/sql/tuning.html
3. http://www.dba-oracle.com/art_otn_cbo_p1.htm

On Oracle Joins

2009-04-02T11:10:00.000+05:30

Some useful links on Oracle SQL optimization (more specifically on the way oracle joins tables)

1. Sachin Arora's explanation on Hash Joins/Nested Loops
2. Optimization of joins
3. Hash joins & Nested Loops on dbaForums
4. @OTN Forums on Nested Loops
5. Hash join Tips
6. Sizing PGA

2009-03-13T12:18:00.000+05:30

Useful scripts for DBA perspective -

http://www.dbapool.com/scripts.php

More stuff on finding free space available in a database -

http://forums.oracle.com/forums/thread.jspa?threadID=624042

A sql that we found useful is here - (from the above link itself) -

SELECT
NVL (b.tablespace_name, NVL (a.tablespace_name, 'UNKOWN')) NAME ,
mbytes_alloc mbytes ,
mbytes_alloc - NVL (mbytes_free, 0) used ,
NVL (mbytes_free, 0) free ,
((mbytes_alloc - NVL (mbytes_free, 0)) / mbytes_alloc) * 100 pct_used,
100 - (((mbytes_alloc - NVL (mbytes_free, 0)) / mbytes_alloc) * 100) pct_free
FROM
(
SELECT
SUM(BYTES) / 1024 / 1024 mbytes_free,
tablespace_name
FROM
SYS.dba_free_space
GROUP BY
tablespace_name
)
a,
(
SELECT
SUM(BYTES) / 1024 / 1024 mbytes_alloc,
tablespace_name
FROM
SYS.dba_data_files
GROUP BY
tablespace_name
)
b
WHERE
a.tablespace_name(+) = b.tablespace_name

UNION ALL

SELECT
f.tablespace_name ,
SUM (ROUND((f.bytes_free + f.bytes_used) / 1024 / 1024, 2) ) "total MB" ,
SUM (ROUND(NVL (p.bytes_used, 0) / 1024 / 1024, 2)) "Used MB" ,
SUM (ROUND ( ((f.bytes_free + f.bytes_used) - NVL (p.bytes_used, 0) ) / 1024 / 1024, 2 ) ) "Free MB" ,
(SUM (ROUND (NVL (p.bytes_used, 0) / 1024 / 1024, 2)) * 100) / (SUM (ROUND ((f.bytes_free + f.bytes_used) / 1024 / 1024, 2))),
100 - (SUM (ROUND (NVL (p.bytes_used, 0) / 1024 / 1024, 2)) * 100) / (SUM (ROUND ((f.bytes_free + f.bytes_used) / 1024 / 1024, 2)))
FROM
SYS.v_$temp_space_header f,
dba_temp_files d ,
SYS.v_$temp_extent_pool p
WHERE
f.tablespace_name(+) = d.tablespace_name AND
f.file_id(+) = d.file_id AND
p.file_id(+) = d.file_id
GROUP BY
f.tablespace_name
ORDER BY
5 DESC --&orderby
;

Compile all invalid packages in a schema : oracle

2009-02-25T16:16:00.000+05:30

Came across this from sql developer, Looks like a very useful script, especially when its made available readymade

begin
FOR cur IN
(SELECT OBJECT_NAME, OBJECT_TYPE, owner
FROM all_objects
WHERE object_type in ('PACKAGE','PACKAGE BODY')
and owner = :OBJECT_OWNER AND status = 'INVALID' )
LOOP
BEGIN
if cur.OBJECT_TYPE = 'PACKAGE BODY'
then
EXECUTE IMMEDIATE 'alter package "' || cur.owner || '"."' || cur.OBJECT_NAME || '" compile body';
else
EXECUTE IMMEDIATE 'alter ' || cur.OBJECT_TYPE || ' "' || cur.owner || '"."' || cur.OBJECT_NAME || '" compile';
end if;
EXCEPTION
WHEN OTHERS THEN NULL;
END;
end loop;
end;

And then another one to check as to what all objects are invalid -

select object_type, count(*)
from user_objects
where status = 'INVALID'
GROUP BY object_type

Oracle : count(1) vs count(*)

2009-02-12T14:47:00.001+05:30

It seems we should prefer using count (*) over count(1)

read the article from asktom here...

http://asktom.oracle.com/pls/asktom/f?p=100:11:0::::P11_QUESTION_ID:40208915257337

there've been other discussions and debates on this as well.

Free Adobe Flex Video Training

2009-01-15T08:59:00.000+05:30

Just found about it...

http://www.adobe.com/devnet/flex/videotraining/?sdid=EGMNU

not been through it yet, but it looks pretty good, combination of pdf ebooks, audio/video stuff and all that...

More Free ebooks on Oracle - Unix - Linux

2009-01-03T10:22:00.000+05:30

Find free to download ebooks on this site -

http://www.itstudy8.org

Changing System name in Ubuntu

2009-01-03T09:46:00.000+05:30

When I installed Ubuntu for the first time, I named my system to reflect the os name (gutsy) and the architecture (64 bit). However, whenever there were system upgrades, the name got irrelevant. For example, when I upgraded to Hardy, the name still remained stuck to the last version of OS.

Therefore, I now thought of renaming my system to separate it from the os versions. This is so very simple to achieve in Ubuntu -

just edit the file /etc/hostname (owned by root, therefore use sudo). Put in the new name, and restart the system.

However, the consequences of this name change will have to be reflected across the network (if you have one). Hostname files on all other systems or dns entries will have to be updated to reflect the new system name.

Useful Articles on Oracle

2008-12-23T09:48:00.000+05:30

Useful Articles on Oracle Cast function -
http://www.psoug.org/reference/cast.html

Performing DML with Collections
1. http://download.oracle.com/docs/cd/B28359_01/appdev.111/b28371/adobjcol.htm#sthref483
2. http://www.psoug.org/reference/collections.html

Answers to my question on whether or not its possible to perform DML on Oracle collections (in combination with Cast)
1. http://www.orafaq.com/forum/t/137485/133715/
2. http://forums.oracle.com/forums/thread.jspa?messageID=3176984

Some Notes on Joins & their syntaxes in Oracle
1. http://oreilly.com/catalog/orsqlpluspr2/chapter/ch01.html

Some articles on Oracle Explain Plan and how to generate/read/understand it
1. http://www.adp-gmbh.ch/ora/explainplan.html
2. http://www.adp-gmbh.ch/ora/sql/join/join_methods.html
3. http://download-uk.oracle.com/docs/cd/B12037_01/server.101/b10752/ex_plan.htm

Some articles on Oracle Global Temp Tables (GTT)
1. http://www.dbasupport.com/oracle/ora8/page2.shtml
2. http://www.dba-oracle.com/t_temporary_tables_sql.htm

On Oracle Memory Usage
http://www.dbasupport.com/oracle/ora11g/memory_usage.shtml

Useful mysql stuff...

2008-12-06T02:50:00.000+05:30

Here's a compilation of my day to day use of mysql related stuff...

Resetting root password in mysql
- Goto this page https://help.ubuntu.com/community/MysqlPasswordReset
To give privileges to a certain user in mysql
- grant all on *.* to username

Finally... IE on Linux / Ubuntu...

2008-12-04T20:18:00.000+05:30

Great..... It works... :)

Initially, when I needed to run a few IE specific sites on my Ubuntu installation, I spent quite some time searching around google, and some more prominent linux sites... But I did not hit success...

Today however, my search bore fruit... I found IEs4Linux project, and its so damned simple.

I am still running Hardy (not Interpid) on my desktop, and wine (the default). I picked up the instructions, kind of updated them myself to work in my hardy, and bingo.. it installed and works...absolutely.

Here's what I did -

Found this page for installation instructions on Ubuntu :
http://www.tatanka.com.br/ies4linux/page/Installation:Ubuntu
Run the following commands -

sudo apt-get update
sudo apt-get install wine cabextract

And then run this -

wget http://www.tatanka.com.br/ies4linux/downloads/ies4linux-latest.tar.gz
tar zxvf ies4linux-latest.tar.gz
cd ies4linux-*
./ies4linux

The last one would open a dialog box and would ask you basic questions about what version do you want, choose what u like and bingo.. it would start installation...
Within minutes, it automatically downloads some files from here and there, and is done.
It just works...

Agile : What is, Comparison, Transition...

2008-12-01T23:54:00.001+05:30

Generally on Agile Methodology -
1. Agile methodology home page - http://www.agilemethodology.org/
2. Manifesto of Agile methodology - http://www.agilemanifesto.org/
3. Working with Agile methods - http://www.agileadvice.com/

Some links on Comparison of waterfall and Agile development methodologies.
1. http://agileintro.wordpress.com/2008/01/04/waterfall-vs-agile-methodology/
2. Youtube -http://www.youtube.com/watch?v=XokJLWp7icI
3. on Agile Advice - http://www.agileadvice.com/archives/2006/05/waterfall_vs_ag.html

Some links on the transition from Waterfall model to Agile methodology...
1. Agile Blog: Transitioning from Waterfall to Agile - Some tips
2. From Udayan
3. Scrum Alliance
4. A Presentation

Blogged with the Flock Browser

Some Oracle data dictionary help...

2008-11-08T01:53:00.000+05:30

To get parameters information for your functions/procedures/packages etc. you can use all_arguments view. A sample query is attached...

select owner, object_name, package_name, argument_name, position, data_type, in_out
from all_arguments
where owner = (case when lc_user is null then user else lc_user end)
and package_name = (case when lc_pkg_nm is null then package_name else lc_pkg_nm end)
order by owner, package_name, object_name, position;

Further another view called all_source is also pretty useful if you have to look at the actual source code of a component.

Following links helped me on this -

My new quad core desktop system...

2008-09-20T07:18:00.000+05:30

Recently managed to get a desktop system home, especially after coming back, it was necessary. The laptop just doesnt provide enough power.

This sure looks like a monster for a desktop system usage.

- Quad Core CPU (Core 2 Quad Q6600) at 2.4 GHz, 8 MB L2 Cache
- 2 GB RAM @800 MHz
- 250 GB Seagate Barracuda Hard Disk
- Here I was a bit disappointed. My local vendor just couldnt manage to get me a 10K rpm or higher hard disk from his supplier. He kept saying it would require a SCSI card and all that configuration. From my understanding it should be a simple SATA or SATA2 disk. But, well, this is the first one in place, may be we'd have more... :)

So, with this system in place, I plan to do some interesting things... e.g. running my long awaited local oracle server so that I can work with apex (http://apex.oracle.com)

Running Hardy on it. To say the least, hardy flies on this system. To start with, everything was detected normally on this configuration and I had no difficulty to make it work with Ubuntu Hardy. Even the live CD worked fine.

Now with the installed copy along with all the updates so far, the system literally flies... The system monitor shows me 4 CPUs and it adds so damned much to my confidence in using this power.... :)

The only issue I have found till now is with my power supply ratings/configuration. It somehow draws a lot of power (the rating on the PSU box says 450W) whereas the Display is not connected to the CPU power (its a TFT anyway). The UPS that I have is a APC with 500 VA rating. The UPS just doesnt hold.

All the power from the UPS comes to just this system and the display. Still it doesnt hold. About the power supply, I was told that the rating (450W) is an indication of how much can this provide, not how much does it need all the time. That might be wrong or right both. I am still figuring that out. No idea till now.

iPhone 3G in India...

2008-08-26T10:26:00.000+05:30

Finally... finally... after such a long wait.. iPhone comes to India.. officially. But, despite all the hype and wait around it, the affordability of it really a huge question mark.

I didnt change my mobile phone for last 2 years, waiting for the iPhone to come to India. I booked an iPhone for me through both the prospective service providers in India, Airtel and Vodafone. It was all in anticipation that once its released in India, there would be huge queues for buying it, as was seen in other parts of world for iPhone 3G release.

However, the launch in India was not as expected. There was hype, but not as many customers. I am not a statistician and therefore cant give any numbers to prove my point, but I guess I have a feeling for the situation.

In my view, the pricing of the iPhone is a big reason for its slow launch. I cant say whether or not it would pick up in future, but right now, the price tag of 31000 and 36000 INR for iPhone is a very very high price. I mean, yes its a good phone, there are nice features, but thirty one thousand rupees... its just too much for a gadget in my view. And then, compare the price of the phone in US, 200 USD.. converts to around 8000 INR and so it shows that apple is selling that phone 4 times as costly as in US.. what kind of business strategy would that be ??? Especially in a cost sensitive market as India...

There would be people buying it, but not like me, not the average Indian IT guy, who knows the features, their meaning and usage and has a wish to own one. I dont know many who'd take up the phone at this price tag.

May be Apple will come back with a price cut in India as well, as they did in US... but will they offer people refunds for the difference.. we'll see...

Till the time apple decides to cut down on the price tag, my wait for owning an iPhone continues...

Data Warehousing on a Shoestring Budget : TDWI

2008-07-23T09:24:00.001+05:30

Another interesting set of articles around low budget data warehousing...

Data Warehousing on a Shoestring Budget

Blogged with the Flock Browser

Ideas on DWH Testing...

2008-07-22T13:23:00.001+05:30

Recently I was asked about strategy on data warehouse testing. Realizing how rarely we talk about this, I went to google for this and spend about half an hour searching.

Found the following links, which still need to be researched/analyzed further.

DM Review : Where are the Articles on Data Warehouse Testing and Validation Strategy?

A Wordpress blog : Strategies for Testing Data Warehouse Applications « Business Intelligence and Datawarehousing

DM Review : Strategies for Testing Data Warehouse Applications

Blogged with the Flock Browser

Lookup Transformations in Informatica

2008-07-22T10:12:00.001+05:30

Lookup is a transformation to look up the values from a relational table/view or a flat file. There are two types of Lookups in Powercenter, namely;

Connected Lookup
Unconnected Lookup

Caching is an important facet of lookup transformation planning. You need to know what kind of data you are dealing with, how frequently do you call the lookup, how frequently does the data change, what is the size of your lookup table etc. among other things. Once you use cache, the trip to database can be avoided, thereby enhancing performance.

Different types of caches can be used with lookup like static, dynamic, persistent, and shared(The dynamic cache cannot be used while creating an un-connected lookup). Each of these has its own identification. For more details, refer to Informatica Transformation Guide.

Lookup is a passive transformation, and can be used either connected or unconnected. Typically, connected lookup is used when you want to do the lookup for all rows. When you have selective lookup requirements, its normally better to go for unconnected lookup. Unconnected Lookup can be used as if its a function call.

To read more, here is a good article -
What is lookup transformation in informatica? - IT Community

Getting Started with ActiveScaffold

2008-05-21T10:06:00.001+05:30

Recently I returned to my rails env to do some more experiments (which might be used for a future project)

To start with, I have seen a bit of active scaffold stuff and was very impressed with it originally already. In the meantime, rails 2 have arrived. Quite a few things have changed and I was thinking that it might have an effect on the overall picture of active scaffold also.

However, checking up on railsforum etc brought out that not much effect has gone down to active scaffold kind of plugins from rails 2.

Starting up active scaffold is really easy as they say on its website also...

try it here... www.activescaffold.com It turned out to be really simple to initialize yourself with it.

Searching a bit more I came across a few tutorials -

Tutorial from Active Scaffold guys

Another one from someone like me, experimenting with activeScaffold...

AkillesBlog » Blog Archive » Ruby on Rails: Experimenting with ActiveScaffold

Blogged with the Flock Browser

Open Solaris build May 2008 ( 2008.05 ) released

2008-05-13T15:30:00.000+05:30

Today I came across a few news items and articles here and there that talked about the release of OpenSolaris projects' May 2008 release.

Though its available for download from the open solaris home page, Sun is offering Free CD shipments for the latest release here : https://www2.sun.de/dct/forms/reg_us_2307_228_0.jsp

I already ordered my copies :)

Also, there is a nice article about bit of unix OS history and open solaris 2008.05 release review here : http://blogs.zdnet.com/BTL/?p=8703

Installing Oracle 11g on Ubuntu server 64bit on vmware server 2 beta

2008-05-13T10:16:00.000+05:30

I have this wish to install a hardy server version on my system. Since I did not want to add another partition, so I chose to do that inside a virtual machine (using vmware). vmware supports 64bit Guest Operating system, as opposed to virtual box which does not. :(

So, I have been able to install the server edition inside the vmware machine. I have never used nfs before, but the tutorial I was following was suggesting to use it.

http://www.pythian.com/blogs/654/installing-oracle-11g-on-ubuntu-linux-710-gutsy-gibbon

However, I am having problems with -

Trying to expand the virtual hard disk inside vmware. I tried it, but havent succeeded yet.
The Oracle installer exits with a strange error message.

I have posted my issues on the ubuntu forums, and havent got any solution yet. Lets see how it goes...

Using Base64 encoding with Oracle / Perl

2008-05-12T19:19:00.000+05:30

For basic encryption purposes, base 64 encoding can be used. In fact its so basic that it would probably be incorrect to call it encryption. Its more encoding, basically preventing the subject text from appearing in plain text.

For example, if in some tool, all the parameters are stored in repository tables, and therefore, its not very nice if the passwords are stored there in plain text. They should at least be made illegible, so that they dont appear in plain text and there is some effort spent in decoding it.

Base64 encoding is an industry standard and almost all programming languages, development toolkits provide plugins for this. For Oracle, there are pre-existing packages providing the functionality.

We can use base 64 encoding on oracle side using Oracle supplied package UTL_ENCODE. On the perl side, we could use cpan package MIME:Base64. Other languages provide different kinds of plug-ins. For example, ruby on rails has a plug in for this, Javascript also allows this to be done relatively easily.

The idea is that, such sensitive data to be entered by customers, in encoded format. So, we also dont see the plain text version. And then the program uses the encoded version, right to the point before actual data needs to be used.

For achieving this, following way of working can be used :

1. On Oracle side, the sensitive data is encoded using a program call equivalent to the following will return an encoded/decoded string for a given input string.

FUNCTION fnc_encrypt(lv_str IN VARCHAR2)
RETURN VARCHAR2
AS
BEGIN
   RETURN utl_raw.cast_to_varchar2(UTL_ENCODE.BASE64_ENCODE(utl_raw.cast_to_raw(lv_str)));
END;

FUNCTION fnc_decrypt(lv_str IN VARCHAR2)
RETURN VARCHAR2
AS
BEGIN
    RETURN utl_raw.cast_to_varchar2(UTL_ENCODE.BASE64_deCODE(utl_raw.cast_to_raw(lv_str)));
END;

Its a design decision whether to put this code in a procedure/package somewhere.

2. On the other end of the processing chain, reverse operation would have to be performed. For example, in Perl, the implementation relies on the cpan package MIME:Base64

use MIME::Base64 ();

and then use the decode/encode function as per the need...

MIME::Base64::decode($PI_CAT_PWD);
MIME::Base64::encode($PI_CAT_PWD);

vmware 2 beta on hardy

2008-05-09T12:04:00.000+05:30

I installed vmware 2 beta on my hardy. It was a surprise and a pleasant one at that, the installation procedure did not require any tweaks at all. It worked out of the box.

Further, when I tried to connect to the web based management interface, I encountered my first problem. vmware requires a log-in with root user, and as any ubuntu user would tell you, the root user is well-hidden in ubuntu. Almost never used, my knowledge on how to set a password to root user was zero.

Again, communities came to help. This thread on vmware communities helped me through that.

http://communities.vmware.com/message/901283#901283

Its a small tip, but none-the-less important. Just run "sudo gnome-terminal" and it would open a root terminal for you. Once there, type "passwd" to change the password for root user and you are done.

now, in the browser, log in to vmware web mgmt interface with user as root, and password you chose for root user.

As a precaution, I gave my normal user admin privileges so that I dont have to login as root all the time.

Oracle wrap utility...

2008-05-08T15:02:00.000+05:30

Oracle's wrap utility helps to hide the application logic from being visible. However, I recently discovered that it doesnt hide string literals.

Hard to grasp though, since my understanding was a bit different. I cant really think of a reason why would they keep strings etc in plain text. Perhaps to force people (like me) not to store their encryption key in a package and wrap it to achieve another level of security...

Here's the standard statement from Oracle about the limitations of wrap utility -

Limitations of the Wrap Utility

String literals, number literals, and names of variables, tables, and columns remain in plain text within the wrapped file. Wrapping a procedure helps to hide the algorithm and prevent reverse-engineering, but it is not a way to hide passwords or table names that you want to be secret.

From : http://download.oracle.com/docs/cd/B10501_01/appdev.920/a96624/c_wrap.htm#LNPLS016

I wonder if there are any tricks to achieve something like that... my investigations are still on.. will report...

xgl - System monitor bug in hardy...

2008-05-08T09:45:00.001+05:30

On Ubuntu's latest release, Hardy Heron, if you are using xserver-xgl package for graphics, you are in for a surprise.

The system monitor carries on a bug in xgl package wrt incorrect reporting of memory usage, and shows processes using zillion bytes of memory.

I started using xserver-xgl from the times in Gutsy when Compiz would not run on my hardware without that. However, I didnt quite notice this until I involved myself with beta release of hardy.

As it happened, I raised this issue on the Ubuntu forums (more than once actually, link-1, link-2), and finally after a few discussions/posts, it was discovered that

xserver-xgl has a few bugs
no more active development is happening on this stream of packages
Compiz should work without needing this package as well.

As a result, I tried removing xgl from my laptop, and voila, memory consumption numbers are from earth again.. the martian numbers have left... :)

Attaching two screenshots here to give the idea of the Martian numbers...

acid3 and webkit on hardy

2008-05-05T11:50:00.000+05:30

I have been fascinated with the rendering of web pages by Safari. Always liked it, it has a certain edge to it when talking about the look and feel. When I read more, it turned out that the rendering engine is similar to Konqueror (somewhere I read that its actually based on open source libraries from Konqueror).

I dont have a mac so I cant really experience it first hand, but the windows release of Safari made me get my first experience.

This weekend, I was surfing net when I came across "webkit". Reading a bit more about it informed me -

WebKit is an open source web browser engine. WebKit is also the name of the Mac OS X system framework version of the engine that's used by Safari, Dashboard, Mail, and many other OS X applications. WebKit's HTML and JavaScript code began as a branch of the KHTML and KJS libraries from KDE. This website is also the home of S60's S60 WebKit development.

This made me think, if its actually open source to start with, the code must be available and may be I can compile it for my system.

Well, I didnt had to look much further... search produced this link...
http://blog.kagou.fr/post/2008/04/21/Test-Webkit-on-Ubuntu-Gutsy-and-Hardy

However, my first attempt at compilation failed, due to lack of one of the packages. libxt-dev

So, now the list of pacakges that should be installed on hardy to compile webkit would look like this :

sudo apt-get install autoconf automake libtool bison flex gperf libicu-dev libxslt-dev libcurl4-openssl-dev libsqlite3-dev libjpeg62-dev libpng12-dev libglibmm-2.4-dev libgtk2.0-dev build-essential libxt-dev

The sample browser, gives me 98 on acid3 (acid3.acidtests.org). Though very minimalistic in capabilities, this is the best score for acid3 that I could get on my system.

Here are my scores
Firefox 3 beta 5 : 71
Flock : 51
Opera 9.5 beta 2 : 78
Midori (webkit based) : 64
Webkit minimal browser (nightly build r32862 ) 98

On my windows partition as well, none of the browsers went beyong 70's. Best being opera at 71.

Linux Kernel sites...

2008-05-03T12:45:00.001+05:30

Found this interesting site. If you want to start knowing or playing around with linux kernel, this site could be a interesting start.

Linux Kernel Newbies

Also interesting would be to watch out for this site... www.kernel.org

Blogged with the Flock Browser

vmware infrastructure web access on vista...

2008-05-01T16:37:00.000+05:30

For those who try to use vmware server on windows vista (unfortunately I am trying these days), the vmware server 2 beta is using tomcat engine to provide web based environment.

The access control is weird to say the least. It doesnt matter if ur user has Administrtor rights on the vista box. You have to have the "Administrator" user and log in as that only. So, you got to make a new user called Administrator and then log in to Vmware Infrastructure Web access.

I found the help on vmware forums. reproducing here for benefit of myself (lest I forget) and others who might come across.

Logon to Vista using your usual account.
Launch the cmd prompt - Make sure you select, 'Run as administrator'
Net user administrator yourFavPassword
Net user administrator /active:yes
Switch User, or logoff
Logon as Administrator Password yourFavPassword
(Your password may be different!)

And then the VI web access works... :)

making vmware work on hardy..

2008-04-30T03:47:00.000+05:30

Had trouble originally making vmware work on my new hardy 64 bit system (behaves almost same on desktop and server variants).

Careful looking up found out two things,

the new beta version (2) of vmware works from web based console, and could take quite some time making it work
vmware on hardy 64 bit requires some tweaking to do before you can really get up and running with it.

Finally managed to get through with help from ubuntuforums.org. Nice fellow has posted his experience with 64 bit version, and it turned out to work exactly the same way for me...

Loads of thanks :) to http://ubuntuforums.org/showpost.php?p=4357442&postcount=10

25 reasons to use Ubuntu... :)

2008-04-22T19:13:00.000+05:30

Well, its a very interesting collection of reasons to use Ubuntu...

Some might not be as 100% agreeable as other might be, but still most are completely agreeable...
Well done...

Found it on this blog-... here u read 25 reasons for using Ubuntu

running a 64bit guest os in vmware

2008-04-22T13:23:00.000+05:30

I tried to install a 64bit guest os and that took much longer to realize the problem.

What I found out (may be you guys already know that) the hard way today, is that, you cant keep your virtual disks on a non-ext3 partition (I was trying to keep them on my ext hard disk - all ntfs) :(

Also, you need to switch on the VT feature from your bios to be able to actually use 64 bit guest os.. My Core2Duo supported that, so luckily I had it already with my hardware. But if you want to do that, make sure ur cpu supports it.

converting .img to .iso and mounting iso

2008-04-21T10:47:00.000+05:30

.img files are CloneCD files created as a result of dumping contents of a Cd into a file. Very similar to .iso files which are more popular for containing cd images.

Many tools, like virtual box allow mounting .iso files as a virtual cd/dvd. Ubuntu repositories provide a tool called ccd2iso for converting the .img to .iso.

just install ccd2iso from the repositories using

sudo apt-get install ccd2iso

Once installed, just launch it for converting

ccd2iso file.img file.iso

This will create the iso equivalent of the img file.

To mount the iso file normally in the linux environment, just type

mount -o loop -t iso9660 filename.iso /mnt/iso

Experiments With Hardy...

2008-04-21T09:21:00.000+05:30

Ubuntu is by far the more stable of the OS's.. nothing new about that.

Last friday the release candidate of the new Hardy came out. It was beta for sometime now, and I had it for a few days in beta phase too..

It was my first time playing around with an OS in its beta stage and sure.. it feels good to be involved..

I found a few things, suggested on the forums, put out bug reports and it all works so nicely. Everybody involved is so open and helping... it was actually very nice exp. A good learning curve as well for a few things... :)

Currently there are a few things still going on with the "Hardy Release Candidate" with differing opinions, but all in all it looks like a good package to me. On my laptop, it works nicely, no issues as such...

In my view, the Ubuntu series has had its share of issues with hardware compatibility. With different people using different set of hardware, it turns out that not everything is 100% picked up by the installer as it should be.

I tried using virtualbox on hardy to install another copy of hardy 64 bit to experiment something, however, virtualbox wont allow that !! People on the ubuntu forums suggest that it could be a "feature" of vbox not to support 64bit cpu as of now... That would be a shame, actually... :(

Checking out vmware has been another shock. Earlier it used to be "FREE" for Linux, however now its only 30day evaluation version. Sounds like just another corporate, not a company supporting open source...

On the 64 bit point, I am really thinking of going 64 bit completely... all out... I have tried it for a few weeks now, and for me its been no problems at all... and the support for packages etc is fine for my needs... Yeah, some more programs need to be compiled by hand, but thats ok for me... :)

Ubuntu directory structure...

2008-04-21T09:13:00.000+05:30

In the *nix operating systems, the tree style of file systems are there from time immemorial. In fact the idea of a inverted tree directory system or all filesystems originated with Unix.

The root of the tree is called "root directory" here and is the top level directory, and all its subdirectories make up the directory hierarchy. This is different as compared to Windows wherein there are drives. Especially diff since each diff hardware device has its own dir tree with its own root directory. In Unix there is only one root dir. Everything else, falls under that tree...

Here's a summary of the standard directories found on a typical Unix/Linux system. Current reference being Ubuntu.

/bin - binary applications (most of your executable files)

/boot - files required to boot (such as the kernel, etc)

/dev - your devices (everything from drives to displays)

/etc - just about every configuration file for your system

/etc/profile.d - contains scripts that are run by /etc/profile upon login.

/etc/rc.d - contains a number of shell scripts that are run on bootup at different run levels. There is also typically an rc.inet1 script to set up networking (in Slackwar), an rc.modules script to load modular device drivers, and an rc.local script that can be edited to run commands desired by the administrator, along the lines of autoexec.bat in DOS.

/etc/rc.d/init.d - contains most of the initialization scripts themselves on an rpm-based system.

/etc/rc.d/rc*.d - where “*'’ is a number corresponding to the default run level. Contains files for services to be started and stopped at that run level. On rpm-based systems, these files are symbolic links to the initialization scripts themselves, which are in /etc/rc.d/init.d.

/etc/skel - directory containing several example or skeleton initialization shells. Often contains subdirectories and files used to populate a new user’s home directory.

/etc/X11 - configuration files for the X Window system

/home - locally stored user files and folders

/lib - system libraries (similar to Program Files)

/lost+found - lost and found for lost files

/media - mounted (or loaded) devices such as cdroms, digital cameras, etc.

/mnt - mounted file systems

/opt - location for “optionally” installed programs

/proc - dynamic directory including information about and listing of processes

/root - “home” folder for the root user

/sbin - system-only binaries (see /bin)

/sys - contains information about the system

/tmp - temporary files

/usr - applications mainly for regular users

/var - mainly logs, databases, etc.

/usr/local/bin - the place to put your own programs. They will not be overwritten with upgrades.

/usr/share/doc - documentation.

Some content shamelessly borrowed from this page :Linux or ubuntu Directory structure

Get rid of the Ubuntu splash screen during boot

2008-04-19T06:07:00.001+05:30

If you want to see Ubuntu booting like the good old text based linux systems, here are some ideas on removing the splash screen from the booting process....

Quickzi: Get rid of the Ubuntu splash screen during boot

Blogged with the Flock Browser

Firefox Smart Keywords

2008-04-19T00:11:00.001+05:30

A very very useful tip.

In firefox its possible to make shortcuts for your favourite links (especially useful for searches etc).

For example, for a typical google image search, you would normally open images.google.com and then type in your required phrase and press search. That constitutes two trips to google image server to accomplish your search needs.

With this method from firefox, you can achieve this in one trip to server, just by doing a small trick with your favourites configuration.

You add a particular url to your favourites list. And later on, from the organize favourites dialog, assign that url a shortcut, a very small phrase e.g. img or images or something that you like.

Now, after this setup, when you want to search for images on keyword india, you'd just have to say your chosen keyword and india, i.e. just type in

img india

in the address bar and press enter.. your search results should be on the way...

More detailing here... Solutions - 25 Useful Firefox Smart Keywords

Blogged with the Flock Browser

A user profile on Ubuntu India - very interesting...

2008-04-16T15:17:00.001+05:30

Just saw this user page by Mr Bharadwaj... on Ubuntu India...
Very interesting writings...

User:Bharadwaj.j@gmail.com - Ubuntu India

Blogged with the Flock Browser

polishlinux.org » GNU/Linux for everyone

2008-04-16T02:52:00.001+05:30

A very very useful/interesting site for people looking for Linux distribution informations. It also has a page for comparing various distributions on some specific parameters

Also, allows individuals to write their views in a feedback/forum form.

polishlinux.org » GNU/Linux for everyone

For comparing distributions, visit http://polishlinux.org/choose/comparison/

Also interesting might be www.distrowatch.com

Blogged with the Flock Browser

chown : not owner !!!

2008-04-14T14:14:00.000+05:30

Apparantly, you cant do a chown in Solaris. Here's more...

http://unix.derkeiler.com/Newsgroups/comp.unix.solaris/2004-03/2579.html

another one here

http://www.itworld.com/Comp/3380/nls_unixchown040923/index.html

The reasoning explained sounds logical. Apparantly (as one of the article also explains), the restriction helps a user fill another user's quota by creating a huge file and chowning it to the other user. Even worse, if the other user doesnt have privs to access the dir hosting that huge file, he cant even delete that.

Looks like a safety precaution, turned into a bit of blockade for normal users... :(

Buying a wireless router for home...

2008-04-14T10:29:00.001+05:30

When it comes to buying a wireless router, the acronyms can be misleading and confusing at times.

While I was looking for articles around the net that explain complex terms, I found a few. Here are some -

How To Buy A Wireless Router: Decoding Wi-Fi 802.11 Networking Standards and Protocols

Wireless network buying guide - CNET reviews

Wireless Router FAQ - What is the Range of a Typical Wi-Fi LAN?

Wireless Routers - Best 802.11g Wireless Broadband Router Reviews

Blogged with the Flock Browser

Hardy 64bit beta Installed...

2008-04-14T08:30:00.000+05:30

This weekend I installed Hardy 64 bit on my Core2Duo notebook and it flew... Just like that.

Earlier when I had gutsy for the first time on this laptop, I had plenty of issues with my hardware. Tweak here and tweak there, I had to do a lot of things to make it work the right way.

However, with Hardy this time, I would say, I felt happy for two reasons.. first it didnt complain about any of my hardware (probably because now my hardware is about a year old, and for gutsy it was so much newer..) and that its 64 bit installation.

I am yet to completely switch to this, especially since I encountered a bug with evolution and that Hardy is still beta, but I do see a way forward going with this version of Ubuntu for me.

As a result of this experiment, right now, I am triple booting, Vista business, Gutsy (32 bit) and Hardy (64 bit).

This is one thing I like so much about the PC notebooks, you can play around with them a lot more than you can do with an equivalent mac... :)

One of the guys on ubuntu forums posted this url for a better understanding of 64 bit systems... 64 bit Systems have more benefits than RAM only

Later on, I found this link, a very interesting collection of 32bit vs 64bit OS (especially Ubuntu) comparison.

Hardware analysis and news..

2008-04-11T13:30:00.000+05:30

Interesting site for hardware news/updates/info...

AnandTech: your source for hardware analysis and news

BigAdmin Feature Article: Using Service Management Facility (SMF) in the Solaris 10 OS: A Quick Example

2008-04-10T13:01:00.000+05:30

A very nice article -

BigAdmin Feature Article: Using Service Management Facility (SMF) in the Solaris 10 OS: A Quick Example: "Introduction"

Extremely Simple Calendar Integration for Rails

2008-04-10T12:16:00.000+05:30

Found this interesting plugin for Ruby on Rails

Beige Sunshine » Blog Archive » Extremely Simple Calendar Integration for Rails

Ruby on Rails - Rails Migrations - Dizzy

2008-04-10T09:46:00.001+05:30

Some tweaks/cheats on Rails migrations -
Ruby on Rails - Rails Migrations - Dizzy

Blogged with the Flock Browser

my technology playground

And now.. a ubuntu laptop concept that works on Solar energy .. wow...

Data Lineage.. what is that ?

Informatica & hadoop... solutions for future ?

Switching defaults in Ubuntu

writing for cloudtimes.org these days

HTML 5

convert Infrmatica Session Logs to text/xml format for out of tool readability

Just found out about this amazing thing...

Exadata - is it really worth the hype

Rejecting records in Fact table loads - Informatica

Setting a useful command prompt in Unix

scripts and hash bang ( #! )

A Note to New Consultants

my experiments with solr :)

Business & Open Source - How both can benefit

Tips for brainstorming...

Big help...

I also feel like saying, 1984...

Country General Mood using Tweets

Oracle Count(1) vs Count(*)

Developing a Rails application using an existing database

Making rails talk to SQL Server Express Edition

winscp vs cuteftp

Parallel Querying using Oracle

Ubuntu 10.04 is here.... Upgrade today...

Implementing Cartesian Product in Informatica Mapping

Hard vs Soft Parse in Oracle...

AWS Application Demo

Web Application Debugging Tools

workshop day GIDS 2010 - 3 - Functional Programming

workshop day gids 2010 - 2 - paypal x

The workshop day @GIDS 2010 - I - cloud computing using Google App engine

Is that a workable idea ?

Fantastic Effort by SaltMarch

Web Changing the face of the world : Ramesh

Marty Hall : Why Ajax : Using Ajax in ur web apps

@GIDS 2010 Bangalore

Apple and India...

Explain plan of Infobright

Discovered BOINC 5 years ago, joined today :)

Connecting to an Oracle Server...which way is faster...

availability of native drivers !!!

Its been a while...

Compile all packages in a schema

MDX Tutorial

Interesting Demos from Informatica

Top 12 Ruby on Rails Tutorials

tweaks around Oracle 11g on Hardy 64bit

Earning money using cloud Computing ??

Blank Subject Reminder in Outlook...

All about Dual Table : AskTom

Oracle SQL tuning - Burleson Consulting

Beware of Question Authorities « H.Tonguç Yılmaz - Oracle Blog

SQL - Difference Between Hash Join & Merge Join

An Extract from AskTom

On SQL Tuning

On Oracle Joins

Compile all invalid packages in a schema : oracle

Oracle : count(1) vs count(*)

Free Adobe Flex Video Training

More Free ebooks on Oracle - Unix - Linux

Changing System name in Ubuntu

Useful Articles on Oracle

Useful mysql stuff...

Finally... IE on Linux / Ubuntu...

Agile : What is, Comparison, Transition...

Some Oracle data dictionary help...

My new quad core desktop system...

iPhone 3G in India...

Data Warehousing on a Shoestring Budget : TDWI

Ideas on DWH Testing...

Lookup Transformations in Informatica

Getting Started with ActiveScaffold

Open Solaris build May 2008 ( 2008.05 ) released

Installing Oracle 11g on Ubuntu server 64bit on vmware server 2 beta

Using Base64 encoding with Oracle / Perl

vmware 2 beta on hardy

Oracle wrap utility...

Limitations of the Wrap Utility