Eclipse - installing Scala plugin manually?

I have been playing around with Scala for some time, and was always using the Scala IDE (www.scala-ide.org) which is based on a relatively older version of Eclipse (Luna).

I recently discovered this, wherein you could install the scala plug-in on a regular Eclipse installation.

Just add the following url as a new update site in your local eclipse installation and you'd be able to install the scala plugin just like that -


http://download.scala-ide.org/sdk/lithium/e44/scala211/stable/site

Removing ports from an existing mapping - Do's and Dont's

Recently, a colleague called in for an issue that he was facing with an Informatica mapping. Let me recreate that situation here -

There is a mapping that get n ports from a source and loads m ports to a target. Standard stuff, nothing special or fishy around there. There comes a change request that says, such and such x number of columns are not required in the target, and since there would be a sure performance penalty for carrying through extra data (however small), the mapping should be changed to remove those ports from the pipeline.

What this gentleman did was to remove the connections for those ports from the source qualifier onwards. That saved him from changing and re-importing the source definition.

However, the mapping execution failed, complaining about a certain error situation. Thats when I got the call :)

whether I could discuss and fix it is something else, but what was the reason of the error - ?  

The very fact that if you are bringing in certain ports in the source qualifier, you HAVE to take them forward.  Thats a rule from informatica's side. Which means, if you are pulling up n ports from the source, you HAVE to expose all those n ports going away from the source qualifier. Or, putting it differently, in whichever way you create a set of ports in your source qualifier, all of them have to be consumed by some transformation object. Not a single one can be left unconnected, on the input side or the output side. Simple.

That was the reason, and the solution was fairly simple, to reduce his effort, either remove the ports from source qualifier as well, or just carry them forward to one transformation. and then drop them onwards.

Having said that, the very reason for which this whole change was initiated, the performance gain that would come through by not having the extra ports being carried forward, would come only when you remove the un-wanted ports from the source itself.

Implementing Cartesian Product in Informatica Mapping

As against pentaho, Informatica doesnt provide a ready made transformation for implementing cartesian product in a mapping. Although, most of us would agree that  its not often that we tend to go for cartesian product joins. [the instinct generally is to do enough to avoid a cartesian product, because its a performance killer in general]

However, when your requirements need this, there is no direct way to do it in informatica joiner transformation.  Either you do it in the db side, by overriding your source qualifier sql statement and building it in there.


However, I have seen that some designers dont like to override sql statements, in such cases you'd have to implement it inside the mapping only. Here's a workaround for achieving that. Here goes -

1. Read both the sources using their own source qualifiers, normally.
2. For both of them, put in an Expression Transformation after the source
3. In both the expression transformations, create an output port with a constant value. For Example, call it dummy1 for stream 1 and assign it a value -1.  Similarly, a port would be created in the second pipeline, lets call it dummy2 and assign it a value -1.
4. Now create a joiner transformation. Link ports [including the one that we created with a constant value] to the joiner from both the expressions.
5. In the join condition, choose to compare the dummy columns. 
6. The rest of the joiner configuration would have to be like any other joiner. Nothing specific.

You might want to keep the smaller source as the master in the joiner, since it would save on the caching.

Before implementing the above solution, be sure to go back and check if its actually required for your application to have cartesian product !!!!

Hard vs Soft Parse in Oracle...

Was reading on asktom, came across this response from Tom, that seemed so direct and clear...

So, shamelessly copying.. :)



[From AskTom]
Parsing

This is the first step in the processing of any statement in Oracle. Parsing is the act of breaking the submitted statement down into its component parts ? determining what type of statement it is (query, DML, DDL) and performing various checks on it.

The parsing process performs two main functions:

o Syntax Check: is the statement a valid one? Does it make sense given the SQL grammar documented in the SQL Reference Manual. Does it follow all of the rules for SQL.

o Semantic Analysis: Going beyond the syntax ? is the statement valid in light of the objects in the database (do the tables and columns referenced exist). Do you have access to the objects ? are the proper privileges in place? Are there ambiguities in the statement ?

For example if there are two tables T1 and T2 and both have a column X, the
query ?select X from T1, T2 where ?? is ambiguous, we don?t know which table to get X from. And so on.

So, you can think of parsing as basically a two step process, that of a syntax check to
check the validity of the statement and that of a semantic check ? to ensure the
statement can execute properly. The difference between the two types of checks are hard for you to see ? Oracle does not come back and say "it failed the syntax check", rather it returns the statement with a error code and message. So for example, this statement
fails with a syntax error:

SQL> select from where 2;

select from where 2

*

ERROR at line 1:

ORA-00936: missing expression

While this statement failed with a semantic error ? if the table NOT_A_TABLE existed and we had permission to access it, this statement would have succeeded:

SQL> select * from not_a_table;

select * from not_a_table

*

ERROR at line 1:

ORA-00942: table or view does not exist

That is the only way to really tell the difference between a semantic and syntactic error ? if the statement COULD have executed given the proper objects and privileges, you had a semantic error, otherwise if the statement could not execute under any circumstances, you
have a syntax error. Regardless ? Oracle will not execute the statement for you!

The next step in the parse operation is to see if the statement we are currently parsing has already in fact been processed by some other session. If it has ? we may be in luck here, we can skip the next two steps in the process, that of optimization and row source generation. If we can skip these next two steps in the process, we have done what is known as a Soft Parse ? a shorter process to getting our query going. If we cannot, if we must do all of the steps, we are performing what is known as a Hard Parse ? we must parse, optimize, generate the plan for the query. This distinction is very important. When developing our applications we want a very high percentage of our queries to be Soft Parsed ? to be able to skip the optimize/generate phases ? as they
are very CPU intensive as well as a point of contention (serialization). If we have to
Hard Parse a large percentage of our queries, our system will function slowly and in some cases ? not at all.

The way this sharing of SQL in Oracle is accomplished is via the shared pool, a piece of memory in the SGA maintained by Oracle. After Oracle parses the query and it passes the syntax and semantic checks ? it will look in the shared pool component of the SGA to see if that same exact query has already been processed by another session.

Since it has performed the semantic check it has already figured out:

o Exactly what tables are involved
o That we have access to the tables (the proper privileges are there)

And so on. Now, it can look at all of the queries in the shared pool that have already been parsed/optimized and generated to see if the work has already been done.

[Raghav]

As has been discussed here, the soft parsing of queries makes our application faster, since it doesn't have to do everything all over again (from parsing point of view).  Bind variables come to help in this case.

For example, a query like the following -

select a,b from tbl1 where c = 2333;

would be parsed and stored under a different identifier than

select a,b from tbl1 where c = 58763;

whereas, essentially they are same queries only, the literal value being different for the where condition.

In this case, the second query would go for hard parse, instead of our expectation of a soft parse. So, what could be done to tell oracle to think that these are actually same queries and there is no need to do the hard parse again. Well, you can do that by using a bind variable, in the sense, removing the only differentiating part of query by a runtime replacement.

Look at the following query -

select a,b from tbl1 where c = :bb;

now, this query can be used for both the examples above, at the runtime (after parsing), the variable :bb would be replaced by the respective literal value, and you have your answer.  However, this gains a lot of performance, since, the second run goes for a soft parse, instead of a hard parse, so, every subsequent run, after the first one, is re-using the parsing information collected the first time.

So, simple recommendation, use bind variables wherever you see a query being reused with different literal values, and use run time substitution of the literals.

Lookup Transformations in Informatica

Lookup is a transformation to look up the values from a relational table/view or a flat file. There are two types of Lookups in Powercenter, namely;

1. Connected Lookup 
2. Unconnected Lookup

Caching is an important facet of lookup transformation planning. You need to know what kind of data you are dealing with, how frequently do you call the lookup, how frequently does the data change, what is the size of your lookup table etc. among other things. Once you use cache, the trip to database can be avoided, thereby enhancing performance.

Different types of caches can be used with lookup like static, dynamic, persistent, and shared(The dynamic cache cannot be used while creating an un-connected lookup). Each of these has its own identification. For more details, refer to Informatica Transformation Guide.

Lookup is a passive transformation, and can be used either connected or unconnected.  Typically, connected lookup is used when you want to do the lookup for all rows. When you have selective lookup requirements, its normally better to go for unconnected lookup. Unconnected Lookup can be used as if its a function call.

To read more, here is a good article -
What is lookup transformation in informatica? - IT Community

Firefox Smart Keywords

A very very useful tip.

In firefox its possible to make shortcuts for your favourite links (especially useful for searches etc).

For example, for a typical google image search, you would normally open images.google.com and then type in your required phrase and press search. That constitutes two trips to google image server to accomplish your search needs.

With this method from firefox, you can achieve this in one trip to server, just by doing a small trick with your favourites configuration.

You add a particular url to your favourites list. And later on, from the organize favourites dialog, assign that url a shortcut, a very small phrase e.g. img or images or something that you like.

Now, after this setup, when you want to search for images on keyword india, you'd just have to say your chosen keyword and india, i.e. just type in

img india

in the address bar and press enter.. your search results should be on the way...

More detailing here... Solutions - 25 Useful Firefox Smart Keywords

Blogged with the Flock Browser

PerlTips - 1

to escape particular characters in a given variable $var ->
$var =~ s/([CHARLIST])/\\$1/g;

(http://www.unix.org.ua/orelly/perl/cookbook/ch01_14.htm)

The complete Perl Bookshelf

http://www.unix.org.ua/orelly/perl/index.htm