Counting columns in a tab delimited file

It sounds so simple, however, when you sit down to write this, specially as a single line expression, it can take a while.

In my experiments I found it rather easy to count it with other delimiters as compared to TAB character.

Here is the command for counting columns

cat <FILENAME>| awk -F'\t' '{print NF}'

 cat can be slow at times, especially with larger files, therefore an alternative without that...

awk -F'\t' '{print NF}' <FILENAME>

 

how to find sql id of a long running sql in oracle

Sql for finding out sql id etc details of some long running query. Often useful for sending kill/monitoring instructions to DBA friends..

select distinct t.sql_id, s.inst_id,s.sid, s.serial#,s.osuser, s.program, s.status, t.sql_text

from gv$session s, gv$sqlarea t

where s.username = '<USERNAME>'

and s.sql_id = t.sql_id

and t.sql_text like '<%provide a segment of sql to identify it%>'

Partitioning in Informatica

Sourced from Internet

All transformations have some basic counters that indicate the number of input rows, output rows, and error rows.

Source Qualifier, Normalizer, and target transformations have additional counters that indicate the efficiency of data moving into and out of buffers. Use these counters to locate performance bottlenecks.

Some transformations have counters specific to their functionality. For example, each Lookup transformation has a counter that indicates the number of rows stored in the lookup cache.

When you view the performance details file, the first column displays the transformation name as it appears in the mapping, the second column contains the counter name, and the third column holds the resulting number or efficiency percentage. If you use a Joiner transformation, the first column shows two instances of the Joiner transformation:

	<Joiner transformation> [M]. Displays performance details about the master pipeline of the Joiner transformation.
	<Joiner transformation> [D]. Displays performance details about the detail pipeline of the Joiner transformation.

When you create multiple partitions, the Integration Service generates one set of counters for each partition. The following performance counters illustrate two partitions for an Expression transformation:

Transformation Name	Counter Name	Counter Value
EXPTRANS [1]	Expression_input rows	8
	Expression_output rows	8
EXPTRANS [2]	Expression_input rows	16
	Expression_output rows	16

Note: When you increase the number of partitions, the number of aggregate or rank input rows may be different from the number of output rows from the previous transformation.

The following table describes the counters that may appear in the Session Performance Details area or in the performance details file:

Transformation	Counters	Description
Aggregator and Rank Transformations	Aggregator/Rank_inputrows	Number of rows passed into the transformation.
	Aggregator/Rank_outputrows	Number of rows sent out of the transformation.
	Aggregator/Rank_errorrows	Number of rows in which the Integration Service encountered an error.
	Aggregator/Rank_readfromcache	Number of times the Integration Service read from the index or data cache.
	Aggregator/Rank_writetocache	Number of times the Integration Service wrote to the index or data cache.
	Aggregator/Rank_readfromdisk	Number of times the Integration Service read from the index or data file on the local disk, instead of using cached data.
	Aggregator/Rank_writetodisk	Number of times the Integration Service wrote to the index or data file on the local disk, instead of using cached data.
	Aggregator/Rank_newgroupkey	Number of new groups the Integration Service created.
	Aggregator/Rank_oldgroupkey	Number of times the Integration Service used existing groups.
Lookup Transformation	Lookup_inputrows	Number of rows passed into the transformation.
	Lookup_outputrows	Number of rows sent out of the transformation.
	Lookup_errorrows	Number of rows in which the Integration Service encountered an error.
	Lookup_rowsinlookupcache	Number of rows stored in the lookup cache.
Joiner Transformation (Master and Detail)	Joiner_inputMasterRows	Number of rows the master source passed into the transformation.
	Joiner_inputDetailRows	Number of rows the detail source passed into the transformation.
	Joiner_outputrows	Number of rows sent out of the transformation.
	Joiner_errorrows	Number of rows in which the Integration Service encountered an error.
	Joiner_readfromcache	Number of times the Integration Service read from the index or data cache.
	Joiner_writetocache	Number of times the Integration Service wrote to the index or data cache.
	Joiner_readfromdisk*	Number of times the Integration Service read from the index or data files on the local disk, instead of using cached data.
	Joiner_writetodisk*	Number of times the Integration Service wrote to the index or data files on the local disk, instead of using cached data.
	Joiner_readBlockFromDisk**	Number of times the Integration Service read from the index or data files on the local disk, instead of using cached data.
	Joiner_writeBlockToDisk**	Number of times the Integration Service wrote to the index or data cache.
	Joiner_seekToBlockInDisk**	Number of times the Integration Service accessed the index or data files on the local disk.
	Joiner_insertInDetailCache*	Number of times the Integration Service wrote to the detail cache. The Integration Service generates this counter if you join data from a single source.
	Joiner_duplicaterows	Number of duplicate rows the Integration Service found in the master relation.
	Joiner_duplicaterowsused	Number of times the Integration Service used the duplicate rows in the master relation.
All Other Transformations	Transformation_inputrows	Number of rows passed into the transformation.
	Transformation_outputrows	Number of rows sent out of the transformation.
	Transformation_errorrows	Number of rows in which the Integration Service encountered an error.
*The Integration Service generates this counter when you use sorted input for the Joiner transformation. **The Integration Service generates this counter when you do not use sorted input for the Joiner transformation.

If you have multiple source qualifiers and targets, evaluate them as a whole. For source qualifiers and targets, a high value is considered 80-100 percent. Low is considered 0-20 percent.

Timezones in Oracle

It might have been written umpteen times here and there, but it always manages to confuse me. So, here it is another time on the internet...

Datatypes
Timestamp/timezone datatype	What Oracle stores	What Oracle displays
TIMESTAMP WITH TIME ZONE	Year, month, day, hour, minute, second, fractional second, and time zone displacement (HH:MI difference from GMT)	Stored value
TIMESTAMP WITH LOCAL TIME ZONE	Year, month, day, hour, minute, second, fractional second; does NOT store time zone information, but instead converts data to the database time zone and stores it w/o time zone information	Converts the stored data to the session's time zone before displaying

With due respect to - http://toolkit.rdbms-insight.com/tz.php

And now.. a Solar Powered ubuntu laptop...

Nick Rutledge has conceptualized a thin laptop that runs Ubuntu, is beautiful and hopes to run on Solar power... isnt that a killer combination...

check out his concept here...

http://nrutledge.blogspot.com/p/ubuntu-laptop-concept.html

Data Lineage.. what is that ?

It is one of those buzzwords, that keep doing the circuit every once in a while. Almost every enterprise wants to do the analysis regarding this, and is almost always hard to find people with knowledge/experience doing this kind of analysis.

For the unaware, Data Lineage is basically (really in very short words) a study of the data from its source to its eventual target, similar to what we'd do for our generation tree, we analyze the generation analysis of the data we are dealing with.

Starting from the source of the data, it travels through different subsystems, sometimes going through transformations, and thus possibly changing shape too...

Informatica had a very interesting blog post around this (already in 2007), which can turn out to be fairly informative.

http://blogs.informatica.com/perspectives/index.php/2007/05/21/data-lineage-where-did-that-data-come-from/

Informatica & hadoop... solutions for future ?

Distributed computing using hadoop has taken the IT industry by a whirlwind in the last few years.  After getting almost "adopted" by yahoo, hadoop has progressed quite fast, and is now maturing slowly but steadily.

More and more enterprise solution providers are annoucing their support for the hadoop platform, hoping to get a pie of the big Data business chunk.  Its possibly a fair thing to expect that the leader in Data Integration business solutions space, Informatica has also announced a tie up with Cloudera, for porting Informatica platform to hadoop.

Though the exact details are yet to come out, the possibilities are endless.  With hadoop (and its inherent distributed computing based on map/reduce technology), informatica can actually think of processing big data in sustainable time frames.

For one my customers, I deal with about 200 million rows of data per day in one job.  Besides the issues with oracle in tuning the query etc, the informatica component itself consumes times in terms of hours.  With map reduce in place, I hope to get that in minutes, oracle issues notwithstanding.

Although word about hadoop is spreading quite fast, its adoption (from buzzword to actual usage in enterprise) is not as fast.  To aid their cause, Informatica and cloudera have started an interesting series of webinars, termed as "hadoop tuesdays".  Its free to join, and they get experts to talk about various related issues around hadoop and big data and informatica.  Its been very useful and informative so far.

Switching defaults in Ubuntu

Ubuntu allows you to have multiple alternatives installed for many software.. for example, java.
You can have the default open jdk installed, and then you can actually have the Sun version installed.

For example, to see what alternatives are installed for your software, try going to /etc/alternatives. Here you'd see many pieces of software with alternatives listed.

With these software installed, you would need to point your system to use one of them as the default, this is important especially after installing a newer version of the software.

In such a case, to switch the alternatives, you need to use this

sudo update-alternatives

If you do a man on update-alternatives, there is a plethora of options to use.

For our example, to configure the default for java, use this

sudo update-alternatives --config java

writing for cloudtimes.org these days

Read some of my content at http://cloudtimes.org/author/raghav/

HTML 5

I attended the .WEB day of GIDS (The Great Indian Developer Summit) 2011 edition.


Among many talks, there were two focusing on HTML5. One by Scott Davis (of http://www.thirstyhead.com/) and Venkat Subramaniam (of http://www.agiledeveloper.com) .  Scott's talk was more on the conceptual and capability side of HTML 5.  Venkat focused more on the implementation and initiating newbies to HTML 5 coding.


Before these discussions, I would not have been able to say much on the capabilities of HTML5. It was more of a buzzword before, however, now its more of another technology holding lot of promise.   I think that should say a lot for the two speakers, that within two sessions, they have been able to lift the standard of know how around a cutting edge technology from buzzword to daily use.


Both these talks, put together were able to provide a rather complete picture. Enumerate the benefits, major improvements, new tags which are bringing in so much functionality to native HTML without need of any third party libraries, plugins etc.


Of course its cutting edge today, since not all browsers support all of the HTML 5 specification. The specification is huge in itself anyway.  As one of the speakers put it, the HTML 5 spec is a combination of HTML plus all of CSS 3 plus a lot of RIA functioanlities based on JavaScript libraries.  One can say that html5 is rather heavy from browser engine side, however, it intends to provide all the features across the browsers (eventually).   Since its a huge spec, not all browsers implement it  ** completely and ** uniformly.


There would be a time when all the browsers (at least the leading ones) would implement it completely (or almost all of it), but till then, the developers would have to live with polyfill (polimorphically backfill) the html5 functionality for non supporting browsers.   A javascript library at www.modernizer.com is a big help in implementing this transparently.


As Scott very aptly put it, "We'd program for the faster animal in the herd, and allow the rest of the slower ones to polyfill. As and when they catch up with the fastest one, need for polyfill will automatically go away".


From what I see in html5 spec (what ever part that I have come to know), it looks very very interesting and powerful.  Lots of current functionality that is implemented today with the help of third party libraries/plugins is going to be implemented natively.


And, let me not forget to mention the one single most important innovation that is coming through with html5, semantic web.  Its not really a set of tags or something similar, rather a concept.   There are tags available in spec, which actually indicate the semantics (meaning) of the content. For example, there is a tag called

. This tag wont do much on its own, but when someone is reading the code, or for that matter the parser program is going through the code, the tag name already says that its a footer.  The tag name actually means something.  This also paves way for future improvements on the implementation side.


Perhaps a separate post for html5 possibilities for mobile applications, a huge area in itself.


Resources

1. www.html5rocks.com
2. www.html5doctor.com
3. www.diveintohtml5.org -> this is a unique one, a complete book on html 5, which is available free of cost, completely online.  One of the finest resources for html5.
4. www.html5demos.com
5. www.modernizer.com  -> javascript library for polyfill

   ## Technorati - CJYMQNJWMX9K