2009-05-25

Extract variable types from a text file using awk

The most difficult tasks in data analysis are not, from my experience, the statistical or analytic tasks. Instead, the most time consuming and frustrating parts of analysis are simply extracting and transforming raw data into a format that can be analyzed by common software. A common task in this vein is to determine what type of variables are contained in a text file containing fields and records (e.g. a comma separated values file or .csv). Most commonly, these data types are one of either strings, integers or decimal numbers called doubles. Some software, such as R does a good job at automatically recognizing what kinds of data are contained in a text file through commands like read.table. Other software, such as the GIS software GRASS, may require you to explicitly input data types of each column of data. I find that to be time consuming, especially when you may have dozens of columns or dozens of text files to work through. I have written the following script in awk (technically, gawk) to automatically determine what type of data is contained in a columnar text file. Awk is a programming language specifically designed to parse and transform these text files. Awk scripts can often developed to be included as short snippets of larger bash scripts. Such is the case here. It could probably be done more easily in Perl, but I was already working on a bash script that this will go into, which I will post later. The task accomplished by the script is a simple one. Create a comma separated string that lists the variable names and data types for each column in a text file. This string will ultimately be used as input to the command v.in.ascii in GRASS, which creates a point vector dataset from a text file.

typeVars.awk

#!/bin/awk -f
##
## file: typeVars.awk
## created by: R. Todd Jobe <toddjobe@unc.edu>
## date: 2009.05.25
## This script will print the variable names and types from a 
## columnar text file.
## Specifically, it can generate a variable declaration for GRASS 
## input.
{ 
    # Set the types to default values if not defined
    if(str=="") str="var"
    if(it=="") it="integer"
    if(dbl=="") dbl="double precision"

    # Get the column name from the 1st row
    if(NR==1){
 for(i=1; i <= NF; i++){
     type[i,1]=$(i)
 }
    }else if(NR==2){

        # Get the column type from the 2nd row
 j=0
 for(i=1; i <= NF; i++){
     if($i ~ /[^-.0-9]/){
  type[i,2]=str
  strf[++j]=i
  if(str=="var"){
      max[i]=length($i)
      type[i,2]=sprintf("%s(%.0f)",str,max[i])
  }
     }else if($i ~ /\./){
  type[i,2]=dbl
     }else{
  type[i,2]=it
     }
 }
    }else{
        # Get the maximum column width for strings
 if(str=="var"){
     for( k in strf ){
  if(length($(k)) > max[k]) {
      max[k] = length ($k)
      type[strf[k],2]=sprintf("%s(%.0f)",str,max[k])
  }
     }
 }
    }
}
END{
    ORS=""
    out=sprintf("%s %s",type[1,1],type[1,2])
    for(l=2;l<i;l++){
 out=sprintf("%s, %s %s",out,type[l,1],type[l,2])
    }
    print out
}

An Example

Here's an example of using the typeVars.awk script. We begin with a csv file of different variable types. typeVars.awk is executed on this csv and the output is a single string of comma separated column names with variable types. By default the output is formatted for use in GRASS and strings are labelled with their maximum length.
bash-3.2$ cat trial.csv
astring,aint,adbl,plot,x,y
abc123,1234,1.23,zyx1234,54,1.45
abc1235,1234,2.587,z*x1234,360,1.45
abc12,12345,1.23,zy.1234,1,1.45
abc1,999999,1.23,zx1234,0,1.45
bash-3.2$ echo `typeVars.awk -F, trial.csv`
astring var(7), aint integer, adbl double precision, plot var(6), x integer, y double precision
Notice -F parameter is specified in the call to typeVars.awk. Whitespace is the default field separator in awk, so in order to parse a csv this needs to be set to a ",".

Labels: ,

275 Comments:

«Oldest ‹Older 201 – 275 of 275
Blogger Girija B said...

the extraction of variable types are more important to all the programming languages. This blog easily explains the extraction of variable types.CCNA Training in Chennai

March 3, 2016 at 11:08 PM  
Blogger Girija B said...

this is one of the notable blog for the variable types extraction method.CCNA Training in Chennai

March 3, 2016 at 11:11 PM  
Blogger Fathima Fazal said...

Variables are highly used during training of java technology. I would like to suggest you that IICT is the best Java Training Institute in Chennai managed by MNC Experts. IICT Reviews shows the standard and quality of training.

March 17, 2016 at 12:01 AM  
Blogger Gopi Perumal said...

Excellent Post, I welcome your interest about to post blogs. It will help many of them to update their skills in their interesting field.
Regards,

Regards,
CCNA Training in Chennai | CCNA Training Institute in Chennai | Best CCNA Training in Chennai

March 26, 2016 at 9:02 AM  
Blogger Ridhima said...


All are saying the same thing repeatedly, but in your blog I had a chance to get some useful and unique information, I love your writing style very much, I would like to suggest your blog in my dude circle, so keep on updates.


Java training in Adyar

April 4, 2016 at 10:28 AM  
Blogger Raghu Sharma said...

Selenium is a testing tool which is an open source which can work very well for the professionals and for testers. If a profesional is knowing manual testing then if he learn seenium then there are good chance for career growth.
selenium training in chennai | selenium training institute in chennai | selenium course in chennai

April 5, 2016 at 7:35 AM  
Blogger Priya said...




The structs is a complex data type declaration used in the C programming language that helps you to define a physically grouped list of variables to be placed under one name in a block of memory.
struts training in chennai | struts training | struts training center in chennai

April 8, 2016 at 8:44 AM  
Blogger Khalid Baid said...


MIDSUMMER SEASON WEDDING WEAR SHOES
Handbag & Clutches For Hot Girls
Front Open Double Shirt
Fashion Gallery Lehenga Choli
Stylo Best Mehndi Designs
HANDBAGS FOR WOMEN FASHION
Latest Sherwani Designs
Bridal Jewellery Set
Zara Shahjahan Eid Dresses
Mehndi Patterns for EID
SUMMER SEASON LADIES DRESSES FASHION
Sophia Tolli Collection
Earrings In Gold Collection
Actress Maya Ali – Fashion Collection
Bridal Gowns Collection
LADIES BLAZER STYLES OUTFITS
MEHNDI DRESS DESIGNS
BRIDAL SHOES
SHIRTS GRAY MAXI SKIRT SKIRTS
BRIDAL DRESSES WESTERN STYLE
LAWN AND CHIFFON OUTFITS
classic lawn suits
mix eid dresses
midsummer kurta
anarkali suits

April 10, 2016 at 12:27 AM  
Blogger Murali Rajesh said...

Selenium is the best tool for software testing automation and it is used globally by the top companies across globe. Get training to use selenium and star yur career as a siftware tester today.
Selenium training in Chennai | Selenium course in Chennai | Selenium training institute in Chennai

April 27, 2016 at 9:55 AM  
Blogger for IT the said...

Interesting Article

Java Training in Chennai | Core Java Training in Chennai

April 28, 2016 at 5:57 AM  
Blogger for IT the said...

Online Java Training | Java EE Training in Chennai

April 28, 2016 at 5:57 AM  
Blogger Chinni Bocha said...

nice article about echology..
Best SEO training in hyderabad all modules are clearly solved and practice with guide.seo training in hyderabadAnd clearly getting knowledge of seo.

May 19, 2016 at 3:11 AM  
Blogger Chinni Bocha said...

nice blog about computational..
SEO training in hyderabad by experts in digital markeing And by prosessional experts in seo.All the training by placement and also guide by the professionals.SEO training in hyderabad

May 19, 2016 at 4:38 AM  
Blogger Bay Max said...

Great blog.More vacancies are available in IT industry.To get a good career join with us.
Regards
Bigdata Training in chennai | Bigdata Courses in chennai



May 20, 2016 at 8:55 AM  
Blogger Chinni Bocha said...

Great sharing .. where there is number of applications for 1 job.,Here learn seo and earn money or get a job..SEO training in hyderabad by experts in digital markeing And by prosessional experts in seo.All the training by placement and also guide by the professionals.SEO training in hyderabad

May 24, 2016 at 12:57 AM  
Blogger mary Brown said...

Java Training in Chennai
Online MVC Training India | Angularjs Training | javase j2ee javaee interview questions | Java Training in Chennai

June 2, 2016 at 2:56 AM  
Blogger Rohith said...

Yes, you are absolutely correct...And it is very informative and very clear and easy to understand.. seo training in chennai

June 25, 2016 at 5:47 AM  
Blogger Vasu N said...

good explain the programming coding
salesforce training in hyderabad

July 4, 2016 at 3:16 AM  
Blogger Madhu Manohar said...

Useful Information
one and only affiliate agency in south INDIA, earn money online from affiliate network in india http://publishers.praiseads.com/

July 5, 2016 at 8:18 AM  
Blogger Venky Praise said...

Freshers jobs way provide freshers jobs ,walkins,direct recruitment,openings for freshers and experienced and recruiting experienced and freshers Walkins

July 8, 2016 at 12:54 AM  
Blogger Venky Praise said...

Helpful Information
Freshers jobs way provide freshers jobs ,walkins,direct recruitment,openings for freshers and experienced and recruiting experienced and freshers Walkins

July 8, 2016 at 1:15 AM  
Blogger Chinni Bocha said...

nice posts..
sap fiori training in hyderabad



July 9, 2016 at 6:42 AM  
Blogger Chinni Bocha said...

nice posts..

SAP Hana online training in hyderabad


July 9, 2016 at 8:54 AM  
Blogger Chinni Bocha said...

EXcellent share about compuaional ecology..
SAP HANA

online training




July 11, 2016 at 7:08 AM  
Blogger Chinni Bocha said...

Excelent posts..
SAP GRC training in

hyderabad

July 11, 2016 at 8:12 AM  
Blogger prema said...

useful information

Online recruitment for bank jobs and government jobs and you can get Notification and application to apply

online for bank jobs and govt jobs

July 14, 2016 at 7:04 AM  
Blogger prema said...

useful information

Online recruitment for bank jobs and government jobs and you can get Notification and application to apply

online for bank jobs and govt jobs

July 14, 2016 at 7:05 AM  
Blogger prema said...

useful information

Online recruitment for bank jobs and government jobs and you can get Notification and application to apply

online for bank jobs and govt jobs

July 14, 2016 at 7:43 AM  
Blogger Balusomu said...

Recruitment voice contains Daily GK Updates, Bank Recruitment, Government jobs, Bank jobs, Interview Tips, Banking News, GK Updates bank recruitment

July 15, 2016 at 6:57 AM  
Blogger Balusomu said...

Recruitment voice contains Daily GK Updates, Bank Recruitment, Government jobs, Bank jobs, Interview Tips, Banking News, GK Updates bank recruitment

July 15, 2016 at 7:08 AM  
Blogger Balusomu said...

Useful Information:
Telugu Cinema Contains Telugu Cinema News, Latest Movie Reviews, Actor, Actress, Movie Galleries And Many More Telugu Cinema

July 15, 2016 at 8:52 AM  
Blogger Balusomu said...

Useful Information:
Telugu Cinema Contains Telugu Cinema News, Latest Movie Reviews, Actor, Actress, Movie Galleries And Many More Telugu Cinema

July 15, 2016 at 9:23 AM  
Blogger prema said...

Useful information:
General Awareness and Current Affairs with Latest Current Affairs Quiz, GK Updates, General Knowledge Last 6 Months Current Affairs, Placement Papers, Model Papers, Interview Questions @ examvalue.com

July 18, 2016 at 8:54 AM  
Blogger prema said...


useful information

Online recruitment for bank jobs and government jobs and you can get Notification and application to apply online for bank jobs and govt jobs

July 28, 2016 at 3:01 AM  
Blogger prema said...

Useful information:
General Awareness and Current Affairs with Latest Current Affairs Quiz, GK Updates, General Knowledge Last 6 Months Current Affairs, Placement Papers, Model Papers, Interview Questions @ examvalue.com

July 28, 2016 at 6:53 AM  
Blogger Rakesh S said...

extract variable types from text file nice posts..
Hadoop training in hyderabad.All the basic and get the full knowledge of hadoop.
hadoop training in hyderabad

August 13, 2016 at 1:06 AM  
Blogger Gunardi omex said...

What’s up, its fastidious piece of writing on the topic of media print, we all understand media is a impressive source of data. Bye

Tangki Fiberglass
Jual Septic Tank
Jual Tangki Kimia
Jual Talang Fiber
Jual Rotameter
Tangki Panel

August 28, 2016 at 5:16 AM  
Blogger mythily mythu said...

Thanks for sharing this information.
/business-Intelligence training in chennai

September 25, 2016 at 3:39 AM  
Blogger mythily mythu said...

This blog is informative.The coding explains about how to extract files from thetext file is clearly explained.
oracle-rac training in chennai

September 25, 2016 at 3:42 AM  
Blogger UMA MAHESWARI said...

• great article!!!!!This is very importent information for us.I like all content and information.I have read it.You know more about this please visit again.
oracle training in chennai

September 29, 2016 at 9:02 AM  
Blogger UMA MAHESWARI said...

Thanks for sharing this information,
unix training in chennai

September 29, 2016 at 9:03 AM  
Blogger vignes waran said...

This blogs informative is very excellent.I want to some coding extract file how to replace name and the extract file name.Help me and update solve for this solution

Hadoop Online Training

October 17, 2016 at 1:40 AM  
Blogger vignes waran said...

This blogs informative is very excellent.I want to some coding extract file how to replace name and the extract file name.Help me and update solve for this solution

Hadoop Online Training

October 17, 2016 at 1:40 AM  
Blogger vignes waran said...

This blogs informative is very excellent.I want to some coding extract file how to replace name and the extract file name.Help me and update solve for this solution

Hadoop Online Training

October 17, 2016 at 1:40 AM  
Blogger for IT the said...

I have read your blog its very attractive and impressive. I like it your blog.

Java Training in Chennai Core Java Training in Chennai Core Java Training in Chennai

Java Online Training Java Online Training Core Java 8 Training in Chennai Core java 8 online training JavaEE Training in Chennai Java EE Training in Chennai

October 27, 2016 at 5:11 PM  
Blogger for IT the said...

Java Online Training Java Online Training Java Online Training Java Online Training Java Online Training Java Online Training

Hibernate Online Training Hibernate Online Training Spring Online Training Spring Online Training Spring Batch Training Online Spring Batch Training Online

October 27, 2016 at 5:11 PM  
Blogger for IT the said...

Java Training Institutes Java Training Institutes Java Spring Hibernate Training Institutes in Chennai | Hibernate Training Institutes in Chennai Spring Training Institutes in Chennai Spring Training Institutes in Chennai

October 27, 2016 at 5:11 PM  
Blogger mythily mythu said...

This blog is informative...
SSRS training in chennai

November 2, 2016 at 1:11 AM  
Blogger UMA MAHESWARI said...

Great articles, first of all Thanks for writing such lovely Post!
hadoop training in chennai

November 4, 2016 at 9:12 AM  
Blogger UMA MAHESWARI said...

Thank you for sharing your useful information.
obiee training in chennai

November 6, 2016 at 12:46 AM  
Blogger 1croreprojects said...

Thanks for sharing this valuable information.
ieee java projects in chennai
ieee dotnet projects in chennai
mba projects in chennai
be projects in chennai
ns2 projects in chennai
mca projects in chennai
bulk projects in chennai
bsc projects in chennai
msc projects in chennai

November 11, 2016 at 1:53 AM  
Blogger surveyforms said...

Actually picture is the presentation of some one’s feelings; it provides the lesson to the viewers.

------------------------
Online form builder

December 22, 2016 at 11:08 AM  
Blogger surveyforms said...

Actually picture is the presentation of some one’s feelings; it provides the lesson to the viewers.

------------------------
Online form builder

December 22, 2016 at 11:09 AM  
Blogger dhivya priya said...

Finding the time and actual effort to create a superb article like this is great thing. I’ll learn many new stuff right here! Good luck for the next post buddy..
Isoft Innovation
Isoft Innovations Facebook

February 24, 2017 at 5:15 AM  
Blogger sunilkumarkuppam said...

Thanks for sharing this information and keep updating us. This is more informatics and it really helped me to know the Android.
Android Training in Chennai | Android Course in Chennai | Android Training Chennai

March 29, 2017 at 8:40 AM  
Blogger Neemu NA said...

Thanks for sharing useful information articles, Hi We at Colan Infotech Private Limited a
Mobile apps development companies in chennai.
we always desire to solicit our customer's fruitful experience with us, we are the top notch
Mobile App Development Company in chennai

April 10, 2017 at 9:34 AM  
Blogger vanu3vj said...

Thanks for the awesome share
We at Colan Infotech Private Limited best web design company in chennai,is Situated in US and India,
will provide you best service in qa testing services
Design Services and Colan Infotech has a group of exceedingly dedicated, inventive and creative experts with an energy for delivering exciting ,
helpful and stylish Web and Mobile Applications, We are one of the best software testing services company
and of course we stepped in bangalore too we are best qa and testing services
can provide quality assurance and testing services,
we are the best among the software testing company india

April 10, 2017 at 9:59 AM  
Blogger Neemu NA said...

Thanks for sharing this information article
Hi we at Colan Infotech Private Limited , a company which is Situated in US and India, will provide you best service and our talented team will assure you best result and we are familiar with international markets, We work with customers in a wide variety of sectors. Our talented team can handle all the aspects of custom application development, we are the best among the dot net development companies in Chennai . asp .net web development company We have quite an extensive experience working with asp .net development services. we are the only asp.net web development company which offer custom services to a wide range of industries by exceeding our client’s expectations. You can even interact directly with the team regarding your project, just as you would with your in-house team. hire asp.net programmers to achieve your dream product.
Custom application development company, asp.net development companies, asp .net web development company,Hire asp .net programmers,asp.net web development services,dot net development companies in chennai Hire asp .net programmers/ hire asp .net developer  . We are the best asp .net development company providing top notch asp .net development services Hire asp .net programmers/ hire asp .net developer  . We are the best asp .net development company providing top notch asp .net development services.

April 11, 2017 at 8:27 AM  
Blogger niyam faz said...

Thanks for sharing useful information article to us keep sharing this info,
Hi We at Colan Infotech Private Limited a
Mobile application development company in chennai,
is Situated in US and India, will provide you best service in
enterprise mobile app development company .
and Colan Infotech has a group of exceedingly dedicated, inventive and creative experts with an energy for delivering exciting , helpful and stylish Web and Mobile Applications, We work with customers in a wide variety of sectors.
We design all of our websites and applications using the responsive web design approach. Our talented team can handle all the aspects of mobility so we are rated as best service provider in
Mobile apps development companies in chennai.
We solidly trust that our customers start things out and there is not a viable alternative for quality of service.
We offer custom services to a wide range of industries by exceeding our client’s expectations. You can even interact directly with the team regarding your project, just as you would with your in-house team. we always desire to solicit our customer's fruitful experience with us, we are the top notch
Mobile App Development Company in chennai
and mobile app development companies in Bangalore. We can provide best
mobile app development chennai .
We can provide cutting edge technology services in
Mobile application development in chennai.
Reach us for mobile app development chennai or just call us for best
mobile app developers in chennai .

April 20, 2017 at 2:23 AM  
Blogger Mehgna Sharma said...

We need to thank admin for publishing useful information.
catia summer trainingin Noida | Android 6months Industrial Training in Noida | Python Training in Noida

May 23, 2017 at 9:11 AM  
Blogger Mehgna Sharma said...

I like your writing. thanks for sharing that with us all.
catia summer trainingin Noida | Android 6months Industrial Training in Noida | Python Training in Noida

May 23, 2017 at 9:14 AM  
Blogger sunilkumarkuppam said...

This article helps me to know about the essential skills for learning the Cloud Computing and keep on updating us. This content is really helped me to know about the importance of Cloud Computing. Cloud Computing Training in Chennai | Cloud Computing Training | Cloud Computing Courses

May 23, 2017 at 10:15 AM  
Blogger Aptron said...

Thanks for sharing such a great information..Its really nice and informative..

scada 6 months project based industrial training
Jvhdl 6 months project based industrial training
ansys 6 months project based industrial training

May 24, 2017 at 8:00 AM  
Blogger sunitha vishnu said...

It is amazing and wonderful to visit your site.Thanks for sharing this information,this is useful to me...
Android Training in Chennai
Ios Training in Chennai

June 17, 2017 at 1:56 AM  
Blogger sapna negi said...

Thank you for sharing this .Extract variable types from a text file using awk". it was really good .
best sap fiori institute in gurgaon|
best sap xi-pi institute in gurgaon

June 21, 2017 at 1:50 AM  
Blogger sunitha vishnu said...

It is amazing and wonderful to visit your site.Thanks for sharing this information,this is useful to me...
Android Training in Chennai
Ios Training in Chennai

August 4, 2017 at 1:25 AM  
Blogger sunitha vishnu said...

It is amazing and wonderful to visit your site.Thanks for sharing this information,this is useful to me...
Android Training in Chennai
Ios Training in Chennai

August 4, 2017 at 2:16 AM  
Blogger kingrani said...

SSC JHT Paper II Exam Results
UPTET Recruitment Notification

August 8, 2017 at 6:33 AM  
Blogger sunitha vishnu said...

It is amazing and wonderful to visit your site.Thanks for sharing this information,this is useful to me...
Android Training in Chennai
Ios Training in Chennai

August 10, 2017 at 2:12 AM  
Blogger sunitha vishnu said...

It is amazing and wonderful to visit your site.Thanks for sharing this information,this is useful to me...
Android Training in Chennai
Ios Training in Chennai

August 11, 2017 at 2:43 AM  
Blogger sunitha vishnu said...

It is amazing and wonderful to visit your site.Thanks for sharing this information,this is useful to me...
Android Training in Chennai
Ios Training in Chennai

August 11, 2017 at 2:43 AM  
Blogger viji said...

you are posting a good information for people and keep maintin it and give more updates too.

Digital Marketing Company in Chennai

Digital Marketing Agency in Chennai

Digital Marketing Companies Chennai

Digital Marketing Services in Chennai

Digital Marketing Company Chennai

Digital Marketing Agency Chennai

August 24, 2017 at 6:20 AM  
Blogger 7th Street Club said...

Nice detail you provided thanx for the data you provided here
ofb semi skilled result
CSIR UGC NET Result
Board exam result 2018
Railway Recruitment

October 3, 2017 at 5:54 AM  
Blogger Guna Sekar said...

Excellent Post, Interesting Article.Indias Fastest Local Search Engine, you can Search Anything, From anywhere at any time CALL360

October 17, 2017 at 2:26 AM  
Blogger Guna Sekar said...

Excellent Post, Interesting Article.Indias Fastest Local Search Engine, you can Search Anything, From anywhere at any time CALL360

October 17, 2017 at 2:28 AM  
«Oldest ‹Older 201 – 275 of 275

Post a Comment

Subscribe to Post Comments [Atom]

<< Home