AWK, My Number 1 Data Manipulation Tool

_{made with canva}

Good day, fellow developers, I haven't shared a post here in a long time. Today, I would like to talk about awk, an incredible tool for all UNIX based systems. Since learning about awk, I've had less stress when manipulating any type of data because awk was designed to make it easier.

If you're looking for a tool to take your data manipulation and analysis to the next level, you should look into Awk. Awk is a powerful tool for shell scripting and data manipulation that allows you to quickly and easily manipulate data on the command line. It's incredibly versatile and can be used to write simple scripts to parse data from text files and to perform complex data manipulation operations.

I used to perform most of my data manipulation stuff with Excel, but I just realized awk does it better and with ease. I now use awk for all of my data manipulation tasks, including text searching, filtering, and modification. Regardless of file size, awk handles data manipulation and analysis operations efficiently. With Awk, you can quickly extract data from a text file, transform it into a more useful format, and analyze it without having to type out lengthy commands or write complex scripts.

In the following demonstration, I used awk to convert a text file containing comma-separated values of student details into a more meaningful format.

You can see in the screenshot below that I have a text file with values separated by commas. You can see that the data is almost not meaningful and can't be used for a presentation. Assuming you want to make a presentation with this data.

To manipulate this data with awk, I can either write the whole script on the command line and execute it or create a reusable script that I can easily reuse for my subsequent data manipulation on similar data.

Here I will be writing the script to manipulate the data in a text editor and saving it as a .awk file, which I can always reuse. The screenshot below shows the script used to convert the data shown below into a nicely formatted data set with additional calculations.An awk script file contains three parts. The first part is "BEGIN", which contains the heading section of your resulting data. The second part contains code to manipulate the data. And the final part, "END," displays whatever you want after manipulation.Check out the script below:

    FS=","
    print " Display of Students Results   "
    
    # printing header row
    printf "%-10s\t%-20s\t%-10s\t%-15s\t%-10s\t%-10s\n", "Stdt ID", "Stdt Name", "Class Mrk(max 70)", "Exm Rslt(max 50)","Final Results(%)", "Status"
    printf "----------------------------------------------------------\n"
}
# process data rows
NR > 1 {
    studentID = $1
    studentName= $2
    CSM = $3
    examMark= $4
    finalResults = ((CSM+examMark)/120)*100
    if (finalResults < 50){
      status = "Failed"}
    else {status="Passed"}
{
 # printing the formatted row
    printf "%-10s\t%-30s\t%-20s\t%-10s\t%-20s\t%-10s\n", studentID ,studentName, CSM, examMark, int(finalResults), status
    # update total number of students
    num_ofStudents += 1
} 
}
# print totals if data rows exist
END { 
        printf "-----------------------------------------------------------\n"
        print "                                    "
        printf "     Total Students:\t%-10d\n", num_ofStudents
}

In the above code, we have all the three parts in our script. To run the script, we open the terminal and run this command awk -f your_script.awk your_data.awk. To save the results in a new txt file say results.txt we add it like this awk -f your_script.awk your_data.awk < results.txt The screenshot below shows how the result will look that when ran on the terminal.

You can see how nicely awk has formatted our data with ease. As if that weren't enough, Awk is also incredibly easy to learn and use. Even if you're new to Linux and shell scripting, you can quickly learn how to use Awk to perform data manipulation and analysis operations with ease. Plus, there are plenty of online resources and tutorials available to help get you up and running with Awk in no time.