Do Know Evil – Tyler Mulligan’s Tips and Tricks

My favorite thing about Linux is the terminal. I use it countless times a day to do all sorts of tasks, like managing game servers or writing scripts to do tedious tasks. One of the most popular things I do in the terminal is search for files or strings inside of files and today I’d like to go over a few methods and tricks for doing this. There are three tools that make this task amazingly easy but combining them is where we find the real power. These three tools are locate, find and grep. I will cover the basic use of these tools with some examples and tricks but I suggest you take a look at the man pages for the tools for addition information (i.e. man locate).

locate

Locate has got to be the most straight forward way to search for files. It uses a database so it’s blazing fast when searching your entire computer, compared to the ‘find’ we’ll cover later. It’s not available by default on all Linux distributions (it is on Ubuntu) so you might have to install it. Since I’ll be mentioning fstab later, I’ll give my examples with it.

Basic syntax is: locate [filename]

tyler@quadjutsu:~$ locate fstab
/etc/fstab
/etc/fstab.orig
/etc/fstab.pre-ntfs-config
/etc/fstab~
/usr/include/fstab.h
/usr/lib/udev/migrate-fstab-to-uuid.sh
/usr/share/apps/katepart/syntax/fstab.xml
/usr/share/doc/m4/examples/fstab.m4
/usr/share/doc/mount/examples/fstab
/usr/share/doc/util-linux/examples/fstab.example2
/usr/share/man/man5/fstab.5.gz
/usr/share/pysdm/fstab.py
/usr/share/pysdm/fstab.pyc

This database will automatically update itself at various times but to force an update type the following command:

tyler@quadjutsu:~$ sudo updatedb

Folder exclusions can be found (and edited) in /etc/updatedb.conf

By default (at least in ubuntu) /media (your mounted media) is not included in this database. This means if you use extra drives you’ve added to your computer and you want them to be searchable through the locate tool, you’ll have to mount them to a directory like /mnt.

To mount a drive on boot, you’ll need to add a line like the following to your /etc/fstab. The one below mounts an ntfs drive to /mnt/mydrive. That folder must exist for the drive to be mounted.

/dev/sda1 /mnt/mydrive ntfs-3g defaults,locale=en_US.utf8 0 0

I found the /dev/sda1 part by listing my harddrive partitions using the following command:

sudo fdisk -l

find

Find is a little more cryptic than locate but it’s a very powerful tool that should be available on most Linux distributions if not all.

Basic syntax is: find [directory] -name “[string]” -print

Find will search recursively through the folder you’re calling it from. The directory isn’t required but it will show you the full path. A neat trick is to use $(pwd) which creates a string for the “present working directory”. It’s also a good habit to use quotes around your name search because you can’t do regular expressions without it. find does not do matching the same way locate does. You’ll have to specific a wild card (*) for partial searches.

tyler@quadjutsu:~/Desktop$ find -name "notes" -print
tyler@quadjutsu:~/Desktop$ find -name "notes*" -print
./notes.txt~
./notes.txt
tyler@quadjutsu:~/Desktop$ find $(pwd) -name "notes*" -print
/home/tyler/Desktop/notes.txt~
/home/tyler/Desktop/notes.txt
tyler@quadjutsu:~/Desktop$ find -name "no[a-z]*" -print
./notes.txt~
./notes.txt

My Buddy From Belgium, MrBougo has asked I make note that -iname makes it case insensitive.

grep

In the context of searching, grep is like a pocket knife. It’s great for limiting returned results and searching through files, which are my two most common uses of it, though I’m sure there are a million others. First I’ll cover the searching files for strings portion and in combined tools we’ll discuss how to refine search results.

In terms of searching strings in files, basic syntax is: grep “[string]” [filename]

grep is case sensitive but can be changed to insensitive with -i. You can access extended regular expressions (which allow for such functions as use of the + sign to signify 1 or more characters) by calling “egrep”

tyler@quadjutsu:~$ grep "tyler" resume.txt
tyler@detrition.net
tyler@quadjutsu:~$ grep "Tyler" resume.txt
Tyler J. Mulligan
tyler@quadjutsu:~$ grep -i "tyler" resume.txt
Tyler J. Mulligan
tyler@detrition.net
tyler@quadjutsu:~$ grep -i "tyl" resume.txt
Tyler J. Mulligan
tyler@detrition.net
tyler@quadjutsu:~$ egrep -i "tyler [a-z.]+" resume.txt
Tyler J. Mulligan
tyler@quadjutsu:~$ egrep "tyler [a-z.]+" resume.txt

MrBougo also mentioned that, grep returns a 1 exit status if it doesnt find, so grep “foo” && grep “bar” will only grep for bar when foo is found.

Combining the tools

There are two very important techniques for linking together commands in the terminal. The pipe | and &&. The pipe passes the output of one command to the next, the && runs a command after the one before is complete.

Starting off slow, we’ll refine our search of grep with another grep. Note that you don’t have to use the quotes as I’ve shown below.

tyler@quadjutsu:~$ grep -i "nexuiz" resume.txt
    * Web Developer and Interaction Designer for Nexuiz / Alientrap
    * Owner and Creator of Nexuiz Ninjaz
tyler@quadjutsu:~$ grep -i "nexuiz" resume.txt |grep -i ninjaz
    * Owner and Creator of Nexuiz Ninjaz

Not that you would really want to do the follow commands but to emphasize how | and && work, the following example shows how you’re running the command twice rather than refining a search:

tyler@quadjutsu:~$ grep -i "nexuiz" resume.txt && grep -i "ninjaz" resume.txt 
    * Web Developer and Interaction Designer for Nexuiz / Alientrap
    * Owner and Creator of Nexuiz Ninjaz
    * Owner and Creator of Nexuiz Ninjaz

Recalling our first example with fstab, there was a lot of extra results we didn’t want. We know know it was in the etc folder, so we can filter our results by that.

tyler@quadjutsu:~$ locate fstab |grep etc
/etc/fstab
/etc/fstab.orig
/etc/fstab.pre-ntfs-config
/etc/fstab~

Another way would be to EXCLUDE folders we don’t want with the -v flag.

tyler@quadjutsu:~$ locate fstab |grep -v usr
/etc/fstab
/etc/fstab.orig
/etc/fstab.pre-ntfs-config
/etc/fstab~

And filtering out the junk like so:

tyler@quadjutsu:~$ locate fstab |grep etc |grep -v "~" |grep -v "\."
/etc/fstab

~ has a special meaning in the terminal so you must quote it to match it. ~ refers to the user’s home directory, in this case /home/tyler. I’ve also had to escape . because it too has special meaning, in the context of regular expressions, it will match any character… if we’re excluding any character, we’re excluding our results .

Quick Tricks

Remember commands is sometimes a hard thing to do, especially when it’s a whole string of complicated commands and regexes. Sometimes, I know I don’t have the mind to take notes but that’s okay (for a grace period) because bash keeps track for me. The history command will list all your recently executed commands and utilizing the grep command, we can filter that list.

tyler@quadjutsu:~$ history |grep locate
  613  locate fstab
  614  locate fstab |grep etc
  624  locate fstab |grep etc |grep -v "~" |grep -v "\."

A quick way to search for files in a directory would be to use locate as the path and grep the results

tyler@quadjutsu:~$ locate ~ |grep notes.txt
/home/tyler/Desktop/notes.txt

These are the basics of finding files and searching through them. Linux provides many tools to reformat and replace strings. There are a million different ways to combine commands and always a faster way to do it. Keep playing and you’ll just get better and better.