Do Know Evil - Tyler Mulligan’s Tips and Tricks

More often than not, when I’m doing form to database interactions, I use the same field names. This means there is some redundant information in my code, making my job more difficult. As such, I’ve started to create feather.php, a SQL statement constructor class that works off data sent by the $_POST or $_GET arrays.

It’s not ready for a public release as there are still some security issues to address (see TODO) but I wanted to get some thoughts, opinions and suggestions on this.

For those of you concerned about using the same field names as database columns, I plan to add associative arrays to allow you use different names.

i.e. something like this:

$field_names array('the_first_name','the_last_name','the_zipcode');
$column_names = array('first_name','last_name','zipcode');

Someone had requested a script that would redirect a list of ip’s to a random list of sites.

Here’s the snippet I came up with:

<?
// list of ip's you wish to redirect
$ips = array('127.0.0.1');
// Sites you wish to redirect to
$sites = array('http://yahoo.com/','http://www.aol.com');
 
foreach($ips as $ip) {
    if ($ip==GetHostByName($REMOTE_ADDR)) {
        header('Location: ' . $sites[rand(0,count($sites))]);
    }
}
?>

To do daily database backups, I use cron jobs. If you aren’t familiar with cron jobs, think of the as scripts that run on a timer. Much like ’scheduled tasks’ on windows. To edit your cron jobs on dreamhost, locate the ‘cron jobs‘ menu item located under ‘goodies‘ on the main menu:

Before we add a cron job though, I want to familiarize you with the script and have you run a test to save you a headache later.

#! /bin/sh
 
# Daily backups on your database with email notification
# Tyler Mulligan
 
# file
dateVar=$(date +%m-%d)          # Date variable to append to filename (default: month-day | 11-24)
savePath="/home/tyler/backups/" # Backups are stored here
fileName="my_db_backup"         # File name minus the date
 
# email
subject="My Database Backup ${dateVar}"
email="db_robot@mydomain.com"
 
# database
username="root"      # username
password=""          # password
hostname="localhost" # hostname
database="database"  # database
 
mysqldump -u $username -p$password -h $hostname $database | gzip  > $savePath$fileName-$dateVar.sql.gz
 
uuencode $savePath$fileName-$dateVar.sql.gz $fileName-$dateVar.sql.gz | mail -s "$subject" $email

I’ve set it up so you fill out your information like any other config file, the last two lines do all the work.

I suggest you create a new text file called ‘dbbackup.txt’, and paste the above code in and saving it for future reference. It’s always good to have a clean slate to start from. Once you have that done, fill in your file, email and database variables and give it a test run.

Create a shell script through ssh and run that.

Copy your personalized database dump code, log into your server and type:

vi my_db_test.sh
[press i]
right click (to paste the copied code)
[press esc][type ":wq"]
[press enter]

Your script is now written to a file called ‘my_db_test.sh’.

chmod +x the file and run it:

chmod +x my_db_test.sh
./my_db_test.sh

Once you receive the email, check your backup directory and verify that the database was properly dumped. If you don’t receive and email, you did something wrong.

If everything worked fine, return to the dreamhost cron job page, add a cron job and paste your working code into your newly created cron job.

Save, wait a day and you should receive your email. It’s a wise idea to grab a copy of your backup every week or so to store locally.

Dreamhost is actually offering a special for their 10th anniversary right now, 500gb disk space, 5TB transfer for only $5.95 a month. I’ve been with them 2 years and I’ve been pleased.

After going to GameDev ‘07 at Harvard with James Lindsay on Saturday, I got some motivation and decided to get my feet wet with the facebook api. I have created my first application which pulls pictures from the website www.xepic.com and displays them in your profile like so:

If you’d like to try out my application, you can find it here.

I love the music on di.fm but I HATE the commercials. They make me physically ill. Since I can’t quite afford an account, I’ve decided the next best thing is muting the commercials. Unfortunately, I often forget that I have it muted and I site there with the sound off until something in my brain clicks. Anyway, the point is, why remember, when I can write a program to remember for me?

I downloaded Auto Hotkey and googled for a hot second and came up with this little gem:

^+m::
SoundSet, +1, , mute

#Persistent
SetTimer, Unmute, 10000
return

Unmute:
SoundSet, +1, , mute
SetTimer, Unmute, Off
exit

Now, whenever I press ctrl+shift+m, it mutes for 10 seconds. Whoop whoop!

Someone overheard a friend and I chatting about a website and it’s tag cloud script. I mentioned that it’s pretty easy to create and that I had already created a php function. He asked me to dig it up, so I present to you a cloud script in it’s simplest form.

Cloud

<?php
 
$ceiling = 50;
 
class cloud {
 
	function setTop($top) {
		global $ceiling;
		$ceiling = $top;
	}
 
	function sizeMe($name,$times) {
		global $ceiling;
		$times = ($times / $ceiling) * 100;
		if ($times <=20)
			$size = 1;
		else if ($times>=11 && $times <=40)
			$size = 2;
		else if ($times>=21 && $times <=60)
			$size = 3;
		else if ($times>=31 && $times <=80)
			$size = 4;
		else
			$size = 5;
		// You should really replace this with a return $size, this is just for demonstration purposes
		echo "<span class=\"a$size\"><a href=\"#\">$name</a></span> ";
	}
}
?>

It’s pretty straight forward and easily modified. You setTop based on the number of entries in your database, then pass $name and the number of times it appears to the sizeMe function. If you have any questions or suggestions please let me know. I tried to keep this simple.

In this example I created classes, a1, a2, a3, a4 and a5 that I defined font-size for in a css file.

Strip Characters From A Filename

<?php
 
function renameStripChars($directory, $pattern, $replacement, $verbose = false) {
  if($curdir = opendir($directory)) {
   while($file = readdir($curdir)) {
     if($file != '.' && $file != '..') {
 
       $dstfile = $directory . '/' . preg_replace($pattern, $replacement, $file); 
	   $srcfile = $directory . '/' . $file;	 
 
	   rename($srcfile, $dstfile);
 
       if(is_dir($srcfile) && $verbose) {
	      renameStripChars($srcfile, $pattern, $replacement, $verbose);
       }
     }
   }
   closedir($curdir);
  }
}
 
// An example of how I used it
renameStripChars('picsTest', '/[^\w\d\.-\s]+/', '', true);
renameStripChars('picsTest', '/[\s]+/', '_', true);
renameStripChars('picsTest', '/[-]+/', '-', true);
renameStripChars('picsTest', '/[_]+/', '_', true);
 
?>

I wrote this function because I had a boatload of images with ridiculous characters in them and I wanted them out before I uploaded them somewhere. Most frameworks have a similar function built in but if you’re looking for a quick fix, this is perfect for you.

For those of you familiar with regular expressions, you’re going to love this little feature in Notepad++, Regex find and replace. If you haven’t used it yet, you’ve been wasting your time doing things the hard way.

Lets say I have a text file with a List of websites in the format as such: URL, Description, Name and I want to turn this list into a bunch of links.

I would look for the following string:

(http://[a-z.]+), ([A-z ']+), ([A-z ]+)

By using parenthesis, I’m defining variables I’d like to call back in my replace string. I have created three, the URL(\1), the Descriptiption(\2) and the Name(\3). Then I call them in my replace string and rearrange them as such:

<a href="\1">\3</a>: \2

Now I just click replace, and like magic they are all transformed into pastable links:

Do Know Evil: Tyler Mulligan’s Blog
detrition: Tyler Mulligan’s Portfolio
Google: Tyler’s Search Engine of Choice

But the fun doesn’t stop there, no no no. Say I had multiple files with the same structure, I would just open said files and click, “Replace in all open documents”.

So there you have your crash course in regex find and replace compliments of Notepad++. It’s it bit limited in some respects, it doesn’t fully support regular expressions but it certainly makes a difference. You can read the details about it’s abilities here.

Once upon a time, I found myself in possession of a vast amount of wallpapers. I’m talking well over 10,000, all in one folder. That’s not to helpful since a lot of them were repeats in different dimensions. After realizing that I couldn’t just add a ’sort by dimension’ field in explorer, I set about on a mission using a method I knew would work, PHP with the GD (Graphics Display) library.

The core of this script is based around the ability for GD to pull the dimensions of a picture quickly and efficiently. The goal is to get php to look in a directory and sort all their files by dimensions we define (this will help filter out any pictures in other dimensions present in the folder).

So the sorting cycle looks something like this:

$totalFiles=count($file);
for ($j=0; $j < $totalFiles; $j++)
{
    $dimension=getimagesize("$dir$file[$j]");
    for($k=0; $k < $totalFolders; $k++)
        if($dimension[0] == $width[$k] && $dimension[1] == $height[$k])
        {
            movefile($dir, $file[$j], "$dir$width[$k]x$height[$k]");
            $sorted++;
            if ($sorted%50==0) { echo $sorted." files sorted | ".round(100*($sorted/$totalFiles),2)."%\n";        }
    }
}

However, the folder is currently non-existent but we’re not going to open up explorer and create them ourself, especially since we plan on having an array hold our dimension values.

Here’s what I did:

$width = array(800,1024,1152,1280,1280,1280,1440,1600,1680,1920);
$height = array(600,768,864,800,960,1024,900,1200,1050,1200);
 
if (count($width) != count($height)) // Make sure the arrays are the same length
    exit ("Non-matching array lengths: ".count($width)." widths with ".count($height)." heights.");
 
$totalFolders=count($width);
for($i=0; $i < $totalFolders; $i++)  // Create directories based on the arrays if they haven't been created already
    if (!file_exists("$dir$width[$i]x$height[$i]"))
    {
        mkdir("$dir$width[$i]x$height[$i]");
        echo "Folder $width[$i]x$height[$i] created\n";
    }

The rest of the code is all pretty straight forward. I added in some job statistics (percentages and amount of time to complete). I have to find my pictures of the output but I believe it went through all the wallpapers in ~110 seconds. Anyway, here’s the script in it’s entirety, I hope it helps :).

<?php
$dir = "pics/";                      // The directory (trailing slash)
$dh = opendir($dir);                 // Open the directory
$m_time = explode(" ",microtime());  // These two lines get the start time
$starttime = $m_time[0] + $m_time[1];//
$width = array(800,1024,1152,1280,1280,1280,1440,1600,1680,1920);
$height = array(600,768,864,800,960,1024,900,1200,1050,1200);
 
if (count($width) != count($height)) // Make sure the arrays are the same length
    exit ("Non-matching array lengths: ".count($width)." widths with ".count($height)." heights.");
 
$totalFolders=count($width);
for($i=0; $i < $totalFolders; $i++)  // Create directories based on the arrays if they haven't been created already
    if (!file_exists("$dir$width[$i]x$height[$i]"))
    {
        mkdir("$dir$width[$i]x$height[$i]");
        echo "Folder $width[$i]x$height[$i] created\n";
    }
// Create an array of the file names
while (false !== ($filename = readdir($dh))) {
   if (!is_dir($dir.$filename)) $file[] = $filename;
}
if (!isset($file))
    exit ("No files in this directory.");
 
$sorted=0;
$totalFiles=count($file);
for ($j=0; $j < $totalFiles; $j++)
{
    $dimension=getimagesize("$dir$file[$j]");
    for($k=0; $k < $totalFolders; $k++)
        if($dimension[0] == $width[$k] && $dimension[1] == $height[$k])
        {
            movefile($dir, $file[$j], "$dir$width[$k]x$height[$k]");
            $sorted++;
            if ($sorted%50==0) { echo $sorted." files sorted | ".round(100*($sorted/$totalFiles),2)."%\n";        }
    }
}
 
$m_time = explode(" ",microtime());  // These two lines get the end time
$endtime = $m_time[0] + $m_time[1];  //
echo "\n".$sorted." files have been sorted in ". round(($endtime - $starttime),3) ." seconds.\n";
echo "That's ".(1/(round(($endtime - $starttime),3)/$sorted))." files per second!";
 
function movefile($dir, $theFile, $newDir)
{
    if (!copy($dir.$theFile, "$newDir/$theFile")) echo "failed to copy $theFile...\n";
    else unlink($dir.$theFile);
}
?>

Note: This script should be called from the command line.

Recently, I was presented with the task of counting the number times filenames repeated in an excel document. Being a 16,000+ row file, with well over 500 unique filenames, there was no simple solution in excel.

Having some experience exporting excel documents using PHP, I knew importing them wouldn’t be too hard. A few google searches later and I came across Excel Reader which fit the bill perfectly.

After sorting my excel document by filename, I created a loop in PHP that would read the first filename, count the number of times it repeated until it changed, then append that data to a string that would be exported to a new excel document once it was done reading the data.

// Starting at 2 to skip the headers | +1 so it compares the last row
for ($i = 2; $i <= $data->sheets[0]['numRows']+1; $i++) {
	if ($data->sheets[0]['cells'][$i][3]==$curFileName) {
		$curFileCount++;
	} else {
		// Output the data line:
"Device","User","Filename","Filecount"
		$stringData =
"\"".$data->sheets[0]['cells'][$i-1][1]."\",\"".$data->sheets[0]['cells'][$i
-1][2]."\",\"".$data->sheets[0]['cells'][$i-1][3]."\",\"".$curFileCount."\"\
n";
		fwrite($fh, $stringData);
 
		// Reset the count and update the curFileName
		$curFileName = $data->sheets[0]['cells'][$i][3];
		$curFileCount=1;
	}
	echo "Row: ".$i."\n";
}

Of course… I didn’t have just one of these files but 6 and manually entering each filename would be a waste of my time. My software solution for this was to create an “Input” and “Output” folder and have PHP cycle through the “Input” folder.

// For every file in the dir "Input", generate a report
if ($handle = opendir('Input')) {
	while (false !== ($orgFileName = readdir($handle))) {
		if($orgFileName != '..' && $orgFileName != '.') {
 
			// Echo the file we're reading, then read that file
			echo $orgFileName."\n";
			$data->read("Input/".$orgFileName);

Eventually it all came together as such:

<?php
 
// Don't change any of this
require_once 'Excel/reader.php';
$data = new Spreadsheet_Excel_Reader();
$data->setOutputEncoding('CP1251');
 
// For every file in the dir "Input", generate a report
if ($handle = opendir('Input')) {
	while (false !== ($orgFileName = readdir($handle))) {
		if($orgFileName != '..' && $orgFileName != '.') {
 
			// Echo the file we're reading
			echo $orgFileName."\n";
			$data->read("Input/".$orgFileName);
 
			// Set the first filename to compare
			$curFileName =
$data->sheets[0]['cells'][2][3]==$curFileName;
			$curFileCount = 0;
 
			// Create a file and open it for writing
			$myFile =
"Output/File_Count-".str_replace(".xls",".csv",$orgFileName);
			$fh = fopen($myFile, 'w') or die("can't open file");
 
			// Starting at 2 to skip the headers | +1 so it
compares the last row
			for ($i = 2; $i <= $data->sheets[0]['numRows']+1;
$i++) {
				if
($data->sheets[0]['cells'][$i][3]==$curFileName) {
					$curFileCount++;
				} else {
					// Output the data line:
"Device","User","Filename","Filecount"
					$stringData =
"\"".$data->sheets[0]['cells'][$i-1][1]."\",\"".$data->sheets[0]['cells'][$i
-1][2]."\",\"".$data->sheets[0]['cells'][$i-1][3]."\",\"".$curFileCount."\"\
n";
					fwrite($fh, $stringData);
 
					// Reset the count and update the
curFileName
					$curFileName =
$data->sheets[0]['cells'][$i][3];
					$curFileCount=1;
				}
				echo "Row: ".$i."\n";
			}
			// Close file
			fclose($fh);
		}
	}
}
?>

My apologies for the less than perfect code, I was on a deadline and perfect code wasn’t a requirement :).

Here are some screenshots of the script in action.

I know… this seems silly because I released 1.0a today… but I honestly didn’t think I’d get this done today.

1.0a links are down for a short while. I want to clean up the script a bit.

litebox 1.0 get it while it’s hot!