I’m still relatively new to Python. I dabble once in a while. Usually when I catch myself writing something overly complicated in bash. I figured doing MySQL from a Python script would be easier and better implemented in Python. Well, I was in for a few surprises I’d like to share with others looking to integrate with MySQL in Python before you encounter the same problems I did.

Getting on with it

As a Python noob, it made sense to me to use the python-mysqldb module. It “works” but not well. Read through the [working] code below and see if you can find the problem and how I circumvented it.

#!/bin/python
# Tyler Mulligan (tyler@doknowevil.net)
# MySQL the WRONG way in Python
import MySQLdb

def connection_settings():
    global DB_ADMIN, DB_ADMIN_PASSWORD, DB_NAME, DB_HOST, DB_USER, DB_PASSWORD
    DB_ADMIN="root"
    DB_ADMIN_PASSWORD="local"
    DB_NAME="foobardb"
    DB_HOST="localhost"
    DB_USER="foobar"
    DB_PASSWORD="local"

def run_admin_sql():
    conn = MySQLdb.Connection(host=DB_HOST, user=DB_ADMIN, passwd=DB_ADMIN_PASSWORD)
    cursor = conn.cursor()

    #cursor.execute("DROP USER %s@%s", (DB_USER, DB_HOST))
    #cursor.execute("DROP DATABASE %s" % DB_NAME)

    cursor.execute("CREATE USER %s@%s IDENTIFIED BY %s", (DB_USER, DB_HOST, DB_PASSWORD))
    cursor.execute("CREATE DATABASE %s" % DB_NAME)
    cursor.execute("GRANT USAGE ON %s.* TO %s@%s IDENTIFIED BY '%s'" % (DB_NAME, DB_USER, DB_HOST, DB_PASSWORD))
    cursor.execute("GRANT ALL PRIVILEGES ON *.* to %s@%s IDENTIFIED BY %s", (DB_USER, DB_HOST, DB_PASSWORD))

    cursor.close()
    conn.close()

    conn = MySQLdb.Connection(db=DB_NAME, host=DB_HOST, user=DB_USER, passwd=DB_PASSWORD)
    cursor = conn.cursor()

    cat_table_sql = """CREATE TABLE cats
    (
    `id` int(3) unsigned NOT NULL AUTO_INCREMENT,
    `name` varchar(50) NOT NULL,
    `age` int(2) NOT NULL,
    `type` int(2) NOT NULL,
    PRIMARY KEY (`id`)
    )"""

    cursor.execute(cat_table_sql)
    cursor.execute("INSERT INTO cats (name, age, type) VALUES (%s, %s, %s)", ("fluffy", 5, 1))
    cursor.execute("INSERT INTO cats (name, age, type) VALUES (%s, %s, %s)", ("meow meow", 6, 1))
    cursor.execute("INSERT INTO cats (name, age, type) VALUES (%s, %s, %s)", ("purrfect", 3, 1))

    cursor.execute("SELECT * FROM cats")

    results = cursor.fetchall()
    cursor.close()
    conn.close()
    return results

connection_settings()
print run_admin_sql()

The problem is parameterization. The python-mysqldb module sucks at it. It’s glaringly obvious when you look at the follow snippet and how I worked around it.

cursor.execute("CREATE USER %s@%s IDENTIFIED BY %s", (DB_USER, DB_HOST, DB_PASSWORD)) # GOOD
cursor.execute("CREATE DATABASE %s" % DB_NAME) # BAD NEWS BEARS
cursor.execute("CREATE DATABASE %s", (DB_NAME)) # but the module won't support this

The difference, is that the first implementation uses the module’s parameterization to escape the variables that are passed to it, making it safe. The second is using Python’s string formatter which means I’d have to do all the escaping and sanitization prior, which is obviously more dangerous and annoying.

The MySQLdb docs bury this little note, “Parameter placeholders can only be used to insert column values. They can not be used for other parts of SQL, such as table names, statements, etc.”

You can implement it this way but you’ll be wasting your time. There are better modules out there to make up for this one’s shortcomings. OurSQL and SQLAlchemy were the two libraries recommended to me in #python on irc.freenode.org. I will make another post when I develop the same example above using one of these libraries.

It’s not ready for a public release as there are still some security issues to address (see TODO) but I wanted to get some thoughts, opinions and suggestions on this.

For those of you concerned about using the same field names as database columns, I plan to add associative arrays to allow you use different names.

i.e. something like this:

$field_names array('the_first_name','the_last_name','the_zipcode');
$column_names = array('first_name','last_name','zipcode');

Before we add a cron job though, I want to familiarize you with the script and have you run a test to save you a headache later.

#! /bin/sh

# Daily backups on your database with email notification
# Tyler Mulligan

# file
dateVar=$(date +%m-%d)          # Date variable to append to filename (default: month-day | 11-24)
savePath="/home/tyler/backups/" # Backups are stored here
fileName="my_db_backup"         # File name minus the date

# email
subject="My Database Backup ${dateVar}"
email="db_robot@mydomain.com"

# database
username="root"      # username
password=""          # password
hostname="localhost" # hostname
database="database"  # database

mysqldump -u $username -p$password -h $hostname $database | gzip  > $savePath$fileName-$dateVar.sql.gz

uuencode $savePath$fileName-$dateVar.sql.gz $fileName-$dateVar.sql.gz | mail -s "$subject" $email

I’ve set it up so you fill out your information like any other config file, the last two lines do all the work.

I suggest you create a new text file called ‘dbbackup.txt’, and paste the above code in and saving it for future reference. It’s always good to have a clean slate to start from. Once you have that done, fill in your file, email and database variables and give it a test run.

Create a shell script through ssh and run that.

Copy your personalized database dump code, log into your server and type:

vi my_db_test.sh
[press i]
right click (to paste the copied code)
[press esc][type ":wq"]
[press enter]

Your script is now written to a file called ‘my_db_test.sh’.

chmod +x the file and run it:

chmod +x my_db_test.sh
./my_db_test.sh

Once you receive the email, check your backup directory and verify that the database was properly dumped. If you don’t receive and email, you did something wrong.

If everything worked fine, return to the dreamhost cron job page, add a cron job and paste your working code into your newly created cron job.

Save, wait a day and you should receive your email. It’s a wise idea to grab a copy of your backup every week or so to store locally.

Dreamhost is actually offering a special for their 10th anniversary right now, 500gb disk space, 5TB transfer for only $5.95 a month. I’ve been with them 2 years and I’ve been pleased.

After a little more research I came about a link to a slightly thinner client in the developer zone under the Generally Available (GA) license. I’ve played around with it a bit and while it wasn’t exactly what I was looking for, it’s helped me during those times I don’t feel like waiting on phpMyAdmin.

Check out this screencast to get a better idea of how this tool works. I’m sure I’ll grow to like this tool a lot more in the future, I just need to learn more about it’s functionality. I thought I’d pass this tip onto you.