Creating high quality thumbnails for PDFs in Linux using Imagemagick

If you’ve been googling for a solution to this, you might have found the quality of your pdf thumbnails images created using imagemagick is less than what you’ve desired. There is a simple but not exactly straightforward reason for this. Imagemagick uses ghostscript to generate these thumbnails from pdfs, unless you specify the density before the filename, you may end up with poor quality thumbnails. I’ve used my resume to demonstrate.

# Create [a low quality] thumbnail image for the first page from a pdf
f="tyler.mulligan-resume-winter-2013.pdf"; convert $f[0] -quality 100 -resize 960x +adjoin $f-%02d.png
# Create [a high quality] thumbnail image for the first page from a pdf
f="tyler.mulligan-resume-winter-2013.pdf"; convert -density 300 $f[0] -quality 100 -resize 960x +adjoin $f-%02da.png

The difference in these thumbnails is minor but enough to drive a graphic designer insane. The font kerning is off and the shapes rasterize imperfectly.

Low Quality

tyler.mulligan-resume-winter-2013.pdf-00

High Qualitytyler.mulligan-resume-winter-2013.pdf-00a

I used the bracket page notation to define the first page of the pdf in the last two commands, filename.pdf[0] (zero index). It’s possible to do the entire PDF at once by removing it.

# Create high quality thumbnail images for a pdf
f="my.pdf"; convert -density 300 $f -quality 100 -resize 960x +adjoin $f-%02d.png

We can toss this in a for loop to perform the operation on a bunch:

# Create high quality thumbnail images from multiple pdfs
for f in *.pdf; do convert -density 300 $f -quality 100 -resize 960x +adjoin $f-%02d.png; done

If you’re still having issues with quality, you can play with the the -scale parameter, adding it after the -quality setting, i.e. “-scale @1500000″.

jQuery and jQuery UI in your Grease Monkey scripts.

Not all websites are made equal. Unfortunately, some of those websites are ones we’re required to use. Be it our job, our interests or the undiscovered, web-browsing can be the orgeon trail of the internet. Cultivating the atmosphere to work for us and bringing the right tools along for the journey are essential for survival, if you’re a pioneer.

I’m tired of websites that don’t provide, can’t provide or will never provide. I’ve called in my boys, Stylish and Grease Monkey on the front lines. However, beyond the crazy happenstance that another user might share a solution to your issue on userstyles.org or userscripts.org you could be left in the cold, without the right tools. I won’t go into detail on Stylish in this post but you might not even need it after you see the script.

Since I’m already connected with Google every which way in my life, why not pretend like the Internet is a big party and websites are your friends. Introduce jQuery from Google’s CDN to other friends (websites) that don’t already know jQuery and jQuery UI and make it personal.

That was the driving force behind this Grease Monkey Script, which was derived from mashing together strd6′s script and Joan Piedra’s script with an update to the latest versions of jQuery and jQuery UI.

// ==UserScript==
// @name           jQuery and jQuery UI
// @author         Tyler "-z-" Mulligan
// @version        0.5
// @namespace      http://www.doknowevil.net
// @description    jQuery 1.6.2 and jQuery UI 1.8.4 loaded from google's CDN with redmond theme.
// @include        http://www.website.com/*
//
// @resource       jQuery               https://ajax.googleapis.com/ajax/libs/jquery/1.6.2/jquery.min.js
// @resource       jQueryUI             https://ajax.googleapis.com/ajax/libs/jqueryui/1.8.14/jquery-ui.min.js
//
// @resource       jQueryUICSS          http://ajax.googleapis.com/ajax/libs/jqueryui/1.8.4/themes/redmond/jquery-ui.css
//
// @resource       ui-bg_inset-hard_100_fcfdfd_1x100.png  http://ajax.googleapis.com/ajax/libs/jqueryui/1.8.4/themes/redmond/images/ui-bg_inset-hard_100_fcfdfd_1x100.png
// @resource       ui-bg_gloss-wave_55_5c9ccc_500x100.png http://ajax.googleapis.com/ajax/libs/jqueryui/1.8.4/themes/redmond/images/ui-bg_gloss-wave_55_5c9ccc_500x100.png
// @resource       ui-bg_glass_85_dfeffc_1x400.png        http://ajax.googleapis.com/ajax/libs/jqueryui/1.8.4/themes/redmond/images/ui-bg_glass_85_dfeffc_1x400.png
// @resource       ui-bg_glass_75_d0e5f5_1x400.png        http://ajax.googleapis.com/ajax/libs/jqueryui/1.8.4/themes/redmond/images/ui-bg_glass_75_d0e5f5_1x400.png
// @resource       ui-bg_inset-hard_100_f5f8f9_1x100.png  http://ajax.googleapis.com/ajax/libs/jqueryui/1.8.4/themes/redmond/images/ui-bg_inset-hard_100_f5f8f9_1x100.png
// @resource       ui-bg_flat_55_fbec88_40x100.png        http://ajax.googleapis.com/ajax/libs/jqueryui/1.8.4/themes/redmond/images/ui-bg_flat_55_fbec88_40x100.png
// @resource       ui-bg_glass_95_fef1ec_1x400.png        http://ajax.googleapis.com/ajax/libs/jqueryui/1.8.4/themes/redmond/images/ui-bg_glass_95_fef1ec_1x400.png
// @resource       ui-icons_469bdd_256x240.png            http://ajax.googleapis.com/ajax/libs/jqueryui/1.8.4/themes/redmond/images/ui-icons_469bdd_256x240.png
// @resource       ui-icons_469bdd_256x240.png            http://ajax.googleapis.com/ajax/libs/jqueryui/1.8.4/themes/redmond/images/ui-icons_469bdd_256x240.png
// @resource       ui-icons_d8e7f3_256x240.png            http://ajax.googleapis.com/ajax/libs/jqueryui/1.8.4/themes/redmond/images/ui-icons_d8e7f3_256x240.png
// @resource       ui-icons_6da8d5_256x240.png            http://ajax.googleapis.com/ajax/libs/jqueryui/1.8.4/themes/redmond/images/ui-icons_6da8d5_256x240.png
// @resource       ui-icons_217bc0_256x240.png            http://ajax.googleapis.com/ajax/libs/jqueryui/1.8.4/themes/redmond/images/ui-icons_217bc0_256x240.png
// @resource       ui-icons_f9bd01_256x240.png            http://ajax.googleapis.com/ajax/libs/jqueryui/1.8.4/themes/redmond/images/ui-icons_f9bd01_256x240.png
// @resource       ui-icons_2e83ff_256x240.png            http://ajax.googleapis.com/ajax/libs/jqueryui/1.8.4/themes/redmond/images/ui-icons_2e83ff_256x240.png
// @resource       ui-icons_cd0a0a_256x240.png            http://ajax.googleapis.com/ajax/libs/jqueryui/1.8.4/themes/redmond/images/ui-icons_cd0a0a_256x240.png
// @resource       ui-bg_flat_0_aaaaaa_40x100.png         http://ajax.googleapis.com/ajax/libs/jqueryui/1.8.4/themes/redmond/images/ui-bg_flat_0_aaaaaa_40x100.png
// @resource       ui-bg_flat_0_aaaaaa_40x100.png         http://ajax.googleapis.com/ajax/libs/jqueryui/1.8.4/themes/redmond/images/ui-bg_flat_0_aaaaaa_40x100.png

// ==/UserScript==

var $;

// Inject jQuery into page... gross hack... for now...
(function() {
    if (typeof unsafeWindow.jQuery == 'undefined') {
        var head = document.getElementsByTagName('head')[0];
 
        var script = document.createElement('script');
        script.type = 'text/javascript';
 
        var jQuery = GM_getResourceText('jQuery');
        var jQueryUI = GM_getResourceText('jQueryUI');
 
        script.innerHTML = jQuery + jQueryUI;
        head.appendChild(script);

    }
    GM_wait();
})();
   
// Check if jQuery's loaded
function GM_wait() {
    if (typeof unsafeWindow.jQuery == 'undefined') {
        window.setTimeout(GM_wait, 100);
    } else {
        $ = unsafeWindow.jQuery.noConflict(true);
        addUIStyles();
        letsJQuery();
    }
}

function addUIStyles() {
    var head = document.getElementsByTagName('head')[0];
    var resources = {
        'ui-bg_inset-hard_100_fcfdfd_1x100.png': GM_getResourceURL('ui-bg_inset-hard_100_fcfdfd_1x100.png'),
        'ui-bg_gloss-wave_55_5c9ccc_500x100.png': GM_getResourceURL('ui-bg_gloss-wave_55_5c9ccc_500x100.png'),
        'ui-bg_glass_85_dfeffc_1x400.png': GM_getResourceURL('ui-bg_glass_85_dfeffc_1x400.png'),
        'ui-bg_glass_75_d0e5f5_1x400.png': GM_getResourceURL('ui-bg_glass_75_d0e5f5_1x400.png'),
        'ui-bg_inset-hard_100_f5f8f9_1x100.png': GM_getResourceURL('ui-bg_inset-hard_100_f5f8f9_1x100.png'),
        'ui-bg_flat_55_fbec88_40x100.png': GM_getResourceURL('ui-bg_flat_55_fbec88_40x100.png'),
        'ui-bg_glass_95_fef1ec_1x400.png': GM_getResourceURL('ui-bg_glass_95_fef1ec_1x400.png'),
        'ui-icons_469bdd_256x240.png': GM_getResourceURL('ui-icons_469bdd_256x240.png'),
        'ui-icons_469bdd_256x240.png': GM_getResourceURL('ui-icons_469bdd_256x240.png'),
        'ui-icons_d8e7f3_256x240.png': GM_getResourceURL('ui-icons_d8e7f3_256x240.png'),
        'ui-icons_6da8d5_256x240.png': GM_getResourceURL('ui-icons_6da8d5_256x240.png'),
        'ui-icons_217bc0_256x240.png': GM_getResourceURL('ui-icons_217bc0_256x240.png'),
        'ui-icons_f9bd01_256x240.png': GM_getResourceURL('ui-icons_f9bd01_256x240.png'),
        'ui-icons_2e83ff_256x240.png': GM_getResourceURL('ui-icons_2e83ff_256x240.png'),
        'ui-icons_cd0a0a_256x240.png': GM_getResourceURL('ui-icons_cd0a0a_256x240.png'),
        'ui-bg_flat_0_aaaaaa_40x100.png': GM_getResourceURL('ui-bg_flat_0_aaaaaa_40x100.png'),
        'ui-bg_flat_0_aaaaaa_40x100.png': GM_getResourceURL('ui-bg_flat_0_aaaaaa_40x100.png')
    };
    
    var style = document.createElement('style');
    style.type = 'text/css';
    
    var css = GM_getResourceText ('jQueryUICSS');
    $.each(resources, function(resourceName, resourceUrl) {
        console.log(resourceName + ': ' + resourceUrl);
        css = css.replace( 'images/' + resourceName, resourceUrl);
    });
    
    style.innerHTML = css;
    head.appendChild(style);
}

// All your GM code must be inside this function
function letsJQuery() {
    //alert($); // check if the dollar (jquery) function works
    //alert($().jquery); // check jQuery version
    $("
I'm in a dialog!
").dialog({ buttons: { "Ok": function() { alert("Ok"); }, "Cancel": function() { $(this).dialog("close"); } } }); }

I also wanted to use this script with Chromium browser, the open-source version of Google Chrome. I found the TamperMonkey Extension for Chrome. Stylish is also available for Chrome.

Generating CSS Based on Images, also SEO-Friendly

I’ve typed out a lot of CSS that’s focused on using a background image to replace text. I’ve been using a cross-browser CSS trick that’s SEO-friendly, I’m not sure if it’s documented anywhere. I came upon the solution myself when I was searching for a clever way to do this years ago and it’s become the standard method for me. However, I dislike the tedious task of typing out the CSS and referencing the image pixels and at work, I don’t want my developers wasting that time either.

I figured it’d be best to come up with a simple solution, so I started coding something out in bash and it worked:

#!/bin/bash
# css.sh - generate common css and html
# Tyler Mulligan (z@interwebninja.com)
# Last Update: 02/16/2011
# MIT License

create_block_image() {
    feh -lr ${1:-.} | awk '{ print $3" "$4" "$8 }' | sed '1d' | while read l; do N=$((N+1));
        cbi_css $l
    done
    feh -lr ${1:-.} | awk '{ print $3" "$4" "$8 }' | sed '1d' | while read l; do N=$((N+1));
        cbi_html_a $l
    done
    feh -lr ${1:-.} | awk '{ print $3" "$4" "$8 }' | sed '1d' | while read l; do N=$((N+1));
        cbi_html_div $l
    done
}
cbi_css() {
    f=${3##*/}
    echo "#${f%.*} {
    display: block;
    background: url('$3') no-repeat 0 0;
    width: 0;
    height: $2px;
    padding-left: $1px;
    overflow: hidden;
}"
}
cbi_html_a() {
    f=${3##*/}
    echo "${f%.*}"
}
cbi_html_div() {
    f=${3##*/}
    echo "
" } s=$1; shift; case $s in --image-block|--cbi|-i) create_block_image $@;; *) help help;; esac

Which ouputs something like:

z@zygon:~/scripts/css$ ./css.sh -i img/
#dumbtubes-fav {
    display: block;
    background: url('img/dumbtubes-fav.png') no-repeat 0 0;
    width: 0;
    height: 14px;
    padding-left: 16px;
    overflow: hidden;
}
#submit_new {
    display: block;
    background: url('img/submit_new.png') no-repeat 0 0;
    width: 0;
    height: 106px;
    padding-left: 244px;
    overflow: hidden;
}
#dumbtubes-logo {
    display: block;
    background: url('img/dumbtubes-logo.png') no-repeat 0 0;
    width: 0;
    height: 70px;
    padding-left: 274px;
    overflow: hidden;
}
dumbtubes-fav
submit_new

To explain the way this CSS works, it uses the padding-left as the actual width, setting the width 0 and then pushes whatever content you have out of view using overflow:hidden to hit it from view. This makes it easy for search engines, keeping your HTML clean and CSS simple.

The bash script relies of “feh” a lightweight image viewer for linux that outputs a list of images and their dimensions. It also generates some sample HTML to quickly drop in and modify.

This is not an elegant solution and I’m not happy with the fact that it relies on feh. I’ve been slowly getting into python and I was amazed at how fast I was able to recreate this script in Python.

#!/usr/bin/python
# css.py - generate common css and html
# Tyler Mulligan (z@interwebninja.com)
# Last Update: 02/17/2011
# MIT License
from PIL import Image
import sys
import os.path

CSS_FORMAT = """#%s {\n\
    display: block;\n\
    background: url('%s') no-repeat 0 0;\n\
    height: %spx;\n\
    width: 0;\n\
    padding: %spx;\n\
    overflow: hidden;\n\
}\n"""

HTML_A_FORMAT = """%s\n"""
HTML_DIV_FORMAT = """
%s
\n""" css="" html_a="" html_div="" for tf in os.listdir(sys.argv[1]): f = os.path.join(sys.argv[1],tf) if os.path.isfile(f) == True : img = Image.open(f) fid = os.path.splitext(tf)[0] (width, height) = img.size[0:2] css += CSS_FORMAT % (fid,f,height,width) html_a += HTML_A_FORMAT % (fid,f,fid,fid) html_div += HTML_DIV_FORMAT % (fid,fid) print css print html_a print html_div

This is just the basic idea, I didn’t spend more than 15 minutes on this rewrite, it does what I need it to do so far. It was suggested by friends if I make it any bigger, to look into a templating engine, such as mako. I hope this script can be useful to someone :). PS, HTML_DIV_FORMAT should be on one line, not sure why my syntax highlighter is trying to put it on 3.

What Makes ASUS and Android an A+?

ASUS has vision, and the eeepc is no doubt a standard setting line of netbooks. Their latest teaming of Android with their latest tablets, eee Pad transformer and the eee Pad Slider provide both ASUS and users with a lightweight familiar system.  It may also be using the same operating system as their phones, possibly other devices.

With ChromeOS, Google is also covering a server-side focused solution to a consumer’s needs.  We see Blackberry trying out a similar solution with WebOS, ASUS however, is focusing on hardware, as usual and doing an impressive job, as usual.

They don’t completely shy away from Windows either, with the Eee Slate running the most impressive hardware, it should be interesting to see how Microsoft holds onto their market being a direct competitor of the iPad.

ASUS doesn’t forget it’s stake in the software portion of this emerging market, no. They’ve invested in a “cloud” for consumers information to be separate from their phones and separate from their carrier. Their information is stored and accessible via their device(s) as an app or access.asus.com. Now all they need to do is invest in a short domain name, ax.as.us.

Other Points, these are cool:

gnu-and-penguin

Non-free Art In Open-source (FOSS) Video Games

Introduction

I’ve been interested in art since I can remember, the marriage of computers and art has always fascinated me. I would spend math classes (sorry Mr. D) fiddling around with my calculator, drawing pixel art and saving them in the storage on my calculator. I would hack up BASIC programs or marvel at ASM based games and sub-shells. Later these activities moved over to computers, websites, software, the internet and open-source video games, such as Nexuiz or Xonotic. My brain has always enjoyed trying to pull the worlds of art and science together but it wasn’t always viewing the two alike.

Originally my feelings about art were 2 dimensional, as I’d expect many to view it. It’s easy for art to be a fleeting glimpse. You see the final product, the lines of the creation process are blurred, the definitions of “source” gets lost.

When I first started contributing 2d art to Nexuiz, I was submitting PSD files. I didn’t realize at the time but I was deterring other artists who were without Photoshop the ability to properly edit the file because of some advanced settings in the file that could not be read by open-source software. This hindered others who were trying to expand on my work or create derivative works.

Now that isn’t a completely fair comparison because source is still being provided, only limited to a number of people. However, the reason I bring it up is because it was the beginning of a turning point for me. As I delved more into Linux and the world of Free Open-Source Software (FOSS), I was realizing how important source is to a community. How source code teaches, how knowledge of techniques are passed on. My involvement with Nexuiz and other (at the time) open-source projects, are what kept this message strong in the back of my head.


GPL Superheros

On Art in Video Games

Source of some art is almost impossible. A hand draw sketch for example, in the digital world however, things are different and art source becomes a very powerful teaching tool. Often, for the creation of maps for these games is not the most straight forward process. The mapping software and map entities require some critical thinking. The most common way to learn to add a feature to your map if you don’t know how, is to identify a map with the feature you’d like to implement, open its .map source file and study how it was done.

The reason this all came up for me today was because of a popular open-source ioquake-based game called Warsow. The code is licensed under GPL and the art has a proprietary, closed-source license. As a core member of the Xonotic team, we faced the decision of how to license Xonotic when we forked from Nexuiz. This decision was not as easy as you’d think because of ‘techniical’ license issues. The code must be GPL because of its lineage, GPL is not so keen on art. Creative Commons is an art license, not so keen on code. They seem like they’d be a good merger but from what I understood they could cause issues for distribution. We’d have to distribute the code and art as separate packages. In the end we licensed everything under GPL to make things easier for ourselves.

I thought Warsow was facing the same problem and today they had a developer Q & A, I decided to ask about this, pardon my bias phrasing.

[-z-] asked: do you think distributing warsow as one package is legal under the licensing terms?
crizis answers: Yes, it is. Even Richard Stallman himself blessed way of having open engine and restricted artwork. All code in Warsow is open source, artwork is not.

I found this interesting and with some conversation with with fellow Xonoticans, I came to learn how Richard Stallman, the founder of free software, feels about art in FOSS and I couldn’t disagree more. How is requiring the source for compiled programs any different than requiring the source for art?

Expanding On My Feelings

In my opinion, non-free art in free software is not in good spirit, it does not help others learn and it can hurt the growth of the game. The distinction between art and code gets further blurred when you see how interactive the artwork is.

I can argue that maps are code. They are a meta file, you can open them up in your text editor and edit them, they are filled with coordinates of brushes and entities, that hold keys with settings for the objects references to textures and can even contain some mild programming. Shader files are used to enhance textures and brushes. It’s not conventional programming, it’s closer to “virtual circuitry” as MrBougo called it.

This information is lost if source is not provided. Mappers have two choices on how to de-construct such features. Reverse engineer or decompile the map. The former likely being a waste of someone’s time and the later being an example of two wrongs not making it right.

I do not mean this as an attack on Warsow, it was only what re-lit my interest in the topic and the views I found by RMS were shocking. I think this should be a topic for discussion because in my eyes, art should fall under a similar license as code in FOSS, especially in software such as a video game where media is more advanced and interactive.

Tips for Using Bash in the Linux Terminal – Part 1

Introduction

Bash is the default shell in the terminal on many Linux and UNIX based operating system, such as Ubuntu or Mac OS X. I’ve mentioned commandlinefu.com before as a great reference for learning some neat tricks with the terminal. I’ve gained a lot from the site and a few others, such as the Advanced Bash Scripting Guide and the bash hackers wiki. I wanted to share some of the tips I use most often, combined with other information that I’ve compiled through my use of the terminal.

I think I should mention that I’ve been using more applications in the terminal recently. Some people view this as backwards but I argue just the opposite. Limiting the amount of times I need to use the mouse and the number of keystrokes I need to make drastically increases my efficiency. Many bash applications are designed around single keystrokes, layered hotkeys and edit “modes”. GUIs have their place but many of my tasks can be completely more accurately and consistently via the terminal.

With that said, lets dive in.

The Basics

Hotkeys

Moving:
Ctrl + a -> Go to the beginning of the line you are currently typing on.
Ctrl + e -> Go to the end of the line you are currently typing on
Alt + f -> Move cursor forward one word on the current line.
Alt + b -> Move cursor backward one word on the current line.

Editing:
Ctrl + u -> Clears from before the cursor position. If you are at the end of the line, clears the entire line.
Ctrl + k -> Clear from after the cursor. If you are at the beginning of the line, clears the entire line.
Ctrl + w -> Delete the word before the cursor.
Ctrl + h -> Same as backspace.
Ctrl + t -> Swap the last two characters before the cursor.
Esc + t -> Swap the last two words before the cursor.

Other:
Ctrl + l -> Clear screen (same as clear command).
Ctrl + c -> Kill the current command or process.
Ctrl + z -> Puts whatever you are running into a suspended background process, fg to restore it.
Ctrl + d -> Exit the current shell.

Oddly, unlike many terminal applications, Bash hotkeys don’t make a lot of sense to me. There are very few that are “intuitive”.

History

Press the up arrow for the last command or:

!! — repeat last command

echo "hello"
!!

Outputs:

z@zentury:~$ echo "hello"
hello
z@zentury:~$ !!
echo "hello"
hello

ctrl+r is one of the best ways to search through your history. it will initialize a reverse search as you type. To go to the next result, press ctrl+r again

Advanced

History Expansion/Modification

!:0 — will repeat the first token

cd ~
ls -la
!:0

!:1-3 — defining a range: 1-3 will repeat the 2nd to 4th tokens (count starts at 0). It’s important to note that double quotes will group tokens together.

echo "hello there" && ls ~
!:3-4

!!:s/find/replace/ — will allow you to replace a part of the command

echo "hello there"
!!:s/hello/hi/

Outputs:

z@zentury:~$ echo "hello there"
hello there
z@zentury:~$ !!:s/hello/hi/
echo "hi there"
hi there

OR even shorter:

^find^replace

echo "hello there"
^hello^hi

Outputs:

z@zentury:~$ echo "hello there"
hello there
z@zentury:~$ ^hello^hi
echo "hi there"
hi there

Sequences and Pattern Expansion

Typically in a Linux or UNIX environment you have access to a command line tool name “seq” which gone over before. However, it’s good to know that bash has built-in sequence expansion and you don’t need to rely on seq.

echo {a..z}
a b c d e f g h i j k l m n o p q r s t u v w x y z

by defining a start and end character with the ‘..’ in between, we tell bash to fill in the rest and echo a list for us. Those are all lowercase, what if you wanted uppercase? simple:

echo {A..Z}
A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

Or both, with a few extra characters in the mix:

echo {A..z}
A B C D E F G H I J K L M N O P Q R S T U V W X Y Z [  ] ^ _ ` a b c d e f g h i j k l m n o p q r s t u v w x y z

It doesn’t always have to be a-z though,

echo {A..G}
A B C D E F G

This also works with numbers:

echo {0..9}
0 1 2 3 4 5 6 7 8 9
echo {0..100}
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100

Descending as well as ascending

echo {9..0}
9 8 7 6 5 4 3 2 1 0

Echo a specific set

echo {1,4,6,9}
1 4 6 9

Applying it

Quickly backup a file

touch file1.txt
cp file1.txt{,.bak}
ls
file1.txt file1.txt.bak

explanation: the first parameter is empty, the second is .bak, this expands to >> cp file1.txt file1.txt.bak << and creates the copy

Convert an image type

If you have image magick installed, you can convert file types pretty easy using this same concept:

sudo apt-get install imagemagick

(To install on Ubuntu)

convert file.{jpg,png}

Permutations

echo {a..c}{a..c}{a..c}
aaa aab aac aba abb abc aca acb acc baa bab bac bba bbb bbc bca bcb bcc caa cab cac cba cbb cbc cca ccb ccc

Stay tuned for Part 2

These are some pretty common techniques I use to reduce the amount of typing and thinking required to complete a task in the terminal. Stay tuned for part 2 and check out some of my creative usages at commandlinefu.com.
-

Customizing Colors for Your .bashrc

I’m working on organizing my dotfiles and part of that process has led me to clean up the way I address colors. I came across a few good sources, for .bashrc colors and decided it’d be better to create my own. I would like to have used the ones I found on the Arch wiki but the aliases were a bit odd to me. They also break consistency as you scroll down. I’m not sure if this is due to it being a collaborative wiki or it’s a way to teach multiple syntaxes/approaches.  In either case I figured the aliases weren’t as “standard” as I thought they might be in that world.

So I decided to copy them the best of the bunch into geany and modify them with regular expressions. I used column selection to speed up some of the tasks, such as Title Case formatting and typing the prefixes.

# define colors
Black='\e[0;30m'    # Black / Regular
Red='\e[0;31m'      # Red
Green='\e[0;32m'    # Green
Yellow='\e[0;33m'   # Yellow
Blue='\e[0;34m'     # Blue
Purple='\e[0;35m'   # Purple
Cyan='\e[0;36m'     # Cyan
White='\e[0;37m'    # White

BBlack='\e[1;30m'   # BBlack / Bold
BRed='\e[1;31m'     # BRed
BGreen='\e[1;32m'   # BGreen
BYellow='\e[1;33m'  # BYellow
BBlue='\e[1;34m'    # BBlue
BPurple='\e[1;35m'  # BPurple
BCyan='\e[1;36m'    # BCyan
BWhite='\e[1;37m'   # BWhite

UBlack='\e[4;30m'   # UBlack / Underline
URed='\e[4;31m'     # URed
UGreen='\e[4;32m'   # UGreen
UYellow='\e[4;33m'  # UYellow
UBlue='\e[4;34m'    # UBlue
UPurple='\e[4;35m'  # UPurple
UCyan='\e[4;36m'    # UCyan
UWhite='\e[4;37m'   # UWhite

BGBlack='\e[40m'    # BGBlack - background
BGRed='\e[41m'      # BGRed
BGGeeen='\e[42m'    # BGGreen
BGYellow='\e[43m'   # BGYellow
BGBlue='\e[44m'     # BGBlue
BGPurple='\e[45m'   # BGPurple
BGCyan='\e[46m'     # BGCyan
BGWhite='\e[47m'    # BGWhite

NC='\e[0m'          # Text Reset / No Color

Here is a test case:

for c in {,B,U,BG}{Black,Red,Green,Yellow,Blue,Purple,Cyan,White}; do echo -e ${!c}$c${NC}; done; echo -e "${NC}"

Here is the same test case without the aliases

for c in {0,1,4}\;{30..37} {40..47}; do echo -e \\e[${!c}${c}m${c}m\\e[0m; done;

Here's a PS1 you can try it with

PS1="${debian_chroot:+($debian_chroot)}\[${BWhite}\]\u\[${NC}\]\[${Yellow}\]@\[${White}\]\h\[${NC}\]:\[${BBlue}\]\w\[${NC}\]$ "

I plan to use these quite often as I build out my .bash_aliases but the most immediate use you might find useful is the PS1 generation tool I’ve setup an alpha of http://interwebninja.com/ps1-o-matic/

ps1-o-matic-0.5 video in action




Below is a screenshot of how I’m revising the script to use only javascript objects rather than relying on html inputs to store values:

I’m currently at the point where I’m using jquery ui’s draggable and sortable to allow segments to be moved around but I have nothing worth sharing yet. Stay tuned, I’ll be checking the source into my github account soon.

Sorry screen, tmux is better (but here are some screen-like hotkeys)

Introduction

If you’re familiar with the command line on Linux or UNIX, you’ve likely heard of a program called “screen”, which allows you to create virtual terminal sessions inside of your current terminal. The major benefit to this is the ability to dettach and reattach screen sessions, leaving your programs to act as if you never left. Additionally, you can have multiple buffers inside your screen that act like tabs, allowing you to flip between.

The major difference between screen and tmux is their ability to split and manage splits. Oh yeah, that and hotkeys. I think they were trying to pay legacy (or force you to change back :-P) by setting up your main key to be “b” instead of “a”, which is an awkward reach. Tux Wears Fedora below shares more screen like hotkeys, as do I with some minor tweaks that I combined with the default example in ubuntu /usr/share.

I became well acquainted at Tux Wears Fedora’s post on tmux migrating from screen.

# ~/.tmux.conf
# By Tyler Mulligan. Public domain.
#
# This configuration file binds many of the common GNU screen key bindings to
# appropriate tmux key bindings. Note that for some key bindings there is no
# tmux analogue and also that this set omits binding some commands available in
# tmux but not in screen.
#
# Note this is a good starting point but you should check out the man page for more
# configuration options if you really want to get more out of tmux

### Unbind existing tmux key bindings (except 0-9).

# Set the prefix to ^A.
unbind C-b
set -g prefix ^A
bind a send-prefix

# Bind appropriate commands similar to screen.
# lockscreen ^X x
unbind ^X
bind ^X lock-server
unbind x
bind x lock-server

# screen ^C c
unbind ^C
bind ^C new-window
bind c
bind c new-window

# detach ^D d
unbind ^D
bind ^D detach

# displays *
unbind *
bind * list-clients

# next ^@ ^N sp n
unbind ^@
bind ^@ next-window
unbind ^N
bind ^N next-window
unbind " "
bind " " next-window
unbind n
bind n next-window

# title A
unbind A
bind A command-prompt "rename-window %%"

# other ^A
unbind ^A
bind ^A last-window

# prev ^H ^P p ^?
unbind ^H
bind ^H previous-window
unbind ^P
bind ^P previous-window
unbind p
bind p previous-window
unbind BSpace
bind BSpace previous-window

# windows ^W w
unbind ^W
bind ^W list-windows
unbind w
bind w list-windows

# quit \
unbind \
bind \ confirm-before "kill-server"

# kill K k
unbind K
bind K confirm-before "kill-window"
unbind k
bind k confirm-before "kill-window"

# redisplay ^L l
unbind ^L
bind ^L refresh-client
unbind l
bind l refresh-client

# More straight forward key bindings for splitting
unbind %
bind | split-window -h
bind v split-window -h
unbind '"'
bind - split-window -v
bind h split-window -v

# History
set -g history-limit 1000

# Pane
unbind o
bind C-s down-pane

# Terminal emulator window title
set -g set-titles on
set -g set-titles-string '#S:#I.#P #W'

# Status Bar
set -g status-bg black
set -g status-fg white
set -g status-interval 1
set -g status-left '#[fg=green]#H#[default]'
set -g status-right '#[fg=yellow]#(cut -d " " -f 1-4 /proc/loadavg)#[default] #[fg=cyan,bold]%Y-%m-%d %H:%M:%S#[default]'

# Notifying if other windows has activities
setw -g monitor-activity on
set -g visual-activity on

# Highlighting the active window in status bar
setw -g window-status-current-bg red

# Clock
setw -g clock-mode-colour green
setw -g clock-mode-style 24

# :kB: focus up
unbind Tab
bind Tab down-pane
unbind BTab
bind BTab up-pane

# " windowlist -b
unbind '"'
bind '"' choose-window

Splitting is what initially caused me to migrate but there are plenty of other features that have lead me to stay. this article outlines the benefits in detail. Once you go tmux, you never go back.

Using Terminal Program "screen" on Linux / UNIX

Introduction

The terminal program “screen” for Linux / UNIX, is a command line tool that allows you to emulate terminals inside a currently running session and detach the ‘screen’ to the background. This program may seem obscure to new users because of it’s abstract nature and unusual key bindings but once you start learning the basics, the advanced usages don’t seem that scary.

If you consider yourself to have an advanced sense of the command line and you’d like to go more advanced out of gate, you should consider skipping ahead to my article on the screen successor, tmux which allows for more advanced splitting (and apparently better code).

Getting Started

Below is a walk through of an average screen scenario I’ve put together to give you an idea of how you might use this program in your workflow.

to start a basic screen session type:

screen

this will put you in a virtual session.

to detach the screen press: ctrl+a, d

to reattach the screen type:

screen -r

detach again, then type:

screen

and detach this. You now have 2 screen sessions open, so when you type screen -r, instead of reattaching, it will list the possible screens to attach. You’ll see something like the following:

There are several suitable screens on:
    31454.pts-2.quadjutsu    (10/10/2009 09:45:51 AM)    (Detached)
    31219.pts-2.quadjutsu    (10/10/2009 09:44:27 AM)    (Detached)
Type "screen [-d] -r [pid.]tty.host" to resume one of them.

So I’d type:

screen -r 31454

to attach the first one. On systems with ‘pkill’ you can type:

pkill screen

to kill all the screen sesions.

That’s one way to separate screens… another is virtual “tabs” within a screen session.

so lets create a new screen session with:

screen

Then type:

ls

so you have some data to reference for which “tab” you’re in

Then hit: ctrl+a, c

This will create a new tab.

to cycle forward through the tabs (next), hit: ctrl+a, n or ctrl+a, [spacebar]
to cycle backwards through the tabs (previous), hit: ctrl+a, p or ctrl+a, [backspace]

to kill a tab, type: exit

I found a good reference for “screen” hotkeys at pixelbeat.org if you’d like to learn more.

MySQL and Python – The Problems and [Some] Solutions

Intro

I’m still relatively new to Python. I dabble once in a while. Usually when I catch myself writing something overly complicated in bash. I figured doing MySQL from a Python script would be easier and better implemented in Python. Well, I was in for a few surprises I’d like to share with others looking to integrate with MySQL in Python before you encounter the same problems I did.

Getting on with it

As a Python noob, it made sense to me to use the python-mysqldb module. It “works” but not well. Read through the [working] code below and see if you can find the problem and how I circumvented it.

#!/bin/python
# Tyler Mulligan (tyler@doknowevil.net)
# MySQL the WRONG way in Python
import MySQLdb

def connection_settings():
    global DB_ADMIN, DB_ADMIN_PASSWORD, DB_NAME, DB_HOST, DB_USER, DB_PASSWORD
    DB_ADMIN="root"
    DB_ADMIN_PASSWORD="local"
    DB_NAME="foobardb"
    DB_HOST="localhost"
    DB_USER="foobar"
    DB_PASSWORD="local"

def run_admin_sql(): 
    conn = MySQLdb.Connection(host=DB_HOST, user=DB_ADMIN, passwd=DB_ADMIN_PASSWORD)
    cursor = conn.cursor()
    
    #cursor.execute("DROP USER %s@%s", (DB_USER, DB_HOST))
    #cursor.execute("DROP DATABASE %s" % DB_NAME)
    
    cursor.execute("CREATE USER %s@%s IDENTIFIED BY %s", (DB_USER, DB_HOST, DB_PASSWORD))
    cursor.execute("CREATE DATABASE %s" % DB_NAME)
    cursor.execute("GRANT USAGE ON %s.* TO %s@%s IDENTIFIED BY '%s'" % (DB_NAME, DB_USER, DB_HOST, DB_PASSWORD))
    cursor.execute("GRANT ALL PRIVILEGES ON *.* to %s@%s IDENTIFIED BY %s", (DB_USER, DB_HOST, DB_PASSWORD))
    
    cursor.close()
    conn.close()
    
    conn = MySQLdb.Connection(db=DB_NAME, host=DB_HOST, user=DB_USER, passwd=DB_PASSWORD)
    cursor = conn.cursor()

    cat_table_sql = """CREATE TABLE cats
    (
    `id` int(3) unsigned NOT NULL AUTO_INCREMENT,
    `name` varchar(50) NOT NULL,
    `age` int(2) NOT NULL,
    `type` int(2) NOT NULL,
    PRIMARY KEY (`id`)
    )"""
    
    cursor.execute(cat_table_sql)
    cursor.execute("INSERT INTO cats (name, age, type) VALUES (%s, %s, %s)", ("fluffy", 5, 1))
    cursor.execute("INSERT INTO cats (name, age, type) VALUES (%s, %s, %s)", ("meow meow", 6, 1))
    cursor.execute("INSERT INTO cats (name, age, type) VALUES (%s, %s, %s)", ("purrfect", 3, 1))
    
    cursor.execute("SELECT * FROM cats")
    
    results = cursor.fetchall()
    cursor.close()
    conn.close()
    return results


connection_settings()
print run_admin_sql()

The problem is parameterization. The python-mysqldb module sucks at it. It’s glaringly obvious when you look at the follow snippet and how I worked around it.

cursor.execute("CREATE USER %s@%s IDENTIFIED BY %s", (DB_USER, DB_HOST, DB_PASSWORD)) # GOOD
cursor.execute("CREATE DATABASE %s" % DB_NAME) # BAD NEWS BEARS
cursor.execute("CREATE DATABASE %s", (DB_NAME)) # but the module won't support this

The difference, is that the first implementation uses the module’s parameterization to escape the variables that are passed to it, making it safe. The second is using Python’s string formatter which means I’d have to do all the escaping and sanitization prior, which is obviously more dangerous and annoying.

The MySQLdb docs bury this little note, “Parameter placeholders can only be used to insert column values. They can not be used for other parts of SQL, such as table names, statements, etc.”

You can implement it this way but you’ll be wasting your time. There are better modules out there to make up for this one’s shortcomings. OurSQL and SQLAlchemy were the two libraries recommended to me in #python on irc.freenode.org. I will make another post when I develop the same example above using one of these libraries.