1. Introduction
  2. What didn't work
  3. What might work elsewhere
  4. What seems to work so far
  5. Things that started to go right
  6. Search, don't categorize
  7. Other things that help
  8. Useful links
  9. Feedback

1. Introduction

This is written from a Unix/Linux perspective, but Mac users running a recent version of the operating system should be able to make sense of it.

Some terms for non-Unix types:

2. What didn't work

The problem: the filesystem on my Unix workstation was a mess. I couldn't find anything without grepping all over creation. About half the time, I'd actually find something useful. Usually, I'd get no hits at all, or I'd match something like a compiled binary and end up hosing my display beyond belief.

This is what didn't do the trick:

2.1. My own categories

I tried about six different categorization schemes, all of which went nowhere. Poking around in the IBM and Red Hat distribution webpages gave me setups that looked like this:

Making the directory tree was easy. Figuring out where to put things under it, not so much.

For example, I'd learn how to use a program like tcpdump, which is great for examining network traffic to find out why two of your systems aren't working and playing nicely together.

Where does it go? Should it be Networks? Well, only systems people can use the program because it's intended for troubleshooting and you can see things like passwords and the contents of email messages in transit, so how about Admin? I'm not storing the software as such, just instructions on how to use it, so how about Documentation?

Notice the Misc category? Where do you suppose most stuff ended up?

2.2. Make a projects directory

A friend of mine had a well-organized system, and I happened to notice that he had a directory called projects under his home directory. Sounded good, so I made one, too.

OK, do I use the project name for each separate project? Oops, some projects have either duplicate or very close names. How about using the name of the person requesting the work? Does a two-minute job count as a project?

In short, all the same defects as making up my own categories, plus some brand new ones.

2.3. Dump it all in $HOME

I tried this during my "to hell with it" phase, and wound up with over 1,000 files in my home directory. Take a wild guess how well that worked out.

2.4. Separate files by filetype

OK, let's try putting text files in one area, PDF files in a second area, web-pages used for presentations in a third area, etc.

All this got me was a bunch of strangely-named files without much in common, in no particular order. At least I could grep through the text files without accidentally getting a match from a PDF file and having gibberish all over my screen.

2.5. Treat it all as email

I noticed someone else storing most of his daily work as email. Anything having to do with a given project all ended up in a mail folder, and he could just use the mail-reader of his choice to navigate around in it. Sounded cool.

658 email folders later, I noticed a tiny problem. Where did I put that security notice about Solaris operating systems? Was it in the security folder or the solaris folder? And why did I put it in this one when it clearly should have gone into the other one? (Pick whatever values you like for "this" or "the other".)

All these methods had one thing in common: putting a bunch of semi-structured information somewhere and expecting it to magically organize itself. Trying to impose additional order on stuff like this is a waste of time when you consider how much new semi-structured information we send and receive every day.

3. What might work elsewhere

A few things I tried that almost worked...

3.1. Dewey Decimal System

I ran into something called CyberDewey, written by David Mundie. This sounded like the neatest thing since sliced bread, especially since he seemed to have the same problems I did when trying to organize files.

I went so far as to buy a copy of the Abridged Dewey Decimal Catalog, which is actually pretty nifty; if you're looking to organize your paper files, you could do a lot worse than use an existing classification scheme like this.

For example, let's say I have an article on "hurricane relief" and I want to know where to file it. I flip to the Relative Index portion of the catalog (about 200 pages out of 1,000 total), look for "hurricanes", and see something like this: 

Hurricanes: 551.55
   Weather forecasting: 551.64
   See also: Disasters

"Disasters" sounds promising, so I try that: 

Disasters: 904
   ...
   Social services: 363.34
   Public administration: 353.9

This gives me some category numbers to check in the front portion of the catalog: 

363.34    Disasters, including floods and war

353.9     Safety administration
          ...
	  Disaster and emergency planning

904       Collected accounts of events

551.55    Atmospheric disturbances, including cyclones, hurricanes, ...

At this point, I have some choices about how to file my article: make a folder called 904 if it's an interview with a Katrina survivor, or 353.9 if it's about FEMA or government response, etc. It's nice because the hard stuff (figuring out the categories and where they go) has all been done for you.

Unfortunately, the stuff available in the Abridged version is a little too general for my job. I also didn't feel like either typing in the categories by hand or forking over $275 for a copy of WebDewey. You can get some of the DDC headings in digital form, but not enough to solve my problem.

See also:

3.2. DMOZ setup

The Open Directory Project wants to be the largest, most comprehensive human-edited directory of the Web. Among other things, they provide a nice set of categories that are well-organized, free, and already in digital form.

It's not hard to take their category list and turn it into a directory tree suitable for a webpage. My homepage includes a section based on that list:

The 00hierarchy link holds the computer-related topics from the DMOZ category list that I thought would be most useful.

This is OK for a setup with a small number of files and reasonably clean delineation between topics, but it didn't quite do the trick for my daily work.

3.3. Canadian Government setup

"But minister, it isn't like this film is the first troublesome thing to come out of Canada. Let us not forget Bryan Adams."

"No, no. The Canadian government has apologized for Bryan Adams on several occasions." --"South Park, Bigger, Longer, and Uncut"

Canada took a stab at making a set of consistent categories for government records management called ARCS. It's a block numeric records classification system based on function and subject. Each functional or subject grouping of records is assigned a unique three- or four-digit number; this is called a primary number, and it's the main building block for the system. This number is used to classify all information related to a subject or function, regardless of physical format.

Most government offices deal with a similar set of administrative requirements, and the ARCS setup is a pretty nice representation. The documentation comes in PDF form and is very thorough. I made a directory tree suitable for web access based on this setup, which you can see here:

The plain-text categories file used to generate the directory tree and indexes is here. A similar system more suitable for colleges and universities can be found here.

This is intended more for administrative work than something like my job.

3.4. Trouble ticket system

I looked at several trouble-ticket/user-request systems, and they're dandy for collaboration between helpdesk people, but they don't really do the trick for organizing my day-to-day information. Here are some of the more interesting packages and papers I saw:

The one that seems to have improved the most is Roundup. Features:

4. What seems to work so far

4.1. Cherry-picking GTD

The idea of doing a complete brain-dump and getting everything stored in a trusted location was probably the most useful thing I got from the "Getting Things Done" book.

My job as a system administrator doesn't change every day, but it's much easier to keep track of things via date rather than via subject. I tend to remember things in time-order, so I finally stopped trying to change the way I work to fit some hierarchy. Instead, I made a directory structure on the machine to match my work habits.

I have a top directory cleverly named notebook with subdirectories in the form yyyy/mmdd, so every day has its own folder: 

me% dtree ~/notebook
notebook:
+-----2000
|      +-----0309
|      |      +-----ssh-and-cvs.txt
|      +-----0411
|      |      +-----infodoc-14272.txt
|      |      +-----infodoc-15583.txt
...
|      +-----1230
|      |      +-----solaris_dev_random.txt
+-----2001
...
+-----2002
...
+-----2003
...
+-----2004
...
+-----2005
...
|      +-----0926
|      |      +-----online-help.txt
|      |      +-----wiki-collaboration.txt
|      |      +-----agenda
|      |      +-----LOG
...
|      +-----1113
|      |      +-----agenda
...

Someone might ask me, "Remember that thing we broke last Wednesday, and then we fixed it on Thursday before anyone noticed?" 

me% date
Tue Sep 27 16:46:48 EDT 2005

me% cal 9 2005
   September 2005
Su Mo Tu We Th Fr Sa
             1  2  3
 4  5  6  7  8  9 10
11 12 13 14 15 16 17
18 19 20 21 22 23 24
25 26 27 28 29 30

me% cd ~/notebook/2005/0921
me% ls -l
total 4
-r--r--r--  1 vogelke  vogelke  424 Sep 21 20:34 LOG
-r--r--r--  1 vogelke  vogelke  117 Sep 21 13:24 that-thing-we-broke

me% cat that-thing-we-broke
Wed, 21 Sep 2005 13:24:44 -0400

    Some blarney about whatever's broke.  Could be an email message,
    plain text, etc.

me% cd ~/notebook/2005/0922
me% ls -l
...
-r--r--r--  1 vogelke  vogelke     88 Sep 22 14:23 how-we-fixed-it
...

me% cat how-we-fixed-it
Thu, 22 Sep 2005 14:08:22 -0400

    Thank the deity of your choice for the backup server.

4.2. Indexed search

Make sure the files you use and modify most often are indexed for rapid retrieval. grep is fine for doing a quick-and-dirty search through a few files, but it doesn't work nearly as well for anything larger.

5. Things that started to go right

The most noticeable improvement was finally being able to find the things that happened on a given date.

Here's what convinced me that I was on the right track: I'd think of a problem, and a fix would occur to me a moment later that was consistent and doable. Most of the scripts and utilities below were either already available or didn't take more than 20 minutes to write.

5.1. Where is today's stuff?

About 95% of the time, I'm dealing with stuff for yesterday, today, and the coming week. A few symbolic links made navigation pretty easy: 

me% date
Tue Sep 27 16:50:30 EDT 2005

me% pwd
/home/vogelke

me% ls -lF
...
lrwxr-xr-x    1 vogelke  18 Sep 26 00:02 yesterday@ -> notebook/2005/0926
lrwxr-xr-x    1 vogelke  18 Sep 27 00:02 today@ -> notebook/2005/0927
lrwxr-xr-x    1 vogelke  18 Sep 27 00:02 tomorrow@ -> notebook/2005/0928
...
lrwxr-xr-x    1 vogelke  18 Sep 27 00:02 tuesday@ -> notebook/2005/0927
lrwxr-xr-x    1 vogelke  18 Sep 27 00:02 wednesday@ -> notebook/2005/0928
lrwxr-xr-x    1 vogelke  18 Sep 27 00:02 thursday@ -> notebook/2005/0929
lrwxr-xr-x    1 vogelke  18 Sep 27 00:02 friday@ -> notebook/2005/0930
lrwxr-xr-x    1 vogelke  18 Sep 27 00:02 saturday@ -> notebook/2005/1001
lrwxr-xr-x    1 vogelke  18 Sep 27 00:02 sunday@ -> notebook/2005/1002
lrwxr-xr-x    1 vogelke  18 Sep 27 00:02 monday@ -> notebook/2005/1003
...

I can get to next Monday's folder by typing cd ~/monday at the prompt, no matter where I am in the filesystem. Since I use the Z-shell, I can also get there by just typing monday. Here are the shell settings: 

cdpath=(.. ~ )  # specify a search path for the cd command.
setopt autocd   # cd to a directory if it's the first word on the command line.

5.2. How to do scheduling?

Remind is a calendar and reminder program for Linux and most UNIX Systems, and it's made to order for this sort of thing. Two good articles on remind can be found here:

The source can be found at http://www.roaringpenguin.com/penguin/remind-03.00.23.tar.gz

5.2.1. Notebook directories

I use remind to generate the symlinks above from a cron job that runs just after midnight every day: 

#!/bin/sh
# mknbdir: make notebook directory for the day.
# should be run just after midnight.
# make symbolic links for today, tomorrow, etc.

PATH=/usr/local/bin:/bin:/sbin:/usr/sbin:/usr/bin
export PATH

die () {
    echo "$@" >& 2
    exit 1
}

cd $HOME
top='notebook'
test -d "$top" || die "$top: dir not found"

#
# Handle pages for today and yesterday.
#

cur="`date +%Y/%m%d`"
test -d "$top/$cur" || mkdir -p "$top/$cur"
test -d "$top/$cur" || die "unable to make $cur"
test -L yesterday && rm yesterday
test -L today && rm today
ln -s $top/$cur today

#
# Handle pages for other dates.
# NOTE: "date" must have "-d" option for this to work.
#

other='yesterday tomorrow sunday monday tuesday wednesday
       thursday friday saturday'

for day in $other
do
    cur=`date -d "$day 03:00" +%Y/%m%d`
    test -d "$top/$cur" || mkdir -p "$top/$cur"
    test -d "$top/$cur" || die "unable to make $cur"
    test -L $day && rm $day
    ln -s $top/$cur $day
done

#
# Keep a year's worth in advance.
# NOTE: must have "remind" installed for this to work.
#

cd $HOME/$top

echo 'REM MSG daily' |
    remind -s+53 - |
    sed -e 's!^\(....\)/\(..\)/\(..\).*!\1/\2\3!' |
while read dir
do
    test -d $dir || mkdir -p $dir
done

exit 0

This script also illustrates using remind to generate dates without having to worry about leap years, etc. To print this week plus the next 52 weeks: 

me% date
Tue Sep 27 13:36:20 EDT 2005

me% echo REM MSG daily | remind -s+53 -
2005/09/25 * * * * daily
2005/09/26 * * * * daily
2005/09/27 * * * * daily
2005/09/28 * * * * daily
2005/09/29 * * * * daily
2005/09/30 * * * * daily
2005/10/01 * * * * daily
2005/10/02 * * * * daily
2005/10/03 * * * * daily
...
2006/09/25 * * * * daily
2006/09/26 * * * * daily
2006/09/27 * * * * daily
2006/09/28 * * * * daily
2006/09/29 * * * * daily
2006/09/30 * * * * daily

5.2.2. Daily agendas

Anything on my todo list for today goes in a file called agenda under today's notebook directory. The a command simply runs remind on today's agenda file. 

me% date
Tue Sep 27 13:36:20 EDT 2005

me% cat ~/today/agenda
REM MSG 3rd quarter training ends today

me% a
Reminders for Tuesday, 27th September, 2005 (today):

3rd quarter training ends today

A slightly more advanced script to display your agenda for the next few days can be found here.

The rem script below lets me edit or create new agendas quickly: 

#!/bin/ksh
# rem: edits the agenda file in today's notebook directory
# (default), or one entered on the command line.  If the
# agenda file doesn't exist, a new one is created.

PATH=/bin:/usr/bin:/usr/local/bin
: ${EDITOR="/usr/bin/vi"}
export PATH EDITOR
umask 022

afile=$HOME/today/agenda

for ac_option
do
    case "$ac_option" in
        -*) ;;
        t)  die "Please specify -to, -tu, or -th" ;;
        s)  die "Please specify -sa or -su" ;;

        y*)   afile=$HOME/yesterday/agenda ;;
        to*)  afile=$HOME/tomorrow/agenda  ;;
        m*)   afile=$HOME/monday/agenda    ;;
        tu*)  afile=$HOME/tuesday/agenda   ;;
        w*)   afile=$HOME/wednesday/agenda ;;
        th*)  afile=$HOME/thursday/agenda  ;;
        f*)   afile=$HOME/friday/agenda    ;;
        sa*)  afile=$HOME/saturday/agenda  ;;
        su*)  afile=$HOME/sunday/agenda    ;;
        *)    afile="$ac_option"           ;;
    esac
done

test -e "$afile" || echo "REM MSG new agenda" > $afile
exec $EDITOR $afile
exit 0

(Additional command line options to specify the day by John Hoffoss, john dot t dot hoffoss at gmail.com)

5.2.3. Timed popup reminders

Let's say I have a meeting at 2pm in room 205, and I'd like a reminder to show up on my screen 5 minutes before it starts. 

me% echo REM MSG Meeting in 205 > ~/today/1355.rem

is all it takes. A small program called showcal runs once every minute looking for a file called hhmm.rem in today's notebook directory. If it finds one, the contents are run through remind and then to another program which handles screen popups.

The showcal program is here.

What if I want a reminder every day at a certain time? I just put the same type of file in the ~/.calendar directory instead: 

me% cd ~/.calendar

me% ls -l
-rw-r--r--  1 vogelke  vogelke    74 Jun  7 17:12 1245.rem
-rw-r--r--  1 vogelke  vogelke    33 May  7 19:03 1455.rem
...
-rw-r--r--  1 vogelke  vogelke    90 Jul 15 12:29 2035.rem
...

me% cat 2035.rem
REM Sun MSG Desperate Housewives 9pm
REM Wed MSG Veronica Mars 9pm

This reminder will show up at 8:35 pm to ensure that I don't miss anything crucial if I happen to be at work late on Sunday or Wednesday. It won't show up on any other days.

5.3. How to record progress?

6. Search, don't categorize

It's a lot easier to break up my workstation text files by frequency of updates, and then set up searching appropriately:

I installed SWISH-E to index anything in text format. That plus one or two shell scripts handles my search needs.

7. Other things that help

7.1. A good shell

I used tcsh and bash for several years. They're fine programs, but I haven't found anything to match the flexibility of the Z-shell:

7.2. A good window manager

I started using the FVWM window manager in 1999, and really got to like it when configured using Eric Raymond's Big Blue-Steel Desktop. However, I wanted to make better use of the open desktop space I had, so I moved to the IceWM window manager a few months ago.

I have four virtual desktops for my basic work environment:

Function keys F1-F4 take me to Desktops 1-4, respectively. F5 locks my keyboard. F6 lets me either restart IceWM or logout.

IceWM takes up very little screen real-estate. I have one toolbar at the bottom of my screen which is barely 3/8ths of an inch high, so I have plenty of room for applications:

My IceWM setup files are here:

My general X-windows setup files are here:

7.3. A decent browser

You need a web browser that does something besides act like a virus delivery service. Either Mozilla or Firefox will do the trick; I prefer Mozilla because I had too much trouble trying to get the keyboard working the way I liked on Firefox.

My only complaint was having the Backspace key take me to the previous link, instead of simply moving the display of the current page back by one screen. I added lines 34-35 to /usr/X11R6/lib/mozilla/res/builtin/htmlBindings.xml to change the Backspace key accordingly: 

 1  <?xml version="1.0"?>

...

32  <handler event="keypress" keycode="VK_LEFT" command="cmd_scrollLeft" />
33  <handler event="keypress" keycode="VK_RIGHT" command="cmd_scrollRight" />

34  <handler event="keypress" keycode="VK_BACK" command="cmd_scrollPageUp" />
35  <handler event="keypress" keycode="VK_DELETE" command="cmd_scrollPageUp" />
36  <handler event="keypress" keycode="VK_HOME" command="cmd_scrollTop"/>
37  <handler event="keypress" keycode="VK_END" command="cmd_scrollBottom"/>

38  <handler event="keypress" key="x" command="cmd_cut" modifiers="accel"/>
39  <handler event="keypress" key="c" command="cmd_copy" modifiers="accel"/>
...

(Unfortunately, this doesn't seem to work with Mozilla-1.7.12. You have to unzip the file toolkit.jar in /usr/X11R6/lib/mozilla/chrome, go into the content directory, modify the platformHTMLBindings.xml file to hold the keymappings you like, and create a new toolkit.jar file holding the modified files.)

Here's a list of keyboard shortcuts.

My Mozilla setup files are here:

7.4. Configurable mail delivery

I use qmail as my message transfer agent because it's secure and extensible. My userid is vogelke, so I own any address starting with vogelke- and I can filter mail to that address accordingly.

I have these qmail files in my home directory: 

me% cat ~/.qmail 
| preline /usr/local/bin/procmail
| (preline /bin/cat; echo) >> $HOME/backup/$USER

The basic .qmail file is for any mail addressed to me without any dash-extensions on the end. Each message is sent through the procmail program for further filtering, and a copy of that same message will also be appended to my backup/vogelke file. 

me% cat ~/.qmail-bcc 
| (echo; preline /bin/cat; echo) >> $HOME/mail/sentmail

When I want to save an outgoing message, I send a BCC to vogelke-bcc, and a copy of the message as seen by the mail delivery software is appended to my mail/sentmail file.

This way, the message in sentmail is identical to the one seen by the recipient, in case any troubleshooting is needed. Some mail-readers can save a copy of your outgoing mail, but they don't do such a good job saving the headers you might need in case there's a delivery problem. 

me% cat ~/.qmail-blog 
| preline /bin/cat | $HOME/bin/newpost

If I want to send mail to someone and have a copy of that message posted to my weblog, I send a BCC to vogelke-blog. A copy of the message is sent to the newpost program, which readies it for web display.

The newpost program is here. 

me% cat ~/.qmail-header 
| (preline formail -XFrom: -XSubject: -XDate: -XTo: -XCc: 
-XMessage-ID: ; echo) >> $HOME/mail/SENT.`/bin/date +%Yw%W`

This example is a little more complex. I like to keep a record of what messages I've sent without having to keep copies of every single message, so I always sends a BCC to vogelke-header. A copy of the outgoing message is sent through the formail program, which extracts the most useful lines from the header and appends them to a file in my mail folder called SENT.year-week: 

me% cd mail

me% ls -l SENT.*
-rw-------  1  vogelke  2297 Apr 22 14:01 SENT.2005w16
-rw-------  1  vogelke  2094 Apr 30 21:31 SENT.2005w17
-rw-------  1  vogelke  1045 May  6 16:57 SENT.2005w18
...
-rw-------  1  vogelke  3954 Sep 18 17:59 SENT.2005w37
-rw-------  1  vogelke  2521 Sep 23 12:37 SENT.2005w38
-rw-------  1  vogelke   445 Sep 27 13:40 SENT.2005w39

me% cat SENT.2005w39
Date: 26 Sep 2005 14:53:45 -0400
Message-ID: <20050926185345.74763.qmail@kev.nowhere.usa>
From: "Karl Vogel" <vogelke@example.com>
To: someone@somewhere.else
Cc: webmaster@example.com
Subject: Re: Web login error

Date: 27 Sep 2005 13:40:40 -0400
Message-ID: <20050927174040.85874.qmail@kev.nowhere.usa>
From: "Karl Vogel" <vogelke@example.com>
To: jane@example.com
Subject: Re: Access problem this morning

me%

This provides me with a record of everyone I've mailed, broken out by week. 

me% cat ~/.qmail-xnote 
| $HOME/bin/xnote

Mail to vogelke-xnote goes through a program called xnote, which displays popup messages. Sending something to vogelke-xnote with the subject line Wake up causes a popup like this to appear:

This is very useful, because any host that can send you mail can pop a message on your screen. The servers I maintain are all set up to send mail to me in this way whenever they shut down nicely and reboot, or whenever hourly checks indicate that disk space is getting tight, etc.

The xnote program is here. It's basically a wrapper for another program called xalarm, which can be downloaded from http://www.x.org/contrib/utilities/

Plain-text versions of the .qmail files are here.

7.5. A good email reader

Mutt is a fast, customizable mail-reader with lots of features:

My setup is almost identical to the one on this site. Here's a screenshot when reading message 1 of 27 from my inbox:

My screen is set to display 47 lines at a time, and most of my email messages are shorter than that, so I rarely have to scroll through multiple pages to see if I need to keep or act on a message.

7.6. Templates

Most wheels aren't worth re-inventing. If you find yourself constantly rewriting the same code snippets or emails, it's time to pick a language and a template setup.

Perl and the Text::Template package suit me fine, but if push comes to shove, any decent scripting language with variable substitution can serve as a template engine.

Here are some of the better articles I've seen on choosing (or writing) a template system.

7.7. Code fragments or cliches

If you spend more than five minutes figuring out how some language function or WordSmasher-2000 utility works, write it down.

I have a directory called ~/cliche which holds things that held me up, or things I don't want to lose, or things I don't feel like typing in again: 

me% dtree ~/cliche
cliche
+-----ASCII
|      +-----alphabet
|
+-----C
|      +-----days-of-week
|      +-----file-locking
...
|      +-----read-directory
|      +-----set-bit-flags
|      +-----temp-file-creation
|
+-----html
|      +-----clickable-table-cells
|      +-----comment-form
|
+-----perl
|      +-----approximate-string
|      +-----argument-handling
|      +-----columnize
|      +-----context-grep
|      +-----counter-file
...
|      +-----using-dumper
|      +-----word-wrap
|      +-----yesterday
|
+-----sh
|      +-----args
|      +-----csort
|      +-----die
|      +-----ksh-random-number
+-----web
|      +-----generate-toc
|      +-----headers
|      +-----param-list

I called it cliche because most of the snippets are the moral equivalent of "I'm just here for the team"; people expect them, and I'll probably end up using them sooner or later.

For example, the file ~/cliche/ASCII/alphabet simply keeps me from having to stumble all over the keyboard if I need to loop through the alphabet for some reason: 

0 1 2 3 4 5 6 7 8 9 a b c d e f g h i j k l m n o p q r s t u v w x y z

The file ~/cliche/perl/yesterday is a 4-line function in the perl language providing the time 24 hours ago, and so on.

8. Useful links

9. Feedback

Feel free to send comments.

Generated from organizing.t2t by txt2tags
$Revision: 1.29 $