Today something great and something horrible happened within 5 minutes of each other. I got a letter from Chase and opened it in front of my computer as i am always prepared for the worst when dealing with scum like that. So I quickly scanned the friendly letter and almost exploded, it was a very polite letter happily informing me that my 1.9% APR on my credit card originally gotten from WaMu is from now on going to be 29.99%. That is not a typo.

ally bank

One of the main reasons I wanted to share this on my blog is because take a look at that website! Someone sure knows how to make money, there is alot to admire about this site for you marketers and developers alike. But seriously, getting a money-market account with no minimums, no monthly fees, no direct-deposit requirements, and none of all those other made-up pirate words is pretty unheard of. If I was a journalist I would want to know who it is that is behind this. But knowing how the banking industry is it’s probably their idea of a “lead-generating-tool” that they can later just buy outright and stick it to all the “leads”. It’s basically the same setup as PayPal and their optional money market account that I use, except unlike paypal you can send and receive money without them taking a big chunk. And who ever heard of such a thing as a no-penalty CD that doesn’t have hidden costs?

Have you heard congress wants to make it legally required to inform your blog readers if you are getting paid to promote a product?
Well don’t worry about me, I never have been paid and I literally just opened my account 20 minutes ago. I’m just anal about not having to pay a bank to let them lend it to others. ally vs Chase is the same thing as Open Source vs Microsoft.

Any other ideas?

Update: Best Free Online Banking originally appeared on AskApache.

My Setup

As this screenshot shows, I only use a handful of add-ons at a time. These buttons allow me to clear the DNS/Cookies/Cache for whichever site I’m on when I click it. Very very helpful for me as a web designer. The 4th button there is just a restart button. Other than those, Firebug, YSlow, LastPass, and Web Developer are the only ones I always use regularly.

I like the idea of the last.fm but it’s not as powerful as the site, which is awesome. Lately I’ve been listening to Kings of Leon Radio…

Slow Firefox

The more add-ons you have, disabled or not, the slower Firefox is. (unless you are running your profile folder in TMPFS). Also, bookmarks and settings like that have a performance hit. I have been using Firefox since it launched way back when, and I have always kept my bookmarks when moving to new machines and new installations.. So with over 5 thousand bookmarks I finally did some debugging and discovered that was a huge cause of Firefox acting slow. Now I am trying to migrate them all over to Google’s Gmarks, which knowing Google will be awesome eventually.

Multiple Firefox Installations, Sorta

The solution to all these problems is to use separate Firefox profiles which use separate folders to store your profile-specific extensions and settings. So I have profiles with upwards of 40 Addons installed and enabled, and another profile that is built for speed… It’s very very slow to be running Firebug and have multiple tabs open.. You can use the profilemanager to load them specific profiles from the command line. I personally use separate icons on my Windows Quick Launch that I just modified the shortcut pointing to firefox to also have the profile commandline. Note also that you can have multiple profiles open and running simultaneously.. This lets you do some crazy networking and other random hacks like having many Firefox instances running each of which is configured to use a separate SOCKS Proxy or network interface, so you can really open up those pipes for some intense txrx.

You can do some very powerful things with Firefox that most people are unaware of, if you are interested start with these:

• Setting up an extension development environment
• Firefox Command Line Options

About This Collection

Web Development Add-ons for Advanced Web Developers. I personally use these to work on Sites, Servers, WordPress, Javascript, PHP, CSS, XHTML, validation, page-loading, SEO, optimizing, and much more. Add this collection.

Created by:AskApache

Updated:September 22, 2009

Share this Collection

• Digg this!
• Post to Facebook
• Add to Delicious
• Post to MySpace
• Share on FriendFeed
• Post to Twitter

Firefox Add-ons for Web Developers originally appeared on AskApache.

Keep in mind that this blog and my research is only a hobby of mine, my job is primarily marketing and sales, so I’m not some licensed expert or anything, or even an unlicensed expert! haha. But it does bother me that those who are tech-savvy enough to run web-hosting companies are happily ripping people off. So this article details the main tools that are used to speed up and optimize your machine by delegating levels of priority to specific processes. Future articles will use these tools alot, so this is meant as an intro.

CPU and Disk I/O

As most of you are aware, there are 2 variables that determine any computer or programs speed. CPU and Disk I/O. CPU determines how fast you can process data, crunch numbers, etc. while disk I/O determines how fast your disks can read and write data to the hard-drive. Wouldn’t it be great if you could easily configure your server to give your httpd, php, and other processes both greater CPU processing and disk IO than your non-important processes like backup scripts, ftp daemons, etc.? We are talking about Linux in this article, so of course YES not only can you do that, you should!

RAM

RAM is like a hard-drive in that data is stored on it, and read/written to it. The difference is that RAM is somewhere around 30x faster than disk I/O, but the cost of that incredible speed is that the data stored on it is only temporary in the sense that it won’t be stored permanently, it is completely erased when your machine is rebooted. RAM is also expensive, and there is a limit to how much a server or machine can have due to hardware limits.

SWAP

SWAP takes off when you run out of RAM but you still want certain data to be read/write quickly. Basically when you start running out of RAM your machine starts supplementing RAM with SWAP storage. SWAP is usually a partition on a second hard-drive disk. There is an upper limit on how much I/O can occur on a disk at one time, and the more I/O takes place, the slower all I/O becomes, so SWAP works well on a separate hard-drive as it will have much faster I/O. On Windows they opted to copy the SWAP mechanism but instead use a file named pagefile.sys, and that is just one reason people in the know do not care for Windows.

CPU

So lets do this, think of your CPU (your processor) as having an amount of 100% processing available when not being used, 0% when its maxed out. CPU’s handle multiple processing tasks simultaneously, so what we will discuss in this article is how to specify HOW MUCH of that processing amount each of your programs (heretofore “processes”) are able to use. Yes, very very cool.

That is correct, you can easily configure your server to provide more of the available processing time to certain programs over others, like you can configure apache and php to utilize 50% of your CPU processing time by themselves, so that all other processes (proftpd, sshd, rsync, etc.) combined can only utilize 50%. The terminology is we can give certain specific processes (like php.cgi, httpd, fast-cgi.cgi) a specific priority, where -19 is the most priority, and +19 is the least amount of priority, or CPU processing time. I know it seems backwards..

The Tools

If you run Windows, you are in the right place… because the following advice will save your life: GET LINUX! Ok, now that that is out of the way, the following are the tools dicussed on this page. All of them are free, open-source, and wonderful. The basic idea of these tools is to control how much CPU is devoted to each process, and also how much Disk IO/Disk traffic is given to each process.

nice

run a program with modified scheduling priority

renice

alter priority of running processes

ionice

set or retrieve the I/O priority for a given pid or execute a new task with a given I/O priority.

iostat

Report Central Processing Unit (CPU) statistics and input/output statistics for devices and partitions.

ulimit

Ulimit provides control over the resources available to processes started by the shell, on systems that allow such control.

chrt

set or retrieve real-time scheduling parameters for a given pid or execute a new task under given scheduling parameters.

taskset

set or retrieve task CPU affinity for a given pid or execute a new task under a given affinity mask.

Part 1: Process Processes Faster

Ok so lets tackle figuring out how to give your response-intensive processes (like apache, php, ruby, perl, java) meaning a request to your server/machine requires a response. For instance, when you requested this page that you are reading at this very second, several things on my server had to happen for you to be able to read this.

First your computer sends out a request to see what server the www.askapache.com domain name is. DNS servers respond with my server IP, so for servers dedicated as nameservers, optimizing the DNS processes like bind would speed that up. Now that your computer knows how to reach my server it sends an HTTP GET request for this url. This request is received by the httpd process that is apache, and apache determines this url should be handled by my custom compiled php5.3.0 binary, because this page is WordPress generated. So the php binary loads up the WordPress /index.php file, which chain-loads several other php files, including wp-config.php containing my MySql database settings. Now php connects to my MySql Server to fetch this articles content, comments, title, tags, etc. and then generates the HTML and hands that back to Apache.

Finally, Apache generates a HTTP RESPONSE and sends the RESPONSE and CONTENT back to your Browser, which then in turn renders the page for your eyes with the necessary javascript, images, css, and other files included in the HTML response.

Too much Processing

Now you see why I’ve opted to write my own caching plugin that takes the php and mysql processes OUT of that equation. Both the php binary and the mysql instance consume CPU processing, and disk IO, to load all their library files, make various network requests and sockets, check permissions, and on and on. And that’s completely ok, the thing is, unless you configure these processes (Apache, PHP, MySQL) they will use the same amount of CPU processing that other processes use, other processes that have very little to do with you reading this sentence. Processes to run my mail server, my FTP server, my SSH server, my cronjobs, cleanup scripts, atd daemon, etc.. and they will get the same amount of CPU!

Another even simpler example is what got me to look into this myself. I wrote a shell script that created hourly, daily, weekly, and monthly backups for all of my websites and sql databases, and set it up to run by cronjob at those set intervals. Eventually I noticed my sites were slower, my php even slower, and sometimes I even saw 503 errors that my host throws up when my server is overloaded. The research that I pursued to prevent that from happening has been hugely eye-opening. What does a backup script do? Mine just created tar archives of all the files in my web root, then gzipped the tar archive saving to a backup server using scp (a file transfer using ssh). This resulted in the following huge problems that seem to have nothing to do with a faster server and speedier website, but they have everything to with it.

1. CPU Bottleneck #1 – tar and gzip use compression algorithms at a low level to create a compressed version, and all that compressing uses a whole lot of crunching – CPU processing
2. DISK IO Bottleneck – Tarring the whole web root directory was creating a ton of disk io, and remember the more disk io that is going on, the less is available for everything else.
3. CPU Bottleneck #2 – Using scp to send my backups was security-smart, but these huge archive files had to be encrypted and sent over the net.

Breaking Bottles

I apologize for being a little long-winded there, but I think it’s important to make sure everyone understands those basic concepts, which are foreign to most people. Once you understand what is causing the bottlenecks, then you can understand the solutions, which actually are incredibly simple and even a novice linux user can easily do. Besides, the net gets a little bit faster every time someone implements this.

nice

Nice allows you to run a program with modified scheduling priority which specifies how much CPU is devoted to a particular process. Run COMMAND with an adjusted niceness, which affects process scheduling. With no COMMAND, print the current niceness.

Nicenesses range from -20 (most favorable scheduling) to 19 (least favorable). -n, --adjustment=N – add integer N to the niceness (default 10). nice +19 tasks get a HZ-independent 1.5%. Running a nice +10 and a nice +11 task means the first will get 55% of the CPU, the other 45%.

nice usage

nice [OPTION] [COMMAND [ARG]...]
 
-n, --adjustment=ADJUST   increment priority by ADJUST first

Examples of nice

Using nice to download a file

nice -n 17 curl -q -v -A 'Mozilla/5.0' -L -O http://wordpress.org/latest.zip

Unzipping a file with nice

nice -n 17 unzip latest.zip

Nice way to build from source

nice -n 2 ./configure
nice -n 2 make
nice -n 2 make install

It is sometimes useful to run non-interactive programs with reduced priority.

$ nice factor `echo '2^9 - 1'|bc`
511: 7 73

Since nice prints the current priority, we can invoke it through itself to demonstrate how it works: The default behavior is to reduce priority by 10.

 $ nice nice
10
$ nice -n 10 nice
10

The ADJUSTMENT is relative to the current priority. The first nice invocation runs the second one at priority 10, and it in turn runs the final one at a priority lowered by 3 more.

$ nice nice -n 3 nice
13

Specifying a priority larger than 19 is the same as specifying 19.

$ nice -n 30 nice
19

Only a privileged user may run a process with higher priority.

$ nice -n -1 nice
nice: cannot set priority: Permission denied
$ sudo nice -n -1 nice
-1

The new scheduler in v2.6.23 addresses all three types of complaints:

To address the first complaint (of nice levels being not “punchy” enough), the scheduler was decoupled from ‘time slice’ and HZ concepts (and granularity was made a separate concept from nice levels) and thus it was possible to implement better and more consistent nice +19 support: with the new scheduler nice +19 tasks get a HZ-independent 1.5%, instead of the variable 3%-5%-9% range they got in the old scheduler.

To address the second complaint (of nice levels not being consistent), the new scheduler makes nice(1) have the same CPU utilization effect on tasks, regardless of their absolute nice levels. So on the new scheduler, running a nice +10 and a nice 11 task has the same CPU utilization “split” between them as running a nice -5 and a nice -4 task. (one will get 55% of the CPU, the other 45%.) That is why nice levels were changed to be “multiplicative” (or exponential) – that way it does not matter which nice level you start out from, the ‘relative result’ will always be the same.

The third complaint (of negative nice levels not being “punchy” enough and forcing audio apps to run under the more dangerous SCHED_FIFO scheduling policy) is addressed by the new scheduler almost automatically: stronger negative nice levels are an automatic side-effect of the recalibrated dynamic range of nice levels.

renice

Renice is similar to the nice command, but it lets you modify the nice of a currently running process. This is nice for shell scripts where you can add this to the top of the script to nicify the whole script to 19.

renice usage

renice priority [ [ -p ] pids ] [ [ -g ] pgrps ] [ [ -u ] users ]
 
-g      Force who parameters to be interpreted as process group ID's.
-u      Force the who parameters to be interpreted as user names.
-p      Resets the who interpretation to be (the default) process ID's.

Examples of renice

From the shell, changes the priority of the shell and all children to 19. From a shell script, does the same but only for the script and its children.

renice 19 -p $$

This runs renice without any output

renice 19 -p $$ &>/dev/null

10 gets more CPU than 19

renice 10 -p $$

change the priority of process ID’s 987 and 32, and all processes owned by users daemon and root.

renice +1 987 -u daemon root -p 32

Part 2: Optimizing Disk I/O

Linux Scheduling Policies

The scheduler is the kernel component that decides which runnable process will be executed by the CPU next. Each process has an associated scheduling policy and a static scheduling priority, sched_priority

Processes scheduled under one of the real-time policies (SCHED_FIFO, SCHED_RR) have a sched_priority value in the range 1 (low) to 99 (high). (As the numbers imply, real-time processes always have higher priority than normal processes.) The following “real-time” policies are also supported, for special time-critical applications that need precise control over the way in which runnable processes are selected for execution:

Currently, Linux supports the following “normal” (i.e., non-real-time) scheduling policies:

SCHED_OTHER: Default Linux time-sharing scheduling

The standard round-robin time-sharing policy

SCHED_BATCH: Scheduling batch processes

This policy is useful for workloads that are non-interactive, but do not want to lower their nice value, and for workloads that want a deterministic scheduling policy without interactivity causing extra preemptions (between the workload’s tasks).

SCHED_IDLE: Scheduling very low priority jobs

This policy is intended for running jobs at extremely low priority (lower even than a +19 nice value with the SCHED_OTHER or SCHED_BATCH policies)

SCHED_FIFO: First In-First Out scheduling

A first-in, first-out policy

SCHED_RR: Round Robin scheduling

A round-robin policy.

Scheduling Classes

IOPRIO_CLASS_RT

This is the realtime io class. The RT scheduling class is given first access to the disk, regardless of what else is going on in the system. Thus the RT class needs to be used with some care, as it can starve other processes. As with the best effort class, 8 priority levels are defined denoting how big a time slice a given process will receive on each scheduling window. This scheduling class is given higher priority than any other in the system, processes from this class are given first access to the disk every time. Thus it needs to be used with some care, one io RT process can starve the entire system. Within the RT class, there are 8 levels of class data that determine exactly how much time this process needs the disk for on each service. In the future this might change to be more directly mappable to performance, by passing in a wanted data rate instead.

IOPRIO_CLASS_BE

This is the best-effort scheduling class, which is the default for any process that hasn’t set a specific io priority. This is the default scheduling class for any process that hasn’t asked for a specific io priority. Programs inherit the CPU nice setting for io priorities. This class takes a priority argument from 0-7, with lower number being higher priority. Programs running at the same best effort priority are served in a round-robin fashion. The class data determines how much io bandwidth the process will get, it’s directly mappable to the cpu nice levels just more coarsely implemented. 0 is the highest BE prio level, 7 is the lowest. The mapping between cpu nice level and io nice level is determined as: io_nice = (cpu_nice + 20) / 5.

IOPRIO_CLASS_IDLE

This is the idle scheduling class, processes running at this level only get io time when no one else needs the disk. A program running with idle io priority will only get disk time when no other program has asked for disk io for a defined grace period. The impact of idle io processes on normal system activity should be zero. This scheduling class does not take a priority argument. The idle class has no class data, since it doesn’t really apply here.

ionice

ionice – get/set program io scheduling class and priority. This program sets the io scheduling class and priority for a program. Since v3 (aka CFQ Time Sliced) CFQ implements I/O nice levels similar to those of CPU scheduling. These nice levels are grouped in three scheduling classes each one containing one or more priority levels:

ionice usage

If no arguments or just -p is given, ionice will query the current io scheduling class and priority for that process.

ionice [-c] [-n] [-p] [COMMAND [ARG...]]

• -c – The scheduling class. 1 for real time, 2 for best-effort, 3 for idle.
• -n – The scheduling class data. This defines the class data, if the class accepts an argument. For real time and best-effort, 0-7 is valid data.
• -p – Pass in a process pid to change an already running process. If this argument is not given, ionice will run the listed program with the given parameters.

ionice Examples

Sets process with PID 89 as an idle io process.

ionice -c3 -p89

Runs ‘bash’ as a best-effort program with highest priority.

ionice -c2 -n0 bash

Returns the class and priority of the process with PID 89

ionice -p89

With the ionice command, you can set the IO priority for a process to one of three classes: Idle (3), Best Effort (2), and Real Time (1). The Idle class means that the process will only be able to read and write to the disk when all other processes are not using the disk. The Best Effort class is the default and has eight different priority levels from 0 (top priority) to 7 (lowest priority). The Real Time class results in the process having first access to the disk irregardless of other process and should never be used unless you know what you are doing.

If we wish to run the updatedb process in the background with an Idle IO class priority, we can run the following:

$ sudo date
$ sudo updatedb &
[1] 16324
$ sudo ionice -c3 -p16324

If we’d rather just lower the Best Effort class priority (defaults to 4) for the command so the process isn’t limited to idle IO periods, we can run the following:

$ sudo date
$ sudo updatedb &
[1] 16324
$ sudo ionice -c2 -n7 -p16324

Again, the Real Time class should not be used as it can prevent you from being able to interact with your system.

You may wonder where you can get the process ID if you don’t know it, can’t remember it, or didn’t start the process (an automatted script may have launched it). You can find process IDs with the ps command.

For example, if I had an updatedb program running in the background, and I wanted to find its process ID, I can run the following:

$ ps -C updatedb
PID TTY TIME CMD
4234 ? 00:00:42 updatedb

This tells me that the process’ process ID (PID) is 4234.

iostat

iostat Usage

iostat [ -c ] [ -d ] [ -N ] [ -n ] [ -h ] [ -k | -m ] [ -t ] [ -V ] [ -x ] [ -z ] [ <device> [...] | ALL ] [ -p [ <device> [,...] | ALL ] ] [ <interval> [ <count> ] ]

-c     The -c option is exclusive of the -d option and displays only the CPU usage report.
-d     The -d option is exclusive of the -c option and displays only the device utilization report.
-k     Display statistics in kilobytes per second instead of blocks per second.  Data displayed are valid only with kernels 2.4 and newer.
-m     Display statistics in megabytes per second instead of blocks or kilobytes per second.  Data displayed are valid only with kernels 2.4 and newer.
-n     Displays the NFS-directory statistic.  Data displayed are valid only with kernels 2.6.17 and newer.  This option is exclusive ot the -x option.
-h     Display the NFS report more human readable.
-p [ { device | ALL } ]   The  -p  option  is  exclusive  of  the -x option and displays statistics for block devices and all their partitions that are used by the system.
-t     Print the time for each report displayed.
-x     Display extended statistics.

iostat Examples

iostat -p ALL 2 1000
avg-cpu:  %user   %nice    %sys %iowait   %idle
            8.34    0.08    1.26    2.27   88.05

Display a single history since boot report for all CPU and Devices.

$ iostat

Display a continuous device report at two second intervals.

$ iostat -d 2

Display six reports at two second intervals for all devices.

$ iostat -d 2 6

Display six reports of extended statistics at two second intervals for devices hda and hdb.

$ iostat -x hda hdb 2 6

Display six reports at two second intervals for device sda and all its partitions (sda1, etc.)

$ iostat -p sda 2 6

Schedule Utils

These are the Linux scheduler utilities – schedutils for short. These programs take advantage of the scheduler family of syscalls that Linux implements across various kernels. These system calls implement interfaces for scheduler-related parameters such as CPU affinity and real-time attributes. The standard UNIX utilities do not provide support for these interfaces — thus this package.

The programs that are included in this package are chrt and taskset. Together with nice and renice (not included), they allow full control of process scheduling parameters. Suggestions for related utilities are welcome, although it is believed (barring new interfaces) that all scheduling interfaces are covered.

I’ve found that quite a few servers do not have this package installed, indicating to you that they might not know what they are doing. Here is how you can install this incredible package, for non-root users. Root users know how to do this, or they shouldn’t be root. Download and install in 1 line provided you have curl. Or just use the following commands.

mkdir -pv $HOME/{dist,source,bin,share/man/man1} && cd ~/dist && curl -O http://ftp.de.debian.org/debian/pool/main/s/schedutils/schedutils_1.5.0.orig.tar.gz && cd ~/source && tar -xvzf ~/dist/sch*z && cd sch* && sed -i -e 's,= /usr/local,=${HOME},g' Makefile && make && make install && make installdoc

mkdir -pv $HOME/{dist,source,bin,share/man/man1}
cd ~/dist && curl -O http://ftp.de.debian.org/debian/pool/main/s/schedutils/schedutils_1.5.0.orig.tar.gz
cd ~/source && tar -xvzf ~/dist/schedutils_1.5.0.orig.tar.gz
cd ~/source/schedutils-1.5.0 && sed -i -e 's,= /usr/local,=${HOME},g' Makefile
make || make -d && make install || make install -d && make installdoc || make installdoc -d

taskset

Taskset is used to set or retrieve the CPU affinity of a running process given its PID or to launch a new COMMAND with a given CPU affinity. CPU affinity is a scheduler property that “bonds” a process to a given set of CPUs on the system. The Linux scheduler will honor the given CPU affinity and the process will not run on any other CPUs. Note that the Linux scheduler also supports natural CPU affinity: the scheduler attempts to keep processes on the same CPU as long as practical for performance reasons. Therefore, forcing a specific CPU affinity is useful only in certain applications.

The CPU affinity is represented as a bitmask, with the lowest order bit corresponding to the first logical CPU and the highest order bit corresponding to the last logical CPU. Not all CPUs may exist on a given system but a mask may specify more CPUs than are present. A retrieved mask will reflect only the bits that correspond to CPUs physically on the system. If an invalid mask is given (i.e., one that corresponds to no valid CPUs on the current system) an error is returned. A user must possess CAP_SYS_NICE to change the CPU affinity of a process. Any user can retrieve the affinity mask.

taskset Usage

taskset [options] [mask | cpu-list] [pid | cmd [args...]]
 
-p, --pid            operate on existing given pid
-c, --cpu-list     display and specify cpus in list format

taskset-examples

The default behavior is to run a new command:

$ taskset 03 sshd -b 1024

You can retrieve the mask of an existing task or set it:

$ taskset -p 700
$ taskset -p 03 700

List format uses a comma-separated list instead of a mask:

$ taskset -pc 0,3,7-11 700

chrt

chrt sets or retrieves the real-time scheduling attributes of an existing PID or runs COMMAND with the given attributes. Both policy (one of SCHED_FIFO, SCHED_RR, or SCHED_OTHER) and priority can be set and retrieved. A user must possess CAP_SYS_NICE to change the scheduling attributes of a process. Any user can retrieve the scheduling information.

chrt Usage

chrt [options] [prio] [pid | cmd [args...]]
 
-p, --pid operate on an existing PID and do not launch a new task
-f, --fifo set scheduling policy to SCHED_FIFO
-m, --max show minimum and maximum valid priorities, then exit
-o, --other set policy scheduling policy to SCHED_OTHER
-r, --rr set scheduling policy to SCHED_RR (the default)

chrt Examples

The default behavior is to run a new command: chrt [prio] -- [command] [arguments]

You can also retrieve the real-time attributes of an existing task:

chrt -p [pid]

Or set them:

chrt -p [prio] [pid]

ulimit – get and set user limits

Ulimit provides control over the resources available to processes started by the shell, on systems that allow such control. One can set the resource limits of the shell using the built-in ulimit command. The shell’s resource limits are inherited by the processes that it creates to execute commands.

ulimit Usage

ulimit [-SHacdfilmnpqstuvx] [limit]

-S

use the `soft’ resource limit

-H

use the `hard’ resource limit

-a

all current limits are reported

-c

the maximum size of core files created

-d

the maximum size of a process’s data segment

-f

the maximum size of files created by the shell

-l

the maximum size a process may lock into memory

-m

the maximum resident set size

-n

the maximum number of open file descriptors

-p

the pipe buffer size

-s

the maximum stack size

-t

the maximum amount of cpu time in seconds

-u

the maximum number of user processes

-v

the size of virtual memory

If LIMIT is given, it is the new value of the specified resource; the special LIMIT values `soft’, `hard’, and `unlimited’ stand for the current soft limit, the current hard limit, and no limit, respectively. Otherwise, the current value of the specified resource is printed. If no option is given, then -f is assumed. Values are in 1024-byte increments, except for -t, which is in seconds, -p, which is in increments of 512 bytes, and -u, which is an unscaled number of processes.

RLIMIT_AS

The maximum size of the process’s virtual memory (address space) in bytes. This limit affects calls to brk(2), mmap(2) and mremap(2), which fail with the error ENOMEM upon exceeding this limit. Also automatic stack expansion will fail (and generate a SIGSEGV that kills the process if no alternate stack has been made available via sigaltstack(2)). Since the value is a long, on machines with a 32-bit long either this limit is at most 2 GiB, or this resource is unlimited.

RLIMIT_CORE

Maximum size of core file. When 0 no core dump files are created. When non-zero, larger dumps are truncated to this size.

RLIMIT_CPU CPU

time limit in seconds. When the process reaches the soft limit, it is sent a SIGXCPU signal. The default action for this signal is to terminate the process. However, the signal can be caught, and the handler can return control to the main program. If the process continues to consume CPU time, it will be sent SIGXCPU once per second until the hard limit is reached, at which time it is sent SIGKILL. (This latter point describes Linux 2.2 through 2.6 behavior. Implementations vary in how they treat processes which continue to consume CPU time after reaching the soft limit. Portable applications that need to catch this signal should perform an orderly termination upon first receipt of SIGXCPU.)

RLIMIT_DATA

The maximum size of the process’s data segment (initialized data, uninitialized data, and heap). This limit affects calls to brk(2) and sbrk(2), which fail with the error ENOMEM upon encountering the soft limit of this resource.

RLIMIT_FSIZE

The maximum size of files that the process may create. Attempts to extend a file beyond this limit result in delivery of a SIGXFSZ signal. By default, this signal terminates a process, but a process can catch this signal instead, in which case the relevant system call (e.g., write(2), truncate(2)) fails with the error EFBIG.

RLIMIT_LOCKS

(Early Linux 2.4 only) A limit on the combined number of flock(2) locks and fcntl(2) leases that this process may establish.

RLIMIT_MEMLOCK

The maximum number of bytes of memory that may be locked into RAM. In effect this limit is rounded down to the nearest multiple of the system page size. This limit affects mlock(2) and mlockall(2) and the mmap(2) MAP_LOCKED operation. Since Linux 2.6.9 it also affects the shmctl(2) SHM_LOCK operation, where it sets a maximum on the total bytes in shared memory segments (see shmget(2)) that may be locked by the real user ID of the calling process. The shmctl(2) SHM_LOCK locks are accounted for separately from the per-process memory locks established by mlock(2), mlockall(2), and mmap(2) MAP_LOCKED; a process can lock bytes up to this limit in each of these two categories. In Linux kernels before 2.6.9, this limit controlled the amount of memory that could be locked by a privileged process. Since Linux 2.6.9, no limits are placed on the amount of memory that a privileged process may lock, and this limit instead governs the amount of memory that an unprivileged process may lock.

RLIMIT_MSGQUEUE

(Since Linux 2.6.8) Specifies the limit on the number of bytes that can be allocated for POSIX message queues for the real user ID of the calling process. This limit is enforced for mq_open(3). Each message queue that the user creates counts (until it is removed) against this limit according to the formula: bytes = attr.mq_maxmsg * sizeof(struct msg_msg *) + attr.mq_maxmsg * attr.mq_msgsize where attr is the mq_attr structure specified as the fourth argument to mq_open(3). The first addend in the formula, which includes sizeof(struct msg_msg *) (4 bytes on Linux/i386), ensures that the user cannot create an unlimited number of zero-length messages (such messages nevertheless each consume some system memory for bookkeeping overhead).

RLIMIT_NICE

(since Linux 2.6.12, but see BUGS below) Specifies a ceiling to which the process’s nice value can be raised using setpriority(2) or nice(2). The actual ceiling for the nice value is calculated as 20 – rlim_cur. (This strangeness occurs because negative numbers cannot be specified as resource limit values, since they typically have special meanings. For example, RLIM_INFINITY typically is the same as -1.)

RLIMIT_NOFILE

Specifies a value one greater than the maximum file descriptor number that can be opened by this process. Attempts (open(2), pipe(2), dup(2), etc.) to exceed this limit yield the error EMFILE. (Historically, this limit was named RLIMIT_OFILE on BSD.)

RLIMIT_NPROC

The maximum number of processes (or, more precisely on Linux, threads) that can be created for the real user ID of the calling process. Upon encountering this limit, fork(2) fails with the error EAGAIN.

RLIMIT_RSS

Specifies the limit (in pages) of the process’s resident set (the number of virtual pages resident in RAM). This limit only has effect in Linux 2.4.x, x < 30, and there only affects calls to madvise(2) specifying MADV_WILLNEED.

RLIMIT_RTPRIO

(Since Linux 2.6.12, but see BUGS) Specifies a ceiling on the real-time priority that may be set for this process using sched_setscheduler(2) and sched_setparam(2).

RLIMIT_RTTIME

(Since Linux 2.6.25) Specifies a limit on the amount of CPU time that a process scheduled under a real-time scheduling policy may consume without making a blocking system call. For the purpose of this limit, each time a process makes a blocking system call, the count of its consumed CPU time is reset to zero. The CPU time count is not reset if the process continues trying to use the CPU but is preempted, its time slice expires, or it calls sched_yield(2). Upon reaching the soft limit, the process is sent a SIGXCPU signal. If the process catches or ignores this signal and continues consuming CPU time, then SIGXCPU will be generated once each second until the hard limit is reached, at which point the process is sent a SIGKILL signal. The intended use of this limit is to stop a runaway real-time process from locking up the system.

RLIMIT_SIGPENDING

(Since Linux 2.6.8) Specifies the limit on the number of signals that may be queued for the real user ID of the calling process. Both standard and real-time signals are counted for the purpose of checking this limit. However, the limit is only enforced for sigqueue(2); it is always possible to use kill(2) to queue one instance of any of the signals that are not already queued to the process.

RLIMIT_STACK

The maximum size of the process stack, in bytes. Upon reaching this limit, a SIGSEGV signal is generated. To handle this signal, a process must employ an alternate signal stack (sigaltstack(2)).

ulimit Examples

Turn off core dumps

ulimit -S -c 0

More Reading

• Please see the SYSSTAT Utilities Home for more performance monitoring tools like sar, sadf, mpstat, iostat, pidstat and sa tools.
• Multitasking from the Linux Command Line + Process Prioritization

Man Pages

1. sched_setscheduler
2. cpuset
3. signal
4. getrlimit
5. ulimit
6. ioprio_get
7. ioprio_set

Kernel Documentation

• information on schedstats (Linux Scheduler Statistics)
• real-time group scheduling
• How and why the scheduler’s nice levels are implemented
• information on scheduling domains
• goals, design and implementation of the Complete Fair Scheduler

Future Discussions:

IO Benchmarking: How, Why and With What

Optimizing Servers and Processes for Speed with ionice, nice, ulimit originally appeared on AskApache.

The first article was meant as a detailed and thorough introduction to speeding up Windows-based PC’s in a way that makes it easy to follow, without getting too specific. So make sure you read that first, and pay the most attention to freeing up RAM, CPU, and Disk IO speed by reducing the number of services and processes that are running, we will deal with defragmentation, hard-drive speed, Disk IO, Prefetching, and Pagefile/Registry Defragmentation now.

This article has some really really really great stuff for ya. It shows which tools (all but one completely free) are the best and you will use them for a long time, they are all very good. That is just a side benefit, as this article is really more of a step-by-step guide to optimizing your system that won’t have to be repeated for at least a year. The result of course is a much more responsive PC.

Whats New

After writing that article I continued my research and testing into the subject on my personal computers. I wanted to test out several additional programs and methods before I wrote about them for you guys, and I found a few really sweet additions that had a very big performance gain for all my computers, from my oldest and slowest PC’s to my new 4K power laptop. This article is primarily focused on optimizing your hard drive data and improving your Disk IO speed, and you will definately see an improvement in speed. It doesn’t get REALLY good until the defragmenting section..

1. Clean Up Hard Drive – Removing unneccessary files
2. Clean Registry – Fixing slow registry problems automatically
3. Ultimate Defragmenting – The best defrag method I use
4. Optimize Physical Hard Disk – Final step that cleans and heals your physical disk

Clean Up Hard Drive

The first step is to clean up all the extra, temporary, and unneccessary files cluttering your hard-drive. The reason is because we will be defragmenting your hard-drive like its never been defragged before, then we are going to go over every single bit and byte of your hard-drive to optimize the physical sectors and storage of your data.. Also we will be running a check of your registry and cleaning out bad links and other slow errors, so get it as clutter-free as possible. DON’T use Windows built-in folder compression, it makes defragmentation worse… DO use 7-zip or winrar to create a solid archive file of any misc directories with a bunch of files… You only need one program to clean your system.

CCleaner

CCleaner is a Small, Fast and Free software that removes unused and temporary files from your system and allows Windows to run faster, more efficiently and gives you more hard disk space. I’ve now been using it for several months and love it. As well as cleaning up old files and settings left by standard Windows components, CCleaner also cleans temporary files and recent file lists for many applications. Including: Firefox, Opera, Safari, Media Player, eMule, Kazaa, Google Toolbar, Netscape, Microsoft Office, Nero, Adobe Acrobat Reader, WinRAR, WinAce, WinZip and more… Google Chrome, Opera, Safari, etc..

• Recycle Bin, Clipboard
• Windows Temporary files, Windows Log files, Chkdsk file fragments
• Recent Documents (on the Start Menu), Run history (on the Start Menu)
• Windows XP Search Assistant history, old Prefetch data, Windows memory dumps after crashes

Download CCleaner

Registry Cleaning

The registry is Windows biggest mistake, (although I’m sure they like it), and basically holds all the information for your programs and Windows. Things like the size of your windows, recent file lists, icon files for different icons, etc.. I’ve never seen a computer that didn’t have some registry issues, so this needs to be cleaned and may have a huge impact on your speed. Real quickly, here are some programs to backup and restore your registry, optimize and defrag your registry, and finally search and clean any errors in your registry.

CCleaner

Yup! CCleaner also takes care of most of the performance issues of your registry. It’s very safe and fast. CCleaner uses an advanced Registry Cleaner to check for problems and inconsistencies. It checks the following:

• ClassIDs, ProgIDs, Application Paths, Icons
• Uninstallers, Shared DLLs, Fonts, Help File references
• File Extensions, ActiveX Controls, Invalid Shortcuts and more…

Download CCleaner

ERUNT – The Emergency Recovery Utility NT

ERUNT is a Registry Backup and Restore utility for Windows NT/2000/2003/XP. I use this to backup my registry automatically or on command.. Ive used it for years and it’s always good to backup before you do anything.

# Here's the command I use (only for advanced users familiar with autoback)
"%ProgramFiles%\ERUNT\AUTOBACK.EXE" %SystemRoot%\ERDNT\#Date# sysreg curuser otherusers /noconfirmdelete /noprogresswindow /days:45 /alwayscreate

Download ERUNT · (Details)

NTGREGOPT – NT Registry Optimizer

NTGREGOPT is a Registry Optimization tool for Windows NT/2000/2003/XP/Vista that minimizes the size of your registry files by simply compacting the registry hives to the minimum size possible.

Registry files in an NT-based system can become fragmented over time, occupying more space on your hard disk than necessary and decreasing overall performance. You should use the NTREGOPT utility regularly, but especially after installing or uninstalling a program, to minimize the size of the registry files and optimize registry access. The program works by recreating each registry hive “from scratch”, thus removing any slack space that may be left from previously modified or deleted keys. It does NOT change the contents of the registry in any way, nor does it physically defrag the registry files on the drive. I recommend using this once every couple weeks. I scheduled it to run automatically.

Download NTGREGOPT · (Details)

Ultimate Defragmenting

I say “Ultimate Defragmenting” because this is the result of a lot of testing of all the various defragmenting software out there, reading a lot of documentation, and running benchmarking to find the fastest results. This is a mix of several individual defragmenting steps combined for a once-a-year ultimate defragmenting session. This is what I use today, and although it’s altogether a long process, each step you’ll add a new tool or skill that you can use by itself from here on out.

ATTENTION: While running MyDefrag/JkDefrag, SpinRite, and UltraDefrag your computer can get very hot and that is very not cool. I set my laptop on a coke can and pointed a small desk fan at it which kept it very very cool, so do what you can to minimize heat during these programs.

PageDefrag

PageDefrag uses advanced techniques to provide you what commercial defragmenters cannot: the ability for you to see how fragmented your paging files and Registry hives are, and to defragment them. In addition, it defragments event log files and Windows 2000/XP hibernation files (where system memory is saved when you hibernate a laptop). One of the limitations of the Windows NT/2000 defragmentation interface is that it is not possible to defragment files that are open for exclusive access. Thus, standard defragmentation programs can neither show you how fragmented your paging files or Registry hives are, nor defragment them. Paging and Registry file fragmentation can be one of the leading causes of performance degradation related to file fragmentation in a system.

I personally keep this enabled for every boot, as it only takes a few seconds after the first time it’s run.

Download PageDefrag

MyDefrag

JkDefrag is a disk defragmenter and optimizer for Windows 2000/2003/XP/Vista/2008/X64. Completely automatic and very easy to use, fast, low overhead, with several optimization strategies, and can handle floppies, USB disks, memory sticks, and anything else that looks like a disk to Windows. Included are a Windows version, a commandline version (for scheduling by the task scheduler or for use from administrator scripts), a screensaver version, a DLL library (for use from programming languages), versions for Windows X64, and the complete sources. (Frequently Asked Questions)

After trying out dozens of degragmenting programs, this is my favorite. I utilize the cool screensaver function and just run sometimes when I’m calling it a day.

Tips and tricks

• Many users start looking for defragmentation/optimization programs when their computer becomes slow. The main reason for a slow computer is a full harddisk. A full harddisk is slow because the distance between files is greater than on a fresh practically empty harddisk. Deleting half the data on a full disk will just about double the speed. The more free diskspace, the faster your computer will be.
• Buy a second harddisk (for example an USB harddisk) and move little used stuff from your primary harddisk to that secondary harddisk. The second disk can also be used for backing up the primary disk.
• When buying a new computer, buy the biggest harddisk you can afford. Investing in a bigger harddisk gives more speed-per-dollar than investing in a faster CPU or investing in more memory.
• Cleanup old junk from your harddisk before running MyDefrag. You can clean Windows files with for example “Start -> Programs -> Accessories -> System Tools -> Disk Cleanup”, or with something like the freeware CCleaner program.
• Reboot before running MyDefrag. This will release files that are in use, so they can be defragmented and optimized.
• Boot into Windows safe mode by pressing F8 when booting, and then run MyDefrag. It will be slower because the Windows disk cache is off in safe mode, but MyDefrag will be able to process (a few) more files.
• Stop your real time virus scanner before running MyDefrag. Virus scanners check all disk activity, making defragmentation and optimization very slow.
• Move the swap file to another volume, reboot, defragment, and move the swap file back. If you don’t have a second volume then temporarily make the swap file small, for example 100Mb.
• Package unused files with a packager such as 7-zip. The packagefile not only takes less harddisk space, but will also defragment and optimize much faster than the individual files. Note: This does not apply to Windows NTFS compression, which will actually make defragmentation and optimization slower.
• The first partition on a harddisk is significantly faster than other partitions. Try to use other partitions only for data that is used less often, such as music, movies, archives, backups, logfiles.
• If you have 2 physical harddisks of the same speed, then place the pagefile on the first partition of the second harddisk.

The way I recommend is to run MyDefrag at the highest level of defragmentation once, which took my fastest PC almost 30 hours. Once that is done you can just run it normally in 20 minutes or so.. This software also has the best defrag information I’ve found to date, so check out the documentation on the site.

Download MyDefrag

UltraDefrag

UltraDefrag is a powerful Open Source Defragmentation tool for Windows Platform. It is very fast, because the defragmenting is done via the kernel-mode driver. There are three interface available : Graphical, Console and Native. I personally like the MyDefrag more because I think it does a better job, but I also use the UltraDefrag tool because it has one very important feature like the PageDefrag tool. It has a native version. That means it can run before Windows loads up by utilizing the bootexecute, the same place that windows chkdsk runs at boot. It also can takeover for Windows builtin prefetcher, to speed up the loading of frequently used programs, which I’ll explain a bit later.

I set ultradefrag up after the MyDefrag 30hour defrag completes, to run at boot and control the prefetching, then I erase any prefetch files currently saved and reboot which lets it defrag the system.

erase /Q "%SYSTEMROOT%\Prefetch\*.*"

Once both MyDefrag and UltraDefrag have run THEN I finally login to windows and don’t open any programs to let the windows OS files get optimized by not doing anything at all for 5 minutes.. Then I reboot and log back in and this time I don’t do anything for 30 minutes and reboot. Finally I log back in and this time I instantly load up 10 of my most frequently used programs, (DreamWeaver, Photoshop, Firefox, Chrome, Notepad2, Thunderbird, Internet Explorer, and a few others) and once they are all loaded I don’t do anything for an hour. Then I reboot and repeat that same process.

This may seem odd or made up but I do my research and this allows your prefetched files to be optimized, including your boot prefetch files. Once that is done I reboot and run all the defrags again. Then I reboot and am ready for the last step.

Download UltraDefrag

Optimize Physical Hard Disk

Now at this point the system is defragged and optimized as much as I can get it, but the last step is to run a program to go over every single bit on our hard-drive-disk to keep the drive clean and healthy and its too technical for me to understand, I just know its amazingly cool and I noticed a big change right away.

HD Tune

HD Tune is a fantastic little utility that you can use to benchmark the DISK IO speed of your various drives, internal and external, fixed and USB, firewire, etc.. Other than using it to determine your fastest drives for moving your program files and temps to, I am just including it in this article because it is an awesome program that you will love.

Download HD Tune

SpinRite

This is the last step in this guide, and was the one thing that surprised me the most in terms of how much of a speed improvement I noticed after using it. SpinRite is the most capable, thorough, and reliable utility that has ever been created for the long term maintenance, recovery, and repair of mass storage systems. SpinRite is not a drive defragmenter. SpinRite operates with the drive’s built-in intelligence to reassign and relocate defective sectors without creating file system fragments. Thus, running SpinRite does not create fragments, but neither does it eliminate any that may exist before it was run. Unlike any other disk utility, SpinRite interfaces directly to the hard disk system’s hardware, rather than working through the system’s operating system or BIOS. FAQ.

The way that we use SpinRite in this article is method #4, Drive Maintenance mode, which reads and writes and verifies every single sector and area of your hard drives, and improves the health of your hard drive a lot. Even on my new 4K dell power laptop, this had a noticeable improvement on speed. After running this 20+ hours for my fastest PC, I rebooted, defragged with jkdefrag, and that is the end of this article.

In case you don’t already know . . . What is SpinRite?

SpinRite is a stand-alone DOS application that specializes in the recovery of marginally or completely unreadable hard and floppy disk data, and in the lifetime maintenance of PC mass storage devices. It earned its stripes many years ago by introducing the concept of non-destructive low-level reformatting and sector interleave optimization. Since then its capabilities have continued to broaden until it has become the premiere tool for disk data recovery and magnetic mass storage drive maintenance. Written in assembly language, SpinRite still performs as well on a clunky old 4.77 megahertz PC/XT as on a screaming 333 megahertz Pentium II.

While SpinRite 6.0 is running, you can toggle through seven displays:

Purchase and Download SpinRite

More Reading

• PCNet File Catch – SpinRite 6.0
• SpinRite 6.0 for Linux Users
• Anticipate Drive Problems Early with SpinRite v6.0
• Black Viper – Windows XP Super Tweaks
• Black Viper – Optimize your Services
• Advanced Windows Shell Tutorial
• Batch Files (Scripts) in Windows
• File System Heirarchy
• Steve Gibson discussing Defragmenting

Windows Optimization – Intense Part II originally appeared on AskApache.

Note: I spent no time on readability, this is primarily a read the code and figure it out article.. This is for advanced users looking for a reference or discussion and for those of you looking to advance. Feedback would be great if you make it that far..

For a better handle on the way I like to structure web site directories, see Optimize a Website for Speed, Security, and Easy Management but note it is a bit outdated compared to what I’m doing now. I don’t have the luxury of using only one type of server, or hosting provider anymore, so I have been working towards making things even more portable in order to move from host to host from server to server without issues i.e. my portable .bash_profile.

So I’ve been basically experimenting various ways to accomplish that and thought I would share what I am currently doing for my benefit and hopefully get some input. All of my WP installs run the development version, and one main idea with my setups is that upgrading is automated. So I really keep the WordPress install clean and use plugins and wp-config.php to do all the customization.

• Portability – Hands-free upgrades and easy to move
• Security – Additional security and protection
• Speed – Less CPU and Disk I/O
• Customization – All my favorite customizations

wp-config.php

These are the main settings I use.. Seriously this is more like an interactive article, because to understand it you will need to do some code grepping. You may want to grab a jolt.

ASKAPACHE_ROOT

The ASKAPACHE_ROOT variable is just a better way for me to be able to include and access all the different files in my site tree. For instance, in my non-wp php files, I can do this:

!defined('ASKAPACHE_ROOT') && require $_SERVER['DOCUMENT_ROOT'] . '/wp-config.php';
include(ASKAPACHE_ROOT . '/includes/custom-download.inc.php');

ASKAPACHE_LOCK

This is one of my all-time favorite hacks, that I think is one of the most useful methods I employ as a web developer. This allows me to use far-future-expire headers for optimum caching, while still forcing browsers to re-validate every day or so automatically, or forcing them to re-validate whenever I change the suffix. This takes advantage of the mod_rewrite trick that I use on EVERY site I run, definately worth learning. Because I practice best-practice web-standards, for every web site I create a single css file and javascript file, which I then add to the template like:

<link rel="stylesheet" type="text/css" media="all" href="http://z.askapache.com/c/apache-0<?php echo ASKAPACHE_LOCK?>.css" />
<script src="http://z.askapache.com/j/apache-0<?php echo ASKAPACHE_LOCK;?>.js" type="text/javascript"></script>

<?php
/**
 * The base configurations of the WordPress.
 *
 * This file has the following configurations: MySQL settings, Table Prefix,
 * Secret Keys, WordPress Language, and ABSPATH. You can find more information by
 * visiting {@link http://codex.wordpress.org/Editing_wp-config.php Editing
 * wp-config.php} Codex page. You can get the MySQL settings from your web host.
 *
 * This file is used by the wp-config.php creation script during the
 * installation. You don't have to use the web site, you can just copy this file
 * to "wp-config.php" and fill in the values.
 *
 * @package WordPress
 */
/* http://codex.wordpress.org/Editing_wp-config.php */
 
/** /home/liet/askapache.com */
!defined('ASKAPACHE_ROOT') && define('ASKAPACHE_ROOT', str_replace('/htdocs','', $_SERVER['DOCUMENT_ROOT']));
 
/** The 008 at the end is for manual tweaking.  time() returns seconds since '00:00:00 1970-01-01 UTC'. */
// http://www.askapache.com/htaccess/mod_rewrite-fix-for-caching-updated-files.html
!defined('ASKAPACHE_LOCK') && define(ASKAPACHE_LOCK', substr(time(),0,5).'008'); // 12533001
 
/** absolute path to the Wordpress directory */
!defined('ABSPATH') && define('ABSPATH', ASKAPACHE_ROOT .'/htdocs/');
 
/**
 * WP_SITEURL, defined since WordPress Version 2.2, allows the WordPress address (URL) to be defined. The valued defined is the address where your WordPress core files reside.
 * It should include the http:// part too. Do not put a slash "/" at the end.
 * Setting this value in wp-config.php overrides the wp_options table value for siteurl and disables the WordPress address (URL) field in the Administration > Settings > General panel.
 */
!defined('WP_SITEURL') && define('WP_SITEURL', 'http://'.$_SERVER['SERVER_NAME']);
 
/**
 * WP_HOME is another wp-config.php option added in WordPress Version 2.2. Similar to WP_SITEURL,
 * WP_HOME overrides the wp_options table value for home but does not change it permanently.
 * home is the address you want people to type in their browser to reach your WordPress blog. It should include the http:// part. Also, do not put a slash "/" at the end.
 */
!defined('WP_HOME') && define('WP_HOME', WP_SITEURL);
 
/** no trailing slash, full paths only */
!defined('WP_CONTENT_DIR') && define( 'WP_CONTENT_DIR', ABSPATH . 'wp-content' );
 
// full url - WP_CONTENT_DIR is defined further up
!defined('WP_CONTENT_URL') && define( 'WP_CONTENT_URL', WP_SITEURL . '/wp-content');
 
/** Allows for the plugins directory to be moved from the default location. @since 2.6.0 */
// full path, no trailing slash
!defined('WP_PLUGIN_DIR') && define( 'WP_PLUGIN_DIR', WP_CONTENT_DIR . '/plugins' );
 
/** Allows for the plugins directory to be moved from the default location. @since 2.6.0 */
// full url, no trailing slash
!defined('WP_PLUGIN_URL') && define( 'WP_PLUGIN_URL', WP_CONTENT_URL . '/plugins' );
 
/** Allows for the plugins directory to be moved from the default location. @since 2.1.0 */
// Relative to ABSPATH.  For back compat.
//!defined('PLUGINDIR') && define( 'PLUGINDIR', 'wp-content/plugins' );
 
/** Number of autosaves to save. TRUE is default and enables post revisions, FALSE disables revisions completely. */
!defined('WP_POST_REVISIONS') && define('WP_POST_REVISIONS', 150);
 
/* ini_set('memory_limit', WP_MEMORY_LIMIT); */
!defined('WP_MEMORY_LIMIT') && define('WP_MEMORY_LIMIT', '64M');
 
/** Only check at this interval for new messages. Default is 5min */
/** @since 2.9  */
!defined('WP_MAIL_INTERVAL') && define('WP_MAIL_INTERVAL', 3600); // 1 hour
 
/** Saves updated post values to post from edit window every x seconds. (default 60)
 * When editing a post, WordPress uses Ajax to auto-save revisions to the post as you edit. You may want to increase this setting for longer delays in between auto-saves, or decrease the setting to make sure you never lose changes.
 * @since 2.5.0 */
!defined( 'AUTOSAVE_INTERVAL' ) && define( 'AUTOSAVE_INTERVAL', 60 );
 
/** @since 2.9.0  */
/** Permanently deletes posts, pages, attachments, and comments which have been in the trash for EMPTY_TRASH_DAYS. */
!defined( 'EMPTY_TRASH_DAYS' ) && define( 'EMPTY_TRASH_DAYS', 300 );

Debugging WordPress

One of my secrets for getting really good at this stuff is to master debugging. There is really not ever a time when I am working on a site that I don’t have color-highlighted logs scrolling automatically in an ssh window. It’s really almost impossible to fix problems with wordpress or do any kind of advanced anything without being able to view debugging info. At first I relied heavily on a custom php.ini being available on the server, but after having to deal with many hosts who don’t allow php.ini files I now rely completely on setting values using ini_set for ultimate portability. Detailed towards the end of this article and is also included in this wp-config.php

/**#@+
 * DEBUGGING STUFF
 */
/** display of notices during development. if false, error_reporting is E_ERROR | E_WARNING | E_PARSE | E_USER_ERROR | E_USER_WARNING | E_RECOVERABLE_ERROR otherwise E_ALL */
!defined('WP_DEBUG') && define('WP_DEBUG', false);
 
/** The SAVEQUERIES definition saves the database queries to a array and that array can be displayed to help analyze those queries.
 *  The information saves each query, what function called it, and how long that query took to execute.  */
!defined('SAVE_QUERIES') && define('SAVE_QUERIES', WP_DEBUG);
 
!defined('ACTION_DEBUG') && define('ACTION_DEBUG', WP_DEBUG);
 
/** This will allow you to edit the scriptname.dev.js files in the wp-includes/js and wp-admin/js directories.  */
!defined('SCRIPT_DEBUG') && define('SCRIPT_DEBUG', WP_DEBUG);

 
/** Add define('WP_DEBUG_LOG', true); to enable php debug logging to WP_CONTENT_DIR/debug.log */
//!defined('WP_DEBUG_LOG') && define('WP_DEBUG_LOG', true);
 
/** This determines whether errors should be printed to the screen as part of the output or if they should be hidden from the user.
 *  Add define('WP_DEBUG_DISPLAY', false); to wp-config.php to use the globally configured setting for display_errors and not force it to On */
!defined('WP_DEBUG_DISPLAY') && define('WP_DEBUG_DISPLAY', false);

Ultimate Security Tweaks

Well, ultimate for WP’s built-in keys and password functions, this is all for wp-config.php keep in mind. This is a very neccessary and recommended step, and is one of the only things I modify for each new installation.

Security KEYS

If like me you are familiar with password-cracking software like John the ripper, rainbow hash tables, l0pht-crack, etc.. then you will like to know that you can specify your own keys and salts for the encryption used by WP. They are AUTH_KEY, AUTH_SALT, SECURE_AUTH_KEY, SECURE_AUTH_SALT, LOGGED_IN_KEY, LOGGED_IN_SALT, NONCE_KEY, NONCE_SALT, SECRET_KEY and SECRET_SALT.

A random and long key gives you better encryption, and exponentially increasing that is using a random and long salt for the encryption. Encryptions with known salts are incredibly easy to decrypt compared to encryptions with secure salts, because the salt + key individually need to be guessed in order to find a matching hash, vs. just the key if the salt is known. See: Locating weak passwords.

A secret key is a hashing salt which makes your site harder to hack and access harder to crack by adding random elements to the password.

In simple terms, a secret key is a password with elements that make it harder to generate enough options to break through your security barriers. A password like “password” or “test” is simple and easily broken. A random, unpredictable password such as “88a7da62429ba6ad3cb3c76a09641fc” takes years to come up with the right combination.

For more information on the technical background and breakdown of secret keys and secure passwords, see:

• WordPress Support Forum – HOWTO: Set up secret keys in WordPress 2.6+
• Wikipedia’s explanation of Password Cracking

I like to use the WordPress.org secret-key service 4 times. That’s because for each key and salt I like to do: (1 key from api +random keyboard input+1 key from api).

/**#@+
 * Authentication Unique Keys.
 *
 * Change these to different unique phrases!
 * You can generate these using the {@link https://api.wordpress.org/secret-key/1.1/ WordPress.org secret-key service}
 * You can change these at any point in time to invalidate all existing cookies.
 * This will force all users to have to log in again.
 *
 * @since 2.6.0
 *
 * Get salt to add to hashes to help prevent attacks.
 *
 * The secret key is located in two places: the database in case the secret key
 * isn't defined in the second place, which is in the wp-config.php file. If you
 * are going to set the secret key, then you must do so in the wp-config.php
 * file.
 *
 * The secret key in the database is randomly generated and will be appended to
 * the secret key that is in wp-config.php file in some instances. It is
 * important to have the secret key defined or changed in wp-config.php.
 *
 * If you have installed WordPress 2.5 or later, then you will have the
 * SECRET_KEY defined in the wp-config.php already. You will want to change the
 * value in it because hackers will know what it is. If you have upgraded to
 * WordPress 2.5 or later version from a version before WordPress 2.5, then you
 * should add the constant to your wp-config.php file.
 *
 * Below is an example of how the SECRET_KEY constant is defined with a value.
 * You must not copy the below example and paste into your wp-config.php. If you
 * need an example, then you can have a
 * {@link https://api.wordpress.org/secret-key/1.1/ secret key created} for you.
 *
 * Salting passwords helps against tools which has stored hashed values of
 * common dictionary strings. The added values makes it harder to crack if given
 * salt string is not weak.
 *
 * @since 2.5
 * @link https://api.wordpress.org/secret-key/1.1/ Create a Secret Key for wp-config.php
 *
 * @return string Salt value from either 'SECRET_KEY' or 'secret' option
 */
define('AUTH_KEY',        'jflkhaskljdfhkljasdhflkjashd;flkjhas;djfh;kajshdflkjashdlfkjhasdlkfhal?p[B+GR{@>{Yq`c|LnG;dvq#| %OA_cbBSU6,rICC1o/c)-|');
define('SECURE_AUTH_KEY', 'jflkhaskljdfhkljasdhflkjashd;flkjhas;djfh;kajshdflkjashdlfkjhasdlkfhal?Vp[Bb15baar8&R-r<[T|?(xhJJABGq+Ux+U$)-Hltp/');
define('LOGGED_IN_KEY',   'jflkhaskljdfhkljasdhflkjashd;flkjhas;djfh;kajshdflkjashdlfkjhasdlkfhal?Vp[B<5n6DG|YWnJ9tY2!M1L)`{-$LW~~Ia%.uCbn!P. 41o2$Z$4');
define('NONCE_KEY',       'jflkhaskljdfhkljasdhflkjashd;flkjhas;djfh;kajshdflkjashdlfkjhasdlkfhal?Vp[Bgu<wM*zewR0.{+m:bmrB?wj!B,4]Wo+4 Avk ApR-D?E');
define('SECRET_KEY',     'jflkhaskljdfhkljasdhflkjashd;flkjhas;djfh;kajshdflkjashdlfkjhasdlkfhal?Vp[B52ugH6muE9r4._iZwoYKUybrqLPpv|d Xr+|yrqhUE');
 
define('AUTH_SALT',        '123423190847olqkfhladhfsldshafasdfasdf09a7f-90a87df98adfyapoiyaf9asd8f70a9s8d7f908a7sdf97W4qCdm~Ky%+%~PPa5b YEmDI%U[W!-B');
define('SECURE_AUTH_SALT', '123423190847olqkfhladhfsldshafasdfasdf09a7f-90a87df98adfyapoiyaf9asd8f70a9s8d7f908a7sdf97W4qCdmad/7o6.AU3%9o-|Kqm]+eUqr-n~:ag');
define('LOGGED_IN_SALT',   '123423190847olqkfhladhfsldshafasdfasdf09a7f-90a87df98adfyapoiyaf9asd8f70a9s8d7f908a7sdf97W4qCdmsLiCv@KJ{#wd(?qe(KcH3!');
define('NONCE_SALT',       '123423190847olqkfhladhfsldshafasdfasdf09a7f-90a87df98adfyapoiyaf9asd8f70a9s8d7f908a7sdf97W4qCdmG9>+wm 2)bS0Pd_+1rx0brX]ND8|');
define('SECRET_SALT',      '123423190847olqkfhladhfsldshafasdfasdf09a7f-90a87df98adfyapoiyaf9asd8f70a9s8d7f908a7sdf97W4qCdm2<>))U|sty)+4vpWooKls/^[vN');
/**#@-*/

Using SSL for Admin and Login

SSL is kinda required from my point of view, it is just way to easy to sniff data off the wire otherwise. At least with SSL you force them to use tools like burpsuite, paros proxy, webscarab, etc..

/** @since 2.6.0  */
!defined('FORCE_SSL_ADMIN') && define('FORCE_SSL_ADMIN', true);
 
/** @since 2.6.0  */
!defined('FORCE_SSL_LOGIN') && define('FORCE_SSL_LOGIN', true);

Mod_Rewrite to Force SSL

This is pretty cool, it forces non-https for all urls except for /wp-admin and wp-login.php, which both require https. It also checks for the logged_in_cookie, and if that is present in the request then it doesn't force non-https. Kinda confusing if you don't have a mod_rewrite cheatsheet.

RewriteCond %{THE_REQUEST} ^$ [OR]
RewriteCond %{REQUEST_URI} ^/(wp-admin|wp-login\.php).*$ [NC,OR]
RewriteCond %{HTTP_COOKIE} ^.*wp_li_sadfsdfasdf11b361cdsdfasdfasd=.*$ [NC]
RewriteRule .* - [S=1]
 
RewriteCond %{HTTPS} =on [OR]
RewriteCond %{HTTP_HOST} !^www\.askapache\.com$ [NC]
RewriteRule .* http://%{SERVER_NAME}%{REQUEST_URI} [R=301,L]
 
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /(wp-admin/.*|wp-login\.php.*)\ HTTP/ [NC]
RewriteCond %{HTTPS} !=on
RewriteRule .* https://%{SERVER_NAME}%{REQUEST_URI} [R=301,L]

File System Permissions

You can get a basic and solid intro on file permissions by reading: Changing File Permissions, or you can check out some of my file permission research.

/** The permissions as octal number, usually 0644 for files, 0755 for dirs.
 *  http://codex.wordpress.org/Changing_File_Permissions
 *  if ( !$wp_filesystem->mkdir($remote_destination, FS_CHMOD_DIR) )
 */
!defined('FS_CHMOD_DIR') && define('FS_CHMOD_DIR', (0755 & ~ umask()));
!defined('FS_CHMOD_FILE') && define('FS_CHMOD_FILE', (0644 & ~ umask()));
/**#@-*/
 
/** Define the timeouts for the connections. Only available after the construct is called to allow for per-transport overriding of the default. */
//stream_set_timeout( $stream, FS_TIMEOUT );
//!defined('FS_TIMEOUT') && define('FS_TIMEOUT', 30);
 
//$this->link = @ftp_connect($this->options['hostname'], $this->options['port'], FS_CONNECT_TIMEOUT);
//!defined('FS_CONNECT_TIMEOUT') && define('FS_CONNECT_TIMEOUT', 30);
 
// function get_filesystem_method($args = array(), $context = false) {
//  $method = defined('FS_METHOD') ? FS_METHOD : false; //Please ensure that this is either 'direct', 'ssh', 'ftpext' or 'ftpsockets'
//!defined('FS_METHOD') && define('FS_METHOD', 'direct');
 
/** These methods for the WordPress core, plugin, and theme upgrades try to determine the WordPress path, as reported by PHP, but symlink trickery can sometimes
 * 'muck this up' so if you know the paths to the various folders on the server, as seen via your FTP user, you can manually define them in the wp-config.php file.
 * FS_METHOD forces the filesystem method. It should only be "direct", "ssh", "ftpext", or "ftpsockets".
 * FTP_BASE is the full path to the "base" folder of the WordPress installation.
 * FTP_CONTENT_DIR is the full path to the wp-content folder of the WordPress installation.
 * FTP_PLUGIN_DIR is the full path to the plugins folder of the WordPress installation.
 * FTP_PUBKEY is the full path to your SSH public key.
 * FTP_PRIKEY is the full path to your SSH private key.
 * FTP_USER is either user FTP or SSH username. Most likely these are the same, but use the appropriate one for the type of update you wish to do.
 * FTP_PASS is the password for the username entered for FTP_USER. If you are using SSH public key authentication this can be omitted.
 * FTP_HOST is the hostname:port combination for your SSH/FTP server. The standard FTP port is 21 and the standard SSH port is 22.
 */
//define('FS_METHOD', 'ftpext');
//define('FTP_BASE', '/path/to/wordpress/');
//define('FTP_CONTENT_DIR', '/path/to/wordpress/wp-content/');
//define('FTP_PLUGIN_DIR ', '/path/to/wordpress/wp-content/plugins/');
//define('FTP_PUBKEY', '/home/username/.ssh/id_rsa.pub');
//define('FTP_PRIKEY', '/home/username/.ssh/id_rsa');
//define('FTP_USER', 'username');
//define('FTP_PASS', 'password');
//define('FTP_HOST', 'ftp.example.org:21');
 
/**
 * Block requests through the proxy.
 *
 * Those who are behind a proxy and want to prevent access to certain hosts may do so. This will
 * prevent plugins from working and core functionality, if you don't include api.wordpress.org.
 *
 * You block external URL requests by defining WP_HTTP_BLOCK_EXTERNAL in your wp-config.php file
 * and this will only allow localhost and your blog to make requests.
 * The constant WP_ACCESSIBLE_HOSTS will allow additional hosts to go through for requests. The format of the
 * WP_ACCESSIBLE_HOSTS constant is a comma separated list of hostnames to allow.
 *
 * @since 2.8.0
 * @link http://core.trac.wordpress.org/ticket/8927 Allow preventing external requests.
/** @since 2.9  */
//!defined('WP_HTTP_BLOCK_EXTERNAL') && define( 'WP_HTTP_BLOCK_EXTERNAL', false );
 
/*
 * The constant WP_ACCESSIBLE_HOSTS will allow additional hosts to go through for requests. The format of the
 * WP_ACCESSIBLE_HOSTS constant is a comma separated list of hostnames to allow.
 *
 * @since 2.8.0
 * @link http://core.trac.wordpress.org/ticket/8927 Allow preventing external requests.
 * $accessible_hosts = preg_split('|,\s*|', WP_ACCESSIBLE_HOSTS);
 * return !in_array( $check['host'], $accessible_hosts ); //Inverse logic, If its in the array, then we can't access it.
 */
//!defined('WP_ACCESSIBLE_HOSTS') && define( 'WP_ACCESSIBLE_HOSTS', 'askapache.com,askapache.org' );

Cookies!

There’s always a little comfort in having non-default cookies for security (against auto-bots), and using shorter names also means smaller HTTP Packets.

The $cookie_hash is my hack to get around the fact that COOKIEHASH isn’t definable in wp-config.

/**#@+
 * COOKIES
 * Used to guarantee unique hash cookies @since 1.5 */
$cookie_hash=md5(WP_SITEURL);
 
/** Set a cookie now to see if they are supported by the browser.
 * setcookie(TEST_COOKIE, 'WP Cookie check', 0, COOKIEPATH, COOKIE_DOMAIN);
 * @since 2.3.0 */
!defined('TEST_COOKIE') && define('TEST_COOKIE', 'wp_tc');
 
/* @since 2.6.0 */
!defined('LOGGED_IN_COOKIE') && define('LOGGED_IN_COOKIE', 'wp_li_' . $cookie_hash);
 
/* @since 2.6.0 */
!defined('SECURE_AUTH_COOKIE') && define('SECURE_AUTH_COOKIE', 'wp_sa_' . $cookie_hash);
 
/* @since 2.5.0 */
!defined('AUTH_COOKIE') && define('AUTH_COOKIE', 'wp_a_' . $cookie_hash);
 
/* @since 2.0.0 */
!defined('PASS_COOKIE') && define('PASS_COOKIE', 'wp_p_' . $cookie_hash);
 
/* @since 2.0.0 */
!defined('USER_COOKIE') && define('USER_COOKIE', 'wp_u_' . $cookie_hash);
 
/* ok unset this var, its not needed as COOKIEHASH will have this value, but is not definable in wp-config.php */
unset($cookie_hash);
 
/** @since 1.2.0 */
!defined('COOKIEPATH') && define('COOKIEPATH', preg_replace('|https?://[^/]+|i', '', WP_HOME . '/' ) );
 
/** @since 1.5.0 */
!defined('SITECOOKIEPATH') && define('SITECOOKIEPATH', preg_replace('|https?://[^/]+|i', '', WP_SITEURL . '/' ) );
 
/** @since 2.6.0 */
!defined('ADMIN_COOKIE_PATH') && define( 'ADMIN_COOKIE_PATH', SITECOOKIEPATH . 'wp-admin' );
 
/** @since 2.6.0 */
!defined('PLUGINS_COOKIE_PATH') && define( 'PLUGINS_COOKIE_PATH', preg_replace('|https?://[^/]+|i', '', WP_PLUGIN_URL)  );
 
/** @since 2.0.0 */
!defined('COOKIE_DOMAIN') && define('COOKIE_DOMAIN', $_SERVER['SERVER_NAME']);

/**
  * The WP_CACHE setting, if true, includes the wp-content/advanced-cache.php script, when executing wp-settings.php.
  * For an advanced caching plugin to use, static because you would only want one
  * if ( defined('WP_CACHE') )@include WP_CONTENT_DIR . '/advanced-cache.php';
  */
!defined('WP_CACHE') && define('WP_CACHE', true);
 
/** WordPress Localized Language, defaults to en_US.
 *
 * Change this to localize WordPress.  A corresponding MO file for the chosen
 * language must be installed to wp-content/languages. For example, install
 * de.mo to wp-content/languages and set WPLANG to 'de' to enable German
 * language support. */
!defined('WPLANG') && define ('WPLANG', 'en_US');
 
/** Stores the location of the language directory. First looks for language folder in WP_CONTENT_DIR
 *   and uses that folder if it exists. Or it uses the "languages" folder in WPINC. @since 2.1.0 */
//!defined('WP_LANG_DIR') && define('WP_LANG_DIR', ABSPATH . WPINC . '/languages');
 
/** LANGDIR defines what directory the WPLANG .mo file resides. If LANGDIR is not defined WordPress looks first to wp-content/languages and then wp-includes/languages for the .mo defined by WPLANG file.  Old static relative path maintained for limited backwards compatibility - won't work in some cases*/
//!defined('LANGDIR') && define('LANGDIR', 'wp-content/languages');
 
/** Stores the location of the WordPress directory of functions, classes, and core content. @since 1.0.0 */
//!defined('WPINC') && define('WPINC', 'wp-includes');

WPMU Stuff

I personally don’t use.

/** Allows for the mu-plugins directory to be moved from the default location. @since 2.8.0 */
//!defined('WPMU_PLUGIN_DIR') && define( 'WPMU_PLUGIN_DIR', WP_CONTENT_DIR . '/mu-plugins' ); // full path, no trailing slash
 
/** Allows for the mu-plugins directory to be moved from the default location. @since 2.8.0 */
//!defined('WPMU_PLUGIN_URL') && define( 'WPMU_PLUGIN_URL', WP_CONTENT_URL . '/mu-plugins' ); // full url, no trailing slash
 
/** Allows for the mu-plugins directory to be moved from the default location. @since 2.8.0 */
//!defined( 'MUPLUGINDIR' ) && define( 'MUPLUGINDIR', 'wp-content/mu-plugins' ); // Relative to ABSPATH.  For back compat.

WordPress Database

This is usually the only thing I have to manually edit when creating a new site, unless I just use the same DB and modify the $table_prefix, (farther down). I run everything I possibly can in UTF-8, but if you don’t already know alot about character sets, wow it is one of the most confusing things so you may want to save learning about that topic for another day. Otherwise the following are helpful (and show how confusing character sets are!)

• Character Sets and Collations MySQL Support
• Converting Database Character Sets
• UTF-8 character sets (UTF-8)

If you ever setup WP to use the builtin membership features, make sure you learn about the CUSTOM_USER_TABLE and CUSTOM_USER_META_TABLE constants, I’ve found them very helpful.

/**#@+
 * MySQL settings
 */
/** The name of the database for WordPress */
define('DB_NAME', 'askapachewpblog75');
 
/** The username to access the database */
define('DB_USER', 'askapache245d');
 
/** The password for the username to access the database */
define('DB_PASSWORD', 'asdfklj2340');
 
/** The hostname to connect to the database at */
define('DB_HOST', 'mysql.askapache.com');
 
/** The charset of the database */
define('DB_CHARSET', 'utf8');
 
/** The collation of the database */
define('DB_COLLATE', 'utf8_general_ci');

$table_prefix

The $table_prefix is the value placed in the front of your database tables. Change the value if you want to use something other than wp_ for your database prefix. Typically this is changed if you are installing multiple WordPress blogs in the same database, and also for enhanced security.

Its a safe and good idea to change this value pre-installation to add more security to your WordPress blog. Exploits attempted against your WordPress blog by malicious crackers often are built with the premise that your blog uses the prefix wp_, by changing the value you mitigate some attack vectors.

/**
 * WordPress Database Table prefix.
 *
 * You can have multiple installations in one database if you give each a unique
 * prefix. Only numbers, letters, and underscores please!
 */
$table_prefix  = 'ar15_';
 
/** CUSTOM_USER_TABLE and CUSTOM_USER_META_TABLE are used to designated that the user and usermeta tables normally utilized by WordPress are not used, instead these values/tables are used to store your user information. */
//!defined('CUSTOM_USER_TABLE') && define('CUSTOM_USER_TABLE', $table_prefix . 'my_users');
//!defined('CUSTOM_USER_META_TABLE') && define('CUSTOM_USER_META_TABLE', $table_prefix . 'my_usermeta');

Setup PHP Ini Settings

 
/** Turns the output of errors on or off, you really never want this on, you should only view errors by reading the log file. */
ini_set('display_errors', WP_DEBUG_DISPLAY);
 
/** Tells whether script error messages should be logged to the server's error log or error_log. */
ini_set('log_errors', 'On');
 
/** http://us.php.net/manual/en/timezones.php */
ini_set('date.timezone', 'America/Indianapolis');
 
/** Where to log php errors */
ini_set('error_log', ASKAPACHE_ROOT . '/logs/php_error.log');
 
/** Set the memory limit, otherwise defaults to '32M' */
ini_set('memory_limit', WP_MEMORY_LIMIT);

Sessions are slow

So I only use sessions when I have a specific use… In this case I need sessions only when one of the tools in the /online-tools/ directory is being used. And that is for the captcha image. In the future I won’t ever use sessions.

if(preg_match( '#^/online-tools/#',$_SERVER['REQUEST_URI'])) session_start();

Include Custom Files

Sure you could use the my-hacks.php that WP allows, or you can just stick your functions in your TEMPLATEPATH/functions.php file, but they are executed only after the wp-settings.php file, which may be too late for your file.

In the past I’ve also used the auto_prepend_file settings to run my script before anything (index.php) but I ran into some issues on different hosts, and it wasn’t as portable.

This is useful because you can have a file with globally available functions that you can use in non-WP areas as well as WP areas. I am moving away from this more and more as I learn more about classes and build plugins instead for portability.

include_once ASKAPACHE_ROOT . '/includes/myfunctions.inc';
 
/** Sets up WordPress vars and included files. */
require_once(ABSPATH . 'wp-settings.php');
?>

Some Useful PHP

I am constantly trying to make my sites and code more portable, so I am using plugins alot more to accomplish things that I use to do with separate php. Here are some examples of minimal php.

add_filter("the_generator", create_function('$a','return "";'));
add_filter('the_content', create_function('$a', 'return ((is_feed())? $a."<p><a href=\"".get_permalink()."\">".get_the_title()."</a> originally appeared on ".get_bloginfo("name").".</p>" : $a);'), 99999);
add_filter('excerpt_length', create_function('$a', 'return 300;'),99);
add_filter('excerpt_more', create_function('$a', 'return "…";'),99);
add_action( 'wp_head', create_function('$a','echo "<link rel=\"pingback\" href=\"'.get_bloginfo('pingback_url').'\" />\n";'), 95 );
add_action( 'wp_head', create_function('$a','echo "<link rel=\"schema.rss\" href=\"http://purl.org/rss/1.0/\" />\n";'), 96 );
add_action( 'wp_head', create_function('$a','echo "<link rel=\"schema.rel\" href=\"http://purl.org/vocab/relationship/\" />\n";'), 97 );
add_action( 'wp_head', create_function('$a','echo "<link rel=\"meta\" type=\"application/rdf+xml\" href=\"/foaf.rdf\" />\n";'), 98 );
add_action( 'wp_head', create_function('$a','echo "<link href=\"/favicon.ico\" rel=\"shortcut icon\" type=\"image/x-icon\" />\n";'), 99 );

Debugging Note

If you read this far than you probably know how important debugging is, but I sometimes like to stick the best tips deep in my articles to make sure only YOU find it. GRTFM isn’t used on this site, it’s mostly a requirement because my writing can get pretty bad.. The point, debugging is more than a crucial requirement if you want to do anything cool. Don’t worry I got you.. check my AskApache Debug Viewer Plugin from the official WP site. It’s pretty close to providing as verbose amount of information that I could possibly figure out how to get out of php, probably more than you have ever seen at least, I focused on quantity. I use it all the time on new installs as there is no setup required and it tells me advanced information about the setup of the server, hacker code for sure.

Here’s a quick function to see set global vars, I just think this is interesting code.

function askapache_global_debug(){
  global $_GET,$_POST,$_COOKIE,$_SESSION,$_ENV,$_FILES,$_SERVER,$_REQUEST,$HTTP_POST_FILES,$HTTP_POST_VARS,$HTTP_SERVER_VARS,$HTTP_RAW_POST_DATA,$HTTP_GET_VARS,$HTTP_COOKIE_VARS,$HTTP_ENV_VARS;
  $gv=create_function('$n','global $$n; ob_start(); if ( is_array($$n) && sizeof($$n)>0 && print("[{$n}]\n") ) print_r($$n);return ob_get_clean();');
  foreach (array('_GET','_POST','_COOKIE','_SESSION','_ENV','_FILES','_SERVER','_REQUEST','HTTP_POST_FILES','HTTP_POST_VARS','HTTP_SERVER_VARS','HTTP_RAW_POST_DATA','HTTP_GET_VARS','HTTP_COOKIE_VARS','HTTP_ENV_VARS') as $k)echo $gv($k);
  print_r(get_defined_constants());
}

Also check the WordPress Codex page: Editing wp-config.php

Advanced WordPress wp-config.php Tweaks originally appeared on AskApache.

The Worst Kind of People

Spammers, and Leechers. They operate like this: Let’s say you have some mp3 files on a server, and SOMEWHERE on the web there is a link to that mp3 file’s location. This includes in javascript files, css files, robots.txt files, the spammers and leechers robots check all those files looking for the type of link they are looking for. Then they try to request that file usually utilizing a number of different types of requests to get access to the file. Then they use it for personal gain, at your peril.

Some robots perform valuable services for the world wide web community, and other leeching programming is pretty cool, so not all these activities are perpetrated by nefarious spammers.

Blocking by IP COOKIE

Ok so if a link exists to your file, it is going to be requested by a robot eventually, so the way to defeat them is by doing something on your site that modifies the way a user would request it. Robots for the most part are not javascript-capable, so the most-common advanced method is to set a cookie using javascript, and then we can check for that cookie in the request for the file using mod_rewrite.

So if your site sets a cookie named fspammers, and furthermore gives that cookie a value of 445, then this is what the request sent by an HTTP Client like Firefox looks like.

GET /hotlink/lovefreedom.mp3 HTTP/1.1
Host: z.askapache.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3 (.NET CLR 3.5.30729)
Accept: image/png,image/*;q=0.8,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: http://www.askapache.com/wordpress/seo-in-wordpress.html
Cookie: fspammers=455

Mod_Rewrite HTTP Headers

The mod_rewrite module has access to ALL the HTTP Headers sent in a request, so for each of the HTTP Headers in the request example above, we can use mod_rewrite to validate.

Mod_Rewrite .htaccess Example

Finally, now that everyone is on the same page about what is really going on, here is the .htaccess code that blocks any requests for anything in the /hotlink/ folder.

Here are the triggers this code blocks access based on.

1. Cookie: Checks if fspammers cookie is present, and that it has the value of 445.
2. HTTP Protocol: Checks if HTTP 1.1 is being used (many robots use 1.0)
3. Host: Checks that the HOST Requested was z.askapache.com
4. Referer: Checks for Referring site is z.askapache.com or www.askapache.com

RewriteEngine On
RewriteBase /
 
RewriteCond %{HTTP_COOKIE} !^.*fspammers=445.*$ [NC,OR]
RewriteCond %{THE_REQUEST} !^[A-Z]{3,9}\ /(.*)\ HTTP/1\.1 [NC,OR]
RewriteCond %{HTTP_HOST} !^z\.askapache\.com$ [NC,OR]
RewriteCond %{HTTP_REFERER} !^http://(www|z)\.askapache\.com.*$ [NC]
RewriteRule ^hotlink/.*$ - [F]</p>

Protecting Files with Advanced Mod_Rewrite Anti-Hotlinking originally appeared on AskApache.

The following undocumented techniques and methods will allow you to utilize mod_rewrite at an “expert level” by showing you how to unlock its secrets.

Most if not all web developers and server administrators struggle with Apache mod_rewrite. It’s very tough and only gets a little easier with practice. Until Now! Get ready to explode your learning curve, I figured something out.

Why mod_rewrite is so tough

I have come to the conclusion, after many hours of zenful thought, that the reason mod_rewrite is so tough is pretty obvious, people are trying to apply regular-expressions to URLs and Variables that they don’t really understand. They understand what they want, but they don’t understand what the URLS and Variables are that they are trying to rewrite.

Hit-Or-Miss with mod_rewrite

A lot of the mod_rewrite “experts” and “gurus” floating around the net absolutely know their mod_rewrite, but what separates them from a beginner or novice is for the most part an understanding of what the URLS and Variables look like that are targeted by the regular expressions. Take this simple rewriterule that rewrites requests made without the www to www.

RewriteEngine On
RewriteBase /
 
RewriteCond %{HTTP_HOST} !^www\.askapache\.com$ [NC]
RewriteRule .+ http://www.askapache.com%{REQUEST_URI}

Pretty simple right? WRONG. Most people could not figure that out..

Why?

The reason intelligent people can’t figure that out is because they have no idea what HTTP_HOST or REQUEST_URI actually looks like. How can you write a rule for something if you don’t know what it looks like? You can’t.

When Not To Use Mod_Rewrite

Ok so heres an important concept that alot of people haven’t heard. You should only use mod_rewrite’s rewriterule when you use a rewritecond or if you are rewriting internally like my feedcount hack.

If you are simply redirecting one url to another, you should definately be using the much easier mod_alias’s redirect and redirectmatch, which is enabled on most Apache servers.

When To Use Mod_Rewrite

So then, you should only use mod_rewrite’s rewriterule when you are checking against one of the Apache Environment Variables to determine whether to rewrite or not. This is where the Apache Documentation is grossly lacking. They don’t tell you what those variables look like, leaving us completely incapable of creating rewrites based on them. Not anymore.

Mod_Rewrite Environment Variables (The Secret)

Here’s the variables I have found accessible by mod_rewrite (both documented and undocumented). A thing to note is that you can set these variables early in an .htaccess file using SetEnv, RewriteRule, Header, etc.. and they will be accessible at the end of the .htaccess file.

• API_VERSION
• AUTH_TYPE
• CONTENT_LENGTH
• CONTENT_TYPE
• DOCUMENT_ROOT
• GATEWAY_INTERFACE
• HTTPS
• HTTP_ACCEPT
• HTTP_ACCEPT_CHARSET
• HTTP_ACCEPT_ENCODING
• HTTP_ACCEPT_LANGUAGE
• HTTP_CACHE_CONTROL

• HTTP_CONNECTION
• HTTP_COOKIE
• HTTP_FORWARDED
• HTTP_HOST
• HTTP_KEEP_ALIVE
• HTTP_PROXY_CONNECTION
• HTTP_REFERER
• HTTP_USER_AGENT
• IS_SUBREQ
• ORIG_PATH_INFO
• ORIG_PATH_TRANSLATED
• ORIG_SCRIPT_FILENAME

• ORIG_SCRIPT_NAME
• PATH
• PATH_INFO
• PHP_SELF
• QUERY_STRING
• REDIRECT_QUERY_STRING
• REDIRECT_REMOTE_USER
• REDIRECT_STATUS
• REDIRECT_URL
• REMOTE_ADDR
• REMOTE_HOST
• REMOTE_IDENT

• REMOTE_PORT
• REMOTE_USER
• REQUEST_FILENAME
• REQUEST_METHOD
• REQUEST_TIME
• REQUEST_URI
• SCRIPT_FILENAME
• SCRIPT_GROUP
• SCRIPT_NAME
• SCRIPT_URI
• SCRIPT_URL
• SCRIPT_USER

• SERVER_ADDR
• SERVER_ADMIN
• SERVER_NAME
• SERVER_PORT
• SERVER_PROTOCOL
• SERVER_SIGNATURE
• SERVER_SOFTWARE
• THE_REQUEST
• TIME
• TIME_DAY
• TIME_HOUR
• TIME_MIN

• TIME_MON
• TIME_SEC
• TIME_WDAY
• TIME_YEAR
• TZ
• UNIQUE_ID

Decoding Mod_Rewrite Variables

So when I realized my problem was that I didn’t know the value of the variable being tested by the RewriteCond, I set out to try and discover how to view those variables.. Keep in mind you can also use RewriteLogging, but its only allowed for root users who can edit the httpd.conf, this is .htaccess.

Setting Environment Variables with RewriteRule

I discovered a multitude of methods to set and view apache environment variables, using various modules and some core tricks, but the method that allows me to view the most environment variables is RewriteRule.. I wanted to use SetEnvIf more, but its just not as powerful as mod_rewrite, due to programming.

This code sets the variable INFO_REQUEST_URI to have the value of REQUEST_URI.

RewriteEngine On
RewriteBase /
RewriteRule .* - [E=INFO_REQUEST_URI:%{REQUEST_URI},NE]

Saving the Apache Variable Values

Now the trick is how to view that environment variable… The method I came up with is nice… We will send the environment variable value in an HTTP Header, as there isn’t much data manipulation/validation so you get an accurate look at the actual value.. At first I tried adding the variable value to a redirection using the query_string.. but a HTTP_USER_AGENT value doesn’t play well as a query_string.

Using RequestHeader in .htaccess

This code takes advantage of the incredible mod_headers apache module to actually ADD a whole new header to YOUR request. Seriously one of the coolest tricks I’ve found yet.. Its almost the same as being able to spoof POST requests! Since Headers can be protected data… especially the HTTP_COOKIE header..

RequestHeader set INFO_REQUEST_URI "%{INFO_REQUEST_URI}e"

Viewing the Variable Values

Now you can use any kind of server-run interpreter like perl, php, ruby, etc., to view all the variable values. All cgi-script handlers like those are able to view request headers..

PHP Code to access Apache Variables

Works even in safe-mode… any interpreter can view HTTP Headers! Note that each of these variables are added as HTTP headers to the request for the script.. kinda confusing.. So each variable sent as a header is prefixed with HTTP_ to denote it was a header.

<?php
header("Content-Type: text/plain");
$INFO=$MISS=array();
foreach($_SERVER as $v=>$r)
{
  if(substr($v,0,9)=='HTTP_INFO')
  {
    if(!empty($r))$INFO[substr($v,10)]=$r;
    else $MISS[substr($v,10)]=$r;
  }
}
 
/* thanks Mike! */
ksort($INFO);
ksort($MISS);
ksort($_SERVER);
 
echo "Received These Variables:\n";
print_r($INFO);
 
echo "Missed These Variables:\n";
print_r($MISS);
 
echo "ALL Variables:\n";
print_r($_SERVER);
?>

Time to Get Crazy

Just create the above php file on your site as /test/index.php or whatever, then create /test/.htaccess which should contain the below .htaccess file snippet. Now just request /test/index.php and be amazed! If you’re looking for more general help check out this excellent mod_rewrite cheat sheet.

Ok, so I’ve prepared the .htaccess code you can use to view the values of all these variables. Just add it to a .htaccess file and make a request. For this test I created an index.php file that printed out all the $_SERVER variables, and made requests to it.

RewriteEngine On
RewriteBase /
RewriteRule .* - [E=INFO_API_VERSION:%{API_VERSION},NE]
RewriteRule .* - [E=INFO_AUTH_TYPE:%{AUTH_TYPE},NE]
RewriteRule .* - [E=INFO_CONTENT_LENGTH:%{CONTENT_LENGTH},NE]
RewriteRule .* - [E=INFO_CONTENT_TYPE:%{CONTENT_TYPE},NE]
RewriteRule .* - [E=INFO_DOCUMENT_ROOT:%{DOCUMENT_ROOT},NE]
RewriteRule .* - [E=INFO_GATEWAY_INTERFACE:%{GATEWAY_INTERFACE},NE]
RewriteRule .* - [E=INFO_HTTPS:%{HTTPS},NE]
RewriteRule .* - [E=INFO_HTTP_ACCEPT:%{HTTP_ACCEPT},NE]
RewriteRule .* - [E=INFO_HTTP_ACCEPT_CHARSET:%{HTTP_ACCEPT_CHARSET},NE]
RewriteRule .* - [E=INFO_HTTP_ACCEPT_ENCODING:%{HTTP_ACCEPT_ENCODING},NE]
RewriteRule .* - [E=INFO_HTTP_ACCEPT_LANGUAGE:%{HTTP_ACCEPT_LANGUAGE},NE]
RewriteRule .* - [E=INFO_HTTP_CACHE_CONTROL:%{HTTP_CACHE_CONTROL},NE]
RewriteRule .* - [E=INFO_HTTP_CONNECTION:%{HTTP_CONNECTION},NE]
RewriteRule .* - [E=INFO_HTTP_COOKIE:%{HTTP_COOKIE},NE]
RewriteRule .* - [E=INFO_HTTP_FORWARDED:%{HTTP_FORWARDED},NE]
RewriteRule .* - [E=INFO_HTTP_HOST:%{HTTP_HOST},NE]
RewriteRule .* - [E=INFO_HTTP_KEEP_ALIVE:%{HTTP_KEEP_ALIVE},NE]
RewriteRule .* - [E=INFO_HTTP_MOD_SECURITY_MESSAGE:%{HTTP_MOD_SECURITY_MESSAGE},NE]
RewriteRule .* - [E=INFO_HTTP_PROXY_CONNECTION:%{HTTP_PROXY_CONNECTION},NE]
RewriteRule .* - [E=INFO_HTTP_REFERER:%{HTTP_REFERER},NE]
RewriteRule .* - [E=INFO_HTTP_USER_AGENT:%{HTTP_USER_AGENT},NE]
RewriteRule .* - [E=INFO_IS_SUBREQ:%{IS_SUBREQ},NE]
RewriteRule .* - [E=INFO_ORIG_PATH_INFO:%{ORIG_PATH_INFO},NE]
RewriteRule .* - [E=INFO_ORIG_PATH_TRANSLATED:%{ORIG_PATH_TRANSLATED},NE]
RewriteRule .* - [E=INFO_ORIG_SCRIPT_FILENAME:%{ORIG_SCRIPT_FILENAME},NE]
RewriteRule .* - [E=INFO_ORIG_SCRIPT_NAME:%{ORIG_SCRIPT_NAME},NE]
RewriteRule .* - [E=INFO_PATH:%{PATH},NE]
RewriteRule .* - [E=INFO_PATH_INFO:%{PATH_INFO},NE]
RewriteRule .* - [E=INFO_PHP_SELF:%{PHP_SELF},NE]
RewriteRule .* - [E=INFO_QUERY_STRING:%{QUERY_STRING},NE]
RewriteRule .* - [E=INFO_REDIRECT_QUERY_STRING:%{REDIRECT_QUERY_STRING},NE]
RewriteRule .* - [E=INFO_REDIRECT_REMOTE_USER:%{REDIRECT_REMOTE_USER},NE]
RewriteRule .* - [E=INFO_REDIRECT_STATUS:%{REDIRECT_STATUS},NE]
RewriteRule .* - [E=INFO_REDIRECT_URL:%{REDIRECT_URL},NE]
RewriteRule .* - [E=INFO_REMOTE_ADDR:%{REMOTE_ADDR},NE]
RewriteRule .* - [E=INFO_REMOTE_HOST:%{REMOTE_HOST},NE]
RewriteRule .* - [E=INFO_REMOTE_IDENT:%{REMOTE_IDENT},NE]
RewriteRule .* - [E=INFO_REMOTE_PORT:%{REMOTE_PORT},NE]
RewriteRule .* - [E=INFO_REMOTE_USER:%{REMOTE_USER},NE]
RewriteRule .* - [E=INFO_REQUEST_FILENAME:%{REQUEST_FILENAME},NE]
RewriteRule .* - [E=INFO_REQUEST_METHOD:%{REQUEST_METHOD},NE]
RewriteRule .* - [E=INFO_REQUEST_TIME:%{REQUEST_TIME},NE]
RewriteRule .* - [E=INFO_REQUEST_URI:%{REQUEST_URI},NE]
RewriteRule .* - [E=INFO_SCRIPT_FILENAME:%{SCRIPT_FILENAME},NE]
RewriteRule .* - [E=INFO_SCRIPT_GROUP:%{SCRIPT_GROUP},NE]
RewriteRule .* - [E=INFO_SCRIPT_NAME:%{SCRIPT_NAME},NE]
RewriteRule .* - [E=INFO_SCRIPT_URI:%{SCRIPT_URI},NE]
RewriteRule .* - [E=INFO_SCRIPT_URL:%{SCRIPT_URL},NE]
RewriteRule .* - [E=INFO_SCRIPT_USER:%{SCRIPT_USER},NE]
RewriteRule .* - [E=INFO_SERVER_ADDR:%{SERVER_ADDR},NE]
RewriteRule .* - [E=INFO_SERVER_ADMIN:%{SERVER_ADMIN},NE]
RewriteRule .* - [E=INFO_SERVER_NAME:%{SERVER_NAME},NE]
RewriteRule .* - [E=INFO_SERVER_PORT:%{SERVER_PORT},NE]
RewriteRule .* - [E=INFO_SERVER_PROTOCOL:%{SERVER_PROTOCOL},NE]
RewriteRule .* - [E=INFO_SERVER_SIGNATURE:%{SERVER_SIGNATURE},NE]
RewriteRule .* - [E=INFO_SERVER_SOFTWARE:%{SERVER_SOFTWARE},NE]
RewriteRule .* - [E=INFO_THE_REQUEST:%{THE_REQUEST},NE]
RewriteRule .* - [E=INFO_TIME:%{TIME},NE]
RewriteRule .* - [E=INFO_TIME_DAY:%{TIME_DAY},NE]
RewriteRule .* - [E=INFO_TIME_HOUR:%{TIME_HOUR},NE]
RewriteRule .* - [E=INFO_TIME_MIN:%{TIME_MIN},NE]
RewriteRule .* - [E=INFO_TIME_MON:%{TIME_MON},NE]
RewriteRule .* - [E=INFO_TIME_SEC:%{TIME_SEC},NE]
RewriteRule .* - [E=INFO_TIME_WDAY:%{TIME_WDAY},NE]
RewriteRule .* - [E=INFO_TIME_YEAR:%{TIME_YEAR},NE]
RewriteRule .* - [E=INFO_TZ:%{TZ},NE]
RewriteRule .* - [E=INFO_UNIQUE_ID:%{UNIQUE_ID},NE]
 
RequestHeader set INFO_API_VERSION "%{INFO_API_VERSION}e"
RequestHeader set INFO_AUTH_TYPE "%{INFO_AUTH_TYPE}e"
RequestHeader set INFO_CONTENT_LENGTH "%{INFO_CONTENT_LENGTH}e"
RequestHeader set INFO_CONTENT_TYPE "%{INFO_CONTENT_TYPE}e"
RequestHeader set INFO_DOCUMENT_ROOT "%{INFO_DOCUMENT_ROOT}e"
RequestHeader set INFO_GATEWAY_INTERFACE "%{INFO_GATEWAY_INTERFACE}e"
RequestHeader set INFO_HTTPS "%{INFO_HTTPS}e"
RequestHeader set INFO_HTTP_ACCEPT "%{INFO_HTTP_ACCEPT}e"
RequestHeader set INFO_HTTP_ACCEPT_CHARSET "%{INFO_HTTP_ACCEPT_CHARSET}e"
RequestHeader set INFO_HTTP_ACCEPT_ENCODING "%{INFO_HTTP_ACCEPT_ENCODING}e"
RequestHeader set INFO_HTTP_ACCEPT_LANGUAGE "%{INFO_HTTP_ACCEPT_LANGUAGE}e"
RequestHeader set INFO_HTTP_CACHE_CONTROL "%{INFO_HTTP_CACHE_CONTROL}e"
RequestHeader set INFO_HTTP_CONNECTION "%{INFO_HTTP_CONNECTION}e"
RequestHeader set INFO_HTTP_COOKIE "%{INFO_HTTP_COOKIE}e"
RequestHeader set INFO_HTTP_FORWARDED "%{INFO_HTTP_FORWARDED}e"
RequestHeader set INFO_HTTP_HOST "%{INFO_HTTP_HOST}e"
RequestHeader set INFO_HTTP_KEEP_ALIVE "%{INFO_HTTP_KEEP_ALIVE}e"
RequestHeader set INFO_HTTP_MOD_SECURITY_MESSAGE "%{INFO_HTTP_MOD_SECURITY_MESSAGE}e"
RequestHeader set INFO_HTTP_PROXY_CONNECTION "%{INFO_HTTP_PROXY_CONNECTION}e"
RequestHeader set INFO_HTTP_REFERER "%{INFO_HTTP_REFERER}e"
RequestHeader set INFO_HTTP_USER_AGENT "%{INFO_HTTP_USER_AGENT}e"
RequestHeader set INFO_IS_SUBREQ "%{INFO_IS_SUBREQ}e"
RequestHeader set INFO_ORIG_PATH_INFO "%{INFO_ORIG_PATH_INFO}e"
RequestHeader set INFO_ORIG_PATH_TRANSLATED "%{INFO_ORIG_PATH_TRANSLATED}e"
RequestHeader set INFO_ORIG_SCRIPT_FILENAME "%{INFO_ORIG_SCRIPT_FILENAME}e"
RequestHeader set INFO_ORIG_SCRIPT_NAME "%{INFO_ORIG_SCRIPT_NAME}e"
RequestHeader set INFO_PATH "%{INFO_PATH}e"
RequestHeader set INFO_PATH_INFO "%{INFO_PATH_INFO}e"
RequestHeader set INFO_PHP_SELF "%{INFO_PHP_SELF}e"
RequestHeader set INFO_QUERY_STRING "%{INFO_QUERY_STRING}e"
RequestHeader set INFO_REDIRECT_QUERY_STRING "%{INFO_REDIRECT_QUERY_STRING}e"
RequestHeader set INFO_REDIRECT_REMOTE_USER "%{INFO_REDIRECT_REMOTE_USER}e"
RequestHeader set INFO_REDIRECT_STATUS "%{INFO_REDIRECT_STATUS}e"
RequestHeader set INFO_REDIRECT_URL "%{INFO_REDIRECT_URL}e"
RequestHeader set INFO_REMOTE_ADDR "%{INFO_REMOTE_ADDR}e"
RequestHeader set INFO_REMOTE_HOST "%{INFO_REMOTE_HOST}e"
RequestHeader set INFO_REMOTE_IDENT "%{INFO_REMOTE_IDENT}e"
RequestHeader set INFO_REMOTE_PORT "%{INFO_REMOTE_PORT}e"
RequestHeader set INFO_REMOTE_USER "%{INFO_REMOTE_USER}e"
RequestHeader set INFO_REQUEST_FILENAME "%{INFO_REQUEST_FILENAME}e"
RequestHeader set INFO_REQUEST_METHOD "%{INFO_REQUEST_METHOD}e"
RequestHeader set INFO_REQUEST_TIME "%{INFO_REQUEST_TIME}e"
RequestHeader set INFO_REQUEST_URI "%{INFO_REQUEST_URI}e"
RequestHeader set INFO_SCRIPT_FILENAME "%{INFO_SCRIPT_FILENAME}e"
RequestHeader set INFO_SCRIPT_GROUP "%{INFO_SCRIPT_GROUP}e"
RequestHeader set INFO_SCRIPT_NAME "%{INFO_SCRIPT_NAME}e"
RequestHeader set INFO_SCRIPT_URI "%{INFO_SCRIPT_URI}e"
RequestHeader set INFO_SCRIPT_URL "%{INFO_SCRIPT_URL}e"
RequestHeader set INFO_SCRIPT_USER "%{INFO_SCRIPT_USER}e"
RequestHeader set INFO_SERVER_ADDR "%{INFO_SERVER_ADDR}e"
RequestHeader set INFO_SERVER_ADMIN "%{INFO_SERVER_ADMIN}e"
RequestHeader set INFO_SERVER_NAME "%{INFO_SERVER_NAME}e"
RequestHeader set INFO_SERVER_PORT "%{INFO_SERVER_PORT}e"
RequestHeader set INFO_SERVER_PROTOCOL "%{INFO_SERVER_PROTOCOL}e"
RequestHeader set INFO_SERVER_SIGNATURE "%{INFO_SERVER_SIGNATURE}e"
RequestHeader set INFO_SERVER_SOFTWARE "%{INFO_SERVER_SOFTWARE}e"
RequestHeader set INFO_THE_REQUEST "%{INFO_THE_REQUEST}e"
RequestHeader set INFO_TIME "%{INFO_TIME}e"
RequestHeader set INFO_TIME_DAY "%{INFO_TIME_DAY}e"
RequestHeader set INFO_TIME_HOUR "%{INFO_TIME_HOUR}e"
RequestHeader set INFO_TIME_MIN "%{INFO_TIME_MIN}e"
RequestHeader set INFO_TIME_MON "%{INFO_TIME_MON}e"
RequestHeader set INFO_TIME_SEC "%{INFO_TIME_SEC}e"
RequestHeader set INFO_TIME_WDAY "%{INFO_TIME_WDAY}e"
RequestHeader set INFO_TIME_YEAR "%{INFO_TIME_YEAR}e"
RequestHeader set INFO_TZ "%{INFO_TZ}e"
RequestHeader set INFO_UNIQUE_ID "%{INFO_UNIQUE_ID}e"

Mod_Rewrite Variables Decoded!

[API_VERSION] => 20020903:12
[AUTH_TYPE] => Digest
[DOCUMENT_ROOT] => /home/user/www_root/askapache.com
[HTTPS] => off
[HTTP_ACCEPT] => text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
[HTTP_COOKIE] => PHPSESSID=752ee6d56e15f305233e30045987e5ce568c034; __qca=1176541225-59967328-5223185;
[HTTP_HOST] => www.askapache.com
[HTTP_REFERER] => http://www.askapache.com/protest/index.php?askapache=awesomeness&you=rock
[HTTP_USER_AGENT] => Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.16) Gecko/20080702 Firefox/2.0.0.16
[IS_SUBREQ] => false
[QUERY_STRING] => e=404
[REMOTE_ADDR] => 22.162.144.211
[REMOTE_HOST] => 22.162.144.211
[REMOTE_PORT] => 4511
[REMOTE_USER] => administrator
[REQUEST_FILENAME] => /home/user/www_root/askapache.com/protest/index.php
[REQUEST_METHOD] => GET
[REQUEST_URI] => /protest/index.php
[SCRIPT_FILENAME] => /home/user/www_root/askapache.com/protest/index.php
[SCRIPT_GROUP] => daemonu
[SCRIPT_USER] => askapache
[SERVER_ADDR] => 208.113.134.190
[SERVER_ADMIN] => webmaster@askapache.com
[SERVER_NAME] => www.askapache.com
[SERVER_PORT] => 80
[SERVER_PROTOCOL] => HTTP/1.1
[SERVER_SOFTWARE] => Apache/2.0.61 (Unix) PHP/4.4.7 mod_ssl/2.0.61 OpenSSL/0.9.7e mod_fastcgi/2.4.2 DAV/2
[THE_REQUEST] => GET /protest/adf HTTP/1.1
[TIME] => 20080820014309
[TIME_DAY] => 20
[TIME_HOUR] => 01
[TIME_MIN] => 43
[TIME_MON] => 08
[TIME_SEC] => 09
[TIME_WDAY] => 3
[TIME_YEAR] => 2008

Request using HTTPS

[API_VERSION] => 20020903:12
[AUTH_TYPE] => Digest
[DOCUMENT_ROOT] => /home/user/www_root/askapache.com
[HTTPS] => on
[HTTP_ACCEPT] => text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
[HTTP_COOKIE] => PHPSESSID=752ee6d56e15f305233e30045987e5ce568c034; __qca=1176541225-59967328-5223185;
[HTTP_HOST] => www.askapache.com
[HTTP_REFERER] => http://www.askapache.com/protest/index.php?askapache=awesomeness&you=rock
[HTTP_USER_AGENT] => Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.16) Gecko/20080702 Firefox/2.0.0.16
[IS_SUBREQ] => false
[QUERY_STRING] => hi=you&whats=&you
[REMOTE_ADDR] => 22.162.144.211
[REMOTE_HOST] => 22.162.144.211
[REMOTE_PORT] => 4605
[REMOTE_USER] => administrator
[REQUEST_FILENAME] => /home/user/www_root/askapache.com/protest/index.php
[REQUEST_METHOD] => GET
[REQUEST_URI] => /protest/index.php
[SCRIPT_FILENAME] => /home/user/www_root/askapache.com/protest/index.php
[SCRIPT_GROUP] => daemonu
[SCRIPT_USER] => askapache
[SERVER_ADDR] => 208.113.134.190
[SERVER_ADMIN] => webmaster@askapache.com
[SERVER_NAME] => www.askapache.com
[SERVER_PORT] => 443
[SERVER_PROTOCOL] => HTTP/1.1
[SERVER_SOFTWARE] => Apache/2.0.61 (Unix) PHP/4.4.7 mod_ssl/2.0.61 OpenSSL/0.9.7e mod_fastcgi/2.4.2 DAV/2
[THE_REQUEST] => GET /protest/index.php?hi=you&whats=&you HTTP/1.1
[TIME] => 20080820015016
[TIME_DAY] => 20
[TIME_HOUR] => 01
[TIME_MIN] => 50
[TIME_MON] => 08
[TIME_SEC] => 16
[TIME_WDAY] => 3
[TIME_YEAR] => 2008

Emulating ErrorDocuments with Mod_Rewrite

The ErrorDocument directive is helpful because an errordocument is called differently then a normal file, and it contains special variables to help an admin debug.

I’ve wanted to use a RewriteCond + a RewriteRule to cause an Apache ErrorDocument to be displayed for a long time… I finally figured it out. Simply use the HTTP STATUS CODE trick in combination with a simple RewriteRule to trigger an Apache ErrorDocument.

This code emulates the internal 404 process Apache goes through.. If the file is not found it requests the /test/trigger-error/404 internally which triggers the 404 ErrorDocument.

ErrorDocument 404 /test/errordocument/404.html
Redirect 404 /test/trigger-error/404
 
RewriteEngine On
RewriteBase /
RewriteCond %{ENV:REDIRECT_STATUS} !=404
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule .* /test/trigger-error/404 [L]

Big Deal.. you might say… well consider that this works with any status code, and using this method you now have the power to trigger any errordocument page based on any kind of rewritecond. I’ll be writing about some practical uses for this powerful method in the coming weeks, but heres a good example now so you can see how it can be used.

This bit of code Triggers the 505 HTTP Version Not Supported When a request is made to the server with a protocol other than 1.1.

ErrorDocument 505 /test/errordocument/505.html
Redirect 505 /test/trigger-error/505
 
RewriteEngine On
RewriteBase /
 
RewriteCond %{ENV:REDIRECT_STATUS} !=505
RewriteCond %{THE_REQUEST} !^[A-Z]{3,9}\ /.*\ HTTP/(0\.9|1\.0|1\.1) [NC]
RewriteRule .* /test/trigger-error/505 [L]

YES! I realize I didn’t explain that very well, I didn’t realize it was that complicated.. I wanted to go into how to use these advanced tricks and methods to achieve some really cool stuff, but explaining just this little bit took me awhile and I’m out of page space!

So play around with this and I’ll post back some of the untapped sicknesses you can give a website with such powerful methods at your disposal.

Ralf S. Engelschall

/*
 *  URL Rewriting Module
 *
 *  This module uses a rule-based rewriting engine (based on a
 *  regular-expression parser) to rewrite requested URLs on the fly.
 *
 *  It supports an unlimited number of additional rule conditions (which can
 *  operate on a lot of variables, even on HTTP headers) for granular
 *  matching and even external database lookups (either via plain text
 *  tables, DBM hash files or even external processes) for advanced URL
 *  substitution.
 *
 *  It operates on the full URLs (including the PATH_INFO part) both in
 *  per-server context (httpd.conf) and per-dir context (.htaccess) and even
 *  can generate QUERY_STRING parts on result.   The rewriting result finally
 *  can lead to internal subprocessing, external request redirection or even
 *  to internal proxy throughput.
 *
 *  This module was originally written in April 1996 and
 *  gifted exclusively to the The Apache Software Foundation in July 1997 by
 *
 *      Ralf S. Engelschall
 *      rse engelschall.com
 *      www.engelschall.com
 */

Crazy Advanced Mod_Rewrite Tutorial originally appeared on AskApache.

Raw Speed Benefit

This eliminates millions of bandwidth and resource robbing 304 If-Modified-Since requests.

Renaming links vs. Renaming files

On the server my files are named apache.css and apache.js, but in the xhtml I point to them using the names apache-113.css and apache-113.js, after I change the file I just add 1 to the number, and the new file is cached. They are internally redirected to apache.css and apache.js (invisible to the user) The concept is similar to a “shortcut” in windows or a symlink in BSD. The trick is that I never actually rename the files on the server. I just rename them in the html. That means apache-135.css is served from the file apache.css but the browser/cache only see and know about apache-135.css.

XHTML

NOTE: You can do your own investigating of this sites source code and HTTP headers to see this whole system in action

<link href="http://z.askapache.com/z/c/apache-0031.css" rel="stylesheet" type="text/css" />
<script src="http://z.askapache.com/z/j/apache-0031.js" type="text/javascript"></script>

mod_rewrite code for htaccess or httpd.conf

Updated: 10/20/2008

RewriteEngine On
RewriteBase /
RewriteCond %{THE_REQUEST} ^(GET|HEAD)\ /z/([a-z]+)/(.+)-([0-9]+)\.(js|css).*\ HTTP/ [NC]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule .+ /z/%2/%3.%5 [NC,L]

Alternate mod_rewrite code

RewriteEngine On
RewriteBase /
RewriteRule ^([cij]+)(/?[a-z]*)/([a-z]+)-([0-9]+)\.([a-z]+)$ /$1$2/$3.$5 [L]

Ideal Caching Scheme

Ok so you want the xhtml to be the only file that isn’t cached without being validated, its simple to setup your own caching scheme.

Bad Cache information for a file with the ?v=foo hack

This object will be fresh for 1 week. It can be validated with Last-Modified. This object requests that a Cookie be set; this makes it and other pages affected automatically stale; clients must check them upon every request. Because it contains a query (‘?’) in the URL, many caches will not keep this object.

mod_rewrite Fix for Caching Updated Files originally appeared on AskApache.

Want to get started without even having to read this page? Type the following in your shell.

curl -O http://z.askapache.com/askapache-bash-profile.txt && source askapache-bash-profile.txt

For those of you power users and server admins that use Bash, ksh, csh, vanilla sh, etc.., or if you are just passionate about shell scripting, because it allows you to get advanced tasks done fast and efficiently, not to mention automated automatically. I give you my .bash_profile file. You should edit it to fit your needs, (especially the exported vars like PATH, LDFLAGS, if you don’t understand something just comment it out) but it’s pretty universal because I work on alot of other people’s servers not to mention many different distros and platforms, and when I get hired to do some server work through a shell, I bring this script along for the ride.

askapache-bash-profile.txt

You can download the latest version: http://z.askapache.com/askapache-bash-profile.txt

The functions and variables below are the way bash sees them, using declare -f, and alias, to make it easier for you to read and understand them. The actual file at http://z.askapache.com/askapache-bash-profile.txt will always be the most updated version, as I use it personally. And it has the whole file the way I wrote it, meaning many extra notes and much simpler to follow. Enjoy!

#!/bin/bash
#
# To install the latest version
#
# curl -O http://z.askapache.com/askapache-bash-profile.txt && source askapache-bash-profile.txt
#
# To run automatically at login: In your ~/.bash_profile or similar login script do
# [[ -r path-to-askapache-bash-profile.txt ]] && source path-to-askapache-bash-profile.txt
#
 
AAPN='AskApache Bash Profile Script'
AAPV='5.6'
AAPT='2009-22-2009'
AAPS=`echo "$0" | sed -e 's,.*/,,'`
 
############################################################################################################################################################
# Copyright (C) 2009 www.AskApache.com
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
############################################################################################################################################################
 
# dont do anything for non-interactive shells
[[ -z "$PS1"  ]] && return
 
# bash defines the following built-in commands:
#   :, ., [, alias, bg, bind, break, builtin, case, cd, command, compgen, complete, continue, declare, dirs
#   disown, echo, enable, eval, exec, exit, export, fc, fg, getopts, hash, help, history, if, jobs, kill,
#   let, local, logout, popd, printf, pushd, pwd, read, readonly, return, set, shift, shopt, source, suspend
#   test, times, trap, type, typeset, ulimit, umask, unalias, unset, until, wait, while
 
#------------------------------------------------------------------------------------------------------------------------------------------------------------
# The first digit selects the set user ID (4) and set group ID (2) and sticky (1) attributes.
# The second digit selects permissions for the user who owns the file: read (4), write (2), and execute (1)
# The third selects permissions for other users in the file's group, with the same values
# The fourth for other users not in the file's group, with the same values.
# [see man chmod, help umask]
#------------------------------------------------------------------------------------------------------------------------------------------------------------
umask 0022
 
#------------------------------------------------------------------------------------------------------------------------------------------------------------
# Trapping Signals to Catch Errors
#------------------------------------------------------------------------------------------------------------------------------------------------------------
# 1      2      3       4      5       6       7      8      9       10      11      12      13      14      15      17      18      19      20      21
# SIGHUP SIGINT SIGQUIT SIGILL SIGTRAP SIGABRT SIGBUS SIGFPE SIGKILL SIGUSR1 SIGSEGV SIGUSR2 SIGPIPE SIGALRM SIGTERM SIGCHLD SIGCONT SIGSTOP SIGTSTP SIGTTIN
# [see man bash, man signal, help trap]
#------------------------------------------------------------------------------------------------------------------------------------------------------------
trap 'echo -e "\n\n\n!!! TERMINATED SIG: [$?] AT LINE:$LINENO AFTER $SECONDS SECONDS !!!\n"|tee -a `tty 2>/dev/null`;' 15
 
#------------------------------------------------------------------------------------------------------------------------------------------------------------
# Advanced Shell Limits
#------------------------------------------------------------------------------------------------------------------------------------------------------------
# -S      use the `soft' resource limit
# -H      use the `hard' resource limit
# -a      all current limits are reported
# -c      the maximum size of core files created
# -d      the maximum size of a process's data segment
# -f      the maximum size of files created by the shell
# -l      the maximum size a process may lock into memory
# -m      the maximum resident set size
# -n      the maximum number of open file descriptors
# -p      the pipe buffer size
# -s      the maximum stack size
# -t      the maximum amount of cpu time in seconds
# -u      the maximum number of user processes
# -v      the size of v
# [see man getrlimit, help ulimit]
#------------------------------------------------------------------------------------------------------------------------------------------------------------
ulimit -S -c 0 # Don't want any coredumps
 
#------------------------------------------------------------------------------------------------------------------------------------------------------------
# Advanced Shell (set)tings
#------------------------------------------------------------------------------------------------------------------------------------------------------------
# e [errexit] Exit immediately if a command exits with a non-zero status.
# B [braceexpand] The shell will perform brace expansion.
# h [hashall] Remember the location of commands as they
# f [noglob]      Disable file name generation (globbing).
# H [histexpand]  Enable ! style history substitution.
# v [verbose]     Print shell input lines as they are read.
# x [xtrace]      Print commands and their arguments as they are executed.
# n [noexec]      Read commands but do not execute them.
# [history] Enable command history
# [see man bash, help set]
#------------------------------------------------------------------------------------------------------------------------------------------------------------
set +C +f +H +v +x +n -b -h -i -m -B
 
#------------------------------------------------------------------------------------------------------------------------------------------------------------
# Advanced Shell (shopt)ions
#------------------------------------------------------------------------------------------------------------------------------------------------------------
# cdable_vars             an argument to the cd builtin command that is not a directory is assumed to be the name of a variable dir to change to.
# cdspell                 minor errors in the spelling of a directory component in a cd command will be corrected.
# checkhash               bash checks that a command found in the hash table exists before execute it.  If no longer exists, a path search is performed.
# checkwinsize            bash checks the window size after each command and, if necessary, updates the values of LINES and COLUMNS.
# cmdhist                 bash attempts to save all lines of a multiple-line command in the same history entry.  Allows re-editing of multi-line commands.
# dotglob                 bash includes filenames beginning with a `.' in the results of pathname expansion.
# execfail                a non-int shell will not exit if it cannot execute the file specified as an argument to the exec builtin command, like int sh.
# expand_aliases          aliases are expanded as described above under ALIASES.  This option is enabled by default for interactive shells.
# extglob                 the extended pattern matching features described above under Pathname Expansion are enabled.
# histappend              the history list is appended to the file named by the value of the HISTFILE variable when shell exits, no overwriting the file.
# hostcomplete            and readline is being used, bash will attempt to perform hostname completion when a word containing a @ is being completed
# huponexit               bash will send SIGHUP to all jobs when an interactive login shell exits.
# interactive_comments    allow a word beginning with # to cause that word and all remaining characters on that line to be ignored in an interactive shell
# lithist                 if cmdhist option is enabled, multi-line commands are saved to the history with embedded newlines rather than using semicolon
# login_shell             shell sets this option if it is started as a login shell (see INVOCATION above).  The value may not be changed.
# mailwarn                file that bash is checking for mail has been accessed since the last checked, ``The mail in mailfile has been read'' is displayed.
# no_empty_cmd_completion bash will not attempt to search the PATH for possible completions when completion is attempted on an empty line.
# nocaseglob              bash matches filenames in a case-insensitive fashion when performing pathname expansion (see Pathname Expansion above).
# nullglob                bash allows patterns which match no files (see Pathname Expansion above) to expand to a null string, rather than themselves.
# progcomp                the programmable completion facilities (see Programmable Completion above) are enabled.  This option is enabled by default.
# promptvars              prompt strings undergo variable and parameter expansion after being expanded as described in PROMPTING above.
# shift_verbose           the shift builtin prints an error message when the shift count exceeds the number of positional parameters.
# sourcepath              the source (.) builtin uses the value of PATH to find the directory containing the file supplied as an argument.
# xpg_echo                the echo builtin expands backslash-escape sequences by default.
# [see man bash, help shopt]
#------------------------------------------------------------------------------------------------------------------------------------------------------------
shopt -s histappend cmdhist extglob no_empty_cmd_completion checkwinsize progcomp sourcepath cdspell cdable_vars checkhash histreedit histverify
shopt -u mailwarn; unset MAILCHECK # dont notify for new mail
 
#----------------------------
# CUSTOM SETTING VARIABLES
#----------------------------
 
# quicker, cleaner ref
export N6=/dev/null
 
# if a program exists in path
function have()
{ unset -v have; PATH=$PATH:/sbin:/usr/sbin:/usr/local/sbin; type $1 &>/$N6 && have="yes"; }
 
[[ -z ${USER:=""} ]] && export USER=`whoami 2>$N6 || id -un 2>$N6 || logname 2>$N6`
[[ -z ${HOME:=""} ]] && export HOME=$(command pwd -L)
 
export HOME=${HOME:-~};
export SHELL=${SHELL:-/bin/bash}
have tty && export SSHTTY=`tty 2>$N6`
 
export TZ='America/Indianapolis'
[[ -z "$HOST" ]] && export HOST=`hostname 2>$N6||uname -n 2>$N6`
 
# DEFINATELY EDIT THIS STUFF, is pretty good at setting path
#JAVA_HOME=/usr/local/jdk
#P=${PATH}:$HOME/bin:$HOME/sbin:/bin:/etc:/sbin:/usr/local/bin:/usr/sbin:/usr/X11R6/bin:/usr/bin:/usr/bin/mh:/usr/libexec:/etc/X11:/etc/X11/xinit:$HOME/.gem/ruby/1.8/bin
#P=${P}:/usr/local/dh/apache/template/bin:/usr/local/dh/apache2/template/bin:/usr/local/dh/apache2/template/build:/usr/local/dh/apache2/template/sbin
#P=${P}:/usr/local/dh/bin:/usr/local/dh/java/bin:/usr/local/dh/java/jre/bin:/usr/local/php5/bin:.
#PATH=$P
GP=; for p in ${PATH//:/ }; do [[ -d $p && -x $p ]] && [[ "${GP}s" == s ]] && GP=$p || GP=${GP}:$p; done; export PATH=$GP
 
#----------------------------
# LIBS / COMPILES - EDIT
#----------------------------
#LDFLAGS="-L${HOME}/lib -L/lib -L/usr/lib -L/usr/lib/libc5-compat -L/lib/libc5-compat -L/usr/i486-linuxlibc1/lib -L/usr/X11R6/lib"; LD_LIBRARY_PATH=$HOME/lib; CPPFLAGS="-I${HOME}/include"
#export LDFLAGS LD_LIBRARY_PATH CPPFLAGS
 
#----------------------------
# MAIL, PROGRAMS, EDITOR
#----------------------------
have nano && export EDITOR="`type -P nano`" && export VISUAL=$EDITOR
have lynx && export BROWSER="`type -P lynx`"
 
#alias man='man -H'
#[[ -r $HOME/.lynx/lynx.cfg ]] && export LYNX_CFG=$HOME/.lynx/lynx.cfg
#[[ -r $HOME/var/locatedb ]] && export LOCATE_PATH=$HOME/var/locatedb
 
alias more='less'
export PAGER=less
export LESSCHARSET='latin1'
have lesspipe && export LESSOPEN='|/usr/bin/lesspipe %s 2>&-' && export LESS='-i -N -w  -z-4 -g -e -M -X -F -R -P%t?f%f :stdin .?pb%pb\%:?lbLine %lb:?bbByte %bb:-...'
 
#------------------------------------------------------------------------------------------------------------------------------------------------------------
# PROMPT
#------------------------------------------------------------------------------------------------------------------------------------------------------------
# \a     an ASCII bell character (07)
# \d     the date in "Weekday Month Date" format (e.g., "Tue May 26")
# \D{format}  the format is passed to strftime(3) and the result is inserted into the prompt string;
# \e     an ASCII escape character (033)
# \h     the hostname up to the first `.'
# \H     the hostname
# \j     the number of jobs currently managed by the shell
# \l     the basename of the shell's terminal device name
# \n     newline
# \r     carriage return
# \s     the name of the shell, the basename of $0 (the portion following the final slash)
# \t     the current time in 24-hour HH:MM:SS format
# \T     the current time in 12-hour HH:MM:SS format
# \@     the current time in 12-hour am/pm format
# \A     the current time in 24-hour HH:MM format
# \u     the username of the current user
# \v     the version of bash (e.g., 2.00)
# \V     the release of bash, version + patchelvel (e.g., 2.00.0)
# \w     the current working directory
# \W     the basename of the current working directory
# \!     the history number of this command
# \#     the command number of this command
# \$     if the effective UID is 0, a #, otherwise a $
# \nnn   the character corresponding to the octal number nnn
# \\     a backslash
# \[     begin a sequence of non-printing characters, which could be used to embed a terminal control sequence into the prompt
# \]     end a sequence of non-printing characters
#------------------------------------------------------------------------------------------------------------------------------------------------------------
 
case ${TERM:-dummy} in linux*|con80*|con132*|console|xterm*|vt*|screen*|putty|Eterm|dtterm|ansi|rxvt|gnome*|*color*)COLORTERM=yes; ;; *) COLORTERM=no; ;; esac; export COLORTERM
PS1="\n[`date +%m/%d`][\u@\h][\w]($SHLVL:\!)\n$ "; X=;R=;
[[ "$COLORTERM" == yes ]] && X='\033[1;37m' && R='\033[0;00m' && PS1="\n[`date +%m/%d`]\e[1;37m[\e[0;32m\u\e[0;35m@\e[0;32m\h\e[1;37m]\e[1;37m[\e[0;31m\w\e[1;37m]($SHLVL:\!)\n\[${X}\]\$ "
export PS1 R X
 
# If set, the value is executed as a command prior to issuing each primary prompt.
have tty && export SSHTTY=$(tty 2>$N6);
[[ -z "$SSHTTY" ]] || export PROMPT_COMMAND='echo -ne "\033]0;$USER:`id -gn 2>$N6`@$HOSTNAME `tty 2>$N6`  [`uptime 2>$N6|sed -e "s/.*: \([^,]*\).*/\1/" -e "s/ //g"` / `command ps aux|wc -l`]  `command w|wc -l` users \007"'
unset PROMPT_COMMAND
 
# used by functions below
export RJ=0;
declare -a CC;
export L1=$(seq -s_ 1 75|tr -d [0-9]);
export L2=$(echo $L1|tr '_' ' ')
 
#---------------
# ls aliases
#---------------
C=' --color=auto'
alias la="command ls -Al${C}"      # show hidden files
alias lx="command ls -lAXB${C}"    # sort by extension
alias lk="command ls -lASr${C}"    # sort by size
alias lc="command ls -lAcr${C}"    # sort by change time
alias lu="command ls -lAur${C}"    # sort by access time
alias lr="command ls -lAR${C}"     # recursive ls
alias lt="command ls -lAtr${C}"    # sort by date
alias lll="stat -c %a\ %N\ %G\ %U \$PWD/*|sort"
 
#---------------
# safety aliases
#---------------
alias chmod='command chmod -c'
alias mkdir='command mkdir -pv'
alias rm='command rm -v'
alias cp='command cp -v'
alias mv='command mv -v'
 
#---------------
# func aliases
#---------------
alias who='command who -ar -pld'
alias which='command type -path'
have tree || alias tree='command -lsFR'
have tree && alias tree='command tree -Csuflapi'
alias top='top -c'
alias vim='command vim --noplugin'
alias du='command du -kh'
alias df='command df -kTh'
[[ $UNAME != Linux ]] && have gsed && alias sed=gsed
#alias php='php -d display_errors=true'
#alias man='man -H'
#alias ldconfig="ldconfig -v -f ${HOME}/etc/ld.so.conf -C ${HOME}/etc/ld.so.cache"
alias path='echo -e ${PATH//:/\\n}'
 
alias pp='command ps -HAcl -F S -A f'
alias p='command ps -HAcl -F S -A f|uniq -w3'
alias ps2='command ps -H'
alias ps1='command ps -lFA'
 
alias df1='command df -iTa'
alias n="${EDITOR}"3
alias inice='ionice -c3 -n7 nice'
alias vdir="ls --format=long"
alias dir="ls --format=vertical"
alias killphp='pkill -9 -f php.cgi\|php5.cgi\|php-cgi\|php\|php4.cgi;pkill -13 -f php.cgi\|php5.cgi\|php-cgi\|php\|php4.cgi'
alias dsiz='du -sk * | sort -n --'
alias lessc='ccze -A |/usr/bin/less -R'
alias h='history'
alias j='jobs -l'
alias ..='cd ..'
alias wtf='watch -n 1 w -hs'
 
# spelling typos
alias kk='ll'
alias xs='cd'
alias vf='cd'
alias moer='more'
alias moew='more'
 
#------------------------------------------------------------------------------------------------------------------------------------------------------------
# history setup
#------------------------------------------------------------------------------------------------------------------------------------------------------------
# HISTCONTROL
#    If  set  to a value of ignorespace, lines which begin with a space character are not entered on the history list.
#    If set to a value of ignoredups, lines matching the last history line are not entered.  A value of ignoreboth combines the two options.
#    If unset, or if set to any other value than those above, all lines read by the parser are saved on the history list, subject to the value of HISTIGNORE.
# HISTFILE
#    The name of the file in which command history is saved (see HISTORY below).  The default value is ~/.bash_history.  If unset, the command history is not saved when an interactive shell exits.
# HISTFILESIZE
#    The  maximum number of lines contained in the history file.  When this variable is assigned a value, the history file is truncated, if necessary, to contain no more than that number of lines.
#    The default value is 500.  The history file is also trun-cated to this size after writing it when an interactive shell exits.
# HISTIGNORE
#    A colon-separated list of patterns used to decide which command lines should be saved on the history list.  Each pattern is anchored at the beginning of the line and must match the complete line (no implicit `*' is appended).
#    Each pattern is  tested against  the  line  after  the  checks  specified  by HISTCONTROL are applied.  In addition to the normal shell pattern matching characters, `&' matches the previous history line.
#    `&' may be escaped using a backslash; the backslash is removed before attempting a match.  The second and subsequent lines of a multi-line compound command are not tested, and are added to the history regardless of the value of HISTIGNORE.
# HISTSIZE
#    The number of commands to remember in the command history (see HISTORY below).  The default value is 500.
#
HISTMASTER=;HISTCONTROL=;HISTIGNORE=;HISTSIZE=;HISTFILESIZE=;HISTFILE=;HISTTIMEFORMAT=;
function asetup_history()
{
  HISTMASTER=${HISTMASTER:-$HOME/backups/.history/combined.log}
  HISTCONTROL=${HISTCONTROL:-'ignoreboth'}
  HISTTIMEFORMAT="%H:%M > "
  HISTIGNORE=${HISTIGNORE:-'clear:ll:ls:./_sbackup.sh:./_logview.sh:ps:updatedb:top'}
  HISTSIZE=${HISTSIZE:-5000000}
  HISTFILESIZE=${HISTFILESIZE:-5000000}
  HISTFILE=${HISTFILE:-$HOME/.bash_history}
  export HISTMASTER HISTCONTROL HISTIGNORE HISTSIZE HISTFILESIZE HISTFILE HISTTIMEFORMAT
}
 
function h1()
{ cat $HISTFILE $HISTMASTER | command grep $1 | sort | uniq | command grep --color=always $1; }
 
function h2()
{ cat $HISTFILE $HISTMASTER | command grep $1 | sort | uniq; }
 
#------------------------------------------------------------------------------------------------------------------------------------------------------------
# personalized colors
#------------------------------------------------------------------------------------------------------------------------------------------------------------
# Attribute codes:        00=none 01=bold 04=underscore 05=blink 07=reverse 08=concealed
# Text color codes:       30=black 31=red 32=green 33=yellow 34=blue 35=magenta 36=cyan 37=white
# Background color codes: 40=black 41=red 42=green 43=yellow 44=blue 45=magenta 46=cyan 47=white
# NORMAL 00       # global default, although everything should be something.
# FILE 00         # normal file
# DIR 01;34       # directory
# LINK 01;36      # symbolic link.
# FIFO 40;33      # pipe
# SOCK 01;35      # socket
# DOOR 01;35      # door
# BLK 40;33;01    # block device driver
# CHR 40;33;01    # character device driver
# ORPHAN 40;31;01 # symlink to nonexistent file
# EXEC 01;32      # executables
LS_COLORS=;LS_OPTIONS=;
function asetup_colors(){
 local L; eval "`dircolors -b`";
 [[ "$COLORTERM" == no ]] && return
 export LS_COLORS
 unset LS_OPTIONS
}
 
function ascript_title(){
  clear; echo; echo; lin 0;
  echo -e "| ${CC[2]}             ___       __    ___                 __                     ${CC[0]} |"
  echo -e "| ${CC[2]}            / _ | ___ / /__ / _ | ___  ___ _____/ /  ___                ${CC[0]} |"
  echo -e "| ${CC[9]}           / __ |(_-</  '_// __ |/ _ \/ _ \`/ __/ _ \/ -_)               ${CC[0]} |"
  echo -e "| ${CC[9]}          /_/ |_/___/_/\_\/_/ |_/ .__/\_,_/\__/_//_/\__/                ${CC[0]} |"
  echo -e "| ${CC[9]}                               /_/                                      ${CC[0]} |";
  lin 1; lin 2 "${1:-$AAPN}"; lin 2 "Version ${2:-$AAPV}"; lin 2 "Build: ${3:-$AAPT}"; lin 3;sleep 2;
}
 
###########################################################################--=--=--=--=--=--=--=--=--=--=--#
###
### FUNCTIONS
###
###########################################################################==-==-==-==-==-==-==-==-==-==-==#
 
#export SMSO=$(tput smso &>$N6);export RMSO=$(tput rmso &>$N6);
#find . -type f -iname '*htaccess*' -print0 | xargs -0 grep -sn -i "THE_REQUEST" | sed "s/THE_REQUEST/${SMSO}\0$RMSO}/gI"
 
function repeat()
{ local i max=$1; shift; for ((i=1; i <= max ; i++)); do eval "$@"; done; }
 
function cuttail()
{ sed -n -e :a -e "1,${2:-10}!{P;N;D;};N;ba" $1; }
 
function quote()
{ echo \'${1//\'/\'\\\'\'}\'; }
 
function dequote()
{ eval echo "$1"; }
 
function get_pids_on_system()
{ command ps axo pid|sed 1d; }
 
function get_pgids_on_system()
{ command ps axo pgid|sed 1d; }
 
function get_users_on_system()
{ have getent &>$N6 && getent passwd|awk -F: '{print $1}' && return; awk 'BEGIN {FS=":"} {print $1}' /etc/passwd; }
 
function get_groups_on_system()
{ have getent &>$N6 && getent group|awk -F: '{print $1}' && return; awk 'BEGIN {FS=":"} {print $1}' /etc/group; }
 
function get_inet_interface_ips()
{ have ip && /bin/ip -o -f inet addr|sed -e '/inet 10./d' -e 's/\/24//g' |awk '{print $4}'|sed -e "s/\\n//g" -e '/.*\.0$/d'; }
 
function personalized_wget()
{
 exec 6>&1 ; exec > ${1:-$HOME/.wgetrc}
 echo 'header = Accept-Language: en-us,en;q=0.5'
 echo 'header = Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5'
 echo 'header = Accept-Encoding: gzip,deflate'
 echo 'header = Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7'
 echo 'header = Keep-Alive: 300'
 echo 'user_agent = Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6'
 echo 'referer = http://www.google.com'
 echo 'robots = off'
 exec 1>&6 6>&-
}
 
function mkdir()
{
 local e=0;
 for f in ${1+"$@"};do
 set fnord `echo ":$f"|sed -e 's/^:\//%/' -e 's/^://' -e 's/\// /g' -e 's/^%/\//'`;shift;pp=;for d in ${1+"$@"};do pp="$pp$d";
 case "$pp" in -*)pp=./$pp; ;; ?:)pp="$pp/";continue ;; esac; [[ ! -d "$pp" ]] && mkdir -v "$pp"||e=$?; pp="$pp/"; done;
 done; return  $e
}
 
function clean_exit()
{ lin 0;lin 1;lin 2 "COMPLETED SUCCESSFULLY";lin 1;lin 3; }
 
function dirty_exit()
{ echo "See ya..";sleep 1;exit; }
 
function set_window_title()
{ echo -n -e "\033]0;$*\007"; }
 
function pd()
{ echo -e  "\n ${CC[15]} ${1:-DONE} $R\n\n" >$SSHTTY; }
 
function cont()
{ local ans; echo -en "\n ${CC[15]}[ ${1:-Press any key to continue} ]$R\n"; read -n 1 ans; }
 
function do_sleep ()
{ echo -en "${CC[6]}${3:-.}" >$SSHTTY;while [[ -d /proc/$RJ ]];do sleep ${2:-3};echo -en "${3:-.}" >$SSHTTY;done;echo -e "${CC[0]}" >$SSHTTY&&sleep 1&&pd; }
 
function pwd()
{ command pwd -LP "$@"; }
 
function kill_jobs(){
 for i in `jobs -p`; do kill -9 $i; done; }
 
function beep_alarm()
{ local i; for i in `seq 0 ${1:-5}`;do echo -en "\a" && sleep 1; done; }
 
function lin()
{ case ${1:-1} in 0)echo -e "\n$R $L1"; ;; 1)echo -e "|$L2|"; ;; 2)echo -en "${R}|${CC[34]}";echo -en "${2:-1}"|sed -e :a -e 's/^.\{1,72\}$/ & /;ta' -e "s/\(.*\)/\1/";echo -e "${R} |${R}" ;; 3)echo -e "$R $L1$R$X\n\n"; ;; esac; }
 
function mp3info()
{ ls *.mp3 |xargs -ix 2>&1 ffmpeg -i x|grep -v "^Must" |grep -v "built\|libavutil\|libavcodec\|configuration\|FFmpeg\|libavformat"; }
 
function tailit()
{ clear;echo "p e a ma md all | FILENAME"; sh ${HOME}/scripts/logview.sh "${2:-askapache.com}" "${1:-all}"; }
 
function pm()
{ local S I=${1:-3};S=$SSHTTY;echo -en "$R\n";case ${2:-0} in 0)echo -en "${CC[6]}==> $X$I$R" >$S; ;; 3)echo -e "\n\n${CC[4]}:: $X$I$R\n" >$S; ;; esac; }
 
function yes_no()
{ local a YN=65; echo -en "${1:-Answer} [y/n] "; read -n 1 a; case $a in [yY]) YN=0; ;; esac; return $YN; }
 
function l()
{ command ls -AhFp --color=auto "$@"; }
 
function la()
{ du *|awk '{print $2,$1}'|sort -n|tr ' ' "\t"; }
 
function ll()
{ command ls -lABls1c --color=auto "$@"; }
 
function stat1()
{ local D=${1:-$PWD/*}; stat -c %a\ %A\ \ A\ %x\ \ M\ %y\ \ C\ %z\ \ %N ${D} |sed -e 's/ [0-9:]\{8\}\.[0-9]\{9\} -[0-9]\+//g' |tr  -d "\`\'"|sort -r; }
 
function stat2()
{ local D=${1:-$PWD/*}; stat -c %a\ %A\ \ A\ %x\ \ M\ %y\ \ C\ %z\ \ %N ${D} |sed -e 's/\.[0-9]\{9\} -[0-9]\+//g'|tr  -d "\`\'"|sort -r; }
 
#--=--=--=--=--=--=--=--=--=--=--#
# processes
#==-==-==-==-==-==-==-==-==-==-==#
function psu()
{ command ps -Hcl -F S f -u ${1:-$USER}; }
 
function ps()
{ [[ -z "$1" ]] && command ps -Hacl -F S -A f && return; command ps "$@"; }
 
function make_nice()
{ have nice || return; pm "Making Nice $$"; pm 0 0; command renice ${1:-19} -p $$; pd; }
 
function make_ionice()
{ have ionice || return; pm "Making IONice $$"; pm 0 0; command ionice -c${1:-3} -n7 -p $$; pd; }
 
function dump_ps_environment()
{ command ps aux | grep ${USER:0:3} | awk '{print $2}' | xargs -t -ipid cat /proc/pid/environ; }
 
function procinfo1()
{ PI=($(strace -s1 procinfo -a 2>&1|sed -e '/^op/!d' -e '/pro/!d' -e '/= -1/d'|sed -e 's%o.*"/proc/\(.*\)".*% \1%g')); for i in ${PI[*]}; do echo -e "\n---===[  /proc/$i  ]\n" && cat /proc/$i && echo -e "\n\n"; done; }
 
function pss(){
  local U PPS PL PX PXX UUS=( $(command ps uax|awk '{print $1}'|command tail -n +2|sort|uniq) ); UL=$((${#UUS[@]} - 1))
  exec 6>&1; exec > ~/proc.$$
  ps aux | grep ${USER:0:3} | awk '{print $2}' | xargs -t -ipid cat /proc/pid/environ
  for UX in $(seq 0 1 $UUS); do U=${UUS[$UX]}; PPS=( $(pgrep -u ${U}) ); PL=$((${#PPS[@]} - 1));
   for PX in $(seq 0 1 $PL); do PXX=${PPS[$PX]};echo -e "\n\n\n----- PROCESS ID: ${PXX} -----\n\n";cat /proc/${PXX}/cmdline 2>$N6 || echo;echo -e "\n\n"; command tree -Csuflapi /proc/Q/${PXX};done
  done
  exec 1>&6 6>&-; cat ~/proc.$$ | more
}
 
function ashow_motd()
{ [[ -r /etc/motd ]] && echo -e "\n\n${CC[2]}`head -n 7 /etc/motd | tail -n 6`${R}\n"; }
 
function ashow_calendar()
{ [[ -d /usr/share/calendar/ ]] && echo -en "\n${CC[5]}" && ( sed = $(echo /usr/share/calendar/calendar*) | sed -n "/$(date +%m\\/%d\\\|%b\*\ %d)/p" ) && echo -en "${R}"; }
 
function ashow_fortune()
{ [[ -x /usr/games/fortune ]] && echo -en "\n${CC[6]}" && /usr/games/fortune -s && echo -en "${R}"; }
 
function askapache()
{
  ascript_title
  ashow_motd
 
  have who && pm "Users" && pm "Logged In" 3 &&  command who -ar -pld
  pm "Current Limits" 3;command ulimit -a
 
  pm "Machine stats";
  have uptime && pm "uptime" 3 && command uptime
  [[ -d /proc ]] && [[ -f /proc/meminfo ]] && cat /proc/meminfo
  have who && pm "Users" 3 && command who
 
  pm "Networking"
  have ip && pm "interfaces" 3 && ip -o addr|sed -e 's/ \{1,\}/\t/g'
  [[ -r /proc/net/sockstat ]] && pm "Sockets" 3 && head -n 2 /proc/net/sockstat
  have ss && pm "Networking Stats" 3 && ss -s
  have netstat && pm "Routing Information" 3 && netstat -r
 
  pm "Disk"
  pm "Mounts" 3; command df -hai
  #[[ -d /sys/block/ ]] && pm "IO Scheduling" 3; for d in /sys/block/[a-z][a-z][a-z]*/queue/*; do [[ -d $d ]] && tree $d; echo "$d => $(cat $d)";done
  have iostat && pm "I/O on Disks" && iostat -p ALL
 
  pm "Processes";
  pm "process tree" 3;  command ps -HAcl -F S -A f | uniq -w3
  have procinfo && pm "procinfo" 3 && procinfo|head -n 13|tail -n 11
}
 
asetup_colors
asetup_history
askapache
 

Custom bash_profile for Advanced Shell Users originally appeared on AskApache.

I’ve used just about every CMS/Blog/Forum/Trac/Gallery/etc) and really didn’t like a lot of the way they coded… I could use php but I didn’t KNOW php.. so I’ve had to learn php also, and it was tough to learn the advanced class usage and all the other language specific (but similar) constructs for php. It was especially difficult (but fun and challenging) to program so as to be compatible with php4 and php5 (Such is WordPress). But I kept at it, and soon you can decide for yourself what to make of it.

I can code in plenty of languages (bash, lua, windows .bat and vbs, ocaml, big fan of all things shell) and can work my way through C and even sorta somewhat with assembly. Assembly is the hardest, by far, I’m into easy and powerful languages like Python, Javascript, perl, php, ruby, and CGI. I’ve used PHP for a long time to do various things, but never to build software projects like this. Once I noticed WordPress’s core .php files and the excellent programming I wanted to try and learn hot to do it. The WordPress code is some of the best I’ve seen. It appears the way they built it was planned, and not just dreamt up while typing that I can’t help but do. Every time I read through the core code I learn a new trick or very nice way to do something. Those guys are really good, and I think WordPress is going to dominate for a long long time.

The Strategy

The Password Protection (passpro) plugin has a lot of complex stuff going on, especially for a newbie to PHP and WordPress like me, so after refactoring the whole thing at least 5 times I decided to modify my approach, and wrote the AskApache Google 404 Plugin as a way to practice on a simpler piece of code, while at the same time providing a plugin of value. Eventually I stopped thinking I could just code the whole thing in one sit-down with a stream-of-consciousness, and had to instead modularize the code and focus in on each part before moving to the next (I go without a plan because its fun, just not the most productive, but again, I’m not a programmer in the scientific sense.).

So I decided I had to really learn how WordPress Plugins work, filters, hooks, actions, and basically comfortability at reverse-engineering code, (Im a beginner for the last time), and so with the upcoming release of the AskApache Google 404 Plugin I have succeeded in making an incredibly stable plugin. That way I only have to worry about what the aapasspro plugin is doing, instead of trying to fit it into a framework.

AskApache Google 404 Upgrade

I think its rather unusual to develop a nice plugin like this 404 handler merely for the purpose of improving upon another plugin, but hey it worked. As of 08/03/2009 14:06PM EST I have about 1 hour left of finishing touches to release this upgrade. But as you cantell by my badly edited posts, I don’t have a lot of time to myself. An hour here and there is about it. So it could be up to 2 weeks before I actually have the time to commit the release to the repo. On a sidenote, have you checked out Windows 7 News? I’ve been contracted to do some technical work for them and thought they had an excellent site.

But keep in mind, the 404 PLugin is just where I practice for the passpro plugin, which truly does have features that no other software like it has ever had. I understand the technology behind this plugin, and know it would really have a great impact on improving the Web (esp. WordPress) for all of us, I’ve just had to learn how to make it.

AskApache Password Protection

Probably still a couple weeks away, this plugin is the ultimate culmination of apache hackers dreams, at least those on shared servers (who may be interested in learning how to bypass security of said servers).. So this is something I have much too fun with doing what I like to do.. network/protocol-level security. I’ve examined the source code for many software packages that I use or have used to audit a server’s security, and this simple php plugin in most instances can enumerate with accuraccy most of the server’s setup in about a minute. The catch (and the file permission problems I had to find a workaround too) is that this software is launched on the server, not remotely against the server.

Some of the software I examined was whiskers, nessus, nmap, hping, mozilla source, wireshark, ncftp, netcat, etc.. The closest comparison to the socket-level class I’ve hacked together to those is wireshark. Except that wireshark only interprets (captures) the data passing over the wire, while this class does that and in fact sends and receives the data like netcat or nmap. Its really more similar to metasploit, and can easily be used to send hex, binary, ascii, or any type of payload to the remote or local host.

The Upgrades Begin

Well I started working on them a long time ago. Both the Password Protection plugin and the Google 404 plugin needed serious work. And I finally have it all figured out. Essentially I would work on one and finish an upgrade, but I just wasn’t happy with it and I wold start all over again, refactoring the code. So as I put the finishing touches on those 2 plugins keep an eye out. They are major upgrades. I was able to meet all the goals I had for them, and came up with a lot of more improvements during the process.One of the main things I needed was a socket-level class to perform all kinds of checks and tests on. I need this also for my crazy cache plugin, which my blog is currently using , and I have a 2 more really nice pplugins I use that also needed access to a network class. I wrote about what I was doing with fsockopen, and I’ve been improving on that example ever since. I use this class to do some really powerful and exciting stuff, but you’ll see it soon enough. As an indication of ‘getting it right’ for the Password Protection plugin, the plugin will now work on Windows, Apache, IIS, Lighthttpd, and will even work running on a blackberry web server. So now everyone using wordpress can at least get some security()

Many of the the other improvements focus on using the fsockopen class and .htaccess tricks to basically enumerate and discover all the different capabilities of your particular server; That way you can learn about all the features and security that are possible for your specific server, and the securty modules wi8ll be geared for that as well. FINALLY this plugin is going to be stable, and I just cant wait to see how people react when they learn all great capability their Apache-based Server has that they didn’t have a clue about. Its amazing in that sense, and hackers will love theh way it works.. but your server admins will love it even more because its entirely 100% focused on helping you to set your site up (if you have Apache) to keep spammers out, to keep virii-serving robots and their log-hogging exploit requests and CPU/Mem robiing 404 errors off of your servers for real. This will have a noticeable affect to whoever is running the server. As you can tell.. I am pumped!

Apache can only be influenced by the main server configs and by .htaccess files. Not by php, not by perl, and the main configs are almost never accessible to the masses. But .htaccess files are. And many hosting providers allow and enable .htaccess files, a configuration file for your web server. The advanced features and capabilities of Apache were out of reach for most of us, it just wasn’t possible to enumerate or access, and most hosting providers are infamous for their lack of .htaccess (customer) support. This plugin goes around those problems to give the power back to the people.

y creating custom .htaccess files containing unpublished .htaccess tricks and techniques and combining that with the use of socket-level networking from WordPress (PHP) using fsockopen, we can effectively enumerate and discover an incredible amount of features and settings you will be able to control and use with this plugin.

Here are a few examples of the capabilities of this plugin, some of which I believe no other software can do.. (Open source free to copy!).

1. Current Version of Apache (Down to the API Version)
2. List of ALL Modules currently enabled by Apache (Such as Mod_Rewrite)
3. List of ALL Directives enabled by EACH enabled Module.
4. Enumerate .htaccess Overrides, Context Permissions
5. Test for any builtin Handlers (like the status handler screenshot)
6. Configure SSI (http://www.askapache.com/htaccess/advanced-htaccess-ssi.html#htaccess-ssi-security)

March 1, 2009
I would focus on the method that WordPress uses. The code they have now (2.8 bleeding-edge) still isn’t where it needs to be, but this is some difficult stuff and they have a brilliant start, it’ll work.. just a question of when.

The main issue with the password protection plugin working for some people and not others is due to file permission configurations. The plugin attempts to write/modify files in your blog’s root directory.

November 05, 2008
To make a long story short, I downloaded each major release of the apache httpd source code starting at version 1.3.0 and finishing with version 2.2.11, I then compiled each version and built a HTTPD from source for all these apache versions.

• 1.3.0
• 1.3.1
• 1.3.11
• 1.3.12
• 1.3.14
• 1.3.17
• 1.3.19
• 1.3.2
• 1.3.20
• 1.3.22
• 1.3.23
• 1.3.24
• 1.3.27
• 1.3.28

• 1.3.29
• 1.3.3
• 1.3.31
• 1.3.32
• 1.3.33
• 1.3.34
• 1.3.35
• 1.3.36
• 1.3.37
• 1.3.39
• 1.3.4
• 1.3.41
• 1.3.6
• 1.3.9

• 2.0.35
• 2.0.36
• 2.0.39
• 2.0.40
• 2.0.42
• 2.0.43
• 2.0.44
• 2.0.45
• 2.0.46
• 2.0.47
• 2.0.48
• 2.0.49
• 2.0.50
• 2.0.51

• 2.0.52
• 2.0.53
• 2.0.54
• 2.0.55
• 2.0.58
• 2.0.59
• 2.0.61
• 2.0.63
• 2.1.3-beta
• 2.1.6-alpha
• 2.1.7-beta
• 2.1.8-beta
• 2.1.9-beta

• 2.2.0
• 2.2.10
• 2.2.2
• 2.2.3
• 2.2.4
• 2.2.6
• 2.2.8
• 2.2.9
• 2.2.10
• 2.2.11

Then I went through each version and determined the compatible modules for that version, and I’m pretty confident that I was also able to find each and every directive allowed by the compatible modules for that version (including core directives). See .htaccess directive list. Basically I can now test a server using a variety of methods and determine almost 100% accurately what version of Apache (down to the API) is running, what modules (and versions) are enabled, and each and every directive that is allowed or disallowed for that version. So this is so awesome because now we can enable all sorts of additional security features.

Htaccess enabled Modules

Here are most of the modules that come with Apache. Each one can have new commands that can be used in .htaccess file scopes.

mod_actions, mod_alias, mod_asis, mod_auth_basic, mod_auth_digest, mod_authn_anon, mod_authn_dbd, mod_authn_dbm, mod_authn_default, mod_authn_file, mod_authz_dbm, mod_authz_default, mod_authz_groupfile, mod_authz_host, mod_authz_owner, mod_authz_user, mod_autoindex, mod_cache, mod_cern_meta, mod_cgi, mod_dav, mod_dav_fs, mod_dbd, mod_deflate, mod_dir, mod_disk_cache, mod_dumpio, mod_env, mod_expires, mod_ext_filter, mod_file_cache, mod_filter, mod_headers, mod_ident, mod_imagemap, mod_include, mod_info, mod_log_config, mod_log_forensic, mod_logio, mod_mem_cache, mod_mime, mod_mime_magic, mod_negotiation, mod_proxy, mod_proxy_ajp, mod_proxy_balancer, mod_proxy_connect, mod_proxy_ftp, mod_proxy_http, mod_rewrite, mod_setenvif, mod_speling, mod_ssl, mod_status, mod_substitute, mod_unique_id, mod_userdir, mod_usertrack, mod_version, mod_vhost_alias

Debugging HTTP protocol

Check this out! I’m particularly happy about this feature, which outputs an exact trace of any requests made by the plugin (such as during the testing phase) by saving the actual raw data sent out on the wire using fsockopen, RX and TX. This is useful for a number of reasons, viewing your headers, finding Redirect Loops, testing RewriteRules, and following the request hop-by-hop for debugging. The below example shows 2 requests for 2 URIs. The first URI is protected using Digest Authentication, the 2nd shows Basic.

 ______________
|  RAW TRACE   |
==================================================================================================================================
GET /htaccess/index.txt?testing=query HTTP/1.1
Host: www.askapache.com
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1) AA_PassPro/1.9 (http://www.askapache.com/)
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us
Accept-Encoding: gzip, deflate
Connection: close
Referer: http://www.askapache.com/
 
HTTP/1.1 401 Authorization Required
Date: Wed, 22 Jul 2009 06:29:58 GMT
Server: Apache
WWW-Authenticate: Digest realm="do or die", nonce="03328f3ec7c7b", algorithm=MD5, domain="/", qop="auth"
Accept-Ranges: bytes
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 882
Connection: close
Content-Type: text/html; charset=UTF-8
 
GET /htaccess/index.txt?testing=query HTTP/1.1
Host: www.askapache.com
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1) AA_PassPro/1.9 (http://www.askapache.com/)
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us
Accept-Encoding: gzip, deflate
Connection: close
Referer: http://www.askapache.com/
Authorization: Digest username="test",realm="do or die",nonce="03328f3ec7c7b",uri="/htaccess/index.txt?testing=query",
cnonce="82d057852a9dc497",nc=00000001,algorithm=MD5,response="9d476e9ea3",qop="auth"
 
HTTP/1.1 200 OK
Date: Wed, 22 Jul 2009 06:29:58 GMT
Server: Apache
Authentication-Info: rspauth="9051b01ee26dd62b3e2b40dada694f45", cnonce="82d057852a9dc497", nc=00000001, qop=auth
Last-Modified: Tue, 21 Jul 2009 23:56:00 GMT
Accept-Ranges: bytes
Cache-Control: max-age=3600
Expires: Wed, 22 Jul 2009 07:29:58 GMT
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 27
Connection: close
Content-Type: text/plain; charset=UTF-8
```````````````````````````````````````````````````````````````````````````````````````````````````````````````````````````````````
 
 ______________
|  RAW TRACE   |
==================================================================================================================================
GET /htaccess/po.txt?testing=query HTTP/1.1
Host: www.askapache.com
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1) AA_PassPro/1.9 (http://www.askapache.com/)
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us
Accept-Encoding: gzip, deflate
Connection: close
Referer: http://www.askapache.com/
 
HTTP/1.1 401 Authorization Required
Date: Wed, 22 Jul 2009 06:29:58 GMT
Server: Apache
WWW-Authenticate: Basic realm="Po Pimping"
Accept-Ranges: bytes
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 878
Connection: close
Content-Type: text/html; charset=UTF-8
 
GET /htaccess/po.txt?testing=query HTTP/1.1
Host: www.askapache.com
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1) AA_PassPro/1.9 (http://www.askapache.com/)
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us
Accept-Encoding: gzip, deflate
Connection: close
Referer: http://www.askapache.com/
Authorization: Basic adfAGAltcA==
 
HTTP/1.1 200 OK
Date: Wed, 22 Jul 2009 06:29:58 GMT
Server: Apache
Last-Modified: Wed, 22 Jul 2009 05:54:39 GMT
Accept-Ranges: bytes
Cache-Control: max-age=3600
Expires: Wed, 22 Jul 2009 07:29:58 GMT
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 27
Connection: close
Content-Type: text/plain; charset=UTF-8
```````````````````````````````````````````````````````````````````````````````````````````````````````````````````````````````````

.htaccess Directives

AcceptFilter, AcceptMutex, AcceptPathInfo, AccessFileName, Action, AddAlt, AddAltByEncoding, AddAltByType, AddCharset, AddDefaultCharset, AddDescription, AddEncoding, AddHandler, AddIcon, AddIconByEncoding, AddIconByType, AddInputFilter, AddLanguage, AddModuleInfo, AddOutputFilter, AddOutputFilterByType, AddType, Alias, AliasMatch, AllowCONNECT, AllowEncodedSlashes, AllowOverride, Anonymous, Anonymous_Authoritative, Anonymous_LogEmail, Anonymous_MustGiveEmail, Anonymous_NoUserID, Anonymous_NoUserId, Anonymous_VerifyEmail, AssignUserId, AuthAuthoritative, AuthBasicAuthoritative, AuthBasicProvider, AuthDBDUserPWQuery, AuthDBDUserRealmQuery, AuthDBM, AuthDBMAuthoritative, AuthDBMGroupFile, AuthDBMType, AuthDBMUserFile, AuthDefaultAuthoritative, AuthDigestAlgorithm, AuthDigestDomain, AuthDigestFile, AuthDigestGroupFile, AuthDigestNcCheck, AuthDigestNonceFormat, AuthDigestNonceLifetime, AuthDigestProvider, AuthDigestQop, AuthDigestShmemSize, AuthGroupFile, AuthLDAPAuthzEnabled, AuthLDAPBindDN, AuthLDAPBindON, AuthLDAPBindPassword, AuthLDAPCharsetConfig, AuthLDAPCompareDNOnServer, AuthLDAPDereferenceAliases, AuthLDAPEnabled, AuthLDAPFrontPageHack, AuthLDAPGroupAttribute, AuthLDAPGroupAttributeIsDN, AuthLDAPRemoteUserAttribute, AuthLDAPRemoteUserIsDN, AuthLDAPStartTLS, AuthLDAPURL, AuthLDAPUrl, AuthName, AuthType, AuthUserFile, AuthzDBMAuthoritative, AuthzDBMType, AuthzDefaultAuthoritative, AuthzGroupFileAuthoritative, AuthzLDAPAuthoritative, AuthzOwnerAuthoritative, AuthzUserAuthoritative, BS2000Account, BalancerMember, BrowserMatch, BrowserMatchNoCase, BufferedLogs, CGIMapExtension, CacheDefaultExpire, CacheDirLength, CacheDirLevels, CacheDisable, CacheEnable, CacheExpiryCheck, CacheFile, CacheForceCompletion, CacheGcClean, CacheGcDaily, CacheGcInterval, CacheGcMemUsage, CacheGcUnused, CacheIgnoreCacheControl, CacheIgnoreHeaders, CacheIgnoreNoLastMod, CacheLastModifiedFactor, CacheMaxExpire, CacheMaxFileSize, CacheMaxStreamingBuffer, CacheMinFileSize, CacheNegotiatedDocs, CacheRoot, CacheSize, CacheStoreNoStore, CacheStorePrivate, CacheTimeMargin, CharsetDefault, CharsetOptions, CharsetSourceEnc, CheckCaseOnly, CheckSpelling, ChildPerUserId, ContentDigest, CookieDomain, CookieExpires, CookieLog, CookieName, CookieStyle, CookieTracking, CoreDumpDirectory, CustomLog, DAV, DAVDepthInfinity, DAVGenericLockDB, DAVMinTimeout, DBDExptime, DBDKeep, DBDMax, DBDMin, DBDParams, DBDPersist, DBDPrepareSQL, DBDriver, Dav, DavDepthInfinity, DavGenericLockDB, DavLockDB, DavMinTimeout, DefaultIcon, DefaultLanguage, DefaultType, DeflateBufferSize, DeflateCompressionLevel, DeflateFilterNote, DeflateMemLevel, DeflateWindowSize, Directory, DirectoryIndex, DirectoryMatch, DirectorySlash, DocumentRoot, DumpIOInput, DumpIOOutput, EnableExceptionHook, EnableMMAP, EnableSendfile, ErrorDocument, ErrorLog, Example, ExpiresActive, ExpiresByType, ExpiresDefault, ExtFilterDefine, ExtFilterOptions, ExtendedStatus, FancyIndexing, FileETag, Files, FilesMatch, FilterChain, FilterDeclare, FilterProtocol, FilterProvider, FilterTrace, ForceLanguagePriority, ForceType, ForensicLog, GprofDir, GracefulShutdownTimeout, Group, Header, HeaderName, HostNameLookups, HostnameLookups, ISAIPFakeAsync, ISAPIAppendLogToErrors, ISAPIAppendLogToQuery, ISAPICacheFile, ISAPIFakeAsync, ISAPILogNotSupported, ISAPIReadAheadBuffer, IdentityCheck, IdentityCheckTimeout, IfDefine, IfModule, IfVersion, ImapBase, ImapDefault, ImapMenu, Include, IndexIgnore, IndexOptions, IndexOrderDefault, IndexStyleSheet, KeepAlive, KeepAliveTimeout, LDAPCacheEntries, LDAPCacheTTL, LDAPCertDBPath, LDAPConnectionTimeout, LDAPOpCacheEntries, LDAPOpCacheTTL, LDAPSharedCacheFile, LDAPSharedCacheSize, LDAPTrustedClientCert, LDAPTrustedGlobalCert, LDAPTrustedMode, LDAPVerifyServerCert, LanguagePriority, Limit, LimitExcept, LimitInternalRecursion, LimitRequestBody, LimitRequestFields, LimitRequestFieldsize, LimitRequestLine, LimitXMLRequestBody, Listen, ListenBacklog, LoadFile, LoadModule, Location, LocationMatch, LockFile, LogFormat, LogLevel, MCacheMaxObjectCount, MCacheMaxObjectSize, MCacheMaxStreamingBuffer, MCacheMinObjectSize, MCacheRemovalAlgorithm, MCacheSize, MMapFile, MaxClients, MaxKeepAliveRequests, MaxMemFree, MaxRequestsPerChild, MaxSpareServers, MaxSpareThreads, MaxSpareThreadsPerChild, MaxThreads, MetaDir, MetaFiles, MetaSuffix, MimeMagicFile, MinSpareServers, MinSpareThreads, ModMimeUsePathInfo, MultiviewsMatch, NWSSLTrustedCerts, NWSSLUpgradeable, NameVirtualHost, NoProxy, NumServers, Options, PassEnv, PerlAccessHandler, PerlAuthenHandler, PerlAuthzHandler, PerlChildExitHandler, PerlChildInitHandler, PerlCleanupHandler, PerlDispatchHandler, PerlFixupHandler, PerlFreshRestart, PerlHandler, PerlHeaderParserHandler, PerlInitHandler, PerlLogHandler, PerlModule, PerlPassEnv, PerlPostReadRequestHandler, PerlRequire, PerlRestartHandler, PerlSendHeader, PerlSetEnv, PerlSetVar, PerlSetupEnv, PerlTaintCheck, PerlTransHandler, PerlTypeHandler, PerlWarn, PidFile, Port, Protocol, ProtocolEcho, Proxy, ProxyBadHeader, ProxyBlock, ProxyDomain, ProxyErrorOverride, ProxyFtpDirCharset, ProxyIOBufferSize, ProxyMatch, ProxyMaxForwards, ProxyPass, ProxyPassInterpolateEnv, ProxyPassMatch, ProxyPassReverse, ProxyPassReverseCookieDomain, ProxyPassReverseCookiePath, ProxyPreserveHost, ProxyReceiveBufferSize, ProxyRemote, ProxyRemoteMatch, ProxyRequests, ProxySet, ProxyStatus, ProxyTimeout, ProxyVia, RLimitCPU, RLimitMEM, RLimitNPROC, ReadmeName, Redirect, RedirectMatch, RedirectPermanent, RedirectTemp, RemoveCharset, RemoveEncoding, RemoveHandler, RemoveInputFilter, RemoveLanguage, RemoveOutputFilter, RemoveType, RequestHeader, Require, RewriteBase, RewriteCond, RewriteEngine, RewriteLock, RewriteLog, RewriteLogLevel, RewriteMap, RewriteOptions, RewriteRule, SSIAccessEnable, SSIEndTag, SSIErrorMsg, SSIStartTag, SSITimeFormat, SSIUndefinedEcho, SSLCACertificateFile, SSLCACertificatePath, SSLCADNRequestFile, SSLCADNRequestPath, SSLCARevocationFile, SSLCARevocationPath, SSLCertificateChainFile, SSLCertificateFile, SSLCertificateKeyFile, SSLCipherSuite, SSLCryptoDevice, SSLEngine, SSLHonorCipherOrder, SSLLog, SSLLogLevel, SSLMutex, SSLOptions, SSLPassPhraseDialog, SSLProtocol, SSLProxyCACertificateFile, SSLProxyCACertificatePath, SSLProxyCARevocationFile, SSLProxyCARevocationPath, SSLProxyCipherSuite, SSLProxyEngine, SSLProxyMachineCertificateFile, SSLProxyMachineCertificatePath, SSLProxyProtocol, SSLProxyVerify, SSLProxyVerifyDepth, SSLRandomSeed, SSLRequire, SSLRequireSSL, SSLSessionCache, SSLSessionCacheTimeout, SSLUserName, SSLVerifyClient, SSLVerifyDepth, Satisfy, ScoreBoardFile, Script, ScriptAlias, ScriptAliasMatch, ScriptInterpreterSource, ScriptLog, ScriptLogBuffer, ScriptLogLength, ScriptStock, SecureListen, SendBufferSize, ServerAdmin, ServerAlias, ServerLimit, ServerName, ServerPath, ServerRoot, ServerSignature, ServerTokens, SetEnv, SetEnvIf, SetEnvIfNoCase, SetHandler, SetInputFilter, SetOutputFilter, StartServers, StartThreads, Substitute, SuexecUserGroup, ThreadLimit, ThreadStackSize, ThreadsPerChild, TimeOut, Timeout, TraceEnable, TransferLog, TypeAuthDBMUserFile, TypesConfig, UnsetEnv, UseCanonicalName, UseCanonicalPhysicalPort, User, UserDir, VirtualDocumentRoot, VirtualDocumentRootIP, VirtualHost, VirtualScriptAlias, VirtualScriptAliasIP, Win32DisableAcceptEx, XBitHack, allow, deny, order, php_admin_flag, php_admin_value, php_flag, php_value

You can view the plugins home page, old, or view it on the wordpress.org site.

An AskApache Plugin Upgrade to Rule them All originally appeared on AskApache.