AskApache Web Devleopment

SEO Secrets of AskApache.com

AskApache — Tue, 29 Apr 2008 01:44:02 +0000

First let me say that I am not into SEO, I don’t read the research, I don’t participate in the SEO community, I only have an interest in regards to the technology used by the search engines. All websites are hosted on servers, mostly Apache, and that is the primary topic of this blog. During the past year of this blog, my research into non-seo areas has turned up some very valuable SEO techniques.. All of them legal, ethical, and genuinely good for the Internet at large.

Some Background

I started this blog in January 2007, my first foray into blogging, and I’ve been extremely successful at achieving top ten google rankings and maintaining on average 15K unique visitors/day (per google analytics) 85% of which come from search engine traffic.

NOTE: I take it for granted that anyone reading AskApache is an expert of some skill, if you aren’t I apologize, I can’t waste time on the easy stuff.

Prerequisite SEO

There are literally hundreds of thousands of SEO articles on the net, 99.9% of which are absolute garbage. Especially in the sense that they just repeat the same 10 year old stuff. However, to do any kind of advanced SEO like I am going to discuss in this article, I am assuming that you, the intelligent reader, has already read those and has a basic understanding of SEO fundamentals like meta tags, titles, keywords, etc.

First, Great Content

The foremost and most important step in achieving any kind of traffic is to produce great content. I’m sure you’ve heard that a million times, but let me break it down how I perceive it. Before I even started to mess with SEO for AskApache.com I began by writing articles. At that point I didn’t have a clue what my blog was going to be about or even if I was going to be doing it after a week.

What is Content

For me, being a top-paid professional web developer, I spend about 80% of my time doing research. I think that is a bit uncommon, but its a throwback from the 10 years I spent in the network/computer security field, where research is 99% of the job, a story for another time perhaps.

So the research I was doing at that time was about best-practice standards-based web design, mainly XHTML Strict, CSS, and unobtrusive javascript. Each of those subjects has become near and dear to my heart, and each should also be mandatory learning for anyone interested in SEO. The best advice I can give towards that end is checking out the CSS, Javascript, and XHTML Strict source code for this page and site. And of course the holy W3.org.

In addition to striving to master those 3 subjects, I was also and always will be researching web programming languages like PHP, Ajax, Ruby, and Server Technology like Apache. Although I should note that my research into Apache and server technologies is more of a hobby than a job requirement, also a throwback to my days in the security industry and of course my love for open source software.

My Content

So basically I was spending 25% of my time at work actually working, and the other 75% of the time I would research how to do something better, faster, the best. Incredibly, I discovered or re-discovered a ton of tips, tricks, and methods to aid me in my work. I was learning so much valuable information that I joined a couple of forums to discuss them and get feedback on making them even better. Soon I realized that I was one of a small few who actually post content to a forum instead of just questions, so I decided to write my tutorials down on a blog, and AskApache was born.

So that is why this blog is comprised of almost 100% tutorials, and why almost all of them are completely original works you won’t find elsewhere. That’s how I create content, but you might do something different. Whatever it is that you do for content, just make sure you are providing VALUE with everything you do. Not to everyone, just stuff that you would consider to have value if you were reading it.

Second, Great Site

Ok so I had 10 or so great articles that I knew would provide value for many web developers, but so what? Nobody cares you know.. That’s when I decided to take a closer look at the software that was running my new blog, WordPress, and I’ve been hacking the code ever since on my never-ending quest to be the best and know the most advanced web development. You’ll see why in a couple paragraphs.

I Mean, a Really Great Site

By great, I mean you need to make it incredibly user-friendly. Every design and development decision you make should be about the visitor. THATS the number one key to success on the net, regardless of endeavor. Here is a list of things you definately need to have before you do SEO, I’m not listing obvious stuff like descriptions, titles, and good writing.

Intuitive and circular, your website should be a spider-web of urls.
Easy to read, plenty of white-space, design is your decision but I like minimalistic.
Super fast rendering. You need effective caching and optimization.
A very helpful 404 error page, hopefully never seen.

Focus in on your URL’s

Many sites that use a CMS of some kind, be it Drupal or WordPress, have hundreds or thousands of URL’s even if they only have 10 actual posts/articles.

Removing Duplicate Content

You’ve all heard this before, but almost no-one has taken it to the level I am going to discuss. Bear with me.

Removing duplicate content is actually a very straightforward process if you know what you are doing, and if you don’t, well that’s why I’m going to quickly explain how to really do a good job.

Locate Duplicate Content and URLS

People misunderstand that you should just not repeat the same paragraph in a different article, that is partially true, but the main impact this has on your site is if you can access the same article from more than a single URL.

I hope you realize you MUST use pretty urls like my site and not codey looking ones with question marks. You can find any potential duplicate urls on wordpress with the rewriterules plugin. Also look at Google’s webmaster tools to look for any duplicate urls, and you can use xenus link sleuth tool as well.

Remove Duplicate Urls with .htaccess

Once you’ve found duplicate urls, you need to instruct google and other search engine robots to be redirected to the correct url. By doing a 301 redirect you tell the search engines NOT to index the bad url, only the good one. Below are some of the .htaccess code I use on this site to accomplish this technique, this is gold I myself use so pay attention. It works.

301 Redirects with mod_rewrite

First lets start with one everyone should know, and the most common, to www or not to www?

RewriteCond %{HTTP_HOST} !^www\.askapache\.com$ [NC]
RewriteRule ^(.*)$ http://www.askapache.com/$1 [R=301,L]

Its a highly rare individual who has seen this one, which forces requests for .html/ to .html

RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /[^\.]+\.html/\ HTTP/ [NC]
RewriteRule ^(.*)\.html/$ http://www.askapache.com/$1.html [R=301,L]

301 Redirects without mod_rewrite

For the rest of the duplicate urls that you find, I like to use Redirect and RedirectMatch.

This redirects requests that start with an & or /& or /( to my homepage.

RedirectMatch 301 ^/&(.*)$ http://www.askapache.com/
RedirectMatch 301 ^/&(.*)$ http://www.askapache.com/
RedirectMatch 301 ^/([\(]+)(.*)$ http://www.askapache.com/

This redirects requests with //whatev to /whatev

RedirectMatch 301 ^//(.*)$ http://www.askapache.com/$1

But this is just a brief look at what you will have to spend some time on. There are detailed guides to doing this with mod_rewrite and using Redirect on my blog. Its time now for some real SEO tips. The heart of the matter, as it were.

Wrap It Up

So I realize that was brief, so I want to really stress 2 things or you won’t take away much from part 1.

Locate Duplicate URLS

This is truly one of the most important things in my personal experience. I personally take this as extreme as I can, I regularly grep my access files, mod_security and error log files looking for bad URLS. I am always checking them out to see if someone has a bad link to me somewhere, or if someone just typed it in wrong. If its a bad link on a site, I will very politely attempt contacting the webmaster about it until they fix it.

Even I, with my many colorful years of Internet travel, was caught off-guard by the variety and creativity and the sheer number of urls people are using to link to my site. I found that often bad links would be published because my URL was just too long, so I shortened the URL’s. Now of course bad links can’t really even touch my site with all my 301’s in place.

Besides grepping your server’s logs, the 2nd best place to locate duplicate urls or just plain wrong urls is by using Googles free webmaster tools. They keep track of all the bad urls linking to your site and allow you to download this data in a .csv spreadsheet format. The first time I checked into this I found over 1,000 bad links, after a couple months with my RewriteRules and 301 Redirects, I’ve narrowed the list down to under 50 most months. That is a powerful reason to use 301 Redirects, as we’ll really get into in part 2.

301 Redirect Bad URLS

Finding the bad urls takes some time, a couple hours even, and then the whole reason you do that is to be able to create 301 Redirects for all of those bad urls to good urls.

One reason that I wasn’t even aware of until several months ago is that when Googlebot locates a bad URL for your site, it tries to access it, and if you haven’t planned for this in advance, your page most likely will return a 200 OK status header, or if you are lucky a 404 Not Found error, both of which really hurt you.

Basically, a 200 response will produce duplicate content in 99.9% of the time, and 404 responses will whisper to Google’s algorithms that you don’t know what you are doing and your site isn’t authoritative. 200 means google will index your site, 404 means google won’t index your site, but it also won’t give up trying for awhile, which takes away from your real urls.

What a 301 Response tells Google

301 Responses were practically invented for user-agents/clients/web-crawling robots like google. They instruct the client, whether that be a persons browser or a googlebot, that the resource/page that they are looking for is actually at a different URL. This is an authoritative response that makes googlebot and other search engines ecstatic because now they can give up on the 200 and 404 responses that didn’t really give them an answer either way.

On the other hand, a great 404 can and should be just as powerful as a 301, but hardly anyone uses them in the correct way according to HTTP 1.1 or 1.0 Specifications. We’ll tear that subject apart further down the road.

I’ll leave this topic for now with one last idea, 301 Redirects when implemented and used correctly, actually redirect the page rank and search engine ranking for itself to the redirected / correct url. That means if you have 1000 unique links pointing to your article, and all of them are incorrect in some way, if you can 301 redirect all of those bad links to your correct link you now have 1000 new good links! It has to be done right and in a classy way though of course.

Now that you have content and a great site, its time to SEO like a mofo.

I just have a few things left before I publish Part 2, which goes into the actual details and SEO Secrets I use. This intro post was a neccessary evil I’m afraid.

Random Posts

seo google searchengineoptimization robots.txt wordpress seo-tips seo-tricks

Mod_Security .htaccess tricks

AskApache — Wed, 23 Apr 2008 06:17:01 +0000

.With over 70% of all attacks now carried out over the web application level, organizations need as much help as they can get in making their systems secure. WAFs are deployed to establish an external security layer that increases security, detects, and prevents attacks before they reach web applications.

« Apache Variable fun (mod_env) | .htaccess Tutorial Index | SetEnvIf and SetEnvIfNoCase Examples

Target Audience:

Web Server Administrators
Web security Adminis
Security consultants and other ballers.
Web Developers

Laying the smack down on attacks

ModSecurityâ„¢ is an Apache Module just like mod_rewrite that is in fact a Web Application Firewall, providing access to every tiny bit of a HTTP Connection. HTTP Headers, Cookie and Post Payloads in their entirety, XML-RPC calls from Ajax, protocol and connection information, etc… its totally stacked.

Mod_Security uses Regex and .htaccess / httpd.conf directives similar to mod_rewrite, allowing for complete control from within .htaccess files and httpd.conf blocks.

Contents

mod_security + mod_rewrite

mod_security is the missing piece if all you know is mod_rewrite. This gives you the ability to scan ALL messages received by your website, including POST, Trackbacks, Pings, Ajax XMLHTTP calls, etc. It lets you create your own rules so that you can stop spam and prevent web application, protocol, and server attacks.

Mod_Security has the ability to parse entire POST_PAYLOADS, specific and individual POST/GET arguments… This is a spammers worst nightmare, and I’m going to make that nightmare reality for them.

Block Spam by examining POST form fields

mod_security gives you the option to block, redirect, handle using an errordocument, PAUSE, close, and chain connections. If someone is spamming your blog from many different IP’s, but they often use the same keywords in a certain field of your form (like .blackjack. in the url field) mod_security lets you examine that specific url field and block all connections that contain the regexp pattern of your choice.

Enable mod_security - DreamHost

To enable mod_security, login to the DreamHost panel and navigate to the “Manage Domains” area, Edit your site and enable the extra security option.

DreamHost, has set itself apart as being the top web host IMHO. They’ve provided the option to enable an Apache module called mod_security for any of your hosted domains. The sysops and tech over there are really doing a great job of staying true to the industry, they are web hosts, not some corporate outsourced bought-out thing. Anyways thanks DH!

Disabling mod_security conditionally per IP

This will make sure that you aren’t processed by mod_security, but this only works if you have a static IP (Get your IP information). Just add this towards the top of your .htaccess file. before your mod_security code. Setting this variable causes the module to be disabled for this specific IP address, this means you won’t run into any problems while posting yourself..

SetEnvIfNoCase Remote_Addr ^208\.113\.183\.103$ MODSEC_ENABLE=Off
 
# You can use multiple SetEnvIf directives to control it further.  
# This only turns it off for your IP + a POST request method.
#
# SetEnvIf Remote_Addr ^208\.113\.183\.103$ MODSEC_ENABLE=Off
# SetEnvIf Request_Method !^POST$ MODSEC_ENABLE=On

Disabling mod_security with .htaccess Authorization

So I did some experimenting to see if there was an alternative way to disable mod_security for the users out there without a static IP address. I found a couple silly solutions that suggested you simply modify your browsers User Agent request header, but thats not very safe is it? No. So I came up with using Basic Authorization.

First you need to setup password protection for the directory that you want mod_security disabled for. In the case of WordPress blogs, you can use the AskApache Password Protect Plugin to get setup. Or you can generate a new htpasswd.

Adding password protection for mod_security bypass

AuthName "htaccess password prompt"
AuthType Basic
AuthUserFile /fullpath-to/.htpasswd
Require valid-user

Magic Authorization shut-off using .htaccess

When you login using this authentication, the environment variables REMOTE_USER and AUTH_TYPE are set (though its hard to find them). Since these are unique to you specifically and hard to spoof, use them to turn off mod_security only after you login. This is added up at the top of your mod_security rules.

SecFilterSelective REMOTE_USER "^yourusername$" "allow"
 
# More variables you can experiment with
# HTTP_Authorization|AUTH_TYPE 
#
# You may have to add this to your mod_rewrite code
#RewriteRule .* - [E=REMOTE_USER:%{HTTP:Authorization}]

AskApache’s MOD_SECURITY config for DreamHost

Custom mod_security .htaccess code

Heres how I start my mod_security code, keep in mind I have a lot to learn.

### ASKAPACHE MOD_SECURITY ###

# Turn the filtering engine On or Off or DynamicOnly for cgi/php/etc
SecFilterEngine On
 
# Only log suspicious requests
SecAuditEngine RelevantOnly
 
# Goes up to 9 but at 2 its overwhelming trust me
SecFilterDebugLevel 0
 
# Make sure that URL encoding is valid
SecFilterCheckURLEncoding On
 
# Unicode encoding check
SecFilterCheckUnicodeEncoding Off
 
# Should mod_security inspect POST payloads
SecFilterScanPOST On
 
# The default rule to apply to inherited rules
SecFilterDefaultAction "deny,log,status:500"

Minimal httpd.conf or .htaccess sample from source


 
# Enable ModSecurity
SecFilterEngine On
 
# Reject requests with status 403
SecFilterDefaultAction "deny,log,status:403"
 
# Some sane defaults
SecFilterScanPOST On
SecFilterCheckURLEncoding On
SecFilterCheckUnicodeEncoding Off
 
# Accept almost all byte values
SecFilterForceByteRange 1 255
 
# Server masking is optional
# SecServerSignature "Microsoft-IIS/5.0"
 
# Designate a directory for temporary files
# storage. It is a good idea to change the
# value below to a private directory, just as
# an additional measure against race conditions
SecUploadDir /tmp
SecUploadKeepFiles Off
 
# Only record the interesting stuff
SecAuditEngine RelevantOnly
# Uncomment below to record responses with unusual statuses
# SecAuditLogRelevantStatus ^5
SecAuditLog logs/modsec_audit.log
 
# You normally won't need debug logging
SecFilterDebugLevel 0
SecFilterDebugLog logs/modsec_debug.log
 
# Only accept request encodings we know how to handle
# we exclude GET requests from this because some (automated)
# clients supply "text/html" as Content-Type
SecFilterSelective REQUEST_METHOD "!^(GET|HEAD)$" chain
SecFilterSelective HTTP_Content-Type "!(^application/x-www-form-urlencoded$|^multipart/form-data;)"
 
# Do not accept GET or HEAD requests with bodies
SecFilterSelective REQUEST_METHOD "^(GET|HEAD)$" chain
SecFilterSelective HTTP_Content-Length "!^$"
 
# Require Content-Length to be provided with
# every POST request
SecFilterSelective REQUEST_METHOD "^POST$" chain
SecFilterSelective HTTP_Content-Length "^$"
 
# Don't accept transfer encodings we know we don't handle
SecFilterSelective HTTP_Transfer-Encoding "!^$"

Block WordPress Spam Forever!

Ok this is my first attempt at this, and I am really excited about the possibilities! This example goes inside your mod_security block towards the bottom, and it sets up a default action for each of the filters below it. This denies the connection, doesn’t log it, and issues a 403 Forbidden Status code, which causes my Apache ErrorDocument to be displayed. This ErrorDocument can be a blank page to minimize bandwidth, or it can be a cgi perl type of script that send you an email, whatever. Many people use different Status Codes for different situations, some like 400, 412, 406, and 410 for spammers. Others prefer 503. You can see all 57 that are available for anyone running Apache and choose your own.


SecFilterSignatureAction deny,nolog,status:403
SecFilterSelective ARG_url "casino|ringtone|lyrics"
SecFilterSelective ARG_comment_post_ID "^$"
 
# reject blog spam from all POST and GET fields
SecFilterSelective ARGS "blockspam|blockspam|blockspam|blockspam|blockspam|blockspam|blockspam|blockspam \
blockspam|blockspam|blockspam|blockspam|blockspam|blockspam|blockspam|blockspam|blockspam|blockspam \
nomorespam|nospam"

The FilesMatch Directive specifies that these rules only apply to these files.
The ARG_url line says that if those words appear in the form field with the id/name of "url", than deny them.
The ARG_comment_post_ID line denies requests that have an empty comment_post_ID field, which a surprisingly large number of spammers forget to add.
The final multi-line section searches ALL get and post items for any of these keywords.

Force Any Connections to be Paused a set number of ms

This is just an example to show you how cool this module is, you really shouldn’t every do this except for very specific instances, because it will end up consuming your servers resources and making a ddos attack more likely. This example forces anyone who doesn’t come from the askapache.com site to have their connection delayed 5000 ms, then processing continues.

SecFilterSelective "HTTP_REFERER" "askapache\.com" log,pass,pause:5000

Only Allow Certain REQUEST_METHODS

I’ve created a free online scanner that you can use to scan your site and see how it handles all 27 REQUEST_METHODS.

# Sends matching requests a 405 Method Not Allowed Status Code
SecFilterSelective REQUEST_METHOD "!^(GET|HEAD|POST|OPTIONS)$" "deny,auditlog,status:405"

Debugging and Logging

First its my best guess that DreamHost is running ModSecurity v1.9.4 so its pretty darn difficult to find information about this modules directives and how to use it. So debugging through the use of trial and error and logging is the best or only way to figure it out.

On DreamHost we lucked out

If you have already been using DreamHosts “extra security” option and use your shell or check your error logs, you will have seen plenty of verbose messages about something or other being blocked. Thats mod_security doing its thing but it takes up resources on everyones servers and makes your error log close to unusable.

Control mod_security logging to your error log

Each filter can have its own actions, so turn logging on (log, auditlog) only for those you want, turn off (nolog, noauditlog) for anything else.

# Not logged
SecFilterDefaultAction "deny,nolog,noauditlog,status:500"
 
# Logged but not as verbose.
SecFilterDefaultAction "deny,nolog,auditlog,status:500"

log: Indicates that a successful match of the rule needs to be logged.
noauditlog: Indicates that a successful match of the rule should not be used as criteria whether the transaction should be logged to the audit log.
nolog: Prevents rule matches from appearing in both the error and audit logs.

Turn Off/On Logging JUST for your IP Address

This is handy when you want to test your rules.

# Turn logging of for your IP
SecFilterSelective REMOTE_ADDR "208\.113\.183\.103" "nolog,noauditlog,pass"
 
# Turn logging on just for your IP
SecFilterSelective REMOTE_ADDR "!^208\.113\.183\.103" "nolog,noauditlog,pass"
SecFilterSelective REMOTE_ADDR "208\.113\.183\.103" "log,auditlog,pass"

Mod_Security Directives for DreamHost

SecFilter: The filtering expression
SecFilterDebugLog: The filename of the filter debugging log file
SecFilterDebugLevel: The level of the debugging log file verbosity
SecFilterSelective: The variable representing areas where filtering is wanted, the filtering regular expression and optional action to take on match
SecFilterEngine: On, Off, or DynamicOnly to determine when will request be filtered
SecServerResponseToken: On or Off to set whether the mod_security token will appear in the server signature
SecFilterScanPOST: On or Off to set whether a request body will be processed
SecFilterDefaultAction: The default action to take on rule match
SecFilterSignatureAction: Base action template for signatures that follow this directive
SecFilterInheritance: On or Off to set whether rules from the parent context will be inherited
SecAuditEngine: On, Off, RelevantOnly or DynamicOrRelevent to determine the level of audit logging
SecAuditLog: The filename of the audit log file
SecUploadDir: The path to the directory where uploaded files should be stored
SecUploadKeepFiles: On or Off to choose whether to keep the uploaded files or not
SecUploadApproveScript: The path to the script that will be called to approve every uploaded file
SecFilterCheckURLEncoding: On or Off to set whether URL encoding validation will be performed
SecFilterCheckUnicodeEncoding: On or Off to set whether Unicode encoding validation will be performed
SecFilterForceByteRange: The first and the last byte value of the range that will be accepted
SecChrootDir: The path of the directory to which server will be chrooted
SecChrootLock: The filename of the lock file used during the chroot process, defaults to “logs/modsec_chroot.lock”
SecServerSignature: The new signature of the server
SecFilterNormalizeCookies: On or Off to determine whether cookie values will be normalized for testing, defaults to On
SecFilterCheckCookieFormat: On or Off to determine whether cookie format will be checked. Defaults to On
SecFilterCookieFormat: version of the Cookie specification to use for parsing. Possible values are 0 and 1.
SecCharset: Configures the charset
SecFilterImport: imports a rule from the parent configuration context.
SecFilterRemove: removes a rule that was inherited from the parent configuration context.
SecFilterInheritanceMandatory: when this directive is set to On then the rules in the parent context cannot be removed from a child context.
SecAuditLogType: whether to use the old audit log format (Serial) or new (Concurrent)
SecAuditLogStorageDir: path to the audit log storage area; absolute, or relative to the root of the server
SecAuditLogParts: list of audit log parts that go into the log.
SecAuditLogRelevantStatus: regular expression that will be used to determine if the response status is relevant for audit logging
SecFilterActionsRestricted: whether to allow rules to override SecFiltersDefaultAction configuration

Just remember each version is different, I think most of the directives in the above list are allowed on DreamHosts install, but I’m not 100%, I’m sure they’ll be upgrading soon anyway.

How I got Started with mod_sec

Subject: mod_security denying blog post

Is there any way you could modify the mod_security rules for me? For the past 2-4 months I have been having a problem on WordPress where I go to edit a post and when I hit submit and POST the changes it gives me a 503 Message, which is odd to use a 503 Service Temporarily Unavailable, that threw me off for at a month thinking the service was unavailable.

Basically this only happens to about 5 of my posts, and almost all of them have something to do with php code, or some other type of programming language. An example is when I try to edit custom-phpini-tips-and-tricks. So what I’ve had to do is go into phpmyadmin and manually edit the database.. I don’t want to turn off the mod_security for my site because I am getting hit all the time by malicious looking bots, is that the only way around this? I have no clue, but if it helps that IP 64.233.167.99 is static for me, and this problem only occurs when posting to the /wp-admin/post.php file.

Reply from DreamHost Support
Unfortunately, the mod_security rules need to be changed on all machines and Apache instances if they are changed on just one as our admins like to keep the servers in sync. So getting mod_security modified isn’t really something that can be done easily. I’ll put in a suggestion as we do host a lot of WordPress blogs, but I can’t guarantee that we’ll be able to do it very quickly.

I know you don’t want to turn mod_security off, but honestly it’s your best bet if you want to be able to post without 503’s right now. If “64.233.167.99″ is always your IP, might I suggest setting up an .htaccess rule in your wp-admin folder to deny all traffic to that folder except for your IP address? I know that doesn’t work if you’re on the road - although adding IPs to allow to an .htaccess file is pretty easy to do - but it will keep the malicious folks out.

Example httpd.conf mod_security rules

Some of the attacks this Apache module has the ability to smack with its unique positioning within Apache HTTP.

HTTP protection - detecting violations of the HTTP protocol and a locally defined usage policy
- SQL Injection
- Cross-Site Scripting (XSS)
- OS Command execution
- Remote code inclusion
- LDAP Injection
- SSI Injection
- Information leak
- Buffer overflows
- File disclosure
Common Web Attacks Protection - detecting common web application security attack
Automation detection - Detecting bots, crawlers, scanners and other surface malicious activity
Trojan Protection - Detecting access to Trojans horses
Errors Hiding - Disguising error messages sent by the server

mod_security links

DreamHost Version 1.9.5 Manual
Reference of Actions, Commands, etc. - Official Site
version 1.9 got_root rules
mod-security-users - mailing list
mod_security online rules generator - Noel Jackson
An Intro to mod_security - Atomic Playboy
Get mod_sec training

MOD_SECURITY: The Most Powerful Server-Side Security Technology I’ve Seen

htaccess Guide Sections

« Apache Variable fun (mod_env) | .htaccess Tutorial Index | SetEnvIf and SetEnvIfNoCase Examples

Random Posts

mod_security -htaccess -MODSEC -SetEnvIf -SecFilter -SecAudit

Undetectable Sniffing On Ethernet

AskApache — Mon, 14 Apr 2008 18:04:09 +0000

This article is NOT about promisc network interfaces, it is about passively sniffing data off the wire without sending/transmitting any data out onto the wire. No transmitting, just receiving. I have been in some tight spots where I had to sniff data off the wire, like a password or username, or capture some packets off the wire and programmatically preform actions based on the captured packets.

Unexpectedly, I stumbled on a method to passively capture packets from the wire without transmitting packets. This is surprising because it is achieved using only software and not messing with the hardware, which is the normal way to passively sniff packets. I’ve only achieved this running various flavors of linux and BSD.

Note: You may be looking for ethernet capture implementations using the best open source network debugging tool ever created, WireShark

Commands to become undetectable on linux/BSD

Bring up the ethernet interface with no IP address and arp ignore mode (stealth mode).

$ ifconfig eth0 hw ether 00:00:00:00:00:00 promisc
$ ifconfig eth0 -arp up

Delete the inet and inet6 address, prevent stack routing.

$ route -nv // should produce nothing, same with netstat -r

what does this do?

You can’t route packets to an interface without an entry in the routing table.
You can’t create an entry in the routing table without an IP address assigned to the interface.

Packet Capture Software

Capture all the packets that pass the wire using Wireshark, the ONLY packet capturing tool I recommend.

I also like using tethereal in a shell script and egrep the output for something I’m looking for:

a basic authorization string
a certain bootfile being called using tftp
spanning trees and switch communication
etc.

Then based on the output I’ll have the shell script perform an action like append to a file or send me a text message to my cell. This could be called an inefficient or sloppy way to do this, but it works great when I’m booting from knoppix or I just need to grab some packets silently.

Caveats and Notes

I did notice that after I had received 490MB of data on my eth0 device, it actually did say that it had Transmitted 12bytes.. so this method is not 100% silent, but without a route or an address it can’t really write data anywhere, so I’m still wondering where those 12bytes went.

I suppose you could create a dummy device in a chroot or something and pass the output from your promisc NIC to the dummy which would route the packets along a GRE or a wireless device or something. I will try something like this for a syslog server.

NOTE: You should also check out the Arp stuff article for more in-depth info about this topic.

Q&A: from a forum discussion

i’m not that familar with linux, so i don’t know if you corrected this with one of these commands, but aren’t there many tools that simply check if the ethernet interface is in promisc. mode? so what’s undetected about this?

Those tools are very easily circumvented. The reason is that they rely on set patterns of behaviour that are common to programs like nmap, ettercap, etc.. This method circumvents every single one of them. Most of those progs rely on sending arbitrary data across the wire and looking for set patterns of replies that indicate a promisc eth. The simple thing about this method is that you are basically stripping the IP stack of its fallback or default settings.

If you understand the definition of promisc mode, and understand how the IP stack in all major OS’s use routeing, arp, MACs, and basic TCP/IP 101, the concept doesn’t seem special.

The key is really that the listening promisc interface has no address, has no routes to any networks or hosts. An ip stack set up this way is not a like a listening service. Its more like a logging host in a honeynet with the TX wires cut off.

Something about passive sniffing: etheral mailing list

If you sniff passive you dont send any data frames? All i know is that a switch only send data based on mac address (and recv port). Where a hub just sends it out to everyone. So how would you start sniffing (on a switch) without doing smth like arp poising (sending data frames)? Also, your MAC and ip gets sent out every x seconds by ARP requests right? Or did i misunderstood the question?

00:00:00:00:00:00 - this is not an invalid mac address? Are you sure you can sniff other computers with this configuration? (diferents computers like yours)

You definately could NOT use this method to use an active sniffing program like ettercap or perform an mitm or something, the whole goal of this method is to NOT transmit a single byte.

The reason you set the mac to 00:00:00:00:00:00 is the same reason you could also set the mac to aa:bb:cc:dd:ee:ff there are a lot of UPnP devices and other types of devices out there that use this mac address as a form of default. The point is just to change your mac to anything other than your real mac so that an IDS or system admin doesn’t see you. Thus, undetected sniffing.

If an admin or IDS saw 12 packets (as in my example) originating from 00:00:00:00:00:00 or aa:bb:cc:dd:ee:ff do you think he would pull out all the stops and think there is someone eavedropping? Of course not.

And yes, I have used this method on knoppix, archlinux, slackware, and FreeBSD. Of course, the differences in ifconfig in different distros need to be taken into account, and you can figure it out for yourself on your own distro by checking out your ifconfig man page.

man -a ifconfig

i would suggest plugging off pathc cord just before and do like that
ifconfig eth0 down
#UNPLUG CORD
ifconfig eth0 hw ether AA:BB:CC:DD:EE:FF
ifconfig eth0 
#PLUG IN CORD
ifconfig eth0 up
and then try to look for outgoing packets. Check for promisc: ifconfig

I would also suggest filling up switchs memory to make it act like a regural HUB so u will be able to sniff with normal oldstyle IP sniffers. You can sniff over switch by filling up its temporary MAC addr table. Then switch to work correctly (from its view its just overload) has to send data to every port just like HUB , and then u r at home. Also check out passive network scanning software: p0f by Lcamtuff, check it out

I haven’t used p0f for about 6 years.. When I used it was just a passive way to detect the OS/versions of a host. It was still pretty beta back then though.

Yes I have overloaded several switches CAM tables adn turned it into “HUB” operating mode.. But man is that ever loud! The ISP called me up and asked me if I knew anything about 2 GIGs of data being sent out from my port on the switch.. (layer 2) ..

Really the best way In my experience is arp poisoning. Ettercap has the basics included in their program. I have also had the experience of literally shutting down a switch from flooding the CAM.. you never really know what will happen. It’s usually not in your favor.

Also, if you start arp poisoning RIGHT after a power-outage (whether you shut off the power or it just went out) a lot of times you have a 15-45 second window where the switches will be acting like HUBs.. If you can poison some key targets in different network segments before the switch starts switching, you can really drop the normal restrictions.

Oh and you can also turn off your arp using ifconfig.

$ ifconfig eth0 -arp

But When you use the method outlined above, of course your NIC never sends out arp requests.. That is the whole point of NOT having an ipv6 or ip address.. and that is also why this method dictates NO ROUTES in the kernel or userland. Your NIC and ip stack just doesn’t have any body to send the arps to! If you aren’t connected to any networks, how can it send a broadcast? And if you don’t have an IP address, where would the arp tell everyone to reply to? See what I mean?

The MAC address being set to a non-suspicious looking address is just to slip by as a network anomololy instead of actually looking like a roque MAC.

In general though, I was talking about using this technique to splice into the wire at an uplink spot. So If I wanted to capture the router passwords for my apartment complex, I would find the router and use a portable hub to connect my packet-sniffing computer in-between the router and the modem or uplink. Then I would call tech-support about something and watch them log into the router.. then I have the password and its almost detectionless because it is completely passive.

I would say that my absolute favorite tool for hacking switches (we are basically just talking about layer-2’s) is hping. I also like nc, socat, vconfig, and dsniff was always fun but its really out of date now, ettercap is probably the king of automated switch hacking, its just not as much fun automated..

I’m sure you are also aware of the fun you can have with port-stealing. This is where you actually make the switch believe that you are coming in from a different port. Its pretty simple but a little harder than arp-poisoning. Most of my experience comes from circumventing vlans and hacking my way into switches on differnet network segments and on differnent vlans. Man talk about off-topic!

Now I got ur point, sorry I ,misunderstood at first totaly agree that ettercap is king nowadays and is most up to date. Anyways , I was always thinking in a little different way, I mean something like “fog cover”. IMO its better to spoof MAC and act as other regular user instead run all loud and silent techniques.
But your idea is very interesing.

Anywayz i think that there will be always something sent to the wire. As a first example that comes to my mind is link speed detection with NLP and FLP.

Thanks but its not an idea.. its a field-tested technique. Also, in my humble opinion, spoofing another host on the network is for newbies, unless you don’t mean just spoofing their MAC and IP. Because that is a really fast way to get caught where I come from. And it only takes 1 single command from a nix. Yes when I first started hacking into switches I was also operating under the assumption that if I spoofed a MAC address and mimicked an allowed machine that was somehow a good thing. First thing I realized was that before I could pretend to be a machine, I had to take that machine offline. If you are lucky and its running windows, that is no problem, you can run any number of techniques against a windows box and take it down. Of course, taking down a host, whether by exploits or DoS is an entirely differnet topic, and I won’t go there here.

The second thing I learned was about layer 2 and layer 3 switching.. and the huge differences between those and layer 1 switches. Mostly I deal with layer 2 switches. But pretending to be a machine is really an entirely different and lengthy topic, and I won’t go there now. It is relevant to the question about port stealing, since that is in essence where I turned after those realizations, and where I began finding more success.

First thing I realized was that before I could pretend to be a machine, I had to take that machine offline. If you are lucky and its running windows, that is no problem, you can run any number of techniques against a windows box and take it down.

True, but why just scan whole subnet for MAC & IP and few hours later just use one of these that are offline.

Actually you are right, good point. I guess the point I wanted to make was that attacking the switch directly, and not relying on spoofing another host, is the best way to go and where I found most of my success. It seems theoretically like a very logical way to attack a switch or even another host, but the fact is, when dealing with layer 2 or layer 3 switches, this will not give you any edge and will just waste your time and most likely tip off an admin to your attempts.

Of course, my experience is really pretty limited to switches that are using port security, which includes vlans, but really just basic port security, where each port on the switch is locked down to a particular mac address, and important routes are static. That is where my experience is at, hacking layer 2 switches with port security ON, and also hacking layer 2 switches with port security ON and VLANs active.

So for easier targets, spoofing another host could work quite well, especially where port security is OFF or the switch is not configured securely or layer 1 switching. I was referring to port security, vlans, trunking, CAM tables, etc.. but mostly just referring to the security mechanisms at play in layer 2 switches vs layer 1. And even layer 2 switches that are configured securely, vs those that are not.

Commands to become undetectable on Windows NT/W2K/XP

NOTE: You are at your own risk if you follow these instructions. Editing your registry is DANGEROUS and should be done with extreme caution. Follow these steps at your OWN risk.

Get your device’s hex value. (’snort -W’ works for this)
open Regedt32
Navigate to: HKEY_LOCAL_MACHINE$\backslash$SYSTEM$\backslash$CurrentControlSet$\backslash$Services$\backslash$Tcpip$\backslash$Parameters$\backslash$
Interfaces$\backslash${XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX}
Select the network card you wish to setup as the monitoring interface (this will be the {XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX} value).
Set IPAddress:REG_MULTI_SZ: to null (Double click on the string, delete data in the Multi-String Editor, then click OK)
Set SubnetMask:REG_MULTI_SZ: to null (Double click on the string, delete data in the Multi-String Editor, then click OK)
Set DefaultGateway:REG_MULTI_SZ: to null (Double click on the string, delete data in the Multi-String Editor, then click OK)
Close the Registry Editor, your changes will be saved automatically.
In a command prompt, run ‘ipconfig’ to verify the interface does not have an IP bound to it.

External Links

Random Posts

ethereal -wireshark -capture -sniffing -sniff-data -invisible-sniffing -wired -ethernet

Blocking Bad Bots and Scrapers with .htaccess

AskApache — Tue, 08 Apr 2008 15:04:11 +0000

This article shows 2 methods of blocking this entire list of bad robots and web scrapers with .htaccess files using SetEnvIfNoCase or using RewriteRules with mod_rewrite

Blocking Bad Robots and Web Scrapers with RewriteRules

ErrorDocument 403 /403.html
 
RewriteEngine On
RewriteBase /
 
# IF THE UA STARTS WITH THESE
RewriteCond %{HTTP_USER_AGENT} ^(aesop_com_spiderman|alexibot|backweb|bandit|batchftp|bigfoot) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(black.?hole|blackwidow|blowfish|botalot|buddy|builtbottough|bullseye) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(cheesebot|cherrypicker|chinaclaw|collector|copier|copyrightcheck) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(cosmos|crescent|curl|custo|da|diibot|disco|dittospyder|dragonfly) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(drip|easydl|ebingbong|ecatch|eirgrabber|emailcollector|emailsiphon) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(emailwolf|erocrawler|exabot|eyenetie|filehound|flashget|flunky) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(frontpage|getright|getweb|go.?zilla|go-ahead-got-it|gotit|grabnet) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(grafula|harvest|hloader|hmview|httplib|httrack|humanlinks|ilsebot) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(infonavirobot|infotekies|intelliseek|interget|iria|jennybot|jetcar) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(joc|justview|jyxobot|kenjin|keyword|larbin|leechftp|lexibot|lftp|libweb) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(likse|linkscan|linkwalker|lnspiderguy|lwp|magnet|mag-net|markwatch) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(mata.?hari|memo|microsoft.?url|midown.?tool|miixpc|mirror|missigua) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(mister.?pix|moget|mozilla.?newt|nameprotect|navroad|backdoorbot|nearsite) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(net.?vampire|netants|netcraft|netmechanic|netspider|nextgensearchbot) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(attach|nicerspro|nimblecrawler|npbot|octopus|offline.?explorer) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(offline.?navigator|openfind|outfoxbot|pagegrabber|papa|pavuk) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(pcbrowser|php.?version.?tracker|pockey|propowerbot|prowebwalker) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(psbot|pump|queryn|recorder|realdownload|reaper|reget|true_robot) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(repomonkey|rma|internetseer|sitesnagger|siphon|slysearch|smartdownload) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(snake|snapbot|snoopy|sogou|spacebison|spankbot|spanner|sqworm|superbot) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(superhttp|surfbot|asterias|suzuran|szukacz|takeout|teleport) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(telesoft|the.?intraformant|thenomad|tighttwatbot|titan|urldispatcher) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(turingos|turnitinbot|urly.?warning|vacuum|vci|voideye|whacker) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(wget|widow|wisenutbot|wwwoffle|xaldon|xenu|zeus|zyborg|anonymouse) [NC,OR]
 
# STARTS WITH WEB
RewriteCond %{HTTP_USER_AGENT} ^web(zip|emaile|enhancer|fetch|go.?is|auto|bandit|clip|copier|master|reaper|sauger|site.?quester|whack) [NC,OR]
 
# ANYWHERE IN UA -- GREEDY REGEX
RewriteCond %{HTTP_USER_AGENT} ^.*(craftbot|download|extract|stripper|sucker|ninja|clshttp|webspider|leacher|collector|grabber|webpictures).*$ [NC]
 
# ISSUE 403 / SERVE ERRORDOCUMENT
RewriteRule . - [F,L]

Block Bad Bots with SetEnvIfNoCase

ErrorDocument 403 /403.html
 
# IF THE UA STARTS WITH THESE
SetEnvIfNoCase ^User-Agent$ .*(aesop_com_spiderman|alexibot|backweb|bandit|batchftp|bigfoot).* bad_web_bot
SetEnvIfNoCase ^User-Agent$ .*(black.?hole|blackwidow|blowfish|botalot|buddy|builtbottough|bullseye).* bad_web_bot
SetEnvIfNoCase ^User-Agent$ .*(cheesebot|cherrypicker|chinaclaw|collector|copier|copyrightcheck).* bad_web_bot
SetEnvIfNoCase ^User-Agent$ .*(cosmos|crescent|curl|custo|da|diibot|disco|dittospyder|dragonfly).* bad_web_bot
SetEnvIfNoCase ^User-Agent$ .*(drip|easydl|ebingbong|ecatch|eirgrabber|emailcollector|emailsiphon).* bad_web_bot
SetEnvIfNoCase ^User-Agent$ .*(emailwolf|erocrawler|exabot|eyenetie|filehound|flashget|flunky).* bad_web_bot
SetEnvIfNoCase ^User-Agent$ .*(frontpage|getright|getweb|go.?zilla|go-ahead-got-it|gotit|grabnet).* bad_web_bot
SetEnvIfNoCase ^User-Agent$ .*(grafula|harvest|hloader|hmview|httplib|httrack|humanlinks|ilsebot).* bad_web_bot
SetEnvIfNoCase ^User-Agent$ .*(infonavirobot|infotekies|intelliseek|interget|iria|jennybot|jetcar).* bad_web_bot
SetEnvIfNoCase ^User-Agent$ .*(joc|justview|jyxobot|kenjin|keyword|larbin|leechftp|lexibot|lftp|libweb).* bad_web_bot
SetEnvIfNoCase ^User-Agent$ .*(likse|linkscan|linkwalker|lnspiderguy|lwp|magnet|mag-net|markwatch).* bad_web_bot
SetEnvIfNoCase ^User-Agent$ .*(mata.?hari|memo|microsoft.?url|midown.?tool|miixpc|mirror|missigua).* bad_web_bot
SetEnvIfNoCase ^User-Agent$ .*(mister.?pix|moget|mozilla.?newt|nameprotect|navroad|backdoorbot|nearsite).* bad_web_bot
SetEnvIfNoCase ^User-Agent$ .*(net.?vampire|netants|netcraft|netmechanic|netspider|nextgensearchbot).* bad_web_bot
SetEnvIfNoCase ^User-Agent$ .*(attach|nicerspro|nimblecrawler|npbot|octopus|offline.?explorer).* bad_web_bot
SetEnvIfNoCase ^User-Agent$ .*(offline.?navigator|openfind|outfoxbot|pagegrabber|papa|pavuk).* bad_web_bot
SetEnvIfNoCase ^User-Agent$ .*(pcbrowser|php.?version.?tracker|pockey|propowerbot|prowebwalker).* bad_web_bot
SetEnvIfNoCase ^User-Agent$ .*(psbot|pump|queryn|recorder|realdownload|reaper|reget|true_robot).* bad_web_bot
SetEnvIfNoCase ^User-Agent$ .*(repomonkey|rma|internetseer|sitesnagger|siphon|slysearch|smartdownload).* bad_web_bot
SetEnvIfNoCase ^User-Agent$ .*(snake|snapbot|snoopy|sogou|spacebison|spankbot|spanner|sqworm|superbot).* bad_web_bot
SetEnvIfNoCase ^User-Agent$ .*(superhttp|surfbot|asterias|suzuran|szukacz|takeout|teleport).* bad_web_bot
SetEnvIfNoCase ^User-Agent$ .*(telesoft|the.?intraformant|thenomad|tighttwatbot|titan|urldispatcher).* bad_web_bot
SetEnvIfNoCase ^User-Agent$ .*(turingos|turnitinbot|urly.?warning|vacuum|vci|voideye|whacker).* bad_web_bot
SetEnvIfNoCase ^User-Agent$ .*(wget|widow|wisenutbot|wwwoffle|xaldon|xenu|zeus|zyborg|anonymouse).* bad_web_bot
SetEnvIfNoCase ^User-Agent$ .*(web(zip|emaile|enhancer|fetch|go.?is|auto|bandit|clip|copier|master|reaper|sauger|site.?quester|whack).* bad_web_bot
SetEnvIfNoCase ^User-Agent$ .*(craftbot|download|extract|stripper|sucker|ninja|clshttp|webspider|leacher|collector|grabber|webpictures).* bad_web_bot
 
Order Deny,Allow
Allow from All
Deny from env=bad_web_bot

Original Bad Bot / Web Scraper List

WebBandit
2icommerce
Accoona
ActiveTouristBot
adressendeutschland
aipbot
Alexibot
Alligator
AllSubmitter
almaden
anarchie
Anonymous
Apexoo
Aqua_Products
asterias
ASSORT
ATHENS
AtHome
Atomz
attache
autoemailspider
autohttp
b2w
bew
BackDoorBot
Badass
Baiduspider
Baiduspider+
BecomeBot
berts
Bitacle
Biz360
Black.Hole
BlackWidow
bladder fusion
Blog Checker
BlogPeople
Blogshares Spiders
Bloodhound
BlowFish
Board Bot
Bookmark search tool
BotALot
BotRightHere
Bot mailto:craftbot@yahoo.com
Bropwers
Browsezilla
BuiltBotTough
Bullseye
BunnySlippers
Cegbfeieh
CFNetwork
CheeseBot
CherryPicker
Crescent
charlotte/
ChinaClaw
Convera
Copernic
CopyRightCheck
cosmos
Crescent
c-spider
curl
Custo
Cyberz
DataCha0s
Daum
Deweb
Digger
Digimarc
digout4uagent
DIIbot
DISCo
DittoSpyder
DnloadMage
Download
dragonfly
DreamPassport
DSurf
DTS Agent
dumbot
DynaWeb
e-collector
EasyDL
EBrowse
eCatch
ecollector
edgeio
efp@gmx.net
EirGrabber
Email Extractor
EmailCollector
EmailSiphon
EmailWolf
EmeraldShield
Enterprise_Search
EroCrawler
ESurf
Eval
Everest-Vulcan
Exabot
Express
Extractor
ExtractorPro
EyeNetIE
FairAd
fastlwspider
fetch
FEZhead
FileHound
findlinks
Flaming AttackBot
FlashGet
FlickBot
Foobot
Forex
Franklin Locator
FreshDownload
FrontPage
FSurf
Gaisbot
Gamespy_Arcade
genieBot
GetBot
Getleft
GetRight
GetWeb!
Go!Zilla
Go-Ahead-Got-It
GOFORITBOT
GrabNet
Grafula
grub
Harvest
Hatena Antenna
heritrix
HLoader
HMView
holmes
HooWWWer
HouxouCrawler
HTTPGet
httplib
HTTPRetriever
HTTrack
humanlinks
IBM_Planetwide
iCCrawler
ichiro
iGetter
Image Stripper
Image Sucker
imagefetch
imds_monitor
IncyWincy
Industry Program
Indy
InetURL
InfoNaviRobot
InstallShield DigitalWizard
InterGET
IRLbot
Iron33
ISSpider
IUPUI Research Bot
Jakarta
java/
JBH Agent
JennyBot
JetCar
jeteye
jeteyebot
JoBo
JOC Web Spider
Kapere
Kenjin
Keyword Density
KRetrieve
ksoap
KWebGet
LapozzBot
larbin
leech
LeechFTP
LeechGet
leipzig.de
LexiBot
libWeb
libwww-FM
libwww-perl
LightningDownload
LinkextractorPro
Linkie
LinkScan
linktiger
LinkWalker
lmcrawler
LNSpiderguy
LocalcomBot
looksmart
LWP
Mac Finder
Mail Sweeper
mark.blonin
MaSagool
Mass
Mata Hari
MCspider
MetaProducts Download Express
Microsoft Data Access
Microsoft URL Control
MIDown
MIIxpc
Mirror
Missauga
Missouri College Browse
Mister
Monster
mkdb
moget
Moreoverbot
mothra/netscan
MovableType
Mozi!
Mozilla/22
Mozilla/3.0 (compatible)
Mozilla/5.0 (compatible; MSIE 5.0)
MSIE_6.0
MSIECrawler
MSProxy
MVAClient
MyFamilyBot
MyGetRight
nameprotect
NASA Search
Naver
Navroad
NearSite
NetAnts
netattache
NetCarta
NetMechanic
NetResearchServer
NetSpider
NetZIP
Net Vampire
NEWT ActiveX
Nextopia
NICErsPRO
ninja
NimbleCrawler
noxtrumbot
NPBot
Octopus
Offline
OK Mozilla
OmniExplorer
OpaL
Openbot
Openfind
OpenTextSiteCrawler
Oracle Ultra Search
OutfoxBot
P3P
PackRat
PageGrabber
PagmIEDownload
panscient
Papa Foto
pavuk
pcBrowser
perl
PerMan
PersonaPilot
PHP version
PlantyNet_WebRobot
playstarmusic
Plucker
Port Huron
Program Shareware
Progressive Download
ProPowerBot
prospector
ProWebWalker
Prozilla
psbot
psycheclone
puf
PushSite
PussyCat
PuxaRapido
Python-urllib
QuepasaCreep
QueryN
Radiation
RealDownload
RedCarpet
RedKernel
ReGet
relevantnoise
RepoMonkey
RMA
Rover
Rsync
RTG30
Rufus
SAPO
SBIder
scooter
ScoutAbout
script
searchpreview
searchterms
Seekbot
Serious
Shai
shelob
Shim-Crawler
SickleBot
sitecheck
SiteSnagger
Slurpy Verifier
SlySearch
SmartDownload
sna-
snagger
Snoopy
sogou
sootle
So-net” bat_bot
SpankBot” bat_bot
spanner” bat_bot
SpeedDownload
Spegla
Sphere
Sphider
SpiderBot
sproose
SQ Webscanner
Sqworm
Stamina
Stanford
studybot
SuperBot
SuperHTTP
Surfbot
SurfWalker
suzuran
Szukacz
tAkeOut
TALWinHttpClient
tarspider
Teleport
Telesoft
Templeton
TestBED
The Intraformant
TheNomad
TightTwatBot
Titan
toCrawl/UrlDispatcher
True_Robot
turingos
TurnitinBot
Twisted PageGetter
UCmore
UdmSearch
UMBC
UniversalFeedParser
URL Control
URLGetFile
URLy Warning
URL_Spider_Pro
UtilMind
vayala
vobsub
VCI
VoidEYE
VoilaBot
voyager
w3mir
Web Image Collector
Web Sucker
Web2WAP
WebaltBot
WebAuto
WebBandit
WebCapture
webcollage
WebCopier
WebCopy
WebEMailExtrac
WebEnhancer
WebFetch
WebFilter
WebFountain
WebGo
WebLeacher
WebMiner
WebMirror
WebReaper
WebSauger
WebSnake
Website
WebStripper
WebVac
webwalk
WebWhacker
WebZIP
Wells Search
WEP Search 00
WeRelateBot
Wget
WhosTalking
Widow
Wildsoft Surfer
WinHttpRequest
WinHTTrack
WUMPUS
WWWOFFLE
wwwster
WWW-Collector
Xaldon
Xenu's
Xenus
XGET
Y!TunnelPro
YahooYSMcm
YaDirectBot
Yeti
Zade
ZBot
zerxbot
Zeus
ZyBorg

Random Posts

robot -bot -.htaccess -scraper -web-scraper -rewriterule -setenvif

Crazy Cache WordPress Plugin Released

AskApache — Tue, 01 Apr 2008 05:25:23 +0000

Isn’t WP-Cache an incredibly useful plugin? If I was only allowed to have one plugin for my WordPress blogs, hands-down I’d choose WP-Cache.

AskApache Crazy Cache lets you cache all the posts on your blog at once.

I’ve used some advanced features of libcurl and fsockopen to make sure that this caching action doesn’t overwhelm your server or result in redundant requests. That could slow down your blog, which I would never, ever, allow, I am very interested in this stuff.. speedy sites that is.

I always wanted the ability to cache all my posts on my blog whenever I wanted, and WP-Cache doesn’t let you do that. So a few months ago I hacked together this kick-butt plugin to do exactly that.

ScreenShot

Installation

This plugin is one of those idiot-proof installations, nuff said.

Download

askapache-crazy-cache at wordpress.org

Want More Speed?

I love you guys and girls who want a faster Internet, we rock. So check these out.

Random Posts

cache -wpcache -wp-cache -wordpess -plugin -askapache -speed