I’m not sure about you, but anything Mario endorses…

Overview and General Ideas

Basic Definitions

CPU – Central Processing Unit

This is what Intel and AMD commercials are about, this chip functions like a calculator, computing all the data.

Hard Drive

This is a glorified CD-ROM Disc that is much faster and has a higher capacity than a CD-ROM. This is where all your files and Operating System is stored on.

RAM – Random Access Memory

Data that is frequently accessed is stored in RAM instead of the Hard Drive because RAM is much much faster.

Page File / Virtual Memory

This is literally a file that is stored on your Hard Drive and is used as a form of RAM when you run out of real RAM. Much slower than RAM (because it is located on the hard drive) it is used to store bits of data used by programs.

Three-pronged attack against the sluggish druggish

1. Increase RAM/Memory

   Reducing the amount of RAM that is wasted by unneeded programs, unused device drivers and devices, background services, visual effects, and various other stuff frees up RAM for your needed programs and processes. The final step of this article details the Cacheman Software that automatically takes care of everything this article doesn’t go into (like registry modifications and hidden tweaks)..

2. Optimize Hard Drive Data

   The attack in order:

   • Reduce Amount of data on drive by removing unneeded data and configuring optimum settings to prevent buildup in the future.
   • Configure settings which minimize the amount of disk activity, e.g., reading and writing to disk.
   • Defragment the data that is left, and setup an automated system to prevent fragmentation in the future.

3. Free Up CPU Time

   Almost all of the following tips will help to decrease the amount of CPU processing time that is wasted on unneeded processes.

Part 1: Increase and Free Memory/RAM

Configure Performance Options

Control Panel -> System -> Advanced -> Performance -> Settings

If you have massive RAM like I do, best appearance is nice. To max your speed change to “Adjust for best performance”.

Change Theme to Classic

The Classic Theme uses less RAM and CPU than the newer XP themes.

Fine-Tune Effects

Disable all for maximum speed. I like to only leave ClearType enabled.

Color quality and Resolution

To sacrifice quality for speed, change the Color quality to a lower setting, and lower the resolution if you want.

Clean up that Desktop!

Your desktop is always loaded in RAM, as it is always available to view. So everything that is ON your desktop is likewise stored in RAM. This means that every icon located on your desktop, which is actually an icon file, is loaded in RAM. What I do is instead of a cluttered Desktop always loaded in RAM, I simply create 1 folder called Desktop on my Desktop, and move everything from my desktop to that folder. That way instead of all those icons taking up memory, now only the single folder icon is taking up RAM. Not a HUGE impact, but it illustrates what RAM is and how you can stop wasting it.

Make Background Solid Color

By setting the background image of your desktop (and folders if you are using custom folder backgrounds) to none, and instead just using a solid color (like blue) you free up RAM. If your desktop uses a high-quality background image (like mine) that is a 5MB jpeg image, that effectively wastes 5MB of your RAM. A solid color background uses none.

Optimize Folder Options

Open up Explorer/My Computer -> Tools -> Folder Options

First change the option to “Use Windows classic folders“. Now goto the View tab and hit the button “Reset all Folders“, which will clear your individual folder viewing settings. Then change the options to the ones I have selected in the image for optimum speed, its up to you which options you don’t want to change. The searching for network folders is a very real cause of slowness.

Page File / Virtual Memory

The optimal way to use a page file is to not use a pagefile and max out your physical RAM. Next to doing that (which Windows won’t let you do anyway, they love pagefiling) the optimal setup is to have the pagefile on a 2nd hard-drive-disk. The computer can only read/write data to/from the hard drive at a certain speed, and only a certain amount of read/writes can be going on at one time.. Think of it like a pipe. Having the pagefile.sys on the same disk that Windows and all the Program’s are loaded on clogs the pipe and limits flow of data in both directions.

Note that moving the page file to a separate partition on the same hard drive would NOT fix this problem or alleviate the bottleneck in any way, so the solution is to get a 2nd hard drive, I personally recommend these 2 $49.99 – 320GB x 7500RPM hard-drives that are very reliable.

Page File Size

The size of your pagefile is up for debate, no clear answer, but I’ve tried a lot of different configurations over the years and here is what I recommend. If you have little RAM (less than 1GB), then you should just let Windows control it. Otherwise use the custom option to set it at a low initial size of 400MB to grow up to 3096MB. I have 4GB of RAM, so I set it to start at 100MB and grow to 5000MB. The general rule is to set it to be 1.5x the amount of RAM you have.

Disable Devices

Disable any devices you never use, like internal modems, firewire ports, floppy drives, etc.. Also disable them in your systems BIOS if you can.

Uninstalling Devices

If you have devices that you used on your computer in the past but don’t anymore, they are still being loaded up by Windows and at the very least hogging some resources. So go through and uninstall them.. (You have to select View -> Show hidden devices).. If you plug them back in they will be re-installed with the “Found new hardware” wizard.

Disable Fast User Switching

Control Panel -> Users -> Change the way users log on

Make sure you disable fast-user-switching unless you absolutely must have it. I’ve found it to be a cool feature if you are a basic computer user not installing programs or tweaking settings.. Nut it can create big problems if you use your computer for all sorts of *advanced* stuff (advanced for Windows)… Plus disabling it will have some speed improvements.

Optimizing Services

Start -> Run -> services.msc

First make sure you export your current services to a backup file before you begin. There are a number of informative articles about this topic on the web.

Removing Unneeded Network Protocols

This actually helps a lot and can have a significant impact on your computer speed, much more so than any impact on network speed improvements. These protocols require all sorts of RAM and are kept loaded. They also represent a security risk as it provides another process to be attacked.

Part 2: Optimize Hard Drive and Data

Recommended Hard Drives

These are the cheapest price’s for these tried and true hard-drives. I love TigerDirect and recommend you check them out.

1. $49.99 Seagate Barracuda 320GB – 7200RPM
2. $49.99 WD3200 Caviar Blue 320GB – 7200RPM
3. $59.99 WD5000 Caviar Blue 500GB – 7200RPM
4. $229.99 WD VelociRaptor 320GB – 10,000RPM, FAST!

File System Tweaks

Start -> Run -> compmgmt.msc
Check the structure and properties of your disks and partitions to see if anything is out of place. Make sure file systems are using NTFS. But some partitions like dell utility partition, or a partition shared by linux/mac/windows on dual-boot systems may be fat or fat32. To convert fat-based to NTFS, defragment MFT, resize NTFS clusters, etc.. you have to use a third part product. I personally use Norton Partition Magic, Paragon Partition Manager, and Acronis Disk Director Suite (plus 20 other various tools).

2nd Hard Drive

*Advanced*
If your lucky or smart enough to have more than one hard drive disk installed, in addition to moving your pagefile to that 2nd hard-drive you should setup the hard drive with at least 2 partitions first. Say you install one of those 320GB hard drives I recommend… Here’s how I set the partitions on the 2nd hard-drive.

1. 6.5GB – S: [SWAP] – First partition is fastest area on drive, this partition is ONLY for the pagefile. Size is 1.5x the max size of your pagefile setting, or 1.5x the amount of RAM.
2. 13.5GB – 2nd partition is not given a drive letter, instead it is mounted as both C:\TMP and C:\TEMP folders.. similarly to optimizing the pagefile, Windows and Programs use the TMP and TEMP folders A LOT so this will really speed up any installations and a lot of Windows itself. *very tricky*
3. 300GB – This is also mounted instead of given a drive letter, mounted as “C:\Documents and Settings” which is where all your settings and user files (Desktop, My Documents) are stored. *very tricky*

Setting up the TMP and “Documents and Settings” folders that way is NOT for beginners, its advanced and dangerous if you do it wrong. But definately has one of the biggest noticeable speed benefits.

Remove Unneccessary Programs

Control Panel -> Add or Remove Programs

Remove all programs that you are positive you don’t need anymore, the more you remove, the better. Microsoft provides a free tool that is simple and helpful when removing programs. Download it at: Windows Installer CleanUp Utility

Remove Windows Components

The only main categories I need are Internet Explorer, Windows Media Player, Update Root Certificates, and Accessories and Utilities

I also remove the Games category which is a subcategory of Accessories and Utilities.

System Restore Optimization

Turn off system restore on any drives that are not crucial… So if you have a 2nd hard-drive that is used for pagefile and temp file storage, turn it off. And you should make sure the settings aren’t too generous, I usually never go higher than 10GB.

Recycle Bin Settings

Keep this setting to a low value, try not to delete stuff you will need later. I personally disable the Recycle Bin completely.. but I come from a unix background so I have no issues with misplacing files.

Remove Unneccessary Fonts

C:\Windows\Fonts

Fonts are loaded up by default by Windows and every time you load a program like Photoshop, Microsoft Word, Outlook, etc., so moving any you never use out of the C:\Windows\Fonts folder saves RAM and CPU and HD space. This is a bit hairy but I sort by size and then check out how long its been since the font has been accessed by looking at the file attributes. The less fonts you have, the faster your computer, but there is obviously a trade-off. You can move all fonts except for a few basic ones included with Windows to a backup folder instead of deleting them, then reboot and if your font is messed up then move some of the fonts from the backup folder back and try again.

Prefetch Cleaning

Delete the contents of the C:\WINDOWS\Prefetch folder and restart the computer. Its good to empty this folder and reboot after major upgrades or about once a month.

Part 3: Finishing the Optimization Process

These 3 tips will have the biggest speed impact for 99% of you, this is some really good stuff.

Disk Cleanup

Start -> Run -> %SystemRoot%\system32\cleanmgr.exe

This wizard is a very useful tool that combines several cleanup tools in one location. It also provides a way to clean up all past restore points save on your disk except the most recent one.

Defragmenting with SmartDefrag

The World’s Most Efficient Defragmenter

What’s the primary cause of slow/unstable PC performance? It’s disk fragmentation. Smart Defrag helps defragment your hard drives more efficiently than any other product on the market –– free or not.

This powerful, award-winning free defragmenter is 100% safe and clean with no adware, spyware, or viruses.

Download Smart Defrag

Cacheman XP Optimization

Cacheman is a Windows software designed to speed up your computer by optimizing several caches, managing RAM and fine tuning a number of system settings. Auto-Optimization makes it suitable for novice and intermediate users yet it is also powerful and versatile enough for computer experts. Backups of settings ensure that all user modifications can be reversed with a single click.

Go Get It

Software Recommendations

From the Sysinternals Suite

Autoruns

See what programs are configured to startup automatically when your system boots and you login. Autoruns also shows you the full list of Registry and file locations where applications can configure auto-start settings.

Process Explorer

Find out what files, registry keys and other objects processes have open, which DLLs they have loaded, and more. This uniquely powerful utility will even show you who owns each process.

Process Monitor

Monitor file system, Registry, process, thread and DLL activity in real-time.

VMMap

See a breakdown of a process’s committed virtual memory types as well as the amount of physical memory (working set) assigned by the operating system to those types. Identify the sources of process memory usage and the memory cost of application features.

Tweak UI Powertoy

This PowerToy gives you access to system settings that are not exposed in the Windows XP default user interface, including mouse settings, Explorer settings, taskbar settings, and more.

Go Get It

The police used inapplicable criminal laws as a basis for a fishing expedition to determine the author of an anonymous email,” said EFF Senior Staff Attorney Matt Zimmerman. “Now, this student has been suspended from his job, and he is without a laptop and other devices he needs to do his schoolwork. His private communications and papers are in the hands of police who are searching for evidence without just cause. Even his cell phone and iPod were taken, clearly an overreach if the goal is tracking the source of an email.

This search warrant is invalid, as there is no probable cause that a crime was committed at all… Every day this student’s private information is in the hands of the police department, he suffers harm to his property interests and his constitutional rights.

The dorm room search stemmed from an investigation into who sent an email to a Boston College mailing list alleging that another student was gay. Police say they know who sent the email and that the sender committed the crimes of “obtaining computer services by fraud or misrepresentation” and obtaining “unauthorized access to a computer system.” However, nothing presented by the investigating officer to obtain the warrant, including the allegation that the student sent the email to the mailing list, could constitute the cited criminal offenses.

Could Happen to Anyone

Think if you were known as someone good with computers at your work or school, and something as small as a mean-spirited email was sent to the whole organization.. Then they come into your office, suspend you from working, and send your cell phone, ipod, disks, backups, and computer to the local police station where it is then hammered at by some average cop who’s only security skill is knowing how to use the special forensic detective programs that are illegal for the general public, but really aren’t all that impressive. This can wreck havoc on your hard-drives, and if it fails at the police station? Sorry no backups!

That Sinking Feeling..

How are individuals supposed to take-on such established authoritative organizations like the Police? When/How do you get your stuff back? How many people do you have to call to even find out IF you’ll get it back or even learn where they’ve taken it? Under criminal investigation, the courts are involved, the state police are involved, and you lost your job and means to make a living.. what can you possibly do?

The Electronic Frontier Foundation (EFF) is representing the student, along with a very good law firm: Fish & Richardson. They’ve petitioned the court for the immediate return of his property and is demanding that investigators be prohibited from any further searches or analysis of his digital data. Massachusetts State Police participated in the search and are overseeing the forensic analysis of the seized property. The damage has already been done, now its time to put the hurt down on them so they don’t trample the Constitution so easily the next time.

The motion to quash the search warrant was filed with assistance from Fish & Richardson attorneys Adam Kessel, Lawrence Kolodney, and Tom Brown. Internet Heroes.

Round Robin Concept

A few months ago I was given a free hosting account on HostGator to evaluate and tempt me away from DreamHost to become a Gator. I get a lot of these types of offers from time to time from the smaller Web companies who read AskApache.. but when I noticed the SSH access was jailed and saw how restrictive the shell was I felt like I was on a windows box.. I want a shell, cpanel sucks. I compile and run everything from the shell so thats was a deal-breaker and I sorta forgot all about it.

The goal is to add the HostGator server to be an exact mirror of the z.askapache.com domain, then to add that server as a 2nd A record to my DNS zone. That way half the visitors to the size will be taking up resources and bandwidth on the HostGator server instead of mine.

Round Robin A records in DNS are intended to evenly distribute queries between each host of the same name. Using some tricks straight out of a hackers toolbox we can verify if the distribution is taking place. (It is.)

DNS – Domain Name System

The Domain Name System is often analogized as a “phone book” for the Internet because it translates human-friendly computer hostnames into IP addresses. For example, www.askapache.com translates to 208.113.134.190. Every request for a human-friendly hostname first needs to be translated to the IP before the server can be queried. Normally each hostname corresponds to exactly 1 IP address. But in a Round Robin setup, the idea is to have the hostname correspond to multiple IP addresses, which are different servers that house the exact same content, resulting in some hosts requesting files from one server, and another host requesting files from the other server, resulting in less CPU resources and bandwidth.

Here is an the zone for z.askapache.com Round Robin records:

Round Robin DNS

Round robin DNS is a technique of load distribution, load balancing, or fault-tolerance provisioning multiple, redundant Internet Protocol service hosts, e.g., Web servers, FTP servers, by managing the Domain Name System’s (DNS) responses to address requests from client computers according to an appropriate statistical model.

In its simplest implementation Round-robin DNS works by responding to DNS requests not only with a single IP address, but a list of IP addresses of several servers that host identical services. The order in which IP addresses from the list are returned is the basis for the term round robin. With each DNS response, the IP address sequence in the list is permuted. Usually, basic IP clients attempt connections with the first address returned from a DNS query so that on different connection attempts clients would receive service from different providers, thus distributing the overall load among servers.

Round robin DNS is often used for balancing the load of geographically-distributed Web servers. For example, a company has one domain name and three identical web sites residing on three servers with three different IP addresses. When one user accesses the home page it will be sent to the first IP address. The second user who accesses the home page will be sent to the next IP address, and the third user will be sent to the third IP address. In each case, once the IP address is given out, it goes to the end of the list. The fourth user, therefore, will be sent to the first IP address, and so forth.

Although easy to implement, round robin DNS has problematic drawbacks, such as those arising from record caching in the DNS hierarchy itself, as well as client-side address caching and reuse, the combination of which can be difficult to manage. Round robin DNS should not solely be relied upon for service availability. If a service at one of the addresses in the list fails, the DNS will continue to hand out that address and clients will still attempt to reach the inoperable service.

Does Round Robin Work

Definately. I can look at the access logs for both servers and see that they are splitting the requests nicely. It is definately not an exact split however, look at these statistics and you’ll see what I mean.

$ dig @ns1.dreamhost.com +authority +all z.askapache.com
 
;; ANSWER SECTION:
z.askapache.com.        14400   IN      A       69.56.174.114
z.askapache.com.        14400   IN      A       64.111.114.111
 
$ dig @ns1.dreamhost.com +authority +all z.askapache.com
 
;; ANSWER SECTION:
z.askapache.com.        14400   IN      A       64.111.114.111
z.askapache.com.        14400   IN      A       69.56.174.114

The effects of caching will distort the effectiveness of any IP address allocation algorithm unless a 0 TTL is used which has the effect of significantly increasing the load on the DNS (and is not always implemented consistently). In this case the cure may be worse than the disease Good news we have good load balancing on our web servers. Bad news we need 17 more DNS servers!. Intuitively, and without running any experiments to verify, we would suggest that given a normal TTL (12 hours or more) and ANY IP allocation algorithm other than a single static list, loads should be reasonably balanced .

Full root server query

Tracing to z.askapache.com[a] via A.ROOT-SERVERS.NET, maximum of 3 retries
A.ROOT-SERVERS.NET [.] (198.41.0.4)
 |\___ L.GTLD-SERVERS.NET [com] (192.41.162.30)
 |     |\___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |\___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      \___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |\___ D.GTLD-SERVERS.NET [com] (192.31.80.30)
 |     |\___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |\___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      \___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |\___ J.GTLD-SERVERS.NET [com] (192.48.79.30)
 |     |\___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |\___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      \___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |\___ H.GTLD-SERVERS.NET [com] (192.54.112.30)
 |     |\___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |\___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      \___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |\___ E.GTLD-SERVERS.NET [com] (192.12.94.30)
 |     |\___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |\___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      \___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |\___ F.GTLD-SERVERS.NET [com] (192.35.51.30)
 |     |\___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |\___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      \___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |\___ I.GTLD-SERVERS.NET [com] (192.43.172.30)
 |     |\___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |\___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      \___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |\___ G.GTLD-SERVERS.NET [com] (192.42.93.30)
 |     |\___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |\___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      \___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |\___ B.GTLD-SERVERS.NET [com] (2001:0503:231d:0000:0000:0000:0002:0030) Not queried
 |\___ B.GTLD-SERVERS.NET [com] (192.33.14.30)
 |     |\___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |\___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      \___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |\___ A.GTLD-SERVERS.NET [com] (2001:0503:a83e:0000:0000:0000:0002:0030) Not queried
 |\___ A.GTLD-SERVERS.NET [com] (192.5.6.30)
 |     |\___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |\___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      \___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |\___ C.GTLD-SERVERS.NET [com] (192.26.92.30)
 |     |\___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |\___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      \___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |\___ M.GTLD-SERVERS.NET [com] (192.55.83.30)
 |     |\___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |\___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      \___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
  \___ K.GTLD-SERVERS.NET [com] (192.52.178.30)
       |\___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
       |\___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
        \___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 
  ns1.dreamhost.com (66.33.206.206)       z.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       z.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       z.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       z.askapache.com -> 69.56.174.114
  ns1.dreamhost.com (66.33.206.206)       z.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       z.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       z.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       z.askapache.com -> 69.56.174.114
  ns1.dreamhost.com (66.33.206.206)       z.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       z.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       z.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       z.askapache.com -> 69.56.174.114
  ns1.dreamhost.com (66.33.206.206)       z.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       z.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       z.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       z.askapache.com -> 69.56.174.114
  ns1.dreamhost.com (66.33.206.206)       z.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       z.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       z.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       z.askapache.com -> 69.56.174.114
  ns1.dreamhost.com (66.33.206.206)       z.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       z.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       z.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       z.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       z.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       z.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       z.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       z.askapache.com -> 69.56.174.114
  ns1.dreamhost.com (66.33.206.206)       z.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       z.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       z.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       z.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       z.askapache.com -> 69.56.174.114
  ns1.dreamhost.com (66.33.206.206)       z.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       z.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       z.askapache.com -> 69.56.174.114
  ns1.dreamhost.com (66.33.206.206)       z.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       z.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       z.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       z.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       z.askapache.com -> 69.56.174.114
  ns1.dreamhost.com (66.33.206.206)       z.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       z.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       z.askapache.com -> 69.56.174.114
  ns1.dreamhost.com (66.33.206.206)       z.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       z.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       z.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       z.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       z.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       z.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       z.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       z.askapache.com -> 69.56.174.114

Live Online DNS Tools

• DNSstuff.com – Your Destination for DNS and Networking Tools
• ZoneCheck
• Quick Check – Pingability.com
• Squishywishywoo: complete dns traversal checking
• ZoneCut DNS
• DNS Sleuth
• DNS test – Domain name DNS test – pweb.cz
• OpenDNS – Support – CacheCheck
• DNSReport
• DNSTrace
• intoDNS
• DNSthru.com
• DNS Doctor
• Power Check

More Reading

• HOWTO – Configure Load Balancing
• Load Sharing with DNS

RFC’s related to DNS

• RFC 920: Specified original TLDs: .arpa, .com, .edu, .org, .gov, .mil and two-character country codes
• RFC 1032: Domain administrators guide
• RFC 1033: Domain administrators operations guide
• RFC 1034: Domain Names – Concepts and Facilities.
• RFC 1035: Domain Names – Implementation and Specification
• RFC 1101: DNS Encodings of Network Names and Other Types
• RFC 1123: Requirements for Internet Hosts — Application and Support
• RFC 1912: Common DNS Operational and Configuration Errors
• RFC 1995: Incremental Zone Transfer in DNS
• RFC 1996: A Mechanism for Prompt Notification of Zone Changes (DNS NOTIFY)
• RFC 2136: Dynamic Updates in the domain name system (DNS UPDATE)
• RFC 2181: Clarifications to the DNS Specification
• RFC 2182: Selection and Operation of Secondary DNS Servers
• RFC 2308: Negative Caching of DNS Queries (DNS NCACHE)
• RFC 2317: Classless IN-ADDR.ARPA delegation
• RFC 2671: Extension Mechanisms for DNS (EDNS0)
• RFC 3597: Handling of Unknown DNS Resource Record (RR) Types
• RFC 3696: Application Techniques for Checking and Transformation of Names
• RFC 4343: Domain Name System (DNS) Case Insensitivity Clarification
• RFC 4592: The Role of Wildcards in the Domain Name System
• RFC 4892: Requirements for a Mechanism Identifying a Name Server Instance
• RFC 5001: DNS Name Server Identifier Option (NSID)
• RFC 5395: Domain Name System (DNS) IANA Considerations

This page contains content by Author of Article from Wikipedia and is licensed under the GNU FDL.

Table of Contents

• Round Robin Concept
• DNS – Domain Name System
• Round Robin DNS
• Efficacy of DNS Load Balancing
• Live Online DNS Tools
• More Reading
• RFC’s related to DNS

For this article I will show how I setup a web mirror of z.askapache.com from my DreamHost server to my HostGator Server. For the transfer and synchronization of the directories we will be using rsync over SSH. We will also be automating this task using a cronjob with no user-interaction, so creating public and private keys for the ssh will be neccessary. Finally, I provide a simple shell script that prevents anyone from logging into your account with the created keys for any purpose other than to rsync.

Table of Contents

1. Rsync Synchronization Magic
2. Securing Rsync with SSH
3. Generate Keys with No Password
4. Custom SSH Connection Info
5. Creating Cronjob for Synchronization
6. Only Allow rsync
7. Rsync/SSH Debugging and Stats
8. Rsync Algorithm

Rsync Synchronization Magic

rsync

rsync is an open source file transfer program for Unix systems that uses the “rsync algorithm” which provides a very fast method for synchronizing files and directories from one location to another while minimizing data transfer using delta encoding when appropriate. An important feature of rsync not found in most similar programs/protocols is that the mirroring takes place with only one transmission in each direction.. It does this by sending just the differences in the files across the link, without requiring that both sets of files are present at one of the ends of the link beforehand.

Securing Rsync with SSH

I NEVER transfer any unencrypted data around unless that data is transported encrypted and to a trusted recipient, (I use HTTPS for WordPress administration) and I haven’t had time to probe the HostGator system for security issues yet, so right away I decided I needed an automated way to securely transfer z.askapache.com files TO hostgator, while not allowing my hostgator account access back on my main server. So if the hostgator account were to get hacked somehow, the cracker/spammer wouldn’t have access back to my main server.

Generate Keys with No Password

First I created a private key, specifically a passwordless host key, meaning to gain access with ssh you only need to supply the key, not a password+key like normal.

[local@dreamhost] $ mkdir -p ~/.ssh && chmod 700 ~/.ssh
 
# Create the public and private keys
# public key at: z.askapache-hostgator.id_rsa.pub
# private key at: z.askapache-hostgator.id_rsa
[local@dreamhost] $ ssh-keygen -t rsa -b 2048 -f ~/.ssh/z.askapache-hostgator.id_rsa
 
# add the public key to remote hosts ~/.ssh/authorized_keys file
[local@dreamhost] $ ssh-copy-id -i ~/.ssh/z.askapache-hostgator.id_rsa.pub remoteuser@remotehost
 
# or use scp + ssh to add the public key
[local@dreamhost] $ scp ~/.ssh/z.askapache-hostgator.id_rsa.pub remoteuser@remotehost:/home/remoteuser/
[local@dreamhost] $ ssh remoteuser@remotehost
[gatoraskapache@gator] $ mkdir -p ~/.ssh && chmod 700 ~/.ssh
[gatoraskapache@gator] $ cat ~/z.askapache-hostgator.id_rsa.pub >> ~/.ssh/authorized_keys
[gatoraskapache@gator] $ chmod 600 ~/.ssh/authorized_keys

Custom SSH Connection Info

This helps alot, by adding this to the very top of my ~/.ssh/config file I don’t have to add all this stuff to the rsync command-line. Basically when I reference connecting to the host ‘gator’ it uses all these options. Very helpful and you can add as many entries as you want.

Host gator
   IdentityFile ~/.ssh/z.askapache-hostgator.id_rsa
   Port 2222
   Protocol 2
   User gatoraskapache
   HostName gator555.hostgator.com
   PasswordAuthentication no

Creating Cronjob for Synchronization

This cronjob runs every 30 minutes, copying all modified files from my local directory ~/z.askapache.com/ to the remote directory ~/public_html/z/. First I backup the current crontab, then I edit the crontab and add this.

crontab -l > .crontab-`date +%F.backup`; crontab -e

*/30 * * * * /usr/bin/rsync -e 'ssh' -rt --delete ~/z.askapache.com/ gator:'~/public_html/z/' 1>/dev/null
@midnight /usr/bin/find ~/z.askapache.com/ -type d ! -perm 755 -exec chmod 755 {} \; 1>/dev/null
@midnight /usr/bin/find ~/z.askapache.com/ -type f ! -perm 644 -exec chmod 644 {} \; 1>/dev/null

Those 2 find commands scheduled to run at midnight simply fix and permissions on files and directories in my static folder. They are all static files so there is no reason they need to have any other permission.

Only Allow rsync

This is SWEET! If you like candy that is.. It’s called each time anything logs into your machine using the password less key we created above, and it simple checks what command the login process is attempting to issue. To set this up you need to edit the ~/.ssh/authorized_keys file on the remote host and prefix your public key that you added with a command directive to execute a script:

command="/home/remoteuser/validate-rsync.sh" ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwBhj6UCS7JbJ08C8pWJqCh2iXZMN7tXpYZh47f4gZZBwrNHZQ== localuser@dreamhost

validate-rsync.sh

For rsync requests it will always be rsync --server at the start of the command, so if the command is anything else then this script:

1. Sends you an email notifying you somethings up..
2. Moves the ~/.ssh folder to ~/.locked-ssh
3. Adds cronjob to move the folder back in about an hour.

#!/bin/bash
# Author: http://www.askapache.com
# Version: 1.2
# Date: 04-08-2009
 
# If the command used to login to ssh correctly starts with 'rsync --server'
# then exit this script and dont process the rest of the script
case "$SSH_ORIGINAL_COMMAND" in 'rsync --server'*) exit 0; ;; esac;
 
# the home directory where the .ssh folder is located
H=/home/remoteuser
 
# if there is a locked ssh folder, kill the rsync and die
[[ -d $H/.lssh ]] && echo "REJECTED" && exit 1
 
EMAIL=webmaster@askapache.com # notified about locked status
OC=$H/old-crontab.txt # the original crontab
NC=$H/new-crontab.txt #the new crontab
 
# When to unlock the rsync
UNLOCK_AT=$(( date +%M\ %k --date='30 minute 1 hour' ))
 
# move the .ssh to .lssh which locks all key-logins
mv $H/.ssh $H/.lssh
 
# mail a notice to the boss
echo $SSH_CONNECTION | mail -s 'RSYNC LOCKED!!!' "$EMAIL"
 
# subshell backs-up crontab then deletes active cron
(

 crontab -l > $OC &>/dev/null || echo -n #backup current crontab
 crontab -r >/dev/null 2>&1 || echo -n # delete current crontab
)
 
# subshell creates new crontab combined with old crontab
(
 # create new crontab
 echo -en "MAILTO='${EMAIL}'\n${UNLOCK_AT} * * * mv $H/.lssh $H/.ssh" >> $NC
 echo -n " && date|mail -s 'UNLOCKED!!!' '${EMAIL}' && crontab $OC || rm $OC && rm $OC" >> $NC

 # add old crontab to new crontab minus any MAILTO lines
 cat $OC | sed '/^MAILTO/d' >> $NC

 # load the new crontab and if it doesnt work notify boss
 crontab $NC || echo "manually mv .lssh to .ssh" mail -s 'CRONTAB PROBLEM!!!' "$EMAIL"

 # remove new crontab
 rm $NC
)
 
exit $?

Here is the cronjob entry it creates… This will enable the rsync again by moving the folder back, then it mails you to alert you that its back up, and finally the original crontab is restored.

MAILTO="askapache@gmail.com"
03 9 * * * mv ~/.locked-ssh ~/.ssh && date|mail -s 'RUNLOCKED!!!' "webmaster@askapache.com" && crontab ~/old-crontab.txt

Rsync/SSH Debugging and Stats

Adding the option -v to the ssh command, ie rsync -e 'ssh -vv' will give you a lot of debugging info.

By adding –stats to your rsync command you can get a detailed look at its efficacy.

Number of files: 14900
Number of files transferred: 0
Total file size: 1456832331 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 320551
File list generation time: 17.393 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 320571
Total bytes received: 20
 
sent 320571 bytes  received 20 bytes  17329.24 bytes/sec
total size is 1456832331  speedup is 4544.21

Rsync Algorithm

The rsync utility uses an algorithm (invented by the Australian computer programmer Andrew Tridgell) for efficiently transmitting a structure (such as a file) across a communications link when the receiving computer already has a different version of the same structure.

The recipient splits its copy of the file into fixed-size non-overlapping chunks, of size S, and computes two checksums for each chunk: the MD4 hash, and a weaker ‘rolling checksum’. It sends these checksums to the sender. Version 30 of the protocol (released with rsync version 3.0.0) now uses MD5 hashes rather than MD4.

The sender computes the rolling checksum for every chunk of size S in its own version of the file, even overlapping chunks. This can be calculated efficiently because of a special property of the rolling checksum: if the rolling checksum of bytes n through n + S − 1 is R, the rolling checksum of bytes n + 1 through n + S can be computed from R, byte n, and byte n + S without having to examine the intervening bytes. Thus, if one had already calculated the rolling checksum of bytes 1–25, one could calculate the rolling checksum of bytes 2–26 solely from the previous checksum, and from bytes 1 and 26.

The rolling checksum used in rsync is based on Mark Adler’s adler-32 checksum, which is used in zlib, and which itself is based on Fletcher’s checksum. The sender then compares its rolling checksums with the set sent by the recipient to determine if any matches exist. If they do, it verifies the match by computing the MD4 checksum for the matching block and by comparing it with the MD4 checksum sent by the recipient.

The sender then sends the recipient those parts of its file that did not match any of the recipient’s blocks, along with assembly instructions on how to merge these blocks into the recipient’s version. In practice, this creates a file identical to the sender’s copy. However, it is in principle possible that the recipient’s copy differs at this point from the sender’s: this can happen when the two files have different chunks that nonetheless possess the same MD4 hash and rolling checksum; the chances for this to happen are in practice extremely remote.

If the sender’s and recipient’s versions of the file have many sections in common, the utility needs to transfer relatively little data to synchronize the files.

While the rsync algorithm forms the heart of the rsync application that essentially optimizes transfers between two computers over TCP/IP, the rsync application supports other key features that aid significantly in data transfers or backup. They include compression and decompression of data block by block using zlib at sending and receiving ends, respectively, and support for protocols such as ssh that enables encrypted transmission of compressed and efficient differential data using rsync algorithm. Instead of ssh, stunnel can also be used to create an encrypted tunnel to secure the data transmitted.

Finally, rsync is capable of limiting the bandwidth consumed during a transfer, a useful feature that few other standard file transfer protocol offer.

This page contains content by Author of Article from Wikipedia and is licensed under the GNU FDL.