DranoK's Linux Wisdom

DIY Render Farm

DranoK — 2008-04-24T15:36:52-05:00

Nice.

This is the story of Helmer. A linux cluster in a IKEA Helmer cabinet.

Quote:

3D computer rendering are very CPU intensive and the best way so speed up slow render problems, are usually to distribute them on to more computers. Render farms are usually very large, expensive and run using ALLOT of energy. I wanted to build something that could be put in my home, not make too much noise and run using very little energy... and be dirt cheep, big problem? :) no computer stuff cost almost nothing these days, it just a matter of finding fun stuff to play with.

Singularity, Part IV: Performance Testing

DranoK — 2008-01-25T13:19:21-06:00

Singularity, Part IV: Performance Testing

Articles in this series
Singularity, Part I: Confessions of a Digital Pack Rat
Singularity, Part II: The Network Upgrade
Singularity, Part III: Construction
Singularity, Part IV: Performance Testing
Singularity, Part V: Conclusion

Intro

With Sin successfully built the big question on our minds was how ZFS would compare to ext3 + md software raid on our hardware. To discover this we spent the first weekend after the server was built running the benchmarks below.

Setup

Unfortunately Ubuntu out of the box had problems with the two array drives sitting on the motherboard rather than the PCI-X controller. We found a bug report for it but didn't feel like updating the kernel just to fix it. Instead we built an 8-disk array instead of a 10-disk for these tests. We're now running a full 10-disk array which obviously performs slightly better than the 8-disk benchmarks below.

What impressed me most about ZFS is that it doesn't need to construct the new disk array. On Linux, md must construct the array, usually by flagging one of the disks as faulty and rebuilding. For both the raid5 and raid6 setups below this took over nine hours under Ubuntu. ZFS, by contrast, doesn't (at least as long as you give it entire disks to use). The array was available for us immediately.

Our tests involved unpacking (not uncompressing) the Linux 2.6.23 kernel tarball, writing a 1GB dd file filled with zeros, and using a benchmarking utility called Bonnie++.

Click here for the rest of this article.

Video Codec Trojans

DranoK — 2007-12-27T17:08:39-06:00

Something to be aware of while downloading anime/movies/whatever:

http://www.securityfocus.com/brief/650

Quote:

Trojan horse programs dressed up like video decoders, or codecs, have become a popular way to attempt to infect the computers of unwary Web surfers.

In other news, Singularity is built and running, I've just been bogged down with other things to finish writing it. Hopefully part 4 will be up tomorrow.

Singularity, Part III: Construction

DranoK — 2007-12-06T02:25:00-06:00

Singularity, Part III: Construction

Articles in this series
Singularity, Part I: Confessions of a Digital Pack Rat
Singularity, Part II: The Network Upgrade
Singularity, Part III: Construction
Singularity, Part IV: Performance Testing
Singularity, Part V: Conclusion

Intro

This part has been delayed a bit. First, at least one memory DIMM was bad. Second, the motherboard had two fried SATA ports. Thrid, it was Thanksgiving weekend, so getting RMAs placed was nearly impossible.

Still, we got everything up and running. I won't do much more talking here. This part is mainly for the pics! ;)

Click here for the rest of this article.

Long live IPv4

DranoK — 2007-11-27T12:29:48-06:00

This is why IPv4 will be around for a long tome to come.

Singularity, Part II: The Network Upgrade

DranoK — 2007-11-15T16:53:17-06:00

Singularity, Part II: The Network Upgrade

Articles in this series
Singularity, Part I: Confessions of a Digital Pack Rat
Singularity, Part II: The Network Upgrade [Image 1] [Image 2]
Singularity, Part III: Construction
Singularity, Part IV: Performance Testing
Singularity, Part V: Conclusion

Home Gigabit Network

Gigabit networking has recently become an option for consumer budgets. This advance has been fueled by the proliferation of cheap, unmanaged switches by Netgear and Cisco.

Most of these were four-port or possibly eight-port creatures, however--most consumers didn't have a need for more than this. Businesses still preferred more expensive managed switches for their production network, and their office network often didn't need it. Especially if everyone is in love with wireless.

This is changing, slowly, as people become disillusioned with cubical wireless and trend back toward wired access. As storage continues to increase the uses for it explode. More and more people start to demand gigabit speeds.

Many corporations have already deployed gigabit managed switches for their office environments. Many small businesses, however, have not. Larger-capacity unmanaged gigabit switches cater to small businesses like these, as well as the enthusiast market.

The Gigabit Myth

People often run network tests and find they're not getting "true" gigabit speeds (128 MiB/s). Typically they blame the switch. To some extent they're correct--cheap, unmanaged switches rarely perform well in a single transfer vs. their pricier cousins. There are other reasons, however.

Even with a RaptorX SATAII drive, the absolute best-case transfer rate you're going to get (buffer -> disk) is 84 MB/s. I'm not sure if this is MB or MiB--regardless it's a far cry from 128MiB/s. And that's just your raw 10k disk speed. If your network ports are built into your motherboard you're going to have a slowdown any time your CPU is busy (like when it's buffering and writing disk data). PCI network cards are typically no better, unless it's a PCI-X or PCI-E with offloaded processing.

Where gigabit networks shine, however, is in multiple transfers. Sure, any single given device may not seem like it's taking advantage of gigabit--but when two or more devices are transferring data at the same time you will definitely notice a difference.

The New Network

We jumped to gigabit for the first time when we built Sol. We quickly ran out of ports, however, and have been hobbling along with a 16-port 10/100 Linksys trunked switch ever since.

With cheaper, higher-capacity gigabit switches now available we took this opportunity to scoop one up.

We also had a cabling rat's nest problem, as most people probably do. You never have the right length of ethernet cables so inevitably end up with 15ft monsters connecting devices which are two feet from each other. We decided we wanted to buy our own cabling kit and crimp new cables ourselves so we could better control the length. We found a good deal on a kit:

We especially enjoyed the Engrish instructions :)

Beyond Gigabit

Now that we had an excess of ports it was time to push Sol beyond gigabit speeds. Since we had two gigabit NICs available here we decided to aggregate them into a single, bonded device.

Network bonding is a method of combining two independent NICs into a single virtual device to provide both redundancy and higher throughput. Under Linux there are many different network bonding modes available, from simple round-robin load balancing to advanced load balancing. For our purposes we chose mode 6, or "Adaptive load balancing". The actual details of network bonding is beyond the scope of this document; more information can be found in your kernel distribution (/usr/src/linux/Documentation/networking/bonding.txt)

Now our current fileserver is running at two gigabit while all our other nodes are at regular gigabit. We are now I/O bound on Sol, however this should speed up simultaneous access. As an added bonus, we can run new ethernet cables and swap ports without causing a network interruption.

Next: Singularity, Part III: Construction

Singularity, Part I: Confessions of a Digital Pack Rat

DranoK — 2007-11-07T16:03:32-06:00

Singularity, Part I: Confessions of a Digital Pack Rat

Articles in this series
Singularity, Part I: Confessions of a Digital Pack Rat
Singularity, Part II: The Network Upgrade [Image 1] [Image 2]
Singularity, Part III: Construction
Singularity, Part IV: Performance Testing
Singularity, Part V: Conclusion

Prologue

My desk is many men's worst nightmare. Papers are strewn everywhere. Pencils pour from every crevice. Network charts which should be on the wall are buried under stacks of paper and cdroms. That paperweight given to me as a gift is probably at the bottom of my trash bin--I trash anything that's not absolutely needed. If I didn't I'd drown.

It's not dirty, you understand. It's cluttered. Disorganized.

It's a stark contrast to my digital world. Here I'm a meticulously-organized pack rat. I keep everything I download--I probably still have the NVidia drivers for my original GForce card somewhere under my vast hierarchical construct.

Such an addiction comes at a cost. My habit quickly outpaced available storage. Luckily for me we live in the future, where gigantic fileservers can be assembled from consumer parts. Where a digital pack rat can thrive.

This is the story of the singularity--the fileserver that will succeed sol and retire jupiter. The third incarnation of primal consumer storage.

Click here to view the full article.

Scheduling jobs with at instead of cron

DranoK — 2007-08-20T12:45:40-05:00

Most Linux admins are well-versed with crontab. All too often, however, I see people using crontab to schedule a job that's only intended to run once. This not only clutters your cron file but can potentially be dangerous if you forget to un-schedule your job before it runs again.

Luckily there's an alternative: at

Prerequisites:

o Familiarity with the bash shell

The at command simply runs a job once at some specified point in the future. Unlike crontabs, which will only use your user environment if you explicitly tell it to, at will use your environment by default.

The basic invocation patterns of at are:

Listing 1: Basic invocation:

1) at time
2) at time < /file/with/commands
3) echo "some command" |at time

In the first example after pressing enter you will be dropped into the at> shell. Here type the commands, one per line, you would like to be executed.

The second example is non-interactive. Instead of typing your commands into the at> shell, put them into the /file/with/commands file instead. Note that you do not put a shabang (#!/bin/sh) at the top of your script! If you have a script you want to run, do not specify "at time < /path/to/script"--instead create a new text file with one line, "/path/to/script" in it, and use that file as your source to at (at time < /path/to/newfile).

The third example is useful if you want to run a single command at a specified time. For example, "echo /path/to/script |at time"

Click here to view the rest of the article

How to tunnel all traffic over DNS

DranoK — 2007-07-25T17:44:37-05:00

Wow, this tickles me in all the right ways. Encapsulating base32-encoded data over TXT records. Brilliant.

http://dnstunnel.de/

Quote:

Well, since all subdomain resolve requests are delegatet (ie., relayed) to your host, you can include arbitrary data in the hostname which your server then can interpret and execute/relay.

The bytes you want to send to the server (upstream) will be encoded using Base32 (if you know what Base64 is, Base32 is just the same except there is no case sensivitiy, for EXAMPLE.COM ist just the same as example.com). After the data, there is a unique ID (since some DNS requests may take longer than others and the UDP protocol has no methods to check this) and either one of the keywords up or down, indicating whether the traffic's up- or downstream. Here is what an example request could look like (transferring something to the server):

ntez375sy2qk7jsg2og3eswo2jujscb3r43as6m6hl2ws
xobm7h2olu4tmaq.lyazbf2e2rdynrd3fldvdy2w3tifi
gy2csrx3cqczxyhnxygor72a7fx47uo.nwqy4oa3v5rx6
6b4aek5krzkdm5btgz6jbiwd57ubnohnknpcuybg7py.6
3026-0.id-32227.up.sshdns.feh.dnstunnel.de

The server's response comes as a DNS TXT record. A TXT record can hold arbitrary ASCII data and can hold uppercase letters as well as lowercase letters and numbers (some other characters, as well). So the responses come Base64 encoded. Such a response might look like the following one:

695-8859.id-39201.down.sshdns.feh.dnstunnel.de. 0 IN TXT
"AAAAlAgfAAAAgQDKrd3sFmf8aLX6FdU8ThUy3SRWGhotR6EsA avqHgBzH2khqsQHQjEf355jS7cT
G+4a8kAmFVQ4mpEEJeBE6IyDWbAQ9a0rgOKcsaWwJ7GdngGm9j pvReXX7S/2oqAIUFCn0M8="
"MHw9tR0kkDVZB7RCfCOpjfHrir7yuiCbt7FpyX8AAAABBQAAA AAAAAAA"

Make a script behave differently if run from cron

DranoK — 2007-07-11T12:48:41-05:00

It can be useful at times to have a shell script behave differently if it's run interactively vs. run from cron or other non-interactive means. For example, you may want the script to send email on certain conditions if it's run from cron, but simply print the message to standard out if it's being run interactively.

Prerequisites:

o Familiarity with the bash shell
o Shell scripting experience

One solution is to wrap the interactive cron with a non-interactive script. This generally isn't the best method as it requires duplicating code in two separate scripts and possibly creating hacked-together ways for the script to interact.

A better solution is to test if the shell is a tty or not.

Listing 1: Test if a shell is a TTY

#!/bin/bash

tty -s
TTYTEST=$?

$? is a special bash variable which means the return code of the most recent command, in this case "tty -s". Unix commands generally return 0 on success, and some number greater than 0 on failure. For "tty -s", a 0 (success) return code means the shell is a TTY, while anything other than 0 means the shell is not. We can now use $TTYTEST to alter the behavior of our script.

Listing 2: Altering script flow based on TTY test

#!/bin/bash

tty -s
TTYTEST=$?

# Random script code here, until we want to do something different based on TTY

if [ $TTYTEST -gt 0 ]; then
  # Code to run if shell is NOT a TTY
else
  # Code to run if the shell IS a TTY
fi

The "-s" option to the tty command tells it to run silently. If you don't specify -s the tty command will print your TTY to standard out if you have one, or "not a tty" if you do not. If you don't want to deal with return codes you can do the following, however personally I think assigning a variable to the return code of "tty -s" is a more elegant way of doing this.

Listing 3: Alternate method to avoid using return codes

#!/bin/bash

TTYTEST=`tty`

# Random script code here, until we want to do something different based on TTY

if [ "$TTYTEST" = 'not a tty' ]; then
  # Code to run if shell is NOT a TTY
else
  # Code to run if the shell IS a TTY
fi

Note that $TTYTEST must be put in quotes since it is a string. The above method can be simplified into a one-line check if you only need to know if the shell is a TTY or not for one if statement.

Listing 4: Quick one-liner to test if a shell is a TTY

#!/bin/bash

if [ "`tty`" = 'not a tty' ]; then
  # Code to run if shell is NOT a TTY
else
  # Code to run if the shell IS a TTY
fi

Just as the $TTYEST variable had to be put in quotes since it was a string, so must `tty` when ran explicitly. Since `tty` will return a string, this must be wrapped in quotes as "`tty`" for it to be used.

Excellent Linux Shell Tricks

DranoK — 2007-06-20T17:31:23-05:00

Some of these are a bit advanced, but very well worth it. I'm not the only one who uses IPTables as a workaround to common problems ;)

http://www.venturecake.com/10-linux-...know-for-once/

Thankyou VentureCake!

A good IBM article on the Linux kernel structure

DranoK — 2007-06-07T12:42:35-05:00

http://www.ibm.com/developerworks/li...nel/index.html

I fucking love these developerWorks articles. They kick major ass. Below is a great (if very simplified) overview of how the different parts of a Linux-based OS work together.

There's many more images in the actual article, as well as succinct overviews of the major components. This article won't make you an expert, but at least it will give you a good idea of how things work ;)

Maintain a tail -f on rotating log files

DranoK — 2007-06-06T17:24:53-05:00

Prerequisites:

o GNU tail
o Familiarity with the bash shell
o Familiarity with the tail command
o Beginner's knowledge of what a file descriptor is

Running tail -f /path/to/somefile.log is one of the first commands a beginner learns. Aside from use on the command line, tail can be used in scripts as well.

Most log files rotate at some point or another (that is, they're renamed), and a new file starts being logged to. This is a problem since tail will follow the file even through a rename. This is because by default tail follows the descriptor, which doesn't actually change when the file gets moved.

Say you're tailing /var/log/messages. This file gets rotated to /var/log/messages.1 and a new /var/log/messages is created. By default your tail will still be following the descriptor for /var/log/messages.1, which probably isn't what you want.

To change this you need to make tail follow the name of the file rather than the file's descriptor. This can be done with:

Listing 1: Tail by name

tail --follow=name /var/log/messages

The normal "tail -f" is equivalent to:

Listing 2: Tail by descriptor (aka, tail -f)

tail --follow=descriptor /var/log/messages

This is a good start, but there's one additional problem. Normally, if a file becomes inaccessible for even a short period of time tail will quit. This is a problem, since there will almost certainly be a delay between when the existing /var/log/messages is moved to /var/log/messages.1 and the new /var/log/messages is created.

To overcome this problem we can use the "--retry" option:

Listing 3: Tail retry option

tail --retry --follow=name /var/log/messages

This is a bit bulky, so luckily GNU tail gives us a single flag to use instead, "-F". The "-F" flag is an alias for "--retry --follow=name":

Listing 4: Final, simplified command to tail by name

tail -F /var/log/messages

Easy as that ;)

Listing 5: Relevant section from the tail man page

       --retry
              keep trying to open a file even if it is inaccessible when  tail
              starts  or  if it becomes inaccessible later -- useful only with
              -f

       -f, --follow[={name|descriptor}]
              output appended data as the file grows; -f, --follow, and --fol-
              low=descriptor are equivalent

       -F     same as --follow=name --retry

Temporarily force a server to queue outgoing mail

DranoK — 2007-05-16T16:11:52-05:00

Email is often used as a tool for servers to send reports, notifications and customer correspondence. It can be difficult to troubleshoot the content of messages, however, without resorting to lots of messy code changes or setting up a network of test mailboxes.

This tutorial will show you how to temporarily stop all mail from being sent on a single server so you can inspect the individual mail messages and delete them before they're delivered to an actual person. Any legit mail will not be lost and will be sent once your debugging is complete.

Prerequisites:

o Familiarity with the bash shell
o Familiarity with firewall concepts
o Familiarity with the SMTP protocol
o Basic familiarity with the iptables command

Most web and app servers run a local MTA (Mail Transfer Agent) for system crons and users to send mail with. Frequently this MTA is bound to the localhost interface (127.0.0.1) so external hosts can't access it.

Most applications use their host's local MTA instead of directly opening an outgoing SMTP connection to some random host on port 25. Apps that don't directly use the local MTA are usually configured to use localhost (127.0.0.1) as their mail server so messages aren't lost if there is a network failure.

All mainstream MTAs store mail messages in a queue if they are not able to be immediately sent (if they are disconnected from the network, for example, or if the relay they are trying to send mail to is down). While a mail message is queued it is stored on the file system in plain text.

We have two requirements while trying to debug our app's outgoing mail:

1) We don't want our test messages to reach actual recipients who may be confused by the mail.
2) We want real mail that wasn't part of our test to be delivered normally when we are finished.

Yes, we could mess with the MTA's configuration to try to do this. We could modify the app sending the messages to store them to a file instead.

A much simpler method, however, is to use iptables instead.

Since we want all outgoing messages to be queued we need to prevent the MTA from reaching any external host over port 25 (SMTP). However, since our app may be trying to send mail to localhost (127.0.0.1), we must ensure that port 25 on localhost remains open. To do this we need two iptable rules:

Listing 1:

iptables -A OUTPUT -p tcp -d 127.0.0.1 --dport 25 -j ACCEPT
iptables -A OUTPUT -p tcp  --dport 25 -j REJECT

Most iptables rules are put on the INPUT chain--that is, they manage incoming traffic. In this case we use "-A OUTPUT" because we want to manage outgoing traffic.

The first rule says to ACCEPT all tcp traffic going to localhost (127.0.0.1) on port 25 (SMTP).

The second rule says to REJECT all tcp traffic going to any destination on port 25.

Since the rule to accept localhost traffic is above the rule to deny all outgoing port 25 traffic we are still able to connect to port 25 of localhost. Also note that it is important to use REJECT instead of DROP, since REJECT will cause the MTA's outgoing connection to fail immediatly, while DROP would cause it to hang until it times out.

Running "iptables -L" should now report something similar to the following:

Listing 2:

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     tcp  --  anywhere             localhost.localdomaintcp dpt:smtp
REJECT     tcp  --  anywhere             anywhere           tcp dpt:smtp reject-with
icmp-port-unreachable

Now all mail being sent will be queued. Your MTA's log file will most likely say the mail has been deferred due to a connection timeout.

Where these messages are stored is entirely dependent on your MTA software and how it was configured. In the case of sendmail, they will likely be in /var/spool/mqueue. Sendmail separates the envelope header from the body. The reasons for this are outside the scope of this article, however it is sufficient to explain that you may need to look at two separate files in order to see an entire email.

Different MTAs use different naming schemes and IDs for these files, however it is generally fairly simple to figure out what each of these files is. Once you find where your MTA stores it's queued messages it shouldn't be difficult to find the debug info you were looking for regardless of how much or little you know about the SMTP protocol and your specific MTA.

Once you have what you need you should delete any files that you don't want sent. Naming schemes vary, but in the case of sendmail all queue files are a string of letters followed by the message ID. It's generally safe to just replace the first 3-5 characters of a file you want to delete in the queue directory with * so you delete all files associated with that message.

For example, if you need to delete a file named "dfj2BIK8802525", use "rm *K8802525" to make sure you delete all associated envelope files. If no legit mail was sent during this time you can safely remove all files in the spool directly.

Remove the iptables rules with "iptables -F" to clear the entire chain. Note that this will remove ALL iptables rules, not just the two you added above. If your had custom iptables rules setup you should either restore them after running "iptables -F" or selectively remove the two rules we added in this tutorial.

The queued mail will now be sent the next time the MTA processes its queue, generally within an hour. Most MTAs have the ability to force processing its queue--using sendmail, for example, you can issue "/usr/sbin/sendmail -q" to force processing of the queue.

See Also:

o IPTables
  http://en.wikipedia.org/wiki/Iptables
o The SMTP Protocol
  http://en.wikipedia.org/wiki/Smtp
o Sendmail
  http://en.wikipedia.org/wiki/Sendmail
o Postfix
  http://en.wikipedia.org/wiki/Postfix_%28software%29

Ugly files and how to get rid of them

DranoK — 2007-03-22T15:24:38-05:00

Prerequisites:

o Familiarity with the bash shell
o Familiarity with the find command
o Beginner's knowledge of what an inode is

I've often ranted about what I consider to be a lack of competency in the IT field. I blame much of this on the interview process--it's difficult to assess a candidate's skill by simply asking questions. Having a candidate sit down in front of a laptop and try to solve an actual problem instead can be much more useful. Below is one of many lab exercises I use.

Listing 1:

Problem:  You are tasked with removing the following files.
          You must remove each file individually.

Neptune:/tmp/example3 # ls -lrat
total 12
drwxrwxrwt 11 root   root  8192 Mar 22 12:50 ..
-rw-r--r--  1 dranok users    0 Mar 22 12:51 -this-file-starts-with-a-dash
-rw-r--r--  1 dranok users    0 Mar 22 12:51 数独」
-rw-r--r--  1 dranok users    0 Mar 22 12:52         how many spaces
-rw-r--r--  1 dranok users    0 Mar 22 12:52 immutable
drwxr-xr-x  2 dranok users 4096 Mar 22 12:52 .

I say each file must be removed individually because in real-life scenarios you're not going to be able to rm -rf the entire directory. In fact, this is a dangerous idea when you have files with strange characters--you may end up deleting something important. Further, there may be files in the directory you need to keep. Besides, running rm -rf on the folder won't always work.

I like this example because each of these files is increasingly difficult to remove.

Most people try to remove the immutable file first. They get the following error, even as root.

Listing 2:

Neptune:/tmp/example3 # rm immutable
rm: remove write-protected regular empty file `immutable'? y
rm: cannot remove `immutable': Operation not permitted

Nope, this isn't an NFS share. so why can't root delete this file? We'll come back to this one.

The "how many spaces" file (which starts with a number of spaces) is the easiest to remove. Just use a wildcard and this one is simple:

Listing 3:

Neptune:/tmp/example3 # rm *how*
Neptune:/tmp/example3 # ls -lrat
total 12
-rw-r--r--  1 dranok users    0 Mar 22 12:51 -this-file-starts-with-a-dash
-rw-r--r--  1 dranok users    0 Mar 22 12:51 数独」
-rw-r--r--  1 dranok users    0 Mar 22 12:52 immutable
drwxrwxrwt 11 root   root  8192 Mar 22 13:00 ..
drwxr-xr-x  2 dranok users 4096 Mar 22 13:04 .

The next easiest is the "-this-file-starts-with-a-dash" file. Most people tend to struggle with this one.

Listing 4:

Neptune:/tmp/example3 # rm -this-file-starts-with-a-dash
rm: invalid option -- t
Try `rm ./-this-file-starts-with-a-dash' to remove the file
Try `rm --help' for more information.
Neptune:/tmp/example3 # rm *dash
rm: invalid option -- t
Try `rm ./-this-file-starts-with-a-dash' to remove the file
Try `rm --help' for more information.

Even wildcards won't work here. The problem is that rm is interpreting the dash as an argument. The easiest answer is to remember that modern Linux commands use getopt, and that two dashes means "end of options":

Listing 5:

Neptune:/tmp/example3 # rm -- *dash
Neptune:/tmp/example3 # ls -lrat
total 12
-rw-r--r--  1 dranok users    0 Mar 22 12:51 数独」
-rw-r--r--  1 dranok users    0 Mar 22 12:52 immutable
drwxrwxrwt 11 root   root  8192 Mar 22 13:00 ..
drwxr-xr-x  2 dranok users 4096 Mar 22 13:08 .

Next we'll tackle the file that contains kanji (数独」). If your terminal (or web browser) doesn't have the Japanese fonts installed this likely shows up as a series of boxes. How do you selectively remove this file?

The answer is to delete the file via its inode. Among other things, an inode is a unique numerical ID for every file on your filesystem.

To see these inode values, use "ls -li":

Listing 6:

Neptune:/tmp/example3 # ls -li
total 0
2573051 -rw-r--r-- 1 dranok users 0 Mar 22 12:52 immutable
2573049 -rw-r--r-- 1 dranok users 0 Mar 22 12:51 数独」

Now we know the inode is 2573049. To remove it, we invoke the find command:

Listing 7:

Neptune:/tmp/example3 # find . -maxdepth 1 -inum 2573049 -ok rm {} \;
< rm ... ./数独」 > ? y
Neptune:/tmp/example3 # ls -lrat
total 12
-rw-r--r--  1 dranok users    0 Mar 22 12:52 immutable
drwxrwxrwt 12 root   root  8192 Mar 22 13:12 ..
drwxr-xr-x  2 dranok users 4096 Mar 22 13:13 .

Easy enough. We use "-ok" here instead of "-exec" since "-ok" will prompt us before running the command. Removing files with strange characters can be dangerous so it's always a good idea to take precautions. We used "-maxdepth 1" to ensure find wouldn't recurse into subdirectories.

That leaves us with that "immutable" file. At this point, even trying to "rm -rf" the directory will fail:

Listing 8:

Neptune:/tmp # rm -rf example3
rm: cannot remove `example3/immutable': Operation not permitted

The first thing to check when root cannot delete a local (non-NFS) file is the extended filesystem attributes. This can be achieved with the "lsattr" command:

Listing 9:

Neptune:/tmp/example3 # lsattr
----i-------- ./immutable

The "i" in the series of dashes means this file is immutable, or write-protected. To remove this flag from the file you must use the change attribute command, chattr:

Listing 10:

Neptune:/tmp/example3 # chattr -i immutable
Neptune:/tmp/example3 # lsattr
------------- ./immutable
Neptune:/tmp/example3 # rm immutable
Neptune:/tmp/example3 # ls -lrat
total 12
drwxrwxrwt 12 root   root  8192 Mar 22 13:15 ..
drwxr-xr-x  2 dranok users 4096 Mar 22 13:18 .

Congratulations! All the files have been removed.

Extended ext2/ext3 file attributes are very useful, however they are outside the scope of this article. Links are provided at the end of this article for more information.

See Also:

o Removing files with strange characters
  http://www.faqs.org/faqs/unix-faq/fa...section-1.html
  http://www.faqs.org/faqs/unix-faq/fa...section-2.html
o Inodes
  http://en.wikipedia.org/wiki/Inode
o The find command
  http://www.die.net/doc/linux/man/man1/find.1.html
o ext2/ext3 extended file attributes 
  http://www.die.net/doc/linux/man/man1/chattr.1.html

How to override DNS on a per-user basis

DranoK — 2007-01-24T17:03:45-06:00

Prerequisites:

o Familiarity with DNS, /etc/hosts and /etc/nsswitch.conf
   http://en.wikipedia.org/wiki/Name_Service_Switch
o Basic understanding of iptables
   http://www.netfilter.org/
   http://iptables-tutorial.frozentux.net/

Lazy QA/Staging

In the staging of production systems it is often desirable to override DNS to point at a local staging server instead of the production one. For example, you may have a large number of scripts that call "www.xyz.com"; changing all of these to "wwwtest.xyz.com" would be a pain in the ass.

Sure, there are good ways around this if you have a decent QA environment. You can use a special QA network with its own DNS server. To some extent you can use rule-based proxies.

A large number of solutions involve editing your local hosts file (/etc/hosts under UNIX, \system32\drivers\etc\hosts on Windows) to override DNS.

This works due to a file named /etc/nsswitch.conf. This very useful file contains the rules that programs use to resolve host names and other directory services (sendmail aliases, autofs, etc).

Basically, when a program wants to connect to "www.xyz.com" it makes a call to gethostbyname(), which consults /etc/nsswitch.conf.

A common /etc/nsswitch.conf might contain the following line:

Listing 1:

hosts:          files dns

This tells gethostbyname() to consult local files first (ie., /etc/hosts). If it can't find what it's looking for there it will consult DNS.

NOTE: Not all programs respect /etc/resolv.conf. For example, the "nslookup", "dig" and "host" commands will return DNS information regardless of what is in /etc/hosts.

Usually this works fine. You'll resolve www.xyz.com normally with the entry commented out in /etc/hosts and resolve it to the staging IP with the entry present.

The problem comes in when you want certain users to use DNS, and certain users to use the overrides in /etc/hosts.

Per-user DNS overrides

The important thing to remember about configuration files in /etc (/etc/hosts, /etc/nsswitch.conf, etc.) is that they are considered system-level confs. That is, they apply to all users. gethostbyname() doesn't care which user called it--it just blindly returns results based on the system-level rules in /etc.

The trick to making this work is understanding how /etc/nsswitch.conf works. Let's take a closer look at our above example, which specifies the search order as "files dns". This means it consults /etc/hosts first, and only if it fails or doesn't find an entry will it try DNS.

The key is, "if it fails."

It's highly unlikely /etc/hosts will ever fail--it's just a flat file. It is possible for DNS to fail, however. In fact, since this requires network access, we can make it fail.

Our goal is for some users to resolve www.xyz.com publicly, while forcing other users to hit staging. We can do this by providing public information via DNS and the staging IPs via /etc/hosts.

Change /etc/nsswitch.conf to consult DNS first, only falling back on /etc/hosts if that fails:

Listing 2:

hosts:          dns files

Now any entry we put into /etc/hosts for "www.xyz.com" will be ignored unless the DNS server fails to respond. The next step is to force DNS to fail for certain users.

Let's say user jdoe has a uid of 503. We want Joe to hit staging. using the "-m owner" feature of iptables we can block him from hitting udp/tcp port 53 (DNS).

Listing 3:

/sbin/iptables -A OUTPUT -p tcp --dport 53 -m owner --uid-owner 503 -j REJECT
/sbin/iptables -A OUTPUT -p udp --dport 53 -m owner --uid-owner 503 -j REJECT

User jdoe will now be unable to make DNS requests since udp/tcp port 53 outgoing is blocked. gethostbyname() will fail and thus fall back to /etc/hosts due to the options in /etc/nsswitch.conf. We can similarly add iptables rules for any user we want to hit staging.

You may need to restart your nscd service depending on how it's configured. nscd is a caching daemon that runs by default on some systems (Solaris, SuSE, etc.). More information can be found below:

http://linuxmanpages.com/man8/nscd.8.php

There is one problem with this method--all hosts must be in /etc/hosts. If jdoe gets bored of QAing and wants to visit slashdot.org, he can't do so unless this is in /etc/hosts or he removes the iptables rule.

Packet filtering with iptables is very powerful and robust. To learn more about it visit the two links provided above in the prerequisites section, as it is a must-have utility for any Linux professional.

Multiple LUNs under Linux 2.4

DranoK — 2007-01-11T17:24:20-06:00

Prerequisites:

1) Ability to build a custom kernel    
    (Refer to your specific distro's documentation)
2) SCSI device addressing
    http://www.tldp.org/HOWTO/SCSI-2.4-HOWTO/scsiaddr.html
3) The sg3_utils package
    http://sg.torque.net/sg/sg3_utils.html

System administrators often wonder why Linux can't use a sane naming convention for its disks like Solaris does--/dev/dsk/c0t1d0s2 is simple to understand--scsi host 0, bus 1, device 0, lun 2. Simple. Elegant.

On Linux 2.4 we get /dev/sdd1 (Under 2.6 we get sysfs and udev, neither of which is much better in my opinion).

Thankfully the sg3_utils package contains the "sg_map" command, which is insanely helpful. Let's take a basic system that has five disks:

Listing 1:

[root@host1 root]# sg_map -x
/dev/sg0  0 0 0 0  8
/dev/sg1  0 0 1 0  1  /dev/st0
/dev/sg2  1 0 0 0  0  /dev/sda
/dev/sg3  1 0 1 0  0  /dev/sdb
/dev/sg4  1 0 2 0  0  /dev/sdc
/dev/sg5  1 0 3 0  0  /dev/sdd
/dev/sg6  1 0 4 0  0  /dev/sde
/dev/sg7  1 0 6 0  3

The -x flag is required to print out the numbers in the middle. We only care about the first four numbers, which are, in order, "host, bus, device, lun" (the fifth digit is scsi type; read the sg_map man page for more info).

On this particular system we have a tape drive on scsi host 0; our five disks are on scsi host 1. They all share the same bus (0) and each disk has its own device id. All disks use only LUN 0, even though each disk is separated into different partitions (/dev/sda1, /dev/sda2, etc). Under Linux different partitions on the same disk do not become different LUNs.

When you're only dealing with individual disks this is annoying but straightforward. Problems can arise, however, If you have a SAN or direct-attached storage. Many of these will arrange their disk volumes by LUN.

Problem is, by default the 2.4 Linux kernel will *only* scan LUN 0.

To change this you must modify your kernel settings. Install your kernel sources as per your distro's documentation (compiling your own kernel is beyond the scope of this article). When you're at the configuration step (whichever you use: make menuconfig, make xconfig, etc) go down to the SCSI section and put a check in the line for "enable scanning of multiple LUNs" Save your .config and finish compiling the kernel as usual.

In your grub or lilo conf file, add "max_scsi_luns=128" to the end of your kernel line:

Listing 2:

title Red Hat Enterprise Linux WS (2.4.21-20.ELsmp-san)
        root (hd0,0)
        kernel /vmlinuz-2.4.21-20.ELsmp-san ro root=LABEL=/ console=tty0
console=ttyS0,9600n8 max_scsi_luns=128

Reboot onto your new kernel and you should now be able to see all your LUNs.

Listing 3:

[root@host2 root]# sg_map -x
/dev/sg0  0 0 1 0  0  /dev/sda
/dev/sg1  2 0 0 0  0  /dev/sdb
/dev/sg2  2 0 0 1  0  /dev/sdc
/dev/sg3  2 0 0 2  0  /dev/sdd
/dev/sg4  2 0 0 3  0  /dev/sde
/dev/sg5  2 0 0 4  0  /dev/sdf
/dev/sg6  2 0 0 5  0  /dev/sdg

This particular server has a single internal disk (/dev/sda on scsi host 0) and serveral SAN volumes (/dev/sdb-/dev/sdg on scsi host 0). Note that all these SAN volumes have the same bus and device identifiers; they are separated only by LUN.

This should be the end of this article, however due to a quirk in how Linux 2.4 handles LUN scanning one last important note must be addressed:

Linux 2.4 scans LUNs in order and will stop scanning if it finds a non-existent LUN.

This is particularly troublesome if you have a SAN and only want certain hosts to see certain volumes. Let's say you have six LUNs numbered 0-5. If you want one host to only see LUNs 0, 1 and 2 this is easy enough--disableing luns 3, 4 and 5 from either your HBA or the SAN itself will cause Linux no problems.

However, let's say you want a host to only see LUN 4. This will cause a problem--Linux will scan LUN 0, find nothing, and move on. It won't even attempt to scan LUNs 1-5. If you want a host to see LUNs 0, 1, 2, 4 and 5 it will only see the first three (if Linux doesn't see LUN 3 it won't bother trying LUNs 4 or 5).

This can lead to messy SAN setups and poses the risk of one machine accidentally mounting a disk it's not supposed to. I'm not aware of a clean solution to this at this time with Linux 2.4 other than separating SAN volumes at a device level instead of using LUNs. If anyone knows a good solution I'd be quite keen on learning about it.

Special Bash Variables I

DranoK — 2007-01-01T00:15:19-06:00

Bash has several built-in special variables. Although most are used only for scripting, there are several that are useful at the command line.

The two I will address in this post are !! and !$.

Repeat entire last line: !!

One of the useful parts of bash is its history system. You can see a history of your commands by typing history at the command prompt. The history command is considered a built-in command because it is not a binary that exists on the filesystem such as ls or uptime. The cd command is another example of a bash built-in.

Built-in bash commands will be covered in more detail in the future, as will the history command. For now, however, know that you can repeat history commands by typing "!" followed by the history ID.

Take the following example:

Listing 1:

dranok@Neptune:~> history
    1  which uptime
    2  cd /tmp
    3  ls -l
    4  uptime

The which command will tell you the full path of a program, assuming it is in a path specified by your $PATH environment variable. If I wanted to run the which uptime command again, instead of typing it out I could instead type !1:

Listing 2:

dranok@Neptune:~> !1
which uptime
/usr/bin/uptime

!1 gets expanded to the actual command which uptime. Note !1 doesn't get saved in your history--if you type history again you'll see the expanded which uptime command. Likewise if you press the up key you'll see the expanded version there as well.

Also note that when you use special bash variables the expanded command is printed on the subsequent line. This is helpful in figuring out exactly what the bash shell actually ran.

So how does this relate to !!? !! is the bash built-in for the last command in your history file--the last command you ran.

Listing 3:

dranok@Neptune:~> cat /tmp/happy
I'm happy!
dranok@Neptune:~> !!
cat /tmp/happy
I'm happy!

This is very useful in conjunction with the sudo command, among other things. For example you might run a command as a non-privileged only to discover you need root access. Running sudo !! is a fast way to achieve this.

Listing 4:

dranok@Neptune:~> cat /tmp/rootonly.txt 
cat: /tmp/rootonly.txt: Permission denied
dranok@Neptune:~> sudo !!
sudo cat /tmp/rootonly.txt 
This file can only be read by root.

Repeat last argument: !$

!$ is similar to !!, however instead of expanding to the entire last command it expands to the last argument of the last command. This is best illustrated by example.

Listing 5:

dranok@Neptune:~> cd /tmp/testdir
-bash: cd: /tmp/testdir: No such file or directory
dranok@Neptune:~> mkdir !$
mkdir /tmp/testdir

In this case we tried to cd into a directory which did not exist. Since the last argument of the preceding command was /tmp/testdir, running mkdir !$ is the equivalent of mkdir /tmp/testdir

Bash contains many other special variables like !! and !$, however these are two of the most ubiquitous and useful.

Free memory on Linux

DranoK — 2006-12-27T13:33:34-06:00

One of the frustrations I deal with on nearly a day-to-day basis is people's misconceptions about how memory is allocated under Linux 2.4 and later.

Listing 1: Tools to report memory usage

* top
* free -m
* cat /proc/meminfo

Let's take a typical example. The "free -m" output below is from a production system that has been running for 80 days.

Listing 2:

             total       used       free     shared    buffers     cached
Mem:          3956       3939         16          0         81       3504
-/+ buffers/cache:        353       3602
Swap:        16378          0      16378

This system has 4GB of physical memory and 16GB of swap. At first glance it looks like 99.57% of the memory is in use. Your first instinct would be to run "top", press capital "M" to sort by memory usage and find out what the hell is taking up so much memory. Doing so will yield no results, however--on this particular system top will show no process taking up more than around 100MB of memory. This can be confirmed with the "ps" command.

So where did all this memory go? The answer lies in the theory of memory management. In particular, the Linux memory management system believes in the following two principles:

1) No matter how much memory is in use, system performance will not be impacted until it is required to use swap space.
2) Unused physical memory is wasted.

Now we can tackle the first line of the "free -m" output.

Listing 3:

             total       used       free     shared    buffers     cached
Mem:          3956       3939         16          0         81       3504

The key lies in the last two columns: buffers and cached. The largest of these is cached. Whenever Linux accesses a file on a filesystem it stores the content of that file in memory. As long as the data doesn't change, subsequent requests for that file will be read from memory instead of physical disk. This vastly improves file access time.

Memory used in such a manner can be freed up almost instantly, thus should really be counted as free memory instead of used. The "-/+ buffers/cache" line of our "free -m" output reflects this.

Listing 4:

             total       used       free     shared    buffers     cached
Mem:          3956       3939         16          0         81       3504
-/+ buffers/cache:        353       3602

Actual used memory here is 353MB. Free memory is 3602MB (Don't add the original 16MB back to this--it's already accounted for). That means over 90% of this system's memory is free and available for use. Quite a bit different than only having 16MB of memory free.

The 3602MB value comes from adding the 3504MB of cache with the 81MB of buffers and 16MB of actual free memory. The 353MB of used memory comes from taking total memory and subtracting the "virtually free" 3602MB value. Note that due to rounding these values may be a bit off.