LINUX: Checking total size of directories

To find the total usage per user for their home directory without finding info on each subfolder, use the du command specifying the max-depth to be = 1 so that subdirectories are not shown on the screen, but they are still calculated in the total.

$ du -h --max-depth=1

Then to sort this list, redirect it to a file such as mysize and run the following to get an ascending list of offenders in the hundreds of MB:

$ du -h --max-depth=1 > mysize

$ grep -e [0-9][0-9][0-9]M mysize | sort

And of course run a simple grep G mysize to see gigabyte offenders.

Changing Line Endings with VI

dos2unix is an easy and popular way to change a file from DOS line endings to Unix line endings. But it isn't foolproof...

When dos2unix doesn't seem to get all the line endings, within VI or VIM or GVIM, try:

:g/^M/s///g

where ^M means ctrl-V ctrl-M

or try

:set fileformat=unix

Note, for those who don't know how to use VI, there are tons or resources online, but a quick crash course (maybe I'll do more in another post): Typing vi, vim, or gvim will typically launch a vi-like editor. They all work by typing a colon (:) to do special commands like save, quit, and much more. To begin typing, you have to tell it to go into insert mode by simply typing "i" for insert (or "a" for append to end of line or "o" for new line then insert). It seems complicated, and it is, but you get used to it and then it is super powerful. Use :x to save and quit when done.

Changing permissions in Linux

Have you ever needed to change permissions on directories (or files) in Linux, but sometimes need to only do it for a certain set? There are a bunch of different tricks to get the job done.

Take as a simple example, wanting to change permissions on all directories so that any future files created under them will have this permission. If you wanted to preserve the files as is, but put what is called a sticky bit onto the directories for all future files, do the following:

$ find -type d | xargs chmod g+s

The left side of the pipe lists all directories from where you are. The right side of the pipe will set the group bit permissions on all dirs in the find list on the left side of the pipe. The group bit will force all files created in those directories from now on to be owned by the user creating it and by the group of the parent directory.

How about if you wanted to find out which file/folders are NOT owned by a certain group (perhaps to change their ownership or permissions)?

find ! -group <name_of_group>

This will list all files/folders that are NOT owned by the group mentioned. You can then put that into a pipe to change ownership or permissions as shown above.

Search & Replace using Perl command

Searches for all files recursively that contain <string2replace> (put any string you want here, this is just for my example) and prints out a list of just the filenames.
The xargs command then passes that list to perl. Our friend perl then does -p to run script on every line of the file, -i to edit in place, -e run the following script (input from command line), 's/???/!!!/g' replaces ??? with !!! and /g says to do it for multiple occurrences on the same line. Use the first command without -i and piped to less to just see an output of all the changes it will make if using -i just to check first. The -i does an inline replace and that is permanent (shows nothing on screen). Without the -i it sends the expected results to standard out (shows it on the screen) but doesn't actually do the change. Think of it as preview-mode.

First test it to see if it looks good.

$ grep -lr <string2replace> . | xargs -n1 perl -p -e 's/<string2replace>/<replacement>/g' | less

Do it for real.

$ grep -lr <string2replace> . | xargs -n1 perl -p -i -e 's/<string2replace>/<replacement>/g'

CodeProject

Using special characters in a blog or XML files

Well, I just had to learn this one for the last post. Certain characters are reserved for HTML and if you place them in your post it will not display properly. Technically speaking, this has to do with encoding standards and such, but that is relevant to what we are doing.

I've come across this problem when automatically generating XML files as well. We have a test harness at work called CPPUNIT that has an option to output results to an xml file. This ASSERTS or error messages are whatever the developer defines in the code, but they often won't think about how that affects XML. XML has these special character restrictions as well (just like HTML for this blog) and if it finds any of them in the XML file, the file becomes invalid. We had to do a search and replace of these special characters in order to ensure our XML files don't become invalid.

Anyway, on to the answer. If you need to place <, >, &, ", or ', then use the following table. Just type in the weird character string with semi-colon at end wherever you want the special character to appear in your blog post or XML file.

< ....... &lt;
> ....... &gt;
& ....... &amp;
" .......  &quot;
' .......   &#39;

There are much more for HTML, but the above are what I find to be the most common and all that you need.

How do I SCP files between machines?

To transfer a file to another machine:

scp <local_filename> <username>@<destination_server>:<remote_location>

Reverse the above if you want to transfer a file from another machine.

CodeProject

EXT3 Error

Have you ever had a Linux server lock up and looking at the output directly on the local monitor will just continuously report the following?

EXT3-fs error (device md0) in start_transaction: Journal has aborted

One possible solution is to type the following:

$ echo 1 > /proc/sys/kernel/sysrq
$ echo b > /proc/sysrq-trigger

CodeProject