UNIX/Linux tips (potpourri)

(Started: August 2006)

I've been using Linux for almost a decade now, so over the years I've collected a number of digital notes on various issues. Some of them could be out of date - they are here mostly as an online reference for myself, but they may also prove useful to others...

Video and audio stuff
Administration
Coding, pdf, Tex, ...

Video and audio stuff

Different ways to deinterlace with MPlayer:

Blend: -vf pp=lb
Bob: -vf tfields=1 -fps 50 -ofps 50
Single field: -vf field=1
Swap fields: -vf phase=t
Adaptive: -vf kerndeint or -vf lavcdeint

Make a DVD with menus using tovid:

tovid -pal -dvd -in A-ha-TakeOnMe.mpg -out A-ha-TakeOnMe.vob

tovid -pal -dvd -in Every\ Breath\ You\ Take.mpg -out \
    Every\ Breath\ You\ Take.vob

makemenu -pal -dvd -align left -textcolor '#FFF' \
    -highlightcolor '#FF0?' -selectcolor '#F00?' \
    -font Helvetica \
    "A-Ha Take On Me" \
    "Police Every Breath You Take" 
    -background Earth.jpg \
    -out Main_menu 

makexml -dvd -menu Main_menu.mpg A-ha-TakeOnMe.vob.mpg \
    Every\ Breath\ You\ Take.vob.mpg MyPopDVD

makedvd MyPopDVD.xml

A one-liner equivalent to DVD Decrypter:
```
dvdbackup -i /dev/dvd -v 3 -o . -M
```

Create QuickTime-compatible .mp4 files:

From a source video at 25 fps, sized at 352x288, 
named test.mpg.

Audio:
bash$ mplayer -really-quiet -ao pcm:file=audio.wav \
	-vo null -vc dummy test.mpg
bash$ normalize audio.wav
bash$ faac -b 96 --mpeg-vers 4 -o audio.aac audio.wav 

Video:
    bash$ mkfifo stream.yuv
(from window 1):
    bash$ mencoder -nosound -ovc raw -noskip \
    	-vf format=i420 \
	-of rawvideo -o stream.yuv test.mpg
(from window 2):
    bash$ x264 --pass 1 --bitrate 256 --stats stats  \
    	--bframes 2 \
	--fps 25.0 -o test.264 stream.yuv 352x288 
    bash$ x264 --pass 2 --bitrate 256 --stats stats  \
    	--bframes 2 \
	--fps 25.0 -o test.264 stream.yuv 352x288 

multiplexing:
MP4Box -add test.264:fps=25 -add audio.aac test.mp4

Batch reading of CDDB entries for many CDs over dialup Internet:

Insert the audio CDs, one by one, and run 
cddb.pl -o
Then, when you've passed them all, connect to the web and
cddb.pl -g
Use the screen output (Perl)...

Flawless ripping of audio CDs:

cdparanoia -p -r  1 - | oggenc -Q -b 96 -r - -o track01.ogg
cdparanoia -p -r  2 - | oggenc -Q -b 96 -r - -o track02.ogg
...

Fool around with sox audio:

mpg123 -s -q voice.mp3 | sox -t raw -r 44100 -s -w -c 2 - \
    -t ossdsp -r 44100 -s -w -c 2 /dev/dsp pitch -490 30 \
    cubic cos
timidity -Ow ~/midi/Movie/swfinal1.mid -o - | sox -t wav - \
    -t ossdsp -r 44100 -s -w -c 1 /dev/dsp filter 0-16000

Cope with difficult VOBSUBs:

You need a complete dump of IFO and VOBs.
First, find out what your subtitles hex id is.
Sometimes, mplayer can't cut it (can't show the sub,
especially when the DVD is the result of DVDShrink).
In theses case, look at the output of...

bash$ mpegdemux -c -k -s all -p all  < VTS_01_1.VOB

and identify lines like:
0000400e: sid=bd[80] MPEG2 pts=25854[0.2873]
0008b80e: sid=bd[21] MPEG2 pts=605454[6.7273]
014ad80e: sid=bd[20] MPEG2 pts=5159454[57.3273]
014cf80e: sid=bd[24] MPEG2 pts=5177454[57.5273]
                  |
------------------'

These numbers (80, 21, 20, 24) are our candidates...
For each one, say 24, try:

cat VTS_01_1.VOB | tcextract -x ps1 -t vob -a 0x24 \
	> subtitle.ps

Now morph this subtitle.ps into a nice vobsub file by...

bash$ tar zxvf ~/stuff/transcode/subtitleripper-0.3-4.tgz
bash$ cd subtitleripper/
bash$ make
bash$ ./subtitle2vobsub -p ../subtitle.ps \
	-i VTS_01_0.IFO -o MAMA

A MAMA.sub and a MAMA.idx have been made.

Better sounding MIDI through timidity and sox:

timidity -Ow -o - xg/Vangelis\ -\ Finale.mid | \
    sox -t .wav - -t .wav - echo 0.1 0.9 | bplay

Forgot the IFO when encoding, VOBSUBs are black:

Simply add this line:

custom colors: ON,tridx:0000,colors: 000000,000000,FF0000,000000

in your .idx file. You might have to move the FF from the 3rd to
any of the other 3 values. If in doubt, permute :‑)

Add subtitles to a DVD from an SRT file:

First, create an SRT file from your text subfile:
    mplayer -dumpsrtsub  ...
    iconv -f UTF-8 -t WINDOWS-1253 ...

Then, lots of WINE...

Use VobEdit to split the input VOB file into its parts (MPEG file (video), 
and whatever audio and subs you want). Use txt2sup to create the SUP file 
from the SRT file - but take care to fix the 4 pallete indexes, so that 
they are all different (initially two of them point to 0!). 
Finally, use IfoEdit:

- DVD Author/new DVD
- Select the Video, audio and subs (SUP) files you want
- select the output dir
- when it finishes, change the palette colors:

In the upper part of the window select VTS_01_0.IFO, then VTS_PGCITI and 
finally VTS_PGC_1. Find the lines describing the subtitle color codes 
(Color 0 Y Cr CB, Color 1 Y Cr CB, Color 2 Y Cr CB and Color 3 Y Cr CB) 
and enter the new values:

Color 0 Y Cr CB = 00 80 80
Color 1 Y Cr CB = 55 80 80
Color 2 Y Cr CB = AA 80 80
Color 3 Y Cr CB = FF 80 80

Click on "Save" to save the modified VTS_01_0.IFO. Check the sub colors
with "mplayer -dvd-device /path/to/VIDEO_TS".

Finally, click "Disc Image" in IfoEdit. Done.

Mass conversion via tovid and make:

%.mpg: %.avi
        tovid -ntsc -dvd -noask -ffmpeg -in "$<" -out "$(basename $@)"

all: $(subst .avi,.mpg,$(wildcard */*.avi))

Creating a DVD slideshow:
Assuming you have a directory called img/ with images of lions, cheetahs, eagles, etc...

bash$ dir2slideshow \
    -o ./ -t 5 -c 1 -n Animals -notitle -p -a /path/to/music1.mp3,/path/to/music2.mp3 img/
(creates ./Animals.txt)

bash$ dvd-slideshow \
    -n Animals -o Animals -p  -smp -f Animals.txt
(creates a nice .vob file)

bash$ makexml Animals/Animals.vob -out MyDisc
(creates MyDisc.xml)

bash$ makedvd MyDisc.xml
(creates MyDisc/ folder, with VIDEO_TS/AUDIO_TS)

Re-multiplex broken H264 MKVs:

Use mkvinfo to identify the track IDs (TIDs), and then note the frame
rate of the video track. Once you figure out the track IDs...

bash$ mkvextract tracks Broken.mkv 1:video 2:audio 3:subs
bash$ mkvmerge --default-language eng -o Fixed.mkv \
		--default-duration 0:23.976fps video audio subs

That was enough to fix all the MKVs I've ever met...

Stabilize video recorded from shaky camera:

This is how I stabilized my iPhone videos:

1. First this:
transcode -J stabilize=accuracy=8,preview  -i IMG_0223.MOV \
    -x mplayer,null -g 640x480

2. Then this:
transcode -J transform -i IMG_0223.MOV -x mplayer,null \
    -g 640x480 -y xvid -o IMG_0223_stable.avi

3. Then extracting and encoding the audio:
mplayer -vo null  -benchmark -ao pcm:file=a.wav IMG_0223.MOV
lame a.wav

4. Finally, merging:
mkvmerge -o final.mkv -A IMG_0223_stable.avi a.wav.mp3

This is the script I used to create a BEFORE/AFTER video:

============
#!/bin/bash
transcode -i v1 -x mplayer,null -g 640x480 -y ppm,null -o f1_
transcode -i v2 -x mplayer,null -g 640x480 -y ppm,null -o f2_
for F in f1_*; do
    echo $F
    montage -background "#444444" -geometry +10+5 \
	$F f2${F#f1} f3${F#f1}
done
ls -1 f3_* > list
RES=$(identify -format "%wx%h" $(head -n 1 list))
transcode -i list -g $RES -V rgb24 -f 30,4 -x imlist,null \
    -y xvid,null -o v1and2.avi
rm f1_* f2_* f3_*
============

Encode screen-video for perfect web publishing via VideoLAN:

# for a 6-core machine:
ffmpeg.exe \
    -i recorded.avi \
    -vcodec libx264 \
    -vpre lossless_fast -crf 22 -threads 6 perfect.flv

Transcode full-HD video to 0.06 bits per pixel via mplayer and x264:
- 1920 x 1080 x 29.97 x 0.06bits/pixel = 3728747bits (3.7MBits)
- We use mplayer to decode the input (it decodes everything) and feed the output to x264 (i.e. the standalone, formal implementation of x264 from the company that makes the codec):
```
bash$ mplayer -nosound -benchmark -vo yuv4mpeg:file=>(x264 \
    --demuxer y4m --threads auto --pass 1 --bitrate 3728 \
    -o Amazing.Scenes.video.mkv - 2>x264.log) IMG_1123.MOV
bash$ mplayer -nosound -benchmark -vo yuv4mpeg:file=>(x264 \
    --demuxer y4m --threads auto --pass 2 --bitrate 3728 \
    -o Amazing.Scenes.video.mkv - 2>x264.log) IMG_1123.MOV
```
- Finally, we copy (a) the audio-only info from the original video (with -D) and (b) the newly encoded video (-A):
```
bash$ mkvmerge  -o AmazingScenes.mkv -A Amazing.Scenes.video.mkv -D IMG_1123.MOV
```
- If we also want to resize down to HD-ready, we should pass -vf scale=1280:720 to the two mplayer invocations.
The generated output showcases the supremacy of x264 - at 0.06 bits per pixel, I have difficulty separating the transcoded result from the original, when looking at both of them on my TV!

System administration, coding, pdf, Tex, ...

Disable LWP and Net::FTP for CPAN
Edit /etc/perl/CPAN/Config.pm and change dontload_hash to...
```
...
'dontload_hash' => {"Net::FTP" => 1, "LWP" =>1 }, # disable Net::FTP and LWP
...
```

Talk to an SSL-enabled site with non-SSL clients: stunnel

From window 1:
    stunnel -d 5000 -c -f -r www.semantix.gr:443
From window 2:
    non-ssl-wget http://localhost:5000

Make VMWARE images run faster:
Add these lines to the .vmx file:

sched.mem.pshare.enable = "FALSE"
prefvmx.useRecommendedLockedMemSize = "TRUE"
prefvmx.minVmMemPct = "100"
mainMem.partialLazySave = "FALSE"
mainMem.partialLazyRestore = "FALSE"
mainMem.useNamedFile = "FALSE"

Make VMWARE Player use more than two cores:

Edit your .vmx file and change the following:

config.version = "8"
virtualHW.version = "7"
numvcpus = "4"
maxvcpus = "4"
cpuid.coresPerSocket="2"

(remember to give adequate memory, too)
memsize = "1024"

Remove a Debian service from startup scripts:

update-rc.d -f nameOfServiceAsExistsUnderEtcInitD remove

Build debian package from its sources:
When I want to install a version of a program that only exists in the unstable branch, I don't want to mess up all my Debian installation - I just...
```
apt-get -b source  -t unstable pbzip2
```
...which creates a nice .deb file, installable with "dpkg -i whatever.deb". If the program has build dependencies, I install them first with...
```
apt-get build-dep pbzip2
```

Redirecting cerr to a stringstream:

// When you try this, don't output to cerr...
streambuf * cerr_strbuf(cerr.rdbuf());
ostringstream output;
cerr.rdbuf(output.rdbuf());
...
// Restore cerr
cerr.rdbuf(cerr_strbuf);

Using Perl as a super-grep:
I prefer to use Perl regexps instead of grep's. Here is a supergrep that uses Perl. It supports the '-i', '-n' and '-v' options that grep has (Update: or you can use pcregrep).

Migrate to a RAID1 config:

My non-RAID config was simple: I had one (and only one!) Linux partition, which was of course, my root partition: /dev/hda5. After adding a second (and much bigger) drive, I decided to migrate to a RAID1 configuration. I first used fdisk -u -l /dev/hda, to find out the size (in sectors) of my partition:

home root# fdisk -u -l /dev/hda

Disk /dev/hda: 40.0 GB, 40020664320 bytes
255 heads, 63 sectors/track, 4865 cylinders, total 78165360 sectors
Units = sectors of 1 * 512 = 512 bytes

   Device Boot    Start       End    Blocks   Id  System
/dev/hda1   *        63   8193149   4096543+   7  HPFS/NTFS
/dev/hda2       8193150  78156224  34981537+   5  Extended
/dev/hda5       8193213  78156224  34981506   83  Linux
home root# bc -l
bc 1.06
Copyright 1991-1994, 1997, 1998, 2000 Free Software Foundation, Inc.
This is free software with ABSOLUTELY NO WARRANTY.
For details type `warranty'. 
78156224-8193213
69963011               <============= The number of sectors =====

I then proceeded to create a partition with exactly the same size, on the new drive (hdb):

home root# fdisk -u /dev/hdb

The number of cylinders for this disk is set to 38913.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSes
   (e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p                                      --------- The number of sectors ---------,
Partition number (1-4): 1                                                       | 
First sector (63-625142447, default 63): 63                                     V
Last sector or +size or +sizeM or +sizeK (63-625142447, default 625142447): +69963011

Command (m for help): t
...(set the filesystem type for /dev/hdb1 to 0xfd, Linux raid auto-detect)
Command (m for help): p
...
Command (m for help): w
...
Command (m for help): q

I then proceeded to create the RAID1 device, md0, setting it up to only have the /dev/hdb1 partition, with the other one set to missing:

home root# mdadm --create /dev/md0 --level=1 --raid-disks=2 missing /dev/hdb1
home root# mkreiserfs /dev/md0
home root# mkdir /mnt/tmp
home root# mount /dev/md0 /mnt/tmp
home root# cp -a /boot /dev /root /.... /... /mnt/tmp/
home root# vi /mnt/tmp/etc/fstab (change the root to be /dev/md0)
home root# vi /boot/grub/menu.lst (change the root=/dev/hda5 to root=/dev/md0)

At this point, after a reboot, the kernel identified the /dev/md0 automatically, and the boot process went fine... Only one thing remained: to add the /dev/hda5 to the array /dev/md0, and allow it to automatically sync up:

home root# mdadm /dev/md0 -a /dev/hda5
home root# cat /proc/mdstat

The last command showed the process of synchronization. When it finished, my RAID1 setup was completed. My fears for faulty drives were finally put to rest...

Man pages with colors (ala OpenBSD console):

Put these in your .bashrc (or whatever your shell initialization script is):

	export LESS_TERMCAP_mb=$'\E[01;31m'
	export LESS_TERMCAP_md=$'\E[01;37m'
	export LESS_TERMCAP_me=$'\E[0m'
	export LESS_TERMCAP_se=$'\E[0m'
	export LESS_TERMCAP_so=$'\E[01;44;33m'
	export LESS_TERMCAP_ue=$'\E[0m'
	export LESS_TERMCAP_us=$'\E[01;32m'

Oh, and make sure 'less' is your PAGER (or MANPAGER).

Setup a RAID1 (mirror) of two loop devices:

bash$ blockdev --getsize /dev/loop0
2048000
bash$ blockdev --getsize /dev/loop1
2048000
(outputs must be the same!)
bash$ echo 0 2048000 mirror core 2 16 nosync 2 \
    /dev/loop0 0 /dev/loop1 0 | dmsetup create myraid1
bash$ ls -l /dev/mapper/myraid1
bash$ blockdev --getsize /dev/mapper/myraid1
2048000

You can find our more examples (e.g. stripe(RAID0) and linear) in the NTFS documentation that comes with the Linux kernel (Documentation/filesystems/ntfs.txt) - look at the Device Mapper section of the file.

Show package sizes under Debian:

dpkg-query -Wf '${Package}\t${Installed-Size}\n'

Create a minimal Debian for chrooted SSH:

root# debootstrap etch /opt/chroot \
	http://ftp.x.y/pub/linux/debian/
root# chroot /opt/chroot
....# adduser userlogin
....# logout
root# apt-get install libpam-chroot
root# echo 'session    required   pam_chroot.so' >> \
	/etc/pam.d/ssh
root# echo 'userlogin  /opt/chroot' >> \
	/etc/security/chroot.conf

...and then, modify the /etc/passwd and /etc/group IDs of the user to mirror those of the chrooted environment.

Use Debian alternatives to set multiple C/C++ compiler versions:

root# update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.6 10
root# update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.7 20
root# update-alternatives --install /usr/bin/cc cc /usr/bin/gcc 30
root# update-alternatives --set cc /usr/bin/gcc
root# update-alternatives --config gcc
root# 
root# update-alternatives --install '/usr/bin/g++' 'g++' '/usr/bin/g++-4.6' 10
root# update-alternatives --install '/usr/bin/g++' 'g++' '/usr/bin/g++-4.7' 20
root# update-alternatives --install /usr/bin/c++ c++ /usr/bin/g++ 30
root# update-alternatives --set c++ /usr/bin/g++
root# update-alternatives --config g++

Allow a user to be sudo-root:
```
root# visudo
...and add a line for account, say, rootfriend:
rootfriend<TAB>ALL=(root) ALL
```
That's it. User rootfriend can do anything root can, as long as he types his own password the first time he "sudo cmd args"...
Backing up with tar:
```
tar jcplSf /mnt/backup/`date +"%Y%m%d%H%M%S"`.tar.bz2 /
```
- j: compress with bzip2
- c: create volume
- p: preserve permissions
- l: stay in this filesystem
- S: handle sparse files
- f: store in this file

Converting a raw filesystem image to/from AES encryption using aespipe in one pass:

Encrypt:
dd if=/dev/XXX bs=512 | aespipe -e aes256 | \
	dd of=/dev/XXX bs=512 conv=notrunc

Decrypt:
dd if=/dev/XXX bs=512 | aespipe -d -e aes256 | \
	dd of=/dev/XXX bs=512 conv=notrunc

Mounting a loop-AES partition/DVD using cryptsetup:

sh# losetup /dev/loop0 /path/to/whatever/file/or/volume
sh# cryptsetup -c aes-plain -h sha512 create crypted /dev/loop0
Enter passphrase:
sh# mount /dev/mapper/crypted /mnt/heaven

Learning about a Samba LAN:

#!/bin/bash
for i in `nmap -sP 192.168.0.0/24 | grep 'appears to be up' \
	| perl -ne '/(192.168.\d+\.\d+)/ && print $1."\n";'`
do
        nmblookup -A $i
done

Using afio to back up (with gzip compression) to CD images:

Make it:
find / -xdev -print | afio -Z -G 1 -s 650m \
	-o /mnt/cdimage.afio
(will prompt you before overwriting cdimage.afio with 
 the next image...)

Use it:
make a symbolic link from Disk to Disk1,
    afio -Z -G 1 -s 0 -i Dsk
and when prompted, change the link to point 
to Dsk2, Dsk3, etc...

Backing up my NTFS partition at work over the network to the server:

Mount the ntfs partition on /iso1:
        mount -t ntfs -o iocharset=utf8 /dev/hda1 /iso1
or, depending on kernel version:
        mount -t ntfs -o nls=utf8 /dev/hda1 /iso1

and in the server, cd into /opt/Backup/Laptop-NTFS/ and run:
        rsync -avz dell:/iso1/ .

Convert a bitmap to a vector for use from LaTEX:

bash$ autotrace -filter-iteration 9 -output-file fig1.eps \
    fig1.pnm
bash$ epstopdf fig1.eps

and use graphicx in the .tex file:

\begin{figure}
\begin{center}
\includegraphics[width=0.85\textwidth]{../fig1} % fig1.pdf
\caption{Blah blah}
\label{figure1}
\end{center}
\end{figure}

Make a two-pages-per-A4 booklet from a PS file:

ps2pdf file.ps
acroread file.ps (print to file, Postscript Level 2)
psbook -s4 newfile.ps | mpage -2 -bA4 -m0 -o > booklet.ps

Cracking Windows passwords in case of human memory failure:

mkdir /var/tmp/a 
cd /mnt/ntfs/WINDOWS/system32/
cp -a config/ /var/tmp/a
cd /var/tmp/a
/usr/local/bin/bkhive ./config/system /tmp/op2.tmp
/usr/local/bin/samdump2 ./config/SAM /tmp/op2.tmp > \
    /tmp/ophcrack.tmp
ophcrack /tmp/ophcrack.tmp 1

Serving files over the Web in a one-liner:

bash$ cd /path/to/datafiles
bash$ python -m SimpleHTTPServer

Using multicores via xargs:
```
find . -type f -iname \*dds | xargs -n 1 -P 4 -I 'crap' convert crap crap.png
```
This command will find all .dds files and execute ImageMagick's "convert" on them... but it will do so by always keeping 4 instances of "convert", and thus keeping 4 cores busy.

Install stuff with CPAN:

bash$ sudo perl -MCPAN -e shell
cpan shell -- CPAN exploration and modules installation (v1.9402)
Enter 'h' for help.

cpan[1]> i /LinkCheck/
...
Module    Rubric::CLI::Command::linkcheck (RJBS/Rubric-0.144.tar.gz)
Module  = W3C::LinkChecker       (SCOP/W3C-LinkChecker-4.5.tar.gz)
...
11 items found

cpan[2]> install W3C::LinkChecker
...
cpan[3]> exit

bash$

Python code for sorting XML in preparation for diff:
Download from here, and use like this:

bash$ sortXML.py old.xml > old.xml.sorted
bash$ sortXML.py new.xml > new.xml.sorted
bash$ xxdiff old.xml.sorted new.xml.sorted

Automatically execute code whenever any files under a folder tree change:

#!/bin/bash

command="$1"
shift
fileSpec="$@"
sentinel=/tmp/t.$$

touch -t197001010000 $sentinel
while :
do
  files=$(find . -newer $sentinel -a '(' $fileSpec ')')
  if [ $? != 0 ]; then
    exit 1;
  fi
  if [ "$files" != "" ]; then
    $command
    touch $sentinel
  fi
  sleep 0.1
done

Python code for converting csv to html:

#!/usr/bin/env python
import csv
import sys

if len(sys.argv)<2:
   print "Usage:", sys.argv[0], "file1 <file2> ..."
   sys.exit(1)
print "<html><head></head><body><table>"
for f in sys.argv[1:]:
   for line in csv.reader(open(f,'r')):
      print "<tr>"
      for elem in line:
         print "<td>", str(elem), "</td>"
      print "</tr>"
print "</table></body></html>"

Build histogram with Gnuplot:
Starting from a 'stats.txt' file, containing frames per second measurements, this script will create 'bins' of size 5 (ie. it will accumulate all sample values from 0-5 into bin 1, 5-10 into bin 2, etc) and then plot the bin heights in a histogram.

#!/usr/bin/perl -w
# 
# "stats.txt" has frames per second, e.g. values from 1 to 78 fps
# This script will generate "stats.png" via gnuplot.
#
# For better fonts, before calling this, use:
#   export GDFONTPATH=/usr/share/fonts/truetype/msttcorefonts/
#   export GNUPLOT_DEFAULT_GDFONT=times
#                  
use strict;
my $total = 0;
my %bins;
open DATA, "stats.txt" or die "No stats\n";
open PLOTDATA, ">histogram.txt";
while(<DATA>) {
    # place sample values in bins of width 5
    $bins{2.5+5*int($_/5)}++; 
    $total++;
}
my @keys = sort { $a <=> $b } keys %bins;
# print the bin cardinalities
foreach my $k (@keys) { 
    print PLOTDATA $k." ";
    print PLOTDATA 100*$bins{$k}/$total;
    print PLOTDATA "\n";
}
# Plot the bins as histogram
open PIPE, "|gnuplot" || die "Can't initialize gnuplot ...\n";
print PIPE <<END
# set term postscript eps enhanced color
# set output "stats.eps"
# set style line 1 lt 1 lw 20
set term png
set output "stats.png"
set yrange [0:17]
set boxwidth 0.8 relative
set xlabel "Frames per second"
set ylabel "Percentage with this speed"
plot 'histogram.txt' using 1:2 with boxes fs solid 0.7  title 'Reported speeds, Oct 2010  '
END

Build animated .gif with Gnuplot:

Just use this:

    print PIPE "set term gif animate 300 optimize\n";
    print PIPE "set output \"graph.gif\"\n";

...and then plot as usual.

Increase number of loop devices (more than 8 squashfs):

Edit/create this file in Debian...

bash$ cat /etc/modprobe.d/local-loop
options loop max_loop=24

Squid simple authentication:
Edit squid.conf (in IPCOP, located under /etc/squid):

Add these lines on top:

  auth_param basic program /usr/lib/squid/ncsa_auth /etc/squid/passwds.proxy
  acl auth_users proxy_auth REQUIRED
  http_access allow auth_users

And maintain the passws.proxy via htpasswd...

SSH productivity tips:

bash$ cat ~/.ssh/config 
ControlMaster auto
ControlPath /tmp/ssh_mux_%h_%p_%r
ControlPersist 4h

More SSH tips are here.

CentOS RPM build from source:

bash$ cd /usr/src/redhat/SOURCES/
bash$ wget http://mirror.mcs.anl.gov/openssh/portable/openssh-5.9p1.tar.gz
bash$ tar zxvf openssh-5.9p1.tar.gz
bash$ cp openssh-5.9p1/contrib/redhat/openssh.spec /usr/src/redhat/SPECS/
bash$ rm -rf  openssh-5.9p1/
bash$ cd /usr/src/redhat/SPECS/
bash$ rpmbuild -bb openssh.spec
bash$ cd /usr/src/redhat/RPMS/i386
bash$ rpm -Uvh ...

Sed and Tr wizardry:

bash$ cat nice.sh
    tr -cs A-Za-z '\n' |
    tr A-Z a-z |
    sort |
    uniq -c |
    sort -rn |
    sed ${1}q

bash$ cat README | nice.sh 5
    3 the
    2 of
    2 a
    1 with
    1 up

Explanation:
- Make one-word lines by transliterating the complement (-c) 
  of the alphabet into newlines (note the quoted newline), 
  and squeezing out (-s) multiple newlines.
- Transliterate upper case to lower case.
- Sort to bring identical words together.
- Replace each run of duplicate words with a single 
  representative and include a count (-c).
- Sort in reverse (-r) numeric (-n) order.
- Pass through the stream editor; quit (q) after printing 
  the number of lines designated by the script's 
  first argument.

Perfect tmux config:

bash$ cat ~/.tmux.conf
set -g prefix C-a
unbind %
bind \ split-window -h
bind - split-window -v
bind-key k select-pane -U
bind-key j select-pane -D
bind-key h select-pane -L
bind-key l select-pane -R

bash$ tmux
Ctrl-a \  (vertical split)
Ctrl-a -  (horiz split)
Ctrl-a cursors (or h/j/k/l): move amongst windows
Ctrl-a d  (detach)
bash$ tmux list-sessions
...
bash$ tmux attach -t 0

HTML entities inside VIM:

$ cat /usr/local/bin/htmlEntities.pl
#!/usr/bin/perl -w
use HTML::Entities;
while(<>) {
        print encode_entities($_);
}
$ vi whatever.html
...
Then select using visual mode and...
    '<,'>!/usr/local/bin/htmlEntities.pl
or using classic ranges:
    (go to start and mark with marker 'a') ma
    (go to end   and mark with marker 'b') mb
    'a,'b!/usr/local/bin/htmlEntities.pl

Use VI mode in bash but keep ESC-.

Inside your .bashrc, add these:

    set -o vi
    # Read my comment in this SO post for explanation:
    #    https://stackoverflow.com/questions/6839273/
    bind -m vi-command ".":insert-last-argument
    # Restore Ctrl-L, Ctrl-A, Ctrl-E, Ctrl-W
    bind -m vi-insert "\C-l.":clear-screen
    bind -m vi-insert "\C-a.":beginning-of-line
    bind -m vi-insert "\C-e.":end-of-line
    bind -m vi-insert "\C-w.":backward-kill-word

My .Xresources:

$ cat .Xresources
*VT100*foreground: wheat
*VT100*background: black
*VT100*saveLines:  2500
*VT100*scrollBar:  true
*VT100*visualBell: true
*XTerm*font:       -b&h-lucida console-medium-r
    -semi condensed--0-0-0-0-m-0-iso8859-7
*XTerm*faceName:   -b&h-lucida console-medium-r
    -semi condensed--0-0-0-0-m-0-iso8859-7
*XTerm*faceSize:   12
*XTerm*renderFont: true
! To write in Greek, set this in your .bashrc
!    XTERM_LOCALE=el_GR.iso88597
! and uncomment this:
!*XTerm*locale:          true

$ grep .Xresources .bashrc
xrdb -merge $HOME/.Xresources

My bash PS1 setting:

$ grep PS1 .bashrc
export PS1='\n\[\e[32;1m\]\u@\h\[\e[37;1m\] \w\]\n\[\e[0m\]\$ '

Create a "drive" for QEMU out of a partition image:

bash$ dd if=/dev/zero of=/tmp/zero count=63
bash$ losetup -fv /tmp/zero
bash$ losetup -fv sda1
bash$ echo -e \
    "unit: sectors\n\n/dev/sda1 : start= 63, size= $( blockdev --getsz sda1 ), Id=7" | \
    sfdisk --force /dev/loop0
bash$ echo -e \
    "0 63 linear /dev/loop0 0\n63 $( blockdev --getsz sda1 ) linear /dev/loop1 0" | \
    dmsetup -v create virtdisk
bash$ fdisk -lu /dev/mapper/virtdisk

There; A new "drive" to pass to QEMU.

Password protect an Apache folder:

bash$ htpasswd -c -m /etc/mypass myaccount
(create password for user 'myaccount')
bash$ cd /var/www/wherever/you/want
bash$ cat > .htaccess
AuthUserFile /etc/mypass
AuthName "Please Log In"
AuthType Basic
require valid-user

And remember to use https, not just http (to avoid sniffers)

At bootup, spawn a screen session via cron:
Inside your crontab, use..

$ crontab -e
...
(Add this line)
@reboot /usr/bin/screen -d -m -S autoLoginScreen /home/ttsiod/bin.local/autoLogin.sh
...

(And this is what exists in the spawned script - automatically logging-in
 into an obscure crappy thingie that loses its wits all the time):

$ cat /home/ttsiod/bin.local/autoLogin.sh
#!/bin/bash
while true ; do
    date
    /home/ttsiod/bin.local/autoLogin.py
    sleep 60
done

This is better than spawning autoLogin.py from within cron, for various reasons: You can attach to the screen (screen -r ...) and see what's happening - and also, some scripts (e.g. using Python pexpect) may not run properly under cron, even after setting the env to the same as the normal one (most likely, missing tty-stuff).

Downgrade a hybrid Debian into "pure" stable or testing:

( echo -n "sudo apt-get install -y --allow-downgrade " ;
  apt-cache madison \
    $(apt-show-versions  | 
          grep -v jessie |
          grep 'newer than version' |
          nth 0 ) | grep ftp.de | awk '{printf("%s=%s\n", $1, $3);}' ) > doit.sh
sudo ./doit.sh

CLang,NERDTree,EasyMotion for VIM:
Install NERDtree and EasyMotion, by downloading from https://www.vim.org and unzip-ing inside your .vim/ folder. For CLang autocompletion it's the same process, with the added caveat that under Arch, I had to use the "libclang" based mode of operation (spawning didn't work for some reason).

After that, in my .vimrc:

let g:clang_library_path='/usr/lib/llvm'
(necessary for using libclang)
autocmd CursorMovedI * if pumvisible() == 0|pclose|endif
autocmd InsertLeave * if pumvisible() == 0|pclose|endif
(auto-closes preview window after you select what to auto-complete with)
nmap <silent> <c-n> :NERDTreeToggle<CR>
(maps NERDTree to Ctrl-N)
let g:NERDTreeDirArrows=0
(tells NERDTree to use standard characters)

Python administration with pip and virtualenv:
After installing easy_install from distro...

    easy_install pip
    pip install virtualenv
    pip install virtualenvwrapper
    mkdir $HOME/.virtualenvs

Edit .bashrc, and change python2 below to your Python...

   export WORKON_HOME=$HOME/.virtualenvs
   export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python2
   source /usr/bin/virtualenvwrapper.sh

Re-login. You can now...

    mkvirtualenv whatever
    workon 
	(shows list)
    workon whatever
	(prompt changes, you are in whatever now...)
    pip freeze 
	(shows list of packages, both system-wide and local)
    pip freeze -l
	(shows list of local packages)
    pip search packagename
    pip install packagename
    pip uninstall packagename
    deactivate
	(back to normal)

If you want a "fresh" environment, that doesn't use the globally installed versions of packages, then instead of...

    mkvirtualenv whatever

...do this:

    mkvirtualenv --no-site-packages whatever

Import sources in a local CVS repository:

mkdir /opt/cvsroot
export CVSROOT=/opt/cvsroot
cd /var/tmp
dtar mysources.tgz
cd NewProggy
cvs import ttsiodras/NewProggy NewProggy start
cd ..
rm -rf NewProggy
cvs checkout ttsiodras/NewProggy
cd ttsiodras/NewProggy 
vi ...
cvs commit ...

Git-svn generated .cvsignore files into .git/info/exclude :

From directory containing .git, execute...

find . -type f -name .cvsignore | \
while read LO ; do \
        PREF="`dirname $LO | cut -c3-`" ; \
        cat "$LO" | \
        while read WOW ; \
                do echo "$PREF/$WOW" ; \
        done ; \
done >> .git/info/exclude

Git status unescaped UTF8 output:
Place quotepath = false in .git/config in the [core] section, or just:
```
$ git config core.quotepath false
```

Create PNGs from a pdf presentation:

gs -dSAFER -dBATCH -dNOPAUSE -dTextAlphaBits=4 \
    -dGraphicsAlphaBits=4 \
    -r85 -q -sDEVICE=png16m -sOutputFile=icfp-pg%02d.png \
    PhDPresentation.pdf

Indent C or C++ code:

Use 
        indent -kr *c

for C, and

        astyle --style=ansi -l *h *cc

for C++.

Show default defines of GCC:
```
echo | gcc -dM -E -
```

Execute queries on MySQL and have html output viewed via w3m:

#!/bin/bash
echo What is the wikidb wikiuser passwd?
read PASSWD
while true 
do 
        echo SQL cmd to execute:  \(\"quit\" to abort\)
        read ANS 
        if [ "$ANS" == "quit" ] 
        then 
                break 
        fi 
        ( echo $ANS | mysql -u wikiuser wikidb -p$PASSWD \
	    -H -q | w3m -o confirm_qq=0 -T text/html ) 
done

Change MySQL tables to UTF8:

bash$ mysql -p
...
mysql> alter table some_table convert to character set utf8 collate utf8_unicode_ci;

Use octave to solve polynomials:

To use octave for solving

     4       3        2
    x  +  60x  - 9261x  - 54000x - 810000 = 0

I typed:

    a = [ 1 60 -9261 -54000 -810000]
    roots(a)

and I got:

ans =

    -128.9492 +   0.0000i
    75.0000 +   0.0000i
    -3.0254 +   8.6372i
    -3.0254 -   8.6372i

Use oprofile for detailed stats on runtime:

opcontrol --no-vmlinux
opcontrol --start
run_your_program
opcontrol --shutdown
opreport -l your_program_binary

Use a PAP-based modem account:

Edit /etc/ppp/pap-secrets and /etc/ppp/options
to fill-in the loginname/password info.

In /etc/ppp/pap-secrets add a line:

* * password

and in /etc/ppp/options, add a line:

lock
name loginname
usepeerdns

Access Internet from within QEMU:

Once you get connected with wvdial or whatever you use...
    echo 1 > /proc/sys/net/ipv4/ip_forward
    iptables -t nat -A POSTROUTING -o ppp0 -j MASQUERADE
and execute qemu...

The simplest of software RAID1, two partitions:

mdadm --create /dev/md0 --level=1 --raid-disks=2 \
	/dev/sda1 /dev/sda2
mdadm -A /dev/md0 /dev/sdb1 /dev/sdb2

Read a damaged CD/DVD valid parts and get the rest with rsync:

As is often the case, when I bring some burned CD/DVD 
from work, I find out that its bad at some offset.
I came up with this Perl script:

---------------------------------------
#!/usr/bin/perl -w
use strict;

my $i=0;

select((select(STDOUT), $| = 1)[0]);
unlink("data");

system("dd if=/dev/zero of=zero bs=2K count=1");

my $step = 1;

print "Sector:             ";
while(1) {
 system("dd if=/cdrom/BadSector of=sector bs=2K skip=$i".
    "count=1 >/dev/null 2>&1");
 if ($? == 0) {
     print sprintf("\b\b\b\b\b\b\b\b%08d", $i);
     system("cat sector >> data");
     $step = 1;
     $i += $step;
 } else {
     system("cat zero >> data");
     $step += $step;
     $i += $step;
     print "\nJumped over $step\nSector:             ";
 }
}
-----------------------------

With the CD/DVD mounted on /cdrom/, it will slowly but 
effectively copy sector by sector of the file mentioned 
in the 'dd' line into the file called 'data'. Reading 
sector-sector proves to be enough to correct a multitude 
of read-errors on my DVD reader, but even if that isn't 
enough, it will quickly jump over the problem areas
in the DVD, writing blank 'sectors' to mark the jumps.

After that, rsync can save the day:

rsync -vvv -B 131072 -e ssh \
	login@xxx.yyy.zzz.kkk:/path/todata data

Connect to the BSD console over a null cable:

slattach -p slip -s 19200 /dev/ttyS0 &
ifconfig  sl0 192.168......

Password-less logins through SSH:
To allow no-password logins to a target account, (say, root :‑), you must add the public key of the account you will login from, into the target account's .ssh/authorized_keys file. To create the public/secret key pair on the source account, use
```
    ssh-keygen -t dsa -b 1024
```
To avoid break-ins if someone steals your private key, use a passphrase when creating the key pair, and use ssh-agent to only type the passphrase once per boot.

Convert a static lib (.a) into a dynamic one (.so):

gcc -shared -o libXxf86vm.so.1.0 \
	-Wl,-soname,libXxf86vm.so.1 \
	-Wl,--whole-archive,libXxf86vm.a,--no-whole-archive

Link with a shared library and store the path (avoid LD_LIBRARY_PATH):

gcc -o exename -L/path/to/dynamiclib/ -lnameofLib \
    -Wl,-R/path/to/dynamiclib/ sourceCode1.c ...

(This assumes that /path/to/dynamiclib/ contains libnameofLib.so)

Undelete from a ReiserFS partition:

First, take a backup of all files in that partition, as the
undeletion process WILL mess up all existing files!

Then,

reiserfsck --rebuild-tree -S \
	-l /var/tmp/recovery.log /dev/hda2

where /dev/hda2 is of course your own partition...

Create live presentations from your X11:

=======================
To generate an XVID AVI
=======================

First, run the server:

    vncserver -geometry 640x480

Then connect to it, and produce the recording file:

    vncrec -record samples.vncrec

(e.g. when asked, connect to localhost:1)
Hit F8 to bring up the menu to Stop.
Then, check it with:

    vncrec -play samples.vncrec

and encode it (at 64kbps) with:

    transcode -i samples.vncrec -x vnc -z -y xvid \
    	-g 1024x768 -o mitsos.avi -w 64

View it with:

    mplayer -vo x11 -fs mitsos.avi 

=======================================
To create an SWF that works for the Web
=======================================

    vncserver -geometry 640x480
    vnc2swf video.swf localhost:1

F9 to start recording
F8 to bring up menu and quit

    cp ../track.mp3 .
    edit_vnc2swf.py -o final.swf -a track.mp3 video.swf

wget fails, needs referrer:

wget -nc -r -l inf -k -p -T 10 -o log \
    --referer=http://doom9.org/software2.htm \
    http://doom9.org/doc-overview.htm

Convert text between different encodings:

iconv -f WINDOWS-1253 -t ISO-8859-7 Subtitle.GR.srt

Programmable debugging with GDB:
While trying to figure out what was going wrong with decoding my mp3PRO files, I used GDB's command lists:

break init_plugins_from_dir
run -ao pcm:file=o.wav -demuxer xmms a.mp3
display/i $pc
set disassembly-flavor intel
cont
tbreak *(decode_file+0x360)
break *(decode_file+0x432)
commands 3
    silent
    echo Reached point where sample rate is final\n
    x /1x $my
    q
end
c
ni
set $my = (*($eax+4)+32)
set $my2 = (*($eax+4)+0x10f78)
echo Initial value of sample rate\n
x /1x $my
x /1x $my2
break *(decode_file+0x39f)
commands 4
    silent
    echo Before call at decode_file+0x39f\n
    x /1x $my
    x /1x $my2
    cont
end
break *(decode_file+0x3a4)
commands 5
    silent
    echo After call at decode_file+0x39f\n
    x /1x $my
    x /1x $my2
    echo \n
    cont
end
set $cnt = 0
break *mp3decDecode
commands 6
    silent
    set $cnt = $cnt + 1
    if $cnt == 3 
	echo Entered mp3decDecode\n
	set $breakDecodeFrame = 1
    end
    cont
end
set $breakDecodeFrame = 0
break *(DecodeFrame__14CMp3PRODecoderPviPibPUcT3+0x1a3)
commands 7
    if  $breakDecodeFrame == 1
	echo This is where the sample rate is changed from\n
	x /1x $my2
	set $my3 = *($my2)+0x98
	x /1x $my3
    else
	cont
    end
end
c

Create syntax-highlighted HTML from code:
```
webcpp file.c file.html
```
Turn the clock back for an application:
Use this code to create a shared library called time-warp.so:
```
gcc -D_GNU_SOURCE -shared -o time-warp.so time-warp.c -ldl
```
...and then, simply execute the desired binary with...
```
LD_PRELOAD=/path/to/time-warp.so /path/to/binary [args]
```
You can change how much back the clock will be set by changing OFFSET_IN_SECONDS in the code.

Dump hex and ASCII data:

bash$ echo Check the MBR data
bash$ dd if=/dev/sdc bs=512 count=1 | hexdump -C
00000000  eb 48 90 d0 bc 00 7c fb  50 07 50 1f fc be 1b 7c  |.H....|.P.P....||
00000010  bf 1b 06 50 57 b9 e5 01  f3 a4 cb bd be 07 b1 04  |...PW...........|
00000020  38 6e 00 7c 09 75 13 83  c5 10 e2 f4 cd 18 8b f5  |8n.|.u..........|
00000030  83 c6 10 49 74 19 38 2c  74 f6 a0 b5 07 b4 03 02  |...It.8,t.......|
00000040  ff 00 00 20 01 00 00 00  00 02 fa 90 90 f6 c2 80  |... ............|
...

Unbuffered pipes (driven to grep, etc):
Install the "expect" package, and use unbuffer:

bash$ unbuffer tail -f /var/log/apache/access.log | grep --line-buffered bozzo | ...

Python debugging:
If you find pdb a bit lacking, install ipdb and put something like this in your code:
```
    if specialConditionAtRuntime() or specialCmdLineArgPassed:
        import ipdb
        ipdb.set_trace()
```
You can then have the nice tab-completion and colored syntax output of IPython during your debugging.

Compile-time C assert:

#define assert_static(e) \
    do { \
        enum { assert_static__ = 1/(e) }; \
    } while (0)

...

    // This will fail at compile time in 64bit compilers
    long x;
    assert_static(sizeof(x) == 4);

Yield (coroutines) in C:
Excellent article by Simon Tatham.

Creating a MinGW import library from a .dll:
Method 1:

pexports RASDLG.DLL | sed 's/^_//' > rasdlg.def
dlltool --input-def rasdlg.def  --dllname RASDLG.DLL --output-lib librasdlg.a -k
cp librasdlg.a /Mingw/lib
ranlib /Mingw/lib/librasdlg.a

Method 2:

pexports /c/WINDOWS/system32/python25.dll >py25.def
dlltool -D python25.dll -d python25.def -l libpython25.a

Simple readline example:

#include <stdio.h>
#include <stdlib.h>
#include <readline/readline.h>
#include <readline/history.h>

int main(void) {
    while(1) {
        char *msg = readline("hello> ");
        if (msg) {
            printf("echo: '%s'\n", msg);
	    add_history(msg);
            free(msg);
        } else 
            exit(1);
    }
    return 0;
}

Simple FFT example:

#!/usr/bin/env python
import math
import scipy
import sys

# Frequency of sampling, in Hz (sampleRate)
if len(sys.argv)>1:
    N = int(sys.argv[1])
else:
    N = 16
# Index of last frequency (N/2 => 0.5*sampleRate)
Nlast = N/2
# Split 360 degress in N parts (don't end with 360, i.e. non-inclusive)
angle = [x*360./N for x in range(0, N)]
print "Angles:", angle
angleInRadians = [x*2.*math.pi/360. for x in angle]
# We are sampling at 16Hz, so we can cope with up to 8Hz
print "Broadcast frequencies of 1, 2, 3 and 8Hz"
data = [
    3.5*math.cos(1.*x)+
    1.0*math.cos(2.*x)+
    2.5*math.cos(3.*x)+
    0.5*math.cos(8.*x)
    for x in angleInRadians]
print "Performing FFT..."
# take samples 0 .. N/2 (inclusive) of the DFT
freq = scipy.fft(data)[0:Nlast+1]
# - DFTs go from N samples to (N/2+1) samples, with
#   notable zeroes in the imag[0] and the imag[N/2]
# - All coeffs must be divided by N/2, except
#   [0] and [N/2], which must be divided by N
# - DC component is stored at freq[0]
# - Maximum frequency (0.5 x sampleRate) is stored at freq[N/2]
freq /= N/2.
freq[0] /= 2.
freq[-1] /= 2.
print "Magnitudes:",
print ",".join("%4.2f"%x for x in abs(freq))


def near(f, g):
    return abs(f-g)<1e-6
print "Verifying coefficients..."
assert(near(freq[1], 3.5))
assert(near(freq[2], 1.0))
assert(near(freq[3], 2.5))
assert(near(freq[8], 0.5))
print "Coefficients OK."

Catch undefined behaviour with clang:

Just compile with:

-fcatch-undefined-behavior

Use ctags for /usr/include/ (skip boost):

$ su -
# cd /usr/include
# ctags -R --exclude=boost/\* .
# exit
$ cat >> .vimrc
set tags+=/usr/include/tags
(Ctrl-D)

Index

Updated: Sat Oct 8 12:33:59 2022