Linux command line
From ScientificComputing
Bash is a powerful scripting language, which can directly been used in a Linux terminal and is often combined with Linux commands. It can simplify the many tasks when you are working on a HPC cluster. Therefore we recommend to study at least some of the basic Linux and bash commands, before you start using the clusters. Below we give examples for bash and the most widely used Linux commands.
Contents
Bash
- specify a variable and print its value to the standard output
[sfux@eu-login-01 ~]$ VAR_A="some string variable" [sfux@eu-login-01 ~]$ echo $VAR_A some string variable
- Print numbers 1-5 to standard output with a for loop
[sfux@eu-login-01 ~]$ for i in {1..5}; do echo $i; done
1
2
3
4
5
- Print all numbers with two digits to standard output without using a for loop
[sfux@eu-login-01 ~]$ echo {0..9}{0..9}
00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33
34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67
68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
- Print 5 random number to standard output and index them
[sfux@eu-login-01 ~]$ for i in {1..5}; do echo "$i-th random number $RANDOM"; done
1-th random number 25702
2-th random number 21605
3-th random number 6648
4-th random number 13259
5-th random number 8814
- Do manipulations on a string (for instance exchange "home" by "work" in "/cluster/home/leonhard")
[sfux@eu-login-01 ~]$ STRING1="/cluster/home/leonhard"
[sfux@eu-login-01 ~]$ echo ${STRING1/home/work}
/cluster/work/leonhard
Linux commands
ls
- Display list of files and folders sorted by time in reversed order (newest files at the bottom), including hidden files.
[sfux@eu-login-01 test]$ ls -ltra total 1619 -rw-r--r-- 1 sfux T0000 166 Jun 13 17:12 file2 -rw-r--r-- 1 sfux T0000 26096 Jun 13 17:13 .file3 -rw-r--r-- 1 sfux T0000 44642 Jun 13 17:14 file4 -rw-r--r-- 1 sfux T0000 1488156 Jun 13 17:14 file1 -rw-r--r-- 1 sfux T0000 73 Jun 13 17:15 file5 drwx------ 71 sfux T0000 117 Jun 13 17:15 .. drwxr-xr-x 2 sfux T0000 7 Jun 13 17:15 .
- Display list of files sorted by file size in reversed order (largest files at the bottom), including hidden files.
[sfux@eu-login-01 test]$ ls -lSra total 1619 drwxr-xr-x 2 sfux T0000 7 Jun 13 17:15 . -rw-r--r-- 1 sfux T0000 73 Jun 13 17:15 file5 drwx------ 71 sfux T0000 117 Jun 13 17:15 .. -rw-r--r-- 1 sfux T0000 166 Jun 13 17:12 file2 -rw-r--r-- 1 sfux T0000 26096 Jun 13 17:13 .file3 -rw-r--r-- 1 sfux T0000 44642 Jun 13 17:14 file4 -rw-r--r-- 1 sfux T0000 1488156 Jun 13 17:14 file1
cd
- Change to your home directory.
cd ~
- Go one level up in the directory hierarchy.
cd ..
- Change to a directory with path: /path/to/directory.
cd /path/to/directory
pwd
- Print working directory.
[sfux@eu-login-01 test]$ pwd /cluster/home/leonhard/test
echo
- Print a string to the standard output.
[sfux@eu-login-01 test]$ echo "this goes to the standard output" this goes to the standard output
less, cat, more
- Display the content of a file.
less file
or
cat file
or
more file
cp
- Copy a file.
[sfux@eu-login-01 test]$ ls -l total 1540 -rw-r--r-- 1 sfux T0000 1488156 Jun 13 17:14 file1 [sfux@eu-login-01 test]$ cp file1 file2 [sfux@eu-login-01 test]$ ls -l total 1540 -rw-r--r-- 1 sfux T0000 1488156 Jun 13 17:14 file1 -rw-r--r-- 1 sfux T0000 1488156 Jun 13 17:26 file2
- Copy a directory.
[sfux@eu-login-01 test]$ ls -l total 1 drwxr-xr-x 2 sfux T0000 2 Jun 13 17:26 testdir1 [sfux@eu-login-01 test]$ cp -r testdir1 testdir2 [sfux@eu-login-01 test]$ ls -l total 2 drwxr-xr-x 2 sfux T0000 2 Jun 13 17:26 testdir1 drwxr-xr-x 2 sfux T0000 2 Jun 13 17:27 testdir2 [sfux@eu-login-01 test]$
mv
- Rename a file.
[sfux@eu-login-01 test]$ ls -l total 1 -rw-r--r-- 1 sfux T0000 10 Jun 13 17:28 file1 [sfux@eu-login-01 test]$ mv file1 file2 [sfux@eu-login-01 test]$ ls -l total 1 -rw-r--r-- 1 sfux T0000 10 Jun 13 17:28 file2
rm
- Delete a file.
[sfux@eu-login-01 test]$ ls -l total 1 -rw-r--r-- 1 sfux T0000 10 Jun 13 17:28 file2 [sfux@eu-login-01 test]$ rm file2 [sfux@eu-login-01 test]$ ls -l total 0
- Delete a directory (no matter if it contains files or not).
[sfux@eu-login-01 test]$ ls -l total 1 drwxr-xr-x 2 sfux T0000 2 Jun 13 17:30 testdir1 [sfux@eu-login-01 test]$ rm -r testdir1 [sfux@eu-login-01 test]$ ls -l total 0
mkdir
- Create a directory.
[sfux@eu-login-01 test]$ ls -l total 0 [sfux@eu-login-01 test]$ mkdir testdir1 [sfux@eu-login-01 test]$ ls -l total 1 drwxr-xr-x 2 sfux T0000 2 Jun 13 17:31 testdir1
rmdir
- Delete a directory (deletes only empty directories).
[sfux@eu-login-01 test]$ ls -l total 1 drwxr-xr-x 2 sfux sdid 2 Jun 13 17:31 testdir1 [sfux@eu-login-01 test]$ rmdir testdir1 [sfux@eu-login-01 test]$ ls -l total 0
man
- Display the manual of a Linux command (output is too large to display it here, but just try it out).
man ls man screen
exit
- Terminate and exit the current shell.
[sfux@eu-login-01 ~]$ exit logout Connection to euler.ethz.ch closed. sfux@calculus:~$
vi, nano
- Try out the text editors by displaying a file.
vi file
or
nano file
grep
- Search for a string within another string or a file.
[sfux@eu-login-01 test]$ ls -l total 2 -rw-r--r-- 1 sfux T0000 0 Jun 13 17:44 file11 -rw-r--r-- 1 sfux T0000 0 Jun 13 17:44 file20 -rw-r--r-- 1 sfux T0000 0 Jun 13 17:43 testfile1 -rw-r--r-- 1 sfux T0000 0 Jun 13 17:43 testfile2 [sfux@eu-login-01-ng test]$ ls -l | grep test -rw-r--r-- 1 sfux T0000 0 Jun 13 17:43 testfile1 -rw-r--r-- 1 sfux T0000 0 Jun 13 17:43 testfile2
sort
- Sort a file line by line
[sfux@eu-login-01 test]$ more file1 c line c b line b d line d a line a [sfux@eu-login-01 test]$ sort file1 a line a b line b c line c d line d
sed
- Powerful tool for string manipulation (even whole files) which has many more functions. Exchange part of a string by another one (see also bash examples)
[sfux@eu-login-01 test]$ echo what a beautiful day | sed s/day/night/ what a beautiful night [sfux@eu-login-01 test]$
awk
- Awk is really a programming language. It can process text, perform arithmetic operations, etc. It is usually used in combination with another command to process/extract information. For example, to find out how many jobs are pending/running on the cluster:
[sfux@eu-login-01 ~]$ bqueues QUEUE_NAME PRIO STATUS MAX JL/U JL/P JL/H NJOBS PEND RUN SUSP clc 94 Open:Active 256 - - - 0 0 0 0 bigmem.4h 88 Open:Active - - - - 264 240 24 0 bigmem.24h 86 Open:Active - - - - 3031 2550 481 0 bigmem.120h 84 Open:Active - 960 - - 834 416 418 0 normal.4h 68 Open:Active - - - - 7520 5431 2089 0 normal.24h 66 Open:Active - - - - 72830 66192 6638 0 normal.120h 64 Open:Active - - - - 13666 5251 8415 0 normal.30d 62 Open:Active - - - - 58 58 0 0 virtual.40d 58 Open:Active - - - - 0 0 0 0 light.5d 10 Open:Active - - - - 1 0 1 0 purgatory 1 Open:Inact - - - - 4 4 0 0
[sfux@eu-login-01 ~]$ bqueues | awk 'BEGIN{p=0;r=0} NR>1{p+=$9;r+=$10} END{printf "Pending:%8i\nRunning:%8i\n",p,r}'
Pending: 80142
Running: 18069
find
Search a string (substitute 'searchstring' by the string you are searching for) in all files contained in the current directory and all its subdirectories.
find . -type f -exec grep -q 'searchstring' {} \; -a -exec ls -l {} \;
du
- Go to a directory and display the total size of it and then the sizes of all subdirectories.
[sfux@eu-login-01 mathematica]$ du -hs 8.6M . [sfux@eu-login-01 mathematica]$ du -h 17K ./parallel 5.0K ./batch_mode 24K ./plot_graphics 1.3M ./ptest 7.3M ./ptest2 8.6M .
tar
- Create a tar archive including several files.
[sfux@eu-login-01 test]$ ls PtCO.t21 t21.C t21.O t21.Pt [sfux@eu-login-01 test]$ tar -cf test.tar * [sfux@eu-login-01 test]$ ls -ltr total 5903 -rw------- 1 sfux T0000 675840 May 23 15:27 t21.Pt -rw------- 1 sfux T0000 442368 May 23 15:27 t21.O -rw------- 1 sfux T0000 442368 May 23 15:27 t21.C -rw------- 1 sfux T0000 1204224 May 23 15:27 PtCO.t21 -rw-r--r-- 1 sfux T0000 2775040 Jun 15 15:48 test.tar
Afterwards you can delete the files that have been collected in the tar archive.
gzip
- Compress a file or an archive (tar and gzip are often used together) using gzip.
[sfux@eu-login-01 test]$ ls -ltr total 2821 -rw-r--r-- 1 sfux T0000 2775040 Jun 15 15:48 test.tar [sfux@eu-login-01 test]$ gzip test.tar [sfux@eu-login-01 test]$ ls -ltr total 1 -rw-r--r-- 1 sfux T0000 730947 Jun 15 15:48 test.tar.gz [sfux@eu-login-01 test]$ gunzip test.tar.gz [sfux@eu-login-01 test]$ ls -ltr total 1 -rw-r--r-- 1 sfux T0000 2775040 Jun 15 15:48 test.tar
bzip2
- Compress a file or an archive using bzip2
[sfux@eu-login-01 test]$ ls -ltr total 2820 -rw-r--r-- 1 sfux T0000 2775040 Jun 15 15:48 test.tar [sfux@eu-login-01 test]$ bzip2 test.tar [sfux@eu-login-01 test]$ ls -ltr total 1 -rw-r--r-- 1 sfux T0000 638158 Jun 15 15:48 test.tar.bz2 [sfux@eu-login-01 test]$ bunzip2 test.tar.bz2 [sfux@eu-login-01 test]$ ls -ltr total 1 -rw-r--r-- 1 sfux T0000 2775040 Jun 15 15:48 test.tar
top
- Dynamic real-time view of of processes on a running system
[sfux@eu-c7-101-14 ~]$ top
top - 08:54:41 up 1 day, 1:46, 1 user, load average: 0.00, 0.00, 0.00
Tasks: 1329 total, 1 running, 1328 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 66068024k total, 2008284k used, 64059740k free, 13256k buffers
Swap: 33554428k total, 0k used, 33554428k free, 264120k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
56697 sfux 20 0 28412 2416 1092 R 0.7 0.0 0:00.06 top
208 root 20 0 0 0 0 S 0.3 0.0 0:02.71 events/13
4668 consul 20 0 28460 18m 7160 S 0.3 0.0 7:00.89 consul
4803 root 0 -20 31220 3952 2144 S 0.3 0.0 3:49.09 lim
1 root 20 0 23504 1604 1284 S 0.0 0.0 0:02.52 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.01 kthreadd
3 root RT 0 0 0 0 S 0.0 0.0 0:00.05 migration/0
4 root 20 0 0 0 0 S 0.0 0.0 0:01.09 ksoftirqd/0
5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 stopper/0
htop
- Interactive process viewer on a running system. Very similar to top, but with a visual interface which allows more interaction when monitoring processes on a compute or login node.
id
- Show groups that your use is a member of
[sfux@eu-login-02 ~]$ id | cut -d "=" -f 4 | tr "," "\n" 104222(sfux-group) 7840(T0003) 7843(T0070) 7854(T0025) 8614(T0002) 29441(ID-A-Users) 36457(T0000) 51200(ID-ALLE-ALLE) 51451(ServDe-otrs-users) 186833(ID-CD-PORTAL-Visitors) 186839(ID-CD-PORTAL-WINDOWS-Visitors) 186842(ID-CD-PORTAL-LINUX-Visitors) 186877(ID-CD-PORTAL-WINDOWS-PKG-Testers) 249231(id-polyhub-testgroup) [sfux@eu-login-02 ~]$
