Linux command line

From ScientificComputing
Jump to: navigation, search

Bash is a powerful scripting language, which can directly been used in a Linux terminal and is often combined with Linux commands. It can simplify the many tasks when you are working on a HPC cluster. Therefore we recommend to study at least some of the basic Linux and bash commands, before you start using the clusters. Below we give examples for bash and the most widely used Linux commands.

Bash

  • specify a variable and print its value to the standard output
[sfux@eu-login-01 ~]$ VAR_A="some string variable"
[sfux@eu-login-01 ~]$ echo $VAR_A
some string variable
  • Print numbers 1-5 to standard output with a for loop
[sfux@eu-login-01 ~]$ for i in {1..5}; do echo $i; done
1
2
3
4
5
  • Print all numbers with two digits to standard output without using a for loop
[sfux@eu-login-01 ~]$ echo {0..9}{0..9}                                                                
00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33
34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67
68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
  • Print 5 random number to standard output and index them
[sfux@eu-login-01 ~]$ for i in {1..5}; do echo "$i-th random number $RANDOM"; done
1-th random number 25702
2-th random number 21605
3-th random number 6648
4-th random number 13259
5-th random number 8814
  • Do manipulations on a string (for instance exchange "home" by "work" in "/cluster/home/leonhard")
[sfux@eu-login-01 ~]$ STRING1="/cluster/home/leonhard"
[sfux@eu-login-01 ~]$ echo ${STRING1/home/work}
/cluster/work/leonhard


Linux commands

ls

  • Display list of files and folders sorted by time in reversed order (newest files at the bottom), including hidden files.
[sfux@eu-login-01 test]$ ls -ltra
total 1619
-rw-r--r--  1 sfux T0000     166 Jun 13 17:12 file2
-rw-r--r--  1 sfux T0000   26096 Jun 13 17:13 .file3
-rw-r--r--  1 sfux T0000   44642 Jun 13 17:14 file4
-rw-r--r--  1 sfux T0000 1488156 Jun 13 17:14 file1
-rw-r--r--  1 sfux T0000      73 Jun 13 17:15 file5
drwx------ 71 sfux T0000     117 Jun 13 17:15 ..
drwxr-xr-x  2 sfux T0000       7 Jun 13 17:15 .
  • Display list of files sorted by file size in reversed order (largest files at the bottom), including hidden files.
[sfux@eu-login-01 test]$ ls -lSra
total 1619
drwxr-xr-x  2 sfux T0000       7 Jun 13 17:15 .
-rw-r--r--  1 sfux T0000      73 Jun 13 17:15 file5
drwx------ 71 sfux T0000     117 Jun 13 17:15 ..
-rw-r--r--  1 sfux T0000     166 Jun 13 17:12 file2
-rw-r--r--  1 sfux T0000   26096 Jun 13 17:13 .file3
-rw-r--r--  1 sfux T0000   44642 Jun 13 17:14 file4
-rw-r--r--  1 sfux T0000 1488156 Jun 13 17:14 file1

cd

  • Change to your home directory.
cd ~
  • Go one level up in the directory hierarchy.
cd ..
  • Change to a directory with path: /path/to/directory.
cd /path/to/directory

pwd

  • Print working directory.
[sfux@eu-login-01 test]$ pwd
/cluster/home/leonhard/test

echo

  • Print a string to the standard output.
[sfux@eu-login-01 test]$ echo "this goes to the standard output"
this goes to the standard output

less, cat, more

  • Display the content of a file.
less file

or

cat file

or

more file

cp

  • Copy a file.
[sfux@eu-login-01 test]$ ls -l
total 1540
-rw-r--r-- 1 sfux T0000 1488156 Jun 13 17:14 file1
[sfux@eu-login-01 test]$ cp file1 file2
[sfux@eu-login-01 test]$ ls -l
total 1540
-rw-r--r-- 1 sfux T0000 1488156 Jun 13 17:14 file1
-rw-r--r-- 1 sfux T0000 1488156 Jun 13 17:26 file2
  • Copy a directory.
[sfux@eu-login-01 test]$ ls -l
total 1
drwxr-xr-x 2 sfux T0000 2 Jun 13 17:26 testdir1
[sfux@eu-login-01 test]$ cp -r testdir1 testdir2
[sfux@eu-login-01 test]$ ls -l
total 2
drwxr-xr-x 2 sfux T0000 2 Jun 13 17:26 testdir1
drwxr-xr-x 2 sfux T0000 2 Jun 13 17:27 testdir2
[sfux@eu-login-01 test]$

mv

  • Rename a file.
[sfux@eu-login-01 test]$ ls -l
total 1
-rw-r--r-- 1 sfux T0000 10 Jun 13 17:28 file1
[sfux@eu-login-01 test]$ mv file1 file2
[sfux@eu-login-01 test]$ ls -l
total 1
-rw-r--r-- 1 sfux T0000 10 Jun 13 17:28 file2

rm

  • Delete a file.
[sfux@eu-login-01 test]$ ls -l
total 1
-rw-r--r-- 1 sfux T0000 10 Jun 13 17:28 file2
[sfux@eu-login-01 test]$ rm file2
[sfux@eu-login-01 test]$ ls -l
total 0
  • Delete a directory (no matter if it contains files or not).
[sfux@eu-login-01 test]$ ls -l
total 1
drwxr-xr-x 2 sfux T0000 2 Jun 13 17:30 testdir1
[sfux@eu-login-01 test]$ rm -r testdir1
[sfux@eu-login-01 test]$ ls -l
total 0

mkdir

  • Create a directory.
[sfux@eu-login-01 test]$ ls -l
total 0
[sfux@eu-login-01 test]$ mkdir testdir1
[sfux@eu-login-01 test]$ ls -l
total 1
drwxr-xr-x 2 sfux T0000 2 Jun 13 17:31 testdir1

rmdir

  • Delete a directory (deletes only empty directories).
[sfux@eu-login-01 test]$ ls -l
total 1
drwxr-xr-x 2 sfux sdid 2 Jun 13 17:31 testdir1
[sfux@eu-login-01 test]$ rmdir testdir1
[sfux@eu-login-01 test]$ ls -l
total 0

man

  • Display the manual of a Linux command (output is too large to display it here, but just try it out).
man ls
man screen

exit

  • Terminate and exit the current shell.
[sfux@eu-login-01 ~]$ exit
logout
Connection to euler.ethz.ch closed.
sfux@calculus:~$

vi, nano

  • Try out the text editors by displaying a file.
vi file

or

nano file

grep

  • Search for a string within another string or a file.
[sfux@eu-login-01 test]$ ls -l
total 2
-rw-r--r-- 1 sfux T0000 0 Jun 13 17:44 file11
-rw-r--r-- 1 sfux T0000 0 Jun 13 17:44 file20
-rw-r--r-- 1 sfux T0000 0 Jun 13 17:43 testfile1
-rw-r--r-- 1 sfux T0000 0 Jun 13 17:43 testfile2
[sfux@eu-login-01-ng test]$ ls -l | grep test
-rw-r--r-- 1 sfux T0000 0 Jun 13 17:43 testfile1
-rw-r--r-- 1 sfux T0000 0 Jun 13 17:43 testfile2

sort

  • Sort a file line by line
[sfux@eu-login-01 test]$ more file1
c line c
b line b
d line d
a line a
[sfux@eu-login-01 test]$ sort file1
a line a
b line b
c line c
d line d

sed

  • Powerful tool for string manipulation (even whole files) which has many more functions. Exchange part of a string by another one (see also bash examples)
[sfux@eu-login-01 test]$ echo what a beautiful day | sed s/day/night/
what a beautiful night
[sfux@eu-login-01 test]$

awk

  • Awk is really a programming language. It can process text, perform arithmetic operations, etc. It is usually used in combination with another command to process/extract information. For example, to find out how many jobs are pending/running on the cluster:
[sfux@eu-login-01 ~]$ bqueues
QUEUE_NAME      PRIO STATUS          MAX JL/U JL/P JL/H NJOBS  PEND   RUN  SUSP 
clc              94  Open:Active     256    -    -    -     0     0     0     0
bigmem.4h        88  Open:Active       -    -    -    -   264   240    24     0
bigmem.24h       86  Open:Active       -    -    -    -  3031  2550   481     0
bigmem.120h      84  Open:Active       -  960    -    -   834   416   418     0
normal.4h        68  Open:Active       -    -    -    -  7520  5431  2089     0
normal.24h       66  Open:Active       -    -    -    - 72830 66192  6638     0
normal.120h      64  Open:Active       -    -    -    - 13666  5251  8415     0
normal.30d       62  Open:Active       -    -    -    -    58    58     0     0
virtual.40d      58  Open:Active       -    -    -    -     0     0     0     0
light.5d         10  Open:Active       -    -    -    -     1     0     1     0
purgatory         1  Open:Inact        -    -    -    -     4     4     0     0
[sfux@eu-login-01 ~]$ bqueues |  awk 'BEGIN{p=0;r=0} NR>1{p+=$9;r+=$10} END{printf "Pending:%8i\nRunning:%8i\n",p,r}'
Pending:   80142
Running:   18069

find

Search a string (substitute 'searchstring' by the string you are searching for) in all files contained in the current directory and all its subdirectories.

find . -type f -exec grep -q 'searchstring' {} \; -a -exec ls -l {} \;

du

  • Go to a directory and display the total size of it and then the sizes of all subdirectories.
[sfux@eu-login-01 mathematica]$ du -hs
8.6M    .
[sfux@eu-login-01 mathematica]$ du -h
17K     ./parallel
5.0K    ./batch_mode
24K     ./plot_graphics
1.3M    ./ptest
7.3M    ./ptest2
8.6M    .

tar

  • Create a tar archive including several files.
[sfux@eu-login-01 test]$ ls
PtCO.t21  t21.C  t21.O  t21.Pt
[sfux@eu-login-01 test]$ tar -cf test.tar *
[sfux@eu-login-01 test]$ ls -ltr
total 5903
-rw------- 1 sfux T0000  675840 May 23 15:27 t21.Pt
-rw------- 1 sfux T0000  442368 May 23 15:27 t21.O
-rw------- 1 sfux T0000  442368 May 23 15:27 t21.C
-rw------- 1 sfux T0000 1204224 May 23 15:27 PtCO.t21
-rw-r--r-- 1 sfux T0000 2775040 Jun 15 15:48 test.tar

Afterwards you can delete the files that have been collected in the tar archive.

gzip

  • Compress a file or an archive (tar and gzip are often used together) using gzip.
[sfux@eu-login-01 test]$ ls -ltr
total 2821
-rw-r--r-- 1 sfux T0000 2775040 Jun 15 15:48 test.tar
[sfux@eu-login-01 test]$ gzip test.tar
[sfux@eu-login-01 test]$ ls -ltr
total 1
-rw-r--r-- 1 sfux T0000 730947 Jun 15 15:48 test.tar.gz
[sfux@eu-login-01 test]$ gunzip test.tar.gz 
[sfux@eu-login-01 test]$ ls -ltr
total 1
-rw-r--r-- 1 sfux T0000 2775040 Jun 15 15:48 test.tar

bzip2

  • Compress a file or an archive using bzip2
[sfux@eu-login-01 test]$ ls -ltr
total 2820
-rw-r--r-- 1 sfux T0000 2775040 Jun 15 15:48 test.tar
[sfux@eu-login-01 test]$ bzip2 test.tar 
[sfux@eu-login-01 test]$ ls -ltr
total 1
-rw-r--r-- 1 sfux T0000 638158 Jun 15 15:48 test.tar.bz2
[sfux@eu-login-01 test]$ bunzip2 test.tar.bz2 
[sfux@eu-login-01 test]$ ls -ltr
total 1
-rw-r--r-- 1 sfux T0000 2775040 Jun 15 15:48 test.tar

top

  • Dynamic real-time view of of processes on a running system
[sfux@eu-c7-101-14 ~]$ top
top - 08:54:41 up 1 day,  1:46,  1 user,  load average: 0.00, 0.00, 0.00
Tasks: 1329 total,   1 running, 1328 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  66068024k total,  2008284k used, 64059740k free,    13256k buffers
Swap: 33554428k total,        0k used, 33554428k free,   264120k cached
 PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                     
 56697 sfux      20   0 28412 2416 1092 R  0.7  0.0   0:00.06 top                                          
   208 root      20   0     0    0    0 S  0.3  0.0   0:02.71 events/13                                    
  4668 consul    20   0 28460  18m 7160 S  0.3  0.0   7:00.89 consul                                       
  4803 root       0 -20 31220 3952 2144 S  0.3  0.0   3:49.09 lim                                          
     1 root      20   0 23504 1604 1284 S  0.0  0.0   0:02.52 init                                         
     2 root      20   0     0    0    0 S  0.0  0.0   0:00.01 kthreadd                                     
     3 root      RT   0     0    0    0 S  0.0  0.0   0:00.05 migration/0                                  
     4 root      20   0     0    0    0 S  0.0  0.0   0:01.09 ksoftirqd/0                                  
     5 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 stopper/0 
           

htop

  • Interactive process viewer on a running system. Very similar to top, but with a visual interface which allows more interaction when monitoring processes on a compute or login node.

id

  • Show groups that your use is a member of
[sfux@eu-login-02 ~]$ id | cut -d "=" -f 4 | tr "," "\n"
104222(sfux-group)
7840(T0003)
7843(T0070)
7854(T0025)
8614(T0002)
29441(ID-A-Users)
36457(T0000)
51200(ID-ALLE-ALLE)
51451(ServDe-otrs-users)
186833(ID-CD-PORTAL-Visitors)
186839(ID-CD-PORTAL-WINDOWS-Visitors)
186842(ID-CD-PORTAL-LINUX-Visitors)
186877(ID-CD-PORTAL-WINDOWS-PKG-Testers)
249231(id-polyhub-testgroup)
[sfux@eu-login-02 ~]$

External links