The Linux Command Line & Shell Scripting

1 The Linux Command line & Shell Scripting1 Updated for 2016-04-13[web] [email] Resources : A Free Book2* Some of the materials covered in today s training is from this book500+ pagesStudy Resources : Tutorial Website3 Study Resources : Cheat Sheets4On the portalTraining -> Slides & HandoutsStudy Resources: Man pages5 Get a Command s help page:man < Command >Press q or Ctrl-C to exit the man pageYou can follow along using: Nucleus Web terminal on the BioHPC portal: or SSH from your from your Resources : Follow Shell and Command line Interfaces7 The interaction between user and the operating system is provide by a shellGraphic User Interfaces & Command line Interfaces By far the most common Shell on Linux is bash(Bourne Again Shell )It has a lot of built-in commandsuser namemachine namecurrent directoryname of the commandoptions/switchesargumentsLinux Basics: The File System8 Files on a Linux system are arranged in a hierarchical directory {uid}{department}{department}{uid}{PI_la b}{uidor project_name} Command line : Files and Directories9 Relative path Absolute path (full path) Relative path path related to the present working directory (pwd)If my current directory is /home2/ydu, relative path is: Trainingpwdwill return : /home2/yduIf change directory to home2, relative path becomes Trainingpwdwill return: /home2 Absolute path specify the location of a file or directory from the root directory (/)/home2/ydu/Training/home2yduTraining (I am here) Linux Command line : Files and Directories10 Editorsvi / vimCryptic commands!

2 Cheat sheet on the tutorial: extensible, customizable text editorQuick tutorial: to tutorial: text editor from your PC or MacMount your directories as network 1: Manage Files/FoldersExample 2: WildcardsExample 3: Disk UsageExample 4: Create Shared Folder/FilesExample 5: Text File Manipulation with Linux Tools(view/find patterns/sort/replace)as important as knowing how to use Linux itselfpopular and convenient in BioDataprocessing and Analysis Example 1: Manage Files/Folders12 CommandWhat does it do?pwdPrintWorking Directory where am I?lsList filesin current directoryls -lahList files with detail (l = long format,a= all files including hidden, h = human readable file sizes)cdTrainingChange directoryto today s training foldermkdirexample1 Make a directory called example1touchexample1 anempty new file inside example1 folderls lahexample1 List filesinside the example1 directorymkdirnew_exampleMake the directorynew_examplecpexample1 new_example/Copy example1 into new_examplecp r example1 newer_exampleCopy example1 recursively (directoryand everything inside) to a directory called newer_example)Example 1: Manage Files/Folders13 CommandWhat does it do?

3 Mvnew_exampleold_exampleMove/rename new_exampledirectory to old_examplermdirold_exampleTry to remove old_examplecd old_exampleChange directory intoold_examplels -alhList files with detail cd ..Change directory toparent folderrmdirold_exampleDeletean empty folderrm r newer_exampleRecursivelydelete all files and folders inside newer_exampleExample 2: Wildcards14* Match any number of charactersls *Any filels notes*Any file beginning with notesls *.txtAny file ending in .txtls *2015*Any file with 2015 somewhere in its name?Match a single characterls data_00?.txtMatches data_001, data002, data_00A etc.[]Match a set of characters (bracket expression)ls data_00[0123456789].txtls data_00[0-9].txtMatches data_001 data_009, not data_00 AExample 3: Disk Usage15quota: display disk usage and limits (How much space do I have)Example 3: Disk Usage16du: estimate file space usage (How much space am I taking up)Total folder size:Detailed usage for each sub-folders:Example 3: Disk Usage17mysandybox usage : of Files : 306 13 Archive files and folders will reduce both space usage and number of filesWe plan to limit the number of files for each user in the 3: Disk Usage18 CommandWhat does it do?

4 (x) everything from a tar archive file (f). Showverbose output (v).cd ..Change directory intoparent directory (CLI)du shexample3 Estimate space usage of folder example3 cd example3 Change directory intoparent directory file1 folder1 Create (c) a compressed archive file (f) adding files and foldersto a file using gzip becomes a .gzfilecd ..Change directory intoparent directory (CLI)du shexample3 Estimate space usage of folder example3 tar zxvfexample3 compressed gzippedtar archive in one stepPermissions19drwxr-xr-x 4 dtrudgian biohpc_admin58 Feb 16 15:13 all_trainingdrwxr-xr-x 7 dtrudgian biohpc_admin140 Feb 12 10:36 Appsdrwxr-xr-x 2 dtrudgian biohpc_admin26 Feb 16 15:12 cli_trainingdrwxr-xr-x 8 dtrudgian Feb 16 14:25 Cluster_Installsdrwxr-xr-x 3 dtrudgian Feb 16 11:49 Desktopdrwxr-xr-x 2 dtrudgian biohpc_admin10 Feb 16 14:10 Documentsdrwxr-xr-x 9 dtrudgian biohpc_admin135 Feb 16 14:32 Downloads-rw-r--r--1 dtrudgian biohpc_admin336 Feb 16 15:16 10 dtrudgian Feb 9 12:45 Gitdrwxr-xr-x 17 dtrudgian Feb 16 15:17 ownClouddrwxr-xr-x 2 dtrudgian biohpc_admin10 Feb 16 14:18 Picturesdrwxr-xr-x 5 dtrudgian biohpc_admin102 Feb 4 11.

5 19 portal_jobsOwnerGroupPermissionsls -lFile Permissions20 Octal Permissions21r = 4w=2x=1 Add up the permissions you need for each class, 5rw= 6rwx= 7-rw-r--r--1 dtrudgian biohpc_admin336 Feb 16 15:16 4 4 Owner can read+writeGroup can readOthers can readPermissions22chmodu/g/a+/-r/w/xfilen ameClass:u = user (owner)g = groupa = all+Add permission-Remove permissionrreadwwritexexecutechmodg+ read/write permissions for the groupchmoda+ execute permission for everyonechmodg-x execute permission for the groupExample 4: Create Shared Folder23 CommandWhat does it do?cd /project/biohpcadmin/shared/Change directory to yourlab s shared folderls -alList files with detail to check permissionmkdirshared_dataCreate sharedfoldercdshared_dataChange directoryto newly created shared_datafoldercp/project/biohpcadmin/ to shared folder, you group members can read and copy itchmodg+ permission to allow group members edit the cd.

6 Change directory to parent folderchmodg+wshared_dataGrant write permission to allow group members create new files inside shared_datafolder You can only apply chmodcommand to the files/folders owned by you. Send email to you need help to set up a shared folder Example 5: Text File Manipulation24 CommandWhat does it do? the contentof in the thecontent of in a pager that allows scrolling & searchinghead n 15 the first 15 lines of n 5 the last 5 lines of n 15 | tail n 5 Shows lines 11-15 of We take the first 15 linesfrom using head, and extract the bottom of that selection by piping it through "elephant" all lines in that contain elephant i"mouse" lines in that contain cat non-case c "at" thenumber of lines in that contain at .grep ^The lines beginningwith In grep water\.$ lines endingwith water . Note that . is a special character so we have to searches for patterns using regular expressions - 5: Text File Manipulation25 CommandWhat does it do?

7 Sort the linesin a file alphabeticallysort -r the lines in a filein reverse alphabetical ordersort nr lines in a file in reverse numerical ordersort > the output of the sort commandinto overwriting existing >> the output of the sort Command into , appendingto the end of the > the error output of the sort Command into > /dev/nullDiscard the error output of the Command which nonsense sort >&1 > the error output into the standardoutput, and direct into the file "s/dog/cat/g" all instances of cat to dog, printing the results(substitute)/old/new/g(lobal)sedc an do a lot - do even more - Shell Scripting26In addition to the interactive mode, bash Shell script is a programing language in itselfrun an entire script of commandsone Command at a timeA little knowledge can make difficult things easy, and time-consuming things quick. Bash Shell Scripting : #!

8 /bin/bashMY_NAME=daveecho Hello $MY_NAMEA variable holds information to be used laterSet them using:name=valueGet their value using:$nameSet the variable MY_NAMERun echo using the value in MY_NAMES pecify we are using bash shellBash Shell Scripting : How to run28chmod+x to run a bash script Method 1: convert scripts into an executable file Method 2: use bash or shBash Shell Scripting : #!/bin/bashNOW=$(date +%Y-%m-%d)DATA_DIR="data_$NOW"mkdir$DATA _DIRmodule load matlabmatlab nodisplay-nosplash< > $DATA_ can assign the output of programs/commands into variables:Create a directory with a name incorporating the datePut the date (YY-MM-DD) into NOWRun matlabwith input , and send the output into our data directoryBash Shell Scripting : If-else Statement #!/bin/bashecho "This scripts checks for the dummy file.

9 "echo " " if [ -f "dummy" ]; thenecho "dummy exists."elseecho "Could not find it! " fifile exists test is -fRead about the available tests hereBash Shell Scripting : Loops about the other loops here#!/bin/bashfor iin $( ls ); doecho "item: $i"doneSet variable ito each value in the output of the ls commandEcho the current value of $ script lists all of the files in the current directory using a for loop:Bash Shell Scripting : Pass Command line Arguments #!/bin/bashecho "Script Name: $0"echo "Total Number of Argument s: $#"echo "1st Argument: $1"echo "2nd Argument: $2"echo "All Arguments are: $*"$* Store all Command line arguments$# Store count of Command line arguments$0 Store name of script itself$1 Store first Command line argument$2 Store second Command line argument$3 Store third Command line Shell Scripting : Reorganize data to fit the software generated by Total Internal Refraction Fluorescence MicroscopeInput data structure required by TEST_INPUT1_101_111_12 Bash Shell Scripting : Reorganize data to fit the software requirement34#!

10 /bin/bashINPUT_FOLDER=$1 # pass input folder name as 1stargument from terminalOUTPUT_FOLDER=$2 # pass input folder name as 2ndargument from terminali=1 # initialize iterator as 1for SUBFOLDER in $(ls $INPUT_FOLDER);do # loop through all foldersecho $SUBFOLDER # display current sub folder nameIMG488_ORG=$INPUT_FOLDER/$ # define 488 image nameIMG561_ORG=$INPUT_FOLDER/$ # define 561 image nameIMG_OUTPUT=${OUTPUT_FOLDER}/$i# display current sub folder namelet "i=$i+1" # increment iteratormkdir$IMG_OUTPUT # create output folder for imagesif [ -f $IMG488_ORG ].