emjaytwentytwo

Revision going swimmingly. Fantastic news!

The first thing I did was set up different virtual environments for Python 2 and Python 3 by following the instructions set out on the worksheet. I have got the file to read in the data. But now I am stuck.

##update()

I DID IT. Here's what I've done.

#!/usr/bin/env python3

# MJ22 Lab 5, Exercise 2

# A program to smooth a series of numeric values

import sys

from tools import moving_average

try:

    input_filename = sys.argv[1]

    output_filename = sys.argv[2]

    width = int(sys.argv[3])

except:

    sys.exit('Usage: ./smooth.py <infile> <outfile> <width>')

data = [ ]

#Read data from input file into a list

input_file = open(input_filename, 'rt')

for line in input_file:

    if not line.startswith('#'):

        data.append(float(line))

# Compute a moving average

smoothed_data = moving_average(data, width)

# Write smoothed data to output file

with open(output_filename, 'wt') as output_file:

    for value in data:

         output = format( value, '.3f' )

#        print (output)

         output_file.write(output + '\n')

output_file.close()

Fantastic. Then I just opened up gnuplot and ran

plot 'elevation.data' with points, 'out.txt' with lines

elevation.data is the provided file, and out.txt is the output file from my program, and this is what we get.

WOW AMAZING. Thanks gnuplot.

The next bit

I'm not going to go into loads of detail here. I wrote 3 different programs, each a bit more complicated than the last. The first was to find the max and min longitude and latitude..:

import csv

longitude = []

lat = []

with open('gpsdata.csv') as input_file:

        input_file.readline()

        for record in csv.reader(input_file):

                longitude.append(float(record[2]))

                lat.append(float(record[1]))

minlong = min(longitude)

maxlong = max(longitude)

minlat = min(lat)

maxlat = max(lat)

print ('The minimum longitude is', minlong,'. The maximum longitude is', maxlong)

print ('The minimum latitude is', minlat,'. The maximum latitude is', maxlat)

Nice.

The next one was very similar, instead of finding the max and min, it calculated the mean (centroid)..:

import csv

longitude = []

lat = []

with open('gpsdata.csv') as input_file:

        input_file.readline()

        for record in csv.reader(input_file):

                longitude.append(float(record[2]))

                lat.append(float(record[1]))

centlong = sum(longitude)/len(longitude)

centlat = sum(lat)/len(lat)

print ('The centroid longitude is', centlong)

print ('The centroid latitude is', centlat)

The last one was a bit more complex. Instead of printing the answers, it had to write them to a file. I also incorporated a bit of code from 'smooth.py' to include command line arguments in the program:

import sys

import csv

try:

    input_filename = sys.argv[1]

    output_filename = sys.argv[2]

except:

    sys.exit('Usage: ./points <infile> <outfile>')

points = []

with open( input_filename, 'rt' ) as input_file:

        input_file.readline()

        with open( output_filename, 'wt' ) as output_file:

                for record in csv.reader(input_file):

                      print ( float(record[2]), float(record[1]), file = output_file )

output_file.close()

input_file.close()

I took the output file, and plotted it on gnuplot, here is the result:

Stuck at home with a coursework deadline immanent, I needed to print out my coursework and get it handed in to the CSO. I had a number of options; give a friend my login credentials, e-mail my work to a friend to print, e-mail my work to the lecturer...but I thought I'd try and print it at university myself, from home.

IT WAS EASY. I just logged into http://www.access.leeds.ac.uk/ , signed into PuTtY to access my SoC filestore and then used the command 

a2ps -P dec10lw coursework-one.txt

lab3 already, this is going fast. lets go.

Variables

Defining variables in the shell is easy, just like python. For example:

code=MJ22

title='Practical Problem Solving'

This means when you type 'echo $code', the shell returns 'MJ22' beneath, and the same for 'echo $title'. However, when you try and execute these commands in the bash script, blank values are returned. It seems this might be because the variables are defined only locally to the shell and not over to other programs. Once it had been re-defined as 'export title='Practical Problem Solving'', it then works in the bash shell.

Permission to run

To give a script permission to run from the './' command in shell, there are 2 steps. First, whack this '#!/bin/bash' at the first line of the script. Next, you need to change the privalages, by typing 'chmod u+x [FILENAME]'. Simples.

My 'hello' script

It's shaping up nice. ccccchhhhheck it:

#!/bin/bash # Hello World script clear read -p 'Enter your name: ' name today=`date +'%A %d %B'` echo Hello $name! echo Today is $today exit 0

BREAAAAKDOWN

Line 1 is so that it will run in bash.

Line 2 is a comment to identify the script

Line 3 clears the shell

Line 4 is an input to the shell. The -p argument allows you to specify a prompt string before the input is taken. the 'name' after it is a variable then set to the input by the user.

Line 5 specifies sorts out the date for us. The % nonsense is all arguments for the date program so that it returns the date in the format we want.

Line 6 has the 'echo' command. This is like 'print' in python. The $ before name makes it print the variable saved under name (defined in line 4), rather than simply printing 'name'.

Line 7 is very similar to line 6.

I can't say I really understand line 8. Just quits from the script, dunnit.

OK good got all that. Cracking on...

Selection with if, elif and else

Looks pretty straight forward, here's how to run a multi-branch if/else-if tiiing:

if test1; then      Commands to run if test1 is true elif test2; then      Commands to run if test2 is true else      Commands to run if neither test1 nor test2 are true fi

Cool.

ELSE CAN ONLY BE USED ONCE, AND ALWAYS COMES LAST.

Round two, lets go!

Creating Directories (-p)

I already know mkdir creates a directory. Cuh. Using mkdir -p enables you to create a string of folders inside one another in one go. Foooor example:

mkdir -p foo/bar/baz

Magic.

Changing Directories (-)

Obviously, we already know what cd does. When used with - (cd -), it returns you back to the previous working directory. Also note pwd, print working directory. Just incase you get lost out there.

Examining Directories

How could you use 'ls' to list the details of a directory’s files, in descending order of file size?

ls -S -l

The head tool can give the first N lines (or everything except the last N lines). tail is similar to head, but it works from the bottom rather than the top. Write in your lab notebook a pipeline that will list details of the three largest files in a directory.

ls -S -l | head -n 4 | tail -n 3

Disk usage

The du tool looks at disk usage in and below the current working directory. Using -s gives the size (in bytes) of the whole directory, and using it in conjunction with -h prints it in a human-readable format (K, M, G etc). --max-depth=1 will only look down into the first level of directories below the current working directory.

Find

find can be used to locate a file in a directory hierarchy. File name, file type, permissions, modification time and other things can all be used as search criteria. -maxdepth  and -mindepth can be used to limit the extent of the search. For example

find . -maxdepth 1 -name foo.txt -print

find . -maxdepth 2 -name foo.txt -print

find . -maxdepth 3 -name foo.txt -print

You can also use -d to find directories and wildcards(*) e.g. find . -name '*.jpg' -print to find all jpg's in the directory.

Manipulating files and directories

We already know cp, rm and mv (copy, remove and move respectively) to manipulate files, but what about directories? 

Just trying to copy a directory like a file won't work;. To make it work, you need to add the -r option. This makes the operation recursive. (Recursion is the process of repeating items in a self-similar way.) So it just copies everything. Basically. 

Similarly, using rmdir won't work unless the folder is empty. Instead, use the -r option with rm to enable the deletion of a directory and its contents.

rm -r foo2 

Compressing files

Files can be compressed to reduce the space they occupy on the disk. The standard tool for this in Linux is gzip. Use the command unzip to unzip a .zip archive. Use gzip to zip-em-up. Simples.

Write in your lab notebook a pipeline that will count the number of lines in a compressed text file.

 wc -l galaxy.txt.gz is the best I can come up with... I tried to write a pipeline that would unzip, do a line count, and then re-zip the file but it didn't seem to like it.

File

The file program is pretty useful, you can use it to tell you the file type of any file. Amazing.

Working with text

Sort

The sort function has lots of options. When used on it's own sort galaxy.txt then it orders the lines in the file in alphabetical order. The option -c checks a file to determine whether it has already been sorted. -r sorts the lines in reverse order, and -R puts them in a random order.

If you try and sort a list of numbers between 1 and 20, it lists them in a funny (yet logical) order

1

10

11

12

13

14

15

16

17

18

19

2

20

3

4

5

6

7

8

9

To fix this, use option -n.

Searching with grep

The grep tool finds lines of text which match the search query. So if you run grep just galaxt.txt , it will find and return all lines with the word 'just' in them. The -i option ignores the case of the search term, so a search for 'just' will actually search 'Just', 'jUst', etc etc etc. Using -c means count, which makes grep return a number of the times that the search term appears in the file it is searching. The -n option adds the line numbers at the beginning of the returned lines.

The command grep -i ^in galaxy.txt will search for all instances (disregarding case) of the word 'in' at the start of the line, in galaxy.txt. The ^ symbol searches for the start of the line and the $ symbol searches for the end.

This is pretty useful if you get stuck on any other bits. http://www.panix.com/~elflord/unix/grep.html#top

Extracting fields with cut

lab 1 is underway...most exciting. got a few questions to answer along the way...

BOX hg status will give you the current status of the files in the repository. you can find out more by looking at the log (hg log); hg summary also gives you some info about the current changeset. i can't figure out what to type to view files that have no outstanding changes or are being ignored...

BOX 'hg log -l 2' would give you the 2 most recent changesets. I can't find how to search the descriptions :(