D.

In random order:

Install Bootstrap

Create website for project on techiemouse.com domain  

Read image data from folder

Install mongo

Create mongo  

Save image data in mongo

Calculate statistics, analyse data

Show data on page using graphs

Learn/use ejs

Read exif data on client side  -> spike and prototype with APIs found

I definitely have a clear idea of what I want. And unlike the other agile projects there is no team there is no product manager, there is no scrum, there is no customer. So how will it work? Do I apply the rules or go with the flow? Or do I go towards a more organised approach?! Waterfall?!

One: Set-up the gitHub repository, and here it is: ExifDataAnalysis 

Two:  Clone the repository and start installing the needed technologies. Most of the package managers and languages I needed were already installed previously. I used: homebrew and nvm to install npm and node.

Three: Creating the dependencies file, also known as package.json. This was initiated using the basic: npm init.

This is where I encountered a first issue: npm init wasn’t recognised as a command in the project folder. To get this fixed:

I checked node version, which was the correct one: 0.10.30

To unblock npm (that was actually installed) I run the command: nvm install 0.10.30 which responded with: v0.10.30 is already installed.

This actually corrected the error and I could run the command and create the package.json. (a weird little bug)

Four: Installing the node framework. I chose express as it’s what we’re using on our work system. But I did have a cheeky look at: Restify, Hapi;

At work we’re allowed and encouraged to start-up Friday projects. About a month ago I moved teams starting from scratch with new people new project, new system, new programming languages and technologies. All the Android knowledge gathered and used in my previous team was no longer needed anymore. This has been a scary and exciting experience.

Now I’m starting to learn more about servers, web and databases and the most amazing thing is: it’s reminds me tremendously of my initial dissertation goal (the one I didn’t manage to get to): big data analysis and user friendly graphs and tables (Can’t say more than that I’m afraid).

Let’s go back to the actual subject of this blog post: Friday projects. I’m starting one! To learn some of the new technologies I’m working with: Node.js, Mongo.db, servers.

My idea is: Creating a web application using a Node.js server that uses node-exif to get exif image data, save it to a MongoDB database, analysing it and returning it in a user-friendly interface, providing details about similarities between the images found.

The project has already been approved so it’s up to me to put al the pieces together: - Setting up Node on server - Setting up website with image upload. - Setting up MongoDB and saving data from images as objects - Displaying analysed data on website.

It’s been a year since I finished uni and since my dissertation has been presented. Life caught up with me and I’ve moved to a new city, got my first job and basically finally started adulthood. 

I want with this blog post to round up my dissertation progress and what I have learned by quoting bits and pieces from my evaluation.

As a computer science student I believe my final project rounded up my whole university experience and knowledge:

Logical thinking/problem solving

starting with pseudo-code that turned into real code and then into refactored code;

working with MySQL within Java and continuously trying to not repeat statements and methods that differ only by one table name or variable;

Working with external libraries, new software and understanding new APIs

JDBC, MySQL, XML

Natural Language Processing, Stanford Libraries

Machine Learning, Weka

Presenting my work

in technical terms - talking to my colleagues and teachers helped me consolidate my communication skills and consolidate my knowledge (through repetition)

in layman terms - how to explain things to people that might not understand programming rules and tasks made me see the project from outside, it’s uses and how it can be improved and adapted to a more general user.

Organisation

source control (and as I keep finding out even now GIT is an extraordinary tool)

Working on my own

the project was my own from start to finish, my plans, my ideas, my code;

there was no other students who to confer with or that could help;

the entire experience proved to me that when I have my next idea I don’t have to be afraid to approach it on my own.

Where everything can be found: My gitHub

I spent these last days reading article after article. It's been over 1000 for sure but ended up with 652 records in my database from which unique articles: 589.

Why the difference? Mostly due to the multiple domain classification. There were many articles that could have been classed as theft and fraud or as murder and assault.

What was tricky? The attempted murders, the articles about crime statistics, the ones with so many strange characters that you wouldn't recognize the words or understand the sentence. 

What I noticed?

- Conspiracy was added as a new class! Also property damage and misconduct (less found than initially thought)

- It get's more and more difficult to label when you find only murders and assaults and not enough fraud or theft;

- Quite a lot of duplicate articles with different IDs especially on conspiracy(I guess and attempt on the Queens life is interesting for all issues)

- I started to look at the articles words: with theft, assault and murder I kept noticing verbs: stealing, robbed, cut, beating, stabbed; with conspiracy was more about: dynamite, bomb, plot(nouns);  with fraud: false, false pretences, falsified, forgery, forged(a mix)

It's time for machine learning.. not much time left..

I have 15911 articles in my database! Words that I queried for were: crime, investigation, arrested and police.

Started the next step: Gathering my training data and test data by reading and labelling 500 articles with the help of my little Helper. This is a new java project that get's a random number between 1 and 15911 shows me the article, abstract, id, text and title and lets me input numbers from 1 to 5 to label the article. Everything then goes into the junction table that connects the Article table with the Domain table. Making things work faster. Already done 100 inputs.

  Now to talk a bit about the findings and the issues.

1. Might need a conspiracy class

"PLOT TO SEIZE THE ARSENAL AT RANGOON. KANGOOX, Thursday. A plot fcr seizing the Areenal and setting fire to the town has been dis- covered by the authorities. Many Burmese have betn arrested in consequence."

2. People assault animals :( .. I guess it goes under assault

"THE GREAT WYRLEY OUTRAGES. The cd tor of a Wolverhampton paper has received another anonymous letter, signed on behalf of the members of the Wyrley Gang, in which the writer asserts that young Edalji is as guilty of the crime of horse maiming, of which he was convicted, as any other member of the gang. The writer adds that the police arc watching the letterboxes, and the members of the gang are watching the police. He wai ns Inspector Campbell that his time is short, as he is \\"a r< gue and liar,\\" and says that the gang is determined to kill him."

3. Arrested because of misconduct and being drunk? ummm do I make a category for that?

"HER 81st APPEARANCE. Florence Jones made her 81st appearance at Cardiff Police-courr, this morning, wnen she wa1 cnarged with being drunk in Louiinunaqoare, on Thursday.Police-constable Wheeler (129) gave evidence in support of thu charge, and prisoner was sent to gaol for fourteen days, the magistrates on the bench being Dr. H. Jr Plline and Alderman Lewis."

4. Not sure how to label this: 

"CARMARTHEN. CRIME IN CARMARTHENSHIRE.The Welshman, in commenting upon the recent murder in Carmarthenshire, makes some startling assertions which, if trueand we cannot believe them without very strong evidence go far to prove that that part of Wales resembles Ireland, at least in one particularthe protection afforded to crime. According to our contemporary, the failure \\"istice in the case in question may most ibly be traced to the fact that witnes ses J their lives will not be safe if they spe e truth. The country is in a state of teri every man, woman, and child seems afraid to speak on the fearful topic;\\" and one man told a police officer that \\"one life has gone, and it was a pity to waste another.\\" Cannot the Welshman push its investigations a little further, and let us know why the murderer is protected? Are the people so much opposed to capital punishment that they would rather commit another murder than allow justice to be carried out, or is there some other reason ? And how comes it that in Wales generally crime is so rare, and in Carmarthenshire murder can be committed with impunity ?"

5. Treason, Bombing attempt, Property damage

"A BARCELONA MYSTERY. Barcelona, Sunday.At 7 o`clock this evening the police here were informed that a suspicious looking objeet had beon foond on tbe Ramble. On investigation the object proved to be an iron cylinder filled with liquid, It was taken to the police station and thance to the Law Courts, where it was placed on the`ground. Immediately a terrific explosion occurred, injuring an official  and doing much damage. Renter,"

6.  Crime against religion?

"A DABWIWIAN PaOTBSSOB INDICTED.  An Austrian Ultramontane paper reports that Professor Dr. Brnehl, the wellknown lecturer on aootomical science in Vienna University, has been indicted for a crime against Church and religion. The Minister of Public Works, Dr. Gautsck von Frankenthurn, asked the rector of the university to call upon Dr. Bruehl to explain some of his recsnt upon the Darwinian theories, in which he i» an ardent i( believer. Dr. Bruehl Is 70 years of age and highly respected."

7. Not enough fraud found and a lot of no crime labelled.

8. What to do with the articles that contain trials and time in court?

  If it's this difficult for me ..how would it be for the computer?

I'll be a SQL ninja by the end of this project. Fought with SQL statements within Java all this time but it's getting there. I like the feeling of accomplishment when after a few struggles I feel like I actually know how the code actually works in the end.

Had a basic understanding of SQL when I started this project and was very comfortable with Java so imagine my surprise when I got stuck in this area. But then again JDBC seems like a different type of programming. It felt clumsy and uncomfortable:  all those Statements and all those executeQueries and result sets and while loops and if statements. I feel like my code is a bit of a mess now (the Database class) and it needs a great deal of refactoring.

But I have done it in the end I have my database now(over 11000 articles and counting stored locally), my tables, even my little Labelling helper that takes a random article and offers me the option to add one or two or three .. of the domain_ids.

Machine learning is the ability of a computer to learn from patterns and improve prediction and behaviour through data .

Developing a machine learning model means having a good amount of labelled training and testing data. But the most important part is choosing the best attributes for the ML algorithm to find patterns that can differentiate between classes. For the project I have chosen to use nouns and verbs and their occurrence in each article as attributes. This mean using one of the Part Of Speech taggers libraries available online.

After a bit of research I found two in Java that are better suited for what I'm looking for: OpenNLP Apache and Stanford NLP.

Choosing one of them wasn't that easy as their results weren't too different but I ended up selecting Stanford due to the options available (multiple features under one umbrella) and its accuracy around 97% for the POS tagger.

The way the library was built was using a machine learning  model that tags the words in a sentence using morphology analysis techniques and an English vocabulary. The result is a string similar to:

The_DT above_JJ sketch_NN shows_VBZ the_DT position_NN of_IN the_DT boy_NN

  Because I am tagging both verbs and nouns I am using an if statement and an option variable.

The method will request two variables then: the text that will be analysed and the result wanted: verb or noun: taggIT(theText, "verb").

To end up with the actual required part of speech I have used just a few simple replace functions

.replace("_VBN", "").replace("_VBZ", "")

.replace("_NN ", " ").replace("_NNS", "")

and then split the sentence into words and only select the ones that don't contain an underscore "_"

for (String word : sentence.split(" ")) {

                     if (!word.contains("_")) {

                           attributeString.append(" " + word);

                     }

The method will return a String that will contain the verbs (or nouns) separated by spaces.

I have chosen to use a String as the result of this method due to the next stage of my attribute processing which is lemmatization. To find a words lemma is to find its general base form. This is particularly useful when using verbs as they can have different forms and tenses that can increase the margin of ML error when having such a variety of attributes in the training set that in the end mean the same thing.

e.g.: be, is, been, was, were / escape escaped escaping

The lemmatizing method uses the Stanford coreNLP library, it splits the string into sentences and tokens(words) and reduces them to the base form using the LemmaAnnotation.class

Not many weeks left, the plan is (as discussed in last meeting with supervisor):

 1. Get all training data in database:

Make list of words the initial search will be done on

Don't add duplicate articles just because they have multiple words from the above list

Test if article is split on two pages and connect the article text

Extract verbs and remove duplicates

2. Database design and set-up

Choose database and design structure

Add all articles to database

Label the training data manually (>500 articles)

3. Machine learning stage 1: Developing the model

Research and play with Weka  (look at Apache Mahout, Java-ML )

Develop the model that classifies the articles.

4. Machine learning stage 2: Testing of Model

Getting new articles to test the model with

Tweaking the model

5. Machine learning stage 3: Applying the model on the rest of the data.

Verify article is not part of training model

Get final result into final database.

All these should be accompanied by constant documentation writing and regular blog posts. (Even if the coding is really the fun/exciting part for me! ) 

After being gone for a few days I felt a bit bad going to the meeting with nothing really important to show.  

Here is what I have until now:

XML from the National Library API is being parsed using DOM implementation in Java.

Getting verbs from sentences works: combining Stanford POS tagger Library and a few .replace and .split and .contains methods.

Object is crated with the necessary data from the Articles and an array of these type objects.

Part of background document ( 540 words ).

Set-up Git and my data/repository is on Dropbox (nice, easy and safe);

The second meeting I got to know the other students that will be working with Amanda and we all discussed our projects. It was interesting to find out that I'm not the only one nervous and with a lot of work ahead. We all are doing research on different domains: natural language processing, machine learning, interactive web programming, data mining, image processing some even need to look outside their domain (biology, insects, DNA, brain connections).

So on Thursday some of the pressure seem to lift after hearing their ideas, their worries, the parts they are afraid of and how much work we all need to do. I'm not alone!

Friday I got some news that brought back the pressure but not in a bad way: the chair in Digital Collections from The National Library of Wales wants to meet with me and talk about how I'll be using their collection. I was excited as I know if I do this right it would be an amazing achievement for me. I must do this right.

Other news:

 I can finally access the API from my own computer. They were nice enough to give me my own username and password.

I understand how the URLs work and I can easily query what I need, it's pretty straight forward.

Started coding JAVA methods to extract the data as a String and now looking into how to parse it to an XML object. This will help me with tokenising, tagging and counting and basically teaching me to using the data in any way I want. For now I hard coded the username and password using  the Authenticator Class.

The plan for the week is:

Coming to university each week day

Read all journals found on NLP

Understand Apache OpenNLP

Play with XML

Think about best way to prepare and hold the articles for data processing

I must admit everything seems new to me at this point and having only 14 weeks to go through it all doesn't seem like enough time.

I admire my friends that have already their graphic interfaces and their buttons and screens that link to each other on a phone .. and I can't wait to actually have something to show for myself. 

There is this reasonably sized list of journals and articles I found and need to go through:

Ricardo Baeza-Yates. Challenges in the interaction of information retrieval and natural language processing. In Computational Linguistics and Intelligent Text Processing, pages 445–456. 2004.

Satanjeev Banerjee and Ted Pedersen. An adapted lesk algorithm for word sense disambiguation using WordNet computational linguistics and intelligent text processing. In Alexander Gelbukh, editor, Computational Linguistics and Intelligent Text Processing, volume 2276 of Lecture Notes in Computer Science, chapter 11, pages 117–171. Springer Berlin / Heidelberg, Berlin, Heidelberg, February 2002.

G. Corso, A. Gulli, and F. Romani. Ranking a stream of news, 2005.

Monika Henzinger, Bay W. Chang, Brian Milch, and Sergey Brin. Query-free news search. In Proceedings of the 12th international conference on World Wide Web, WWW ’03, pages 1–10, New York, NY, USA, 2003. ACM.

G. Holmes, A. Donkin, and I. H. Witten. WEKA: a machine learning workbench. In Intelligent Information Systems,1994. Proceedings of the 1994 Second Australian and New Zealand Conference on, pages 357–361, August 2002.

Karen S. Jones. A statistical interpretation of term specificity and its application in retrieval. In Journal of Documentation, volume 28, pages 11–21, 1972.

Karen S. Jones. Towards better NLP system evaluation. In HLT ’94: Proceedings of the workshop on Human Language Technology, pages 102–107, Morristown, NJ, USA, 1994. Association for Computational Linguistics.

Jure Leskovec, Lars Backstrom, and Jon Kleinberg. Meme-tracking and the dynamics of the news cycle. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, KDD ’09, pages 497–506, New York, NY, USA, 2009. ACM.

David D. Lewis and Karen S. Jones. Natural language processing for information retrieval. Communications of the ACM, 39(1):92–101, 1996.

Zhiwei Li, Bin Wang, Mingjing Li, and Wei Y. Ma. A probabilistic model for retrospective news event detection. In Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, SIGIR ’05, pages 106–113, New York, NY, USA, 2005. ACM.

Levon Lloyd, Dimitrios Kechagias, and Steven Skiena. Lydia: A system for Large-Scale news analysis. In Mariano Consens and Gonzalo Navarro, editors, String Processing and Information Retrieval, volume 3772 of Lecture Notes in Computer Science, chapter 18, pages 161–166. Springer Berlin / Heidelberg, Berlin, Heidelberg, 2005.

Dragomir Radev, Jahna Otterbacher, Adam Winkel, and Sasha B. Goldensohn. NewsInEssence: summarizing online news topics. Commun. ACM, 48(10):95–98, October 2005.

Yusuke Shinyama and Satoshi Sekine. Named entity discovery using comparable news articles. In Proceedings of the 20th international conference on Computational Linguistics, COLING ’04, Stroudsburg, PA, USA, 2004. Association for Computational Linguistics.

K. Sparck. Some points in a time. Computational Linguistics, 31(1):1–14, March 2005.

Ian H. Witten and Eibe Frank. Data Mining: Practical Machine Learning Tools and Techniques with Java Implementations (The Morgan Kaufmann Series in Data Management Systems). Morgan Kaufmann, 1 edition, October 1999.

Ron Zacharski. A programmer’s guide to data mining. http://guidetodatamining.com/, 2012.

My supervisor, Amanda welcomed me with an understanding smile .. "It's normal to be nervous, it's the first week!" and just like that my nervousness went out the window.

In the first part of the meeting I wasn't sure which way to go with the amount of data I have. Little by little we discussed all my ideas:

Sentiment Analysis ?! - maybe interesting and easy with blogs and tweets but a bit tough on formal newspapers.

Interactive Wales Map with all the local news split on domains and mood? - is this really a project for one semester? - should think methodically, in steps that can be ended at any point and still offer a finite product.

I got a first introduction into machine learning: articles and domains and attributes to connect them both together. Maybe think of: specific word frequency (football, tennis - they would appear in sport articles), pair of words, maybe the length of the article or its position in the newspaper (first page? third page?) Is there a pattern?

Are only crime articles easier to analyse rather than classifying every article? There could be  different types:  theft, murder, attempted suicide, property damage, bodily harm. Does the crime rate spiked in specific years? Did it depend on age or sex?

Why can't I access the API from my computer? :( It seems like at the moment only computer science machines are allowed to access the API. Still need to try accessing it through SSH. The good thing is that the data on it is a fraction of the one they have on their servers just so we can play with it without disturbing the real user.

Homework:

- Concordance and co-occurrence

- Mutual Information (transinformation)

- Karen Sparck Jones - her study on words used in similar context and her contribution reagrding inverse document frequency

- Part-of-speech tagging

- Machine Learning - Weka (Data Mining - Mark Hall, Ian Witten), Orange, SCIKIT ML

- Google places vs. Open Street map - gazetteer (list of places)

A productive meeting and I must admit I was more relaxed than expected (I don't usually feel at ease about discussing my worries and asking questions face to face) .

"The project will be an analysis of the data collection provided from the historical newspapers found at the National Library. The newspapers are digitised and have a basic API that will be used to search, request and extract specific data. The point of the project is to discover interesting themes and concepts within the articles and using data mining processes and statistics to present them in an interesting way to the user."

This is how it starts - my final year project at Aberystwyth University. Doesn't seem very clear right? It wasn't for me - I haven't done any machine learning or statistics during my time at this university => Terrifying! But also exciting, I mean it's not like I haven't been through this a couple of times before. I guess I always knew my previous interest in data gathering and analysis during my marketing degree won't fade. I just need to mix this up with a programming language and some web development. Easy?!?