#fablre

Create a fashion corpus. Fashion corpus would be roughly created using Wikipedia articles, RSS feeds from fashion magazines which will require scraping in some form, etc. The whole superset would have to be cleaned up in a big way, possibly using NLTK python libraries (Natual Language Processing) to remove stopwords, repetitions, and more. 

List of sources:

1. Wikipedia - explore metadata to pinpoint/ download fashion specific content. Apparently Wiki has structured downloadable data available at https://en.wikipedia.org/wiki/Wikipedia:Database_download but the question to answer is how the 13 gigs of data would be sorted and the relevant data associated with fashion would be downloaded. It’s a somewhat stupid approach to download all 13 gigs and then use stopwords to find fashion content. Wondering on the approach. Shoutout to #Wikipedia though for an amazingly rich content they are making freely available. 

Interestingly, Wikipedia itself provides an API wrapper in python for querying Wiki database. However, that didn’t seem useful. Here is what we are at: using Special Export downloads articles but not inside subcategories and using PetScan gives a list of all articles inside categories and its subcategories depending on “depth” input. The workaround is to get a list of all articles inside a category re: Fashion using PetScan, then download all articles using Special Export, following which we would be working with Gensim probably to do a document analysis and discard articles which are in terms of tolerance level remote to fashion.   

So, summarily after some trials and lot of errors, I have come to the conclusion that the best way to get the content from Wiki is to use the Wikipedia API calls which doesn’t take markups into consideration and renders the output in a plain and simple readable English format. The only issue remains is the number of API calls and rate limits associated with it. Since there is no strict limit mentioned by Wikipedia folks, I am using time.sleep with a uniform distribution of seconds of code sleep. For now it seems there is no other choice especially given that wikipedia api wrapper doesn’t come with good documentation touching on the API calls rate limitation, as opposed to Tweepy documentations for Twitter API call. For now however the API calls are working fine and the expectation is to download all articles associated with Fashion (used Index of fashion related articles here rather than the Categories as described earlier).

Snippet of code: (mostly brute and might be a genuine programmer’s nightmare)

import wikipedia import os import time from requests import exceptions from random import uniform input_file = open('article_list.txt', 'r') output_file = open('article_content.txt', 'a') for line in input_file:    # time.sleep(uniform(1, 3))    try:        article = wikipedia.page(line)        print article.title        output_file.write(article.title.encode('utf-8'))        output_file.write('\n')        time.sleep(uniform(2, 4))        output_file.write(article.content.encode('utf-8'))        output_file.write('\n')    except:        exceptions.ConnectionError        try:            time.sleep(uniform(3, 5))            article = wikipedia.page(line)            print article.title            output_file.write(article.title.encode('utf-8'))            output_file.write('\n')            time.sleep(uniform(1, 2))            output_file.write(article.content.encode('utf-8'))            output_file.write('\n')        except:            exceptions.ConnectionError            time.sleep(uniform(4, 6))            article = wikipedia.page(line)            print article.title            output_file.write(article.title.encode('utf-8'))            output_file.write('\n')            time.sleep(uniform(2, 3))            output_file.write(article.content.encode('utf-8'))            output_file.write('\n')            continue        continue    # output_file.write(article.title)    # output_file.write(wikipedia.summary(line,sentences=1)    # output_file.write(article.content) output_file.close() print line os.system('say "your program has finished"')

And yes, everything on the community edition of PyCharm 

Error catching: 1. HTTP Connection errors - owing to rate limitations on API

Error catching: 2. DisambiguationError - owing to multiple meanings of one article title

Have been bypassing both of these for now. And till now we have a collection of 700 articles and ongoing. 

Update on 4th Dec: Corpus of around 4k documents created out of wikipedia. Data pre-processing using nltk toolkit and bit of regex. 

LDA - gensim package (ldamodel.py tweaked)

Source Code

import glob import nltk import chardet from collections import defaultdict import logging from lda_model import LdaModel import gensim import pyLDAvis.gensim import warnings warnings.filterwarnings('ignore') # basic logging comments logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s',

level=logging.INFO) # setting folder for file retrieval folder = '/Users/abhinavprakash/PycharmProjects/Data/new_texts/' # detecting encoding def encoding_detection(rawtext):    return chardet.detect(rawtext).values()[1] # POS tagging and removal of other parts of speech. Making sm def pos_tagging(tokens):    return [item[0].lower() for item in nltk.pos_tag(tokens)

if item[1] in ('FW', 'NN', 'NNS', 'NNP', 'NNPS')] raw_list = [] for filename in glob.glob(folder+'*'):    coding = encoding_detection(open(filename).read())    raw_list.append(nltk.word_tokenize(open(filename).read()

.decode(coding))) # raw_list = [nltk.word_tokenize(open(filename).read()

.decode('utf-8')) for filename in glob.glob(folder+'*')] print raw_list[0:10] raw_list_POS = [pos_tagging(row) for row in raw_list] print raw_list_POS[0:10] total_length = len(raw_list_POS) frequency = defaultdict(float) for texts in raw_list_POS:    for text in set(texts):        frequency[text] += 1 print frequency tokens = [[token for token in text if 1.0 < frequency[token]

< 0.4*total_length] for text in raw_list_POS] tokens = [[token for token in text if token not in           ('=', '==', '===', '%', ']', '[', '-', 'c. ',

'>', '<', 'ii', 'iii' '====', 'women', 'designer',

'men', 'century','woman', 'man', 'centuries', 'mm',

'product', 'products', 'i', 'g.', "i'll", "you're",

"it's", "we've",'"i', 'ms.', 'mr.', 'm43', 'id',

'c/o', 'd.getelementbyid', 'window.__stp', "'https",

'//widgets.rewardstyle.com/js/shopthepost.js',            'window.__stp.init', 'd.location', "'object",

'd.createelement', '/^http', 'shopthepost-script',            'javascript', "that's", "don't", '|', 'x', 'k', '=====',

"i'm", "i've", "'shopthepost-script")]           for text in tokens] print 'Tokens below:' print tokens[0:10] # creating and saving BoW representation dictionary = gensim.corpora.Dictionary(tokens) dictionary.save('token_dict_BoW.dict') print 'Dictionary saved' corpus = [dictionary.doc2bow(text) for text in tokens] gensim.corpora.MmCorpus.serialize('token_BoW.mm', corpus) print 'Corpus saved' # loading BoW representation id2word = gensim.corpora.Dictionary.load('token_dict_BoW.dict') mm = gensim.corpora.MmCorpus('token_BoW.mm') # running lda lda1 = LdaModel(corpus=mm, iterations=5000,

id2word=id2word, num_topics=50, update_every=1, chunksize=500, passes=14) print 'LDA run' # lda.print_topics(20) data1 = pyLDAvis.gensim.prepare(lda1, corpus, dictionary) pyLDAvis.show(data1)

Results (first cut, needs optimization - don’t kill me)