Sentiment Analysis of Tweets with AYLIEN Text Analysis API
At AYLIEN, we do our best to make sure our users get up and running and calling our API in the shortest time possible. As part of a new initiative, we are going to be sharing use case ideas, source code and fully functional apps to help you get the most out of our API. For this edition of the blog, we are going to focus on a pretty common use case that a lot of users want to use our API for, analyzing Tweets.
There is a wealth of insight that can be extracted from Tweets. You can read more on Analyzing Tweets and Social data in our previous blog; Why is Sentiment Analysis important from a business perspective.
Today, weâre going to provide you with the source code for a functioning app that mines Twitter for keywords, extracts Tweets and analyzes the text in the Tweets. As part of the process, weâll run two analysis endpoints on each Tweet, Sentiment Analysis on all of the Tweets and Hashtag Suggestion on Tweets that contain a URL.
What you'll need to get going:
Twitter API access: Get your api_key, consumer_secret key and access token to make calls to the Twitter API here.
Node.js running on your machine: If you donât already have Node.js on your machine you can download it here.
Twit: A Twitter API client library for Node.js. You can download it here.
A text editor: You can use any editor, We recommend Sublime Text and you can download it here.
AYLIEN Text Analysis API access: Get your Application ID and Application Key here. See our âgetting startedâ blog for details on how to sign up.
Overview of the code in action
To give an overview of what can be achieved, we will first look at the code in action. The complete code snippet is given at the end of this blog for you to copy and paste.
Step1. Setup your Environment
Ensure Node.js is running on your machine, download the twit client library from GitHub, get access to the Twitter API and finally, open an AYLIEN Text Analysis API account.
Open your text editor and copy and paste the code snippet (provided at the bottom of this blog) and save the file as, tweetsentiment.js. Next, open command prompt and Navigate to the folder where you saved the code snippet.
The application takes two command line parameters which you chose; a keyword for the Twitter query and the number of Tweets the query should return.
Run the code by typing âtweetsentiment websummit 3â. In this case we are querying the keyword âwebsummitâ and asking for 3 Tweets to be returned.
Once the Tweets are returned by the Twitter API they are fed to AYLIEN Text Analysis API, where the polarity will be determined and where the optimal Hashtags for URLâs will be generated.
Note: Ensure you replace the YOUR_APP_ID and YOUR_APP_KEY placeholders in the code with the application id and application key which you received when you signed up for the AYLIEN API. You will also need to fill in your specific Twitter API credentials that you received from Twitter. All going well you should see an output on the command line similar to that shown below.
Tweet Text : RT @IndoBusiness: The #WebSummit is drawing to a close. #Bono up soon. Watch live here: http://t.co/onpdYvoIy4 Or follow our blog here: htt. Sentiment Polarity : neutral Polarity Confidence : 0.9702560119839743 Hashtags : [ '#RyderCup', '#PeterThiel', '#Davos', '#AdrianGrenier', '#PaulMcGinley', '#FoundersFund' ]
Tweet Text : RT @FierceClever: I swear to god, if I hear the word "di srupt" one more time... #websummit Sentiment Polarity : negative Polarity Confidence : 0.8947368421052632 Hashtags : No Hastags available as no Url specified in the Tweet
Tweet Text : Having a great day at the #websummit . Were at stand ECM 243 in the village if anyone would like to pop over before closing! Sentiment Polarity : positive Polarity Confidence : 0.9230769230769231 Hashtags : No Hastags available as no Url specified in the Tweet
Taking the first result as an example, you can see that the Tweet itself is displayed followed by the âsentiment polarityâ of the Tweet (positive, neutral or negative) and the âpolarity confidenceâ i.e. the confidence that the sentiment returned was correct (a number between 0 and 1). Finally, if the Tweet contained a URL embedded in the Tweet a list of optimal hashtags is generated for that webpage/article.
Itâs worth looking at the two parts of the solution that do most of the heavy lifting :
Querying Twitter is very straight forward using the twit client and requires just one line of code:
T.get('search/tweets', { q: process.argv[2], count: process.argv[3] }, function(err, data, response) { data.statuses.forEach(function(s) { ...
The above line of code uses the supplied command line arguments to query Twitter, it then passes the returned results one by one to the function that will feed the body of the Tweet and the embedded URL (if any) to the AYLIEN API endpoints for analysis.
2. Analyzing the Tweets that are returned.
The function below, takes the following arguments, the AYLIEN endpoint to call (Sentiment, Hashtags, Entities etc.) the parameters which the endpoint should work on (i.e. we indicate whether we are passing a piece of text or a URL for analysis and we also pass the actual text or URL) and a callback function to call when the analysis is complete.
function call_api(endpoint, parameters, callback) { var postData = querystring.stringify(parameters); var request = https.request({ host: 'api.aylien.com', path: '/api/v1/' + endpoint, headers: { 'Accept': 'application/json', 'Content-Type': 'application/x-www-form-urlencoded', 'Content-Length': postData.length, 'X-AYLIEN-TextAPI-Application-ID': APPLICATION_ID, 'X-AYLIEN-TextAPI-Application-Key': APPLICATION_KEY, } }, function(response) { var data = ""; response.on('data', function(chunk) { data += chunk; }); response.on('end', function() { callback(JSON.parse(data)); }); }); request.write(postData); request.end(); }
The examples we have used, analyze the Tweets for Sentiment and Hashtag Suggestions. Itâs up to you what endpoints you wish to use. Maybe you want to extract entities or concepts from the Tweets as well. A full list of our endpoints can be found in our documentation.
var Twit = require('./node_modules/twit') //Twitter API client library var https = require('https'), querystring = require('querystring'); //AYLIEN API Credentials const APPLICATION_KEY = YOUR_APPLICATION_KEY, APPLICATION_ID = YOUR_APPLICATION_ID; //Twitter API Credentials var T = new Twit({ consumer_key: YOUR_TWITTER_CONSUMER_KEY , consumer_secret: YOUR_TWITTER_CONSUMER_SECRET , access_token: YOUR_TWITTER_ACCESS_TOKEN , access_token_secret: YOUR_TWITTER_ACCESS_TOKEN_SECRET }) var analysisResults = {}; var parameters; var i = process.argv[3] * 2; //Counter to track when Asynchronous API call have completed console.log("Processing your request. Please wait...") console.log("\n"); T.get('search/tweets', { q: process.argv[2], count: process.argv[3] }, function(err, data, response) { data.statuses.forEach(function(s) { var returnedUrls = s.entities.urls; analysisResults[s.id] = {}; analysisResults[s.id].text = s.text; parameters = {'text': s.text}; callAylienAPIs(parameters,outputResults); function callAylienAPIs(parameters, callback) { call_api('sentiment', parameters, function(result) { var a = {}; a.endpoint = 'sentiment'; a.polarity = result.polarity; a.polarity_confidence = result.polarity_confidence; analysisResults[s.id].sentiment = a; i--; if (i == 0) { callback(); } }) if (returnedUrls.length > 0 ) { var url_paramaters = {'url' : returnedUrls[0].expanded_url }; call_api('hashtags', url_paramaters, function(result) { var a = {}; a.endpoint = 'hashtags'; a.hashtags = result.hashtags; analysisResults[s.id].hashtags = a; i--; if (i == 0) { callback(); } }) } else { var a = {}; a.endpoint = 'hashtags'; a.hashtags = 'No Hastags available as no Url specified in the Tweet'; analysisResults[s.id].hashtags = a; i--; if (i == 0) { callback(); } } } }); }) function outputResults() { for (var key in analysisResults) { console.log("Tweet Text : ", analysisResults[key].text); console.log("Sentiment Polarity : ", analysisResults[key].sentiment.polarity); console.log("Polarity Confidence : ", analysisResults[key].sentiment.polarity_confidence); console.log("Hashtags : ", analysisResults[key].hashtags.hashtags); console.log("\n"); } } function call_api(endpoint, parameters, callback) { var postData = querystring.stringify(parameters); var request = https.request({ host: 'api.aylien.com', path: '/api/v1/' + endpoint, headers: { 'Accept': 'application/json', 'Content-Type': 'application/x-www-form-urlencoded', 'Content-Length': postData.length, 'X-AYLIEN-TextAPI-Application-ID': APPLICATION_ID, 'X-AYLIEN-TextAPI-Application-Key': APPLICATION_KEY, } }, function(response) { var data = ""; response.on('data', function(chunk) { data += chunk; }); response.on('end', function() { callback(JSON.parse(data)); }); }); request.write(postData); request.end(); }
