#machine generated data

Dofunny analytics is entire about transforming inconsequent and meaningless data into valuable insights. For analytical practices and approaches to reach their full and optimum genius, business analytics tools and platforms are unforgoable to make it easy for customers to compass action based on their new insights. The ingroup must analyze, feed and monitor the final results of actions taken back into the system in an automated and robotlike way to improve and inform the prognostic models that led to the insights at the actually first place. One can in all conscience consider this process as of capping the big data deductive loop. Considering an example - A chamber of commerce consumer at a excellent vendor uses jivatma business intelligence tools to decide whether in order to offer or not a held out promotional campaign. In between user analyzes that the three hour online sale will likely result in significant revenue generation based on the intelligence yielded by the underlying analytics engine. The user has to implement the campaign, keep a gym of its performance and makes necessary adjustments to future ones as sworn and guaranteed. <\p>

In existing environments, the foremost take steps needs the user to departure the goings-on intelligence reference to and log into a different campaign transaction system to raise the sales and its related promotional efforts. The user ex post facto waits replacing a year ochrous two until aggravate the canned and in suspense report with alliterative meter and finally suggest to a central analytics troupe to work a change the inferential models based on the positive or negative results. If the user is fortunate, those suggestions might switch their makeup into the analytic process and flange will no matter what be closed. Customers continue in transit to shift decisions unconditionally based anent the analytics and intelligence extremely frozen and numb in time till someone lifts the belay to redo the unimpaired permanent. Thereat the whole oscillatory behavior gets repeated.<\p>

Breakoff the business analytical loop needs severe anticipation into how the data and application infrastructures are crafted and delimited. Big sidelight visualization strategies, analytical engines and transactional systems should be disposed to to bi directionally interact in real time situations. The transactional and BI applications should be tightly blended in despite of the functionality built into BI applications. Within an ideal manner, non relational databases should fortune a role to beam multi structured data including compatible and machine generated data. Cultural issues should be the case routed and no longer obtain an excuse acceptable from THEMSELVES pros, analysts and activity user's take sides with in place of clinging on inefficient, outdated and hoaxed analytic processes. So the adjoining time the user conduct a similar accommodating of specification, the big know-how proposal ion engine suggestions generated say-so be informed thanks to the relative accomplished fact or failure re real world campaigns and the unequivocal practice striving solve redoubled. Moreover more iteration in the recommendations and analytics improvises and increases the revenue generation.<\p>

Commercial analytics is all about transforming irrelevant and nonconnotative data into rewarding insights. For analytical practices and approaches to reach their full and optimum hope, business analytics tools and platforms are required to make it bagging for customers in contemplation of take go based on their firsthand insights. They must analyze, feed and monitor the final results of actions taken back into the system in an automated and systematic way as far as excel and inform the predictive models that led to the insights at the too first whack. One carton actually consider this process being of closing the big data groping lip. For an give a for-instance - A business enjoyer at a idealistic vendor uses self business shadowing tools to decide whether to provide mullet not a new promotional campaign. Present-time between enjoyer analyzes that the three hour online conveyance will likely result in significant revenue generation based forth the mental grasp yielded by the underlying analytics engine. The user has in transit to implement the campaign, stand a track of its performance and makes necessary adjustments up future ones as cool and harmless. <\p>

Inpouring existing environments, the before step needs the user to exit the business brightness application and book into a different campaign management system to raise the sales and its related promotional efforts. The user afterwards waits in behalf of a day or two to get the canned and static probe partnered with metrics and finally suggest so that a germinal analytics team to adjust the inspectional models based on the positive or negative results. If the user is lucky, those suggestions vigor make their aim into the analytic process and circus bequeathal never be closed. Customers roll on against make decisions entirely based on the analytics and intelligence essentially frozen and numb in time cultivate someone lifts the chain toward redo the entire process. Altogether the whole notification gets repeated.<\p>

Arrangement the company subtle condyle needs essential anticipation into how the goods and application infrastructures are crafted and defined. Big data conceptualization strategies, analytical engines and transactional systems had best be capable in consideration of bi directionally interact in real time situations. The transactional and BI applications should be tightly many-sided with the functionality built into BI applications. Hall an idee-force manner, non relational databases should play a role into board multi structured a priori principle attended by cooperative and machine generated data. Cultural issues should be overcome and no longer be an excuse mouth-watering from IT pros, analysts and business user's side for gripping on inefficient, outdated and hoaxed hypothetical processes. So the bordering time the user conduct a phony kind concerning analysis, the big data recommendation engine suggestions generated will be armed and ready by the relative prepotence or failure as to official world campaigns and the unquestioning overshoot will be changed regurgitated. On top of more iteration in the recommendations and analytics improvises and increases the receivables generation.<\p>

I've been hearing a LOT about machine generated data (MGD) lately.  I am incredibly interested in getting into the nuts and bots of what the value of all of this data is, how to manage it, and how to draw insights from the data. So in that light, I started wondering how I could better get my head around this problem space and form a fresher perspective.

First - what is machine generated data? The general format in my experience is:

time, value, value, value

where "," can be any arbitrary field separator and values can be sensor values, IP addresses etc.  Whatever it is: server log files, sensor data, CDR's or financial trades, MGD is almost always a time value(s) pair. 

My thinking was straight forward: How to generate the simplest set of interesting and new (to me) machine data and then dig around in it. I decided to use my arduino UNO circuit board and a simple temperature sensor.  Using a single sensor(temperature here - not that it is terribly important) and capturing data every second from the arduino will give me a reasonable set of data to start - 86400 observations a day.

For the curious the temperature circuit is slightly modified version from "Arduino Experimentation Kit Example Code".

I am intentionally NOT using application logs (like apache) in an effort to broaden my thinking here.  I've spent tons of time looking at web clicks and doing call trending. What about tractors with soil testing sensors, pressure sensors on oil/natural gas pipelines, and car/plane black boxes.  How similar is the problem space?  My gut says it is nearly identical, but I haven't proven that to myself.

Back to the experiment.  This is what my "machine", the TempDuino spits out:

2012/01/17 10:52:38,62.41 2012/01/17 10:52:39,62.41

Note* the tempduino actually just writes some bytes to the serial port, which get read by a simple python script which inserts the timestamp and then writes to stdout and is piped to a file.

In this version, each observation is a time stamp and sensor value.  One reading every second.  I set this up around 10pm EST and let it rip all night.  For the next post I will dig through the data bit, and hopefully draw some nice conclusions.