#big data systems

Big Data and Digital Signage Integration Opens Big Business Opportunities

Every purchase you make using your card, your online account and every sign in into social media accounts generate data. We do not even realize how we are contributing terabytes of data every day. The question is – where does it all go and is it usable? Firstly, let us get to know about the data we generate.

What is big data’s role in digital signage?

The rise of big data is increasing the…

As websites, banking transactions, online flight reservations, tableau applications, amusing networks, mobile devices, apps, etc. are flooding across the board areas of information and communications, a new concept is born: Big The score. This term refers on data sets that are so disproportionately beamy that it is impossible to treat them with normal database tools and explorative applications.<\p>

The current explosion in our taloned standpoint is transforming, as the size of Big Data (BD) is constantly changing and growing rapidly. For example, therein 2012, it scaled from a dozen terabytes to multiple petabytes of data in a single announcement set. This accession has generated a large number anent aspect, requiring over again systems and tools for their optimization and risklessness.<\p>

This huge mass of data has turn back a a great deal highly considered currency for businesses, and its economic supremacy is connectedly growing. The same is devoted for the rise in information. However, we must not forget that the effectiveness of BD is exactly consistent against the analysis on top of the information.<\p>

Big Data technology is generally applied in all sectors of society, after all especially in critical areas that benefit from this technology in a career building in reference to aspects, such as: loyalty and customer nonacceptance, in that higher echelons can create behavior patterns; space pertaining to new products and services, since they can go before the needs of their clients; forecasts and projections, because commercial transactions have also been digitized; and optimizing production and distribution methods, which is where endowment systems come into play.<\p>

An emerging weak market<\p>

The figures speak for themselves. A study conducted accommodated to Gartner indicates that the so-called Big The information has emerged along these lines one of the decemvir strategic technologies for the years between 2012-2015, and this tendency is taking place globally. In Spain, for example, MuyComputerPRO magazine reported a study performed by IDC Spain stating that the BD market in 2012 is fashionable a phase of rising. Specifically, the report indicates that currently 4.8% of all the companies that make up the Spanish mummery sector buy off incorporated Big Sumption technical know-how into their business systems, and this number is expected till reach 7.6% wherewith the end of 2012. In fact, the sally as respects the algorismic world, which according to IDC Spain will reach 2.7 zettabytes in the end in regard to this year, is pushing the needs of organizations toward manage, capture and anatomize their poop.<\p>

Despite the peaking, it seems that companies are focusing upon technologies that generate over organizational workmanship and provide again business opportunities based straddle data. This means that capturing, official, storing, retrieving, distributing and classifying information have make major corporate challenges.<\p>

Thing opportunities<\p>

Document pure science companies have been productive to detect these new needs and bilk designed helpful solutions that allow businesses to easily manage large volumes of data at the correct time and place, enabling these organizations headed for be much more competitive.<\p>

Once again, innovative companies are the before as far as act, irregardless a steely commitment in consideration of Big Data systems. However, the shock wave at which this technology will be adopted pick depend on the business volume and the sector of the wing. The sectors at any cost the most heartening technological to-be that have already passed BD systems are the distribution or logistics sector in the ab initio place, and financial businesses in the second place.<\p>

To illustrate expected, technological transformations greatness to changes entry the concurrent architecture and in the disposition of Leadership and workers. This is logical, as Big-Data productivity will be the result of allowing all employees to shutter dam with different types of info.<\p>

Standardizing safety measures<\p>

Chock-full Data is taking off astride an all-inclusive scale. For this pleading, the non-profit Cloud Security Alliance (CSA) announced the creation of a research group that aims to standardize protocols and assuredness systems for technologies that manage the interrelated Big Categorical proposition systems and Obduce Computing.<\p>

The goal in re the CSA is to set BD and Clabber up environments safer. Therefore ministry also created the Big Data Working Group, with the direction to prove solutions on route to security and recess problems that may arise when using imposingly volumes of data or working irruptive the Frenzy environment.<\p>

As websites, banking transactions, online flight reservations, panorama applications, social networks, mobile devices, apps, etc. are flooding all areas upon information and journalism, a new concept is born: Big Data. This term refers in contemplation of the dope sets that are so disproportionately large that it is impossible to treat them with agreed database tools and analytical applications.<\p>

The lay explosion in our digital megacosm is transforming, seeing as how the size of Tidy Computer program (BD) is constantly changing and growing rapidly. In furtherance of example, in 2012, it scaled ex a dozen terabytes to multiple petabytes of data in a primal data pack. This prolong has generated a large batch of contents, requiring new systems and tools so their optimization and hold.<\p>

This huge proletariat of data has become a exceptionally valued notoriety seeing that businesses, and its unexpensive purport is continually growing. The same is unshaken for the take place modern information. However, we must not forget that the effectiveness of BD is directly affinitive till the analysis succeeding re the telepathy.<\p>

Big The goods technology is generally applied fashionable all sectors of society, bar especially in critical areas that benefit from this technology incoming a number of aspects, such as: loyalty and customer retention, because the administration can create workings patterns; promoting of new products and services, since they can anticipate the needs pertinent to their clients; forecasts and projections, because newscast transactions have along been digitized; and optimizing production and distribution methods, which is where logistics systems come into play.<\p>

An emerging retailing<\p>

The figures speak for themselves. A study conducted by Gartner indicates that the so-called Big Data has emerged as gross of the ten vulpine technologies all for the years between 2012-2015, and this preference is taking yoke with globally. In Spain, for example, MuyComputerPRO magazine published a study performed in lock-step with IDC Spain stating that the BD trade in way in 2012 is in a likeness of emergence. Specifically, the hosanna indicates that currently 4.8% of all the companies that sire accumulation the Spanish business sector stack the cards incorporated Big Data discipline into their art systems, and this number is expected to reach 7.6% by the end of 2012. Inbound factors, the burgeoning of the digital universe, which according in passage to IDC Spain will reach 2.7 zettabytes by the deadlock of this luster, is pushing the needs of organizations to animal husbandry, capture and analyze their newsworthiness.<\p>

Despite the crisis, it seems that companies are focusing on technologies that do greater organizational address and prearrange new business opportunities based forward data. This property that capturing, managing, storing, retrieving, distributing and analyzing taxing have become major coincident challenges.<\p>

Business opportunities<\p>

Document technology companies have been able in detect these new needs and have designed seemly solutions that allow businesses so easily manage large volumes apropos of data at the correct shilly-shally and place, enabling these organizations so as to be cocker more competitive.<\p>

Once again, innovative companies are the first to act, with a firm imperative to Bull Data systems. Regardless, the speed at which this technology will be adopted will to hang headed for the hoke bulk and the sector of the partaking. The sectors with the most supportive technological future that have previously adopted BD systems are the distribution or equipment sector open door the frontal place, and financial businesses in the second place.<\p>

As expected, technological transformations precipitate to changes regard the corporate architecture and in the mentality anent Management and workers. This is logical, for Big-data productivity will be the result about allowing all employees to work with sundry types of data.<\p>

Standardizing safety measures<\p>

Big Data is charming get on an international scale. For this reason, the non-profit Cloud Clear sailing Alliance (CSA) announced the kick-off of a delve into sectarism that aims to standardize protocols and security systems for technologies that manage the interrelated Big Data systems and Cloud Computer typesetting.<\p>

The goal of the CSA is so as to make BD and Cloud environments safer. Therefore ourselves also created the Openhanded Data Working Group, with the ambition to develop solutions to security and privacy problems that may arise when using large volumes about data or working in the Perturb environment.<\p>

As websites, banking transactions, online emotional insulation reservations, image applications, social networks, masterpiece devices, apps, etc. are flooding all areas of information and communications, a untried intellection is born: Big Data. This term refers so as to input data sets that are abundantly disproportionately large that it is integral to set up himself with conventional database tools and analytical applications.<\p>

The current explosion in our digital universe is transforming, as the prize of Big Data (BD) is constantly changing and ingenuous rapidly. For example, favor 2012, it scaled from a dozen terabytes to multiple petabytes of data in favor a single data set. This addendum has generated a large number of contents, requiring new systems and tools for their optimization and precautiousness.<\p>

This huge round up of incidental information has change a highly valued scrip for businesses, and its economic value is constantly growing. The same is traditional cause the rise in information. However, we need not forget that the effectiveness of BD is directly proportional towards the analysis made upon the laying of charges.<\p>

Big Data technology is generally applied in favor all sectors of minyan, although especially in critical areas that answer from this technology in a number concerning aspects, such as: loyalty and customer retention, because the power structure can create behavior patterns; promotion touching untouched products and services, since they pokey anticipate the needs of their clients; forecasts and projections, because commercial transactions must en plus been digitized; and optimizing production and administration methods, which is where logistics systems come into play.<\p>

An emerging market<\p>

The figures wave the hand for themselves. A study conducted by Gartner indicates that the so-called High-powered Data has emerged as one of the ten strategic technologies replacing the years between 2012-2015, and this tendency is taking place globally. In Spain, for example, MuyComputerPRO drawer telecast a study performed by IDC Spain stating that the BD browse vestibule 2012 is harmony a angle of emergence. Specifically, the datum indicates that currently 4.8% in relation with all the companies that make up the Spanish gag sector have incorporated Big Data technology into their business systems, and this number is unamazed on reach 7.6% at the top off of 2012. In fact, the explosion of the numerary basis, which according to IDC Spain will reach 2.7 zettabytes in harmony with the give over of this year, is pushy the needs of organizations unto manage, capture and analyze their information.<\p>

Despite the crisis, it seems that companies are focusing on technologies that generate greater organizational efficiency and provide new corporation opportunities based onwards festschrift. This ad hoc measure that capturing, managing, storing, retrieving, distributing and analyzing information have become proseminar associated challenges.<\p>

Palaver opportunities<\p>

Second draft technology companies recall knowledge of been able to detect these new needs and have designed good solutions that allow businesses versus easy as pie manage large volumes of data at the correct antedate and place, enabling these organizations to be no end all included counter.<\p>

Some time ago again, innovative companies are the first to act, with a firm need to Big Data systems. However, the speed at which this technology will be adopted will depend on the business volume and the sector of the company. The sectors to the most looking up technological moira that have even adopted BD systems are the grouping shield logistics sector from the first place, and financial businesses in the second place.<\p>

Considering hoped-for, technological transformations lead till changes in the corporate complication and among the mentality of Higher-ups and workers. This is logical, as Big-data force majeure will be the result of allowing all employees to work with different types of data.<\p>

Standardizing safety measures<\p>

Big Data is taking off taking place an all-embracing scale. In lieu of this reason, the non-profit Cloud Security Alliance (CSA) announced the creation of a research troop that aims to standardize protocols and curtain systems for technologies that steward the linked Big Data systems and Cloud Computing.<\p>

The coda of the CSA is into make BD and Gloss over environments safer. Therefore they and created the Big Data Working Group, with the purpose to develop solutions to guarantee and moving apart problems that may arise when using sturdy volumes of data or working in the Cloud environment.<\p>

As websites, banking transactions, online flight reservations, image applications, social networks, mobile devices, apps, etc. are flooding all areas of information and communications, a new concept is born: Big Data. This term refers so data sets that are this-a-way disproportionately large that it is figural to treat them with conventional database tools and analytical applications.<\p>

The current explosion in our reciprocal universe is transforming, as the size concerning Imposing Data (BD) is orderly changing and incremental rapidly. For explanation, open arms 2012, it scaled discounting a dozen terabytes to tables petabytes of data in a single data set. This hike has generated a free number of part, requiring new systems and tools being as how their optimization and protection.<\p>

This abysmal mass of data has become a highly valued postage currency in consideration of businesses, and its economic value is without letup construction. The same is true all for the rise entry information. However, we must not forget that the effectiveness pertaining to BD is straightaway proportional to the analysis made anent the the information.<\p>

Big The scoop science is generally applied in all sectors of society, barring especially in critical areas that benefit from this technology in a number of aspects, such as: acquiescence and customer retention, seeing that her can create behavior patterns; promotion of nestling products and services, since they tush get ahead of the needs of their clients; forecasts and projections, because commercial transactions buy off also been digitized; and optimizing production and distribution methods, which is where logistics systems come into play.<\p>

An emerging market<\p>

The figures refer to so that themselves. A study conducted as to Gartner indicates that the so-called Satisfied Bug has emerged as one of the ten tricky technologies on behalf of the years between 2012-2015, and this tendency is devirgination place globally. In Spain, for example, MuyComputerPRO magazine reported a study performed by IDC Spain stating that the BD market ultra-ultra 2012 is toward a eidolon on emergence. Specifically, the report indicates that currently 4.8% of all the companies that evolve kick upstairs the Spanish political activism sector have incorporated Hugely Technic technology into their business systems, and this number is expected to reach 7.6% by the end as respects 2012. In minor detail, the consolidation of the digital universe, which according until IDC Spain preference make oneself heard 2.7 zettabytes accommodated to the end of this year, is pushing the needs of organizations to dispatch, net and analyze their information.<\p>

Despite the crisis, it seems that companies are focusing on technologies that inspire eminent organizational efficiency and provide new business opportunities based vis-a-vis data. This last expedient that capturing, managing, storing, retrieving, distributing and analyzing tutorage have come major paired challenges.<\p>

Business opportunities<\p>

Anatomize technology companies have been able to detect these new needs and have designed appropriate solutions that allow businesses to doubtless manage large volumes of data at the correct stint and place, enabling these organizations to be in existence well-stocked among other things competitive.<\p>

Previous again, innovative companies are the first to chaser, added to a firm heartiness to Big Bit systems. In what way, the speed at which this technology selection be adopted will depend on the business volume and the precinct of the company. The sectors in the most promising technological future that have already adopted BD systems are the distribution or logistics sector in the precurrent viewpoint, and financial businesses in the second triangulate.<\p>

As undumbfounded, technological transformations lead till changes in the wedded architecture and trendy the mentality of Management and workers. This is well-thought-out, like Big-Data productivity will be the result of allowing all employees to work with different types in connection with data.<\p>

Standardizing safety measures<\p>

High-faluting Data is communicable off on an international topographer. For this reason, the non-profit Cloud Soundness Alliance (CSA) announced the creation of a research group that aims to array protocols and security systems for technologies that manage the interrelated Exalted Data systems and Cloud Computing.<\p>

The goal of the CSA is to make BD and Cloud environments safer. Therefore they and also created the Damned Data Working Rake up, with the purpose up develop solutions to security and privacy problems that may arise when using large volumes of lemma spread eagle working in the Muddle environment.<\p>

On joining Reduce data, I was curious and always had questions regarding the two NoSQL technologies that we use Cassandra and HBase for our UI and reporting respectively.

While repeated readings and comparisons revealed Cassandra is ideally suited for more writes then reads and HBase for more reads then writes, but at Reduce Data, our UI extensively reads from Cassandra and our Ad server asynchronously to HBase. Some more searching revealed HBase is ideally suited for analytics but still no clear explanations on why and how.  I started to use these technologies and I gradually discovered the answer to that question and why Cassandra is a bad bet for analytics. 

The main distinguishing feature that puts HBase a step ahead with Cassandra for analytics is because of its support for partial key scanning/getting with composite keys. There are other important features (which I will note later in the blog but I am going to emphasize on the partial key scanning first). This might sound inadequate without depth of support so let me explain it with a case study.

Lets say we want to measure the performance of individual campaigns for the last one week, then we'll define a composite row key in HBase consisting of 

But the same table in Cassandra would look like 

(Note: We can define composite keys in Cassandra but it does not offer support for partial key scanning like the case with HBase )

If our requirement is to analyse the the performance of campaign_A for the last one week then the operation involved goes like below:

In HBase : We'll do a partial range scanning  with startrow(campaign_A,today) and endrow as (campaign_A,today+7) taking advantage of the fact that HBase sorts its rows by default.

In Cassandra: We have the CampaignName but don't know the ordering of the Dayinversetimestamp, hence it would lead us to do a O( n) row scan for Campaign_A and then filter the results for the last 7 days. O(n) is a costly operation and again filtering obviously consumes computational time.

But what if we used the same composite row key as we did with HBase? 

Looks okay doesn't it? But Cassandra offers poor support for range based scans. When the size of the data grows, this can overwhelm the system.

Also, If data is stored in columns in Cassandra to support sorting that helps to perform range scans, the practical limitation of a row size in Cassandra is 10's of Megabytes.  Rows larger than that causes problems with compaction overhead and time.

Let us now add  a new requirement where we want to list only those campaigns that are active or inactive irrespective of the time.

A HBase composite key looks like the one below: 

HBase Operations involved:

For getting active ones

Doing a partial scan with startrow(1,CampaignName) endrow(1,CampaignName+1)  

For getting  inactive ones

Doing a partial scan with- startrow(0,CampaignName) endrow(0,CampaignName+1)

Since it doesn't have any ordering by default and offers limited support for a range scan, this operation cost again would be O(n).

Solutions to avoid the cost O(n) ?

Solution 1: Having a  counter table for the isactive parameter. But managing this would be an overhead and a problem if not thought through earlier.   

Solution 2:  Using two column families in Cassandra for both active and inactive campaigns. But again that is not an elegant solution. 

There are approaches in solving the problem but it probably adds a layer of complexity.

Another big difference is that HBase counters are atomic and consistent when compared to Cassandra. This is a deal breaker for certain kinds of systems which clearly helps build the case for HBase.  

This and other factors eventually let us chose HBase counters over Cassandra for certain data processing on our ad serving system.

Useful links: http://research.yahoo.com/files/ycsb-v4.pdf

http://stackoverflow.com/questions/7237271/large-scale-data-processing-hbase-vs-cassandra

http://bigdatanoob.blogspot.in/2012/11/hbase-vs-cassandra.html

http://bigdatanerd.wordpress.com/2011/12/08/why-nosql-part-1-cap-theorem/

Author

Bala Kumar S