Devoppa

Just a reminder for system exit codes commonly seen on system operation & development.

(will add missing parts)

Exit Code ========= 0    successful termination 1    Catch-all for general errors 2    Misuse of shell built-ins (according to Bash documentation) .. 64    base value for error messages 64    command line usage error 65    data format error 66    cannot open input 67    addressee unknown 68    host name unknown 69    service unavailable 70    internal software error 71    system error (e.g., can't fork) 72    critical OS file missing 73    can't create (user) output file 74    input/output error 75    temp failure; user is invited to retry 76    remote error in protocol 77    permission denied 78    configuration error 78    maximum listed value .. 126    Command invoked cannot execute 127    "command not found" 128    Invalid argument to exit (meaning invalid exit code) 128+n Fatal error signal "n" 255    Exit status out of range (exit takes only integer args in the range 0 - 255)

Also see:

Sometimes your MR job is lenient enough to allow few task failures but the default setting of the leniency is set to fail a job on any task failures.

# of failed Map Tasks exceeded allowed limit. FailedCount: 1.

There isn't anywhere to change this setting in mapred-default.xml.

This customization actually exists as followed in [HADOOP-1144]

Add following to your mapred-site.xml

<property>    <name>mapred.max.map.failures.percent</name>    <value>10</value>    <description>Defaults to 0, meaning any map task failure results to the job failure.</description> </property>

<property>    <name>mapred.max.reduce.failures.percent</name>    <value>10</value>    <description>Defaults to 0, meaning any reduce task failure results to the job failure.</description> </property>

These configurations are also related to

JobConf.getMaxMapTaskFailuresPercent() JobConf.getMaxReduceTaskFailuresPercent()

Note. Keep in mind that a task failes after 4 retries (mapred.map.max.attempts, mapred.reduce.max.attempts).

I needed to expand the number of open files or processes for linux systems because HBase demands it. But one day realized that the limits.conf won't take the * wildcard for 'nproc'. It absuredly works for 'nofile'.

So here is how /etc/security/limits.conf suppose to look

Adding a datanode could be done in few easy steps, but removing one takes little bit of more care.

Even hadoop keeps replications among all datanodes in the cluster, it's probably not a good idea to shutdown a datanode to remove it. It's always possible in Hadoop that some data blocks are under-replicated and only stored in one datanode. Removing that datanode causes a dataloss. So always decommission it and then remove.

Bullet steps to decommissioning a datanode (hdfs 1.0.3)

Make sure dfs.hosts.exclude is set in the hdfs-site.xml and the exclude file exists.

Make sure the target datanode (dn15) is live for load-balancing

Add the hostname or ip to the exclude file. (eg. add dn15 to ${HADOOP_HOME}/conf/datanode.exclude) (this should be done in all hosts to prevent heartbeats among datanodes)

Run "hadoop dfsadmin -refreshNodes" in master. This will initiate the decommissioning process and start re-replicating the blocks in dn15) You'll see something like this in the Namenode web

You can click on the "Decommissioning Nodes" link for more details. Decommissiong of the dn15 (600GB being used by the time of the decommission) took about 6 hours re-replicating 80,000 blocks. Yes, it takes a while. If you want to cancel the decommissioning, delete the hostname in the exclude file (datanode.exclude) and rerun the "hadoop dfsadmin -refreshNodes"

Once the decommission is complete you will see the state of the datanode being changed to "Decommission Complete."

Now you can safely remove or shutdown the node.

If you want to add back the node to the cluster, you will need to delete the hostname from the exclude file first and then start the datanode daemon.

If you are just trying out hadoop & hbase, you don't really need those luxurious servers you won't even be able to afford anyways. HBase will work on a single machine, but to get the feeling of a cluster setup, minimum of two is ideal. The easiest way is to download or create pre-configured hadoop and hbase VM and copy the VM how ever many servers you need. BAMM! you have your hadoop eco already!!!

But that's not really going to be any use in production. It's time to upgrade your cluster to somewhat productive cans.

Hardware

In order to make an extensive use of something called "Locality Group" in Hadoop, it's better to run Hadoop and HBase in co-located nodes. But putting multiple processes in one machine will require a machine that's little above commodity hardware.

Master Server

Master server doesn't require huge space of disk storage. It will only process META data from Hadoop or HBase and these don't take that much of space.

But to relieve its SPOF issue, it's highly recommended to have hardware redundancies such as double NIC (network), double PSU (power), and RAID disk configuration.

Master will require enough memory to operate Namenode, Secondary Namenode, JobTracker, HBase Master, and OS itself.

Slave Server

Slaves are whom actually do works. It is recommended that the servers have enough storage and memory.

Add enough disks how ever many storage you want, but avoid setting up RAID disk stripping on them. It will slow down the whole system down! (see next sections for best ways to mount them)

Because HBase regionservers make very extensive use of in-memory technologies, adding higher memory space than master is ideal.

* Avoid allocating too much of heap memory on each process. A process with large heap space will slow down the fragmentation process of JVM and this cleaning up stage will run on foreground. But it is also good to not squench the memory out of each process. If the allocated heap space is running out OS will start to use swap memory, and swap is known to be slower than RAM memory... duh So bar any possibility that your OS will use swap 

Try to find the sweet spot of memory space.

Filesystem

There's a lot of talk of whether Ext3 or Ext4 should be used for a Hadoop cluster. Ext3 runs well for a Hadoop cluster without any problems. But Ext4 is more recent generation of the EXTended filesystems and it supports larger max storage and individual file size. Ext4 also provides with faster IO than Ext3, but you won't usually realize the difference because a bottleneck for speed reduction is on something else such as network or CPU computation speed. So yet, it's really doesn't make a huge difference but I decided to stick to more recent version of Extended Filesystem for my HBase clusters.

---

Can't imagine how someone would have come what with such idea. All they did was adding a spy cam to a badminton ball. And voila! the creation was born.

I dunno much about cameras but this cam will give extra axis to taking a photo from sky, just like a sky cam. It will give more dynamics and effects to capturing movements.

OK, so it's been quite a while (like 2 months) since I have decided to position myself closer to Hadoop world of Bigdata than any other possible solutions. The reason being Hadoop has been starting to become pretty much what's called a de facto standard basement for cloud and bigdata.  Hadoop's extendability to connect it self to HBase and Hive, Mahout, Chuckwa has been a huge eye-gazer lately, and some people likes to call it HStack or just Hadoop Stack.

There are bountiful number of online tutorials on how to begin Hadoop itself.

For HBase I recommend reading and listening to HBase tutorials and videos first. http://hbase.apache.org/book.html#other.info

Learn how it is different from RDBMS and how hadoop powers HBase fundamentally. And once you get the idea of HBase on HStack, go through the OReilly's HBase reference book by Lars George 2011. It will help you a lot and it holds more recent references than one on hbase.apache.org.

http://ofps.oreilly.com/titles/9781449396107/

I'll research and highlight things I learn from this book and my team of bigdata gru. So you will find many things that you will read from the HBase book in here but also some other useful stuff.

This is just a quick comparison of HBase and Cassandra for my first Tumblr post. It is somewhat biased for HBase to look more cool and shiny because I've been lately doing more study on HBase. I'll come up with more detailed individual research on later posts. Hope this helps.

HBase Features

Auto load balancing and fail-over

BigTable descendent

Google designed Bigdata for intercontinental datacenters and reliable replication

CP of CAP theorem (Consistency, Partition Tolerance)

Compression and Compaction for more efficient data cache performance

Automatic multiple shards per server

Runs on Hadoop Framework

HDFS → data replication, end-to-end checksums, auto re-balancing

MapReduce → Data processing and analytics

Distributes data load by row key in sorted order

If a region server is down, write will be blocked for the data until it is redistributed

Replication: data+log backup method (log shipping)

Write procedure

The data is written to the HBase write-ahead-log in RAM, then it is then flushed to disk

The file on disk is automatically replicated due to the Hadoop Filesystem’s nature

The data enters a “Replication Log”, where it is piped to another Data Center.

  When to use

 When application has a variable schema where each row is slightly different

 When data is stored in collections and keyed on the same value

 When you need key based access to data when storing or retrieving (random access)

 When RDB can’t add columns fast enough and most of them contain NULL in each row

 Tips for using HBase

 HBase use ZooKeeper for various distributed coordination services such as master election

 Use proper NTP and DNS settings among all nodes

 Monitoring of the HBase cluster is very important (mem, cpu, disk, network on each node)

 Use a key prefix that distributes well (ex: do not use timestamp)

 Keep the reasonable number of regions based on memstore and RAM size, and limit RegionServer JVM java heal to 12GB to minimize long GC (Garbage Collection) pauses 

 Don’t

 HBase does not replace RDB; no sql, optimizer, transactions, and joins

 HBase is CPU & Memory intensive with sporadic large seq IO access; avoid sharing resources with other tasks such as MR

Use Case

Facebook Messages – replaced existing MySQL and Cassandra infrastructure

Cassandra Features

Eventual Consistency

AP of CAP theorem(Availability, Partition Tolerance)

Configurable CAP model

Key-Value store

Distributes load across all nodes evenly

BigTable & Dynamo based hybrid architecture

Amazon designed Dynamo on top of close located high fiber network for better performance

Write never fails for high availability

Relies on read for resolving conflicts (expected to be slow on reads)

Use Vector Clock to find most update-to-date data (latest timestamps)

Share state and replicate data using P2P sharing model (Gossip)

Hinted Handoff - basically a hint is given to other live nodes to replay write operations when a dead node comes back on

Write Procedure

The value is written to the “Coordinator” node

A duplicate value is written to another node in the same cluster

A third and fourth value are written from the Coordinator to another cluster across the high-speed fiber

A fifth and sixth value are written from the Coordinator to a third cluster across the fiber

Any conflicts are resolved in the cluster by examining timestamps and determining the “best” value.

Use Case

Cassandra is used to implement Facebook Inbox Search on 2008

Quick diff of HBase and Cassandra

HBase has simpler consistency model than Cassandra

HBase takes consistency win

HBase has SPOF on namenode. Cassandra does not

Cassandra has simpler implementation and easier to hack

Cassandra wins stress test (reliability) done by Adku

Cassandra wins performance test done by Adku

Cassandra has larger developer community (starting to change since 2010)

Indeed Job Trend Comparison

The following is a fairly recent one from SimplyHired. It shows similar curve as the data reported from Indeed. But it also shows that HBase has been growing steadily since Jan 11 whereas Cassandra shows lots of ups and downs. MongoDB is another big NoSQL solution you might want to keep in mind for your data ware-housing.

References

http://www.facebook.com/note.php?note_id=454991608919 Nov 16, 2010

http://www.cloudera.com/blog/2011/04/hbase-dos-and-donts/ Apr 20, 2011

http://blog.adku.com/2011/02/hbase-vs-cassandra.html Feb 2, 2011