Buttercola: System Design: Introduction to Giraph

Buttercola: System Design: Introduction to Giraph

1. What is Giraph?
Apache Giraph is an iterative graph processing system built for high scalability. For example, it is currently used at Facebook to analyze the social graph formed by users and their connections. Giraph originated as the open-source counterpart to Pregel, the graph processing architecture developed at Google and described in a 2010 paper. Both systems are inspired by the Bulk Synchronous Parallel model of distributed computation introduced by Leslie Valiant. Giraph adds several features beyond the basic Pregel model, including master computation, sharded aggregators, edge-oriented input, out-of-core computation, and more. With a steady development cycle and a growing community of users worldwide, Giraph is a natural choice for unleashing the potential of structured datasets at a massive scale. 

2. Existing Solutions:

1. MapReduce in Hadoop
1. Classic map reduce overhead (job startup / shutdown, reloading data from HDFS, shuffling)
2. Immediate results are stored in to disks. Too much disk accesses. 
3. MapReduce programming model not a good fit for graph algorithms. 
2. Google's Pregel:
1. Requires its own computing infrastructure
2. Not open sourced
3. MPI
1. not fault-tolerant 
2. Too generic

3. MPI v.s Giraph

• Giraph vertex centric API is higher level (and narrower) than MPI
• Vertex message queue v.s process level messaging  
• Data distribution, check-pointing handled by Giraph 
• Giraph aggreators are user level MPI_Allreduce reductions.
• Scheduled via Hadoop clusters.

3. Bulk Synchronous Parallel Moel (BSP):

• Computation consists of a series of super steps
• Superstep is an atomic unit of computation where operations can happen in parallel
• During a superstep, components are assigned tasks and receive unordered messages from previous superstep. 
• 
  
• Components communicate through point-to-point messaging.
• All (or a subset of) components can be synchronized through the superstep concept 

Read full article from Buttercola: System Design: Introduction to Giraph