System Design for Big Data [Count Occurrences] | N00tc0d3r

System Design for Big Data [Count Occurrences] | N00tc0d3r

Given a large number (millons or billons) of records (integers, IPs, URLs, query key words, etc.),

• find out top k most frequent records;
• find out duplicates;
• find out whether a given number x is in the set or not.

One common requirement here is how to handle big data efficiently when they are not able to be all loaded into memory.

Read full article from System Design for Big Data [Count Occurrences] | N00tc0d3r