All About Programming: April 2015

jquery - Find object by id in array of javascript objects - Stack Overflow

As you are already using jQuery, you can use the grep function which is intended for searching an array:

var result = $.grep(myArray, function(e){ return e.id == id; });

The result is an array with the items found. If you know that the object is always there and that it only occurs once, you can just use result[0].foo to get the value. Otherwise you should check the length of the resulting array. Example:

Read full article from jquery - Find object by id in array of javascript objects - Stack Overflow

javascript - Fatal error: Unable to find local grunt - Stack Overflow

Solution for v1.4:

1. npm install -g grunt-cli  2. npm init     fill all details and it will create a package.json file.  3. npm install grunt (for grunt dependencies.)

Read full article from javascript - Fatal error: Unable to find local grunt - Stack Overflow

Hello Newman: A REST Client for Scala | PayPal Engineering Blog

At StackMob, we ran a service oriented architecture to power our products. To build out that architecture we ran a distributed system composed of many services inside our firewall.

Every service in the system used HTTP for transport and JSON for serialization to communicate over the wire. The challenge we faced was this: How do we easily and flexibly send and receive JSON data over HTTP in Scala? We had the same challenge for building servers and clients.

When we began investigating the existing pool of HTTP clients, we turned to the massive JVM community for high quality clients that could handle our needs. We found a lot of them! I'll highlight two clients with which we gained significant experience.

Read full article from Hello Newman: A REST Client for Scala | PayPal Engineering Blog

Using a Postman http client for efficient HTTP testing - Agiliq Blog | Django web app development

POSTMAN HTTP and REST client is one of its kind which makes the API testing process effortless. It is available as both a Google Packaged App and a Google Chrome in-browser app. It has got clean and intuitive user interface, with all the important features accessible within one click. The performance of the API testing process gets drastically affected by following features it has:

Read full article from Using a Postman http client for efficient HTTP testing - Agiliq Blog | Django web app development

Install node.js modules in eclipse - Stack Overflow

I dont think that there is an eclipse integrated way to do this. My suggestion is that you download node.js+npm from http://nodejs.org/download/ and then open up a terminal/cmd and in your node.js project directory do "npm install myPackage"

Read full article from Install node.js modules in eclipse - Stack Overflow

Import and export schema in cassandra - Stack Overflow

To export keyspace schema:

cqlsh -e "DESC KEYSPACE user" > user_schema.cql

To export entire database schema:

cqlsh -e "DESC SCHEMA" > db_schema.cql

To import schema open terminal at 'user_schema.cql' ('db_schema.cql') location (or you can specify the full path) and open cqlsh shell. Then use the following command to import keyspace schema:

source 'user_schema.cql'

To import full database schema:

source 'db_schema.cql'

Read full article from Import and export schema in cassandra - Stack Overflow

IDE for Node.js / Javascript? - Stack Overflow

Jetbrains WebStorm/PhpStorm (most comprehensive); $100 for commercial license, $50 personal, $25 academic, free for open source developers upon application approval

Komodo IDE (full Node.js debugging support since version 8); $382

Nodeclipse, an Eclipse plugin for Node.js

Cloud9 - cloud-based IDE

Aptana Studio

vim

Sublime Text

Nide

Read full article from IDE for Node.js / Javascript? - Stack Overflow

Watch Movies and Other Transferred Videos on Google Glass | Google Glass How To | Glass Apps Source

Viewing Experience. Viewing short videos that you've recorded on Google Glass is pretty neat and not too uncomfortable. With that said, viewing a video over an hour or so on Google Glass is going to get uncomfortable; after-all you are only focusing on the video with one eye. Viewing a movie on Google Glass is possible, but it's definitely not going to compare to watching a movie or video on a normal television screen.

Read full article from Watch Movies and Other Transferred Videos on Google Glass | Google Glass How To | Glass Apps Source

Linux Commando: How to get the process start date and time

The venerable ps command deserves first consideration.

Most Linux command-line users are familiar with either the standard UNIX notation or the BSD notation when it comes to specifying ps options.

If ps -ef is what you use, that is the UNIX notation.

Read full article from Linux Commando: How to get the process start date and time

Speed | Functional Programming: Why Should You Care? | InformIT

This is also related to performance. Compiler optimizations work by transforming the code that the programmer has written into something that has identical (observable) semantics but a different implementation. A trivial example is common subexpression elimination. For example, if you write:

Read full article from Speed | Functional Programming: Why Should You Care? | InformIT

javascript - What is the purpose of Node.js module.exports and how do you use it? - Stack Overflow

module.exports is the object that's actually returned as the result of a require call.

The exports variable is initially set to that same object (i.e. it's a shorthand "alias"), so in the module code you would usually write something like this:

var myFunc1 = function() { ... };  var myFunc2 = function() { ... };  exports.myFunc1 = myFunc1;  exports.myFunc2 = myFunc2;

to export (or "expose") the internally scoped functions myFunc1 and myFunc2.

And in the calling code you would use:

var m = require('mymodule');  m.myFunc1();

where the last line shows how the result of require is (usually) just a plain object whose properties may be accessed.

NB: if you overwrite exports then it will no longer refer to module.exports. So if you wish to assign a new object (or a function reference) to exports then you should also assign that new object to module.exports

Read full article from javascript - What is the purpose of Node.js module.exports and how do you use it? - Stack Overflow

Creating & Using Modules In Node.js & Understanding Paths | 10K-LOC

Javascript libraries are commonly referred to as "modules". The modules are typically imported by other Javascript files to use their functionality. Modules could contain class definitions, or helper functions.

Read full article from Creating & Using Modules In Node.js & Understanding Paths | 10K-LOC

Node.js Handbook - How `require()` Actually Works | FredKSchott

Curious, I dug into Node core to find out what was happening under the hood. But instead of finding a single function, I ended up at the heart of Node's module system: module.js. The file contains a surprisingly powerful yet relatively unknown core module that controls the loading, compiling, and caching of every file used. require(), it turned out, was just the tip of the iceberg.

Read full article from Node.js Handbook - How `require()` Actually Works | FredKSchott

Error Handling in Nodejs - Developer Center - Joyent

Error handling is a pain, and it's easy to get by for a long time in Node.js without dealing with many errors correctly. But building robust Node.js apps requires dealing properly with errors, and it's not hard to learn how. If you're really impatient, skip down to the "Summary" section for a tl;dr.

Read full article from Error Handling in Nodejs - Developer Center - Joyent

Understanding module.exports and exports in Node.js

As developers, we often face situations where we need to use unfamiliar code. A question will arise during these moments. How much time should I invest in understanding the code that I'm about to use? A typical answer is learn enough to start coding; then explore that topic further when time permits. Well, the time has come to gain a better understanding of module.exports and exports in Node.js. Here's what I have learned.

Read full article from Understanding module.exports and exports in Node.js

folders | npm Documentation

Local install (default): puts stuff in ./node_modules of the current package root.

Global install (with -g): puts stuff in /usr/local or wherever node is installed.

Install it locally if you're going to require() it.

Install it globally if you're going to run it on the command line.

If you need both, then install it in both places, or use npm link.

Read full article from folders | npm Documentation

node.js - Nodejs Cannot find module - Stack Overflow

Just to quote from here:

https://www.npmjs.org/doc/files/npm-folders.html

Install it locally if you're going to require() it.
Install it globally if you're going to run it on the command line.
If you need both, then install it in both places, or use npm link.

Read full article from node.js - Nodejs Cannot find module - Stack Overflow

Hello Node! - How To Node - NodeJS

In programming literature it has become the standard to create a hello world program as the first example. This article will go through a few simple hello world type examples with everything from simple terminal output to an http server that uses an external framework for some semantic sugar.

Then we'll shift gears and go through a real example that teaches enough to get you up on your feet writing your own web application using node.JS.

Read full article from Hello Node! - How To Node - NodeJS

Rest Api Documentation | Don't Make the Same Mistake Twice

SpringDoclet is a Javadoc doclet that generates documentation on Spring Framwork artifacts in a project. The detection of Spring artifacts is based on the presence of Spring annotations on Java classes and methods.

Read full article from Rest Api Documentation | Don't Make the Same Mistake Twice

Don't rape HTTP: If-(None-)Match & the 412 HTTP status code -- Alessandro Nadalin, forward-thinking technologist

Don't rape HTTP: If-(None-)Match & the 412 HTTP status code — Alessandro Nadalin, forward-thinking technologist

If the Etag of the resource /people/alessandro-nadalin is the same of the server’s one, the PUT updates the user.

But if they don’t match, the server responds with a 412 Precondition Failed status code, which means that the resource is out of date.

Read full article from Don't rape HTTP: If-(None-)Match & the 412 HTTP status code — Alessandro Nadalin, forward-thinking technologist

Net::HTTPServerException: 412 "Precondition Failed"

Every single time this has happened to me, I've had stop and scratch my head. Why can't I remember what causes this!? Well, in my case it's usually a misspelled cookbook or recipe name. For example, I might have a role that looks something like this:

Read full article from Net::HTTPServerException: 412 "Precondition Failed"

406 Error - Not Acceptable - InMotion Hosting

Web browsers make a request for information from the server. When this happens, it sends an Accept header. This tells the server in what formats the browser can accept the data. If the server cannot send data in a format requested in the Accept header, the server sends the 406 Not Acceptable error.

The error can also be generated by the mod_security module. Mod_security, a type of firewall program that runs on Apache web server, scans for violations of the rules it has set. If an action occurs that violates one of these rules, the server will throw a 406 error.

Read full article from 406 Error - Not Acceptable - InMotion Hosting

AWS Developer Forums: How to cancel all the whole Amazon web ...

If you'd like to cancel your AWS account, you'll first need to cancel all of your opt-in services:

1. Visit the Account Activity page: https://aws-portal.amazon.com/gp/aws/developer/account/index.html

2. Under the name of each opt-in service, click the "View/Edit Service" link.

3. Click the "Cancel Service" option for each.

Then contact our customer service team directly to close your account using this web form:

https://aws-portal.amazon.com/gp/aws/html-forms-controller/contactus/aws-account-and-billing

Read full article from AWS Developer Forums: How to cancel all the whole Amazon web ...

Evaluating spring @value annotation as primitive boolean - Stack Overflow

Looks like you're missing the PropertyPlaceholderConfigurer. You need to register it as a bean factory post processor. Theoretically this could be done like this:

Read full article from Evaluating spring @value annotation as primitive boolean - Stack Overflow

java - Check parameters annotated with @Nonnull for null? - Programmers Stack Exchange

Well, consider why you don't write

this.name = Preconditions.checkNotNull(name);  // Might be necessary  that.name = Preconditions.checkNotNull(this.name);  // Who knows?

Programming isn't black magic. Variables don't suddenly become Null. If you know a variable isn't Null, and you know it's not changed, then do not check it.

If you do check it, some programmer in the future (possibly you) will spend a lot of time finding out why that check is there. The check must be there for a reason, after all, so what are you overlooking?

Read full article from java - Check parameters annotated with @Nonnull for null? - Programmers Stack Exchange

Not null arguments in Java public methods | Gualtiero Testa

Important: the @Nonnull annotation is simply an annotation, a kind of hint for the tools. It is not a run-time check and it does not enforce or guarantee that nobody can call the method with a null argument. So our "if arg == null" check should stay there.

Read full article from Not null arguments in Java public methods | Gualtiero Testa

null - Which @NotNull Java annotation should I use? - Stack Overflow

I wouldn't use anything that is not under the javax namespace at all (even though I love what Lombok and IntelliJ are doing). Otherwise, you might be creating a dependency on something other than what the run-time gives you for something that is pretty much semantics. Maybe for some projects, that's ok, but that'd be a deal-breaker for me.

I would use javax.validation.constraints.NotNull because that's already here with Java EE 6.

The javax.annotation.NonNull might not be here until Java 8 (as Stephen pointed out). And the others are not standard annotations.

It would have been nice if annotations were extensible. That way you could define your own non-null annotation inheriting/extending from anything. Then when standards get ironed out, all you would have to do would be to redefine your own custom annotation.

Read full article from null - Which @NotNull Java annotation should I use? - Stack Overflow

Eclipse IDE - Tomcat version 6.0 only supports J2EE 1.2, 1.3, 1.4, and Java EE 5 Web modules

Eclipse IDE – Tomcat version 6.0 only supports J2EE 1.2, 1.3, 1.4, and Java EE 5 Web modules

Import a Java web project in Eclipse, build with Maven, once create a Tomcat server instance, unable to add the Java web project, and showing

Read full article from Eclipse IDE – Tomcat version 6.0 only supports J2EE 1.2, 1.3, 1.4, and Java EE 5 Web modules

The Byzantine Generals Problem | Dr Dobb's

The Byzantine General's problem is one of many in the field of agreement protocols. In 1982, Leslie Lamport described this problem in a paper written with Marshall Pease and Robert Shostak. Lamport framed the paper around a story problem after observing what he felt was an inordinate amount of attention received by Dijkstra's Dining Philosopher problem.

Read full article from The Byzantine Generals Problem | Dr Dobb's

Quantum Byzantine agreement - Wikipedia, the free encyclopedia

Byzantine fault tolerant protocols are algorithms that are robust to arbitrary types of failures in distributed algorithms. With the advent and popularity of the Internet, there is a need to develop algorithms that do not require any centralized control that have some guarantee of always working correctly. The Byzantine agreement protocol is an essential part of this task. In this article the quantum version of the Byzantine protocol,^[1] which works in constant time is described.

Read full article from Quantum Byzantine agreement - Wikipedia, the free encyclopedia

[Tomcat-users] Header names lower case - Grokbase

I noticed the Tomcat implementation of HttpServletRequest.getHeaderNames() returns all header names in lower case. Is there any possibility to get them with their original case?

Read full article from [Tomcat-users] Header names lower case - Grokbase

java - Spring RestTemplate - how to enable full debugging/logging of requests/responses? - Stack Overflow

I finally found a way to do this in the right way. Most of the solution comes from How do I configure Spring and SLF4J so that I can get logging?

It seems there are two things that need to be done :

Add the following line in log4j.properties : log4j.logger.httpclient.wire=DEBUG

Read full article from java - Spring RestTemplate - how to enable full debugging/logging of requests/responses? - Stack Overflow

java - Spring RestTemplate GET with parameters - Stack Overflow

OK, so I'm being an idiot and I'm confusing query parameters with url parameters. I was kinda hoping there would be a nicer way to populate my query parameters rather than an ugly concatenated String but there we are. It's simply a case of build the URL with the correct parameters. If you pass it as a String Spring will also take care of the encoding for you.

Read full article from java - Spring RestTemplate GET with parameters - Stack Overflow

java - Generics with Spring RESTTemplate - Stack Overflow

ParameterizedTypeReference has been introduced in 3.2 M2 to workaround this issue.

Read full article from java - Generics with Spring RESTTemplate - Stack Overflow

How to read authorization header in JAX-RS service - Stack Overflow

You should use @Context HttpServletRequest request to inject request in your method, like this:

public Response log(@Context HttpServletRequest req) throws JSONException

Read full article from How to read authorization header in JAX-RS service - Stack Overflow

How Can I View a Site's SSL Certificate? | Chron.com

If using Chrome, click the "Customize and control Google Chrome" icon containing three lines or the wrench icon. Click "Options" or "Settings" and click the "Under the Hood" tab or click the "Show advanced settings" link. Click the "Manage certificates" button to access SSL certificates. Click on the certificate you want to view to highlight it. Click the "View" button to see the certificate's content.

Read full article from How Can I View a Site's SSL Certificate? | Chron.com

Show SSL certificates with cURL | UnixBulletin

cURL is a very useful tool when you want to get data from sites using SSL encryption, among other things we can see the SSL certificate that is using.

Read full article from Show SSL certificates with cURL | UnixBulletin

(9) How do I disable "Command-q" in OS X? - Quora

You can reassign menu commands in Mac's System Preferences menu to different shortcuts, either for individual applications or for all applications. This will let you effectively reassign Cmd-Q to anything (or nothing), if you'd like.

Read full article from (9) How do I disable "Command-q" in OS X? - Quora

(9) How do I unbind "Ctrl-Q" or "Command-Q" for quit on Google Chrome? (shortcut manager fails) - Quora

In the Chrome menu (top left of the screen), you can enable a "warn before quitting" option - this lets you keep using Cmd+Q to quit, but avoids the accidental activation by requiring you to hold it for a few seconds.

Read full article from (9) How do I unbind "Ctrl-Q" or "Command-Q" for quit on Google Chrome? (shortcut manager fails) - Quora

Create Keyboard Shortcuts for Chrome Extensions

Chrome 22 (currently in the Dev channel) has a cool feature that lets you set keyboard shortcuts for extension buttons.

Just open the extensions page (click the wrench icon, then select Tools and Extensions), scroll to the bottom of the page and click "Configure commands". You can enter shortcuts for all the extensions that use browser actions, a fancy name for the buttons displayed next to the Omnibox.

Read full article from Create Keyboard Shortcuts for Chrome Extensions

dzhibas/SublimePrettyJson

Using Command Palette Ctrl+Shift+P find "Pretty JSON: Minify (compress) JSON" (you can search for part of it like 'json minify') this will make selection or full buffer as single line JSON which later you can use in command lines (curl/httpie) or somewhere else...

Read full article from dzhibas/SublimePrettyJson

JsonTree Sublime Text 3 plugin | Flyclops Tech Blog

So, I wrote this wonky, almost-working plugin for Sublime Text 3 (which is a "beta" but substantially better than 2) called JsonTree. It works by hitting a key combination (cmd-ctrl-j in OS X, or ctrl-alt-j in Windows/Linux) and brings up a tree-view of your JSON document that is browsable and searchable. Screenshot:

Read full article from JsonTree Sublime Text 3 plugin | Flyclops Tech Blog

sublimetext2 - Replace \n with actual new line in Sublime Text - Stack Overflow

Turn on Regex Search and Replace (icon most to the left in search and replace bar or shortcut Alt+R)

Find What: \\n
Replace with: \n

Read full article from sublimetext2 - Replace \n with actual new line in Sublime Text - Stack Overflow

Chrome Running in Background? Here's How to Stop It

Click on the Chrome menu (or press Alt+E)

Select Settings

Click on the link titled 'Show advanced settings'

Under the section headed 'System' untick the box next to "Continue running background apps when Google Chrome is closed"

Read full article from Chrome Running in Background? Here's How to Stop It

What Is a HAR FIle? | MaxCDN One

HAR stands for HTTP Archive. This is a common format for recording HTTP tracing information. This file contains a variety of information, but for our purposes, it has a record of each object being loaded by a browser. Each of these objects' timings is recorded.

Read full article from What Is a HAR FIle? | MaxCDN One

Getting Started · Caching Data with Spring

In its most basic setup, the annotation requires a CacheManager to serve as a provider for the relevant cache. In this example you use an implementation that delegates to a ConcurrentHashMap. The CachingConfigurer interface provides more advanced configuration options.

Read full article from Getting Started · Caching Data with Spring

Best Way To Pretty Print (Format) JSON On The Command-Line

Developers tend to work with APIs a lot and these days most of these APIs are JSON. These JSON strings aren't exactly easy to read unless they are formatted well. There are many services online that can pretty print JSON for you, but that's annoying. I love the command-line and whenever I am playing with new APIs or writing my own I mostly use CURL which means I need a good way to pretty print my JSON on the command-line. It should be simple, quick, easy to remember, easy to install – we're not trying to solve complex algorithms, just format some JSON.

Read full article from Best Way To Pretty Print (Format) JSON On The Command-Line

java - Transform and convert a List to Set with Guava - Stack Overflow

This creates an ImmutableSet, which does not accept null. So if you want your Set to contain null, you'll have to use another solution, like the one you're currently using.

Note that, if it's the creation of the temporary collection that bothers you, you shouldn't be bothered. No copy is made. The collection is simply a view over the original list.

Read full article from java - Transform and convert a List to Set with Guava - Stack Overflow

java - Transform and convert a List to Set with Guava - Stack Overflow

This creates an ImmutableSet, which does not accept null. So if you want your Set to contain null, you'll have to use another solution, like the one you're currently using.

Note that, if it's the creation of the temporary collection that bothers you, you shouldn't be bothered. No copy is made. The collection is simply a view over the original list.

Read full article from java - Transform and convert a List to Set with Guava - Stack Overflow

Groovy Goodness: Lazy Initialization of Properties - Messages from mrhaki

Sometimes we want the value of a property only to be calculated on first use. The calculation of the value can be time consuming or requires a lot of resources. In Groovy we can use the @Lazy transformation to define a property we want to be initialized lazily. The value is not calculated or assigned until the value is requested. The transformation also allows a soft parameter. This defines the field as soft reference.

Read full article from Groovy Goodness: Lazy Initialization of Properties - Messages from mrhaki

Guava Lazy Init - Above, Below, and Beyond Tech Talk

Note to self: How to lazily initialize a value with Java/Guava

Read full article from Guava Lazy Init - Above, Below, and Beyond Tech Talk

Microservice Registration and Discovery with Spring Cloud and Netflix's Eureka

The microservice style of architecture is not so much about building individual services so much as it is making the interactions between services reliable and failure-tolerant. While the focus on these interactions is new, the need for that focus is not. We've long known that services don't operate in a vacuum. Even before cloud economics, we knew that - in a practical world - clients should be designed to be immune to service outages. The cloud makes it easy to think of capacity as ephemeral, fluid. The burden is on the client to manage this intrinsic complexity.

Read full article from Microservice Registration and Discovery with Spring Cloud and Netflix's Eureka

Press Ctrl+Shift+V to Paste Text Without Formatting in Google Chrome

Text pasted into your browser, when using a rich text editor like the one found in Gmail, automatically takes on the formatting of the source text. Google Chrome now supports text stripping via Ctrl+Shift+V (Cmd+Shift+V on Mac), your pasted text is format-free.

Read full article from Press Ctrl+Shift+V to Paste Text Without Formatting in Google Chrome

Create documents with plain text formatting in Google Docs - Web Applications Stack Exchange

If you are using Chrome, you can use the keyboard shortcut Ctrl + Shift + V to paste the unformatted version of the copied text into Google Docs (or any other web page).

Read full article from Create documents with plain text formatting in Google Docs - Web Applications Stack Exchange

CAP Theorem: Revisited

Consistency - A read is guaranteed to return the most recent write for a given client.

Availability - A non-failing node will return a reasonable response within a reasonable amount of time (no error or timeout).

Partition Tolerance - The system will continue to function when network partitions occur.

Read full article from CAP Theorem: Revisited

(8) What is the relation between SQL, NoSQL, the CAP theorem and ACID? - Quora

The CAP Theorem

Published by Eric Brewer in 2000, the theorem is a set of basic requirements that describe any distributed system (not just storage/database systems).

Let's imagine a distributed database system with multiple servers being used by an outside person requesting information. Here's how the CAP theorem applies:

Read full article from (8) What is the relation between SQL, NoSQL, the CAP theorem and ACID? - Quora

Eureka! Why You Shouldn't Use ZooKeeper for Service Discovery - N choose K / Knew Tech Blog

In CAP terms, ZooKeeper is CP, meaning that it's consistent in the face of partitions, not available. For many things that ZooKeeper does, this is a necessary trade-off. Since ZooKeeper is first and foremost a coordination service, having an eventually consistent design (being AP) would be a horrible design decision. Its core consensus algorithm, Zab, is therefore all about consistency.

Read full article from Eureka! Why You Shouldn't Use ZooKeeper for Service Discovery - N choose K / Knew Tech Blog

How do I drop a table and reclaim the space in Cassandra 2.0? - Stack Overflow

The truncate is to attempt to avoid DROP/RECREATE issues and zombie data coming back. In any case once you've dropped a keyspace it's best not to reuse that name. A fix for DROP/RECREATE is coming in 2.1.

Read full article from How do I drop a table and reclaim the space in Cassandra 2.0? - Stack Overflow

Deleting snapshot files | DataStax Cassandra 1.2 Documentation

The nodetool clearsnapshot command removes all existing snapshot files from the snapshot directory of each keyspace. You should make it part of your back-up process to clear old snapshots before taking a new one.

Read full article from Deleting snapshot files | DataStax Cassandra 1.2 Documentation

See the Size of All Your Folders in OS X's Finder

If you get annoyed as I do that sometimes you can't see the complete size of a folder in Finder, here is the way to do it: Switch to list view (View > as List); right click on an empty space and open up Show View Options; check "Calculate all sizes." And voila, it will now detect the complete size.

Read full article from See the Size of All Your Folders in OS X's Finder

TN1 - Apache Curator - Apache Software Foundation

When your watcher is called, it should return as quickly as possible. All ZooKeeper watchers are serialized - processed by a single thread. Thus, no other watchers can be processed while your watcher is running. For example, a Curator user had a watcher handler something like this:

Read full article from TN1 - Apache Curator - Apache Software Foundation

Uncovering mysteries of InputFormat: Providing better control for your Map Reduce execution.

Data split is a fundamental concept in Hadoop Map Reduce framework which defines both the size of individual Map tasks and its potential execution server. The Record Reader is responsible for actual reading records from the input file and submitting them (as key/value pairs) to the mapper. There are quite a few publications on how to implement a custom Record Reader (see, for example, [1]), but the information on splits is very sketchy. Here we will explain what a split is and how to implement custom splits for specific purposes.

Read full article from Uncovering mysteries of InputFormat: Providing better control for your Map Reduce execution.

(8) How does Hadoop handle split input records? - Quora

Hadoop will use your RecordReader and InputFormat to figure out how to 1. create splits and 2. parse data within each split into records (or K/V objects) that can be passed to the mapper. If an InputSplit (which you get to create in your input format) doesn't map exactly to an HDFS block, Hadoop's FileInputFormat (and people that extend it) will Do The Right Thing(tm) by performing a partial network read to complete the record using the first few bytes from the next block. If you want to see the gory details, the source to TextInputFormat (which in turn extends FileInputFormat) is where all the logic lives. You almost certainly want to extend FileInputFormat so you get this behavior for free.

Read full article from (8) How does Hadoop handle split input records? - Quora

8 commands to check cpu information on Linux

The cpu information includes details about the processor, like the architecture, vendor name, model, number of cores, speed of each core etc. There are quite a few commands on linux to get those details about the cpu hardware, and here is a brief about some of the commands.

Read full article from 8 commands to check cpu information on Linux

[SPARK-3466] Limit size of results that a driver collects for each action - ASF JIRA

Right now, operations like collect() and take() can crash the driver with an OOM if they bring back too many data. We should add a spark.driver.maxResultSize setting (or something like that) that will make the driver abort a job if its result is too big. We can set it to some fraction of the driver's memory by default, or to something like 100 MB.

Read full article from [SPARK-3466] Limit size of results that a driver collects for each action - ASF JIRA

Of Algebirds, Monoids, Monads, and other Bestiary for Large-Scale Data Analytics - Michael G. Noll

Have you ever asked yourself what monoids and monads are, and particularly why they seem to be so attractive in the field of large-scale data processing? Twitter recently open-sourced Algebird, which provides you with a JVM library to work with such algebraic data structures. Algebird is already being used in Big Data tools such as Scalding and SummingBird, which means you can use Algebird as a mechanism to plug your own data structures – e.g. Bloom filters, HyperLogLog – directly into large-scale data processing platforms such as Hadoop and Storm. In this post I will show you how to get started with Algebird, introduce you to monoids and monads, and address the question why you should get interested in those in the first place.

SimHash | aNeurone

Google has patented his "SimHash" algorithm. Here I provide the Java source codes and an online demo for study purpose. Please leave comment if you find bugs.

Read full article from SimHash | aNeurone

Redis new data structure: the HyperLogLog - Antirez weblog

Long story short this is what HyperLogLog does: it hashes every new element you observe. Part of the hash is used to index a register (the coin+paper pair, in our previous example. Basically we are splitting the original set into m subsets). The other part of the hash is used to count the longest run of leading zeroes in the hash (our run of heads). The probability of a run of N+1 zeroes is half the probability of a run of length N, so observing the value of the different registers, that are set to the maximum run of zeroes observed so far for a given subset, HyperLogLog is able to provide a very good approximated cardinality.

Read full article from Redis new data structure: the HyperLogLog - Antirez weblog

Shuffling in Spark vs Hadoop MapReduce | Wei Shung Chung

"…a sort-based shuffle implementation that takes advantage of an Ordering for keys (or just sorts by hashcode for keys that don't have it) would likely improve performance and memory usage in very large shuffles. Our current hash-based shuffle needs an open file for each reduce task, which can fill up a lot of memory for compression buffers and cause inefficient IO. This would avoid both of those issues."

Read full article from Shuffling in Spark vs Hadoop MapReduce | Wei Shung Chung

(8) Divide and conquer is faster? (proof by example needed) - Quora

Quicksort. In this case the method divides the problem into two smaller ones, and so forth. When the data are randomly distributed, the problem is divided into two similarly sized chunks. In this case, the division is approximately half the size of the whole chunk and the algorithm exploits locality adequately. When data are pre-sorted, the chunks are not of equal size and the algorithm does not exploit locality at all.

Read full article from (8) Divide and conquer is faster? (proof by example needed) - Quora

coderwall.com : establishing geek cred since 1305712800

Considering that you already have rootfs write access and and have switched on your chromebook to dev mode, it is possible to install NodeJS dev env on chromebook (keeping your OS to ChromeOS. This article is not about ChrUbuntu)

Read full article from coderwall.com : establishing geek cred since 1305712800

Data Structures and Algorithm: Find maximum and minimum

Given an array of integers, find the maximum and minimum of the array.

Read full article from Data Structures and Algorithm: Find maximum and minimum

database - Row count of a column family in Cassandra - Stack Overflow

If you are working on a large data set and are okay with a pretty good approximation, I highly recommend using the command:

nodetool --host <hostname> cfstats

Read full article from database - Row count of a column family in Cassandra - Stack Overflow

shell - How do I grep for multiple patterns? - Unix & Linux Stack Exchange

The portable way is to use the newer syntax, extended regular expressions. You need to pass the -E option to grep to select it. On Linux, you can also type egrep instead of grep -E (on other unices, you can make that an alias).

grep -E 'foo|bar' *.txt

Another possibility when you're just looking for any of several patterns (as opposed to building a complex pattern using disjunction) is to pass multiple patterns to grep. You can do this by preceding each pattern with the -e option.

grep -e foo -e bar *.txt

Read full article from shell - How do I grep for multiple patterns? - Unix & Linux Stack Exchange

Counting keys in Cassandra | Richard Low's blog

Today a colleague asked me: how can I find out how many keys I just inserted into Cassandra? You'd expect any half-decent database to be able to tell you how much stuff it has got. Cassandra being (somewhat better than) half-decent can, but there are many subtleties that are worth understanding.

Read full article from Counting keys in Cassandra | Richard Low's blog

OSX Tips - Turn off sleep mode from the command line | DGKApps - Blog

OSX Tips – Turn off sleep mode from the command line | DGKApps – Blog

If you’ve ever needed to turn off sleep mode in Mac OSX 10.8 (Mountain Lion) from the command line, here’s how you do it:

sudo pmset sleep 0

Read full article from OSX Tips – Turn off sleep mode from the command line | DGKApps – Blog

How to Transfer a Local MySQL Database to Another Server | DeveloperSide.NET

These instructions will show you how to export a single database from MySQL into an SQL file, that you can move/copy to the remote Host, and import it into that Host's MySQL server. The local and remote Hosts can be either Windows or Linux (it will work identically).

Read full article from How to Transfer a Local MySQL Database to Another Server | DeveloperSide.NET

Install MySQL on Mac OSX using Homebrew | Joe Fallon's Blog

Recently, while doing some development on my Mac, I realized I didn't have MySQL installed. I could have loaded up an instance of Ubuntu 12.04 LTS on VirtualBox and used that. However, I thought it would be much more convenient to have it available directly instead in a virtualized environment. Here are the instructions for installing it on a Mac using Homebrew.

Read full article from Install MySQL on Mac OSX using Homebrew | Joe Fallon's Blog

java - A ResourcePool could not acquire a resource from its primary factory or source - Stack Overflow

For anyone that finds this question in the future. What I was doing wrong was that I was using the jtds driver and I forgot to add that in the url. So in my properties file what I should have done was:

Read full article from java - A ResourcePool could not acquire a resource from its primary factory or source - Stack Overflow

java - @Async does not work with task:executor - Stack Overflow

So, the problem was my maven-aspectj-plugin. I found the solution here. All I need to do is to add mode="aspectj" to the task:annotation-driven.

Read full article from java - @Async does not work with task:executor - Stack Overflow

A Sample Usage of Java’s Future and Spring’s @Async | Pam's Blog

A Sample Usage of Java's Future and Spring's @Async | Pam's Blog

Yesterday, a colleague asked about Java Threading. We have this sending-of-files-via-FTP requirement that is already working. Our users wanted us to add functionality that will allow them to do the sending asynchronously. The initial plan was to put the sending of the files in another Thread manually. This means that we have to create a Runnable implementation and manage the Threads ourselves.

Read full article from A Sample Usage of Java's Future and Spring's @Async | Pam's Blog

Spring 3.2 Rundown, Async Support | Not Purely Technical

In a normal web application, the client (browser) sends HTTP requests to the server, which processes information and delivers an HTTP response. While waiting for the response, the browser normally pauses and indicates that it is waiting for a response. To alleviate the client becoming "unresponsive", this is nowadays overcome mostly by sending the request as an AJAX request. This solves the client's problems to a great degree.

Read full article from Spring 3.2 Rundown, Async Support | Not Purely Technical

IT Happens Here!: How To Use @Async in Spring Web Application?

I decided to give the @Async annotation a try! The recently added @Async annotation in the Spring Framework 3.0 seems to be an easy fit for applications that would require asynchrous calls to be invoked while the application is running or deployed! Here is a sample code that can help you with making your @Async annotation work with ease!

Read full article from IT Happens Here!: How To Use @Async in Spring Web Application?

Kim Saabye Pedersen: Spring @Async

I'll give a short demo a Spring's @Async annotation. Observe the following class (which is a Spring component):

Read full article from Kim Saabye Pedersen: Spring @Async

Spring XML file configuration hierarchy help/explanation - Stack Overflow

It still depends somewhat on exactly which config files you were including in which ApplicationContexts, but at the very least I can say that by doing this, you removed a lot of beans from the DispatcherServlet's context which were causing problems. In particular, your correctly-configured @Transactional beans in the root context would no longer be shadowed by beans in the child context and would be injected into your controllers, so your persistence stuff would work then.

So... the main thing to take away is that you have two related ApplicationContexts. You have to remain aware of that fact and stay in control of which beans go in which context.

Read full article from Spring XML file configuration hierarchy help/explanation - Stack Overflow

java - Spring @Async Not Working - Stack Overflow

What this means is that if you had a static method Foo.bar(), calling it in that manner wouldn't execute it in async, even if it was annotated with @Async. You'll have to annotate Foo with @Component, and in the calling class get an @Autowired instance of Foo.

Read full article from java - Spring @Async Not Working - Stack Overflow

jsinghfoss $> ls - #SpringFramework 3.1 - @Async Not Working [SOLVED]

Solution - Is to make sure the context loading (through component scan) is not clashing/or overridden in the child context (i.e. servlet-context.xml). We need to make sure we exclude the file by using exclude-filter (type - regex) in the <context-component-scan> element in the servlet-context.xml file.

Read full article from jsinghfoss $> ls - #SpringFramework 3.1 - @Async Not Working [SOLVED]

Spring @Async Method not Working << KGASOFT - SOFTWARE DEVELOPMENT BLOG

Spring @Async Method not Working « KGASOFT – SOFTWARE DEVELOPMENT BLOG

Posted on Interface: Interface Java } } To demonstrate the issue, we use the following unit test. The first testcase calls the start() method, which in turns calls the doAsync() method of the same class. This does not work as expected and a new thread is NOT created when the doAsync() method is called. The second test case, demonstrates the correct way to do it. Call the doAsync() directly from another class. Unit Test: Unit Test October 6, 2013 Website Comment You may use these HTML tags and attributes:

Read full article from Spring @Async Method not Working « KGASOFT – SOFTWARE DEVELOPMENT BLOG

java - Spring 3.x - @Async methods not called concurrently by task executor - Stack Overflow

The issue is that you're calling the methods internally so they're not proxied. For the @Async to work you need to retrieve the object from your application context and invoke the methods on the retrieved copy. Invoking it internally will not work.

The same thing happens if you try to invoke internal @Transactional methods. See the Note: section at the end of the Spring docs explaining @Transactional for details.

Additionally, the way you're immediately calling .get() on the Future return values is incorrect. If you want them to happen in parallel you should submit all the tasks then retrieve each one via .get().

Read full article from java - Spring 3.x - @Async methods not called concurrently by task executor - Stack Overflow

cassandra - SELECT COUNT(*) return 0 but I have 800 rows - Stack Overflow

Read full article from cassandra - SELECT COUNT(*) return 0 but I have 800 rows - Stack Overflow

Christopher Batey's Blog: Cassandra: Datastax Java driver retry policy

The DefaultRetryPolicy retries with the following behaviour:

Read timeout: When enough replica are available but the data did not come back within the configured read time out
Write timeout: Only if the initial phase of a batch write times out - see cassandra batch statement
Unavailable timeout: Never

Read full article from Christopher Batey's Blog: Cassandra: Datastax Java driver retry policy

Command-line Tool to find Java Heap Size and Memory Used (Linux)? - Stack Overflow

Each Java process has a pid, which you first need to find with the jps command.

Once you have the pid, you can use jstat -gc [insert-pid-here] to find statistics of the behavior of the garbage collected heap. jstat -gccapacity [insert-pid-here] will present information about memory pool generation and space capabilities.

Read full article from Command-line Tool to find Java Heap Size and Memory Used (Linux)? - Stack Overflow

Calling java methods asynchronous using spring | Level Up Lunch

Lets add @EnableAsync to our Application, which is an annotation that turns on springs async method execution. This annotation has similar proxying behaviors like @Cacheable. One thing to note, if you are calling multiple methods wrapped with @Async within the same class you will need to refactor the method into a new class due to the proxying behaviors.

Read full article from Calling java methods asynchronous using spring | Level Up Lunch

linux - Test if a port on a remote system is reachable (without telnet) - Super User

Nice and verbose! From the man pages.
Single port:

nc -zv 127.0.0.1 80

Multiple ports:

nc -zv 127.0.0.1 22 80 8080

Range of ports:

nc -zv 127.0.0.1 20-30

Read full article from linux - Test if a port on a remote system is reachable (without telnet) - Super User

Maximum URL length is 2,083 characters in Internet Explorer

Microsoft Internet Explorer has a maximum uniform resource locator (URL) length of 2,083 characters. Internet Explorer also has a maximum path length of 2,048 characters. This limit applies to both POST request and GET request URLs.

Read full article from Maximum URL length is 2,083 characters in Internet Explorer

java - Spring Cache Abstraction with multi-value queries - Stack Overflow

The query cache can indeed cache a list of results per query input. Be aware thought, that only the IDs of the returned entities will be saved in the query cache. You have to have the entity cache enabled separately for the returned entity type itself, if you want the properties to be cached too.

Read full article from java - Spring Cache Abstraction with multi-value queries - Stack Overflow

java - Mockito: List Matchers with generics - Stack Overflow

when(mock.process(Matchers.anyListOf(Bar.class)));

Read full article from java - Mockito: List Matchers with generics - Stack Overflow

WordCount example in Java 8 on Spark -- Medium

WordCount example in Java 8 on Spark — Medium

Since Spark 1.0 will release soon, I quick love the Java 8 on Spark. It is more convenient for our team and it is readable than scala version. I never try to understand the scala, and it is not the right time. I need Java 8's good language and some functional language feature.

Read full article from WordCount example in Java 8 on Spark — Medium

A Reluctant Web Developer : Postman Chrome App

Run Postman collections in the command line

Run a collection directly from a URL

Switch environments within Newman

Generate output files to be imported on Postman

Integrate with build tools using the -stopOnError flag

Read full article from A Reluctant Web Developer : Postman Chrome App

Removing all duplicates from a List in Java | Baeldung

This quick tutorial is going to show you how to clean up the duplicate elements from a List – first using plain Java, then Guava and finally a Java 8 Lambda based solution.

Read full article from Removing all duplicates from a List in Java | Baeldung

java - Remove duplicates from List using Guava - Stack Overflow

I love Louis' answer for the simplicity of it (and because it's the only answer that doesn't require 2 full iterations), but unfortunately in the real world, you often encounter situations where null does occur. Here's a slightly longer null-safe version:

Read full article from java - Remove duplicates from List using Guava - Stack Overflow

java - How to make a new list with a property of an object which is in another list - Stack Overflow

ists.transform(studentList, StudentToId.INSTANCE);

Surely it will loop in order to extract all ids, but remember guava methods returns view and Function will only be applied when you try to iterate over the List<Integer>
If you don't iterate, it will never apply the loop.

Read full article from java - How to make a new list with a property of an object which is in another list - Stack Overflow

Inspired by Actual Events: Guava Stopwatch

I have already alluded to several caveats to keep in mind when using Guava's Stopwatch. First, two successive start() methods should not be invoked on a given instance of Stopwatch without first stopping it with stop() before making the second call to stop(). Stopwatch has an instance method isRunning() that is useful for detecting a running stopwatch before trying to start it again or even before trying to stop one that has already been stopped or was never started. Most of these issues such as starting the stopwatch twice without stopping it or stopping a stopwatch that is not running or was never started lead to IllegalStateExceptions being thrown. Guava developers make use of their own Preconditions class to ascertain these aberrant conditions and to the throwing of these exceptions. The further implication of this, which is spelled out in the Javadoc documentation, is that Stopwatch is not thread-safe and should be used in a single-thread environment.

Read full article from Inspired by Actual Events: Guava Stopwatch

collections - Java: how to transform from List to Map without iterating - Stack Overflow

Guava has Maps.uniqueIndex(Iterable values, Function keyFunction) and Multimaps.index(Iterable values, Function keyFunction), but they don't transform the values. There are some requests to add utility methods that do what you want, but for now, you'll have to roll it yourself using Multimaps.index() and Multimaps.transformValues():

Read full article from collections - Java: how to transform from List to Map without iterating - Stack Overflow

Apache Spark User List - Visualizing the DAG of a Spark application

We are looking for a tool that would let us visualize the DAG generated by a Spark application as a simple graph.
This graph would represent the Spark Job, its stages and the tasks inside the stages, with the dependencies between them (either narrow or shuffle dependencies).

Read full article from Apache Spark User List - Visualizing the DAG of a Spark application

[SPARK-1103] [WIP] Automatic garbage collection of RDD, shuffle and broadcast data by tdas · Pull Request #126 · apache/spark · GitHub

This PR allows Spark to automatically cleanup metadata and data related to persisted RDDs, shuffles and broadcast variables when the corresponding RDDs, shuffles and broadcast variables fall out of scope from the driver program. This is still a work in progress as broadcast cleanup has not been implemented.

Read full article from [SPARK-1103] [WIP] Automatic garbage collection of RDD, shuffle and broadcast data by tdas · Pull Request #126 · apache/spark · GitHub

spark/programming-guide.md at master · apache/spark · GitHub

At a high level, every Spark application consists of a driver program that runs the user's main function and executes various parallel operations on a cluster. The main abstraction Spark provides is a resilient distributed dataset (RDD), which is a collection of elements partitioned across the nodes of the cluster that can be operated on in parallel. RDDs are created by starting with a file in the Hadoop file system (or any other Hadoop-supported file system), or an existing Scala collection in the driver program, and transforming it. Users may also ask Spark to persist an RDD in memory, allowing it to be reused efficiently across parallel operations. Finally, RDDs automatically recover from node failures.

Read full article from spark/programming-guide.md at master · apache/spark · GitHub

[SPARK-732] Recomputation of RDDs may result in duplicated accumulator updates - ASF JIRA

Currently, Spark doesn't guard against duplicated updates to the same accumulator due to recomputations of an RDD. For example:

Read full article from [SPARK-732] Recomputation of RDDs may result in duplicated accumulator updates - ASF JIRA

Stream groupingBy | Level Up Lunch

In java 8 the idea of grouping objects in a collection based on the values of one or more of their properties is simplified by using a Collector. A collector is another new interface introduced in Java 8 for defining how to perform a reduction operation on a stream and with the functional nature allows you to achieve a grouping with a single statment. It might even spark debates with your DBA for control or their lack of understanding. Should it happen in code or in the database? In the example below, we will peform a number of group by statements to show the simplicity and flexibility of the interface.

Read full article from Stream groupingBy | Level Up Lunch

The CAP Theorem

Labels

Popular Posts