Spark Configuration - Spark 1.0.2 Documentation

Spark Configuration - Spark 1.0.2 Documentation

Spark Properties

Spark properties control most application settings and are configured separately for each application. These properties can be set directly on a SparkConf passed to your SparkContext. SparkConf allows you to configure some of the common properties (e.g. master URL and application name), as well as arbitrary key-value pairs through the set() method. For example, we could initialize an application as follows:

val conf = new SparkConf()               .setMaster("local")               .setAppName("CountingSheep")               .set("spark.executor.memory", "1g")  val sc = new SparkContext(conf)

Read full article from Spark Configuration - Spark 1.0.2 Documentation