[SPARK-3466] Limit size of results that a driver collects for each action - ASF JIRA

[SPARK-3466] Limit size of results that a driver collects for each action - ASF JIRA

Right now, operations like collect() and take() can crash the driver with an OOM if they bring back too many data. We should add a spark.driver.maxResultSize setting (or something like that) that will make the driver abort a job if its result is too big. We can set it to some fraction of the driver's memory by default, or to something like 100 MB.

Read full article from [SPARK-3466] Limit size of results that a driver collects for each action - ASF JIRA