This PR allows Spark to automatically cleanup metadata and data related to persisted RDDs, shuffles and broadcast variables when the corresponding RDDs, shuffles and broadcast variables fall out of scope from the driver program. This is still a work in progress as broadcast cleanup has not been implemented.
Read full article from [SPARK-1103] [WIP] Automatic garbage collection of RDD, shuffle and broadcast data by tdas · Pull Request #126 · apache/spark
No comments:
Post a Comment