PySpark for Pandas users: getting past the RAM wall
Pandas keeps the whole dataset in RAM and computes as you type. PySpark builds a plan first and runs it across a cluster. Here is what that changes for loading data, window functions, and the shuffle settings nobody tunes.