Azure Daily 2022 by Gordon

Generally available: Apache Spark 3.1.2 in Azure HDinsight

The latest release of Apache Spark 3.1 includes:

  • Adaptive query execution
  • Convert sort merge join to broadcast hash join
  • Spark catalyst optimizer
  • Dynamic partition pruning
  • Create new Spark 3.1 clusters (not Spark 3.0 clusters (preview))

For additional details, review the document Apache Spark 3.1 on Microsoft Tech Community.

For a complete list of improvements, review the Apache Spark 3.1 release notes.

For more details on migration, review the migration guide.

Add to Set
Add to Set
Share
Share on Twitter Share on Facebook Share on LinkedIn Share on Reddit Share on WhatsApp Send via Email Copy Link to Clipboard
Referenced in:

Comments

Copyright © Snippset 2022. All Rights Reserved.