Preface

sortBy, Sort as the name suggests , stay Spark in , Use sortBy You can sort a set of data to be processed , This set of data is not limited to numbers , It can also be tuples and other types ;

sortBy

Function signature

def sortBy[K](

f: (T) => K , ascending: Boolean = true , numPartitions: Int = this.partitions.length)

(implicit ord: Ordering[K], ctag: ClassTag[K]): RDD[T]

Function description

This operation is used to sort the data . Before sorting , Data can be passed through f Function to process , And then according to f Function processing

To sort the results of , The default is ascending . New after sorting RDD The number of partitions is the same as the original RDD The number of partitions is one

Cause . There is... In the middle shuffle The process of ;

Case presentation

Next, sort the data in a set , Save to local file directory

import org.apache.spark.{SparkConf, SparkContext}

object SortBy_Test {

  def main(args: Array[String]): Unit = {
    val sparkConf = new SparkConf().setMaster("local[*]").setAppName("Operator")
    val sc = new SparkContext(sparkConf)

    val rdd = sc.makeRDD(List(1,2,3,4,5,6,7,9), 2)
    rdd.sortBy(num => num)
    rdd.saveAsTextFile("E:\\output")

    sc.stop()
  }

}

Run the above code , You can see that two files are generated in the local directory

  Open separately 2 File , You can find , The data is sorted in two different files

 

Put... In the set tuple Data according to key Sort the output

import org.apache.spark.{SparkConf, SparkContext}

object SortBy_Test {

  def main(args: Array[String]): Unit = {
    val sparkConf = new SparkConf().setMaster("local[*]").setAppName("Operator")
    val sc = new SparkContext(sparkConf)

    var rddStr = sc.makeRDD(List(
      ("a",3),("d",2),("e",7)
    ),2)

    rddStr.sortBy(t => t._1)
    rddStr.collect().foreach(println)

    sc.stop()
  }

}

Run the above code , Observe the console output