Preface

Believed to have been used mysql Yes sql In the sentence distinct Keywords are not unfamiliar , Use distinct Keyword can be used to de duplicate the queried data , stay Spark in , You can make a similar understanding ;

Function signature

def distinct()(implicit ord: Ordering[T] = null): RDD[T]

def distinct( numPartitions: Int )(implicit ord: Ordering[T] = null): RDD[T]

Function description

Remove duplicate data from a dataset

Case study ： De duplication of a set of numbers in a set

import org.apache.spark.{SparkConf, SparkContext}

object Distinct_Test {

  def main(args: Array[String]): Unit = {
    val sparkConf = new SparkConf().setMaster("local[*]").setAppName("Operator")
    val sc = new SparkContext(sparkConf)

    val rdd  = sc.makeRDD(List(1,2,3,4,5,3,5,2,2))
    rdd.distinct().collect().foreach(println)
    sc.stop()
  }

}

Run the above program , Observe the console output , It can be found that duplicate elements are finally output only once