Preface

filter, It can be understood as filtering , Intuitive, , Is to filter a group of data according to the specified rules ,filter This operator is in Java Or in other languages , It can easily help us filter the desired data from a set of data ;

Function signature

def filter(f: T => Boolean ): RDD[T]

Function description

Filter the data according to the specified rules , Consistent data retention , Data that does not conform to the rules is discarded . When the data is filtered , The partition does not change , But the data in the partition may be uneven , In the production environment , There may be Data skew ;

Case a , Filter out even numbers from a set of data

import org.apache.spark.rdd.RDD
import org.apache.spark.{SparkConf, SparkContext}

object Filter_Test {

  def main(args: Array[String]): Unit = {
    val sparkConf = new SparkConf().setMaster("local[*]").setAppName("Operator")
    val sc = new SparkContext(sparkConf)

    val rdd  = sc.makeRDD(List(1,2,3,4,5,6))
    val result = rdd.filter(
      item => item % 2 ==0
    )
    result.collect().foreach(println)


    sc.stop()

  }

}

Run this code , Observe the console output

 

Case 2 , Filter out from log file 2015 year 5 month 17 The data of

The contents of the log file are as follows ： 

import org.apache.spark.rdd.RDD
import org.apache.spark.{SparkConf, SparkContext}

object Filter_Test {

  def main(args: Array[String]): Unit = {
    val sparkConf = new SparkConf().setMaster("local[*]").setAppName("Operator")
    val sc = new SparkContext(sparkConf)


    val rdd: RDD[String] = sc.textFile("E:\\code-self\\spi\\datas\\apache.log")
    rdd.filter(
      line =>{
        val datas = line.split(" ")
        val time = datas(3)
        time.contains("17/05/2015")
      }
    ).collect().foreach(println)

    sc.stop()

  }

}

Run the above code , Observe the console output ,