Spark FAQ sorting - must see before interview

One 、job、stage、Task What is the relationship between ？

• One job It can contain more than one stage

• One stage Contains multiple task

Two 、job、stage、Task What is the relationship between ？

• Every time a task is submitted , It creates a job, That is to call action Operator will create job【 When the operator is called, the return value is not RDD Type can be classified as Action operator 】
• Divide according to wide dependence and narrow dependence stage, If it's broad dependence , Just add a new one stage
• Task The number is actually the number of partitions

3、 ... and 、 What is wide dependence 、 Narrow dependence ？

• If a father RDD The partition is divided into several subdomains RDD The use of , It's just wide dependence 【 Superbirth 】 If a father RDD The partition is divided into several subdomains RDD The use of , It's just wide dependence 【 Superbirth 】
• If a father RDD The partition is only used by one child RDD Partition use , It's narrow dependence 【 only 】 If a father RDD The partition is only used by one child RDD Partition use , It's narrow dependence 【 only 】
• abstract class Dependencyabstract class Dependency
  • abstract class NarrowDependency extend Dependency There is an abstract way ,getParents（）
    • class OneToOneDependency extend NarrowDependency Implement abstract methods getParents（）
    • class RangeDependency extend NarrowDependency Implement abstract methods getParents（）class RangeDependency extend NarrowDependency Implement abstract methods getParents（）
  • class ShuffleDependency extend Dependencyclass ShuffleDependency extend Dependency

Four 、Action Operator and Transformation What is an operator , List some ？

• Action The operator will create job, Will be executed immediately . for example ：take ,first,collect,foreach,foreachPartition.

• Transformation Not immediately , But there will be some dependencies recorded , And functions . for example ：map,filter,flatMap,reduceByKey,groupByKey wait .

5、 ... and 、reduceByKey and groupByKey What's the difference? ？

reduceByKey：reduceByKey The results will be sent to reducer I've been talking to everyone before mapper In Ben
To carry out merge, It's a bit like in MapReduce Medium combiner. The advantage of doing so is ,
stay map Do it once reduce after , The amount of data will be greatly reduced , This reduces transmission , Guarantee reduce
The end can calculate the result faster .

groupByKey：groupByKey For each RDD Medium value Values are aggregated to form a sequence
(Iterator), This operation occurred at reduce End , Therefore, it is bound to transmit all data through the network ,
Cause unnecessary waste . At the same time, if the amount of data is very large , It may also cause OutOfMemoryError.

Conclusion ：

So we're doing a lot of data reduce It is recommended to use reduceByKey. It can not only raise the speed
degree , It can also prevent the use of groupByKey Memory overflow caused by .

6、 ... and 、RDD Five attributes

7、 ... and 、Spark The architecture and job submission process of