HQL statement execution process

HQL statement execution process:

1. Syntax parsing: Antlr defines the grammatical rules of SQL, completes SQL lexical, grammatical parsing, and converts SQL into an abstract syntax tree AST Tree;
2. Semantic parsing: traverse the AST Tree and abstract the basic unit of query QueryBlock;
3. Generate a logical execution plan: traverse QueryBlock and translate it into an execution operation tree OperatorTree;
4. Optimize the logic execution plan: The logic layer optimizer performs OperatorTree transformation, merges unnecessary ReduceSinkOperators, and reduces the amount of shuffle data;
5. Generate physical execution plan: traverse OperatorTree and translate into MapReduce tasks;
6. Optimize the physical execution plan: The physical layer optimizer transforms MapReduce tasks to generate the final execution plan.

How is HQL resolved into MR job?

1. Hive uses Antlr to implement syntax parsing. According to the SQL parsing rules formulated by Antlr, complete the lexical/syntax parsing of SQL statements, and convert SQL into abstract syntax tree AST.

2. Traverse the AST to generate the basic query unit QueryBlock. QueryBlock is the most basic unit of SQL, including three parts: input source, calculation process, and output.

3. Traverse QueryBlock to generate OperatorTree. The MapReduce task finally generated by Hive consists of OperatorTree in both Map and Reduce phases.Operator is to complete a single specific operation in the Map phase or the Reduce phase.QueryBlock generates Operator Tree by traversing the attributes of the saved syntax of the QB and QBParseInfo objects generated in the previous process.

4. **Optimize OperatorTree.**Most logic layer optimizers achieve the purpose of reducing MapReduce Job and shuffle data volume by transforming OperatorTree and merging operators

5. OperatorTree generates MapReduce Job. Traverse OperatorTree and translate into MR tasks.

   • Generate MoveTask for output table
   • Depth-first traversal from one of the root nodes of OperatorTree down
   • ReduceSinkOperator marks the boundaries of Map/Reduce, the boundaries between multiple jobs
   • Traverse other root nodes and encounter JoinOperator to merge MapReduceTask
   • Generate StatTask update metadata
   • Sever the operator relationship between Map and Reduce

6. Optimize the task. Use the physical optimizer to optimize the MR task to generate the final execution task

[HIVE] sqlStatement converted into mapreduce - Mr.Ming2 - Blog Park