Spark cluster environment construction

1. Install jdk

Installation environment: CentOS-7.0.1708 Installation method: Source code installation software: jdk-6u45-linux-x64.bin Download address: http://www.Oracle.com/technetwork/Java/javase/downloads/java-archive-downloads-javase6-419409.html

The first step: change the permission

chmod 775 jdk-6u45-linux-x64.bin

Step 2: Execute jdk installation

./jdk-6u45-linux-x64.bin

Step 3: Configure environment variables

The configuration of environment variables is divided into several ways, according to your choice:

Method 1: configure JAVA_HOME and PATH and CLASS_PATH in the vi /etc/profile file

Because this setting will take effect on the shell of all users, it will have an impact on system security.

Add at the end of this file:

export JAVA_HOME=/usr/local/softWare/jdk1.6.0_45

export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

export PATH=$PATH:$JAVA_HOME/bin

Executing source /etc/profile is that the configuration takes effect immediately

Method 2:

Modify the .bashrc file to configure environment variables:

#vi.bashrc

export JAVA_HOME=/usr/local/softWare/java/jdk1.6.0_45

export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

export PATH=$PATH:$JAVA_HOME/bin

After the configuration is complete, use the logout command to log out, and then log in again to make it take effect.

Verify if the installation was successful, use java -version to check.

2. Install scala download path: https://downloads.lightbend.com/scala/2.12.8/scala-2.12.8.tgz scala-2.12.8.tgz Upload and decompress the download package tar -zxvf scala-2.12.8.tgz rm -rf scala-2.12.8.tgz Configure environment variables vi /etc/profile export SCALA_HOME=/usr/local/scala-2.12.8 export PATH=$PATH:$JAVA_HOME/bin:$SCALA_HOME/bin to other nodes: scp -r scala-2.12.8 192.168.0.109:/usr/local/ scp -r scala-2.12.8 192.168.0.110:/usr/local/scp /etc/profile 192.168.0.109:/etc/ scp /etc/profile 192.168.0.110:/etc/ Make environment variables effective: source /etc/profile Verify: scala -version

3.ssh password-free login reference https://blog.51cto.com/13001751/2487972

4. Install HadoopReference https://blog.51cto.com/13001751/2487972

5. Install spark Upload and decompress the download package cd /usr/local/ tar -zxvf spark-2.4.5-bin-hadoop2.7.tgz cd /usr/local/spark-2.4.5-bin-hadoop2.7/conf/ #Enter the spark configuration directory mv spark-env.sh.template spark-env.sh #Copy from the configuration template vi spark-env.sh #Add configuration content export SPARK_HOME=/usr/local/spark-2.4.5-bin-hadoop2.7 export SCALA_HOME=/usr/local/scala-2.12.8 export JAVA_HOME=/usr/local/jdk1.8.0_191 export HADOOP_HOME=/usr/local/hadoop-2.7.7 export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$SCALA_HOME/bin export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop export SPARK_MASTER_IP=spark1 SPARK_LOCAL_DIRS=/usr/local/spark-2.4.5-bin-hadoop2.7 SPARK_DRIVER_MEMORY=1G export SPARK_LIBARY_PATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib:$HADOOP_HOME/lib/native, vi slaves spark2 spark3 scp -r /usr/local/spark-2.4.5-bin-hadoop2.7 [email protected]:/usr/local/ scp -r /usr/local/spark-2.4.5-bin-hadoop2.7 [email protected]:/usr/local/ ./sbin/start-all.sh (do not start-all.sh directly, this command is hadoop)