Pseudo Distributed installation spark

spark install （ Pseudo Distributed installation ）

Upload files

use xtfp Upload the file to /root/softwares

The configuration file

cd /root/softwares
# decompression 
tar -zxvf spark-2.4.0-bin-hadoop2.7.tgz -C /usr/local/
cd /usr/local/
# rename 
mv spark-2.4.0-bin-hadoop2.7/ spark-2.4.0
cd spark-2.4.0/conf/
# To configure spark-env.sh file 
cp spark-env.sh.template spark-env.sh
vim spark-env.sh
<!-- Master node IP-->
export SPARK_MASTER_HOST=192.168.239.239

<!-- Task submission port -->
export SPARK_MASTER_PORT=7077 

<!-- Every worker Use 2 nucleus , It depends on the kernel when your virtual machine is created -->
export SPARK_WORKER_CORES=2

<!-- Every worker Use 3g Memory , It depends on the memory when your virtual machine is created -->
export SPARK_WORKER_MEMORY=1g 

<!-- modify spark The default port of the monitoring window is 8080-->
export SPARK_MASTER_WEBUI_PORT=8888
# To configure slaves file 
cp slaves.template slaves
vi slaves
# Default slaves Now let's have a mainframe 
# Because it is equipped with a single machine , Not the cluster , So there's no need to Slaves( To configure worker Slave node ), Namely localhost, Don't change the settings 
# To spark-config.sh To configure jdk environment variable 
cd /usr/local/spark-2.4.0/sbin/
vim spark-config.sh 
export JAVA_HOME=/usr/local/jdk1.8.0_171 

start-up spark

To the master node spark Of sbin Run under

./start-all.sh
# Check the process 
jps
#shell operation 
cd /usr/local/spark-2.4.0/bin/
./spark-shell
# visit , Verify success 
http://192.168.239.239:4040/executors/
http://192.168.239.239:8080/