当前位置:网站首页>flink on yarn集群部署

flink on yarn集群部署

2022-08-11 11:53:00 gamespiay123

下载flink

wget  https://www.apache.org/dyn/closer.lua/flink/flink-1.14.5/flink-1.14.5-bin-scala_2.12.tgz

eg:   hadoop-master-01(NN)

        hadoop-slave-01(DN)

        hadoop-slave-02(DN)

在hadoop-master-01服务器,当前目录/home/bigdata (zookeeper服务器  hadoop-zookeeper-01,hadoop-zookeeper-02,hadoop-zookeeper-03)

cd /home/bigdata

tar –zxvf  flink-1.14.5-bin-scala_2.12.tgz

mv  flink-1.14.5-bin-scala_2.12  flink

cd conf配置以下

修改flink-conf.yaml文件

jobmanager.rpc.address: hadoop-master-01

jobmanager.rpc.port: 6123

jobmanager.memory.process.size: 1600m

taskmanager.memory.process.size: 4096m

taskmanager.memory.flink.size: 2048m

taskmanager.memory.jvm-metaspace.size: 1024m

taskmanager.numberOfTaskSlots: 4

parallelism.default: 1

high-availability: zookeeper

high-availability.zookeeper.quorum: hadoop-zookeeper-01:2181,hadoop-zookeeper-02:2181,hadoop-zookeeper-03:2181

state.backend: filesystem

state.checkpoints.dir: hdfs://hadoop-master-01:9000/flink/flink-checkpoints

state.savepoints.dir: hdfs://hadoop-master-01:9000/flink/flink-savepoints

jobmanager.execution.failover-strategy: region

rest.port: 8081

rest.address: hadoop-master-01

web.submit.enable: false

io.tmp.dirs: /home/bigdata/flink/tmp

taskmanager.memory.network.fraction: 0.1

taskmanager.memory.network.min: 64mb

taskmanager.memory.network.max: 1gb

jobmanager.archive.fs.dir: hdfs://hadoop-master-01:9000/flink/completed-jobs/

historyserver.web.address: 0.0.0.0

historyserver.web.port: 8082

historyserver.archive.fs.dir: hdfs://hadoop-master-01:9000/flink/completed-jobs/

historyserver.archive.fs.refresh-interval: 10000

修改masters文件

hadoop-master-01:8081

修改workers文件(三台服务器都可以做works执行任务)

hadoop-master-01

hadoop-slave-01

hadoop-slave-02

scp -rp  /home/bigdata/flink   [email protected]:/home/bigdata/flink/

scp -rp  /home/bigdata/flink   [email protected]:/home/bigdata/flink/

cd  /home/bigdata/flink/bin

启动flink集群

./start-cluster.sh

hadoop-master-01执行jps可以看到

ResourceManager     (yarn)

TaskManagerRunner (flink)

StandaloneSessionClusterEntrypoint  (flink)

NameNode  (hadoop)

SecondaryNameNode (hadoop)

hadoop-slave-01和hadoop-slave-02执行jps可以看到

TaskManagerRunner (flink)

NodeManager  (yarn)

DataNode (hadoop)

测试集群效果

cd  /home/bigdata/flink/examples/streaming

flink run -m yarn-cluster  -yjm 4096 -ytm 4096  WordCount.jar  --output  hdfs://hadoop-master-01:9000/flink/test/out.txt

hadoop fs -cat hdfs://hadoop-master-01:9000/flink/test/out.txt

原网站

版权声明
本文为[gamespiay123]所创,转载请带上原文链接,感谢
https://blog.csdn.net/gamespiay123/article/details/126270980