当前位置:网站首页>flink on yarn集群部署
flink on yarn集群部署
2022-08-11 11:53:00 【gamespiay123】
下载flink
wget https://www.apache.org/dyn/closer.lua/flink/flink-1.14.5/flink-1.14.5-bin-scala_2.12.tgz
eg: hadoop-master-01(NN)
hadoop-slave-01(DN)
hadoop-slave-02(DN)
在hadoop-master-01服务器,当前目录/home/bigdata (zookeeper服务器 hadoop-zookeeper-01,hadoop-zookeeper-02,hadoop-zookeeper-03)
cd /home/bigdata
tar –zxvf flink-1.14.5-bin-scala_2.12.tgz
mv flink-1.14.5-bin-scala_2.12 flink
cd conf配置以下
修改flink-conf.yaml文件
jobmanager.rpc.address: hadoop-master-01
jobmanager.rpc.port: 6123
jobmanager.memory.process.size: 1600m
taskmanager.memory.process.size: 4096m
taskmanager.memory.flink.size: 2048m
taskmanager.memory.jvm-metaspace.size: 1024m
taskmanager.numberOfTaskSlots: 4
parallelism.default: 1
high-availability: zookeeper
high-availability.zookeeper.quorum: hadoop-zookeeper-01:2181,hadoop-zookeeper-02:2181,hadoop-zookeeper-03:2181
state.backend: filesystem
state.checkpoints.dir: hdfs://hadoop-master-01:9000/flink/flink-checkpoints
state.savepoints.dir: hdfs://hadoop-master-01:9000/flink/flink-savepoints
jobmanager.execution.failover-strategy: region
rest.port: 8081
rest.address: hadoop-master-01
web.submit.enable: false
io.tmp.dirs: /home/bigdata/flink/tmp
taskmanager.memory.network.fraction: 0.1
taskmanager.memory.network.min: 64mb
taskmanager.memory.network.max: 1gb
jobmanager.archive.fs.dir: hdfs://hadoop-master-01:9000/flink/completed-jobs/
historyserver.web.address: 0.0.0.0
historyserver.web.port: 8082
historyserver.archive.fs.dir: hdfs://hadoop-master-01:9000/flink/completed-jobs/
historyserver.archive.fs.refresh-interval: 10000
修改masters文件
hadoop-master-01:8081
修改workers文件(三台服务器都可以做works执行任务)
hadoop-master-01
hadoop-slave-01
hadoop-slave-02
scp -rp /home/bigdata/flink [email protected]:/home/bigdata/flink/
scp -rp /home/bigdata/flink [email protected]:/home/bigdata/flink/
cd /home/bigdata/flink/bin
启动flink集群
./start-cluster.sh
hadoop-master-01执行jps可以看到
ResourceManager (yarn)
TaskManagerRunner (flink)
StandaloneSessionClusterEntrypoint (flink)
NameNode (hadoop)
SecondaryNameNode (hadoop)
hadoop-slave-01和hadoop-slave-02执行jps可以看到
TaskManagerRunner (flink)
NodeManager (yarn)
DataNode (hadoop)
测试集群效果
cd /home/bigdata/flink/examples/streaming
flink run -m yarn-cluster -yjm 4096 -ytm 4096 WordCount.jar --output hdfs://hadoop-master-01:9000/flink/test/out.txt
hadoop fs -cat hdfs://hadoop-master-01:9000/flink/test/out.txt
边栏推荐
- 13-盒子模型
- SQL Runtime SLX中的优化设计有哪些?
- EastWave Application: Live Demonstration of Negative Refraction Phenomenon
- 云原生(三十四) | Kubernetes篇之平台存储系统实战
- TX12 + ExpressLRS RC configuration and control link problem summary 915 MHZ
- 锂电池充电芯片IC
- [10 o'clock open class]: Optimization of AV1 encoder and its application in streaming media and real-time communication
- flutter面试题收集
- 金九银十面试复习回顾及总结:算法+框架+Redis+分布式+JVM
- RTP协议浅析
猜你喜欢
随机推荐
黑海港口外运农产品问题联合协调中心:正努力加速粮食外运
22-网格布局
seata相关图形,dljd,cat
为什么最好的光刻机来自荷兰,而不是芯片大国美国?
【深度学习】小结1-入门两周学习感受
公共经济学(开卷)期末复习题
13-盒子模型
【ManageEngine】网络带宽管理工具
路测数据合并
【剑指offer-牛客网刷题】第一篇-斐波拉契数列-C实现
面试官:Redis Zset的实现为什么用跳表,而不用平衡树?
堆排序 O(nlogn) ----选择排序
MySQL之JDBC编程增删改查
NLP标注工具Brat的简单使用
C-V2X八大误区澄清和发展辩思
router.beforEach
集合面试题
ST表和区间最值
威佐夫博弈
【学生毕业设计】基于web学生信息管理系统网站的设计与实现(13个页面)