当前位置:网站首页>Dolphinscheduler集成Flink任务踩坑记录
Dolphinscheduler集成Flink任务踩坑记录
2022-04-23 06:03:00 【若小鱼】
1、关于Flink打包
flink任务编写完成,在本地运行调试正常后,我打包提交到Dolphinscheduler平台进行测试。运行后没多久就报错:
[taskAppId=TASK-10-108-214]:[138] - -> java.lang.NoClassDefFoundError: org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer
at com.bigdata.flink.FlinkKafka.main(FlinkKafka.java:30)
Caused by: java.lang.ClassNotFoundException: org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer
看日志详情,我们可以看到任务已经开始准备提交了,但是在环境里找不到FlinkKafkaConsumer。
flink run -m yarn-cluster -ys 1 -yjm 1G -ytm 2G -yqu default -p 1 -sae -c com.bigdata.flink.FlinkKafka flink-job/bigdata-flink.jar
我的打包方式是通过IDEA 菜单的Build Artfacts…打出的jar文件很小,而且依赖包也都没有被打进去。
于是换了一种方式:
- 配置pom文件
<build>
<!--<pluginManagement>-->
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
<version>1.8</version>
<executions>
<execution>
<id>add-source</id>
<phase>generate-sources</phase>
<goals>
<goal>add-source</goal>
</goals>
<configuration>
<sources>
<source>src/main/scala</source>
<source>src/test/scala</source>
</sources>
</configuration>
</execution>
<execution>
<id>add-test-source</id>
<phase>generate-sources</phase>
<goals>
<goal>add-test-source</goal>
</goals>
<configuration>
<sources>
<source>src/test/scala</source>
</sources>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.1.5</version>
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
<configuration>
<scalaVersion>2.11.8</scalaVersion>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.8</source>
<target>1.8</target>
<encoding>utf-8</encoding>
</configuration>
<executions>
<execution>
<phase>compile</phase>
<goals>
<goal>compile</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<configuration>
<appendAssemblyId>false</appendAssemblyId>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<archive>
<manifest>
<!-- 此处指定main方法入口的class -->
<mainClass>com.bigdata.flink.FlinkKafka</mainClass>
</manifest>
</archive>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>assembly</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
<!--</pluginManagement>-->
</build>
- 执行maven下的package命令;可以看到这种打包方式执行后的jar包里包含了所需的依赖包。
2、HADOOP_CLASSPATH环境变量配置
环境依赖的问题解决后,在执行中又出现了HADOOP_CLASSPATH环境变量的报错。
taskAppId=TASK-10-112-218]:[138] - ->
------------------------------------------------------------
The program finished with the following exception:
org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: No Executor found. Please make sure to export the HADOOP_CLASSPATH environment variable or have hadoop in your classpath. For more information refer to the "Deployment" section of the official Apache Flink documentation.
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372)
at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246)
at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054)
at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
Caused by: java.lang.IllegalStateException: No Executor found. Please make sure to export the HADOOP_CLASSPATH environment variable or have hadoop in your classpath. For more information refer to the "Deployment" section of the official Apache Flink documentation.
这次需要修改worker所在节点的flink文件。
- 找到 flink/bin/flink文件
- 在第一行加入
export HADOOP_CLASSPATH=`hadoop classpath`
- 无需重启集群机器,直接重跑flink任务即可。
版权声明
本文为[若小鱼]所创,转载请带上原文链接,感谢
https://blog.csdn.net/weixin_44162809/article/details/121563639
边栏推荐
- Kubernetes coredns FAQ reference
- Apache Atlas 编译及安装记录
- How does VirtualBox modify the IP network segment assigned to the virtual machine in the "network address translation (NAT)" network mode
- 基于ECS搭建云上博客(体验有礼)
- Exception record-8
- 如何通过dba_hist_active_sess_history分析数据库历史性能问题
- Chaos vous emmène au projet chaos.
- 异常记录-8
- try catch 不能捕获异步错误
- RAC环境数据库节点参数设置不当导致监听无法连接问题排查
猜你喜欢
Build an OSS based image sharing website - polite feedback
MySQL【sql性能分析+sql调优】
Oracle redo log产生量大的查找思路与案例
基於ECS搭建雲上博客(雲小寶碼上送祝福,免費抽iphone13任務詳解)
Chaos帶你快速上手混沌工程
Prometheus Thanos快速指南
Construire un blog Cloud basé sur ECS (bénédiction sur le Code Cloud Xiaobao, explication détaillée de la tâche iphone13 gratuite)
Static interface method calls are not supported at language level ‘5‘ 异常解决
qs. In the stringify interface, the input parameter is converted into a & connected string (with the application / x-www-form-urlencoded request header)
Dolphinscheduler调度spark任务踩坑记录
随机推荐
异常记录-6
EMR Based offline data analysis - polite feedback
Redis 详解(基础+数据类型+事务+持久化+发布订阅+主从复制+哨兵+缓存穿透、击穿、雪崩)
Dolphinscheduler调度sql任务建表时The query did not generate a result set异常解决
基於ECS搭建雲上博客(雲小寶碼上送祝福,免費抽iphone13任務詳解)
Prometheus Thanos与Cortex组件比较
Introduction to the top 12 domestic databases in 2021
prometheus告警记录持久化(历史告警保存与统计)
基于ECS搭建云上博客(体验有礼)
Virtio and Vhost_ Net introduction
Ansible basic commands, roles, built-in variables and tests judgment
Implementation of multi tenant read and write in Prometheus cortex
Thanos Compact组件测试总结(处理历史数据)
Use the SED command to process text efficiently
How to use tiup to deploy a tidb V5 0 cluster
Dolphinscheduler调度spark任务踩坑记录
SSM项目在阿里云部署
关于我
Exception record-7
Apache Atlas 编译及安装记录