当前位置:网站首页>org.apache.parquet.schema.InvalidSchemaException: A group type can not be empty. Parquet does not su
org.apache.parquet.schema.InvalidSchemaException: A group type can not be empty. Parquet does not su
2022-04-23 14:02:00 【百夜﹍悠ゼ】
1.读取cvs文件数据写入parquet文件
val spark =SparkSession.builder().appName("test").master("local").getOrCreate()
val file = "hdfs://clusters/test/demo.csv"
val frame = spark.read.option("header","true").csv(file)
frame.printSchema()
val newfile = file.split("\\.")(0)+".parquet"
frame.write.parquet(newfile)
报错提示:
org.apache.parquet.schema.InvalidSchemaException: A group type can not be empty. Parquet does not support empty group without leaves. Empty group: spark_schema
网上找了一大堆资料,大体解释都是说dataframe的schema结构问题,但本文并非此问题。
经过N次尝试,最终确定问题所在是 .parquet(newfile)。代码程序查询,发现此parquet方法调用的parquet包存在两个,默认使用了第一个包,而然这个包并非是spark-sql下的parquet包。
然后,就抱着试试的心态删除了版本1.8.1的parquet包,留下和spark依赖中的parquet包,重新运行代码,直捣黄龙。

版权声明
本文为[百夜﹍悠ゼ]所创,转载请带上原文链接,感谢
https://blog.csdn.net/AlierSnow/article/details/118966472
边栏推荐
- Port occupied 1
- 网站_收藏
- visio安装报错 1:1935 2:{XXXXXXXX...
- leetcode--380.O(1) 时间插入、删除和获取随机元素
- Haruki Murakami -- Excerpt from "what do I talk about when I talk about running"
- [VMware] address of VMware Tools
- [code analysis (7)] communication efficient learning of deep networks from decentralized data
- 9月8日,临去松山湖的前夜
- 变长参数__VA_ARGS__ 和 写日志的宏定义
- Android: answers to the recruitment and interview of intermediate Android Development Agency in early 2019 (medium)
猜你喜欢

Choreographer full resolution

Quartus Prime硬件实验开发(DE2-115板)实验一CPU指令运算器设计

基础知识学习记录

Kettle--控件解析

记录一个奇怪的bug:缓存组件跳转之后出现组件复制

程序编译调试学习记录

crontab定时任务输出产生大量邮件耗尽文件系统inode问题处理

mysql新表,自增id长达20位,原因竟是......

微信小程序的订阅号开发(消息推送)

Crontab timing task output generates a large number of mail and runs out of file system inode problem processing
随机推荐
2021年秋招,薪资排行NO
UML Unified Modeling Language
Yarn online dynamic resource tuning
多重继承虚基类习题
力扣刷题 101. 对称二叉树
redis如何解决缓存雪崩、缓存击穿和缓存穿透问题
PySide2
[VMware] address of VMware Tools
生产环境——
Quartus Prime硬件实验开发(DE2-115板)实验一CPU指令运算器设计
烟雾传感器(mq-2)使用详细教程(基于树莓派3b+实现)
Redis docker 安装
New关键字的学习和总结
Business case | how to promote the activity of sports and health app users? It is enough to do these points well
leetcode--380.O(1) 时间插入、删除和获取随机元素
Interesting talk about network protocol
Strange bug of cnpm
Spark入门基本操作
Qt Designer怎样加入资源文件
The latest development of fed digital currency