当前位置:网站首页>org. apache. parquet. schema. InvalidSchemaException: A group type can not be empty. Parquet does not su
org. apache. parquet. schema. InvalidSchemaException: A group type can not be empty. Parquet does not su
2022-04-23 16:43:00 【A hundred nights】
1. Read cvs File data write parquet file
val spark =SparkSession.builder().appName("test").master("local").getOrCreate()
val file = "hdfs://clusters/test/demo.csv"
val frame = spark.read.option("header","true").csv(file)
frame.printSchema()
val newfile = file.split("\\.")(0)+".parquet"
frame.write.parquet(newfile)
Wrong presentation :
org.apache.parquet.schema.InvalidSchemaException: A group type can not be empty. Parquet does not support empty group without leaves. Empty group: spark_schema
I found a lot of information on the Internet , The general explanation is that dataframe Of schema Structural problems , This is not the question of this article .
after N Attempts to , Finally, the problem is .parquet(newfile). Code program query , Found this parquet Method called parquet There are two packages , The first package is used by default , However, this bag is not spark-sql Under the parquet package .
then , I deleted the version with a try mentality 1.8.1 Of parquet package , Stay and spark Dependent parquet package , Rerun code , press forward to the enemy's capital .
版权声明
本文为[A hundred nights]所创,转载请带上原文链接,感谢
https://yzsam.com/2022/04/202204231401520905.html
边栏推荐
- Detailed explanation of file operation (2)
- Deeply understand the relevant knowledge of 3D model (modeling, material mapping, UV, normal), and the difference between displacement mapping, bump mapping and normal mapping
- Redis "8" implements distributed current limiting and delay queues
- 阿里研发三面,面试官一套组合拳让我当场懵逼
- ∑GL-透视投影矩阵的推导
- Custom implementation of Baidu image recognition (instead of aipocr)
- 伪分布安装spark
- ◰ GL shader handler encapsulation
- Zhongang Mining: Fluorite Flotation Process
- ◰GL-阴影贴图核心步骤
猜你喜欢
MySql主从复制
NVIDIA graphics card driver error
PyMySQL
Sail soft implements a radio button, which can uniformly set the selection status of other radio buttons
PyTorch:train模式与eval模式的那些坑
LVM与磁盘配额
MySQL master-slave replication
G008-HWY-CC-ESTOR-04 华为 Dorado V6 存储仿真器配置
Use case labeling mechanism of robot framework
LVM and disk quota
随机推荐
Change the icon size of PLSQL toolbar
Take according to the actual situation, classify and summarize once every three levels, and see the figure to know the demand
How to choose the wireless gooseneck anchor microphone and handheld microphone scheme
Server log analysis tool (identify, extract, merge, and count exception information)
[pyGame games] how did angry birds, a mobile game that became popular all over the world 10 years ago, dominate the list? Classic return
Qipengyuan horizon credible meta universe social system meets diversified consumption and social needs
The most detailed Backpack issues!!!
◰ GL shadow map core steps
欣旺达:HEV和BEV超快充拳头产品大规模出货
七朋元视界可信元宇宙社交体系满足多元化的消费以及社交需求
Public variables of robotframework
Introduction to how to set up LAN
批量制造测试数据的思路,附源码
Construction of promtail + Loki + grafana log monitoring system
最詳細的背包問題!!!
NVIDIA graphics card driver error
Modify the test case name generated by DDT
MySQL master-slave replication
Detailed explanation of file operation (2)
05 Lua 控制结构