当前位置:网站首页>org. apache. parquet. schema. InvalidSchemaException: A group type can not be empty. Parquet does not su

org. apache. parquet. schema. InvalidSchemaException: A group type can not be empty. Parquet does not su

2022-04-23 16:43:00 A hundred nights

1. Read cvs File data write parquet file

	val spark =SparkSession.builder().appName("test").master("local").getOrCreate()
	val file = "hdfs://clusters/test/demo.csv"
    val frame = spark.read.option("header","true").csv(file)
    frame.printSchema()
    val newfile = file.split("\\.")(0)+".parquet"
    frame.write.parquet(newfile)

Wrong presentation :

org.apache.parquet.schema.InvalidSchemaException: A group type can not be empty. Parquet does not support empty group without leaves. Empty group: spark_schema

I found a lot of information on the Internet , The general explanation is that dataframe Of schema Structural problems , This is not the question of this article .
after N Attempts to , Finally, the problem is .parquet(newfile). Code program query , Found this parquet Method called parquet There are two packages , The first package is used by default , However, this bag is not spark-sql Under the parquet package .
then , I deleted the version with a try mentality 1.8.1 Of parquet package , Stay and spark Dependent parquet package , Rerun code , press forward to the enemy's capital .

 Insert picture description here

版权声明
本文为[A hundred nights]所创,转载请带上原文链接,感谢
https://yzsam.com/2022/04/202204231401520905.html