join

两个DataFrame根据某个条件进行关联。类似的还有crossJoin返回一个笛卡尔积表

Parameters

otherDataFrame

Right side of the join

onstr, list or Column, optional

a string for the join column name, a list of column names, a join expression (Column), or a list of Columns. If on is a string or a list of strings indicating the name of the join column(s), the column(s) must exist on both sides, and this performs an equi-join.

howstr, optional

default inner. Must be one of: inner, cross, outer, full, fullouter, full_outer, left, leftouter, left_outer, right, rightouter, right_outer, semi, leftsemi, left_semi, anti, leftanti and left_anti.

cond = [df['name'] == df1['name'], df['age'] == df1['age']]
df.join(df1, cond, 'outer').select(df.name, df3.age)

lit

Creates a column of literal value.

file = '/mnt/dbwarehouse/raw/user.csv'
df = df.withColumn('ingest_file',lit(file))\
.withColumn('converted',lit(True))
display(df)

na.fill & fillna

na.fill:整表填充，将整个表里的空值 且 字段类型能够对应得上得地方 填充。

fillna:用法同na.fill。

df.na.fill(50).show() # 可以看到name列为文本，则不进行填充
+---+------+-----+
|age|height| name|
+---+------+-----+
| 10|    80|Alice|
|  5|    50|  Bob|
| 50|    50|  Tom|
| 50|    50| null|
+---+------+-----+

df.na.fill(False).show() # 只有spy列为bool，故只填充这一列
+----+-------+-----+
| age|   name|  spy|
+----+-------+-----+
|  10|  Alice|false|
|   5|    Bob|false|
|null|Mallory| true|
+----+-------+-----+

df.na.fill({
    'age': 50, 'name': 'unknown'}).show() # 按字段名指定填充得值
+---+------+-------+
|age|height|   name|
+---+------+-------+
| 10|    80|  Alice|
|  5|  null|    Bob|
| 50|  null|    Tom|
| 50|  null|unknown|
+---+------+-------+