Skip to content

add flatten-dataframe#186

Open
slavlotski wants to merge 1 commit into
MrPowers:mainfrom
slavlotski:feat/flatten-dataframe
Open

add flatten-dataframe#186
slavlotski wants to merge 1 commit into
MrPowers:mainfrom
slavlotski:feat/flatten-dataframe

Conversation

@slavlotski
Copy link
Copy Markdown

Closes #47.

Adds a new public helper chispa.flatten_dataframe(df, sep="_") that turns a DataFrame containing nested columns into a fully flat one. It walks the schema and, for each complex
field:

  • StructType → expands every sub-field into its own top-level column named (e.g. name.firstname → name_firstname).
  • ArrayType → applies explode_outer, so each element becomes its own row (null-preserving).
  • MapType → collects the distinct keys present in the column and turns each into a column.

The walk repeats until no nested types remain, so deeply nested / mixed schemas (struct of map of array of struct, etc.) are flattened in one call.

@slavlotski slavlotski mentioned this pull request May 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flatten dataframe

1 participant