How to Append a Python List to Another List (Series) of a Polars DataFrame?

To append a Python list to each list in a column of a Polars DataFrame, you can use the list.concat() method. This method efficiently merges the existing list values in each row with the new list you wish to add.

Appending a Python list to another list (Series) in a Polars DataFrame involves adding new elements to the existing list values within a column. In this article, I’ll walk you through how to perform this operation effectively using Polars.

Key Points –

• Polars supports the List data type, allowing each cell in a column to store a list of values.
• The list.concat() method efficiently appends one list to another in a column.
• You can append lists containing integers, booleans, strings, or any other supported data type.
• list.concat() creates a new DataFrame with the modified column; it does not modify the original DataFrame in-place.
• The list.concat() method works even if the original list in the column is empty.
• You can append lists stored in variables or dynamically generated lists.
• Appending lists can be done to each row in the DataFrame without needing to iterate manually.
• When using list.concat(), the list being appended is broadcast across all rows in the specified column.

Usage of Append a Python List to Another List (Series)

You can append a Python list to another list (Series) within a polars DataFrame column using the list.concat() method. This allows you to combine or extend lists that are already present in a column, making it easier to work with nested data structures or dynamic lists.

First, let’s create a Polars DataFrame.

import polars as pl

# Create a DataFrame with a list column
df = pl.DataFrame({
    "values": [[5, 10,], [15, 20], [25, 30]]
})
print("Original DataFrame:\n", df)

Yields below output.

To append integers to a list column in a Polars DataFrame, you can use the list.concat() function, which concatenates lists in the DataFrame with the given integer values.

# Append integers to the list column (e.g., append [35, 40] to each list)
df2 = df.with_columns(pl.col("values").list.concat([pl.lit([35, 40])]))
print("After appending integers to the list column:\n", df2)

# Append integers to the list column 
df2 = df.with_columns(pl.col("values").list.concat([35, 40]))
print("After appending integers to the list column:\n", df2)

Here,

• pl.lit([35, 40]): This creates a literal expression for the list [35, 40].
• list.concat(): Concatenates the existing list in each row of the values column with the new list [35, 40].

Append a Single-Element List

To append a single-element list to a list column in a Polars DataFrame, you can follow a similar approach using list.concat() with a single-element list. Here’s how you can append a single-element list (e.g., [50]) to each list in the values column.

# Append a single-element list (e.g., [50]) to each list in the "values" column
df2 = df.with_columns(pl.col("values").list.concat([pl.lit([50])]))
print("DataFrame after appending a single-element list to the list column:\n", df2)

# Output:
# DataFrame after appending a single-element list to the list column:
# shape: (3, 1)
┌──────────────┐
│ values       │
│ ---          │
│ list[i64]    │
╞══════════════╡
│ [5, 10, 50]  │
│ [15, 20, 50] │
│ [25, 30, 50] │
└──────────────┘

Here,

• pl.lit([50]): This creates a literal expression for the single-element list [50].
• list.concat(): This concatenates the existing list in the values column with the new list [50] for each row.

Append to Empty Lists

To append to empty lists in a Polars DataFrame, you can use the list.concat() method, even if the lists are initially empty. Here’s how you can append a list (e.g., [10, 20]) to each empty list in the values column.

import polars as pl

# Create a DataFrame with empty lists in the column
df = pl.DataFrame({
    "values": [[], [], []]  # Empty lists in each row
})

# Append a list (e.g., [10, 20]) to each empty list
df2 = df.with_columns(pl.col("values").list.concat([pl.lit([10, 20])]))
print("DataFrame after appending to empty lists:\n", df2)

# Output:
# DataFrame after appending to empty lists:
# shape: (3, 1)
┌───────────┐
│ values    │
│ ---       │
│ list[i64] │
╞═══════════╡
│ [10, 20]  │
│ [10, 20]  │
│ [10, 20]  │
└───────────┘

Here,

• All rows in the values column contain empty lists [].
• We use list.concat([pl.lit([10, 20])]) to append the list [10, 20] to each of the empty lists.
• After the operation, the values column contains the lists [10, 20] in each row.

Append a List of Booleans

To append a list of booleans to a list column in a Polars DataFrame, you can use the list.concat() method, similar to appending any other type of list. Here’s an example where we append a boolean list (e.g., [True, False]) to each list in the values column.

import polars as pl

df = pl.DataFrame({"values": [[True], [False]]})

df2 = df.with_columns(pl.col("values").list.concat([pl.lit([True, False])]))
print("DataFrame after appending a list of booleans to the list column:\n", df2)

# Output:
# DataFrame after appending a list of booleans to the list column:
# shape: (2, 1)
┌──────────────────────┐
│ values               │
│ ---                  │
│ list[bool]           │
╞══════════════════════╡
│ [true, true, false]  │
│ [false, true, false] │
└──────────────────────┘

Here,

• The values column contains lists of boolean.
• We use list.concat([pl.lit([True, False])]) to append the boolean list [True, False] to each list in the column.
• After the operation, the values column contains lists of boolean and booleans, such as [True, True, false]

Append Dynamic List Stored in a Variable

To append a dynamic list stored in a variable to a list column in a Polars DataFrame, simply pass the variable directly into the list.concat() method.

import polars as pl

# Create a DataFrame with a list column
df = pl.DataFrame({
    "values": [[1, 2], [3, 4], [5, 6]]
})

# Define a dynamic list to append
dynamic_list = [7, 8]

# Append the dynamic list to each list in the "values" column
df2 = df.with_columns(pl.col("values").list.concat([pl.lit(dynamic_list)]))
print("DataFrame after appending the dynamic list to the list column:\n", df2)

# Output:
# DataFrame after appending the dynamic list to the list column:
# shape: (3, 1)
┌─────────────┐
│ values      │
│ ---         │
│ list[i64]   │
╞═════════════╡
│ [1, 2, … 8] │
│ [3, 4, … 8] │
│ [5, 6, … 8] │
└─────────────┘

Here,

• The values column contains lists of integers.
• The list [7, 8] is stored in the variable dynamic_list.
• We use list.concat([pl.lit(dynamic_list)]) to append the contents of dynamic_list to each list in the values column.
• After the operation, the values column contains lists where [7, 8] has been appended to each list.

Append Strings to List Column

To append strings to a list column in a Polars DataFrame, you can use the list.concat() method just like with numeric or boolean lists.

import polars as pl

# Create a DataFrame with string lists
df = pl.DataFrame({
    "Tech": [["Spark"], ["Pandas", "Pyspark"], ["Polars"]]
})

# Define the string list to append
extra_words = ["Hadoop", "C"]

# Append the string list to each list in the "words" column
df2 = df.with_columns(
    pl.col("Tech").list.concat([pl.lit(extra_words)]))
print("DataFrame after appending strings to the list column:\n", df2)

# Output:
# DataFrame after appending strings to the list column:
# shape: (3, 1)
┌──────────────────────────────┐
│ Tech                         │
│ ---                          │
│ list[str]                    │
╞══════════════════════════════╡
│ ["Spark", "Hadoop", "C"]     │
│ ["Pandas", "Pyspark", … "C"] │
│ ["Polars", "Hadoop", "C"]    │
└──────────────────────────────┘

Conclusion

In summary, appending a Python list to a list column in a Polars DataFrame is both simple and efficient with the list.concat() method. Whether you’re dealing with integers, booleans, empty lists, or lists stored in variables, Polars offers a flexible and robust solution for managing nested list data.

Happy Learning!!

Related Articles

• How to Effectively Create Duplicate Rows in Polars?
• Strip Entire Polars DataFrame
• How to Update the Polars DataFrame
• Mapping a Python Dict to a Polars Series
• Polars Counting Elements in List Column
• Convert Polars Casting a Column to Decimal
• How to Convert Struct to Series in Polars?
• How to Remove Duplicate Columns in Polars?
• Conditional Assignment in Polars DataFrame
• Check if any Value in a Polars DataFrame is True
• How to Transform a Series of a Polars DataFrame?
• Add a New Column into an Existing Polars DataFrame

Reference

• https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.concat_list.html