Polars DataFrame shift() Usage & Examples

In Polars, the shift() function is used to shift the values in a column up or down by a specified number of positions, filling the empty positions with null or a given value. This is particularly useful for computing row-wise differences or aligning time series data.

In this article, I will explain the Polars DataFrame shift() function by using its syntax, parameters, and usage to generate a new DataFrame with all columns shifted by n positions.

Key Points –

• Shifts all columns in a Polars DataFrame up or down by a specified number of positions.
• Default shift is down (n=1), meaning values move downward, and the top rows are filled with null.
• Negative n shifts up, moving values upward and placing null at the bottom.
• Supports fill_value, allowing replacement of null with a specified value (e.g., 0, "missing", etc.).
• Does not modify the original DataFrame; instead, it returns a new shifted DataFrame.
• Works on all columns simultaneously, making it useful for time-series data and feature engineering.
• For shifting specific columns, use with_columns(pl.col("column_name").shift(n)) instead of DataFrame.shift().
• Useful for lagging operations, such as creating features for machine learning models or financial analysis.

Polars DataFrame shift() Introduction

Let’s know the syntax of the shift() method.

# Syntax of shift()
DataFrame.shift(n: int = 1, *, fill_value: IntoExpr | None = None) → DataFrame

Parameters of the Polars DataFrame shift()

Following are the parameters of the DataFrame shift() method.

• n (int, default=1) – Number of positions to shift (positive shifts down, negative shifts up).
• fill_value (optional) – Value to fill missing entries (default is null).

Return Value

This function returns a new DataFrame with all columns shifted by n positions.

Usage of Polars DataFrame shift() Method

The shift() method in Polars is used to move values within a DataFrame up or down by a specified number of positions while filling the empty spaces with null or a custom value.

Now, Let’s create a Polars DataFrame.

import polars as pl

data = {'Column1': [2, 4, 6, 8, 10],
        'Column2': [3, 5, 7, 9, 11]}
df = pl.DataFrame(data)
print("Original DataFrame:\n",df)

Yields below output.

You can use the shift() method to move all column values down by 1 position (default behavior).

# Shift all columns down by 1 (default)
df2 = df.shift(1)
print("Shifted DataFrame:\n", df2)

Here,

• The last four rows move down by one position.
• The first row is now null (empty) because there’s no previous value to shift down.

Shifting All Columns Up (n=-1)

Alternatively, to shift all columns up by 1 position in a Polars DataFrame, use the shift() function with n=-1.

# Shift all columns up by 1 position
df2 = df.shift(-1)
print("Shifted DataFrame (Up by 1):\n", df2)

# Output:
# Shifted DataFrame (Up by 1):
# shape: (5, 2)
┌─────────┬─────────┐
│ Column1 ┆ Column2 │
│ ---     ┆ ---     │
│ i64     ┆ i64     │
╞═════════╪═════════╡
│ 4       ┆ 5       │
│ 6       ┆ 7       │
│ 8       ┆ 9       │
│ 10      ┆ 11      │
│ null    ┆ null    │
└─────────┴─────────┘

Here,

• The first four rows move up by one position.
• The last row becomes null because there’s no value to shift up.

Shift with a Custom Fill Value

To shift all columns in a Polars DataFrame while replacing null values with a custom fill value, use the fill_value parameter in the shift() function.

# Shift all columns down by 1 and fill nulls with 0
df2 = df.shift(1, fill_value=0)
print("Shifted DataFrame with Custom Fill Value:\n", df2)

# Output:
# Shifted DataFrame with Custom Fill Value:
# shape: (5, 2)
┌─────────┬─────────┐
│ Column1 ┆ Column2 │
│ ---     ┆ ---     │
│ i64     ┆ i64     │
╞═════════╪═════════╡
│ 0       ┆ 0       │
│ 2       ┆ 3       │
│ 4       ┆ 5       │
│ 6       ┆ 7       │
│ 8       ┆ 9       │
└─────────┴─────────┘

Here,

• The first row, which would have been null, is now filled with 0.
• The remaining values are shifted down by 1.
• This method is useful when you want to avoid null values in your dataset.

Shift Only a Specific Column

To shift only a specific column in a Polars DataFrame, use the with_columns() method along with shift() function.

# Shift only 'Column1' down by 2
df2 = df.with_columns(df["Column1"].shift(2))
print("Shifted DataFrame (Only 'Column1' shifted):\n", df2)

# Output:
# Shifted DataFrame (Only 'Column1' shifted):
# shape: (5, 2)
┌─────────┬─────────┐
│ Column1 ┆ Column2 │
│ ---     ┆ ---     │
│ i64     ┆ i64     │
╞═════════╪═════════╡
│ null    ┆ 3       │
│ null    ┆ 5       │
│ 2       ┆ 7       │
│ 4       ┆ 9       │
│ 6       ┆ 11      │
└─────────┴─────────┘

Here,

• Only Column1 is shifted down by 2, while Column2 remains unchanged.
• The first two values of Column1 become null due to the shift.

Shift with Negative n and Fill Value

Similarly, to shift a column up (negative n) while replacing null values with a custom fill value, use the shift() function with fill_value.

# Shift all columns up by 1 and fill missing values with -1
df2 = df.shift(-1, fill_value=-1)
print("Shifted DataFrame with Negative Shift and Custom Fill Value:\n", df2)

# Output:
# Shifted DataFrame with Negative Shift and Custom Fill Value:
# shape: (5, 2)
┌─────────┬─────────┐
│ Column1 ┆ Column2 │
│ ---     ┆ ---     │
│ i64     ┆ i64     │
╞═════════╪═════════╡
│ 4       ┆ 5       │
│ 6       ┆ 7       │
│ 8       ┆ 9       │
│ 10      ┆ 11      │
│ -1      ┆ -1      │
└─────────┴─────────┘

Here,

• All values are shifted up by 1 position (n=-1).
• The last row, which would have been null, is now replaced with -1.
• This method ensures there are no missing values in the DataFrame.

Compute Differences Using shift()

Finally, to compute differences between consecutive rows in a Polars DataFrame, use shift() to align values and subtract.

# Compute differences by subtracting the shifted DataFrame
df2 = df.with_columns((df - df.shift(2)))
print("DataFrame with Differences:\n", df2)

# Output:
# DataFrame with Differences:
# shape: (5, 2)
┌─────────┬─────────┐
│ Column1 ┆ Column2 │
│ ---     ┆ ---     │
│ i64     ┆ i64     │
╞═════════╪═════════╡
│ null    ┆ null    │
│ null    ┆ null    │
│ 4       ┆ 4       │
│ 4       ┆ 4       │
│ 4       ┆ 4       │
└─────────┴─────────┘

Here,

• shift(2) moves two rows down, aligning previous values.
• Subtracting creates row-wise differences.
• The first two rows are null because there is no previous value.

Conclusion

In conclusion, the shift() function in Polars is a powerful tool for manipulating DataFrame values by shifting them up or down. It is commonly used in time-series analysis, calculating differences, creating lagged values, and aligning data efficiently.

Happy Learning!!

Related Articles

• Polars DataFrame sample() Method
• Polars DataFrame quantile() Method
• Polars DataFrame max() Method
• Polars DataFrame drop() Method
• Polars DataFrame select() Method
• Polars Cast String to Integer
• Convert Polars Cast Int to String
• Convert Polars Cast String to Float
• Convert Polars Cast Float to String
• Polars DataFrame schema() Usage & Examples

Reference

• https://docs.pola.rs/api/python/stable/reference/dataframe/api/polars.DataFrame.set_sorted.html