Pandas Series.shift() Function

In pandas, the shift() function is used to shift the index by the desired number of periods with an optional time frequency. It is particularly useful for time-series data manipulation but can be applied to any Series.

In this article, I will explain the Series.shift() function and using its syntax, parameters, and usage how to shift the values of the Series either forward or backward by specifying the number of periods.

Key Points –

• Series.shift() is used to shift the index by a desired number of periods with an optional time frequency.
• It allows you to shift the values of a Series either forward or backward by specifying the number of periods.
• Shifting forward moves the values to later positions in the Series while shifting backward moves the values to earlier positions.
• When shifting, the values at the beginning or end of the Series are replaced with NaN (by default) or a specified fill value.
• The shift() function is commonly used in time series analysis for tasks like calculating differences between consecutive values or creating lagged or lead variables for modeling purposes.
• Pandas Series shift() function is essential for temporal analysis tasks such as time series forecasting and analyzing time-dependent relationships between variables.

Syntax of Pandas Series.shift() Function

Following is the syntax of the pandas Series.shift() function.

# Syntax of Series.shift() function
Series.shift(periods=1, freq=None, axis=0, fill_value=None)

Parameters of the Series.shift()

Following are the parameters of the Series.shift() function.

• periods – This parameter specifies the number of periods to shift. If positive, it shifts the index forward, creating a lag; if negative, it shifts the index backward, creating a lead. By default, it is set to 1.
• freq – An optional parameter specifying the frequency of the data. If the index of the Series is Datetime-like, this parameter can be used to specify the frequency. This is useful for time series data. By default, it is set to None.
• axis – This parameter indicates the axis along which the shift operation is applied. By default, it is set to 0, which means the shift is applied along the index.
• fill_value – An optional parameter specifying the value to use for newly introduced missing values after the shift operation. By default, NaN (missing value) is used.

Return Value

It returns a new Series object with the index shifted by the specified number of periods. The values in the Series remain the same, but their corresponding index positions change based on the shift operation. If the index is Datetime-like and a frequency is specified, the index values are shifted accordingly based on that frequency.

Shifting Forward the Pandas Series

Let’s create a Pandas Series using Python lists and then apply the shift() function to the given Series to get the updated Series.

import pandas as pd

# Create a Series
series = pd.Series([5, 10, 15, 20, 25, 30])
print("Original Series:\n",series)

Yields below output.

You can shift the pandas Series forward by specifying the number of periods you want to shift using the shift() function. By default, this method shifts the values of the Series by one position. This means each element in the series moves to the next position, and the first element becomes NaN (Not a Number). Let’s call the shift() function with the given Series(no arguments) to shift its values forward by one position.

# Shift the series forward by one period
shifted_forward = series.shift()
print("Shifted forward by one period:\n", shifted_forward)

The above code has returned a new series with the same values but shifted forward by one position. The first element in the shifted series is NaN, and the rest of the elements are shifted by one position compared to the original series.

Yields below output.

Using Negative Periods to Shift Backward

To shift the Pandas Series backward by a specific number of periods, use the shift() function. To do this, you need to specify a negative value for the parameter periods of the shift() function. This will shift the Series values backward by the specified number of periods.

# Shift the Series backward by two periods
shifted_backward = series.shift(-2)
print("Shifted backward by two periods:")
print(shifted_backward)

# Output:
# Shifted backward by two periods:
# 0    15.0
# 1    20.0
# 2    25.0
# 3    30.0
# 4     NaN
# 5     NaN
#dtype: float64

In the above example, series.shift(-2) shifts the series backward by two periods, filling the last two positions with NaN.

Shifting with specified Fill Value

You can specify a fill value to replace the missing values after shifting a Pandas Series. By default, the shift() function represents the missing values with NaN values after completing the shifting process. If you want to get the new Series with specified missing values you can set the fill_value parameter with the specified value. In this example, I will set the fill_value with 4. So, it will replace the missing values with 4 after shifting the Series.

# Shift the series forward by two periods with a fill value of 4
shifted_with_fill = series.shift(2, fill_value=4)
print("Shifted forward by two periods with fill value:\n",shifted_with_fill)

# Output:
# Shifted forward by two periods with fill value:
# 0     4
# 1     4
# 2     5
# 3    10
# 4    15
# 5    20
# dtype: int64

Shifting with Frequency (Datetime Index)

When you’re working with time series data and have a datetime index, you can shift a Pandas Series by a specified number of periods while maintaining the frequency of the datetime index.

import pandas as pd

# Create a sample series with datetime index
dates = pd.date_range('2024-01-01', periods=5)
series = pd.Series([5, 10, 15, 20, 25], index=dates)

# Shift the series forward by one period with a frequency of one day ('D')
result = series.shift(1, freq='D')
print("Shifted forward by one period with frequency 'D':\n",result)

# Output:
# Shifted forward by one period with frequency 'D':
# 2024-01-02     5
# 2024-01-03    10
# 2024-01-04    15
# 2024-01-05    20
# 2024-01-06    25
# Freq: D, dtype: int64

In the above example, series.shift(1, freq='D') shifts the series forward by one period while maintaining the frequency of the datetime index as one day (‘D’). The resulting series has the same index frequency, but the values are shifted accordingly.

Shifting Time Series Data

Similarly, Shifting time series data is a common operation in time series analysis, often used for creating lagged variables or aligning data at different time points. To shift the index forward by 1 month in a time series data, you can use the shift() function from pandas with the freq parameter set to 'M' for months.

import pandas as pd

# Create a time series with datetime index
dates = pd.date_range(start='2024-01-01', periods=6, freq='D')
values = [5, 10, 15, 20, 25, 30]
series = pd.Series(values, index=dates)

# Shift the index forward by one month
shifted_series = series.shift(periods=1, freq='M')
print("Shifted Time Series (Forward by One Month):\n",shifted_series)

# Output:
# Shifted Time Series (Forward by One Month):
# 2024-01-31     NaN
# 2024-02-29    5.0
# 2024-03-31   10.0
# 2024-04-30   15.0
# 2024-05-31   20.0
# 2024-06-30   25.0
# Freq: M, dtype: float64

In this example, we created a time series with a daily frequency. Then, we shifted the index forward by one month using the shift() function, resulting in a new time series with a monthly frequency. The values are aligned according to the new index, and missing values (NaN) are introduced where necessary. This operation allows for analyzing the data at different temporal resolutions or for creating lagged variables for time series forecasting.

Frequently Asked Questions on Pandas Series.shift() Function

Conclusion

In the article, I have explained the shift() function and using its syntax, parameters, and usage how to shift the values of a Series either forward or backward by specifying the number of periods. Additionally, I have demonstrated how to shift time series data that has a datetime index while maintaining the frequency of the data.

Happy Learning!!

Related Articles

• Pandas Series astype() Function
• Pandas Series concat() Function
• Pandas Series where() Function
• Change the Index Order in Pandas Series
• Use pandas.to_numeric() Function
• Pandas Series.mean() Function 
• Convert Pandas Series to String 
• Pandas Series.quantile() Function
• Pandas Series.diff() Function
• Pandas Series any() Function
• Pandas Series map() Function
• Pandas Series round() Function
• Pandas Series Drop duplicates() Function
• What is a Pandas Series Explained With Examples

References

• https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.shift.html