Pandas Convert List of Dictionaries to DataFrame

Use from_dict(), from_records(), json_normalize() methods to convert list of dictionaries (dict) to pandas DataFrame. Dict is a type in Python to hold key-value pairs. Key is used as a column name and value is used for column value when we convert dict to DataFrame. When a key is not found for some dicts and it exists on other dicts, it creates a DataFrame with NaN for non-existing keys.

In this article, we will see how to convert a list of dictionaries (dict) to a pandas DataFrame using pd.DataFrame(), pd.DataFrame.from_dict(), and pd.DataFrame.from_records(). Each dictionary in the list can have similar or different keys but different values.

Key Points –

• Convert a list of dictionaries to a DataFrame directly with pd.DataFrame(list_of_dicts).
• Pandas arranges columns based on the order of keys in the first dictionary by default.
• If some dictionaries lack certain keys, Pandas will insert NaN for missing values in those columns.
• Use the columns parameter to control which columns appear in the DataFrame or their order.
• Specify a custom index by using the index parameter during DataFrame creation or set the index later.
• For better performance, especially with large datasets, consider using pd.DataFrame.from_records().
• If dictionaries contain nested structures, use json_normalize() to flatten them into a DataFrame format.

Quick Examples of Convert a List of Dictionaries to a DataFrame

If you are in a hurry, below are some quick examples of how to convert a list of dictionaries (dict) to a Pandas DataFrame.

# Quick examples of convert a list of dictionaries to a dataframe

# Example 1: Convert a list of dictionaries 
# Using from_records() method
df = pd.DataFrame.from_records(technologies)

# Example 2: Set Custom index by using index parameter
df = pd.DataFrame.from_records(technologies, index=['1', '2', '3'])

# Example 3: Convert a list of dictionaries 
# By from_dict method
df = pd.DataFrame.from_dict(data)

# Example 4: Dictionary orientations of column
df=pd.DataFrame.from_dict(technologies, orient='columns')

# Example 5: Convert a list of dictionaries 
# Using json_normalize()
df=pd.json_normalize(technologies)

Create a DataFrame from List of Dict

If you have a list of dictionaries (dict), it is easy to create a DataFrame by using the DataFrame constructor. For more examples refer to how to create a pandas DataFrame with examples.

# Create a DataFrame from list 
# Of dictionaries with default indexes
import pandas as pd

# List of dict object
technologies=[{'Courses':'Spark','Duration':'30days','Discount':1000}, 
              {'Courses':'python','Fee':25000,'Courses_Fee':'Spark'}, 
              {'Fee':30000,'Duration':'35days','Duration_Discount':'10days'}]
              
# Create DataFrame from list of dic object
df=pd.DataFrame(technologies)
print(df)

Yields below output. Note that when a key is not found for some dicts and it exists on other dicts, it creates a DataFrame with NaN for non-existing keys. In case you would like to change the NaN values refer to How to replace NaN/None values with empty String.

Using from_records() Method Convert a List of Dictionaries

The from_records() method is used to convert a list of dictionaries to DataFrame. It can also be used to convert structured or record ndarray to DataFrame and is used to create a DataFrame from a structured ndarray, sequence of tuples or dicts, or from another DataFrame.

# Convert a list of dictionaries 
# Using from_records() method
df = pd.DataFrame.from_records(technologies)
print(df)

Yields the same output as above.

Set Custom Index by Using Index Parameter

To set a custom index while converting a list of dictionaries to a Pandas DataFrame, you can use the index parameter of the pd.DataFrame() constructor or any of its related methods (from_records(), from_dict()). For example, the index parameter is set to the custom_index list, assigning custom index values to the DataFrame. Y

In the second example, let’s see how to set a custom index by using index parameter. If you already have a DataFrame, you can set the Index to the DataFrame by using df.index.

# Convert list of dictionaries 
# To DataFrame with custom index
custom_index = ['1', '2', '3']
df = pd.DataFrame(technologies, index=custom_index)
print(df)

# Set Custom index by using index parameter
df = pd.DataFrame.from_records(technologies,index=['1', '2', '3'])
print(df)

Yields below output.

# Output:
  Courses Duration  Discount      Fee Courses_Fee Duration_Discount
1   Spark   30days    1000.0      NaN         NaN               NaN
2  python      NaN       NaN  25000.0       Spark               NaN
3     NaN   35days       NaN  30000.0         NaN            10days

Convert a List of Dictionaries by Using from_dict() Method

The pd.DataFrame.from_dict() method can be used to convert a list of dictionaries to a Pandas DataFrame. This method creates a DataFrame from a dictionary or a list of dictionaries. Each dictionary in the list will represent a row in the DataFrame, and the keys of the dictionaries will be used as column names.

Use pd.DataFrame.from_dict() to transform a list of dictionaries to pandas DatFrame. This function is used to construct DataFrame from dict of array-like or dicts.

# Convert a List of Dictionaries by from_dict method
df = pd.DataFrame.from_dict(technologies)
print(df)

Yields below output.

# Output:
  Courses Duration  Discount      Fee Courses_Fee Duration_Discount
0   Spark   30days    1000.0      NaN         NaN               NaN
1  python      NaN       NaN  25000.0       Spark               NaN
2     NaN   35days       NaN  30000.0         NaN            10days

Dictionary Orientations of Columns and Index

There are two primary types of dictionary orientations which are called Columns and index. It is to make the distinction between the different types of  dictionary orientations with the  orient='columns'.

# Dictionary orientations of column
df=pd.DataFrame.from_dict(technologies, orient='columns')
print(df)

Yields the same output as above.

Convert a List of Dictionaries by Using json_normalize()

If we want to convert an object to a JSON string, we have to note that NaN’s and None will be converted to null and datetime objects will be converted to UNIX timestamps. json_normalize() function works with lists of dictionaries (dict).

# Convert a list of dictionaries 
# Using json_normalize
df=pd.json_normalize(technologies)
print(df)

Yields the same output as above.

Complete Examples to Convert a List of Dictionaries of Pandas DataFrame

# Create a DataFrame from list of Dictionaries with default indexes.
import pandas as pd
technologies=[{'Courses':'Spark','Duration':'30days','Discount':1000},{'Courses':'python','Fee':25000,'Courses_Fee':'Spark'},{'Fee':30000,'Duration':'35days','Duration_Discount':'10days'}]
df=pd.DataFrame(technologies)
print(df)

# Convert a List of dictionaries using from_records() method
df = pd.DataFrame.from_records(technologies)
print(df)

# Set Custom index by using index parameter
df = pd.DataFrame.from_records(technologies, index=['1', '2', '3'])
print(df)

# Convert a List of Dictionaries by from_dict method
df = pd.DataFrame.from_dict(data)
print(df)

# Dictionary orientations of column
df=pd.DataFrame.from_dict(technologies, orient='columns')
print(df)

# Convert a list of dictionaries using json_normalize
df=pd.json_normalize(technologies)
print(df)

Frequently Asked Questions on Convert List of Dictionaries to DataFrame

Conclusion

In this article, you have learned about how to convert a list of dictionaries to pandas DataFrame by from_record(), from_dict(), json_normalize() with the examples.

Related Articles

• Pandas Series.isin() Function
• Pandas.Series.combine()
• Convert PySpark DataFrame to Pandas
• convert pandas to pyspark dataframe
• Rename Specific Columns in Pandas
• Pandas Create Empty DataFrame
• Pandas Convert Column to Int in DataFrame
• Pandas Convert JSON to DataFrame
• Pandas Convert Integer to Datetime Type
• Pandas Convert Datetime to Date Column
• pandas convert column to numpy array
• Convert Pandas Column to Lowercase
• Pandas Convert String to Integer
• How to Convert Pandas DataFrame to List?
• Pandas – Convert DataFrame to Dictionary (Dict)
• Pandas Empty DataFrame with Specific Column Types
• Retrieve Number of Columns From Pandas DataFrame
• Create Pandas DataFrame With Working Examples

References

• https://www.geeksforgeeks.org/create-a-pandas-dataframe-from-list-of-dicts/