Pandas Convert List of Dictionaries to DataFrame

  • Post author:
  • Post category:Pandas
  • Post last modified:November 27, 2023

Use from_dict(), from_records(), json_normalize() methods to convert list of dictionaries (dict) to pandas DataFrame. Dict is a type in python to hold key-value pairs. Key is used as a column name and value is used for column value when we convert dict to DataFrame. When a key is not found for some dicts and it exists on other dicts, it creates a DataFrame with NaN for non-existing keys.

In this article, we will see how to convert a list of dictionaries (dict) to a pandas DataFrame using pd.DataFrame(), pd.DataFrame.from_dict(), and pd.DataFrame.from_records(). Each dictionary in the list can have similar or different keys but different values.

1. Quick Examples of Convert a List of Dictionaries to a DataFrame

If you are in a hurry, below are some quick examples of how to convert a list of dictionaries (dict) to a Pandas DataFrame.


# Below are quick examples

# Create a list of dictionary objects
technologies=[{'Courses':'Spark','Duration':'30days','Discount':1000} 
              {'Courses':'python','Fee':25000,'Courses_Fee':'Spark'}, 
              {'Fee':30000,'Duration':'35days','Duration_Discount':'10days'}]

# Convert a List of dictionaries using from_records() method.
df = pd.DataFrame.from_records(technologies)

# Set Custom index by using index parameter.
df = pd.DataFrame.from_records(technologies, index=['1', '2', '3'])

# Convert a List of Dictionaries by from_dict method.
df = pd.DataFrame.from_dict(data)

# Dictionary orientations of column.
df=pd.DataFrame.from_dict(technologies, orient='columns')

# Convert a list of dictionaries using json_normalize().
df=pd.json_normalize(technologies)

1. Create a DataFrame from List of Dict

If you have a list of dictionaries (dict), it is easy to create a DataFrame by using the DataFrame constructor. For more examples refer to how to create a pandas DataFrame with examples.


# Create a DataFrame from list of Dictionaries with default indexes.
import pandas as pd

# List of dict object
technologies=[{'Courses':'Spark','Duration':'30days','Discount':1000} 
              {'Courses':'python','Fee':25000,'Courses_Fee':'Spark'}, 
              {'Fee':30000,'Duration':'35days','Duration_Discount':'10days'}]

# Create DataFrame from list of dic object
df=pd.DataFrame(technologies)
print(df)

Yields below output. Note that when a key is not found for some dicts and it exists on other dicts, it creates a DataFrame with NaN for non-existing keys. In case you would like to change the NaN values refer to How to replace NaN/None values with empty String.


# Output:
  Courses Duration  Discount      Fee Courses_Fee Duration_Discount
0   Spark   30days    1000.0      NaN         NaN               NaN
1  python      NaN       NaN  25000.0       Spark               NaN
2     NaN   35days       NaN  30000.0         NaN            10days

2. Using from_records() Method Convert a List of Dictionaries

The from_records() method is used to convert a list of dictionaries to DataFrame. It can also be used to convert structured or record ndarray to DataFrame and is used to create a DataFrame from a structured ndarray, sequence of tuples or dicts, or from another DataFrame.


# Convert a List of dictionaries using from_records() method.
df = pd.DataFrame.from_records(technologies)
print(df)

Yields same output as above.

3. Set Custom Index by Using Index Parameter

In this section, let’s see how to set a custom index by using index parameter. If you already have a DataFrame, you can set the Index to the DataFrame by using df.index.


# Set Custom index by using index parameter.
df = pd.DataFrame.from_records(technologies,index=['1', '2', '3'])
print(df)

Yields below output.


# Output:
  Courses Duration  Discount      Fee Courses_Fee Duration_Discount
1   Spark   30days    1000.0      NaN         NaN               NaN
2  python      NaN       NaN  25000.0       Spark               NaN
3     NaN   35days       NaN  30000.0         NaN            10days

4. Convert a List of Dictionaries by Using from_dict() Method

Use pd.DataFrame.from_dict() to transform a list of dictionaries to pandas DatFrame. This function is used to construct DataFrame from dict of array-like or dicts. 


# Convert a List of Dictionaries by from_dict method.
df = pd.DataFrame.from_dict(technologies)
print(df)

Yields below output.


# Output:
  Courses Duration  Discount      Fee Courses_Fee Duration_Discount
0   Spark   30days    1000.0      NaN         NaN               NaN
1  python      NaN       NaN  25000.0       Spark               NaN
2     NaN   35days       NaN  30000.0         NaN            10days

5. Dictionary Orientations of Columns and Index

There are two primary types of dictionary orientations which are called Columns and index. It is to make the distinction between the different types of  dictionary orientations with the  orient='columns'.


# Dictionary orientations of column.
df=pd.DataFrame.from_dict(technologies, orient='columns')
print(df)

Yields same output as above.

6. Convert a List of Dictionaries by Using json_normalize()

If we want to convert an object to a JSON string, we have to note that NaN’s and None will be converted to null and datetime objects will be converted to UNIX timestamps. json_normalize() function works with lists of dictionaries (dict).


# Convert a list of dictionaries using json_normalize.
df=pd.json_normalize(technologies)
print(df)

Yields same output as above.

7. Complete Examples to Convert a List of Dictionaries of Pandas DataFrame


# Create a DataFrame from list of Dictionaries with default indexes.
import pandas as pd
technologies=[{'Courses':'Spark','Duration':'30days','Discount':1000},{'Courses':'python','Fee':25000,'Courses_Fee':'Spark'},{'Fee':30000,'Duration':'35days','Duration_Discount':'10days'}]
df=pd.DataFrame(technologies)
print(df)

# Convert a List of dictionaries using from_records() method.
df = pd.DataFrame.from_records(technologies)

# Set Custom index by using index parameter.
df = pd.DataFrame.from_records(technologies, index=['1', '2', '3'])

# Convert a List of Dictionaries by from_dict method.
df = pd.DataFrame.from_dict(data)

# Dictionary orientations of column.
df=pd.DataFrame.from_dict(technologies, orient='columns')

# Convert a list of dictionaries using json_normalize.
df=pd.json_normalize(technologies)

Conclusion

In this article, you have learned about how to convert a list of dictionaries to pandas DataFrame by from_record(), from_dict(), json_normalize() with the examples.

References

Tags:

Leave a Reply

You are currently viewing Pandas Convert List of Dictionaries to DataFrame