• Post author:
  • Post category:Pandas
  • Post last modified:March 27, 2024
  • Reading time:15 mins read
You are currently viewing Pandas Concat Two DataFrames Explained

Use pandas.concat() to concatenate/merge two or multiple pandas DataFrames across rows or columns. When you concat() two pandas DataFrames on rows, it creates a new Dataframe containing all rows of two DataFrames basically it does append one DataFrame with another. When you use concat() on columns it performs the join operation.

In this article, I will explain how to concatenate two pandas DataFrame by rows and columns with examples

pandas concat() Key Points

  • By default concat() method performs an append operation meaning, it appends each DataFrame at the end of another DataFrame and creates a single DataFrame
  • When you use concat() to join two DataFrames, it supports only inner and outer joins, and by default, it performs outer join.
  • Using concat you can join or append multiple pandas DataFrames

Related: In Pandas, you can also concatenate Pandas DataFrame columns.

1. Quick Examples of Concat Two Pandas DataFrames

If you are in a hurry, below are some quick examples of how to concatenate two DataFrames using the concat() method.


# Below are some quick examples

# Using pandas.concat() to concat two DataFrame
data = [df, df1]
df2 = pd.concat(data)

# Use pandas.concat() method to ignore_index 
df2 = pd.concat([df, df1], ignore_index=True, sort=False)

# Using pandas.concat() Method
data = [df, df1]
df2 = pd.concat(data, ignore_index=True, sort=False)

# Using pandas.concat() to join concat two DataFrames
data = pd.concat([df, df1], axis=1, join='inner')

# Using DataFrame.append() method
data = df.append(df1)

# Use DataFrame.append() 
df2 = df.append(df1, ignore_index=True)

# Appending multiple DataFrame
data = df.append([df1, df2])

2. pandas concat() Syntax and Usage

Below is the syntax of the pandas.concat() method.


# Syntax of concat() function
pandas.concat(objs, axis=0, join='outer', ignore_index=False, keys=None, levels=None, names=None, verify_integrity=False, sort=False, copy=True)

2.1 Use pandas.concat() to Concat Two DataFrames

First, let’s create two Pandas DataFrames with different content, and then, you can apply the concat() method to concat the given DataFrames.


import pandas as pd
df = pd.DataFrame({'Courses': ["Spark","PySpark","Python","pandas"],
                    'Fee' : [20000,25000,22000,24000]})

df1 = pd.DataFrame({'Courses': ["Pandas","Hadoop","Hyperion","Java"],
                    'Fee': [25000,25200,24500,24900]})
print("First DataFrame:\n", df)
print("Second DataFrame:\n", df1)

Yields below output.

Pandas concat two DataFrames

You can use pandas.concat() method to concat two DataFrames by rows meaning appending two DataFrames. By default, it performs append operations similar to a union where it bright all rows from both DataFrames to a single DataFrame. The below example demonstrates append using concat().


# Using pandas.concat() to concat two DataFrames
data = [df, df1]
df2 = pd.concat(data)
print("After concatenating the two DataFrames:\n", df2)

Yields below output:

Pandas concat two DataFrames

If you notice in the above example, it just added the row index as-is from two DataFrame, sometimes you may want to reset the index. You can do so by using the ignore_index=True param.


# Use pandas.concat() method to ignore_index 
df2 = pd.concat([df, df1], ignore_index=True, sort=False)
print(df2)

Yields below output.


# Output:
    Courses    Fee
0     Spark  20000
1   PySpark  25000
2    Python  22000
3    pandas  24000
4    Pandas  25000
5    Hadoop  25200
6  Hyperion  24500
7      Java  24900

3. Using pandas.concat() to Join Two DataFrames

As I said above pandas.concat() method is also used to join two pandas DataFrams on columns. In order to do so use axis=1, join='inner'. By default, pd.concat() is a row-wise outer join.


import pandas as pd
df = pd.DataFrame({'Courses':["Spark","PySpark","Python","pandas"],
                      'Fee' :[20000,25000,22000,24000]})
  
df1 = pd.DataFrame({'Duration':['30day','40days','35days','60days'],
                      'Discount':[1000,2300,2500,2000,]}) 

# Using pandas.concat() to join concat two DataFrames
df2 = pd.concat([df, df1], axis=1, join='inner')
print(df2)

Yields below output.


# Output:
   Courses    Fee Duration  Discount
0    Spark  20000    30day      1000
1  PySpark  25000   40days      2300
2   Python  22000   35days      2500
3   pandas  24000   60days      2000

4. Concatenate Multiple DataFrames Using pandas.concat()

You can also use pandas.concat() method to concatenate multiple DataFrames.


import pandas as pd
df = pd.DataFrame({'Courses': ["Spark", "PySpark", "Python", "Pandas"],
                    'Fee' : ['20000', '25000', '22000', '24000']}) 
  
df1 = pd.DataFrame({'Courses': ["Unix", "Hadoop", "Hyperion", "Java"],
                    'Fee': ['25000', '25200', '24500', '24900']})
  
df2 = pd.DataFrame({'Duration':['30day','40days','35days','60days','55days'],
                    'Discount':[1000,2300,2500,2000,3000]})
  
# Appending multiple DataFrame
df3 = pd.concat([df, df1, df2])
print(df3)

Yields below output.


# Output:
    Courses    Fee Duration  Discount
0     Spark  20000      NaN       NaN
1   PySpark  25000      NaN       NaN
2    Python  22000      NaN       NaN
3    Pandas  24000      NaN       NaN
0      Unix  25000      NaN       NaN
1    Hadoop  25200      NaN       NaN
2  Hyperion  24500      NaN       NaN
3      Java  24900      NaN       NaN
0       NaN    NaN    30day    1000.0
1       NaN    NaN   40days    2300.0
2       NaN    NaN   35days    2500.0
3       NaN    NaN   60days    2000.0
4       NaN    NaN   55days    3000.0

4. Use DataFrame.append() to Concat Two DataFrames

Alternatively, you can use pandas.DataFrame.append() method to concatenate DataFrames on rows. For E.x, df.append(df1) appends df1 to the df DataFrame.


import pandas as pd
df = pd.DataFrame({'Courses': ["Spark","PySpark","Python","pandas"],
                    'Fee' : [20000,25000,22000,24000]})

df1 = pd.DataFrame({'Courses': ["Pandas","Hadoop","Hyperion","Java"],
                    'Fee': [25000,25200,24500,24900]})

# Using DataFrame.append() to concat Two DataFrames
df2 = df.append(df1)
print(df2)

Yields below output.


# Output:
    Courses    Fee
0     Spark  20000
1   PySpark  25000
2    Python  22000
3    pandas  24000
0    Pandas  25000
1    Hadoop  25200
2  Hyperion  24500
3      Java  24900

Use ignore_index=True param to reset the index on the combined DataFrame.


# Use DataFrame.append() 
df2 = df.append(df1, ignore_index=True)
print(df2)

Yields below output.


# Output:
    Courses    Fee
0     Spark  20000
1   PySpark  25000
2    Python  22000
3    pandas  24000
4    Pandas  25000
5    Hadoop  25200
6  Hyperion  24500
7      Java  24900

Frequently Asked Questions

What is the purpose of concatenating two DataFrames in Pandas?

Concatenating two DataFrames in Pandas is a common operation used to combine two or more DataFrames along a particular axis (either rows or columns) to create a single, larger DataFrame. It is often used to merge or stack data from different sources for further analysis.

How can I concatenate two DataFrames in Pandas?

You can use the pd.concat() function in Pandas to concatenate two or more DataFrames.

Can I concatenate DataFrames with different columns?

You can concatenate DataFrames with different columns. By default, pd.concat() aligns the columns based on their names. Missing columns will be filled with NaN values.

What is the difference between concatenating along the rows (axis=0) and along the columns (axis=1)?

Concatenating along axis=0 (the default) combines DataFrames vertically, stacking them on top of each other, while concatenating along axis=1 combines DataFrames horizontally, extending them side by side.

How do I specify the axis for concatenation?

To specify the axis explicitly using the axis parameter in pd.concat(). For example, to concatenate along columns, use axis=1, and to concatenate along rows, use axis=0.

Conclusion

In this article, I have explained how to concatenate two pandas DataFrames using pandas.concat() and DataFrame.append() methods with examples. concat() method is also used to concatenate multiple pandas DataFrames.

Happy Learning !!

References

Naveen Nelamali

Naveen Nelamali (NNK) is a Data Engineer with 20+ years of experience in transforming data into actionable insights. Over the years, He has honed his expertise in designing, implementing, and maintaining data pipelines with frameworks like Apache Spark, PySpark, Pandas, R, Hive and Machine Learning. Naveen journey in the field of data engineering has been a continuous learning, innovation, and a strong commitment to data integrity. In this blog, he shares his experiences with the data as he come across. Follow Naveen @ LinkedIn and Medium

This Post Has One Comment

  1. Anonymous

    good for out standing khnowlgde

Comments are closed.