Fix Python – Pandas Left Outer Join results in table larger than left table


Asked By – Terence Chow

From what I understand about a left outer join, the resulting table should never have more rows than the left table…Please let me know if this is wrong…

My left table is 192572 rows and 8 columns.

My right table is 42160 rows and 5 columns.

My Left table has a field called ‘id’ which matches with a column in my right table called ‘key’.

Therefore I merge them as such:

combined = pd.merge(a,b,how='left',left_on='id',right_on='key')

But then the combined shape is 236569.

What am I misunderstanding?

Now we will see solution for issue: Pandas Left Outer Join results in table larger than left table


You can expect this to increase if keys match more than one row in the other DataFrame:

In [11]: df = pd.DataFrame([[1, 3], [2, 4]], columns=['A', 'B'])

In [12]: df2 = pd.DataFrame([[1, 5], [1, 6]], columns=['A', 'C'])

In [13]: df.merge(df2, how='left')  # merges on columns A
   A  B   C
0  1  3   5
1  1  3   6
2  2  4 NaN

To avoid this behaviour drop the duplicates in df2:

In [21]: df2.drop_duplicates(subset=['A'])  # you can use take_last=True
   A  C
0  1  5

In [22]: df.merge(df2.drop_duplicates(subset=['A']), how='left')
   A  B   C
0  1  3   5
1  2  4 NaN

This question is answered By – Andy Hayden

This answer is collected from stackoverflow and reviewed by FixPython community admins, is licensed under cc by-sa 2.5 , cc by-sa 3.0 and cc by-sa 4.0