Fix Python – Assign pandas dataframe column dtypes


Asked By – hatmatrix

I want to set the dtypes of multiple columns in pd.Dataframe (I have a file that I’ve had to manually parse into a list of lists, as the file was not amenable for pd.read_csv)

import pandas as pd
print pd.DataFrame([['a','1'],['b','2']],

I get

ValueError: entry not a 2- or 3- tuple

The only way I can set them is by looping through each column variable and recasting with astype.

dtypes = {'x':'object','y':'int'}
mydata = pd.DataFrame([['a','1'],['b','2']],
for c in mydata.columns:
    mydata[c] = mydata[c].astype(dtypes[c])
print mydata['y'].dtype   #=> int64

Is there a better way?

Now we will see solution for issue: Assign pandas dataframe column dtypes


Since 0.17, you have to use the explicit conversions:

pd.to_datetime, pd.to_timedelta and pd.to_numeric

(As mentioned below, no more “magic”, convert_objects has been deprecated in 0.17)

df = pd.DataFrame({'x': {0: 'a', 1: 'b'}, 'y': {0: '1', 1: '2'}, 'z': {0: '2018-05-01', 1: '2018-05-02'}})


x    object
y    object
z    object
dtype: object


   x  y           z
0  a  1  2018-05-01
1  b  2  2018-05-02

You can apply these to each column you want to convert:

df["y"] = pd.to_numeric(df["y"])
df["z"] = pd.to_datetime(df["z"])    

   x  y          z
0  a  1 2018-05-01
1  b  2 2018-05-02


x            object
y             int64
z    datetime64[ns]
dtype: object

and confirm the dtype is updated.

OLD/DEPRECATED ANSWER for pandas 0.12 – 0.16: You can use convert_objects to infer better dtypes:

In [21]: df
   x  y
0  a  1
1  b  2

In [22]: df.dtypes
x    object
y    object
dtype: object

In [23]: df.convert_objects(convert_numeric=True)
   x  y
0  a  1
1  b  2

In [24]: df.convert_objects(convert_numeric=True).dtypes
x    object
y     int64
dtype: object

Magic! (Sad to see it deprecated.)

This question is answered By – Andy Hayden

This answer is collected from stackoverflow and reviewed by FixPython community admins, is licensed under cc by-sa 2.5 , cc by-sa 3.0 and cc by-sa 4.0