[Python] pandas_gbq upload dataframe error: Error while reading data, error message: CSV...

I have a dataframe which I want to upload to BigQuery. The dataframe has 5 columns with the following types and looks like this:

name object
type object
population int32
geometry geometry
geojson object
dtype: object




name is name of an area as a string, type is its type (province, city, etc) also a string. population is the area population. geometry is a shapely multipolygon and finally, geojson is the multipolygon converted to geojson using df['geojson'] = df['geometry'].apply(lambda x: json.dumps(shapely.geometry.mapping(x)))

The exact error reads pandas_gbq.gbq.GenericGBQException: Reason: 400 Error while reading data, error message: CSV processing encountered too many errors, giving up. Rows: 855402; errors: 8; max bad: 0; error percent: 0, so it seems like row 855402 is the problematic one.

The first thing I did was to check whether this row had the same data types, which it does

>>> df.iloc[855402:855403].dtypes
name object
type object
population int32
geometry geometry
geojson object
dtype: object


I also checked whether the geometry file is valid and is also is

>>> df.iloc[855402]['geometry'].is_valid
True


What do I need to do to be able to upload this dataframe to BigQuery?

Continue reading...