[Python] Scaling OSMNX library's 'nearest_edges' function on huge spark dataset

I am trying to scale the distance value returned from the 'nearest_edges' function(from the OSMNX library) on a huge dataset using the lat and long columns as my inputs to the creation of my mutlidigraph. It takes forever to run and sometimes returns null. Is there any other solution? I created a user defined function (code below) so I can apply that function to the dataset using the long lat columns of that dataset.

My code below: import osmnx as ox @udf(returnType=T.DoubleType()) def get_distance_to_road (lat_dd=None,long_dd=None,dist_bbox=None): try: location = (lat_dd,long_dd)

G = ox.graph_from_point(
center_point=location,
dist=dist_bbox, #meter
simplify=True,
retain_all=True,
truncate_by_edge=True,
network_type='all'
)

Gp = ox.project_graph(G)
point_geom_proj, crs = ox.projection.project_geometry(Point(reversed(location)), to_crs=Gp.graph['crs'])
distance = np.round(ox.nearest_edges(Gp, point_geom_proj.x, point_geom_proj.y, return_dist=True)[1],2).item()

except Exception:
distance = None
return distance #meter

Continue reading...