[Python] Unable to Bypass Cloudflare in Cloud Environment Scraping

Summary


I am trying to implement a scraping process that bypasses Cloudflare in an OCI environment, but I am not successful. It works in my local environment using several methods, but it fails in the cloud environment. Should I assume that OCI's IP addresses are blacklisted by Cloudflare?

If anyone has knowledge about this, I would appreciate your guidance.

What I've Tried


Scraping with Python

• nodriver: Module
• drissionpage: Module
• flaresolverr: Docker proxy

Environment

Local

• Machine: Mac mini with M2 chip
• macOS: Sonoma 14.6.1

Cloud

• OKE (Oracle Linux, ARM-based)
• OCI Instance (Oracle Linux, ARM-based, Ubuntu)

Success/Failure

nodriver

• Local: Successfully bypasses Cloudflare on Mac
• Cloud: Fails to bypass Cloudflare on OCI instance (though the process finishes normally)

drissionpage

• Local: Successfully bypasses Cloudflare on Mac
• Cloud: Fails to bypass Cloudflare on OCI instance (though the process finishes normally)

flaresolverr

• Local: Successfully bypasses Cloudflare by running Docker Compose to set up a proxy container and using the proxy in Python
• Cloud (OKE): Fails to bypass Cloudflare with the proxy container running on a pod and Python using the proxy (process finishes normally)
• Cloud (OCI Instance): Fails to bypass Cloudflare with the proxy container running on the instance and Python using the proxy (process finishes normally)

Continue reading...