[Python] issues in fetching data from firebase

I have uploaded the data on firebase which has 12,000 products, and each product has multiple sections, firstly I want to fetch one of the sections from each product. Example: product 1 -> _id, name, price etc. product 2 -> _id, name, price etc. I want to fetch _id from all 12,000 products but no matter what I try it is only able to retrieve some 2k data.

This is the python code I have written

import firebase_admin
from firebase_admin import db, credentials
import sys
import requests
import logging

# Set the default encoding to utf-8
sys.stdout.reconfigure(encoding='utf-8')

# Configure logging
logging.basicConfig(level=logging.INFO)

# Initialize Firebase
cred = credentials.Certificate("credentials.json")
firebase_admin.initialize_app(cred, {"databaseURL": "https://---.firebasedatabase.app/"})

# Create a reference to the Firebase Realtime Database
ref = db.reference("/products")

# Fetch all data directly
def fetch_all_data():
try:
# Fetch all products from the database
data = ref.get() # This assumes that your data is directly under the /products node
if data is None:
logging.warning("No data returned from Firebase.")
return data
except (requests.RequestException, firebase_admin._http_client.TransportError) as e:
logging.error(f"Network error: {e}. Retrying...")
return fetch_all_data() # Retry the data fetch

# Fetch all products
products = fetch_all_data()

# Check if the products data is available
if products:
total_count = len(products)
id_list = [product.get("_id") for product in products if isinstance(product, dict) and product.get("_id")]

logging.info(f"Total products with _id: {total_count}")
logging.info(f"List of _id values: {id_list[:20]}...") # Print first 20 ids as a sample
else:
logging.error("No products found in the database")

logging.info("Length of List of _id values: %d", len(id_list)) # Print length of the list



I checked the rules of my realtime database and enabled them as true still it is not giving me 12k data

{
"rules": {
".read": "true",
".write": "true",
}
}

Continue reading...