[Python] AWS DynamoDB download speed with Boto3

I am using python/Boto3 to download logs from a health tracking device. I wrote a python app that lets you run a paginated query for a a given user & timeframe:


def paginated_query(self, tstart, tend, user):
paginator = self.datapoints_table.meta.client.get_paginator('query')
page_iterator = paginator.paginate(
TableName=self.table_name,
KeyConditionExpression= Key('userId').eq(user_id) & Key('timestamp').between(tstart, tend)
)



The query itself works and executes within a few milliseconds.

Then, to download the data and process it with python, I convert the generator into a list:

page_iterator = self.paginated_query(tstart, tend, user)

log_list = []
for p in page_iterator:
items = p.get('Items', []) # download the data
log_list.extend(items)


This last snippet takes a very long time depending on how much data I am trying to download. I tried to process pages in parallel, instead of sequentially in a for loop, but it makes no difference in the execution time. I measured a data download rate from DynamoDB/AWS at 1.8MB/s max, which seems much slower than what my connection could achieve. I can only assume there is a bottleneck on AWS side.

Does anyone know how to make the data download faster?

Continue reading...