Improving Python S3 Client Performance with Rust

Experimental Results

Result 1 — GET Large Objects

Result 2 — GETs on FlashBlade vs AWS

Result 3 — Listing Objects

Code Walkthrough

maturin build --release
python3 -m pip install fasts3/target/wheels/*.whl
import boto3
import fasts3
s3r = boto3.resource('s3', endpoint_url=ENDPOINT_URL) # boto3
s = fasts3.FastS3FileSystem(endpoint=ENDPOINT_URL) # fasts3 (rust)
# boto3 download_fileobj()
bytes_buffer = io.BytesIO()
s3r.meta.client.download_fileobj(Bucket=BUCKET, Key=SMALL_OBJECT, Fileobj=bytes_buffer)
# fasts3 get_objects
contents = s.get_objects([BUCKETPATH])
pub fn new(endpoint: String) -> FastS3FileSystem
pub fn ls(&self, path: &str) -> PyResult<Vec<String>>
pub fn get_objects(&self, py: Python, paths: Vec<String>) -> PyResult<PyObject>

What About Small Objects?

Summary

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store