Object Storage via Fuse Filesystems

  • The application expecting files requires only moderate performance and does not have complicated dependencies on POSIX semantics.
  • You are using the filesystem adaptor for either reads or writes of the data, but not both. If your application is both reading and writing files, then best to use a real filesystem for the working data and copy only the final results to an object store.
  • You are using the adaptor because one part of your data pipeline is an application that expects files, whereas other applications expect objects.

Fuse Best Practices and Limitations

  • Do not expect ownership or permissions to work right, control permissions with your S3 key policies instead.
  • Do not use renames (‘mv’ command)
  • Lots of directory listing operations
  • Write to files sequentially and avoid random writes or appending to existing files.
  • Do not use symlinks or hard links
  • Do not expect consistency across clients, avoid sharing files through multiple clients with fuse mounts.
  • No really large files (1TB or larger)

Installation and Mounting Instructions

Basics

sudo apt install s3fs
sudo mkdir -p /mnt/fuse_s3fs && sudo chown $USER /mnt/fuse_s3fss3fs $BUCKETNAME /mnt/fuse_s3fs -o url=http://10.62.64.200 -o use_path_request_style
wget -N https://github.com/kahing/goofys/releases/latest/download/goofys
chmod a+x goofys
sudo mkdir -p /mnt/fuse_goofys && sudo chown $USER /mnt/fuse_goofys./goofys --endpoint=http://10.62.64.200 $BUCKETNAME /mnt/fuse_goofys
goofys <bucket:prefix> <mountpoint>
[fb]
type = s3
env_auth = true
region = us-east-1
endpoint = http://10.62.64.200
rclone --vfs-cache-mode writes mount fb:$BUCKETNAME /mnt/fuse_rclone &

Simulating a Directory Structure with Object Keys

Permissions

Angle 1: Reader

$ touch foo
touch: failed to close ‘foo’: Permission denied
$ ls
foo linux-5.12.13

$ ls
linux-5.12.13
$ cat pod.yaml
cat: pod.yaml: Permission denied

Angle 2: Writer

Enabling Deletions

Full Control

Bonus: Undo an Accidental Deletion

#!/usr/bin/python3
import boto3
import sys
FB_DATAVIP='10.62.64.200'if len(sys.argv) != 3:
print("Usage: {} bucketname key".format(sys.argv[0]))
sys.exit(1)
bucketname = sys.argv[1]
key = sys.argv[2]
s3 = boto3.resource('s3', endpoint_url='http://' + FB_DATAVIP)
kwargs = {'Bucket' : bucketname, 'Prefix' : key}
pageresponse = s3.meta.client.get_paginator('list_object_versions').paginate(**kwargs)for pageobject in pageresponse:
if 'DeleteMarkers' in pageobject.keys() and pageobject['DeleteMarkers'][0]['Key'] == key:
print("Undeleting s3://{}/{}".format(bucketname, key))
s3.ObjectVersion(bucketname, key, pageobject['DeleteMarkers'][0]['VersionId']).delete()
./s3-undelete.py phrex temp/pod.yaml
Undeleting s3://phrex/temp/pod.yaml

Performance Testing

  • All tests run against a small 9-blade FlashBlade
  • Client is 16 core, 96 GB DRAM, Ubuntu 20.04
  • Ramdisk used as the source or sink for write and read tests respectively
  • A direct S3 performance test gets 1.1GB/s writes and 1.5GB/s reads.
  • I also compare with a high-performance NFS filesystem, backed by the same FlashBlade, to illustrate the fuse-client overhead.
  • Tested goofys version 0.24.0, s3fs version v1.86, and rclone version 1.50.2

Throughput results

for i in {1..24}; do
cp /mnt/ramdisk/file_1G /mnt/$d/temp/file_1G_$i
done
for i in {1..24}; do
cp /mnt/ramdisk/file_1G /mnt/$d/temp/file_1G_$i &
done
wait

Metadata results

Comparing to AWS

Summary

  • Best to use for only one part of your data workflow, either simple writing or reading of files.
  • Do not rely on POSIX filesystem features like permissions, file renames, random overwrites, etc.
  • Prefer goofys as a fuse client choice because of superior performance

--

--

--

Data science, software engineering, hacking

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Creating Main Menu for App

The Principles of Agile Software (Part 1)

How to Use Gatsby as the Storefront of your Open Source Ecommerce Store

Neon Pixel 2 | Stream Overlay Package

Debug Arduino Mega code with Atmel-Ice via JTAG

Using BigCommerce Storefront APIs to Create Custom Product Display Page Experiences

Spark partitioning: the fine print

A Guide to Thorough Code Review

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Joshua Robinson

Joshua Robinson

Data science, software engineering, hacking

More from Medium

Comsysto’s Fast & Curious #4

Comsysto’s Fast & Curious title picture

Making Road Safety more inclusive thanks to AI

How I Reindex Elasticsearch Without Downtime

[Kernel] CGroups