Considerations When Using Amazon S3 as a File SystemNow that we’ve looked at the advantages of using Amazon S3 as a mounted drive, we should consider some of the points before using this approach.When you are using Amazon S3 as a file system, you might observe a network delay when performing IO centric operations such as creating or moving new folders or files. The performance depends on your network speed as well distance from Amazon S3 storage region.Since Amazon S3 is not designed for atomic operations, files cannot be modified, they have to be completely replaced with modified files. This doesn’t impact your application as long as it’s creating or deleting files; however, if there are frequent modifications to a file, that means replacing the file on Amazon S3 repeatedly, which results in multiple put requests and, ultimately, higher costs.As files are transferred via HTTPS, whenever your application tries to access the mounted Amazon S3 bucket first time, there is noticeable delay. Future or subsequent access times can be delayed with local caching.Each object has a maximum size of 5GB. When considering costs, remember that Amazon S3 charges you for performing IO operations. The overall object might cost less, comparing General Purpose SSD to IOPS storage for example, but the cost for IO could be higher.NetApp can help cut Amazon AWS storage costs with powerful storage efficiencies and other benefits, more of which we’ll discuss below.NetApp Solutions for Using Amazon S3 for File System StorageOne way that NetApp offers you a shortcut in using Amazon S3 for file system storage is with Cloud Volumes ONTAP (formerly ONTAP Cloud). Cloud Volumes ONTAP has a number of storage optimization and data management efficiencies, and the one that makes it possible to use Amazon S3 as a file system is data tiering.
With data tiering to Amazon S3 Cloud Volumes ONTAP can send infrequently-accessed files to S3 (the cold data tier), where prices are lower than on Amazon EBS. When the data is needed for active use, it can be brought back up to Amazon EBS (the hot data tier), where the costs are higher but the performance is much better than on Amazon S3.
There is another way to leverage Amazon S3 as part of a file system that you run in AWS. With Cloud Volumes ONTAP data tiering, you can create an NFS/CIFS share on Amazon EBS which has back-end storage in Amazon S3. From this S3-backed file share you could mount from multiple machines at the same time, effectively treating it as a regular file share. (Note that in this case that you would only be able to access the files over NFS/CIFS from Cloud Volumes ONTAP and not through Amazon S3.) The savings of storing infrequently used file system data on Amazon S3 can be a huge cost benefit over the native AWS file share solutions.
It is possible to move and preserve a file system in Amazon S3, from where the file system would remain fully usable and accessible. However, one consideration is how to migrate the file system to Amazon S3. One option would be to use Cloud Sync. Cloud Sync is NetApp’s solution for fast and easy data migration, data synchronization, and data replication between NFS and CIFS file shares, Amazon S3, NetApp StorageGRID® Webscale Appliance, and more. Cloud Sync can also migrate and transfer data to and from Amazon EFS, AWS’s native file share service.ConclusionFrom the steps outlined above you can see that it’s simple to mount S3 bucket to EC2 instances, servers, laptops, or containers.
Mounting Amazon S3 as drive storage can be very useful in creating distributed file systems with minimal effort, and offers a very good solution for media content-oriented applications. But since you are billed based on the number of GET, PUT, and LIST operations you perform on Amazon S3, mounted Amazon S3 file systems can have a significant impact on costs, if you perform such operations frequently.
This mechanism can prove very helpful when scaling up legacy apps, since those apps run without any modification in their codebases. Having a shared file system across a set of servers can be beneficial when you want to store resources such as config files and logs in a central location. However, AWS does not recommend this due to the size limitation, increased costs, and decreased IO performance. But for some users the benefits of added durability in a distributed file system functionality may outweigh those considerations.With NetApp, you might be able to mitigate the extra costs that come with mounting Amazon S3 as a file system with the help of Cloud Volumes ONTAP and Cloud Sync. When used in support of mounting Amazon S3 as a file system you get added benefits, such as Cloud Volumes ONTAP’s cost-efficient data storage and Cloud Sync’s fast transfer capabilities, lowering the overall amount you spend for AWS services.