As an IT architect who works on big data and cloud platform solutions, one common challenge is uploading files, especially files larger than 1TB. The problem becomes exasperated when there is no browser plug-in for a reliable and secure upload.
We solved this problem inside NetApp IT by implementing a highly scalable, reliable, and secure upload solution using StorageGRID®, a software-defined, object-based storage solution from NetApp. Because StorageGRID leverages S3, we were able to secure and upload over https with pre-assigned URLS for added security. There were no client-side plug-ins as with the UDP transport layer communication protocol. As a TCP-based solution, the file uploads were fast and reliable. Our solution supports up to 2 TB with multi-part upload features including pause, resume, and cancel features. It was fast and easy to implement with well known technology stacks like S3 SDK, Angular, and Node.
The best way to quickly upload large objects reliably is by splitting files into smaller 100MG segments using multipart uploading. Here’s some sample code to show how it was done.
Also, I recommend you store the credentials locally in ~/.aws/credentials or in a properties file—and never in code.
aws_access_key_id = PPIF2PCJK7KXXXXXXXXX
aws_secret_access_key = ZM8L9BIRkK5UTjmLm6NXxxMXXXxxXxxxxXXXxXxx
How it works
As shown in the process diagram below, a user selects his nearest geographic location from a dropdown menu. The location choices are North America (NA), Europe, Middle East & Africa (EMEA), or Asia Pacific (APAC). For NA and EMEA, the file will be stored in the location’s corresponding StorageGRID. For APAC, the file will be stored in AWS, then later synced between AWS US-East region and StorageGRID. Once the file is in StorageGRID, it will be routed accordingly for core processing or data processing.
The NetApp-on-NetApp blogs feature advice from subject matter experts from NetApp IT who share their real experiences using NetApp’s industry-leading data management solutions to support business goals. Visit www.NetAppIT.com to learn more.