Aws S3 Formdata Post Requires Exactly One File Upload Per Request

• Linguistic communication Support
• Node.js
• Directly to S3 File Uploads in Node.js

This article was contributed by Will Webberley

Will is a computer scientist and is enthused past most all aspects of the engineering science domain. He is specifically interested in mobile and social computing and is currently a researcher in this expanse at Cardiff Academy.

Straight to S3 File Uploads in Node.js

English language — 日本語に切り替える

Last updated March 09, 2022

Table of Contents

• Uploading straight to S3
• Overview
• Prerequisites
• Initial setup
• Directly uploading
• Running the app
• Summary

Web applications oftentimes require the ability to let users to upload files such as images, movies and archives. Amazon S3 is a pop and reliable storage choice for these files.

This article demonstrates how to create a Node.js application that uploads files directly to S3 instead of via a web application, utilising S3'south Cross-Origin Resources Sharing (CORS) back up. The Limited web framework is used to facilitate request-treatment in the examples below, but the procedure should be almost identical in whatsoever Node.js application.

Uploading directly to S3

A complete example of the code discussed in this article is available for direct use in this GitHub repository.

The principal advantage of direct uploading is that the load on your application'due south dynos would exist considerably reduced. Using app-side processes for receiving files and transferring to S3 can needlessly tie up your dynos and volition hateful that they will not exist able to answer to simultaneous web requests as efficiently.

The application uses client-side and app-side JavaScript for signing the requests. The actual upload is carried out asynchronously and so that you can decide how to handle your application'south flow after the upload has completed (for example, you may wish to redirect users to another page upon successful upload rather than a full page refresh).

An instance unproblematic business relationship-editing scenario is used as a guide for completing the diverse steps required to achieve the straight upload and to relate the awarding of this to a wider range of use-cases. More information on this scenario is provided afterward.

Overview

S3 is comprised of a set of buckets, each with a globally unique proper noun, in which individual files (known equally objects) and directories, can exist stored.

For uploading files to S3, you will need an Admission Key ID and a Surreptitious Access Key, which act as a username and countersign. The access cardinal account will need to have sufficient admission privileges to the target saucepan in order for the upload to be successful.

Delight see the S3 Article for more information on this, creating buckets and handling your hallmark keys.

In general, the method described in this commodity follows these elementary steps:

• A file is selected for upload by the user in their spider web browser;
• The user's browser makes a request to your spider web application on Heroku, which produces a temporary signature with which to sign the upload asking;
• The temporary signed request is returned to the browser in JSON format;
• The browser then uploads the file directly to Amazon S3 using the signed asking supplied by your Node.js awarding.

This guide includes data on how to implement the client-side and app-side code to form the complete arrangement. After post-obit the guide, you lot should take a working barebones arrangement, assuasive your users to upload files to S3. Nonetheless, information technology is usually worth adding actress functionality to help improve the security of the system and to tailor it for your ain particular uses. Pointers for this are mentioned in the appropriate parts of the guide.

The signature generation on the server uses AWS'southward official SDK, as explained after. Please see their documentation for information on the features of this SDK.

Prerequisites

• The Heroku CLI has been installed;
• Node.js has been installed;
• A Heroku application has been created for the current project;
• An AWS S3 bucket has been created;
• You have AWS hallmark keys that have write access to the bucket.

Initial setup

S3 setup

Yous will now need to edit some of the permissions properties of the target S3 bucket so that the final request has sufficient privileges to write to the bucket. In a web-browser, sign in to the AWS console and select the S3 section. Select the advisable saucepan and click the Permissions tab. A few options are now provided on this page (including Block public access, Admission Control Listing, Bucket Policy, and CORS configuration).

Firstly, ensure that "Cake all public access" is turned off, and in particular turn off "Cake public admission to buckets and objects granted through new access control lists" and "Block public access to buckets and objects granted through any access control lists" for the purposes of this project. Setting upwardly the bucket in this way allows us to read its contents without signed URLs, but this may not be suitable for services running in product.

Next, you will need to configure the bucket's CORS (Cantankerous-Origin Resources Sharing) settings, which will let your application to access content in the S3 saucepan. Each rule should specify a set up of domains from which access to the bucket is granted and also the methods and headers permitted from those domains.

For this to work in your application, click 'Edit' and enter the post-obit JSON for the bucket's CORS settings:

            [     {         "AllowedHeaders": [             "*"         ],         "AllowedMethods": [             "Go",             "Caput",             "Mail",             "PUT"         ],         "AllowedOrigins": [             "*"         ],         "ExposeHeaders": []     } ]                      

Click 'Save changes' and close the editor.

This tells S3 to allow whatever domain admission to the bucket and that requests tin comprise any headers, which is generally fine for testing. When deploying, you lot should change the 'AllowedOrigin' to only accept requests from your domain.

If you wish to use S3 credentials specifically for this application, then more keys tin can be generated in the AWS account pages. This provides further security, since you can designate a very specific fix of requests that this prepare of keys are able to perform. If this is preferable to you, so you will need to configure your IAM users in the Edit bucket policy option in your S3 bucket. In that location are various guides on AWS'southward web pages detailing how this can be achieved.

App setup

If your app hasn't yet been setup, then it is useful to do so at this stage. To get started, create a directory somewhere on your local automobile:

            $ mkdir NodeDirectUploader                      

At present create two further subdirectories of NodeDirectUploader/ to respectively comprise your HTML pages and back up files:

            $ cd NodeDirectUploader $ mkdir views $ mkdir public                      

Node'southward packet managing director, npm, should have been installed by default along with Node and can be used to handle the installation and updates of the required packages for your app. To begin this, run Node's interactive package setup tool in the root of your app directory:

            $ npm init                      

The tool will ask some questions about your app, including its proper noun, description, licensing, and version-control, and create a file called package.json in the app's root. This file uses your responses to maintain information about your app, which you tin can edit freehand equally you develop further.

The same file can be used to easily declare your app's dependencies, which will facilitate the deployment and share-ability of your app. To do so, edit bundle.json and add a "dependencies" JSON object to incorporate the following parcel dependencies:

            {   "name": "NodeDirectUploader",   "version": "0.0.1",    ...    "dependencies": {     "aws-sdk": "2.10",     "ejs": "2.x",     "express": "four.x"   } }                      

These dependencies can so be installed using npm:

            $ npm install                      

Utilize of these packages will become clear later, and installation of them in this way allows for greater control of your per-app dependencies as your apps grow.

Heroku setup

In order for your application to access the AWS credentials for signing upload requests, they will need to be added as configuration variables in Heroku:

            $ heroku config:set AWS_ACCESS_KEY_ID=thirty AWS_SECRET_ACCESS_KEY=yyy Adding config vars and restarting app... washed, v21     AWS_ACCESS_KEY_ID => thirty     AWS_SECRET_ACCESS_KEY => yyy                      

In addition to the AWS access credentials, set your target S3 saucepan'south name:

            $ heroku config:fix S3_BUCKET=zzz Adding config vars and restarting app... done, v21     S3_BUCKET     => zzz                      

Using config vars is preferable over configuration files for security reasons. Avoid placing passwords and access keys directly in your application'due south code or in configuration files. Please see the commodity Configuration and Config Vars for more than information.

Setting upwardly local environment variables for your app is useful for running and testing your app locally. For more than data, see the Set up your local surroundings variables section of the Heroku Local article. Information on launching your app locally is provided later in this article.

Think to add the .env file to your .gitignore, since this file should simply be used for local testing.

Direct uploading

The processes and steps required to accomplish a direct upload to S3 volition be demonstrated through the use of a simple profile-editing scenario for the purposes of this article. This case will involve the user being permitted to select an avatar paradigm to upload and enter some bones data to exist stored equally part of their account.

In this scenario, the following procedure will take place:

• The user is presented with a web folio, containing elements encouraging the user to choose an paradigm to upload as their avatar and to enter a username and their own proper name.
• An chemical element is responsible for maintaining a preview of the chosen prototype by the user. By default, and if no image is chosen for upload, a default avatar image is used instead (making the image-upload finer optional to the user in this scenario).
• When a user selects an prototype to be uploaded, the upload to S3 is handled automatically and asynchronously with the process described earlier in this article. The image preview is then updated with the selected paradigm once the upload is consummate and successful.
• The user is then gratuitous to movement on to filling in the rest of the information.
• The user then clicks the "submit" button, which posts the username, proper name and the URL of the uploaded epitome to the Node awarding to be checked and/or stored. If no image was uploaded by the user earlier the default avatar image URL is posted instead.

Setting up the client-side code

No third-political party lawmaking is required to complete the implementation on the client-side.

The HTML and JavaScript can at present be created to handle the file selection, obtain the asking and signature from your Node application, and then finally make the upload request.

Firstly, create a file called account.html in your application's views/ directory and populate the head and other necessary HTML tags appropriately for your application. In the body of this HTML file, include a file input and an element that will contain status updates on the upload progress. In addition to this, create a grade to allow the user to enter their username and full proper noun and a subconscious input chemical element to hold the URL of the called avatar image:

To see the completed HTML file, please see the appropriate lawmaking in the companion repository.

            <input blazon="file" id="file-input"> <p id="status">Delight select a file</p> <img id="preview" src="/images/default.png">  <form method="POST" action="/save-details">   <input type="subconscious" id="avatar-url" proper noun="avatar-url" value="/images/default.png">   <input type="text" proper noun="username" placeholder="Username"><br>   <input type="text" name="full-proper noun" placeholder="Full name"><br><br>   <input type="submit" value="Update profile"> </form>                      

The #preview chemical element initially holds a default avatar image (which would become the user's avatar if a new image is not called), and the #avatar-url input maintains the current URL of the user's chosen avatar epitome. Both of these are updated by the JavaScript, discussed below, when the user selects a new avatar.

Thus when the user finally clicks the submit push button, the URL of the avatar is submitted, forth with the username and full name of the user, to your desired endpoint for server-side handling.

The customer-side lawmaking is responsible for achieving 2 things:

1. Retrieve a signed request from the app with which the paradigm tin be PUT to S3
2. Actually PUT the image to S3 using the signed asking

JavaScript's XMLHttpRequest objects can be created and used for making asynchronous HTTP requests.

To reach this, commencement create a <script> block and write some lawmaking that listens for changes in the file input, in one case the document has loaded, and starts the upload process.

            (() => {   document.getElementById("file-input").onchange = () => {     const files = certificate.getElementById('file-input').files;     const file = files[0];     if(file == nothing){       return warning('No file selected.');     }     getSignedRequest(file);   }; })();                      

The code too determines the file object itself to exist uploaded. If one has been selected properly, it proceeds to call a function to obtain a signed PUT request for the file. Side by side, therefore, write a part that accepts the file object and retrieves an appropriate signed request for it from the app.

            function getSignedRequest(file){   const xhr = new XMLHttpRequest();   xhr.open('Go', `/sign-s3?file-proper name=${file.proper noun}&file-type=${file.blazon}`);   xhr.onreadystatechange = () => {     if(xhr.readyState === 4){       if(xhr.status === 200){         const response = JSON.parse(xhr.responseText);         uploadFile(file, response.signedRequest, response.url);       }       else{         alert('Could not get signed URL.');       }     }   };   xhr.send(); }                      

If the proper noun (file.proper noun) and/or mime type (file.type) of the file you upload contains special characters (such as spaces), and then they should be encoded outset (east.g. encodeURIComponent(file.name)).

The in a higher place function passes the file'south name and mime type as parameters to the GET request since these are needed in the structure of the signed asking, equally will be covered later in this article. If the retrieval of the signed request was successful, the function continues by calling a function to upload the bodily file:

            function uploadFile(file, signedRequest, url){   const xhr = new XMLHttpRequest();   xhr.open('PUT', signedRequest);   xhr.onreadystatechange = () => {     if(xhr.readyState === 4){       if(xhr.condition === 200){         document.getElementById('preview').src = url;         document.getElementById('avatar-url').value = url;       }       else{         warning('Could not upload file.');       }     }   };   xhr.transport(file); }                      

This role accepts the file to exist uploaded, the signed request, and generated URL representing the eventual retrieval URL of the avatar image. The latter two arguments volition exist returned as function of the response from the app. The office, if the request to S3 is successful, and then updates the preview chemical element to the new avatar prototype and stores the URL in the hidden input so that information technology can exist submitted for storage in the app.

Now, once the user has completed the rest of the form and clicked submit, the name, username, and avatar image tin can all exist posted to the same endpoint.

If y'all find that the page isn't working as y'all intend later on implementing the organisation, then consider using panel.log() to record whatsoever errors that are revealed inside the onreadystatechange office and apply your browser'southward error console to assist diagnose the trouble.

Information technology is good practice to inform the user of any prolonged activity in any grade of application (web- or device-based) and to display updates on changes. Therefore a loading indicator could exist displayed betwixt selecting a file and the upload being completed. Without this sort of information, users may suspect that the page has crashed, and could attempt to refresh the page or otherwise disrupt the upload process.

Setting up the app-side Node lawmaking

This section discusses the use of Node.js for generating a temporary signature with which the upload asking tin can be signed. This temporary signature uses AWS authentication credentials (the access key and secret fundamental) as a basis for the signature, merely users will non have directly admission to this information. Later on the signature has expired, then upload requests with the aforementioned signature volition not be successful.

To encounter the completed Node file, please see the appropriate code in the companion repository.

Start by creating your main application file, app.js, in the root of your application directory and set up your skeleton application appropriately:

            const express = require('express'); const aws = crave('aws-sdk');  const app = express(); app.set('views', './views'); app.utilize(express.static('./public')); app.engine('html', require('ejs').renderFile); app.listen(process.env.PORT || 3000);  const S3_BUCKET = process.env.S3_BUCKET;                      

In some scenarios, it may be necessary to check that the environment's PORT var is a number by using Number(process.env.PORT).

The packages installed with npm are imported at the tiptop of the application. The Limited app is then prepare-upward and finally the saucepan name is loaded from the surround.

You should now configure your AWS region. To do so, update the imported aws object. For case:

            aws.config.region = 'european union-west-1';                      

Retrieve to use the region that your target bucket resides in. If y'all demand it, use this folio to discover your region.

Adjacent, in the same file, yous volition need to create the views responsible for returning the right data back to the user's browser when requests are made to various URLs. Inside the app.js file, define the view for requests to /account to render the folio business relationship.html, which contains the course for the user to complete:

            app.get('/account', (req, res) => res.render('account.html'));                      

Now create the view, in the same JavaScript file, that is responsible for generating and returning the signature with which the customer-side JavaScript can upload the image. This is the beginning request fabricated past the customer earlier attempting an upload to S3. This view responds with requests to /sign-s3:

            app.get('/sign-s3', (req, res) => {   const s3 = new aws.S3();   const fileName = req.query['file-proper name'];   const fileType = req.query['file-type'];   const s3Params = {     Bucket: S3_BUCKET,     Key: fileName,     Expires: 60,     ContentType: fileType,     ACL: 'public-read'   };    s3.getSignedUrl('putObject', s3Params, (err, data) => {     if(err){       panel.log(err);       render res.end();     }     const returnData = {       signedRequest: data,       url: `https://${S3_BUCKET}.s3.amazonaws.com/${fileName}`     };     res.write(JSON.stringify(returnData));     res.cease();   }); });                      

This lawmaking uses the aws-sdk module to create a signed URL that the browser can use to make a PUT request to S3. In addition, the prospective URL of the object to be uploaded is produced as a combination of the S3 bucket name and the object name. This URL and the signed request are and then returned to the browser in JSON format.

The Expires parameter describes the number of seconds for which the signed URL will exist valid for. In some circumstances, such as when uploading big files, a larger value may be more appropriate in lodge to extend the validity of the signed URL.

Initialising the s3 object automatically loads the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY variables that were prepare into the surround earlier.

You may wish to assign another, customised name to the object instead of using the i that the file is already named with, which is useful for preventing accidental overwrites in the S3 bucket. This proper name could exist related to the ID of the user's account, for instance. If not, you should provide some method for properly quoting the name in instance there are spaces or other awkward characters nowadays. In addition, this is the phase at which yous could provide checks on the uploaded file in gild to restrict access to sure file types. For example, a uncomplicated bank check could exist implemented to allow only .png files to proceed across this bespeak.

Finally, in app.js, create the view responsible for receiving the account information later on the user has uploaded an avatar, filled in the form, and clicked submit:

            app.mail('/save-details', (req, res) => {   // TODO: Read POSTed form data and do something useful });                      

This office is currently just a stub that you'll need to complete in lodge to allow the app to read and store the submitted profile information and to correctly associate it with the rest of the user's business relationship details.

Running the app

Everything should now be in place to perform the directly uploads to S3. To test the upload, salve any changes and utilise heroku local to kickoff the awarding:

Y'all volition demand a Procfile for this to be successful. See Getting Started with Node.js on Heroku for more information. Besides remember to correctly set your environment variables on your own machine earlier running the application locally.

            $ heroku local 15:44:36 web.1  | started with pid 12417                      

Printing Ctrl+C to return to the prompt. If your awarding is returning 500 errors (or other server-based issues), then get-go your server in debug style and view the output in the Concluding emulator to help set up your problem:

            $ DEBUG=limited:* node app.js                      

Summary

This article covers uploading to Amazon S3 straight from the browser using Node.js to temporarily sign the upload request. Although the guide and companion lawmaking focuses on the Express framework, the thought should easily carry over to other Node applications.

lofgrensquing.blogspot.com

Source: https://devcenter.heroku.com/articles/s3-upload-node

Share this post