Streaming uploads

Cloud Storage supports streaming data to a bucket without requiring that the data first be saved to a file. This is useful when you want to upload data but don't know the final size at the start of the upload, such as when generating the upload data from a process, or when compressing an object on-the-fly.

Using checksum validation when streaming

Because a checksum can only be supplied in the initial request of an upload, it's often not feasible to use Cloud Storage's checksum validation when streaming. It's recommended that you always use checksum validation, and you can manually do so after a streaming upload completes; however, validating after the transfer completes means that any corrupted data is accessible during the time it takes to confirm the corruption and remove it.

If you require checksum validation prior to the upload completing and the data becoming accessible, then you shouldn't use a streaming upload. You should use a different upload option that performs checksum validation prior to finalizing the object.

Required roles

To get the permissions that you need to stream uploads, ask your administrator to grant you one of the following roles:

• For uploads that include an Object Retention Lock, ask your administrator to grant you the Storage Object Admin (roles/storage.objectAdmin) IAM role for the bucket.

• For all other cases, ask your administrator to grant you the Storage Object User (roles/storage.objectUser) IAM role for the bucket.

These predefined roles contain the permissions required to stream uploads to Cloud Storage. To see the exact permissions that are required, expand the Required permissions section:

You can also get these permissions with other predefined roles or custom roles.

For information about granting roles on buckets, see Set and manage IAM policies on buckets.

Stream an upload

The following examples show how to perform a streaming upload from a process to a Cloud Storage object:

namespace gcs = ::google::cloud::storage;
using ::google::cloud::StatusOr;
[](gcs::Client client, std::string const& bucket_name,
   std::string const& object_name, int desired_line_count) {
  std::string const text = "Lorem ipsum dolor sit amet";
  gcs::ObjectWriteStream stream =
      client.WriteObject(bucket_name, object_name);

  for (int lineno = 0; lineno != desired_line_count; ++lineno) {
    // Add 1 to the counter, because it is conventional to number lines
    // starting at 1.
    stream << (lineno + 1) << ": " << text << "\n";
  }

  stream.Close();

  StatusOr<gcs::ObjectMetadata> metadata = std::move(stream).metadata();
  if (!metadata) throw std::move(metadata).status();
  std::cout << "Successfully wrote to object " << metadata->name()
            << " its size is: " << metadata->size()
            << "\nFull metadata: " << *metadata << "\n";
}

using Google.Cloud.Storage.V1;
using System;
using System.IO;

public class UploadFileSample
{
    public void UploadFile(
        string bucketName = "your-unique-bucket-name",
        string localPath = "my-local-path/my-file-name",
        string objectName = "my-file-name")
    {
        var storage = StorageClient.Create();
        using var fileStream = File.OpenRead(localPath);
        storage.UploadObject(bucketName, objectName, null, fileStream);
        Console.WriteLine($"Uploaded {objectName}.");
    }
}