How to create an S3 batch job

### Overview Nick Rimmer demonstrates how to configure and run an Amazon Web Services (AWS) S3 batch job to perform large-scale batch operations on S3 objects. In this walkthrough, a job is defined as the basic unit of work containing all necessary execution parameters, and the practical example demonstrates copying five test files from a source bucket (`techsnipsUK`) to a destination bucket (`mytechsnip bucket`).

### Key Topics - **S3 Inventory Configuration**: Setting up an inventory on the source bucket to generate an object list in CSV format, creating the destination bucket policy, and locating the resulting `manifest.json`. - **Batch Operations Capabilities**: Understanding supported operations such as PUT (copy), AWS Lambda invocation, tag replacement, access control list (ACL) replacement, and object restoration. - **Job Configuration & Options**: Configuring the manifest format, choosing operational targets, acknowledging overwrite behavior, selecting storage classes and encryption options, and preserving object metadata and tags. - **Execution, Permissions, and Reporting**: Utilizing an Identity and Access Management (IAM) role (`S3 batch role`), submitting the job, tracking status, confirming and running the job, and inspecting the completion report.

### Technical Details To initiate S3 batch operations, an inventory of the source bucket must first be established. Under bucket management, an inventory report is named and configured with a destination bucket (`techsnipsUK inventory`), outputting in default CSV format for current object versions with default fields and no encryption. The speaker notes it can take up to 48 hours to deliver the first report and that a bucket policy is automatically created on the inventory destination bucket to allow writing inventory files.

Once generated, the inventory's `manifest.json` file serves as the input manifest for creating the batch job. In the AWS console under batch operations, the user selects the manifest format (S3 inventory report) and points to the manifest path in the EU London region. When selecting the operation, available choices include PUT (copy), invoking a Lambda function, replacing tags, replacing ACLs, or restoring files. For the demonstration, the PUT operation is selected targeting `mytechsnip bucket`.

The demonstration proceeds with versioning unenabled, requiring an acknowledgment that existing objects with the same name will be overwritten. The storage class is set to Standard, encryption is left as none (with S3-managed keys or KMS noted as alternatives), ACL permissions remain identical to source objects, and options to copy existing tags and metadata are checked. Priority and description are set, and a completion report covering all tasks is directed to the inventory bucket. An IAM role named `S3 batch role` with full S3 access is specified to grant necessary execution permissions. After reviewing details, the job is created, confirmed, and run. Upon completion, the job status shows 100% completion across all five objects, and the downloaded completion report verifies individual file names, bucket locations, and successful status.

### Takeaways S3 batch operations allow administrators to run operations across defined lists of objects without writing custom orchestration scripts. Creating a batch copy job requires an inventory manifest (such as `manifest.json` generated from an S3 inventory report), an appropriate IAM role with sufficient permissions, explicit handling of overwrite acknowledgments when versioning is off, and optional configuration of completion reporting for operational auditing.