S3 Imports - In-Depth Overview

Introduction

This article provides an in-depth overview of how to configure S3 imports. It applies to any of the DataSet integrations that use a monitor with type: "s3Bucket" to download logs from S3. Some examples would include Import Logs from an S3 Bucket or Import Amazon ELB Access Logs.

Please note that this page was designed to complement the existing DataSet documentation. The included examples were intended to demonstrate a functioning S3 import setup. Your implementation may vary due to the requirements of your platform.

Configuration

Please refer to the "Appendix: Create IAM Role" section of the DataSet integration that you will be implementing. As a prerequisite, please complete these steps first, then compare your settings against the examples below if unexpected behavior occurs. Please use the following instructions as a reference for the steps below.

Step 1

Create an IAM role that provides access to our (DataSet) AWS account as a trusted entity (step 6). For additional security, a unique 56 character identifier that is associated with your DataSet account will be used to restrict access (step 7). The unique identifier is obtained from the DataSet documentation page; you need to be signed in to your DataSet account in order to view it. For example,

From the IAM page, click "Roles" -> Select the role you created -> "Trust Relationships"

The JSON configuration should be similar to the following:

The sts:ExternalId field is the DataSet account identifier we mentioned above

Step 2

Additional permissions are granted to the role so that it can access your SQS queue and S3 bucket (steps 9-11).

From the IAM page, click "Roles" -> Select the role you created -> "Permissions" -> Select the policy you created

The policy will be similar to:

Of note,

• Be sure to confirm that the Action field matches the following SQS / S3 values:
  

  "Action": [
    "sqs:DeleteMessage",
    "s3:GetObject",
    "sqs:ReceiveMessage",
    "sqs:GetQueueAttributes"
  ],

• Please review the format of the S3 bucket and SQS queue
  • An example S3 ARN is
    

     "arn:aws:s3:::<Your S3 Bucket>/*",

  • SQS ARN includes the AWS region, your numerical AWS account number, and the SQS queue name. For example,
    

    "arn:aws:sqs:<AWS Region>:<Your AWS Account Number>:<Your SQS Queue>"

Step 3

SQS Queue

Access the SQS queue page and select the SQS queue that you created.

Click the "Edit" button and review the "Access Policy". Our example policy is as follows:

{
  "Version": "2012-10-17",
  "Id": "example-ID",
  "Statement": [
    {
      "Sid": "example-statement-ID",
      "Effect": "Allow",
      "Principal": {
        "Service": "s3.amazonaws.com"
      },
      "Action": "sqs:SendMessage",
      "Resource": "arn:aws:sqs:<AWS Region>:<Your AWS Account Number>:<Your SQS Queue>",
      "Condition": {
        "StringEquals": {
          "aws:SourceAccount": "<Your AWS Account Number>"
        },
        "ArnLike": {
          "aws:SourceArn": "arn:aws:s3:*:*:<Your S3 Bucket>"
        }
      }
    }
  ]
}

Step 4

S3 Bucket

From the S3 page, click the bucket that you created. Click the "Properties" tab -> Scroll down to "Event Notifications" -> Click "Create Event Notification"

Only two sections will need to be updated:

Select "Put" and "Post" from the "Event Types" section

Choose "SQS queue" and enter your SQS queue ARN in the "Destination" section

Step 5

The DataSet S3 monitor configuration will be similar to:

...
  {
    type: "s3Bucket",
    region: "<AWS Region>",
    roleToAssume: "arn:aws:iam::<AWS Account Number>:role/<AWS Role>",
    queueUrl: "https://sqs.<AWS Region>.amazonaws.com/<AWS Account Number>/<SQS Queue>",
    fileFormat: "<format>",
    hostname: "<hostname>",
    parser: "<parser>" 
  },
...

Please see this page for all available options

Once established, it can take in upwards of 15 minutes before logs are available.

Check for New Logs

As logs are written to the S3 bucket, they will be imported to your DataSet account once per minute. From your DataSet account, run a search query of:

tag contains 's3'

And you should see results similar to the following, which indicate that logs are successfully being imported from S3:

Similarly, you could search for S3 logs directly with a query of

source='s3Bucket'

for the imported log events