Importing Splunk Data

You can import data from your Splunk instance into Scalyr easily and efficiently, retaining context. The following article will show you how to perform this import. There are two methods to doing this. 

High Volume

If you are sending data at high volume >1GB, it is best to place the files in S3 and have the scalyr servers process the data. 

Prerequisites
Splunk

S3

Scalyr

Instructions

1. Setup and S3 Bucket

2. Create an SQS queue, and configure your S3 bucket to publish new-object notifications to the queue.
3. Configure IAM

4. Setup an S3 Monitor.

Your monitor config will look like this

  monitors: [
    {
      type: "s3Bucket",
      region: "us-east-1",
      roleToAssume: "arn:aws:iam::account-id:role/role-name-with-path",
      queueUrl: "https://sqs.us-east-1.amazonaws.com/nnnnnnnnnnnn/scalyr-s3-bucket-foo"
      fileFormat: "text_gzip",
      hostname: "foo",
      parser: "foo"
    }
  ]

5. Add the parser

// Parser for log files containing JSON records.

{
attributes: {
// Tag all events parsed with this parser so we can easily select them in queries.
dataset: "splunk"
},

formats: [
{
format: "${parse=json}$", repeat: true
rewrites: [
{
input: "result_indextime",
output: "timestamp",
match: ".*",
replace: "$0"
}, 
{
input: "resultSource",
output: "logfile",
match: ".*",
replace: "$0"
}
{
input: "resultHost",
output: "serverHost",
match: ".*",
replace: "$0"
}
]
}
]
}

6. export the Splunk file 

cd /opt/splunk/bin
./splunk search "query" -output json -maxout 0 > scalyr_testuat_s3.log

7. GZip it into multiple files 

split -l 40000 "scalyr_testuat_s3.log" "splunk3.gz.part-" && gzip -9 splunk3.gz.part*

8. Add to the s3 bucket

Low volume

If you are importing a low volume of data < 1GB, you can export from the Splunk CLI or UI as a JSON object and configure the Scalyr Agent to stream it to Scalyr.

Prerequisites
Splunk 
Scalyr Agent
Access to Splunk CLI
Full permission in Scalyr account

Objectives 
Migrate this Splunk data 
To Scalyr 

Instructions

You can add your Splunk data into your Scalyr account by performing the following steps.

Splunk

1. Define the query in Splunk for data you want to export. Example

index=* 

2. Navigate to your Splunk instance and invoke the CLI (Here is more information on the Splunk CLI)

cd /opt/splunk/bin
./splunk search "query" -output json -maxout 0

If this is working, great, let's install Scalyr

Scalyr

Agent
1. Install the Scalyr agent. 

curl -sO https://www.scalyr.com/install-agent.sh
sudo bash ./install-agent.sh --set-api-key "api-key"
sudo scalyr-agent-2 start

2. Configure the Scalyr Agent

vim /etc/scalyr-agent-2/agent.json

Configure the agent to point to a directory and configure the json parser. 

{
path: "/tmp/splunk/*.log",
attributes: {parser:"splunk-json"}
}

Parser
3. Configure the parser. Navigate to your parsers page
4. Add parser
5. Name (splunk-json)
6. Paste the default splunk-json parser

// Parser for log files containing JSON records.

{
attributes: {
// Tag all events parsed with this parser so we can easily select them in queries.
dataset: "splunk"
},

formats: [
{
     format: "${parse=json}$", repeat: true
rewrites: [
     {
        input: "result_indextime",
        output: "timestamp",
        match: ".*",
        replace: "$0"
     }, 
     {
       input: "resultSource",
       output: "logfile",
       match: ".*",
       replace: "$0"
     }
     {
       input: "resultHost",
       output: "serverHost",
       match: ".*",
       replace: "$0"
     }
  ]
  }
  ]
}

Once the agent is installed and configured and the parser is installed and configured, we are ready to export.

Export

1. Export data from Splunk to Scalyr by entering the command from above and pointing it to the file we just configured.
(For the Splunk search command, set the -maxout value to 0 for unlimited output. If there are issues with the dataset, you can use the dump command)

./splunk search "index=_internal earliest=09/14/2015:23:59:00 latest=09/16/2015:01:00:00 " -output rawdata -maxout 0 > /tmp/splunk/output.log 

2. Navigate to your scalyr account and you should see your data populate.