You can import data from your Splunk instance into Scalyr easily and efficiently, retaining context. The following article will show you how to perform this import. There are two methods to doing this.
High Volume
If you are sending data at high volume >1GB, it is best to place the files in S3 and have the scalyr servers process the data.
Prerequisites
Splunk
S3
Scalyr
Instructions
2. Create an SQS queue, and configure your S3 bucket to publish new-object notifications to the queue.
3. Configure IAM
4. Setup an S3 Monitor.
Your monitor config will look like this
monitors: [ { type: "s3Bucket", region: "us-east-1", roleToAssume: "arn:aws:iam::account-id:role/role-name-with-path", queueUrl: "https://sqs.us-east-1.amazonaws.com/nnnnnnnnnnnn/scalyr-s3-bucket-foo" fileFormat: "text_gzip", hostname: "foo", parser: "foo" } ]
5. Add the parser
// Parser for log files containing JSON records.
{
attributes: {
// Tag all events parsed with this parser so we can easily select them in queries.
dataset: "splunk"
},
formats: [
{
format: "${parse=json}$", repeat: true
rewrites: [
{
input: "result_indextime",
output: "timestamp",
match: ".*",
replace: "$0"
},
{
input: "resultSource",
output: "logfile",
match: ".*",
replace: "$0"
}
{
input: "resultHost",
output: "serverHost",
match: ".*",
replace: "$0"
}
]
}
]
}
6. export the Splunk file
cd /opt/splunk/bin
./splunk search "query" -output json -maxout 0 >scalyr_testuat_s3.log
7. GZip it into multiple files
split -l 40000 "scalyr_testuat_s3.log" "splunk3.gz.part-" && gzip -9 splunk3.gz.part*
8. Add to the s3 bucket
Low volume
If you are importing a low volume of data < 1GB, you can export from the Splunk CLI or UI as a JSON object and configure the Scalyr Agent to stream it to Scalyr.
Prerequisites
Splunk
Scalyr Agent
Access to Splunk CLI
Full permission in Scalyr account
Objectives
Migrate this Splunk data
To Scalyr
Instructions
You can add your Splunk data into your Scalyr account by performing the following steps.
Splunk
1. Define the query in Splunk for data you want to export. Example
index=*
2. Navigate to your Splunk instance and invoke the CLI (Here is more information on the Splunk CLI)
cd /opt/splunk/bin
./splunk search "query" -output json -maxout 0
If this is working, great, let's install Scalyr
Scalyr
Agent
1. Install the Scalyr agent.
curl -sO https://www.scalyr.com/install-agent.sh
sudo bash ./install-agent.sh --set-api-key "api-key"
sudo scalyr-agent-2 start
2. Configure the Scalyr Agent
vim /etc/scalyr-agent-2/agent.json
Configure the agent to point to a directory and configure the json parser.
{
path: "/tmp/splunk/*.log",
attributes: {parser:"splunk-json"}
}
Parser
3. Configure the parser. Navigate to your parsers page
4. Add parser
5. Name (splunk-json)
6. Paste the default splunk-json parser
// Parser for log files containing JSON records.
{
attributes: {
// Tag all events parsed with this parser so we can easily select them in queries.
dataset: "splunk"
},
formats: [
{
format: "${parse=json}$", repeat: true
rewrites: [
{
input: "result_indextime",
output: "timestamp",
match: ".*",
replace: "$0"
},
{
input: "resultSource",
output: "logfile",
match: ".*",
replace: "$0"
}
{
input: "resultHost",
output: "serverHost",
match: ".*",
replace: "$0"
}
]
}
]
}
Once the agent is installed and configured and the parser is installed and configured, we are ready to export.
Export
1. Export data from Splunk to Scalyr by entering the command from above and pointing it to the file we just configured.
(For the Splunk search command, set the -maxout value to 0 for unlimited output. If there are issues with the dataset, you can use the dump command)
./splunk search "index=_internal earliest=09/14/2015:23:59:00 latest=09/16/2015:01:00:00 " -output rawdata -maxout 0 > /tmp/splunk/output.log
2. Navigate to your scalyr account and you should see your data populate.
Comments
0 comments
Please sign in to leave a comment.