Overriding standard file extensions processing in Splunk

Image

Few months ago, we came across an interesting problem: proprietary software would dump binary logs in private format that we would need to parse and forward over to Splunk in readable format. One issue was that the software would use .dat file extension that is well-known to Splunk. A combination of these two factors created a situation where our inputs.conf and props.conf changes would not work as Splunk would "know better" and ignore our configurations:

props.conf:
[source::/opt/acme/log/traffic/*/*]
sourcetype=acme_traffic
NO_BINARY_CHECK=true
invalid_cause = archive
unarchive_cmd = /opt/splunkforwarder/bin/acme2text.py

inputs.conf
[monitor:///opt/acme/log/traffic/*/*]
index = main
sourcetype =acme_traffic
_meta = env::NO_ENV buildno::NO_BUILD_NO product::ACME-TRAFFIC

With these settings, unarchive_cmd would never get executed and raw content of binary .dat files is indexed without any field extractions. After few frustrating hours spent on reading documentation, Splunk forums and StackOverflow, as well as tinkering and tweaking, something interesting showed up in Splunk debug logs - apparently, Splunk's internal processing has a priority of 10000 (well, it IS over 9000!)

Solution to the problem then became quite straightforward:

props.conf:
[source::/opt/acme/log/traffic/*/*]
sourcetype=acme_traffic
NO_BINARY_CHECK=true
invalid_cause = archive
unarchive_cmd = /opt/splunkforwarder/bin/acme2text.py
priority = 10002

inputs.conf
[monitor:///opt/acme/log/traffic/*/*]
index = main
sourcetype =acme_traffic
_meta = env::NO_ENV buildno::NO_BUILD_NO product::ACME-TRAFFIC

Setting priority value to anything higher than 10000 will override Splunk internal configuration and force it to use user-defined processing.

Egor Cole

Egor brings extensive software development expertise and has a strong track record of managing complex projects through to successful completion. Egor is often recommended as an IT professional who can turn complex processes into simplified IT systems that improve business operations, create positive ROI’s and have a lasting impact.