Splunk HTTP Event Collector provider

Operator runbook for sending IdentityMesh audit events straight to Splunk Enterprise or Splunk Cloud through the HTTP Event Collector (HEC). It is one of the provider kinds described in siem-providers.md, which explains the model, the delivery guarantees and the event itself. Read that page first.

The provider sends one HEC event per audit event, with your index, source and source type, and the IdentityMesh event as the JSON body. With indexer acknowledgement switched on, a batch counts as delivered only once Splunk confirms it has been indexed, not merely received.

When to choose Splunk HEC

• Splunk is your SIEM and you want audit events pushed within seconds, without a Universal Forwarder on the IdentityMesh hosts.
• You want proof of indexing: with indexer acknowledgement the product’s delivery position only moves once Splunk has written the events to an index.
• You want each stream in its own source type, so searches, retention and access rules can tell them apart.

If you already collect the Windows event log with a Universal Forwarder, the default Windows event log provider also reaches Splunk with no further setup. The HEC provider adds acknowledgement of indexing and removes the forwarder hop.

Prepare Splunk

Do this once, in Splunk Web, as a Splunk administrator.

1. Create an index. Settings → Indexes → New Index, for example identitymesh. Size its retention for your audit obligations; it holds the evidence.
2. Switch the collector on. Settings → Data inputs → HTTP Event Collector → Global Settings. Set All Tokens to Enabled, keep Enable SSL ticked, and note the port (8088 unless changed).
3. Create a token. Settings → Data inputs → HTTP Event Collector → New Token.
   • Name it, for example IdentityMesh audit.
   • Tick Enable indexer acknowledgement if you want proof of indexing (recommended for evidence). If you tick it here, switch on Use indexer acknowledgement on the provider too.
   • On the next page leave the source type as Automatic: the provider sets it on every event.
   • Under Allowed indexes add the index from step 1, and make it the default index.
   • Copy the token value. It is shown in full only here.

On Splunk Cloud, ask your Splunk Cloud administrator for the HEC address of your stack and a token; the address usually uses port 443.

The certificate Splunk presents on the collector port must be trusted by the Admin API host. The certificate Splunk ships with by default is self-signed: replace it with one from your certificate authority, or add that authority to the Local Machine trusted root store on the Admin API host.

Add the provider

1. Open Settings → SIEM Integrations and choose Add provider.
2. Pick the kind Splunk HEC and give the provider a name. The name appears in every event as provider.
3. Fill in the settings below and paste the HEC token.
4. Choose which streams and kinds this provider receives. See Limiting what leaves the product.
5. Choose Save.
6. Press Test. It sends one sample event through the real send path with the saved settings and token, including the acknowledgement wait when that is on, and reports what Splunk answered. After a later change, save again before you test: Test always uses what is saved.

The test event has the kind SiemProviderTest on the admin stream and a seq of 0. Find it with index=identitymesh kind=SiemProviderTest.

Settings reference

What Splunk receives

Each audit event becomes one HEC event:

{
  "time": 1793610847.421,
  "host": "im-api-01",
  "source": "identitymesh",
  "sourcetype": "identitymesh:siem",
  "index": "identitymesh",
  "event": { "product": "IdentityMesh", "seq": 12345, "stream": "admin", "kind": "POST /api/connectors", "...": "..." }
}

Splunk extracts the JSON fields at search time, so stream, kind, seq, outcome, actor.upn, subject.id and every detail member are searchable by name. The per-stream members are listed in the field reference per stream.

Search-time JSON extraction is on by default in Splunk. If your Splunk administrator has changed that default, add a stanza for each source type you use to props.conf on the search heads (or in Splunk Cloud, as a source type with the same setting):

[identitymesh:siem]
KV_MODE = json

# One more stanza per entry in Source type per stream, for example:
[identitymesh:vault]
KV_MODE = json

Use KV_MODE = json and not INDEXED_EXTRACTIONS = json: with both, or with indexed extractions on a HEC event, every field appears twice.

Indexer acknowledgement

Without acknowledgement, an HTTP 200 from Splunk means the collector received the batch. It does not prove the batch reached an index.

With Use indexer acknowledgement on:

1. The provider sends the batch with the channel header. Splunk answers with an acknowledgement id.
2. The provider asks the collector’s acknowledgement endpoint about that id every two seconds.
3. When Splunk reports the id as indexed, the batch is delivered and the provider’s position and sequence move forward.
4. If Splunk has not confirmed within 60 seconds, the send counts as a retry: the provider backs off and sends the same batch again, with the same seq numbers. Splunk may then hold some events twice, with the same provider, stream and seq. Deduplicate on those three, as in Using the sequence.

A retry ends the provider’s turn in that pass, and a provider stops for the pass after about a minute of sending, so a slow indexer delays only this provider. Its other streams go on the next pass.

If the token does not have acknowledgement enabled, Splunk answers without an acknowledgement id, or the acknowledgement endpoint answers “ACK is disabled”. Either way the provider is parked with that reason: switch acknowledgement on for the token, or off on the provider.

What the answers mean

A parked provider sends nothing until you correct its settings and save. Saving resumes delivery from the same position, so nothing is skipped. The backlog, last error and Stalled flag are on the Operations page.

Certificate validation

Skip certificate validation turns off the check of the collector’s certificate. Anyone able to intercept the traffic could then read the token and the events, so it is meant for a test Splunk only.

The setting is honoured only when the Admin API host permits it: the host runs in the Development environment, or its configuration sets Siem:AllowInsecureTls to true (in the Admin API’s appsettings.json, or as an environment variable). Otherwise the portal refuses to save the setting, and a provider that already has it has its certificate checked as normal. When the setting is in effect, the IdentityMesh event log gets a warning every 15 minutes. The relay agent follows the same rule for its own connection.

The better fix is always to trust the certificate: replace Splunk’s default certificate, or add your certificate authority to the Local Machine trusted root store on the Admin API host.

Sample searches

One per stream, assuming the index identitymesh and the default source type. Use them as starting points for alerts.

admin: refused administrator calls in the last day, by person.

index=identitymesh sourcetype="identitymesh:siem" stream=admin outcome=denied earliest=-24h
| stats count, values(kind) AS actions BY actor.upn
| sort - count

object: more than 100 deletes by one actor in five minutes.

index=identitymesh sourcetype="identitymesh:siem" stream=object kind=Delete
| bin _time span=5m
| stats count BY _time, actor.upn, actor.source
| where count > 100

vault: credential views and failed or forced rotations.

index=identitymesh sourcetype="identitymesh:siem" stream=vault kind IN ("CredentialView", "ForcedRotation", "RotationFailed")
| table _time, kind, actor.upn, actor.ip, subject.id, outcome

group: owners and members added, by who added them.

index=identitymesh sourcetype="identitymesh:siem" stream=group kind IN ("OwnerAdded", "MemberAdded")
| stats count BY actor.upn, kind, subject.id
| sort - count

run: connectors whose sync runs failed or were interrupted this week.

index=identitymesh sourcetype="identitymesh:siem" stream=run kind IN ("Failed", "Interrupted") earliest=-7d
| stats count, latest(_time) AS lastFailure BY subject.id, detail.runType
| convert ctime(lastFailure)

workflow: workflow runs that failed, timed out or were rejected.

index=identitymesh sourcetype="identitymesh:siem" stream=workflow kind IN ("Failed", "TimedOut", "Rejected")
| table _time, detail.workflowName, detail.workflowVersion, kind, detail.attempt

Delivery check: gaps in the sequence, per provider and stream.

index=identitymesh sourcetype="identitymesh:siem" kind!=SiemProviderTest
| dedup provider, stream, seq
| sort 0 provider, stream, seq
| streamstats current=f last(seq) AS previous BY provider, stream
| where isnotnull(previous) AND seq != previous + 1
| table provider, stream, previous, seq

Each stream counts on its own, so the search compares each event with the one before it in the same stream. A row here means an event never arrived, or arrived outside the time range searched. Widen the range before raising an incident. Test events have seq 0 and are left out.

If you set Source type per stream, replace the sourcetype term with the stream’s own source type, or drop it and keep stream=.

Related

• siem-providers.md: the provider model, the event and the field reference per stream.
• siem-integration.md: every surface the product emits, and the Universal Forwarder recipe for log files.
• siem-webhook.md: the HTTPS provider, for collectors other than Splunk HEC.
• event-log.md: the IdentityMesh log that the stall and certificate warnings write to.