Site integration

You integrate Divolte Collector into your site by simply including a small piece of JavaScript.

This takes care of logging all pageviews and exposes a JavaScript module. This module can be used to interact with Divolte Collector in the browser and log custom events.

Structured data in Avro

To preserve the sanity of developers and data scientists alike, all data should come with a schema. CSV is not a schema. The common log format is not a schema. JSON is not a schema. A schema defines which fields exist and what their types are. Using a schema allows you to inspect data without making assumptions about which fields are available.

Divolte Collector uses Apache Avro for storing data. Avro requires a schema for all data, yet it allows for full flexibility through schema evolution.

Through a special feature of Divolte Collector called mapping, you can map any part of incoming events onto any field in your schema. Mapping also allows for complex constructs such as mapping fields conditionally or setting values based on URL patterns or other incoming event data.

Avro schema

{
  "namespace": "com.example.record",
  "type": "record",
  "name": "MyEventRecord",
  "fields": [
    { "name": "location", "type": "string" },
    { "name": "pageType", "type": "string" },
    { "name": "timestamp", "type": "long" }
  ]
}

Divolte Collector mapping

mapping {
  map clientTimestamp() onto 'timestamp'
  map location() onto 'location'

  def u = parse location() to uri
  section {
    when u.path().equalTo('/checkout') apply {
      map 'checkout' onto 'pageType'
      exit()
    }
    map 'normal' onto 'pageType'
  }
}

IP to geolocation lookup

If enabled, Divolte Collector will perform on-the-fly ip2geo lookups using databases provided by MaxMind. You can use either the light version of their database, which is downloadable for free or use their more accurate subscription database, which comes with a commercial license.

Note that it's not possible for us to enable this feature by default as redistribution of the MaxMind database is restricted. Configuration is however very simple: just put the path of the database file in the Divolte Collector configuration.

If you have a subscription license to the MaxMind database, Divolte Collector will reload the database as updates appear without a restart.

Custom events

Log anything from the browser. As with other web tracking tools you can fire custom events from your pages using JavaScript. Whether it's an add-to-basket, checkout or product image zoom, just add a custom event if you want to track it.

Custom events can have parameters in the form of arbitrary JavaScript objects and these are easily mapped onto your Avro records. They are part of your own schema. You can extract top-level object members directly by name or use JSONPath expressions to extract values, arrays or complete objects from the event payload.

In JavaScript

divolte.signal('searchResultClick', {
    productId: 309125,
    searchPhrase: 'sneakers',
    filters: [
        { name: 'size', value: 10 },
        { name: 'color': value: 'red' }
    ]
})

In Divolte Collector

// Using direct values
map eventParameters().value('searchPhrase') onto 'searchPhrase'
map eventParameters().value('productId') onto 'product'

// Using JSONPath extractions
map eventParameters().path('$.filters[*].name') onto 'filterKeys'
map eventParameters().path('$.filters[*].value') onto 'filterValues'

Open source

Divolte Collector is released under the Apache License, Version 2.0.

It's never a good idea to be locked in to a vendor for your data collection. Similarly, sending your clickstream data to cloud providers can present issues. Better to take control and free yourself from data ownership issues, closed formats and license or service fees for obtaining for your own data.