Data Feed Specification

🚧
This Is Only Needed For Platforms Without A Clerk.io Plugin
The data feed is only needed for platforms without an extension, following the custom setup guide.
See help.clerk.io for a list of all supported platforms

The first step of integrating Clerk.io with any store is to sync the stores product-, category- and order-data with Clerk.io.

This is done via a data feed in JSON format as described in this document.

📘
Remember: Always Use a JSON library to create the feed
Most programming languages have a built-in JSON library, that automatically parses and writes data in JSON. This means that you never have to manually type JSON.
Check the official JSON homepage, for a library for your programming language.

The base structure of the feed contains the different data types you can send to Clerk.io and the feed configuration.

{
  "products": [ ... ],
  "categories": [ ... ],
  "orders": [ ... ],
  "customers": [ ... ],
  "pages": [ ... ],

  "config": {
    "created": 1567069830,
    "strict": false
  }
}

{ 
  "products": [
    {
      "id": 123,
      "name": "Green Lightsaber",
      "description": "Antiuque rebel lightsaber",
      "price": 99995.95,
      "image": "http://your-store.com/images/antique-rebel-lightsaber.jpg",
      "url": "http://your-store.com/antique-rebel-lightsaber",
      "brand": "Je’daii",
      "categories": [987, 654]
    },
    {
      "id": 789,
      "name": "Death Star Deluxe",
      "description": "Death Star - Guaranteed idiot proof",
      "price": 99999999999999.95,
      "image": "http://your-store.com/images/death-star.jpg",
      "url": "http://your-store.com/death-star",
      "brand": "Imperial Inc.",
      "categories": [345678]
    }
  ],

  "categories": [
    {
      "id": 1,
      "name": "Imperial Goods",
      "subcategories": [42, 25],
      "url": "http://your-store.com/imperial-goods"
    },
    {
      "id": 42,
      "name": "Tatooine",
      "subcategories": [],
      "url": "http://your-store.com/imperial-goods/tatooine"
    },
    {
      "id": 25,
      "name": "Coruscant",
      "subcategories": [],
      "url": "http://your-store.com/imperial-goods/coruscant"
    }
  ],

  "sales": [
    {
      "id": 123458,
      "customer": 789,
      "email": "[email protected]",
      "products": [{"id":456,"quantity":1,"price":200.00}, {"id":789,"quantity":2,"price":120.00}],
      "time": 1389871120
    },

    {
      "id": 123456,
      "customer": 456,
      "email": "[email protected]",
      "products": [{"id":456,"quantity":1,"price":200.00}, {"id":789,"quantity":2,"price":120.00},{"id":123,"quantity":2,"price":60.00}],
      "time": 1389870977
    },

    {
      "id": 123457,
      "customer": "",
      "products": [{"id":789,"quantity":2,"price":120.00}],
      "time": 1389871090
    }
  ],
    
  "customers": [
    {
      "id": 135,
      "name": "Luke Skywalker",
      "email": "[email protected]",
      "gender": "male",
      "subscribed_to_newsletter": true
    },
    {
      "id": 1,
      "name": "Darth Vader",
      "email": "[email protected]",
      "gender": "male",
      "age": 45,
      "interests": ["loghtsaber", "force"],
      "subscribed_to_newsletter": false
    }
  ],
  
  "created": 1234567890,
  "strict": false
}

Here is a table of what each datatype is:

Entry	Content
`products`	A list of all the products in the store.
`categories`	A list of all the categories in the store.
`orders`	A list of all orders made in the store.
`customers`	A list of your registered customers in your store.
`pages`	A list of the pages (cms, blog etc,) in your store.

Here is the feed config that dictates how the feed should be read. The entire config is optional.

Config	Behaviour
`created`	Optional_. A timestamp indicating when the feed was last updated. This is used for product versioning if real time product updates are pushed via the API. This must be a unix timestamp
`strict`	Optional. Should the feed be parsed in _strict mode or nor. If set to `true` all values will be interpreted as is but if set to `false` string encoded numbers etc will be parsed to numbers etc. By default `strict` is set to `false`, and this setting is especially recommended to keep if you are using PHP.

Config

Behaviour

created

Optional_. A timestamp indicating when the feed was last updated. This is used for product versioning if real time product updates are pushed via the API.

This must be a unix timestamp

strict

Optional. Should the feed be parsed in _strict mode or nor. If set to true all values will be interpreted as is but if set to false string encoded numbers etc will be parsed to numbers etc. By default strict is set to false, and this setting is especially recommended to keep if you are using PHP.

The following sections describe the structure of products, categories and orders.

🚧
A note on Value Types
Null values
Unchecked null values are a sure way for errors to sneak in over time. If an attribute does not exist for a given product, category or order simply just omit the attribute.
ID value types
We highly recommend using integers as IDs but it is possible to use strings as well. You must always commit to 1 type in your feed, meaning all IDs for both products, categories, orders, pages and customers must be of the same type.
Attribute names
Attribute names can only contain alphanumerical values (A-Z, 0-9) and underscores.
Thus, a valid attribute name could be brand_name.
Using dashes or special characters in the attribute names will cause them to be ignored in the sync.

Products

A product is represented as a product object. A product object is a JSON object where each key, value pair corresponds to a product attribute name, attribute value.

All attribute names must be strings and values can be either bool, int, float, string or a list of the former.

A product can have any number of attributes but must at least contain the following marked as Required:

Attribute	Importance	Content
`id`	Required	The product ID.
`name`	Required	The product name.
`description`	Required	The product description.
`price`	Required	The product current selling price.
`list_price`	Optional	The product original list price.
`image`	Required	The full URL for the product image. This will be used for thumbnails when displaying products. We recommend a maximum image size of 200x200px.
`url`	Required	The full URL for the product page.
`categories`	Required	A list of the category IDs for the product categories.
`created_at`	Required	Unix timestamp for when the product was created.
`brand`	Optional	The product brand as a string.
`sku`	Optional	The product SKU (Stock Keeping Unit).
`stock`	Optional	Stock count of this product.
`age`	Optional	The number of days since the product was created.
`on_sale`	Optional	`true` if this product is on sale, else `false`.
`gender`	Optional	Is the product for a specific gender? Add that information.
`keywords`	Optional	Keywords or synonyms that should be searchable for the product.
`margin`	Optional	An indication of the profit made from selling the product. Can be in percent, a scale between 1-5 or any other number that can be used to compare earnings between products.
`index`	Optional	If `false`, the product will not be indexed and thus not shown in any results. It will still be kept in the database as historic data so results like alternatives and personal recommendations can be based on it. If `true`, the product will be indexed as normal. Defaults to `true` if not sent.
`color_names`	Optional	A list of colors the product is available in, like "Green", "Red", "Blue".
`color_codes`	Optional	A list of specific HEX color codes for the available colors, like "#664386", "#7c1292", "#4d7810"
`reviews_amount`	Optional	The amount of reviews a product has received.
`reviews_ag`	Optional	The average review score a product has received.

[
  {
    "id": 135,
    "name": "Lightsaber",
    "description": "Antique Rebel Lightsaber",
    "price": 99995.95,
    "image": "https://galactic-empire-merch.com/images/a-r-lightsaber.jpg",
    "url": "https://galactic-empire-merch.com/antique-rebel-lightsaber",
    "brand": "Je’daii",
    "categories": [987, 654],
    "created_at": 1199145600,
    "color_names": ["Green","Red"],
    "color_codes": ["#7CFC00","#FF3131"],
    "reviews_amount": 164,
    "reviews_avg": 4.8
  },
  {
    "id": 261,
    "name": "Death Star Deluxe",
    "description": "Death Star - Guaranteed idiot proof",
    "price": 99999999999999.95,
    "image": "https://galactic-empire-merch.com/images/death-star.jpg",
    "url": "https://galactic-empire-merch.com/death-star",
    "brand": "Imperial Inc.",
    "categories": [345678],
    "created_at": 1197565600
  }
]

Key	Importance	Content
`id`	Required	The category ID.
`name`	Required	The category name.
`url`	Required	The category URL.
`subcategories`	Required	The IDs of the category's immediate subcategories.
`image`	Optional	An image representing the category
`description`	Optional	Text describing the details of the category

Orders

A sale/order is represented as a sale object with the following entries:

Key	Importance	Content
`id`	Required	The sale/order ID.
`products`	Required	The products in the order. Each product is an object with an ID, quantity, and unit price.
`time`	Required	The time of the order as a Unix Timestamp.
`email`	_Required__*	The customer email.
`customer`	Optional	The customer ID, if any.

*Note that email is required to use our Email and Audience products.

[
  {
    "id": 123458,
    "customer": 789,
    "email": "[email protected]",
    "products": [{"id":456,"quantity":1,"price":200.00}, {"id":789,"quantity":2,"price":120.00}],
    "time": 1389871120
  },

  {
    "id": 123456,
    "customer": 456,
    "email": "[email protected]",
    "products": [{"id":456,"quantity":1,"price":200.00}, {"id":789,"quantity":2,"price":120.00},{"id":123,"quantity":2,"price":60.00}],
    "time": 1389870977
  },

  {
    "id": 123457,
    "customer": "",
    "products": [{"id":789,"quantity":2,"price":120.00}],
    "time": 1389871090
  },
]

📘
Orders are kept once synced
Orders are stored as a log and thus never deleted. After the first successful Data Sync, and when sales tracking is working, you can remove orders from the feed to reduce the feed file size.

Customers

Each Customer is represented as a customer object where each key-value pair corresponds to a Customers attribute name and value. Just as for product objects, customer objects allow you to have any number attributes.

All attribute names must be strings, and values can be either bool, int, float, string or a list of the former.

A customer can have any number of attributes but must at least contain the following marked as Required:

Key	Importance	Content
`id`	Required	The customer ID.
`name`	Required	The customers full name.
`email`	Required	The customers email.
`subscribed`	Optional	Boolean indicating whether the customer has subscribed to newsletters. This must be true for Clerk.io to send marketing emails to this customer.
`zip`	Optional	The customers zip code.
`gender`	Optional	The customers gender.
`age`	Optional	The customers age.
`is_b2b`	Optional	Boolean indicating whether a customer is B2B or not.

[
  {
    "id": 135,
    "name": "Luke Skywalker",
    "email": "[email protected]",
    "subscribed": true,
    "gender": "male",
    "zip": "1134",
    "is_b2b": "false"
  },
  {
    "id": 165,
    "name": "Darth Vader",
    "email": "[email protected]",
    "subscribed": false,
    "gender": "male",
    "age": 45,
    "interests": ["lightsaber", "force"],
    "is_b2b": true
  }
]

Pages

Each Page is represented as a page object where each key-value pair corresponds to a Page attribute name and value. Just as for product objects, customer objects allow you to have any number of attributes.

All attribute names must be strings, and values can be either bool, int, float, string or a list of the former.

A product can have any number of attributes but must at least contain the following marked as Required:

Key	Importance	Content
`id`	Required	The main id of the
`type`	Required	The type of the content. Used to separate different types of pages such as CMP pages, blog posts and landing pages.
`url`	Required	The URL of the content.
`image`	Optional	The main image for the content.
`title`	Required	The main page title.
`text`	Required	The main page text.

Besides the required fields any page can accept any number of optional fields.

[
  {
    "id": 135,
    "type": "cms",
    "url": "https://galactic-empire-merch.com/imperial-goods/tatooine",
    "title": "Open Hours",
    "text": "The main text about our opening hours...",
  },
  {
    "id": 1354,
    "type": "blog",
    "url": "https://galactic-empire-merch.com/imperial-goods/tatooine",
    "title": "New Blog Post",
    "text": "The main text about our opening hours...",
    "keywords": ["blog", "post", "new"]
  }
]

Data Security

🚧
Do this as the very last step
This is an optional feature, so skip this until your have made a full working implementation and are ready to go into production.

Your data is extremely business-sensitive so security is of the highest priority!

We recommend that the JSON feed only accepts an SSL encrypted connection and uses HTTP Authentication if possible.

In addition, these basic security measures provide an additional layer of security so you can verify that the request to download the feed is from a trusted source (ie. us). The system is based on a shared secret; your account Private API Key is found under your Account Settings.

All requests via HTTP or HTTPS are given two query parameters hash and salt. salt is just a random string used to salt the hash function. hash is a SHA512 hash computed from the Private API Key in the following way:

hash = SHA512(salt + private_key + str(int(floor(unix_timestamp() / 100))))

Where + is the concatenation operator and unix_timestamp() returns the current Unix Timestamp.

An example request could be the following URL:

https://example.com/clerk-product-feed.php?salt=f4Ke...A02X&hash=4DFF...340F

Thus by recomputing the hash you can verify that the request issuer knows your private key, and is trustworthy, so that the feed data can be provided safely.

Multi-language feed

📘
This is an optional step
Most webshops should handle language-versioning by creating individual Stores, but it is also possible to achieve in a single feed.
The language must be specified with its exact name, and currently the following are supported:

danish

dutch

english

finnish

french

german

italian

norwegian

portuguese

russian

spanish

swedish

If your language is not on the above list, choose a language using the same root as yours, or simply "english". Clerk.io will still work well but grammar-neutralisation in Search will not optimal

🚧
Important!
All attributes containing text that should be searchable, must have a translation, even if the text is identical between languages. Examples include data like SKU's, brands, types and more.
Otherwise, the attributes might not be searchable for all languages.

Clerk.io supports multi-language texts in a single feed, by sending each attribute as a dictionary consisting of a language and a translation .

Example feed with products and categories:

{ 
  "products": [
    {
      "id": 123,
      "name": {
        "danish":"Grønt lyssværd",
        "english":"Green Lightsaber"
      	},
      "description": {
        "danish":"Antikt rebel lyssværd",
        "english":"Antiuque rebel lightsaber"
      	},
      "price": 999.50,
      "image": {
        "danish":"http://your-store.com/dk/images/antique-rebel-lightsaber.jpg",
        "english":"http://your-store.com/uk/images/antique-rebel-lightsaber.jpg"
      	},
      "url": {
        "danish":"http://your-store.com/dk/antique-rebel-lightsaber",
        "english":"http://your-store.com/uk/antique-rebel-lightsaber"
      	},
      "brand": {
        "danish":"Je'daii",
        "english":"Je'daii"
      	},
      "sku": {
      	"danish": "LUK3-SW0RD",
        "english": "LUK3-SW0RD"
      },
      "categories": [987, 654]
    },
    {
      "id": 456,
      "name": {
        "danish":"Deluxe Dødsstjerne",
        "english":"Death Star Deluxe"
      	},
      "description": {
        "danish":"Dødsstjerne - garanteret idiotsikker",
        "english":"Death Star - Guaranteed idiot proof"
      	},
      "price": 9999999,
      "image": {
        "danish":"http://your-store.com/dk/images/death-star.jpg",
        "english":"http://your-store.com/uk/images/death-star.jpg"
      	},
      "url": {
        "danish":"http://your-store.com/dk/death-star",
        "english":"http://your-store.com/uk/death-star"
      	},
      "brand": {
        "danish":"Imperial A/S",
        "english":"Imperial Inc."
      },
      "sku": {
      	"danish": "D4R7H-V4D3R",
        "english": "D4R7H-V4D3R"
      },
      "categories": [987, 654]
    },
  ],

  "categories": [
    {
      "id": 1,
      "name": {
        "danish":"Gode sager fra imperiet",
        "english":"Imperial Goods"
      	},
      "subcategories": [42, 25],
      "url": {
        "danish":"http://your-store.com/dk/imperial-goods",
        "english":"http://your-store.com/uk/imperial-goods"
      	},
    },
    {
      "id": 1,
      "name": {
        "danish":"Tatooine slik",
        "english":"Tatooine candy"
      	},
      "subcategories": [],
      "url": {
        "danish":"http://your-store.com/dk/imperial-goods/tatooine",
        "english":"http://your-store.com/uk/imperial-goods/tatooine"
      	},
    },
  ]
}

When you set your language configuration in the frontend, the corresponding language will be fetched from the API.

Examples:

<!-- Start of Clerk.io E-commerce Personalisation tool - www.clerk.io -->
  <script type="text/javascript">
    (function(w,d){
      var e=d.createElement('script');e.type='text/javascript';e.async=true;
      e.src=(d.location.protocol=='https:'?'https':'http')+'://cdn.clerk.io/clerk.js';
      var s=d.getElementsByTagName('script')[0];s.parentNode.insertBefore(e,s);
      w.__clerk_q=w.__clerk_q||[];w.Clerk=w.Clerk||function(){w.__clerk_q.push(arguments)};
    })(window,document);

    Clerk('config', {
      key: 'fpUlOw4iBn8nnFfvdKMAH4w2KpALBV5J',
      language: 'danish'
    });
  </script>
  <!-- End of Clerk.io E-commerce Personalisation tool - www.clerk.io -->

curl -X POST \
     -H 'Content-Type: application/json' \
     -d '{"key": "fpUlOw4iBn8nnFfvdKMAH4w2KpALBV5J",
          "limit": 30,
          "offset": 60,
          "visitor": "jgy543",
          "exclude": [42],
          "filter": "price > 100",
          "labels": ["Bestsellers"],
          "attributes":["id","name","url","image"],
          "language": "danish"}' \
     http://api.clerk.io/v2/recommendations/popular

Syncing your data feed

When your feed has the above structure you can upload it to Clerk.io by going to Data Sync in my.clerk.io.

Choose JSON as the sync method, paste the feed url and choose the language of the feed. Click Update Settings once you are done.

Afterwards, click Start Sync to synchronise the feed.

This Is Only Needed For Platforms Without A Clerk.io Plugin

Remember: Always Use a JSON library to create the feed

A note on Value Types

Products

Categories

Orders

Orders are kept once synced

Customers

Pages

Data Security

Do this as the very last step

Multi-language feed

This is an optional step

Important!

Syncing your data feed