Data Feed Specification

🚧

This Is Only Needed For Platforms Without A Clerk.io Plugin

The data feed is only needed for platforms without an extension, following the custom setup guide.

See help.clerk.io for a list of all supported platforms

The first step of integrating Clerk.io with any store is to sync the stores product-, category- and order-data with Clerk.io.

This is done via a data feed in JSON format as described in this document.

📘

Remember: Always Use a JSON library to create the feed

Most programming languages have a built-in JSON library, that automatically parses and writes data in JSON. This means that you never have to manually type JSON.

Check the official JSON homepage, for a library for your programming language.

The base structure of the feed contains the different data types you can send to Clerk.io and the feed configuration.

{
  "products": [ ... ],
  "categories": [ ... ],
  "orders": [ ... ],
  "customers": [ ... ],
  "pages": [ ... ],

  "config": {
    "created": 1567069830,
    "strict": false
  }
}
{ 
  "products": [
    {
      "id": 123,
      "name": "Green Lightsaber",
      "description": "Antiuque rebel lightsaber",
      "price": 99995.95,
      "image": "http://your-store.com/images/antique-rebel-lightsaber.jpg",
      "url": "http://your-store.com/antique-rebel-lightsaber",
      "brand": "Je’daii",
      "categories": [987, 654]
    },
    {
      "id": 789,
      "name": "Death Star Deluxe",
      "description": "Death Star - Guaranteed idiot proof",
      "price": 99999999999999.95,
      "image": "http://your-store.com/images/death-star.jpg",
      "url": "http://your-store.com/death-star",
      "brand": "Imperial Inc.",
      "categories": [345678]
    }
  ],

  "categories": [
    {
      "id": 1,
      "name": "Imperial Goods",
      "subcategories": [42, 25],
      "url": "http://your-store.com/imperial-goods"
    },
    {
      "id": 42,
      "name": "Tatooine",
      "subcategories": [],
      "url": "http://your-store.com/imperial-goods/tatooine"
    },
    {
      "id": 25,
      "name": "Coruscant",
      "subcategories": [],
      "url": "http://your-store.com/imperial-goods/coruscant"
    }
  ],

  "sales": [
    {
      "id": 123458,
      "customer": 789,
      "email": "[email protected]",
      "products": [{"id":456,"quantity":1,"price":200.00}, {"id":789,"quantity":2,"price":120.00}],
      "time": 1389871120
    },

    {
      "id": 123456,
      "customer": 456,
      "email": "[email protected]",
      "products": [{"id":456,"quantity":1,"price":200.00}, {"id":789,"quantity":2,"price":120.00},{"id":123,"quantity":2,"price":60.00}],
      "time": 1389870977
    },

    {
      "id": 123457,
      "customer": "",
      "products": [{"id":789,"quantity":2,"price":120.00}],
      "time": 1389871090
    }
  ],
    
  "customers": [
    {
      "id": 135,
      "name": "Luke Skywalker",
      "email": "[email protected]",
      "gender": "male",
      "subscribed_to_newsletter": true
    },
    {
      "id": 1,
      "name": "Darth Vader",
      "email": "[email protected]",
      "gender": "male",
      "age": 45,
      "interests": ["loghtsaber", "force"],
      "subscribed_to_newsletter": false
    }
  ],
  
  "created": 1234567890,
  "strict": false
}

Here is a table of what each datatype is:

Entry

Content

products

A list of all the products in the store.

categories

A list of all the categories in the store.

orders

A list of all orders made in the store.

customers

A list of your registered customers in your store.

pages

A list of the pages (cms, blog etc,) in your store.

Here is the feed config that dictates how the feed should be read. The entire config is optional.

Config

Behaviour

created

  • Optional_. A timestamp indicating when the feed was last updated. This is used for product versioning if real time product updates are pushed via the API.

This must be a unix timestamp

strict

  • Optional. Should the feed be parsed in _strict mode or nor. If set to true all values will be interpreted as is but if set to false string encoded numbers etc will be parsed to numbers etc. By default strict is set to false, and this setting is especially recommended to keep if you are using PHP.

The following sections describe the structure of products, categories and orders.

🚧

A note on Value Types

Null values
Unchecked null values are a sure way for errors to sneak in over time. If an attribute does not exist for a given product, category or order simply just omit the attribute.

ID value types
We highly recommend using integers as IDs but it is possible to use strings as well. You must always commit to 1 type in your feed, meaning all IDs for both products, categories, orders, pages and customers must be of the same type.

Attribute names
Attribute names can only contain alphanumerical values (A-Z, 0-9) and underscores.
Thus, a valid attribute name could be brand_name.

Using dashes or special characters in the attribute names will cause them to be ignored in the sync.

Products

A product is represented as a product object. A product object is a JSON object where each key, value pair corresponds to a product attribute name, attribute value.

All attribute names must be strings and values can be either bool, int, float, string or a list of the former.

A product can have any number of attributes but must at least contain the following marked as Required:

Attribute

Importance

Content

id

Required

The product ID.

name

Required

The product name.

description

Required

The product description.

price

Required

The product current selling price.

list_price

Optional

The product original list price.

image

Required

The full URL for the product image. This will be used for thumbnails when displaying products. We recommend a maximum image size of 200x200px.

url

Required

The full URL for the product page.

categories

Required

A list of the category IDs for the product categories.

created_at

Required

Unix timestamp for when the product was created.

brand

Optional

The product brand as a string.

sku

Optional

The product SKU (Stock Keeping Unit).

stock

Optional

Stock count of this product.

age

Optional

The number of days since the product was created.

on_sale

Optional

true if this product is on sale, else false.

gender

Optional

Is the product for a specific gender? Add that information.

keywords

Optional

Keywords or synonyms that should be searchable for the product.

margin

Optional

An indication of the profit made from selling the product. Can be in percent, a scale between 1-5 or any other number that can be used to compare earnings between products.

index

Optional

If false, the product will not be indexed and thus not shown in any results. It will still be kept in the database as historic data so results like alternatives and personal recommendations can be based on it.

If true, the product will be indexed as normal.

Defaults to true if not sent.

color_names

Optional

A list of colors the product is available in, like "Green", "Red", "Blue".

color_codes

Optional

A list of specific HEX color codes for the available colors, like "#664386", "#7c1292", "#4d7810"

reviews_amount

Optional

The amount of reviews a product has received.

reviews_ag

Optional

The average review score a product has received.

[
  {
    "id": 135,
    "name": "Lightsaber",
    "description": "Antique Rebel Lightsaber",
    "price": 99995.95,
    "image": "https://galactic-empire-merch.com/images/a-r-lightsaber.jpg",
    "url": "https://galactic-empire-merch.com/antique-rebel-lightsaber",
    "brand": "Je’daii",
    "categories": [987, 654],
    "created_at": 1199145600,
    "color_names": ["Green","Red"],
    "color_codes": ["#7CFC00","#FF3131"],
    "reviews_amount": 164,
    "reviews_avg": 4.8
  },
  {
    "id": 261,
    "name": "Death Star Deluxe",
    "description": "Death Star - Guaranteed idiot proof",
    "price": 99999999999999.95,
    "image": "https://galactic-empire-merch.com/images/death-star.jpg",
    "url": "https://galactic-empire-merch.com/death-star",
    "brand": "Imperial Inc.",
    "categories": [345678],
    "created_at": 1197565600
  }
]

Categories

A category is represented as a category object. They can have any number of attributes but must at least contain the following marked as Required:

Key

Importance

Content

id

Required

The category ID.

name

Required

The category name.

url

Required

The category URL.

subcategories

Required

The IDs of the category's immediate subcategories.

image

Optional

An image representing the category

description

Optional

Text describing the details of the category

[
  {
    "id": 1,
    "name": "Imperial Goods",
    "subcategories": [42, 25],
    "url": "https://galactic-empire-merch.com/imperial-goods"
  },
  {
    "id": 42,
    "name": "Tatooine",
    "subcategories": [],
    "url": "https://galactic-empire-merch.com/imperial-goods/tatooine"
  },
  {
    "id": 25,
    "name": "Coruscant",
    "subcategories": [],
    "url": "https://galactic-empire-merch.com/imperial-goods/coruscant"
  }
]

Orders

A sale/order is represented as a sale object with the following entries:

Key

Importance

Content

id

Required

The sale/order ID.

products

Required

The products in the order. Each product is an object with an ID, quantity, and unit price.

time

Required

The time of the order as a Unix Timestamp.

email

  • _Required__*

The customer email.

customer

Optional

The customer ID, if any.

*Note that email is required to use our Email and Audience products.

[
  {
    "id": 123458,
    "customer": 789,
    "email": "[email protected]",
    "products": [{"id":456,"quantity":1,"price":200.00}, {"id":789,"quantity":2,"price":120.00}],
    "time": 1389871120
  },

  {
    "id": 123456,
    "customer": 456,
    "email": "[email protected]",
    "products": [{"id":456,"quantity":1,"price":200.00}, {"id":789,"quantity":2,"price":120.00},{"id":123,"quantity":2,"price":60.00}],
    "time": 1389870977
  },

  {
    "id": 123457,
    "customer": "",
    "products": [{"id":789,"quantity":2,"price":120.00}],
    "time": 1389871090
  },
]

📘

Orders are kept once synced

Orders are stored as a log and thus never deleted. After the first successful Data Sync, and when sales tracking is working, you can remove orders from the feed to reduce the feed file size.

Customers

Each Customer is represented as a customer object where each key-value pair corresponds to a Customers attribute name and value. Just as for product objects, customer objects allow you to have any number attributes.

All attribute names must be strings, and values can be either bool, int, float, string or a list of the former.

A customer can have any number of attributes but must at least contain the following marked as Required:

Key

Importance

Content

id

Required

The customer ID.

name

Required

The customers full name.

email

Required

The customers email.

subscribed

Optional

Boolean indicating whether the customer has subscribed to newsletters. This must be true for Clerk.io to send marketing emails to this customer.

zip

Optional

The customers zip code.

gender

Optional

The customers gender.

age

Optional

The customers age.

is_b2b

Optional

Boolean indicating whether a customer is B2B or not.

[
  {
    "id": 135,
    "name": "Luke Skywalker",
    "email": "[email protected]",
    "subscribed": true,
    "gender": "male",
    "zip": "1134",
    "is_b2b": "false"
  },
  {
    "id": 165,
    "name": "Darth Vader",
    "email": "[email protected]",
    "subscribed": false,
    "gender": "male",
    "age": 45,
    "interests": ["lightsaber", "force"],
    "is_b2b": true
  }
]

Pages

Each Page is represented as a page object where each key-value pair corresponds to a Page attribute name and value. Just as for product objects, customer objects allow you to have any number of attributes.

All attribute names must be strings, and values can be either bool, int, float, string or a list of the former.

A product can have any number of attributes but must at least contain the following marked as Required:

Key

Importance

Content

id

Required

The main id of the

type

Required

The type of the content. Used to separate different types of pages such as CMP pages, blog posts and landing pages.

url

Required

The URL of the content.

image

Optional

The main image for the content.

title

Required

The main page title.

text

Required

The main page text.

Besides the required fields any page can accept any number of optional fields.

[
  {
    "id": 135,
    "type": "cms",
    "url": "https://galactic-empire-merch.com/imperial-goods/tatooine",
    "title": "Open Hours",
    "text": "The main text about our opening hours...",
  },
  {
    "id": 1354,
    "type": "blog",
    "url": "https://galactic-empire-merch.com/imperial-goods/tatooine",
    "title": "New Blog Post",
    "text": "The main text about our opening hours...",
    "keywords": ["blog", "post", "new"]
  }
]

Data Security

🚧

Do this as the very last step

This is an optional feature, so skip this until your have made a full working implementation and are ready to go into production.

Your data is extremely business-sensitive so security is of the highest priority!

We recommend that the JSON feed only accepts an SSL encrypted connection and uses HTTP Authentication if possible.

In addition, these basic security measures provide an additional layer of security so you can verify that the request to download the feed is from a trusted source (ie. us). The system is based on a shared secret; your account Private API Key is found under your Account Settings.

All requests via HTTP or HTTPS are given two query parameters hash and salt. salt is just a random string used to salt the hash function. hash is a SHA512 hash computed from the Private API Key in the following way:

hash = SHA512(salt + private_key + str(int(floor(unix_timestamp() / 100))))

Where + is the concatenation operator and unix_timestamp() returns the current Unix Timestamp.

An example request could be the following URL:

https://example.com/clerk-product-feed.php?salt=f4Ke...A02X&hash=4DFF...340F

Thus by recomputing the hash you can verify that the request issuer knows your private key, and is trustworthy, so that the feed data can be provided safely.

Multi-language feed

📘

This is an optional step

Most webshops should handle language-versioning by creating individual Stores, but it is also possible to achieve in a single feed.

The language must be specified with its exact name, and currently the following are supported:

  • danish
  • dutch
  • english
  • finnish
  • french
  • german
  • italian
  • norwegian
  • portuguese
  • russian
  • spanish
  • swedish

If your language is not on the above list, choose a language using the same root as yours, or simply "english". Clerk.io will still work well but grammar-neutralisation in Search will not optimal

🚧

Important!

All attributes containing text that should be searchable, must have a translation, even if the text is identical between languages. Examples include data like SKU's, brands, types and more.

Otherwise, the attributes might not be searchable for all languages.

Clerk.io supports multi-language texts in a single feed, by sending each attribute as a dictionary consisting of a language and a translation .

Example feed with products and categories:

{ 
  "products": [
    {
      "id": 123,
      "name": {
        "danish":"Grønt lyssværd",
        "english":"Green Lightsaber"
      	},
      "description": {
        "danish":"Antikt rebel lyssværd",
        "english":"Antiuque rebel lightsaber"
      	},
      "price": 999.50,
      "image": {
        "danish":"http://your-store.com/dk/images/antique-rebel-lightsaber.jpg",
        "english":"http://your-store.com/uk/images/antique-rebel-lightsaber.jpg"
      	},
      "url": {
        "danish":"http://your-store.com/dk/antique-rebel-lightsaber",
        "english":"http://your-store.com/uk/antique-rebel-lightsaber"
      	},
      "brand": {
        "danish":"Je'daii",
        "english":"Je'daii"
      	},
      "sku": {
      	"danish": "LUK3-SW0RD",
        "english": "LUK3-SW0RD"
      },
      "categories": [987, 654]
    },
    {
      "id": 456,
      "name": {
        "danish":"Deluxe Dødsstjerne",
        "english":"Death Star Deluxe"
      	},
      "description": {
        "danish":"Dødsstjerne - garanteret idiotsikker",
        "english":"Death Star - Guaranteed idiot proof"
      	},
      "price": 9999999,
      "image": {
        "danish":"http://your-store.com/dk/images/death-star.jpg",
        "english":"http://your-store.com/uk/images/death-star.jpg"
      	},
      "url": {
        "danish":"http://your-store.com/dk/death-star",
        "english":"http://your-store.com/uk/death-star"
      	},
      "brand": {
        "danish":"Imperial A/S",
        "english":"Imperial Inc."
      },
      "sku": {
      	"danish": "D4R7H-V4D3R",
        "english": "D4R7H-V4D3R"
      },
      "categories": [987, 654]
    },
  ],

  "categories": [
    {
      "id": 1,
      "name": {
        "danish":"Gode sager fra imperiet",
        "english":"Imperial Goods"
      	},
      "subcategories": [42, 25],
      "url": {
        "danish":"http://your-store.com/dk/imperial-goods",
        "english":"http://your-store.com/uk/imperial-goods"
      	},
    },
    {
      "id": 1,
      "name": {
        "danish":"Tatooine slik",
        "english":"Tatooine candy"
      	},
      "subcategories": [],
      "url": {
        "danish":"http://your-store.com/dk/imperial-goods/tatooine",
        "english":"http://your-store.com/uk/imperial-goods/tatooine"
      	},
    },
  ]
}

When you set your language configuration in the frontend, the corresponding language will be fetched from the API.

Examples:

<!-- Start of Clerk.io E-commerce Personalisation tool - www.clerk.io -->
  <script type="text/javascript">
    (function(w,d){
      var e=d.createElement('script');e.type='text/javascript';e.async=true;
      e.src=(d.location.protocol=='https:'?'https':'http')+'://cdn.clerk.io/clerk.js';
      var s=d.getElementsByTagName('script')[0];s.parentNode.insertBefore(e,s);
      w.__clerk_q=w.__clerk_q||[];w.Clerk=w.Clerk||function(){w.__clerk_q.push(arguments)};
    })(window,document);

    Clerk('config', {
      key: 'fpUlOw4iBn8nnFfvdKMAH4w2KpALBV5J',
      language: 'danish'
    });
  </script>
  <!-- End of Clerk.io E-commerce Personalisation tool - www.clerk.io -->
curl -X POST \
     -H 'Content-Type: application/json' \
     -d '{"key": "fpUlOw4iBn8nnFfvdKMAH4w2KpALBV5J",
          "limit": 30,
          "offset": 60,
          "visitor": "jgy543",
          "exclude": [42],
          "filter": "price > 100",
          "labels": ["Bestsellers"],
          "attributes":["id","name","url","image"],
          "language": "danish"}' \
     http://api.clerk.io/v2/recommendations/popular

Syncing your data feed

When your feed has the above structure you can upload it to Clerk.io by going to Data Sync in my.clerk.io.

Choose JSON as the sync method, paste the feed url and choose the language of the feed. Click Update Settings once you are done.

Afterwards, click Start Sync to synchronise the feed.

2270

Integration configuration