Data Feed Specification

🚧

This Is Only Needed For Platforms Without A Clerk.io Plugin

The data feed is only needed for platforms without an extension, following the custom setup guide.

See help.clerk.io for a list of all supported platforms

The first step of integrating Clerk.io with any store is to sync the stores product-, category- and order-data with Clerk.io.

This is done via a data feed in JSON format as described in this document.

📘

Remember: Always Use a JSON library to create the feed

Most programming languages have a built-in JSON library, that automatically parses and writes data in JSON. This means that you never have to manually type JSON.

Check the official JSON homepage, for a library for your programming language.

The base structure of the feed contains the different data types you can send to Clerk.io and the feed configuration.

{
  "products": [ ... ],
  "categories": [ ... ],
  "orders": [ ... ],
  "customers": [ ... ],
  "pages": [ ... ],

  "config": {
    "created": 1567069830,
    "strict": false
  }
}
{ 
  "products": [
    {
      "id": 123,
      "name": "Green Lightsaber",
      "description": "Antiuque rebel lightsaber",
      "price": 99995.95,
      "image": "http://your-store.com/images/antique-rebel-lightsaber.jpg",
      "url": "http://your-store.com/antique-rebel-lightsaber",
      "brand": "Je’daii",
      "categories": [987, 654]
    },
    {
      "id": 789,
      "name": "Death Star Deluxe",
      "description": "Death Star - Guaranteed idiot proof",
      "price": 99999999999999.95,
      "image": "http://your-store.com/images/death-star.jpg",
      "url": "http://your-store.com/death-star",
      "brand": "Imperial Inc.",
      "categories": [345678]
    }
  ],

  "categories": [
    {
      "id": 1,
      "name": "Imperial Goods",
      "subcategories": [42, 25],
      "url": "http://your-store.com/imperial-goods"
    },
    {
      "id": 42,
      "name": "Tatooine",
      "subcategories": [],
      "url": "http://your-store.com/imperial-goods/tatooine"
    },
    {
      "id": 25,
      "name": "Coruscant",
      "subcategories": [],
      "url": "http://your-store.com/imperial-goods/coruscant"
    }
  ],

  "sales": [
    {
      "id": 123458,
      "customer": 789,
      "email": "[email protected]",
      "products": [{"id":456,"quantity":1,"price":200.00}, {"id":789,"quantity":2,"price":120.00}],
      "time": 1389871120
    },

    {
      "id": 123456,
      "customer": 456,
      "email": "[email protected]",
      "products": [{"id":456,"quantity":1,"price":200.00}, {"id":789,"quantity":2,"price":120.00},{"id":123,"quantity":2,"price":60.00}],
      "time": 1389870977
    },

    {
      "id": 123457,
      "customer": "",
      "products": [{"id":789,"quantity":2,"price":120.00}],
      "time": 1389871090
    }
  ],
    
  "customers": [
    {
      "id": 135,
      "name": "Luke Skywalker",
      "email": "[email protected]",
      "gender": "male",
      "subscribed_to_newsletter": true
    },
    {
      "id": 1,
      "name": "Darth Vader",
      "email": "[email protected]",
      "gender": "male",
      "age": 45,
      "interests": ["loghtsaber", "force"],
      "subscribed_to_newsletter": false
    }
  ],
  
  "created": 1234567890,
  "strict": false
}

Here is a table of what each datatype is:

EntryContent
productsA list of all the products in the store.
categoriesA list of all the categories in the store.
ordersA list of all orders made in the store.
customersA list of your registered customers in your store.
pagesA list of the pages (cms, blog etc,) in your store.

Here is the feed config that dictates how the feed should be read. The entire config is optional.

ConfigBehaviour
createdOptional. A timestamp indicating when the feed was last updated. This is used for product versioning if real time product updates are pushed via the API.

This must be a unix timestamp
strictOptional. Should the feed be parsed in strict mode or nor. If set to true all values will be interpreted as is but if set to false string encoded numbers etc will be parsed to numbers etc. By default strict is set to false, and this setting is especially recommended to keep if you are using PHP.

The following sections describe the structure of products, categories and orders.

🚧

A note on Value Types

Null values
Unchecked null values are a sure way for errors to sneak in over time. If an attribute does not exist for a given product, category or order simply just omit the attribute.

ID value types
We highly recommend using integers as IDs but it is possible to use strings as well. You must always commit to 1 type in your feed, meaning all IDs for both products, categories, orders, pages and customers must be of the same type.

Attribute names
Attribute names can only contain alphanumerical values (A-Z, 0-9) and underscores.
Thus, a valid attribute name could be brand_name.

Using dashes or special characters in the attribute names will cause them to be ignored in the sync.

Products

A product is represented as a product object. A product object is a JSON object where each key, value pair corresponds to a product attribute name, attribute value.

All attribute names must be strings and values can be either bool, int, float, string or a list of the former.

A product can have any number of attributes but must at least contain the following marked as Required:

AttributeImportanceContent
idRequiredThe product ID.
nameRequiredThe product name.
descriptionRequiredThe product description.
priceRequiredThe product current selling price.
list_priceOptionalThe product original list price.
imageRequiredThe full URL for the product image. This will be used for thumbnails when displaying products. We recommend a maximum image size of 200x200px.
urlRequiredThe full URL for the product page.
categoriesRequiredA list of the category IDs for the product categories.
created_atRequiredUnix timestamp for when the product was created.
brandOptionalThe product brand as a string.
skuOptionalThe product SKU (Stock Keeping Unit).
stockOptionalStock count of this product.
ageOptionalThe number of days since the product was created.
on_saleOptionaltrue if this product is on sale, else false.
genderOptionalIs the product for a specific gender? Add that information.
keywordsOptionalKeywords or synonyms that should be searchable for the product.
marginOptionalAn indication of the profit made from selling the product. Can be in percent, a scale between 1-5 or any other number that can be used to compare earnings between products.
indexOptionalIf false, the product will not be indexed and thus not shown in any results. It will still be kept in the database as historic data so results like alternatives and personal recommendations can be based on it.

If true, the product will be indexed as normal.

Defaults to true if not sent.
color_namesOptionalA list of colors the product is available in, like "Green", "Red", "Blue".
color_codesOptionalA list of specific HEX color codes for the available colors, like "#664386", "#7c1292", "#4d7810"
reviews_amountOptionalThe amount of reviews a product has received.
reviews_agOptionalThe average review score a product has received.
[
  {
    "id": 135,
    "name": "Lightsaber",
    "description": "Antique Rebel Lightsaber",
    "price": 99995.95,
    "image": "https://galactic-empire-merch.com/images/a-r-lightsaber.jpg",
    "url": "https://galactic-empire-merch.com/antique-rebel-lightsaber",
    "brand": "Je’daii",
    "categories": [987, 654],
    "created_at": 1199145600,
    "color_names": ["Green","Red"],
    "color_codes": ["#7CFC00","#FF3131"],
    "reviews_amount": 164,
    "reviews_avg": 4.8
  },
  {
    "id": 261,
    "name": "Death Star Deluxe",
    "description": "Death Star - Guaranteed idiot proof",
    "price": 99999999999999.95,
    "image": "https://galactic-empire-merch.com/images/death-star.jpg",
    "url": "https://galactic-empire-merch.com/death-star",
    "brand": "Imperial Inc.",
    "categories": [345678],
    "created_at": 1197565600
  }
]

Categories

A category is represented as a category object. They can have any number of attributes but must at least contain the following marked as Required:

KeyImportanceContent
idRequiredThe category ID.
nameRequiredThe category name.
urlRequiredThe category URL.
subcategoriesRequiredThe IDs of the category's immediate subcategories.
imageOptionalAn image representing the category
descriptionOptionalText describing the details of the category
[
  {
    "id": 1,
    "name": "Imperial Goods",
    "subcategories": [42, 25],
    "url": "https://galactic-empire-merch.com/imperial-goods"
  },
  {
    "id": 42,
    "name": "Tatooine",
    "subcategories": [],
    "url": "https://galactic-empire-merch.com/imperial-goods/tatooine"
  },
  {
    "id": 25,
    "name": "Coruscant",
    "subcategories": [],
    "url": "https://galactic-empire-merch.com/imperial-goods/coruscant"
  }
]

Orders

A sale/order is represented as a sale object with the following entries:

KeyImportanceContent
idRequiredThe sale/order ID.
productsRequiredThe products in the order. Each product is an object with an ID, quantity, and unit price.
timeRequiredThe time of the order as a Unix Timestamp.
emailRequired*The customer email.
customerOptionalThe customer ID, if any.

*Note that email is required to use our Email and Audience products.

[
  {
    "id": 123458,
    "customer": 789,
    "email": "[email protected]",
    "products": [{"id":456,"quantity":1,"price":200.00}, {"id":789,"quantity":2,"price":120.00}],
    "time": 1389871120
  },

  {
    "id": 123456,
    "customer": 456,
    "email": "[email protected]",
    "products": [{"id":456,"quantity":1,"price":200.00}, {"id":789,"quantity":2,"price":120.00},{"id":123,"quantity":2,"price":60.00}],
    "time": 1389870977
  },

  {
    "id": 123457,
    "customer": "",
    "products": [{"id":789,"quantity":2,"price":120.00}],
    "time": 1389871090
  },
]

📘

Orders are kept once synced

Orders are stored as a log and thus never deleted. After the first successful Data Sync, and when sales tracking is working, you can remove orders from the feed to reduce the feed file size.

Customers

Each Customer is represented as a customer object where each key-value pair corresponds to a Customers attribute name and value. Just as for product objects, customer objects allow you to have any number attributes.

All attribute names must be strings, and values can be either bool, int, float, string or a list of the former.

A customer can have any number of attributes but must at least contain the following marked as Required:

KeyImportanceContent
idRequiredThe customer ID.
nameRequiredThe customers full name.
emailRequiredThe customers email.
subscribedOptionalBoolean indicating whether the customer has subscribed to newsletters. This must be true for Clerk.io to send marketing emails to this customer.
zipOptionalThe customers zip code.
genderOptionalThe customers gender.
ageOptionalThe customers age.
is_b2bOptionalBoolean indicating whether a customer is B2B or not.
[
  {
    "id": 135,
    "name": "Luke Skywalker",
    "email": "[email protected]",
    "subscribed": true,
    "gender": "male",
    "zip": "1134",
    "is_b2b": "false"
  },
  {
    "id": 165,
    "name": "Darth Vader",
    "email": "[email protected]",
    "subscribed": false,
    "gender": "male",
    "age": 45,
    "interests": ["lightsaber", "force"],
    "is_b2b": true
  }
]

Pages

Each Page is represented as a page object where each key-value pair corresponds to a Page attribute name and value. Just as for product objects, customer objects allow you to have any number of attributes.

All attribute names must be strings, and values can be either bool, int, float, string or a list of the former.

A product can have any number of attributes but must at least contain the following marked as Required:

KeyImportanceContent
idRequiredThe main id of the
typeRequiredThe type of the content. Used to separate different types of pages such as CMP pages, blog posts and landing pages.
urlRequiredThe URL of the content.
imageOptionalThe main image for the content.
titleRequiredThe main page title.
textRequiredThe main page text.

Besides the required fields any page can accept any number of optional fields.

[
  {
    "id": 135,
    "type": "cms",
    "url": "https://galactic-empire-merch.com/imperial-goods/tatooine",
    "title": "Open Hours",
    "text": "The main text about our opening hours...",
  },
  {
    "id": 1354,
    "type": "blog",
    "url": "https://galactic-empire-merch.com/imperial-goods/tatooine",
    "title": "New Blog Post",
    "text": "The main text about our opening hours...",
    "keywords": ["blog", "post", "new"]
  }
]

Data Security

🚧

Do this as the very last step

This is an optional feature, so skip this until your have made a full working implementation and are ready to go into production.

Your data is extremely business-sensitive so security is of the highest priority!

We recommend that the JSON feed only accepts an SSL encrypted connection and uses HTTP Authentication if possible.

In addition, these basic security measures provide an additional layer of security so you can verify that the request to download the feed is from a trusted source (ie. us). The system is based on a shared secret; your account Private API Key is found under your Account Settings.

All requests via HTTP or HTTPS are given two query parameters hash and salt. salt is just a random string used to salt the hash function. hash is a SHA512 hash computed from the Private API Key in the following way:

hash = SHA512(salt + private_key + str(int(floor(unix_timestamp() / 100))))

Where + is the concatenation operator and unix_timestamp() returns the current Unix Timestamp.

An example request could be the following URL:

https://example.com/clerk-product-feed.php?salt=f4Ke...A02X&hash=4DFF...340F

Thus by recomputing the hash you can verify that the request issuer knows your private key, and is trustworthy, so that the feed data can be provided safely.

Multi-language feed

📘

This is an optional step

Most webshops should handle language-versioning by creating individual Stores, but it is also possible to achieve in a single feed.

The language must be specified with its exact name, and currently the following are supported:

  • danish
  • dutch
  • english
  • finnish
  • french
  • german
  • italian
  • norwegian
  • portuguese
  • russian
  • spanish
  • swedish

If your language is not on the above list, choose a language using the same root as yours, or simply "english". Clerk.io will still work well but grammar-neutralisation in Search will not optimal

🚧

Important!

All attributes containing text that should be searchable, must have a translation, even if the text is identical between languages. Examples include data like SKU's, brands, types and more.

Otherwise, the attributes might not be searchable for all languages.

Clerk.io supports multi-language texts in a single feed, by sending each attribute as a dictionary consisting of a language and a translation .

Example feed with products and categories:

{ 
  "products": [
    {
      "id": 123,
      "name": {
        "danish":"Grønt lyssværd",
        "english":"Green Lightsaber"
      	},
      "description": {
        "danish":"Antikt rebel lyssværd",
        "english":"Antiuque rebel lightsaber"
      	},
      "price": 999.50,
      "image": {
        "danish":"http://your-store.com/dk/images/antique-rebel-lightsaber.jpg",
        "english":"http://your-store.com/uk/images/antique-rebel-lightsaber.jpg"
      	},
      "url": {
        "danish":"http://your-store.com/dk/antique-rebel-lightsaber",
        "english":"http://your-store.com/uk/antique-rebel-lightsaber"
      	},
      "brand": {
        "danish":"Je'daii",
        "english":"Je'daii"
      	},
      "sku": {
      	"danish": "LUK3-SW0RD",
        "english": "LUK3-SW0RD"
      },
      "categories": [987, 654]
    },
    {
      "id": 456,
      "name": {
        "danish":"Deluxe Dødsstjerne",
        "english":"Death Star Deluxe"
      	},
      "description": {
        "danish":"Dødsstjerne - garanteret idiotsikker",
        "english":"Death Star - Guaranteed idiot proof"
      	},
      "price": 9999999,
      "image": {
        "danish":"http://your-store.com/dk/images/death-star.jpg",
        "english":"http://your-store.com/uk/images/death-star.jpg"
      	},
      "url": {
        "danish":"http://your-store.com/dk/death-star",
        "english":"http://your-store.com/uk/death-star"
      	},
      "brand": {
        "danish":"Imperial A/S",
        "english":"Imperial Inc."
      },
      "sku": {
      	"danish": "D4R7H-V4D3R",
        "english": "D4R7H-V4D3R"
      },
      "categories": [987, 654]
    },
  ],

  "categories": [
    {
      "id": 1,
      "name": {
        "danish":"Gode sager fra imperiet",
        "english":"Imperial Goods"
      	},
      "subcategories": [42, 25],
      "url": {
        "danish":"http://your-store.com/dk/imperial-goods",
        "english":"http://your-store.com/uk/imperial-goods"
      	},
    },
    {
      "id": 1,
      "name": {
        "danish":"Tatooine slik",
        "english":"Tatooine candy"
      	},
      "subcategories": [],
      "url": {
        "danish":"http://your-store.com/dk/imperial-goods/tatooine",
        "english":"http://your-store.com/uk/imperial-goods/tatooine"
      	},
    },
  ]
}

When you set your language configuration in the frontend, the corresponding language will be fetched from the API.

Examples:

<!-- Start of Clerk.io E-commerce Personalisation tool - www.clerk.io -->
  <script type="text/javascript">
    (function(w,d){
      var e=d.createElement('script');e.type='text/javascript';e.async=true;
      e.src=(d.location.protocol=='https:'?'https':'http')+'://cdn.clerk.io/clerk.js';
      var s=d.getElementsByTagName('script')[0];s.parentNode.insertBefore(e,s);
      w.__clerk_q=w.__clerk_q||[];w.Clerk=w.Clerk||function(){w.__clerk_q.push(arguments)};
    })(window,document);

    Clerk('config', {
      key: 'fpUlOw4iBn8nnFfvdKMAH4w2KpALBV5J',
      language: 'danish'
    });
  </script>
  <!-- End of Clerk.io E-commerce Personalisation tool - www.clerk.io -->
curl -X POST \
     -H 'Content-Type: application/json' \
     -d '{"key": "fpUlOw4iBn8nnFfvdKMAH4w2KpALBV5J",
          "limit": 30,
          "offset": 60,
          "visitor": "jgy543",
          "exclude": [42],
          "filter": "price > 100",
          "labels": ["Bestsellers"],
          "attributes":["id","name","url","image"],
          "language": "danish"}' \
     http://api.clerk.io/v2/recommendations/popular

Syncing your data feed

When your feed has the above structure you can upload it to Clerk.io by going to Data Sync in my.clerk.io.

Choose JSON as the sync method, paste the feed url and choose the language of the feed. Click Update Settings once you are done.

Afterwards, click Start Sync to synchronise the feed.

2270

Integration configuration