A/B Testing Results

Here you can fetch reports on how your experiment is performing. You can fetch results for combinations of experiment's groups, goals, and treatment rules.

The parameters fill in the question "in this group, how many times this treatment was triggered, and how many times this goal was matched after said treatment was triggered in the same session."

Weekly experiment results API v3¶

Get results of an experiment between two dates.

POST https://api.giosg.com/api/reporting/v3/orgs/<organization_id>/experiments/<experiment_id>/weekly

This endpoint returns:

200 if the request was successful
401 if you are not authenticated
403 if you do not have active subscription
403 if you do not have access to the organization
403 if you do not have reporting permission

Request format¶

An example request

POST https://api.giosg.com/api/reporting/v3/orgs/aaa1d32b-e82c-11e4-b081-6c4008adf7e8/experiments/b0a1d32b-e82c-11e4-b081-6c4008adf7e8/weekly

Attribute	Type	Description
`start_date`	ISO 8601 string	Experiment results are fetched from this date onwards. e.g. `2017-03-11`
`end_date`	ISO 8601 string	Experiment results are fetched up to this date (inclusive). e.g. `2017-03-17`
`control_group_goal_rule_id`	JSON object	The IDs of the experiment group, rule and goal which others are compared against when computing statistical significance. If not provided, these test statistics will not be calculated.
`group_goal_rule_ids`	Array of JSON objects	Identifiers for the weekly data rows to be included in the result. Each row is identified by combination of group_id, goal_id, and rule_id.

Each object in group_goal_rule_ids contains following values

Attribute	Type	Description
`group_id`	ID	The ID of the experiment group to calculate the results for.
`goal_id`	ID	The ID of the goal to calculated the results for.
`rule_id`	ID	The ID of the treatment rule to calculated the results for.

An example request

{
  "start_date": "2017-03-11",
  "end_date": "2017-03-17",
  "control_group_goal_rule_id": {
    "group_id": "0bf27670-0e18-11e7-a1bb-60f81db14236",
    "goal_id": "0bf27123-ce3b-11e9-a32f-2a2ae2dbcce4",
    "rule_id": "0bf27321-ce3b-11e9-a32f-2a2ae2dbcce4"
  },
  "group_goal_rule_ids": [
    {
      "group_id": "c67b7210-ce3b-11e9-a32f-2a2ae2dbcce4",
      "goal_id": "c67b7123-ce3b-11e9-a32f-2a2ae2dbcce4",
      "rule_id": "c67b7321-ce3b-11e9-a32f-2a2ae2dbcce4"
    },
    {
      "group_id": "0bf27670-0e18-11e7-a1bb-60f81db14236",
      "goal_id": "0bf27123-ce3b-11e9-a32f-2a2ae2dbcce4",
      "rule_id": "0bf27321-ce3b-11e9-a32f-2a2ae2dbcce4"
    }
  ]
}

Response attributes¶

An example response

{
  "start_date": "2017-03-11",
  "end_date": "2017-03-17",
  "control_group_goal_rule_id": {
    "group_id": "0bf27670-0e18-11e7-a1bb-60f81db14236",
    "goal_id": "0bf27123-ce3b-11e9-a32f-2a2ae2dbcce4",
    "rule_id": "0bf27321-ce3b-11e9-a32f-2a2ae2dbcce4"
  },
  "experiment_id": "b0a1d32b-e82c-11e4-b081-6c4008adf7e8",
  "organization_id": "aaa1d32b-e82c-11e4-b081-6c4008adf7e8",
  "by_week": [
    {
      "group_id": "c67b7210-ce3b-11e9-a32f-2a2ae2dbcce4",
      "goal_id": "c67b7123-ce3b-11e9-a32f-2a2ae2dbcce4",
      "rule_id": "c67b7321-ce3b-11e9-a32f-2a2ae2dbcce4",
      "start_time": "2019-09-02T00:00:00.000Z",
      "end_time": "2019-09-08T23:59:59.999Z",
      "unique_visitors_with_treatment_count": 105,
      "unique_visitors_with_goal_after_treatment_count": 20,
      "conversion": "0.109",
      "statistical_significance": "0.82"
    },
    {
      "group_id": "0bf27670-0e18-11e7-a1bb-60f81db14236",
      "goal_id": "0bf27123-ce3b-11e9-a32f-2a2ae2dbcce4",
      "rule_id": "0bf27321-ce3b-11e9-a32f-2a2ae2dbcce4",
      "start_time": "2019-09-02T00:00:00.000Z",
      "end_time": "2019-09-08T23:59:59.999Z",
      "unique_visitors_with_treatment_count": 104,
      "unique_visitors_with_goal_after_treatment_count": 1,
      "conversion": "0.009",
      "statistical_significance": null
    },
    {
      "group_id": "c67b7210-ce3b-11e9-a32f-2a2ae2dbcce4",
      "goal_id": "c67b7123-ce3b-11e9-a32f-2a2ae2dbcce4",
      "rule_id": "c67b7321-ce3b-11e9-a32f-2a2ae2dbcce4",
      "start_time": "2019-09-09T00:00:00.000Z",
      "end_time": "2019-09-15T23:59:59.999Z",
      "unique_visitors_with_treatment_count": 60,
      "unique_visitors_with_goal_after_treatment_count": 3,
      "conversion": "0.109",
      "statistical_significance": "0.82"
    },
    {
      "group_id": "0bf27670-0e18-11e7-a1bb-60f81db14236",
      "goal_id": "0bf27123-ce3b-11e9-a32f-2a2ae2dbcce4",
      "rule_id": "0bf27321-ce3b-11e9-a32f-2a2ae2dbcce4",
      "start_time": "2019-09-09T00:00:00.000Z",
      "end_time": "2019-09-15T23:59:59.999Z",
      "unique_visitors_with_treatment_count": 62,
      "unique_visitors_with_goal_after_treatment_count": 0,
      "conversion": "0.000",
      "statistical_significance": null
    }
  ],
  "totals": [
    {
      "group_id": "c67b7210-ce3b-11e9-a32f-2a2ae2dbcce4",
      "goal_id": "c67b7123-ce3b-11e9-a32f-2a2ae2dbcce4",
      "rule_id": "c67b7321-ce3b-11e9-a32f-2a2ae2dbcce4",
      "unique_visitors_with_treatment_count": 165,
      "unique_visitors_with_goal_after_treatment_count": 23,
      "statistical_significance": "0.82",
      "conversion": "0.139"
    },
    {
      "group_id": "0bf27670-0e18-11e7-a1bb-60f81db14236",
      "goal_id": "0bf27123-ce3b-11e9-a32f-2a2ae2dbcce4",
      "rule_id": "0bf27321-ce3b-11e9-a32f-2a2ae2dbcce4",
      "unique_visitors_with_treatment_count": 166,
      "unique_visitors_with_goal_after_treatment_count": 1,
      "statistical_significance": null,
      "conversion": "0.006"
    }
  ]
}

Root level fields

Attribute	Type	Description
`start_date`	ISO 8601 string	The first date of the interval
`end_date`	ISO 8601 string	The last date of the interval (inclusive)
`organization_id`	ID	The ID of the organization this experiment belongs to.
`experiment_id`	ID	The ID of the experiment.
`control_group_goal_rule_id`	JSON object	The IDs of the group, rule and goal which others are compared against. This group will not have statistical significance calculated.
`by_week`	Array of JSON objects	The weekly results
`totals`	Array of JSON objects	Sums of weekly counts for the whole time range.

Each object in by_week contains following values

Attribute	Type	Description
`start_time`	ISO 8601 string	The starting moment of the week this data point describes
`end_time`	ISO 8601 string	The ending moment of the week this data point describes
`group_id`	ID	The id of the experiment group
`goal_id`	ID	The id of the goal
`rule_id`	ID	The id of the rule
`unique_visitors_with_treatment_count`	integer	The weekly count of unique visitors that were treated with the configured treatment (if a treatment is configured for the experiment)
`unique_visitors_with_goal_after_treatment_count`	integer	The weekly count of unique visitors that achieved the configured goal after being treated
`conversion`	float	The weekly conversion rate calculated by `unique_visitors_with_goal_after_treatment_count/unique_visitors_with_treatment_count`
`statistical_significance`	float or null	The test statistics on whether the group has statistically significant difference in conversion rate compared to control group. See Definition of statistical significance below

Each object in totals contains following values

Attribute	Type	Description
`group_id`	ID	The id of the group
`goal_id`	ID	The id of the goal
`rule_id`	ID	The id of the rule
`unique_visitors_with_treatment_count`	integer	The sum of all weekly counts of unique visitors that were treated with the configured treatment (if a treatment is configured for the experiment).
`unique_visitors_with_goal_after_treatment_count`	integer	The sum of all weekly counts of unique visitors that achieved the configured goal after being treated in this group.
`conversion`	float	The total conversion rate calculated by `unique_visitors_with_goal_after_treatment_count/unique_visitors_with_treatment_count`
`statistical_significance`	float or null	The test statistics on whether the group has statistically significant difference in conversion rate compared to control group. See Definition of statistical significance below

Weekly experiment results API v2¶

Get results on a specific experiment during a time range of n weeks.

The results are calculated on resolution of one week because the results are likely vary a lot on daily basis. Week by week comparison should be more robust.

Unlike other reporting APIs, experiment results are fetched based on the last date in the interval. The data is available only for past weeks. The last full week, starting from Monday, before the given last_date is considered as the first week in the report.

GET https://api.giosg.com/api/reporting/v2/orgs/<organization_id>/experiments/<experiment_id>/weekly/?last_date=<end_date>&control_group_id=<control_group_id>&weeks=<weeks>

This endpoint returns:

200 if the request was successful
401 if you are not authenticated
403 if you do not have active subscription
403 if you do not have access to the organization
403 if you do not have reporting permission
404 if the experiment results were not found

Request parameters¶

An example request

GET https://api.giosg.com/api/reporting/v2/orgs/aaa1d32b-e82c-11e4-b081-6c4008adf7e8/experiments/b0a1d32b-e82c-11e4-b081-6c4008adf7e8/weekly/?end_date=2017-03-17&control_group_id=0bf27670-0e18-11e7-a1bb-60f81db14236&weeks=4

Parameter	Type	Description
`organization_id`	ID	ID of the organization
`experiment_id`	ID	The ID of the experiment
`last_date`	ISO 8601 string	Experiment results are fetched up to this date for the given interval. e.g. `2017-03-20`
`control_group_id`	ID	The ID of the group against which others are compared when computing statistical significance. If not provided, these test statistics will not be calculated.
`weeks`	integer	For how many weeks the results are accumulated.

Response attributes¶

An example response

{
  "weeks": 2,
  "end_date": "2018-10-15",
  "resolution": "week",
  "control_group_id": "0bf27670-0e18-11e7-a1bb-60f81db14236",
  "experiment_id": "b0a1d32b-e82c-11e4-b081-6c4008adf7e8",
  "organization_id": "aaa1d32b-e82c-11e4-b081-6c4008adf7e8",
  "groups_by_week": [
    {
      "group_name": "A (Algorithm)",
      "group_id": "f1b8fda8-0e17-11e7-a5af-60f81db14236",
      "end_date": "2018-10-14",
      "unique_visitor_count": 505,
      "visitor_with_treatment_count": 105,
      "visitor_with_goal_count": 55,
      "conversion": "0.109",
      "confidence_interval_low": "0.123",
      "confidence_interval_high": "0.223",
      "statistical_significance": "0.82"
    },
    {
      "group_name": "B (Random)",
      "group_id": "0bf27670-0e18-11e7-a1bb-60f81db1423",
      "end_date": "2018-10-14",
      "unique_visitor_count": 1050,
      "visitor_with_treatment_count": 120,
      "visitor_with_goal_count": 1,
      "conversion": "0.00095",
      "confidence_interval_low": "0.02",
      "confidence_interval_high": "0.10",
      "statistical_significance": null
    },
    {
      "group_name": "A (Algorithm)",
      "group_id": "f1b8fda8-0e17-11e7-a5af-60f81db14236",
      "end_date": "2018-10-07",
      "unique_visitor_count": 505,
      "visitor_with_treatment_count": 105,
      "visitor_with_goal_count": 55,
      "conversion": "0.109",
      "confidence_interval_low": "0.115",
      "confidence_interval_high": "0.323",
      "statistical_significance": "0.72"
    },
    {
      "group_name": "B (Random)",
      "group_id": "0bf27670-0e18-11e7-a1bb-60f81db1423",
      "end_date": "2018-10-07",
      "unique_visitor_count": 1045,
      "visitor_with_treatment_count": 120,
      "visitor_with_goal_count": 1,
      "conversion": "0.00095",
      "confidence_interval_low": "0.03",
      "confidence_interval_high": "0.1",
      "statistical_significance": null
    }
  ],
  "group_totals": [
    {
      "group_name": "A (Algorithm)",
      "group_id": "f1b8fda8-0e17-11e7-a5af-60f81db14236",
      "unique_visitor_count": 1010,
      "visitor_with_treatment_count": 210,
      "visitor_with_goal_count": 110,
      "statistical_significance": "0.87",
      "conversion": "0.109"
    },
    {
      "group_name": "B (Random)",
      "group_id": "0bf27670-0e18-11e7-a1bb-60f81db1423",
      "unique_visitor_count": 2095,
      "visitor_with_treatment_count": 240,
      "visitor_with_goal_count": 2,
      "statistical_significance": null,
      "conversion": "0.00095"
    }
  ]
}

Root level fields

Attribute	Type	Description
`end_date`	ISO 8601 string	The last date of the interval
`resolution`	string	Resolution of the intervals. Always "week"
`units`	integer	How many intervals are included. If the interval is "week" and units is 4 then there are 4 weeks of data in the response.
`organization_id`	ID	The organization_id this experiment belongs to
`experiment_id`	ID	Id of the experiment
`control_group_id`	ID	The group id against which the difference of other groups is tested. This group will not have statistical significance calculated because no reason to compare to itself.
`groups_by_week`	Array of JSON objects	The weekly results for each group
`group_totals`	Array of JSON objects	Sums of weekly counts per group for the whole time range.

Each object in groups_by_week contains these values

Attribute	Type	Description
`end_date`	ISO 8601 string	The last date of the week this data point describes
`group_name`	string	The name of the Experiment group
`group_id`	ID	The id of the group
`unique_visitor_count`	integer	The weekly count of unique visitors in this group
`visitor_with_treatment_count`	integer	The weekly count of unique visitors that were treated with the configured treatment in this group (if a treatment is configured for the experiment)
`visitor_with_goal_count`	integer	The weekly count of unique visitors that achieved the configured goal in this group
`conversion`	float	The weekly conversion rate calculated by `visitor_with_goal_count/unique_visitor_count`
`confidence_interval_low`	float	Lower bound of 95% confidence interval, see Definition of Confidence Intervals below
`confidence_interval_high`	float	Upper bound of 95% confidence interval, see Definition of Confidence Intervals below
`statistical_significance`	float	The test statistics on whether the group has statistically significant difference in conversion rate compared to control group. See Definition of statistical significance below

Each object in group_totals contains these values

Attribute	Type	Description
`group_name`	string	The name of the Experiment group
`group_id`	ID	The id of the group
`unique_visitor_count`	integer	The sum of all weekly counts of unique visitors that were assigned to this group during the time period. If the same visitor was assigned to the same group the next week they will be counted as an unique visitor for both weeks.
`visitor_with_treatment_count`	integer	The sum of all weekly counts of unique visitors that were treated with the configured treatment (if a treatment is configured for the experiment).
`visitor_with_goal_count`	integer	The sum of all weekly counts of unique visitors that achieved the configured goal in this group.
`statistical_significance`	float	The test statistics on whether the group has statistically significant difference in conversion rate compared to control group. See Definition of statistical significance below
`conversion`	float	The total conversion rate calculated by `visitor_with_goal_count/unique_visitor_count`

Weekly cumulated experiment results API v1¶

Get weekly accumulated results on a specific experiment. By "weekly accumulated", we mean that e.g. the unique visitor counts, visitors with goal counts are cumulated over the period. The first week contains counts only from the first week of data. The counts for the second week contain both first and second week et cetera.

The results are calculated on resolution of one week because the results are likely vary a lot on daily basis. Week by week comparison should be more robust.

Unlike other reporting APIs, experiment results are fetched based on the last date in the interval. This is due to cumulative calculation requirements.

GET https://api.giosg.com/api/reporting/v1/orgs/<organization_id>/experiments/<experiment_id>/weekly/?end_date=<end_date>&control_group_id=<control_group_id>&weeks=<weeks>

Request parameters¶

An example request

GET https://api.giosg.com/api/reporting/v1/orgs/aaa1d32b-e82c-11e4-b081-6c4008adf7e8/experiments/b0a1d32b-e82c-11e4-b081-6c4008adf7e8/weekly/?end_date=2017-03-17&control_group_id=0bf27670-0e18-11e7-a1bb-60f81db14236&weeks=4

Parameter	Type	Description
`organization_id`	ID	ID of the organization
`experiment_id`	ID	The ID of the experiment
`end_date`	ISO 8601 string	Experiment results are fetched up to this date for the given interval. e.g. `2017-03-20`
`control_group_id`	ID	The ID of the group against which others are compared when computing statistical significance. If not provided, these test statistics will not be calculated.
`weeks`	integer	For how many weeks the results are accumulated. The valid week numbers at the moment are 1, 4 or 8 weeks.

Response attributes¶

An example response

{
  "weeks": 4,
  "end_date": "2017-03-17",
  "resolution": "week",
  "control_group_id": "0bf27670-0e18-11e7-a1bb-60f81db14236",
  "experiment_id": "b0a1d32b-e82c-11e4-b081-6c4008adf7e8",
  "organization_id": "aaa1d32b-e82c-11e4-b081-6c4008adf7e8",
  "groups_by_week": [{
      "group_name": "A (Algorithm)",
      "group_id": "f1b8fda8-0e17-11e7-a5af-60f81db14236",
      "end_date": "2017-02-21",
      "unique_visitor_count": 14234,
      "visitor_with_treatment_count": 1234,
      "visitor_with_goal_count": 123,
      "conversion": "0.18",
      "confidence_interval_low": "0.123",
      "confidence_interval_high": "0.223",
      "statistical_significance": "0.82"
    }, {
      "group_name": "B (Random)",
      "group_id": "0bf27670-0e18-11e7-a1bb-60f81db1423",
      "end_date": "2017-02-21",
      "unique_visitor_count": 1235,
      "visitor_with_treatment_count": 1352,
      "visitor_with_goal_count": 1234,
      "conversion": "0.05",
      "confidence_interval_low": "0.02",
      "confidence_interval_high": "0.10",
      "statistical_significance": null
    }, {
      "group_name": "A (Algorithm)",
      "group_id": "f1b8fda8-0e17-11e7-a5af-60f81db14236",
      "end_date": "2017-03-01",
      "unique_visitor_count": 19234,
      "visitor_with_treatment_count": 24124,
      "visitor_with_goal_count": 4214,
      "conversion": "0.18",
      "confidence_interval_low": "0.115",
      "confidence_interval_high": "0.323",
      "statistical_significance": "0.72"
    }, {
      "group_name": "B (Random)",
      "group_id": "0bf27670-0e18-11e7-a1bb-60f81db1423",
      "end_date": "2017-03-01",
      "unique_visitor_count": 14234,
      "visitor_with_treatment_count": 1234,
      "visitor_with_goal_count": 123,
      "conversion": "0.07",
      "confidence_interval_low": "0.03",
      "confidence_interval_high": "0.1",
      "statistical_significance": null
    },
    ...
  ],
  "group_sizes": [{
      "group_name": "A (Algorithm)",
      "group_id": "f1b8fda8-0e17-11e7-a5af-60f81db14236",
      "end_date": "2017-03-01",
      "unique_visitor_count": 19234
    }, {
      "group_name": "B (Random)",
      "group_id": "0bf27670-0e18-11e7-a1bb-60f81db1423",
      "end_date": "2017-03-01",
      "unique_visitor_count": 14234
    }, {
      "group_name": "A (Algorithm)",
      "group_id": "f1b8fda8-0e17-11e7-a5af-60f81db14236",
      "end_date": "2017-02-21",
      "unique_visitor_count": 19234
    }, {
      "group_name": "B (Random)",
      "group_id": "0bf27670-0e18-11e7-a1bb-60f81db1423",
      "end_date": "2017-02-21",
      "unique_visitor_count": 14234
    }
    ...
  ]
}

Root level fields

Attribute	Type	Description
`end_date`	ISO 8601 string	The last date of the interval
`resolution`	string	Resolution of the intervals. Always "week"
`units`	integer	How many intervals are included. If the interval is "week" and units is 4 then there are 4 weeks of cumulative data in the response.
`organization_id`	ID	The organization_id this experiment belongs to
`experiment_id`	ID	Id of the experiment
`control_group_id`	ID	The group id against which the difference of other groups is tested. This group will not have statistical significance calculated because no reason to compare to itself.
`groups_by_week`	Array of JSON objects	The cumulative weekly results for each group
`group_sizes`	Array of JSON objects	Experiment group sizes for each week. These counts are not cumulative and thus this data can be used to determine group sizes for each individual week.

Each object in cumulative_groups_by_week contains these values

Attribute	Type	Description
`group_name`	string	The name of the Experiment group
`group_id`	ID	The id of the group
`unique_visitor_count`	integer	The cumulative weekly counts of unique visitors ordered from earliest to most recent
`visitor_with_treatment_count`	integer	The cumulative weekly counts of unique visitors that were treated with the configured treatment (if a treatment is configured for the experiment)
`visitor_with_goal_count`	integer	The cumulative weekly counts of unique visitors that achieved the configured goal
`conversion`	float	The weekly conversion rate calculated by `visitor_with_goal_count/unique_visitor_count`
`confidence_interval_low`	float	Lower bound of 95% confidence interval, see Definition of Confidence Intervals below
`confidence_interval_high`	float	Upper bound of 95% confidence interval, see Definition of Confidence Intervals below
`statistical_significance`	float	The test statistics on whether the group has statistically significant difference in conversion rate compared to control group. See Definition of statistical significance below

Each object in group_sizes contains these values

Attribute	Type	Description
`group_name`	string	The name of the Experiment group
`group_id`	ID	The id of the group
`end_date`	ISO 8601 string	The last date of the interval
`unique_visitor_count`	integer	The count of unique visitors that were assigned to this group during this week. If the same visitor was assigned to the same group the next week they will be counted as an unique visitor for both weeks.

Definition of confidence intervals¶

The visitors are modeled by a binomial distribution with parameter p, where success is defined as reaching the goal. We model the posterior distribution for the conversion rate p with a beta distribution. The 95% confidence interval is the most narrow interval that captures 95% of the probability mass. This is also known as highest posterior density or highest density intervals.

Definition of statistical significance¶

At giosg, we use a Bayesian approach to estimate the statistical significance of differences in conversion rates. The test is essentially a null-hypothesis significance test (NHST) were we test whether the treatment group has a statistically significantly higher conversion rate than the control group. Both groups are modeled by Beta-distributions with uniform uninformative priors (Beta(1, 1)), and the statistical significance is estimated in a Monte Carlo simulation. Due to the approach, you might see slight variations in the statistical significance for the same results, but only in the third decimal.