Skip to content

A/B Testing Results

Here you can fetch reports on how your experiment is performing. You can fetch results for combinations of experiment's groups, goals, and treatment rules.

The parameters fill in the question "in this group, how many times this treatment was triggered, and how many times this goal was matched after said treatment was triggered in the same session."

Weekly experiment results API v3

Get results of an experiment between two dates.

POST https://api.giosg.com/api/reporting/v3/orgs/<organization_id>/experiments/<experiment_id>/weekly

This endpoint returns:

  • 200 if the request was successful
  • 401 if you are not authenticated
  • 403 if you do not have active subscription
  • 403 if you do not have access to the organization
  • 403 if you do not have reporting permission

Request format

An example request

1
POST https://api.giosg.com/api/reporting/v3/orgs/aaa1d32b-e82c-11e4-b081-6c4008adf7e8/experiments/b0a1d32b-e82c-11e4-b081-6c4008adf7e8/weekly
Attribute Type Description
start_date ISO 8601 string Experiment results are fetched from this date onwards. e.g. 2017-03-11
end_date ISO 8601 string Experiment results are fetched up to this date (inclusive). e.g. 2017-03-17
control_group_goal_rule_id JSON object The IDs of the experiment group, rule and goal which others are compared against when computing statistical significance. If not provided, these test statistics will not be calculated.
group_goal_rule_ids Array of JSON objects Identifiers for the weekly data rows to be included in the result. Each row is identified by combination of group_id, goal_id, and rule_id.

Each object in group_goal_rule_ids contains following values

Attribute Type Description
group_id ID The ID of the experiment group to calculate the results for.
goal_id ID The ID of the goal to calculated the results for.
rule_id ID The ID of the treatment rule to calculated the results for.

An example request

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
{
  "start_date": "2017-03-11",
  "end_date": "2017-03-17",
  "control_group_goal_rule_id": {
    "group_id": "0bf27670-0e18-11e7-a1bb-60f81db14236",
    "goal_id": "0bf27123-ce3b-11e9-a32f-2a2ae2dbcce4",
    "rule_id": "0bf27321-ce3b-11e9-a32f-2a2ae2dbcce4"
  },
  "group_goal_rule_ids": [
    {
      "group_id": "c67b7210-ce3b-11e9-a32f-2a2ae2dbcce4",
      "goal_id": "c67b7123-ce3b-11e9-a32f-2a2ae2dbcce4",
      "rule_id": "c67b7321-ce3b-11e9-a32f-2a2ae2dbcce4"
    },
    {
      "group_id": "0bf27670-0e18-11e7-a1bb-60f81db14236",
      "goal_id": "0bf27123-ce3b-11e9-a32f-2a2ae2dbcce4",
      "rule_id": "0bf27321-ce3b-11e9-a32f-2a2ae2dbcce4"
    }
  ]
}

Response attributes

An example response

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
{
  "start_date": "2017-03-11",
  "end_date": "2017-03-17",
  "control_group_goal_rule_id": {
    "group_id": "0bf27670-0e18-11e7-a1bb-60f81db14236",
    "goal_id": "0bf27123-ce3b-11e9-a32f-2a2ae2dbcce4",
    "rule_id": "0bf27321-ce3b-11e9-a32f-2a2ae2dbcce4"
  },
  "experiment_id": "b0a1d32b-e82c-11e4-b081-6c4008adf7e8",
  "organization_id": "aaa1d32b-e82c-11e4-b081-6c4008adf7e8",
  "by_week": [
    {
      "group_id": "c67b7210-ce3b-11e9-a32f-2a2ae2dbcce4",
      "goal_id": "c67b7123-ce3b-11e9-a32f-2a2ae2dbcce4",
      "rule_id": "c67b7321-ce3b-11e9-a32f-2a2ae2dbcce4",
      "start_time": "2019-09-02T00:00:00.000Z",
      "end_time": "2019-09-08T23:59:59.999Z",
      "unique_visitors_with_treatment_count": 105,
      "unique_visitors_with_goal_after_treatment_count": 20,
      "conversion": "0.109",
      "statistical_significance": "0.82"
    },
    {
      "group_id": "0bf27670-0e18-11e7-a1bb-60f81db14236",
      "goal_id": "0bf27123-ce3b-11e9-a32f-2a2ae2dbcce4",
      "rule_id": "0bf27321-ce3b-11e9-a32f-2a2ae2dbcce4",
      "start_time": "2019-09-02T00:00:00.000Z",
      "end_time": "2019-09-08T23:59:59.999Z",
      "unique_visitors_with_treatment_count": 104,
      "unique_visitors_with_goal_after_treatment_count": 1,
      "conversion": "0.009",
      "statistical_significance": null
    },
    {
      "group_id": "c67b7210-ce3b-11e9-a32f-2a2ae2dbcce4",
      "goal_id": "c67b7123-ce3b-11e9-a32f-2a2ae2dbcce4",
      "rule_id": "c67b7321-ce3b-11e9-a32f-2a2ae2dbcce4",
      "start_time": "2019-09-09T00:00:00.000Z",
      "end_time": "2019-09-15T23:59:59.999Z",
      "unique_visitors_with_treatment_count": 60,
      "unique_visitors_with_goal_after_treatment_count": 3,
      "conversion": "0.109",
      "statistical_significance": "0.82"
    },
    {
      "group_id": "0bf27670-0e18-11e7-a1bb-60f81db14236",
      "goal_id": "0bf27123-ce3b-11e9-a32f-2a2ae2dbcce4",
      "rule_id": "0bf27321-ce3b-11e9-a32f-2a2ae2dbcce4",
      "start_time": "2019-09-09T00:00:00.000Z",
      "end_time": "2019-09-15T23:59:59.999Z",
      "unique_visitors_with_treatment_count": 62,
      "unique_visitors_with_goal_after_treatment_count": 0,
      "conversion": "0.000",
      "statistical_significance": null
    }
  ],
  "totals": [
    {
      "group_id": "c67b7210-ce3b-11e9-a32f-2a2ae2dbcce4",
      "goal_id": "c67b7123-ce3b-11e9-a32f-2a2ae2dbcce4",
      "rule_id": "c67b7321-ce3b-11e9-a32f-2a2ae2dbcce4",
      "unique_visitors_with_treatment_count": 165,
      "unique_visitors_with_goal_after_treatment_count": 23,
      "statistical_significance": "0.82",
      "conversion": "0.139"
    },
    {
      "group_id": "0bf27670-0e18-11e7-a1bb-60f81db14236",
      "goal_id": "0bf27123-ce3b-11e9-a32f-2a2ae2dbcce4",
      "rule_id": "0bf27321-ce3b-11e9-a32f-2a2ae2dbcce4",
      "unique_visitors_with_treatment_count": 166,
      "unique_visitors_with_goal_after_treatment_count": 1,
      "statistical_significance": null,
      "conversion": "0.006"
    }
  ]
}

Root level fields

Attribute Type Description
start_date ISO 8601 string The first date of the interval
end_date ISO 8601 string The last date of the interval (inclusive)
organization_id ID The ID of the organization this experiment belongs to.
experiment_id ID The ID of the experiment.
control_group_goal_rule_id JSON object The IDs of the group, rule and goal which others are compared against. This group will not have statistical significance calculated.
by_week Array of JSON objects The weekly results
totals Array of JSON objects Sums of weekly counts for the whole time range.

Each object in by_week contains following values

Attribute Type Description
start_time ISO 8601 string The starting moment of the week this data point describes
end_time ISO 8601 string The ending moment of the week this data point describes
group_id ID The id of the experiment group
goal_id ID The id of the goal
rule_id ID The id of the rule
unique_visitors_with_treatment_count integer The weekly count of unique visitors that were treated with the configured treatment (if a treatment is configured for the experiment)
unique_visitors_with_goal_after_treatment_count integer The weekly count of unique visitors that achieved the configured goal after being treated
conversion float The weekly conversion rate calculated by unique_visitors_with_goal_after_treatment_count/unique_visitors_with_treatment_count
statistical_significance float or null The test statistics on whether the group has statistically significant difference in conversion rate compared to control group. See Definition of statistical significance below

Each object in totals contains following values

Attribute Type Description
group_id ID The id of the group
goal_id ID The id of the goal
rule_id ID The id of the rule
unique_visitors_with_treatment_count integer The sum of all weekly counts of unique visitors that were treated with the configured treatment (if a treatment is configured for the experiment).
unique_visitors_with_goal_after_treatment_count integer The sum of all weekly counts of unique visitors that achieved the configured goal after being treated in this group.
conversion float The total conversion rate calculated by unique_visitors_with_goal_after_treatment_count/unique_visitors_with_treatment_count
statistical_significance float or null The test statistics on whether the group has statistically significant difference in conversion rate compared to control group. See Definition of statistical significance below

Weekly experiment results API v2

Get results on a specific experiment during a time range of n weeks.

The results are calculated on resolution of one week because the results are likely vary a lot on daily basis. Week by week comparison should be more robust.

Unlike other reporting APIs, experiment results are fetched based on the last date in the interval. The data is available only for past weeks. The last full week, starting from Monday, before the given last_date is considered as the first week in the report.

GET https://api.giosg.com/api/reporting/v2/orgs/<organization_id>/experiments/<experiment_id>/weekly/?last_date=<end_date>&control_group_id=<control_group_id>&weeks=<weeks>

This endpoint returns:

  • 200 if the request was successful
  • 401 if you are not authenticated
  • 403 if you do not have active subscription
  • 403 if you do not have access to the organization
  • 403 if you do not have reporting permission
  • 404 if the experiment results were not found

Request parameters

An example request

1
GET https://api.giosg.com/api/reporting/v2/orgs/aaa1d32b-e82c-11e4-b081-6c4008adf7e8/experiments/b0a1d32b-e82c-11e4-b081-6c4008adf7e8/weekly/?end_date=2017-03-17&control_group_id=0bf27670-0e18-11e7-a1bb-60f81db14236&weeks=4
Parameter Type Description
organization_id ID ID of the organization
experiment_id ID The ID of the experiment
last_date ISO 8601 string Experiment results are fetched up to this date for the given interval. e.g. 2017-03-20
control_group_id ID The ID of the group against which others are compared when computing statistical significance. If not provided, these test statistics will not be calculated.
weeks integer For how many weeks the results are accumulated.

Response attributes

An example response

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
{
  "weeks": 2,
  "end_date": "2018-10-15",
  "resolution": "week",
  "control_group_id": "0bf27670-0e18-11e7-a1bb-60f81db14236",
  "experiment_id": "b0a1d32b-e82c-11e4-b081-6c4008adf7e8",
  "organization_id": "aaa1d32b-e82c-11e4-b081-6c4008adf7e8",
  "groups_by_week": [
    {
      "group_name": "A (Algorithm)",
      "group_id": "f1b8fda8-0e17-11e7-a5af-60f81db14236",
      "end_date": "2018-10-14",
      "unique_visitor_count": 505,
      "visitor_with_treatment_count": 105,
      "visitor_with_goal_count": 55,
      "conversion": "0.109",
      "confidence_interval_low": "0.123",
      "confidence_interval_high": "0.223",
      "statistical_significance": "0.82"
    },
    {
      "group_name": "B (Random)",
      "group_id": "0bf27670-0e18-11e7-a1bb-60f81db1423",
      "end_date": "2018-10-14",
      "unique_visitor_count": 1050,
      "visitor_with_treatment_count": 120,
      "visitor_with_goal_count": 1,
      "conversion": "0.00095",
      "confidence_interval_low": "0.02",
      "confidence_interval_high": "0.10",
      "statistical_significance": null
    },
    {
      "group_name": "A (Algorithm)",
      "group_id": "f1b8fda8-0e17-11e7-a5af-60f81db14236",
      "end_date": "2018-10-07",
      "unique_visitor_count": 505,
      "visitor_with_treatment_count": 105,
      "visitor_with_goal_count": 55,
      "conversion": "0.109",
      "confidence_interval_low": "0.115",
      "confidence_interval_high": "0.323",
      "statistical_significance": "0.72"
    },
    {
      "group_name": "B (Random)",
      "group_id": "0bf27670-0e18-11e7-a1bb-60f81db1423",
      "end_date": "2018-10-07",
      "unique_visitor_count": 1045,
      "visitor_with_treatment_count": 120,
      "visitor_with_goal_count": 1,
      "conversion": "0.00095",
      "confidence_interval_low": "0.03",
      "confidence_interval_high": "0.1",
      "statistical_significance": null
    }
  ],
  "group_totals": [
    {
      "group_name": "A (Algorithm)",
      "group_id": "f1b8fda8-0e17-11e7-a5af-60f81db14236",
      "unique_visitor_count": 1010,
      "visitor_with_treatment_count": 210,
      "visitor_with_goal_count": 110,
      "statistical_significance": "0.87",
      "conversion": "0.109"
    },
    {
      "group_name": "B (Random)",
      "group_id": "0bf27670-0e18-11e7-a1bb-60f81db1423",
      "unique_visitor_count": 2095,
      "visitor_with_treatment_count": 240,
      "visitor_with_goal_count": 2,
      "statistical_significance": null,
      "conversion": "0.00095"
    }
  ]
}

Root level fields

Attribute Type Description
end_date ISO 8601 string The last date of the interval
resolution string Resolution of the intervals. Always "week"
units integer How many intervals are included. If the interval is "week" and units is 4 then there are 4 weeks of data in the response.
organization_id ID The organization_id this experiment belongs to
experiment_id ID Id of the experiment
control_group_id ID The group id against which the difference of other groups is tested. This group will not have statistical significance calculated because no reason to compare to itself.
groups_by_week Array of JSON objects The weekly results for each group
group_totals Array of JSON objects Sums of weekly counts per group for the whole time range.

Each object in groups_by_week contains these values

Attribute Type Description
end_date ISO 8601 string The last date of the week this data point describes
group_name string The name of the Experiment group
group_id ID The id of the group
unique_visitor_count integer The weekly count of unique visitors in this group
visitor_with_treatment_count integer The weekly count of unique visitors that were treated with the configured treatment in this group (if a treatment is configured for the experiment)
visitor_with_goal_count integer The weekly count of unique visitors that achieved the configured goal in this group
conversion float The weekly conversion rate calculated by visitor_with_goal_count/unique_visitor_count
confidence_interval_low float Lower bound of 95% confidence interval, see Definition of Confidence Intervals below
confidence_interval_high float Upper bound of 95% confidence interval, see Definition of Confidence Intervals below
statistical_significance float The test statistics on whether the group has statistically significant difference in conversion rate compared to control group. See Definition of statistical significance below

Each object in group_totals contains these values

Attribute Type Description
group_name string The name of the Experiment group
group_id ID The id of the group
unique_visitor_count integer The sum of all weekly counts of unique visitors that were assigned to this group during the time period. If the same visitor was assigned to the same group the next week they will be counted as an unique visitor for both weeks.
visitor_with_treatment_count integer The sum of all weekly counts of unique visitors that were treated with the configured treatment (if a treatment is configured for the experiment).
visitor_with_goal_count integer The sum of all weekly counts of unique visitors that achieved the configured goal in this group.
statistical_significance float The test statistics on whether the group has statistically significant difference in conversion rate compared to control group. See Definition of statistical significance below
conversion float The total conversion rate calculated by visitor_with_goal_count/unique_visitor_count

Weekly cumulated experiment results API v1

Get weekly accumulated results on a specific experiment. By "weekly accumulated", we mean that e.g. the unique visitor counts, visitors with goal counts are cumulated over the period. The first week contains counts only from the first week of data. The counts for the second week contain both first and second week et cetera.

The results are calculated on resolution of one week because the results are likely vary a lot on daily basis. Week by week comparison should be more robust.

Unlike other reporting APIs, experiment results are fetched based on the last date in the interval. This is due to cumulative calculation requirements.

GET https://api.giosg.com/api/reporting/v1/orgs/<organization_id>/experiments/<experiment_id>/weekly/?end_date=<end_date>&control_group_id=<control_group_id>&weeks=<weeks>

Request parameters

An example request

1
GET https://api.giosg.com/api/reporting/v1/orgs/aaa1d32b-e82c-11e4-b081-6c4008adf7e8/experiments/b0a1d32b-e82c-11e4-b081-6c4008adf7e8/weekly/?end_date=2017-03-17&control_group_id=0bf27670-0e18-11e7-a1bb-60f81db14236&weeks=4
Parameter Type Description
organization_id ID ID of the organization
experiment_id ID The ID of the experiment
end_date ISO 8601 string Experiment results are fetched up to this date for the given interval. e.g. 2017-03-20
control_group_id ID The ID of the group against which others are compared when computing statistical significance. If not provided, these test statistics will not be calculated.
weeks integer For how many weeks the results are accumulated. The valid week numbers at the moment are 1, 4 or 8 weeks.

Response attributes

An example response

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
{
  "weeks": 4,
  "end_date": "2017-03-17",
  "resolution": "week",
  "control_group_id": "0bf27670-0e18-11e7-a1bb-60f81db14236",
  "experiment_id": "b0a1d32b-e82c-11e4-b081-6c4008adf7e8",
  "organization_id": "aaa1d32b-e82c-11e4-b081-6c4008adf7e8",
  "groups_by_week": [{
      "group_name": "A (Algorithm)",
      "group_id": "f1b8fda8-0e17-11e7-a5af-60f81db14236",
      "end_date": "2017-02-21",
      "unique_visitor_count": 14234,
      "visitor_with_treatment_count": 1234,
      "visitor_with_goal_count": 123,
      "conversion": "0.18",
      "confidence_interval_low": "0.123",
      "confidence_interval_high": "0.223",
      "statistical_significance": "0.82"
    }, {
      "group_name": "B (Random)",
      "group_id": "0bf27670-0e18-11e7-a1bb-60f81db1423",
      "end_date": "2017-02-21",
      "unique_visitor_count": 1235,
      "visitor_with_treatment_count": 1352,
      "visitor_with_goal_count": 1234,
      "conversion": "0.05",
      "confidence_interval_low": "0.02",
      "confidence_interval_high": "0.10",
      "statistical_significance": null
    }, {
      "group_name": "A (Algorithm)",
      "group_id": "f1b8fda8-0e17-11e7-a5af-60f81db14236",
      "end_date": "2017-03-01",
      "unique_visitor_count": 19234,
      "visitor_with_treatment_count": 24124,
      "visitor_with_goal_count": 4214,
      "conversion": "0.18",
      "confidence_interval_low": "0.115",
      "confidence_interval_high": "0.323",
      "statistical_significance": "0.72"
    }, {
      "group_name": "B (Random)",
      "group_id": "0bf27670-0e18-11e7-a1bb-60f81db1423",
      "end_date": "2017-03-01",
      "unique_visitor_count": 14234,
      "visitor_with_treatment_count": 1234,
      "visitor_with_goal_count": 123,
      "conversion": "0.07",
      "confidence_interval_low": "0.03",
      "confidence_interval_high": "0.1",
      "statistical_significance": null
    },
    ...
  ],
  "group_sizes": [{
      "group_name": "A (Algorithm)",
      "group_id": "f1b8fda8-0e17-11e7-a5af-60f81db14236",
      "end_date": "2017-03-01",
      "unique_visitor_count": 19234
    }, {
      "group_name": "B (Random)",
      "group_id": "0bf27670-0e18-11e7-a1bb-60f81db1423",
      "end_date": "2017-03-01",
      "unique_visitor_count": 14234
    }, {
      "group_name": "A (Algorithm)",
      "group_id": "f1b8fda8-0e17-11e7-a5af-60f81db14236",
      "end_date": "2017-02-21",
      "unique_visitor_count": 19234
    }, {
      "group_name": "B (Random)",
      "group_id": "0bf27670-0e18-11e7-a1bb-60f81db1423",
      "end_date": "2017-02-21",
      "unique_visitor_count": 14234
    }
    ...
  ]
}

Root level fields

Attribute Type Description
end_date ISO 8601 string The last date of the interval
resolution string Resolution of the intervals. Always "week"
units integer How many intervals are included. If the interval is "week" and units is 4 then there are 4 weeks of cumulative data in the response.
organization_id ID The organization_id this experiment belongs to
experiment_id ID Id of the experiment
control_group_id ID The group id against which the difference of other groups is tested. This group will not have statistical significance calculated because no reason to compare to itself.
groups_by_week Array of JSON objects The cumulative weekly results for each group
group_sizes Array of JSON objects Experiment group sizes for each week. These counts are not cumulative and thus this data can be used to determine group sizes for each individual week.

Each object in cumulative_groups_by_week contains these values

Attribute Type Description
group_name string The name of the Experiment group
group_id ID The id of the group
unique_visitor_count integer The cumulative weekly counts of unique visitors ordered from earliest to most recent
visitor_with_treatment_count integer The cumulative weekly counts of unique visitors that were treated with the configured treatment (if a treatment is configured for the experiment)
visitor_with_goal_count integer The cumulative weekly counts of unique visitors that achieved the configured goal
conversion float The weekly conversion rate calculated by visitor_with_goal_count/unique_visitor_count
confidence_interval_low float Lower bound of 95% confidence interval, see Definition of Confidence Intervals below
confidence_interval_high float Upper bound of 95% confidence interval, see Definition of Confidence Intervals below
statistical_significance float The test statistics on whether the group has statistically significant difference in conversion rate compared to control group. See Definition of statistical significance below

Each object in group_sizes contains these values

Attribute Type Description
group_name string The name of the Experiment group
group_id ID The id of the group
end_date ISO 8601 string The last date of the interval
unique_visitor_count integer The count of unique visitors that were assigned to this group during this week. If the same visitor was assigned to the same group the next week they will be counted as an unique visitor for both weeks.

Definition of confidence intervals

The visitors are modeled by a binomial distribution with parameter p, where success is defined as reaching the goal. We model the posterior distribution for the conversion rate p with a beta distribution. The 95% confidence interval is the most narrow interval that captures 95% of the probability mass. This is also known as highest posterior density or highest density intervals.

Definition of statistical significance

At giosg, we use a Bayesian approach to estimate the statistical significance of differences in conversion rates. The test is essentially a null-hypothesis significance test (NHST) were we test whether the treatment group has a statistically significantly higher conversion rate than the control group. Both groups are modeled by Beta-distributions with uniform uninformative priors (Beta(1, 1)), and the statistical significance is estimated in a Monte Carlo simulation. Due to the approach, you might see slight variations in the statistical significance for the same results, but only in the third decimal.