5  Resampling with code

Chapter 2 used simulation and resampling from tables of random numbers, dice, and coins. Making random choices in this way can make it easier to understand the process, but of course, physical methods of making random outcomes can be slow and boring.

We saw that short computer programs can do a huge number of resampling trials in a less than a second. The flexibility of a programming language makes it possible to simulate many different outcomes and tests.

Programs can build up tables of random numbers, and do basic tasks like counting the number of values in a row or taking proportions. With these simple tools, we can simulate many problems in probability and statistics.

In this chapter, we will model another problem using Python, but this chapter will add three new things.

• The problem we will work on is a little different from the ambulances problem from Chapter 2. It is a real problem about deciding whether a new cancer treatment is better than the alternatives, and it introduces the idea of making a model of the world, to ask questions about chances and probabilities.

• We will slow down a little to emphasize the steps in solving this kind of problem. First we work out how to simulate a single trial. Then we work out how to run many simulated trials.

• We sprinted through the code in Chapter 2, with the promise we would come back to the details. Here we go into more detail about some ideas from the code in the last chapter. These are:

  • Storing several values together in one place, with arrays.
  • Using functions (code recipes) to apply procedures.
  • Comparing numbers to other numbers.
  • Counting numbers that match a condition.

In the next chapter, we will talk more about using arrays to store results, and for loops to repeat a procedure many times.

5.1 Statistics and probability

We have already emphasized that statistics is a way of drawing conclusions about data from the real world, in the presence of random variation; probability is the way of reasoning about random variation. This chapter introduces our first statistical problem, where we use probability to draw conclusions about some important data — about a potential cure for a type of cancer. We will not make much of the distinction between probability and statistics here, but we will come back to it several times in later chapters.

5.2 A new treatment for Burkitt lymphoma

Burkitt lymphoma is an unusual cancer of the lymphatic system. The lymphatic system is a vein-like network throughout the body that is involved in the immune reaction to disease. In developed countries, with standard treatment, the cure rate for Burkitt lymphoma is about 90%.

In 2006, researchers at the US National Cancer Institute (NCI), tested a new treatment for Burkitt lymphoma (Dunleavy et al. 2006). They gave the new treatment to 17 patients, and found that all 17 patients were doing well after two years or more of follow up. By “doing well”, we mean that their lymphoma had not progressed; as a short-hand, we will say that these patients were “cured”, but of course, we do not know what happened to them after this follow up.

Here is where we put on our statistical hat and ask ourselves the following question — how surprised are we that the NCI researchers saw their result of 17 out of 17 patients cured?

At this stage you might and should ask, what could we possibly mean by “surprised”? That is a good and important question, and we will discuss that much more in the chapters to come. For now, please bear with us as we do a thought experiment.

Let us forget the 17 out of 17 result of the NCI study for a moment. Imagine that there is another hospital, called Saint Hypothetical General, just down the road from the NCI, that was also treating 17 patients with Burkitt lymphoma. Saint Hypothetical were not using the NCI treatment, they were using the standard treatment.

We already know that each patient given the standard treatment has a 90% chance of cure. Given that 90% cure rate, what is the chance that 17 out of 17 of the Hypothetical group will be cured?

You may notice that this question about the Hypothetical group is similar to the problem of the 16 ambulances in Chapter 2. In that problem, we were interested to know how likely it was that 3 or more of 16 ambulances would be out of action on any one day, given that each ambulance had a 10% chance of being out of action. Here we would like to know the chances that all 17 patients would be cured, given that each patient has a 90% chance of being cured (and a 10% chance of relapse).

5.3 A physical model of the hypothetical hospital

As in the ambulance example, we could make a physical model of chance in this world. For example, to simulate whether a given patient is cured or not by a 90% effective treatment, we could throw a ten sided die and record the result. We could say, arbitrarily, that a result of 0 means “not cured”, and all the numbers 1 through 9 mean “cured” (typical 10-sided dice have sides numbered 0 through 9).

We could roll 17 dice to simulate one “trial” in this random world. For each trial, we record the number of dice that show numbers 1 through 9 (and not 0). This will be a number between 0 and 17, and it is the number of patients “cured” in our simulated trial.

Figure 5.1 is the result of one such trial we did with a set of 17 10-sided dice we happened to have to hand:

Figure 5.1: One roll of 17 10-sided dice

The trial in Figure 5.1 shows are four dice with the 0 face uppermost, and the rest with numbers from 1 through 9. Therefore, there were 13 out of 17 not-zero numbers, meaning that 13 out of 17 simulated “patients” were “cured” in this simulated trial.

We could repeat this simulated trial procedure 100 times, and we would then have 100 counts of the not-zero numbers. Each of the 100 counts would be the number of patients cured in that trial. We can ask how many of these 100 counts were equal to 17. This will give us an estimate of the probability we would see 17 out of 17 patients cured, given that any one patient has a 90% chance of cure. For example, say we saw 15 out of 100 counts were equal to 17. That would give us an estimate of 15 / 100 or 0.15 or 15%, for the probability we would see 17 out of 17 patients cured.

So, if Saint Hypothetical General did see 17 out of 17 patients cured with the standard treatment, they would be a little surprised, because they would only expect to see that happen 15% of the time. But they would not be very surprised — 15% of the time is uncommon, but not very uncommon.

5.4 A trial, a run, a count and a proportion

Here we stop to emphasize the steps in the process of a random simulation.

1. We decide what we mean by one trial. Here one trial has the same meaning in medicine as resampling — we mean the result of treating 17 patients. One simulated trial is then the simulation of one set of outcomes from 17 patients.
2. Work out the outcome of interest from the trial. The outcome here is the number of patients cured.
3. We work out a way to simulate one trial. Here we chose to throw 17 10-sided dice, and count the number of not zero values. This is the outcome from one simulation trial.
4. We repeat the simulated trial procedure many times, and collect the results from each trial. Say we repeat the trial procedure 100 times; we will call this a run of 100 trials.
5. We count the number of trials with an outcome that matches the outcome we are interested in. In this case we are interested in the outcome 17 out of 17 cured, so we count the number of trials with a score of 17. Say 15 out of the run of 100 trials had an outcome of 17 cured. That is our count.
6. Finally we divide the count by the number of trials to get the proportion. From the example above, we divide 15 by 100 to 0.15 (15%). This is our estimate of the chance of seeing 17 out of 17 patients cured in any one trial. We can also call this an estimate of the probability that 17 out of 17 patients will be cured on any on trial.

Our next step is to work out the code for step 2: simulate one trial.

5.5 Simulate one trial with code

We can use the computer to do something very similar to rolling 17 10-sided dice, by asking the computer for 17 random whole numbers from 0 through 9.

Whole numbers

A whole number is a number that is not negative, and does not have fractional part (does not have anything after a decimal point). 0 and 1 and 2 and 3 are whole numbers, but -1 and \(\frac{3}{5}\) and 11.3 are not. The whole numbers from 0 through 9 are 0, 1, 2, 3, 4, 5, 6, 7, 8, 9.

We have already discussed what we mean by random in Section 2.2.

We will be asking the computer to generate many random numbers. So, before we start, we again import Numpy and get its random number generator:

import numpy as np

# Ask for Numpy's default random number generator and name
# it `rnd`.  `rnd` is short for "random".
rnd = np.random.default_rng()

5.6 From numbers to arrays

We next need to prepare the sequence of numbers that we want NumPy to select from.

We have already seen the idea that Python has values that are individual numbers. Remember, a variable is a named value. Here we attach the name a to the value 1.

a = 1
# Show the value of "a"
a

1

NumPy also allows values that are sequences of numbers. NumPy calls these sequences arrays.

Here we make a array that contains the 10 numbers we will select from:

# Make an array of numbers, store with the name "some_numbers".
some_numbers = np.array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
# Show the value of "some_numbers"
some_numbers

array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])

Notice that the value for some_numbers is an array, and that this value contains 10 numbers.

Put another way, some_numbers is now the name we can use for this collection of 10 values.

Arrays are very useful for simulations and data analysis, and we will be using these for nearly every example in this book.

5.7 Functions

Functions are another tool that we will be using everywhere, and that you seen already, although we have not introduced them until now.

You can think of functions as named production lines.

For example, consider the Python function np.round

# We load the Numpy library so we have access to the Numpy functions.
import numpy as np

np.round is the name for a simple production line, that takes in a number, and (by default) sends back the number rounded to the nearest integer.

What is an integer?

An integer is a positive or negative whole number.

In other words, a number is an integer if the number is either a whole number (0, 1, 2 …), or a negative whole number (-1, -2, -3 …). All of -208, -2, 0, 10, 105 are integers, but \(\frac{3}{5}\), -10.3 and 0.2 are not.

We will use the term integer fairly often, because it is a convenient way to name all the positive and negative whole numbers.

Think of a function as a named production line. We sent the function (production line) raw material (components) to work on. The production line does some work on the components. A finished result comes off the other end.

Therefore, think of np.round as the name of a production line, that takes in a component (in this case, any number), and does some work, and sends back the finished result (in this case, the number rounded to the nearest integer.

The components we send to a function are called arguments. The finished result the function sends back is the return value.

• Arguments: the value or values we send to a function.
• Return value: the values the function sends back.

See Figure 5.2 for an illustration of np.round as a production line.

Figure 5.2: The round function as a production line

In the next few code cells, you see examples where np.round takes in a not-integer number, as an argument, and sends back the nearest integer as the return value:

# Put in 3.2, round sends back 3.
np.round(3.2)

np.float64(3.0)

# Put in -2.7, round sends back -3.
np.round(-2.7)

np.float64(-3.0)

Like many functions, np.round can take more than one argument (component). You can send round the number of digits you want to round to, after the number of you want it to work on, like this (see Figure 5.3):

# Put in 3.1415, and the number of digits to round to (2).
# round sends back 3.14
np.round(3.1415, 2)

np.float64(3.14)

Figure 5.3: round with optional arguments specifying number of digits

Notice that the second argument — here 2 — is optional. We only have to send round one argument: the number we want it to round. But we can optionally send it a second argument — the number of decimal places we want it to round to. If we don’t specify the second argument, then round assumes we want to round to 0 decimal places, and therefore, to the nearest integer.

5.8 Functions and named arguments

In the example above, we sent round two arguments. round knows that we mean the first argument to be the number we want to round, and the second argument is the number of decimal places we want to round to. It knows which is which by the position of the arguments — the first argument is the number it should round, and second is the number of digits.

In fact, internally, the round function also gives these arguments names. It calls the number it should round — a — and the number of digits it should round to — decimals. This is useful, because it is often clearer and simpler to identify the argument we are specifying with its name, instead of just relying on its position.

If we aren’t using the argument names, we call the round function as we did above:

# Put in 3.1415, and the number of digits to round to (2).
# round sends back 3.14
np.round(3.1415, 2)

np.float64(3.14)

In this call, we relied on the fact that we, the people writing the code, and you, the person reading the code, remembers that the second argument (2) means the number of decimal places it should round to. But, we can also specify the argument using its name, like this (see Figure 5.5):

# Put in 3.1415, and the number of digits to round to (2).
# Use the name of the number-of-decimals argument for clarity:
np.round(3.1415, decimals=2)

np.float64(3.14)

Figure 5.4: The round function with argument names

Figure 5.5: The np.round function with argument names

Here Python sees the first argument, as before, and assumes that it is the number we want to round. Then it sees the second, named argument — decimals=2 — and knows, from the name, that we mean this to be the number of decimals to round to.

In fact, we could even specify both arguments by name, like this:

# Put in 3.1415, and the number of digits to round to (2).
np.round(a=3.1415, decimals=2)

np.float64(3.14)

We don’t usually name both arguments for round, as we have above, because it is so obvious that the first argument is the thing we want to round, and so naming the argument does not make it any more clear what the code is doing. But — as so often in programming — whether to use the names, or let Python work out which argument is which by position, is a judgment call. The judgment you are making is about the way to write the code to be most clear for your reader, where your most important reader may be you, coming back to the code in a week or a year.

How do you know what names to use for the function arguments?

You can find the names of the function arguments in the help for the function, either online, or in the notebook interface. For example, to get the help for np.round, including the argument names, you could make a new cell, and type np.round?, then execute the cell by pressing Shift-Enter. This will show the help for the function in the notebook interface.

5.9 Ranges

Now let us return to the variable some_numbers that we created above:

# Make an array of numbers, store with the name "some_numbers".
some_numbers = np.array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
# Show the value of "some_numbers"
some_numbers

array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])

In fact, we often need to do this: generate a sequence or range of integers, such as 0 through 9.

Pick a number from 1 through 5

Ranges can be confusing in normal speech because it is not always clear whether they include their beginning and end. For example, if someone says “pick a number between 1 and 5”, do they mean to pick from all of the numbers, including the first and last (any of 1 or 2 or 3 or 4 or 5)? Or do they mean only the numbers that are between 1 and 5 (so 2 or 3 or 4)? Or do they mean all the numbers up to, but not including 5 (so 1 or 2 or 3 or 4)?

To avoid this confusion, we will nearly always use “from” and “through” in ranges, meaning that we do include both the start and the end number. For example, if we say “pick a number from 1 through 5” we mean one of 1 or 2 or 3 or 4 or 5.

Creating ranges of numbers is so common that Python has a standard Numpy function np.arange to do that.

# An array containing all the numbers from 0 through 9.
some_numbers = np.arange(0, 10)
some_numbers

array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])

Notice that we send np.arange the arguments 0 and 10. The first argument, here 0, is the start value. The second argument, here 10, is the stop value. Numpy (in the arange function) understands this to mean: start at 0 (the start value) and go up to but do not include 10 (the stop value).

You can therefore read np.arange(0, 10) as “the sequence of integers starting at 0, up to, but not including 10”.

Like np.round, the arguments to np.arange also have names, so, we could also write:

# An array containing all the numbers from 0 through 9.
# Now using named arguments.
some_numbers = np.arange(start=0, stop=10)
some_numbers

array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])

So far, we have sent arange two arguments, but we can also send just one argument, like this:

# An array containing all the numbers from 0 through 9.
some_integers = np.arange(10)
some_integers

array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])

When we sent arange a single argument, like this, arange understands this to mean we have sent just the stop value, and that is should assume a start value of 0.

Again, if we wanted, we could send this argument by name:

# An array containing all the numbers from 0 through 9.
# Specify the stop value by explicit name, for clarity.
some_integers = np.arange(stop=10)
some_integers

array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])

Here are some more examples of np.arange:

# All the integers starting at 10, up to, but not including 15.
# In other words, 10 through 14.
np.arange(10, 15)

array([10, 11, 12, 13, 14])

# Here we are only sending one value (7). np.arange understands this to be
# the stop value, and assumes 0 as the start value.
# In other words, 0 through 6
np.arange(7)

array([0, 1, 2, 3, 4, 5, 6])

5.10 range in Python

So far you have seen ranges of integers using np.arange. The np. prefix refers to the fact that np.arange is a function from the Numpy module (library). The a in arange signals that the result np.arange returns is an array:

arr = np.arange(7)
# Show the result
arr

array([0, 1, 2, 3, 4, 5, 6])

# Show what type of thing this is.
type(arr)

<class 'numpy.ndarray'>

We do often use np.arange to get a range of integers in a convenient array format, but Python has another way of getting a range of integers — the range function.

The range function is very similar to np.arange, but it is not part of Numpy — it is basic function in Python — and it does not return an array of numbers, it returns something else. Here we ask for a range from 0 through 6 (0 up to, but not including 7):

# Notice no `np.` before `range`.
r = range(7)
r

range(0, 7)

Notice that the thing that came back is something that represents or stands in for the number 0 through 6. It is not an array, but a specific type of thing called — a range:

type(r)

<class 'range'>

The range above is a container for the numbers 0 through 6. We can get the numbers out of the container in many different ways, but one of them is to convert this container to an array, using the np.array function. The np.array function takes the thing we pass it, and makes it into an array. When we apply np.array to r above, we get the numbers that r contains:

# Get the numbers from the range `r`, convert to an array.
a_from_r = np.array(r)
# Show the result
a_from_r

array([0, 1, 2, 3, 4, 5, 6])

The range function has the same start and stop arguments that np.arange does, and with the same meaning:

# 3 up to, not including 12.
# (3 through 11)
r_2 = range(3, 12)
r_2

range(3, 12)

np.array(r_2)

array([ 3,  4,  5,  6,  7,  8,  9, 10, 11])

You may reasonably ask — why do I need this range thing, if I have the very similar np.arange? The answer is — you don’t need range, and you can always use np.arange where you would use range, but for reasons we will go into later (Section 6.6.3), range is a good option when we want to represent a sequence of numbers as input to a for loop. We cover for loops in more detail in Section 6.6.2, but for now, the only thing to remember is that range and np.arange are both ways of expressing sequential ranges of integers.

5.11 Choosing values at random

We can use the rnd.choice function to select a single value at random from the sequence of numbers in some_integers.

More on rnd.choice

The rnd.choice function will be a fundamental tool for taking many kinds of samples, and we cover it in more detail in Chapter 7.

# Select an integer from the choices in some_integers.
my_integer = rnd.choice(some_integers)
# Show the value that results.
my_integer

np.int64(5)

Like np.round (above), rnd.choice is a function.

Note 5.1: Functions and methods

Actually, to be precise, we should call rnd.choice a method. A method is a function attached to a value. In this case the function choice is attached to the value rnd. That’s not an important distinction for us at the moment, so please forgive our strategic imprecision, and let us continue to say that rnd.choice is a function.

As you remember, a function is a named production line. In our case, the production line has the name rnd.choice.

We sent rnd.choice. a value to work on — an argument. In this case, the argument was the value of some_integers.

Figure 5.6 is a diagram illustrating an example run of the rnd.choice function (production line).

Figure 5.6: Example run of the rnd.choice function

Here is the same code again, with new comments.

# Send the value of "some_integers" to rnd.choice
# some_integers is the *argument*.
# Put the *return* value from the function into "my_number".
my_number = rnd.choice(some_integers)
# Show the value that results.
my_number

np.int64(4)

5.12 Creating arrays with sampling

In the code above, we asked Python to select a single number at random — because that is what rnd.choice does by default.

In fact, the people who wrote rnd.choice, wrote it to be flexible in the work that it can do. In particular, we can tell rnd.choice to select any number of values at random, by adding a new argument to the function.

In our case, we would like Numpy to select 17 numbers at random from the sequence of some_integers.

To do this, we add an argument to the function that tells it how many numbers we want it to select.

# Get 17 values from the *some_integers* array.
# Store the 17 numbers with the name "a"
a = rnd.choice(some_integers, 17)
# Show the result.
a

array([4, 5, 9, 8, 2, 9, 1, 5, 8, 2, 1, 8, 2, 6, 6, 5, 0])

As you can see, the function sent back (returned) 17 numbers. Because it is sending back more than one number, the thing it sends back is an array, where the array has 17 elements.

5.12.1 sum — adding all the values

Bear with us for a short diversion. You will see why we made this diversion soon.

NumPy has a function np.sum that will add up all the numbers in an array.

You can see the contents of a above.

np.sum adds all the numbers in the array together, to give the sum of the array. The sum is just the result of adding all the values in the array. Put another way, it is the result of adding the second element to the first, then adding third element to the result, and the fourth element to the result, and so on.

np.sum(a)

np.int64(81)

5.13 Counting results

We now have the code to do the equivalent of throwing 17 ten-sided dice. This is the basis for one simulated trial in the world of Saint Hypothetical General.

Our next job is to get the code to count the number of numbers that are not zero in the array a. That will give us the number of patients who were cured in simulated trial.

Another way of asking this question, is to ask how many elements in a are greater than zero.

5.13.1 Comparison

To ask whether a number is greater than zero, we use comparison. Here is a greater than zero comparison on a single number:

n = 5
# Is the value of n greater than 0?
# Show the result of the comparison.
n > 0

True

> is a comparison — it asks a question about the numbers either side of it. In this case > is asking the question “is the value of n (on the left hand side) greater than 0 (on the right hand side)?” The value of n is 5, so the question becomes, “is 5 greater than 0?” The answer is Yes, and Python represents this Yes answer as the value True.

In contrast, the comparison below boils down to “is 0 greater than 0?”, to which the answer is No, and Python represents this as False.

p = 0
# Is the value of p greater than 0?
# Show the result of the comparison.
p > 0

False

So far you have seen the results of comparison on a single number. Now say we do the same comparison on an array. For example, say we ask the question “is the value of a greater than 0”? Remember, a is an array containing 17 values. We are comparing 17 values to one value (0). What answer do you think NumPy will give? You may want to think a little about this before you read on.

As a reminder, here is the current value for a:

# Show the current value for "a"
a

array([4, 5, 9, 8, 2, 9, 1, 5, 8, 2, 1, 8, 2, 6, 6, 5, 0])

Now you have had some time to think, here is what happens:

# Is the value of "a" greater than 0
# Show the result of the comparison.
a > 0

array([ True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True, False])

There are 17 values in a, so the comparison to 0 means there are 17 comparisons, and 17 answers. NumPy therefore returns an array of 17 elements, containing these 17 answers. The first answer is the answer to the question “is the value of the first element of a greater than 0”, and the second is the answer to “is the value of the second element of a greater than 0”.

Let us store the result of this comparison to work on:

# Is the value of "a" greater than 0
# Store as another array "q".
q = a > 0
# Show the value of r
q

array([ True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True, False])

5.14 Counting True values with sum

Notice above that there is one True element in q for every element in a that was greater than 0. It only remains to count the number of True values in q, to get the count of patients in our simulated trial who were cured.

We can use the NumPy function np.sum to count the number of True elements in an array. As you can see above, np.sum adds up all the elements in an array, to give a single number. This will work as we want for the q array, because Python counts False as equal to 0 and True as equal to 1:

# Question: is False equal to 0?
# Answer - Yes! (True)
False == 0

True

# Question: is True equal to 1?
# Answer - Yes! (True)
True == 1

True

Therefore, the function sum, when applied to an array of True and False values, will count the number of True values in the array.

To see this in action we can make a new array of True and False values, and try using np.sum on the new array.

# An array containing three True values and two False values.
trues_and_falses = np.array([True, False, True, True, False])
# Show the new array.
trues_and_falses

array([ True, False,  True,  True, False])

The sum operation adds all the elements in the array. Because True counts as 1, and False counts as 0, adding all the elements in trues_and_falses is the same as adding up the values 1 + 0 + 1 + 1 + 0, to give 3.

We can apply the same operation on q to count the number of True values.

# Count the number of True values in "q"
# This is the same as the number of values in "a" that are greater than 0.
b = np.sum(q)
# Show the result
b

np.int64(16)

5.15 The procedure for one simulated trial

We now have the whole procedure for one simulated trial. We can put the whole procedure in one cell:

# Procedure for one simulated trial

# Get 17 values from the *some_integers* array.
# Store the 17 numbers with the name "a"
a = rnd.choice(some_integers, 17)
# Is the value of "a" greater than 0
q = a > 0
# Count the number of True values in "q"
b = np.sum(q)
# Show the result of this simulated trial.
b

np.int64(17)

5.16 Repeating the trial

Now we know how to do one simulated trial, we could just keep running the cell above, and writing down the result each time. Once we had run the cell 100 times, we would have 100 counts. Then we could look at the 100 counts to see how many were equal to 17 (all 17 simulated patients cured on that trial). At least that would be much faster than rolling 17 dice 100 times, but we would also like the computer to automate the process of repeating the trial, and keeping track of the counts.

Please forgive us as we race ahead again, as we did in the last chapter. As in the last chapter, we will use a results array called z to store the count for each trial. As in the last chapter, we will use a for loop to repeat the trial procedure many times. As in the last chapter, we will not explain the counts array of the for loop in any detail, because we are going to cover those in the next chapter.

Let us now imagine that we want to do 100 simulated trials at Saint Hypothetical General. This will give us 100 counts. We will want to store the count for each trial.

To do this, we make an array called z to hold the 100 counts. We have called the array z, but we could have called it anything we liked, such as counts or results or cecilia.

# An array to hold the 100 count values.
# Later, we will fill this in with real count values from simulated trials.
z = np.zeros(100)

Next we use a for loop to repeat the single trial procedure.

Notice that the single trial procedure, inside this for loop, is the same as the single trial procedure above — the only two differences are:

• The trial procedure is inside the loop, and
• We are storing the count for each trial as we go.

We will go into more detail on how this works in the next chapter.

# Procedure for 100 simulated trials.

# An array to store the counts for each trial.
z = np.zeros(100)

# Repeat the trial procedure 100 times.
for i in np.arange(100):
    # Get 17 values from the *some_integers* array.
    # Store the 17 numbers with the name "a".
    a = rnd.choice(some_integers, 17)
    # Is the value of "a" greater than 0.
    q = a > 0
    # Count the number of True values in "q".
    b = np.sum(q)
    # Store the result at the next position in the "z" array.
    z[i] = b
    # Now go back and do the next trial until finished.
# Show the result of all 100 trials.
z

array([16., 15., 15., 16., 16., 12., 15., 11., 16., 13., 12., 16., 15.,
       16., 15., 16., 14., 15., 14., 15., 15., 15., 14., 15., 17., 15.,
       14., 15., 16., 17., 15., 17., 16., 17., 14., 16., 15., 15., 15.,
       17., 17., 13., 16., 13., 16., 14., 14., 15., 15., 15., 14., 15.,
       15., 15., 17., 16., 17., 14., 15., 14., 16., 16., 15., 15., 16.,
       15., 15., 16., 17., 15., 17., 15., 10., 15., 15., 14., 14., 13.,
       16., 14., 17., 17., 16., 14., 15., 16., 17., 14., 15., 15., 16.,
       16., 17., 16., 13., 15., 15., 14., 17., 15.])

Finally, we need to count how many of the trials results we stored in z gave a “cured” count of 17.

We can ask the question whether a single number is equal to 17 using the double equals comparison: ==.

s = 17
# Is the value of s equal to 17?
# Show the result of the comparison.
s == 17

True

Note

5.17 Single and double equals

Notice that the double equals == means something entirely different to Python than the single equals =. In the code above, Python reads s = 17 to mean “Set the variable s to have the value 17”. In technical terms the single equals is called an assignment operator, because it means assign the value 17 to the variable s.

The code s == 17 has a completely different meaning.

It means “give True if the value in s is equal to 17, and False otherwise”. The == is a comparison operator — it is for comparing two values — here the value in s and the value 17. This comparison, like all comparisons, returns an answer that is either True or False. In our case s has the value 17, so the comparison becomes 17 == 17, meaning “is 17 equal to 17?”, to which the answer is “Yes”, and Python sends back True.

We can ask this question of all 100 counts by asking the question: is the array z equal to 17, like this:

# Is the value of z equal to 17?
were_cured = z == 17
# Show the result of the comparison.
were_cured

array([False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False,  True, False, False,
       False, False,  True, False,  True, False,  True, False, False,
       False, False, False,  True,  True, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
        True, False,  True, False, False, False, False, False, False,
       False, False, False, False, False,  True, False,  True, False,
       False, False, False, False, False, False, False, False,  True,
        True, False, False, False, False,  True, False, False, False,
       False, False,  True, False, False, False, False, False,  True,
       False])

Finally we use sum to count the number of True values in the were_cured array, to give the number of trials where all 17 patients were cured.

# Count the number of True values in "were_cured"
# This is the same as the number of values in "z" that are equal to 17.
n_all_cured = np.sum(were_cured)
# Show the result of the comparison.
n_all_cured

np.int64(15)

n_all_cured is the number of simulated trials for which all patients were cured. It only remains to get the proportion of trials for which this was true, and to do this, we divide by the number of trials.

# Proportion of trials where all patients were cured.
p = n_all_cured / 100
# Show the result
p

np.float64(0.15)

From this experiment, we see that there is roughly a one-in-six chance that all 17 patients are cured when using a 90% effective treatment.

5.18 What have we learned from Saint Hypothetical?

We started with a question about the results of the NCI trial on the new drug. The question was — was the result of their trial — 17 out of 17 patients cured — surprising.

Then, for reasons we did not explain in detail, we changed tack, and asked the same question about a hypothetical set of 17 patients getting the standard treatment in Saint Hypothetical General.

That Hypothetical question turns out to be fairly easy to answer, because we can use simulation to estimate the chances that 17 out of 17 patients would be cured in such a hypothetical trial, on the assumption that each patient has a 90% chance of being cured with the standard treatment.

The answer for Saint Hypothetical General was — we would be somewhat surprised, but not astonished. We only get 17 out of 17 patients cured about one time in six.

Now let us return to the NCI trial. Should the trial authors be surprised by their results? If they assumed that their new treatment was exactly as effective as the standard treatment, the result of the trial is a bit unusual, just by chance. It is up to us to decide whether the result is unusual enough to make us think that the actual NCI treatment might in fact have been more effective than the standard treatment.

You will see this move again and again as we go through the book.

• We take something that really happened — in this case the 17 out of 17 patients cured.
• Then we imagine a hypothetical world in which the results only depend on chance.
• We do simulations in that hypothetical world to see how often we get a result like the one that happened in the real world.
• If the real world result (17 out of 17) is an unusual, surprising result in the simulations from the hypothetical world, we take that as evidence that the real world result might not be due to chance alone.

We have just described the main idea in statistical inference. If that all seems strange and backwards to you, do not worry, we will go over that idea many times in this book. It is not a simple idea to grasp in one go. We hope you will find that, as you do more simulations, and think of more hypothetical worlds, the idea will start to make more sense. Later, we will start to think about asking other questions about probability and chance in the real world.

5.19 Conclusions

Can you see how each of the operations that the computer carries out are analogous to the operations that you yourself executed when you solved this problem using 10-sided dice? This is exactly the procedure that we will use to solve every problem in probability and statistics that we must deal with. Either we will use a device such as coins or dice, or a random number table as an analogy for the physical process we are interested in (patients being cured, in this case), or we will simulate the analogy on the computer using the Python program above.

The program above may not seem simple at first glance, but we think you will find, over the course of this book, that these programs become much simpler to understand than the older conventional approach to such problems that has routinely been taught to students for decades.

Dunleavy, Kieron, Stefania Pittaluga, John Janik, Nicole Grant, Margaret Shovlin, Richard Little, Robert Yarchoan, Seth Steinberg, Elaine S. Jaffe, and Wyndham H. Wilson. 2006. “Novel Treatment of Burkitt Lymphoma with Dose-Adjusted EPOCH-Rituximab: Preliminary Results Showing Excellent Outcome.” Blood 108 (11): 2736–36. https://doi.org/10.1182/blood.V108.11.2736.2736.