2 What is data?

Last modified on 31. March 2026 at 10:21:06

“The limits of my language mean the limits of my world.” — Ludwig Wittgenstein

Emma sat on the wooden stage. She need time to recover from what she had seen behind the curtains. What she had seen beneath the wood had now become reality. Dust danced in the light. Single light beams hit some dust particles from time to time and exploded into golden rain. Sparkles dropped on the wood.

“How is gold made?” Emma asked.

“This was a long break. I’m a bit surprised by your question. I thought we should discuss your findings from behind the stage. Not gold.” Jeff answers from his seat on the left.

“Gold cannot be formed through fusion in a star. This is only possible up to iron. You need a supernova or a collision of two neutron stars.” Emma spoke absently into the darkness.

“Interesting topic. Was this observed? The collision of two neutron stars and the resulting gold spewing out,” Jeff laughed.

“No, it was a computer simulation. Artificial data, if you will.” Emma looked around. She decided that it was time to begin the play.

“Then it’s not real, and it’s not an explanation for me at all.” Jeff stated.

“But the simulation was based on real observations.” Emma stood up and started building the scene for the play.

2.1 What is observed?

Figure 2.1: My son and I are observing the people riding bicycles on an early spring day. Given their clothing and headwear, my son tried to guess what season it really was. Some cyclists seem to be dressed for spring, others for winter, and one young woman for summer. He was very confused because it was really hard to guess. Why was there such a wide variation?

Can we observe anything? What does it mean to observe? We see things, but our vision is already influenced by our understanding of what they are. Additionally, by naming things and processes, we are already following a theory. “The sun is rising” is a popular example of a saying connected with the geocentric theory, which states that the Earth is at the center of the solar system. Therefore, our language is already influenced by our observations. We cannot speak about something we observe without giving the observation some meaning. It is possible, but it demands a high level of concentration.

Figure 2.2: An observation is a measurement or sensory input within a theoretical and linguistic framework. An observation cannot be made without a context. There is no such thing as an objective observation.

This book requires a highly specific type of observation. We need data. Very rarely is any observation we make based on first-hand data. For example, if you look outside your window and observe a bird, you will learn some information about it. But is this data? No, it is just information. Data is information, but structured information. Data is a type of observation that is stored in a specific way. We translate observations and information about them into a set of numbers. Not every observation can be represented as a number in a data sheet or table. As their names suggest, data sheets and tables are two-dimensional. A sheet is a type of paper on which organised observations can be written down. The same is true for a table. It is a structured type of observation.

2.2 What is data?

We will now think about data. This will happen after we have discussed the science in the chapter. It will also happen if you have skipped the chapter entirely. What is data? If you had asked me when I was a teenager, I would have referred to Star Trek: The Next Generation and Commander Data. He was an android without any emotions. Pure objectivity. He was an artificial being. However, this is not the type of data we want to discuss. We want to discuss data that has been observed and measured. Furthermore, our data is numeric or has a numeric counterpart. We use descriptive words, but rarely. Later on, we want to run algorithms and calculations with our data. Therefore, the data must be open to calculations.

Data are observations represented by numbers and letters. Can we use data in any format? No, you cannot. There is a general format for how data should be stored. In our case, we have observations in the rows and variables of measurement in the columns. There can only be one observation per row. This format is called the long format. Another possibility is the wide format, where the observations are scattered around and do not share the same rows. The wide format may be useful for time series, but I do not recommend it.

The following Figure 2.3 shows a typical data set used in data science and statistics. It is also typical for those working in the life sciences. The matrix has $n$ rows and $p$ columns. To select a cell, we can write the row and column information in brackets. The rows are indicated by $i$ and the columns by $j$. This is a simple way of extracting information from a data table. However, these manipulations are not normally used anymore because they are very error-prone. This is a simple yet powerful way of storing information. We translate our observations into numbers and letters, which we then put into a two-dimensional matrix.

Figure 2.3: Theoretical data set. It has $n$ rows and $p$ columns. The row and column positions can be used to access each cell of this two-dimensional matrix, which is indicated by the grey shaded area.

Figure 2.4 might be interesting, but it is not very helpful in the daily working process. Therefore, I will show you an annotated data table with example observations labelled with numbers and letters. The first columns contain an identifier for each observation, or ID. I always include one in my data sets because I always want to know which individual is which. This is even more important when working with time series or repeated measurements. After the ID column, we find the experimental variables. We can also refer to these as ‘X’ variables, as we will plot them on the x-axis in our visualizations later on. We will often set the experimental variables before we start the experiment. Then, we find the measurements on the individuals. These variables are called outcomes or ‘Y’ variables, as they will be the variables on our y-axis later on. The terminology used varies from scientific field to scientific field. I will therefore talk about terminology in a separate chapter later on.

Figure 2.4: Annotated dataset. The individuals’ observations are shown in the rows. The columns show the single identifier of the observations, the experimental factors set before the experiment, and the measured variables after the experiment has concluded.

Can we live with this two-dimensional representation of data? Not really. If you look at different fields, you will find a variety of other data tables. In genetics, we do not have data sets of this kind when we look at thousands of genes. Therefore, we have at least two data files: One contains all the information on the patients or individuals, and the other contains all the genetic information. Most of the time, the genetic file is saved in a transposed form. This means that the variables or genes are in the rows and the individuals are in the columns. This is because it is easier to change rows in a data set than columns. This gives us an advantage when it comes to saving and manipulating the data. However, this makes it slightly more complicated to work with such data. Further, geology has its own data format, while other fields use different ways of storing data. Storage methods sometimes depend on the analysis software used or the storage and calculation capacities required.

Here, we will focus on the presented data set in Figure 2.4, making some modifications to the columns if we are working with time series or repeated measurements. However, we will stick to one data set and will not consider complicated representations. We will talk about the experimental variables and the different types of measurements, both of which are represented by different numbers and letters. There are still many topics to consider if we want to work with data to obtain information and knowledge. In the following sections, we will cover different attributes and limitations of data that are important for successful work.

2.3 Meta information of data

The data seems easy enough. We have a two-dimensional table with numbers and letters. That could be it. However, there are some things to consider. Information is often hidden in data. Where does the data come from? What are the internal generation processes? Are there unknown or known structures beneath the numbers and letters that cannot be seen by the row and column names?

First, we need to consider the source of the data. Depending on where you obtain your data, there may be more regularities to consider than in other fields.

Second, we must discuss how the data is generated. Are you conducting a controlled experiment where you only change one influential variable and observe what happens? Or are you doing nothing and only observing the data? Does your role in the generation process play an important role?

Third, we would like to discuss the replication. We need replications in our data in order to perform statistical analyses later on. However, there are two types of replication. They can appear similar in a dataset, but they have a significant impact on your analysis and the answers to your questions.

Is the data valid? This may seem like a simple question, but it is sometimes very important. If you are a statistician working with data, your possibilities are limited. Often, you are not an expert in the field in which you are conducting the analysis. You could argue that men should, on average, be larger than women. However, if the data is more specific, you need help to determine its validity.

Next, we will examine artificial or simulated data. This special type of data is used mainly in statistical research and teaching, and we will use it in this book. Non-statisticians should not use artificial data in their daily research to cheat their data sheets.

Is all this information lost or hidden? In a sense, yes, because if you don’t have any information on the generation process, or if you can only see some cryptic column names, it might be impossible for you to gain any deeper knowledge of the data. However, it is always possible to find a pattern in anything.

Garbage in garbage out

2.3.1 Source of data

Where does your data come from? Does it matter? Yes, because it has a huge influence on your work and how you share it. First, let’s define the different sources of data. One type is human data, such as blood samples or questionnaires. Humans can share information about social, economic, and other factors. Human data is often used in medicine and the social sciences. Then, we can examine data from animals. Mice are a special case because much basic research is conducted on mice before moving on to humans. The rest of the animal studies include all animals relevant to agriculture and leisure activities, such as dogs and horses. Naturally, there is a great deal of interest in chickens, pigs, and cows because of their role in food production. Finally, we have plants, including fungi and other plant-based sources. Our focus is mostly on crops that are interesting to agriculture, but the field is broad. On the sidelines, we have cell cultures, which have their own problems and limitations.

Figure 2.5 shows a visualization of the different sources of data and their connection to regulation and openness. We often refer to data from humans, animals and plants as in vivo data because it originates from living beings. This includes mice, which are the general animal model for humans. If artificial or simulated data is used in statistics, these data sources are referred to as in silico data. In my view and in the context of this book, artificial data is common, but this is a biased perspective on data. Most of the time, data is generated by experiments or observations rather than by algorithms. Nevertheless, human data is largely regulated. Human data cannot be shared openly. The same is partly true for animal models. Plant data and some types of cell cultures are easier to share. If you simulate your data, you can share it as freely as you like.

Figure 2.5: Figure showing examples of sources of data. On the left are the three main *in vivo* sources of data: human, animal and plant, in order of decreasing regulatory requirements. Cell cultures can be conducted from each of these sources. On the left is the *in silico* source of data, which uses artificial or simulated data. The openness and shareability of data moves in the opposite direction to the level of regulation.

Finally, we must consider the ethical and cultural issues involved. You cannot do whatever you want with patients and animals. Depending on your location and nationality, the process of working with or sharing human or animal data is highly regulated. You may need an ethical vote from an Ethics Committee at your institution or approval for an animal experiment from a different committee. Sometimes cultural borders hinder work with specific animal breeds. All of this is beyond the scope of this book. If you are unsure whether your experiment can be conducted and data collected, please contact a statistician at your institution or a nearby university. As a general rule, never upload any human patient data, with or without personal identifiers, to an open database without consulting the relevant legal authorities first.

2.3.2 Experimental vs. observed data

There is a lot that could be written about experimental and observed data. The terminology can be a little confusing because ‘experimental data’ consists of observations that have been clearly recorded. Therefore, I prefer to refer to it as data that comes from a controlled experiment. Conversely, observed data is data generated without any influence from an experimenter. In this book, we will focus primarily on experimental data. However, many of the algorithms introduced can also be used with observed data. There are no real limitations to their application. The problem lies in interpretation. With experimental data, you know which influencers are changing because you are the one changing them. With observed data, however, you can never be sure whether you are observing correlation or causality because you are not controlling any of the influencing factors. If you are missing information, you may draw the wrong conclusions.

In a clinical context, we often refer to observed data as real-world data, which can come from anywhere. This can be data collected from wearable devices such as clocks and smartphones. It can also be collected from operations or other examinations of patients during normal clinical processes. This data is merely observed; there is no controlled assignment of patients to treatments or other controlled activities. In this context, we can also speak of experimental data from randomised clinical trials (RCTs) and observational data as real-world data from clinical routine.

We live in a data-driven world. Or, we could say that we live in an information-driven world. There is so much information available that we believe everything is clear and well known. Sometimes, however, we realise that this is not the case. We only have partial information in very specific areas where we gather it. Someone might think that weather forecasting should be easy. We have satellites, we know past temperatures, wind speeds and air pressure. Yet we still have problems predicting the weather a week in advance or thunderstorms a few hours in advance. We know that the conditions are right for a thunderstorm, but we only find out where it will build up once we observe it.

2.3.3 Biological vs. technical replication

When I first started studying statistics, I found the replications a little irritating. I was familiar with numbers and calculations from school mathematics lessons. There we had a formula or equation and I put in numbers to calculate something. I never did this more than once. Why should I calculate the same thing again and again? Why is there replication in statistical data? We need replication, because we need a probabilistic process to do statistics in science. By the probabilistic process we can then calculate probabilities. Without a probabilistic process and replication we cannot have any probabilities. Perhaps the explanation comes a little early, but we will revisit this topic throughout the book. It is important to remember here that we want to calculate probabilities. Therefore, we need replications in order to calculate these probabilities.

The following Figure 2.6 illustrates the various types of replication. In general, we divide replication into biological and technical replication. We are also able to conduct biological replication at many points in time. This may sound confusing, but the difference lies in the time between the measurements. Let us start with biological replication. We measure an outcome Y on different biological individuals. Depending on our research question, we might measure only a few individuals or hundreds to thousands. These individuals can be humans, plants or mice. Technical replication involves taking measurements many times on the same individual at the same time. This allows us to ensure that the measurement was correct. For example, you could take your body weight three times and then combine the three measurements to give one overall result. If a large amount of time passes between the individual measurements, we speak of a repeated measurement. In this case, we are still measuring the same individual, but at different points in time. We want to know how body weight changes in humans over weeks. This is a biological replication at different points in time.

Figure 2.6: Different types of replication can be observed in data tables and experiments. **(A)** First, there is biological replication, where a measured $Y$ is repeatedly observed in different biological individuals. **(B)** The second type is repeated measurement, where a measured Y is observed on each individual at different points in time. The time between each measurement is long. **(C)** Third, technical replication involves measuring individuals in a row at one point in time. The time between each observation is negligible. The values measured for each individual are mostly combined into a single measure.

Where is the replication in the technical replication?

A word of caution: Should be all the values the same?

Should we not measure always the same value?

2.3.4 Internal and external validity

Can we trust the data we are working with? Is the data valid, and what can we do with it? This also depends on the information buried in the data. If we can find information in our data, we can make decisions or predictions. However, these decisions and predictions will only be useful for future eventualities if the data from the past is connected to the future.

Internal validity is often considered a prerequisite for external validity, as a study must first be true for its participants before it can be generalized to the broader population.

Examples of failures in internal validity

Moving plants

foo

Mice with big ears

bar

Electrified children

hust

https://pmc.ncbi.nlm.nih.gov/articles/PMC6188693/

https://en.wikipedia.org/wiki/External_validity

https://www.researchgate.net/profile/Jolaine-Draugalis/publication/260267611_Establishing_the_internal_and_external_validity_of_experimental_studies/links/56c2042208ae44da37ff4feb/Establishing-the-internal-and-external-validity-of-experimental-studies.pdf

https://journals.sagepub.com/doi/pdf/10.1177/17456916221136117

https://www.verywellmind.com/internal-and-external-validity-4584479#:~:text=External%20validity%20relates%20to%20how%20applicable%20the,%20Maturation%20%20Statistical%20regression%20*%20Testing

https://dovetail.com/research/internal-validity/

Threats to internal data validity

History
Social
Blinding
Positive and negative control
Maturation and learning
Changing in instruments

From there, we have the famous saying ‘garbage in, garbage out’, which is illustrated in Figure 2.8. If we input garbage data into a statistical algorithm, we will find a pattern. The pattern will usually be of poor quality too. Quite often, we do this in a setting where we don’t even understand the algorithm. Therefore, we put something into an algorithm and hope that something useful will come out. If we get something, we are very happy and continue changing nothing in case everything is lost. Welcome to one possible cause of the replication crisis.

Figure 2.8: Visualisation of the saying ‘garbage in, garbage out’. In other words, if the input data is bad, the statistical algorithm cannot work miracles. The output will also be garbage, regardless of the statistical algorithm. In such a setting, the algorithm is also seen as a black box because, if the data is not understood, neither will the algorithm be.

Guidance on Validity of Clinical Studies¹ and²

2.3.5 Simulation and artificial data

You might be surprised to learn that, in statistical research, we statisticians work with simulated data. This data is also known as artificial data. This data has no experimental background and is entirely simulated. We simulate our measured values based on the experimental values. Therefore, we need to make some assumptions about the simulation process. Consequently, we can conclude that our artificial data is model-laden. We need a model to connect the experimental and measured variables. We can call this connection a model. If we change the model, the generated data will also change.

In this book, we will simulate our data. This may seem counterintuitive, as we will use data with predefined effects and differences. What is the point of such data? What can we learn from it? The advantage of simulated data is that we already know what we want to find with our statistical algorithms. The advantage of using simulated data is that we already know what we are looking for with our statistical algorithms. Otherwise, we would not know whether there were no patterns in the data or whether our algorithm was unable to detect them.

In vivo versus in silico.

2.4 Ways to work with data

2.4.1 Long format vs. wide format

There are two types of arranging your data in a 2D data table. If we put each observation in one line, we speak of

A question of time

Entity–attribute–value model

Wide, or unstacked data is presented with each different data variable in a separate column.

Narrow, stacked, or long data is presented with one column containing all the values and another column listing the context of the value

Figure 2.9: Two types of data storage exist in the wide and long format. It is important to remember that we need the long format for visualization and analysis in R. **(A)** The wide format is also called unstacked data. We use this type of data if we are measuring time series or changing data from the same subject. This format is very efficient for storage but difficult to analyze. **(B)** The long format is also called stacked or narrow data. The long format is the standard for data analysis. The measured variable Y is contained in only one column.

Two types of data storage in the wide and long format. It is important to remeber, that we need the long format for the visualitation and analysis in R. The wide format is also called unstacked data. We use this type of data, if we measure time series data or data which is changing in the same subject. In this applications, the format is very effiecent to store but difficult to analyse. The long format is also called stacked or narrow data. The long format is the standard format for the analysis of data. The measured variable Y is only contained in one column.

2.4.2 The elephant in the room is Excel

⁴

⁵

What problems are there with Excel?

Reach and shareability. If we do something in Excel by point and click, nobody else can do it if they are not observing us.
Scalability and efficiency. If we have one task, it will take some time. If we have the same task with different numbers, it will take twice as long. We cannot really automate any processes.
Excel is error-prone. If we make a mistake, we will not really understand why things did not work. We can click again, but we have no protocol to follow. Others cannot understand what we have done.
Excel is neither open nor free to use. We need a licence to use Excel and are unable to share any files with people who do not have access to Microsoft products.

2.4.3 Tidy data

2.5 R packages used

Do we want here to simulate data?

2.6 Data

Is your data the fuel of discovery or the test of discovery?

R Code [show / hide]

jump_weight_tbl <- tibble(x = c(0.6, 1, 2.3, 3.5, 5.2, 7.1, 8.4, 9.2, 10),
                          y = 0.15*x^3 - 2.2*x^2 + 8.8*x + 3.2 + rnorm(9, 0, 0.5)) |>
  mutate_all(round, 1) |> 
  rename(weight_mg = x, jump_length_cm = y)

Table 2.1: foo.

weight_mg	jump_length_cm
0.6	8.5
1.0	9.6
2.3	13.7
3.5	13.2
5.2	11.0
7.1	8.3
8.4	10.9
9.2	14.3
10.0	21.1

Equation 2.1

\[ y = 0.15\cdot x^3 - 2.2\cdot x^2 + 8.8 \cdot x + 3.2 \tag{2.1}\]

2.7 Outro

What do we want to do in this book? We want to perform inductive reasoning based on data. Statistical modeling is inductive reasoning, so it is also based on data. Therefore, we will need data to perform statistical analyses in any possible combination. In other words, we live in a data-driven world.

In this book, we will follow an inductive approach. First, we make observations, then we try to find patterns in them. Therefore, our data is the fuel of discovery. Conversely, we can also use data to test a hypothesis. Both are valid and are used. Since this book is about statistics and data science, we cannot generate new theories. We are limited by the tools we have and the data available to us. This book may be read by those with a background in natural science. Then, you can use the statistical tools differently.

2.8 Alternatives

Further tutorials and R packages on XXX

2.9 Glossary

term: what does it mean.

2.10 The meaning of “Models of Reality” in this chapter.

itemize with max. 5-6 words

2.11 Summary

References

[1]

European Commission. Guidance on the Validity of Clinical Studies for Joint Clinical Assessment. Directorate-General for Health; Food Safety; 2024.

[2]

Toumi M, Falissard B, Jouini A, Aballéa S, Boyer L. Clinical trial validity guidance from the HTACG: Looking for chicken teeth. Journal of Market Access & Health Policy. 2025;13(2):15-15.

[3]

Dormann CF, Ellison AM. Statistics by Simulation: A Synthetic Data Approach. Princeton University Press; 2025.

[4]

Hassenstein MJ, Jung K. Ten simple rules for effective research data management. PLOS Computational Biology. 2025;21(12):e1013779.

[5]

Wilkinson MD, Dumontier M, Aalbersberg IjJ, Appleton G, Axton M, Baak A, et al. The FAIR guiding principles for scientific data management and stewardship. Scientific data. 2016;3(1):1-9.

```{r echo = FALSE, warning = FALSE, message = FALSE} source("init.R") source("images/part_0/part_0_data.R") ``` # What is data? {#sec-what-is-data} *Last modified on `r format(fs::file_info("chapter-02-what-is-data.qmd")$modification_time, '%d. %B %Y at %H:%M:%S')`* > *"The limits of my language mean the limits of my world." --- Ludwig Wittgenstein* Emma sat on the wooden stage. She need time to recover from what she had seen behind the curtains. What she had seen beneath the wood had now become reality. Dust danced in the light. Single light beams hit some dust particles from time to time and exploded into golden rain. Sparkles dropped on the wood. "How is gold made?" Emma asked. "This was a long break. I'm a bit surprised by your question. I thought we should discuss your findings from behind the stage. Not gold." Jeff answers from his seat on the left. "Gold cannot be formed through fusion in a star. This is only possible up to iron. You need a supernova or a collision of two neutron stars." Emma spoke absently into the darkness. "Interesting topic. Was this observed? The collision of two neutron stars and the resulting gold spewing out," Jeff laughed. "No, it was a computer simulation. Artificial data, if you will." Emma looked around. She decided that it was time to begin the play. "Then it's not real, and it's not an explanation for me at all." Jeff stated. "But the simulation was based on real observations." Emma stood up and started building the scene for the play. ## What is observed? ![My son and I are observing the people riding bicycles on an early spring day. Given their clothing and headwear, my son tried to guess what season it really was. Some cyclists seem to be dressed for spring, others for winter, and one young woman for summer. He was very confused because it was really hard to guess. Why was there such a wide variation?](images/data_window_view.png){fig-align="center" width="373" #fig-data-window-view} Can we observe anything? What does it mean to observe? We see things, but our vision is already influenced by our understanding of what they are. Additionally, by naming things and processes, we are already following a theory. "The sun is rising" is a popular example of a saying connected with the geocentric theory, which states that the Earth is at the center of the solar system. Therefore, our language is already influenced by our observations. We cannot speak about something we observe without giving the observation some meaning. It is possible, but it demands a high level of concentration. ```{r} #| message: false #| echo: false #| warning: false #| fig-align: center #| fig-height: 0.75 #| fig-width: 7.5 #| fig-cap: "An observation is a measurement or sensory input within a theoretical and linguistic framework. An observation cannot be made without a context. There is no such thing as an objective observation." #| label: fig-observation p_observation_data ``` This book requires a highly specific type of observation. We need data. Very rarely is any observation we make based on first-hand data. For example, if you look outside your window and observe a bird, you will learn some information about it. But is this data? No, it is just information. Data is information, but structured information. Data is a type of observation that is stored in a specific way. We translate observations and information about them into a set of numbers. Not every observation can be represented as a number in a data sheet or table. As their names suggest, data sheets and tables are two-dimensional. A sheet is a type of paper on which organised observations can be written down. The same is true for a table. It is a structured type of observation. ## What is data? We will now think about data. This will happen after we have discussed the science in the chapter. It will also happen if you have skipped the chapter entirely. What is data? If you had asked me when I was a teenager, I would have referred to Star Trek: The Next Generation and Commander Data. He was an android without any emotions. Pure objectivity. He was an artificial being. However, this is not the type of data we want to discuss. We want to discuss data that has been observed and measured. Furthermore, our data is numeric or has a numeric counterpart. We use descriptive words, but rarely. Later on, we want to run algorithms and calculations with our data. Therefore, the data must be open to calculations. Data are observations represented by numbers and letters. Can we use data in any format? No, you cannot. There is a general format for how data should be stored. In our case, we have observations in the rows and variables of measurement in the columns. There can only be one observation per row. This format is called the long format. Another possibility is the wide format, where the observations are scattered around and do not share the same rows. The wide format may be useful for time series, but I do not recommend it. The following @fig-02-data-2d-matrix shows a typical data set used in data science and statistics. It is also typical for those working in the life sciences. The matrix has $n$ rows and $p$ columns. To select a cell, we can write the row and column information in brackets. The rows are indicated by $i$ and the columns by $j$. This is a simple way of extracting information from a data table. However, these manipulations are not normally used anymore because they are very error-prone. This is a simple yet powerful way of storing information. We translate our observations into numbers and letters, which we then put into a two-dimensional matrix. ```{r} #| message: false #| echo: false #| warning: false #| fig-align: center #| fig-height: 3.25 #| fig-width: 7 #| fig-cap: "Theoretical data set. It has $n$ rows and $p$ columns. The row and column positions can be used to access each cell of this two-dimensional matrix, which is indicated by the grey shaded area." #| label: fig-02-data-2d-matrix p_data_2d_matrix ``` @fig-02-data-2d-matrix-annotated might be interesting, but it is not very helpful in the daily working process. Therefore, I will show you an annotated data table with example observations labelled with numbers and letters. The first columns contain an identifier for each observation, or ID. I always include one in my data sets because I always want to know which individual is which. This is even more important when working with time series or repeated measurements. After the ID column, we find the experimental variables. We can also refer to these as 'X' variables, as we will plot them on the x-axis in our visualizations later on. We will often set the experimental variables before we start the experiment. Then, we find the measurements on the individuals. These variables are called outcomes or 'Y' variables, as they will be the variables on our y-axis later on. The terminology used varies from scientific field to scientific field. I will therefore talk about terminology in a separate chapter later on. ```{r} #| message: false #| echo: false #| warning: false #| fig-align: center #| fig-height: 3.25 #| fig-width: 7 #| fig-cap: "Annotated dataset. The individuals' observations are shown in the rows. The columns show the single identifier of the observations, the experimental factors set before the experiment, and the measured variables after the experiment has concluded." #| label: fig-02-data-2d-matrix-annotated p_data_2d_matrix_annotated ``` Can we live with this two-dimensional representation of data? Not really. If you look at different fields, you will find a variety of other data tables. In genetics, we do not have data sets of this kind when we look at thousands of genes. Therefore, we have at least two data files: One contains all the information on the patients or individuals, and the other contains all the genetic information. Most of the time, the genetic file is saved in a transposed form. This means that the variables or genes are in the rows and the individuals are in the columns. This is because it is easier to change rows in a data set than columns. This gives us an advantage when it comes to saving and manipulating the data. However, this makes it slightly more complicated to work with such data. Further, geology has its own data format, while other fields use different ways of storing data. Storage methods sometimes depend on the analysis software used or the storage and calculation capacities required. Here, we will focus on the presented data set in @fig-02-data-2d-matrix-annotated, making some modifications to the columns if we are working with time series or repeated measurements. However, we will stick to one data set and will not consider complicated representations. We will talk about the experimental variables and the different types of measurements, both of which are represented by different numbers and letters. There are still many topics to consider if we want to work with data to obtain information and knowledge. In the following sections, we will cover different attributes and limitations of data that are important for successful work. ## Meta information of data The data seems easy enough. We have a two-dimensional table with numbers and letters. That could be it. However, there are some things to consider. Information is often hidden in data. Where does the data come from? What are the internal generation processes? Are there unknown or known structures beneath the numbers and letters that cannot be seen by the row and column names? First, we need to consider the source of the data. Depending on where you obtain your data, there may be more regularities to consider than in other fields. Second, we must discuss how the data is generated. Are you conducting a controlled experiment where you only change one influential variable and observe what happens? Or are you doing nothing and only observing the data? Does your role in the generation process play an important role? Third, we would like to discuss the replication. We need replications in our data in order to perform statistical analyses later on. However, there are two types of replication. They can appear similar in a dataset, but they have a significant impact on your analysis and the answers to your questions. Is the data valid? This may seem like a simple question, but it is sometimes very important. If you are a statistician working with data, your possibilities are limited. Often, you are not an expert in the field in which you are conducting the analysis. You could argue that men should, on average, be larger than women. However, if the data is more specific, you need help to determine its validity. Next, we will examine artificial or simulated data. This special type of data is used mainly in statistical research and teaching, and we will use it in this book. Non-statisticians should not use artificial data in their daily research to cheat their data sheets. Is all this information lost or hidden? In a sense, yes, because if you don't have any information on the generation process, or if you can only see some cryptic column names, it might be impossible for you to gain any deeper knowledge of the data. However, it is always possible to find a pattern in anything. Garbage in garbage out ### Source of data Where does your data come from? Does it matter? Yes, because it has a huge influence on your work and how you share it. First, let's define the different sources of data. One type is human data, such as blood samples or questionnaires. Humans can share information about social, economic, and other factors. Human data is often used in medicine and the social sciences. Then, we can examine data from animals. Mice are a special case because much basic research is conducted on mice before moving on to humans. The rest of the animal studies include all animals relevant to agriculture and leisure activities, such as dogs and horses. Naturally, there is a great deal of interest in chickens, pigs, and cows because of their role in food production. Finally, we have plants, including fungi and other plant-based sources. Our focus is mostly on crops that are interesting to agriculture, but the field is broad. On the sidelines, we have cell cultures, which have their own problems and limitations. @fig-source-data shows a visualization of the different sources of data and their connection to regulation and openness. We often refer to data from humans, animals and plants as *in vivo* data because it originates from living beings. This includes mice, which are the general animal model for humans. If artificial or simulated data is used in statistics, these data sources are referred to as *in silico* data. In my view and in the context of this book, artificial data is common, but this is a biased perspective on data. Most of the time, data is generated by experiments or observations rather than by algorithms. Nevertheless, human data is largely regulated. Human data cannot be shared openly. The same is partly true for animal models. Plant data and some types of cell cultures are easier to share. If you simulate your data, you can share it as freely as you like. ```{r} #| message: false #| echo: false #| warning: false #| fig-align: center #| fig-height: 2.25 #| fig-width: 7 #| fig-cap: "Figure showing examples of sources of data. On the left are the three main *in vivo* sources of data: human, animal and plant, in order of decreasing regulatory requirements. Cell cultures can be conducted from each of these sources. On the left is the *in silico* source of data, which uses artificial or simulated data. The openness and shareability of data moves in the opposite direction to the level of regulation." #| label: fig-source-data p_source_data ``` Finally, we must consider the ethical and cultural issues involved. You cannot do whatever you want with patients and animals. Depending on your location and nationality, the process of working with or sharing human or animal data is highly regulated. You may need an ethical vote from an Ethics Committee at your institution or approval for an animal experiment from a different committee. Sometimes cultural borders hinder work with specific animal breeds. All of this is beyond the scope of this book. If you are unsure whether your experiment can be conducted and data collected, please contact a statistician at your institution or a nearby university. As a general rule, never upload any human patient data, with or without personal identifiers, to an open database without consulting the relevant legal authorities first. ### Experimental vs. observed data There is a lot that could be written about experimental and observed data. The terminology can be a little confusing because 'experimental data' consists of observations that have been clearly recorded. Therefore, I prefer to refer to it as data that comes from a controlled experiment. Conversely, observed data is data generated without any influence from an experimenter. In this book, we will focus primarily on experimental data. However, many of the algorithms introduced can also be used with observed data. There are no real limitations to their application. The problem lies in interpretation. With experimental data, you know which influencers are changing because you are the one changing them. With observed data, however, you can never be sure whether you are observing correlation or causality because you are not controlling any of the influencing factors. If you are missing information, you may draw the wrong conclusions. In a clinical context, we often refer to observed data as real-world data, which can come from anywhere. This can be data collected from wearable devices such as clocks and smartphones. It can also be collected from operations or other examinations of patients during normal clinical processes. This data is merely observed; there is no controlled assignment of patients to treatments or other controlled activities. In this context, we can also speak of experimental data from randomised clinical trials (RCTs) and observational data as real-world data from clinical routine. We live in a data-driven world. Or, we could say that we live in an information-driven world. There is so much information available that we believe everything is clear and well known. Sometimes, however, we realise that this is not the case. We only have partial information in very specific areas where we gather it. Someone might think that weather forecasting should be easy. We have satellites, we know past temperatures, wind speeds and air pressure. Yet we still have problems predicting the weather a week in advance or thunderstorms a few hours in advance. We know that the conditions are right for a thunderstorm, but we only find out where it will build up once we observe it. ### Biological vs. technical replication When I first started studying statistics, I found the replications a little irritating. I was familiar with numbers and calculations from school mathematics lessons. There we had a formula or equation and I put in numbers to calculate something. I never did this more than once. Why should I calculate the same thing again and again? Why is there replication in statistical data? We need replication, because we need a probabilistic process to do statistics in science. By the probabilistic process we can then calculate probabilities. Without a probabilistic process and replication we cannot have any probabilities. Perhaps the explanation comes a little early, but we will revisit this topic throughout the book. It is important to remember here that we want to calculate probabilities. Therefore, we need replications in order to calculate these probabilities. The following @fig-replicate-data illustrates the various types of replication. In general, we divide replication into biological and technical replication. We are also able to conduct biological replication at many points in time. This may sound confusing, but the difference lies in the time between the measurements. Let us start with biological replication. We measure an outcome Y on different biological individuals. Depending on our research question, we might measure only a few individuals or hundreds to thousands. These individuals can be humans, plants or mice. Technical replication involves taking measurements many times on the same individual at the same time. This allows us to ensure that the measurement was correct. For example, you could take your body weight three times and then combine the three measurements to give one overall result. If a large amount of time passes between the individual measurements, we speak of a repeated measurement. In this case, we are still measuring the same individual, but at different points in time. We want to know how body weight changes in humans over weeks. This is a biological replication at different points in time. ```{r} #| message: false #| echo: false #| warning: false #| fig-align: center #| fig-height: 3 #| fig-width: 8.5 #| fig-cap: "Different types of replication can be observed in data tables and experiments. **(A)** First, there is biological replication, where a measured $Y$ is repeatedly observed in different biological individuals. **(B)** The second type is repeated measurement, where a measured Y is observed on each individual at different points in time. The time between each measurement is long. **(C)** Third, technical replication involves measuring individuals in a row at one point in time. The time between each observation is negligible. The values measured for each individual are mostly combined into a single measure." #| label: fig-replicate-data p_measure + p_repeat_measure + p_tech_rep + plot_layout(width = c(0.3,0.75,1)) + plot_annotation(tag_levels = 'A', tag_prefix = '(', tag_suffix = ')') & theme(plot.tag = element_text(size = 16, face = "bold")) ``` Where is the replication in the technical replication? ::: callout-note ## A word of caution: Should be all the values the same? Should we not measure always the same value? ::: ### Internal and external validity Can we trust the data we are working with? Is the data valid, and what can we do with it? This also depends on the information buried in the data. If we can find information in our data, we can make decisions or predictions. However, these decisions and predictions will only be useful for future eventualities if the data from the past is connected to the future. Internal validity is often considered a prerequisite for external validity, as a study must first be true for its participants before it can be generalized to the broader population. ```{r} #| message: false #| echo: false #| warning: false #| fig-align: center #| fig-height: 4.25 #| fig-width: 8 #| fig-cap: "fooo" #| label: fig-data-internal-external p_internal_external ``` ::: callout-note ## Examples of failures in internal validity **Moving plants** foo **Mice with big ears** bar **Electrified children** hust ::: https://pmc.ncbi.nlm.nih.gov/articles/PMC6188693/ https://en.wikipedia.org/wiki/External_validity https://www.researchgate.net/profile/Jolaine-Draugalis/publication/260267611_Establishing_the_internal_and_external_validity_of_experimental_studies/links/56c2042208ae44da37ff4feb/Establishing-the-internal-and-external-validity-of-experimental-studies.pdf https://journals.sagepub.com/doi/pdf/10.1177/17456916221136117 https://www.verywellmind.com/internal-and-external-validity-4584479#:~:text=External%20validity%20relates%20to%20how%20applicable%20the,*%20Maturation%20*%20Statistical%20regression%20*%20Testing https://dovetail.com/research/internal-validity/ Threats to internal data validity - History - Social - Blinding - Positive and negative control - Maturation and learning - Changing in instruments From there, we have the famous saying 'garbage in, garbage out', which is illustrated in @fig-data-garbage. If we input garbage data into a statistical algorithm, we will find a pattern. The pattern will usually be of poor quality too. Quite often, we do this in a setting where we don't even understand the algorithm. Therefore, we put something into an algorithm and hope that something useful will come out. If we get something, we are very happy and continue changing nothing in case everything is lost. Welcome to one possible cause of the replication crisis. ```{r} #| message: false #| echo: false #| warning: false #| fig-align: center #| fig-height: 1.2 #| fig-width: 7 #| fig-cap: "Visualisation of the saying 'garbage in, garbage out'. In other words, if the input data is bad, the statistical algorithm cannot work miracles. The output will also be garbage, regardless of the statistical algorithm. In such a setting, the algorithm is also seen as a black box because, if the data is not understood, neither will the algorithm be." #| label: fig-data-garbage p_data_garbage ``` [Guidance on Validity of Clinical Studies](https://health.ec.europa.eu/document/download/9f9dbfe4-078b-4959-9a07-df9167258772_en?filename=hta_clinical-studies-validity_guidance_en.pdf)@EC2024HTA and @toumi2025clinical ### Simulation and artificial data You might be surprised to learn that, in statistical research, we statisticians work with simulated data. This data is also known as artificial data. This data has no experimental background and is entirely simulated. We simulate our measured values based on the experimental values. Therefore, we need to make some assumptions about the simulation process. Consequently, we can conclude that our artificial data is model-laden. We need a model to connect the experimental and measured variables. We can call this connection a model. If we change the model, the generated data will also change. In this book, we will simulate our data. This may seem counterintuitive, as we will use data with predefined effects and differences. What is the point of such data? What can we learn from it? The advantage of simulated data is that we already know what we want to find with our statistical algorithms. The advantage of using simulated data is that we already know what we are looking for with our statistical algorithms. Otherwise, we would not know whether there were no patterns in the data or whether our algorithm was unable to detect them. @dormann2025statistics In vivo versus in silico. ## Ways to work with data ### Long format vs. wide format There are two types of arranging your data in a 2D data table. If we put each observation in one line, we speak of A question of time [Entity–attribute–value model](https://en.wikipedia.org/wiki/Entity%E2%80%93attribute%E2%80%93value_model) Wide, or unstacked data is presented with each different data variable in a separate column. Narrow, stacked, or long data is presented with one column containing all the values and another column listing the context of the value ```{r} #| message: false #| echo: false #| warning: false #| fig-align: center #| fig-height: 3.75 #| fig-width: 6.5 #| fig-cap: "Two types of data storage exist in the wide and long format. It is important to remember that we need the long format for visualization and analysis in R. **(A)** The wide format is also called unstacked data. We use this type of data if we are measuring time series or changing data from the same subject. This format is very efficient for storage but difficult to analyze. **(B)** The long format is also called stacked or narrow data. The long format is the standard for data analysis. The measured variable Y is contained in only one column." #| label: fig-long-wide p_wide + p_long + plot_layout(width = c(0.6,0.4)) + plot_annotation(tag_levels = 'A', tag_prefix = '(', tag_suffix = ')') & theme(plot.tag = element_text(size = 16, face = "bold")) ``` Two types of data storage in the wide and long format. It is important to remeber, that we need the long format for the visualitation and analysis in R. The wide format is also called unstacked data. We use this type of data, if we measure time series data or data which is changing in the same subject. In this applications, the format is very effiecent to store but difficult to analyse. The long format is also called stacked or narrow data. The long format is the standard format for the analysis of data. The measured variable Y is only contained in one column. ### The elephant in the room is Excel @hassenstein2025ten @wilkinson2016fair What problems are there with Excel? - Reach and shareability. If we do something in Excel by point and click, nobody else can do it if they are not observing us. - Scalability and efficiency. If we have one task, it will take some time. If we have the same task with different numbers, it will take twice as long. We cannot really automate any processes. - Excel is error-prone. If we make a mistake, we will not really understand why things did not work. We can click again, but we have no protocol to follow. Others cannot understand what we have done. - Excel is neither open nor free to use. We need a licence to use Excel and are unable to share any files with people who do not have access to Microsoft products. ### Tidy data ## R packages used Do we want here to simulate data? ```{r} #| message: false #| echo: false #| warning: false #| fig-align: center #| fig-height: 3.25 #| fig-width: 7 #| fig-cap: "foo. **(A)** foo. **(B)**" #| label: fig-02-cabin-tibble p_tibble_empty <- ggplot() + theme_void() + # theme_minimal() + coord_cartesian(xlim = c(0, 10), ylim = c(0.5, 10.5)) + scale_x_continuous(breaks = seq(-10,10,1), expand = expansion(mult = c(0, 0))) + scale_y_continuous(breaks = seq(-10,10,1), expand = expansion(mult = c(0, 0))) + geom_cabin(x = 0.5, y = 0.5) + labs(title = "A tibble()") + theme(plot.title = element_text(size = 16, face = "bold"), plot.subtitle = element_text(size = 12, face = "italic")) p_tibble_filled <- ggplot() + theme_void() + # theme_minimal() + coord_cartesian(xlim = c(0, 10), ylim = c(0.5, 10.5)) + scale_x_continuous(breaks = seq(-10,10,1), expand = expansion(mult = c(0, 0))) + scale_y_continuous(breaks = seq(-10,10,1), expand = expansion(mult = c(0, 0))) + geom_cabin(x = 0.5, y = 0.5) + geom_hook(x = 1.7, y = 8.25, width = 1.3, angle = 0, hook_text = str_c(c("cat", "cat", "cat", "...", "dog", "dog", "dog"), collapse = "\n"), label = FALSE) + geom_hook(x = 3.7, y = 8.25, width = 1.3, angle = 0, hook_text = str_c(c("33.2", "31.7", "36.8", "...", "22.1", "19.7", "24.3"), collapse = "\n"), label = FALSE) + geom_hook(x = 5.7, y = 8.25, width = 1.3, angle = 0, hook_text = str_c(c("10.1", "12.4", "15.6", "...", "22.4", "18.1", "25.6"), collapse = "\n"), label = FALSE) + geom_hook(x = 7.7, y = 8.25, width = 1.3, angle = 0, hook_text = str_c(c("yes", "yes", "no", "...", "no", "yes", "no"), collapse = "\n"), label = FALSE) + labs(title = "A tibble()") + theme(plot.title = element_text(size = 16, face = "bold"), plot.subtitle = element_text(size = 12, face = "italic")) p_tibble_empty + p_tibble_filled + plot_layout(ncol = 2) + plot_annotation(tag_levels = 'A', tag_prefix = '(', tag_suffix = ')') & theme(plot.tag = element_text(size = 16, face = "bold")) ``` ```{r} #| message: false #| echo: false #| warning: false #| fig-align: center #| fig-height: 3 #| fig-width: 4 #| fig-cap: "foo." #| label: fig-02-cabbinet ggplot() + theme_void() + # theme_minimal() + coord_cartesian(xlim = c(0, 10), ylim = c(0.5, 10)) + scale_x_continuous(breaks = seq(-10,10,1), expand = expansion(mult = c(0, 0))) + scale_y_continuous(breaks = seq(-10,10,1), expand = expansion(mult = c(0, 0))) + geom_cabin(x = 0.5, y = 0.5) + geom_hook(x = 1.5, y = 8.25, width = 0.8, angle = 0, hook_text = str_c(c(2.1, 3.2, 2.1, 3.2, 2.1, 3.2, 2.1, 3.2), collapse = "\n"), label_text = "hust") + geom_hook(x = 2.75, y = 8.25, width = 0.8, angle = 0, hook_text = str_c(c(2.1, 3.2, 2.1, 3.2, 2.1, 3.2, 2.1, 3.2), collapse = "\n"), label_text = "hust") + geom_hook(x = 4, y = 8.25, width = 0.8, angle = 0, hook_text = str_c(c(2.1, 3.2, 2.1, 3.2, 2.1, 3.2, 2.1, 3.2), collapse = "\n"), label_text = "hust") + geom_hook(x = 5.25, y = 8.25, width = 0.8, angle = 0, hook_text = str_c(c(2.1, 3.2, 2.1, 3.2, 2.1, 3.2, 2.1, 3.2), collapse = "\n"), label_text = "hust") + geom_hook(x = 5.25, y = 8.25, width = 0.8, angle = 0, hook_text = str_c(c(2.1, 3.2, 2.1, 3.2, 2.1, 3.2, 2.1, 3.2), collapse = "\n"), label_text = "hust") + geom_hook(x = 6.5, y = 8.25, width = 0.8, angle = 0, hook_text = str_c(c(2.1, 3.2, 2.1, 3.2, 2.1, 3.2, 2.1, 3.2), collapse = "\n"), label_text = "hust") + geom_hook(x = 7.75, y = 8.25, width = 0.8, angle = 0, hook_text = str_c(c(2.1, 3.2, 2.1, 3.2, 2.1, 3.2, 2.1, 3.2), collapse = "\n"), label_text = "hust") ``` ## Data Is your data the fuel of discovery or the test of discovery? ```{r} jump_weight_tbl <- tibble(x = c(0.6, 1, 2.3, 3.5, 5.2, 7.1, 8.4, 9.2, 10), y = 0.15*x^3 - 2.2*x^2 + 8.8*x + 3.2 + rnorm(9, 0, 0.5)) |> mutate_all(round, 1) |> rename(weight_mg = x, jump_length_cm = y) ``` ```{r} #| echo: false #| message: false #| warning: false #| label: tbl-02-jump-weight #| tbl-cap: "foo." jump_weight_tbl|> kable(align = "c", "pipe") ``` @eq-jump-weight $$ y = 0.15\cdot x^3 - 2.2\cdot x^2 + 8.8 \cdot x + 3.2 $$ {#eq-jump-weight} ```{r} #| message: false #| echo: false #| warning: false #| fig-align: center #| fig-height: 4 #| fig-width: 10 #| fig-cap: "foo." #| label: fig-02-science-to-model p1 <- ggplot(jump_weight_tbl, aes(x = weight_mg, y = jump_length_cm)) + theme_book() + scale_x_continuous(breaks = 0:10, limits = c(0, 10)) + scale_y_continuous(breaks = seq(7, 21, 2), limits = c(7, 22)) + labs(x = "Flea weight in [mg]", y = "Jump length in [cm]", title = "Science") + theme(axis.text = element_blank()) p2 <- ggplot(jump_weight_tbl, aes(x = weight_mg, y = jump_length_cm)) + theme_book() + geom_point() + scale_x_continuous(breaks = 0:10, limits = c(0, 10)) + scale_y_continuous(breaks = seq(7, 21, 2), limits = c(7, 22)) + labs(x = "weight_mg", y = "jump_length_cm", title = "Data") p3 <- ggplot(jump_weight_tbl, aes(x = weight_mg, y = jump_length_cm)) + theme_book() + geom_function(fun = \(x) 0.15*x^3 - 2.2*x^2 + 8.8*x + 3.2, size = 1) + scale_x_continuous(breaks = 0:10, limits = c(0, 10)) + scale_y_continuous(breaks = seq(7, 21, 2), limits = c(7, 22)) + labs(x = "Influencer (X)", y = "Outcome (Y)", title = "Model") + theme(axis.text = element_blank()) p1 + p2 + p3 + plot_layout(ncol = 3) + plot_annotation(tag_levels = 'A', tag_prefix = '(', tag_suffix = ')') & theme(plot.tag = element_text(size = 16, face = "bold")) ``` ```{r} #| message: false #| echo: false #| warning: false #| fig-align: center #| fig-height: 4.5 #| fig-width: 5.5 #| fig-cap: "foo." #| label: fig-02-science-with-model ggplot(jump_weight_tbl, aes(x = weight_mg, y = jump_length_cm)) + theme_book() + geom_point(size = 3, shape = 21, fill = col_pal(1, 0.5)) + geom_function(fun = \(x) 0.15*x^3 - 2.2*x^2 + 8.8*x + 3.2) + scale_x_continuous(breaks = 0:10, limits = c(0, 10)) + scale_y_continuous(breaks = seq(7, 21, 2), limits = c(7, 22)) + labs(x = "Flea weight in [mg]", y = "Jump length in [cm]", title = "Science with data and model") ``` ## Outro What do we want to do in this book? We want to perform inductive reasoning based on data. Statistical modeling is inductive reasoning, so it is also based on data. Therefore, we will need data to perform statistical analyses in any possible combination. In other words, we live in a data-driven world. In this book, we will follow an inductive approach. First, we make observations, then we try to find patterns in them. Therefore, our data is the fuel of discovery. Conversely, we can also use data to test a hypothesis. Both are valid and are used. Since this book is about statistics and data science, we cannot generate new theories. We are limited by the tools we have and the data available to us. This book may be read by those with a background in natural science. Then, you can use the statistical tools differently. ## Alternatives Further tutorials and R packages on XXX ## Glossary term : what does it mean. ## The meaning of "Models of Reality" in this chapter. - itemize with max. 5-6 words ## Summary ## References {.unnumbered}