Data With Dorrie

Data

There are a few things we should always look at ahead of working with the data.

The Data Structure

Before really doing any meaningful analysis or visualization, it's important to make sure that the data and its structure make sense. If the dataset is publicly available and widely used, there's a good chance you can find documentation on how it's structured. Otherwise, like in this case, we can turn the data itself to get a picture of what it can tell us.

Here's the data again, for reference:

Looking at the data, we can see that there's a row for each hour of the week (giving us 168 rows total), each with 3 columns:

• Time: When the activity happened, in MM/DD/YYYY HH:MM AM/PM format
• Shri: The main activity that I did during the hour
• Dorrie: The main activity that Dorrie did during the hour

Assessing Data Quality

After getting an initial sense of how the data is structured, a good next step can be checking that the data is high quality. For this dataset, that might mean making sure that we have:

• No Missing Data: There should be one row for each hour, and each hour should have an activity both Shri and Dorrie.
• No Duplicated Data: There should not be more than one row for each hour, and there should only be one activity for Shri and Dorrie.
• Consistent Data: Data that represents the same activity should have the same exact label. For example, ideally we can avoid having a label for "Eating" and another value for "Eat", since they represent the same thing.
• Clarity: If a data point looks strange or like an outlier, we should investigate it. For example, if Dorrie usually sleeps around 16 hours a day and we notice a day where she didn't sleep, then either something extraordinary happened or there's a data error.

The complexity of this step and the tooling involved will vary significantly based on the scope of the dataset and project. Large corporations have entire teams dedicated to this step, and it's not uncommon for this to take up the majority of a data scientist's time. For our purposes, the dataset is relatively simple and its quality can be validated using a combination of manual inspection and functions in a tool like Excel or Google Sheets.

Exploring The Categories

Whenever we're dealing with categorical data (data that uses labels instead of numbers to describe things), it's helpful to get a sense of all of the distinct categories that exist. This should be relatively straightforward in most data processing tools.

In this dataset, here are my distinct activity categories:

And Dorrie's distinct activity categories:

Listing out the distinct categories helps us start to understand some of the constraints around the data.

For example, we can get a sense of the specificity of the data. With activities like "Exercise" and "TV", there is no way from the given data to figure out what type of exercise was done or what show was watched. Any dataset will have a level of fidelity (or precision) that we need to be aware of.

On the flipside, we can also see that there are some activities that are more specific, like "Basketball with Friends". These activities can be grouped together with other activities to form broader categories, such as "Leisure" or "Productivity". Annotating the data with these broader groups is an example of adding metadata, which is basically data that describes our data (you can see why it's meta!). It's a lot easier to make an existing data point more general than it is to make it more specific.

Having this type of understanding of the data is important for the next step, which is thinking about how to explore the data.