1.1.1. Packages for Data Science#

Python is general-purpose programming language, so data structures and functions specialized for research computing are not built-in. Instead, the community provides these through packages. Nevertheless, and to the community’s credit, Python is a leading language for research computing and data science. This section introduces some of the fundamental packages for research computing.

NumPy provides $n$-dimensional arrays (such as vectors and matrices) and a broad collection of mathematical functions. NumPy is the primary way to do linear algebra efficiently in Python. Many other packages depend on or are compatible with NumPy.

For data science, data frames, which represent tables of data, are another fundamental data structure. Several competing packages provide data frames and related functions:

• Pandas is the oldest and most widely-used data frame package. It provides a broad set of features but also has many quirks. Pandas is generally less efficient than other packages in terms of compute time and memory usage.

• Polars is specifically designed to be efficient, prevent bugs, and present a consistent programming interface. Polars is what we currently recommend for most people.

• DuckDB treats data frames as tables in a database. This makes DuckDB extremely efficient and also means all operations on DuckDB data frames must be written in Structured Query Language (SQL) rather than Python. If you’re comfortable with SQL or need to work with big data (hundreds of gigabytes or more), DuckDB is a good choice.

• Ibis provides data frames that can use any of the other packages under-the-hood. Ibis uses DuckDB by default, so it has similar efficiency but a Python programming interface (SQL is also supported).

We’ll use Polars, because it’s substantially more efficient than Pandas and provides early warnings for certain kinds of common mistakes.

Other notable packages for research computing in Python:

• Jupyter provides interactive notebooks and IPython, a more convenient command-line prompt for Python.

• SciPy provides even more mathematical functions, to supplement NumPy.

• Matplotlib provides a rich collection of visualization functions and is the foundation for many data visualization packages.

• statsmodels provides functions to fit statistical models (where the focus is inference and interpretability).

• scikit-learn provides functions to fit machine learning models (where the focus is prediction). The package’s excellent documentation also provides a light but practical introduction to the models.

This is by no means an exhaustive list. You’re likely to encounter many more packages as you learn and use Python.

pixi init python_basics
cd python_basics
pixi add python jupyter numpy polars plotnine

pixi run jupyter lab

1.3.1. Variables#

Python and most other programming languages allow you to create named values called variables. You can create a variable with the assignment operator = by writing a name on the left-hand side and a value or expression on the right hand side. For example, to save the estimated area of the triangle in a variable called area, you can write:

area = 3 * 4 / 2

In Python, variable names can consist of any combination of letters and underscores (_). Names can also include numbers, but can’t start with a number. Spaces, dots (.), and other symbols are not allowed in variable names. So geese, top50dogs and nine_lives are valid variable names, but goose teeth, tropical.fish and 9_lives are not.

The main reason to use variables is to temporarily save results from expressions so that you can use them in other expressions. For instance, now you can use the area variable anywhere you want the area of the triangle.

Notice that when you assign a result to a variable, Python doesn’t automatically display that result. If you want to see the result as well, you have to enter the variable’s name as a separate expression:

area

6.0

Another reason to use variables is to make an expression clearer and more general. For instance, you might want to compute the area of several triangles with different bases and heights. Then the expression 3 * 4 / 2 is too specific. Instead, you can create variables base and height, then rewrite the expression as base * height / 2. This makes the expression easier to understand, because the reader does not have to intuit that 3 and 4 are the base and height in the formula. Here’s the new code to compute and display the area of a triangle with base 3 and height 4:

base = 3
height = 4
area = base * height / 2
area

6.0

Now if you want to compute the area for a different triangle, all you have to do is change base and height and run the code again (Python will not update area until you do this). Writing code that’s general enough to reuse across multiple problems can be a big time-saver in the long run. Later on, you’ll see ways to make this code even easier to reuse.

1.3.2. Strings#

Python treats anything inside single or double quotes as literal text rather than as an expression to evaluate. In programming jargon, a piece of literal text is called a string. You can use whichever kind of quotes you prefer, but the quote at the beginning of the string must match the quote at the end.

'Hi'

'Hi'

"Hello!"

'Hello!'

Numbers and strings are not the same thing, so for example Python considers 1 different from "1". You can use the number 1 in mathematical expressions, but not the string "1".

1.3.3. Comparisons#

Besides arithmetic, you an also use Python to compare values. Programming tasks often involve comparing values. Use comparison operators to do so:

Notice that the “equal to” operator is two equal signs. This is to distinguish it from the assignment = operator.

Here are a few examples:

1.5 < 3

True

"a" > "b"

False

3 == 3.14

False

"hi" == "hi"

True

When you make a comparison, Python returns a Boolean value. There are only two possible Boolean values: True and False. Booleans are commonly used for expressions with yes-or-no responses.

Boolean values are values, so you can use them in other computations. For example:

True

True

True == False

False

1 < x < 2

1 < x and x < 2

1.3.4. Calling Functions#

Python can do a lot more than just arithmetic. Most of Python’s features are provided through functions, pieces of reusable code. You can think of a function as a machine that takes some inputs and uses them to produce some output. In programming jargon, the inputs to a function are called arguments, the output is called the return value, and when you use a function, you’re calling the function.

To call a function, write its name followed by parentheses. Put any arguments to the function inside the parentheses. For example, the function to round a number to a specified decimal place is named round. So you can round the number 8.153 to the nearest integer with this code:

round(8.153)

8

Many functions accept more than one argument. For instance, the round function accepts two arguments: the number to round, and the number of decimal places to keep. When you call a function with multiple arguments, separate the arguments with commas. So to round 8.153 to 1 decimal place:

round(8.153, 1)

8.2

When you call a function, Python assigns the arguments to the function’s parameters. Parameters are special variables that represent the inputs to a function and only exist while that function runs. For example, the round function has parameters number and ndigits. The next section, Getting Help, explains how to look up the parameters for a function.

Some parameters have default arguments. A parameter is automatically assigned its default argument whenever the parameter’s argument is not specified explicitly. As a result, assigning arguments to these parameters is optional. For instance, the ndigits parameter of round has a default argument (round to the nearest integer), so it is okay to call round without setting ndigits, as in round(8.153). In contrast, the numbers parameter does not have a default argument. Getting Help explains how to look up the default arguments for a function.

Python normally assigns arguments to parameters based on their position. The first argument is assigned to the function’s first parameter, the second to the second, and so on. So in the code above, 8.153 is assigned to number and 1 is assigned to ndigits.

You can make Python assign arguments to parameters by name with =, overriding their positions. So some other ways you can write the call above are:

round(8.153, ndigits = 1)

8.2

round(number = 8.153, ndigits = 1)

8.2

round(ndigits = 1, number = 8.153)

8.2

All of these are equivalent. When you write code, choose whatever seems the clearest to you. Leaving parameter names out of calls saves typing, but including some or all of them can make the code easier to understand.

Parameters are not regular variables, and only exist while their associated function runs. You can’t set them before a call, nor can you access them after a call. So this code causes an error:

number = 4.755
round(ndigits = 2)

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[19], line 2
      1 number = 4.755
----> 2 round(ndigits = 2)

TypeError: round() missing required argument 'number' (pos 1)

In the error message, Python says that you forgot to assign an argument to the parameter number. You can keep the variable number and correct the call by making number an argument (for the parameter number):

round(number, ndigits = 2)

4.75

Or, written more explicitly:

round(number = number, ndigits = 2)

4.75

The point is that variables and parameters are distinct, even if they happen to have the same name. The variable number is not the same thing as the parameter number.

1.3.5. Objects & Attributes#

Python represents data as objects. Numbers, strings, data structures, and functions are all examples of objects.

An attribute is an object attached to another object. An attribute usually contains metadata about the object to which it is attached. You can access attributes by typing a . after an object.

When an attribute is a function, it’s called a method. For example, all strings have a capitalize method. Here’s the code to capitalize a string:

"snakes everywhere!".capitalize()

'Snakes everywhere!'

The built-in dir function lists all of the attributes attached to an object. Here are the attributes for a string:

dir("hi")

['__add__',
 '__class__',
 '__contains__',
 '__delattr__',
 '__dir__',
 '__doc__',
 '__eq__',
 '__format__',
 '__ge__',
 '__getattribute__',
 '__getitem__',
 '__getnewargs__',
 '__getstate__',
 '__gt__',
 '__hash__',
 '__init__',
 '__init_subclass__',
 '__iter__',
 '__le__',
 '__len__',
 '__lt__',
 '__mod__',
 '__mul__',
 '__ne__',
 '__new__',
 '__reduce__',
 '__reduce_ex__',
 '__repr__',
 '__rmod__',
 '__rmul__',
 '__setattr__',
 '__sizeof__',
 '__str__',
 '__subclasshook__',
 'capitalize',
 'casefold',
 'center',
 'count',
 'encode',
 'endswith',
 'expandtabs',
 'find',
 'format',
 'format_map',
 'index',
 'isalnum',
 'isalpha',
 'isascii',
 'isdecimal',
 'isdigit',
 'isidentifier',
 'islower',
 'isnumeric',
 'isprintable',
 'isspace',
 'istitle',
 'isupper',
 'join',
 'ljust',
 'lower',
 'lstrip',
 'maketrans',
 'partition',
 'removeprefix',
 'removesuffix',
 'replace',
 'rfind',
 'rindex',
 'rjust',
 'rpartition',
 'rsplit',
 'rstrip',
 'split',
 'splitlines',
 'startswith',
 'strip',
 'swapcase',
 'title',
 'translate',
 'upper',
 'zfill']

1.3.6. Comments#

Like most programming languages, Python provides a way to mark parts of your code as comments: expressions to ignore rather than run. Use comments to plan, explain, and document your code. You can also temporarily comment out code to prevent it from running, which can be helpful for testing and debugging.

In Python, comments begin with number sign # and extend to the end of the line:

# This is a comment.

Python will ignore comments when you run your code.

1.4.1. When Something Goes Wrong#

As a programmer, sooner or later you’ll run some code and get an error message or result you didn’t expect. Don’t panic! Even experienced programmers make mistakes regularly, so learning how to diagnose and fix problems is vital.

Try going through these steps:

1. If Python printed a warning or error message, read it! If you’re not sure what the message means, try searching for it online.

2. Check your code for typographical errors, including incorrect capitalization, whitespace, and missing or extra commas, quotes, and parentheses.

3. Test your code one line at a time, starting from the beginning. After each line that assigns a variable, check that the value of the variable is what you expect. Try to determine the exact line where the problem originates (which may differ from the line that emits an error!).

If none of these steps help, try asking online. Stack Overflow is a popular question and answer website for programmers. Before posting, make sure to read about how to ask a good question.

1.5.1. Importing Modules#

You can import a module with the import command. Python will run the module’s code so that you can use any functions or data structures it defines. There are many modules built into Python that provide extra functionality. In addition, every package consists of one or more modules.

Most packages have a main module with the same name as the package. So the NumPy package provides a module called numpy, and the Polars package provides a module called polars. Make sure you have NumPy installed, then try loading the numpy module:

import numpy

A handful of modules print out a message when loaded, but the vast majority do not. Thus you can assume the import command was successful if nothing is printed. If something goes wrong while loading a module, Python will print out an error message explaining the problem.

Once a module is imported, you can access its functions by typing the name of the module, a dot ., and then the name of the function. For instance, to use the round function provided by NumPy:

numpy.round(3.3)

np.float64(3.0)

Typing the full name of a module is inconvenient, so the import command allows you to define an alias when you import a module. For popular packages, there’s usually a conventional alias for the main module. The conventional alias for numpy is np. Using the conventional alias is a good habit, because it makes it easier for other people to understand your code. Use the as keyword to set an alias when you import a module:

import numpy as np

Now you can call NumPy functions by typing np instead of numpy:

np.round(3.4)

np.float64(3.0)

1.5.2. Jupyter Notebooks#

For data science tasks, it is also common to use a Jupyter notebook with extension .ipynb to store code. In addition to Python code, Jupyter notebooks have full support for formatted text, images, and code from other programming languages such as Julia and R. The tradeoff is that Jupyter notebooks can only be viewed in a web browser.

Jupyter notebooks are more convenient than Python scripts for interactive work such as data analysis and learning or experimenting with the language. On the other hand, Python scripts are more appropriate for long-running code that does not require user interaction (such as web scrapers or scientific simulations) and for developing packages and software. The remainder of this reader assumes you’re using a Jupyter notebook rather than the Python console or a Python script, unless otherwise noted.

You can create a new Jupyter notebook in JupyterLab with this menu option:

File -> New -> Notebook

JupyterLab will prompt you to select a kernel for the notebook. The kernel is the software used to run code in the notebook. For a notebook that will contain Python code, you should choose a Python kernel.

After you select the kernel, you’ll see a pane like this:

Jupyter notebooks are subdivided into cells. You can create as many cells as you like, but each cell can only contain one kind of content, usually code or text.

New cells are code cells by default. You can run a code cell by clicking on the cell and pressing Shift-Enter. The notebook will display the result and create a new empty code cell below the result:

You can convert a code cell to a text cell by clicking on the cell and selecting the “Markdown” option from the cell type dropdown menu:

Markdown is a simple language you can use to add formatting to your text. For example, surrounding a word with asterisks, as in Let *sleeping* dogs lie, makes the surrounded word italic. You can find a short, interactive tutorial about Markdown here. If you “run” a text cell by pressing Shift-Enter, the notebook will display the text with any formatting you added.

1.6.1. Absolute & Relative Paths#

A path that starts from the root directory, like all of the ones we’ve seen so far, is called an absolute path. The path is “absolute” because it unambiguously describes where a file or directory is located. The downside is that absolute paths usually don’t work well if you share your code.

For example, suppose Ada uses the path /Programs/ada/cats.csv to load the cats.csv file in her code. If she shares her code with another pioneer of computing, say Gladys, who also has a copy of cats.csv, it might not work. Even though Gladys has the file, she might not have it in a directory called ada, and might not even have a directory called ada on her computer. Because Ada used an absolute path, her code works on her own computer, but isn’t portable to others.

On the other hand, a relative path is one that doesn’t start from the root directory. The path is “relative” to an unspecified starting point, which usually depends on the context.

For instance, suppose Ada’s code is saved in the file analysis.ipynb, which is in the same directory as cats.csv on her computer. Then instead of an absolute path, she can use a relative path in her code:

cats.csv

The context is the location of analysis.ipynb, the file that contains the code. In other words, the starting point on Ada’s computer is the ada directory. On other computers, the starting point will be different, depending on where the code is stored.

Now suppose Ada sends her corrected code in analysis.ipynb to Gladys, and tells Gladys to put it in the same directory as cats.csv. Since the path cats.csv is relative, the code will still work on Gladys’ computer, as long as the two files are in the same directory. The name of that directory and its location in the file system don’t matter, and don’t have to be the same as on Ada’s computer. Gladys can put the files in a directory /Users/gladys/from_ada/ and the path (and code) will still work.

Relative paths can include directories. For example, suppose that Charles wants to write a relative path from the Users directory to a cool selfie he took. Then he can write:

charles/cool_hair_selfie.jpg

You can read this path as, “Starting from wherever you are, go to the charles directory, and from there go to the cool_hair_selfie.jpg file.” In other words, the relative path depends on the context of the code or program that uses it.

When you write paths, there are three shortcuts you can use. These are most useful in relative paths, but also work in absolute paths:

• . means the current directory.

• .. means the directory above the current directory.

• ~ means the home directory. Each user has their own home directory, whose location depends on the operating system and their username. Home directories are typically found inside C:/Users/ on Windows, /Users/ on macOS, and /home/ on Linux.

As an example, suppose Ada wants to write a (relative) path from the ada directory to Charles’ cool selfie. Using these shortcuts, she can write:

../charles/cool_hair_selfie.jpg

Read this as, “Starting from wherever you are, go up one directory, then go to the charles directory, and then go to the cool_hair_selfie.jpg file.” Since /Users/ada is Ada’s home directory, she could also write the path as:

~/../charles/cool_hair_selfie.jpg

This path has the same effect, but the meaning is slightly different. You can read it as “Starting from your home directory, go up one directory, then go to the charles directory, and then go to the cool_hair_selfie.jpg file.”

The .. and ~ shortcut are frequently used and worth remembering. The . shortcut is included here in case you see it in someone else’s code. Since it means the current directory, a path like ./cats.csv is identical to cats.csv, and the latter is preferable for being simpler. There are a few specific situations where . is necessary, but they fall outside the scope of this text.

1.6.2. The Working Directory#

Absolute & Relative Paths explained that relative paths have a starting point that depends on the context where the path is used. The working directory is the starting point Python uses for relative paths. Think of the working directory as the directory Python is currently “at” or watching.

Python’s built-in os module provides functions to manipulate the working directory. The function os.getcwd returns the absolute path for the current working directory, as a string. It doesn’t require any arguments:

import os

os.getcwd()

'/home/nick/mill/datalab/teaching/python_basics'

On your computer, the output from os.getcwd will likely be different. This is a very useful function for getting your bearings when you write relative paths. If you write a relative path and it doesn’t work as expected, the first thing to do is check the working directory.

The related os.chdir function changes the working directory. It takes one argument: a path to the new working directory. Here’s an example:

os.chdir("..")

# Now check the working directory.
os.getcwd()

'/home/nick/mill/datalab/teaching'

Another function that’s useful for dealing with the working directory and file system is os.listdir. The os.listdir function returns the names of all of the files and directories inside of a directory. It accepts a path to a directory as an argument, or assumes the working directory if you don’t pass a path. For instance:

# List files and directories in /home/.
os.listdir("/home/")

# List files and directories in the working directory.
os.listdir()

['bml-model',
 'python_intensive_training',
 'python_basics',
 'intro_to_machine_learning',
 'install_guides',
 'workshop_index',
 'reproducible_research',
 '2022_python_bootcamp',
 'testing',
 'intro_to_the_command_line',
 'intermediate_python',
 'workshop_regression',
 'workshop_ocr_python',
 'dynamic_data_viz',
 'adventures_in_data_science',
 'data_viz.md',
 'template_python_workshop',
 'ollama',
 'intro_qgis',
 'ocr',
 'template_r_workshop',
 'data_viz_principles',
 'r_basics',
 'statistics_notes',
 'archive',
 'jupyterbook-notes.md',
 '2021_remote_computing',
 '2024_adventures_in_data_science',
 'cradle',
 'intermediate_r',
 'workshop_handouts',
 'intro_to_sql',
 'intro_to_remote_computing',
 'julia_basics']

As usual, since you have a different computer, you’re likely to see different output if you run this code. If you call os.listdir with an invalid path or an empty directory, Python raises a FileNotFoundError:

os.listdir("/this/path/is/fake/")

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
Cell In[35], line 1
----> 1 os.listdir("/this/path/is/fake/")

FileNotFoundError: [Errno 2] No such file or directory: '/this/path/is/fake/'

1.7.1. Hello, Data!#

The California least tern is a endangered subspecies of seabird that nests along the coast of California and Mexico. The California Department of Fish and Wildlife (CDFW) monitors least tern nesting sites across the state to estimate breeding pairs, fledglings, and predator activity in each annual breeding season.

The CDFW publishes most of the data it collects to the California Open Data portal. The examples in this and subsequent chapters use a cleaned 2000-2023 version of the California least tern data.

Let’s use Polars to read the California least tern data set. The default file name is 2000-2023_ca_least_tern.csv, which suggests it’s a CSV file. The Polars function to read a CSV file is read_csv. The function’s first and only required argument is the path to the CSV file. In the following code, the path to the California least tern data set is data/2000-2023_ca_least_tern.csv, but it might be different for you, depending on Python’s working directory and where you saved the file. We’ll save the result from the read_csv function in a variable called terns. We can use this variable to access the data in subsequent code.

import polars as pl

terns = pl.read_csv("data/2000-2023_ca_least_tern.csv")

If you tried running the line of code above and got an error message, pay attention to what the error message says, and remember the strategies to get help in Getting Help. The most common mistake when reading a file is incorrectly specifying the path, so first check that you got the path right.

If the code ran without errors, it’s a good idea to check that the data set looks like what the documentation describes. When working with a new data set, it usually isn’t a good idea to print the whole thing (at least until you know how big it is). Large data sets can take a long time to print, and the output can be difficult to read.

Instead, use the .head method to print only the beginning, or head, of the data set:

terns.head()

If you run this code and see a similar table, then congratulations, you’ve read your first data set into Python! ✨

Packages for Data Science explained that we use data frames to represent tabular data. Typically, each row in a data frame corresponds to a single subject and is called an observation. Each column corresponds to a measurement of the subject and is called a feature or covariate.

You can check to make sure Polars has indeed created a data frame with the type function (more about this function in Data Types):

type(terns)

polars.dataframe.frame.DataFrame

Everything looks good here.

1.8.1. Summarizing Data#

The .glimpse method provides a structural summary of a data frame. The method lists the data frame’s shape and column names, as well as the type of data in each column and a few example values. Try calling .glimpse on the terns data frame:

terns.glimpse()

Rows: 791
Columns: 43
$ year                <i64> 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000
$ site_name           <str> 'PITTSBURG POWER PLANT', 'ALBANY CENTRAL AVE', 'ALAMEDA POINT', 'KETTLEMAN CITY', 'OCEANO DUNES STATE VEHICULAR RECREATION AREA', 'RANCHO GUADALUPE DUNES PRESERVE', 'VANDENBERG SFB', 'SANTA CLARA RIVER MCGRATH STATE BEACH', 'ORMOND BEACH', 'NBVC POINT MUGU'
$ site_name_2013_2018 <str> 'Pittsburg Power Plant', 'NA_NO POLYGON', 'Alameda Point', 'Kettleman', 'Oceano Dunes State Vehicular Recreation Area', 'Rancho Guadalupe Dunes Preserve', 'Vandenberg AFB', 'Santa Clara River', 'Ormond Beach', 'NBVC Point Mugu'
$ site_name_1988_2001 <str> 'NA_2013_2018 POLYGON', 'Albany Central Avenue', 'NA_2013_2018 POLYGON', 'NA_2013_2018 POLYGON', 'NA_2013_2018 POLYGON', 'NA_2013_2018 POLYGON', 'NA_2013_2018 POLYGON', 'NA_2013_2018 POLYGON', 'NA_2013_2018 POLYGON', 'NA_2013_2018 POLYGON'
$ site_abbr           <str> 'PITT_POWER', 'AL_CENTAVE', 'ALAM_PT', 'KET_CTY', 'OCEANO_DUNES', 'RGDP', 'VAN_SFB', 'S_CLAR_MCG', 'ORMOND', 'PT_MUGU'
$ region_3            <str> 'S.F._BAY', 'S.F._BAY', 'S.F._BAY', 'KINGS', 'CENTRAL', 'CENTRAL', 'CENTRAL', 'SOUTHERN', 'SOUTHERN', 'SOUTHERN'
$ region_4            <str> 'S.F._BAY', 'S.F._BAY', 'S.F._BAY', 'KINGS', 'CENTRAL', 'CENTRAL', 'CENTRAL', 'VENTURA', 'VENTURA', 'VENTURA'
$ event               <str> 'LA_NINA', 'LA_NINA', 'LA_NINA', 'LA_NINA', 'LA_NINA', 'LA_NINA', 'LA_NINA', 'LA_NINA', 'LA_NINA', 'LA_NINA'
$ bp_min              <f64> 15.0, 6.0, 282.0, 2.0, 4.0, 9.0, 30.0, 21.0, 73.0, 166.0
$ bp_max              <f64> 15.0, 12.0, 301.0, 3.0, 5.0, 9.0, 32.0, 21.0, 73.0, 167.0
$ fl_min              <i64> 16, 1, 200, 1, 4, 17, 11, 9, 60, 64
$ fl_max              <i64> 18, 1, 230, 2, 4, 17, 11, 9, 65, 64
$ total_nests         <i64> 15, 20, 312, 3, 5, 9, 32, 22, 73, 252
$ nonpred_eggs        <i64> 3, None, 124, None, 2, 0, None, 4, 2, None
$ nonpred_chicks      <i64> 0, None, 81, 3, 0, 1, 27, 3, 0, None
$ nonpred_fl          <i64> 0, None, 2, 1, 0, 0, 0, None, 0, None
$ nonpred_ad          <i64> 0, None, 1, 6, 0, 0, 0, None, 0, None
$ pred_control        <str> None, None, None, None, None, None, None, None, None, None
$ pred_eggs           <i64> 4, None, 17, None, 0, None, 0, None, None, None
$ pred_chicks         <i64> 2, None, 0, None, 4, None, 3, None, None, None
$ pred_fl             <i64> 0, None, 0, None, 0, None, 0, None, None, None
$ pred_ad             <i64> 0, None, 0, None, 0, None, 0, None, None, None
$ pred_pefa           <str> 'N', None, 'N', None, 'N', None, 'N', None, 'N', None
$ pred_coy_fox        <str> 'N', None, 'N', None, 'N', None, 'N', None, 'N', None
$ pred_meso           <str> 'N', None, 'N', None, 'N', None, 'N', None, 'Y', None
$ pred_owlspp         <str> 'N', None, 'N', None, 'N', None, 'N', None, 'N', None
$ pred_corvid         <str> 'Y', None, 'N', None, 'N', None, 'N', None, 'N', None
$ pred_other_raptor   <str> 'Y', None, 'Y', None, 'N', None, 'Y', None, 'Y', None
$ pred_other_avian    <str> 'N', None, 'Y', None, 'Y', None, 'N', None, 'N', None
$ pred_misc           <str> 'N', None, 'N', None, 'N', None, 'N', None, 'N', None
$ total_pefa          <i64> 0, None, 0, None, 0, None, 0, None, None, None
$ total_coy_fox       <i64> 0, None, 0, None, 0, None, 0, None, None, None
$ total_meso          <i64> 0, None, 0, None, 0, None, 0, None, None, None
$ total_owlspp        <i64> 0, None, 0, None, 0, None, 0, None, None, None
$ total_corvid        <i64> 4, None, 0, None, 0, None, 0, None, None, None
$ total_other_raptor  <i64> 2, None, 6, None, 0, None, 3, None, None, None
$ total_other_avian   <i64> 0, None, 11, None, 4, None, 0, None, None, None
$ total_misc          <i64> 0, None, 0, None, 0, None, 0, None, None, None
$ first_observed      <str> '2000-05-11', None, '2000-05-01', '2000-06-10', '2000-05-04', '2000-05-07', '2000-05-07', '2000-06-06', None, '2000-05-21'
$ last_observed       <str> '2000-08-05', None, '2000-08-19', '2000-09-24', '2000-08-30', '2000-08-13', '2000-08-17', '2000-09-05', None, '2000-08-12'
$ first_nest          <str> '2000-05-26', None, '2000-05-16', '2000-06-17', '2000-05-28', '2000-05-31', '2000-05-28', '2000-06-06', '2000-06-08', '2000-06-01'
$ first_chick         <str> '2000-06-18', None, '2000-06-07', '2000-07-22', '2000-06-20', '2000-06-22', '2000-06-20', '2000-06-28', '2000-06-26', '2000-06-24'
$ first_fledge        <str> '2000-07-08', None, '2000-06-30', '2000-08-06', '2000-07-13', '2000-07-20', '2000-07-15', '2000-07-24', '2000-07-17', '2000-07-16'

The next chapter explains data types in more detail. For now, just take note that there are multiple types (i64, str, and f64 in the terns data frame).

In contrast to the .glimpse method, the .describe method provides a statistical summary of a data frame:

terns.describe()

1.8.2. Summarizing Columns#

You can select individual columns with bracket notation. Put the name of the column in quotes and place that inside of square brackets []. For example, to select the total_nests column:

terns["total_nests"]

Polars provides a variety of methods to compute on columns. For instance, you can use the .mean method to compute the mean:

terns["total_nests"].mean()

162.84546615581098

Similarly, you can use the .min method to compute the smallest value in a column:

terns["total_nests"].min()

0

For columns of categories, statistics like means and minimums aren’t defined. Instead, it’s often informative to count the number of observations of each category. You can do this with the .value_counts method:

terns["year"].value_counts()

We’ll explain more ways to work with data frames and columns in the next chapter.

1.9.1. Exercise#

In a string, an escape sequence or escape code consists of a backslash \ followed by one or more characters. Escape characters make it possible to:

• Write quotes or backslashes in a string

• Use spaces in file paths

• Write characters that don’t appear on your keyboard (for example characters in a different script system)

For example, the escape sequence \n means “newline character.” A full list of these sequences is available at W3Schools.

1. Assign a string that contains a newline to the variable newline. Then display newline via the Python console.

2. The print function renders output in a properly formatted manner. Use this function to print newline.

3. How does the output between these two displays differ? Why do you think this is?

1.9.2. Exercise#

1. Chose a directory on your computer that you’re familiar with, such as your current working directory. Determine the path to the directory, then use os.listdir to display its contents. Do the files displayed match what you see in your systems file browser?

2. Send a path to os.path.exists and inspect its output. What does this function do? See if you can change its output. If you can, why did it change?

1.9.3. Exercise#

1. Open the help file for the Polars read_csv function. The separator parameter controls which characters Polars looks for when determining the columns in a file. What is the default character?

2. A TSV file is similar to CSV files, except it uses tabs to delimit columns. Tabs are represented by escape sequences in Python. Find the right sequence and explain how you would load a TSV file with read_csv.