According to PyPI’s most downloaded packages for February 2025
There’s a lot of boring, utility packages. Let’s import the top 15,000, and look at the first 15
import pandas as pd
df = pd.read_csv("spreadsheets/top-pypi-packages.csv")
df.head(15)| download_count | project | |
|---|---|---|
| 0 | 1346601849 | boto3 |
| 1 | 670262655 | urllib3 |
| 2 | 609967137 | botocore |
| 3 | 576250793 | requests |
| 4 | 563169502 | certifi |
| 5 | 533963799 | charset-normalizer |
| 6 | 515259652 | setuptools |
| 7 | 512857052 | idna |
| 8 | 492528430 | grpcio-status |
| 9 | 479677890 | typing-extensions |
| 10 | 474695177 | packaging |
| 11 | 454054215 | aiobotocore |
| 12 | 453387776 | python-dateutil |
| 13 | 415989804 | s3transfer |
| 14 | 404306024 | six |
BORING. For reference
| Package(s) | Description |
|---|---|
boto3, botocore, aiobotocore and s3transfer |
These hook you into Amazon’s Web Service (AWS). Connects with a quick runtime. Machine learning, AI etc. |
urllib3, requests and certifi |
We’ll look at these first, they’re useful. For making HTTP (internet) requests. certifi is for certificates, used to be in requests. |
charset-normalizer and idna |
For fixing/translating encodings |
setuptools and packaging |
For package development |
grpcio-status |
For using GRPC, used for connecting services. |
typing-extensions, python-dateutil and six |
Extensions to base Python libraries. typing-extensions adds typing to older versions of Python 3 (it’s a new thing). python-dateutil expands Python’s datetime module. six provides cross-compatibility between Python 2 and 3 (because \(2\times 3 = 6\)). |
count = df.head(15)["download_count"].sum()
print(f"Together, these packages been downloaded {count} times in the last MONTH. That's 8.5 billion downloads!")Together, these packages been downloaded 8502971755 times in the last MONTH. That's 8.5 billion downloads!It’s worth pointing out that a lot of these downloads could be automatic - scripts running and restarting, creating virtual environments every time etc.
So far, the only ones I want to look at are urllib3/requests. Let’s look to the next 15
df[15:31]| download_count | project | |
|---|---|---|
| 15 | 381826344 | numpy |
| 16 | 360961023 | pyyaml |
| 17 | 359178671 | s3fs |
| 18 | 355011082 | pip |
| 19 | 336751364 | fsspec |
| 20 | 325104210 | google-api-core |
| 21 | 309681352 | cryptography |
| 22 | 287715421 | cffi |
| 23 | 283246117 | pydantic |
| 24 | 282364469 | pycparser |
| 25 | 278848477 | pandas |
| 26 | 276988719 | attrs |
| 27 | 262452647 | markupsafe |
| 28 | 260586799 | rsa |
| 29 | 252369285 | pyasn1 |
| 30 | 251786378 | wheel |
Aha! Here’s some more.
| Package(s) | Description |
|---|---|
numpy |
A numerical Python package. Powerful. |
pyyaml, cffi and pycparser |
For connecting with other languages. pyyaml is a parser for YAML, yet another markup language. cffi is for interacting with C code, while pycparser parses it directly. |
s3fs |
More AWS. |
pip and wheel |
For downloading packages and wheels. pip for packages, wheel for manipulating wheels. |
fsspec |
For file system management. |
google-api-core |
For interacting with Google’s API |
cryptography and rsa |
For creating secret keys and decoding, etc. Fun! |
pydantic |
For validating data - really, for making sure that your data fits the variables/types you expect. |
pandas |
For manipulating data. Glorified Excel. |
attrs |
For improving classes. Fun, but a bit too advanced. |
markupsafe |
For creating strings that escape characters for HTML and XML |
pyasn1 |
Allows use of ASN.1 for managing network protocols |
A bit more variety! Today, we’ll look at the following packages from this top-30 list, in order:
requestsnumpycryptographyrequestsThe requests package lets you send HTTP requests. Typically, these are GET, for getting things, and POST, for sending them.
import requests
r = requests.get("https://uqpug.github.io/")
r<Response [200]>Every HTTP request gets a status response. The code 200 means OK, so “<Response [200]>” means it worked.
Our response has a lot of attributes. The main one is r.text
Note - if you’re pulling from an API, usually you’ll get JSON code in response. Don’t use
r.textthere, user.json().
r.text'<!DOCTYPE html>\n<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en"><head>\n\n<meta charset="utf-8">\n<meta name="generator" content="quarto-1.6.42">\n\n<meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes">\n\n\n<title>UQ Python User Group (UQ PUG) – UQPUG (Python User Group)</title>\n<style>\ncode{white-space: pre-wrap;}\nspan.smallcaps{font-variant: small-caps;}\ndiv.columns{display: flex; gap: min(4vw, 1.5em);}\ndiv.column{flex: auto; overflow-x: auto;}\ndiv.hanging-indent{margin-left: 1.5em; text-indent: -1.5em;}\nul.task-list{list-style: none;}\nul.task-list li input[type="checkbox"] {\n width: 0.8em;\n margin: 0 0.8em 0.2em -1em; /* quarto-specific, see https://github.com/quarto-dev/quarto-cli/issues/4556 */ \n vertical-align: middle;\n}\n</style>\n\n\n<script src="site_libs/quarto-nav/quarto-nav.js"></script>\n<script src="site_libs/quarto-nav/headroom.min.js"></script>\n<script src="site_libs/clipboard/clipboard.min.js"></script>\n<script src="site_libs/quarto-search/autocomplete.umd.js"></script>\n<script src="site_libs/quarto-search/fuse.min.js"></script>\n<script src="site_libs/quarto-search/quarto-search.js"></script>\n<meta name="quarto:offset" content="./">\n<script src="site_libs/quarto-html/quarto.js"></script>\n<script src="site_libs/quarto-html/popper.min.js"></script>\n<script src="site_libs/quarto-html/tippy.umd.min.js"></script>\n<script src="site_libs/quarto-html/anchor.min.js"></script>\n<link href="site_libs/quarto-html/tippy.css" rel="stylesheet">\n<link href="site_libs/quarto-html/quarto-syntax-highlighting-2f5df379a58b258e96c21c0638c20c03.css" rel="stylesheet" id="quarto-text-highlighting-styles">\n<script src="site_libs/bootstrap/bootstrap.min.js"></script>\n<link href="site_libs/bootstrap/bootstrap-icons.css" rel="stylesheet">\n<link href="site_libs/bootstrap/bootstrap-a352ab7406acfd56ea2326799fe2b2a3.min.css" rel="stylesheet" append-hash="true" id="quarto-bootstrap" data-mode="light">\n<script id="quarto-search-options" type="application/json">{\n "location": "navbar",\n "copy-button": false,\n "collapse-after": 3,\n "panel-placement": "end",\n "type": "overlay",\n "limit": 50,\n "keyboard-shortcut": [\n "f",\n "/",\n "s"\n ],\n "show-item-context": false,\n "language": {\n "search-no-results-text": "No results",\n "search-matching-documents-text": "matching documents",\n "search-copy-link-title": "Copy link to search",\n "search-hide-matches-text": "Hide additional matches",\n "search-more-match-text": "more match in this document",\n "search-more-matches-text": "more matches in this document",\n "search-clear-button-title": "Clear",\n "search-text-placeholder": "",\n "search-detached-cancel-button-title": "Cancel",\n "search-submit-button-title": "Submit",\n "search-label": "Search"\n }\n}</script>\n\n\n</head>\n\n<body class="nav-sidebar floating nav-fixed">\n\n<div id="quarto-search-results"></div>\n <header id="quarto-header" class="headroom fixed-top">\n <nav class="navbar navbar-expand-lg " data-bs-theme="dark">\n <div class="navbar-container container-fluid">\n <div class="navbar-brand-container mx-auto">\n <a class="navbar-brand" href="./index.html">\n <span class="navbar-title">UQPUG (Python User Group)</span>\n </a>\n </div>\n <div class="quarto-navbar-tools tools-end">\n</div>\n <div id="quarto-search" class="" title="Search"></div>\n </div> <!-- /container-fluid -->\n </nav>\n <nav class="quarto-secondary-nav">\n <div class="container-fluid d-flex">\n <button type="button" class="quarto-btn-toggle btn" data-bs-toggle="collapse" role="button" data-bs-target=".quarto-sidebar-collapse-item" aria-controls="quarto-sidebar" aria-expanded="false" aria-label="Toggle sidebar navigation" onclick="if (window.quartoToggleHeadroom) { window.quartoToggleHeadroom(); }">\n <i class="bi bi-layout-text-sidebar-reverse"></i>\n </button>\n <nav class="quarto-page-breadcrumbs" aria-label="breadcrumb"><ol class="breadcrumb"><li class="breadcrumb-item"><a href="./index.html">General information</a></li></ol></nav>\n <a class="flex-grow-1" role="navigation" data-bs-toggle="collapse" data-bs-target=".quarto-sidebar-collapse-item" aria-controls="quarto-sidebar" aria-expanded="false" aria-label="Toggle sidebar navigation" onclick="if (window.quartoToggleHeadroom) { window.quartoToggleHeadroom(); }"> \n </a>\n </div>\n </nav>\n</header>\n<!-- content -->\n<div id="quarto-content" class="quarto-container page-columns page-rows-contents page-layout-article page-navbar">\n<!-- sidebar -->\n <nav id="quarto-sidebar" class="sidebar collapse collapse-horizontal quarto-sidebar-collapse-item sidebar-navigation floating overflow-auto">\n <div class="sidebar-menu-container"> \n <ul class="list-unstyled mt-1">\n <li class="sidebar-item">\n <div class="sidebar-item-container"> \n <a href="./index.html" class="sidebar-item-text sidebar-link active">\n <span class="menu-text">General information</span></a>\n </div>\n</li>\n <li class="sidebar-item sidebar-item-section">\n <div class="sidebar-item-container"> \n <a class="sidebar-item-text sidebar-link text-start" data-bs-toggle="collapse" data-bs-target="#quarto-sidebar-section-1" role="navigation" aria-expanded="true">\n <span class="menu-text">Demonstrations</span></a>\n <a class="sidebar-item-toggle text-start" data-bs-toggle="collapse" data-bs-target="#quarto-sidebar-section-1" role="navigation" aria-expanded="true" aria-label="Toggle section">\n <i class="bi bi-chevron-right ms-2"></i>\n </a> \n </div>\n <ul id="quarto-sidebar-section-1" class="collapse list-unstyled sidebar-section depth1 show"> \n <li class="sidebar-item sidebar-item-section">\n <div class="sidebar-item-container"> \n <a class="sidebar-item-text sidebar-link text-start collapsed" data-bs-toggle="collapse" data-bs-target="#quarto-sidebar-section-2" role="navigation" aria-expanded="false">\n <span class="menu-text">Top Modules</span></a>\n <a class="sidebar-item-toggle text-start collapsed" data-bs-toggle="collapse" data-bs-target="#quarto-sidebar-section-2" role="navigation" aria-expanded="false" aria-label="Toggle section">\n <i class="bi bi-chevron-right ms-2"></i>\n </a> \n </div>\n <ul id="quarto-sidebar-section-2" class="collapse list-unstyled sidebar-section depth2 "> \n <li class="sidebar-item">\n <div class="sidebar-item-container"> \n <a href="./Demonstrations/top_modules/top_modules.html" class="sidebar-item-text sidebar-link">\n <span class="menu-text">The most popular Python modules</span></a>\n </div>\n</li>\n </ul>\n </li>\n </ul>\n </li>\n <li class="sidebar-item sidebar-item-section">\n <div class="sidebar-item-container"> \n <a class="sidebar-item-text sidebar-link text-start" data-bs-toggle="collapse" data-bs-target="#quarto-sidebar-section-3" role="navigation" aria-expanded="true">\n <span class="menu-text">Current notebooks - 2025</span></a>\n <a class="sidebar-item-toggle text-start" data-bs-toggle="collapse" data-bs-target="#quarto-sidebar-section-3" role="navigation" aria-expanded="true" aria-label="Toggle section">\n <i class="bi bi-chevron-right ms-2"></i>\n </a> \n </div>\n <ul id="quarto-sidebar-section-3" class="collapse list-unstyled sidebar-section depth1 show"> \n <li class="sidebar-item">\n <div class="sidebar-item-container"> \n <a href="./Current notebooks - 2025/16-Mar25.html" class="sidebar-item-text sidebar-link">\n <span class="menu-text">2025 Mar 6<sup>th</sup> – UQ PUG 16</span></a>\n </div>\n</li>\n </ul>\n </li>\n <li class="sidebar-item sidebar-item-section">\n <div class="sidebar-item-container"> \n <a class="sidebar-item-text sidebar-link text-start" data-bs-toggle="collapse" data-bs-target="#quarto-sidebar-section-4" role="navigation" aria-expanded="true">\n <span class="menu-text">Archive - 2024</span></a>\n <a class="sidebar-item-toggle text-start" data-bs-toggle="collapse" data-bs-target="#quarto-sidebar-section-4" role="navigation" aria-expanded="true" aria-label="Toggle section">\n <i class="bi bi-chevron-right ms-2"></i>\n </a> \n </div>\n <ul id="quarto-sidebar-section-4" class="collapse list-unstyled sidebar-section depth1 show"> \n <li class="sidebar-item">\n <div class="sidebar-item-container"> \n <a href="./Archive - 2024/05-Feb24.html" class="sidebar-item-text sidebar-link">\n <span class="menu-text">2024 Feb 7<sup>th</sup> – UQ PUG 5</span></a>\n </div>\n</li>\n <li class="sidebar-item">\n <div class="sidebar-item-container"> \n <a href="./Archive - 2024/06-Mar24.html" class="sidebar-item-text sidebar-link">\n <span class="menu-text">2024 Mar 7<sup>th</sup> – UQ PUG 6</span></a>\n </div>\n</li>\n <li class="sidebar-item">\n <div class="sidebar-item-container"> \n <a href="./Archive - 2024/07-Apr24.html" class="sidebar-item-text sidebar-link">\n <span class="menu-text">2024 Apr 11<sup>th</sup> – UQ PUG 7</span></a>\n </div>\n</li>\n <li class="sidebar-item">\n <div class="sidebar-item-container"> \n <a href="./Archive - 2024/08-May24.html" class="sidebar-item-text sidebar-link">\n <span class="menu-text">2024 May 2<sup>nd</sup> – UQ PUG 8</span></a>\n </div>\n</li>\n <li class="sidebar-item">\n <div class="sidebar-item-container"> \n <a href="./Archive - 2024/09-Jul24.html" class="sidebar-item-text sidebar-link">\n <span class="menu-text">2024 July 4<sup>th</sup> – UQ PUG 9</span></a>\n </div>\n</li>\n <li class="sidebar-item">\n <div class="sidebar-item-container"> \n <a href="./Archive - 2024/12-Oct24.html" class="sidebar-item-text sidebar-link">\n <span class="menu-text">2024 Oct 3<sup>rd</sup> – UQ PUG 12</span></a>\n </div>\n</li>\n <li class="sidebar-item">\n <div class="sidebar-item-container"> \n <a href="./Archive - 2024/13-Nov24.html" class="sidebar-item-text sidebar-link">\n <span class="menu-text">2024 Nov 7<sup>th</sup> – UQ PUG 13</span></a>\n </div>\n</li>\n <li class="sidebar-item">\n <div class="sidebar-item-container"> \n <a href="./Archive - 2024/14-Dec24.html" class="sidebar-item-text sidebar-link">\n <span class="menu-text">2024 Dec 5<sup>th</sup> – UQ PUG 14</span></a>\n </div>\n</li>\n </ul>\n </li>\n <li class="sidebar-item sidebar-item-section">\n <div class="sidebar-item-container"> \n <a class="sidebar-item-text sidebar-link text-start" data-bs-toggle="collapse" data-bs-target="#quarto-sidebar-section-5" role="navigation" aria-expanded="true">\n <span class="menu-text">Archive - 2023</span></a>\n <a class="sidebar-item-toggle text-start" data-bs-toggle="collapse" data-bs-target="#quarto-sidebar-section-5" role="navigation" aria-expanded="true" aria-label="Toggle section">\n <i class="bi bi-chevron-right ms-2"></i>\n </a> \n </div>\n <ul id="quarto-sidebar-section-5" class="collapse list-unstyled sidebar-section depth1 show"> \n <li class="sidebar-item">\n <div class="sidebar-item-container"> \n <a href="./Archive - 2023/1-Sep23.html" class="sidebar-item-text sidebar-link">\n <span class="menu-text">2023 Sep 7<sup>th</sup> – UQ PUG 1</span></a>\n </div>\n</li>\n <li class="sidebar-item">\n <div class="sidebar-item-container"> \n <a href="./Archive - 2023/2-Oct23.html" class="sidebar-item-text sidebar-link">\n <span class="menu-text">2023 Oct 5<sup>th</sup> – UQ PUG 2</span></a>\n </div>\n</li>\n <li class="sidebar-item">\n <div class="sidebar-item-container"> \n <a href="./Archive - 2023/3-Nov23.html" class="sidebar-item-text sidebar-link">\n <span class="menu-text">2023 Nov 2<sup>nd</sup> – UQ PUG 3</span></a>\n </div>\n</li>\n <li class="sidebar-item">\n <div class="sidebar-item-container"> \n <a href="./Archive - 2023/4-Dec23.html" class="sidebar-item-text sidebar-link">\n <span class="menu-text">2023 Dec 7<sup>th</sup> – UQ PUG 4</span></a>\n </div>\n</li>\n </ul>\n </li>\n </ul>\n </div>\n</nav>\n<div id="quarto-sidebar-glass" class="quarto-sidebar-collapse-item" data-bs-toggle="collapse" data-bs-target=".quarto-sidebar-collapse-item"></div>\n<!-- margin-sidebar -->\n <div id="quarto-margin-sidebar" class="sidebar margin-sidebar">\n <nav id="TOC" role="doc-toc" class="toc-active">\n <h2 id="toc-title">On this page</h2>\n \n <ul>\n <li><a href="#about-us" id="toc-about-us" class="nav-link active" data-scroll-target="#about-us">About us</a>\n <ul class="collapse">\n <li><a href="#who-are-we" id="toc-who-are-we" class="nav-link" data-scroll-target="#who-are-we">Who are we?</a></li>\n <li><a href="#when-do-we-meet" id="toc-when-do-we-meet" class="nav-link" data-scroll-target="#when-do-we-meet">When do we meet?</a></li>\n </ul></li>\n <li><a href="#how-it-works" id="toc-how-it-works" class="nav-link" data-scroll-target="#how-it-works">How it works</a></li>\n <li><a href="#this-website" id="toc-this-website" class="nav-link" data-scroll-target="#this-website">This website</a></li>\n <li><a href="#general-resources" id="toc-general-resources" class="nav-link" data-scroll-target="#general-resources">General Resources</a></li>\n </ul>\n</nav>\n </div>\n<!-- main -->\n<main class="content" id="quarto-document-content">\n\n<header id="title-block-header" class="quarto-title-block default">\n<div class="quarto-title">\n<h1 class="title">UQ Python User Group (UQ PUG)</h1>\n</div>\n\n\n\n<div class="quarto-title-meta">\n\n \n \n \n </div>\n \n\n\n</header>\n\n\n<p>Welcome to the UQ Python User Group (UQ PUG). We meet monthly, and our next gathering is on the 6th of March. We meet in room 12-N109 (Central Library).</p>\n<section id="about-us" class="level2">\n<h2 class="anchored" data-anchor-id="about-us">About us</h2>\n<section id="who-are-we" class="level3">\n<h3 class="anchored" data-anchor-id="who-are-we">Who are we?</h3>\n<p>UQ PUG is a <strong>gathering for Python users of any proficiency</strong>, to help each other solve problems, share resources and tips, discover new features and hang out as a friendly community.</p>\n</section>\n<section id="when-do-we-meet" class="level3">\n<h3 class="anchored" data-anchor-id="when-do-we-meet">When do we meet?</h3>\n<ul>\n<li>Every <strong>first Thursday of the month</strong> from <strong>10:00 am</strong> to 11:30am</li>\n<li>In-person at UQ Central Library 12-N109</li>\n<li>Find the next gathering on the <a href="https://web.library.uq.edu.au/library-services/training#keyword=Python%20User%20Group;campus=;weekstart=">UQ Library Training Page</a> and book in through StudentHub</li>\n<li>Contact us at <a href="mailto:training@library.uq.edu.au">training@library.uq.edu.au</a> for any questions or support</li>\n</ul>\n</section>\n</section>\n<section id="how-it-works" class="level2">\n<h2 class="anchored" data-anchor-id="how-it-works">How it works</h2>\n<p>Each month, we gather at UQ Central Library to help solve each other’s problems and discover new features. We begin with a brief demonstration before diving into group discussion.</p>\n<p>Bring along your python questions, problems, conundrums, opinions and victories to share with the group. We’ll try our best to help you out!</p>\n</section>\n<section id="this-website" class="level2">\n<h2 class="anchored" data-anchor-id="this-website">This website</h2>\n<p>On this website we contain the notebooks from our sessions (currently since Oct 24) and details from specific demonstrations.</p>\n</section>\n<section id="general-resources" class="level2">\n<h2 class="anchored" data-anchor-id="general-resources">General Resources</h2>\n<p>Here are a few links to some general resources.</p>\n<ul>\n<li><a href="https://docs.python.org/3/">Python Documentation</a></li>\n<li><a href="https://pypi.org/">PyPI</a>, the python package repository</li>\n<li><a href="https://www.anaconda.com/">Anaconda</a>, an environment manager with python applications (e.g. Spyder) available</li>\n<li><a href="https://web.library.uq.edu.au/library-services/training">UQ Training Sessions</a></li>\n<li>Library training email - <a href="mailto:training@library.uq.edu.au">training@library.uq.edu.au</a></li>\n<li><a href="https://au.linkedin.com/learning/">Online courses</a> with LinkedIn Learning (use <a href="https://web.library.uq.edu.au/library-services/training/linkedin-learning-online-courses">UQ credentials</a>)</li>\n</ul>\n<p><a href="googlec027a6a2057eb177.html">For admin</a></p>\n\n\n</section>\n\n</main> <!-- /main -->\n<script id="quarto-html-after-body" type="application/javascript">\nwindow.document.addEventListener("DOMContentLoaded", function (event) {\n const toggleBodyColorMode = (bsSheetEl) => {\n const mode = bsSheetEl.getAttribute("data-mode");\n const bodyEl = window.document.querySelector("body");\n if (mode === "dark") {\n bodyEl.classList.add("quarto-dark");\n bodyEl.classList.remove("quarto-light");\n } else {\n bodyEl.classList.add("quarto-light");\n bodyEl.classList.remove("quarto-dark");\n }\n }\n const toggleBodyColorPrimary = () => {\n const bsSheetEl = window.document.querySelector("link#quarto-bootstrap");\n if (bsSheetEl) {\n toggleBodyColorMode(bsSheetEl);\n }\n }\n toggleBodyColorPrimary(); \n const icon = "\ue9cb";\n const anchorJS = new window.AnchorJS();\n anchorJS.options = {\n placement: \'right\',\n icon: icon\n };\n anchorJS.add(\'.anchored\');\n const isCodeAnnotation = (el) => {\n for (const clz of el.classList) {\n if (clz.startsWith(\'code-annotation-\')) { \n return true;\n }\n }\n return false;\n }\n const onCopySuccess = function(e) {\n // button target\n const button = e.trigger;\n // don\'t keep focus\n button.blur();\n // flash "checked"\n button.classList.add(\'code-copy-button-checked\');\n var currentTitle = button.getAttribute("title");\n button.setAttribute("title", "Copied!");\n let tooltip;\n if (window.bootstrap) {\n button.setAttribute("data-bs-toggle", "tooltip");\n button.setAttribute("data-bs-placement", "left");\n button.setAttribute("data-bs-title", "Copied!");\n tooltip = new bootstrap.Tooltip(button, \n { trigger: "manual", \n customClass: "code-copy-button-tooltip",\n offset: [0, -8]});\n tooltip.show(); \n }\n setTimeout(function() {\n if (tooltip) {\n tooltip.hide();\n button.removeAttribute("data-bs-title");\n button.removeAttribute("data-bs-toggle");\n button.removeAttribute("data-bs-placement");\n }\n button.setAttribute("title", currentTitle);\n button.classList.remove(\'code-copy-button-checked\');\n }, 1000);\n // clear code selection\n e.clearSelection();\n }\n const getTextToCopy = function(trigger) {\n const codeEl = trigger.previousElementSibling.cloneNode(true);\n for (const childEl of codeEl.children) {\n if (isCodeAnnotation(childEl)) {\n childEl.remove();\n }\n }\n return codeEl.innerText;\n }\n const clipboard = new window.ClipboardJS(\'.code-copy-button:not([data-in-quarto-modal])\', {\n text: getTextToCopy\n });\n clipboard.on(\'success\', onCopySuccess);\n if (window.document.getElementById(\'quarto-embedded-source-code-modal\')) {\n const clipboardModal = new window.ClipboardJS(\'.code-copy-button[data-in-quarto-modal]\', {\n text: getTextToCopy,\n container: window.document.getElementById(\'quarto-embedded-source-code-modal\')\n });\n clipboardModal.on(\'success\', onCopySuccess);\n }\n var localhostRegex = new RegExp(/^(?:http|https):\\/\\/localhost\\:?[0-9]*\\//);\n var mailtoRegex = new RegExp(/^mailto:/);\n var filterRegex = new RegExp("https:\\/\\/uqpug\\.github\\.io\\/");\n var isInternal = (href) => {\n return filterRegex.test(href) || localhostRegex.test(href) || mailtoRegex.test(href);\n }\n // Inspect non-navigation links and adorn them if external\n \tvar links = window.document.querySelectorAll(\'a[href]:not(.nav-link):not(.navbar-brand):not(.toc-action):not(.sidebar-link):not(.sidebar-item-toggle):not(.pagination-link):not(.no-external):not([aria-hidden]):not(.dropdown-item):not(.quarto-navigation-tool):not(.about-link)\');\n for (var i=0; i<links.length; i++) {\n const link = links[i];\n if (!isInternal(link.href)) {\n // undo the damage that might have been done by quarto-nav.js in the case of\n // links that we want to consider external\n if (link.dataset.originalHref !== undefined) {\n link.href = link.dataset.originalHref;\n }\n }\n }\n function tippyHover(el, contentFn, onTriggerFn, onUntriggerFn) {\n const config = {\n allowHTML: true,\n maxWidth: 500,\n delay: 100,\n arrow: false,\n appendTo: function(el) {\n return el.parentElement;\n },\n interactive: true,\n interactiveBorder: 10,\n theme: \'quarto\',\n placement: \'bottom-start\',\n };\n if (contentFn) {\n config.content = contentFn;\n }\n if (onTriggerFn) {\n config.onTrigger = onTriggerFn;\n }\n if (onUntriggerFn) {\n config.onUntrigger = onUntriggerFn;\n }\n window.tippy(el, config); \n }\n const noterefs = window.document.querySelectorAll(\'a[role="doc-noteref"]\');\n for (var i=0; i<noterefs.length; i++) {\n const ref = noterefs[i];\n tippyHover(ref, function() {\n // use id or data attribute instead here\n let href = ref.getAttribute(\'data-footnote-href\') || ref.getAttribute(\'href\');\n try { href = new URL(href).hash; } catch {}\n const id = href.replace(/^#\\/?/, "");\n const note = window.document.getElementById(id);\n if (note) {\n return note.innerHTML;\n } else {\n return "";\n }\n });\n }\n const xrefs = window.document.querySelectorAll(\'a.quarto-xref\');\n const processXRef = (id, note) => {\n // Strip column container classes\n const stripColumnClz = (el) => {\n el.classList.remove("page-full", "page-columns");\n if (el.children) {\n for (const child of el.children) {\n stripColumnClz(child);\n }\n }\n }\n stripColumnClz(note)\n if (id === null || id.startsWith(\'sec-\')) {\n // Special case sections, only their first couple elements\n const container = document.createElement("div");\n if (note.children && note.children.length > 2) {\n container.appendChild(note.children[0].cloneNode(true));\n for (let i = 1; i < note.children.length; i++) {\n const child = note.children[i];\n if (child.tagName === "P" && child.innerText === "") {\n continue;\n } else {\n container.appendChild(child.cloneNode(true));\n break;\n }\n }\n if (window.Quarto?.typesetMath) {\n window.Quarto.typesetMath(container);\n }\n return container.innerHTML\n } else {\n if (window.Quarto?.typesetMath) {\n window.Quarto.typesetMath(note);\n }\n return note.innerHTML;\n }\n } else {\n // Remove any anchor links if they are present\n const anchorLink = note.querySelector(\'a.anchorjs-link\');\n if (anchorLink) {\n anchorLink.remove();\n }\n if (window.Quarto?.typesetMath) {\n window.Quarto.typesetMath(note);\n }\n if (note.classList.contains("callout")) {\n return note.outerHTML;\n } else {\n return note.innerHTML;\n }\n }\n }\n for (var i=0; i<xrefs.length; i++) {\n const xref = xrefs[i];\n tippyHover(xref, undefined, function(instance) {\n instance.disable();\n let url = xref.getAttribute(\'href\');\n let hash = undefined; \n if (url.startsWith(\'#\')) {\n hash = url;\n } else {\n try { hash = new URL(url).hash; } catch {}\n }\n if (hash) {\n const id = hash.replace(/^#\\/?/, "");\n const note = window.document.getElementById(id);\n if (note !== null) {\n try {\n const html = processXRef(id, note.cloneNode(true));\n instance.setContent(html);\n } finally {\n instance.enable();\n instance.show();\n }\n } else {\n // See if we can fetch this\n fetch(url.split(\'#\')[0])\n .then(res => res.text())\n .then(html => {\n const parser = new DOMParser();\n const htmlDoc = parser.parseFromString(html, "text/html");\n const note = htmlDoc.getElementById(id);\n if (note !== null) {\n const html = processXRef(id, note);\n instance.setContent(html);\n } \n }).finally(() => {\n instance.enable();\n instance.show();\n });\n }\n } else {\n // See if we can fetch a full url (with no hash to target)\n // This is a special case and we should probably do some content thinning / targeting\n fetch(url)\n .then(res => res.text())\n .then(html => {\n const parser = new DOMParser();\n const htmlDoc = parser.parseFromString(html, "text/html");\n const note = htmlDoc.querySelector(\'main.content\');\n if (note !== null) {\n // This should only happen for chapter cross references\n // (since there is no id in the URL)\n // remove the first header\n if (note.children.length > 0 && note.children[0].tagName === "HEADER") {\n note.children[0].remove();\n }\n const html = processXRef(null, note);\n instance.setContent(html);\n } \n }).finally(() => {\n instance.enable();\n instance.show();\n });\n }\n }, function(instance) {\n });\n }\n let selectedAnnoteEl;\n const selectorForAnnotation = ( cell, annotation) => {\n let cellAttr = \'data-code-cell="\' + cell + \'"\';\n let lineAttr = \'data-code-annotation="\' + annotation + \'"\';\n const selector = \'span[\' + cellAttr + \'][\' + lineAttr + \']\';\n return selector;\n }\n const selectCodeLines = (annoteEl) => {\n const doc = window.document;\n const targetCell = annoteEl.getAttribute("data-target-cell");\n const targetAnnotation = annoteEl.getAttribute("data-target-annotation");\n const annoteSpan = window.document.querySelector(selectorForAnnotation(targetCell, targetAnnotation));\n const lines = annoteSpan.getAttribute("data-code-lines").split(",");\n const lineIds = lines.map((line) => {\n return targetCell + "-" + line;\n })\n let top = null;\n let height = null;\n let parent = null;\n if (lineIds.length > 0) {\n //compute the position of the single el (top and bottom and make a div)\n const el = window.document.getElementById(lineIds[0]);\n top = el.offsetTop;\n height = el.offsetHeight;\n parent = el.parentElement.parentElement;\n if (lineIds.length > 1) {\n const lastEl = window.document.getElementById(lineIds[lineIds.length - 1]);\n const bottom = lastEl.offsetTop + lastEl.offsetHeight;\n height = bottom - top;\n }\n if (top !== null && height !== null && parent !== null) {\n // cook up a div (if necessary) and position it \n let div = window.document.getElementById("code-annotation-line-highlight");\n if (div === null) {\n div = window.document.createElement("div");\n div.setAttribute("id", "code-annotation-line-highlight");\n div.style.position = \'absolute\';\n parent.appendChild(div);\n }\n div.style.top = top - 2 + "px";\n div.style.height = height + 4 + "px";\n div.style.left = 0;\n let gutterDiv = window.document.getElementById("code-annotation-line-highlight-gutter");\n if (gutterDiv === null) {\n gutterDiv = window.document.createElement("div");\n gutterDiv.setAttribute("id", "code-annotation-line-highlight-gutter");\n gutterDiv.style.position = \'absolute\';\n const codeCell = window.document.getElementById(targetCell);\n const gutter = codeCell.querySelector(\'.code-annotation-gutter\');\n gutter.appendChild(gutterDiv);\n }\n gutterDiv.style.top = top - 2 + "px";\n gutterDiv.style.height = height + 4 + "px";\n }\n selectedAnnoteEl = annoteEl;\n }\n };\n const unselectCodeLines = () => {\n const elementsIds = ["code-annotation-line-highlight", "code-annotation-line-highlight-gutter"];\n elementsIds.forEach((elId) => {\n const div = window.document.getElementById(elId);\n if (div) {\n div.remove();\n }\n });\n selectedAnnoteEl = undefined;\n };\n // Handle positioning of the toggle\n window.addEventListener(\n "resize",\n throttle(() => {\n elRect = undefined;\n if (selectedAnnoteEl) {\n selectCodeLines(selectedAnnoteEl);\n }\n }, 10)\n );\n function throttle(fn, ms) {\n let throttle = false;\n let timer;\n return (...args) => {\n if(!throttle) { // first call gets through\n fn.apply(this, args);\n throttle = true;\n } else { // all the others get throttled\n if(timer) clearTimeout(timer); // cancel #2\n timer = setTimeout(() => {\n fn.apply(this, args);\n timer = throttle = false;\n }, ms);\n }\n };\n }\n // Attach click handler to the DT\n const annoteDls = window.document.querySelectorAll(\'dt[data-target-cell]\');\n for (const annoteDlNode of annoteDls) {\n annoteDlNode.addEventListener(\'click\', (event) => {\n const clickedEl = event.target;\n if (clickedEl !== selectedAnnoteEl) {\n unselectCodeLines();\n const activeEl = window.document.querySelector(\'dt[data-target-cell].code-annotation-active\');\n if (activeEl) {\n activeEl.classList.remove(\'code-annotation-active\');\n }\n selectCodeLines(clickedEl);\n clickedEl.classList.add(\'code-annotation-active\');\n } else {\n // Unselect the line\n unselectCodeLines();\n clickedEl.classList.remove(\'code-annotation-active\');\n }\n });\n }\n const findCites = (el) => {\n const parentEl = el.parentElement;\n if (parentEl) {\n const cites = parentEl.dataset.cites;\n if (cites) {\n return {\n el,\n cites: cites.split(\' \')\n };\n } else {\n return findCites(el.parentElement)\n }\n } else {\n return undefined;\n }\n };\n var bibliorefs = window.document.querySelectorAll(\'a[role="doc-biblioref"]\');\n for (var i=0; i<bibliorefs.length; i++) {\n const ref = bibliorefs[i];\n const citeInfo = findCites(ref);\n if (citeInfo) {\n tippyHover(citeInfo.el, function() {\n var popup = window.document.createElement(\'div\');\n citeInfo.cites.forEach(function(cite) {\n var citeDiv = window.document.createElement(\'div\');\n citeDiv.classList.add(\'hanging-indent\');\n citeDiv.classList.add(\'csl-entry\');\n var biblioDiv = window.document.getElementById(\'ref-\' + cite);\n if (biblioDiv) {\n citeDiv.innerHTML = biblioDiv.innerHTML;\n }\n popup.appendChild(citeDiv);\n });\n return popup.innerHTML;\n });\n }\n }\n});\n</script>\n</div> <!-- /content -->\n\n\n\n\n</body></html>'This is the HTML for the website. If you want to webscrape, the next thing to do is import the BeautifulSoup package. We’ll leave it there.
numpyThe numpy package is huge. There’s far too much to cover. But we’ll take a look two basics.
The first is arrays, a vector-like object that beats lists:
import numpy as np
np.array([1,2,3,4])array([1, 2, 3, 4])basic_list = [1,2,3,4]
basic_list * 2
# This is annoying![1, 2, 3, 4, 1, 2, 3, 4]Why does it beat lists? Watch:
print([1,2,3,4]*3)
print(np.array([1,2,3,4])*3)[1, 2, 3, 4, 1, 2, 3, 4, 1, 2, 3, 4]
[ 3 6 9 12]Arrays can be multidimensional - matrices or tensors!
import numpy as np
top_row = [1,2,3]
bottom_row = [4,5,6]
np.array([top_row, bottom_row])array([[1, 2, 3],
[4, 5, 6]])Let’s make two matrices and multiply them, with matrix multiplication:
\[\begin{pmatrix}1&2&3\\4&5&6\end{pmatrix}\begin{pmatrix}1&2\\3&4\\5&6\end{pmatrix}\]
We use the @ symbol to denote matrix multiplication (with numpy)
A = np.array([[1,2,3],[4,5,6]])
B = np.array([[1,2],[3,4],[5,6]])
A @ Barray([[22, 28],
[49, 64]])You can also create really long arrays. This is useful for graphing. Using np.arange(start, stop, interaval), and matplotlib.pyplot to plot,
\[ y = \frac{\cos(2π\times x)}{x} \]
import matplotlib.pyplot as plt
x = np.arange(0.1, 10, 0.0001)
y = np.sin(2*np.pi*x) / x
plt.plot(x,y)
cryptographyFor making secret keys that require tokens to decript!
Basically, we use a key to encrypt our token,
\[ \text{Secret} \xrightarrow{\text{Key}} \text{Token} \]
Then use the same key to decrypt it.
\[\text{Key} + \text{Token} = \text{Secret}\]
We make a key that can be used to generate tokens (i.e. encrypted messages). Then, you use the same key to decode the message. Fernet is one technique to do it:
from cryptography.fernet import Fernet
key = Fernet.generate_key()
keyb'YraYOb2WUiH67-h7s0yP6I-fIJh1Yr6V8f-KV5ahfPE='Now that we have a key, we store it into a Fernet object. Then, we turn our secret into a token:
# Store the key
f = Fernet(key)
# Encrypt our message
token = f.encrypt(b"my secrets")
tokenb'gAAAAABn03Rf_Sb6onZ48Q3haIiW_u41a62snfHj402ncke3fYypzqv5KogwL3XpUP_Sxps362YLsaW47qcThgXD2oZvwYhsdw=='The
b"...means bytes, so its bytecoded.
Finally, we use the key (stored in f) to decrypt the message from the token
f.decrypt(token)b'my secrets'