Skip to content

Python

Python🔗

Category Tools
Environment venv module, (Ana)Conda, Micromamba
Package managers pip, pipx
Package sources PyPI
Linting & syntax Black, pycodestyle
References Standard Library, W3C Python Reference

Installation and Versions🔗

  • (virtual) environments
    • Python module venv
    • virtualenv
    • (Ana)Conda
    • Micromamba

      micromamba is a tiny version of the mamba package manager. It is a statically linked C++ executable with a separate command line interface. It does not need a base environment and does not come with a default version of Python.

  • change system python version: Linux: default applications
  • 2to3 converter: Automated Python 2 to 3 code translation

Functions/methods🔗

Docstrings🔗

Convention for argument description:

1
2
3
4
5
def func(x):
'''
This function handles x.
:param x: the ominous x
'''

Decorators🔗

Article intro @towardsdatascience

Annotate: Argument Metadata🔗

Introduced with Python 3.9

Example

1
2
3
4
5
6
7
8
9
from typing import Annotated
def get_velocity(distance: Annotated[float,'meters'], time: Annotated[float,'seconds']) -> float:
  return distance/time
## or (possibly in another file/module)
Meters = Annotated[float,'meters']
Seconds = Annotated[float,'seconds']
MeterPerSecond = Annotated[float,'meter per second']
def get_velocity(distance: Meters, time: Seconds) -> MeterPerSecond:
  return distance/time

Files🔗

The with statement

  • simplifies exception handling by encapsulating common preparation and cleanup tasks
  • no call of close() needed
  • examples
1
2
with open("welcome.txt") as file: # Use file to refer to the file object
  data = file.read()

Data Types🔗

Dictionaries🔗

W3schools Dictionaries Reference | RealPython on Dictionaries (iteration)

Merge two dictionaries stackoverflow

  • Python >= 3.9.0: z = x | y
  • Python >= 3.5: z = {**x, **y}
  • Python <= 3.4:
1
2
3
4
def merge_two_dicts(x, y):
    z = x.copy()   # start with keys and values of x
    z.update(y)    # modifies z with keys and values of y
    return z

Pictures🔗

Simple edits with pillow library

Libraries, packages and modules🔗

Importing Modules🔗

See documentation on the import system.

Package Sources and Managers🔗

Package managers

Package sources

  • PyPI Python Package Index

awesome-python: curated list of awesome Python frameworks, libraries, software and resources Github

Why Python packaging sucks (minor rant)
  • PyPI search sucks: query for jupyter-lab doesn’t return jupyterlab (why do I have to guess packager’s use of hyphens?!)
  • not obvious how to have pip upgrade installed packages (version not pinned by me, e.g. in requirements.txt)

Technical🔗

argparse: parse command line arguments and create help page
Documentation | @golinuxcloud

Visualisation, GUI, visuals🔗

Python Enhancements, Coding, Optimisation🔗

  • functools: implements decorator cache that can significantly speed up things like recursive function calls
  • Colander: dictionary/JSON/YAML data validation
    Example

Image Processing, Data Extraction and Processing🔗

  • camelot-py: table data extraction: library Camelot
    • can be tweaked to influence extraction quality
    • can extract multiple tables
    • can read from URL
    • flavor: how PDF is parsed. Options are lattice and stream
    • see Data Science Pandas etc for examples
  • Mito: work with Pandas dataframes like (Excel) tables: select, format, formulas, filter, etc.
    1 | 2 | 3 | Basic Visualisation
    • read tables: CSV, …
    • simple analysis: describe
    • fill holes: fillna
  • Pillow: Python Imaging Library fork. Example: see rembg
  • rembg: remove image backgrounds

    Example 
    1
2
3
4
5
6
7
8
    # Remove Background of Images
# pip install rembg
# pip install pillowfrom rembg import remove as rem
from PIL import Imagedef Remove_bg(img):
    output = "removed_bg.png"
    input = Image.open(img)
    output_img = rem(input)
    output_img.save(output)Remove_bg('input.png')
  • language-tool-python: a grammar checker for Python

    This is a Python wrapper for LanguageTool. LanguageTool is open-source grammar tool, also known as the spellchecker for OpenOffice. This library allows you to make to detect grammar errors and spelling mistakes through a Python script or through a command-line interface.

    Example 
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
    # Proofread your Documents
# pip install language-tool-pythonimport language_tool_python as ltp

def Proofread():
    checker = ltp.LanguageTool('en-US')
    text = 'A quick broun fox jumpps over a a little lazy dog.'
    correction = checker.correct(text)
    print("Your Original Text:", text)
    print("Corrected Text:", correction)
    Proofread()

Web Stuff🔗

1
python -m http.server

Warning: http.server is not recommended for production. It only implements basic security checks.

  • google-search: what the name implies

    Example 
    1
2
3
4
5
6
7
    # Search on Google
# pip install google-searchfrom googlesearch.googlesearch import GoogleSearchdef Get_Google_Search(query):
    Google = GoogleSearch()
    r = Google.search(query, num_results=10)
    for data in r.results:
        print("Title: " + data.title)
        print("Content: " + data.getText())Get_Google_Search("python programming")
  • scrapy: web scraping, website information extraction, sitecrawler, data mining, monitoring, automated testing
  • public-apis: organised hundreds of free APIs that could be used for software and web development
    Examples:
    • Fun Facts: randomly generates a fun fact every time we call it
    • Colormind API: can be used to generate gorgeous colour codes that potentially can be used in our data visualisation
    • Government APIs

Hardware, DIY, IoT🔗

  • home-assistant core: Open source home automation that puts local control and privacy first. Powered by a worldwide community of tinkerers and DIY enthusiasts. Perfect to run on a #tech/hw/RaspberryPi or a local server
    Website | Github

Obscure🔗

  • real-time-voice-cloning
  • face_recognition: The world’s simplest facial recognition api for Python and the command line Github
  • Speech to text AI

    Example 
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
    # Convert Speech to Text
#pip install SpeechRecognitionimport speech_recognition as srdef SpeechToText():Ai = sr.Recognizer()
  with sr.Microphone() as source:
      listening = Ai.listen(source, phrase_time_limit = 6)
  try:
      command = Ai.recognize_google(listening).lower()
      print("You said: " + command)

  except sr.UnknownValueError:
      print("Sorry Can't understand, Try again")
      SpeechToText()

Advanced🔗

Decorators🔗

Decorator to wrap a function with a memoizing callable that saves up to the maxsize most recent calls. It can save time when an expensive or I/O bound function is periodically called with the same arguments.

  • @cache part of module functools, smaller and faster than python SL @lru_cache() with a size limit

Simple lightweight unbounded function cache. Sometimes called “memoize”.

Filetypes🔗

YAML🔗

Testing, Linters🔗

Essential Python Tools: Linters and formatters

Linters

Testing #dev/build/python

  • virtualenvwrapper - a useful set of scripts for creating and deleting virtual environments
  • pew - provides a set of commands to manage multiple virtual environments
  • tox - a generic virtualenv management and test automation command line tool, driven by a tox.ini configuration file
  • nox - a tool that automates testing in multiple Python environments, similar to tox, driven by a noxfile.py configuration file

Projects🔗

Project Managers🔗

  • Hatch
    > Hatch is a modern, extensible Python project manager. See the Why Hatch? page for more information.

Virtual Environment🔗

  1. create: python3 -m venv .venv
  2. activate: source .venv/bin/activate
  3. deactivate: deactivate
  4. Install dependencies: e.g. pip install -r requirements.txt
  5. Save them: pip freeze > requirements.txt
    Can then be installed again

Tests🔗

  1. by convention start filenames with prefix test_, followed by name of script, same for test functions
  2. can be run with pytest (pip install pytest)
  3. can be used from sub-directory if empty file __init__.py is created (used to mark directories as Python package directories

Documentation🔗

Creating a Package🔗

Cookiecutter: there are several templates out there for Python packages to help one get started with authorship, licensing, etc.

Data Science: Pandas etc🔗

Medium: # 5 Python Libraries That Will Help Automate Your Life

Python Automation Tutorial - Extracting Table from PDF
Using Camelot to extract data into Pandas dataframe from PDF with multiple tables

Jupyter: Notebook, Lab🔗

Tags: #dev/python/jupyter #dev/python/conda #dev/python/anaconda #dev/python/miniconda

References🔗

Essential Python Tools
awesome-python-applications: Free software that works great, and also happens to be open-source Python.