Create a new python projects

When creating a new Python repository, some guidelines help keep our code standardized.

Recommended libraries

Development & testing

pytest: Primary testing framework for writing and running tests.
pytest-cov: Test coverage reporting plugin for pytest.
black: Opinionated code formatter that ensures consistent code style.
flake8: Linter for style enforcement.
pylint: Comprehensive linter for error detection and quality enforcement.
mypy: Static type checker.
isort: Utility to sort imports.

Package manager & build system

poetry: Modern packaging and dependency management.

Common utilities

typer: Library for building CLI applications.
python-dotenv: Environment variable management.
pydantic: Data validation and settings management using Python type annotations.
fastapi: Modern, high-performance web framework for building APIs.
structlog: Structured logging library.
httpx: Asynchronous and performant HTTP client.
rich: Terminal formatting library for rich text.
sqlmodel: Intuitive and robust ORM.
tqdm: Fast, extensible progress bar for CLI.

Recommended folder structure

Depending on the type of project, for example, API service, CLI application or library, the folder structure can be varied. The following structure is for a standard CLI application.

project_name/
├── .gitlab/                     # GitLab-specific configuration
│   ├── issue_templates/         # Issue templates
│   └── merge_request_templates/ # MR templates
├── .gitlab-ci.yml               # CI/CD configuration
├── project_name/                # Main package directory
│   ├── __init__.py              # Package initialization
│   ├── cli.py                   # Command-line interface entry points
│   ├── config.py                # Configuration handling
│   └── core/                    # Core functionality
│       └── __init__.py
├── tests/                       # Test directory
│   ├── __init__.py
│   ├── conftest.py              # pytest fixtures and configuration
│   └── test_*.py                # Test modules
├── docs/                        # Documentation
├── scripts/                     # Utility scripts
├── README.md                    # Project overview
├── CONTRIBUTING.md              # Contribution guidelines
├── LICENSE                      # License information
├── pyproject.toml               # Project metadata and dependencies (Poetry)

Linter configuration

We should consolidate configurations into pyproject.toml as much as possible.

`pyproject.toml`

[tool.black]
line-length = 120

[tool.isort]
profile = "black"

[tool.mypy]
python_version = 3.12
ignore_missing_imports = true

[tool.pylint.main]
jobs = 0
load-plugins = [
  # custom plugins
]

[tool.pylint.messages_control]
enable = [
  # custom plugins
]

[tool.pylint.reports]
score = "no"

`setup.cfg`

[flake8]
extend-ignore = E203,E501
extend-exclude = **/__init__.py,.venv,tests
indent-size = 4
max-line-length = 120

Example Makefile

# Excerpt from project Makefile showing common targets

# lint
.PHONY: install-lint-deps
install-lint-deps:
    @echo "Installing lint dependencies..."
    @poetry install --only lint

.PHONY: format
format: black isort

.PHONY: black
black: install-lint-deps
    @echo "Running black format..."
    @poetry run black ${CI_PROJECT_DIR}

.PHONY: isort
isort: install-lint-deps
    @echo "Running isort format..."
    @poetry run isort ${CI_PROJECT_DIR}

.PHONY: lint
lint: flake8 check-black check-isort check-pylint check-mypy

.PHONY: flake8
flake8: install-lint-deps
    @echo "Running flake8..."
    @poetry run flake8 ${CI_PROJECT_DIR}

.PHONY: check-black
check-black: install-lint-deps
    @echo "Running black check..."
    @poetry run black --check ${CI_PROJECT_DIR}

.PHONY: check-isort
check-isort: install-lint-deps
    @echo "Running isort check..."
    @poetry run isort --check-only ${CI_PROJECT_DIR}

.PHONY: check-pylint
check-pylint: install-lint-deps install-test-deps
    @echo "Running pylint check..."
    @poetry run pylint ${CI_PROJECT_DIR}

.PHONY: check-mypy
check-mypy: install-lint-deps
    @echo "Running mypy check..."
    @poetry run mypy ${CI_PROJECT_DIR}

# test
.PHONY: test
test: install-test-deps
    @echo "Running tests..."
    @poetry run pytest

.PHONY: test-coverage
test-coverage: install-test-deps
    @echo "Running tests with coverage..."
    @poetry run pytest --cov=duo_workflow_service --cov=lints --cov-report term --cov-report html

Example GitLab CI Configuration

# Excerpt from .gitlab-ci.yml showing linting and testing jobs

image: python:3.13

stages:
  - lint
  - test

variables:
  PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"
  POETRY_CACHE_DIR: "$CI_PROJECT_DIR/.cache/poetry"
  POETRY_VERSION: "2.1.2"

cache:
  key: ${CI_COMMIT_REF_SLUG}
  paths:
    - $PIP_CACHE_DIR
    - $POETRY_CACHE_DIR
    - .venv/

# Base template for Python jobs
.poetry:
  before_script:
    - pip install poetry==${POETRY_VERSION}
    - poetry config virtualenvs.in-project true
    - poetry add --dev black isort flake8 pylint mypy pytest pytest-cov

# Linting jobs
black:
  extends: .poetry
  stage: lint
  script:
    - poetry run black --check ${CI_PROJECT_DIR}

isort:
  extends: .poetry
  stage: lint
  script:
    - poetry run isort --check-only ${CI_PROJECT_DIR}

flake8:
  extends: .poetry
  stage: lint
  script:
    - poetry run flake8 ${CI_PROJECT_DIR}

pylint:
  extends: .poetry
  stage: lint
  script:
    - poetry run pylint ${CI_PROJECT_DIR}

mypy:
  extends: .poetry
  stage: lint
  script:
    - poetry run mypy ${CI_PROJECT_DIR}

# Testing jobs
test:
  extends: .poetry
  stage: test
  script:
    - poetry run pytest --cov=duo_workflow_service --cov-report=term --cov-report=xml:coverage.xml --junitxml=junit.xml
  coverage: '/TOTAL.+?(\d+\%)/'
  artifacts:
    when: always
    reports:
      junit: junit.xml
      coverage_report:
        coverage_format: cobertura
        path: coverage.xml

Adding reviewer roulette

We recommend reviewer roulette to distribute review workload across reviewers and maintainers. A pool of Python Reviewers is available for small Python projects and can be configured following these steps.

To create a pool of reviewers specific to a project:

Follow the GitLab Dangerfiles instructions to add the configuration to your project.
Implement the Danger Reviewer component in your GitLab CI pipeline to automatically trigger the roulette.