Python Hydra

Part 1: Project Configuration

There are several ways to configure projects:

  • Command-line arguments
  • configure file
{
"mysql":{
"host":"localhost",
"user":"root",
"passwd":"my secret password",
"db":"write-math"
},
"other":{
"preprocessing_queue":[
"preprocessing.scale_and_center",
"preprocessing.dot_reduction",
"preprocessing.connect_lines"
],
"use_anonymous":true
}
}

Part 2: Basic Idea behind Hydra

  • composite configuration files: each configuration file is used for one aspect of the project; compose the configuration just like composing code with Hydra
  • dynamically create a hierarchical configuration by composition and override it through config files and the command line

Part 3: Hydra Components

3.1 compose function

The Compose API is useful when @hydra.main() is not applicable. For example:

  • If you want to compose multiple configuration objects (Example with Ray): write out the default setting
results = []    
for model in ["alexnet", "resnet"]:
for dataset in ["cifar10", "imagenet"]:
overrides = [f"dataset={dataset}", f"model={model}"]
run_cfg = compose(overrides=overrides)
ret = train.remote(overrides, run_cfg)
results.append(ret)
  • In parts of your application that does not have access to the command line
from omegaconf import OmegaConf 
from hydra.experimental import compose, initialize
if __name__ == "__main__":
# initialize the Hydra subsystem.
# This is needed for apps that cannot have a standard
# @hydra.main() entry point
initialize(config_path="conf")
cfg = compose("config.yaml",
overrides=["db=mysql", "db.user=${env:USER}"])
print(OmegaConf.to_yaml(cfg, resolve=True))
def test_with_initialize() -> None:
with initialize(config_path="../hydra_app/conf"):
# config is relative to a module
cfg = compose(config_name="config",
overrides=["app.user=test_user"])
with initialize(config_path="cloud_app/conf"):
cfg = compose(overrides=["+db=mysql"])
print(cfg)

3.2 Reuse configuration module

defaults:
- db@source: mysql
- db@destination: mysql

3.3 Instantiate

configuration.yaml

trainer:  
_target_: my_app.Trainer
optimizer:
_target_: my_app.Optimizer
algo: SGD
lr: 0.01
dataset:
_target_: my_app.Dataset
name: Imagenet
path: /datasets/imagenet

codes

class Optimizer:    
algo: str
lr: float
def __init__(self, algo: str, lr: float) -> None:
self.algo = algo
self.lr = lr
def __repr__(self) -> str:
return f"Optimizer(algo={self.algo},lr={self.lr})"
@hydra.main(config_name="config")
def my_app(cfg: DictConfig) -> None:
optimizer = instantiate(cfg.trainer.optimizer)
print(optimizer)

Examples

Reference

Blog

--

--

--

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

The perfect programming language

Future Trends in Cloud Computing All Point to Optimization

Netplan or Interfaces File

Deploying a go application and a redis cluster in minikube

Profits & Buybacks

Thoughts about functional programming

Preparing your download request…

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
ifeelfree

ifeelfree

More from Medium

How to decide to buy cryptocurrency using Python?

FinTech Companies Can’t Stay Away from Python Anymore

Implementing a few algorithms with python

Mastering Python Fundamental in 3 Days | Day 3 | Looping With Python