Python OmegaConf

ifeelfree
3 min readJan 10, 2021

--

Table of Contents

· Part 1: What is OmegaConf?
· Part 2: OmegaConf In Details
· 2.1 DictConfig
· 2.2 Interpolation
· 2.3 Merging Multiple YAMLs
· 2.4 Config Flags
· 2.5 Structured Configs
· Reference

Part 1: What is OmegaConf?

Introduction

OmegaConf is a YAML based hierarchical configuration system. It supports the following functions regarding to YAML:

  • reading/writing YAML file as a well-designed class DictConfig
  • interpolations
  • merging several YAML files
  • command line overrides

Installation

pip install omegaconf

Part 2: OmegaConf In Details

2.1 DictConfig

Creation

from omegaconf import OmegaConf
conf = OmegaConf.create()

conf is the DictConfig class object, and we can create it from

  • dictionary OmegaConf.create()
  • list OmegaConf.create()
  • YAML file OmegaConf.load()
  • YAML string OmegaConf.create()
  • command line sys.argv=['script.py', 'srv.port=82'] and OmegaConf.from_cli()
  • dataclasses OmegaConf.structured()

Correspondence

  • If the required key is not available, None will return. In the above example, conf.my_name==None . Like Dict in Python, we can use get to given missing key a value. For example, conf.get('missing', 'new_value'), then conf.missing=='new_value' Like Dict in Python, we can add a new key (a new dictionary)
  • ??? means that the value must be set before it is called.
name:
first_name:???
conf = OmegaConf.load('must_set.yaml')
print(conf)
conf.name.first_name="tompas"
print(conf)
  • OmegaConf.to_yaml is used to transform DictConfigto str.
  • Reading and writing YAML file
# reading
from omegaconf import OmegaConf
conf = OmegaConf.load('ab.yaml')
# writing
OmegaConf.save(config=conf, f="file_name_to_save")

2.2 Interpolation

config.yaml

server:
host: localhost
port: 80
client:
url: http://${server.host}:${server.port}/
server_port: ${server.port}

Code

conf = OmegaConf.load('config.yaml')
print(conf)
conf = OmegaConf.create(OmegaConf.to_yaml(conf, resolve=True))
print(conf)

Output

{'server': {'host': 'localhost', 'port': 80}, 'client': {'url': 'http://${server.host}:${server.port}/', 'server_port': '${server.port}'}}{'server': {'host': 'localhost', 'port': 80}, 'client': {'url': 'http://localhost:80/', 'server_port': 80}}

Other types of interpolation

(1) env resolvers

# env.yaml
user:
name: ${env:USER}
home: /home/${env:USER}
# codes
conf = OmegaConf.load('env.yaml')
print(conf)
import os
os.environ["USER"]="TOM"
conf = OmegaConf.create(OmegaConf.to_yaml(conf, resolve=True))
print(conf)

(2) customer resolver

conf = OmegaConf.load('custm.yaml')
print(conf)
func=lambda x:int(x)+100
OmegaConf.register_resolver('value_may_change', func)
conf = OmegaConf.load('custm.yaml')
conf = OmegaConf.create(OmegaConf.to_yaml(conf, resolve=True))
print(conf)

Both env resolver and customer resolver needs to be set before they can be used.

2.3 Merging Multiple YAMLs

Codes

conf=OmegaConf.merge(conf_1, conf_2, conf_3)

Example

2.4 Config Flags

(1) make DictConfig behave more like a regular class, and we can temporally change it

conf = OmegaConf.create({"a":[23, 45]})
conf.b=[1,2]
print(conf)
OmegaConf.set_struct(conf, True)
from omegaconf import OmegaConf, open_dict
with open_dict(conf):
conf.c=[7,8]
print(conf)

(2) read-only flag

2.5 Structured Configs

Structured Configs is an enhanced version of dataclass from Python package.

(1) Primitive types it supports

int, float, bool, str, Enums

from dataclasses import dataclass
from enum import Enum
class Height(Enum):
SHORT=0
TALL=1
@dataclass
class SimpleType:
num: int=10
val_f: float=3.14
is_awesome: bool=True
description: str="text"
h: Height=Height.TALL
conf = OmegaConf.create(
SimpleType(num=200)
)
print(conf)

The difference between dataclass from Python with DictConfig is that DictConfig will perform type checking or type conversion.

(2) Nesting

@dataclass
class Wheel:
radius:float=3
@dataclass
class CircusBike:
front: Wheel=Wheel(radius=10)
rear: Wheel=Wheel(radius=20)

conf = OmegaConf.create(CircusBike)
print(conf)

Reference

Official Website

--

--