Introduction
PAZ is a hierarchical perception library in Python.
Selected examples:
PAZ is used in the following examples (links to real-time demos and training scripts):
Probabilistic 2D keypoints | 6D head-pose estimation | Object detection |
---|---|---|
Emotion classifier | 2D keypoint estimation | Mask-RCNN (in-progress) |
---|---|---|
3D keypoint discovery | Haar Cascade detector | 6D pose estimation |
---|---|---|
Implicit orientation | Attention (STNs) | |
---|---|---|
All models can be re-trained with your own data (except for Mask-RCNN, we are working on it here).
Hierarchical APIs
PAZ can be used with three diferent API levels which are there to be helpful for the user's specific application.
High-level
Easy out-of-the-box prediction. For example, for detecting objects we can call the following pipeline:
from paz.pipelines import SSD512COCO
detect = SSD512COCO()
# apply directly to an image (numpy-array)
inferences = detect(image)
There are multiple high-level functions a.k.a. pipelines
already implemented in PAZ here. Those functions are build using our mid-level API described now below.
Mid-level
While the high-level API is useful for quick applications, it might not be flexible enough for your specific purporse. Therefore, in PAZ we can build high-level functions using our a mid-level API.
Mid-level: Sequential
If your function is sequential you can construct a sequential function using SequentialProcessor
. In the example below we create a data-augmentation pipeline:
from paz.abstract import SequentialProcessor
from paz import processors as pr
augment = SequentialProcessor()
augment.add(pr.RandomContrast())
augment.add(pr.RandomBrightness())
augment.add(pr.RandomSaturation())
augment.add(pr.RandomHue())
# you can now use this now as a normal function
image = augment(image)
You can also add any function not only those found in processors
. For example we can pass a numpy function to our original data-augmentation pipeline:
augment.add(np.mean)
There are multiple functions a.k.a. Processors
already implemented in PAZ here.
Using these processors we can build more complex pipelines e.g. data augmentation for object detection: pr.AugmentDetection
Mid-level: Explicit
Non-sequential pipelines can be also build by abstracting Processor
. In the example below we build a emotion classifier from scratch using our high-level and mid-level functions.
from paz.applications import HaarCascadeFrontalFace, MiniXceptionFER
import paz.processors as pr
class EmotionDetector(pr.Processor):
def __init__(self):
super(EmotionDetector, self).__init__()
self.detect = HaarCascadeFrontalFace(draw=False)
self.crop = pr.CropBoxes2D()
self.classify = MiniXceptionFER()
self.draw = pr.DrawBoxes2D(self.classify.class_names)
def call(self, image):
boxes2D = self.detect(image)['boxes2D']
cropped_images = self.crop(image, boxes2D)
for cropped_image, box2D in zip(cropped_images, boxes2D):
box2D.class_name = self.classify(cropped_image)['class_name']
return self.draw(image, boxes2D)
detect = EmotionDetector()
# you can now apply it to an image (numpy array)
predictions = detect(image)
Processors
allow us to easily compose, compress and extract away parameters of functions. However, most processors are build using our low-level API (backend) shown next.
Low-level
Mid-level processors are mostly built from small backend functions found in: boxes, cameras, images, keypoints and quaternions.
These functions can found in paz.backend
:
from paz.backend import boxes, camera, image, keypoints, quaternion
For example, you can use them in your scripts to load or show images:
from paz.backend.image import load_image, show_image
image = load_image('my_image.png')
show_image(image)
Additional functionality
-
PAZ has built-in messages e.g.
Pose6D
for an easier data exchange with other libraries or frameworks such as ROS. -
There are custom callbacks e.g. MAP evaluation for object detectors while training
-
PAZ comes with data loaders for the multiple datasets: OpenImages, VOC, YCB-Video, FAT, FERPlus, FER2013.
-
We have an automatic batch creation and dispatching wrappers for an easy connection between you
pipelines
and tensorflow generators. Please look at the examples and the processorpr.SequenceWrapper
for more information.