AiSpace¶

AiSpace provides highly configurable framework for deep learning model development, deployment and conveniently use of pre-trained models (bert, albert, opt, etc.).

Features¶

Highly configurable, we manage all hyperparameters with inheritable Configuration files.
All modules are registerable, including models, dataset, losses, optimizers, metrics, callbacks, etc.
Standardized process
Multi-GPU Training
Integrate multiple pre-trained models, including chinese
Simple and fast deployment using BentoML
Integrated Chinese benchmarks CLUE

Quickstart¶

Training¶

python -u aispace/trainer.py \
    --schedule train_and_eval \
    --config_name CONFIG_NAME \
    --config_dir CONFIG_DIR \
    [--experiment_name EXPERIMENT_NAME] \
    [--model_name MODEL_NAME] \
    [--gpus GPUS] 

Training with resumed model¶

python -u aispace/trainer.py \
    --schedule train_and_eval \
    --config_name CONFIG_NAME \
    --config_dir CONFIG_DIR \
    --model_resume_path MODEL_RESUME_PATH \
    [--experiment_name EXPERIMENT_NAME] \
    [--model_name MODEL_NAME] \
    [--gpus GPUS] 

–model_resume_path is a path to initialization model.

Average checkpoints¶

python -u aispace/trainer.py \
    --schedule avg_checkpoints \
    --config_name CONFIG_NAME \
    --config_dir CONFIG_DIR \
    --prefix_or_checkpoints PREFIX_OR_CHECKPOINGS \
    [--ckpt_weights CKPT_WEIGHTS] \
    [--experiment_name EXPERIMENT_NAME] \
    [--model_name MODEL_NAME] \
    [--gpus GPUS] 

–prefix_or_checkpoints is paths to multiple checkpoints separated by comma.

–ckpt_weights is weights same order as the prefix_or_checkpoints.

Deployment¶

Generate deployment files before deployment, you need to specify the model path (–model_resume_path) to be deployed like following.

python -u aispace/trainer.py \
    --schedule deploy \
    --config_name CONFIG_NAME \
    --config_dir CONFIG_DIR \
    --model_resume_path MODEL_RESUME_PATH \
    [--experiment_name EXPERIMENT_NAME] \
    [--model_name MODEL_NAME] \
    [--gpus GPUS] 

We use BentoML as deploy tool, so your must implement the deploy function in your model class.

Output file structure¶

The default output path is save, which may has multiple output directories under name as:

{experiment_name}_{model_name}_{dataset_name}_{random_seed}_{id}

Where id indicates the sequence number of the experiment for the same task, increasing from 0.

Take the text classification task as an example, the output file structure is similar to the following:

test_bert_for_classification_119_0
├── checkpoint                  # 1. checkpoints
│   ├── checkpoint
│   ├── ckpt_1.data-00000-of-00002
│   ├── ckpt_1.data-00001-of-00002
│   ├── ckpt_1.index
|   ...
├── deploy                      # 2. Bentoml depolyment directory
│   └── BertTextClassificationService
│       └── 20191208180211_B6FC81
├── hparams.json                # 3. Json file of all hyperparameters
├── logs                        # 4. general or tensorboard log directory
│   ├── errors.log              # error log file
│   ├── info.log                # info log file
│   ├── train                
│   │   ├── events.out.tfevents.1574839601.jshd-60-31.179552.14276.v2
│   │   ├── events.out.tfevents.1574839753.jshd-60-31.profile-empty
│   └── validation
│       └── events.out.tfevents.1574839787.jshd-60-31.179552.151385.v2
├── model_saved                 # 5. last model saved
│   ├── checkpoint
│   ├── model.data-00000-of-00002
│   ├── model.data-00001-of-00002
│   └── model.index
└── reports                     # 6. Eval reports for every output or task
    └── output_1_classlabel     # For example, text classification task
        ├── confusion_matrix.txt
        ├── per_class_stats.json
        └── stats.json

Configuration¶

We use yaml to manage various configurations, and inheritance and override can be implemented between configurations The configurations of specific task inherit base configuration directly or indirectly.

Base¶

configs/base.yml

This is the most basic configuration file, including the default configuration about training, logging, etc.

You can read this configuration carefully to understand the possibility of configurability.

Pretrain¶

configs/pretrain

This kind of configuration file adds pretrained item compared to other configurations mainly, and includes base.yml.

Specific task¶

For example:

configs/glue_zh/tnews.yml

Dataset¶

BaseDataset¶

The base class BaseDataset inherits the tfds.core.GeneratorBasedBuilder and Registry, which makes subclasses registerable.

Custom dataset¶

Take the glue_zh dataset as an example as following:

@BaseDataset.register("glue_zh")
class GlueZh(BaseDataset):
    ...

The development follows tensorflow_dataset’s specification.

Model¶

BaseModel¶

The base class BaseModel inherits the tf.keras.Model and Registry, which makes subclasses registerable. It also implements deploy method for helping generate deployment files.

Custom Models¶

Take the bert_for_classification model as an example as following:

@BaseModel.register("bert_for_classification")
class BertForSeqClassification(BaseModel):
    ...

The registered name of the model BertForSeqClassification is bert_for_classification. And the implementation of other functions follows tf.keras.Model’s specification.

Deployment¶

We use BentoML as deploy tool, so your must implement the deploy function in your model class and a bentoml service class.

Take model bert_for_classification as an example:

Custom bentoml service¶

For more detailed information, please visit BentoML.

__all__ = [
    "BertTextClassificationService"
]

import os, sys
import tensorflow as tf

sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(os.path.abspath(__file__)), "../" * 4)))

from bentoml import api, env, BentoService, artifacts
from bentoml.artifact import TensorflowSavedModelArtifact, PickleArtifact
from bentoml.handlers import JsonHandler

import numpy as np
from scipy.special import softmax

from aispace.datasets.tokenizer import BertTokenizer
from aispace.utils.hparams import Hparams

@artifacts([
        TensorflowSavedModelArtifact('model'),
        PickleArtifact('tokenizer'),
        PickleArtifact("hparams"),
    ])
@env(pip_dependencies=['tensorflow-gpu==2.0.0', 'numpy==1.16', 'scipy==1.3.1', "tensorflow-datasets==1.3.0"])
class BertTextClassificationService(BentoService):

    def preprocessing(self, text_str):
        input_ids, token_type_ids, attention_mask = self.artifacts.tokenizer.encode(text_str)
        return input_ids, token_type_ids, attention_mask

    def decode_label_idx(self, idx):
        return self.artifacts.hparams.dataset.outputs[0].labels[idx]

    @api(JsonHandler)
    def title_predict(self, parsed_json):
        input_data = {
            "input_ids": [], "token_type_ids": [], "attention_mask": []
        }
        if isinstance(parsed_json, (list, tuple)):
            pre_input_data = list(zip(*list(map(self.preprocessing, parsed_json))))
            input_data['input_ids'].extend(pre_input_data[0])
            input_data['token_type_ids'].extend(pre_input_data[1])
            input_data['attention_mask'].extend(pre_input_data[2])
        else:  # expecting type(parsed_json) == dict:
            pre_input_data = self.preprocessing(parsed_json['text'])
            input_data['input_ids'].append(pre_input_data[0])
            input_data['token_type_ids'].append(pre_input_data[1])
            input_data['attention_mask'].append(pre_input_data[2])

        input_data['input_ids'] = tf.constant(input_data['input_ids'], name="input_ids")
        input_data['token_type_ids'] = tf.constant(input_data['token_type_ids'], name="token_type_ids")
        input_data['attention_mask'] = tf.constant(input_data['attention_mask'], name="attention_mask")
        prediction = self.artifacts.model(input_data, training=False)
        prediction_normed = softmax(prediction[0].numpy(), -1)
        prediction_idx = np.argmax(prediction_normed, -1).tolist()
        prediction_confidence = np.max(prediction_normed, -1).tolist()
        ret = {
            "predictions": []
        }
        for idx, confidence in zip(prediction_idx, prediction_confidence):
            cur_label = self.decode_label_idx(idx)
            new_ret = {
                "label": cur_label,
                "confidence": confidence
            }
            ret["predictions"].append(new_ret)

        return ret

Deploy function¶

Deploy function in model class as following

def deploy(self):
    """Return path of deployment files"""
    from aispace.datasets.tokenizer import BertTokenizer
    from .bento_services import BertTextClassificationService
    tokenizer = BertTokenizer(self._hparams.dataset.tokenizer)
    bento_service = \
        BertTextClassificationService.pack(
            model=self,
            tokenizer=tokenizer,
            hparams=self._hparams,
        )
    saved_path = bento_service.save(self._hparams.get_deploy_dir())
    return saved_path

Generate deployment files¶

To generate deployment files, you need to specify the model path (–model_resume_path) to be deployed and run following script.

python -u aispace/trainer.py \
    --schedule deploy \
    --config_name CONFIG_NAME \
    --config_dir CONFIG_DIR \
    --model_resume_path MODEL_RESUME_PATH \
    [--experiment_name EXPERIMENT_NAME] \
    [--model_name MODEL_NAME] \
    [--gpus GPUS] 

Examples¶

glue_zh/tnews¶

Tnews is a task of Chinese GLUE, which is a short text classification task from ByteDance.

Run Tnews classification

python -u aispace/trainer.py \
    --experiment_name test \
    --model_name bert_for_classification \
    --schedule train_and_eval \
    --config_name tnews \
    --config_dir ./configs/glue_zh \
    --gpus 0 1 2 3  \

Specify different pretrained model, please change includes and pretrained.name in config file.

Model	Accuracy	Macro_precision	Macro_recall	Macro_f1
bert-base-chinese-huggingface	65.020	64.987	62.484	63.017
albert_base_zh	62.160	62.514	59.267	60.377
albert_base_zh_additional_36k_steps	61.760	61.723	58.534	59.273
albert_small_zh_google	62.620	63.819	58.992	59.387
albert_large_zh	61.830	61.980	59.843	60.200
albert_tiny	60.110	57.118	55.559	56.077
albert_tiny_489k	61.130	57.875	57.200	57.332
albert_tiny_zh_google	60.860	59.500	57.556	57.702
albert_xlarge_zh_177k	63.380	63.603	60.168	60.596
albert_xlarge_zh_183k	63.210	67.161	59.220	59.599
albert_base_zh_google	59.340	59.787	55.659	56.788
albert_large_zh_google	58.210	59.465	54.548	55.308
chinese_wwm	64.000	62.747	64.509	63.042
chinese_wwm_ext	65.020	65.048	62.017	62.688
chinese_roberta_wwm_ext	64.860	64.819	63.275	63.591
chinese_roberta_wwm_large_ext	65.700	62.342	61.527	61.664
ERNIE_stable-1.0.1	66.330	66.903	63.704	64.524
ERNIE_1.0_max-len-512	66.010	65.301	62.230	62.884
chinese_xlnet_base	65.110	64.377	64.862	64.169
chinese_xlnet_mid	66.000	66.377	63.874	64.708
chinese_electra_small	60.370	60.223	57.161	57.206
chinese_electra_small_ex	59.900	58.078	55.525	56.194
chinese_electra_base	60.500	60.090	58.267	58.909
chinese_electra_large	60.500	60.362	57.653	58.336
nezha-base	58.940	57.909	55.650	55.630
nezha-base-wwm	58.800	60.060	54.859	55.831

NOTE: The hyper-parameters used here have not been fine-tuned.

glue_zh/cmrc2018¶

python -u aispace/trainer.py \
    --experiment_name test \
    --model_name bert_for_qa \
    --schedule train_and_eval \
    --enable_xla False \
    --config_name cmrc2018 \
    --config_dir ./configs/glue_zh \
    --gpus 0 1 2 3 \
    > err.log 2>&1 &

Model	F1	EM
bert-base-chinese-huggingface	71.718	44.419
albert_base_zh	69.463	41.643
albert_base_zh_google	68.538	39.320
chinese_wwm	72.081	44.419
chinese_roberta_wwm_ext	71.523	44.362
ERNIE_stable-1.0.1	83.835	64.898
ERNIE_1.0_max-len-512	83.363	65.293
chinese_electra_small	72.172	46.314

glue_zh/csl¶

python -u aispace/trainer.py \
    --experiment_name test \
    --model_name bert_for_classification \
    --schedule train_and_eval \
    --config_name csl \
    --config_dir ./configs/glue_zh \
    --gpus 0 1 \
    > csl_err.log 2>&1 &

Model	Accuracy	Macro_precision	Macro_recall	Macro_f1
ERNIE_1.0_max-len-512	83.000	83.439	83.000	82.943

glue_zh/drcd¶

python -u aispace/trainer.py \
    --experiment_name test \
    --model_name bert_for_qa \
    --schedule train_and_eval \
    --enable_xla False \
    --config_name drcd \
    --config_dir ./configs/glue_zh \
    --gpus 0 1 \
    > drcd_err.log 2>&1 &

Model	F1	EM
ERNIE_1.0_max-len-512	85.657	75.433

glue_zh/afqmc¶

python -u aispace/trainer.py \
    --experiment_name test \
    --model_name bert_for_classification \
    --schedule train_and_eval \
    --config_name afqmc \
    --config_dir ./configs/glue_zh \
    --gpus 0 1 \
    > afqmc_err.log 2>&1 &

Model	Accuracy	Macro_precision	Macro_recall	Macro_f1
ERNIE_1.0_max-len-512	72.405	67.489	66.750	67.071

glue_zh/iflytek¶

python -u aispace/trainer.py \
    --experiment_name test \
    --model_name bert_for_classification \
    --schedule train_and_eval \
    --config_name iflytek \
    --config_dir ./configs/glue_zh \
    --gpus 0 1 \
    > iflytek_err.log 2>&1 &

Model	Accuracy	Macro_precision	Macro_recall	Macro_f1
ERNIE_1.0_max-len-512	58.753	30.406	32.275	28.965

glue_zh/cmnli¶

python -u aispace/trainer.py \
    --experiment_name test \
    --model_name bert_for_classification \
    --schedule train_and_eval \
    --config_name cmnli \
    --config_dir ./configs/glue_zh \
    --gpus 0 1 \
    > cmnli_err.log 2>&1 &

Model	Accuracy	Macro_precision	Macro_recall	Macro_f1
ERNIE_1.0_max-len-512	78.759	78.750	78.679	78.593

glue_zh/wsc¶

python -u aispace/trainer.py \
    --experiment_name test \
    --model_name bert_for_relation_extract \
    --schedule train_and_eval \
    --config_name wsc \
    --config_dir ./configs/glue_zh \
    --gpus 0 1 \
    > wsc_err.log 2>&1 &

Model	Accuracy	Macro_precision	Macro_recall	Macro_f1
ERNIE_1.0_max-len-512	59.615	58.507	55.969	54.117

dureader/robust¶

python -u aispace/trainer.py \
    --experiment_name test \
    --model_name bert_for_qa \
    --schedule train_and_eval \
    --enable_xla False \
    --config_name dureader_robust \
    --config_dir ./configs/qa \
    --gpus 0 1 \
    > err.log 2>&1 &

|Model|F1|EM| |—|—|—| |bert-base-chinese-huggingface|66.624|51.856| |chinese_wwm|67.007|53.434| |chinese_roberta_wwm_ext|65.521|50.274| |ERNIE_stable-1.0.1|75.268|61.675| |ERNIE_1.0_max-len-512|83.609|72.328|

dureader/yesno¶

python -u aispace/trainer.py \
    --experiment_name test \
    --model_name bert_for_classification \
    --schedule train_and_eval \
    --enable_xla False \
    --config_name dureader_yesno \
    --config_dir ./configs/qa \
    --gpus 0 1 \
    > err.log 2>&1 &

Model	Accuracy	Macro_precision	Macro_recall	Macro_f1
bert-base-chinese-huggingface	76.565	73.315	69.958	71.230
ERNIE_stable-1.0.1	85.756	82.919	81.627	82.213
ERNIE_1.0_max-len-512	86.122	83.847	80.636	81.965

AiSpace¶

Features¶

Quickstart¶

Training¶

Training with resumed model¶

Average checkpoints¶

Deployment¶

Output file structure¶

Configuration¶

Base¶

Pretrain¶

Specific task¶

Dataset¶

BaseDataset¶

Custom dataset¶

Model¶

BaseModel¶

Custom Models¶

Deployment¶

Custom bentoml service¶

Deploy function¶

Generate deployment files¶

Examples¶

glue_zh/tnews¶

glue_zh/cmrc2018¶

glue_zh/csl¶

glue_zh/drcd¶

glue_zh/afqmc¶

glue_zh/iflytek¶

glue_zh/cmnli¶

glue_zh/wsc¶

dureader/robust¶

dureader/yesno¶

Indices and tables¶