Faster Build with Software Mirrors

There are several ways to accelerate the building process. We will introduce how to use software mirrors (docker hub mirror, pip index mirror, conda channel mirror and so on) in envd.

Setting up docker hub pull through mirror

If docker image pulls are too slow, you can add a pull through registry mirror to the docker. It may bring additional performance improvements with a pull through registry mirror since network roundtrips to docker hub are reduced.

You can set the mirror when envd bootstrap:

bash

envd bootstrap --dockerhub-mirror https://docker.mirrors.sjtug.sjtu.edu.cn

Here are some registry mirrors:

https://docker.mirrors.sjtug.sjtu.edu.cn in China.
https://hub-mirror.c.163.com in China.

Using envdlib to set up the pip/conda/apt mirrors

You can use envdlib to set up the pip/conda/apt mirrors all in one line!

python

envdlib = include("https://github.com/tensorchord/envdlib")

envdlib.mirror(pip="tuna", conda="tuna", apt="tuna")

For more information about this function, check the API reference page.

Setting up pip index mirror

Mirroring or caching of PyPI can be used to speed up local package installation, allow offline work, handle corporate firewalls or just plain Internet flakiness.

PyPI index mirror can be set with config.pip_index(url="<index>", extra_url=<extra>):

python

config.pip_index(url="https://pypi.tuna.tsinghua.edu.cn/simple")

envd will use the index to pull python packages. Here are some pip indexes:

https://pypi.tuna.tsinghua.edu.cn/simple in China.
https://mirror.sjtu.edu.cn/pypi/web/simple in China.

Setting up apt source mirror

Apt downloads packages from one or more software repositories (sources) and installs them onto your computer.

Official Ubuntu apt source is used in envd by default, but you can change to a mirror which is close to you:

python

config.apt_source(source="""
# See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to
# newer versions of the distribution.
deb https://mirror.sjtu.edu.cn/ubuntu focal main restricted
# deb-src https://mirror.sjtu.edu.cn/ubuntu focal main restricted
deb https://mirror.sjtu.edu.cn/ubuntu focal-updates main restricted
# deb-src https://mirror.sjtu.edu.cn/ubuntu focal-updates main restricted
deb https://mirror.sjtu.edu.cn/ubuntu focal universe
# deb-src https://mirror.sjtu.edu.cn/ubuntu focal universe
deb https://mirror.sjtu.edu.cn/ubuntu focal-updates universe
# deb-src https://mirror.sjtu.edu.cn/ubuntu focal-updates universe
deb https://mirror.sjtu.edu.cn/ubuntu focal multiverse
# deb-src https://mirror.sjtu.edu.cn/ubuntu focal multiverse
deb https://mirror.sjtu.edu.cn/ubuntu focal-updates multiverse
# deb-src https://mirror.sjtu.edu.cn/ubuntu focal-updates multiverse
deb https://mirror.sjtu.edu.cn/ubuntu focal-backports main restricted universe multiverse
# deb-src https://mirror.sjtu.edu.cn/ubuntu focal-backports main restricted universe multiverse
deb http://archive.canonical.com/ubuntu focal partner
# deb-src http://archive.canonical.com/ubuntu focal partner
deb https://mirror.sjtu.edu.cn/ubuntu focal-security main restricted universe multiverse
# deb-src https://mirror.sjtu.edu.cn/ubuntu focal-security main restricted universe multiverse
""")

Setting up conda channel mirror

Mirroring or caching of conda can be used to speed up local package installation, allow offline work, handle corporate firewalls or just plain Internet flakiness.

You can set up the conda channel mirror via:

python

config.conda_channel(channel="""
channels:
  - defaults
show_channel_urls: true
default_channels:
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/msys2
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free
custom_channels:
  conda-forge: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  msys2: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  bioconda: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  menpo: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  pytorch: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  pytorch-lts: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  simpleitk: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
""")

Setting up R mirror

To be added

Faster Build with Software Mirrors #

Setting up docker hub pull through mirror #

Using envdlib to set up the pip/conda/apt mirrors #

Setting up pip index mirror #

Setting up apt source mirror #

Setting up conda channel mirror #

Setting up R mirror #

Faster Build with Software Mirrors

Setting up docker hub pull through mirror

Using envdlib to set up the pip/conda/apt mirrors

Setting up pip index mirror

Setting up apt source mirror

Setting up conda channel mirror

Setting up R mirror