managing extra dependencies

[edublancas]

As our testing suite becomes more complex, we must add more dependencies for running the tests. For example, JupySQL now has a suite of integration tests requiring complex dependencies to be installed (see ploomber/jupysql#344)

We're currently managing extra dependencies in the setup.py file, which causes problems because some dependencies require C/fortran libraries. Hence, we must include pre-setup steps to ensure this gets installed properly. The same applies to installing documentation dependencies.

A better solution would be to use conda files (e.g., environment.integration.yml) for this, since installing these dependencies with conda is a lot easier.

e.g., we could run:

# setup development environment (basic dependencies for unit test)
pkgmt setup

# include integration testing dependencies
pkgmt setup --extras integration

# include documentation dependencies
pkgmt setup --extras doc

# include documentation and integration
pkgmt setup --extras doc --extras integration

With these change, setup.py will only contain dependencies required for end-users, while all development dependencies will be moved as environment.X.yml files.

We might still have optional dependencies in setup.py but those should be to enable optional features.

Note that if we move forward with this, we should document it in the contributing guide.

Note: don't forget to remove the pip install awscli in the rreadthedocs config