Compatibility Issue with Python 3.12.3 and Databricks SQL CLI

[jitkaran]

I wanted to bring to your attention an issue we've encountered in our project regarding the compatibility of Python 3.12.3 with Databricks SQL CLI.

Currently, we are facing challenges with installing and using Databricks SQL CLI on our DevOps build server, which is running Python 3.12.3. Despite attempting to install the latest available version, Databricks SQL CLI 0.3.3, we have been unsuccessful in getting it to work properly.

We suspect that this issue might be related to compatibility with other packages such as NumPy or wheel. Could you please confirm if this is a known compatibility issue with Python 3.12.3 and suggest the best approach to overcome it?

PS C:\Windows\system32> pip3 install databricks-sql-cli==0.3.3
Collecting databricks-sql-cli==0.3.3
Using cached databricks_sql_cli-0.3.3-py3-none-any.whl.metadata (4.6 kB)
Requirement already satisfied: Pygments<3.0.0,>=2.11.2 in c:\program files\python312\lib\site-packages (from databricks-sql-cli==0.3.3) (2.17.2)
Requirement already satisfied: cli-helpers<3.0.0,>=2.2.1 in c:\program files\python312\lib\site-packages (from databricks-sql-cli==0.3.3) (2.3.1)
Requirement already satisfied: click<9.0.0,>=8.1.2 in c:\program files\python312\lib\site-packages (from databricks-sql-cli==0.3.3) (8.1.7)
Requirement already satisfied: configobj<6.0.0,>=5.0.6 in c:\program files\python312\lib\site-packages (from databricks-sql-cli==0.3.3) (5.0.8)
Collecting databricks-sql-connector==2.5.2 (from databricks-sql-cli==0.3.3)
Using cached databricks_sql_connector-2.5.2-py3-none-any.whl.metadata (4.1 kB)
Collecting numpy==1.23.4 (from databricks-sql-cli==0.3.3)
Using cached numpy-1.23.4.tar.gz (10.7 MB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
ERROR: Exception:
Traceback (most recent call last):
File "C:\Program Files\Python312\Lib\site-packages\pip_internal\cli\base_command.py", line 180, in exc_logging_wrapper
status = run_func(*args)
^^^^^^^^^^^^^^^
File "C:\Program Files\Python312\Lib\site-packages\pip_internal\cli\req_command.py", line 245, in wrapper
return func(self, options, args)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python312\Lib\site-packages\pip_internal\commands\install.py", line 377, in run
requirement_set = resolver.resolve(
^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python312\Lib\site-packages\pip_internal\resolution\resolvelib\resolver.py", line 95, in resolve
result = self._result = resolver.resolve(
^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python312\Lib\site-packages\pip_vendor\resolvelib\resolvers.py", line 546, in resolve
state = resolution.resolve(requirements, max_rounds=max_rounds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python312\Lib\site-packages\pip_vendor\resolvelib\resolvers.py", line 427, in resolve
failure_causes = self._attempt_to_pin_criterion(name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python312\Lib\site-packages\pip_vendor\resolvelib\resolvers.py", line 239, in _attempt_to_pin_criterion
criteria = self._get_updated_criteria(candidate)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python312\Lib\site-packages\pip_vendor\resolvelib\resolvers.py", line 230, in _get_updated_criteria
self._add_to_criteria(criteria, requirement, parent=candidate)
File "C:\Program Files\Python312\Lib\site-packages\pip_vendor\resolvelib\resolvers.py", line 173, in _add_to_criteria
if not criterion.candidates:
^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python312\Lib\site-packages\pip_vendor\resolvelib\structs.py", line 156, in bool
return bool(self._sequence)
^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python312\Lib\site-packages\pip_internal\resolution\resolvelib\found_candidates.py", line 155, in bool
return any(self)
^^^^^^^^^
File "C:\Program Files\Python312\Lib\site-packages\pip_internal\resolution\resolvelib\found_candidates.py", line 143, in
return (c for c in iterator if id(c) not in self._incompatible_ids)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python312\Lib\site-packages\pip_internal\resolution\resolvelib\found_candidates.py", line 47, in _iter_built
candidate = func()
^^^^^^
File "C:\Program Files\Python312\Lib\site-packages\pip_internal\resolution\resolvelib\factory.py", line 182, in _make_candidate_from_link
base: Optional[BaseCandidate] = self._make_base_candidate_from_link(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python312\Lib\site-packages\pip_internal\resolution\resolvelib\factory.py", line 228, in _make_base_candidate_from_link
self._link_candidate_cache[link] = LinkCandidate(
^^^^^^^^^^^^^^
File "C:\Program Files\Python312\Lib\site-packages\pip_internal\resolution\resolvelib\candidates.py", line 290, in init
super().init(
File "C:\Program Files\Python312\Lib\site-packages\pip_internal\resolution\resolvelib\candidates.py", line 156, in init
self.dist = self._prepare()
^^^^^^^^^^^^^^^
File "C:\Program Files\Python312\Lib\site-packages\pip_internal\resolution\resolvelib\candidates.py", line 222, in _prepare
dist = self._prepare_distribution()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python312\Lib\site-packages\pip_internal\resolution\resolvelib\candidates.py", line 301, in _prepare_distribution
return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python312\Lib\site-packages\pip_internal\operations\prepare.py", line 525, in prepare_linked_requirement
return self._prepare_linked_requirement(req, parallel_builds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python312\Lib\site-packages\pip_internal\operations\prepare.py", line 640, in _prepare_linked_requirement
dist = _get_prepared_distribution(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python312\Lib\site-packages\pip_internal\operations\prepare.py", line 71, in _get_prepared_distribution
abstract_dist.prepare_distribution_metadata(
File "C:\Program Files\Python312\Lib\site-packages\pip_internal\distributions\sdist.py", line 54, in prepare_distribution_metadata
self._install_build_reqs(finder)
File "C:\Program Files\Python312\Lib\site-packages\pip_internal\distributions\sdist.py", line 124, in _install_build_reqs
build_reqs = self._get_build_requires_wheel()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python312\Lib\site-packages\pip_internal\distributions\sdist.py", line 101, in _get_build_requires_wheel
return backend.get_requires_for_build_wheel()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python312\Lib\site-packages\pip_internal\utils\misc.py", line 745, in get_requires_for_build_wheel
return super().get_requires_for_build_wheel(config_settings=cs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python312\Lib\site-packages\pip_vendor\pyproject_hooks_impl.py", line 166, in get_requires_for_build_wheel
return self._call_hook('get_requires_for_build_wheel', {
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python312\Lib\site-packages\pip_vendor\pyproject_hooks_impl.py", line 321, in _call_hook
raise BackendUnavailable(data.get('traceback', ''))
pip._vendor.pyproject_hooks._impl.BackendUnavailable: Traceback (most recent call last):
File "C:\Program Files\Python312\Lib\site-packages\pip_vendor\pyproject_hooks_in_process_in_process.py", line 77, in build_backend
obj = import_module(mod_path)
^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python312\Lib\importlib_init.py", line 90, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 1387, in _gcd_import
File "", line 1360, in _find_and_load
File "", line 1310, in _find_and_load_unlocked
File "", line 488, in _call_with_frames_removed
File "", line 1387, in _gcd_import
File "", line 1360, in _find_and_load
File "", line 1331, in _find_and_load_unlocked
File "", line 935, in _load_unlocked
File "", line 995, in exec_module
File "", line 488, in call_with_frames_removed
File "C:\Users***\AppData\Local\Temp\pip-build-env-1wpkad50\overlay\Lib\site-packages\setuptools_init.py", line 10, in
import distutils.core
ModuleNotFoundError: No module named 'distutils'

Additionally, we are curious to know if there are plans for a new release or update that would make Databricks SQL CLI compatible with the latest version of Python. Your expert guidance and insights on this matter would be greatly appreciated.

We are eagerly awaiting your response and appreciate your assistance in resolving this compatibility issue

[kpocius]

Duplicates #54

[rpanman-sonatype]

I'm getting the same error as this. Python 3.12 removed distuils which seems to be required to install databricks-sql-cli. To me #54 looks different because that's a lack of cmake. Just wondering if there's any work ongoing to support Python 3.12? Thanks!

[rpanman-sonatype]

I had a look at this a bit more. It seems that pyproject.toml is selecting numpy version 1.23.4 for python>=3.8

databricks-sql-cli/pyproject.toml

Line 22 in d538d24

{version = "1.23.4", python = ">=3.8"}

Numpy supports Python 3.12 from 1.26.0 hence the issue when we try to install on Python 3.12

[bennyblanc0]

Running the following command in Python 3.12.3 seemed to allow progression of the databricks-sql-cli install:

pip install setuptools

... but it now gets stuck further on at:

C:\Users\bsole>pip install databricks-sql-cli
Collecting databricks-sql-cli
Using cached databricks_sql_cli-0.3.3-py3-none-any.whl.metadata (4.6 kB)
Collecting Pygments<3.0.0,>=2.11.2 (from databricks-sql-cli)
Using cached pygments-2.19.1-py3-none-any.whl.metadata (2.5 kB)
Collecting cli-helpers<3.0.0,>=2.2.1 (from databricks-sql-cli)
Using cached cli_helpers-2.3.1-py3-none-any.whl.metadata (2.4 kB)
Requirement already satisfied: click<9.0.0,>=8.1.2 in c:\users\bsole\appdata\local\programs\python\python313\lib\site-packages (from databricks-sql-cli) (8.1.8)
Collecting configobj<6.0.0,>=5.0.6 (from databricks-sql-cli)
Using cached configobj-5.0.9-py2.py3-none-any.whl.metadata (3.2 kB)
Collecting databricks-sql-connector==2.5.2 (from databricks-sql-cli)
Using cached databricks_sql_connector-2.5.2-py3-none-any.whl.metadata (4.1 kB)
Collecting numpy==1.23.4 (from databricks-sql-cli)
Using cached numpy-1.23.4.tar.gz (10.7 MB)
Installing build dependencies ... done
Getting requirements to build wheel ... error
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [32 lines of output]
Traceback (most recent call last):
File "C:\Users\bsole\AppData\Local\Programs\Python\Python313\Lib\site-packages\pip_vendor\pyproject_hooks_in_process_in_process.py", line 353, in
main()
~~~~^^
File "C:\Users\bsole\AppData\Local\Programs\Python\Python313\Lib\site-packages\pip_vendor\pyproject_hooks_in_process_in_process.py", line 335, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
~~~~^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\bsole\AppData\Local\Programs\Python\Python313\Lib\site-packages\pip_vendor\pyproject_hooks_in_process_in_process.py", line 112, in get_requires_for_build_wheel
backend = _build_backend()
File "C:\Users\bsole\AppData\Local\Programs\Python\Python313\Lib\site-packages\pip_vendor\pyproject_hooks_in_process_in_process.py", line 77, in build_backend
obj = import_module(mod_path)
File "C:\Users\bsole\AppData\Local\Programs\Python\Python313\Lib\importlib_init.py", line 88, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 1387, in _gcd_import
File "", line 1360, in _find_and_load
File "", line 1310, in _find_and_load_unlocked
File "", line 488, in _call_with_frames_removed
File "", line 1387, in _gcd_import
File "", line 1360, in _find_and_load
File "", line 1331, in _find_and_load_unlocked
File "", line 935, in load_unlocked
File "", line 1026, in exec_module
File "", line 488, in call_with_frames_removed
File "C:\Users\bsole\AppData\Local\Temp\pip-build-env-0nmestdl\overlay\Lib\site-packages\setuptools_init.py", line 16, in
import setuptools.version
File "C:\Users\bsole\AppData\Local\Temp\pip-build-env-0nmestdl\overlay\Lib\site-packages\setuptools\version.py", line 1, in
import pkg_resources
File "C:\Users\bsole\AppData\Local\Temp\pip-build-env-0nmestdl\overlay\Lib\site-packages\pkg_resources_init.py", line 2172, in
register_finder(pkgutil.ImpImporter, find_on_path)
^^^^^^^^^^^^^^^^^^^
AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.