Any Python coders/developers help me package and distribute a tool I made?

The Lounge General Discussion
1

A while back I made a Python command-line tool which takes a WAV file as input and does transient (and other kinds) of detection (using the Aubio library) and outputs a Kit for Synthstrom Deluge. It worked well if you could actually go through all the steps of installing up-to-date Python, Pip and several other modules.

Fast forward to quarantine and I decided to resurrect it (I hadn’t bothered updating it since before Synthstrom changed the Instrument/Kit XML structure). I’ve got it all working again as a script but I wanted to be able to package it and stick it on Github so that people could download and install it much easier (with all the dependencies taken care of etc.)

I got as far as creating a package structure and all the required files (with the help of the ‘cookiecutter’ tool) and have managed to create a Github repository and that side of things is kind of working out. You can clone it and check it out (link below).

However, there are still something not quite right.

First off, running setup (“python setup.py install”) throws a permission error (I’m on OSX Catalina) saying it’s not possible to install into the default Python folder. I did some reading and found a suggestion to use the --user option (so “python setup.py install --user” which went through the process of installing stuff (aubio, numpy etc. which are dependencies) but at the end there is nothing installed on my system that I can run. I’m clearly doing something wrong but I’m exhausted from all the reading and configuring.

Any help would be hugely appreciated! Link to the repository. Everything is included so you can see how I’ve got it set up to install:

1 Like
2

I’m no expert, but I’ll try to take a dive and see if there is anything I can offer. I wonder if the first issue is Catalina itself.

3

I’m not a python developer, but are you sure it didn’t install your script and all its dependencies to somewhere like $HOME/.local/ ?

The script itself would be installed as $HOME/.local/bin/delslice or something like that.

1 Like
4

The default location for packages requires admin/root privs to write to (create files in).

The --user option lets you install packages to a directory that you as a non-admin user control and can write too.

But then how does python know which directories to check for packages when it starts running?

It uses the PYTHONPATH environment variable, which is a list of directories separated by the : character.

So I think that is the reason you can’t install the packages atm (because you’re not an admin user) and the packages you did install with --user arent being found because you havent set the PYTHONPATH env variable.

Theres some more info here Using PYTHONPATH — Functional MRI methods and How do I install a Python module for use on Linux systems at SEAS?

and the details of the env var from here are 1. Command line and environment — Python 3.13.2 documentation

PYTHONPATH
Augment the default search path for module files. The format is the same as the shell’s PATH: one or more directory pathnames separated by os.pathsep (e.g. colons on Unix or semicolons on Windows). Non-existent directories are silently ignored.

In addition to normal directories, individual PYTHONPATH entries may refer to zipfiles containing pure Python modules (in either source or compiled form). Extension modules cannot be imported from zipfiles.

The default search path is installation dependent, but generally begins with prefix/lib/pythonversion (see PYTHONHOME above). It is always appended to PYTHONPATH.

An additional directory will be inserted in the search path in front of PYTHONPATH as described above under Interface options. The search path can be manipulated from within a Python program as the variable sys.path.

1 Like
5

Thanks all. I’ll do some more reading.

Something I did discover is that by default ‘python’ is still pointing at Python 2.7. If instead I use ‘python3 setup.py install’ it actually works with no errors. Apart from that it doesn’t seem to install my actual delslice module anywhere. Examining the build folder (python3 setup.py build) within my project, it appears that the setup process has install my .py file into a folder called ‘lib’ which is making me wonder if it’s treating my package as a library instead of an executable module.

6

Python developer here. Because of the whole Python 2 vs 3 debacle, Python isn’t really useable with lots of external packages unless you use it in conjunction with virtualenv or the equivalent Python3 module venv. These install your project dependencies on a local per- project basis (rather like npm with JavaScript), rather than in the traditional global manner. Install globally and you will likely suffer untold 2 vs 3 issues. With virtualenv/venv installed you then simply include your deps within the requirements.txt file and pip install -r requirements.txt. Dependencies can include git projects which are configured as pip packages, which yours seems to be (setup.py). Working with venv is much simpler than it sounds. Stack Overflow is your friend. Good luck!

1 Like
7

Ps here’s a (non music related) gist which shows my standard Python project setup

See in particular INSTALL.md, setup.sh, requirements.txt, .gitignore

INSTALL.md first line assumes you have virtualenv and Python 3.6 installed. But you could achieve the same effect with Python 3 venv module and whatever version of Python 3 u want

2 Likes
8

Thank you!

I managed to create and activate a virtual env (using python3 venv) and managed to pip3 install my project into the venv folder. The same problem still exists though: the installation process installs ‘delslice.py’ (my module) into the lib folder (VENV/lib/python3.7/site-packages/) and as such if I try to invoke the module using ‘python3 delslice.py’ I just get a file not found error. If I type the path explicitly from within the VENV folder ‘python3 lib/python3.7/site-packages/delslice.py’ it works fine so everything else seems to be installing correctly.

9

It’s installed as a module. Try python3 -m delslice which tells the interpreter to run the specified module as a script.

1 Like
10

that sounds right

11

OK I think I might have made some real progress. Would anyone mind grabbing the repository and see if you can install it? As far as I can tell all you need is to clone the git into a folder and the from the folder do ‘pip3 install .’ and then ‘python3 delslice’ to run it. There’s an test audio file included. To process it do this:

python3 delslice --input ABORT_15.WAV --output DSTEST.XML

and it should throw some stuff on screen about slicing and output the file DSTEST.XML

12

Sorry but I can’t get it to work. At least I can install the package via pip but I can’t seem to run the code.

I have a feeling your package structure is messed up, in particular with all the __main__ packages and main() functions. These are not very pythonic and I suspect are confusing pip. You would be better getting rid of anything main or __main__ and if you must maybe have a demo() package or function.

Eg for a package structure that works take a look at this -

Note in particular -

  • no __main__ or main anywhere
  • drop- dead simple setup.py

with the above in place you can then import anything you want from the packages. This may not be what u want but you can then go about restricing access once it’s working.

13

Hmmm. I removed the main file and now only have ‘init.py’ (empty) and ‘delslice.py’ in the ‘delslice’ subfolder. I’ve also removed the ‘entry_points’ entry from setup.py

I seem to still be able to install it with ‘pip3 install .’ but when i try to run it with ‘python3 delslice’ i get a error:

/usr/local/bin/python3: can’t find ‘main’ module in ‘delslice’

Seems removing main broke it but I’m slightly out of my depth (paddling hard!)

14

Why don’t u try a simple example to test package access.

Eg add the following

delslice/hello.py
def hello_world():
    print ("Hello World delslice!")

(also make sure you still have a delslice/__init__.py file)

Then assuming packaged up and pip- imported from git via requirements.txt, can u do the following -

from delslice import hello

hello.hello_world()
15

Yep, that worked but even though I can do:

python3
import etc. (as you described)

i can’t simply do ‘python3 delslice’ or ‘python3 hello’ or anything to invoke it from the command line. When I had it set up with --main–.py it worked.

I truly appreciate your help and input - thank you!

16

OK. Not sure from the above what is working / not working for you now. If the hello example I described worked then I would suggest refactoring the code to use that specific pattern. Alternatively if your existing main stuff works, even though maybe not perfect, but you’re okay with that, maybe just run with that. Good luck :slight_smile: