thraxil.org:

Reply to: Django Deployment with virtualenv and pip

(Whoo, long comment. If I had my own blog set up I'd post this there, as a response .. but, uh, I don't .. sorry.)

I'll admit I was both confused and skeptical of this approach when I first started using it at work, but after a couple of months of developing a Django project under this system I'm really digging it. It's tremendously convenient, easy to get started with a project and to replicate a build, and fits my brain well. I think it's conceptually somewhat similar to the buildout approach, but IMO it's far more straightforward and comprehensible, and I like its better integration with virtualenv.

I've only had a couple of cases where this system hasn't been fully sufficient for my needs. In both cases, I think the solution is best done outside this system and then integrated with it.

h2. Installing external scripts / development tools

I've written "flunctests":http://pypi.python.org/pypi/flunc for my Django project. Flunctests are twill scripts, so they're executed through a simulated browser over HTTP against a running webserver. The tests are invoked by a command-line script that's provided when you install flunc. So I've declared flunc as a "libs.txt" dependency for my project, packaged the project's tests alongside the code, and .. have a bash script that invokes the tests: source ./ve/bin/activate; flunc the_test_suite -t http://localhost:8000 -w

That's annoying .. I don't really want to have to go inside the virtualenv directly or activate it. There's also an organizational problem with my approach: technically flunc and the application's tests are downstream of the Django project I'm developing, not a part of it. When we deploy the project to a production server, flunc is just an extraneous library to install; even if we decide to run the project's flunctests against a site in production, running them remotely against the site's external URL is a much more accurate test of the deployment than connecting to the production server and running them from the deployment's environment itself.

But the standard arguments apply against a global installation of flunc.

I think that solving this is outside the scope of the system you're describing. The idea I'm considering is something like the following .. it seems a bit weird with the nested virtualenvs, but it's the most logical approach I've come up with so far:

  1. Create a separate Python package that contains the project's flunctests and declares a dependency on flunc in setup.py
  2. Put together a "development installation" script that: creates a new virtualenv; installs my flunctest package within that new virtualenv, preferably through an SVN checkout and python setup.py develop; and installs my Django project, through an SVN checkout of the top-level project directory and bootstrap.py

h2. Installing shared development libraries

The distinction between project dependencies (declared in ./requirements, installed in the virtualenv's site-packages) and project code (checked into the project's source repository directly as code and not formally installed anywhere) makes it a bit hard to use shared development libraries that you want to hack on frequently without forking. These are the sorts of things that I always prefer to package as an egg, do a source checkout of, and python setup.py develop. But this is hard to do with no "outer" virtual environment.

When I put it that way it seems like the same solution is in order -- an external virtual environment that installs development eggs and then installs my project itself. These really are more like project dependencies, though, so the separation that seems natural for development tools feels artificial and awkward in this case.

"Fassembler":http://blog.ianbicking.org/2008/06/19/my-experience-writing-a-build-system/ has this nice feature for this use case where you can specify dependencies in your requirements file as "editable", with the flag -e. For editable dependencies, fassembler does a source checkout and python setup.py develop. (Pip may have this feature as well .. fassembler/poacheggs/pip/workingenv/virtualenv all have these very similar concepts of requirements files with slightly different featuresets and syntaxes, so I have a lot of trouble remembering their overlap.)

In my experience this hasn't been quite the right place for the feature, though; I think that whether to install a dependency as "editable" is more of a personal and case-by-case decision than a formal specification to declare in the project's requirements. In terms of workflow I think that the "plonenext":http://svn.plone.org/svn/plone/plonenext/3.2/README.txt tool "gets it right.":http://dev.plone.org/plone/browser/plonenext/3.3/README.txt#L24 I think a similar command-line tool could be installed into an external virtualenv.


formatting is with Markdown syntax. Comments are not displayed until they are approved by a moderator. Moderators will not approve unless the comment contributes value to the discussion.

namerequired
emailrequired
url
remember info?