Thursday, February 7, 2013

Windows WebDAV Double Authentication 401 Requets

Speed up trick for Windows 7 WebDAV Client

One way to conveniently develop remotely with Eclipse + PyDev is to run WebDAV on the remote server and map it as a networked drive in Windows (PyDev over Eclipse Target Managmenet does not play nicely). WebDAV works rather well except it is not the fastest especially on initial project creation /project refreshing and if your share is protected with a username/password; below is how I improved the performance.

Apparently by protocol design and due to security Windows 7 client does not cache it's authentication and so on each request first an anonymous attempt is made followed by an authenticated one:

...
72.38.184.18 - - [22/Jan/2013:23:05:04 +0000] "PROPFIND /Files HTTP/1.0" 401 751
72.38.184.18 - username [22/Mar/2011:23:05:04 +0000] "PROPFIND /Files HTTP/1.0" 301 495
72.38.184.18 - - [22/Jan/2013:23:05:04 +0000] "PROPFIND /Files/ HTTP/1.0" 401 751
72.38.184.18 - username [22/Mar/2011:23:05:04 +0000] "PROPFIND /Files/ HTTP/1.0" 207 1175
72.38.184.18 - - [22/Jan/2013:23:05:07 +0000] "PROPFIND /Files HTTP/1.0" 401 751
...

After a bit of research I was able to implement a work around, allow anonymous access to the PROPFIND requests which are mainly used for directory listings and authenticated access for everything else. Refreshing a python/djagno project will take half as long for example 21876 files took 11 minutes with no authentication vs 21 with authentication

## Development HTTP Site
<VirtualHost *:80>
  ServerAdmin FOO@FOO.com
  ServerName development.FOO.com
  ServerAlias www.development.FOO.com

  # Log file location and settings; logs within project is ok as long as 'links' are made to system 'var/log/apache'
  ErrorLog /var/log/apache2/development.FOO.com-error.log
  CustomLog /var/log/apache2/development.FOO.com-vhost_combined-access.log vhost_combined

  # Canonical to always strip www - see: http://stackoverflow.com/questions/88011/make-apache-automatically-strip-off-the-www
  RewriteCond %{HTTP_HOST} ^www\.(.+)$
  RewriteRule ^(.*)$ ${SERVER_PROTOCOL}://%1/$1 [R=301,L,NC]

  # Authenticated access for the development site version - because without this Google will find you!
  # Just in case we also prevent serving of the password logins file if it is stored in a serving folder.
  Redirect /apache-logins.htdigest http://development.FOO.com
  <Location />
    DAV On
    DirectoryIndex disabled
    Options +Indexes

    AuthType Digest
    AuthName "development.FOO.com"
    # AuthDigestDomain which urls (and any under it) this applies - should match location
    AuthDigestDomain /
    AuthDigestProvider file
    AuthUserFile /srv/www/django/development.FOO.com/apache-logins.htdigest
    # uncomment the LimitExcept to receive a small boost for non caching Windows WebDAV client by allowing
    # anonymous directory listing; see http://serverfault.com/questions/250578/webdav-and-windows-7-client
    <LimitExcept PROPFIND>
     Require valid-user
    </LimitExcept>
  </Location>

  WSGIProcessGroup development.FOO.com
  # You can further limit processes, threads and set a inactivity-timer so deamon get unloaded
  WSGIDaemonProcess development.FOO.com display-name=%{GROUP}
  WSGIScriptAlias / /srv/www/django/development.FOO.com/apache-django-development.wsgi

  # Serve static / media files through apache instance and alias/map them to specific urls. to maximize security
  # `Options -Indexes` is enabled to prevent directory listing
  Options -Indexes
  Alias /robots.txt /srv/www/django/development.FOO.com/src/django-project/static/robots.txt
 #Alias /sitemap.xml /srv/www/django/development.FOO.com/src/django-project/static/sitemap.xml
  Alias /favicon.ico /srv/www/django/development.FOO.com/src/django-project/static/favicon.ico
  Alias /media /srv/www/django/development.FOO.com/src/django-project/static/
  Alias /static /srv/www/django/development.FOO.com/src/django-project/static/

</VirtualHost>

Hope this helps you out, feel free to follow me on twitter or google plus: @danielsokolowski and +1 this.

Monday, February 4, 2013

GeoDjango + Spatialite (SQLite3) Setup Issues

Because there are fresh ones every time I start a new project!

As of May 03 2013 I have opted to install virtualenv --system-site-packages and running aptitude-install python-pysqlite2 spatialite-bin gdal-bin; the site are not as self contained any more but sure beats fighting with the issues.

All my Django projects are self contained (as much as possible) with Virtualenv and use Spatialite database (SQLite3 +geo spatial stuff) installed so to enable GeoDjango functionality. This post is a list of issues I encounter related to that setup and how I resolved them; as I seem to run into a new one on every project start. And as always if you found this useful do +1, or follow me on twitter @danielsokolows

/usr/include/spatialite.h:61: note: previous declaration of ‘spatialite_init’ was here

Say what? --- this one threw me for a three hour fun time. It happened because I executed 'pip install pyspatialite' will install the 3.X vesion which is is broken and craps out with:

setup.py", line 97, in get_amalgamation 
        download_file = pattern.findall(download_page)[0] 
    IndexError: list index out of range

So then I tried an older 'pip install pyspatialite==2.6.2-spatialite.2.4.0-4' which results in:

...
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -DMODULE_NAME="spatialite.dbapi2" -DVERSION="3.0.1" -DSQLITE_ENABLE_FTS3=1 -DSQLITE_ENABLE_RTREE=1 -DSQLITE_ENABLE_COLUMN_METADATA=1 -DOMIT_FREEXL=1 -I/Library/Frameworks/GEOS.framework/unix/include/ -I/Library/Frameworks/PROJ.framework/unix/include/ -Iamalgamation -I/usr/include/python2.6 -c src/module.c -o build/temp.linux-i686-2.6/src/module.o
In file included from src/module.c:24:src/connection.h:36: error: conflicting types for ‘spatialite_init’
/usr/include/spatialite.h:61: note: previous declaration of ‘spatialite_init’ was here
error: command 'gcc' failed with exit status 1
----------------------------------------
Command /srv/www/django/development..com/virtualenv/bin/python -c "import setuptools; __file__='/srv/www/django/development..com/virtualenv/build/pyspatialite/setup.py'; execfile('/srv/www/django/development..com/virtualenv/build/pyspatialite/setup.py')" install --single-version-externally-managed --record /tmp/pip-vq4xsp-record/install-record.txt --install-headers /srv/www/django/development..com/virtualenv/include/site/python2.6 failed with error code 1
Storing complete log in /root/.pip/pip.log

Which after pulling hair out I finally realized is because pip is caching the previous partial install and does not attempt a new download. To resolve this (clearing pip cache did nothing as there was no '~/.pip/cache' to clear) install another package and uninstall it and then try again:

(virtualenv)root@klisrv03:development..com>$  pip install pysqlite
Downloading/unpacking pysqlite
  Downloading pysqlite-2.6.3.tar.gz (74Kb): 74Kb downloaded
 ...
Successfully installed pysqlite
Cleaning up...
(virtualenv)root@klisrv03:development..com>$  pip uninstall pysqlite
Uninstalling pysqlite:
 ...
Proceed (y/n)? y
  Successfully uninstalled pysqlite
(virtualenv)root@klisrv03:development..com>$  pip install pyspatialite==2.6.2-spatialite.2.4.0-4
...
Successfully installed pyspatialite
Cleaning up..

That was easy and obvious!

ImproperlyConfigured: Only versions of pysqlite 2.5+ are compatible with SpatiaLite and GeoDjango.

EASY! Just run 'pip install pysqlite' to get a newer version then what your current python stack includes...not so fast as then you get:

ImproperlyConfigured: The pysqlite library does not support C extension loading. Both SQLite and pysqlite must be configured to allow the loading of extensions to use SpatiaLite.

The solution is to try installing from source as per django docs or using something like 'pip install pyspatiallite==2.6.2-spatialite.2.4.0-4'

Still getting: ImproperlyConfigured: Only versions of pysqlite 2.5+ are compatible with SpatiaLite and GeoDjango then make sure to patch in support as per GeoDjango missing pyspatialite support or grab my github-danielsokolowski-django patched version.

error: geos_c.h: No such file or directory

   gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -DMODULE_NAME="spatialite.dbapi2" -DSQLITE_ENABLE_FTS3=1 -DSQLITE_ENABLE_RTREE=1 -DSQLITE_ENABLE_COLUMN_METADATA=1 -I/Library/Frameworks/GEOS.framework/unix/include/ -I/Library/Frameworks/PROJ.framework/unix/include/ -Iamalgamation -I/usr/include/python2.6 -c amalgamation/spatialite.c -o build/temp.linux-i686-2.6/amalgamation/spatialite.o
    amalgamation/spatialite.c:94:20: error: geos_c.h: No such file or directory

To resolve run aptitude install libgeos-dev.

error: proj_api.h: No such file or directory

    gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -DMODULE_NAME="spatialite.dbapi2" -DSQLITE_ENABLE_FTS3=1 -DSQLITE_ENABLE_RTREE=1 -DSQLITE_ENABLE_COLUMN_METADATA=1 -I/Library/Frameworks/GEOS.framework/unix/include/ -I/Library/Frameworks/PROJ.framework/unix/include/ -Iamalgamation -I/usr/include/python2.6 -c amalgamation/spatialite.c -o build/temp.linux-i686-2.6/amalgamation/spatialite.o
    amalgamation/spatialite.c:98:22: error: proj_api.h: No such file or directory

Execute aptitude install libproj-dev.

error: python.h: No such file or directory

Execute aptitude install python-dev.

src/connection.h:33:21: fatal error: sqlite3.h: No such file or directory

Execute aptitude install libsqlite3-dev.

src/connection.h:34:24: fatal error: spatialite.h: No such file or directory

Execute aptitude install libspatialite-dev.

ImportError: cannot import name SpatialReference

You are missing the required for Spatialite + Geodjango GDAL library, install it by running aptitude install gdal-bin