Saturday, December 14, 2013

How to enable VPN over ICMP / DNS in SoftEther VPN

Or where is the damn VPN over ICMP / DNS setting in SoftEther

Because the infrequent times I need to install a SoftEther server I seem to waste half an hour trying to locate this setting.

BUT, it's already enabled by default.

Well enjoy this tip and if this has helped do thank me by following me on twitter @danielsokolows.

Sunday, December 8, 2013

CloudAtCost Buyer Beware

March 14, 2014 Update - I have sold am selling my 'Developer 3' for life instance


See performance stats below and email me at 'cloudatcost-dev3-for-sale [at] danols.com' if interested.

Below is latest performance snapshot - it's usable but not for what I need. Make me an offer in the comment section, the first reasonable amount will be accepted and we'll use a paypal for the transaction.

oot@debian7:~# . test-linode.sh 
$ openssl speed sha1
...
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
sha1             34431.17k    90492.32k   200161.54k   303043.38k   328540.16k

$ dd if=/dev/zero of=test bs=64k count=5k oflag=dsync && rm test
5120+0 records in
5120+0 records out
335544320 bytes (336 MB) copied, 84.5891 s, 4.0 MB/s

...
Resolving ipv4.download.thinkbroadband.com (ipv4.download.thinkbroadband.com)... 80.249.99.148
Length: 536870912 (512M) [application/zip]
Saving to: `/dev/null'

...
 536,870,912 1.18M/s   in 4m 25s  

Super cheap hosting but too good to be true

I've bought their for life Development 3 package about a month or so ago and aside from the rough administration panel at that time I was very impressed at the perfromance and support. The CloudAtCost instance was beating my good old and tried Linode VPS server and tickets were being answsered within hours - please use the previous or this Linode link if you decide to sign up as I get a small kickback .

Fast forward to today and I feel I might have fallen victim to the "Too good to be true" experience. My tickets go unanswered, there was a two week unable to re-image due to maintance message, and performance has hit rock bottom as seen per the stats below

CloudAtCost Dev 3 Package
512MB download ---> file avg speed 842K/s
dd if=/dev/zero of=test bs=64k count=5k oflag=dsync ---> 3.6 MB/s
SHA1 OpenSSL 8192 Test: 322022.06k

Linode 1GB Package
512MB download ---> file avg speed 6.92MB/s
dd if=/dev/zero of=test bs=64k count=5k oflag=dsync ---> 26.7 MB/s
SHA1 OpenSSL 8192 Test: 275749.84k

But let's be fair here and assume that CloudAtCost is simply swamped with the explosion of people jumping at their for life cloud packages, and so with that in mind I reserve the final verdict if I wasted my money to say 6-12 months from now. However the early lesson here is clearly that you get what you pay for.

Testing script used

Below is very rought testing script I use when comparing new services to Linode.

#!/bin/sh

#echo 'time arc a -mx temp.arc linux'
#rm temp.arc
#time arc a -mx temp.arc linux

echo 'time openssl speed sha1'
time openssl speed sha1

echo 'time tar cf - linux | bzip2 -9 - > temp.tar.bz2'
rm temp.tar.bz2
time tar cf - linux | bzip2 -9 - > temp.tar.bz2

echo 'time dd if=/dev/zero of=test bs=64k count=5k oflag=dsync && rm test'
time dd if=/dev/zero of=test bs=64k count=5k oflag=dsync && rm test

time wget --output-document=/dev/null http://ipv4.download.thinkbroadband.com/512MB.zip
#wget ftp://ftp.kernel.org/pub/linux/kernel/v2.4/linux-2.4.0.tar.bz2
#time tar -xvf linux-2.4.0.tar.bz2

Feel free to comment or follow me on twitter @danielsokolow.

Feb 01 2014 Update

New 2.0 control panel with better usability but network connection performance has further deteriorated.

CloudAtCost Dev 3 Package
512MB download ---> file avg speed 472KB/s
dd if=/dev/zero of=test bs=64k count=5k oflag=dsync ---> 10.7 MB/s
SHA1 OpenSSL 8192 Test: 379376.98k

Starbound Dedicated Server Setup

Setting up persistent Starbound on Debian

Starbound is a Terraria survival/build like game set in space. Based on watching few videos and playing it for a bit I concluded that except for the over abundant life and loot on every planet this game appeals to me very much. And so along with my minecraft.danols.com server I have decided to run a starbound.danols.com 24/7 dedicated one for Starbound; below is how I did it.

Get a dedicated Debian linux server

If you don't have yet a linux server I recommend going with Linode Xen VPS hosting - the basic first package will suffice. I have been with them for over 5 years now and can not say enough good things about them.

Yes, there are chepaer alternatives out there but heed the warning that you get what you pay for. With the basic package I am able to run a Freeswtich VOIP server, Minecraft, email, and few websites. Their setup instructions are solid so use that to provision a Debian 7 instance.

Please use the above link or this 7d884fa5262b62b8735502da003fee34061db49b referral code as I will get a small kick back if you do sign up.

Pick a domain address for your Starbound server

If you don't have a domain provider yet I recommend 1&1 Internet. I have used GoDaddy and 1and1 side-by-side over the last few years and I lean towards 1and1 since their prices are good and you also get private registration for free.

Once you get a domain, edit 1and1 or GoDaddy DNS (Domain Resolution Servers) entries and point them at Linode's servers.

After you get the DNS setup with your domain registrat, log into your Linode account and adjust the DNS entires there to point to your server, this is only needed if you are running the server under a subdomain like myself. Below is a screen shot of my starbound.danols.com settings.

To confirm everything is setup correctly ping your domain and if there is a response you are done with this step. This means that now when players want to join your server they just type your domain address instead of your IP.

root@sh1:/usr/local# ping starbound.danols.com
PING starbound.danols.com (69.164.217.6) 56(84) bytes of data.
64 bytes from sh1.danols.com (69.164.217.6): icmp_seq=1 ttl=64 time=0.051 ms
64 bytes from sh1.danols.com (69.164.217.6): icmp_seq=2 ttl=64 time=0.065 ms
64 bytes from sh1.danols.com (69.164.217.6): icmp_seq=3 ttl=64 time=0.064 ms
--- starbound.danols.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.051/0.060/0.065/0.006 ms

Copy the Linux server binaries from your Steam's Starbound folder

Your windows Steam based installation of Starbound includes Linux execetubales to run a dedicated Starbound server. Using a SFTP WinSCP client copy <install directory>\Steam\SteamApps\common\Starbound\linux64\*.* files to /usr/local/starbound-server/linux64/ and <install directory>\Steam\SteamApps\common\Starbound\assets\*.* files to /usr/local/starbound-server/assets.

The final folder structures looks like this:

/usr/local/starbound-server/
                           /assets/  (files from asssets in steam folder)
                           /linux64/     (files from linux64 steam folder)

If you are curious why are we placing our Starbound server under '/usr/local/' folder structure feel free to read Debian FilesystemHierarchyStandard wiki.

Create a starbound-server user and the system startup script

To SSH remote into your server I use Kitty which is a putty clone with extra handy futures such as the ability to store passwords.

To create a limited starbound user which the server will run under execute adduser command as follows:

root@sh1:/usr/local/starbound-server# adduser --home /usr/local/starbound-server/ --shell /bin/false --no-create-home --ingroup daemon --disabled-password --disabled-login starbound-server
Adding user `starbound-server' ...
Adding new user `starbound-server' (1005) with group `daemon' ...
Not creating home directory `/usr/local/starbound-server/'.
Changing the user information for starbound-server
Enter the new value, or press ENTER for the default
        Full Name []: Starbound Server
        Room Number []:
        Work Phone []:
        Home Phone []:
        Other []:
Is the information correct? [Y/n]

Now to create the game server startup init system script execute nano /usr/local/starbound-server/starbound-server-init.d-script.sh and paste the following script contents into it.

#!/bin/sh
### BEGIN INIT INFO
# Provides:      starbound
# Required-Start:    $local_fs $remote_fs $network $syslog $named
# Required-Stop:    $local_fs $remote_fs $network $syslog $named
# Default-Start:    2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: starts the starbound server
# Description:      starts starbound using start-stop-daemon
### END INIT INFO

### NOTE: after linking it into init.d run `update-rc.d starbound-server.sh defaults`

### Settings
PID=/var/run/starbound-server.pid
DAEMON=/usr/local/starbound-server/linux64/starbound_server   # Server BIN
DAEMON_FOLDER=/usr/local/starbound-server/linux64/            # Folder where starbound_server file lives
DAEMON_OPTS=""                              
NAME=Starbound                                            # Name of process
DESC="Starbound server"                                   # Desc of process
USER=starbound-server                                     # User used to run the process (do not use root)


# Check if user exists
if ! id -u $USER > /dev/null 2>&1; then
        echo "The user does not exist; execute below commands to crate and try again:"
        echo "  root@sh1:~# adduser --home /usr/local/starbound-server/ --shell /bin/false --no-create-home --ingroup daemon --disabled-password --disabled-login $USER"
        echo "  ..."
        echo "  root@sh1:~# chown starbound-server:daemon /usr/local/starbound-server/ -R"
        exit 1
fi

### Check if server file exists and is executable
if [ ! -e $DAEMON ]; then echo "File $DAEMON does not exist"; exit 1; fi
if [ ! -x $DAEMON ]; then echo "Server file is not executable; run 'chmod +x $DAEMON' to fix."; exit 1; fi

set -e

. /lib/lsb/init-functions


start() {
        # start the server
        # echo "start-stop-daemon --start --make-pidfile --pidfile $PID --chuid $USER --chdir $DAEMON_FOLDER --exec $DAEMON -- $DAEMON_OPTS"
        start-stop-daemon --start --make-pidfile --pidfile $PID --chuid $USER  --chdir $DAEMON_FOLDER  --exec $DAEMON -- $DAEMON_OPTS > /dev/null &
       
}

stop() {
        # gracefully stop the server
        start-stop-daemon --stop --signal INT --pidfile $PID --chuid $USER --oknodo --exec $DAEMON
}

case "$1" in
        start)
                log_daemon_msg "Starting $DESC" "$NAME"
                start
                log_end_msg $?
                ;;

        stop)
                log_daemon_msg "Stopping $DESC" "$NAME"
                stop
                log_end_msg $?
                ;;
        restart)
                log_daemon_msg "Restarting $DESC" "$NAME"
                stop
                sleep 1
                start
                log_end_msg $?
                ;;

        status)
                status_of_proc -p $PID "$DAEMON" starbound
                ;;

        *)
                echo "Usage: $NAME {start|stop|backup|restart|status}" >&2
                exit 1
                ;;
esac

exit 0

error while loading shared libraries: libvorbisfile.so.3: cannot open shared object file: No such file or directory

You might get the above error message so let's run aptitude install libvorbisfile3 which resolved the missing library issue for me.

Final steps and test run

Ensure that all files are owned by the startbound-server user - do this by executing chown starbound-server:daemon /usr/local/starbound-server/ -R. Add the startup script to the boot process by linking it from '/etc/init.d' folder and adding it to the startup sequence as follows:

root@sh1:/usr/local/starbound-server/assets# ln -s /usr/local/starbound-server/starbound-server-init.d-script.sh /etc/init.d/starbound-server.sh
root@sh1:/usr/local/starbound-server/assets# update-rc.d starbound-server.sh defaults

Test the entire setup by starting the game server /etc/init.d/starbound-server.sh start, and logging into your new shiny server. Feel free to follow me on Twitter @danielsokolow and comment if you have any questions.

And lastly do log into my persistent server at starbound.danols.com to check it out.

Troubleshooting

Best way to troubleshoot is to start the server manually and look for any errors in the console, to start the server manually do as follows:

root@sh1:/etc/init.d# cd /usr/local/starbound-server/linux64/
root@sh1:/usr/local/starbound-server/bin# ./starbound_server
Info: Creating Star::Root with 1 assets sources and config file: './starbound.config'
Info: Loading Star::Assets from: '../assets'
Info: Loading Star::Root...
...

Warn: Slow asset 0.081 : /objects/generic/capsulemed/capsulemed.png:default
Warn: Perf: UniverseServer::createWorld millis: 6830
Warn: Perf: WorldServerThread::sync millis: 363
Warn: Perf: UniverseServer::doTriggeredStorage millis: 363
Warn: Perf: UniverseServer::run.innerloop millis: 473
Info: Shutting down world alpha:-61930447:-6722809:-4350314:8:6

Tips

Tailing the log in style

The log is very noisy, but you can mitiage some of that by using a good log coloriser like 'ccze'; to install and use execute the following:

root@sh1:~# aptitude install ccze
...
root@sh1:~# tail -f -n 50 /usr/local/starbound-server/linux64/starbound_server.log | ccze -A
...

Auto re-start the server if it dies

I've noticed the server seems to stops running after a while which - I attribute this to the beta code quality - as a work arround I created the following cron entry which just executes /etc/init.d/starbound-server start; the start-stop-deamon takes care of not spawning a new server if it is already running for us.

I like to be explicit so I anmed the below as /usr/local/starbound-server/starbound-server-cron.d-entry and linked it as /etc/cron.d/starbound-server-start-if-needed. Note that files in cron.d must not have periods in them.

# On Debian system symlink this file into your `/etc/cron.d` folder like so
#
# > ln -s /usr/local/starbound-server/starbound-server-cron.d-entry /etc/cron.d/starbound-server-start-if-needed
#
# Do ensure permissions and ownership are as follows (see http://askubuntu.com/questions/54857/can-symlinks-be-use$
#
#    -rwxr--r--  1 root daemon  215 Dec 12 15:01 starbound-server-cron.d-entry

 */15 * * * * root /etc/init.d/starbound-server start

Thursday, November 21, 2013

Eclipse ANT View/Editor Missing

Installing the Eclipse ANT plugin directly

All Eclipse distributions have ANT Java based build support (you'll find 'org.apache.ant_*' in your Eclipse install) however unless you also install the Eclipse Java development tools (JDT) you will be missing the ANT Editor view.

Rather than installing the entire JDT package we can install the ANT UI component directly using Eclipse's P2 director through the command line as follows

 $ cd 
 $ ./eclipse -application org.eclipse.equinox.p2.director -repository http://download.eclipse.org/releases/kepler -installIU org.eclipse.ant.ui
 Installing org.eclipse.ant.ui 3.5.400.v20130514-1341.
 Operation completed in 19162 ms.

Afterwards to activate the editor go to 'Window > Show View > Other > Ant' or right click 'XML' files and choose 'Open with > Other > Ant Editor'.

If you found this useful feel free to follow me on twitter twitter @danielsokolowski.

Thursday, November 14, 2013

Eclipse ZIP Plugin Install

Working with zipped staticresources in Eclipse IDE

As I work toward mastering Salesforce's APEX language one of the thing that surprised me is the convention to store static resources (js, css, images, etc.) as a zipped archive in the Salesforce.com cloud; my uneducated guess this is so to avoid the Data limit imposted.

To Edit staticresources in Eclipse IDE directly (most of my Team uses Sublime + Maven which I am evaluating and those experiences are for another blog) one could use the Eclipse Zip Editor plugin, however it does not have a handy url install and requires a manual one.

Installing Eclipse Zip Plugin

There is no friendly site install url, and selecting the local or archive install results in Eclipse throwing an error; the only way I was able to install is using Eclipse dropins folder mechanism.

  • Download Eclipse Zip Editor form their site
  • Unzip the plugin
  • In the unziped folder navigate to the subfolder plugins
  • Copy the file
  • Navigate to your Eclipse IDE installation and to the folder 'dropins' and paste the jar file
  • Restart Eclipse

Using the Eclipse Zip Plugin

After Eclipse IDE reboots it might not be obvious how to use the plugin, to do so navigate to a zip staticresource, right click, and choose the ZIP Editor from the list

Feel free to comment and follow me on twitter @danielsokolows.

Friday, October 11, 2013

Dual Strike Sidewinder Xpadder Profile

How to macro program the Dual Strike in Windows 8

By far I consider the Dual Strike Sidewinder the best Frist Person Shooter (FPS) controller device.

Sadly though this controller is no longer manufactured and although Windows 8 does include the needed drivers the bundled software that allows you to program and create macros no longer works - without this functionality the controller is nothing but a shadow of it's potential.

Xpadder to the rescue

I stumbled onto this utility after reading this post and in my testing it is the perfect solution, not only for this controller but for any that you mihgt have and not only for controller aware games! That is if you ever wanted to play Minecraft, Xenominer, or Starmine with your favorite controller well with Xpadder you can make that happen.

Yes it's not free and costs $9.99, yes it's not the best looking software, and yes the site look like it was made at the same time as my Dual Strike Sidewinder stopped selling but it does work and is easy to use.

Xpadder Dual Strike Profile

Save yourself couple of hours and download my ready made SideWinder Dual Strike USB version 1.xpaddercontroller profile file where all the buttons/functions are mapped but macros are left blank to be defined as per your needs.

If you found this useful do to follow me on twitter: @danielsokolows.

Mouse Mapping Flaws

As I continued utilziing the software I've noticed that there is huge room for improvement when it comes to mouse movement emulation. I believe there should be an ability to map anolog (joy)stick position to a view port rather than accelerated movement, see my Better Mouse Emulation post.

Saturday, September 28, 2013

Minecraft server update cron script

Automatically update crafbukkit server to latest development version

Modify as need and place or link the below script into your '/etc/cron.daily/' folder; remember not to use dots in your cron files as they will not be read - for example name it or soft link it as 'update-crafbukkit-minecraft'.

#!/bin/sh
#
# file: update-craftbukkit-minecraft.sh

/etc/init.d/minecraft-server.sh stop
wget -N http://dl.bukkit.org/latest-dev/craftbukkit.jar
# got the link by reading here: http://wiki.bukkit.org/Setting_up_a_server and guessing to change '-rb' to '-dev'
rm minecraft_server.jar
ln -s craftbukkit.jar minecraft_server.jar
/etc/init.d/minecraft-server.sh start

You can find my minecraft setup and my minecraft init.d startup script at an older post Minecraft Server on Debian.

Friday, August 2, 2013

Normalize '__unicode__' accross django models

Don't repeat your self, use monkeypatching - a better default '__unicode__'

This latest top-secret project for sure is keeping me busy and time wise leaves little to keep blogging, however it being Friday and all and because you're worth it here is a Django expert tip.

Django's default '__unicode__' representation leaves a bit to be desired to say the least. In a nut shell it returns the class name + 'object' for all instances and so requires that all your models have a '__unicode__' defined to have a useful string represenation. Well with a bit of friendly monkeys we can patch it right up as per the code below:

from django.db.models.base import Model
import logging
import hashlib
import re
from django.conf import settings

logger = logging.getLogger(__name__)


#---------------------------------------------------------------------------------------------------------------------- 
logger.info("Patching 'django.db.models.base.Model': adding get_pk_display() method that return pk like string "\
  " that is based on class name and pk of the model, ex. #NG000-341-000 or #NGNone, or #NGXYA") 

# confirm signature of code we are patching and warn if it has changed
#raise Exception(hashlib.md5(str(Model)).hexdigest())
if not '300fd49fc5099da23c0929fb1b7b48cd' == \
  hashlib.md5(str(Model)).hexdigest():
 logger.warn("Hexdigest md5 signature of 'django.db.models.base.Model' does not match expected value."
     " There might be a chance patch is broken so please review code and update as needed.")

def get_pk_display(self):
 try:
  return u'#{0}{1:09}'.format(''.join(re.findall('[A-Z]', self.__class__.__name__)), self.pk)
 except: # most likely non integer based pk or not yet saved i.e. pk == None
  return u'#{0}{1}'.format(''.join(re.findall('[A-Z]', self.__class__.__name__)), self.pk)

# do the actual patch
Model.get_pk_display = get_pk_display
del get_pk_display  # clean up namespace after patching 
#----------------------------------------------------------------------------------------------------------------------


#---------------------------------------------------------------------------------------------------------------------- 
logger.info("Patching 'django.db.models.base.Model': adding default __unicode__() method that return self.name" \
  " prepended with self.get_pk_display if settings.DEBUG = True. If self.name does not exist then only" \
  " self.get_pk_display is returned") 

# confirm signature of code we are patching and warn if it has changed
#raise Exception(hashlib.md5(str(Model)).hexdigest())
if not '300fd49fc5099da23c0929fb1b7b48cd' == \
  hashlib.md5(str(Model)).hexdigest():
 logger.warn("Hexdigest md5 signature of 'django.db.models.base.Model' does not match expected value."
     " There might be a chance patch is broken so please review code and update as needed.")
def __unicode__(self):
 try:
  return u'{0}{1}'.format(self.get_pk_display() + ': ' if settings.DEBUG else '', self.name)
 except:
  return self.get_pk_display()
Model.__unicode__ = __unicode__ 

I store the above file in the project root with an app labled '_monkeypatches' in a file named 'patch_django_db_models_base_model.py' - nomenclature and organization is an art in itself but I prefer verbose for my own sake and the team - and below is what it does in plain English:

  • Every model in your django project gets a new method 'get_pk_display' – it takes the capital letters of the class name and the instances default pk and renders a front end friendly id of object. For example UserProfile with pk 7 would be rendered as '#UP000000007'. This is a handy method in itself as for example I use it heavily on the current project that requires information is hidden until certian conditions are met.
  • Every model get's a new shiny __unicode__ that returns the objects 'self.name' (from 4+ years of Django it's uncommon for my objects not to need a 'name' field) and prepends it with the output 'get_pk_display' IF settings.DEBUG is True – if object does not have 'self.name' defined then 'get_pk_display' is just returned, ex '#C000000312 Danols Web Engineering'.

I tell you – the above is rather handy and now I am in the process of wiping out uneeded __unicode__ and cleaning up templates.

Thoughts, comments and feel free to share your nuggets of Django wisdom.

Be great – @danielsokolow

Wednesday, June 19, 2013

Django Custom Chainable QuerySets

How to reuse custom defined 'models.Manager' QuerySets

Django recommends to define custom query sets on your sublcassed models.Manager, there is one problem however that these are not chainable, that is you can only call your custom queryset once - assuming I have defined an '.active()' queryset the following will not work:

>> PaymentTerm.objects.active().active()
Traceback (most recent call last):
  File "", line 1, in 
AttributeError: 'PaymentTermManager' object has no attribute 'active'
>>>

Current solutions on the web involve creating a subclassed 'models.Manager' and a custom 'QuerySet' which do work but are to code verbose for my linking. I believe the below approach is simpler, cleaner and more succinct.

class OfferManager(models.Manager):
 
 ...
 
 STATUS_CHOICES = (
  (STATUS_DISABLED, "Disabled"),
  (STATUS_ENABLED, "Enabled"),
  (STATUS_NEGOTIATED, "Negotiated"),
  (STATUS_ARCHIVED, "Archived"),
 )
 QUERYSET_PUBLIC_KWARGS = {'status__gte': STATUS_ENABLED}
 QUERYSET_ACTIVE_KWARGS = {'status': STATUS_ENABLED}
  def get_query_set(self):
   """ Assing a modified QuerySet to include our chainable QuerySets """
   class OfferManagerQuerySet(QuerySet): # clone QuerySet
                pass
  # explictly copy our QuerySet methods
  OfferManagerQuerySet.public = self.public
  OfferManagerQuerySet.active = self.active

  ...
 
  return QuerySetDynamicallyCreated(self.model, using=self._db)
 def public(self):
  """ Returns all entries accessible through front end site"""
  return self.all().filter(**OfferManager.QUERYSET_PUBLIC_KWARGS)
 public.chainable = True
 def active(self):
  """ returns offers that are open to negotiation """
  return self.public().filter(**OfferManager.QUERYSET_ACTIVE_KWARGS)

 ...

Django patch for model.Manager chainable QuerySets

Please take a look at Django Ticket #20625 with a github Django branch for new proposed method for defining chainable filters based on this idea which works through a 'chainable == True' attribute as shown below:

class OfferManager(models.Manager):

    ...
    
    def public(self):
        """ Returns all entries accessible through front end site"""
        return self.all().filter(...)
    public.chainable = True     # instructs to dynamically tranplat this method onto
                                # returned QuerySet as .public(...) 
                                # effectively providing chainable custom QuerySets

    def active(self):
        """ Returns offers that are open to negotiation """
        return self.public().filter(**OfferManager.QUERYSET_ACTIVE_KWARGS)
                                    # an example of how to reffer to OfferManager
                                    # constants as 'self' context changes
    active.chainable = True
    ...

As always feel free to comment or follow me on twitter @danielsokolowski

Thursday, June 6, 2013

CommandError: Unable to serialize database: Error encountered checking Geometry returned from GEOS C function "GEOSWKBReader_read_r".

ERROR django.contrib.gis libgeos.py@80: GEOS_ERROR: ParseException: Unknown WKB type 0

These cryptic messages are caused by using the vanilla Django `model.Manager` with your geo enabled models. This caused me a frew hours of grief trying to figure it out, thinking it's the import data, or an issue with the third party app, etc. but the fix is to just extend your custom model manager from the `gis.models.GeoManager` which is super easy to forget:


from django.contrib.gis.db import models

class MyCustomManager(models.GeoManager):
    """
    Additional methods / constants to Base's objects manager - using a GeoManager is fine even for plain models:
    
    ``BaseManager.objects.public()`` - all instances that are asccessible through front end
    """
    # Model (db table) wide constants - we put these and not in model definition to avoid circular imports.
    # one can access these constants through .objects.STATUS_DISABLED or ImageManager.STATUS_DISABLED
    STATUS_DISABLED = 0
    STATUS_ENABLED = 100
    STATUS_ARCHIVED = 500
    STATUS_CHOICES = (
        (STATUS_DISABLED, "Disabled"),
        (STATUS_ENABLED, "Enabled"),
        (STATUS_ARCHIVED, "Archived"),
    )
    # We keep status field and custom queries naming a little different as it is not one-to-one mapping in all situations
    QUERYSET_PUBLIC_KWARGS = {'status__gte': STATUS_ENABLED} # Because you can't yet chain custom manager filters ex. 
                                                             #'public().open()' we provide access this way and we can use directly in 'ForeignKey.limit_choices_to'
                                                             # workaround - http://stackoverflow.com/questions/2163151/custom-queryset-and-manager-without-breaking-dry  
    def public(self):
        """ Returns all entries someway accessible through front end site"""
        return self.filter(**self.QUERYSET_PUBLIC_KWARGS)      # def active(self):

If this post hepls you do say thank you by commeting or following me on twitter. .

Wednesday, June 5, 2013

Django import fixtures IntegrityError: Problem installing fixtures:

IntegrityError: Problem installing fixtures: The row in table 'geonames_alternatename' with primary key '1830' has an invalid foreign key: geonames_alternatename.locality_id contains a value '109436' that does not have a corresponding value in geonames_locality.geonameid.

In a situation where a third party app data export works just fine but the fixutre fails due to any IntegrityError is most likley cause by an overiden `objects` manager that returns only a subset. The solution is to explicitly tell django to export using the default model managers using the `--all` argument for example like so:

dumpdata geonames --indent=4 --all --settings=settings_development > geonames_ext/fixtures/initial_data.json

This is clearly a situation where explicit is better then implicit, that is when defining custom model managers do not touch the `get_query_set` method rather instead add an explict method; for post completeness below is my Eclipse PyDev template I utlize when defining new models.

from django.db import models
#from django.contrib.db import models
from django.core.urlresolvers import reverse
from django.core.urlresolvers import NoReverseMatch
from djanfrom django.utils.encoding import iri_to_uri
from django.utils.http import urlquote
from django.core.exceptions import ValidationError
from django.core.validators import MinValueValidator, RegexValidator

class ${model_name}Manager(models.Manager):
    """
    Additional methods / constants to ${model_name}'s objects manager:
    
    ``${model_name}Manager.objects.public()`` - all instances that are asccessible through front end
    """
 ### Model (db table) wide constants - we put these and not in model definition to avoid circular imports.
    ### One can access these constants through .objects.STATUS_DISABLED or ImageManager.STATUS_DISABLED
    STATUS_DISABLED = 0
    STATUS_ENABLED = 100
    STATUS_ARCHIVED = 500
    STATUS_CHOICES = (
        (STATUS_DISABLED, "Disabled"),
        (STATUS_ENABLED, "Enabled"),
        (STATUS_ARCHIVED, "Archived"),
    )
 ### We keep status field and custom queries naming a little different as it is not one-to-one mapping in all situations
 ### Note: workarrund - http://stackoverflow.com/questions/2163151/custom-queryset-and-manager-without-breaking-dry
    QUERYSET_PUBLIC_KWARGS = {'status__gte': STATUS_ENABLED} # because you can't yet chain custom manager filters ex.: 
                                                             # 'public().open()' we provide access the QUERYSET_KWARGS
    def public(self):
        """ Returns all entries someway accessible through front end site"""
        return self.filter(**self.QUERYSET_PUBLIC_KWARGS)   # def active(self):
 # """ Returns all active entries, for example entries that can be selected in forms, searched on, etc""
 #def active(self)
 # return self.filter(status=self.STATUS_ENABLED) 

class ${model_name}(models.Model):
    """
    Main entity representing ${model_name} object
    """
    ### model options - "anything that's not a field"
    class Meta:
        ordering = ['order', 'name']
        get_latest_by = 'order'
        #order_with_respect_to = 
        #permissions = [["can_deliver_pizzas", "Can deliver pizzas"]]
        #unique_together = [["driver", "restaurant"]]
        #verbose_name = "pizza"
        #verbose_name_plural = "stories"
    
    ### Python class methods
    def __unicode__(self):
  """ Retruns a unicode representation for the instance of this model """
  if settings.DEBUG:
   return u'PK%s: %s - %s' % (self.pk, self.name, self.FOO)
  return u'%s - %s' % (self.name,  self.FOO)
    
    ### Django established method
    #def get_absolute_url(self):
    #    """ Returns the relative url mapping for the instance of this model if it exists or None otherwise"""
    #    return iri_to_uri(reverse('Company', kwargs={'pk': self.pk, 'slug': urlquote(slugify(self.__uniocode__()))}))
    #    return reverse ('${model_name}', kwargs={'slug': self.slug})
    def save(self, *args, **kwargs):
        if self.project: # enforce that company matches project's company
            self.company = self.project.company
        super(Login, self).save(*args, **kwargs) # Call the "real" save() method.        

    def clean(self):
        """ Check if icon is needed or not """
        if self.project and (self.company != self.project.company):
            raise ValidationError('Project must belong to the company specified.')
        
    ### extra model functions
    def get_next_order():
        """ Returns the next available integer for the order field """
        try:
            return ${model_name}.objects.latest('order').last + 1
        except: 
            return 1
            
    def get_upload_path(instance, filename):
        """ returns a dynamic path for filefields/imagefieds """
        return '%s/%s/%s' % (instance._meta.app_label, instance._meta.module_name, filename)
    
    def get_geopoint_srid4326(self):
        """ 
        Returns a geopoint in spatial projection 4326 which is lat and lng coordinate system. 
        
        We are doing this so we can utilise lat lng in the template as sqlite dosen't support
        all distance calculations in 4326 so we used srid 900913 which is not lat lng based.
        """
        pnt = GEOSGeometry(self.geopoint.wkt) # we have to make copy of point as transform 
        pnt.set_srid(self.geopoint.get_srid())                                      # works in place and dosen't return a new point
        return pnt.transform(4326)
    ### custom managers
    objects = ${model_name}Manager()
    #objects = models.GeoManager() # geodjango objects manager
    
    ### model DB fields
    # add  blank=False, default='' if you don't want to display a 'blank choice' when rendering modelchoicefield
    status = models.IntegerField(blank=False, default=None, choices=${model_name}Manager.STATUS_CHOICES, default=${model_name}Manager.STATUS_ENABLED)
    order = models.PositiveIntegerField(help_text='Display order modifier, leave default if unsure', unique=False, default=get_next_order, validators=[MinValueValidator(1)])
    name = models.CharField(help_text='Name', max_length=20)
    # rather than database slug use PK and a dynamic slug from name, links will still work using ID lookup
 #slug = models.SlugField(help_text='A short label used in URL generation containing only letters, numbers, underscores or hyphens; hyphens being preferred for SEO optimizations',
 #           unique=True, null=True, validators=[RegexValidator(r'.+')])
    description = models.TextField(help_text='Optional HTML allowed description', blank=True)
    # content = models.TextField(help_text='Optional content', blank=True) # this is not needed keep it simple
    phone = models.CharField(help_text='Main phone number ex. 613-333-4444.', max_length=12, 
                validators=RegexValidator(regex='\d\d\d-\d\d\d-\d\d\d\d', 
                           message="Please enter phone number in following format: 613-333-4444", 
                           code="Unrecognized Phone"))
 services = models.ManyToManyField(LawyerService, limit_choices_to= {'status', LawyerService.objects.live()}, 
                    help_text='Available Family law services.')
    ### GeoDjango-specific fields
    #geopoint = models.PointField()

As always, if this has hepled you say thank you by commeting or following me on twitter.

Friday, May 3, 2013

Unable to configure handler 'console': __init__() got an unexpected keyword argument 'strm'

Django and upgraded from 2.6 to Python 2.7

Hopefuly this quick post will save someone time, I recently upgraded our Debian servers from stable (squeeze) to testing (wheezy).

Unfortunately this resulted in an odd error:

Traceback (most recent call last):
  File "manage.py", line 12, in 
    execute_from_command_line(sys.argv)
  File "/srv/www/django/development.FOO.com/virtualenv/src/github-django-django-stable/django/core/management/__init__.py", line 453, in execute_from_command_line
    utility.execute()
  File "/srv/www/django/development.FOO.com/virtualenv/src/github-django-django-stable/django/core/management/__init__.py", line 392, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/srv/www/django/development.FOO.com/virtualenv/src/github-django-django-stable/django/core/management/__init__.py", line 263, in fetch_command
    app_name = get_commands()[subcommand]
  File "/srv/www/django/development.FOO.com/virtualenv/src/github-django-django-stable/django/core/management/__init__.py", line 109, in get_commands
    apps = settings.INSTALLED_APPS
  File "/srv/www/django/development.FOO.com/virtualenv/src/github-django-django-stable/django/conf/__init__.py", line 53, in __getattr__
    self._setup(name)
  File "/srv/www/django/development.FOO.com/virtualenv/src/github-django-django-stable/django/conf/__init__.py", line 49, in _setup
    self._configure_logging()
  File "/srv/www/django/development.FOO.com/virtualenv/src/github-django-django-stable/django/conf/__init__.py", line 84, in _configure_logging
    logging_config_func(self.LOGGING)
  File "/usr/lib/python2.7/logging/config.py", line 777, in dictConfig
    dictConfigClass(config).configure()
  File "/usr/lib/python2.7/logging/config.py", line 575, in configure
    '%r: %s' % (name, e))
ValueError: Unable to configure handler 'console': __init__() got an unexpected keyword argument 'strm'
(virtualenv)root@klisrv10:/srv/www/django/development.FOO.com/src/django-project#

The solution is relatively simple but not easily foundable in Google or Bing. The logging definition in settings.py needs to be updated to use the new Python 2.7 'stream' keyword argument instead of 'strm'. My updated default project logging settings below:

LOGGING = {
 'version': 1,
 'disable_existing_loggers': True,
 'filters': {
        'require_debug_false': {
            '()': 'django.utils.log.RequireDebugFalse'
        }
    },
 'formatters': {
  'verbose': {
   'format': '%(levelname)s %(name)s %(asctime)s %(module)s %(process)d %(thread)d %(pathname)s@%(lineno)s: %(message)s'
  },
  'simple': {
   'format': '%(levelname)s %(name)s %(filename)s@%(lineno)s: %(message)s'
  },
 },
  # let the 'handlers' get all messages and filter level in 'loggers'
 'handlers': {
  'null': {
   #'level':'DEBUG',
   'class':'django.utils.log.NullHandler',
  },
  'console':{
   #'level':'WARNING',
   'class':'logging.StreamHandler',
   'formatter': 'simple',
   'stream': sys.stderr,
   # 'stream': sys.stdout
   # see https://code.google.com/p/modwsgi/wiki/ApplicationIssues#Writing_To_Standard_Output
   
  },
  'mail_admins': {
   #'level': 'ERROR',
   'filters': ['require_debug_false'],
   'class': 'django.utils.log.AdminEmailHandler',
   'formatter': 'verbose'
  }
 },
 'loggers': {
  # catch all logger ex. any logger = logging.getLogger(__name__)
  '': { 
   'handlers': ['mail_admins', 'console'],
   'level': 'WARNING',
  },
 }
}

As always if you found this useful feel free to follow me here or via twitter @danielsokolowski.

Friday, April 19, 2013

mppe_compress[0]: osize too small! (have: 1404 need: 1408)

Windows VPN and PPTP Connectivity Timeouts

In my pursuit for a personal cloud and convenient secure remote file system so as to be able to develop directly on the server I have researched, tested and decided against SFTP, WebDAV, and SAMBA/NFS over internet. The only approach still standing is the built in Windows VPN and PPTP + SAMBA/CIFS:


However there was one bug that prevented some sites/services from working and timing out; the /var/log/syslog showed the following intermittently:

Apr 19 10:15:15 sh1 pptpd[11330]: GRE: accepting packet #107
Apr 19 10:15:15 sh1 pptpd[11330]: GRE: accepting packet #108
Apr 19 10:15:15 sh1 pptpd[11330]: GRE: accepting packet #109
Apr 19 10:15:15 sh1 kernel: mppe_compress[0]: osize too small! (have: 1404 need: 1408)
Apr 19 10:15:15 sh1 kernel: ppp0: ppp: compressor dropped pkt
Apr 19 10:15:15 sh1 kernel: mppe_compress[0]: osize too small! (have: 1404 need: 1408)

This due to the way MPPE Microsoft point-to-point Encryption encodes data which results in the packet size being bigger then what was agreed in the VPN handshake - is my guess. There is a reported bug from 2005 which sadly hast not yet been addressed.

Fixing the issue by increasing the MTU

You can't fix this issue by modifying the MTU/MRU settings in '/etc/ppp/options' directly, you have to adjust the MTU after the PPP connection is up and this can be accomplished by adding a custom 'ip-up' script.

Below is my workaround script, place it into file '/etc/ppp/ip-up.d/mppefixmtu' and ensure that it is executable ('chmod +x mppefixmtu'):

#!/bin/sh
CURRENT_MTU="`ifconfig $1 | grep -Po '(?<=MTU:)([0-9]+)'`"
FIXED_MTU="`expr $CURRENT_MTU + 4`"
ifconfig $1 mtu $FIXED_MTU
echo "Increased MTU for $1 to $FIXED_MTU (from $CURRENT_MTU) to fix MPPE Microsoft Point-to-Point bug #330973"

Troubleshooting

You can review script errors by examining the file '/var/log/ppp-ipupdown.log', however if the file doesn't exist then you must create it to enable ip-up/ip-down script logging - don't forget to restart pppd.

As always if you found this useful feel free to follow me here or via twitter @danielsokolowski.

Tuesday, April 16, 2013

Another Awesome Free Linode VPS Upgrade

Double memory, double CPU and 10x internet traffic.

Our hosting of choice for self managed servers is Linode as this company continuously stays competitive and keeps giving back to it's customers. In the last two months they doubled the memory, updated CPU cores, increased transfer cap, and storage space you can read their blog for more info.
Plan RAM Disk XFER CPU Price
Linode 1G 1 GB 24 GB 2 TB 8 cores (1x priority) $20 / mo
Linode 2G 2 GB 48 GB 4 TB 8 cores (2x priority) $40 / mo
Linode 4G 4 GB 96 GB 8 TB 8 cores (4x priority) $80 / mo
Linode 8G 8 GB 192 GB 16 TB 8 cores (8x priority) $160 / mo
Linode 16G 16 GB 384 GB 20 TB 8 cores (16x priority) $320 / mo
Linode 24G 24 GB 576 GB 20 TB 8 cores (24x priority) $480 / mo
Linode 32G 32 GB 768 GB 20 TB 8 cores (32x priority) $640 / mo
Linode 40G 40 GB 960 GB 20 TB 8 cores (40x priority) $800 / mo


In my opinion the best VPS for a low cost $20 a month plan.

So at these specification I believe you can't get a better deal. Their $20 a month plan even rivals some dedicated servers as per my OVHS experience and with the doubled memory one can now run a small Minecraft server even better.

If you do sign up do you use my referral code: 7d884fa5262b62b8735502da003fee34061db49b so that I do get a small kick back for a sign up.

Linode failed to boot for unknown reason.

There was a serious issue when upgrading: after the upgrade none of the linodes would boot. However the support team responded within minutes to our support ticket and instructed me to use the 'Latest 64 big (3.8.4-x86_64-linode1)' kernel option. So if you run into unable to boot option after the upgrade change your linode's configuration profile as shown below:



And if you found this useful feel free to follow me here or via twitter @danielsokolowski.

Monday, April 1, 2013

Linphone Portable Version

Linphone PortableApps Beta Version

Linphone is an excelent multi-line, multi-platform - iOS, Blackberry, Windows, Mac and Linux - opensource VOIP SIP client I have been using for over a year now; I have tried a number of VOIP clients before settling on it - it's so awesome it cooks breakfast in the morning and it deserved a PortableApps version so I created one.


To download my beta version click LinphonePortable_3.5.99.0-snapshot-mar-21-2013_English.paf.exe and report any issues on this forum http://portableapps.com/node/36863. If you find this useful do +1 this or follow me @danielsokolow.

Sunday, March 31, 2013

Storing passwords in PuTTY

How to save SSH username/password for auto login in PuTTy

The answer is you can't do it...at least in plain PuTTy. However there is an awesome fork with that let's you store the username and password and other additional features called KiTTy.



So grab yourself a copy and +1 this if you do, thanks.

Tuesday, March 26, 2013

Eclipse Auto Refresh Browser on Save

Automatically refresh the browser window when saving a file in Eclipse IDE

All of my Django webdesign (yes SEOing there) work is done in the excellent and plugin rich Eclipse IDE on a Windows machine. The edit/preview workflow pattern is make some change, save the files (CTR+S), swtich to browser (ALT+Tab), refresh the windows (F5), switch back to Eclipse IDE (ALT+Tab) and rinse and repeat. Even though I can do this quickly it is repetitive and cumbersome and the seconds it takes turn into minutes and hours wasted over many projects.

Googling you will find posts that show how to automate the Alt+Tab F5 Alt+Tab sequence using VBScript and setting it as an automatic builder in Eclipse.


Now this is a good enough solution and it does work very well except that it does not work with all browsers. Google Chrome being one of them where the SendKeys seem to be ignored. However using an alternative utility NirCmd I was able to make it work as follows:
  • Download NirCmd and extract 'nircmd.exe' somewhere into your project (its KISS this way).
  • Create the following 'nircmd-chrome-focus+f5.bat' batch file somewhere in your project:

    ECHO Off
    REM This little batch files calls the awesome nircmd utility to focus Chrome window send an F5 and swtich
    REM back to Eclipse. This is no longer possible with VBScript in WIN7 as Chrome can only be focused but won't 
    REM accept key sends unless a click is made. Seriously donate to the NirSoft for making this tool. 
    
    %~dp0nircmd.exe win activate ititle "- Google Chrome"
    %~dp0nircmd.exe win max ititle "- Google Chrome"
    %~dp0nircmd.exe sendkey f5 press
    %~dp0nircmd.exe win activate ititle "- Eclipse"
    %~dp0nircmd.exe win max ititle "- Eclipse" 
    
    
  • Add the batch file as a project builder 'Project > Properties > Builder > New' and ensure 'Project > Build Automatically' and 'Build Options' > 'During auto builds' is selected; see above screenshot for an idea. 
Feel free to comment or share this if it has hepled you, also consider donating to NirSoft and check out some of their other utilities.

Friday, March 15, 2013

/bin/bash^M: bad interpreter Error

Eclipse file saving and /bin/bash^M: bad interpreter

The source of the error is having a bash file saved with line delimiter in Windows style rather the Unix format. To fix this you can use the 'dos2unix' linux utiltiy:

$> aptitude install dos2unix
The following NEW packages will be installed:
  dos2unix 
0 packages upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
                                         
$> dos2unix <file_to_fix>.sh 
dos2unix: converting file <file_to_fix>.sh to UNIX format ...

Eclipse new line delimter setting

But this post is about setting the Eclipse IDE to save in the Unix format, so let's get to it.

To do so, right click on the 'project' in your navigation pane, select 'properties', and under the 'Resource' tab ensure to specify the 'New text file line delimiter' is set to Unix as shown above; while you at it select the same encoding as on your server which most likley is UTF-8.

If you have 'Preferences > General > Editors > Text Editors > Show whitespace characters' enabled then you will be able to notice the encoding at a glance.

Do plus +1, comment, or share this if it helped you.

Thursday, February 7, 2013

Windows WebDAV Double Authentication 401 Requets

Speed up trick for Windows 7 WebDAV Client

One way to conveniently develop remotely with Eclipse + PyDev is to run WebDAV on the remote server and map it as a networked drive in Windows (PyDev over Eclipse Target Managmenet does not play nicely). WebDAV works rather well except it is not the fastest especially on initial project creation /project refreshing and if your share is protected with a username/password; below is how I improved the performance.

Apparently by protocol design and due to security Windows 7 client does not cache it's authentication and so on each request first an anonymous attempt is made followed by an authenticated one:

...
72.38.184.18 - - [22/Jan/2013:23:05:04 +0000] "PROPFIND /Files HTTP/1.0" 401 751
72.38.184.18 - username [22/Mar/2011:23:05:04 +0000] "PROPFIND /Files HTTP/1.0" 301 495
72.38.184.18 - - [22/Jan/2013:23:05:04 +0000] "PROPFIND /Files/ HTTP/1.0" 401 751
72.38.184.18 - username [22/Mar/2011:23:05:04 +0000] "PROPFIND /Files/ HTTP/1.0" 207 1175
72.38.184.18 - - [22/Jan/2013:23:05:07 +0000] "PROPFIND /Files HTTP/1.0" 401 751
...

After a bit of research I was able to implement a work around, allow anonymous access to the PROPFIND requests which are mainly used for directory listings and authenticated access for everything else. Refreshing a python/djagno project will take half as long for example 21876 files took 11 minutes with no authentication vs 21 with authentication

## Development HTTP Site
<VirtualHost *:80>
  ServerAdmin FOO@FOO.com
  ServerName development.FOO.com
  ServerAlias www.development.FOO.com

  # Log file location and settings; logs within project is ok as long as 'links' are made to system 'var/log/apache'
  ErrorLog /var/log/apache2/development.FOO.com-error.log
  CustomLog /var/log/apache2/development.FOO.com-vhost_combined-access.log vhost_combined

  # Canonical to always strip www - see: http://stackoverflow.com/questions/88011/make-apache-automatically-strip-off-the-www
  RewriteCond %{HTTP_HOST} ^www\.(.+)$
  RewriteRule ^(.*)$ ${SERVER_PROTOCOL}://%1/$1 [R=301,L,NC]

  # Authenticated access for the development site version - because without this Google will find you!
  # Just in case we also prevent serving of the password logins file if it is stored in a serving folder.
  Redirect /apache-logins.htdigest http://development.FOO.com
  <Location />
    DAV On
    DirectoryIndex disabled
    Options +Indexes

    AuthType Digest
    AuthName "development.FOO.com"
    # AuthDigestDomain which urls (and any under it) this applies - should match location
    AuthDigestDomain /
    AuthDigestProvider file
    AuthUserFile /srv/www/django/development.FOO.com/apache-logins.htdigest
    # uncomment the LimitExcept to receive a small boost for non caching Windows WebDAV client by allowing
    # anonymous directory listing; see http://serverfault.com/questions/250578/webdav-and-windows-7-client
    <LimitExcept PROPFIND>
     Require valid-user
    </LimitExcept>
  </Location>

  WSGIProcessGroup development.FOO.com
  # You can further limit processes, threads and set a inactivity-timer so deamon get unloaded
  WSGIDaemonProcess development.FOO.com display-name=%{GROUP}
  WSGIScriptAlias / /srv/www/django/development.FOO.com/apache-django-development.wsgi

  # Serve static / media files through apache instance and alias/map them to specific urls. to maximize security
  # `Options -Indexes` is enabled to prevent directory listing
  Options -Indexes
  Alias /robots.txt /srv/www/django/development.FOO.com/src/django-project/static/robots.txt
 #Alias /sitemap.xml /srv/www/django/development.FOO.com/src/django-project/static/sitemap.xml
  Alias /favicon.ico /srv/www/django/development.FOO.com/src/django-project/static/favicon.ico
  Alias /media /srv/www/django/development.FOO.com/src/django-project/static/
  Alias /static /srv/www/django/development.FOO.com/src/django-project/static/

</VirtualHost>

Hope this helps you out, feel free to follow me on twitter or google plus: @danielsokolowski and +1 this.

Monday, February 4, 2013

GeoDjango + Spatialite (SQLite3) Setup Issues

Because there are fresh ones every time I start a new project!

As of May 03 2013 I have opted to install virtualenv --system-site-packages and running aptitude-install python-pysqlite2 spatialite-bin gdal-bin; the site are not as self contained any more but sure beats fighting with the issues.

All my Django projects are self contained (as much as possible) with Virtualenv and use Spatialite database (SQLite3 +geo spatial stuff) installed so to enable GeoDjango functionality. This post is a list of issues I encounter related to that setup and how I resolved them; as I seem to run into a new one on every project start. And as always if you found this useful do +1, or follow me on twitter @danielsokolows

/usr/include/spatialite.h:61: note: previous declaration of ‘spatialite_init’ was here

Say what? --- this one threw me for a three hour fun time. It happened because I executed 'pip install pyspatialite' will install the 3.X vesion which is is broken and craps out with:

setup.py", line 97, in get_amalgamation 
        download_file = pattern.findall(download_page)[0] 
    IndexError: list index out of range

So then I tried an older 'pip install pyspatialite==2.6.2-spatialite.2.4.0-4' which results in:

...
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -DMODULE_NAME="spatialite.dbapi2" -DVERSION="3.0.1" -DSQLITE_ENABLE_FTS3=1 -DSQLITE_ENABLE_RTREE=1 -DSQLITE_ENABLE_COLUMN_METADATA=1 -DOMIT_FREEXL=1 -I/Library/Frameworks/GEOS.framework/unix/include/ -I/Library/Frameworks/PROJ.framework/unix/include/ -Iamalgamation -I/usr/include/python2.6 -c src/module.c -o build/temp.linux-i686-2.6/src/module.o
In file included from src/module.c:24:src/connection.h:36: error: conflicting types for ‘spatialite_init’
/usr/include/spatialite.h:61: note: previous declaration of ‘spatialite_init’ was here
error: command 'gcc' failed with exit status 1
----------------------------------------
Command /srv/www/django/development..com/virtualenv/bin/python -c "import setuptools; __file__='/srv/www/django/development..com/virtualenv/build/pyspatialite/setup.py'; execfile('/srv/www/django/development..com/virtualenv/build/pyspatialite/setup.py')" install --single-version-externally-managed --record /tmp/pip-vq4xsp-record/install-record.txt --install-headers /srv/www/django/development..com/virtualenv/include/site/python2.6 failed with error code 1
Storing complete log in /root/.pip/pip.log

Which after pulling hair out I finally realized is because pip is caching the previous partial install and does not attempt a new download. To resolve this (clearing pip cache did nothing as there was no '~/.pip/cache' to clear) install another package and uninstall it and then try again:

(virtualenv)root@klisrv03:development..com>$  pip install pysqlite
Downloading/unpacking pysqlite
  Downloading pysqlite-2.6.3.tar.gz (74Kb): 74Kb downloaded
 ...
Successfully installed pysqlite
Cleaning up...
(virtualenv)root@klisrv03:development..com>$  pip uninstall pysqlite
Uninstalling pysqlite:
 ...
Proceed (y/n)? y
  Successfully uninstalled pysqlite
(virtualenv)root@klisrv03:development..com>$  pip install pyspatialite==2.6.2-spatialite.2.4.0-4
...
Successfully installed pyspatialite
Cleaning up..

That was easy and obvious!

ImproperlyConfigured: Only versions of pysqlite 2.5+ are compatible with SpatiaLite and GeoDjango.

EASY! Just run 'pip install pysqlite' to get a newer version then what your current python stack includes...not so fast as then you get:

ImproperlyConfigured: The pysqlite library does not support C extension loading. Both SQLite and pysqlite must be configured to allow the loading of extensions to use SpatiaLite.

The solution is to try installing from source as per django docs or using something like 'pip install pyspatiallite==2.6.2-spatialite.2.4.0-4'

Still getting: ImproperlyConfigured: Only versions of pysqlite 2.5+ are compatible with SpatiaLite and GeoDjango then make sure to patch in support as per GeoDjango missing pyspatialite support or grab my github-danielsokolowski-django patched version.

error: geos_c.h: No such file or directory

   gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -DMODULE_NAME="spatialite.dbapi2" -DSQLITE_ENABLE_FTS3=1 -DSQLITE_ENABLE_RTREE=1 -DSQLITE_ENABLE_COLUMN_METADATA=1 -I/Library/Frameworks/GEOS.framework/unix/include/ -I/Library/Frameworks/PROJ.framework/unix/include/ -Iamalgamation -I/usr/include/python2.6 -c amalgamation/spatialite.c -o build/temp.linux-i686-2.6/amalgamation/spatialite.o
    amalgamation/spatialite.c:94:20: error: geos_c.h: No such file or directory

To resolve run aptitude install libgeos-dev.

error: proj_api.h: No such file or directory

    gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -DMODULE_NAME="spatialite.dbapi2" -DSQLITE_ENABLE_FTS3=1 -DSQLITE_ENABLE_RTREE=1 -DSQLITE_ENABLE_COLUMN_METADATA=1 -I/Library/Frameworks/GEOS.framework/unix/include/ -I/Library/Frameworks/PROJ.framework/unix/include/ -Iamalgamation -I/usr/include/python2.6 -c amalgamation/spatialite.c -o build/temp.linux-i686-2.6/amalgamation/spatialite.o
    amalgamation/spatialite.c:98:22: error: proj_api.h: No such file or directory

Execute aptitude install libproj-dev.

error: python.h: No such file or directory

Execute aptitude install python-dev.

src/connection.h:33:21: fatal error: sqlite3.h: No such file or directory

Execute aptitude install libsqlite3-dev.

src/connection.h:34:24: fatal error: spatialite.h: No such file or directory

Execute aptitude install libspatialite-dev.

ImportError: cannot import name SpatialReference

You are missing the required for Spatialite + Geodjango GDAL library, install it by running aptitude install gdal-bin

Sunday, January 27, 2013

HTTPS and drop in Google Web Master Tools Impressions

Switched site to HTTPS and impressions and visits plummeted

In an attempt to resolve the (No Provided) issue in Google Analytics I have switched our site Kingston Search Engine Optimization [yes I am doing SEO backlinking there :)] to HTTPS and experienced a massive drop in impressions and visits.

Google Analytics reported no change in actual visits so the issue was left alone to hopefully sort itself out; but it didn't. By the way switching to HTTPS will not resolve the missing keywords in GA, if you analyze the browser requests you will notice Google just plainly strips them from the referral header information.

Solution to HTTPS impression drop

This was a mystery for a while until today when I found a solution which is very simple. By default Google Web Master Tools creates and tracks sites under HTTP protocol, and treats HTTPS as different sites entirely, just like it treats www.webdesign.danols.com and webdesign.danols.com as separate sites. Creating a new entry through 'Add a Site' and including the protocol resolves the issue, ex. 'https://webdesign.danols.com'; and luckily Google Web Master Tools will keeps all the history data from before the HTTPS switch.

If this post helps I ask that you do +1 or share it, or even better follow me on twitter @danielsokolows

Thursday, January 10, 2013

Django Template Eclipse Editor Plugin

Aptana Eclipse Django template editor alternative

A solid lightweight alternative to Aptana Django template editor Eclipse plug-in. Definitively worth at a test drive - which is what I am doing now - since I have refrained from installing the Aptana alternative due to it's dependencies.

You can download it here: Django Editor - plug-in for Eclipse. One shortcoming however is that HTML and JS editing is bare bones with no content assist, this is expected thought and can be mitigated by switching to a specialized editor as needed.

Friday, January 4, 2013

Setting up ZOTAC ZBOX nano AD Remote for XBMC

Getting the Windows, Info and Colored buttons to work

The included Zotac remote is Windows MCERemote compatible but the additional buttons (Windows, Info, Colored buttons) will not work out of the box. The XBMC MCERemote add-on will fail with:

 * Warning
 * The ReportMappingTable registry key is not present
 * No Microsoft remote is installed

Good News

But before I begin, if this helps you please do +1 or share it as that helps me.

So take the MCERemote registry files, this, this,0 and this thread and a 3 hours and you get a tweaked Zotac Remote Registry Fix below; it labels all keys I could discover and maps some as follow:

  • The Windows remote key to Win keyboard key (perfect if you are running Windows 8)
  • The Colored Buttons to Ctrl 1 through 4
  • The Info button to 'Ctrl D' - XBMC display info
  • [T] Teletext? key to send 'T' - XBMC toggles subtitlies
  • The '≡' right of 'T' to send 'C' - XBMC display contextual menu

Copy and paste the below into a file on your dekstop named 'zotact.reg' and double click on it and then restart your computer. If you have any issues feel free to contact me and I will try to assist and if you do find this helpful do follow me on twitter: @danielsokolows.

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Services\HidIr\Remotes\745a17a0-74d3-11d0-b6fe-00a0c90f57df]
"RemoteName"="RC6 based MCE remote for Philips(OVU710001/00 OTS)"
"EnableDebounce"=dword:00000000
"EnableVendorPassThrough"=dword:00000001
"DecoderID"=dword:00000001
"CodeSetID"=dword:00000000
"CodeBitLen"=dword:00000020
"CodeSetNum0"=dword:00000001
"CodeSetNum1"=dword:00000002
"CodeSetNum2"=dword:00000003
"CodeSetNum3"=dword:00000004
"CodeSetNumMask"=dword:00007000
"CodeSetNumShiftBits"=dword:0000000c
"CodeMatchMask"=dword:ffff0f00
"CodeMatchValue"=dword:80340400
"DataMask"=dword:000000ff
"DataShiftBits"=dword:00000000
"ReportLength"=dword:00000003
"ReportDescriptor"=hex:05,0c,09,01,a1,01,85,01,19,00,2a,3c,02,15,00,26,3c,02,\
  95,01,75,10,81,00,c0,06,bc,ff,09,88,a1,01,85,02,19,01,29,ff,15,00,25,01,95,\
  01,75,08,81,00,c0,05,01,09,80,a1,01,85,03,19,01,29,ff,15,01,25,ff,95,01,75,\
  08,81,00,c0,05,01,09,06,a1,01,85,04,05,07,19,e0,29,e8,75,01,95,08,81,02,19,\
  00,29,90,26,ff,00,75,08,95,01,81,00,c0
"ReportMappingTable"=hex:\
5d,00,00,00,04,08,00,\ ; Green Windows key - modified to send Keyboard Window key. Original: 5d,00,00,00,02,0d,00,
83,00,00,00,01,24,02,\ ; Button '<-' (Back)
cb,00,00,00,04,01,07,\ ; Info '(i)' - modified to send ctrl-D. Original cb,00,00,00,01,09,02,
58,00,00,00,04,00,52,\ 
59,00,00,00,04,00,51,\
5a,00,00,00,04,00,50,\
5b,00,00,00,04,00,4f,\
5c,00,00,00,04,00,28,\
2c,00,00,00,01,b0,00,\ ; Play 
30,00,00,00,01,b1,00,\ ; Pause
31,00,00,00,01,b7,00,\ ; Stop
37,00,00,00,01,b2,00,\ ; Record
28,00,00,00,01,b3,00,\ ; Forward
29,00,00,00,01,b4,00,\ ; Rewind
21,00,00,00,01,b6,00,\ ; Button '|<'
20,00,00,00,01,b5,00,\ ; Button '>|'
10,00,00,00,01,e9,00,\ ; Up Arrow
11,00,00,00,01,ea,00,\ ; Down Arrow
0d,00,00,00,01,e2,00,\ ; Mute
4a,00,00,00,02,2b,00,\
1e,00,00,00,01,9c,00,\ ; Channel '+' Button
1f,00,00,00,01,9d,00,\ ; Channel '-' Button
0c,00,00,00,03,82,00,\
8e,00,00,00,03,83,00,\ ; Red Power '|'
3d,00,00,00,03,82,00,\ ; Red Power 'O'
00,00,00,00,04,00,27,\ ; Button '0'
01,00,00,00,04,00,1e,\ ; Button '1'
02,00,00,00,04,00,1f,\ ; Button '2'
03,00,00,00,04,00,20,\ ; Button '3'
04,00,00,00,04,00,21,\ ; Button '4'
05,00,00,00,04,00,22,\ ; Button '5'
06,00,00,00,04,00,23,\ ; Button '6'
07,00,00,00,04,00,24,\ ; Button '7'
08,00,00,00,04,00,25,\ ; Button '8'
09,00,00,00,04,00,26,\ ; Button '9'
3a,00,00,00,04,00,29,\ ; Button 'CLEAR'
34,00,00,00,04,00,28,\ ; Button 'ENTER'
32,00,00,00,04,02,20,\ ; Button '#'
33,00,00,00,04,02,25,\ ; Button '*'
2f,00,00,00,04,00,17,\ ; Button '[T]' - modified to send 'T' 
6d,00,00,00,04,05,1e,\ ; Red - modified to send 'Ctrl-Alt-1'. Original: 6d,00,00,00,02,5b,00,
6e,00,00,00,04,05,1f,\ ; Green - modified to send 'Ctrl-Alt-2'. Original: 6e,00,00,00,02,5c,00,
6f,00,00,00,04,05,20,\ ; Yellow - modified to send 'Ctrl-Alt-3'. Original: 6f,00,00,00,02,5d,00,
70,00,00,00,04,05,21,\ ; Blue - modified to send 'Ctrl-Alt-4'. Original: 70,00,00,00,02,5e,00,
3e,00,00,00,04,00,06,\ ; Button '=' to the right of [T] modified to send 'C'. Orignal: 3e,00,00,00,01,8d,00,
2e,00,00,00,02,25,00,\
87,00,00,00,02,47,00,\
9b,00,00,00,02,48,00,\
89,00,00,00,02,49,00,\
9c,00,00,00,02,4a,00,\
4f,00,00,00,02,50,00,\
9d,00,00,00,02,3c,00,\
9e,00,00,00,02,3d,00,\
54,00,00,00,02,24,00,\
85,00,00,00,02,4b,00,\
4e,00,00,00,02,4c,00,\
4b,00,00,00,02,4d,00,\
c9,00,00,00,02,51,00,\
42,00,00,00,02,28,00,\
a8,00,00,00,02,32,00,\
a9,00,00,00,02,33,00,\
aa,00,00,00,02,34,00,\
ab,00,00,00,02,35,00,\
ac,00,00,00,02,36,00,\
ad,00,00,00,02,37,00,\
ae,00,00,00,02,38,00,\
af,00,00,00,02,39,00,\
b1,00,00,00,02,3a,00,\
b2,00,00,00,02,80,00,\
94,00,00,00,02,81,00,\
95,00,00,00,02,6f,00,\
84,00,00,00,01,08,02,\
f7,00,00,00,02,27,00,\
0f,00,00,00,02,4f,00,\
78,00,00,00,02,3e,00,\
79,00,00,00,02,3f,00,\
7a,00,00,00,02,40,00,\
8d,00,00,00,02,6a,00,\
8f,00,00,00,02,2c,00,\
50,00,00,00,02,2d,00,\
b3,00,00,00,02,52,00,\
b4,00,00,00,02,53,00,\
b5,00,00,00,02,54,00,\
b6,00,00,00,02,55,00,\
c0,00,00,00,02,56,00,\
c2,00,00,00,02,57,00,\
c1,00,00,00,02,58,00,\
c3,00,00,00,02,59,00,\
40,00,00,00,02,2e,00,\
ef,00,00,00,02,2f,00,\
cd,00,00,00,02,30,00,\
80,00,00,00,02,31,00,\
90,00,00,00,02,44,00,\
f8,00,00,00,02,78,00,\
0e,00,00,00,02,41,00,\
0b,00,00,00,02,42,00

FreeSWITCH Call Audio Stuttering Issues

Reduce VOIP audio stuttering by increasing deamon priority

On occassion our FreeSWITCH Linode VPS stutter which I believe is because the server overall gets a little busy with everything else that is installed. One noticeable way tp reduce or almost eliminate this is to run the FreeSWITCH VOIP daemon in real time priority. But before I begin please do +1 or share if it helps you as it helps me.

On Ubuntu/Debian chaning FreeSWITCH priority is easily accomplished by modifying the `/etc/init.d/freeswitch` start-up script and adding the `rc` real time execute options to the `DAEMON_ARGS` argument – but if you want set the priority even higher scroll to bottom of post.

An example of a FreeSWITCH init.d script can be found on the FreeSWITCH Wiki page which - I think is more complete then the sample included in the source files.

#!/bin/bash
### BEGIN INIT INFO
# Provides:          freeswitch
# Required-Start:    $local_fs $remote_fs $network
# Required-Stop:     $local_fs $remote_fs $network
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Description:       FreeSWITCH debian init script.
# Author: Matthew Williams
#
### END INIT INFO
# Do NOT "set -e"

# PATH should only include /usr/* if it runs after the mountnfs.sh script
PATH=/sbin:/usr/sbin:/bin:/usr/bin:/usr/local/bin
DESC="Freeswitch"
NAME=freeswitch
DAEMON=/usr/local/freeswitch/bin/$NAME
DAEMON_ARGS="-nc -np" # -rp is for real-time priority, -np for normal system priority
PIDFILE=/usr/local/freeswitch/run/$NAME.pid
SCRIPTNAME=/etc/init.d/$NAME

FS_USER=freeswitch
FS_GROUP=daemon

# Exit if the package is not installed
[ -x "$DAEMON" ] || exit 0

# Read configuration variable file if it is present
[ -r /etc/default/$NAME ] && . /etc/default/$NAME

# Load the VERBOSE setting and other rcS variables
. /lib/init/vars.sh

# Define LSB log_* functions.
# Depend on lsb-base (>= 3.0-6) to ensure that this file is present.
. /lib/lsb/init-functions

#
# Function that sets ulimit values for the daemon
#
do_setlimits() {
        ulimit -c unlimited
        ulimit -d unlimited
        ulimit -f unlimited
        ulimit -i unlimited
        ulimit -n 999999
        ulimit -q unlimited
        ulimit -u unlimited
        ulimit -v unlimited
        ulimit -x unlimited
        ulimit -s 240
        ulimit -l unlimited
        return 0
}

#
# Function that starts the daemon/service
#
do_start()
{
 # Set user to run as
        if [ $FS_USER ] ; then
   DAEMON_ARGS="`echo $DAEMON_ARGS` -u $FS_USER"
        fi
 # Set group to run as
        if [ $FS_GROUP ] ; then
          DAEMON_ARGS="`echo $DAEMON_ARGS` -g $FS_GROUP"
        fi

        # Return
        #   0 if daemon has been started
        #   1 if daemon was already running
        #   2 if daemon could not be started
        start-stop-daemon --start --quiet --pidfile $PIDFILE --exec $DAEMON --test > /dev/null -- \
                || return 1
        do_setlimits
        start-stop-daemon --start --quiet --pidfile $PIDFILE --exec $DAEMON -- \
                $DAEMON_ARGS \
                || return 2
        # Add code here, if necessary, that waits for the process to be ready
        # to handle requests from services started subsequently which depend
        # on this one.  As a last resort, sleep for some time.
}

#
# Function that stops the daemon/service
#
do_stop()
{
        # Return
        #   0 if daemon has been stopped
        #   1 if daemon was already stopped
        #   2 if daemon could not be stopped
        #   other if a failure occurred
        start-stop-daemon --stop --quiet --retry=TERM/30/KILL/5 --pidfile $PIDFILE --name $NAME
        RETVAL="$?"
        [ "$RETVAL" = 2 ] && return 2
        # Wait for children to finish too if this is a daemon that forks
        # and if the daemon is only ever run from this initscript.
        # If the above conditions are not satisfied then add some other code
        # that waits for the process to drop all resources that could be
        # needed by services started subsequently.  A last resort is to
        # sleep for some time.
        start-stop-daemon --stop --quiet --oknodo --retry=0/30/KILL/5 --exec $DAEMON
        [ "$?" = 2 ] && return 2
        # Many daemons don't delete their pidfiles when they exit.
        rm -f $PIDFILE
        return "$RETVAL"
}

#
# Function that sends a SIGHUP to the daemon/service
#
do_reload() {
        #
        # If the daemon can reload its configuration without
        # restarting (for example, when it is sent a SIGHUP),
        # then implement that here.
        #
        start-stop-daemon --stop --signal 1 --quiet --pidfile $PIDFILE --name $NAME
        return 0
}

case "$1" in
  start)
        [ "$VERBOSE" != no ] && log_daemon_msg "Starting $DESC" "$NAME"
        do_start
        case "$?" in
                0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
                2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
        esac
        ;;
  stop)
        [ "$VERBOSE" != no ] && log_daemon_msg "Stopping $DESC" "$NAME"
        do_stop
        case "$?" in
                0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
                2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
        esac
        ;;
  #reload|force-reload)
        #
        # If do_reload() is not implemented then leave this commented out
        # and leave 'force-reload' as an alias for 'restart'.
        #
        #log_daemon_msg "Reloading $DESC" "$NAME"
        #do_reload
        #log_end_msg $?
        #;;
  restart|force-reload)
        #
        # If the "reload" option is implemented then remove the
        # 'force-reload' alias
        #
        log_daemon_msg "Restarting $DESC" "$NAME"
        do_stop
        case "$?" in
          0|1)
                do_start
                case "$?" in
                        0) log_end_msg 0 ;;
                        1) log_end_msg 1 ;; # Old process is still running
                        *) log_end_msg 1 ;; # Failed to start
                esac
                ;;
          *)
                # Failed to stop
                log_end_msg 1
                ;;
        esac
        ;;
  *)
        #echo "Usage: $SCRIPTNAME {start|stop|restart|reload|force-reload}" >&2
        echo "Usage: $SCRIPTNAME {start|stop|restart|force-reload}" >&2
        exit 3
        ;;
esac

Setting the deamon priority even higher

To set the priority even higher you used to be able to pass the system default priority `np` option to FreeSWITCH executable and then the desired nice level directly to the `start-stop-deamon` but as of Jan 2013 '-np' option defaults to priority 19.

Use the below work around which changes the priority after freeswitch starts up.

#!/bin/bash
### BEGIN INIT INFO
...
DAEMON_ARGS="-nc -rp" # -rp is for real-time priority, -np for normal system priority, -lp for low priority
...
        sstart-stop-daemon --start --quiet --pidfile $PIDFILE --exec $DAEMON --test > /dev/null -- \
                || return 1
        do_setlimits
        start-stop-daemon --start --quiet --pidfile $PIDFILE --exec $DAEMON -- \
                $DAEMON_ARGS \
                || return 2
        ...
        # increase priority further, using -20 is not recommended
        sleep 15 # wait for all children to start
        renice -n -19 -u freeswitch

Use UDP Transport instead of TCP

The last tweak that might help is using UDP. This transport protocol has less overhead as for one it does not require packet acknowledgement. Check your client and confirm the UDP is selected; speaking of VOIP clients I recommend open source Linphone - a multi line and multi OS (Android, Windows, Linux, etc) client; the latest snapshot builds work the best in my experience.


FreeSWITCH Command Line Options

For a list of all available FreeSWITCH command options see this Command Line wiki page.