Monday, February 27, 2012

Getting ready for the PyCon Pandas tutorial

Today I received an email with a link to instructions for preparing a computer for the PyCon Tutorial: Data analysis in Python with pandas, complete with the following instructions for:

Installing PyQt on OS X and Linux
Installation of PyQt from source on OS X and Linux is much more involved.
OS X: http://www.oak-tree.us/blog/index.php/2009/05/12/pyqt-mac

Note the lack of instructions for installing on Linux. Thrown to the wolves, I guess...

In the interest of helping other Linux users (or at least Ubuntu 10.10 users) out, here are the steps for setting up a Python virtual environment with the necessary libraries, using virtualenvwrapper and pip.

mkvirtualenv pandas

pip install numpy
pip install scipy
pip install matplotlib
pip install ipython
sudo apt-get install libzmq-dev
pip install pyzmq
pip install pygments
pip install pandas
pip install python-dateutil
pip install pyqt
pip install tornado
ipython notebook --pylab=inline

Here is a list of the library versions that were installed:
(pandas)tylere@tylere-laptop:~$ pip freeze

Pygments==1.4
ipython==0.12
matplotlib==1.1.0
numpy==1.6.1
pandas==0.7.0
python-dateutil==1.5
pyzmq==2.1.11
scipy==0.10.0
tornado==2.2
wsgiref==0.1.2
yolk==0.4.3

Tuesday, January 24, 2012

pyKML at the 2012 American Meteorological Society Annual Meeting

Tomorrow I am giving a talk at the American Meteorological Society Annual Meeting entitled:

Using the pyKML Library to Format Weather and Climate Data for Viewing in Google Earth

It is at 11:45, if you happen to be at the meeting. The talk presents numerous examples of using pyKML, but because of the static format required by the conference (i.e. Powerpoint or PDF) I thought it might be useful to provide a number of links to related content. Enjoy!

pyKML:

Examples:

Misc:

Wednesday, September 21, 2011

Installing GDAL in a Python Virtual Environment

Python virtual environments are great for managing package versions, but installing Python packages that have shared library dependencies can be a pain. It seems that ever since I have been using Python virtual environments, I have been trying to figure out how to install the GDAL (the geospatial workhorse) Python bindings in a virtual environment.

Well today, I finally succeeded. The following recipe seems to work for GDAL 1.8.1. It assumes that you have virtualenv and the virtualenvwrapper already installed.

(1) Build and install GDAL with Python bindings on your system

GDAL_VER=1.8.1 
GDAL_PATH=/usr/local/gdal/$GDAL_VER 
GEOS_VER=3.2.2 
GEOS_PATH=/usr/local/geos/$GEOS_VER 
mkdir ~/src/gdal 
cd ~/src/gdal 
wget http://download.osgeo.org/gdal/gdal-$GDAL_VER.tar.gz 
tar xzf gdal-$GDAL_VER.tar.gz 
cd gdal-$GDAL_VER 
./configure --prefix=$GDAL_PATH --with-geos=$GEOS_PATH/bin/geos-config --with-python >& log_configure.out 
make >& log_make.out 
make install >& log_make_install.out 
sudo sh -c "echo $GDAL_PATH'/lib' > /etc/ld.so.conf.d/gdal.conf" 
sudo ldconfig

(2) Create a Python Virtual Environment

mkvirtualenv --no-site-packages osgeo-test

(3) Create a symbolic link from within the virtual environment to gdal-config

This is needed because the setup.py file of the GDAL Python bindings requires gdal-config.

# create a symbolic link in the virtual environment to the specific version of GDAL that you want to use 
(osgeo-test)$ ln -s ~/src/gdal/gdal-1.8.1/apps/gdal-config ~/.virtualenvs/osgeo-test/bin/gdal-config

(4) Download the GDAL Python package using PIP

Download, but don't install, since we need to configure the build extension parameters.

(osgeo-test)$ pip install --no-install GDAL

(5) Set the build extension parameters using setup.py

(osgeo-test)$ cd ~/.virtualenvs/osgeo-test/build/GDAL 
(osgeo-test)$ python setup.py build_ext --gdal-config=/usr/local/gdal/1.8.1/bin/gdal-config --library-dirs=/usr/local/gdal/1.8.1/lib --libraries=gdal --include-dirs=/usr/local/gdal/1.8.1/include

(6) Install the configured package using PIP

(osgeo-test)$ cd 
(osgeo-test)$ pip install --no-download GDAL

(7) test it out

(osgeo-test)$ cd 
(osgeo-test)$ python 
Python 2.6.5 (r265:79063, Apr 16 2010, 13:57:41) 
[GCC 4.4.3] on linux2 
Type "help", "copyright", "credits" or "license" for more information. 
>>> from osgeo import gdal 
>>> gdal.__version__ 
'1.8.1' 
>>>

Thanks

Thanks to Hobu for pushing out a new version of the GDAL Python bindings to PyPI that made this possible.

References

http://openblockproject.org/docs/install/common_install_problems.html

Saturday, June 7, 2008

Kiteboarding at Harsens Island

Ray and I headed out to Harsens Island for Kiteboarding. Ray had been there a few times before, but it was my first time out there. I got a kick out of the car ferry. While on the island it is hard to imagine that you are just outside of Detroit... it felt more like a barrier island on the east coast. The wind was a bit to light for most of the day, but Ray was able to get up and go when it occasionally picked up.

View Larger Map

Wednesday, June 4, 2008

GeoDjango on Slicehost: Getting Started (1 of 4)

Intro

This series of posts describes a relatively inexpensive way of setting up GeoDjango on a server that is publicly accessible. Specifically, I will go over how to setup GeoDjango on a Slicehost VPS slice that is running Ubuntu 8.04 (Hardy). The posts assume you have a local computer running Ubuntu Linux.

References

The setup process was drawn from a variety of other sources. If you need more information on any of the topics, refer to the following pages:

This first post describes the preliminary steps of signing up for a slice and creating a SSH keypair. The subsequent posts will cover the initial configuration of Ubuntu 8.04 LTS, building/installing the necessary software, and creating a test GeoDjango project.

Several of the parameters described in this series of posts are just for demonstration purposes, and will vary for each user. For these posts the following values were used:

Slice IP: 11.222.333.444
Slice Password: secret
Slice Name: myslice
Admin user: demo

Getting a Slice

The first step is to purchase a slice from Slicehost. I signed up for the smallest size slice (256MB, 10GB storage, 100GB bandwidth, currently @ $20/month), but these instructions should work for larger slices too (although the free memory stats will differ). After signing up for a slice, you will receive an IP for your slice and a Slice Manager Password.

It's probably too early to tell, but if you find these instructions useful consider signing up using my referral link. If I get enough referral credits, I'll probably upgrade to a larger slice...

https://manage.slicehost.com/customers/new?referrer=947119543

If you need to rebuild your slice (i.e. erase everything and start over) go to: https://manage.slicehost.com/ and login to the SliceManager. Click on the name of your slice, then select Rebuild. Select the '"Ubuntu 8.04 LTS (hardy)" image, give it a name (such as myslice), and rebuild it.

Setting up an SSH Key Pair

The first step for setting up a secure connection between your local computer and the slice is generating a SSH Key Pair. If you end up rebuilding your slice, you can reuse the SSH key pair.

On your LOCAL COMPUTER, type in the following to generate an SSH key pair.

$ mkdir ~/.ssh
$ ssh-keygen -t rsa

This will create a public (~/.ssh/id_rsa.pub) and private (~/.ssh/id_rsa) key files. The public key will be later transferred to your slice.

Ok, that was all of the preliminaries. On to Part 2 - Ubuntu 8.04 Configuration

GeoDjango on Slicehost: Ubuntu 8.04 Configuration (2 of 4)

OS Configuration

This post goes over how to configure your Ubuntu 8.04 Slice on Slicehost.

(general reference: http://articles.slicehost.com/2008/4/25/ubuntu-hardy-setup-page-1)

Log On to your Slice as root

Note: you will need the ip and password for the slice which was provided to you when the slice was created.

ssh root@11.222.333.444

Change the root password for your slice by entering

passwd

and specifying a new password for the root user account. Logout and SSH to connect again to verify that the new password works. NOTE: If you rebuilt the slice, you may need to remove an entry from the local ~/.ssh/known_hosts file in order to connect.

Create an admin (non-root) user account

Add a user and give them a password...

adduser demo

Give the user admin privileges by opening the /etc/sudoers file

nano /etc/sudoers

and add the following to the end of the file:

demo   ALL=(ALL) ALL

SSH Configuration

(general reference: http://articles.slicehost.com/2008/4/25/ubuntu-hardy-setup-page-1 (SSH Copy section & SSH Config section))
Copy your public key from your local computer to your slice by running the following on your LOCAL COMPUTER...

scp ~/.ssh/id_rsa.pub demo@11.222.333.444:/home/demo/

Note: you will need to provide the demo user's password that you created earlier.

Now on the SLICE computer, setup the SSH permissions...

mkdir /home/demo/.ssh
mv /home/demo/id_rsa.pub /home/demo/.ssh/authorized_keys
chown -R demo:demo /home/demo/.ssh
chmod 700 /home/demo/.ssh
chmod 600 /home/demo/.ssh/authorized_keys

Make changes to the default SSH configuration...

nano /etc/ssh/sshd_config

Replace the content of /etc/ssh/sshd_config with the following...

# Package generated configuration file
# See the sshd(8) manpage for details

# What ports, IPs and protocols we listen for
Port 30000
# Use these options to restrict which interfaces/protocols sshd will bind to
#ListenAddress ::
#ListenAddress 0.0.0.0
Protocol 2
# HostKeys for protocol version 2
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
#Privilege Separation is turned on for security
UsePrivilegeSeparation yes

# Lifetime and size of ephemeral version 1 server key
KeyRegenerationInterval 3600
ServerKeyBits 768

# Logging
SyslogFacility AUTH
LogLevel INFO

# Authentication:
LoginGraceTime 120
PermitRootLogin no
StrictModes yes

RSAAuthentication yes
PubkeyAuthentication yes
#AuthorizedKeysFile     %h/.ssh/authorized_keys

# Don't read the user's ~/.rhosts and ~/.shosts files
IgnoreRhosts yes
# For this to work you will also need host keys in /etc/ssh_known_hosts
RhostsRSAAuthentication no
# similar for protocol version 2
HostbasedAuthentication no
# Uncomment if you don't trust ~/.ssh/known_hosts for RhostsRSAAuthentication
#IgnoreUserKnownHosts yes

# To enable empty passwords, change to yes (NOT RECOMMENDED)
PermitEmptyPasswords no

# Change to yes to enable challenge-response passwords (beware issues with
# some PAM modules and threads)
ChallengeResponseAuthentication no

# Change to no to disable tunnelled clear text passwords
PasswordAuthentication no

# Kerberos options
#KerberosAuthentication no
#KerberosGetAFSToken no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes

# GSSAPI options
#GSSAPIAuthentication no
#GSSAPICleanupCredentials yes

X11Forwarding no
X11DisplayOffset 10
PrintMotd no
PrintLastLog yes
TCPKeepAlive yes
#UseLogin no

#MaxStartups 10:30:60
#Banner /etc/issue.net

# Allow client to pass locale environment variables
AcceptEnv LANG LC_*

Subsystem sftp /usr/lib/openssh/sftp-server

UsePAM no
UseDNS no
AllowUsers demo

IP Table Configuration

Just follow the instructions on http://articles.slicehost.com/2008/4/25/ubuntu-hardy-setup-page-1 (iptables section))

Save any existing rules...

iptables-save > /etc/iptables.up.rules

Create a new list of iptable rules...

nano /etc/iptables.test.rules

Paste in the following content...

*filter

#  Allows all loopback (lo0) traffic and drop all traffic to 127/8 that doesn't use lo0
-A INPUT -i lo -j ACCEPT
-A INPUT -i ! lo -d 127.0.0.0/8 -j REJECT

#  Accepts all established inbound connections
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

#  Allows all outbound traffic
#  You can modify this to only allow certain traffic
-A OUTPUT -j ACCEPT

# Allows HTTP and HTTPS connections from anywhere (the normal ports for websites)
-A INPUT -p tcp --dport 80 -j ACCEPT
-A INPUT -p tcp --dport 443 -j ACCEPT

#  Allows SSH connections
#
# THE -dport NUMBER IS THE SAME ONE YOU SET UP IN THE SSHD_CONFIG FILE
#
-A INPUT -p tcp -m state --state NEW --dport 30000 -j ACCEPT

# Allow ping
-A INPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT

# log iptables denied calls
-A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables denied: " --log-level 7

# Reject all other inbound - default deny unless explicitly allowed policy
-A INPUT -j REJECT
-A FORWARD -j REJECT

COMMIT

Update the rules...

iptables-restore < /etc/iptables.test.rules

Check to see that the IP table rules were updated...

iptables -L

Save the rules...

iptables-save > /etc/iptables.up.rules

Make sure the rules are applied after a reboot...

nano /etc/network/interfaces

Add the single line after just after 'iface lo inet loopback'

...
auto lo
iface lo inet loopback
pre-up iptables-restore < /etc/iptables.up.rules

# The primary network interface
...

Reload the SSH configuration...

/etc/init.d/ssh reload

* Reloading OpenBSD Secure Shell server's configuration sshd [ OK ]

Test the login by opening up a new shell, and logging in as the admin users...

ssh -p 30000 demo@11.222.333.444

Note that if you are unable to connect, you may need to remove stored host keys in the ~/.ssh/known_hosts file. Once you verify that you can login as the admin user, you can close the connection that uses the root login.

Updates to the OS

To get a more useful bash shell that the default, edit the ~/.bashrc file...

nano ~/.bashrc

and add the following to the end

export PS1='\[\033[0;35m\]\h\[\033[0;33m\] \w\[\033[00m\]: '
alias free='free -m'
alias update="sudo aptitude update"
alias install="sudo aptitude install"
alias upgrade="sudo aptitude safe-upgrade"
alias remove="sudo aptitude remove"

Reload the configuration changes for the bash shell...

source ~/.bashrc

Update the OS with any released patches...

sudo aptitude update

Set the locale...

sudo locale-gen en_US.UTF-8
sudo /usr/sbin/update-locale LANG=en_US.UTF-8

Update and upgrade...

sudo aptitude safe-upgrade
sudo aptitude full-upgrade

Install the package of essential build programs...

sudo aptitude install build-essential

Ok, now the OS is configured and updated... you are now ready to proceed to Part 3 - installing the software applications.
...

GeoDjango on Slicehost: Installing Software (3 of 4)

This page goes through installing the necessary software applications for running GeoDjango.

Python

Python 2.5 is installed by default with Ubuntu 8.04. Ok, that was easy, lets move on to something else...

Apache HTTP Server

(Ref: http://articles.slicehost.com/2008/4/25/ubuntu-hardy-installing-apache-and-php5)

Install the Apache packages...

sudo aptitude install apache2 apache2.2-common apache2-mpm-prefork apache2-utils libexpat1 ssl-cert python-central libapache2-mod-python

Edit the Apache configuration file...

sudo nano /etc/apache2/apache2.conf

Add this line to the end of apache2.conf (replace 'myslice' with whatever you named your slice)

ServerName myslice

Restart Apache...

sudo apache2ctl graceful
sudo /etc/init.d/apache2 restart

The Apache Web server should have started. Verify this by browsing to http://11.222.333.444
You should see the following message...

It Works!

Subversion

Install the subversion application from the repository...

sudo apt-get install subversion

GeoDjango

Checkout the GeoDjango code...

svn co http://code.djangoproject.com/svn/django/branches/gis ~/django_gis

Setup a symbolic links so that Python knows how to find the GeoDjango code and the shell knows how to find the Django admin script.

sudo ln -s ~/django_gis/django /usr/lib/python2.5/site-packages/django
sudo ln -s ~/django_gis/django/bin/django-admin.py /usr/local/bin

Test that the Django Admin script can be found by entering the following...

django-admin.py help

... which should return a help message.

PostgreSQL

Install the PostgreSQL packages...

sudo aptitude install postgresql python-psycopg2 postgresql-server-dev-8.3 flex

Test to see that PostgreSQL is working...

sudo -u postgres psql postgres




Welcome to psql 8.3.1, the PostgreSQL interactive terminal.

Type:  \copyright for distribution terms
\h for help with SQL commands
\? for help with psql commands
\g or terminate with semicolon to execute query
\q to quitpostgres=# \q

Next we need to alter the default configuration to allow django to connect to the postgres server.

sudo nano /etc/postgresql/8.3/main/pg_hba.conf

Update the access records to the following...

# Database administrative login by UNIX sockets
local   all         postgres                          ident sameuser

# TYPE  DATABASE    USER        CIDR-ADDRESS          METHOD
# "local" is for Unix domain socket connections only
#local   all         all                               ident sameuser
local   all     all     password

# IPv4 local connections:
#host    all         all         127.0.0.1/32          md5
host    all     all     127.0.0.1       255.255.255.255 password

# IPv6 local connections:
#host    all         all         ::1/128               md5

Restart the PostgreSQL server, so that it loads the new access configuration...

sudo /etc/init.d/postgresql-8.3 restart

GEOS, proj4, PostGIS, and GDAL

(Refs: http://trac.osgeo.org/geos/ and http://code.djangoproject.com/wiki/GeoDjangoInstall)

First, make a directory for downloading source code...

mkdir ~/downloads

The following commands download, build, and install the GEOS, proj4, PostGIS, and GDAL libraries. Run them all at once if you are feeling lucky, or one at a time if you want to watch the messages. Time to get a cup of coffee... this takes a while.

cd ~/downloads
wget http://geos.refractions.net/downloads/geos-3.0.0.tar.bz2
tar xjf geos-3.0.0.tar.bz2
cd geos-3.0.0
./configure
make
sudo make install
 
cd ~/downloads
wget ftp://ftp.remotesensing.org/proj/proj-4.6.0.tar.gz
wget ftp://ftp.remotesensing.org/proj/proj-datumgrid-1.3.tar.gz
tar xzf proj-4.6.0.tar.gz
cd ~/downloads/proj-4.6.0/nad
tar xzf ../../proj-datumgrid-1.3.tar.gz
cd ~/downloads/proj-4.6.0
./configure
make
sudo make install
 
cd ~/downloads
wget http://postgis.refractions.net/download/postgis-1.3.3.tar.gz
tar xzf postgis-1.3.3.tar.gz
cd postgis-1.3.3
./configure
make
sudo make install
 
cd ~/downloads
wget http://download.osgeo.org/gdal/gdal-1.5.1.tar.gz
tar xzf gdal-1.5.1.tar.gz
cd gdal-1.5.1
./configure --datadir=/usr/local/share
make
sudo make install

The GEOS library is now available in /usr/local/include/
The Proj4 files are now in /usr/local/bin/proj/

Update the shared library configuration file...

sudo nano /etc/ld.so.conf

and add the following line to the end

/usr/local/lib

Update the shared libraries...

sudo /sbin/ldconfig

Test that GDAL is installed correctly...

gdalinfo




Usage: gdalinfo [--help-general] [-mm] [-stats] [-nogcp] [-nomd]
[-noct] [-checksum] [-mdd domain]* datasetname

Test that Python, Django, GDAL, and GEOS are all working together

Start up Python...

python




Python 2.5.2 (r252:60911, Apr 21 2008, 11:17:30) 
[GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

Check to see if Python can use the GDAL library...

>>> from django.contrib.gis.gdal import HAS_GDAL
>>> print HAS_GDAL




True
>>> from django.contrib.gis.tests import test_gdal
>>> test_gdal.run()




.......................
BEGIN - expecting IllegalArgumentException; safe to ignore.

ERROR 1: IllegalArgumentException: points must form a closed linestring

END - expecting IllegalArgumentException; safe to ignore.

......................
----------------------------------------------------------------------
Ran 45 tests in 0.094s

OK

Now check to see if Python can use the GEOS library...

>>> from django.contrib.gis.tests import test_geos
>>> test_geos.run()




Testing WKT output. ... ok
Testing HEX output. ... ok
Testing KML output. ... ok
Testing the Error handlers. ... 
BEGIN - expecting GEOS_ERROR; safe to ignore.

GEOS_ERROR: ParseException: Expected number but encountered ','
GEOS_ERROR: ParseException: Unknown WKB type 255

END - expecting GEOS_ERROR; safe to ignore.

GEOS_ERROR: ParseException: Unexpected EOF parsing WKB
ok
Testing WKB output. ... ok
Testing creation from HEX. ... ok
Testing creation from WKB. ... ok
Testing EWKT. ... ok
Testing GeoJSON input/output (via GDAL). ... ok
Testing equivalence with WKT. ... ok
Testing Point objects. ... ok
Testing MultiPoint objects. ... ok
Testing LineString objects. ... ok
Testing MultiLineString objects. ... ok
Testing LinearRing objects. ... ok
Testing Polygon objects. ... ok
Testing MultiPolygon objects. ... 
BEGIN - expecting GEOS_NOTICE; safe to ignore.

GEOS_NOTICE: Duplicate Rings at or near point 60 300

END - expecting GEOS_NOTICE; safe to ignore.

ok
Testing Geometry __del__() on rings and polygons. ... ok
Testing Coordinate Sequence objects. ... ok
Testing relate() and relate_pattern(). ... ok
Testing intersects() and intersection(). ... ok
Testing union(). ... ok
Testing difference(). ... ok
Testing sym_difference(). ... ok
Testing buffer(). ... ok
Testing the SRID property and keyword. ... ok
Testing the mutability of Polygons and Geometry Collections. ... ok
Testing three-dimensional geometries. ... ok
Testing the distance() function. ... ok
Testing the length property. ... ok
Testing empty geometries and collections. ... ok
Testing `ogr` and `srs` properties. ... ok
Testing use with the Python `copy` module. ... ok
Testing `transform` method. ... ok
Testing `extent` method. ... ok
Testing pickling and unpickling support. ... ok

----------------------------------------------------------------------
Ran 36 tests in 0.246s

OK

If the tests ran correctly, you now have GeoDjango installed. You can now proceed to Part 4 - Creating a GeoDjango project.