Oct 302011
 
Moving Servers

Moving servers from Strato (can’t complain, good enough service, nothing ever broke, few but regular planned outages, and not expensive either), to Linode (quite a bit more expensive for the same performance, but less performance is ok too, so it’s about the same price). The main advantage of Linode is the location: Tokyo.

Main differences:

  • Location Germany vs Japan
  • Ping round trip from home decreased from 310ms to 30ms (and 18ms is my ADSL line already)
  • CPU changed from 1 core Opteron 2347 HE to 4 core Xen L5520 (but in reality it’s only 1 core anyway, however the new one is about 3 times faster according to http://www.webhostingtalk.com/showthread.php?s=&threadid=308055)
  • RAM decreased from 2GB to 0.5GB (but I only use about 0.4GB anyway)
  • Disk size decreased from 80GB incl. backup plus 80GB ftp, to 20GB and no backups
  • Data traffic decreased from 3TB to 200GB (but I never crossed 100GB)
  • Virtualizer changed from Virtuozzo to Xen
  • I now have some transparency about oversubscription of the underlying server
  • DNS is totally different: Strato uses a simple wildcard scheme: anything with *.studiokubota.com is pointing to the server. Linode uses a more traditional scheme where I need to define those DNS entries manually. But now I have control about it and can point hosts in my domain to any IP I like.
  • IP addresses decreased from 2 to 1 (but I used the extra IP only for VPN)
  • I need to use another registrar (I actually don’t need to, but it’s much cheaper if I do)

As you can see, it’s a mixed bag, but I am, generally ok although I’ll lose the ability to do proxy services in Germany, which allowed me to see some content some companies like to prohibit me from watching (streaming movies).

As a reminder to myself, here the additional commands/configs I had to do to move the server with all stuff I need:

# This is pretty much copy & paste. Run as root on the new server.
OLD=the.old.server
NEW=the.new.server
MYSQLPASSWORD=THEMYSQLPASSWORD

# User accounts

# Add encrypted password
echo >>/etc/passwd "harald:*:2000:100:Harald Kubota:/home/harald:/bin/bash"
# Copy root .ssh directory from old server

cd ~root

rsync -av $OLD:~root/.ssh .
# Install packages
aptitude update
aptitude full-upgrade
aptitude install locate
aptitude install apache2 imagemagick openvpn  gcc rsnapshot
aptitude install libapache2-mod-perl2 libapache2-mod-php5 libapache2-mod-perl2-doc
aptitude install phpmyadmin bsd-mailx
aptitude install postfix spamassassin procmail dovecot-common dovecot-imapd
# Configure postfix as Internet site with its local name as hostname

aptitude install mysql-server

# Set mysql "root" password, do not initialize the DB beside creating it

# Installing MT5 support files

aptitude install php5

aptitude install bsd-mailx exim4 libsoap-lite-perl libdbi-perl libimage-size-perl \
libmime-charset-perl libmime-encwords-perl perlmagick libxml-libxslt-perl \
 libxml-simple-perl libxml-libxml-perl libmime-tools-perl
aptitude install libgd-gd2-perl libio-compress-zlib-perl libarchive-any-perl
# copy /var/www

rsync -avz --progress $OLD:/var/www/ /var/www/

# For test purposes, create a symbolic link to one of the existing web pages

cd /var/www
ln -s harald.studiokubota.com $NEW

# Copy apache config

rsync -av $OLD:/etc/apache2/ /etc/apache2/

# Copy backup job from /etc/cron.daily/

rsync -av $OLD:/etc/cron.daily/mysqldump /etc/cron.daily/

# Mail: postfix, dovecot, spamassassin

rsync -av $OLD:/etc/postfix /etc/
# Modify main.cf to contain all domain names and fix the hostname too
# Also add virtual names and rebuild via
postmap /etc/postfox/virtual

rsync -av $OLD:/etc/dovecot /etc/
rsync -av $OLD:/etc/spamassassin /etc/
rsync -ac $OLD:/etc/default/spamassassin /etc/default/

mkdir /var/log/spamassassin
groupadd spamd
useradd -d /var/log/spamassassin -M -g spamd -s /bin/false spamd
chown spamd:spamd /var/log/spamassassin

# To learn ham/spam by putting spam in Spam and ham in Ham, add those to your user crontab:
#10 19 * * * sudo sa-learn --spam -u spamd --dir /home/harald/Maildir/.Spam/* -D
#10 20 * * * sudo sa-learn --ham -u spamd --dir /home/harald/Maildir/.Ham/* -D

# On both servers stop spamassassin and copy the DB over
ssh $OLD "service spamassassin stop"
service spamassassin stop
rsync -av $OLD:/var/log/spamassassin .
# Start spamassassin again
ssh $OLD "service spamassassin start"
service spamassassin start
service postfix restart
service dovecot restart

# Restore mysql DB

d=`date +%d`
cat >/tmp/dump.sh <<EOF
FN="/var/tmp/dbdump-${d}.bz2"
rm -f "\$FN"
mysqldump -p$MYSQLPASSWORD --all-databases | bzip2 -9 >\$FN
EOF
scp /tmp/dump.sh $OLD:/var/tmp/
ssh $OLD "bash /var/tmp/dump.sh"

# Restore on NEW server

rsync -av --progress $OLD:/var/tmp/dbdump-${d}.bz2 /var/tmp/
bunzip2 </var/tmp/dbdump-${d}.bz2 | mysql -u root -p
mysqladmin -p$MYSQLPASSWORD flush-privileges

# Other things

rsync -av --progress $OLD:/etc/openvpn /etc/
rsync -av --progress $OLD:/etc/default/openvpn /etc/default/

Files to modify manually later:

  • /etc/openvpn/openvpn.conf
  • some more files in /etc/openvpn to create/configure (it’s obvious), and adjust the files on the VPN client too
Feb 112011
 
XBee - Again

xbee-python is out in version 2.0.0 which officially supports the ZigBee (Series 2) version of the Digi XBee modules. Thus time to update my test programs to this new library.

Here the result:

#!/usr/bin/python

# This is a simple demo to remotely turn a LED on and off
# 2011-02-11 Harald Kubota

import serial
from xbee import ZigBee
import time

PORT='/dev/ttyUSB0'
BAUD_RATE=9600
ser = serial.Serial(PORT, BAUD_RATE)

# ZB XBee here. If you have Series 1 XBee, try XBee(ser) instead
xbee=ZigBee(ser)

#MAC, number written on the back of the XBee module
# CO3 = my coordinator
# EP1 = my endpoint with the LED on pin 11
device={
        "CO3":'\x00\x13\xa2\x00\x40\x52\x8d\x8a',
        "EP1":'\x00\x13\xa2\x00\x40\x4a\x61\x84'
}
#64 bit address
led=False

#change remote device function
xbee.remote_at(dest_addr_long=device["EP1"],command='D2',parameter='\x02')
xbee.remote_at(dest_addr_long=device["EP1"],command='D1',parameter='\x03')
xbee.remote_at(dest_addr_long=device["EP1"],command='IR',parameter='\x04\x00')
xbee.remote_at(dest_addr_long=device["EP1"],command='IC',parameter='\x02')

while 1:
        #set led status
        led=not led
        if led:
                xbee.remote_at(dest_addr_long=device["EP1"],command='D4',parameter='\x04')
        else:
                xbee.remote_at(dest_addr_long=device["EP1"],command='D4',parameter='\x05')
        # wait 1 second
        time.sleep(1)

ser.close()

Time to learn Python. I was mostly guessing my way through the Python code of this library. Luckily it’s far easier to read and understand than some other languages. Maybe I am just not a friend of putting $ signs in front of variables.

Jan 102011
 
WLAE-AG300N Ethernet Bridge

I bought recently 2 of those to replace a wireless connection I used before (see here for the details). I wanted to write a bit about those as they have some quirks.

Pros:

  • Quite small and light.
  • Work as wireless bridge, access point and wireless repeater.
  • Has 2 Ethernet (10/100 MBit/s) ports for 2 devices, or more if connected to switches.
  • Works in the 2.4 and 5 GHz band, so you can turn on microwaves if you operate in the 5 GHz band.
  • AOSS/WPS works as advertised: you push buttons on them, and they find each other. For bridging, it’s super-simple. Default is 5 GHz band too.
  • Operation is transparent as it should be.

Cons:

  • Those are not routers and cannot do NAT.
  • The web interface is in Japanese if you buy them in Japan, and the corresponding US model has yet no firmware to download. Good thing I have Google Translate to help me with most of the text.
  • They can only use either 2.4 or 5 GHz. Important when they work as wireless repeaters.

Quirks:

  • Default IP is 192.168.11.100. They also use DHCP if they can. I should move to another network.
  • Changing the channel is not simple. I still don’t know how to do that. Seems that this is no longer possible if I have DFS. I cannot find a way to turn this off, but then, I don’t need to as it picked the correct band anyway.
  • The docs from the Buffalo US site shows small differences between the US and JP firmware. It’s not just translated. But the English manual is a big help.
  • That manual also writes:
    This sticker shows the AirStation’s SSID, default encryption key, and WPS PIN code.  By default, encryption is disabled for AirStations sold in Asia
    That explained why initially the wireless LED was blinking orange, as that manual also said: Blinking :  AOSS/WPS error. After setting a password, the blinking became a solid green (which means: 5 GHz wireless).
  • I have 2 and both are configured identical. Using AOSS buttons, one became a master and the other one a client. I don’t know which one is which though.
  • I set up fixed IPs on both. On one of them it works just fine: no DHCP requests. On the other one, constant DHCP requests. The DHCP server I have offers a good one (identical to the one I configured manually), but it’s not being accepted. At the same time, that unit works just fine with the assigned manual IP address. When I configure it to use DHCP, it immediately grabs one and is happy with it. Looks like a bug to me.
Jan 102011
 
OpenVPN Setup

My provider YahooBB!, who does otherwise a good job, blocks outgoing SMTP (25/tcp) traffic. That is generally a good thing since it limits spam caused by infected home PCs, but you cannot opt-out even if you know what you are doing and have no infected PCs, and a need to send out email.

Sending out emails via email client is no issue of course. But sendmail/postfix/etc. need a relay and the YahooBB! relay only accepts emails with its own email address. I don’t even use that one.

Well, there is always a technical solution for a technical problem. In this case: my virtual server in Germany. That one can send out emails just fine, so all I need to do, is to let it relay my emails from home. I just need a VPN connection from home to that machine.

I wanted to do this many times anyway, so time to do this now.

One problem I did not expect: The virtual server I use uses OpenVZ (or something similar), which does not allow or have a tap interface for bridging. So I need to use tun (which luckily exists), and use a tunnel and a P2P or generally a routed access. Browsing along with Google searching for a rather simple example for a network-to-client connection, it turned out to be not that easy and my router which is using OpenWRT, needs the OpenVPN package first. So I build a new flash image…and lost Internet access completely. Router simply not responding anymore. It can be de-bricked via TFTP, but given the limited time I have, no time for that, so I replaced it with another spare router which already has the OpenVPN package installed. Configuring the router part was simple. Wireless performance sucked though and in the system log I found a lot messages like:

ath: DMA failed to stop in 10 ms


causing patcket loss of about 15% overall. So I ordered those. With the laggard Internet connection I now had for the time being I figured out that OpenWRT does not do a good job of making it easy to create a network-to-client VPN. Me never having done this (I used OpenSWAN long time ago, but apparently too long to remember much of it), did not help here either.

So in the end I used a point-to-point tunnel from my virtual server in Germany to my file server at home.

Turned out to be surprisingly simple with those really nice instructions I found here. While the example is for CentOS and I use Debian, but that’s easy to fix. Small minor fixes on the script side were needed. For my and everyone’s benefit, here my instructions which heavily lean on the ones from John Malkowski from vpsnoc.com.

# Quick and dirty OpenVPN install script
# Tested on Centos 5.x 32bit, openvz minimal CentOS OS templates
# Please submit feedback and questions at [email protected]
# John Malkowski vpsnoc.com 01/04/2010
# Adjusted and used as an example for Debian by Harald Kubota 2011-01-09

ip="the.ip.address.on.the.server"

aptitude install openvpn

cd /etc/openvpn/

cp -R /usr/share/doc/openvpn/examples/easy-rsa/ /etc/openvpn/
cd /etc/openvpn/easy-rsa/2.0/
chmod +rwx *

# Edit vars here. I like emacs.

emacs vars

. ./vars
./clean-all
source ./vars
echo -e "\n\n\n\n\n\n\n" | ./build-ca
./build-key-server server
./build-dh
cp keys/{ca.crt,ca.key,server.crt,server.key,dh1024.pem} /etc/openvpn/

# Repeat for every client, you can choose more meaningful names too

for i in client1 client2 client3 ; do

./build-key $i
cd keys/

client="
# client $i
log /tmp/openvpn.log
verb 3
client
remote $ip 1194
dev tun
comp-lzo
ca ca.crt
cert $i.crt
key $i.key
route-delay 2
route-method exe
#redirect-gateway def1
#dhcp-option DNS 10.8.0.1"

echo "$client" > $HOSTNAME.conf

tar czf $i-keys.tgz ca.crt ca.key $i.crt $i.csr $i.key $HOSTNAME.conf
mv $i-keys.tgz /root

done

opvpn='
log /tmp/openvpn.log
verb 3
dev tun
server 10.8.0.0 255.255.255.0
ifconfig-pool-persist ipp.txt
ca ca.crt
cert server.crt
key server.key
dh dh1024.pem
push "route 10.8.0.0 255.255.255.0"
#push "redirect-gateway"
comp-lzo
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
group nogroup
daemon'

echo "$opvpn" > /etc/openvpn/openvpn.conf

# Enable IP forwarding
echo 1 > /proc/sys/net/ipv4/ip_forward
#iptables -t nat -A POSTROUTING -s 10.8.0.0/24 -o venet0 -j MASQUERADE
#iptables-save > /etc/sysconfig/iptables
#sed -i 's/eth0/venet0/g' /etc/sysconfig/iptables # dirty vz fix for iptables-save
#echo "net.ipv4.ip_forward=1" >> /etc/sysctl.conf

/etc/init.d/openvpn start


That was the server part. For the client, copy the (e.g.) client1-keys.tgz package and extract into /etc/openvpn/ on the client.
A simple /etc/init.d/openvpn start will then initiate the VPN connection.

Note that the default gateway is not changed. This is a strict P2P connection. Default traffic will be not re-directed. If you want to redirect all traffic to the VPN server, then un-comment the redirect-gateway related messages. You have most likely have to set up NAT on the VPN server too. Since this is not needed in my case, I leave this as an exercise to the reader.

Jan 092011
 
Wuala on a Headless System

From my colleague Soren I heard first about Wuala. Did sound interesting back then. But I never found a use for it. I have a total of about 30G online storage space and hardly know how to use it. Too complex to connect to and too slow to use.

Recently when looking at PogoPlug Pro model, I found it quite intriguing as now I have a mobile Android device which could use it well. However The PogoPlug cannot be bought (via normal channels) in Japan. Or even sent to Japan. Curse you Amazon!  Buffalo to the rescue! Except theirs is not yet available. Looks promising though as it includes 2 disks already. I’ll look into this when it’s available.

If anyone wonders, what PogoPlug and Wuala have in common, then stop wondering. Not much beside the “storage” and “cloud”. They solve different problems.

Anway, that’s why I looked into Wuala again. And hey, it works quite well. Integration with Linux is great Shows up as a mounted filesystem. Speed is slow though. Time to upgrade that old ADSL into something more speedy.

Now the important message of this posting: How to make Wuala run on a headless server. The solution was found here and here. I used the latter. Works as advertised and starts up nicely in the background via screen.

What I am waiting for now is to connect to that Wuala instance from my notebook (see here in the Wuala forum).

Aug 062010
 
USL5P - It still lives

I bought one of those some time ago (about 2005). It’s basically a small computer with a funny CPU (SH4, 266 MHz), RAM (64 MB) network (Fast Ethernet) and 5 USB 2.0 ports, 4 buttons, and some LEDs. Inside is a CF card for the OS, and originally you are supposed to connect USB storage devices which are then exported via Samba. Since it was basically a LANTANK with the internal IDE disk(s) replaced with a CF card, it did not take long until it was hacked. All you needed to do was to take out the CF card, modify some files (e.g. set a root password and enable telnet), and put it back. Suddenly you had a cut down accessible Debian. At first I used Gentoo (specifically from here) but when Gentoo broke on my desktop (trying to replace the shared glibc with another, newer and incompatible one on a running system is bad), I jumped to Ubuntu (Kubuntu) on my desktop and and plain Debian (or Ubuntu Server) on my servers.

So some month ago I updated it to the latest Debian release (unstable/unreleased) as the previous installation was from 2007 and updates were no longer available.

Here some pointers where to find useful information about this device and how to configure it to be on Debian sid/squeeze:

/etc/apt/sources.list should contain:

deb http://ftp.debian-ports.org/debian/ unstable main
deb http://ftp.debian-ports.org/debian/ unreleased main
And if anyone wonders why I would use such a slow machine: It runs on 5V, using max 2.2A (and that’s mostly for the USB ports which can draw 0.5A each). It also has no moving parts, it’s really small, and it just works (mostly thanks to Debian’s SH4 port). And it can do anything Linux can do when it comes to networking and USB.
To put some numbers to the “slow”: compiling a recent Linux kernel takes 8h.
A more modern similar machine would be this.
Dec 192009
 
MZK-W04NU OpenWRT

When you have one of those routers like the Planex (AKA PCi) MZK-W04NU, the first thing to do if to upgrade the firmware to an ‘open’ version, so it does what you want it to do, not what the manufacturer had in mind.

In the case of the above router it has WiFi 802.11n, a 4 port switch and one more NIC and one USB port. That’s enough to make it do a lot of things: router, wireless router, or bridge, or firewall, or media server, or NFS/CIFS server, or print server. You are limited by the CPU (MIPS 4k@400MHz) and RAM (32 MByte).

Step one is to update the firmware to something else, e.g. OpenWRT.

In the end you will have one way or another a “webui” file which you can use to update the firmware via the firmware update link in the web interface.

Once you have OpenWRT installed, it’s easier: you need a “bin” file (about 7.5MB in size), and upload it via:

mtd -r write openwrt-ar71xx-mzk-w04nu-squashfs.bin firmware

Then wait (it takes about 3min), it will reboot, and there you go: a new firmware.

Home Automation

 Uncategorized  No Responses »
Sep 292008
 
Home Automation

If you did not know, the purpose of using this microcontroller is to create something like this: It’s an alarm clock which turns on the light gradually, simulating the light levels similar to a sunrise. Everyone knows it’s easy to wake up in the morning in summer. I hope make me wake up easily in winter too.

Philips does not sell that one in Japan (and not in US either), and amazon.de does not send electronic stuff outside the German speaking countries, so I’m on my own here. National sells something similar for even more money, but where’s the fun if you buy ready-built stuff…

While planning this project, I set my eye on the larger picture: complete home automation. Controll all light and power sockets from a central place. Have sensors like humidity/temperature/touch to tell a computer what to do. Like: at 6am gradually turn on the light in the bed room. If it’s cold, then turn on the heater in bed and living room. When I leave the bed room, turn off light and heater shortly later. When I leave the house, turn off all lights and all heaters. Would work fine in a single household. In my case, it would not work at all though. But the point is, I could make that.

The question now is: how to connect all those sensors and output devices. 4 choices I see:

  • use extra cables
  • use existing cables
  • use wireless (radio)
  • use wireless (IR)

Using new cables is not very attractive as this is rather ugly. There’s no raised floor in our appartment, so using
EIA-485 or cabled Ethernet is out for now.

Using existing (power) cables would be good if there would exist something easy to use. Something like X10. It communicates via power lines, but has problems with reliability in modern households and when more than one power phase is in use. I could not find anything for Japan. Maybe not possible to use or not allowed to use.

One could use Ethernet-over-power-lines. Maybe using one Ethernet connected controller per room would be ok, but some rooms in this apartment we live in are rather large, so the problem of connecting a light on/off switch still exists as I would need to run a cable from the switch to the microcontroller. But the amount of cables would be far smaller than if everything would have a cable go to a central patch room.

IR has the problem of not being able to cross rooms, so a lot of repeaters would be needed (1 per room).

Wireless does sound expensive. Wi-Fi would be neat as it interfaces well to any computer, but it’s expensive if you have a simple on/off switch and want to automize it: You’d need Ethernet (most small microcontrollers don’t have it), and a wireless bridge. Those are usually quite large and need their own power. I count about 20 lamps and about 24 switches. That’s a lot of Wi-Fi going on. Bluetooth would be better, but the Bluetooth stack is quite complicated and ready-to-use modules which take out the complexity are quite expensive.

An alternative I heard about years ago is
ZigBee: a mesh capable wireless network in the (crowded) 2.4GHz range, with slow transmission rate, but cheap to implement. “Cheap” is what they mentioned for Bluetooth too, but this at US$65 is not what I call cheap. 44 of those (20 lamps and 24 switches) plus the microcontroller to talk to it.

ZigBee is now called IEEE 802.15.4. That means it’s hopefully not going to die too soon. The best part is: at about US$25 (see here as an example) it’s much cheaper than e.g. Bluetooth.

And now the best part: those XBee modules, which implement ZigBee and offer:

  • a serial interface with either a pass-through-mode or a more capable API mode
  • 10 digital I/O pins
  • 6 ADC channels
  • PWM output even
  • runs on 3.3V as the LPC2148 does
  • it’s small
  • it’s comparably cheap

Most switches or sockets don’t need anything but a single I/O pin, so in my example of 20 lamps and 24 switches, it’ll be 44 of those modules. If sockets/switches are physically near, then they can be combined, reducing the number likely by one third. That would be at US$25 per piece, US$750 for 30 of them. Quite doable.

I’ll get some of those modules and see what they can do.

Jun 112008
 

It works with Linux!

Making it work involves 2 things:

  1. Setting up /dev/rfcomm, which is setting up /etc/bluetooth/rfcomm.conf (and restarting the bluetooth stack) and
  2. Setting up the configuration for the modem.

In the latter case I used KPPP and it was very simple. The only issue I had was using 921600 bit/s is too fast and no modem is found. using 460800 bit/s was ok though.

May 182008
 

It works! Took a long time to figure out, but this is how to make a N800 (or Windows machine) connect to the Internet via Bluetooth via an emobile H11T:

  • Phone number is *99#
  • Account is em, password em
  • Do not turn on any compression
  • The last one is critical: Windows does not do this, so it works. The N800 turns it on by default, so it fails. MacOSX turns it on too (see http://www.taniwha.org.uk/) which is how I found this out. Usually if compression is not supported, it should be negotiated during the PPP handshake. But that’s not how it works in real life.

    It took me about 5 calls to the phone support of emobile to find this out. When they just wanted to replace the phone with a new one, the last test was to reset all settings and try again, and then it worked finally.

    The issue which made this experience painful is: if you try to connect with compression on, then the phone will not allow any connections any more. To fix this, you need to reset your phone. Which means losing all your data, time setting, network settings, Bluetooth pairings etc.