Saturday, 4 October 2014

Experiments with Pexpect

The Pexpect python module is used to start up and control new processes from python code.  For the Manchester CoderDojo I needed to incrementally get output from a spawned process over the course of a long run.  To test this I created a simple module that counted to 20 slowly:


"""
Count to 20 in one second steps
"""
from time import sleep

for i in range(1,21):
    print(str(i))
    sleep(1)

Next I created a python module to spawn off a process to call the counter and print the output.

"""
Call the slow-running counter and print the results.
"""

import pexpect

output = pexpect.run('python count-slow.py')
print(output)

The problem of course is that the run command blocks until all the output is produced.  The solution allowing you to  see the output as it appears is to use the pexpect.spawn() function.  This function to creates a sub-process object from which you can read one line at a time.  Pyexpect.readline() will block until it receives a line.  An empty string indicates the process has completed.

"""
Call the slow-running counter and print the results.
"""

import pexpect

child = pexpect.spawn('python count-slow.py')
next_line = child.readline()
while next_line != '':
    print(next_line.strip())
    next_line = child.readline()

Formatting Python Code for the Web with Pygments

I've been unsatisfied with Blogger's lack of easy handling for code formatting.  To fix this I installed Pygments.  I can now type,

pygmentize -f html -O noclasses count-slow.py

...and end up with output that I can paste into the Blogger HTML editor window.

The result is something like this:

"""
Count to 20 in one second steps
"""
from time import sleep

for i in range(1,20):
    print(str(i))
    sleep(1)




Friday, 3 October 2014

Vagrant Working for the Manchester CoderDojo Minecraft Server Build

I've got a basic vagrant setup working for the Manchester CoderDojo Minecraft server build.  Instructions on the GitHub page.

Next task is to get the chunked responses working so that you can see the python output appear in the browser as the code is running.

Tip: Copying Large Files from Remote Desktop Sessions

I just had the problem of having to grab a large file from a Windows remote desktop session.  The usual cut-remote-paste-local magic wasn't going to manage the file size.  It turns out that RDP maps "\\TSCLIENT\" on the remote server to your local PC.  You can therefore use robocopy to bring the large file over locally using the restart mode in case of problems. 

For example to grab "big file.zip" from the current folder on the remote box to the c:\tmp folder locally run this:

robocopy /z /eta .  \\TSCLIENT\C\tmp "big file.zip"

Monday, 29 September 2014

Vagrant and Docker Workflow

I've recently been looking into using both Vagrant and Docker to develop my projects.  This is the work flow that I've settled on...

  • I'll use Vagrant to support my development work
  • I'll use Docker as the mechanism for getting my projects hosted
  • Start a project with a Vagrant set up file that will launch a Debian vm and apt-get Docker installed on it
  • The Vagrant VM is used to build the code
  • The result of of the development code is to build a Docker image of the project
  • The Docker image is what is then used to deliver the project's services

As an example I created vagrant-docker-flask on GitHub to show how this works.

  1. Clone the git repo: git clone https://github.com/davegoopot/vagrant-docker-flask.git
  2. In the cloned directory run:  vagrant up
  3. You now have the dev machine image installed that you can connect to using: vagrant ssh
  4. Inside vagrant run: cd /vagrant/flaskserver/
  5. Then build the flask server image with: sudo docker build -t="local:flask" .
  6. Start up a container from the image with:  sudo docker run -d -p 5000:5000 local:flask python demo.py
  7. The flask server will run and be accessible from the original host machine at http://localhost:5000/
  8. To work on the code, you can now edit flask/demo.py, run the docker build and the docker run command again and see your changes

Monday, 5 May 2014

Making the Raspberry Pi Email IP on Boot

I needed my Raspberry Pi to email its IP address to me when it booted in order to ease finding the IP address when running headless.  Here's how I did it:

  1. There are some really good instructions here: http://elinux.org/RPi_Email_IP_On_Boot_Debian
  2. In order for it to work I had to set up an application password for gmail since I'm using two-factor authentication to login
  3. Reboot and it sent me an email.  That was painless!

Twilio Interface for the Rube Goldberg Machine

Here's how I built the Twilio interface to the Rube Goldberg machine...

  1. Install the Twilio python libraries: sudo pip install twilio
  2. Set up a feature branch for Twilio: git checkout -b twilio-plugin
  3. Set up tests and code to load credentials from twilio.secret file
  4. From the secret file load the account_sid and auth_token
  5. Write the update_state() method to send an SMS with Twilio
  6. Decided not to unittest the SMS sending as this is just simple interface code with no logic
  7. Used the python code documented at:  https://www.twilio.com/user/account/developer-tools/api-explorer/message-create
  8. With a bit of Blu Tack, and some dominoes the system worked: http://youtu.be/Vj7IvwIN0r4

Sunday, 4 May 2014

Fixing Anki Flashcard Dates

I've been using Anki for a while to learn things.  Recently I made the mistake of editing some existing cards to have a new meaning.  The problem was that the algorithm for when the card would be next presented to me did not reset itself on the edit.  I was left wanting to see the edited card, but Anki telling me that I'd have to wait for three months first.

Here's how I fixed it:

  1. Installed the full PC version of Anki:  sudo apt-get install anki
  2. Connect the full PC version and sync up the deck in question
  3. Realise that the Debian stable version of Anki is out of date to work with the server
  4. Set up Apt-Pinning for Anki:
    1. Add a testing entry to /etc/apt/sources.list
    2. Edit /etc/apt/preferences and set it up like this
    3. Package: *
      Pin: release a=stable
      Pin-Priority: 900
      
      Package: *
      Pin: release a=testing
      Pin-Priority: 600
    4. Run aptitude, update
    5. Find the anki package in the aptitude UI
    6. Pick the 2.0... version of Anki and install it -- being careful to make sure aptitude picks the correct version
  5. With the PC client running click "browse"
  6. Select the required deck
  7. Select all the cards
  8. Do Edit->Reschedule then set the schedule for zero days
  9. Close the browser and reopen the deck, all the cards are now ready for review
  10. Sync back up to the server and back down the phone again

Monday, 28 April 2014

Taking Stock of Rube Goldberg Branches

Doing a quick review of the Rube Goldberg branches in git:

  • Master - Needs pylint fixing
  • file-plugin - work complete.  Deleted
  • gpio-impl - work complete.  Deleted 
  • http-plug - work still to do.  Merge conflicts from master.  Added Trello item
  • pip-packaging -- work still to do in order to figure out how to use from pip install only. Merged master.  Added Trello
  • scratch-plug - work complete. Deleted. Trello for sorting out tests

Sunday, 27 April 2014

Working on the Minecraft-Scratch Interface for the Rube Goldberg Machine

For the Rube Goldberg project at the CoderDojo I need a way to interact with Scratch from Python.  Here's my steps of how I went about it:

  1. After reading around I realised that I'd have to run Scratch 1.4 and the Python interaction would come from scratchpy - https://github.com/pilliq/scratchpy
  2. Installed scratch 1.4 on Windows
  3. pip install scratchpy
  4. Enabled Scratch remote sensing - http://wiki.scratch.mit.edu/wiki/Remote_Sensor_Connections#Enabling
  5. Played with the interaction - http://youtu.be/YG99uam8pKA
  6. Create new branch for scratch plugin git checkout -b scratch-plug
  7. Check all tests pas
  8. python setup.py test
  9. Tests failed as wasn't on Rasp PI.  After some poking around I discovered that I'd installed the GPIO package on to my development virtual box environment, and so now the package was loading okay and the case that I had set up in the test file to spot the lack of GPIO wasn't working.  I deleted out the GPIO package and it all worked again...phew!
  10. Making the scratch target -- use a broadcast of either "BLOCK-PRESENT" or "BLOCK-ABSENT"
  11. Create a scratchplug.py module
  12. Implemented the target code
  13. Tested that it worked using a file source
  14. Not quite sure how to make the unit tests work.  Will park that problem for now and come back to it later
  15. Now to make the source method
  16. The problem is that the scratch.receive() method blocks until it receives a broadcast, but the Rube Goldberg architecture needs to poll
  17. The solution will be to use the multiprocessing module as shown in this answer on Stack Overflow - http://stackoverflow.com/questions/492519/timeout-on-a-python-function-call
  18. Got a bit stuck coordinating the output with Queues since I didn't realise that multiprocessing had its own implementation of Queue -- you live and learn
  19. Committed that on to the scratch-plug and pushed to GitHub
  20. Made a quick demo of the Minecraft light going into scratch and scratch button turning on a light in Minecraft - https://www.youtube.com/watch?v=cpD-RegWybA

Sunday, 23 February 2014

Python Minecraft CoderDojo -- Play Testers Sought

I've updated the sheets for the Manchester (UK) CoderDojo Python-Minecraft project.  I'm looking for play-testers.  If you are interested, take a look at this on the Minecraft Forum:

Friday, 21 February 2014

Jitsi VOIP Client DTMF Settings

It took me ages to find out how to modify the Jitsi DTMF settings today.  In case you have the same problem, here is a pictorial guide:












Monday, 17 February 2014

The Corn Laws

The Corn Laws where tariffs imposed on the import of corn to the UK from 1815 to 1846.  The intention of the tariffs were to protect the interests of corn producing landowners.  While landowners benefited from the protection of their markets, low-paid workers experienced poverty and famine from having to spend a large proportion of their income on bread.  The struggle to repeal the Corn Laws led to the collapse of the Conservative government and the resignation of Robert Peel as Prime Minister.  Following their repeal corn prices fell in the UK resulting in the reduction in the amount of land dedicated to cereal production and a similar reduction in the demand for agricultural labour.

Thursday, 13 February 2014

Mainline vs. Trunk Development

In software source control (git, mercurial, etc.) you branch the code base in order to isolate different development efforts from one another. In a "mainline" source control model fixes are made on the branches and then merged into the mainline. In a "trunk" source control model fixes are made on the single trunk and then pulled into the branches as needed for a release.
Paul Hammant provides a nice picture of this in which he expresses his preference for the trunk model here: http://paulhammant.com/images/mainline_vs_trunk.png

Monday, 10 February 2014

The Pigeonhole Principle

The Pigeonhole Principle is a counting argument method of mathematical proof.  It says that if you have more items to distribute than pigeonholes to put them into then at least one of the pigeonholes must contain more than one item.  This simple principle becomes interesting when you extend it to countably infinite sets, e.g. the natural numbers.  See: http://en.wikipedia.org/wiki/Hilbert%27s_paradox_of_the_Grand_Hotel

Friday, 10 January 2014

Five Sentences a Day

In order to have confidence that you have learned something it is a good idea to write something about it. I'd like to learn something new each day. But, who has the time to write an essay each day? Combining the idea of http://five.sentenc.es/ with the desire to write something each day I hereby declare I'm going to write five sentences, or fewer, each day.

Wednesday, 24 July 2013

Getting Apache Ant Working in Cygwin

I've just had some fun setting up Ant under Cygwin.  It all came down to getting the PATHs working correctly when the installed directories had spaces in their names.  The solution was to use cygpath to first shorten into dos format then unix-ify the paths.  e.g.

LONG_PATH=/cygdrive/c/Program\ Files/Java/jdk1.7.0_25/

DOS_PATH=`cygpath -d "$LONG_PATH"`




echo $DOS_PATH
C:\PROGRA~1\Java\JDK17~1.0_2\

UNIX_PATH=`cygpath -u $DOS_PATH`

echo $UNIX_PATH
/cygdrive/c/PROGRA~1/Java/JDK17~1.0_2/

So a one-liner to do the whole lot

UNIX_PATH=`cygpath -u $(cygpath -d "$LONG_PATH")`

Sunday, 9 June 2013

SFTP Chroot Setup for Group ScriptCraft Session

For the next Madlab CoderDojo I'm going to try running a group session scripting Minecraft with Javascript using ScriptCraft.  One problem with this is that there will be a room full of kids with laptops that I've never seen before who will need access to the ScriptCraft server to edit Javascript files.  My plan is to use Notepad++ for Windows and TextWrangler for Mac machines, and SFTP over to the server.  In order to restrict any potential damage I'm going to use the SSH server to enforce an SFTP chroot jail.  This is the server side of the setup...
  1. sudo adduser coderdojo
  2. sudo vi /etc/ssh/sshd_config
  3. Add the following to the end of the config file:
    Match User coderdojo
        ChrootDirectory /home/coderdojo
        AllowTCPForwarding no
        X11Forwarding no
        ForceCommand internal-sftp
  4. sudo /etc/init.d/ssh restart
  5. Finally the user's directory needs to be owned by root so:  sudo chown root:root /home/coderdojo
  6. Create a folder on the scriptcraft server for the coderdojo files:  sudo mkdir /home/coderdojo/dojo
  7. sudo chown coderdojo:coderdojo /home/coderdojo/dojo
  8. Now set up a symbolic link from the ScriptCraft plugin directory back to the coderdojo user's directory: sudo ln -s  /home/coderdojo/dojo /home/scriptcraft/bukkit/js-plugins/dojo
  9. sudo chmod a+w /home/coderdojo/dojo

With this setup in place, users can use SFTP to edit files on the ScriptCraft server that are then picked up and run by the ScriptCraft Javascript module.

The instructions I followed for this setup are on the Debian Admin site.

ScriptCraft Server Updated for Latest "classroom" Module

Walter Higgins from ScriptCraft has just added a "classroom" module to ScriptCraft. (Here's what changed on GitHub).  Here are the steps I followed to upgrade the Madlab Miners ScriptCraft server:

  1. Log on to the server as the scriptcraft user
  2. wget http://scriptcraftjs.org/download/2013-06-09/ScriptCraft.jar
  3. Re-attach to the scriptcraft screen session  (screen -r)
  4. Type "stop" at the server prompt, and wait for it to stop (takes several seconds to save the chunks from all the Multiverse worlds)
  5. mv ~/ScriptCraft.jar ~/bukkit/plugins
  6. Restart the server using the bash file: ./startserver.sh
  7. Once the server is up login and check that you can turn on the classroom with: /js classroom.allowScripting(true)
  8. Works like a charm!

Saturday, 8 June 2013

Method for Archiving Gmail

My sister-in-law runs a fashion business and as a consequence sends and receives a  lot of big emails.  Even Google Apps for Business' 25GB mail box allowance is used up within a couple of years.  I needed a way to archive emails out of Gmail and on to local storage.  Here's how I do it:

  1. Install the getmail tool.  I'm running it under Cygwin on Windows, but it should run equally well on Linux.  Here are the Cygwin steps:
    1. Run the Cygwin installer and make sure that you have python installed
    2. Make a directory for the getmail install: mkdir getmail
    3. cd getmail
    4. Download the latest release: wget http://pyropus.ca/software/getmail/old-versions/getmail-4.43.0.tar.gz
    5. tar -zxvf getmail-4.43.0.tar.gz
    6.  cd getmail-4.43.0
    7.  python setup.py install
  2. Log into the Gmail web UI of the account you want to archive
  3. Search for the files that you want to archive.  E.g. to find all the files sent or received in 2011 search for:  "before: 2012/01/01 after: 2010/12/31"
  4. Now we need to give all the search results a unique label.  The trick here is that you need to label all the search results not just the first page, so:
    1. In the Gmail web UI click the select "all" check box
    2. When you do this, Gmail will ask "All 20 conversations on this page are selected. Select all conversations that match this search", click on the link to make sure you select all of them
    3. Click on the label icon
    4. Fill something unique in the "Label as:" box, e.g. "Mail2011"
    5. Click the "Create New" link
    6. The "New Label" dialogue box will pop up.  Click "Create"
    7. A warning box will explain that this will affect all conversations in the search.  Click "OK"
    8. Wait a few seconds until the "Loading..." box goes way to give Google time to make all its changes
  5. Next we need to set up a config file that will let you download all the selected emails
    1. First make a new directory for the archive
    2. mkdir emailarchive
    3. cd emailarchive
    4. Then edit a new config file
    5. vi getmail.conf
    6. Adapt my config to your use:
      [retriever]
      type = SimpleIMAPSSLRetriever
      server = imap.gmail.com
      username = user@gmailaccount.com
      password = topsecret
      mailboxes = ("Mail2011",)
      port = 993


      [destination]
      type = Mboxrd
      path = /home/dave/emailarchive/Mail2011.mbox


      [options]
      received = false
      delivered_to = false
      read_all = false
      verbose = 1

  6. The next step is a gotcha, the archive file has to exist before you try to run the getmail program, so do: touch Mail2011.mbox
  7. You are ready to run the program now with:
    getmail --getmaildir /home/dave/emailarchive --rcfile getmail.conf
  8. Some time later you will have all the emails downloaded to a local mbox format mail store
  9. Finally you can now go back into Gmail, select the label you created earlier and delete all the emails.  Note that Gmail won't free up your space until the emails are emptied from your trash folder

If you need instructions on how to read mbox format mail store files take a look at these instructions for several different mail reading clients.

Thanks to Matt Cutts for outlining the general approach to Gmail archiving that I have adapted here.