Saturday, 4 October 2014

Experiments with Pexpect

The Pexpect python module is used to start up and control new processes from python code.  For the Manchester CoderDojo I needed to incrementally get output from a spawned process over the course of a long run.  To test this I created a simple module that counted to 20 slowly:


"""
Count to 20 in one second steps
"""
from time import sleep

for i in range(1,21):
    print(str(i))
    sleep(1)

Next I created a python module to spawn off a process to call the counter and print the output.

"""
Call the slow-running counter and print the results.
"""

import pexpect

output = pexpect.run('python count-slow.py')
print(output)

The problem of course is that the run command blocks until all the output is produced.  The solution allowing you to  see the output as it appears is to use the pexpect.spawn() function.  This function to creates a sub-process object from which you can read one line at a time.  Pyexpect.readline() will block until it receives a line.  An empty string indicates the process has completed.

"""
Call the slow-running counter and print the results.
"""

import pexpect

child = pexpect.spawn('python count-slow.py')
next_line = child.readline()
while next_line != '':
    print(next_line.strip())
    next_line = child.readline()

Formatting Python Code for the Web with Pygments

I've been unsatisfied with Blogger's lack of easy handling for code formatting.  To fix this I installed Pygments.  I can now type,

pygmentize -f html -O noclasses count-slow.py

...and end up with output that I can paste into the Blogger HTML editor window.

The result is something like this:

"""
Count to 20 in one second steps
"""
from time import sleep

for i in range(1,20):
    print(str(i))
    sleep(1)




Friday, 3 October 2014

Vagrant Working for the Manchester CoderDojo Minecraft Server Build

I've got a basic vagrant setup working for the Manchester CoderDojo Minecraft server build.  Instructions on the GitHub page.

Next task is to get the chunked responses working so that you can see the python output appear in the browser as the code is running.

Tip: Copying Large Files from Remote Desktop Sessions

I just had the problem of having to grab a large file from a Windows remote desktop session.  The usual cut-remote-paste-local magic wasn't going to manage the file size.  It turns out that RDP maps "\\TSCLIENT\" on the remote server to your local PC.  You can therefore use robocopy to bring the large file over locally using the restart mode in case of problems. 

For example to grab "big file.zip" from the current folder on the remote box to the c:\tmp folder locally run this:

robocopy /z /eta .  \\TSCLIENT\C\tmp "big file.zip"

Monday, 29 September 2014

Vagrant and Docker Workflow

I've recently been looking into using both Vagrant and Docker to develop my projects.  This is the work flow that I've settled on...

  • I'll use Vagrant to support my development work
  • I'll use Docker as the mechanism for getting my projects hosted
  • Start a project with a Vagrant set up file that will launch a Debian vm and apt-get Docker installed on it
  • The Vagrant VM is used to build the code
  • The result of of the development code is to build a Docker image of the project
  • The Docker image is what is then used to deliver the project's services

As an example I created vagrant-docker-flask on GitHub to show how this works.

  1. Clone the git repo: git clone https://github.com/davegoopot/vagrant-docker-flask.git
  2. In the cloned directory run:  vagrant up
  3. You now have the dev machine image installed that you can connect to using: vagrant ssh
  4. Inside vagrant run: cd /vagrant/flaskserver/
  5. Then build the flask server image with: sudo docker build -t="local:flask" .
  6. Start up a container from the image with:  sudo docker run -d -p 5000:5000 local:flask python demo.py
  7. The flask server will run and be accessible from the original host machine at http://localhost:5000/
  8. To work on the code, you can now edit flask/demo.py, run the docker build and the docker run command again and see your changes

Monday, 5 May 2014

Making the Raspberry Pi Email IP on Boot

I needed my Raspberry Pi to email its IP address to me when it booted in order to ease finding the IP address when running headless.  Here's how I did it:

  1. There are some really good instructions here: http://elinux.org/RPi_Email_IP_On_Boot_Debian
  2. In order for it to work I had to set up an application password for gmail since I'm using two-factor authentication to login
  3. Reboot and it sent me an email.  That was painless!

Twilio Interface for the Rube Goldberg Machine

Here's how I built the Twilio interface to the Rube Goldberg machine...

  1. Install the Twilio python libraries: sudo pip install twilio
  2. Set up a feature branch for Twilio: git checkout -b twilio-plugin
  3. Set up tests and code to load credentials from twilio.secret file
  4. From the secret file load the account_sid and auth_token
  5. Write the update_state() method to send an SMS with Twilio
  6. Decided not to unittest the SMS sending as this is just simple interface code with no logic
  7. Used the python code documented at:  https://www.twilio.com/user/account/developer-tools/api-explorer/message-create
  8. With a bit of Blu Tack, and some dominoes the system worked: http://youtu.be/Vj7IvwIN0r4

Sunday, 4 May 2014

Fixing Anki Flashcard Dates

I've been using Anki for a while to learn things.  Recently I made the mistake of editing some existing cards to have a new meaning.  The problem was that the algorithm for when the card would be next presented to me did not reset itself on the edit.  I was left wanting to see the edited card, but Anki telling me that I'd have to wait for three months first.

Here's how I fixed it:

  1. Installed the full PC version of Anki:  sudo apt-get install anki
  2. Connect the full PC version and sync up the deck in question
  3. Realise that the Debian stable version of Anki is out of date to work with the server
  4. Set up Apt-Pinning for Anki:
    1. Add a testing entry to /etc/apt/sources.list
    2. Edit /etc/apt/preferences and set it up like this
    3. Package: *
      Pin: release a=stable
      Pin-Priority: 900
      
      Package: *
      Pin: release a=testing
      Pin-Priority: 600
    4. Run aptitude, update
    5. Find the anki package in the aptitude UI
    6. Pick the 2.0... version of Anki and install it -- being careful to make sure aptitude picks the correct version
  5. With the PC client running click "browse"
  6. Select the required deck
  7. Select all the cards
  8. Do Edit->Reschedule then set the schedule for zero days
  9. Close the browser and reopen the deck, all the cards are now ready for review
  10. Sync back up to the server and back down the phone again

Monday, 28 April 2014

Taking Stock of Rube Goldberg Branches

Doing a quick review of the Rube Goldberg branches in git:

  • Master - Needs pylint fixing
  • file-plugin - work complete.  Deleted
  • gpio-impl - work complete.  Deleted 
  • http-plug - work still to do.  Merge conflicts from master.  Added Trello item
  • pip-packaging -- work still to do in order to figure out how to use from pip install only. Merged master.  Added Trello
  • scratch-plug - work complete. Deleted. Trello for sorting out tests

Sunday, 27 April 2014

Working on the Minecraft-Scratch Interface for the Rube Goldberg Machine

For the Rube Goldberg project at the CoderDojo I need a way to interact with Scratch from Python.  Here's my steps of how I went about it:

  1. After reading around I realised that I'd have to run Scratch 1.4 and the Python interaction would come from scratchpy - https://github.com/pilliq/scratchpy
  2. Installed scratch 1.4 on Windows
  3. pip install scratchpy
  4. Enabled Scratch remote sensing - http://wiki.scratch.mit.edu/wiki/Remote_Sensor_Connections#Enabling
  5. Played with the interaction - http://youtu.be/YG99uam8pKA
  6. Create new branch for scratch plugin git checkout -b scratch-plug
  7. Check all tests pas
  8. python setup.py test
  9. Tests failed as wasn't on Rasp PI.  After some poking around I discovered that I'd installed the GPIO package on to my development virtual box environment, and so now the package was loading okay and the case that I had set up in the test file to spot the lack of GPIO wasn't working.  I deleted out the GPIO package and it all worked again...phew!
  10. Making the scratch target -- use a broadcast of either "BLOCK-PRESENT" or "BLOCK-ABSENT"
  11. Create a scratchplug.py module
  12. Implemented the target code
  13. Tested that it worked using a file source
  14. Not quite sure how to make the unit tests work.  Will park that problem for now and come back to it later
  15. Now to make the source method
  16. The problem is that the scratch.receive() method blocks until it receives a broadcast, but the Rube Goldberg architecture needs to poll
  17. The solution will be to use the multiprocessing module as shown in this answer on Stack Overflow - http://stackoverflow.com/questions/492519/timeout-on-a-python-function-call
  18. Got a bit stuck coordinating the output with Queues since I didn't realise that multiprocessing had its own implementation of Queue -- you live and learn
  19. Committed that on to the scratch-plug and pushed to GitHub
  20. Made a quick demo of the Minecraft light going into scratch and scratch button turning on a light in Minecraft - https://www.youtube.com/watch?v=cpD-RegWybA

Sunday, 23 February 2014

Python Minecraft CoderDojo -- Play Testers Sought

I've updated the sheets for the Manchester (UK) CoderDojo Python-Minecraft project.  I'm looking for play-testers.  If you are interested, take a look at this on the Minecraft Forum:

Friday, 21 February 2014

Jitsi VOIP Client DTMF Settings

It took me ages to find out how to modify the Jitsi DTMF settings today.  In case you have the same problem, here is a pictorial guide:












Monday, 17 February 2014

The Corn Laws

The Corn Laws where tariffs imposed on the import of corn to the UK from 1815 to 1846.  The intention of the tariffs were to protect the interests of corn producing landowners.  While landowners benefited from the protection of their markets, low-paid workers experienced poverty and famine from having to spend a large proportion of their income on bread.  The struggle to repeal the Corn Laws led to the collapse of the Conservative government and the resignation of Robert Peel as Prime Minister.  Following their repeal corn prices fell in the UK resulting in the reduction in the amount of land dedicated to cereal production and a similar reduction in the demand for agricultural labour.

Thursday, 13 February 2014

Mainline vs. Trunk Development

In software source control (git, mercurial, etc.) you branch the code base in order to isolate different development efforts from one another. In a "mainline" source control model fixes are made on the branches and then merged into the mainline. In a "trunk" source control model fixes are made on the single trunk and then pulled into the branches as needed for a release.
Paul Hammant provides a nice picture of this in which he expresses his preference for the trunk model here: http://paulhammant.com/images/mainline_vs_trunk.png

Monday, 10 February 2014

The Pigeonhole Principle

The Pigeonhole Principle is a counting argument method of mathematical proof.  It says that if you have more items to distribute than pigeonholes to put them into then at least one of the pigeonholes must contain more than one item.  This simple principle becomes interesting when you extend it to countably infinite sets, e.g. the natural numbers.  See: http://en.wikipedia.org/wiki/Hilbert%27s_paradox_of_the_Grand_Hotel

Friday, 10 January 2014

Five Sentences a Day

In order to have confidence that you have learned something it is a good idea to write something about it. I'd like to learn something new each day. But, who has the time to write an essay each day? Combining the idea of http://five.sentenc.es/ with the desire to write something each day I hereby declare I'm going to write five sentences, or fewer, each day.