Streaming Sound Between Devices


Today I spent a little time figuring out solutions for "casting" audio from one device to another. Here are some notes on how to make this happen.

Background

I have an old laptop setup as a media-center PC connected to my TV and receiver. This is pretty much the only set of speakers we have in the apartment so whenever we say "I'd like to listen to some music" we almost always mean "through these speakers". However we don't really have a good source of music these days. Sometimes we listen to youtube, sometimes amazon music, sometimes google music. Sometimes we play audio files from a laptop or desktop.

Ideally, I'd figure out a good media management solution, but until that day a decent stopgap and otherwise generally useful feature would be "play this music here on this device I'm using but make the sound come out of those nice speakers over there".

I'm currently able to listen to Google Play Music or Youtube from Firefox or Chrome on either Ubuntu system in the house (desktop or laptop) or from Google Play music or Amazon Prime Music from my Android phone. Amazon Prime Music is apparently broken (no audio) on Ubuntu period...

In theory this solution should work for any ubuntu application and any app on Android.

Other Options

I've previously tried or briefly entertained the following solutions:

Pulse Audio

My current solution is to "enable network access to local sound devices" on the Pulse Audio server running on the media center laptop connected to the receiver. The solution came from askubuntu.com but I'll describe it here as well.

Install paprefs

First, install the pulse audio preferences tool on the sound server:

$ sudo apt-get install paprefs

Configure pulse-audio

Run paprefs uncheck all of the options on all tabs except for

  • "Enable network access to local sound devices"
  • "Don't require authentication"

Restart pulse

Restart the pulse audio server with:

$ pulseaudio -k

Test the client

Test the client by downloading a sample audio file and playing it through the network to your audio server:

$ wget http://www.signalogic.com/melp/EngSamples/Orig/female.wav
$ paplay --server=192.168.x.x:4713 female.wav

Use the ip address of the actual server. If you don't know it, you can right click on the "Network Manager" applet and click "Connection Information" or you can run (in a terminal) ifconfig and read through the output.

Stream browser audio

Run your browser in an environment with PULSE_SERVER=192.168.x.x:4713 and give youtube a try. This worked for me on ubuntu 16.04 (on both client and server) and with chrome and firefox. Amazon prime streaming however doesn't seem to work.

Stream android audio

Solutions came from a couple places:

On the server:

$ sudo apt install pulseaudio-module-bluetooth
$ pactl load-module module-bluetooth-discover

Connect the phone and PC using bluetooth. For me, I had to initialize the pairing from the PC, rather than the phone.

In android bluetooth settings for the PC connection, click Use For: "Media Audio" (this was already checked for me).

At this point it should just work. If you go to the unity-control-center "Sound" section (or "Sound" on the dash menu) you should see the phone listed under the "Input" tab.

python-like kwargs in C++


When dealing with functions that have lots of optional parameters, or at least for which resonable defaults are readily available, it's often a bit of a frustration in C++. Generally defaults are specified during a function declaration as in:

double BinarySearch(std::function fn,
                    int max_depth = 16, double epsilon = 1e-9,
                    double lower_bound = 0, double upper_bound = 100);

And then if we call BinarySearch with only one parameter then the call will use the default values for the rest. But what if I want to specify custom bound, and use the defaults for the other parameters? Admittedly, this is a somewhat contrived example since bounds are less likely to be optional then the others, and we could reorder them better going from most-likely-to-be- specified to least, but it's easy to see how something more flexible would be desirable.

Consider then the following two code snippets. Which is more readable?

First snippet:

double solution = BinarySearch(fn, 0, 100);

Second snippet:

double solution = BinarySearch(fn, lower_bound = 0, upper_bound = 100);

I really like the way that optional arguments work in python with kwargs. I'd love to have that same kind of functionality in C++. kwargs.h implements one mechanism of achieving this.

How does it work

kwargs takes advantage of variadic templates in C++ to build up a single data structure which contains all of the optional parameters (I'll call this a "parameter pack"). Each optional parameter of type T is stored in a structure of type Arg<tag,T> where tag is a unique numeric identifier associated with a particular optional argument key.

The parameter pack data structure derives from Arg<tag,T> for each (tag,T) pair that shows up in the list of optional arguments.

Overloading of the assignment (=) operator gives us an opportunity for building the (tag,T) pairs within the parameter list of the function call.

Get it

See the source and doxygen on github.

Example

I'll conclude with an example usage:

#include 
#include "kwargs.h"

// these are tags which will uniquely identify the arguments in a parameter
// pack
enum Keys {
  c_tag,
  d_tag
};

// global symbols used as keys in list of kwargs
kw::Key c_key;
kw::Key d_key;

// a function taking kwargs parameter pack
template 
void foo(int a, int b, Args kwargs) {
  // first, we construct the parameter pack from the parameter pack
  kw::ParamPack params(kwargs);

  std::cout << "foo:\n--------"
            << "\na: " << a
            << "\nb: " << b
  // We can attempt to retrieve a key while providing a default fallback value.
  // If c_key is in kwargs then this will return the value associated with
  // that key, and will have the correct type. Note that the type of the default
  // parameter in this case is const char*.
            << "\nc: " << kw::Get(params,c_key,"null");
  // We can also do stuff conditionally based on whether or not arg exists in
  // the param pack. We still need to provide a default value, since we need to
  // know the return type of the Get function when the key is not in kwargs.
  // In this case, the default value wont ever be used at runtime.
  if( kw::ContainsTag::result ) {
    std::cout << "\nd: " << kw::Get(params,d_key,0);
  }

  std::cout << "\n\n";
}

int main( int argc, char** argv )
{
  foo(1, 2);
  foo(1, 2, c_key=3);
  foo(1, 2, c_key=3, d_key=4);
  foo(1, 2, d_key=4);
}

Full Disk Encryption with USB master key


When I decided to go with full disk encryption on my machines, I had a pretty hard time figuring out exactly what to do. Not only were there tools and commands to learn, but there was quite a bit of design involved in the process. In this post, I will describe the setup I chose, and the scripts I use to make things a bit more convenient.

Partition Design

partition size label file system mount point notes
sda1 200MiB boot ext4 /boot
sda2 20GiB luks/ext4 /
sda3 218GiB luks/ext4 /home
sda4 6GiB Extended Partition
sda5 4096MiB recovery ext4 / Minimal Ubuntu for recovery
sda6 1900MiB luks/swap

Creating the partition layout

After deciding on a partition scheme, the next step was to create the partitions and format them. I created a 14.04 USB installer and booted the machine in the "Try Ubuntu" mode.

This part is easy. I used gparted to create the partition layout above. For the three encrypted partitions (marked luks/… above) I left them "unformatted".

Formatting the encrypted partitions

Instructions for the next step came mostly from the Arch wiki. The process involves first formatting the encrypted partitions with a Linux Unified Key Setup (LUKS) header. The luks header contains the AES master key as well as key information for up to eight passphrases (note, in this context a passphrase may be a file of arbitrary data) allowing access to that master key.

Initially, I chose to lock down the root and home partitions using a keyfile passphrase, and later added a password as a passphrase, so I will go through that process.

First, we need to create a keyfile for each partition. I used /dev/random to source random bytes (rather than /dev/urandom which is faster but pseudorandom). Knowing that /dev/random sources user interface event arrival times for randomness I made sure to do lots of typing while I was reading off a key. I chose to use a keysize of 512bytes. I initially accomplished this with

$ cat /dev/random > root.key

while watching in another terminal with ls -l until the file was over 512 bytes. Then I used

$ truncate -s 512 root.key

to reduce the size to 512. After doing this a few times I created a small script to provide saner feedback on how much data had been read from /dev/random

#!/usr/bin/python
# key_progress.py
import sys

max = 512
bytes = 0
while( bytes < max ):
  data = sys.stdin.read(1)
  bytes += len(data)
  fraction = bytes/float(max)
  digits = int(10*fraction)
  bars = '='*digits
  spaces = ' '*(10-digits)
  percent = 100*fraction
  text = "\r[%s%s] : %04.2f%%" % (bars,spaces,percent)
  sys.stdout.write(text)
  sys.stdout.flush()
sys.stdout.write('\n')

Usage is like

$ cat /dev/random | tee -a root.key | key_progress.py

Next, I set up the root partition. As described in the wiki, we create the luks header with

# cryptsetup -cipher=aes-xts-plain64 -key-file=~/root.key -key-size=512 -hash=sha51 -iter-time=5000 -use-random luksFormat /dev/sda2

Please research the different cipher options but at the time of this writing my research indicates that aes-xts is generally the best option for a general- use large block device (hard drive).

Note that one may omit the -key-file=/path/to/file flag to specify a password instead of a keyfile, but I chose to start with a keyfile and add a password later.

Once the block device has a proper luks header we can expose the encrypted block device as an unencrypted block device with the following command.

# cryptsetup luksOpen /dev/sda2 root

This will expose the unencrypted blocks of /dev/sda2 in /dev/mapper/root as if it were a raw block device. We can then create an ext4 filesystem on that encrypted device via

# mkfs.ext4 /dev/mapper/root

I repeated the same steps for the home partition:

# cat /dev/random | tee -a home.key | key_progress.py
# cryptsetup -cipher=aes-xts-plain64 -key-file=~/home.key -key-size=512 -hash=sha51 -iter-time=5000 -use-random luksFormat /dev/sda3
# cryptsetup luksOpen /dev/sda3 home
# mkfs.ext4 /dev/mapper/home

Installing Ubuntu 14.04

The installation process is another of the easy steps. Just launch "install ubuntu" from the desktop icon. When it asks where to install choose "do something else" and specify the partitions manually.

device type mount point format?
/dev/sda1 ext4 /boot No
/dev/mapper/root ext4 / No
/dev/mapper/home ext4 /home No
/dev/sda6 swap

The rest of the installation is business as usual

Post-install work

After installing, and before rebooting we need to do some work on the new system.

First, let's setup an encrypted swap. If the ubuntu live-usb system happens to be using our formatted swap space then we need to turn off the swap. Then we run the following

# swapoff -a
# cryptsetup -d /dev/random create cryptswap /dev/sda6
# mkswap -f /dev/mapper/cryptswap -v1

At this point I realized that I had no way to provide the keyfile for unlocking the root partition at boot so I added a password to encrypted root partition.

# cryptsetup -key-file=/root.key luksAddKey /dev/sda2

Now we need to setup the new systems cryptab. Start by mounting the filesystem for the new installation.

# cd /mnt
# mkdir root
# mount /dev/mapper/root root
# mount /dev/mapper/home root/home
# mount /dev/sda1 root/boot

Now we edit /etc/cryptab to setup the encrypted volumes. My crypttab looks like the following:

# root filesystem
root UUID=fdc69d8e-ede4-40f1-8a38-d69ba73c1c82 none luks

# home filesystem
home UUID=74c4c574-2396-434e-9f0f-ea9f8270c44e /root/home.key luks

# swap
cryptswap /dev/sda6 /dev/urandom swap

You can find the UUID of the relevant partitions by running blkid as root.

Note that I've placed home.key in /root so that once the root partition is decrypted the home partition can be decrypted without requiring a second input of the password.

Now that the cryptab is setup, we need to rebuild the initramfs for boot. We do this by chroot-ing into the new system and running update-initramfs -u

# mount -o bind /proc root/proc
# mount -o bind /dev root/dev
# mount -o bind /dev/pts root/dev/pts
# mount -o bind /sys root/sys
# chroot root
# update-initramfs -u

And that's it. Reboot now and ubuntu will ask you for a password to expose the root partition before mounting it. You may notice a warning about /home not being mounted and given options to wait, skip, or do something else. Just give it a few more seconds. I guess the timeout built in for mounting /home isn't long enough to also deal with decrypting it.

USB Master Key

The process of entering the password every boot is troublesome and, to be honest, my ability to recall secure passwords is not to be trusted. In order to address this shortcoming, I decided to create a USB stick whose sole purpose was to act like a password. I do not consider this insecure because, should I lose the USB key, I can revoke the encryption passphrase from the LUKS header and make it no longer usable on my system.

I considered several options for how to go about creating the USB key. I considered a separate boot initramfs with grub installed on the key to actually boot from the key. I also considered storing the keyfile on a filesystem and searching removable filesystems at boot, or trying to identify the disk by partition label or UUID. Ultimately, I decided instead to use an unformatted partition of the USB key to store the keyfile.

The following can be performed on target system, however I suggest creating a backup of the boot/initramfs-…-generic file. If you mess anything up you can drop to grub (press shift 5 times at boot) and alter the grub boot command to use the backup initramfs to get back into a useable system.

First I created the keyfile as usual:

$ cat /dev/random | tee -a keymaster_root.key | key_progress.py
$ truncate -s 512 keymaster_root.key

Then I partitioned the USB key with an unformatted partition of 1MiB and the rest as LUKS/ext4. The second partition is somewhat irrelevant for this discussion except to show that the USB key is still useful for storage.

Now copy the keyfile to the unformatted space, and add the key to an unused slot of the LUKS header for the root partition

# dd if=keymaster_root.key of=/dev/sdb1 bs=1
# cryptsetup luksAddKey /dev/sdb1 keymaster_root.key

Now we find the unique name for the USB stick, particularly it's first partition so we can identify it when present:

$ ls -l /dev/disk/by-id | grep sdb1

In my case the USB key is given the path /dev/disk/by-id/usb- ADATA_USB_Flash_Drive_1411416161600078-0:0-part1. I then edited my crypttab as follows:

# /etc/crypttab
# root
root UUID=fdc69d8e-ede4-40f1-8a38-d69ba73c1c82 /dev/disk/by-id/usb-ADATA_USB_Flash_Drive_1411416161600078-0:0-part1 luks,keyscript=/sbin/usb_keymaster

# home
home UUID=74c4c574-2396-434e-9f0f-ea9f8270c44e /root/nadie_chain_root_home.key luks

# swap
cryptswap /dev/sda6 /dev/urandom swap

The keyscript is a script which dm-crypt runs to get the passphrase. It expects the passphrase on stdout. I used the script from the stack overflow link above but took the initial parts of the script from the blog linked above (which makes the script work with plymouth).

The script attempts to read from the raw device provided in ${CRYPTTAB_KEY} (which is the third column in crypttab). If the device file does not exist after 5 seconds, it drops to asking for a password.

#!/bin/sh
#/sbin/usb_keymaster

# define counter-intuitive shell logic values (based on /bin/true & /bin/false)
# NB. use FALSE only to *set* something to false, but don't test for
# equality, because a program might return any non-zero on error
TRUE=0
FALSE=1

# set DEBUG=$TRUE to display debug messages, DEBUG=$FALSE to be quiet
DEBUG=$TRUE

# default path to key-file on the USB/MMC disk
KEYFILE=".keyfile"

# maximum time to sleep waiting for devices to become ready before
# asking for passphrase
MAX_SECONDS=2

# is plymouth available? default false
PLYMOUTH=$FALSE
if [ -x /bin/plymouth ] && plymouth -ping; then
    PLYMOUTH=$TRUE
fi

# is usplash available? default false
USPLASH=$FALSE
# test for outfifo from Ubuntu Hardy cryptroot script, the second test
# alone proves not completely reliable.
if [ -p /dev/.initramfs/usplash_outfifo -a -x /sbin/usplash_write ]; then
    # use innocuous command to determine if usplash is running
    # usplash_write will return exit-code 1 if usplash isn't running
    # need to set a flag to tell usplash_write to report no usplash
    FAIL_NO_USPLASH=1
    # enable verbose messages (required to display messages if kernel boot option "quiet" is enabled
    /sbin/usplash_write "VERBOSE on"
    if [ $? -eq $TRUE ]; then
        # usplash is running
        USPLASH=$TRUE
        /sbin/usplash_write "CLEAR"
    fi
fi

# is stty available? default false
STTY=$FALSE
STTYCMD=false
# check for stty executable
if [ -x /bin/stty ]; then
    STTY=$TRUE
    STTYCMD=/bin/stty
elif [ `(busybox stty >/dev/null 2>&1; echo $?)` -eq $TRUE ]; then
    STTY=$TRUE
    STTYCMD="busybox stty"
fi

# print message to usplash or stderr
# usage: msg  "message" [switch]
# command: TEXT | STATUS | SUCCESS | FAILURE | CLEAR (see 'man usplash_write' for all commands)
# switch : switch used for echo to stderr (ignored for usplash)
# when using usplash the command will cause "message" to be
# printed according to the usplash  definition.
# using the switch -n will allow echo to write multiple messages
# to the same line
msg ()
{
    if [ $# -gt 0 ]; then
        # handle multi-line messages
        echo $2 | while read LINE; do
            if [ $PLYMOUTH -eq $TRUE ]; then
                # use plymouth
                plymouth message -text="$LINE"
            elif [ $USPLASH -eq $TRUE ]; then
                # use usplash
                /sbin/usplash_write "$1 $LINE"
            else
                # use stderr for all messages
                echo $3 "$2″ >&2
            fi
        done
    fi
}

dbg ()
{
    if [ $DEBUG -eq $TRUE ]; then
        msg "$@"
    fi
}

# read password from console or with usplash
# usage: readpass "prompt"
readpass ()
{
    if [ $# -gt 0 ]; then
        if [ $PLYMOUTH -eq $TRUE ]; then
            PASS="$(plymouth ask-for-password -prompt "$1″)"
        elif [ $USPLASH -eq $TRUE ]; then
            usplash_write "INPUTQUIET $1
            PASS="$(cat /dev/.initramfs/usplash_outfifo)"
        else
            [ $STTY -ne $TRUE ] && msg TEXT "WARNING stty not found, password will be visible"
            /lib/cryptsetup/askpass "$1# echo -n "$1″ >&2
            # $STTYCMD -echo
            # read -r PASS /dev/null
            # [ $STTY -eq $TRUE ] && echo >&2
            # $STTYCMD echo
        fi
    fi
    echo -n "$PASS"
}

dbg STATUS "Executing usb_keymaster …"

# flag tracking key-file availability
OPENED=$FALSE

for TRY in 1 2 3 4 5
do
  if ! [ -e "${CRYPTTAB_KEY}" ]; then
    dbg TEXT "Waiting for USB stick to be recognized [${TRY}]"
    sleep 1
  fi
done # for TRY
if [ -e "${CRYPTTAB_KEY}" ]; then
  dd if="${CRYPTTAB_KEY}" bs=1 skip=0 count=512 2>/dev/nul
  OPENED=$TRUE
fi


# clear existing usplash text and status messages
[ $USPLASH -eq $TRUE ] && msg STATUS "                               " && msg CLEAR ""

if [ $OPENED -ne $TRUE ]; then
  # dbg TEXT "Failed to find suitable USB/MMC key-file …"
  readpass "$(printf "Unlocking the disk $CRYPTTAB_SOURCE ($CRYPTTAB_NAME)\nEnter passphrase: ")"
else
  # dbg TEXT "Success loading key-file from $SFS ($LABEL)"
  # msg TEXT "Unlocking the disk $CRYPTTAB_SOURCE ($CRYPTTAB_NAME)"
  msg "Unlocking ${CRYPTTAB_SOURCE} (${CRYPTTAB_NAME}) from USB key"
fi

#
[ $USPLASH -eq $TRUE ] && /sbin/usplash_write "VERBOSE default"

Be sure to chmod +x /sbin/usb_keymaster to make the script executable.

In order to get this script into the initramfs we need to add the following hook file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
#!/bin/sh
#/etc/initramfs-tools/hooks/usb_keymaster

PREREQ=""

prereqs() {
  echo "$PREREQ"
}

case "$1″ in
  prereqs)
    prereqs
    exit 0
  ;;
esac

. "${CONFDIR}/initramfs.conf"
. /usr/share/initramfs-tools/hook-functions

copy_exec /sbin/usb_keymaster /sbin

Lastly, we need to ensure that the USB drivers are loaded at boot time (this may not be necessary with 14.04). Edit /etc/initramfs-tools/modules.

# List of modules that you want to include in your initramfs.
# They will be loaded at boot time in the order below.
#
# Syntax:  module_name [args ...]
uhci_hcd
ehci_hcd
usb_storage

Lastly update the initramfs with

update-initramfs -u

Skype in 64bit Ubuntu 13.04


I've had some trouble getting skype to work in ubuntu 13.04. The first problem arose after switching to the nvidia binary graphics drivers. Skype kept trying to load the 64bit libraries instead of the 32bit compatability ones. This can be repaired by changing the linker's search path. The command to launch skype then is:

   ~$ LD_LIBRARY_PATH=/usr/lib32 skype

The second problem was that skype could not find pulse audio, so it was trying to look up hardware devices using ALSA (or so I've guessed by stdout). The solution is to install libpulse0:i386

   ~$ sudo apt-get install libpulse0:i386

After restarting skype my microphone and audio devices are all listed as "pulse audio" instead of having a dropdown for various options.

Also, as of skype 4.2.0.11 screen sharing appears to now be available in the linux version… finally.

Range Based for-loops in C++11


One of the new features of C++11 that is a complete win in my opinion is the support for range-based for-loop syntax. Judicious use allows for significantly more compact and readable code. However, one thing that is lacking from this feature is the ability to iterate over a range of integers. This isn't a problem, however, because it is very easy to implement.

// This file compiles as a single unit, you can test it by saving it 
// as range_for.cpp and compiling with 
// g++ -std=c++11 -o range_for range_for.cpp
// Adjust the interface as you see fit

// Implementation (put this in a header)
// --------------------------------------------


/// encapsulates a range of integral numbers for use in a c++11 range-based for 
/// loop
template< typename T>
struct Range
{
    /// the begin() and end() functions must return an iterator with a specific
    /// interface
    struct Iterator
    {
        T val;  ///< storage for the actual value

        /// implicit construction from the value type
        Iterator( T val ):val(val){}

        /// the range-based for loops attempt to dereference an iterator using
        /// this operator
        T operator*(){ return val; }

        /// the range-based for loops quit when the comparison of iter != end
        /// returns false
        bool operator !=( T other ){ return val != other; }

        /// iterators in a range-based for loop must have the prefix increment
        /// operator
        Iterator& operator++(){ ++val; return *this; }

        /// we likely want to implicitly convert to the value type
        operator T(){ return val; }
    };

    private:
        T m_begin;  ///< the first integral value
        T m_end;    ///< one past the last integral value

    public:
        /// construct a range [begin, end)
        Range( T begin, T end ):
            m_begin(begin),
            m_end(end)
        {}

        /// interface required by range-based for loop
        Iterator begin(){ return m_begin; }

        /// interface required by range-based for loop
        Iterator end()  { return m_end;   }
};


/// return an object which can be used in a range-based for loop
template 
Range range( T begin, T end )
{
    return Range(begin,end);
}




// Usage 
// ---------------------------------------------

#include 

int main(int argc, char** argv)
{
    std::cout << "Static ranges\n----------------\n";
    std::cout << "\nint:\n";
    for( int i : range(1,5) )
        std::cout << "   i: " << i << "\n";

    std::cout << "\nunsigned int:\n";
    for( unsigned int i : range(1u,5u) )
        std::cout << "   i: " << i << "\n";

    std::cout << "\nlong:\n";
    for( long i : range(1L,5L) )
        std::cout << "   i: " << i << "\n";

    std::cout << "\nDynamic range (program args)\n----------------\n";
    for( int i : range(0,argc) )
        std::cout << i << " : " << argv[i] << "\n";

    return 0;
}

The range<T>( T begin, T end ) function template returns a Range&ltT> object which supports the required interface for range-based loops. It has a begin and end method, both of which return an iterator. The iterator has the same storage says as the integral type used to instantiate the templates. It is implicitly convertable to and from the integral type. Futhermore, it has the prefix ++ operator, and can be "dereferenced" to get the stored integral value.

The implementation is likely to yield compiled code equivalent to a normal for loop.

The output of the demo program above is:

josh@Nadie:~/Desktop$ g++ -std=c++11 -o range_for range_for.cpp 
josh@Nadie:~/Desktop$ ./range_for arg1 arg2 arg3 arg4 arg5
Static ranges
-----------

int:
   i: 1
   i: 2
   i: 3
   i: 4

unsigned int:
   i: 1
   i: 2
   i: 3
   i: 4

long:
   i: 1
   i: 2
   i: 3
   i: 4

Dynamic range (program args)
-----------
0 : ./range_for
1 : arg1
2 : arg2
3 : arg3
4 : arg4
5 : arg5
josh@Nadie:~/Desktop$

Making a list of everything you've installed with apt-get


As a clean install of ubuntu is often the only reliable way to upgrade, it's useful to know everything that you've installed manually. You could get a list of everything that's marked in for installation with something like:

dpkg -get-selections | grep -v deinstall > my-selections

but that list tends to be very long and it includes all the dependencies. So if you install a meta package which depends on a specific version of some other package you'll be trying to install that specific version after upgrade, which is not what we want. In that case all we want is the meta package.

Luckly, apt-get keeps a history of every time it is called. These history files are in /var/log/apt/history.log.(\d).gz. Each entry in that list contains a "Commandline: …" line which lists the command line that was executed. Thus we can get a list of everything we've installed by grepping through these files.

The first script is a perl script which parses "Commandline: …" entries:

#!/usr/bin/perl
# parse_apt_log.pl
while(<>)
{
    if(/^Commandline: apt-get install(.+)/)
    {
        unless(/-reinstall/)
        {
            print join("\n", split(/\s+/, $1) );
        }
    }
}

and the second is a bash script which pipes the contents of those history files to the perl script

#!/bin/bash

outfile="$(mktemp)"

gunzip -c /var/log/apt/history.log.*.gz | ./parse_apt_log.pl > $outfile

sort "$outfile"
rm "$outfile"

Which gives an output like:

josh@Nadie:~/Desktop$ ./make_pkg_list.sh
astyle
bitcoin-qt
debathena-pharos-support
dhex
digikam
docbook2x
docbook5-xml
docbook-xsl-ns
firefox
flip
fontforge
fop
gdm
git-svn
gnucash
gyp
icedtea6-plugin
icedtea-7-plugin
jhead
latexml
libapache2-mod-fastcgi
libavcodec-extra-53
libav-dbg
libavdevice-extra-53
libavfilter-extra-2
libavformat-extra-53
libavutil-extra-51
libcairo2-dbg
libcrypto++-dev
libcurl4-gnutls-dev
libfcgi-dev
libgdal1-dev
libglade2.0-cil
libgnome-desktop-dev
libgtkglext1-dev
libjpeg-turbo-progs
libjson0-dev
libmosquitto0
libpcre3-dev
libqtsysteminfo1
libtelepathy-qt4-2
libtidy-dev
libusb-0.1-4
libusb-1.0-0-dev
libusb-dev
libxml2-dev
libxml2-utils
linux-generic
markdown
meld
mtpfs
mtp-tools
network-manager-openvpn
okular
openjdk-7-jre
owncloud-client
pcsxr
pcsxr
pdfedit
python-rosinstall
qt-sdk
qtsixa
ragel
ros-fuerte-desktop-full
ros-fuerte-pr2-*
ros-fuerte-pr2-simulator
ros-fuerte-rqt
rtmpdump
sixad
smplayer
sound-juicer
spawn-fcgi
sqlitebrowser
synapse
ufraw
ufraw-batch
uncrustify
universalindentgui
untex
vlc-dbg
xdotool
xmlstarlet
xmlto
xsltproc

You can reinstall this list on the new system with sudo apt-get installtr '\n' ' ' < package-list.txt``, where tr will remove the newlines.

Other install notes for upgrading to ubuntu 13.04:

To install deb packages from the command line along with their dependencies install gdebi-core from the repositories. Then run, for instance sudo gdebi google-chrome-stable-current_amd64.deb.

To remove overlay scrollbars: gsettings set com.canonical.desktop.interface scrollbar-mode normal

To allow rawdisk vmdk's in Virtualbox the user must be a member of the disk group: sudo usermod -a -G disk user

C++ FreeFontStack


I've been doing a lot of research into free software font technologies and as a consequence I've been playing around a lot with Freetype2 and Fontconfig (actually the library interface libfontconfig). These are "straight up" c-libraries and so I (personally) find the interface a bit "clunky" in C++ code. Don't get me wrong, I wouldn't presume to criticize their design as it is clear the authors/maintainers are far more knowledgeable about this stuff than I am… but from a users perspective I desire a bit more.

So… I've written some C++ wrappers for freetype2 and fontconfig. The main goals in these projects are:

  • Don't change the usage patterns of the library
  • Provide a pointer-like type which automatically performs reference counting for reference counted objects
  • Provide a pointer-like type which exposes member-functions where relevant
  • Provide a pointer-like type which still allows access to the public data members of the underlying structures
  • Allow access to the native c-pointer for use in parts of the library that I haven't finished wrapping

So far… cppfontconfig wraps more or less all the functionality of libfontconfig. I'd still like to add some interface for assignment by touples in order to grab error codes when desired without the error being part of the function signature. Perhaps in the future.

cppfreetype isn't nearly as complete (freetype2 is a much larger library). I still haven't decided the appropriate way to deal with structures which are normally stack allocated. I do however have wrappers for FT_Face, FT_GlyphSlot, and an iterface for iterating over contours and points in an FT_Outline.

These libraries are released as GPLv3. I have public git repositories and publicly accessible doxygen documentation available in the links below. No bug tracker though since I'm giving up on redmine.

Tweaking JabRef in Ubuntu


A couple of months ago I switched from using Mendeley to JabRef. JabRef is a lot less pretty and lacks a lot of the features of Mendeley, but it a much more solid piece of software. I particularly love that it works with a native bibtex file as a backend, so I can keep my bib file under version control, check it out as a git-remote/svn:external in my latex projects, and have a use the same cite keys in all my documents.

There were a couple of annoyances that I had with JabRef that I managed to work around so I figured I would document them here.

Look and Feel

The "native" java look and feel has all the aesthetic bells and whistles of a 90s era unix desktop, and drives me nuts. Luckily, jabref allows you to specify the look and feel class in it's advanced preferences. Got to options->preferences->advanced and check "Use other look and feel". I use the Gnome desktop so I set this to com.sun.java.swing.plaf.gtk.GTKLookAndFeel and now Jabref looks like it belongs.

JabRef with GTK look and feel

Tiny Treeview for Groups

I like to organize my references (rather than simply search for them) to help me find multiple references for a similar topic. JabRef, however, limits you to only 10 or so rows in the treeview for groups. I found this to be rather frustrating. Unfortunately, you can't change the number of rows dynamically (by clicking and dragging) but there is an undocumented preference entry which sets the number of visible rows. Start by going to options->preferences and the clicking "export preferences" to export a preference file for JabRef with all your current preferences. Save it as a .xml file. Now open that with your favorite text editor and add the following entry within <node name="jabref"><map></map></node>

<entry key="groupsVisibleRows" value="30″/>

Feel free to change "30″ to whatever you like. Now the groups treeview will be a bit larger and more useful. Also, take a look at all the other preferences which you didn't know jabref had and feel free to play around with them to tune your jabref experience.

HTC Incredible 2 USB charging port repair


So I happened to come across a broken HTC Incredible 2. My sister had replaced it with an iPhone when the Incredible stopped charging. She had taken it back to the Verizon shop where they tried replacing the battery but it still wouldn't charge. It seems like such a waste to throw away such a fine piece of hardware which clearly works but can't recharge the battery.

Long story short, a lot of people have had a similar problem and had success by simply replacing the USB chargin port of the phone. As it turns out the repair is not even that difficult. The phones design is incredibly modular, with everything connected to the mainboard by ribbon cables. In this post I document the process of disassembling the HTC Incredible 2 and replacing the USB charging port.

The part I ordered was For HTC Incredible S II 2 Charging Charger Port USB Dock Connector Part Repair. The cost was $9 shipped and it came with a screwdrive and plastic pry tool.

In addition to these tools, I also used a pair of tweezers, an exacto knife, and a very small phillips screw driver. Start by removing the outer cover. This can be done with the plastic pry tool or a coin, or a credit card with one corner shaved down a bit. There is a little tab at the bottom of the phone to get you started.

Then remove the battery and the simcard. These are pretty easy, just take them out.

Now there are seven (7) star screws to remove in order to get the inner housing cover off. One of them is covered with a green "void" sticker (removing this screw will void the warranty, which I assume is already voided). If you dont have a star screwdriver (though one came with my replacement part) a small flathead should work. you can look at the next picture to see where the screws are located.

I used a piece of paper and take to diagram where all the parts came from as I took them off. This is a handy way to keep track of where stuff goes when you put it back together. Anyway, now that the screws are removed we may remove the inner back of the case.

For me, the case took some work. Note that the volume rocker switch is a part of the piece we are removing so it is safe to pry under it, however the little piece of plastic over the headphone jack is not part of the piece we are removing, so don't try to pry it out too. During this process, the USB protective housing and the little rubber cover to what looks like an antenna port came out. I just added them to my diagram so I knew how to put them back.

Now we can see the (presumable) culprit of the charging problem. The USB charging port has some corrosion on it. I've read that many people have had problems charging and that replacing this part will fix those problems. I'm making an $9 bet that this is my problem too.

Now we have to disconnect two ribbon cables on the right side of the main board. They are covered with some ESD tape. I saved the tape and put it back on when I reassembled the phone. I'm not sure if they're useful but I figured perhaps they'd relieve some stress on the attach points.

To remove the ribbon cables pop the tab on the back of the attach point. With the tab up, the ribbon cable should slide right out.

Now remove the two small philips screws. One is holding a piece of plastic over the headphone jack. The other is holding the main board onto the case.

Next we need to pop out the vibrator motor. Once the motor is removed from it's holder, you may leave it connected to the mainboard. It's not held in with any tabs so just pull it on out with your pry tool.

Now it's time to separate the mainboard from the housing. The mainboard is still connected by a ribbon cable on the left side, so lift up from the right side. The board is held down by two attach points. It fits under the headphone jack at the top, and it is clipped in place at the vibrator motor housing. I used the pry tool to separate the board from the clip at the vibrator motor housing, and then pulled slightly downward to remove it from under the headphone jack.

Now we can remove the ribbon cable that is attaching the board to the screen. In hindsight this is not really necessary since our part is already exposed, but I guess it make things a bit easier later. Start by removing the plastic tape covering the ribbon cable attach point. Then pop the lever, and slide out the ribbon cable.

Now that the mainboard is separated from the housing we can remove the USB port. It's just another ribbon cable so remove the tape, flip the tab, and slide out the part.

The we simply swap in the new part. Insert the ribbon cable, make sure it's snug inside the attach point, then close the tab.

Now we reattach the mainboard to the housing by the big ribbon cable on the left (screen?). This was a bit tricky since the ribbon cable is kind of short. I had to pull a five finger maneuver to get the thing in a position where I could insert the cable without it sliding back out. Once the cable is in, close the tab.

Now simply flip the mainboard back over so that it is lying on the housing. Check the camera to make sure it didn't slide out. Mine had slid out and was sitting at a funny angle so I had to shift it back into place. Also make sure that the two small ribbon cables on the right side are pulled out and aren't buried under the mainboard.

Now we can snap the mainboard back into place, and return the vibrator motor back to it's home.

Now screw back in the two phillips screws. One holds the plastic cover over the headphone jack, and the other holds the mainboard onto the housing.

Now we reattach the small ribbon cables on the right side. Slide them into the attach point, close the lever, and then replace the tape (if you like).

Now we can snap the cover back on. Start by replacing the USB port cover. This fits into the housing by two tabs that are under the port. Replace the rubber cap on the left side if it fell off. Also return the volume rocker to it's place inside the back of the housing (the part that you took off) if it fell out.

Then simply snap the back of the housing into place, and screw all the screws back in.

Now you can replace the battery, simcard, and outer cover.

Plug in the phone, and if nothing starts sparking/smoking perhaps you have indeed fixed your problem. Not that this is proof, but the phone now at least tells me it's charging the battery. We'll see if it ever gets to 100%.

Notes on running the same Windows 7 installation in a virtualbox and natively


It seems the initial install can be done either natively or in a virtualbox. I chose to install in virtualbox first, so that I didn't have to worry about changing the boot order of drives in bios.

I created a raw disk vmdk using the command

VBoxManage internalcommands createrawvmdk-filename Alucard_win7.vmdk -rawdisk /dev/sdb

Note that I did not restrict partitions or specify an MBR file. This is because I want windows to actually put its loader in the MBR of the /dev/sdb disk.

I created a virtualmachine, attaching this disk as a SATA drive, and putting the windows 7 install iso as the cd drive. After installation, I did not enter a serial number, but just "continued". After windows loaded I ran "Windows Loader" installation to handle the serial number issues.

Anyway, after this all happened Ubuntu had trouble reading from /dev/sdb. It could not enumerate the partitions. In order to solve this I rebooted, changed boot disk order, and booted into windows natively. It worked without requiring the repair disk, but note that in some tutorials they say that you'll need to repair the installation.

After rebooting again, and changing the boot disk order, ubuntu started and could see the partitions on /dev/sdb. I probably didn't need to change boot disk order, but probably just needed to restart so that the drive was re- enumerated.

Anyway, after this, I ran

sudo update-grub

and windows 7 was found. The grub entry that was created would boot windows 7 natively without any problems.