Wikipedia Graph Generator

I've been working on this project for the past month or so, but since it pretty much works I've lost interest. For those interested, this sort of graph is an example of generated output, and shows pages as vertices and inter-page links as directed edges. The scripts can be found here: wikidown-20091110.zip

Since the data set is quite large (the PostgreSQL database dump compressed is over 550 MB), it'd be much easier for anyone who wants to check this out to generate the data. The process is as follows:

If the above image doesn't load, open the image location in a separate window or tab.

1. Run psql -U postgres wikidown2 < schema_create2
2. Download this file from Wikipedia: enwiki-latest-pages-articles.xml.bz2 (5.3 GB)
3. Run wikixml2csv.py enwiki-latest-pages-articles.xml.bz2 pages.lst links.lst I originally used CSV here, but switched to an ASCII group separator later. The '.csv' suffixes are vestigial.
4. Run csvlistfilter.sh links.lst links-sorted.lst.
5. Run csvpagefilter.py pages.lst pages-presorted.lst.
6. Run sort -k1nru pages-presorted.lst pages-sorted.lst.
7. Unfortunately, due to the behavior of Python's hash algorithm and Postgres's tree algorithm, there will be a duplicate title. The only real solution is to keep trying the following steps and edit pages.lst accordingly. Remember than whenever you delete a row in pages.lst, you must also delete links in links.lst that point to it.
8. Run bzip2 links.lst and bzip2 pages.lst.
9. Run csv2psql.py pages.lst.bz2 links.lst.bz2. This step is considerably faster if both files are mounted in tmpfs (RAM), but only if your machine has enough RAM. tmpfs is not available on Windows.

Once you've run those steps, you can run subgraph.py

Addendum: "Leaf Hat Linux" Stickers

In response to my previous post, it appears that I'll have to have 1000 printed and they'll be 4" wide by 2" tall. I'll raise the price to $0.40 per sticker, but will only charge $3.00 per ten stickers or $20.00 per hundred stickers.

Sorry for the inconvenience; it was news to me too. The quality won't suffer, though, since it's all still not rastorized.

"Leaf Hat Linux" Stickers

I'm having 500 "Leaf Hat Linux" stickers printed. These are high-quality stickers, with no rasterization done at any point, and will be 3.5" wide by 1.25" tall.

If anyone wants one or more, contact me. I'll probably set the price at $0.35 per sticker.

sizeof/bsizeof

I thought it was about time that I make a few convenient changes to the old sizeof utility. The primary change here is adding support of printing either bytes (as bsizeof, which is useful for piping to sort -k1nr) or size in larger units as appropriate (as sizeof).

sizeof.c

#include <sys/types.h>
#include <dirent.h>
#include <stdio.h>
#include <string.h>
#include <libgen.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>
#include <limits.h>

off_t getsize(const char * path) {
off_t ret = 0;
struct stat info;
struct dirent * file;
DIR * dirinfo;
char d_name[PATH_MAX];
size_t pathlen = strlen(path);
char * app = d_name + pathlen + 1;
if(pathlen < PATH_MAX) {
strcpy(d_name, path);
strcat(d_name, "/");
if(!stat(path, &info)) {
if(S_ISREG(info.st_mode))
ret = info.st_size;
else if(S_ISDIR(info.st_mode)) {
dirinfo = opendir(path);
while((file = readdir(dirinfo)) != NULL) {
if(strcmp(".", file->d_name) && strcmp("..", file->d_name) && pathlen + file->d_reclen + 1 < PATH_MAX) {
strcpy(app, file->d_name);
ret += getsize(d_name);
}
}
closedir(dirinfo);
}
}
}
return ret;
}
int compute_size_string(char * str, off_t size) {
if(size > (1 << 30))
sprintf(str, "%.2lf GiB", ((double) size) / ((double)(1 << 30)));
else if(size > (1 << 20))
sprintf(str, "%.2lf MiB", ((double) size) / ((double)(1 << 20)));
else if(size > (1 << 10))
sprintf(str, "%.2lf kiB", ((double) size) / ((double)(1 << 10)));
else
sprintf(str, "%lu B", size);
return 0;
}
void print_size_computed(off_t size, const char * name) {
char size_string[1024];
compute_size_string(size_string, size);
printf("%-10s %s\n", size_string, name);
}
void print_size(off_t size, const char * name) {
printf("%-40lu %s\n", size, name);
}

int main(int argc, char ** argv) {
int i;
off_t size;
char * bname = basename(argv[0]);
void (* handler)(off_t, const char *) = strcmp(bname, "bsizeof") ? &print_size_computed : &print_size;
for(i = 1; i < argc; i++) {
size = getsize(argv[i]);
if(size)
handler(size, argv[i]);
}
return 0;
}

Dream: Mountain Spring and Waterfall

I know for a fact that I've had this dream before, which is why it's interesting. The overarching plot starts off with me wanting to run away from home, and through the course I visit multiple different locations. I'll only describe the snapshots that I can recall.

1. I start by walking down my homestreet, which isn't so abnormal. I make a left at the tee.
2. A thought runs through my mind, "If I keep going this way, through the mountains, I'll end up in Everett." I even see the mountains in the distance. Strangely, there aren't any actual mountains between where I live and Everett in real life.
3. I end up passing through a rural area reminiscent of the rural Cascade foothills, with light forest and a lot of grass on the slopes.
4. Eventually, I arrive at a massive spring, where the water is clear and the bed is made of white rock. There are executives of some sort ("suits") in the water, and I mention to one that I've been in the pool before. He only utters a bit of poetry in response.
5. My next stop is where the road I'm walking along makes a light right turn, with the stream originating from the spring separating it from the base of a cliff. A high road runs across the top of the cliff.
6. I exit the forest, only to come to a series of waterfalls in a somewhat more barren landscape. I had been off the road for some time, instead opting for a paved path originating from it. The path splits, with one end capped with a balcony and the other direction crossing a bridge similar to this one, only using steel tubing instead of wood. The bridge crosses a very tall, yet moderately wide waterfall, which levels off for a hundred meters, then cascades down a much wider set of horseshoe falls. I take out my camera and snap photos of the scenery, but when I get to the bridge I hesitate, remembering my last experience. The bridge begins to violently sway in a manner similar to that of a boat as I try to cross it, forcing me to return to the path.
7. I climb various other paths to take photos of odd sculptures that adorn the ridge visible from the balcony, where other tourists do the same.

It's an interesting dream, but I only recall having had it once before.

ロマンスの神様 - 広瀬香美

This song is so happy and warm that I feel like my head's going to explode with rainbows. It sounds silly, but you'd have to listen to it to see what I mean.

• ロマンスの神様 [Java applet, YouTube]

勇気と愛が世界を救う　絶対いつか出会えるはずなの
沈む夕日に淋しく一人　こぶし握りしめる私
週休二日　しかもフレックス　相手はどこにでもいるんだから
今夜飲み会　期待している　友達の友達に

目立つにはどうしたらいいの　一番の悩み
性格良ければいい　そんなの嘘だと思いませんか？

Boy Meets Girl　幸せの予感　きっと誰かを感じてる
Fall In Love　ロマンスの神様　この人でしょうか

ノリと恥じらい必要なのよ　初対面の男の人って
年齢　住所　趣味に職業　さりげなくチェックしなくちゃ
待っていました　合格ライン　早くサングラス取って見せてよ
笑顔が素敵　真顔も素敵　思わず見とれてしまうの

幸せになれるものならば　友情より愛情
「帰りは送らせて」と　さっそくOK　ちょっと信じられない

Boy Meets Girl　恋してる瞬間　きっとあなたを 感じてる
Fall In Love　ロマンスの神様　願いをかなえて
Boy Meets Girl　恋する気持ち　何より素敵な宝物
Fall In Love　ロマンスの神様　どうもありがとう

よくあたる星占いに　そう言えば書いてあった
今日 会う人と結ばれる　今週も　来週も　さ来週もずっと　oh yeah!

Boy Meets Girl　土曜日 遊園地　一年たったらハネムーン
Fall In Love　ロマンスの神様　感謝しています
Boy Meets Girl　いつまでも　ずっとこの気持ちを忘れたくない
Fall In Loveロ　マンスの神様　どうもありがとう

flatten.py

Working with images a lot, thanks to my webcomic, becomes a lot easier when certain things are automated. For example, this script takes care of flattening an image (by removing alpha information), and even passes the output through pngcrush when applicable. It also includes a default output convention:

flatten.py

#!/usr/bin/python
from sys import argv, stderr, exit
from subprocess import Popen
from optparse import OptionParser, OptionValueError
from os import remove
from os import close as os_close
from re import compile as re_compile
from os.path import join, isfile
from tempfile import mkstemp


size = re_compile('^(.+)x(.+)$')

def check_size(option, opt_str, value, parser):
m = size.search(value)
if m:
parser.values.size = m.group(1), m.group(2)
else:
raise OptionValueError('Invalid size: %s' % value)

filext = re_compile('^(.+)\.(.+)$')
filext_png = re_compile('\.png$')

img_type = ['Bilevel', 'Grayscale', 'GrayscaleMatte', 'Palette', 'PaletteMatte', 'TrueColor', 'TrueColorMatte', 'ColorSeparate', 'ColorSeparationMatte', 'Optimize']
img_cspace = ['CMY', 'CMYK', 'Gray', 'HSB', 'HSL', 'HWB', 'Lab', 'Log', 'OHTA', 'Rec601Luma', 'Rec601YCbCr', 'Rec709Luma', 'Rec709YCbCr', 'RGB', 'sRGB', 'Transparent', 'XYZ', 'YCbCr', 'YCC', 'YIQ', 'YPbPr', 'YUV']

parser = OptionParser(usage = 'Usage: %prog [ options ] input [ output ]')
parser.add_option('-s', '--size', dest = 'size', type = 'string', action = 'callback', callback = check_size, default = None, help = 'The output size: WIDTHxHEIGHT')
parser.add_option('-n', '--no-clobber', dest = 'clobber', default = True, action = 'store_false', help = 'The output size: WIDTHxHEIGHT')
parser.add_option('-f', '--filter', dest = 'filter', default = 'Catrom', help = 'The filter to resize with if necessary.')
parser.add_option('-b', '--background', dest = 'background', default = 'white', help = 'The background when flattening.')
parser.add_option('-F', '--no-flatten', dest = 'flatten', default = True, action = 'store_false', help = 'Don\'t flatten the output.')
parser.add_option('-t', '--type', dest = 'type', type = 'string', default = None, help = 'The type of the output image: analogous to both -type and -colorspace in ImageMagick')
parser.add_option('-d', '--depth', dest = 'depth', type = 'int', default = None, help = 'The depth of the output image.')

options, args = parser.parse_args()
if len(args):
input = args.pop(0)
if isfile(input):
output = None
pngout = False
if len(args):
output = args.pop(0)
if filext_png.search(output):
pngout = True
else:
m = filext.search(input)
if not m is None:
output = '%s_small.%s' % (m.group(1), m.group(2))
if m.group(2) == 'png':
pngout = True
else:
output = '%s_small.png' % input
pngout = True
if not options.clobber and isfile(output):
print >>stderr, 'Output file exists:', output
exit(2)
# ACTUAL PROCESSING
args = ['convert', input]

if options.flatten:
args += ['-background', options.background, '-flatten', '+matte']
if not options.size is None:
args += ['-filter', options.filter, '-resize', '%sx%s' % options.size]

if not options.type is None:
if options.type in img_type:
args += ['-type', options.type]
if not options.depth is None:
args += ['-depth', str(options.depth)]
elif options.type in img_cspace:
args += ['-colorspace', options.type]
if not options.depth is None:
args += ['-depth', str(options.depth)]

if pngout:
fd, tmpfile = mkstemp()
os_close(fd)
args.append('png:' + tmpfile)
Popen(args).communicate()
Popen(('pngcrush', '-rem', 'sRGB', tmpfile, output)).communicate()
remove(tmpfile)
else:
args.append(output)
Popen(args).communicate()
# END
else:
print >>stderr, 'No such file:', input
exit(2)
else:
parser.print_help()
exit(1)% python flatten.py
Usage: flatten.py [ options ] input [ output ]

Options:
-h, --help show this help message and exit
-s SIZE, --size=SIZE The output size: WIDTHxHEIGHT
-n, --no-clobber The output size: WIDTHxHEIGHT
-f FILTER, --filter=FILTER
The filter to resize with if necessary.
-b BACKGROUND, --background=BACKGROUND
The background when flattening.
-F, --no-flatten Don't flatten the output.
-t TYPE, --type=TYPE The type of the output image: analogous to both -type
and -colorspace in ImageMagick
-d DEPTH, --depth=DEPTH
The depth of the output image.