I got it.

Having used Subversion for a couple of years and CVS for more years than I care to remember, I realized that a central repository system was fundamentally broken. It requires one to keep so much in one’s head. I almost always have to execute a svn status before every commit because I can’t remember about files that have moved, changed, deleted, added etc. This also adds a lot of cruft to the metadata of files that have had substantial changes across branch merges.

On the other hand, Git only cares about changes between trees. It cares about content and not files. You diff trees, not files. This is all very beautiful.

I have only scratched the surface. I cheated and took the crash course for people coming from the Subversion world. I haven’t even explored how merges work across branches: they are about five of them in Git.

One day, I will get all of it. Until then, I’ll trust the judgement of the incredibly smart people who work on Git. This is another of Linus’ masterpieces.

PS: Here’s a fairly good explanation of the guts of Git: Git for computer scientists.

Morning Tutorial

As processors become faster and faster, memory is not able to keep up the demand made by applications, further exacerbated by multicore technology. As as result, applications spend most of their time waiting for data to arrive from memory. This is know as the “memory wall.” This tutorial had presenters from Rambus, Intel and AMD who described their respective approaches to dramatically improve the situation. Most of the technology showcased is still in the prototype stage. I should also never get confused between the terms bandwidth, latency and throughput.

Afternoon Tutorial

The second half of the day had various people from Nvidia showcasing the CUDA technology. They had just released version 2.0. I’ve seen most of the material a couple of times before, so I’m only going to write about things I found really interesting:

• really simple extensions to the C language. My biggest concern with this toolkit is if we will ever see a return on investment in writing for CUDA. What if Nvidia were to abandon this effort a few years down the line? One of the product managers alleviated this concern for me — the language is so simple that it should be trivial to write a source-to-source translator to convert it to plain multithreaded code.

• the nvcc compiler can spit out code for their GPUs or plain multithreaded code that can be compiled with gcc or visual studio.

• the CUDA FFT library supports three-dimensional transforms of Hermitian sequences. This special optimization is twice as fast as the regular complex-to-complex transform.

• hardware support for various types of interpolation, filtering and boundary conditions owing to its graphics roots.

• hardware support for sin, exp, pow, rcp (reciprocal) that can be done in one clock cycle.

• future: fortran/c++ support, multiple GPU devices, debugger, profiler, GPU cluster support.

Conference Day 1

Cars that drive themselves: One of the most interesting talks on the first day was the keynote was Sebastian Thurn. I had attended his talk before at SFU when he had come over just after winning the DARPA challenge n 2005. I was impressed then and I’m continued to be impressed by the quality of work that he and his team puts out. This time he spent a lot of time describing the challenges in the next DARPA Urban Challenge. Two interesting points from his talk:

• the occupancy of freeways is really bad. With automated cars, we can improve the “throughput” of the freeways.
• you can think of these automated cars as train carriages, only attaching/detaching themselves as required.

Mobile Media Processing: Three really impressive technologies.

• a 300 mW single chip television from Telegent. Most of the world is still on analog television. Take technology from the 50s and apply the latest process and methodologies to it.
• voice enhancer based on human audio system from Audience. Based on the Fast Cochlea Transform, they can pick out just your voice and segment it from the background. Enables really advanced voice manipulation techniques — useful for ransom negotiation. Maybe not.
• Nvidia Tegra. Really cool HD video in the palm of your hand.

Supercomputing: The reason I was there.

• Cell broadband engine: For some reason I was finding it difficult to stay focussed as this was not x86.

• Anton: Written about this earlier. They can do a nonbonded computation in one clock cycle! That’s awesome. They get massive speedups with careful organization of memory and treating their hardware as a massive stream processor. All of the messy conditionals and correction terms are treated separately. Interesting to see that they get better performance by doing more computation and reducing unpredictability.

• D.E. Shaw’s specialized ASIC for molecular dynamics which I’ve written about earlier and IBM’s PowerXCell powering Roadrunner.
• Upcoming architectures: AMD’s 780G and Intel’s Nehalem (dot products of special interest to me.)
• Chips tuned for network or IO (Sun’s Rock, Fujitsu’s SPARC64VII and Intel’s Tukwila.)
• Algorithmic content: Roofline models for automatic tuning of kernels (good addition to Demmel’s talk on the future of linear algebra from MMDS.)
• Intel’s Larrabee: response to “the can of whoop-ass” (detailed architectural paper from SIGRAPH.)
• CUDA: useful for a class of algorithms (based on memory access.)

I’m going to be trying something new this time — live blogging. I’ll try to push constant updates to my twitter stream : gane5h.

I’ll be staying at the Sheraton in Palo Alto. Drop me a line if you want to meetup for a chat.

First some rules:

• Singular matrices have determinant zero. Simple, but often overlooked.
• If you require a spectral decomposition at a later stage, the easiest is the product of the eigenvalues (the matrix cookbook is a handy reference.)
• For triangular matrices, it’s the product of the diagonal entries.
• The determinant of the product of two matrices is the product of their determinants.

Using the above rules, the first step is to factor your matrix into two triangular matrices. The LU decomposition achieves this. In general, a permutation matrix allows all ones on the diagonal of L. If you only care about the absolute value of the determinant, simply computing the product of the diagonal values of U gets you the determinant.

If your matrix is symmetric and positive definite, then you can use a specialization known as Cholesky decomposition which makes L the transpose of U (doing half the work with half the storage.) The determinant is then the product of the square of the diagonal elements. Diffusion tensors are covariance matrices (symmetric, positive definite) and so this decomposition always applies.

Each day had a theme:

• Data Analysis and Data Applications
• Networked Data and Algorithmic Tools
• Statistical, Geometric, and Topological Methods
• Machine Learning and Dimensionality Reduction

The breadth of topics was quite exhaustive. I mostly pushed my own agenda: streaming algorithms. There’s so much to write here that I won’t even attempt to.

Besides streaming algorithms, the presentations on mathematical topics were really interesting. Some of it I’ve previously seen from my day-to-day work, some of it was new. Of particular interest to me were the following:

• Graph Sparsification : Never seen anything like this before.
• Massive Terrain Data : Real smart use of offline datastrutures.
• Symmetries in point cloud data : I’m intimately familiar with this style of mathematics from my previous work.
• Pathway Analysis in Protein Folding : Puts bread on the table.
• Intersection SVMs : Didn’t know this was a well known concept in machine learning known as the kernel trick. Goes by Reproducing Kernel Hilbert Space in my neck of the woods and also a precursor to the above mentioned image matching algorithm.
• Manifold regularization : Fréchet means anyone?
• Sufficient Dimension Reduction
• Semi-definite programming : Some mathematical insights to a couple of engineering problems (where <1e-4 is good enough) that’s making my life difficult.
• Spectral Algorithms
• Matrix/Tensor Factorization
• Future of Parallel Linear Algebra

Thanks to my friend Krishna who let me sleep on the floor in his house, thereby saving me from the grips of boredom.

The 2008 Workshop on Algorithms for Modern Massive Data Sets (MMDS 2008) will address algorithmic, mathematical, and statistical challenges in modern statistical data analysis. The goals of MMDS 2008 are to explore novel techniques for modeling and analyzing massive, high-dimensional, and nonlinearly-structured scientific and internet data sets, and to bring together computer scientists, statisticians, mathematicians, and data analysis practitioners to promote cross-fertilization of ideas.

These are my notes (disclaimer: I’m no expert, corrections are welcome.)

Most algorithm engineers thus far are happy with algorithms that are linear, i.e., in the number of data points in the dataset. With web-scale data, linear is not good enough: sub-linear is required. These streams of data are typically unbounded and do not fit in main memory. They cannot even be stored, as it is infeasible to go back and reload them.

The Frequency Problem is used as a model problem over data streams, for example, computing means/variances, medians, top-k frequent items, distinct elements etc. One approach is to approximate the computation over the stream via random sampling.

There are four topics that keep recurring when looking at proofs of streaming algorithms using randomization:

• Markov’s inequality
• Chebyschev’s inequality
• Holder’s inequality
• Chernoff bounds

This is really key. As an example, suppose we wanted to run some complicated algorithm (offline) over the packets flowing through a router. As this would a huge number of packets, we won’t have the luxury of storing all packets, but only a subset. By sampling the packets with probability p, we can estimate the amount of memory required to store the packets as the expectation value of the binomial variable with parameter p and n, where n is the number of packets going through the router.

The use of one of the bounds over the other is a matter of how tight the bound is. The Markov inequality only uses the expectation value of the random variable while the other use higher moments. They do this by using the Markov inequality to a moment generating function exp(tX) of the random variable X.

You’ll want to check the preliminary program for the kind of topics that are going to be covered. While you are at it, check out the workshop webpage from 2006. I’ll go find something to wipe the drool off my chin…

The Singular Value Decomposition is the swiss-knife of linear algebra. Every matrix Y can be factored into three matrices: U, S, and V as

U and V are orthogonal matrices and S is a diagonal matrix. Some uses of this factorization of the matrix Y: calculating 2-norms, Frobenius norms, ranks, null spaces and ranges, (pseudo)inverses and determinants, and by extension solving systems of equations (exact, over- and under- determined), eigenvalues and eigenvectors, and approximations.

Almost every problem in engineering becomes an optimization problem (as in reducing the error under some norm) and the method of least squares makes extensive use of the SVD.

The stage has been set for the next few posts of mine.

The above is an actual plot that I created for some Hall measurements I was doing. I was supposed for find functional relationships between temperature and majority charge carriers, which in my case were electrons because of the n-type doping. The simple case was a least squares fit: scipy.optimize.leastsq to the rescue. The more complicated part was solving a non-linear equation for roots and then doing a least squares fit. The root-finding module in scientific python provides lots of options. At this point, I can confidently say that this environment has more features than Octave.

Just today, I wanted to use the Fourier method on a differential equation (plug: the advantages of which are here) and numerical python with fft, fftshift and fftfreq are exact substitutes for their Matlab equivalents. You can also put actual LaTeX equations on plots, which is a major plus.

That is all.

Most high performance numerical codes use some variant of Fortran or just raw C (which is my staple language.) In that case, as Micheal Feathers rightly points out, unit tests can “collide” with C.

I’ll want to expand on his solution of link seams.

GNU/Linux (and other Unix-like operating systems I think), provide a mechanism of loading shared object files at runtime. Using dlopen(3) and dlsym(3) you can re-create much of the reflection functionality that the Java community enjoys. So by loading your unit test dynamically at runtime, and having a predetermined symbol that behaves as the entry point, you can nicely wrap your module/function with a unit test. Interface this with ctest from the cmake project, and you are set. If you want to have a private function whose symbol isn’t visible on the outside, make it static. The nm utility will give you a list of public symbols in an object.

I’m pretty sure this is supported across the board. I remember doing this under DOS using DJGPP (remember that?) so this isn’t exactly novel.

I’ve used radial and tangential fields in electromagnetics (think solenoids and inductors) countless times, so it should be trivial to generate one, shouldn’t it? Unfortunately, I was getting mixed up in the minus signs somewhere and for the life of me couldn’t find out where. I checked and rechecked my math. I checked the usage of atan2(3) instead of atan(3). Finally, I had to resort to generating them by rotating the Cartesian basis and regenerating the tensors from the spectral components. Hacky, cludgy and doesn’t follow the DRY principle but this is just a test case. Yay tangential field:

To check the robustness of my algorithms later on, I need to add some noise to the fields and see how well they perform. Plagued by partial volume effects, diffusion tensor data are inherently very noisy, so it’ll be good to include noise as part of the algorithm development process. Right now, modeling noise in tensors is a very complicated process because the tensors themselves are built through a linear regression from diffusion weighted images. The noise is definitely not gaussian. Gaussian noise is easy, and that’s what I’m doing now until I fully understand how noise is transformed through the regression.

Another matter of complication is that regularizing (or denoising/smoothing) these fields is an active area of research. Extensions to the standard anisotropic edge-preserving filters like Perona-Malik are non-trivial (at least to me.) One of the technicalities of diffusion tensors is that they are positive definite. The positive definiteness is a physical manifestation as diffusion can only be zero at absolute zero (a great story for sci-fi.) I picked one of the many extensions just to test my workflow and here are the results. Pretty good I’d say for all the complications.

Lastly, opensource visualization tools seem to trip over the simplest of tasks. MayaVi seems to be the only one that can update the VTK pipeline when the source data changes. In other programs I have to rebuild the pipeline each time. As you can imagine, this gets tiring really fast.