Analysing data files by drawing pictures

Not many people have to analyse completely undocumented data files to see what they are – but, every now and then, I do.

One surprisingly useful technique is to map the possible values of a byte (0 to 255) to unique colours, then display the file as a picture, one pixel per byte.

Here’s an old data file:

But, that’s just drawing the pixels out at a random width (250 pixels wide, in this case).

Starting a new line after every 269 pixels, you get this:

Note the diagonal lines. Since the lines are at 45 degrees, and slanting top-left to bottom right, you can tell that the display width needs to be 1 more pixel wide.

Drawing the file out with 270 pixel rows, one gets:

Nice vertical lines show that one has found the record size, the first step on decoding the data.

Other types of file don’t display so nicely. Here’s a couple of graphics files (a .gif and a .png)

No obvious patterning, but there’s still information to gain. In particular, it’s clear that both files have a header section which has much more structure than the image data that follows it. The fact that a file contains two or more different types of data can be very useful – sometimes you would have to step through hundreds of pages with a hex editor  to spot it.

Wallpaper design for the new house?

The product of my messing around with a graphics program.

Mechanics just don’t try as hard anymore

From “Popular Science” magazine, 1930.

 

 

And, the same magazine, 1940.

 

Salvador Dali, taking his pet anteater for a walk

Not your everyday sort of pet.

(Not sure where I swiped this image from, there’s many copies of it out on the aetherweb.)

 

A wonderful use for all those dismantled computer parts

One of Brenda Guyton‘s clever pieces.

 

The benefits of a classical education

 

Again, from an old Playboy magazine, I suspect.

Some (older) proverbs

This was two lists of proverbs, one English and the other African, published as a competition (to match them up) in Popular Science magazine in 1926.

ENGLISH:

1. Married in haste, we repent at leisure.
2. Answer a fool according to his folly.
3. One swallow does not make a summer.
4. Out of the frying-pan into the fire.
5. Robbing Peter to pay Paul.
6. Birds of a feather flock together.
7. First catch your hare.
8. Sour grapes.
9. Adding insult to injury.
10. Curses come home to roost.
11. Distance lends enchantment.
12. Milk for babes.
13. We can all endure the misfortunes of others.

AFRICAN:

1. One tree does not make a forest.
2. I nearly killed the bird. No one can eat nearly in a stew.
3. Full-belly child says to hungry-belly child, “keep good cheer.”
4. Distant firewood is good firewood.
5. Ashes fly in the face of him who throws them.
6. If the boy says he wants to tie the water with a string, ask him if he means the water in the pot or the water in the lagoon.
7. Cocoanut is not good for birds to eat.
8. He runs away from the sword and hides himself in the scabbard.
9. A fool of Ika and an idiot from Iluka meet together to make friends.
10. The ground-pig said: “I do not feel so angry with the man who killed me as with the man who dashed me on the ground after.”
11. Quick loving a woman means quick not loving a woman.
12. If the stomach is not strong, do not eat cockroaches.
13. No one should draw water from the spring to supply the river.

Interesting both in the comparison between cultures, and in way some of the once-common English proverbs have faded away.

After his death, the robot came back as C-Through-P-O

Using BabelFish for Story Ideas

A common enough technique – here’s the way I do it.

(1) Grab a bunch of headlines from your favourite news feed. Here I’ve grabbed the “Latest News Headlines” and “Editors Picks” from www.stuff.co.nz. You want short, punchy headlines. Strip out any numbers or timestamps.

Jeff Lima signs with South Sydney Rabbitohs
Quade has one more chance to impress Deans
NZ dollar down as euro weakens
Weak market dragged down
Rena captain and officer sent to jail
Gwaze jury locked in deliberations
Tobacco tax passes first hurdle
Google data shows Microsoft piracy problems
Belgrave Finance director faces jail
Budget strikes enviable balance – Key
Driver crashes into house, flees through fence
Roving cows bully drinkers out of beer
Google buys Motorola; Googlerola anyone?
Breaking down the daily grind

(2) Off to Yahoo! Babel Fish translation site http://babelfish.yahoo.com/.

(3) Paste the block of text (don’t worry about line breaks) and translate ENGLISH –> PORTUGUESE

Os sinais de Jeff Lima com Sydney sul Rabbitohs Quade têm uma mais possibilidade imprimir para baixo o dólar dos decanos NZ enquanto o euro se enfraquece Mercado fraco arrastado para baixo Capitão e oficial de Rena emitidos à cadeia Júri de Gwaze fechado nas deliberações Obstáculo das passagens do imposto do tabaco primeiro O diretor da finança de Belgrave dos problemas da pirataria de Microsoft das mostras dos dados de Google enfrenta a cadeia O orçamento golpeia o contrapeso invejável – chave O excitador deixa de funcionar na casa, fuje através dos bebedores recorrido da intimidação das vacas da cerca fora da cerveja Google compra Motorola; Googlerola qualquer um? Dividindo a moagem diária

(4) And again, this time PORTUGUESE –> FRENCH

Les signes de Jeff Lime avec Sydney sud Rabbitohs Quade ont un plus possibilités d’imprimer en bas le dollar des doyens NZ comme l’euro s’affaiblissent Marché faible traîné en bas Capitaine et agent de Renne émis à la chaîne Jury de Gwaze fermé dans les délibérations Obstacle des passages de l’impôt du tabac premier O directeur des finances de Belgrave des problèmes de la piraterie de Microsoft de tu montres des données de Google affronte à la chaîne budgétise frappe le contrepoids enviable – clé l’excitateur laisse fonctionner à la maison, fuje à travers les buveurs agréable de l’intimidation des vaches de près dehors de la bière Google achat Motorola ; Googlerola quiconque un ? En divisant la mouture quotidienne

(5) This time, we do FRENCH –> DUTCH

De tekens van Jeff Lime met zuidelijk Sydney Rabbitohs Quade hebben meer mogelijkheden d’ de dollar van de oudsten NZ als l’ beneden drukken; euro s’ verzwakken beneden meegesleepte Markt zwakke Kapitein en agent van Rendier die aan de lopende band Jury van Gwaze worden uitgezonden, die in de beraadslagingen Hinderpaal van de overgang van l’ is gesloten; belasting van de leidende tabak eerste O van de financiën van Belgrave van de problemen van de piraterij van Microsoft van jij trotseren de horloges van de gegevens van Google aan de lopende band in de begroting opnemen aanslag het benijdenswaardige tegengewicht – sleutel l’ excitateur laat aan het huis, fuje door de alcoholisten aangenaam van l’ werken; intimidatie van de koeien nauwkeurig buiten van het bier Google aankoop Motorola; Googlerola ieder die een? Door dagelijkse mouture te verdelen

(6) At last, back we go DUTCH -> ENGLISH

The signs of Jeff Lime with Southern Sydney Rabbitohs Quade have more possibilities d’ the dollar of the seniors NZ as l’ downstairs press; euro s’ weaken below dragged market weak captain and agent of reindeer who is transmitted to the conveyor jury of Gwaze, which in the deliberations obstacle of the passage of l’ has been closed; tax of leading tobacco first O of the finances of Belgrave of the problems of the piracy of Microsoft of you face the watches of the data of Google to the conveyor in the budget take attack the enviable counterweight – key l’ excitateur leave fuje agreeably to the house, by the alcoholics of l’ to work; intimidation of the cows closely of the beer Google purchase Motorola; Googlerola everyone which? By dividing daily mouture

(7) Now for the (virtual) highlighter pen. Look for short but slightly odd, or sometimes surreal, phrases. Notice that interesting chunks often came from the end of one mangled headline and the start of the next.

The signs of Jeff Lime with Southern Sydney Rabbitohs Quade have more possibilities d’ the dollar of the seniors NZ as l’ downstairs press; euro s’ weaken below dragged market weak captain and agent of reindeer who is transmitted to the conveyor jury of Gwaze, which in the deliberations obstacle of the passage of l’ has been closed; tax of leading tobacco first O of the finances of Belgrave of the problems of the piracy of Microsoft of you face the watches of the data of Google to the conveyor in the budget take attack the enviable counterweight – key l’ excitateur leave fuje agreeably to the house, by the alcoholics of l’ to work; intimidation of the cows closely of the beer Google purchase Motorola; Googlerola everyone which? By dividing daily mouture

(8) Collect the phrases.

• The dollar of the seniors.
• Weak captain and agent of reindeer.
• Transmitted to the conveyor jury.
• The deliberations obstacle.
• The finances of Belgrave.
• You face the watches.
• Attack the enviable counterweight.
• Intimidation of the cows.
• The beer Google.

(9) Look at them, and try to think of a scenario for each one, in which the phrase – or the idea behind it – makes sense.  How do you get appointed as an agent of reindeer? Who is intimidating the cows, and why was it neccessary? What obstacle prevents deliberations (perhaps the removal of chairs to make meetings shorter)?

1960s Targets of a Young Omnivore’s Lust

Two magic-users, two spacewomen, two secret agents (yes, I know one’s in there twice, but she deserves to be), a backwood’s girl, and a woman with a strange companion.