Back to landing page

Some random links

Some of these I've saved through Reminders while browsing from my phone. Those usually end up with the source website as part of the name. Some are duplicates.

Write to us at [email protected] with any questions. More information: https://penrose.ink/siggraph20.html

This is largely a discoverability problem1. The solution the Internet has settled on seems to be “Emacs distributions” like Doom, Spacemacs or Prelude that glue together dozens (sometimes hundreds) of addons to deliver a batteries included, finely tuned and user-friendly experience from first launch. While it’s not for me, this does work great 2, and many of these packages will probably make their way into the default Emacs experience in due time.

root@kali:~# binwalk -B ddwrt-linksys-wrt1200ac-webflash.bin DECIMAL HEXADECIMAL DESCRIPTION ---------------------------------------------------------------------------–— 0 0x0 TRX firmware header, little endian, image size: 37883904 bytes, CRC32: 0x95C5DF32, flags: 0x1, version: 1, header size: 28 bytes, loader offset: 0x1C, linux kernel offset: 0x0, rootfs offset: 0x0 28 0x1C uImage header, header size: 64 bytes, header CRC: 0x780C2742, created: 2018-10-10 02:12:20, image size: 2150281 bytes, Data Address: 0x8000, Entry Point: 0x8000, data CRC: 0xA097CFEA, OS: Linux, CPU: ARM, image type: OS Kernel Image, compression type: none, image name: "DD-WRT" 92 0x5C Linux kernel ARM boot executable zImage (little-endian) 2460 0x99C device tree image (dtb) 23432 0x5B88 xz compressed data 23776 0x5CE0 xz compressed data 2117484 0x204F6C device tree image (dtb) 3145756 0x30001C UBI erase count header, version: 1, EC: 0x0, VID header offset: 0x800, data offset: 0x1000

Commonly in Australia and New Zealand, "cutting down the tall poppy" is used to describe those who deliberately put down another for their success and achievements.In Japan, a similar common expression is "the nail that sticks up gets hammered down".

The paradox may have analogies in electrical power grids and biological systems. It has been suggested that in theory, the improvement of a malfunctioning network could be accomplished by removing certain parts of it. The paradox has been used to explain instances of improved traffic flow when existing major roads are closed.

The absolute temperature (Kelvin) scale can be understood loosely as a measure of average kinetic energy. Usually, system temperatures are positive. However, in particular isolated systems, the temperature defined in terms of Boltzmann's entropy can become negative. The possibility of negative temperatures was first predicted by Lars Onsager in 1949. Onsager was investigating 2D vortices confined within a finite area, and realized that since their positions are not independent degrees of freedom from their momenta, the resulting phase space must also be bounded by the finite area. Bounded phase space is the essential property that allows for negative temperatures, and can occur in both classical and quantum systems. As shown by Onsager, a system with bounded phase space necessarily has a peak in the entropy as energy is increased. For energies exceeding the value where the peak occurs, the entropy decreases as energy increases, and high-energy states necessarily have negative Boltzmann temperature. A system with a truly negative temperature on the Kelvin scale is hotter than any system with a positive temperature. If a negative-temperature system and a positive-temperature system come in contact, heat will flow from the negative- to the positive-temperature system. A standard example of such a system is population inversion in laser physics. Temperature is loosely interpreted as the average kinetic energy of the system's particles. The existence of negative temperature, let alone negative temperature representing "hotter" systems than positive temperature, would seem paradoxical in this interpretation. The paradox is resolved by considering the more rigorous definition of thermodynamic temperature as the tradeoff between internal energy and entropy contained in the system, with "coldness", the reciprocal of temperature, being the more fundamental quantity. Systems with a positive temperature will increase in entropy as one adds energy to the system, while systems with a negative temperature will decrease in entropy as one adds energy to the system.Thermodynamic systems with unbounded phase space cannot achieve negative temperatures: adding heat always increases their entropy. The possibility of a decrease in entropy as energy increases requires the system to "saturate" in entropy. This is only possible if the number of high energy states is limited. For a system of ordinary (quantum or classical) particles such as atoms or dust, the number of high energy states is unlimited (particle momenta can in principle be increased indefinitely). Some systems, however (see the examples below), have a maximum amount of energy that they can hold, and as they approach that maximum energy their entropy actually begins to decrease. The limited range of states accessible to a system with negative temperature means that negative temperature is associated with emergent ordering of the system at high energies. For example in Onsager's point-vortex analysis negative temperature is associated with the emergence of large-scale clusters of vortices. This spontaneous ordering in equilibrium statistical mechanics goes against common physical intuition that increased energy leads to increased disorder.

This uses the special `:memory:` filename to open an in-memory database. Then it uses two `-cmd` options to turn on CSV mode and import the `taxi.csv` file into a table called `taxi`. Then it runs the SQL query.

You can get `taxi.csv` by downloading the compressed file from [here](https://github.com/multiprocessio/dsq/blob/43e72ff1d2c871082fed0ae401dd59e2ff9f6cfe/testdata/taxi.csv.7z) and running:

7z e -aos taxi.csv.7z

I figured this out while commenting on [this issue](https://github.com/multiprocessio/dsq/issues/70).

The output looks like this: 

"",128020,32.2371511482553
0,42228,17.0214016766151
1,1533197,17.6418833067999
2,286461,18.0975870711456
3,72852,17.9153958710923
4,25510,18.452774990196
5,50291,17.2709248175672
6,32623,17.6002964166367
7,2,87.17
8,2,95.705
9,1,113.6

Add `-cmd '.mode column'` to output in columns instead: 


$ sqlite3 :memory: -cmd '.mode csv' -cmd '.import taxi.csv taxi' -cmd '.mode column' \
    'SELECT passenger_count, COUNT(*), AVG(total_amount) FROM taxi GROUP BY passenger_count'
passenger_count  COUNT(*)  AVG(total_amount)
---------------  --------  -----------------
                 128020    32.2371511482553
0                42228     17.0214016766151
1                1533197   17.6418833067999
2                286461    18.0975870711456
3                72852     17.9153958710923
4                25510     18.452774990196
5                50291     17.2709248175672
6                32623     17.6002964166367
7                2         87.17
8                2         95.705
9                1         113.6

Or use `-cmd '.mode markdown'` to get a Markdown table: 

| passenger_count | COUNT(*) | AVG(total_amount) |
|-----------------|----------|-------------------|
|                 | 128020   | 32.2371511482553  |
| 0               | 42228    | 17.0214016766151  |
| 1               | 1533197  | 17.6418833067999  |
| 2               | 286461   | 18.0975870711456  |
| 3               | 72852    | 17.9153958710923  |
| 4               | 25510    | 18.452774990196   |
| 5               | 50291    | 17.2709248175672  |
| 6               | 32623    | 17.6002964166367  |
| 7               | 2        | 87.17             |
| 8               | 2        | 95.705            |
| 9               | 1        | 113.6             |

A full list of output modes can be seen like this: 

% sqlite3 -cmd '.help mode'
.mode MODE ?TABLE?       Set output mode
   MODE is one of:
     ascii     Columns/rows delimited by 0x1F and 0x1E
     box       Tables using unicode box-drawing characters
     csv       Comma-separated values
     column    Output in columns.  (See .width)
     html      HTML  code
     insert    SQL insert statements for TABLE
     json      Results in a JSON array
     line      One value per line
     list      Values delimited by "|"
     markdown  Markdown table format
     quote     Escape answers as for SQL
     table     ASCII-art table
     tabs      Tab-separated values
     tcl       TCL list elements

## Other options

There are a whole bunch of other tools that can be used for this kind of thing!

My own [sqlite-utils memory](https://simonwillison.net/2021/Jun/19/sqlite-utils-memory/) command can load data from JSON, CSV or TSV into an in-memory database and run a query against it. It's a LOT slower than using `sqlite3` directly though.

[dsq](https://github.com/multiprocessio/dsq) is a tool that does this kind of thing (and a lot more). Author Phil Eaton compiled [a collection of benchmarks](https://github.com/multiprocessio/dsq#benchmark) of other similar tools, and his [benchmarking script](https://github.com/multiprocessio/dsq/blob/43e72ff1d2c871082fed0ae401dd59e2ff9f6cfe/scripts/benchmark.sh) demonstrates how to use each one of them.

While cliche nowadays, you’re no doubt familiar with the “80/20 rule” in data analytics, and probably even experienced it yourself: 80% of your time is spent gathering, cleansing, and storing data, while 20% of your time is spent actually analyzing it and getting real work done. You often end up stuck between the document model of JSON and the relational model of SQL databases.