Fernando Ipar

emacs config update

Mon, 11 Sep 2023 00:00:00 -0500

Time for the sporadic update about my emacs configuration journey.

I’m now on 29.1, and as part of the upgrade process, I refactored my config into more, smaller files, to make it easy to selectively disable features while troubelshooting startup problems.

Since I use both a personal and a work machine, maintaining emacs’ configuration in sync has been a challenge. I’ve now gone back to keeping this on a private git repo, and I created a simple macro to support adding config snippets depending on the machine name:

For example, at work I’m using Github Copilot (through the neovim plugin and https://github.com/zerolfx/copilot.el) and I set this up on a file named copilot.el on my ~/.emacs.d/ with this content:

I’m now simplifying my shortcuts a little bit by using hydras, resulting in things like this:

Finally, I’m now using modus vivendi, which is now included on 29 (so I’m linking that repo just as a reference).

Documenting emacs hyper shortcuts

Wed, 29 Apr 2020 00:00:00 -0500

I have written before about my discovery, and adoption of emacs shortcuts based on the Hyper key, and this time I want to share a simple script to generate basic documentation of them. The script works on the assumption that shortcuts that involve defining another shortcut (i.e., using kbd) will include some user-friendly documentation in the form of a comment, and will include this, instead of the kbd form, in the documentation. This is not perfect. For example, sometimes an extra closing parenthesis will show up. But perfect is the enemy of good, and this output is good enough for me right now.

The color scheme is because I have physical sticker labels attached to some of my keys, and I am using [ and ] as ‘map’ keys to have two more layers of hyper shortcuts for each existing one (say, if ‘H-f’ focuses on the current org subtree, ‘H-] f’ reverts that, and I call the second form ‘Hyper Map 1 f’), and for mnemonics I have colored [ green, and ] red, and I am using the colors on individual key labels to let me know what is there for each level.

Why a pdf with the shortcuts if I have the labels, you may ask? Because while some keys are set in stone (I don’t anticipate to ever change the shorcuts for getting me the day’s agenda, or for invoking capture), others are more dynamic, and depend on how frequently I use (or stop using) a specific function. It’s a pain to be relabeling these so I’m now just relying on the pdf document instead.

Given the assumption described before, here’s the script that gets me a pdf with my shortcuts (that looks like the one shown in the post’s image):

Questions to ask when evaluating a database

Fri, 17 Apr 2020 00:00:00 -0500

I’ve been evaluating a few databases recently while working on a way to process application logs, and here’s a list of questions I ask myself during the process.

This list is biased to operations, so I’m explicitly ignoring equally important questions that apply to the development and usage side, like “can it enforce user-defined integrity constraints on the data?”.

Without further ado, I want to know what happens when

The process crashes,
The OS crashes,
It runs out of disk space,
It runs out of memory,
A system call (e.g., fork) fails,
The system clock goes back (or forward) in time,
The process is SIGSTOPped and SIGCONTed later,
Multiple copies of the process are started with the same (or a different) config,
The storage subsystem silently changes data on the write path,
DNS fails,
The network fails, or its performance degrades, or packets arrive out of order,
The hostname changes,
There’s a surge in new connections,
A hardware resource saturates.

More questions can be considered, (usually some will come up as subquestions while answering one of the above), but I’ve found this is a good enough list to start working on tests to verify how the database will behave for the use case.

And what about the pic, you ask? It’s a dead fish on the beach. Fish out of water. I want to know how the database dies, so to speak. Does it make an irrecoverable mess? Does it die as cleanly as possible and make an autopsy possible?

Minimal networking knowledge recommended for DBAs

Wed, 25 Sep 2019 00:00:00 -0500

This article is a cross post from Pythian’s blog

In a recent post, I summarized what I consider to be the minimal topics one should be familiar with in order to do technical work with databases.

One thing I left out, except for it being implicitly included in “Hardware Basics” and “Operating Systems Basics”, is networking. In this post, I’ll go through what I consider the minimal networking knowledge one needs to know when working with databases.

Protocol Concepts

While there are some niche network protocols in use (in my experience, when working with mainframes, for example), database practitioners may well spend their whole lives dealing mostly or only with TCP/IP, so that’s where I suggest focusing your learning efforts.

A great guide to the basics is “TCP/IP Illustrated, Vol 1: The Protocols”. While a bit old, the fundamentals are still relevant, and the book is very clearly written, and packed with tcpdump captures supporting the explanations. I think a good way to learn with it, while then checking out what has changed since it was written, is going to the RFC index and checking out RFCs mentioned in the book for a protocol you’re studying. If more recent RFCs are available, they’ll be mentioned in the index with an “obsoleted-by” note.

This is a vast field, and not all of it is of immediate relevance to a database practitioner, so I’m providing the following list of what I consider the core networking knowledge you’ll need to work with (and primarily to troubleshoot) databases:

TCP’s three-way handshake
TCP connection termination
The MSL and its relationship with connection termination and establishment
Slow start
Flow Control
How ARP works
How DNS works
The MTU and Path MTU
Routing, with emphasis on routing errors and their meaning (for example, what’s the difference between no route to host and a timeout?)
Basics of UDP and what it does not provide when compared with TCP (if you’re troubleshooting something that uses it, like Galera)

Networking Tools

I’ll mention tools available on GNU/Linux here because that’s the most popular OS by far for the databases I currently work with, but once you know the protocol basics, you’re just one quick Internet-search away from discovering which tool you must use on another OS (for example, entering “How to find out my IP in Windows” in Google gets me instructions on how to use ipconfig on the very first result returned). If you’re not on GNU/Linux but you are working on another OS that’s part of the Unix family tree, there are two useful resources that can help you:

The Unix Rosetta Stone,
The USE Method’s Rosetta Stone.

You’ll need to become familiar with the following commands to find out which IP address the host you’re working has, how to change it, or add/remove an alias, and how to modify the routing table or the firewall rules. For this, I recommend reading the man pages for the following commands:

ip (You may find a lot of older resources online mentioning ifconfig, which is what I learned to use, but ip is what should be used now),
ethtool,
arp,
iptables/ip6tables,
netstat or ss.

Sometimes, you’ll end up working on a puzzling database problem that involves the network, and you’ll want to know if the client is sending what you actually mean to send, the server is sending the response the client ends up seeing, etc. In those (and other) cases, you’ll want to capture network traffic and analyze it to see what is being sent over the pipe.

For this, I recommend getting familiar with tcpdump and wireshark. I also recommend that you get familiar with these tools as you learn the protocols because it is an interrelated process:

You’ll read how the three-way handshake works.
You’ll want to go and establish a TCP connection while capturing traffic, then analyze the traffic and see if it matches what the protocol says should happen.

This is how the examples in the Stevens book I recommended earlier were made, but I think it’s priceless to come up with your own captures as you learn this.

Wrapping up

As database professionals, while we are specifically responsible for the health of database systems, our world does not end up there. Imagine a colleague (or client) that can’t connect to the database. You can’t just say, “The database logs show no connection attempts with your user; there’s nothing I can do.” You can instead get into a joint troubleshooting session, verifying that the client machine can reach the database server, at the right port, that a connection can be established, etc. When the problem is in the network, sometimes the solution is out of your hands (imagine a firewall sitting between you and the user, filtering traffic). But if you go to the network team with substantial evidence that shows how the connection can’t be established because it times out, despite the right routes being available, it will make their job easier, which in turns lets them be more effective at helping you resolve the problem. Great engineers always go the extra mile. Plus, troubleshooting network problems will be a welcome distraction from troubleshooting database problems!

Top 8 things every database practitioner should know

Thu, 05 Sep 2019 00:00:00 -0500

This article is a cross post from Pythian’s blog

I’ve been interviewing and mentoring candidates for technical database positions for almost ten years, and a common question from those who either don’t make it or feel they’re not ready yet to apply to the position they’re looking for is: What else should I learn to prepare for this?

This post is an attempt at summarizing my advice in this regard, covering what I think are the most important minimum topics one should be familiar with when applying for any position that involves doing technical work with databases.

They are:

Database Models
Transactional concepts
Data structures
Consensus algorithms
Development and automation fundamentals
Hardware basics
Operating Systems Basics
Performance Optimization

Database Models

You need to start at the beginning, and when it comes to databases, I think that’s database models. A database model can be considered a specification for how data will look like to the users of the database, including which data types and operations are supported. Unless you’re developing a database system from scratch, you never choose a model directly; instead, you pick it by proxy when you decide which database system to use.

For this reason, if your responsibilities include selecting a database based on a set of requirements, the only way to make an informed choice is by knowing which models are available, and which implementations they have.

While studying the models, always keep in mind that, as practitioners, we’ll be making choices based not just on ideal models but also on real-world constraints (e.g., budget constraints), so more often than not we’ll find ourselves bending the rules. As in the case of making informed decisions, the only way to properly bend (or break) the rules is by first knowing them very well.

I think the absolute minimum here is getting familiar with the relational model, as it inspired a lot of the database systems currently in use. You may have heard comments such as “when you get to a certain size, the relational model does not scale”. (A mistaken statement, since the relational model is a logical data model and has nothing to say about physical representation, so if anything, what does not scale is a specific implementation of the relational model. This statement makes as much sense as saying that arithmetic does not scale because your pocket calculator overflows at relatively small numbers, but I digress), but the fact is, there is still a vast market of job opportunities for people supporting database operations for medium-sized businesses, where a relational database implementation is a great fit.

Next in line would be the most popular contemporary non-relational models. This is the world of object, document, graph, and time-series databases, among others.

Finally, while not worth spending a lot of time (except to satisfy your historical curiosity), I think it pays to skim through pre-relational models, mostly so you become aware of 1) why the relational model became so popular, and 2) what pre-relational mistakes other contemporary products may be making. These would be the network and hierarchical models.

Several textbooks can help with this, but my bare minimum recommendation is to read “Database in Depth” by C.J.Date.

Transactional Concepts

Another critical concept in database systems is transaction support. You should be familiar with the ACID acronym and be able to evaluate the transactional guarantees of a database system, as well as the transactional requirements of a system you may design or build.

When using an SQL-based system, it’s essential to understand the transaction isolation levels provided by it. Make sure you go to the relevant documentation instead of relying just on the level names, as not all databases implement a given level in the same way.

As mentioned, the first resource to learn this should be the official documentation for the product you’re using or evaluating. If you want to go deeper, “Transactional Information Systems: Theory, Algorithms, and the Practice of Concurrency Control and Recovery” is an excellent resource.

For distributed systems, “Consistency Models” is a must-read.

Data Structures

Data structures define the organization of data both in memory and in persistent storage. The fundamental ones you need to be familiar with to work with databases are B Trees, LSM Trees, and Hashes.

Others that you may need to become familiar with depending on the products you use are Bitmap Indexes, Bloom Filters, and Sparse Indexes.

Finally, if you want to get involved in developing or customizing database systems (e.g., patching MySQL, PostgreSQL, etc.) you should also be familiar with more fundamental data structures such as lists, stacks, and queues, among others.

There are many sources of information about this, but I recommend 3:

Some “Fundamentals” textbook. I learned with Knuth (which, for data structures, involves Volumes 1 and 3), but there are several excellent texts out there. Just check out a few samples on the web to find one where the style suits you and go for it. Online university course notes and bibliography are a great resource for finding out “where do I learn more?” too. The paper presenting a specific data structure, or a good write up of it. For example, here’s one for LSM Trees (I recommend The Morning Paper as a friendly way to keep up with our field). The relevant source code, if available. Finally, if you want to design your own data structure, I recommend “The Periodic Table of Data Structures” to get ideas of what you could try.

Consensus Algorithms

As soon as our databases involve more than one system communicating, they need some protocol for how to agree on what the global system’s state should be.

Consensus algorithms are the theoretical foundation for this, and if you plan to work with distributed databases, you should become familiar (this means “can understand how it works and what guarantees are provided,” not “can implement it”) with some. The choice will be dictated by the system you use, but if you want to learn this before using a system, or, as in the case of learning database models, so you’re better informed when you have to choose which distributed system to use, I think it pays to become familiar with Raft as a starting point. Paxos can be an excellent second step, but bear in mind one of the key reasons why Raft was designed was to provide a consensus algorithm that was more understandable than Paxos, so be ready to spend some time if you want to go down this road (Notice I linked to “Paxos Made Simple” instead of to the original paper, even though they’re both by the same author).

Another good piece of advice I can give here is, as practitioners, we should treat consensus algorithms like we (hopefully) treat cryptographic algorithms: you don’t make up your own. Don’t get me wrong: it’s perfectly fine to try to create something new (in fact, I encouraged you to design your own data structure just a few paragraphs above), but this is not something you “learn by doing,” much less by writing production software.

Development And Automation Fundamentals

The first thing you need to become proficient at is the language of the database you work with. The only caveat is that, even if you currently don’t use an SQL-based database, I think you should still become familiar with it since it is supported (in some form of another, with several inconsistencies between systems) by most database systems currently in use. As in the case of transaction isolation levels, I think the official documentation from the product should be read. However, that can be supplemented here by a more general SQL reference. My recommendation would be “SQL and Relational Theory” by C.J.Date.

You should also be confident with scripting. Any scripting language available on the platform(s) you work should be good here. I’m a heavy bash user, but these days a lot of people go straight to Python for this. Unless the position you’re applying for lists a specific requirement, any language will do, since you’ll be able to apply the concepts from it into other scripting languages in the future if needed.

Since database work involves a lot of automation these days, it’s essential to become familiar with some DevOps concepts and tools. At the very least, you’ll want to know the basics of automatic provisioning with Ansible, Chef, or some other such tool.

If you want to work on an Open Source database and you enjoy programming, it’s a great idea to become familiar with its source code, and, in turn, with the programming language it is written in. This may seem like overkill at first, but it is priceless when you need to troubleshoot hard-to-track problems or understand stack traces and other error messages.

Hardware Basics

You should be familiar with the overall architecture of the actual computers where the database software runs.

Databases are meant (among other things) to provide a reliable and convenient abstraction for data storage so that users can just worry about their logic, and not care about how things are done under the hood. However, as the people supporting (I’m using the term broadly here) databases, we spend a significant part of our time under this hood.

Among other things, this means you should be able to find out and properly interpret hardware specs such as storage device IOPS, understand what hyperthreading is, etc. All of this is tied together and, for example, what drives you have available may end up dictating which data structure you’ll use (e.g., some are better at reducing wear for SSDs or dealing with slower storage).

There are several great books on computer architecture, but I think the best recommendation I can make for this and the next section is “Systems Performance: Enterprise and The Cloud” by Brendan Gregg.

Operating Systems Basics

This is very much tied with Hardware and involves you knowing about swapping/paging, what Virtual Memory is, how the I/O subsystem works (including how to make sure a write made it to persistent storage, or what schedulers you have available).

It’s also vital to know what tracing and monitoring tools your OS has available, so you can dig deeper when a performance problem requires it, and understand what resource is the bottleneck, for example.

Performance Optimization

Finally, you need to be proficient at suitable methods for performance analysis and optimization. Here, the Gregg book is recommended again, as it introduces the USE method, which I often use when diagnosing performance problems.

The book “Optimizing Oracle Performance” by Cary Millsap is also recommended. If you don’t plan to work on Oracle: don’t be put off by the title. I’ve never worked on Oracle, and I still found it very helpful. I’d say about 2/3rds of the book is generic and applies to whatever database system you work with, as it consists of the presentation of “Method R” for performance optimization.

As a final recommendation, for capacity planning, I recommend reading “Guerrilla Capacity Planning” by Neil Gunther. This book introduces the Universal Scalability Law and includes some heavy theoretical material (which I had to re-read several times) plus good practical examples for how to use it to estimate hardware and software scalability.

Parting words

I hope you found this useful. Besides studying all the things I’ve mentioned on this post, I have one last recommendation to make: get involved in an online community related (e.g., the MySQL Community Slack) to the technologies you want to get better at since frequently senior people are available and will provide insightful help to people asking questions.

My org-mode workflow

Wed, 23 Jan 2019 00:00:00 -0600

A recent Hacker News discussion inspired me to write a short post on my org-mode workflow, with the hope that it may help others.

The thing I like the most about org-mode is that it supports heavy customization. This is not surprising, since Emacs itself is all about customization. You don’t just use an editor, you build the editor you want using this Text Editing Framework, so to speak.

I came to emacs precisely because of org-mode, after enough people recommended it. I came from vim and the first turn-down for me where the keyboard shortcuts. I partially improved that by using evil-mode, which in my view gives you the best of two worlds: the power of Emacs and the ergonomics of VIM. That is, of course, if you think VIM is an ergonomic text editor (I do).

The second huge improvement to the usability of Emacs (and org-mode) for me came via the Hyper key. I have written about this before and while my setup has changed a bit since that was written, the idea still holds.

I use a Kinesis Freestyle2 Blue keyboard and combined with heavy use of the Hyper key (all but 7 keys on my keyboard are currently mapped as hyper key shortcuts!) and slight typing changes (Never use the pinky for Shift, remap Caps-Lock as Control, that kind of thing), I can spend my typing time without incurring in physical pain.

So enough for the intro, here’s how I use org-mode to manage my work and personal life:

Capture templates and a global Inbox

I use org-mode’s capture templates to create new entries whenever I need to get something off my head but without losing track of it. Typically that means creating follow-up To Do items after checking my email, getting a new ticket assigned to me, etc. This lets me go through my backlog at the start of the day (I have a daily “check-in” routine for this) and triage every item in the queue to make sure I don’t drop the ball on anything that’s urgent, while still allowing me not to waste mental space trying to track all tasks.

When going through the backlog, I classify things in three categories:

What I can do right now because I expect it to be fast. “Fast” is a subjective concept here. I never read the book, but from talking with others that have, my approach is inspired by Getting Things Done, and I believe the author sets the threshold at something that takes 5 minutes or less.
What I don’t need to do right now but I must do at some specific time. Things with a deadline. These go to a ToDo entry that is scheduled and has a deadline (Going back to my comment about shortcuts, I have Hyper keys setup to do both these things). It’s important to set this properly so that I can feel confident that once I’m done with the capture process, I can move on to the next item in the backlog and forget about this one, while still knowing I won’t drop the ball on it.
What I would like to do eventually, would be nice if I do, etc. Things that fall in this category tend to be ideas to improve a project, write a blog post, learn something new. Stuff I’d like to remember as “I would love to do this Some Day” but that is not time sensitive. I typically go to these items whenever I’m stuck, or I find myself procrastinating.

By default, my capture templates go to the global Inbox. This is a heading in my Todo.org file. I’ll talk more about files in the next section but for now what matters is that I will usually refile items to their file, or to their header in the global one.

By the end of this process, I have a better picture of how my day (and upcoming days) will look like, and I can plan how to tackle it accordingly.

My org files

As mentioned, I have a Todo.org file, which I call my “global” file. It holds my Inbox, my Personal heading (where I file anything personal that doesn’t merit a file of its own) and a Trashcan. I borrowed the idea of a Trashcan for ideas as opposed to just deleting the “bad ones” right away from Robert Pirsig’s description of his workflow in Lila. He used cards and my analogy is an org-mode heading (or subheading, etc) is the equivalent to one of his cards.

Besides this, I have some things in their own org file. Examples include:

Community.org to track work on community contributions, including ideas of the “would be nice to do” type, but also work with (usually soft) deadlines on bug reports and blog posts.
One file per each large-enough project I’m involved in at the moment, and/or for each active client. When going through my email, if I have a request from a client, after capturing it I will refile it to the appropriate file.
On-demand ephemeral files for things that I need on the spot. Say I need to hop on a call on short notice. I used to add a headline to an existing file and work there. Now I just create a file only for this, work there, and if needed, refile whatever is important to another place when the call is done. Otherwise, it’s easy to move the file to the Attic and be done with it. One of the goals of my setup is that I can both minimize “waste”, which I define as things that I don’t need and clutter my attention, while still letting me keep track of where I’ve been (something important so you have a decent chance of getting to where you’re going). I refer to this exit-target as Attic instead of Trashcan because I typically don’t delete things, unless I’m done with a client and the contract requires that I remove all information about this. Otherwise, having a history of my work as back as I have been doing it is quite useful to plan new things. Org-mode is all about text files so with today’s drives, there’s no way you can ever have disk-space problems because you have too many org files.

My snippets

I make use of snippets (via yasnippet) to help me work faster and avoid errors in repetitive work, but also to keep track of my steps. Specifically, on this last item, I have a ‘tsnow’ snippet that expands to the current timestamp (date + time) and when working on a case (say, a bug report), I will create an org header with a brief description of what I’m doing and then invoke the snippet. This creates a worklog that I can go back to when needed and is immensely helpful to me, especially if I need to interrupt my work for some time and then go back to it. When this happens, I also create a special log header called “Status at <tsnow>” where I do a short brain dump of what I’ve found so far and what I think my next steps should be. A nice side effect is that I can export the worklog in plain text (or HTML or PDF too) and upload it to a ticketing system.

Within each timestamped header I also make use of snippets to create EXAMPLE and SRC blocks as needed, to include code snippets and console output of what I have been doing. For more info on these type of blocks, I refer you to the fine manual.

Quite often, these worklogs become narratives with code interweaved. I like to think of this as “Literate Troubleshooting”, but that may be just because I am a heavy Knuth fan :)

Agenda

Org provides an agenda view and while by default it works fine, I have a custom view that shows me anything that is scheduled for today (or for another day in the week, in the weekly view) in the IN PROGRESS, TODO, and SCHEDULED states. In org mode, headers can have a state, and those are useful to organize your workflow. Mine are: TODO, IN PROGRESS, SCHEDULED,DONE, WAITING FOR FEEDBACK, WAITING ON CUSTOMER, SUSPENDED, REASSIGNED, and CANCELLED.

Focusing

While org-mode is very nice to organize your thoughts, tasks, and many other things, multi-header org files can get cluttered quickly. If this becomes too much cognitive noise to you (it does to me at times) I recommend using the focus and widen options. I have each of this assigned to a Hyper key so that I can easily make anything except that heading I’m currently interested in, disappear, and then reappear as needed.

Tracking time

If tracking time is important for you, org’s clock will come in handy. I basically run with the clock on all the time, and just have it switch automatically as I go from heading to heading, which is kind of saying from task to ask. Off for lunch? No problem, I just use my “Interruption” capture template before going afk. As a result, at the end of the week, I can get a nice report of where I’ve spent my time.

I hope you found this useful, and I hope you give Emacs and org-mode a chance. They’re both beautiful software that can help you get a lot done.

What is a good cache hit ratio?

Sat, 19 May 2018 00:00:00 -0500

What is a good cache hit ratio?

Taking MySQL as an example: what is a good query cache hit ratio?

At the time of writing this, I get over 600 results if I enter that question into Google and limit the results to stackexchange.com. If I change ‘ratio’ to ‘rate,’ that increases to over 3000 results, which makes sense, as most people argue that it’s the rate what you usually want to know about.

There are several answers to any of these questions, but I prefer to borrow from Hofstadter’s and Pirsig’s interpretation of a Zen Koan and answer ‘Mu’: unask the question.

What that means to me (and I am aware this is my interpretation of other people’s interpretation of a concept in a culture foreign to them) is that it does not make sense to answer the question given the context. To be more specific, I don’t think a (MySQL query) cache hit ratio can be good or bad. It can tell us something about a workload but nothing more. Additionally, what it tells us is aggregate information about this workload.

Let’s say the hit rate is 60% hits over an hour. It does not tell us which queries were hits and which misses, nor what is the range of response time for a given query fingerprint (to borrow pt-query-digest’s term). Without this info, it does not tell us if the query cache is beneficial for the workload or not. The same applies for a hit rate of 90% hits over an hour, by the way, as explained by Peter Zaitsev for memcached in this post.

The reason I think this is not a valid question, at least not in a broad performance optimization/application health check context is that it tells us nothing about the things that actually matter about a database: is work being done at the rate and response time the client (or users) expect? Quoting Cary Millsap, there are only three acceptable units for (database) statistics that make sense to end users:

your local currency
The duration by which you’ll improve someone’s response time
The number of business actions per unit of time by which you’ll improve someone’s throughput

A cache hit rate is measured in hits by time, which is none of the above because such a ratio tells you nothing useful about costs (or savings), response time or throughput.

Running sysbench-based benchmarks against Cassandra

Mon, 26 Mar 2018 00:00:00 -0500

This article is a cross post from Pythian’s blog

I was recently discussing benchmarking options for Cassandra with some colleagues and given my background with MySQL, sysbench was the first tool I thought of.

Sysbench is a high performance and flexible benchmark tool that can be used to run both Database and Operating System based experiments. In case you’re not familiar with it, there’s an excellent introduction to it by my colleague Martin Arrieta here.

One interesting thing about this tool is that while it includes out of the box support for MySQL and PostgreSQL, it uses Lua for scripting so other databases can be supported provided there’s a Lua driver for it.

Naturally, that means my quest for a way to run sysbench against Cassandra started with the search for a Cassandra Lua driver, which led me to https://github.com/thibaultcha/lua-cassandra

The next task was looking for a simple way to deploy benchmark clients without the need to install dependencies on the host OS. These days my answer to that involves Docker and what I found for this was a handy image from Severalnines: https://hub.docker.com/r/severalnines/sysbench/

This image was a good starting point but did not fully support my use case as I need to install custom Lua modules on the container, which requires installing some additional packages on it.

Given my full stack for this is Open Source I went ahead and modified the Dockerfile for this image to add what I needed. I felt this was a change that could benefit others too as I’m probably not the only one using sysbench to run experiments against databases that don’t have a driver bundled with it, so I submitted the following PR which has been merged already: https://github.com/ashraf-s9s/sysbench-docker/pull/1

Putting it all together, I can now launch sysbench against a Cassandra cluster to test the performance of different schemas and workloads.

To give a simple example let’s consider the following Lua script:

    #!/usr/bin/env sysbench

    function event()
    local cassandra = require "cassandra"

    local peer= assert(cassandra.new {
	host = "172.17.0.2",
	port = 9042,
	keyspace = "test"
    })

    assert(peer:connect())

    assert(peer:execute("select * from user"))
    peer:close()
    end

A couple of comments about the script:

172.17.0.2 is the IP address of the Cassandra node I want to connect to. In my case, this is another container, but be sure to change this as needed if you want to reproduce this test (or refer to this gist to see how I ran mine). For the script to work, the Cluster must have a ‘test’ keyspace with a ‘user’ table on it (as you can see from the query, the table structure does not matter here). We can use sysbench to execute it via docker like so:

    telecaster:sysbench-docker fipar$ docker run -v ~/src/:/src/ --name=sb -it severalnines/sysbench bash -c 'luarocks install lua-cassandra --local; luarocks install luasocket --local; /src/tmp/benchmark1.lua run'
    Warning: The directory '/root/.cache/luarocks' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing /usr/local/lib/luarocks/rocks/luarocks/2.4.3-1/bin/luarocks with sudo, you may want sudo's -H flag.
    Installing https://luarocks.org/lua-cassandra-1.2.3-0.rockspec
    ...snip...
    Installing https://luarocks.org/luasocket-3.0rc1-2.src.rock
    ...snip...
    sysbench 1.0.13 (using bundled LuaJIT 2.1.0-beta2)

    Running the test with following options:
    Number of threads: 1
    Initializing random number generator from current time


    Initializing worker threads...

    Threads started!


    General statistics:
	total time:                          10.0017s
	total number of events:              2792

    Latency (ms):
	    min:                                    2.17
	    avg:                                    3.57
	    max:                                   48.12
	    95th percentile:                        4.65
	    sum:                                 9958.60

    Threads fairness:
	events (avg/stddev):           2792.0000/0.00
	execution time (avg/stddev):   9.9586/0.00 You can see I am installing the required Lua modules when starting the container. If a benchmark will be executed several times (which is usually the case) a better approach would be to further customize the Dockerfile to include the necessary modules. I have not done that in the PR though because I think that would bloat the existing image. You can also see that I’m making my machine’s src directory available on the container via the -v ~/src/:/src/. That’s why I can then execute the script from /src/ on the container. Be sure to adjust this as needed to point to a directory tree where the lua script can be found on your machine.

In conclusion, if you have benchmarking needs and have not considered sysbench, don’t be put off if your database of choice is not listed as supported: as long as there’s a Lua driver for it there’s a good chance that you will be able to use sysbench for the task!

Measuring the potential overhead of pmm-client on MySQL workloads

Mon, 22 Jan 2018 00:00:00 -0600

This article is a cross post from Pythian’s blog

Having good historial metrics monitoring in place is critical for properly operating, maintaining and troubleshooting database systems, and Percona Monitoring and Management is one of the options we recommend to our clients for this.

One common concern among potential users is how using this may impact their database’s performance. As I could not find any conclusive information about this, I set out to do some basic tests and this post shows my results.

To begin, let me describe my setup. I used the following Google Cloud instances:

One 4 vCPU instance for the MySQL server
One 2 vCPU instance for the sysbench client
One 1 vCPU instance for the PMM server

I used Percona Server 5.7 and PMM 1.5.3 installed via Docker. Slow query log was enabled with long_query_time set to 0 for all tests.

I ran sysbench with 200k, 1M and 10M rows using the legacy oltp script with the pareto and special distributions, and up to 64 client threads. After several runs and reviews of the results, I settled on 1M rows with pareto for extended tests, as other combinations showed minor variations on the results from this one.

I am well aware a synthetic workload is not representative but I think the results are still useful, though I would love to measure this on a real life workload (do let me know in the comments if you have done this already).

In a nutshell, I found some impact in performance (measured as throughput in transactions per second) when running sysbench with the PMM exporters which in my case was eliminated when I configured them to serve their metrics by HTTP instead of HTTPS.

The following graph shows box plots for throughput for pmm enabled or disabled, for a different number of threads, with and without ssl:

We can see that with SSL enabled there is a noticeable drop in throughput when the exporters are running, while this is not the case when SSL is disabled.

I arrived at the conclusion that it was worth repeating the tests with SSL disabled after creating Flame Graphs from perf captures during sample runs. On them, the only significant increases were due to the exporters (mysqld_exporter and node_exporter, the qan exporter did not have any noticeable impact during my tests). The results from the tests show that this analysis pointed me in the right direction so while they are worth of separate blog posts, it is worth to at least recommend our readers to get familiar with this performance analysis tool.

Next is a scatter plot of throughput over time with ssl enabled:

On it we get a more clear picture of the impact of having the exporters running during the test.

Next is the same graphs but with SSL disabled:

Now it is much more difficult to differentiate the runs.

This is confirmed if we look at the 99 percentile of throughput for each case (here for 32 threads):

PMM	SSL	tps (p99)
enabled	enabled	1167.518
enabled	disabled	1397.397
disabled	disabled	1429.097

Conclusion PMM is a very good Open Source option for monitoring but as every instrumentation and monitoring layer you add to your stack, it won’t come for free. My very simple tests show that its impact may be significant under some scenarios, yet if it’s bad enough it may be mitigated by using HTTP instead of HTTPS for the exporters. Given the events that are unfolding in IT security as I type this, it may seem reckless to recommend disabling SSL as an “optimization”, but I think good engineering is all about informed tradeoffs and if you’re running this on a secure private network, how risky is it to expose monitoring metrics over HTTP instead of HTTPS? I would love to read answers to this question in the comments!

Finally, I think a similar cost is probably paid for the TLS layer on the pmm-server end. It would be very interesting to see an experiment like this repeated but on a different scenario: one pmm-server with several monitored clients.

Expanding your keyboard on emacs with the Hyper key

Tue, 02 Jan 2018 00:00:00 -0600

I recently stumbled upon the hyper and super keys on emacs here.

What I like about this setup is that I rarely use the Function key in my day to day usage, so this is making good use of a key that’s both mostly unused and conveniently located in my keyboard, to the effect of running potentially complex tasks (like exporting the currently selected org mode subtree to a pdf and opening it) with just two keys.

Here’s my current configuration:

;; hyper key shortcuts: http://ergoemacs.org/emacs/emacs_hyper_super_keys.html
(setq ns-function-modifier 'hyper)  ; make Fn key do Hyper

(global-set-key (kbd "H-a") 'org-agenda)
(global-set-key (kbd "H-d") 'delete-other-windows)
(global-set-key (kbd "H-i") 'org-clock-in)
(global-set-key (kbd "H-o") 'org-clock-out)
(global-set-key (kbd "H-s") 'org-schedule)
(global-set-key (kbd "H-b") 'ido-switch-buffer)
(global-set-key (kbd "H-r") 'org-archive-subtree-default)
(global-set-key (kbd "H-m") 'evil-mc-make-all-cursors)
(global-set-key (kbd "H-n") 'evil-mc-undo-all-cursors)
(global-set-key (kbd "H-k") 'epa-encrypt-region)
(global-set-key (kbd "H-l") 'epa-decrypt-region)
(global-set-key (kbd "H-j") 'eww)
(global-set-key (kbd "H-v") 'darkroom-mode)
(global-set-key (kbd "H-x") 'toggle-frame-fullscreen)


(defun open-ansi-term-localhost ()
(interactive)
(ansi-term "/bin/bash" "localhost"))

(global-set-key (kbd "H-t") 'open-ansi-term-localhost)
;; C-c C-e C-s l o  export current subtree to latex pdf and open
(global-set-key (kbd "H-e") (kbd "C-c C-e C-s l o"))
;; same, but html buffer and open
(global-set-key (kbd "H-w") (kbd "C-c C-e C-s h H"))
;; same, but ascii and open
(global-set-key (kbd "H-q") (kbd "C-c C-e C-s t A"))
;; plain org export
(global-set-key (kbd "H-y") 'org-export-dispatch) 
;; org capture
(global-set-key (kbd "H-c") 'org-capture) 
;; eval lisp
(global-set-key (kbd "H-z") 'org-ctrl-c-ctrl-c)

Something interesting here is that the target can be a function or another keyboard combination. The latter is useful if you can’t easily find the function you want, or if you want to pass arguments along with it, like I am doing in my org export shortcuts.

As for how to keep up with the growing amount of hyper key based shortcuts, I am using the old school technology of little sticker labels with a descriptive text on each key. It’s cheap and works wonders until I build the corresponding muscle memory.

Nothing groundbreaking for seasoned emacs users but I thought it could be useful to newbies such as myself (I have been using the editor in one form or another for several years already, but with evil-mode, and using only a small subset of its features).