If you’re just querying for a word, it’s usage is pretty straightforward, and in fact, you can use the SOUNDS LIKE operator to build expressions such as this:

SELECT expr FROM table expr WHERE field0 SOUNDS LIKE ‘inputWord’

However, if you’re storing multiple-word strings, things get a little more complicated, since they can’t be compared by their soundex. Rather, the soundex returned will be associated with the whole phrase. If at a later time, you want to search for a subpart of this phrase, there’s no way for you to do this.

Well, at least not directly, but by using an auxiliary table to store soundex strings, a couple of stored procedures, and a trigger, it can be done with little effort to programmers that use the database.

Let’s assume we have a very simple table called soundex_text with the following structure:

create table if not exists soundex_text (
	id int unsigned not null auto_increment primary key,
	description text
) Engine = Innodb;

The field we want to query by proximity is description, therefore we create the following auxiliary table to store soundex values:

create table if not exists soundex_text_index (
	soundex_text_id int unsigned not null references soundex_text(id),
	soundex char(4)
) Engine = Innodb;

We now create a stored procedure and a trigger to populate the auxiliary table automatically every time a row is inserted in the main table.

-- adapted from example at http://forums.mysql.com/read.php?60,78776,242420#msg-242420
-- posted by jim smith: http://forums.mysql.com/profile.php?60,3154903
CREATE PROCEDURE update_soundex_text_index (sStringIn text,splitChar varchar(1), soundex_text_id int)
BEGIN
DECLARE comma INT DEFAULT 0;
DECLARE mylist TEXT DEFAULT sStringIn;
DECLARE temp TEXT DEFAULT '';
DECLARE strlen int DEFAULT LENGTH(sStringIn);
DECLARE insert_id int DEFAULT soundex_text_id;

/* find the first instance of the spliting character */
SET comma = LOCATE(splitChar,mylist);
/* Insert each split variable into the temp table */
WHILE strlen > 0 DO
	IF comma = 0 THEN
		SET temp = TRIM(mylist);
		SET mylist = '';
		SET strlen = 0;
	END IF;
	IF comma != 0 THEN
		SET temp = TRIM(SUBSTRING(mylist,1,comma-1));
		SET mylist = TRIM(SUBSTRING(mylist FROM comma+1));
		SET strlen = LENGTH(mylist);
		-- Sample handling of special chars you might want removed from individual words
		-- before storing their soundex.
		SELECT REPLACE(temp,',','') INTO temp;
		SELECT REPLACE(temp,';','') INTO temp;
		SELECT REPLACE(temp,':','') INTO temp;
	END IF;
	IF temp != '' THEN
		insert into soundex_text_index (soundex_text_id,soundex) values (insert_id,substring(soundex(temp) from 1 for 4));
	END IF;
	SET comma = LOCATE(splitChar,mylist);
END WHILE;

END//

drop trigger if exists soundex_text_bi//
create trigger soundex_text_bi
before insert
on soundex_text
for each row
begin
	SET @id = last_insert_id();
	call update_soundex_text_index (NEW.description, ' ', @id);
end;//
delimiter ;

Finally, a stored procedure to automatically query the table using the auxiliary table implicitly:

delimiter //
create procedure query_soundex_text(sStringIn text, splitChar varchar(1))
BEGIN
DECLARE comma INT DEFAULT 0;
DECLARE mylist TEXT DEFAULT sStringIn;
DECLARE temp TEXT DEFAULT '';
DECLARE strlen int DEFAULT LENGTH(sStringIn); 

create temporary table results (id int unsigned, description text);

/* find the first instance of the spliting character */
SET comma = LOCATE(splitChar,mylist);
/* Insert each split variable into the temp table */
WHILE strlen > 0 DO
	IF comma = 0 THEN
		SET temp = TRIM(mylist);
		SET mylist = '';
		SET strlen = 0;
	END IF;
	IF comma != 0 THEN
		SET temp = TRIM(SUBSTRING(mylist,1,comma-1));
		SET mylist = TRIM(SUBSTRING(mylist FROM comma+1));
		SET strlen = LENGTH(mylist);
		-- Sample handling of special chars you might want removed from individual words
		-- before storing their soundex.
		SELECT REPLACE(temp,',','') INTO temp;
		SELECT REPLACE(temp,';','') INTO temp;
		SELECT REPLACE(temp,':','') INTO temp;
	END IF;
	IF temp != '' THEN
		insert into results select st.id, st.description from soundex_text st, soundex_text_index sti where sti.soundex = substring(soundex(temp) from 1 for 4);
	END IF;
	SET comma = LOCATE(splitChar,mylist);
END WHILE;

select distinct * from results;
drop table results;

END//
delimiter ;

Using this is pretty straightforward.
Just insert some sample data into the table, and give it a shot!:

mysql> insert into soundex_text(description) values ('This is a sample text row');
Query OK, 1 row affected (0.00 sec)

mysql> call query_soundex_text('semple',' ');
+------+---------------------------+
| id   | description               |
+------+---------------------------+
|    3 | This is a sample text row |
+------+---------------------------+
1 row in set (0.00 sec)

Query OK, 0 rows affected (0.00 sec)

mysql> call query_soundex_text('THis is a samPLE taxt row',' ');
+------+---------------------------+
| id   | description               |
+------+---------------------------+
|    3 | This is a sample text row |
+------+---------------------------+
1 row in set (0.00 sec)

Query OK, 0 rows affected (0.00 sec)

mysql> call query_soundex_text('rock',' ');
Empty set (0.01 sec)

Query OK, 0 rows affected (0.01 sec)

To make it even easier, you could modify the querying stored procedure and always assume that the splitting character is a whitespace.

You can download a zipfile with a sql script with all this code for you to load into a MySQL database right here.

Enjoy!

Related posts:

1. Generating data with dbmonster In my last post I included some sample data...
2. Extending procedure_analyse My previous post explored a stored procedure that extended procedure_analyse...

Related posts brought to you by Yet Another Related Posts Plugin.

Why is it that writers, musicians and, well, any respected and professional artist or scientist gets his/her work reviewed before it’s published, but the same isn’t true in IT?

My guess? Most programmers aren’t neither artists, nor scientists. They’re an anonymous resource in a badly oiled machine that’s always behind schedule.

Well, so long, Knuth!

Related posts:

1. Symbolics Lisp Machine Wanna have your own Symbolics Lisp Machine? Here’s the info....

Related posts brought to you by Yet Another Related Posts Plugin.

So, without further ado,  here it goes:

#!/bin/bash 

[ $# -eq 0 ] && {
        echo "usage: salt <length>">&2
        exit
}
strings </dev/urandom | while read line; do
        echo $line | tr '\n\t ' $RANDOM:0:1 >> /tmp/.salt.$$
        salt=$(cat /tmp/.salt.$$)
        [ ${#salt} -ge $1 ] && salt=${salt:0:$1} && echo $salt && break
done
rm -f /tmp/.salt.$$

I had to use $1 and not a var, and echo the salt right from inside the while, because the ‘|’ creates another shell, so I can’t pass variables to or from the while in this case.

If you want to keep things simple, you can go perl and just do

#!/usr/bin/perl
use Crypt::Salt;
my $length = shift;
print salt($length);

But if you ever find yourself in a server with no cpan, the first option might prove useful

Related posts:

1. Generating data with dbmonster In my last post I included some sample data...
2. Running commands from the shell with a timeout (pt 2) Here’s an improved version of the safecmd script. This one...
3. Running commands from the shell with a timeout Sometimes, in a shell script, you need to run a...

Related posts brought to you by Yet Another Related Posts Plugin.

Highbase is currently comprised of several shell scripts  and some C code. It’s actually a good project (talk about self promotion) that hasn’t reached a stable release yet just because

1. It hasn’t been tested enough in production environments
2. I’ve been amazingly busy during the last years. Lots of work, and lots of parenting in the last two and a half years in particular.

Still, it’s real close to a release candidate, and I will continue testing and fixing the main branch. However, I believe writing a new version in Erlang has many advantages:

1. Erlang was designed from the ground up with reliability in mind (among other things. It’s quite popular now due to it’s multi-core-friendlyness, but that’s not really useful for a project like this), so it’s only natural to use it to develop a high availability solution (you know, the “best tool for the job” mentality)
2. Except for the mysql service verification (I wouldn’t want to go with odbc), and perhaps gratuitious ARP (I’m not sure at this point), rewriting in Erlang would remove all other dependencies in third party packages. In general, it would make the solution more mantainable and hence more reliable (see point 1).

I already wrote a first draft in Erlang about a year ago, but this was when I was just learning the language, so I didn’t take advantage of it’s best features for reliability. Actually, the problem was I wrote my version using ‘pure’ Erlang, dismissing the OTP part. Erlang as a language is OK (it has anything you need to write reliable,concurrent software). Erlang/OTP is a powerful thing, in the sense that it abstracts you, as a programmer, from the low level chores of distributed, concurrent and fault tolerant programming. You just have to focus on your programming logic, and by following some conventions (Behaviours, which for OO people could loosely be related to Interfaces and Base Clases) you get a lot of extra functionality for free.

How?

This new rewrite will be done using the gen_server behaviour, together with events. All this free to use from OTP.

Here’s a state diagram with an initial draft of the slave routine (done with Visual Paradigm):

Without reading a line of code, here you can get a grasp of the power of OTP. The slave routine is, in it’s normal state, just waiting for an event. So far, I’ve identified the following events:

• service down
• link down (erlang link down, would probably be due to the master node being down)
• shutdown request

In the first case, just as today, we first attempt to restart the service, and if this fails, we go for a takeover. If restart fails, we first to a failover (i.e., shutdown of the master node, this is the same algorithm executed on our current code branch).

In the second case, if the node is down, it’s just a takeover. If the node is up, we do a takeover with ARP spoofing, because we assume there’s something weird with the other node. This part of the algorithm can be improved (we could do another verification, go back to the waiting for event state for N times, etc., this is just a draft).

Still, one of the improvements over the current version is that I don’t have to handle any loops, OTP handles that for me, all I have to do is write the callback functions (waiting_for_event, restarting_service, etc). Less code = less chance of errors.

When?

I’m currently revewing the design of the new algorithm. I hope to have a draft in two weeks (this is a slow week for me, my daughter is starting preschool education next week and hence my house is kinda crazy).

I’m also overdue with regular Highbase releases so I’ll try to get up to date with both trees.

The current erlang tree is mysql-ha-erl (legacy name, before the trademark issues). I’ll surely be creating a new tree, highbase-erl, during the next few days. Official release news will be broadcasted here from the official site.

Related posts:

1. highbase beta-0.9.4 Heavy testing, headphones with no music, lots of Sanchez Gran...
2. I love playing Monopoly This is the latest speed test I run against Santiago...

Related posts brought to you by Yet Another Related Posts Plugin.

According to the site, it aims to counter XSS, SQL Injection, header injection, directory traversal, RFE/LFI, DoS and LDAP attacks, and unknown attack patterns,  through it’s Centrifuge component.

Installation is simple. Just download it, copy the lib directory to a directory in your project structure, or add it to your include_path.

I actually chose a mix of both ways, so it’s automatically included when I distribute my applications.

Here’s how to use it:


ini_set('include_path',ini_get('include_path').PATH_SEPARATOR.PRIVATE_ROOT.DIRECTORY_SEPARATOR.'phpids'.DIRECTORY_SEPARATOR.'lib');

Here we’re just modifying the include path in order to add phpids.
PRIVATE_ROOT is a constant that I’ve defined in my app, which defines the root of the application. This is not accessible from the web server (following the recommendation of the Dispatch method, from the PHP Security Consortium. However, this is a particular case, in most situations, I’d recommend using a framework that already takes a similar pattern into account, like CakePHP).
phpids is where I’ve copied this IDS. The downloaded package has a similar root directory name, like phpids-x.y.n, depending on the version number.


require_once 'IDS/Init.php';


Require the Init.php file

Then, whenever you need to access the request variables (in my case, this was just in one point in my application so I just had to modify one function), add something like this:


$request = array(
'REQUEST' => $_REQUEST,
'GET' => $_GET,
'POST' => $_POST,
'COOKIE' => $_COOKIE
);
$init = IDS_Init::init(PRIVATE_ROOT. DIRECTORY_SEPARATOR .'phpids'.DIRECTORY_SEPARATOR.'lib/IDS/Config/Config.ini');

$ids = new IDS_Monitor($request, $init);
$result = $ids->run();

if (!$result->isEmpty()) {
trigger_error($result);
}

If the $result object is not empty, the IDS detected an attack attempt and therefore you should stop processing the request.

And now the fun part.

In order to test this, I set up a very simple, very insecure test page.
Here’s the php code:

error_reporting(E_ALL);
ini_set('include_path',ini_get('include_path').':'.'../src/phpids/lib');
require 'IDS/Init.php';
function checkIds()
{
   $request = array(
     'REQUEST' => $_REQUEST,
     'GET' => $_GET,
     'POST' => $_POST,
     'COOKIE' => $_COOKIE
);

$init = IDS_Init::init('/home/fipar/workspace/at-intranet/src/phpids/lib/IDS/Config/Config.ini');
$ids = new IDS_Monitor($request,$init);
$result = $ids->run();
if (!$result->isEmpty()) {
     trigger_error($result);
}
}

if (isset($_REQUEST['sql'])) {
     checkIds();
     $sql = $_REQUEST['sql'];
     print 'Form says '.$sql.'
';
}

The html page has just an input type text with the name ‘sql’, and a submit button.

I tried the following inputs:

• Hello, which goes by ok
• ‘ and 1, which generates errors for the REQUEST and POST variables, stating
  

  Detects classic SQL injection probings 1/2 | Tags: sqli, id, lfi | ID: 42

• <a href=”http://www.google.com”>www.google.com</a>, which generates errors for the REQUEST and POST variables, stating
  

  • finds html breaking injections including whitespace attacks | Tags: xss, csrf | ID: 1
  • Detects JavaScript object properties and methods | Tags: xss, csrf, id, rfe | ID: 17
  • Detects basic SQL authentication bypass attempts 2/3 | Tags: sqli, id, lfi | ID: 45

No related posts.

Here’s the info.

I have enough room, I just live a few thousand miles too far, and have to deal with very picky Customs officials.

No related posts.

The problem is, sometimes, people who skip on frameworks, don’t know what they’re doing.
Or, as the Tao of Programming says:

There once was a master programmer who wrote unstructured programs. A novice programmer, seeking to imitate him, also began to write unstructured programs. When the novice asked the master to evaluate his progress, the master criticized him for writing unstructured programs, saying, “What is appropriate for the master is not appropriate for the novice. You must understand the Tao before transcending structure.”

Anyway, for novices and masters alike, here’s a great resource of common programming errors one should avoid while working on a project. Some are web oriented, but most are applicable in any environment.

No related posts.

Here’s the code:

#!/bin/bash 

timeout=$1
command=$2
shift 2

check_pid_name()
{
        command=$1
        childpid=$2
        [ -d /proc/$childpid ] || return 1
        [ $(grep -c $command /proc/$childpid/cmdline 2>/dev/null) -gt 0 ] && return 0 || return 1
}

[ -z "$command" ] && echo "Usage: safecmd   [args]">&2 && exit 1

safe_run()
{
        command=$1
        shift
        $command $*
        kill $$
}

safe_run $command $* &
childpid=$!
sleep $timeout 

check_pid_name $command $childpid && {
        kill $childpid
        sleep 0.1
        check_pid_name $command $childpid && kill -9 $childpid
        echo "$command $* timed out"
}

You can try it how like this, for example:

./safecmd 2 sleep 4

which will output

./safecmd.1: line 34: 11739 Terminated               safe_run $command $*
sleep 4 timed out

or ./safecmd 2 sleep 1

which will output

Terminated

Indicating that sleep 1 finished successfully

This is still missing something. You can’t test for the success of the command you pass safecmd, if it exits successfully you’ll see the same return code than if it would have exited with an error.

In order to improve this, as far as I can tell, you have to use two scripts, one just as the wrapper (to use one separate script instead of the function safe_run()). That’s the way this is handled in Highbase, and I’ll post a full example in my next post on this subject.

Related posts:

1. Running commands from the shell with a timeout Sometimes, in a shell script, you need to run a...
2. Generating random salts from bash From the ‘just because it can be done’ column, here...

Related posts brought to you by Yet Another Related Posts Plugin.

In order to avoid this situations, here’s a small shell script that will execute any command with a guaranteed timeout:

#!/bin/bash

timeout=$1
command=$2

check_pid_name()
{
        command=$1
        childpid=$2
        [ -d /proc/$childpid ] || return 1
        [ $(grep -c $command /proc/$childpid/cmdline 2>/dev/null) -gt 0 ] && return 0 || return 1
}

[ -z "$command" ] && echo "I need a command to run">&2 && exit 1

[ $# -eq 2 ] && shift || shift 2

$command $* &
childpid=$!

sleep $timeout 

check_pid_name $command $childpid && {
        kill $childpid
        sleep 0.1
        check_pid_name $command $childpid && kill -9 $childpid
        echo "$command $* timed out"
}

As you can see, it’s quite simple. You just have to start the desired command in the background, sleep for the timeout, and then kill the process if it’s still running.

Yes, this script has a defect, all the commands will wait $timeout to run, but that’s easy to solve. I’ll post an improved (but slightly more complex) version in my next post.

Related posts:

1. Running commands from the shell with a timeout (pt 2) Here’s an improved version of the safecmd script. This one...
2. Generating random salts from bash From the ‘just because it can be done’ column, here...

Related posts brought to you by Yet Another Related Posts Plugin.