Here’s the code:

#!/bin/bash

usage()
{
cat <&2

usage : multi-fifo target0 [target1 [target2 [...]]]

Where each targetN is a program you want to send the input multi-fifo receives

EOF

}

[ $# -eq 0 ] && usage && exit 1

i=0
pipeline="tee /dev/null"
while [ -n "$1" ]; do
target=$1
fifo=/tmp/multi-fifo-$$.$i
mkfifo $fifo
eval "$target < $fifo" &
pipeline="$pipeline | tee $fifo"
i=$((i+1))
shift
done

eval "cat | $pipeline"

j=0
while [ $j -lt $i ]; do
rm -f /tmp/multi-fifo-$$.$j
j=$((j+1))
done

Usage is pretty straightforward:


tar cjvf - . | ./multi-fifo "nc host1 9999" "nc host2 9999" >/dev/null


This way, the script will create a fifo for each netcat client, then send it’s stdin to a pipeline that writes to both fifos using tee. Useful for example if you want to stream a backup to multiple destination servers for cloning.

Related posts:

1. Using the ENUM data type to increase performance While going through the DATA TYPES section of the Certification...
2. Generating data with dbmonster In my last post I included some sample data...
3. Running commands from the shell with a timeout (pt 2) Here’s an improved version of the safecmd script. This one...

Related posts brought to you by Yet Another Related Posts Plugin.

If you’re just querying for a word, it’s usage is pretty straightforward, and in fact, you can use the SOUNDS LIKE operator to build expressions such as this:

SELECT expr FROM table expr WHERE field0 SOUNDS LIKE ‘inputWord’

However, if you’re storing multiple-word strings, things get a little more complicated, since they can’t be compared by their soundex. Rather, the soundex returned will be associated with the whole phrase. If at a later time, you want to search for a subpart of this phrase, there’s no way for you to do this.

Well, at least not directly, but by using an auxiliary table to store soundex strings, a couple of stored procedures, and a trigger, it can be done with little effort to programmers that use the database.

Let’s assume we have a very simple table called soundex_text with the following structure:

create table if not exists soundex_text (
	id int unsigned not null auto_increment primary key,
	description text
) Engine = Innodb;

The field we want to query by proximity is description, therefore we create the following auxiliary table to store soundex values:

create table if not exists soundex_text_index (
	soundex_text_id int unsigned not null references soundex_text(id),
	soundex char(4)
) Engine = Innodb;

We now create a stored procedure and a trigger to populate the auxiliary table automatically every time a row is inserted in the main table.

-- adapted from example at http://forums.mysql.com/read.php?60,78776,242420#msg-242420
-- posted by jim smith: http://forums.mysql.com/profile.php?60,3154903
CREATE PROCEDURE update_soundex_text_index (sStringIn text,splitChar varchar(1), soundex_text_id int)
BEGIN
DECLARE comma INT DEFAULT 0;
DECLARE mylist TEXT DEFAULT sStringIn;
DECLARE temp TEXT DEFAULT '';
DECLARE strlen int DEFAULT LENGTH(sStringIn);
DECLARE insert_id int DEFAULT soundex_text_id;

/* find the first instance of the spliting character */
SET comma = LOCATE(splitChar,mylist);
/* Insert each split variable into the temp table */
WHILE strlen > 0 DO
	IF comma = 0 THEN
		SET temp = TRIM(mylist);
		SET mylist = '';
		SET strlen = 0;
	END IF;
	IF comma != 0 THEN
		SET temp = TRIM(SUBSTRING(mylist,1,comma-1));
		SET mylist = TRIM(SUBSTRING(mylist FROM comma+1));
		SET strlen = LENGTH(mylist);
		-- Sample handling of special chars you might want removed from individual words
		-- before storing their soundex.
		SELECT REPLACE(temp,',','') INTO temp;
		SELECT REPLACE(temp,';','') INTO temp;
		SELECT REPLACE(temp,':','') INTO temp;
	END IF;
	IF temp != '' THEN
		insert into soundex_text_index (soundex_text_id,soundex) values (insert_id,substring(soundex(temp) from 1 for 4));
	END IF;
	SET comma = LOCATE(splitChar,mylist);
END WHILE;

END//

drop trigger if exists soundex_text_bi//
create trigger soundex_text_bi
before insert
on soundex_text
for each row
begin
	SET @id = last_insert_id();
	call update_soundex_text_index (NEW.description, ' ', @id);
end;//
delimiter ;

Finally, a stored procedure to automatically query the table using the auxiliary table implicitly:

delimiter //
create procedure query_soundex_text(sStringIn text, splitChar varchar(1))
BEGIN
DECLARE comma INT DEFAULT 0;
DECLARE mylist TEXT DEFAULT sStringIn;
DECLARE temp TEXT DEFAULT '';
DECLARE strlen int DEFAULT LENGTH(sStringIn); 

create temporary table results (id int unsigned, description text);

/* find the first instance of the spliting character */
SET comma = LOCATE(splitChar,mylist);
/* Insert each split variable into the temp table */
WHILE strlen > 0 DO
	IF comma = 0 THEN
		SET temp = TRIM(mylist);
		SET mylist = '';
		SET strlen = 0;
	END IF;
	IF comma != 0 THEN
		SET temp = TRIM(SUBSTRING(mylist,1,comma-1));
		SET mylist = TRIM(SUBSTRING(mylist FROM comma+1));
		SET strlen = LENGTH(mylist);
		-- Sample handling of special chars you might want removed from individual words
		-- before storing their soundex.
		SELECT REPLACE(temp,',','') INTO temp;
		SELECT REPLACE(temp,';','') INTO temp;
		SELECT REPLACE(temp,':','') INTO temp;
	END IF;
	IF temp != '' THEN
		insert into results select st.id, st.description from soundex_text st, soundex_text_index sti where sti.soundex = substring(soundex(temp) from 1 for 4);
	END IF;
	SET comma = LOCATE(splitChar,mylist);
END WHILE;

select distinct * from results;
drop table results;

END//
delimiter ;

Using this is pretty straightforward.
Just insert some sample data into the table, and give it a shot!:

mysql> insert into soundex_text(description) values ('This is a sample text row');
Query OK, 1 row affected (0.00 sec)

mysql> call query_soundex_text('semple',' ');
+------+---------------------------+
| id   | description               |
+------+---------------------------+
|    3 | This is a sample text row |
+------+---------------------------+
1 row in set (0.00 sec)

Query OK, 0 rows affected (0.00 sec)

mysql> call query_soundex_text('THis is a samPLE taxt row',' ');
+------+---------------------------+
| id   | description               |
+------+---------------------------+
|    3 | This is a sample text row |
+------+---------------------------+
1 row in set (0.00 sec)

Query OK, 0 rows affected (0.00 sec)

mysql> call query_soundex_text('rock',' ');
Empty set (0.01 sec)

Query OK, 0 rows affected (0.01 sec)

To make it even easier, you could modify the querying stored procedure and always assume that the splitting character is a whitespace.

You can download a zipfile with a sql script with all this code for you to load into a MySQL database right here.

Enjoy!

Related posts:

1. Generating data with dbmonster In my last post I included some sample data...
2. Extending procedure_analyse My previous post explored a stored procedure that extended procedure_analyse...

Related posts brought to you by Yet Another Related Posts Plugin.

Here’s an improved version. I’ve followed Arjen Lentz‘s suggestion and added support for the max_elements and max_memory parameters.

I also added a new Indexed column to the output, which is an ENUM(‘No’,'Yes’,'Overindexed’). Yes and No are self-explanatory, while Overindexed means the column is present as the left-most part of more than one index. This is useless, it just presents a performance penalty for MySQL (it needs to update more indexes) and if, for instance, you have columns A and B, and you have KEY(A) and KEY (A,B), mysql can use the second index to search for A alone too.

Here’s the updated version:

/*

extended procedure analyse
(C) 2009 Fernando Ipar
mail(at)fernandoipar.com

GPLv2

*/
drop procedure if exists extended_procedure_analyse;
delimiter //
create procedure extended_procedure_analyse(databaseName varchar(64), tableName varchar(64), max_elements int, max_memory int)
begin

	drop temporary table if exists procedure_analyse_output;
	drop temporary table if exists tmp_pao;

	create temporary table procedure_analyse_output
	(
	Field_name varchar(64),
	Min_value int,
	Max_value int,
	Min_length int,
	Max_length int,
	Empties_or_zeros int,
	Nulls int,
	Avg_value_or_avg_length float,
	Std float,
	Optimal_fieldtype text,
	Actual_fieldtype text,
	Indexed enum ('No','Yes','Overindexed') default 'No'
	);	

	set @table = concat(databaseName,'.',tableName);
	set @dbName = databaseName;
	set @tbName = tableName;
	set @maxEle = max_elements;
	set @maxMem = max_memory;

	set @qry = concat('insert into procedure_analyse_output (Field_name,Min_value,Max_value,Min_length,Max_length,Empties_or_zeros,Nulls,Avg_Value_or_avg_length,Std,Optimal_fieldtype) select * from ', @table,' procedure analyse(',@maxEle,',',@maxMem,')');
	prepare myStmt from @qry;
	execute myStmt;

	update procedure_analyse_output set Field_name = replace(Field_name,  CONCAT(databaseName,'.',tableName,'.'),'');

	prepare myStmt from 'update procedure_analyse_output pao, information_schema.columns c set pao.Actual_Fieldtype = c.column_type where table_schema = ? and table_name = ? and column_name = pao.Field_name';
	execute myStmt using @dbName,@tbName;

	set @qry = concat('select count(*) as `Total_number_of_rows` from ',@table);
	prepare myStmt from @qry;

	execute myStmt;

	set @qry = concat('update procedure_analyse_output pao set Indexed = "Yes\" where exists (select 1 from information_schema.statistics where table_schema = ? and table_name = ? and column_name = pao.Field_name)');
	prepare myStmt from @qry;
	execute myStmt using @dbName,@tbName;

	create temporary table tmp_pao as select * from procedure_analyse_output;
	prepare myStmt from 'update tmp_pao set Indexed = "Overindexed" where exists (select Field_name,count(*) from procedure_analyse_output pao inner join information_schema.statistics s on pao.Field_name = s.column_name where table_schema = ? and table_name = ? and seq_in_index = 1 and pao.Field_name = tmp_pao.Field_name group by Field_name having count(*) > 1)';
	execute myStmt  using @dbName,@tbName;

	select * from tmp_pao;

	drop temporary table procedure_analyse_output;
	drop temporary table tmp_pao; 

end;
//
delimiter ;

And here’s a sample output:

mysql> call extended_procedure_analyse('test','Account',4,100)\G
*************************** 1. row ***************************
Total_number_of_rows: 1
1 row in set (0.04 sec)

*************************** 1. row ***************************
             Field_name: InternalAID
              Min_value: 2147483647
              Max_value: 2147483647
             Min_length: 18
             Max_length: 18
       Empties_or_zeros: 0
                  Nulls: 0
Avg_value_or_avg_length: 1.65632e+17
                    Std: 0
      Optimal_fieldtype: BIGINT(18) UNSIGNED NOT NULL
       Actual_fieldtype: bigint(20)
                Indexed: Yes
*************************** 2. row ***************************
             Field_name: accountID
              Min_value: 12
              Max_value: 12
             Min_length: 2
             Max_length: 2
       Empties_or_zeros: 0
                  Nulls: 0
Avg_value_or_avg_length: 2
                    Std: NULL
      Optimal_fieldtype: ENUM('12') NOT NULL
       Actual_fieldtype: varchar(255)
                Indexed: Overindexed
*************************** 3. row ***************************
             Field_name: acctBalance
              Min_value: 2147483647
              Max_value: 2147483647
             Min_length: 11
             Max_length: 11
       Empties_or_zeros: 0
                  Nulls: 0
Avg_value_or_avg_length: 1.00051e+20
                    Std: 0
      Optimal_fieldtype: BIGINT(11) UNSIGNED NOT NULL
       Actual_fieldtype: double
                Indexed: Yes
*************************** 4. row ***************************
             Field_name: ownerID
              Min_value: 2147483647
              Max_value: 2147483647
             Min_length: 19
             Max_length: 19
       Empties_or_zeros: 0
                  Nulls: 0
Avg_value_or_avg_length: 6.57541e+18
                    Std: 0
      Optimal_fieldtype: BIGINT(19) UNSIGNED NOT NULL
       Actual_fieldtype: bigint(20)
                Indexed: No
4 rows in set (0.24 sec)

Query OK, 0 rows affected, 8 warnings (0.24 sec)

There’s a hg repository to handle the project. We’re working with @fedesilva to create a standalone java app that will present this and more info in a friendly manner to sysadmins, and to handle the creation and destruction of the stored procedure automatically.

Related posts:

1. Making use of procedure analyse() SELECT Field0[,Field1,Field2,...] FROM TABLE PROCEDURE ANALYSE() is a nice tool...

Related posts brought to you by Yet Another Related Posts Plugin.

Why is it that writers, musicians and, well, any respected and professional artist or scientist gets his/her work reviewed before it’s published, but the same isn’t true in IT?

My guess? Most programmers aren’t neither artists, nor scientists. They’re an anonymous resource in a badly oiled machine that’s always behind schedule.

Well, so long, Knuth!

Related posts:

1. Symbolics Lisp Machine Wanna have your own Symbolics Lisp Machine? Here’s the info....

Related posts brought to you by Yet Another Related Posts Plugin.

So, without further ado,  here it goes:

#!/bin/bash 

[ $# -eq 0 ] && {
        echo "usage: salt <length>">&2
        exit
}
strings </dev/urandom | while read line; do
        echo $line | tr '\n\t ' $RANDOM:0:1 >> /tmp/.salt.$$
        salt=$(cat /tmp/.salt.$$)
        [ ${#salt} -ge $1 ] && salt=${salt:0:$1} && echo $salt && break
done
rm -f /tmp/.salt.$$

I had to use $1 and not a var, and echo the salt right from inside the while, because the ‘|’ creates another shell, so I can’t pass variables to or from the while in this case.

If you want to keep things simple, you can go perl and just do

#!/usr/bin/perl
use Crypt::Salt;
my $length = shift;
print salt($length);

But if you ever find yourself in a server with no cpan, the first option might prove useful

Related posts:

1. Generating data with dbmonster In my last post I included some sample data...
2. Running commands from the shell with a timeout (pt 2) Here’s an improved version of the safecmd script. This one...
3. Running commands from the shell with a timeout Sometimes, in a shell script, you need to run a...

Related posts brought to you by Yet Another Related Posts Plugin.

This results are from a 2048 down /256 up Kb/s connection for which I pay U$S 52/monthly.
Yeah, I feel your envy already.

The Administración Nacional de Telecomunicaciones of Uruguay runs my ISP. I could choose another one, but the link to the outside world would eventually be handled by the state.

I called tech support twice this week, and it seems this results are due to a problem with the international (I get real fast to anything in Uruguay or Argentina, I guess it’s my fault for getting customers outside of that comfort zone!) link. They don’t have an estimated date of repair, and they don’t need to, since, what else I’m I going to do? Probably take a plane and sit on a console in front of the damn database.

Uruguay: redefining the term ‘Sneaker Net’

No related posts.

However, I didn’t want to let it pass. I checked up on erlybird a couple of times before, but it was only available as a source project, and you needed a particular netbeans version in order to build the module.

I’m as geek as it gets, but I’m also 30, married, and a parent. I would have done that 12 years ago, now ./configure && make && make install is as far as I take it unless I’m really into it.

So 0.17.0 (released last November) fixes all this, it’s a netbeans module, which you can install the usual way. It requires NetBeans 6.5, but that’s not an issue for me since I’ve been using that version for quite a while already.

Here’s a link with more info.

Kudos to both admins for the great job they’ve done.

No related posts.

According to the site, it aims to counter XSS, SQL Injection, header injection, directory traversal, RFE/LFI, DoS and LDAP attacks, and unknown attack patterns,  through it’s Centrifuge component.

Installation is simple. Just download it, copy the lib directory to a directory in your project structure, or add it to your include_path.

I actually chose a mix of both ways, so it’s automatically included when I distribute my applications.

Here’s how to use it:


ini_set('include_path',ini_get('include_path').PATH_SEPARATOR.PRIVATE_ROOT.DIRECTORY_SEPARATOR.'phpids'.DIRECTORY_SEPARATOR.'lib');

Here we’re just modifying the include path in order to add phpids.
PRIVATE_ROOT is a constant that I’ve defined in my app, which defines the root of the application. This is not accessible from the web server (following the recommendation of the Dispatch method, from the PHP Security Consortium. However, this is a particular case, in most situations, I’d recommend using a framework that already takes a similar pattern into account, like CakePHP).
phpids is where I’ve copied this IDS. The downloaded package has a similar root directory name, like phpids-x.y.n, depending on the version number.


require_once 'IDS/Init.php';


Require the Init.php file

Then, whenever you need to access the request variables (in my case, this was just in one point in my application so I just had to modify one function), add something like this:


$request = array(
'REQUEST' => $_REQUEST,
'GET' => $_GET,
'POST' => $_POST,
'COOKIE' => $_COOKIE
);
$init = IDS_Init::init(PRIVATE_ROOT. DIRECTORY_SEPARATOR .'phpids'.DIRECTORY_SEPARATOR.'lib/IDS/Config/Config.ini');

$ids = new IDS_Monitor($request, $init);
$result = $ids->run();

if (!$result->isEmpty()) {
trigger_error($result);
}

If the $result object is not empty, the IDS detected an attack attempt and therefore you should stop processing the request.

And now the fun part.

In order to test this, I set up a very simple, very insecure test page.
Here’s the php code:

error_reporting(E_ALL);
ini_set('include_path',ini_get('include_path').':'.'../src/phpids/lib');
require 'IDS/Init.php';
function checkIds()
{
   $request = array(
     'REQUEST' => $_REQUEST,
     'GET' => $_GET,
     'POST' => $_POST,
     'COOKIE' => $_COOKIE
);

$init = IDS_Init::init('/home/fipar/workspace/at-intranet/src/phpids/lib/IDS/Config/Config.ini');
$ids = new IDS_Monitor($request,$init);
$result = $ids->run();
if (!$result->isEmpty()) {
     trigger_error($result);
}
}

if (isset($_REQUEST['sql'])) {
     checkIds();
     $sql = $_REQUEST['sql'];
     print 'Form says '.$sql.'
';
}

The html page has just an input type text with the name ‘sql’, and a submit button.

I tried the following inputs:

• Hello, which goes by ok
• ‘ and 1, which generates errors for the REQUEST and POST variables, stating
  

  Detects classic SQL injection probings 1/2 | Tags: sqli, id, lfi | ID: 42

• <a href=”http://www.google.com”>www.google.com</a>, which generates errors for the REQUEST and POST variables, stating
  

  • finds html breaking injections including whitespace attacks | Tags: xss, csrf | ID: 1
  • Detects JavaScript object properties and methods | Tags: xss, csrf, id, rfe | ID: 17
  • Detects basic SQL authentication bypass attempts 2/3 | Tags: sqli, id, lfi | ID: 45

No related posts.

Here’s the info.

I have enough room, I just live a few thousand miles too far, and have to deal with very picky Customs officials.

No related posts.

MySQL Proxy sits between clients and servers and offers many possibilities, load balancing being the one that is of particular interest to me (and some Highbase users who requested this feature).

Here‘s the official announcement.

Related posts:

1. Updated mysql-proxy benchmarking script (for proxy 0.7) My previous post contained a lua script for MySQL proxy...
2. Using MySQL Proxy to benchmark query performance By transparently sitting between client and server on each request,...
3. MySQL Certification self study I’m taking the MySQL Certification exams soon, and while I’d...

Related posts brought to you by Yet Another Related Posts Plugin.