You might be wondering  why I am saying all these here. Well, I suspected that the problem was actually because of my SQL query for delta load.  But to be sure, I wanted to see the query being executed by Solr DataImportHandler. As always I turned to Google for assistance, and I finally reached the MySQL documentation on the General Query Log. Voila! This was exactly what I wanted. All I had to do was use the –log=[filename] parameter and all my queries will be logged to the specified log file. Nice, isn’t it?

Now I have to stop my running MySQL server and restart it with the –log switch, in addition to the other regular options. But there was a problem, I was not sure of the other required parameters. You can use the ps utility, when the MySQL server is running, to find out the normal parameters.

ps -ax | grep mysql

For me the output was

/usr/local/mysql/bin/mysqld –basedir=/usr/local/mysql –datadir=/usr/local/mysql/data –user=mysql –pid-file=/usr/local/mysql/data/localhost.pid –port=3306 –socket=/tmp/mysql.sock

Now shutdown the MySQL server.

// On Mac
/Library/StartupItems/MySQLCOM/MySQLCOM stop
// For other Linux/Unix variants try
/etc/init.d/mysqld restart
service mysql restart

Start mysqld with –log option

/usr/local/mysql/bin/mysqld --basedir=/usr/local/mysql --datadir=/usr/local/mysql/data --user=mysql --pid-file=/usr/local/mysql/data/localhost.pid --port=3306 --socket=/tmp/mysql.sock --log=/tmp/query.log

The general query log contains lots of irrelevant information. To view the log after filtering out the unwanted details use tail and grep as given below

tail -f /tmp/query.log | grep -v Connect | grep -v Quit

The amount of information added to the file is quite large. If you are using this on a production server, I recommend turning off the logging once you are done.

awk '{if (NR == 1)print >> "header.txt"; else print >> "body.txt";}' input.txt
Where NR is a built-in variable that contains the number of the current record / line,
input.txt is the input file,
header.txt is output file for header,
body.txt is the output file for the remaining content

Awk reads the input file (input.txt) line by line and checks whether the current line is the first line. If so the content is appended ( >> ) to the file header.txt. Else the content is appended to body.txt.

wget http://www.joycebabu.com/my-photo.jpg
Where http://www.joycebabu.com/my-photo.jpg is the file to be downloaded.

By default wget saves the file with the same name as the fetched file. For example, the above command will save the file as my-photo.jpg in the current working directory. If a file already exists with the given name, the downloaded file will be named as my-photo.jpg.1, my-photo.jpg.2 etc until a non existent filename is found.

It is possible to explicitly specify a different name for the downloaded file. This is possible using the -O switch (--output-document also works, but I believe short is sweet). The new command is

wget -O photo.jpg http://www.joycebabu.com/my-photo.jpg
Where photo.jpg is the new filename.

But be careful while using the -O switch. If there is an existing file with the same name, it will be overwritten with the new file.

I have been using the Secure Copy (scp) utility for copying files between my local server and development server. Sometimes I have to copy more than one file. Previously I used to copy the files one at a time. This is very annoying, as you have to type the password every time you use the command . But it is possible to copy multiple files using scp, just like the copy (cp) utility.

When you have to copy multiple files to your remote server, the syntax is similar to the cp command.

scp file1.sql file2.sh joyce@joycebabu.com:~/upload

Where file1.sql and file2.sh are the files to be copied, joyce is the username, joycebabu.com is the hostname and ~/upload is the destination directory on the remote server.

In order to download multiple files from the remote server, the command to be used is

scp joyce@joycebabu.com:"file1.log file2.log" ~/logs

Where file1.log and file2.log are the files to be downloaded and ~/logs is the destination directory on the local server. Notice the quotes around the filenames. This ensures that the filenames list is not parsed by the local shell and is passed to the remote shell. Similarly, when you want to download files using wildcards (*.php, files_?.log etc), you should enclose the name within quotes to ensure that the expansion is done by the remote server.

The -r option can be used to copy directories recursively.

scp -r joyce@joycebabu.com:~/logs ~/logs

This may not be a lifesaver tip and the time gained by this method may be small. After all, when a large number of files are to be transferred, I use FTP or tar my files and copy it. But at times when things go wrong, even this small gain can help.

As usual, I approached Google for a solution and it introduced me to two different solutions – AWK (a programming language) and Sed (a unix utility). There is only a very slight difference between the two commands.

awk ‘/– Table structure for table .tbl_first./,/– Table structure for table .tbl_second./{print}’ mydumpfile.sql > mytable.sql

sed -ne ‘/– Table structure for table .tbl_first./,/– Table structure for table .tbl_second./p’ mydumpfile.sql > mytable.sql

Here tbl_first is the table I wanted to extract and tbl_second was the table below that. The above commands will search the file mydumpfile.sql and extract the text between the start string (– Table structure for table .tbl_first.) and end string (– Table structure for table .tbl_second.). The dots before and after the table name are wildcard character to match the engrave character, which has a special meaning in shell commands. The {print} option (p in sed) prints the extracted string, which is then redirected to the file mytable.sql.

But that didn’t solve my problem completely. I was not sure of the order of tables in the .sql file. This time  grep (another powerful unix utility) came to my rescue. The following command lists all the tables in the file mydumpfile.sql in the same order in which they appear in the file.

grep ‘Table structure’ mydumpfile.sql | cut -d’`’ -f2

I don’t know a lot about shell commands. But with my very limited experience I can say that they are extremely powerful. Two small lines of code saved me a lot of time.