A subset of Python (30 Jun 2013)

Over the years, I collected implementations of an algorithm to solve the subset sum problem in various languages. A few days ago, I experimented a little with Python, and here is the deplorably non-idiomatic and naïve result - apologies offered.


import random
import array
import sys

numbers = array.array('i')
flags = array.array('c')
solutions = 0

def find_solutions(k, target_sum):
    global solutions

    if target_sum == 0:
        print "  Solution:",
        for i in range(0, len(numbers)):
            if flags[i] != 0:
                print numbers[i],
        print
        solutions = solutions + 1
    else:
        if k < len(numbers):
            if (numbers[k] * (len(numbers)-k+1)) >= target_sum:
                if target_sum >= numbers[k]:
                    flags[k] = 1
                    find_solutions(k+1, target_sum - numbers[k])
                    flags[k] = 0
                find_solutions(k+1, target_sum)

def find_subset_sum(target_sum):
    global solutions
    global flags
    print "Subsets which sum up to %s:" % target_sum
    flags = [0] * len(numbers)
    find_solutions(0, target_sum)
    print "Found", solutions, "different solutions"

def subset_sum_test(size):
    global numbers
    total = 0

    print "Random values:\n  ",
    for i in range(0, size):
        numbers.append(random.randint(0, 1000))
        total = total + numbers[i]
        print numbers[i],
    print

    numbers = sorted(numbers, reverse = True)
    target_sum = total/2
    find_subset_sum(target_sum)

subset_sum_test(15 if len(sys.argv) < 2 else int(sys.argv[1]))


JavaScript: The Good Parts (04 May 2013)

js_singleton.png

Finally, after all those years, I got around to reading "JavaScript: The Good Parts" - pretty much from cover to cover.

All in all, a great book! However, it left me even more confused about how JavaScript sets "this" than I was before. Fortunately, I found the discussion at http://stackoverflow.com/questions/3127429/javascript-this-keyword and the article at http://www.digital-web.com/articles/scope_in_javascript/ which both clarified open questions for me. But I am pretty sure I will forget JavaScript's context subtleties fairly quickly, and then I will have to look them up again. Oh well.

Crockford's approach is to focus on a subset of JavaScript which he considers sane, Of course there is much debate on what belongs into the subset and what doesn't. For example, Crockford bans the ++ and -- operators because they remind him too much of obfuscated C++ code. I did not find this particular advice too convincing, and it seems I am not alone.

To experiment with code from the book and test some code of my own, I used the jsc command-line JS interpreter on the Mac, and also toyed with http://jsfiddle.net/. (On Ubuntu, I guess I could have used rhino.)


Scripting VPN connections (20 Aug 2011)

Like many other companies, my company provides VPN access to its employees so that we can stay connected from our home offices or on the road. Most of the time, I connect to the company network through a web portal which downloads, installs and runs Juniper's "Network Connect" software on the Windows client system. That's all fine and dandy, except that I am a command-line guy and find it way too clumsy to fire up a web browser just in order to "dial in".

Fortunately, Juniper's Network Connect client has a command-line interface, and so here is a trivial DOS batch script which can be used to establish a connection in "I-don't-need-no-stinkin'-buttons" mode.

The script assumes that the Network Connect client has been installed and run in the usual manner (i.e. from the web portal) at least once. It will attempt to auto-detect the VPN host and user name, so in most cases all you have to specify is password information. Oh, and the script assumes you want to connect to the "SecurID(Network Connect)" realm by default, which requires entering a PIN and a number displayed on your RSA SecurID token.

@echo off
REM Launch Juniper Network Connect client from the command line
REM Written by Claus Brod in 2011, see
REM http://www.clausbrod.de/Blog/DefinePrivatePublic20110820JuniperNetworkConnect

REM --------------------------------------------------------
setlocal enableextensions

call :find_juniper_client NCCLIENTDIR
if "x%NCCLIENTDIR%"=="x" (
  echo ERROR: Cannot find Network Connect client.
  goto :end
)

rem CONFIGURE: Set your preferred VPN host here.
set url=define-your-vpn-host-here
ping -n 1 %url% >nul
if not errorlevel 1 goto :validhost

rem Try to auto-detect the VPN host from the config file
set NCCLIENTCONFIG="%NCCLIENTDIR%\..\Common Files\config.ini"
if exist %NCCLIENTCONFIG% for /f "delims=[]" %%A in ('findstr [[a-z0-9]\. %NCCLIENTCONFIG% ^| findstr /V "Network Connect"') do set url=%%A
ping -n 1 %url% >nul
if errorlevel 1 (
  echo ERROR: Host %url% does not ping. Please check your configuration.
  goto :end
)

:validhost
call :read_no_history url %url% "VPN host"

set user=guest
call :read_no_history user %user% "Username"

rem CONFIGURE: Set your preferred realm here. By default, the script
rem assumes two-stage authentication using a PIN and RSA SecurID.

set realm="SecurID(Network Connect)"
call :read_no_history realm %realm% "Realm"

REM TODO: Hide password input
set password=""
call :read_no_history password %password% "Enter PIN + token value for user %user%:"
if x%password%==x (
  echo ERROR: No password specified
  goto :end
)

cls

echo Launching Juniper Network Connect client in
echo   %NCCLIENTDIR%...
"%NCCLIENTDIR%\nclauncher.exe" -url %url% -u %user% -p %password% -r %realm%
goto :end

REM --------------------------------------------------------
:find_juniper_client
setlocal
set CLIENT=

rem search registry first
for /f "tokens=1* delims=       " %%A in ('reg query "HKLM\SOFTWARE\Juniper Networks" 2^>nul') do set LATESTVERSION="%%A"
if x%LATESTVERSION%==x"" goto :eof
for /f "tokens=2* delims=        " %%A in ('reg query %LATESTVERSION% /v InstallPath 2^>nul ^| findstr InstallPath') do set CLIENT=%%B

rem if nothing found, check filesystem
if "x%CLIENT%"=="x" for /d %%A in ("%ProgramFiles(x86)%\Juniper Networks\Network Connect*") do set CLIENT=%%A
if "x%CLIENT%"=="x" for /d %%A in ("%ProgramFiles%\Juniper Networks\Network Connect*") do set CLIENT=%%A

endlocal & set "%~1=%CLIENT%"
goto :eof


REM --------------------------------------------------------
REM read_no_history promptvar default promptmessage
:read_no_history
setlocal
set msg=%~3
if not "x%~2"=="x" (
  set msg="%~3 (default: %~2): "
)
set /P RNH_TEMP=%msg% <nul
set RNH_TEMP=

REM call external script to avoid adding to our own command history
set RNH_CMDFILE=%TEMP%\temp$$$.cmd
  (
    echo @echo off
    echo set var_=%2
    echo set /p var_=
    echo echo %%var_%%
  )> "%RNH_CMDFILE%"

for /f "delims=," %%A in ('%RNH_CMDFILE%') do set RNH_TEMP=%%A
del %RNH_CMDFILE%
endlocal & if not x%RNH_TEMP%==x set "%~1=%RNH_TEMP%"
goto :eof


REM --------------------------------------------------------
:end
endlocal

The above script is meant to be used along with the Windows version of the Network Connect client. For the Linux client, Paul D. Smith provides an excellent script and great instructions at http://mad-scientist.us/juniper.html.

See below for the direct download link.

PS: The code is now available from github as well, see https://github.com/clausb/nclauncher.


head replacement on Windows (16 Jun 2009)

A co-worker needed to convert a Cygwin-dependent script to something that runs on a bare-bones Windows system. The interesting part of the task was finding a replacement for the good ol' head command-line utility.

Fortunately, this is fairly simple using a few lines of VBScript and the Windows Scripting Host. First, here's the VBScript code:

lines = WScript.Arguments(0)
Do Until WScript.stdin.AtEndOfStream Or lines=0
  WScript.Echo WScript.stdin.ReadLine
  lines = lines-1
Loop

This is an extremely stripped-down version of head's original functionality, of course. For example, the code above can only read from standard input, and things like command-line argument validation and error handling are left as an exercise for the reader big grin

Assuming you'd save the above into a file called head.vbs, this is how you can display the first three lines of a text file called someinputfile.txt:

   type someinputfile.txt | cscript /nologo head.vbs 3

Enjoy!


Speeding through the crisis (22 Apr 2009)

That darn ol' MP3 player. Five years old, but still looks cute. Stubbornly refuses to break, too, so no excuse to go out and buy a new one. Which, of course, I wouldn't do anyway these days. You know, the crisis and all - who has the guts to make investments like this now. I mean, a new player could easily cost me as much as 30 euros! wink

So I'm sticking to the old hardware, and it works great, except for one thing: It cannot set bookmarks. Sure, it remembers which file I was playing most recently, but it doesn't know where I was within that file. Without bookmarks, resuming to listen to that podcast of 40 minutes length which I started into the other day is an awkward, painstakingly slow and daunting task.

But then, those years at university studying computer science needed to finally amortize themselves anyway, and so I set out to look for a software solution!

The idea was to preprocess podcasts as follows:

  • Split podcasts into five-minute chunks. This way, I can easily resume from where I left off without a lot of hassle.
  • While I'm at it, speed up the podcast by 15%. Most podcasts have more than enough verbal fluff and uhms and pauses in them, so listening to them in their original speed is, in fact, a waste of time. Of course, I don't want all my podcasts to sound like Mickey Mouse cartoons, of course, so I need to preserve the original pitch.
  • Most of the time, I listen to technical podcasts over el-cheapo headphones in noisy environments like commuter trains, so I don't need no steenkin' 320kbps bitrates, thank you very much.
  • And the whole thing needs to run from the command line so that I can process podcasts in batches.

I found it surprisingly difficult to find the single right tool for the purpose, so after experimenting for a while, I wrote the following bash script which does the job.

#! /bin/bash
#
# Hacked by Claus Brod, 
#   http://www.clausbrod.de/Blog/DefinePrivatePublic20090422SpeedingThroughTheCrisis
#
# prepare podcast for mp3 player:
#  - speed up by 15%
#  - split into small chunks of 5 minutes each
#  - recode in low bitrate
#
# requires:
#  - lame
#  - soundstretch
#  - mp3splt

if [ $# -ne 1 ]
then
  echo Usage: $0 mp3file >&2
  exit 2
fi

bn=`basename "$1"`
bn="${bn%.*}"

lame --priority 0 -S --decode "$1" - | \ 
  soundstretch stdin stdout -tempo=15 | \ 
  lame --priority 0 -S --vbr-new -V 9 - temp.mp3

mp3splt -q -f -t 05.00 -o "${bn}_@n" temp.mp3
rm temp.mp3

The script uses lame, soundstretch and mp3splt for the job, so you'll have to download and install those packages first. On Windows, lame.exe, soundstretch.exe and mp3splt.exe also need to be accessible through PATH.

The script is, of course, absurdly lame with all its hardcoded filenames and parameters and all, and it works for MP3 files only - but it does the job for me, and hopefully it's useful to someone out there as well. Enjoy!


"Don't quote me on this", I said (03 Apr 2007)

A good while ago, I discussed how the idiosyncratic command-line parsing rules in cmd.exe can hurt code which uses C runtime APIs such as system, and how to soothe the pain using mildly counterintuitive, yet simple quoting rules:

  • Quote the command
  • Quote all arguments
  • If both the command and at least one of the arguments contain blank characters, quote the whole command line as well.

This works for almost all situations except if you use start:

  C:\>start /b "c:\Program Files\Microsoft Visual Studio 8\Common7\Tools\Bin\uuidgen" 
            -ofoobar.txt

Try the above in a command prompt (everything in one line!): "The system cannot find the file -ofoobar.txt".

When asking for syntax help (start /?), we get:

I:\>start /?
Starts a separate window to run a specified program or command.

START ["title"] [/D path] [/I] [/MIN] [/MAX] [/SEPARATE | /SHARED]
      [/LOW | /NORMAL | /HIGH | /REALTIME | /ABOVENORMAL | /BELOWNORMAL]
      [/AFFINITY <hex affinity>] [/WAIT] [/B] [command/program]
      [parameters]

    "title"     Title to display in  window title bar.
    ...

Note the optional title argument and how it can create a syntactical ambiguity. In "start some command", which part of the command line is the title and which is the command? This may or may not be the reason why start trips over command names which contain blank characters - but the optional title argument also provides the workaround for the problem:

  C:\>start "eureka" /b "c:\Program Files\Microsoft Visual Studio 8\Common7\Tools\Bin\uuidgen"
          -ofoobar.txt

In fact, an empty title string will do just as well:

  C:\>start "" /b "c:\Program Files\Microsoft Visual Studio 8\Common7\Tools\Bin\uuidgen"
          -ofoobar.txt

The above example is a little contrived - but the problem is real; recently, it affected some customers when we changed installation paths for our products at CoCreate.


Getting rid of nul, or: How I learnt to love UNC (29 Apr 2006)

Every now and then, some tool on my system runs berserk and starts to generate files called nul. This is a clear indication that there's something going wrong with output redirection in a script, but I still have to figure out exactly what's going on. Until then, I need at least a way to get rid of those files.

Yes, that's right, you cannot delete a file called nul that easily - neither using Windows Explorer nor via the DOS prompt. nul is a very special filename for Windows - it is an alias for the null device, i.e. the bit bucket where all the redirected output goes, all those cries for help from software which we are guilty of ignoring all the time.

UNC path notation to the rescue: To remove a file called nul in, say c:\temp, you can use the DOS del command as follows:

  del \\.\c:\temp\nul

Works great for me. But since I rarely use UNC syntax, I sometimes forget how it looks like. Worse, the syntax requires to specify the full path of the nul file, and I hate typing those long paths. So I came up with the following naïve batch file which does the job for me. It takes one argument which specifies the relative or absolute path of the nul file. Examples:

   rem remove nul file in current dir
   delnul.bat nul
   rem remove nul file in subdir
   delnul.bat foo\nul
   rem remove nul file in tempdir
   delnul.bat c:\temp\nul

For the path completion magic, I'm using the for command which has so many options that my brain hurts whenever I read its documentation. I'm pretty sure one could build a Turing-complete language using just for...

  @echo off
  set fullpath=
  for %%i IN (%1x) DO set fullpath=%%~di%%~pi

  set filename=
  for %%i IN (%1x) DO set filename=%%~ni
  if not "x%filename%" == "xnulx" (echo Usage: %0 [somepath\]nul && goto :eof)
 
  echo Deleting %fullpath%nul...
  del \\.\%fullpath%nul

DelinvFile takes this a lot further; it has a Windows UI and can delete many other otherwise pretty sticky files - nul is not the only dangerous file name; there's con, aux, prn and probably a couple of other magic names which had a special meaning for DOS, and hence also for Windows.


A subset of Ruby (16 Apr 2006)

A programming language which inspires an author to write something like why's (poignant) guide to Ruby must be truly special. So I gave in readily to the temptation to try the language, especially since Ruby's dynamic type system and lambda-like expressions appeal to the Lisp programmer in me.

A while ago, I blogged about the subset sum problem, and so writing a version of that algorithm in Ruby was an obvious choice.

$solutions = 0
$numbers = []
$flags = []

def find_solutions(k, target_sum)
  if target_sum == 0
    # found a solution!
    (0..$numbers.length).each { |i| if ($flags[i]) then print $numbers[i], " "; end }
    print "\n"
    $solutions = $solutions + 1
  else
    if k < $numbers.length
      if target_sum >= $numbers[k]
        $flags[k] = true
        find_solutions k+1, target_sum-$numbers[k]
        $flags[k] = false
      end
      find_solutions k+1, target_sum
    end
  end
end

def find_subset_sum(target_sum)
  print "\nNow listing all subsets which sum up to ", target_sum, ":\n"
  $solutions = 0
  (0..$numbers.length()).each { |i| $flags[i] = false }
  find_solutions 0, target_sum
  print "Found ", $solutions, " different solutions.\n"
end

def subset_sum_test(size)
  total = 0
  target_sum = 0
  (0..size).each { |i| $numbers[i] = rand(1000); total += $numbers[i]; print $numbers[i], " " }
  target_sum = total/2
  find_subset_sum target_sum
end

subset_sum_test 25

Comparing this to my other implementations in various languages, this solution is shorter than the Lisp version, and roughly the same length as the VB code I wrote. I'm pretty sure that as I learn more about Ruby I will be able to improve quite a bit on the above naïve code, so it will probably become even shorter.

But even after the first few minutes of coding in this language, I'm taking away the impression that I can produce code which looks a lot cleaner than, say, Perl code. Well, at least cleaner than the Perl code which I usually write ... big grin


Pasting my own dogfood, part 2 (09 Apr 2006)

Yesterday, I introduced the problem of how to automatically test Windows clipboard code in applications. The idea is to move from manual and error-prone clickety-click style testing to an automatic process which produces reliable results.

Clipboard viewer Unbeknownst to many, Windows ships with a fairly interesting tool called the ClipBook Viewer (clipbrd.exe), which monitors what the clipboard contains, and will even display the formats it knows about.

This is quite helpful while developing and debugging clipboard code. However, ClipBook Viewer can even help with test automation since it can save the current clipboard contents to *.CLP files and load them back into the clipboard later.

Which, in fact, is almost all we need to thoroughly and reliably test clipboard code: We run some apps which produce a good variety of clipboard formats which our own application needs to deal with. We select some data, copy them to the clipboard, then save the clipboard contents as a *.CLP file from ClipBook Viewer.

Once we have created a reasonably-sized clipboard file library, we run ClipBook viewer and load each one of those clipboard files in turn. After loading, we switch to our own app, paste the data and check whether the incoming data makes sense to us. Not bad at all!

But alas, I could not find a way to automate the ClipBook Viewer via the command-line or COM interfaces. If someone knows about such interfaces, I'm certainly most interested to hear about them.

Luck would have it that only recently, I blogged about poor man's automation via SendKeys. The idea is to write a small shell script which runs the target application (here: clipbrd.exe), and then simulate how a user presses keys to use the application.

clipbrd.exe can be started with the name of a *.CLP file in its command line, and will then automatically load this file. However, before it pushes the contents of the file to clipboard, it will ask the user for confirmation in a message box. Well, in fact, first it will try to establish NetDDE connections, and will usually waste quite a bit of time for this. The following script tries to take this into account:

Set WshShell = WScript.CreateObject("WScript.Shell")
WshShell.Run("clipbrd.exe c:\temp\clip.CLP")
WScript.Sleep 5000 ' Wait for "Clear clipboard (yes/no)?" message box
WshShell.SendKeys "{ENTER}"

Now we could add some more scripting glue code to control our own application, have it execute its "Paste" functionality and verify that the data arrives in all its expected glory.

The above code is not quite that useful if we need to run a set of tests in a loop; the following modified version is better suited for that purpose. It assumes that all *.CLP files are stored in c:\temp\clipfiles.

Set WshShell = WScript.CreateObject("WScript.Shell")
WshShell.Run("clipbrd.exe")
WScript.Sleep 20000

startFolder="c:\temp\clipfiles"
set folder=CreateObject("Scripting.FileSystemObject").GetFolder(startFolder)
for each file in folder.files
  WScript.Echo "Now testing " & file.Path
  OpenClipFile(file.Path)
  ' Add here:
  ' - Activate application under test
  ' - Have it paste data from the clipboard
  ' - Verify that the data comes in as expected
next

' Close clipbrd.exe
WshShell.AppActivate("ClipBook Viewer")
WshShell.SendKeys "%F"
WScript.Sleep 1000
WshShell.SendKeys "x"

Sub OpenClipFile(filename)
  WshShell.AppActivate("ClipBook Viewer")
  WshShell.SendKeys "%W"  ' ALT-W for Windows menu
  WScript.Sleep 500
  WshShell.SendKeys "1"   ' Activate Clipboard window
  WScript.Sleep 500
  WshShell.SendKeys "%F"  ' ALT-F for File menu
  WScript.Sleep 1000
  WshShell.SendKeys "O"
  WScript.Sleep 1000
  WshShell.SendKeys filenam9e
  WScript.Sleep 1000
  WshShell.SendKeys "{ENTER}"
  WScript.Sleep 1000  ' Wait for "Clear clipboard (yes/no)?"
  WshShell.SendKeys "{ENTER}"
End Sub

I'm sure a VBscript hacker could tidy this up considerably and use it to form a complete test suite. However, while this approach finally gives us some degree of automation, it is still lacking in several ways:

  • The format for the *.CLP file is undocumented, so we cannot add clipboard data of our own, unless we first copy it to the clipboard, then save it from there using ClipBook Viewer.
  • Automation via sending keys is a very brittle approach. For instance, the above code was written for the English version of clipbrd.exe. The German or French or Lithuanian versions of clipbrd.exe might have completely different keyboard shortcuts.
  • I shudder when looking at those magic delay time values which the code is ridden with - what if we run on a really slow system? On a system which has even more networking problems than the one which I tested the code with?
  • Any process (or user) stealing the window focus while the test is running will break the test.

Hence, next time: Look Ma, no SendKeys!


Regular recursion (25 Mar 2006)

So I tried to come up with some simple code in VBscript which recursively searches a directory for file names of arbitrary patterns. This is what I got working:

Sub recursiveSearch(dir, regex)
  for each file in dir.files
    if regex.Test(file.Name) Then
      WScript.Echo("File matches: " & file.Path)
    End if
  next

  for each folder in dir.SubFolders
    recursiveSearch folder, regex
  next
End Sub

startFolder="c:\temp"
set folder=CreateObject("Scripting.FileSystemObject").GetFolder(startFolder)

Set regex=new RegExp
regex.Pattern = "^Foo\d{3}[0-9a-zA-Z]\.txt$"
' File name starts with 'Foo', followed by three digits, then either
' a digit or letter, and has a .txt extension.
recursiveSearch folder, regex

Somehow I've got a hunch that there may be an easier way to do this. Blogosphere, any ideas?


When asked for a TWiki account, use your own or the default TWikiGuest account.


Don't quote me on this (18 Mar 2006)

Let us assume that I'm a little backward and have a peculiar fondness for the DOS command shell. Let us further assume that I also like blank characters in pathnames. Let us conclude that therefore I'm hosed.

But maybe others out there are hosed, too. Blank characters in pathnames are not exactly my exclusive fetish; others have joined in as well (C:\Program Files, C:\Documents and Settings). And when using software, you might be running cmd.exe without even knowing it. Many applications can run external helper programs upon user request, be it through the UI or through the application's macro language.

The test environment is a directory c:\temp\foo bar which contains write.exe (copied from the Windows system directory) and two text files, one of them with a blank in its filename.

Now we open a DOS shell:

C:\>dir c:\temp\foo bar
 Volume in drive C is IBM_PRELOAD
 Volume Serial Number is C081-0CE2

 Directory of c:\temp

File Not Found

 Directory of C:\

File Not Found

C:\>dir "c:\temp\foo bar"
 Volume in drive C is IBM_PRELOAD
 Volume Serial Number is C081-0CE2

 Directory of c:\temp\foo bar

03/18/2006  03:08 PM    <DIR>          .
03/18/2006  03:08 PM    <DIR>          ..
01/24/2006  11:19 PM             1,516 foo bar.txt
01/24/2006  11:19 PM             1,516 foo.txt
03/17/2006  09:44 AM             5,632 write.exe
               3 File(s)          8,664 bytes
               2 Dir(s)  17,448,394,752 bytes free

Note that we had to quote the pathname to make the DIR command work. Nothing unusual here; quoting is a fact of life for anyone out there who ever used a DOS or UNIX shell.

Trying to start write.exe by entering c:\temp\foo bar\write.exe in the DOS shell fails; again, we need to quote:

C:\>"c:\temp\foo bar\write.exe"

And if we want to load foo bar.txt into the editor, we need to quote the filename as well:

C:\>"c:\temp\foo bar\write.exe" "c:\temp\foo bar\foo bar.txt"

Still no surprises here.

But let's suppose we want to run an arbitrary command from our application rather than from the command prompt. The C runtime library provides the system() function for this purpose. It is well-known that under the hood system actually runs cmd.exe to do its job.

#include <stdio.h>
#include <process.h>

int main(void)
{
  char *exe = "c:\\temp\\foo bar\\write.exe";
  char *path = "c:\\temp\\foo bar\\foo bar.txt";

  char cmdbuf[1024];
  _snprintf(cmdbuf, sizeof(cmdbuf), "\"%s\" \"%s\"", exe, path);

  int ret = system(cmdbuf);
  printf("system(\"%s\") returns %d\n", cmdbuf, ret);
  return 0;
}

When running this code, it reports that system() returned 0, and write.exe never starts, even though we quoted both the name of the executable and the text file name.

What's going on here? system() internally runs cmd.exe like this:

  cmd.exe /c "c:\temp\foo bar\write.exe" "c:\temp\foo bar\foo bar.txt"

Try entering the above in the command prompt: No editor to be seen anywhere! So when we run cmd.exe programmatically, apparently it parses its input differently than when we use it in an interactive fashion.

I remember this problem drove me the up the freakin' wall when I first encountered it roughly two years ago. With a lot of experimentation, I found the right magic incantation:

  _snprintf(cmdbuf, sizeof(cmdbuf), "\"\"%s\" \"%s\"\"", exe, path);
  // originally: _snprintf(cmdbuf, sizeof(cmdbuf), "\"%s\" \"%s\"", exe, path);

Note that I quoted the whole command string another time! Now the executable actually starts. Let's verify this in the command prompt window: Yes, something like cmd.exe /c ""c:\temp\foo bar\write.exe" "c:\temp\foo bar\foo bar.txt"" does what we want.

I was reminded of this weird behavior when John Scheffel, long-time user of our flagship product OneSpace Designer Modeling and maintainer of the international CoCreate user forum, reported funny quoting problems when trying to run executables from our app's built-in Lisp interpreter. John also found the solution and documented it in a Lisp version.

Our Lisp implementation provides a function called sd-sys-exec, and you need to invoke it thusly:

(setf exe "c:/temp/foo bar/write.exe")
(setf path "c:/temp/foo bar/foo bar.txt")
(oli:sd-sys-exec (format nil "\"\"~A\" \"~A\"\"" exe path))

Kudos to John for figuring out the Lisp solution. Let's try to decipher all those quotes and backslashes in the format statement.

Originally, I modified his solution slightly by using ~S instead of ~A in the format call and thereby saving one level of explicit quoting in the code:

  (format nil "\"~S ~S\"" exe path))

This is much easier on the eyes, yet I overlooked that the ~S format specifier not only produces enclosing quotes, but also escapes any backslash characters in the argument that it processes. So if path contains a backslash (not quite unlikely on a Windows machine), the backslash will be doubled. This works surprisingly well for some time, until you hit a UNC path which already starts with two backslashes. As an example, \\backslash\lashes\back turns into \\\\backslash\\lashes\\back, which no DOS shell will be able to grok anymore.

John spotted this issue as well. Maybe he should be writing these blog entries, don't you think? smile

From those Lisp subtleties back to the original problem: I never quite understood why the extra level of quoting is necessary for cmd.exe, but apparently, others have been in the same mess before. For example, check out this XEmacs code to see how complex correct quoting can be. See also an online version of the help pages for CMD.EXE for more information on the involved quoting heuristics applied by the shell.

PS: A very similar situation occurs in OneSpace Designer Drafting as well (which is our 2D CAD application). To start an executable write.exe in a directory c:\temp\foo bar and have it open the text file c:\temp\foo bar\foo bar.txt, you'll need macro code like this:

LET Cmd '"C:\temp\foo bar\write.exe"'
LET File '"C:\temp\foo bar\foo bar.txt"'
LET Fullcmd (Cmd + " " + File)
LET Fullcmd ('"' + Fullcmd + '"')  { This is the important line }
RUN Fullcmd

Same procedure as above: If both the executable's path and the path of the data file contain blank characters, the whole command string which is passed down to cmd.exe needs to be enclosed in an additional pair of quotes...

PS: See also http://blogs.msdn.com/b/twistylittlepassagesallalike/archive/2011/04/23/everyone-quotes-arguments-the-wrong-way.aspx


When asked for a TWiki account, use your own or the default TWikiGuest account.


Rapid proto-typing (03 Mar 2006)

Much to my dismay, I found myself in a situation where the following hack is useful. I shudder at the thought of actually using it because of its inherent instability, but sometimes it's better than a poke in the eye with C#.

If you're automating an application which, while executing a command, may pop up error or warning messages and wait for user input, you may need to explicitly send a keystroke to that application. Fortunately, this is reasonably simple using cscript.exe, the WSH Shell object and VBscript:

Set WshShell = WScript.CreateObject("WScript.Shell")
WshShell.AppActivate ("Appname as it appears in the main window title")
WshShell.SendKeys "{ENTER}"

While testing this, I learnt that the application name parameter to AppActivate can actually be an abbreviation. For instance, if you run Word, its main window title is usually something like "gazonk.doc - Microsoft Word". AppActivate actually uses a simple best-match algorithm so that the following will still work as expected:

Set WshShell = WScript.CreateObject("WScript.Shell")
WshShell.AppActivate ("Microsoft Word")
WshShell.SendKeys "foo"

The SendKeys method turns out to be pretty convenient since it allows to describe non-printable characters with a special notation, such as {BREAK} for the Break key, {PGUP} and {PGDN} for moving pagewise, {DEL}, {HOME}, all the function keys et cetera.


When asked for a TWiki account, use your own or the default TWikiGuest account.


Environmental unconsciousness (4.2.2006)

Chances are that - by looking at my earlier blog entry on batch files - you think I'm a DOS lamer. Nothing could be further from the truth, because I'm really a UNIX lamer. (OK, so what really shaped my thinking even before that was the phrase "38911 bytes free". But I digress.)

So I still write little one-off scripts using bash, typically in a Cygwin environment. One of these scripts recently ran berserk, reporting lots of errors like this one:

  ./foo.sh: line 42: /usr/bin/find: Resource temporarily unavailable

I couldn't really figure out what resources the shell was talking about. Memory? Certainly not - the test system had ample memory, and was hardly using any. Files or disk space? Nope, lots of free disk space everywhere, and noone was fighting over access to shared files or so. Too many processes? Process Explorer wouldn't think so. Hmmm...

This test script then revealed the truth:

typeset -i limit=2200

# Create a file with 2200 environment variable definitions
rm -f exportlist
typeset -i i=0
while [ $i -lt $limit ]
do
  echo "export FOO$i=$i" >>exportlist
  let i=i+1
done

# Import the environment definitions
source ./exportlist 

# Are we still alive?
env | wc
find . -name exportlist

Run this script and watch it balk miserably about unavailable resources. So it's the environment which filled up and caused my scripts to fail! And indeed, the system for which the problem was originally reported uses a lot of environment variables for some reason, and this broke my script.

Once I had found out that much, it was easy to google for the right search terms and learn more: In this Cygwin mailing list discussion, Mike Sieweke explains that we are actually suffering from a Windows limitation here - apparently, the environment cannot grow larger than 32K. Christopher Faylor, chief maintainer of Cygwin, even recommends a workaround, but I haven't tested that one yet; instead, I helped to clean up the polluted environment on the affected PC, and henceforth, no waldsterben anymore on that system.

32K - this would have filled almost all of those 38911 memory bytes assigned for BASIC programs on my good ol' 64...


When asked for a TWiki account, use your own or the default TWikiGuest account.


Audio kills the video star (Jan 29, 2006)

Movie podcasts are the next big thing after blogging and podcasting. For the time being, I'll stick to my blog, thanks very much for asking, but I do listen to a lot of podcasts while commuting or exercising. Occasionally, I also watch some of the Channel 9 videos where Microsoft engineers and employees talk about their work. No matter what you think about the company in general, everybody knows that Microsoft hires smart people, so there is a lot to learn from them.

Many of those videos contain demos or at least feature casually-dressed geeks scribbling frantically on whiteboards, which, of course, is a must-see (ahem). But quite a few videos could be enjoyed almost just as well in pure audio format. Unfortunately, most of the Channel 9 content is in video format (*.wmv) only, which will neither fit nor play on my 512 MB MP3 player.

I'm pretty much a newbie in all things video, and so I was glad that Minh Truong suggested a way to convert WMV to WMA using Windows Media Encoder.

This actually works fine, but it's a lot of settings to remember (see the screenshots below), and it produces WMA instead of MP3 or OGG format which I'd prefer.

Fortunately, I found that Windows Media Encoder actually ships with a script called WMCmd.vbs which takes a gazillion parameters and automates the conversion process! And indeed, the following trivial command line produces a WMA audio file from a WMV video:

cd c:\Program Files\Windows Media Components\Encoder
cscript.exe WMCmd.vbs -input c:\temp\foo.wmv -output c:\temp\foo.wma -audioonly

There are a number of options to control the quality and encoding of the output which I haven't explored at all.

So now I only need to find a reasonable WMA-to-MP3 converter which can be used from the command line. batchenc and dBpowerAMP Music Converter look like they could help with that part of the job, but I'm not sure. Sounds like I have a plan for next weekend big grin


When asked for a TWiki account, use your own or the default TWikiGuest account.


Batch-as-batch-can! (27.1.2006)

So here I confess, not without a certain sense of pride: Sometimes I boldly go where few programmers like to go - and then I write a few lines in DOS batch language.

Most of the time, it's not as bad as many people think. Its bad reputation mostly stems from the days of DOS and Windows 95, but since the advent of Windows NT, the command processor has learnt quite a few new tricks. While its syntax remains absurd enough to drive some programmers out of their profession, you can now actually accomplish most typical scripting tasks with it. In particular, the for statement is quite powerful.

Anyway - a while ago, one of my batchfiles started to act up. The error message was "The system cannot find the batch label specified - copyfile". The batch file in question had a structure like this:

@echo off
rem copy all pdb files in the current directory into a backup directory
set pdbdir=c:\temp\pdbfiles

for /r %%c in (*.pdb) do call :copyfile "%%c" %pdbdir%
if errorlevel 1 echo Error occurred while copying pdb files

echo All pdb files copied.
goto :eof

rem copyfile subroutine
:copyfile
  echo Copying %1 to %2...
  copy /Y %1 %2 >nul
  goto :eof

I know what you're thinking - no, this is not the problem. This is how you write subroutines in DOS batch files. Seriously. And yes, the above script can of course be replaced by a single copy command. The original script couldn't; it performed a few extra checks for each and every file to be copied in the :copyfile subroutine, but it also contained a lot of extra fluff which distracts from the actual problem, so what you're seeing here is a stripped-down version.

The error message complained that the label copyfile could not be found. Funny, because the label is of course there. (The leading colon identifies it as a label.) And in fact, the very same subroutine could be called just fine from elsewhere in the same batch file!

For debugging, I removed the @echo off statement so that the command processor would log all commands it executes; this usually helps to find most batch file problems. But not this one - removing the echo "fixed" the bug. I added the statement again - now I got the error again. Removed the echo statement - all is fine.

Oh great. It's a Heisenbug. So I added the echo statement back in again and stared at the script hoping to find the problem by the old-fashioned method of "flash of inspiration".

No inspiration in sight, though. Not knowing what to do, I added a few empty lines between the for and the if errorlevel statement and ran the script again - no error message! Many attempts later, I concluded that it's the sheer length of the script file which made the difference between smooth sailing and desperation. By the way, the above demo script works just fine, of course, because I stripped it down for publication.

Google confirmed my suspicion: Apparently, there are cases where labels cannot be found even though they are most certainly in the batch file. Maybe the length of the label names matters - Microsoft Knowledge Base Article 63071 suggests that only the first eight characters of the label are significant. However, copyfile has exactly eight characters!

I still haven't solved this puzzle. If you're a seasoned batch file programmer sent to this place by Google and can shed some light on this, I could finally trust that script again...

-- ClausBrod - 27 Jan 2006


When asked for a TWiki account, use your own or the default TWikiGuest account.

From http://help.wugnet.com/windows/system-find-batch-label-ftopict615555.html, I tentatively conclude that you need two preconditions for this to hit you:

  • the batch file must not use CRLF line endings
  • the label you jump to must span a block boundary

As to your remark "And in fact, the very same subroutine could be called just fine from elsewhere in the same batch file": in my experience, the subroutine gets called just fine when you get this error.

Regards,

Reinder

-- Reinder - 20 Jun 2008


What's my email address? (4 Jan 2006)

On various occasions, I had already tried to make sense out of directory services such as LDAP and Microsoft's ADSI. Now, while that stuff is probably not rocket science, the awkward terminology and syntax in this area have always managed to shy me away; most of the time, there was another way to accomplish the same without going through LDAP or ADSI, and so I went with that.

This time, the task was to retrieve the email address (in SMTP format) for a given user. In my first attempt, I tried to tap the Outlook object model, but then figured that a) there are a few systems in the local domain which do not have Outlook installed and b) accessing Outlook's address info causes Outlook to display warnings to the user reporting that somebody apparently is spelunking around in data which they shouldn't be accessing. Which is probably a good idea, given the overwhelming "success" of Outlook worms in the past, but not exactly helpful in my case.

However, everybody here is connected to a Windows domain server and therefore has access to its AD services, so that sounded like a more reliable approach. I googled high and low, dissected funky scripts I found out there and put bits of pieces of them together again to form this VBscript code:

  user="Claus Brod"
  context=GetObject("LDAP://rootDSE").Get("defaultNamingContext")
  ou="OU=Users,"
  Set objUser = GetObject("LDAP://CN=" & user & "," & ou & context)
  WScript.Echo(objUser.mail)

  groups=objUser.Get("memberOf")
  For Each group in groups
    WScript.Echo("  member of " & group)
  Next

This works, but the OU part of the LDAP string (the "ADsPath") depends on the local organizational structure and needs to be adapted for each particular environment; I haven't found a good way to generalize this away. Hints most welcome.

PS: For those of you on a similar mission, Richard Mueller provides some helpful scripts at http://www.rlmueller.net/freecode3.htm.


When asked for a TWiki account, use your own or the default TWikiGuest account.



to top


You are here: Blog > WebLeftBar > BlogOnSoftwareScripting

r1.14 - 30 Jun 2013 - 13:02 - ClausBrod to top

Blog
RSS
  2013: 9 - 8 - 7 - 6 - 5
  2012: 2 - 10
  2011: 1 - 8 - 9 - 10 - 12
  2010: 11 - 10 - 9 - 4
  2009: 11 - 9 - 8 - 7 -
     6 - 5 - 4 - 3
  2008: 5 - 4 - 3 - 1
  2007: 12 - 8 - 7 - 6 -
     5 - 4 - 3 - 1
  2006: 4 - 3 - 2 - 1
  2005: 12 - 6 - 5 - 4
  2004: 12 - 11 - 10
  C++
  COM & .NET
  Lisp
  Scripting
  Windows
  CoCreate Modeling
  OpenSource
  Java
  Stuff
Changes
Index
Search
Maintenance
Impressum
Home



Jump:

Copyright © 1999-2014 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback