Beware of unknown enumerators (26.1.2006)

The other day, I was testing COM clients which accessed a collection class via a COM-style enumerator (IEnumVARIANT). And those clients crashed as soon as they tried to do anything with the enumerator. Of course, the same code had worked just fine all the time before. What changed?

In COM, a collection interface often implements a function called GetEnumerator() which returns the actual enumerator interface (IEnumVARIANT), or rather, a pointer to the interface. In my case, the signature of that function was:

   HRESULT GetEnumerator(IUnknown **);

Didn't I say that GetEnumerator is supposed to return an IEnumVARIANT pointer? Yup, but for reasons which I may cover here in one of my next bonus lives, that signature was changed from IEnumVARIANT to IUnknown. This, however, is merely a syntactic change - the function actually still returned IEnumVARIANT pointers, so this alone didn't explain the crashes.

Well, I had been bitten before by smart pointers, and it happened again this time! The COM client code declared a smart pointer for the enumerator like this:

  CComPtr<IEnumVARIANT> enumerator = array->GetEnumerator();

This is perfectly correct code as far as I can tell, but it causes a fatal avalanche:

• The compiler notices that GetEnumerator returns an IUnknown pointer. This doesn't match the constructor of this particular smart pointer which expects an argument of type IEnumVARIANT *.
• So the compiler looks for other matching constructors.
• It doesn't find a matching constructor in CComPtr itself, but CComPtr is derived from CComPtrBase which has an undocumented constructor CComPtrBase(int).
• To match this constructor, the compiler converts the return value of GetEnumerator() into a bool value which compresses the 32 or 64 bits of the pointer into a single bit! (Ha! WinZip, can you beat that?)
• The boolean value is then passed to the CComPtrBase(int) constructor.
• To add insult to injury, this constructor doesn't even use its argument and instead resets the internally held interface pointer to 0.

Any subsequent attempt to access the interface through the smart pointer now crashes because the smart pointer tries to use its internal interface pointer - which is 0.

All this happens without a single compiler or runtime warning. Now, of course it was our own silly fault - the GetEnumerator declaration was bogus. But neither C++ nor ATL really helped to spot this issue. On the contrary, the C++ type system (and its implicit type conversions) and the design of the ATL smart pointer classes collaborated to hide the issue away from me until it was too late.

to top