I hit bug 7235187 today when I tried to set a TDE master key in a wallet created for OID authentication. Seems it will be fixed in 10.2.0.5 / 11.2. Workaround is to have two wallets.
WALLET_LOCATION=(SOURCE=(METHOD = FILE)(METHOD_DATA =(DIRECTORY = /opt/oracle/product/admin/xxx/wallet)))
# Try to create the key
SQL> ALTER SYSTEM SET ENCRYPTION KEY AUTHENTICATED BY "password";
ALTER SYSTEM SET ENCRYPTION KEY AUTHENTICATED BY "password"
ERROR at line 1:
ORA-28378: Wallet not open after setting the Master Key