I have created a top-level custom entity and everything works great in the web client, including the left nav, creating groups from it, and filtering.  However, I get a javascript error when I try to do a lookup.  I right click on the new left-nav item and choose Lookup and the little lookup dialog appears with my default group fields listed but when I type in a search criteria and click Lookup I receive a very vague javascript error popup.  The default group works fine too.

Try clearing the browser cache. I have seen this cause javascript errors on even the out of the box screens that a cache clearance has helped with.