It is impossible to open a JFrame with an applet. If you want to do any graphics, you can do it in the rectangular region of the applet on a page. The graphics can be basically as good as your skills allow them to be, though.
Computers are fast enough now that you could render some 3D graphics in an applet using purely CPU-based processing and keep a reasonable frame-rate. There are some ways to use OpenGL for 3D hardware acceleration in Java but have not tried hard at putting this in a Java applet.