Now 2005.May 24 morning, at 08:38 got linear test code working to queue up any output written to writer of wrapped response object, then at very end request all that text to be converted to HTML as a string, which test loop prints on terminal. For now, all the output being queued is via print and println, all plain text, no structured output implemented yet, but the framework is in place. After grocery trip, updated index file to include milestones, uploaded that and previous day's log, installed on JavaClass WebSite. Sent e-mail to Sukhjit asking for more time to complete project. Collected all files to-date into single directory, zipped, uuencoded, uploaded, uudecoded, e-mailed to Yahoo! Mail account so I'll have them to try installing stuff on campus lab J2EE system. On campus: Unzipped the e-mail attachment, compiled toplevel Loginout, configured properties file for Windows, set up data source, tried to run the test program but it failed because I had forgotten to create the Users table. Compiled Users, tried to run it, but it failed because MicroSoft ACCESS doesn't accept the SQL command for creating a new table. Sent question to instructor. Also on campus: Tried to find jar file instructor told me about, found it under slightly different name. Tried to find batch file instructor told me about, found several batch files but none seem to be the right one, sent question to instructor. Now Wednesday morning, May.25. I've been investigating a problem with 'out' variable in JSP and related output stream which is part of response object, finally at 07:55 I found the problem: The response.getWriter() object writes all output to the stream to the client *immediately*, whereas the 'out' variable in JSP merely buffers up all the output and then sends to the client steam at the very end of the running of the JSP/servlet, which means it gets written after the very last of the direct output. So now at 08:05 I need to investigate what happens when servlet is called from JSP. Results: Just like from within a JSP itself, all output to response.getWriter() appears before the first output via the 'out' variable from the JSP. But if an exeption occurs, even in the JSP after return from the servlet, all output from both streams is lost. But if I say out.flush() in the JSP, it clears buffers in both the servlet writer and then the JSP writer, so then if an exception occurs, both sets of output appear before the exception report. Summary: JSP prints aaa, JSP calls servlet, servlet prints bbb, servlet returns, JSP prints ccc, JSP flushes buffers, JSP throws exception eee. Sequence of appearance in Web page is bbb aaa ccc eee. So now I know a new trick when debugging dataflow JSPs: Whenever I'm about to run some untested code that might bomb out with an exception, call out.flush() just before it, and if calling several lines of untested code call out.flush() again between each consecutive pair of such untested code. Of course this does nothing to protect from compiler diagnostics which throw servlet exception which prevents any of my code from being executed whatsoever. It protects me only from runtime exceptions in second line of new code causing loss of output from first line of new code. So now I need to decide which of these two output streams to use when debugging dataflow that I intend, when it's all debugged, to move into regular class files used in servlets. I think the best answer is *never* to use the 'out' variable of the JSP, instead *always* use the writer embedded in the response object. This decision applies to (1) DataFlow.jsp, (2) DataFlow.java/class called from jump1.jsp, (3) DispatcherServlet.java/class called from jump1.jsp, (4) DispatcherServlet.java/class installed directly under J2EE/TomCat servlet container. By making this decision, I'll be sure of getting consistent results as my new code miagrates from DataFlow.jsp through the other intermediary operational modes to finally arrive at a fullfledged working servlet running as a legitimate J2EE/TomCat serverside application. Note that if I were developing a proper JSP application, then the above would not be correct, because JSPs are supposed to write output to the stream bound to the page variable 'out', not directly to the printer of the response object. But I'm not trying to develop a proper JSP application, I'm just using a JSP as a trick to debug code intended for a DispatcherServlet.java/class, so using the servlet stream instead of the JSP page stream seems appropriate. Next to edit DataFlow.jsp to effect that decision, setting sOut to be the server output stream, and replacing all references to out to be sOut instead. Starting edit at 09:29 ... done, cleaning up formatting of trace output to be more legible ... done, starting to add actual unwrapping code for HTTP case ... first draft done at 10:52, cleaning up dummy code that was there before ... done, now reconciling this latest version of DataFlow.jsp with earlier version of DataFlow.java, now 11:33 as I continue with this needed task ... done, and cleaned up and clarified relationship between JSP 'out' page stream and writer used by servlet, including looking at HTML source of output from DataFlow.jsp, working perfectly now. Next one final check of jump1.jsp -> DataFlow.java/class, including looking at HTML source in browser, but first need to start J2EE stop/restart cycle, IP, next recompile DataFlow.java, oops undefined formatting constants that were set up in DataFlow.jsp, need to copy their definitions over ... done and recompiling again ... shit, I hate absolutely being required to declare exceptions or else my code won't even compile!! Gotta put back in the try/catch block I mistakenly took out, done but I forgot to move the declaration outside the try block so it'd be accessible later, so another recompile required, done at 12:16 with no compiler errors, time to copy to system directory and try it ... done, and J2EE is back up already, so refreshing jump1.jsp->DataFlow.java/class Web page at 12:18 ... output looks fine, but viewing source shows several blank lines after the servlet returns before the text at the very top of the JSP, and I have no idea where that's coming from, but it doesn't mess up the display, and this is just a debug rig anywy, so it's not worth tracking down. Now things are completed and stable enough that it's time for me to start developing UI-independent login form and conversion of it to HTML, not under JSP at first, only in null test rig. But first I need to try running the current DataFlow.java/class under a NULL test rig. -- After nap. Nap interrupted after less than 20 minutes by loud noise from outside, and I haven't been able to get back to any restful state, so resuming under duress at 13:44. Converting DataFlow.java to properly structured DispatcherServlet.java, first copying structured prototype in comment in DataFlow.jsp directly across while reconciling with what was already in DispatcherServlet.java ... done at 14:00, next converting inline code from DataFlow.java to fit structured format ... done at 16:02, ready to try on server for the first time ... not getting the new code (claims no such method, the unwrapper I just finished writing) because HSA is in the cache, so gotta bring down server, now 16:10 ... now 16:12 and restarting server ... up at 16:17, started reload of Web page, ... hey, it works! Now is the big moment, to switch to running the servlet from a test rig instead of under the J2EE/HTTP servlet container. Taking a break (dishwashing & rest) first. Resuming work at 18:57. Starting to develop NULL test rig for servlets. Got that working and then converted inline HTML text for head and title into formal setTitle call, whereby the HTML gets generated automatically by the unwrapper so the writer of the innerds of the servlet doesn't have to know any such HTML syntax. Output is in both s-expression and HTML format in the toplevel debug rig. Got that all working at 21:10. Next to get XML output to do same ... done at 21:22. Next to design a nested-list structure to represent a form for the user to fill out to then submit to the DispatcherServlet ... something like this (in s-expression notation, you really don't want to see how verbose this would be in XML do you?)): (FORM (FIELDS (METHOD "post") (ACTION "http://localhost:8080/date/jump2.jsp") (NAME "loginForm")) (HIDDEN "jsession" "4poiv455tyn3gi") (FLOW "Your ID: " (TEXTFIELD (NAME "login") (SIZE 17) (MAXLENGTH 32) (VALUE ""))) (FLOW "Password: " (PASSWORDFIELD (NAME "passwd") (SIZE 17) (MAXLENGTH 32))) (SUBMIT (NAME "checkpw") (VALUE "Sign In")) ) Note that parts listed at the top level of the FORM structure, after the first item which must be the FIELDS, are defined to stack vertically. If you want anything to flow horizontally as a group, enclose all such items within an extra level of FLOW structure. Methods defined as follows: - static MyList makeFormFields(String method, String action, String name) - static MyList makeHidden(String name, String value) - static MyList makeTextField(String name, int size, int maxlength, String value) - static MyList makePasswordField(String name, int size, int maxlength) - static MyList makeSubmitButton(String name, String value) - constructor FlowPane() - .add(Object elem) - constructor StackOfPanes() - .add(Object elem) - MyList makeForm(MyList formfields, StackOfPanes sop) Once such an object has been built, the return value from makeForm gets passed to .writeObject(theObject) It's now 22:26 and I've written most of the code for building such an object, except stubs for most of the static constructor methods, and I'm ready to make the first test rig for it and apply TDD to fill in the stubs with real code. ... After midnight, at 00:22, I've finished writing and testing all the code to build all the parts of the form, all the way up to the two main parts of the form (the FIELDS, and the StackOfPanes that constitutes the main body of the form). After a night's sleep I should be able to quickly finish building the FORM object by simply putting those two pieces together, then start coding the conversion of it to HTML. May.26 at 07:09, finished code to build a FORM object and print in XML notation (conversion to s-expression notation was already general enough). Starting work to convert FORM object to HTML notation ... took long nap and resumed ... done and working almost correctly at 12:27 ... fixed and working perfectly at 12:40. All the pieces are finished! Next to finish building the toplevel test rig, and modify the DispatcherServlet to check parameters and put up the login form if the user is not yet logged in ... partly done before going to class. Before class, in campus lab: Tried what Suhkjit suggested in e-mail: I was able to compile the HelloW.java file, but not able to figure out how to run it on server. After class, in campus lab: Guessed the URL to use, but it bombed with 500 invalid class type. Sukhjit spent a lot of time claiming I should be able to figure it out myself before he finally edited the batch file to make a new copy missing some of the stuff that switched control to another batch file thereby erasing all changes it had made to the current DOS shell. By running the new batch file, I was finally able, for the very first time ever, to compile a servlet and have it run correctly on MS-Windows. So finally I have the capability, when I've finished debugging my Lab#3 on the laptop, of then porting to the campus lab computer and running it there too. Back at home: Continued work on NULL test rig, but was too tired and had to crash. Woke up a few hours later and finished getting NULL test rig working well enough to pass simulated encoded-HTML-form-contents (i.e. tag-value pairs as string in format tag1=val1&tag2=val2...) to be incorporated into the request-object parameters to then be accessible to the servlet via getParameterNames() and getParamater(tag). It's now May.27 at 02:19. Starting to write the control logic for DispatcherServlet, whereby it detects whether user is connecting for the first time (hence needs to get login form) or is reconnecting in the middle of a session where some task is in progress and form contents need to be processed. ... OK, in test rig I have working the decision between initial connection (therefore emit login form, done), and submitting login form (therefore run business logic for login, not yet connected to that code I wrote weeks ago), and running Duke's stupid name dummy test (working), at 04:10 starting J2EE server to verify it all works for real ... while it was starting up I remembered to copy all the latest versions of class files to the system directory, done, by now the server was already up, started lynx and clicked on link to run jump2.jsp which links to servlet, and after I accepted the cookie, up came the login form! I filled it out, and tried to submit it, but couldn't connect to host. I checked the URL where I was and the URL where I was trying to connect, only difference was port 8000 originally but port 8080 from submit button in login form. Started shutdown of server, edited source to fix the port number, started recompilation, at 04:28 waiting for either to finish ... compiler finished first, copied new class file to /tmp/, J2EE shutdown finished now too, invoked J2EE restart, copied new class file from /tmp/ to system area, waiting for J2EE server to finish restarting ... blam: J2EE server listen port: 1050 java.lang.RuntimeException: Could not initialize j2ee server. Possible cause could be another instance of the server already running. at com.sun.enterprise.iiop.POAProtocolMgr.initializeNaming(POAProtocolMgr.java:134) at com.sun.enterprise.server.J2EEServer.run(J2EEServer.java:222) at com.sun.enterprise.server.J2EEServer.main(J2EEServer.java:913) java.lang.RuntimeException: Could not initialize j2ee server. Possible cause could be another instance of the server already running. at com.sun.enterprise.iiop.POAProtocolMgr.initializeNaming(POAProtocolMgr.java:134) at com.sun.enterprise.server.J2EEServer.run(J2EEServer.java:222) at com.sun.enterprise.server.J2EEServer.main(J2EEServer.java:913) java.lang.RuntimeException: Could not initialize j2ee server. Possible cause could be another instance of the server already running. at com.sun.enterprise.server.J2EEServer.run(J2EEServer.java:350) at com.sun.enterprise.server.J2EEServer.main(J2EEServer.java:913) J2EE server reported the following error: Could not initialize j2ee server. Possible cause could be another instance of the server already running. Error executing J2EE server ... I'm supposed to run cleanup now, but last time I did that it destroyed all my deployed applications and I had to spend two hours just to get one (1) dummy application back up so I could patch my code into it like I'm doing. So I just tried again to run j2ee -verbose, and it did: J2EE server listen port: 1050 Naming service started:1050 Binding DataSource, name = jdbc/InventoryDB, url = jdbc:cloudscape:rmi:CloudscapeDB;create=true Binding DataSource, name = jdbc/Cloudscape, url = jdbc:cloudscape:rmi:CloudscapeDB;create=true Binding DataSource, name = jdbc/DB1, url = jdbc:cloudscape:rmi:CloudscapeDB;create=true Binding DataSource, name = jdbc/DB2, url = jdbc:cloudscape:rmi:CloudscapeDB;create=true Binding DataSource, name = jdbc/EstoreDB, url = jdbc:cloudscape:rmi:CloudscapeDB;create=true Binding DataSource, name = jdbc/XACloudscape, url = jdbc/XACloudscape__xa Binding DataSource, name = jdbc/XACloudscape__xa, dataSource = COM.cloudscape.core.RemoteXaDataSource@2b6c89 Starting JMS service... Initialization complete - waiting for client requests Binding: < JMS Destination : jms/Queue , javax.jms.Queue > Binding: < JMS Destination : myQueue , javax.jms.Queue > Binding: < JMS Destination : jms/Topic , javax.jms.Topic > Binding: < JMS Cnx Factory : jms/QueueConnectionFactory , Queue , No properties > Binding: < JMS Cnx Factory : QueueConnectionFactory , Queue , No properties > Binding: < JMS Cnx Factory : jms/TopicConnectionFactory , Topic , No properties > Binding: < JMS Cnx Factory : TopicConnectionFactory , Topic , No properties > Starting web service at port: 8000 Starting secure web service at port: 7000 J2EE SDK/1.3 Starting web service at port: 9191 J2EE SDK/1.3 Loading jar:/usr/java/j2sdkee1.3/repository/adsl-63-192-44-196.roadguy.com/applications/DateApp1116664022695Server.jar /usr/java/j2sdkee1.3/repository/adsl-63-192-44-196.roadguy.com/applications/DateApp1116664022695Server.jar Created Context:/date J2EE server startup complete. so I don't know why it failed the first time I tried to restart it. Anyway, going to refresh Web page and see if the correct port number works at 04:40 ... good, trace output to J2EE system window shows it is making the correct decisions, and output to Web page shows the under construction notice as it should. Way past time for bed at 04:43. After waking up May.27 daytime, converted Loginout to change static methods to be instance methods instead, changed login method to save explanation (why login failed) text in instance variable StringBuffer details instead of immediately printing to output stream, added method getDetails, updated main (unit-test rig) to do things per above changes, tested it, works fine! Began adapting DispatcherServlet, in case of verifying data from login form, to call login method in Loginout, testing in toplevel NULL test rig, it works, time now 10:32. Made one small change to trace output, compiling at 10:36 ... done, tested, all working at 10:42. Now to bring up J2EE server to do a live test ... at 10:43 waiting for it to restart ... started up lynx and all ready to click on link to servlet ... bombed out again: J2EE server listen port: 1050 org.omg.CORBA.INTERNAL: minor code: 1398079697 completed: No at com.sun.corba.ee.internal.iiop.GIOPImpl.createListener(GIOPImpl.java:256) at com.sun.corba.ee.internal.iiop.GIOPImpl.getEndpoint(GIOPImpl.java:205) at com.sun.corba.ee.internal.iiop.GIOPImpl.initEndpoints(GIOPImpl.java:140) at com.sun.corba.ee.internal.POA.POAORB.getServerEndpoint(POAORB.java:488) at com.sun.corba.ee.internal.POA.POAImpl.pre_initialize(POAImpl.java:154) at com.sun.corba.ee.internal.POA.POAImpl.(POAImpl.java:112) at com.sun.corba.ee.internal.POA.POAORB.makeRootPOA(POAORB.java:110) at com.sun.corba.ee.internal.POA.POAORB$1.evaluate(POAORB.java:128) at com.sun.corba.ee.internal.core.Future.evaluate(Future.java:21) at com.sun.corba.ee.internal.corba.ORB.resolveInitialReference(ORB.java:2421) at com.sun.corba.ee.internal.corba.ORB.resolve_initial_references(ORB.java:2356) at com.sun.enterprise.server.J2EEServer.run(J2EEServer.java:193) at com.sun.enterprise.server.J2EEServer.main(J2EEServer.java:913) java.lang.RuntimeException: Unable to create ORB. Possible causes include TCP/IP ports in use by another process at com.sun.enterprise.server.J2EEServer.run(J2EEServer.java:203) at com.sun.enterprise.server.J2EEServer.main(J2EEServer.java:913) java.lang.RuntimeException: Unable to create ORB. Possible causes include TCP/IP ports in use by another process at com.sun.enterprise.server.J2EEServer.run(J2EEServer.java:203) at com.sun.enterprise.server.J2EEServer.main(J2EEServer.java:913) java.lang.RuntimeException: Unable to create ORB. Possible causes include TCP/IP ports in use by another process at com.sun.enterprise.server.J2EEServer.run(J2EEServer.java:350) at com.sun.enterprise.server.J2EEServer.main(J2EEServer.java:913) J2EE server reported the following error: Unable to create ORB. Possible causes include TCP/IP ports in use by another process Error executing J2EE server ... tried again, waiting for second attempt in a row ... it failed again: org.omg.CORBA.INTERNAL: minor code: 1398079697 completed: No at com.sun.corba.ee.internal.iiop.GIOPImpl.createListener(GIOPImpl.java:256) at com.sun.corba.ee.internal.iiop.GIOPImpl.getEndpoint(GIOPImpl.java:205) at com.sun.corba.ee.internal.iiop.GIOPImpl.initEndpoints(GIOPImpl.java:140) at com.sun.corba.ee.internal.POA.POAORB.getServerEndpoint(POAORB.java:488) at com.sun.corba.ee.internal.POA.POAImpl.pre_initialize(POAImpl.java:154) at com.sun.corba.ee.internal.POA.POAImpl.(POAImpl.java:112) at com.sun.corba.ee.internal.POA.POAORB.makeRootPOA(POAORB.java:110) at com.sun.corba.ee.internal.POA.POAORB$1.evaluate(POAORB.java:128) at com.sun.corba.ee.internal.core.Future.evaluate(Future.java:21) at com.sun.corba.ee.internal.corba.ORB.resolveInitialReference(ORB.java:2421) at com.sun.corba.ee.internal.corba.ORB.resolve_initial_references(ORB.java:2356) at com.sun.enterprise.server.J2EEServer.run(J2EEServer.java:193) at com.sun.enterprise.server.J2EEServer.main(J2EEServer.java:913) java.lang.RuntimeException: Unable to create ORB. Possible causes include TCP/IP ports in use by another process at com.sun.enterprise.server.J2EEServer.run(J2EEServer.java:203) at com.sun.enterprise.server.J2EEServer.main(J2EEServer.java:913) java.lang.RuntimeException: Unable to create ORB. Possible causes include TCP/IP ports in use by another process at com.sun.enterprise.server.J2EEServer.run(J2EEServer.java:203) at com.sun.enterprise.server.J2EEServer.main(J2EEServer.java:913) java.lang.RuntimeException: Unable to create ORB. Possible causes include TCP/IP ports in use by another process at com.sun.enterprise.server.J2EEServer.run(J2EEServer.java:350) at com.sun.enterprise.server.J2EEServer.main(J2EEServer.java:913) J2EE server reported the following error: Unable to create ORB. Possible causes include TCP/IP ports in use by another process Error executing J2EE server ... The problem seems repeatable this time, so I'll try cleanup and hope it doesn't destroyed my deployed applications again like it did last time ... it says: This will delete all deployed applications and clean all log files Do you want to proceed (y/n) I'll say no this time, but the message seems like it'll simply abort the entire cleanup process leaving me unable to restart the server at all. The cleanup script was not run So how the fuck can I recover from this without losing my deployed application?? I tried restarting j2ee a third time and it bombed again as the previous two times. Two more things to try before shutting down the entire operating system: Look in /var/lock/ to see if anything suspicious there, nope. Shut down cloudscape before restarting j2ee, at 11:05 ... it's down at 11:07, trying to restart j2ee again now ... at 11:09 it has done the Binding DataSource stuff and is now doing the Starting JMS service... finally back up at 11:15 which is 33 minutes after I first tried to bring it back up, restarting cloudscape as I typed that ... it seems to be back up at 11:18, although it doesn't print any confirmation message after "[RmiJdbc] RmiJdbcServer bound in rmi registry", total time spent 36 minutes for getting j2ee up and cloudscape down and up. Now copying recently-changed class files to system area ... done at 11:22, clicking on Web link to jump2 which calls servlet, waiting for JSP to compile ... done and offered me cookie at 11:26, accepted, servlet exception: java.util.MissingResourceException: DispatcherServlet.doGetAdapter caught excep tion from doGetMain:java.util.MissingResourceException: Can't find bundle for b ase name lab3, locale en_US OK, I think I know what that is. I put the lab3.properties file on that directory and never removed it, but a few days ago when I couldn't start j2ee and I did cleanup and it destroyed all my deployed applications and I had to spend a couple hours re-deploying the one application I am adapting for Lab#3, it destroyed the directory with that one properties file and all my copies of class files, and I copied all the class files back onto it to resume development of code, but since I wasn't using any database stuff or Lab#3 configuration stuff I didn't need the copy of the properties file on that system directory, until now. Copying that one file now ... done at 11:33, refreshing Web page ... crap, j2ee server has cached the fact that it already looked for that file and found it missing, so I need to stop and restart the server again, at 11:37 I've issued command to shut it down ... it's down, starting it back up ... up at 11:55, resubmitting POST data, waiting for it to respond ... it took until 12:11 for cloudscape to respond but that case worked, login rejected, no such user. I immediately went back to login form and submitted a valid user with wrong password, waiting for it to respond ... 12:25 it finally responded correctly, now I submitted valid user with correct password ... at 12:33 it responded correctly that the user was already logged in, so I immediately submitted valid user with valid password and not logged in, waiting for response ... at 12:43 it responded that user already logged in, so did I goof?? Looked around, found user that I'm *sure* isn't logged in already, submitted that login at 12:49, waiting for response ... at 12:55 it bombed: A Servlet Exception Has Occurred java.util.MissingResourceException: DispatcherServlet.doGetAdapter caught excep tion from doGetMain:java.util.MissingResourceException: Query (SELECT Seq from Users WHERE Name='Victor') failed, stub: java.sql.SQLException: no such object in table at DispatcherServlet.doGetAdapter(DispatcherServlet.java:94) at DispatcherServlet.doGet(DispatcherServlet.java:46) There most definitely was such a record when I checked earlier today, and I haven't deleted it, so something is going wrong for no apparent reason. Let me try the same login attempt from the test rig ... login succeeded like it should have the first time. Maybe cloudscape was in a funny state for a moment when I tried from the servlet? No way to know. Because running the J2EE server was such a royal pain, yet running only the NULL test rig (viewing the page layout in s-expression and/or XML and/or raw HTML text notation) tends to miss some things that would be obvious in a full visual presentation, I decided it was time to try writing software to convert from my UI-independent structured screen-layout objects to the corresponding swing GUI JComponents and show them on-screen. I started work on the conversion, with simple JFrame to show the result, but had a problem that stumped me for an hour: How to avoid buttons expanding to fill all available space within the contentpane of the JFrame? I had to leave to go to dentist appointment with the issue unresolved. In the evening I randomly tried putting the component inside an extra layer of JPanel inside the JFrame.contentpane, and that fixed the problem. So I then proceeded to implement a few of the components (button directly, textfield and passwordfield translated from UI-independent structures). Now it's bedtime at 22:41.