Google App Engine (Java) App stopped working (all versions) -

all versions of app stopped working.

similar unable load several versions of appengine project , seeing harddeadlineexceeded exceptions

from logs:

failed startup of context com.google.apphosting.utils.jetty.runtimeappenginewebappcontext@7881db{/,/base/data/home/apps/[my_app]/946.357282374521459350} com.google.apphosting.api.deadlineexceededexception: request (d505d1e88148a047) started @ 2012/03/08 04:17:00.300 utc , still executing @ 2012/03/08 04:18:00.917 utc.     @ java.util.zip.zipfile.read(native method)     @ java.util.zip.zipfile.access$1200(zipfile.java:57)     @ java.util.zip.zipfile$zipfileinputstream.read(zipfile.java:476)     @ java.util.zip.zipfile$1.fill(zipfile.java:259)     @ java.util.zip.inflaterinputstream.read(inflaterinputstream.java:158)     @ java.io.datainputstream.readfully(datainputstream.java:195)     @ java.util.jar.jarfile.hasclasspathattribute(jarfile.java:482)     @ java.util.jar.javautiljaraccessimpl.jarfilehasclasspathattribute(javautiljaraccessimpl.java:33)     @ sun.misc.urlclasspath$jarloader.getclasspath(urlclasspath.java:1176)     @ sun.misc.urlclasspath.getloader(urlclasspath.java:374)     @ sun.misc.urlclasspath.findresource(urlclasspath.java:201)     @ java.net.urlclassloader$2.run(urlclassloader.java:379)     @ java.security.accesscontroller.doprivileged(native method)     @ java.net.urlclassloader.findresource(urlclassloader.java:376)     @ com.google.apphosting.runtime.security.userclassloader.findresource(userclassloader.java:723)     @ java.lang.classloader.getresource(classloader.java:977)     @ org.mortbay.resource.resource.newsystemresource(resource.java:203)     @ org.mortbay.jetty.webapp.webxmlconfiguration.configuredefaults(webxmlconfiguration.java:159)     @ org.mortbay.jetty.webapp.webappcontext.startcontext(webappcontext.java:1230)     @ org.mortbay.jetty.handler.contexthandler.dostart(contexthandler.java:517)     @ org.mortbay.jetty.webapp.webappcontext.dostart(webappcontext.java:467)     @ org.mortbay.component.abstractlifecycle.start(abstractlifecycle.java:50)     @ com.google.apphosting.runtime.jetty.appversionhandlermap.createhandler(appversionhandlermap.java:202)     @ com.google.apphosting.runtime.jetty.appversionhandlermap.gethandler(appversionhandlermap.java:171)     @ com.google.apphosting.runtime.jetty.jettyservletengineadapter.servicerequest(jettyservletengineadapter.java:123)     @ com.google.apphosting.runtime.javaruntime$requestrunnable.run(javaruntime.java:422)     @ com.google.tracing.tracecontext$tracecontextrunnable.runincontext(tracecontext.java:449)     @ com.google.tracing.tracecontext$tracecontextrunnable$1.run(tracecontext.java:455)     @ com.google.tracing.tracecontext.runincontext(tracecontext.java:695)     @ com.google.tracing.tracecontext$abstracttracecontextcallback.runininheritedcontextnounref(tracecontext.java:333)     @ com.google.tracing.tracecontext$abstracttracecontextcallback.runininheritedcontext(tracecontext.java:325)     @ com.google.tracing.tracecontext$tracecontextrunnable.run(tracecontext.java:453)     @ com.google.apphosting.runtime.threadgrouppool$poolentry.run(threadgrouppool.java:251)     @ java.lang.thread.run(thread.java:679) c 2012-03-07 23:18:01.030 uncaught exception servlet javax.servlet.unavailableexception: initialization failed.     @ com.google.apphosting.runtime.jetty.appversionhandlermap.createhandler(appversionhandlermap.java:211)     @ com.google.apphosting.runtime.jetty.appversionhandlermap.gethandler(appversionhandlermap.java:171)     @ com.google.apphosting.runtime.jetty.jettyservletengineadapter.servicerequest(jettyservletengineadapter.java:123)     @ com.google.apphosting.runtime.javaruntime$requestrunnable.run(javaruntime.java:422)     @ com.google.tracing.tracecontext$tracecontextrunnable.runincontext(tracecontext.java:449)     @ com.google.tracing.tracecontext$tracecontextrunnable$1.run(tracecontext.java:455)     @ com.google.tracing.tracecontext.runincontext(tracecontext.java:695)     @ com.google.tracing.tracecontext$abstracttracecontextcallback.runininheritedcontextnounref(tracecontext.java:333)     @ com.google.tracing.tracecontext$abstracttracecontextcallback.runininheritedcontext(tracecontext.java:325)     @ com.google.tracing.tracecontext$tracecontextrunnable.run(tracecontext.java:453)     @ com.google.apphosting.runtime.threadgrouppool$poolentry.run(threadgrouppool.java:251)     @ java.lang.thread.run(thread.java:679)   also, seen in browser:   uncaught exception servlet javax.servlet.unavailableexception: initialization failed.     @ com.google.apphosting.runtime.jetty.appversionhandlermap.createhandler(appversionhandlermap.java:211)     @ com.google.apphosting.runtime.jetty.appversionhandlermap.gethandler(appversionhandlermap.java:171)     @ com.google.apphosting.runtime.jetty.jettyservletengineadapter.servicerequest(jettyservletengineadapter.java:123)     @ com.google.apphosting.runtime.javaruntime$requestrunnable.run(javaruntime.java:422)     @ com.google.tracing.tracecontext$tracecontextrunnable.runincontext(tracecontext.java:449)     @ com.google.tracing.tracecontext$tracecontextrunnable$1.run(tracecontext.java:455)     @ com.google.tracing.tracecontext.runincontext(tracecontext.java:695)     @ com.google.tracing.tracecontext$abstracttracecontextcallback.runininheritedcontextnounref(tracecontext.java:333)     @ com.google.tracing.tracecontext$abstracttracecontextcallback.runininheritedcontext(tracecontext.java:325)     @ com.google.tracing.tracecontext$tracecontextrunnable.run(tracecontext.java:453)     @ com.google.apphosting.runtime.threadgrouppool$poolentry.run(threadgrouppool.java:251)     @ java.lang.thread.run(thread.java:679) 

another error in logs:

2012-03-08 12:28:26.248 com.google.inject.internal.util.$finalizablereferencequeue$systemloader loadfinalizer: not allowed access system class loader. 2012-03-08 12:28:27.143 com.google.inject.internal.util.$finalizablereferencequeue : failed start reference finalizer thread. reference cleanup occur when new references created. java.lang.reflect.invocationtargetexception @ com.google.appengine.runtime.request.process-58eed86e9936f59f(request.java) @ sun.reflect.nativemethodaccessorimpl.invoke0(native method) @ sun.reflect.nativemethodaccessorimpl.invoke(nativemethodaccessorimpl.java:57) @ sun.reflect.delegatingmethodaccessorimpl.invoke(delegatingmethodaccessorimpl.java:43) @ java.lang.reflect.method.invoke(method.java:43) @ com.google.inject.internal.util.$finalizablereferencequeue.(finalizablereferencequeue.java:124) @ com.google.inject.internal.util.$mapmaker$queueholder.(mapmaker.java:787) @ com.google.inject.internal.util.$mapmaker$weakentry.(mapmaker.java:946) @ com.google.inject.internal.util.$mapmaker$strength$1.newentry(mapmaker.java:312) @ com.google.inject.internal.util.$mapmaker$strategyimpl.newentry(mapmaker.java:498) @ com.google.inject.internal.util.$mapmaker$strategyimpl.newentry(mapmaker.java:419) @ com.google.inject.internal.util.$customconcurrenthashmap$computingimpl.get(customconcurrenthashmap.java:2029) @ com.google.inject.internal.annotations$annotationchecker.hasannotations(annotations.java:116) @ com.google.inject.internal.annotations.isbindingannotation(annotations.java:180) @ com.google.inject.internal.annotations.findbindingannotation(annotations.java:161) @ com.google.inject.internal.providermethodsmodule.getkey(providermethodsmodule.java:129) @ com.google.inject.internal.providermethodsmodule.createprovidermethod(providermethodsmodule.java:116) @ com.google.inject.internal.providermethodsmodule.getprovidermethods(providermethodsmodule.java:83) @ com.google.inject.internal.providermethodsmodule.configure(providermethodsmodule.java:73) @ com.google.inject.spi.elements$recordingbinder.install(elements.java:223) @ com.google.inject.spi.elements$recordingbinder.install(elements.java:232) @ com.google.inject.spi.elements.getelements(elements.java:101) @ com.google.inject.internal.injectorshell$builder.build(injectorshell.java:133) @ com.google.inject.internal.internalinjectorcreator.build(internalinjectorcreator.java:103) @ com.google.inject.guice.createinjector(guice.java:95) @ com.google.inject.guice.createinjector(guice.java:72) @ com.google.inject.guice.createinjector(guice.java:62) @ [myapp].server.guice.guiceinjectorfactory$guiceberry.getinjector(guiceinjectorfactory.java:20) @ [myapp].server.guice.guiceinjectorfactory.getinjector(guiceinjectorfactory.java:29) @ com.google.inject.servlet.guiceservletcontextlistener.contextinitialized(guiceservletcontextlistener.java:45) @ org.mortbay.jetty.handler.contexthandler.startcontext(contexthandler.java:548) @ org.mortbay.jetty.servlet.context.startcontext(context.java:136) @ org.mortbay.jetty.webapp.webappcontext.startcontext(webappcontext.java:1250) @ org.mortbay.jetty.handler.contexthandler.dostart(contexthandler.java:517) @ org.mortbay.jetty.webapp.webappcontext.dostart(webappcontext.java:467) @ org.mortbay.component.abstractlifecycle.start(abstractlifecycle.java:50) @ com.google.tracing.tracecontext$tracecontextrunnable.runincontext(tracecontext.java:449) @ com.google.tracing.tracecontext$tracecontextrunnable$1.run(tracecontext.java:455) @ com.google.tracing.tracecontext.runincontext(tracecontext.java:695) @ com.google.tracing.tracecontext$abstracttracecontextcallback.runininheritedcontextnounref(tracecontext.java:333) @ com.google.tracing.tracecontext$abstracttracecontextcallback.runininheritedcontext(tracecontext.java:325) @ com.google.tracing.tracecontext$tracecontextrunnable.run(tracecontext.java:453) @ java.lang.thread.run(thread.java:679) caused by: java.security.accesscontrolexception: access denied (java.lang.runtimepermission modifythreadgroup) @ java.security.accesscontrolcontext.checkpermission(accesscontrolcontext.java:355) @ java.security.accesscontroller.checkpermission(accesscontroller.java:567) @ java.lang.securitymanager.checkpermission(securitymanager.java:549) @ java.lang.threadgroup.checkaccess(threadgroup.java:315) @ java.lang.thread.init(thread.java:353) @ java.lang.thread.(thread.java:479) @ com.google.inject.internal.util.$finalizer.(finalizer.java:92) @ com.google.inject.internal.util.$finalizer.startfinalizer(finalizer.java:81) ... 42 more c 2012-03-08 12:28:59.290 uncaught exception servlet com.google.apphosting.runtime.harddeadlineexceedederror: request (58eed86e9936f59f) started @ 2012/03/08 17:27:56.854 utc , still executing @ 2012/03/08 17:28:59.245 utc. @ java.io.fileinputstream.readbytes(native method) @ java.io.fileinputstream.read(fileinputstream.java:231) @ java.io.bufferedinputstream.fill(bufferedinputstream.java:235) @ java.io.bufferedinputstream.read1(bufferedinputstream.java:275) @ java.io.bufferedinputstream.read(bufferedinputstream.java:334) @ com.google.inject.internal.asm.$classreader.a(unknown source) @ com.google.inject.internal.asm.$classreader.(unknown source) @ com.google.inject.internal.util.$linenumbers.(linenumbers.java:62) @ com.google.inject.internal.util.$stacktraceelements$1.apply(stacktraceelements.java:36) @ com.google.inject.internal.util.$stacktraceelements$1.apply(stacktraceelements.java:33) @ com.google.inject.internal.util.$mapmaker$strategyimpl.compute(mapmaker.java:549) @ com.google.inject.internal.util.$mapmaker$strategyimpl.compute(mapmaker.java:419) @ com.google.inject.internal.util.$customconcurrenthashmap$computingimpl.get(customconcurrenthashmap.java:2041) @ com.google.inject.internal.util.$stacktraceelements.formember(stacktraceelements.java:53) @ com.google.inject.internal.errors.formatsource(errors.java:690) @ com.google.inject.internal.errors.format(errors.java:555) @ com.google.inject.provisionexception.getmessage(provisionexception.java:59) @ java.lang.throwable.getlocalizedmessage(throwable.java:284) @ java.lang.throwable.tostring(throwable.java:360) @ java.lang.string.valueof(string.java:2838) @ java.lang.stringbuffer.append(stringbuffer.java:236) @ org.mortbay.component.abstractlifecycle.setfailed(abstractlifecycle.java:183) @ org.mortbay.component.abstractlifecycle.start(abstractlifecycle.java:56) @ org.mortbay.jetty.servlet.servlethandler.initialize(servlethandler.java:662) @ org.mortbay.jetty.servlet.context.startcontext(context.java:140) @ org.mortbay.jetty.webapp.webappcontext.startcontext(webappcontext.java:1250) @ org.mortbay.jetty.handler.contexthandler.dostart(contexthandler.java:517) @ org.mortbay.jetty.webapp.webappcontext.dostart(webappcontext.java:467) @ org.mortbay.component.abstractlifecycle.start(abstractlifecycle.java:50) @ com.google.apphosting.runtime.jetty.appversionhandlermap.createhandler(appversionhandlermap.java:202) @ com.google.apphosting.runtime.jetty.appversionhandlermap.gethandler(appversionhandlermap.java:171) @ com.google.apphosting.runtime.jetty.jettyservletengineadapter.servicerequest(jettyservletengineadapter.java:123) @ com.google.apphosting.runtime.javaruntime$requestrunnable.run(javaruntime.java:422) @ com.google.tracing.tracecontext$tracecontextrunnable.runincontext(tracecontext.java:449) @ com.google.tracing.tracecontext$tracecontextrunnable$1.run(tracecontext.java:455) @ com.google.tracing.tracecontext.runincontext(tracecontext.java:695) @ com.google.tracing.tracecontext$abstracttracecontextcallback.runininheritedcontextnounref(tracecontext.java:333) @ com.google.tracing.tracecontext$abstracttracecontextcallback.runininheritedcontext(tracecontext.java:325) @ com.google.tracing.tracecontext$tracecontextrunnable.run(tracecontext.java:453) @ com.google.apphosting.runtime.threadgrouppool$poolentry.run(threadgrouppool.java:251) @ java.lang.thread.run(thread.java:679) 2012-03-08 12:28:59.323 request caused new process started application, , caused application code loaded first time. request may take longer , use more cpu typical request application. w 2012-03-08 12:28:59.323 problem encountered process handled request, causing exit. cause new process used next request application. if see message frequently, may throwing exceptions during initialization of application. (error code 104)

this caused application taking long start up. while application still initializing itself, exceeds hard per-request time limit (30 seconds, in practice apparently more 60 seconds).

i've seen these well, , think it's combination of application being expensive load, , wrong on appengine's side; appears take longer load classes should. either try slimming down application, or wait google fix underlying issue.

the log messages com.google.inject.internal.util.$finalizablereferencequeue$systemloader loadfinalizer red herring, normal when using guice on appengine , shouldn't harm.