Upgrades to versions >= 16.1.1 do not apply the ScriptingService configuration introduced in 16.1.1 (#1034)

[FedericoBrignola]

Describe the bug
Commit #1034 introduced a new script type that is not added to the configuration data store during the upgrade process.
As a result, instances upgraded to OpenAM 16.1.1 and above are left in an inconsistent state.

To Reproduce
Steps to reproduce the behavior:

1. Start a new OpenDJ instance:

podman run --rm -ti -p 1389:1389 -e 'ADD_BASE_ENTRY=--addBaseEntry' -e 'BASE_DN=dc=openam,dc=openidentityplatform,dc=org' opendj:latest

2. Install OpenAM 16.0.6 using the new OpenDJ instance:
   • Host: host.containers.internal
   • Port: 1389
   • Password: password
3. Upgrade the installation to OpenAM 16.1.1
   • The upgrade process completes successfully and reports no errors.
4. Navigate to Configure → Global Services.
   The page is empty and the following API call returns a 500 Internal Server Error:
   http://localhost:8080/openam/json/global-config/services?_action=nextdescendents
   See the stack trace below.

Expected behavior
The upgrade process should add the new script configuration under ou=ScriptingService, resulting in the same configuration structure as a fresh OpenAM 16.1.1 installation.

Screenshots
ScriptingService configuration of a new OpenAM 16.0.6 installation (remains unchanged after upgrading to 16.1.1):

ScriptingService configuration of a new OpenAM 16.1.1 installation. (Note the additional script configuration):

Additional context
Stack trace from debug/CoreSystem

ERROR: A runtime exception occurred during the CREST request handling
java.lang.IllegalStateException: Exception from invocation expected to be handled by promise
	at org.forgerock.json.resource.AnnotatedMethod.invoke(AnnotatedMethod.java:100)
	at org.forgerock.json.resource.AnnotatedMethod.invoke(AnnotatedMethod.java:64)
	at org.forgerock.json.resource.AnnotatedSingletonHandler.handleRead(AnnotatedSingletonHandler.java:63)
	at org.forgerock.json.resource.Router.handleRead(Router.java:328)
	at org.forgerock.json.resource.FilterChain$Cursor.handleRead(FilterChain.java:113)
	at org.forgerock.authz.filter.crest.AuthorizationFilters$AuthorizationFilter$6.apply(AuthorizationFilters.java:378)
	at org.forgerock.authz.filter.crest.AuthorizationFilters$AuthorizationFilter$6.apply(AuthorizationFilters.java:374)
	at org.forgerock.util.promise.Promises$CompletedPromise.thenAsync(Promises.java:255)
	at org.forgerock.util.promise.Promises$CompletedPromise.thenAsync(Promises.java:244)
	at org.forgerock.util.promise.Promises$CompletedPromise.thenAsync(Promises.java:223)
	at org.forgerock.authz.filter.crest.AuthorizationFilters$AuthorizationFilter.filterRead(AuthorizationFilters.java:374)
	at org.forgerock.json.resource.FilterChain$Cursor.handleRead(FilterChain.java:111)
	at org.forgerock.json.resource.FilterChain.handleRead(FilterChain.java:260)
	at org.forgerock.json.resource.Router.handleRead(Router.java:328)
	at org.forgerock.openam.core.rest.sms.tree.SmsRouteTree.handleRead(SmsRouteTree.java:400)
	at org.forgerock.openam.core.rest.sms.tree.SmsRouteTree.getSingletonInstance(SmsRouteTree.java:350)
	at org.forgerock.openam.core.rest.sms.tree.SmsRouteTree.readInstances(SmsRouteTree.java:340)
	at org.forgerock.openam.core.rest.sms.tree.SmsRouteTree.handleAction(SmsRouteTree.java:269)
	at org.forgerock.json.resource.Router.handleAction(Router.java:245)
	at org.forgerock.openam.core.rest.sms.tree.SmsRouteTree.handleAction(SmsRouteTree.java:300)
	at org.forgerock.openam.core.rest.sms.SmsRequestHandler.handleAction(SmsRequestHandler.java:648)
	at org.forgerock.json.resource.Router.handleAction(Router.java:245)
	at org.forgerock.json.resource.FilterChain$Cursor.handleAction(FilterChain.java:63)
	at org.forgerock.openam.rest.fluent.AuditFilter.filterAction(AuditFilter.java:88)
	at org.forgerock.openam.rest.fluent.AuditFilterWrapper.filterAction(AuditFilterWrapper.java:60)
	at org.forgerock.json.resource.FilterChain$Cursor.handleAction(FilterChain.java:61)
	at org.forgerock.openam.rest.fluent.CrestLoggingFilter.filterAction(CrestLoggingFilter.java:75)
	at org.forgerock.json.resource.FilterChain$Cursor.handleAction(FilterChain.java:61)
	at org.forgerock.openam.rest.ContextFilter.filterAction(ContextFilter.java:58)
	at org.forgerock.json.resource.FilterChain$Cursor.handleAction(FilterChain.java:61)
	at org.forgerock.openam.rest.AuthenticationEnforcer.filterAction(AuthenticationEnforcer.java:137)
	at org.forgerock.json.resource.FilterChain$Cursor.handleAction(FilterChain.java:61)
	at org.forgerock.json.resource.FilterChain.handleAction(FilterChain.java:230)
	at org.forgerock.json.resource.InternalConnection.actionAsync(InternalConnection.java:37)
	at org.forgerock.json.resource.http.RequestRunner.visitActionRequest(RequestRunner.java:149)
	at org.forgerock.json.resource.http.RequestRunner.visitActionRequest(RequestRunner.java:95)
	at org.forgerock.json.resource.Requests$ActionRequestImpl.accept(Requests.java:185)
	at org.forgerock.json.resource.http.RequestRunner.handleResult(RequestRunner.java:141)
	at org.forgerock.json.resource.http.HttpAdapter$1.apply(HttpAdapter.java:711)
	at org.forgerock.json.resource.http.HttpAdapter$1.apply(HttpAdapter.java:708)
	at org.forgerock.util.promise.Promises$CompletedPromise.thenAsync(Promises.java:255)
	at org.forgerock.util.promise.Promises$CompletedPromise.thenAsync(Promises.java:244)
	at org.forgerock.json.resource.http.HttpAdapter.doRequest(HttpAdapter.java:708)
	at org.forgerock.json.resource.http.HttpAdapter.doAction(HttpAdapter.java:612)
	at org.forgerock.json.resource.http.HttpAdapter.handle(HttpAdapter.java:276)
	at org.forgerock.http.handler.Handlers$HandlerDescribableAsDescribableHandler.handle(Handlers.java:154)
	at org.forgerock.http.filter.OptionsFilter.filter(OptionsFilter.java:77)
	at org.forgerock.http.handler.Handlers$1.handle(Handlers.java:61)
	at org.forgerock.http.routing.Router.handle(Router.java:100)
	at org.forgerock.http.routing.Router.handle(Router.java:100)
	at org.forgerock.http.routing.ResourceApiVersionRoutingFilter.filter(ResourceApiVersionRoutingFilter.java:64)
	at org.forgerock.http.handler.Handlers$1.handle(Handlers.java:61)
	at org.forgerock.caf.authentication.framework.AuthenticationFramework.grantAccess(AuthenticationFramework.java:220)
	at org.forgerock.caf.authentication.framework.AuthenticationFramework$3.apply(AuthenticationFramework.java:212)
	at org.forgerock.caf.authentication.framework.AuthenticationFramework$3.apply(AuthenticationFramework.java:205)
	at org.forgerock.util.promise.Promises$CompletedPromise.thenAsync(Promises.java:255)
	at org.forgerock.util.promise.Promises$CompletedPromise.thenAsync(Promises.java:244)
	at org.forgerock.caf.authentication.framework.AuthenticationFramework.validateRequest(AuthenticationFramework.java:170)
	at org.forgerock.caf.authentication.framework.AuthenticationFramework$1.apply(AuthenticationFramework.java:155)
	at org.forgerock.caf.authentication.framework.AuthenticationFramework$1.apply(AuthenticationFramework.java:152)
	at org.forgerock.util.promise.PromiseImpl$7.handleStateChange(PromiseImpl.java:485)
	at org.forgerock.util.promise.PromiseImpl.handleCompletion(PromiseImpl.java:567)
	at org.forgerock.util.promise.PromiseImpl.addOrFireListener(PromiseImpl.java:555)
	at org.forgerock.util.promise.PromiseImpl.thenAsync(PromiseImpl.java:477)
	at org.forgerock.util.promise.PromiseImpl.thenAsync(PromiseImpl.java:468)
	at org.forgerock.caf.authentication.framework.AuthenticationFramework.processMessage(AuthenticationFramework.java:147)
	at org.forgerock.caf.authentication.framework.AuthenticationFilter.filter(AuthenticationFilter.java:97)
	at org.forgerock.http.handler.Handlers$1.handle(Handlers.java:61)
	at org.forgerock.openam.http.GuiceHandler.handle(GuiceHandler.java:59)
	at org.forgerock.openam.http.HttpRoute$6.handle(HttpRoute.java:215)
	at org.forgerock.http.routing.Router.handle(Router.java:100)
	at org.forgerock.http.swagger.OpenApiRequestFilter.filter(OpenApiRequestFilter.java:69)
	at org.forgerock.http.handler.Handlers$1.handle(Handlers.java:61)
	at org.forgerock.openam.http.ApiDescriptorFilter.filter(ApiDescriptorFilter.java:122)
	at org.forgerock.http.handler.Handlers$1.handle(Handlers.java:61)
	at org.forgerock.openam.http.OpenAMHttpApplication$1.filter(OpenAMHttpApplication.java:71)
	at org.forgerock.http.handler.Handlers$1.handle(Handlers.java:61)
	at org.forgerock.http.filter.TransactionIdInboundFilter.filter(TransactionIdInboundFilter.java:60)
	at org.forgerock.http.handler.Handlers$1.handle(Handlers.java:61)
	at org.forgerock.http.servlet.HttpFrameworkServlet.service(HttpFrameworkServlet.java:255)
	at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:658)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:197)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:142)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:57)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:166)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:142)
	at org.forgerock.openam.cors.CORSFilter.doFilter(CORSFilter.java:65)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:166)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:142)
	at org.forgerock.openam.validation.ResponseValidationFilter.doFilter(ResponseValidationFilter.java:45)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:166)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:142)
	at org.forgerock.openam.headers.SetHeadersFilter.doFilter(SetHeadersFilter.java:87)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:166)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:142)
	at com.sun.identity.setup.AMSetupFilter.doFilter(AMSetupFilter.java:112)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:166)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:142)
	at org.forgerock.openam.audit.context.AuditContextFilter.doFilter(AuditContextFilter.java:52)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:166)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:142)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:165)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:88)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:535)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:115)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:86)
	at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:782)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:71)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:348)
	at org.apache.coyote.http2.StreamProcessor.service(StreamProcessor.java:476)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:71)
	at org.apache.coyote.http2.StreamProcessor.process(StreamProcessor.java:102)
	at org.apache.coyote.http2.StreamRunnable.run(StreamRunnable.java:35)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:976)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:494)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:80)
	at java.base/java.lang.Thread.run(Thread.java:1474)
Caused by: java.lang.reflect.InvocationTargetException
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:119)
	at java.base/java.lang.reflect.Method.invoke(Method.java:565)
	at org.forgerock.json.resource.AnnotatedMethod.invoke(AnnotatedMethod.java:96)
	... 116 more
Caused by: org.forgerock.json.JsonException: Unable to validate attributes
	at org.forgerock.openam.core.rest.sms.SmsJsonConverter.toJson(SmsJsonConverter.java:199)
	at org.forgerock.openam.core.rest.sms.SmsJsonConverter.toJson(SmsJsonConverter.java:168)
	at org.forgerock.openam.core.rest.sms.SmsJsonConverter.toJson(SmsJsonConverter.java:142)
	at org.forgerock.openam.core.rest.sms.SmsGlobalSingletonProvider.addOrganisationAttributes(SmsGlobalSingletonProvider.java:146)
	at org.forgerock.openam.core.rest.sms.SmsResourceProvider.getJsonValue(SmsResourceProvider.java:452)
	at org.forgerock.openam.core.rest.sms.SmsResourceProvider.getJsonValue(SmsResourceProvider.java:440)
	at org.forgerock.openam.core.rest.sms.SmsSingletonProvider.handleRead(SmsSingletonProvider.java:153)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	... 118 more
Caused by: Message:Data validation failed for the attribute, forgerock-oauth2-provider-access-token-modification-script

	at com.sun.identity.sm.AttributeValidator.validate(AttributeValidator.java:410)
	at com.sun.identity.sm.AttributeValidator.validate(AttributeValidator.java:372)
	at com.sun.identity.sm.ServiceSchemaImpl.validateAttrValues(ServiceSchemaImpl.java:594)
	at com.sun.identity.sm.ServiceSchemaImpl.validateAttributes(ServiceSchemaImpl.java:345)
	at com.sun.identity.sm.ServiceSchemaImpl.validateAttributes(ServiceSchemaImpl.java:314)
	at com.sun.identity.sm.ServiceSchema.validateAttributes(ServiceSchema.java:749)
	at org.forgerock.openam.core.rest.sms.SmsJsonConverter.toJson(SmsJsonConverter.java:192)
	... 125 more

Vote to raise the priority 🖐️

[vharseko]

Thanks for the detailed report and the reproduction steps — confirmed.

Root cause: the SMS only processes the <Configuration> section of a service XML when the whole service is registered as new, and the upgrade framework diffs <Schema> only. So the OAUTH2_ACCESS_TOKEN_MODIFICATION script context and its default global script (d22f9a0c-426a-4466-b95e-d0f125b0d5fa), introduced in the <GlobalConfiguration> section of scripting.xml by #1034, are never created when upgrading an instance that already had the Scripting Service. At the same time the new OAuth2 Provider attribute forgerock-oauth2-provider-access-token-modification-script is added to the schema with its default pointing at the now-missing script — its single_choice validation then fails and breaks the whole global-config/services?_action=nextdescendents call, which is why the Global Services page comes up empty.

Fix: #1104 adds an upgrade step that creates any global Scripting Service sub-configurations present in the bundled scripting.xml but missing from the configuration store. Because it works by diffing, upgrading an already-affected 16.1.1/16.1.2 installation to a release containing the fix will also repair it automatically — no manual cleanup needed.

Workarounds for already-upgraded instances in the meantime:

Option 1 — quick relief (removes the 500, leaves the new feature disabled): reset the dangling default so it no longer references the missing script:

ssoadm set-attr-defs -u amadmin -f /path/to/pwd.txt \
  -s OAuth2Provider -t organization \
  -a forgerock-oauth2-provider-access-token-modification-script=[Empty]

Option 2 — full repair (restores the same state as a fresh 16.1.1+ install): recreate the two missing sub-configurations. The most faithful way is to export the corresponding entries from a fresh 16.1.1+ installation and import them into the upgraded config store:

ou=OAUTH2_ACCESS_TOKEN_MODIFICATION,ou=default,ou=GlobalConfig,ou=1.0,ou=ScriptingService,ou=services,<base-dn>
  (plus its ou=engineConfiguration child)
ou=d22f9a0c-426a-4466-b95e-d0f125b0d5fa,ou=globalScripts,ou=default,ou=GlobalConfig,ou=1.0,ou=ScriptingService,ou=services,<base-dn>

Alternatively the same can be done with ssoadm create-sub-cfg -s ScriptingService -g (-b OAUTH2_ACCESS_TOKEN_MODIFICATION -i scriptContext, -b OAUTH2_ACCESS_TOKEN_MODIFICATION/engineConfiguration -i engineConfiguration, -b globalScripts/d22f9a0c-426a-4466-b95e-d0f125b0d5fa -i globalScript), taking the attribute values (engine whitelist, script body) from the OAUTH2_ACCESS_TOKEN_MODIFICATION section of scripting.xml and access-token-modification.groovy.

Note that either way OAuth2 token issuance itself is unaffected by this bug: when the referenced script is missing, tokens are issued unmodified.

[joelcharlebois]

Is there a way to implement the workarounds without the ssoadm tools (if not installed or available...)? For example, quick fix Option 1 using OpenDJ control panel and modify affected entries via the GUI (easiest for some people), or via ldif modify commands?

Will a future upgrade restore the reference to the script after implementing the Option 1 quick fix?

Quick Update 1: I have never configured OAuth2Provider in my instance, and I have no "GlobalConfig" nor "OrganizationalConfig" sub-entries under "ou=1.0,ou=OAuth2Provider,ou=services,base-dn". As such, I believe the workaround option 1 would fail anyway. Please advise about this, and the prior question.

Quick Update 2: In my case, for a quick fix via an LDAP browser, would modifying and setting the global default attribute schema value to empty for the "forgerock-oauth2-provider-access-token-modification-script" attribute achieve the same quick fix goal? Excerpt of the attribute schema below.

            <AttributeSchema cosQualifier="default"  i18nKey="a104ac"  isSearchable="no"  listOrder="natural"  name="forgerock-oauth2-provider-access-token-modification-script"  order="85"  propertiesViewBeanURL="../XUI/%23realms/{0}/scripts/edit/{1}"  resourceName="accessTokenModificationScript"  syntax="string"  type="single_choice"  uitype="scriptSelect" >
                    <ChoiceValues>
                        <ChoiceValuesClassName className="org.forgerock.openam.scripting.service.ScriptChoiceValues" >
                            <AttributeValuePair>
                                <Attribute name="ContextId" ></Attribute>
                                <Value>OAUTH2_ACCESS_TOKEN_MODIFICATION</Value>
                            </AttributeValuePair>
                        </ChoiceValuesClassName>
                    </ChoiceValues>
                    <DefaultValues>
                        <Value>d22f9a0c-426a-4466-b95e-d0f125b0d5fa</Value>
                    </DefaultValues>
                </AttributeSchema>

Thanks.

[maximthomas]

Hi @joelcharlebois,
You can use the ssoadm.jsp tool as an alternative to the ssoadm command line tool.
https://doc.openidentityplatform.org/openam/admin-guide/chap-admin-tools#openam-ssoadm-jsp-overview

[joelcharlebois]

Thank you @maximthomas. I will review that.

Would modifying the "sunServiceSchema" entry section I listed above, located at "ou=1.0,ou=OAuth2Provider,ou=services, base-dn", setting the listed default value to empty, would that achieve the same result as the Option 1 quick relief proposed much further above by @vharseko?

Thanks.