DXA 2.0 backwards compatibility testing

Our test environment is the regular DD4T training environment. It uses SDL Web 8.5 and has a working DD4T 2.2 application.

We will test the following version: UNKNOWN

DXA 2.0 source code can be found here: https://github.com/sdl/dxa-web-application-dotnet

This document describes the backwards compatibility as SDL envisions it: DD4T migration support.pdf.

Configuration options

Data source	Data format	Application	Test
Content service	DD4T 2	DD4T 2	If time permits: scenario 4
Model service	DD4T 2	DD4T 2	Scenario 1
Content service	DXA 2	DD4T 2	Not supported
Model service	DXA 2	DD4T 2	Scenario 2
Content service	DD4T 2	DXA 2	Not supported
Model service	DD4T 2	DXA 2	Scenario 3
Content service	DXA 2	DXA 2	Not supported
Model service	DXA 2	DXA 2	No (SDL will test this)

Test scenarios

We will test the following scenarios:

Number	Description	Result Java	Issues found / remarks
1	Install the model service. In the original application, replace the SDLWeb85 provider with DD4T.Providers.DxaModelService. Data is still DD4T 2		Filename 'index' is added automatically if the URL does not end with '.html' or another extension. This is not a bug but it is new functionality to DD4T users out there. I would like to understand how it works: why is 'index' appended and not something else? Is this configurable? The java docs have some minor issues, but are easily fixed (Auto Config, beans tags wrong, contructors wrong) The Java guys get back together this Thursday to further test this Missing PageTemplate, ComponentTemplate info, see issue 6
2	Upload the DXA templates and modify our templates. Republish all pages so that the broker contains DXA2-style JSON. Retest #1.		There are differences between the JSON that is returned by the model service when it is published by the DD4T 2 TBBs, or the same page published with the DXA R2 TBBs: In the converted JSON, rich text contains xhtml namespaces, in the native JSON it doesn't. DD4T contains a rich text resolver which removes these namespaces (this may not be a problem but it's a risk) - see issue 3 In the native JSON, the page metadata is an empty object (perhaps I need to configure something in the DXA R2 templates?) - see issue 4 Links in rich text fields are resolved by the model service. In DD4T this is done in the web application. Some customers have written custom code to resolve links to TCM URIs to URLs. This is no longer possible - see issue 5 When accessing a DXA page through the web application, we are getting an error with the ClientCacheKeyEnhancer. This is apparently a known issue with the CTP version, we will retest with the latest build.
3	Create a DXA 2 application, create some models and views, and test against DD4T2-style JSON through the model service	On Hold	.NET version: model conversion DD4T > R2 fails if the page template has no metadata. See issue 9. Other than that, model conversion works DD4T application with DXA Model Service Provider: TESTING IN PROGRESS With PT metadata, the model is converted and looks good. We still need to test if the DXA Web App really works with this data. Java Version: Stopped testing. We were using publication Id 8 to test DD4T 2 style Json against a DXA web app, but in that publication, the required _all.json and presumbably other required config wasnt published. I don't have access, so Quirijn Slings, would it be possible to publish that in publication 8? The new URLs to test with, are: Discovering the content service capability: http://ec2-52-214-122-181.eu-west-1.compute.amazonaws.com:8082/discovery.svc/Environment/ContentServiceCapability Model service - request a page published with DD4T JSON and converted to R2: http://ec2-52-214-122-181.eu-west-1.compute.amazonaws.com:8089/PageModel/tcm/2//dd4t.html?modelType=R2 Model service - request a page published with DXA R2 JSON and converted to DD4T: http://ec2-52-214-122-181.eu-west-1.compute.amazonaws.com:8089/PageModel/tcm/2//dxa.html?modelType=DD4T Note: the publication ID has changed to 2.
4	Replace DD4T Nuget packages with DLLs built from the DD4T copy inside DXA	NA
5.	Load the dd4t dependencies in the Java DXA from Maven		Java: confirmed latest code is in. The problem is that if the the dd4t 2 code needs to be fixed we need to update both code bases.

Test data

We used a single test page with 4 component presentations and copied that. In one version we used DD4T TBBs, in the other we used the DXA R2 TBBs. We set the Entity TBBs to include template related data and follow linked components to level 2 (just like the DD4T TBBs).

For each page, we captured the output of three times:

The result of the preview process in the CM
The output of the model service when called from the browser
The output of the model service provider

This gives us the following JSON:

DD4T output (preview): dd4t2-preview.json
DD4T output (retrieved from model service in the browser): dd4t2-from-modelservice.json
DD4T output (model service provider): dd4t2-from-modelserviceprovider.json
DXA output (preview): dxa2-preview.json
DXA output (model service): dxa2-from-modelservice.json
DXA output (model service provider): dxa2-from-modelserviceprovider.json

Issues (to be solved before the release)

Number

Prio

Issue

2 - ?

Medium

Links are resolved in the model service. This makes it impossible for customers to write their own linking logic, which many of them have done. Please make this a configurable feature and default to false.

DXA publishes rich text like this:

Hello <a href="tcm:90-123">world</a>

It resolves this into:

Hello <a href="/world">world</a>

Although this is great, DD4T has link resolvers in the application which expect xlink:href attributes. The rich text should therefore be resolved into this:

Hello <a href="/world" xlink:href="tcm:90-123" xmlns:xlink="http://www.w3.org/1999/xlink">world</a>

NOTE: latest version allows you to chose whether or not to resolve links AND remove namespaces. SDL is still discussing internally whether to split these to settings into:

Resolve links (true or false, if true adds href="URL" without a namespace)
Remove XMLNS (true or false, if true removes all namespace declarations and all xlink:* attributes)

3 - ?

Low

When you start the model service before the discovery and content services are started, it returns the following error (even after the other services have been started):

{
"timestamp": "2018-01-29T17:13:22.830+0000",
"status": 500,
"error": "Internal Server Error",
"exception": "com.google.gson.JsonSyntaxException",
"message": "com.google.gson.stream.MalformedJsonException: Invalid escape sequence at line 1 column 1194 path $.claimValues..DOCUMENT_ROOT",
"path": "/PageModel/tcm/2//dd4t.html"
}

This may not be high prio or even a real issue, as long as there is a dependency configured from the model service to the discovery service. (QS)

8 - TSI-3024

High

There should be no copies of the legacy DD4T libraries in the DXA codebase. This is confusing to the customers and does not have any real advantages. We need to align releases (and hopefully not release DD4T at all anymore, other than defect fixes).

Fixed issues

Number	Issue	Fix
	DD4T.Providers.DxaModelService DLL is built with dependency on DLLs that are not added to the dependencies section in the Nuspec. Dependencies used in the DLL, are: DD4T.ContentModel 2.2.1 DD4T.ContentModel.Contracts 2.2.1 (I installed Nuget version 2.2.2) DD4T.Core.Contracts 2.2.2 (I installed Nuget version 2.2.7) DD4T.Utils 2.2.2 (I installed Nuget version 2.2.7) Tridion.ContentDelivery 10.1.0.1003 (Nuget version 10.1.0) Sdl.Web.Delivery.Caching 10.1.0.1003 (Nuget version 10.1.0) This should be reflected in the Nuspec so they are installed automatically. The result right now is a type loading exception. Workaround is to manually configure bindings in the Web.config, but this shouldn't be necessary.
	ASP.NET MVC presents the URL to the action method WITHOUT a leading slash (so 'en/index.html' instead of '/en/index.html'). In DD4T, the slash is not prepended to the URL in the framework itself. Instead, it is normally done in the concrete controller implementation (in other words: the customer knows they have to add the leading slash themselves). The DXA 2 model service provider (I think it is in the provider, could be in the client as well) prepends the slash to the URL itself. As a result, all existing implementations return a 404, because the URL that is looked up in the broker, is '//en/index'.	Fixed by checking if the URL starts with a single slash, and only prepend a slash if it doesn't.
	Page cannot be deserialized when JSON is generated by DXA R2 TBBs. The error thrown in the application is 'Value cannot be null, parameter name: source'. The line that causes the error, is made red below (from the ContentModels.cs class file in DD4T.Model): [XmlIgnore] IList<IComponentPresentation> IPage.ComponentPresentations { get { return ComponentPresentations.ToList<IComponentPresentation>(); } } Looking at the JSON generated by the model service, I think it is simple: the conversion to DD4T-style JSON is not happening. The DD4T and DXA2 models are the same for the top level properties of the page (like Id and Title) but fail on the ComponentPresentations element, which does not exist on that level in DXA2. Update QS 23 Jan: the root cause has been identified by Alexey: it fails if the navigation.json page is not published. Since the point of this compatibility is to support existing DD4T customers who want to profit from the model service, it makes no sense to force them to create this page and publish it (they probably have an alternative navigation mechanism in place already). We need a version of the model service that does not fail when the navigation.json page is missing. Update QS 23 Jan (later on the day): the process also needs schemas.json. Bart is recommending to publish the Publish Settings page, which in turn publishes the above-mentioned JSON files as BinaryVariants. We will do that and retest. Update Jan 25: When running a dd4t-2-java app against http://ec2-52-17-130-35.eu-west-1.compute.amazonaws.com:8081/content.svc, it seems that the ModelService doesnt convert PageTemplate info. We need to decide whether this is a blocking issue or not. Quirijn Slings - we first also need to know how this page was published (with DXA or DD4T TBBs) Update Jan 30: everything works if a. you have published the publish settings page and the navigation page, and b. you have assigned DXA metadata to your DXA page template. This is something that must be made clear in the release notes.
	Access any page, for example http://localhost:83/further-information and 500 error is returned. In the site.log file, errors related to the model service are found and a URL it is trying to call. The errors are always similar to: DXA Model Service returned an error response: Status=500, Error='Internal Server Error', Exception='java.lang.NullPointerException', Message='No message available', Path='/PageModel/tcm/16/further-information' Copy the model service URL found in the log file and request it in the browser. http://ec2-52-17-130-35.eu-west-1.compute.amazonaws.com:8089/PageModel/tcm/16/further-information?includes=INCLUDE A perfectly normal JSON is retrieved. Access again http://localhost:83/further-information and now it works great. When trying to access any other page, the error cycle appears. Update KS: Tested on a fresh 8.5 instance with the latest hotfix CD_8.5.0.8403. The same results are observed as the last time. The only difference is that this time cachine was left as default (web.config of the web application wasn't changed at all) so that returning to the previous page did not return 500 since it was probably retrieved from the cache. Still, the main issue is there: Requesting of the new pages does not work unless the model service is not requested directly and via the web application. Update KS: Issue fix after the new version of DXA app was delivered by SDL.
	~~When using "Generate DXA R2 Page Model", the metadata of the page is not included in the JSON that is returned by the model service.~~ Sorry, apparently there is a Meta property and a Metadata property and I just got confused.
5	Fatal	Model conversion from R2 to DD4T fails. I request a page containing DXA R2 JSON and publish it with URL /dxa.html. The JSON looks like this (when retrieved with raw=true): dxa-raw.json. I then try to convert it to DD4T by calling the model service with this url: http://ec2-52-214-122-181.eu-west-1.compute.amazonaws.com:8089/PageModel/tcm/2/dxa?modelType=DD4T This results in this error: { "timestamp": "2018-01-30T13:18:29.967+0000", "status": 500, "error": "Internal Server Error", "exception": "java.lang.NullPointerException", "message": "No message available", "path": "/PageModel/tcm/2/dxa" } The dxa log of the model service shows this error: 2018-01-30 13:18:29,943 INFO c.s.d.m.s.DefaultPageModelService - Found R2 model while requested DD4T, need to process R2 and convert, request PageRequestDto(publicationId=2, uriType=tcm, path=/dxa, includePages=INCLUDE, contentType=MODEL, dataModelType=DD4T, expansionDepth=100, depthCounter=com.sdl.dxa.common.dto.DepthCounter@9f) 2018-01-30 13:18:29,953 DEBUG o.s.b.f.s.DefaultListableBeanFactory - Returning cached instance of singleton bean 'cacheManager' 2018-01-30 13:18:29,953 DEBUG o.s.b.f.s.DefaultListableBeanFactory - Returning cached instance of singleton bean 'cacheManager' 2018-01-30 13:18:29,954 DEBUG o.s.b.f.s.DefaultListableBeanFactory - Returning cached instance of singleton bean 'cacheManager' 2018-01-30 13:18:29,954 DEBUG o.s.w.s.m.m.a.ExceptionHandlerExceptionResolver - Resolving exception from handler [public org.springframework.http.ResponseEntity com.sdl.dxa.modelservice.controller.PageModelController.getPage(java.lang.String,int,com.sdl.dxa.common.dto.PageRequestDto$PageInclusion,com.sdl.dxa.common.dto.DataModelType,boolean,javax.servlet.http.HttpServletRequest) throws com.sdl.webapp.common.api.content.ContentProviderException,com.fasterxml.jackson.core.JsonProcessingException]: java.lang.NullPointerException 2018-01-30 13:18:29,954 DEBUG o.s.w.s.m.a.ResponseStatusExceptionResolver - Resolving exception from handler [public org.springframework.http.ResponseEntity com.sdl.dxa.modelservice.controller.PageModelController.getPage(java.lang.String,int,com.sdl.dxa.common.dto.PageRequestDto$PageInclusion,com.sdl.dxa.common.dto.DataModelType,boolean,javax.servlet.http.HttpServletRequest) throws com.sdl.webapp.common.api.content.ContentProviderException,com.fasterxml.jackson.core.JsonProcessingException]: java.lang.NullPointerException 2018-01-30 13:18:29,955 DEBUG o.s.w.s.m.s.DefaultHandlerExceptionResolver - Resolving exception from handler [public org.springframework.http.ResponseEntity com.sdl.dxa.modelservice.controller.PageModelController.getPage(java.lang.String,int,com.sdl.dxa.common.dto.PageRequestDto$PageInclusion,com.sdl.dxa.common.dto.DataModelType,boolean,javax.servlet.http.HttpServletRequest) throws com.sdl.webapp.common.api.content.ContentProviderException,com.fasterxml.jackson.core.JsonProcessingException]: java.lang.NullPointerException 2018-01-30 13:18:29,955 DEBUG o.s.w.s.DispatcherServlet - Could not complete request java.lang.NullPointerException: null at com.sdl.dxa.modelservice.service.processing.conversion.ToDd4tConverterImpl._buildPageTemplate(ToDd4tConverterImpl.java:201) at com.sdl.dxa.modelservice.service.processing.conversion.ToDd4tConverterImpl.convertToDd4t(ToDd4tConverterImpl.java:137) at com.sdl.dxa.modelservice.service.DefaultPageModelService._processDd4tPageModel(DefaultPageModelService.java:99) at com.sdl.dxa.modelservice.service.DefaultPageModelService.loadLegacyPageModel(DefaultPageModelService.java:80) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at org.springframework.cache.interceptor.CacheInterceptor$1.invoke(CacheInterceptor.java:52) Note that the DXA page template that we used on this page, has metadata assigned to it.
4	High	Model conversion (to R2) fails if page template has no metadata (which is a valid scenario in DD4T). The error is in the class ToR2ConverterImpl, in the _loadIncludes method. Please check if the pageTemplate has metadata at all before checking if it contains a certain field.
1	Low	XHTML namespaces are removed by the model service. This is probably not a problem for any customer, but it is a difference so we should put it in the release notes.
7 - TSI-3028	High	Importing the DXA CMS-items into an existing BluePrint hierarchy which already has a configured Topology with discovery services and everything, is too hard. It can really only be done by importing too much and then rewiring the topology manually. We cannot assume that customers will figure this out. Update: it is really much easier now with the new verison of the master-only.zip! The scenario should be described in the documentation, but it looks good now.