While working on the integrating Sitecore’s CM’s functionality with third party Digital Asset Management(DAM) system, I came across one interesting issue where I wanted my container applications to run on https.
After integrating the third party DAM with Sitecore CMS, it found out that, while using local website on developer machine, the developer is not able to load the SSO login page. While loading that page below error messages logged in browser’s console.
Access to the WebCrypto API is restricted to secure origins. Compact View requires HTTPS when used outside localhost (for development).
As clearly stated I am not using localhost as host to access my cm and seems like CM needs to be securely severed over https where as we, for local development, use non secure protocol http.
The solution is simple looking form 35,000 feet. Needs CM urls to be serving content over the secure HTTPS protocol. But when actually started digging in details, it is fun and very learning experience.
Performed following steps to achieve this on our local docker development environment.
Open the startup/createcert.ps1 for editing and change following parameter’s default value
$certificatepassword – from b to more secured passowrd string
$dnsNameList – by default the value is *.dev.local. Change this value to match your host pattern. You can specify individual host comma(,) separated.
Alternatively, you can can pass those two as parameter while firing createcert.ps1 command.
3. Open docker-compose.yml file and navigate to cm service and perform following changes:
Add new environment parameter HOST_HEADER and set cm host value to it. If you have multiple host(this is what in my case), specify them semicolon(;) separated list. For example host1.dev.local;host2.dev.local
Under the volumes bind the new volume for folder startup to c:\startup path of container.
Under the port bind the https’s default secure port 443 with the next available container port
The container host port needs to be new port which is not been use by any other services.
Lastly, change the entypoint to the path of startup.ps1 file under the startup folder bound under the volumes in above step for startup folder.
After all above changes to CM service of docker-compose.yml file will lokk something like below
The QA team, we are working with has raised a very interesting issue while testing newly created data report using Sitecore Poershell Extenstion(SPE).
The data report was executing perfectly but while trying to export the data in any of pre-defined formats offered by SPE, it was not working.
The Sitecore Powershell is allowing export of report data in various industrial standard formats out of the box as listed below:
These formats have option links in the ribbon above but when clicking, none of those producing any output.
First step is to try to reproduce the issue. We tried on all the available environments and it is happening on every environment except for the local environment.
I though the issue may have to do with some missing config or SPE. So, we quickly try to re-install the SPE module on CM with no over-write option for both files and content on Integration Environment. But that does not fix the issue. ☹
So, we started listing what are the major difference from modules and features point of view between local development and other environments on Azure. It turns out that all other environments have exactly the same piece of code, configurations and modules installed except for Federated Authentication using Sitecore Identity Server for CM.
We did quick search over the internet and one of my colleague found out that there is one file which is disabled for SPE which needs to be enable if we are using sitecore 9.1 and higher(we are running 9.3) along with identity server.
The file needs to be enabled is Spe.IdentityServer.config file which is part of the SPE package.
Once package is installed, the file is located at path <<website Root>>\App_Config\Include\Spe\Spe.IdentityServer.config.disabled. The content of this disabled file is as below.
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/" xmlns:role="http://www.sitecore.net/xmlconfig/role/" xmlns:security="http://www.sitecore.net/xmlconfig/security/">
<sitecore role:require="Standalone or ContentManagement" security:require="Sitecore">
<processor type="Sitecore.Owin.Authentication.Pipelines.CookieAuthentication.ValidateIdentity.ValidateSiteNeutralPaths, Sitecore.Owin.Authentication">
<!-- This entry corrects the infinite loop of ExecuteCommand in the SPE Console -->
Tip: The best way to get these config enable in Docker container world, is to leave the above mentioned .disabled file as is. Create new configuration file with above content in Visual Studio solution deploying to z folder under app_config\include on your binding volume path which will be taken care by Sitecore config patching.
While configuring Coveo search on my local development environment using Docker, something got my attention on Coveo’s diagnosis page(<<Sitecore CM Host>>/sitecore modules/Web/Coveo/Admin/CoveoDiagnosticPage.aspx).
This page has all the green ticks except for the one. This is Coveo Search REST Endpoint. See image below for more details.
As usually, I copied the API end point URL and hit in the Browser and it serve the request with no error. It is HTTP-200 response with valid JSON result.
The error in the log file also not giving any lead either.
Coveo.SearchProvider.Applications.BaseVerifier.VerifyComponent(:0) ERROR StateVerifier has encountered an error while verifying the component "Coveo Search REST Endpoint".<br>Exception: Coveo.SearchProvider.Applications.StateVerifierException<br>Message: Failed to obtain resource located at 'http://sicore930.dev.local/coveo/rest/'. See inner exception for details.<br>Source: Coveo.SearchProvider.Applications<br>at Coveo.SearchProvider.Applications.CommonStateVerifier.TryPingRestEndpoint(String p_Endpoint)<br>at Coveo.SearchProvider.Applications.CommonStateVerifier.<>c__DisplayClass11_0.b__0()<br>at Coveo.SearchProvider.Applications.BaseVerifier.VerifyComponent(Func`1 p_VerifyMethod, String p_ComponentName)
I opened docker powershell command for the cm container and tried to request the host from within container.
PS> Docker container ls
Grab the name or container id
PS> Docker exec -it <<container name/id>> cmd
This will open a powershell window connected to the container specified.
Now things are little clear to me. The Coveo diagnosis page is using current host to make REST API request from IIS server running on the container. The container does not have host file entry to reseolve the host url.
So now, I started searching how to make entry in container’s host file??? I was sure, I am not the first person who wanted this kind of stuff.
While setting my local environment for development on Sitecore 9.3 using docker, I came across below issue.
[Win32Exception (0x80004005): The network path was not found]
[SqlException (0x80131904): A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server)]
System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection) +1341
System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection) +159
System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection) +382
System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions) +307
System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource`1 retry) +198
System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry) +422
Sitecore.Data.DataProviders.Sql.DataProviderCommand..ctor(IDbCommand command, DataProviderTransaction transaction, Boolean openConnection) +113
Sitecore.Data.DataProviders.NullRetryer.Execute(Func`1 action, Action recover) +293
Sitecore.Data.DataProviders.NullRetryer.Execute(Func`1 action, Action recover) +293
Sitecore.Data.DataProviders.Sql.SqlDataApi.CreateReader(String sql, Object parameters) +281
Sitecore.Data.SqlServer.SqlServerDataProvider.ExecutePreLoadItemDefinitionSql(String sql, Object parameters, SafeDictionary`2 prefetchData) +52
Sitecore.Data.DataProviders.Retryer.ExecuteNoResult(Action action, Action recover) +539
Sitecore.Data.SqlServer.SqlServerDataProvider.LoadInitialItemDefinitions(String condition, Object parameters, SafeDictionary`2 prefetchData) +237
Sitecore.Data.DataProviders.Sql.SqlDataProvider.GetPrefetchData(ID itemId) +62
Unicorn.Data.DataProvider.UnicornSqlServerDataProvider.GetItemDefinition(ID itemId, CallContext context) +242
Sitecore.Data.DataProviders.DataProvider.GetItemDefinition(ID itemID, CallContext context, DataProviderCollection providers) +156
Sitecore.Data.DataSource.GetItemInformation(ID itemID) +88
Sitecore.Data.DataSource.GetItemData(ID itemID, Language language, Version version) +32
Sitecore.Data.Engines.TemplateEngine.GetTemplate(ID templateId) +184
Sitecore.XA.Foundation.SitecoreExtensions.Extensions.DatabaseExtensions.GetContentItemsOfTemplate(Database database, ID templateId) +126
Sitecore.XA.Foundation.Multisite.SiteResolvers.EnvironmentSitesResolver.ResolveAllSites(Database database) +63
Sitecore.Sites.SiteCollection.AddRange(IEnumerable`1 sites) +221
Sitecore.XA.Foundation.Multisite.Pipelines.Initialize.InitSiteManager.Process(PipelineArgs args) +85
(Object , Object ) +9
Sitecore.Pipelines.CorePipeline.Run(PipelineArgs args) +490
Sitecore.Pipelines.DefaultCorePipelineManager.Run(String pipelineName, PipelineArgs args, String pipelineDomain, Boolean failIfNotExists) +236
Sitecore.Pipelines.DefaultCorePipelineManager.Run(String pipelineName, PipelineArgs args, String pipelineDomain) +22
Sitecore.Nexus.Web.HttpModule.Init(HttpApplication app) +1165
System.Web.HttpApplication.RegisterEventSubscriptionsWithIIS(IntPtr appContext, HttpContext context, MethodInfo handlers) +584
System.Web.HttpApplication.InitSpecial(HttpApplicationState state, MethodInfo handlers, IntPtr appContext, HttpContext context) +168
System.Web.HttpApplicationFactory.GetSpecialApplicationInstance(IntPtr appContext, HttpContext context) +277
System.Web.Hosting.PipelineRuntime.InitializeApplication(IntPtr appContext) +369
[HttpException (0x80004005): A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server)]
System.Web.HttpRuntime.FirstRequestInit(HttpContext context) +532
System.Web.HttpRuntime.EnsureFirstRequestInit(HttpContext context) +111
System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest wr, HttpContext context) +724
The error speaks itself, that the CM and/or CD was not able to connect to underlying SQL server. SO, I immediately opens the SSMS and try to connect to my SQL server running as container on my local and I am successfully able to connect both using IP address as well as using alias(in my case it’s sql).
So what’s the issue??? When I am able to connect to SQL server running in container using outside of the Docker created network than why my CM and/or CD instance is not able to connect to it within the same network???
I check inward and outward binding ports setting in docker-compose.yml file for SQL service. Double check connection strings for server name, user name and password and that all looks correct.
Thanks to one of my colleague, who points out my docker-compose file was missing hostname for SQL server service.
I have added them to my docker-compose.yml for SQL, SOLR and XCONNECT and happy days.
This is going to be a short blog for fixing issue I came across while working on custom pipeline creation and execution for Sitecore’s Data Exchange Framework(DEF or DxF).
While the Pipeline Batch item is selected, buttons on DATA EXCHANGE toolbar at top does not behave how they should.
Immediately after selecting pipeline item, I noticed all buttons in ribbon is visible and clickable. Secondly, clicking on Run Pipeline Batch Button does not works. I was neither starting execution of pipeline nor displaying any visible error messages. See below image for more details:
I started investigation and collecting events what could have done the damage. Found that before last deployment it was behaving the way it should be. Huuummm… that’s good hint, isn’t it? The last release just deployed files to CMS server. This is soothing to my ears as investment scope is drastically reduced to files. The content automatically gone out of equation.
To fix this, I tried install DEF SDK package first with just deploying file artifacts and skipping any content installation.
Unfortunately, that does not work ….!!! 😦
Next best bet is obvious and try to re-install the DEF framework package with deploying files only.
And that does the trick ….!!!! (Dance)
The file \App_Config\Sitecore\DataExchange\Sitecore.DataExchange.Local.config was missing. Placing this file from Package has actually fixed the issue.
Happy Data Sync and import using Sitecore Data Exchange Framework. 🙂
This morning when I login to one of none production environment I found that the graph was not loading on Sitecore CMS launch pad.
When I did into more details and found out the ajax request to get the data are failing with server error(HTTP-500). As shown in below image the response of the API call does not providing any information or clue.
Immediately, I jumped onto logs and found following entry in there.
The remote server returned an error: (403) Forbidden.
The full exception looks like below.
50616 09:48:50 ERROR [Experience Analytics]: System.Net.WebException: The remote server returned an error: (403) Forbidden.
at Sitecore.Xdb.Reporting.Datasources.Remote.RemoteReportDataSourceProxy.GetData(ReportDataQuery query)
at Sitecore.Xdb.Reporting.ReportDataProvider.GetCachedDataForQuery(ReportDataQuery query, Func`1 ifNotAvailableInCache, Nullable`1 ifModifiedSince)
at Sitecore.Xdb.Reporting.ReportDataProvider.ExecuteQueryWithCache(ReportDataQuery query, ReportDataSource dataSource, CachingPolicy cachingPolicy)
at Sitecore.Xdb.Reporting.ReportDataProvider.GetData(String dataSourceName, ReportDataQuery query, CachingPolicy cachingPolicy)
at Sitecore.ExperienceAnalytics.Api.ReportDataService.ExecuteQuery(IReportQueryData queryData, CachingPolicy cachingPolicy)
at Sitecore.ExperienceAnalytics.Api.ReportingService.RunQuery(ReportQuery reportQuery)
at Sitecore.ExperienceAnalytics.Api.Http.AnalyticsDataController.Get(ReportQuery reportQuery)
After further investigation found out the Reporting keys were not aligned between CM and Reporting server in the ConnectionStrings.config file.
While using Razl syncing couple of large content sub-trees under Sitecore this morning, it has prompted for below error after progressing for while.
Error 'The HTTP request to 'http://xxx.xxx/_CMP/RazlService.asmx' has exceeded the allotted timeout of 00:04:59.9990000. The time allotted to this operation may have been a portion of a longer timeout.' getting Sitecore items.
The fix for above issue is to increase timeout period for the http request to classic web service. After applying following fix, it has started working as expected.
Navigate to the root folder of the website mentioned in above error message
navigate to _CMP folder and open web.config file for editing
Add following line to the web.config file under <system.web> node
So far in Part – 1 and Part – 2 of this series, the Sitecore Publishing Service up & running in Azure PaaS and the Sitecore Publishing module is installed & configured.
This last part will trying to address few additional scenarios with publishing service. Scenarios like, the Sitecore databases are hosted in Azure SQL and Sitecore implementation have multiple publishing targets.
Sitecore database in Azure(Azure SQL)
If publishing service hosted on Azure web app and the databases are also in Azure SQL Database, there are few additional configurations needed for the service to work seamlessly with bigger publishing payload.
Follow below steps to configure publishing service for Azure Environment
Navigate to \configurations\system.webServer\aspNetCore node in wwwRoot\web.config file of publishing service in app service.
replace the arguments attribute with the following
arguments=" --environment azure"
Save the web.config
Stop and start app service
By default Sitecore publishing service is running under the “production” environment and that patches the configurations for production. After above change, service will start with Azure environment and it will patch azure environment specific configurations.
Navigate to wwwroot/config/azure
rename file sc.publishing.sqlazure.connections.xml.example to sc.publishing.sqlazure.connections.xml. Remove .example extension and make sure it has .xml extension.
New publishing target…!!!???
If other than default Internet publishing target is being used in CMS, there are few additional configurations needs to be made. So lets have a look at those.
I am assuming the additional publishing target also linked to additional database other than default web database. If so, the new connection string also needs to be added to publishing service.
New connection string
In this scenario, will take example of new web database in Azure SQL US region and need to add new target to publish content to web database to this US region.
Add new connection string to Publishing service. lets name it web_us
Sitecore.Framework.Publishing.Host configuration setconnectionstring web_us "<US web database connection string>"
Open above file and follow below steps:
navigate to Settings\Sitecore\Publishing\Services\DefaultConnectionFactory\Options\Connections
Search for the child node Internet, duplicate the whole node along with children. For example, name it InternetUS
Within that InternetUS node navigate to child node ConnectionString and change the value of the node to following to match the name of the connection sting in steps above.
Navigate to Settings\Sitecore\Publishing\Services\StoreFactory\Options\Stores\Targets
Search for the child node Internet, duplicate the whole node along with children. For example, name it InternetUS
Change the value of the ConnectionName to InternetUs. This must match the name of the name of the new node name under Settings\Sitecore\Publishing\Services\DefaultConnectionFactory\Options\Connections. Created in one of the step above.
Set the id of new Publishing Target item in Sitecore CMS. For example,
Set the value of child node ScDatabase to newly added databsae name. In our example it would be web_us.
One can select log filter level to one of the following:
Trace : The most detailed log messages, may contain sensitive application data. These messages are disabled by default and should never be enabled in a production environment.
Debug : Logs that are used for interactive investigation during development should use the Debug level. These logs should primarily contain information useful for debugging and have no long-term value.
Information : Track the general flow of the application using logs at the Information level. These logs should have value in the long term.
Warning : Warnings should highlight an abnormal or unexpected event in the application flow. This event does not cause the application execution to stop, but can signify sub-optimal performance or a potential problem for the future.
Error : An error should be logged when the current flow of execution is stopped due to a failure. These should indicate a failure in the current activity, not an application-wide failure. These will mainly be unhandled exceptions and recoverable failures.
Critical : A critical log should describe an unrecoverable application or system crash, or a catastrophic failure that requires immediate attention.
To overwrite these default log level filters, use wwwroot/config/sitecore/sc.logging.xml file.
The Development environment mode sets the Sitecore log level filter to Debug. So the service can be run under Development Environment for more detailed logging.
After all these configuration changes restart the publishing service.
Now that Publishing service is deployed, configure and up in Part – 1, the next bit is to install and configure Sitecore publishing service module in Sitecore CM and CD. This part is very simple and strait forward. So lets get started…
The Sitecore publishing module is nothing but Sitecore package and needs to be installed the same way as other Sitecore packages. This package contains all the necessary artifacts for this module to perform it’s job.
The Sitecore Publishing Module can be downloaded from here.
Just check the the version compatibility with Sitecore version where you wanted to implement this publishing service/module. For this you need to navigate to individual release page and check the compatibility detail at start of the page under the main heading.
Install publishing module
To install Sitecore publishing module, one can use OOTB Sitecore installation wizard.
***Important! Copy and save this information***
BEFORE YOU CLICK NEXT:
- Ensure you have installed and configured the Sitecore Publishing Service (this module only enables integration with the service)
Documentation detailing how to install the service is available seperately.
[Warning] This module will not work without a properly configured service instance. No items will be able to be published.
AFTER YOU CLOSE THE WIZARD:
After the package is installed, follow these steps to complete the Sitecore Publishing Module installation:
- Configure the service endpoints:
Add a configuration file which overrides the 'PublishingServiceUrlRoot' setting to point to your service module
Make sure the address contains a trailing slash
<?xml version="1.0" encoding="utf-8"?>
- Configure the Content Delivery Servers:
Ensure that the following file is in the website 'App_Config/Modules/PublishingService' directory:
Ensure that the following files exist in the website 'bin' directory:
Just before installation process starts, installer will prompt the information about configurations and artifacts needed for the Sitecore Module and services to work. As instructed, copy and save that instruction shown in above(From version 3.1.3).
Once installation of the module is successful, in CM perform following steps:
Navigate to <website root>/App_Config/Modules/PublishingService folder in CM app service
Make sure the URL is post-fix with forward slash(/) at the end.
Navigate to <website root>/App_Config/Modules/PublishingService folder in both the CM and CD app services
Copy Sitecore.Publishing.Service.Delivery.config file from CM to above folder in CD
Navigate to <website root>/bin folder in both the CM and CD app services
Copy below library files to CD’s bin folder from CM
At this point, the publishing will start working, if there is only one default Internet publishing target is there. The Publishing Module has overwritten few of Sitecore default settings for publishing. So after installation of publishing module, one may notice change in UI for the Publishing. Please read here for more details about visual changes and changes in types of publishing.
The last part -3 will cover some specific configuration for Azure PaaS environment and some advance setting of Sitecore Publishing Service. So stay tuned and enjoy content publishing….!!!
While working on finding solution for slow publishing issue of more than 10,000 items in Azure PaaS, I came across this very powerful add-on Sitecore Publishing Services.
Believe it or not but once it is installed, configured and up, it reduced the publishing time drastically. In our case, publishing of those 10,000 items were taking around 1 hour 30 mins(hhhuuuffff…), merely took 12 mins to publish.(Super duper awesome, isn’t it…??? This makes everyone’s life easy)
Thus, I decided to write blog on whole solution. Blog will become very lengthy if I put everything in one. Thus, splitting it in three parts.
Part 1 will cover installation and basic configuration of publishing service. Part 2 will cover installation of publishing module in CMS. The third part will cover advance configuration of publishing service.
So, lets get started…
You can download both Sitecore publishing service and publishing module from Sitecore’s development site here. It is the designed and developed by Sitecore itself.
Just check the version compatibility with Sitecore version where you wanted to implement this publishing service/module. see image below for more details. (As I have downloaded first the latest and greatest version 4 at the time I am writing this blog for Sitecore 9.0.1 and later figure out it is not compatible.)
Install publishing service on Azure PaaS
With my current client all their Sitecore implementation is in Azure PaaS. It absolutely makes sense to have this Sitecore Publishing service also setting in Azure PaaS. So, follow below steps to get the publishing services up as app service.
Download the service application as zip file from above link. Select the one which is matching your host architecture type either 32 bit or 64 bit.
Open Kudu – Advance tool for the web app where you want to host publishing service
in Kudu editor open command console by navigate to Debug Console menu option and select CMD
navigate to site root folder using following command
Drag and drop service zip file to folder view area above console. This will upload the service package to Azure web app.
Once upload completed, run following command on console below the folder view
unzip -o *.zip
The site’s wwwroot folder has only one zip file which is recently uploaded publishing service zip file.
Once unzipping of application completed, execute following commands in CMD. This will set connection stings of core, master and web database in publishing service. Going forward, these connection strings will be used for all publishing jobs.
After executing above command the console displays the result of the command. The below screenshot have result with already updates schema for all the databases as I have previously upgraded the schema.
Once above commands are successfully executed, now it is the time to check the status of the publishing service web application.
In the browser open below URL
http://<<azure app service name>>.azurewebsites.net/api/publishing/maintenance/status
Above web api will return following JSON result if all the configurations are correctly set.
This are basic steps to deploy Sitecore Publishing services up and running in azure PaaS. In the Part – 2, will cover installation of Sitecore Publishing Module in Sitecore CM/CA with configuration.