[Error] ["XdbContextLoggingPlugin"] XdbContext Batch Execution Exception

Recently we have upgraded our Sitecore to version 9.0.1 XP0 version on my local machine and used same databases & XConnect in to different environments. Understanding and setting up XConnect in local environment was a bit of hardship, particularly with the SSL certificates. With various blog posts online I could finally make my local environment working.

We had some series of issues with migration of XConnect on various environments but what annoyed the most was  XdbContext Batch execution exception in XConnect logs.

Stack trace in logs showed us

Sitecore.Xdb.Collection.Failures.DataProviderException: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server) ---> Sitecore.Xdb.Collection.Failures.DataProviderException.

As every other developer my natural inclination was that any one of the databases in connection strings was in accessible or the user did not have necessary permissions. Even after checking database servers and permissions XConnect was still giving same exception.

After lot of deliberation we found that _ShardManagement.ShardsGlobal table in Xdb.Collection.ShardMapManager database contains server name which is used by XConnect to communicate with various services.



Since I had copied databases from my local environment to other environments, ShardMapManager had my local db name in it. UAT environment XConnect was trying to connect to my local DB and was giving XdbContextLogging exception.

Replaced server name with respective environments and everything started working fine :)

Why Sitecore Recommends SOLR For Site Search?

Before we understand what SOLR is and why Sitecore recommends it for search, let us understand some basics regarding different type of databases we have.

SQL	NoSQL
Table based databases	Document based databases
Have predefined schema	Dynamic schema
Vertically scalable. Manage increasing load by increasing CPU, RAM etc. on single server.	Horizontally scalable. We can add few more servers easily to handle large traffic.
Good fit for complex queries.	Not good fit for complex queries.
Based on ACID properties (Atomicity, Consistency, Isolation and Durability)	Based on Brewers CAP theorem (Consistency, Availability and partition tolerance)
Good fit for transactional type applications.	Not stable enough for complex transactional applications.
MS-SQL, Oracle, MySQL, SQLite, Postgres	MongoDB, Redis, BigTable, RavenDb, Cassandra, Hbase, Neo4j and CouchDb

Full text search of SQL:

Pros: For smaller databases where we have limited content and fixed schema, full text search of SQL is ideal.

Cons: But since SQL is not horizontally scalable, performing search with limited resources (CPU, RAM etc.) can be a huge set back to the application.

Full text search with NOSQL:  

Pros: NoSQL should be used when Schema (data requirements) isn’t clear at the outset or if we are dealing with massive amounts of unstructured data. Think of non-relational databases more like folders, assembling related information of all types. For example, if a blog application used a NoSQL database, each file could store data for a blog post: social likes, photos, text, metrics, links etc. 

Cons: But storing and retrieving data in bulk like this requires extra processing effort and more storage than highly organized SQL data.

Full Text Search Engines:

Full text search engines excel at quickly and effectively searching large volumes of unstructured text and returning these documents based on how well they match the user’s query. They also have the ability to quickly facet, or categorize data or search results based on specific values of specific fields.

Example: Lucene, Solr, ElasticSearch, KinoSearch, Sphinix, Xapian.

Lucene: Lucene is able to achieve fast search responses because, instead of searching the text directly, it searched an index instead.

This is like retrieving pages in a book related to a keyword by scanning the index at the back of a book, as opposed to searching every word of every page of the book.

Advantages of Lucene:

1. Indexing: Indexing is the process of crawling through content and storing them on disk. Most of the search engines only support batch indexing; once they create an index for a set of documents, adding new documents becomes difficult without reindexing all the documents. Lucene supports both Incremental and batch indexing. Lucene allows easy adding of documents to an existing index.
2. Data Sources: Many search engines can only index files or webpages. Lucene in addition can index data from a database, or where multiple virtual documents exist in a single file, such as a ZIP archive.
3. Content Tagging: Lucene supports content tagging by treating documents as collection of fields, and supports queries that specify which fields to search.
4. Stemming: Reducing a word to its root form is called Stemming. Often, a user desires a query for one word to match other similar words. For example, a query for “jump” should probably also match words “jumped”, “jumper”, or “jumps”.
5. Stop-word processing: Common words, such as “a”, “and”, “the” etc. add little value to a search index. Lucene uses StopAnalyzer class which eliminates stop words from the input stream.
6. Query features: Lucene supports full Boolean queries, and queries and also has ability to search multiple indexes at once and merge the results to give a meaningful relevance score.
7. Concurrency: Lucene allows users to search an index transactionally, even if another user is simultaneously updating the index.
8. Non-English support: Lucene allows us to perform language-specific filtering.

Disadvantages of Lucene:

1. Scaling: Lucene works perfectly fine for single server environment. But in multi-server environment indexes have to be copied to each server which is error prone.
2. NO UI: We have to depend on external tools like LUKE to analyse indexes and run queries during development.

Lucene Vs Solr

A simple way to conceptualize the relationship between Solr and Lucene is that of a car and its engine. You can’t drive an engine, but you can drive a car. Similarly, Lucene is a programming library which you can’t use as-is, whereas Solr is a complete application which can be used out-of-box.

Solr = Lucene + additional features

Solr is an application or HTTP wrapper on top of Lucene. Content from the Lucene index can be retrieved using an HTTP GET query in XML, JSON, or binary formats.

Extra features in addition to Lucene

1. Faceted Search: Dynamically clusters search results into drill-down categories.
2. Built-in Sorting: Automatic features to sort search results by a variety of characteristics.
3. Web Admin Interface: SOLR application provides us with a web UI to run queries and check indexed data in various formats. We need not depend on external tools like LUKE.
4. Hit Highlighting: Shows a snippet of a document in the search results that surrounds the search terms.
5. HTTP query: Pass a number of optional request parameters to the request handler to control what information is returned.

Conclusion: Sitecore strongly recommends going with SOLR when we have scaled environment.  Lucene can only be used for a development or single server environment.

Sitecore Custom Validation Example

Following example is used to create an Even number validator using sitecore

Step 1: Create a new class inherited from StandardValidator class

using Sitecore.Data.Validators;

using System.Runtime.Serialization;

namespace Demo.Validators

{

    [Serializable]

    public class EvenNumberValidator : Sitecore.Data.Validators.StandardValidator 

    {

        public EvenNumberValidator() { }

        public EvenNumberValidator( SerializationInfo info,StreamingContext context):base(info,context)

        {

        }

    }

}

Step 2: Override Name property and Evaluate & GetMaxValidatorResult methods.

using Sitecore.Data.Validators;

using System.Runtime.Serialization;

namespace Demo.Validators

{

    [Serializable]

    public class EvenNumberValidator : Sitecore.Data.Validators.StandardValidator 

    {

        public EvenNumberValidator() { }

        public EvenNumberValidator( SerializationInfo info,StreamingContext context):base(info,context)

        {

        }

        public override string Name

        {

            get {

                    return (this.ToString()); // returns validator name to be displayed in error message

                }

        }

        protected override ValidatorResult Evaluate() //executes on validation

        {

            string val = GetControlValidationValue(); //fetches value of a control. GetItem() can be used to fetch complete item

            int n;

            bool isEven = false;

            bool isNumeric = int.TryParse(val, out n);

            if(isNumeric)

            {

                isEven = n % 2 == 0 ? true : false;

            }

            if(isEven)

            {

                return (ValidatorResult.Valid); // validation success

            }

            else

            {

                //validation failed

                Text = String.Format("{0} is not an even number!!",val); //Error message to be displayed

                return (GetFailedResult(ValidatorResult.Error)); //error level

            }

        }

        protected override ValidatorResult GetMaxValidatorResult()//blocks operations if we return FatalError or CriticalError

        {

            return (GetFailedResult(ValidatorResult.Error));

        }

       

    }

}

If the result of the GetMaxValidatorResult() method is FatalError or CriticalError, then Sitecore will block operations such as save or workflow commands in the user interface until validation completes.

Validation Bar and Quick Action Bar validators never block user interface operations.

Step 3: Create a new validator (here Even Number Validation)  at /sitecore/system/Settings/Validation Rules/Field Rules section using /sitecore/templates/System/Validation/Validation Rule template. Set Title, Description and Type properties.

Step 4:  Assign the validator to any field in a template. Here I have assigned it to “Validate Button” type of validation.

Step 5: Perform validation using “Validation” button under “Review” tab.

Sitecore Validation Examples

For each item, Sitecore invokes the item validation rules defined in the item or the standard values item associated with its data template, as well as global validation rules.

For each data template field, Sitecore invokes the validation rules defined in the data template field definition item, as well as the validation rules defined in the data template field type validation rules definition item.

After you stop editing a field value, Sitecore automatically invokes validation rules asynchronously, and then updates the user interface after validation completes.

Validators.UpdateDelay property in web.config controls the length of time between editing activity and the invocation of validators.

Validators.AutomaticUpdate property when set to false would disable automatic validation feature.

Validation Levels

Field Level: We can configure validators to be used for validation at the individual field level. For example, required field validator for a single line text field.  All field level validators are present at /sitecore/system/Settings/Validation Rules/Field Rules.

Field Type Level: We can configure validators to be used for validation of all occurrences of a particular field type. For example, all occurrences of DropList field type in sitecore tree. All field type level validators are present at /sitecore/system/Settings/Validation Rules/Field Types

Item Level: We can configure validators for individual items in its presentation details. All item level validators are present at /sitecore/system/Settings/Validation Rules/Item Rules

Standard Values Level: Configure validation for all items based on a data template by configuring validation rules in the standard values of that data template.

Global Level: Configure validation that applies to all items.

Sitecore Validation can be done in four different ways

1. Quick Action Bar.
2. Validator Bar.
3. User chooses Validate command in the Proofing group on the Review tab.
4. User chooses a specific workflow command.

     1. Quick Action Bar

     Steps to display field validation error messages on Quick Action Bar

     Step 1: Add validator for individual field in template in Quick Action Bar Section.

     Step 2: Right click on Quick Action Bar and enable “Validation Rules”

       



































      Step 3: Creating new page would display validation error on Quick Action Bar if “Page Title” value is empty.

       Note: We can save and publish the page even with validation errors.

     2. User chooses Validate command in the Proofing group on the Review tab.

    Step 1: Add validator for individual field in template in Validate Button Section.

    Step 2: Click on “Validation” button under “Review” tab without entering “Page Title” value.

     3. Validation Bar

     Steps to display error message on Validation Bar

    Step 1: Add validator for individual field in template in Validator Bar Section.

   Step 2: Select the page and click on "Save" button without entering "Page Title" value.

    4. Workflow

    Step 1: Create a workflow definition item “PageReview” in “/system/Workflows” section using “/sitecore/templates/System/Workflow/Workflow” template.

   Step 2: Create three workflow states “Draft”, “AwaitingApproval” & “Approved” using “System/Workflow/State” template under “PageReview” workflow definition item.

  

     Step 3:  Select workflow definition item and set the Initial State field to Draft.

    Step 4: Create “Submit” command under “Draft” state using template “/System/Workflow/Command”

    Step 5: Create “Approve” and “Reject” commands under “AwaitingApproval” state using template “/System/Workflow/Command”.

    Step 6: Set “Next State” of “Approve” command to “Approved” state and “Reject” command to “Draft” state and “Submit” command to “AwaitingApproval”.

  Step 7: Add “Validation Action” to the “Approve” command using “System/Workflow/Validation Action” template.

    Step 8: In the type field, enter the following

   Sitecore.Workflows.Simple.ValidatorsAction, Sitecore.kernel

   In Max Result Allowed field enter “Warning”

   Fill Error result fields with message “You cannot approve an item with validation errors”

   Step 9: Define the Final state to “Approved” state.

   Step 10: Assign the workflow to page template. Select “_Standard Values”, click on “View” tab in main menu and select “Standard Fields”

   Assign “Default workflow” to the “PageReview” workflow item.

   

   Step 11: Assign “Required” validator to “Page Title” field

   Step 12: Create a new page based on the template. New page would show a workflow warning at the top.

  Step 13: Click on “Submit” action under “Review” tab.

   Step 14: Click on “Approve” button to trigger work flow validation modal.