Myopicmage, the code experience

Sitecore 7 added a content search API to interact with Lucene and Solr. I'm sure anyone who has ever worked with search will tell you that search is hard, as it requires a lot of customisation that is entirely per-site, and what works for someone else might not work for you.

I'm here to tell you what worked for me, a really specific use case involving Sitecore 8.1 update 4, Apache Solr 6.2, and searching 7 regions with 4 different languages.

We're using some internal libraries on top of the content search API, but they eventually make the same calls as everyone else.

We started out with a fairly standard content search, which... mostly worked, even across languages. Condensed form:

var context = SearchIndex.CreateSearchContext(); var query = context.GetQueryable<oursearchresults>(); query.Content.Like(queryArgs);

There are actually a few issues with this approach:

The way our site is set up, 90% of the content we care about is actually in an item's components, not on the item itself.

This treats all languages the same way. Sitecore will send the same query to solr no matter the language being searched: _content:(*queryArgs*)

This will only give exact matches (even though .Like() is used)

Issue 1 is solved with a computed index field.

public class VisualizationField : MediaItemContentExtractor { public override object ComputeFieldValue(IIndexable indexable) { string baseValue = base.ComputeFieldValue(indexable) as string; Item indexItem = indexable as SitecoreIndexableItem; if (!ShouldIndexItem(indexItem)) { return baseValue; } var dataSources = Globals.LinkDatabase .GetReferences(indexItem) .Where(link => ShouldProcessLink(link, indexItem)) .Select(link => link.GetTargetItem()) .Where(targetItem => targetItem != null && targetItem.Versions.Count > 0) .Distinct(); var result = new StringBuilder(); if (!string.IsNullOrEmpty(baseValue)) { result.AppendLine(baseValue); } foreach (var dataSource in dataSources.Where(ShouldIndexDataSource)) { dataSource.Fields.ReadAll(); foreach (var field in dataSource.Fields.Where(ShouldIndexField)) { result.AppendLine(field.Value); } } return result.ToString(); } }

The ShouldProcess and ShouldIndex methods check to see whether or not something is actually related, and whether or not something should be put into the solr index based on some pretty basic parameters (correct content type, whether or not the component is actually being rendered).

Issue 2 caused me a great deal of stress until I stumbled across a blog post from the Sitecore 7 era. Sitecore added the concept of CultureExecutionContexts, which is a really fancy way of saying you can tell Sitecore to send over a search for content_t_{lang} instead of just _content by using this:

var context = SearchIndex.CreateSearchContext(); var culture = new CultureInfo(Sitecore.Context.Language.Name); var cultureCtx = new CultureExecutionContext(culture); var query = context.GetQueryable<oursearchresults>(cultureCtx); query.Content.Like(queryArgs);

And now your solr queries will look like this:

`content_t_{lang}:(*queryArgs*)`

Huzzah! You're searching specific languages! The problem quickly becomes, now you're doing language-specific exact match queries, which isn't very helpful.

Enter stemming algorithms.

The basic idea is that you give solr a word like engineer, and it boils the word down to the word's stem, so that you can run queries like engineer, engineers, engineered, or engineering and it will give you the same results. There are stemmers for basically every language you can think of, and the solr documentation explains how to use them far better than I ever could. The example schema.xml file generated by Sitecore actually contains basic analyzers that work fairly well. You will likely want to tweak them to fit your needs, but for an out-of-the-box solution, they work.

Once you've put the correct analyzers in place, restarted solr (this is important, solr does not pick up schema changes on the fly), and reindexed, you should now be getting decent search results in multiple languages.

Now is when language-specifics come into play. One of the languages this client supports is Polish, which does not come with out-of-the-box support from solr. Thankfully, there are already instructions for how to set that up.

The problem language for us, so far, has been German. German is what's known as a fusional language, which means that they tend to make new words by shoving old ones together. For instance, the German word for engineer is "ingenieur" and the word for civil engineer is "bauingenieur." This creates an issue for our search purposes, as "bauingenieur" and "ingenieur" should both return results for "ingenieur." The problem is solved with the Dictionary Compound Word Token Filter, a solr filter that will break words like bauingenieur down into their components "bau" and "ingenieur," so your results become what you'd expect. This requires a German word list, which can be a bit tricky to find, but once you have it, it works beautifully.

At this point, our search results have become downright useful and accurate (though we haven't implemented nice-to-haves like spellchecking and synonym searches), but there's a subtle bug. Sitecore isn't sending over the _content field to solr for each individual language properly. If your setup is like ours, with a very thin item and all of the pertinent content in subcomponents, the _content field in the index is going to be very sparse, basically containing nothing but the content in the top level item itself.

This is a subtle bug, and one that took several hours of debugging and someone far more versed in Sitecore than me to finally solve, but the issue is in the computed index field for the _content field.

var dataSources = Globals.LinkDatabase .GetReferences(indexItem) .Where(link => ShouldProcessLink(link, indexItem)) .Select(link => link.GetTargetItem()) .Where(targetItem => targetItem != null && targetItem.Versions.Count > 0) .Distinct();

This code will only get the components in Sitecore's default language. The rest of the code will properly put the correct language content from the top-level item into the index, but one of the checks it makes is whether or not a component is in the layout of that item in that version and language. If you have an item that only exists in the default Sitecore language, this works fine, but for any other language it's not going to get any of the subcomponents.

I haven't found any documentation about this, but the solution that is working for us is bringing in a LanguageSwitcher:

using (var switcher = new LanguageSwitcher(indexItem.Language)) { public class VisualizationField : MediaItemContentExtractor { public override object ComputeFieldValue(IIndexable indexable) { string baseValue = base.ComputeFieldValue(indexable) as string; Item indexItem = indexable as SitecoreIndexableItem; if (!ShouldIndexItem(indexItem)) { return baseValue; } var dataSources = Globals.LinkDatabase .GetReferences(indexItem) .Where(link => ShouldProcessLink(link, indexItem)) .Select(link => link.GetTargetItem()) .Where(targetItem => targetItem != null && targetItem.Versions.Count > 0) .Distinct(); var result = new StringBuilder(); if (!string.IsNullOrEmpty(baseValue)) { result.AppendLine(baseValue); } foreach (var dataSource in dataSources.Where(ShouldIndexDataSource)) { dataSource.Fields.ReadAll(); foreach (var field in dataSource.Fields.Where(ShouldIndexField)) { result.AppendLine(field.Value); } } return result.ToString(); } } }

So Entity Framework Core doesn't support lazy loading of navigational properties. This isn't the worst thing in the world, but it does become a bit of a problem when given this fact:

If you change the query so that it no longer returns instances of the entity type that the query began with, then the include operators are ignored.

From the MS docs:

In the following example, the include operators are based on the Blog, but then the Select operator is used to change the query to return an anonymous type. In this case, the include operators have no effect.

using (var context = new BloggingContext()) { var blogs = context.Blogs .Include(blog => blog.Posts) .Select(blog => new { Id = blog.BlogId, Url = blog.Url }) .ToList(); }

So now, because of this, I have this monstrosity of a query:

var submissions = await _db.Submissions .Include(x => x.answers) .ThenInclude(x => x.question) .Include(x => x.answers) .ThenInclude(x => x.answer) .Include(x => x.answers) .ThenInclude(x => x.cardchoices) .ThenInclude(x => x.card) .Include(x => x.answers) .ThenInclude(x => x.choiceanswers) .ThenInclude(x => x.answer) .Where(x => x.surveyid == id) .Where(x => x.answers.Any()) .Select(x => x.answers .OrderBy(y => y.question.sortorder) .Select(y => new { question = y.question.sortorder, answer = y.answer.text, text = y.text, cards = string.Join(", ", y.cardchoices.Select(z => z.card.gameid)), selections = y.choiceanswers.Select(z => z.answer) }) ) .ToListAsync();

State management is hard. That's just a fact of life. It's why god-awful solutions like redux exist. It's why people write blog post after blog post about state management. It's arguably the reason that functional programming has made the resurgence it has. State management remains an unsolved problem.

Unless you ask the React community.

Just use redux!

I did use redux, on two other projects. And if I have to write another

function setThing(thing) { return { type: SET_THING, thing } } function thingReducer(action, state = initialState) { switch (action.type) { case SET_THING: return { ...state, action.thing }; default: return state; } }

I will scream.

So I decided to do my own state management on ohheck.help. this.setState() is very tempting in the early days, when your project is small, or when you're not dealing with much data. This isn't my first react project, so I'm familiar with how to structure things to minimize usage of this.state, and to do my best to rely on props. But when you bring react-router into the picture, things get a little more complicated. What happens in a standard master/detail situation?

For instance, I have a list of goups, and I want to pull up a subunit detail page. I've already downloaded the groups and subunit data. When I link to the subunit page, what do I do?

In react-router 3, you'd either rely on redux, or fetch the data from the server again on component load. Or maybe shove the data into localStorage.

However, in 4, there's a handy addition to <link>:

item.subunits.map((innerItem: Subunit, innerIndex: number) => <div key={innerIndex}> <Link to={{ pathname: `/dashboard/subunits/${innerItem.id}`, state: innerItem }}> {innerItem.name} </Link> </div> )

pathname remains the same as 3, but the new addition is state, which you can use to pass arbitrary data from one route to another. So in my Subunit component:

componentDidMount = () => { if (!this.props.location.state) { this.getData(this.props.match.params.id); } else if (this.props.location.state.id != this.props.match.params.id) { this.getData(this.props.match.params.id); } else { this.setState({ subunit: new Subunit(this.props.location.state), loading: false }); } }

There are three conditions to check:

there's no state data from being <link>ed to

we've received state by being <link>ed to, but for some reason the url doesn't match.

we've received data from a <link>

In the first two cases, the solution is to fetch() the data from the server, but we can skip that in the case that someone has clicked on a link to get to this page. It makes your logic a little more complicated, but it saves a request or two.

public abstract class Common { public int id { get; set; } public DateTime created { get; set; } public string createdby { get; set; } public DateTime modified { get; set; } public string modifiedby { get; set; } }

and then all of my classes inherit from that one. The thing is, manually updating those four properties on every write to the database is, let's face it, a pain. So during my trials with entity framework core 2.0, I stumbled across a stack overflow post which pointed me at the following code:

public override Task<int> SaveChangesAsync(CancellationToken cancellationToken = default(CancellationToken)) { var changeSet = ChangeTracker.Entries<Common>(); if (changeSet != null) { foreach (var entry in changeSet.Where(x => x.State != EntityState.Unchanged)) { if (entry.State == EntityState.Added) { entry.Entity.created = DateTime.Now; entry.Entity.createdby = user; } entry.Entity.modified = DateTime.Now; entry.Entity.modifiedby = user; } } return base.SaveChangesAsync(cancellationToken); }

I'm not sure how to have a fallback route?

<Route exact path="/dashboard" component={a.AdminHome} /> <Route path="/dashboard/responses/:id" component={a.Responses} /> <Route exact path="/dashboard/responses" render={() => <h3>Please go back and select a survey.</h3>} /> <Route path="/dashboard/bycard/:id" component={a.SurveysByCard} /> <Route exact path="/dashboard/bycard" render={() => <h3>Please go back and select a survey.</h3>} /> <Route path="/dashboard/groups/:id" component={a.SingleGroup} /> <Route exact path="/dashboard/groups" component={a.Groups} /> <Route path="/dashboard/subunits/:id" component={a.SingleSubunit} /> <Route exact path="/dashboard/subunits" component={a.Subunits} /> <Route path="/dashboard/idols/:id" component={a.SingleIdol} /> <Route exact path="/dashboard/idols" component={a.Idols} /> <Route path="/dashboard/cards/:id" component={a.SingleCard} /> <Route exact path="/dashboard/cards" component={a.AllCards} /> <Route path="/dashboard/survey/:id" component={a.SurveyView} /> <Route exact path="/dashboard/survey" component={a.NewSurvey} />

renderCards = () => this.state.cards.map( (item: Card, index: number) => <div className="pure-u-1-4" key={index}> <img src={item.imageurl} style={{ width: '250px', height: '350px' }} /> </div> )

In ConfigureServices() inside of Startup.cs:

services.AddMvc().AddJsonOptions(options => { options.SerializerSettings.ReferenceLoopHandling = ReferenceLoopHandling.Ignore; options.SerializerSettings.NullValueHandling = NullValueHandling.Ignore; });

One of the neatest things I've learned is that controller type signatures in .net core have changed.

In all of the documentation, controller methods look like this:

public IActionResult Page() => View();

That's still necessary if you're going to return something like a View(), but if you're going to return an object, asp.net core supports WebApi-style method signatures, where you directly return an object.

public async Task<List<SomeDto>> AllThings() => await _ctx.Things.ToListAsync();

This is a bit cleaner than what you used to have to do:

public async Task<IActionResult> AllThings() => Json(await _ctx.Things.ToListAsync());

I've been working on a site named ohheck.help for some friends. It's a personal passion project, because I like to know things.

The site is written entirely in asp.net core 1.1, using entity framework core on postgresql. It was started in asp.net core 2.0, but entity framework 2.0 is not ready for prime time (even using sqlite), and the provider for postgres couldn't even implement FirstOrDefault(), which was a bit of a problem.

As a mostly windows/web developer, it's been really interesting to learn more about how to host stuff like this on linux. Nginx is kind of crazy, and actually running outside of IIS is different.

This isn't my first asp.net core application, but it's the first one I've put a huge amount of effort into.

At the time of this writing, the whole project is about 13k lines of code (admittedly not all of them mine), but that's actually not counting dependencies.

Anyway this blog is mostly to document stuff I've learned about .net core and typescript along the way.

It's so clean and nice.