Anniversary edition of F# Weekly #43, 2015 – Three years together

3years

Welcome to F# Weekly,

This is an anniversary edition of F# Weekly, we are together for 3 years already! Just wow! It was an amazing time, a lot of things changed in the F# world during this period, but one thing stays the same – you still read F# Weekly every week, thank you for that!

In order to celebrate this date, I ask you to join “F# Advent Calendar in English 2015” and make this December unbelievably cool!

FsAdvent

A roundup of F# content from this past week:

News

Videos/Presentations

Blogs

F# vNext News 

New Releases

That’s all for now. Have a great week.

Previous F# Weekly edition – #42Subscribe

F# Advent Calendar in English 2015

FsAdvent

Update(10/27/2015): There are a lot of people, who want to participate, so we’ve decided to extend the timeline and double number of slots. Please do not be shy and books free slots.

Last year we ran an amazing event “F# Advent Calendar in English 2014“. It was incredible December full of F# and Christmas spirit. Every day astonishing authors around the globe posted new F#-related articles. It was an extraordinary time.

December is close enough, so it is a good time to plan something special for F# Advent Calendar. Have you done something special this year? Do you have any unique experience you are willing to share? Have your project incredibly evolved this year? Are there any good ideas for the post, but you didn’t have time to write it? The time has come – it is right now! You have a chance to share your story with the globe! Join F# Advent Calendar and hurry up!

Rules

Rules are very simple:

  1. Choose F# related topic for your blog post and reserve the date on Twitter or leave a comment to this post. Please note that you do not have to announce the topic until the date.
  2. Prepare a blog post in English
  3. Publish your post on specified date (according to the calendar)
  4. Post link to your post on Twitter with hashtags #fsharp and #FsAdvent.

Calendar

Date Author Post Title
 Nov 29 (Sunday)  Rachel Reese  How Jet Build Microservices with F#
 Nov 29 (Sunday)  Jamie Dixon  Creating Dynamic Uris For Visual Studio Web Tests
 Nov 30 (Monday)  Steffen Forkmann  F# advent calendar: Using Async.Choice in Paket
 Nov 30 (Monday)  Bohdan Szymanik  Sharpen up your legacy app(s) performance with a bit of F#
 Dec 01 (Tuesday)  Mark Seemann  Recurse
 Dec 01 (Tuesday)  Kristian Schmidt  A roll of the Liar’s dice
 Dec 02 (Wednesday)  Mike Janger  Taking Ionide Out for a Spin
 Dec 02 (Wednesday)  Tomasz Jaskuλa  Data Science tools in F# through univariante linear regression
 Dec 03 (Thursday)  Phillip Trelford  Calendar Types
 Dec 03 (Thursday)  Jeremy Abbott  F# Events, Reactive Programming and Async Workflows
 Dec 04 (Friday)  Richard Dalton  Azure WTF#
 Dec 04 (Friday)  Edgar Sánchez  Calculating a cannon ball trajectory, the fun way
 Dec 05 (Saturday)  Scott Wlaschin  Thirteen ways of looking at a turtle
Thirteen ways of looking at a turtle (part 2)
Thirteen ways of looking at a turtle – addendum
 Dec 05 (Saturday)  Sean Trelford  No 1 at Christmas
 Dec 06 (Sunday)  Andrea Magnorsky  Computation expressions and microphones
More Computation expressions
 Dec 06 (Sunday)  Christopher Atkins  F# 2015 Advent Cookies
 Dec 07 (Monday)  Sergey Tihon  Application contracts with Swagger powered APIs for .NET or Why SwaggerProvider
 Dec 07 (Monday)  Aaron Powell  What’s the time Mr Wolf?
 Dec 08 (Tuesday)  Isaac Abraham  F#, .NET and the Open Source situation
 Dec 08 (Tuesday)  Jonathan Wood  A Quick Look At F# In Visual Studio Code
 Dec 09 (Wednesday)  Reed Copsey, Jr.  Christmas Trees in WPF using FSharp.ViewModule
 Dec 09 (Wednesday)  Peter Bayne  The trips and traps of creating a Generative Type Provider in F#
 Dec 10 (Thursday)  Daniel Egloff  Algo Trading with F# and GPUs
 Dec 11 (Friday)  @TeaDrivenDev  Making Busy Progress in F#
 Dec 11 (Friday)  Reid Evans  Providing Value with Trivial Abstraction in F#
 Dec 12 (Saturday)  Eriawan Kusumawardho  What’s new in F# 4.0 in Visual Studio 2015
 Dec 12 (Saturday)  Riccardo Terrell  Solving the Santa Claus Problem in F#
 Dec 13 (Sunday)  Marcus Griep  Chiron: JSON + Ducks + Monads
 Dec 13 (Sunday)  @lenadroid  Learn the machine! #fsharp #accord.net
 Dec 14 (Monday)  Tomas Jansson  F#, event sourcing and CQRS tutorial… and agents
 Dec 14 (Monday)  Alex Casquete  Building an Hypermedia REST API with F# and Suave.IO
 Dec 15 (Tuesday)  Evelina Gabasova  The Star Wars social network
 Dec 15 (Tuesday)  Stachu Korick  Pseudocode-Driven Development with F#
 Dec 16 (Wednesday)  Yan Cui  Advent of Code F# – Day 16
 Dec 16 (Wednesday)  Paulmichael Blasucci  A Mixed-Paradigm Recipe for Exposing Native Code
 Dec 17 (Thursday)  Kunjan Dalal  1729
 Dec 17 (Thursday)  Jérémie Chassaing  Ukulele Fun for XMas !
 Dec 18 (Friday)  Anton Tcholakov  Using F# for scientific instrument control
 Dec 18 (Friday)  Matt Hawkins  ReST vs CQRS: The Trigger Pattern
 Dec 19 (Saturday)  Michael Newton  Angels From the Realms of Glory
 Dec 19 (Saturday)  Steven Pemberton  Let It Snow! A basic particle system in F# and WPF
 Dec 20 (Sunday)  Juan M Gómez  Developing mobile apps at the speed of light
 Dec 20 (Sunday)  Jorge Fioranelli  Reactive Messaging Patterns with F# and Akka.NET
 Dec 21 (Monday)  Steffen Forkmann  Automatic re-build and background tasks for suave.io websites
 Dec 21 (Monday)  Tomasz Heimowski  Property-based testing XSLT
 Dec 22 (Tuesday)  Mathias Brandewinder  Hacking together @wbfacts, a World Bank Twitter Bot
 Dec 22 (Tuesday)  Chad Boyer  F# Advent Calendar 2015
 Dec 23 (Wednesday)  Carsten König  F# advent 2015 – some fun with lambda calculus
 Dec 23 (Wednesday)  Troy Kershaw  Getting Started with SignalR using F# and OWIN
 Dec 24 (Thursday)  Matthew Sottile  Comparing trees, functionally
 Dec 24 (Thursday)  Craig Stuntz  Designing for Problems Too Big to Test
 Dec 25 (Friday)  Richard Griffiths  Monogame SnowFlakes – 2015
 Dec 25 (Friday)  Louie Bacaj  F# Powered Realtime Dashboard
 Dec 26 (Saturday)  Adam Granicz  WebSharper – a year in review
 Dec 26 (Saturday)  Chris Dobson  F#, Minecraft and a Raspberry Pi
 Dec 27 (Sunday)  @squeekeeper  Generating Markov text from YouTube comments
 Dec 27 (Sunday)  Indy Garcia  Twitter Local
 Dec 28 (Monday)  Pierre Irrmann  Visualizing F# Advent Calendar contributors
 Dec 29 (Tuesday)  Tamizh Vendan  Implementing API Gateway in F# Using Rx and Suave
 Dec 31 (Thursday)  Tomas Petricek  Happy New Year 2016 around the World
Jan 1 (Friday)  F# Software Foundation  Welcome to 2016 – A Call to Action

F# Weekly #42, 2015

Welcome to F# Weekly,

A roundup of F# content from this past week:

News

Videos/Presentations

Blogs

F# vNext News 

That’s all for now. Have a great week.

Previous F# Weekly edition – #41Subscribe

F# Weekly #41, 2015

Welcome to F# Weekly,

A roundup of F# content from this past week:

News

Videos/Presentations

Blogs

F# vNext News

New releases

That’s all for now. Have a great week.

Previous F# Weekly edition – #40Subscribe

SharePoint 2013: Content Enrichment and performance of 3rd party services

There are a lot of cases when people write Content Enrichment service for SharePoint 2013 to integrate it with some 3rd party tagging/enrichment API. Such integration very often leads to huge crawl time degradation, the same happened in my case too.

The first thought was to measure the time that CPES (Content Processing Enrichment Service) spends on calls. I wrapped all my calls to external system in timers and run Full Crawl again. The results were awful, calls were in average from 10 to 30 times slower from CPES than from POC console app. Code was the same and of course perfect, so, probably my 3rd party service was not able to handle such an incredible load that was generated by my super-fast SharePoint Farm.

If you are in the same situation with fast SharePoint, awesome CPES implementation and bad 3rd party service, you probably should try async integration approach described in “Advanced Content Enrichment in SharePoint 2013 Search” by Richard diZerega.

But what if the problem is in my source code… what if execution environment affects execution time of REST calls… Some time later, I found a nice post “Quick Ways to Boost Performance and Scalability of ASP.NET, WCF and Desktop Clients” from Omar Al Zabir. This article describes the notion of Connection Manager:

By default, system.net has two concurrent connections per IP allowed. This means on a webserver, if it’s calling WCF service on another server or making any outbound call to a particular server, it will only be able to make two concurrent calls. When you have a distributed application and your web servers need to make frequent service calls to another server, this becomes the greatest bottleneck

WHOA! That means that it does not matter how fast our SharePoint Search service is. Our CPES executes no more than two calls to 3rd party service at the same time and queues other. Let’s fix this unpleasant injustice in web.config:

 <system.net>
   <connectionManagement>
     <add address="*" maxconnection="128"/>
   </connectionManagement>
 </system.net>

Now performance degradation should be fixed and if your 3rd party service is good enough your crawl time will be reasonable, otherwise you need to go back to advice from Richard diZereg.

SharePoint 2013: Content Enrichment for Large Files

There are a couple of guides on how to write Content Enrichment services for SharePoint 2013. One of them is official MSDN article “How to: Use the Content Enrichment web service callout for SharePoint Server“.

This article advice you two configuration steps to adjust max size of document that will be processed by CPES (Content Processing Enrichment Service).

  1. Modify web.config to accept messages up to 8 MB, and configure readerQuotas to be a sufficiently large value.
    <bindings>
     <basicHttpBinding>
       <!-- The service will accept a maximum blob of 8 MB. -->
       <binding maxReceivedMessageSize = "8388608">
       <readerQuotas maxDepth="32"
         maxStringContentLength="2147483647"
         maxArrayLength="2147483647" 
         maxBytesPerRead="2147483647" 
         maxNameTableCharCount="2147483647" /> 
       <security mode="None" />
       </binding>
     </basicHttpBinding>
    </bindings>
    
  2. Modify SPEnterpriseSearchContentEnrichmentConfiguration.
    $ssa = Get-SPEnterpriseSearchServiceApplication
    $config = New-SPEnterpriseSearchContentEnrichmentConfiguration
    $config.Endpoint = http://Site_URL/ContentEnrichmentService.svc
    $config.InputProperties = "Author", "Filename"
    $config.OutputProperties = "Author"
    $config.SendRawData = $True
    $config.MaxRawDataSize = 8192
    Set-SPEnterpriseSearchContentEnrichmentConfiguration –SearchApplication
    $ssa –ContentEnrichmentConfiguration $config
    

The concept is generally good, but what if you need to process files larger than 8MB? Let’s try to increase this number up to 300Mb for example (I think that ideally this limit should be not less than max file size allowed for your web apps).

Let’s change both values and run full crawl of SharePoint site. After that, if you are lucky, you will see something like that in your “Error Breakdown”:

crawl_errors_001

WAT? Something went wrong, but what it was … Let’s investigate ULS logs on the machine with Search Service. After a couple of unforgettable minutes of reading ULS logs, I’ve found the following error message:

[Microsoft.CrawlerFlow-cb9134ec-91c6-4bac-89f9-a0cc9fe1e481] Microsoft.Ceres.Evaluation.Engine.ErrorHandling.HandleExceptionHelper : Evaluation failure detected: Operator : ContentEnrichment Operator type : ContentEnrichmentClient Error id : 3206 Correlation id : 60ef1afd-038a-4f64-8230-2b2493923f80 Partition id : 0c37852b-34d0-418e-91c6-2ac25af4be5b Message : Failed to send the item to the content processing enrichment service. 49691C90-7E17-101A-A91C-08002B2ECDA9:#9: https://mysite.com/MyDoc.pptx id : ssic://780174 System.ServiceModel.EndpointNotFoundException: There was no endpoint listening
at http://MyServer:8081/ContentProcessingEnrichmentService.svc that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details. —> System.Net.WebException: The remote server returned an error: (404) Not Found.
at System.Net.HttpWebRequest.GetResponse()
at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout) –

WAT? “The remote server returned an error: (404) Not Found”. Does the service not exist sometimes? How could it be? Let’s go to IIS log (on the machine where your CPES is installed). Path to the IIS logs should look similar to this c:\inetpub\logs\LogFiles\W3SVC3\.

iislog

It is true – CPES sometimes returns 404.13 status. Let’s google what this status code means.

404.13 – Content length too large. The request contains a Content-Length header. The value of the Content-Length header is larger than the limit that is allowed for the server.

Seems that IIS is not ready yet to receive our 300Mb files. There is one more parameter in web config that should be tweaked to  handle really large files, this parameter is maxAllowedContentLength (default value is 30000000, that is ~30Mb). Let’s change it in web.config:

 <system.webServer>
   <security>
     <requestFiltering>
       <requestLimits maxAllowedContentLength="314572800" />
     </requestFiltering>
   </security>
 </system.webServer>

Recrawl your content once again, and Voila, strange errors gone! Enjoy your content enrichment!)

F# Weekly #40, 2015

Welcome to F# Weekly,

A roundup of F# content from this past week:

News

Videos/Presentations

Blogs

F# vNext News

New releases

That’s all for now. Have a great week.

Previous F# Weekly edition – #39Subscribe