This post is about one more FAKE use case. It will be not usual, but I hope useful script.
The problem I have faced to is recompilation of Stanford NLP products to .NET using IKVM.NET. I am sick of doing it manually. I posted instructions on how to do it, but I think that not many people have tried to do it. I believe that I can automate it end to end from downloading *.jar files to building NuGet packages. Of course, I have chosen FAKE for this task (Thanks to Steffen Forkmann for help with building NuGet packages).
The build scenario is the following:
- Download zip archive with *.jar files and trained models from Stanford NLP site (They can be large, up to 200Mb like for Stanford Parser, and I do not want to store all this stuff in my repository)
- Download IKVM.NET compiler as a zip archive. (It is not distributed with NuGet package and is not referenced from IKVM.NET site. It is really tricky to find it for the first time)
- Unzip all downloaded archives.
- Carefully recompile all required *.jar files considering all references.
- Sign all compiled assemblies to be able to deploy them to the GAC if needed.
- Compile NuGet package.
Steps 1-5 are not covered by FAKE OOTB tasks and I needed to implement them by myself. Since I wanted to use F# 3.0 features and .NET 4.5 capabilities (like System.IO.Compression.FileSystem.ZipFile for unzipping) I have chosen pre-release version of FAKE 2 that uses .NET 4 runtime. Pre-release version of FAKE can be restored from NuGet as follows:
"nuget.exe" "install" "FAKE" "-Pre" "-OutputDirectory" "..\build" "-ExcludeVersion"
Requirements: For sure, I do not want to download files from the Internet during each build. Before downloading files, I want to check their presence on the file system, if they are missed then start downloading. During downloading, I want to see the progress status to be sure that everything works. The code that does it:
#r "System.IO.Compression.FileSystem.dll" let downloadDir = @".\Download\" let restoreFile url = let downloadFile file url = printfn "Downloading file '%s' to '%s'..." url file let BUFFER_SIZE = 16*1024 use outputFileStream = File.Create(file, BUFFER_SIZE) let req = System.Net.WebRequest.Create(url) use response = req.GetResponse() use responseStream = response.GetResponseStream() let printStep = 100L*1024L let buffer = Array.create<byte> BUFFER_SIZE 0uy let rec download downloadedBytes = let bytesRead = responseStream.Read(buffer, 0, BUFFER_SIZE) outputFileStream.Write(buffer, 0, bytesRead) if (downloadedBytes/printStep <> (downloadedBytes-int64(bytesRead))/printStep) then printfn "\tDownloaded '%d' bytes" downloadedBytes if (bytesRead > 0) then download (downloadedBytes + int64(bytesRead)) download 0L let file = downloadDir @@ System.IO.Path.GetFileName(url) if (not <| File.Exists(file)) then url |> downloadFile file file let unZipTo toDir file = printfn "Unzipping file '%s' to '%s'" file toDir Compression.ZipFile.ExtractToDirectory(file, toDir) let restoreFolderFromUrl folder url = if not <| Directory.Exists folder then url |> restoreFile |> unZipTo (folder @@ @"..\") let restoreFolderFromFile folder zipFile = if not <| Directory.Exists folder then zipFile |> unZipTo (folder @@ @"..\")
Compiler should be able to rebuild any number of *.jar files with predefined dependencies and sign result *.dll files if required.
let ikvmc = restoreFolderFromUrl @".\temp\ikvm-7.3.4830.0" "http://www.frijters.net/ikvmbin-7.3.4830.0.zip" @".\temp\ikvm-7.3.4830.0\bin\ikvmc.exe" let ildasm = @"c:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\Bin\x64\ildasm.exe" let ilasm = @"c:\Windows\Microsoft.NET\Framework64\v2.0.50727\ilasm.exe" type IKVMcTask(jar:string) = member val JarFile = jar member val Version = "" with get, set member val Dependencies = List.empty<IKVMcTask> with get, set let timeOut = TimeSpan.FromSeconds(120.0) let IKVMCompile workingDirectory keyFile tasks = let getNewFileName newExtension (fileName:string) = Path.GetFileName(fileName).Replace(Path.GetExtension(fileName), newExtension) let startProcess fileName args = let result = ExecProcess (fun info -> info.FileName <- fileName info.WorkingDirectory <- FullName workingDirectory info.Arguments <- args) timeOut if result<> 0 then failwithf "Process '%s' failed with exit code '%d'" fileName result let newKeyFile = let file = workingDirectory @@ (Path.GetFileName(keyFile)) File.Copy(keyFile, file, true) Path.GetFileName(file) let rec compile (task:IKVMcTask) = let getIKVMCommandLineArgs() = let sb = Text.StringBuilder() task.Dependencies |> Seq.iter (fun x -> compile x x.JarFile |> getNewFileName ".dll" |> bprintf sb " -r:%s") if not <| String.IsNullOrEmpty(task.Version) then task.Version |> bprintf sb " -version:%s" bprintf sb " %s -out:%s" (task.JarFile |> getNewFileName ".jar") (task.JarFile |> getNewFileName ".dll") sb.ToString() File.Copy(task.JarFile, workingDirectory @@ (Path.GetFileName(task.JarFile)) ,true) startProcess ikvmc (getIKVMCommandLineArgs()) if (File.Exists(keyFile)) then let dllFile = task.JarFile |> getNewFileName ".dll" let ilFile = task.JarFile |> getNewFileName ".il" startProcess ildasm (sprintf " /all /out=%s %s" ilFile dllFile) File.Delete(dllFile) startProcess ilasm (sprintf " /dll /key=%s %s" (newKeyFile) ilFile) tasks |> Seq.iter compile
Using this helper function, build scripts come out pretty straightforward and easy. For example, recompilation of Stanford Parser looks as follows:
Target "RunIKVMCompiler" (fun _ -> restoreFolderFromUrl @".\temp\stanford-parser-full-2013-06-20" "http://nlp.stanford.edu/software/stanford-parser-full-2013-06-20.zip" restoreFolderFromFile @".\temp\stanford-parser-full-2013-06-20\edu" @".\temp\stanford-parser-full-2013-06-20\stanford-parser-3.2.0-models.jar" [IKVMcTask(@"temp\stanford-parser-full-2013-06-20\stanford-parser.jar", Version=version, Dependencies = [IKVMcTask(@"temp\stanford-parser-full-2013-06-20\ejml-0.19-nogui.jar", Version="0.19.0.0")])] |> IKVMCompile ikvmDir @".\Stanford.NLP.snk" )