Hacks too much

I've been slowly converting the memory model to be a lot more efficient (still to be seen if that works), but testing, and getting a set of tests that work has wasted a lot more time than I should have! It has me wondering, actually, if this isn't the right way to do things - or perhaps I should be moving back to something else not involving Rx programming. On the other hand, having everything hanging off the ctor is just so clean!

The main problem seems to be time travel. Rx has some nice things like a test scheduler which helps a lot with testing in Rx and time. You can put in a delay, and then use the scheduler to release the delay when you want. Sweet.

The problem comes when you start involving multiple threads and SemaphoreSlim's... at some point it seems like the test scheduler just stops working [I didn't try to debug it itself]. Rather, I switched all the tests to use the normal schedulers and found other ways to record what was going on, and make sure the testing was being done right.

I've not seen much about this on stack overflow.

The old setup worked just fantastic when having to deal with small ATLAS meetings. Even longer meetings like an ATLAS week seemed to work ok.

But CHEP would cause lots of faults, and crashes. Mostly out of memory errors.

Also, virtualization was turned on automatically for these long lists. However, the way my code worked was once the meeting was loaded every single talk for a single day would be rendered to an image and displayed. Regardless of it being displayed or not.

So this means it is a change. In part, this is about playing better with Rx than I was in the first time. So had I been smarter, perhaps this re-write and re-factoring could have been avoided.

When the meeting is opened, and by default all talks are to be downloaded, then fire off downloads. The files are placed in the cache, but nothing more is done.

When a PDF Page guy is in view, it subscribes to an observable, which causes the PDF file to be loaded from the cache, turned into a PDF file, and rendered. Once loaded, the PDF file is held onto as long as anyone is interested in it. I’m not 100% confident of this logic, we will have to check it when all is done. But everything should be released when it goes out of view.

All information needed to display the images from a pre-rendered talk (e.g. one that has been looked at before) should be cached, so the PDF file is not rendered. This means when you load up a meeting and see the grid of slides, if this isn’t the first time, no PDF files should be loaded, and no rendering should run.

First thing was to move it to use the new universal projects and PCL's. They have advanced significantly since I first wrote this code. In particular, it is now time to move to Windows.Net.Http, which involves a lot of changes. In particular, how the credentials are used.

The basic things are the same (see previous posts on CERNSSO). In particular, the fact that a Windows App has only the local cert store. What is nice, now, is that also exists on Windows Phone - so with 8.1 they are identical finally. It wasn't possible to do anything with certificate stores on WP previously.

But this makes me wonder. Why can't I add the cert and make the web request, allow redirects, and let the underlying system (which mimics a browser) just work? That would be a lot simpler than doing what I'm currently doing. Testing that is going to require some effort. :-)

Found some example code on accessing certificates and associating them with a request in the windows runtime here.

Parses the header and symbols list from dumpbin

Both of those have to be correlated

Removed one or two things as per the bindexplib code

Can be found on GitHub

Tested for identical output on TUri.obj, which is 32 bit, only.

Turns out that MS has a dumpbin command which should give us the raw information. We then have to parse its output to produce a .def file.

The initial idea for this came from some stack overflow discussion:

The main question and answer that describes how to get at this list.

Some details encountered getting at symbols, un-inited variables, etc.

What do the columns mean. In particular the () means function, and without means data.

While some suggest linkermember, that doesn't seem to work. On the other hand, from a quick hand scan and text edit search, the /symbols option seems like just what we need.

Next job (assuming each stage goes ok)

Make sure this does something sensible on x64 object files.

Write a script that will generate the proper output, and compare a few files.

Managed to figure out how to distribute ROOT in nuget, finally. The idea is a cheat. The nuget package is tiny - downloads a targets and props file. During the build, the installation on the local machine is checked. If it is matches the version requested, then all include files, etc., point to that and the build runs. If not, it will download the proper version from root.cern.ch, unpack it in the user's TEMP directory, and then use that. It will also define a few important variables for debug-running. There are two problems: intellisense doesn't work until VS rescans the project if a download had to occur and there is an incorrect local installation of ROOT. And the debugger environment variable modifications won't work until the person reloads the project. The latter should be a bug, I think. ;-) GitHub contains what is needed, including custom MSBUILD commands.

After conversation with Bertrand I realized that the vc12 bug isn't actually too long a file - it is a nested namespace: namespace ROOT { namespace Fit { namespace ROOT {}}}. That second ROOT causes all hell to break loose. Created a single file, 40 lines long, removed a lot of the cruft, and the bug still occurs. Sweet. GitHub was updated, as was the bug report in connect.

I built a version of Pythia8.1 and put it on nuget. Build and package instructions can be found in GitHub.

In order to do that I used the CoApp tools for C++ packages. But those tools don't allow you to use custom MSBuild commands. So I added some source code, submitted a pull request. It is still flawed, but need some input from experts before I fix it (event at the level of it being the correct approach!).

There are a few problems:

Doesn't work with debug builds

Can't include it properly in a CLI test project (seems to totally mess things up).

static/dynamic: this indicates the libraries it is built against. For us, this is always dynamic.

Release/Debug: this is given to understand a set of macro definitions and also the checked STL library when debug is specified, and not otherwise. It also switches optimization on/off, perhaps writes a PDB.

Unfortunately, in ROOT's case, it doesn't actually do debug/release this way.

All builds are API compatible with the Release version of ROOT

The debug version turns off optimization and writes a debug program database.

I think the reason this is done is because they link against external libraries and they are built only release (or something like that).

ROOT debug mode is everything that we would normally consider debug except we link against the regular libraries.

This is unfortunate, because the whole world thinks of "debug" being one thing, and ROOT thinks of it being another.

The only reason to use debug ROOT libraries is if you want to look inside ROOT code and see what happens.

So, this means that a "debug" pivot for the rest of the world can't link against a "debug" pivot for ROOT. I think that brings us to the conclusion:

You can't have any debug pivots for the ROOT library. It just isn't composable with other libraries in the open source world!

Under what reasons would you ever use the debug libraries in ROOT? Just when you want to debug something going on in the base library. Having the program database is very nice in this particular case, btw. So it can definitely be useful.

The solution ClrPlus seems to build without errors. One minor warning that means nothing... Both Debug and Release builds.

Building the CoApp.Powershell guy - that seems to cause the trouble.

The first project to build ("Unused") fails with some command called "handle". But it doesn't call a VSIDE error when you right click and build!

Handle looks like a sys internals tool! Ah ha! Time (I guess) to start keeping track of this stuff we find out?

Ops. Looks like our clone is way behind. Did a merge (git fetch upstream, git merge upstream/master). Cool that worked without totally blowing away the local repro! However, lots of changes came down.

Definitely have to build the solution twice due to a nuget issue. So that should be in the instructions.

The unused variable warning is still present. Moving right along... Still have the failure with handle. May actually be a non-failure in that I don't have anything open. But still.

Get warnings about missing references that don't seem important. Removed them from my copy:

Icon.Zip

MSBuild.ExtensionPack

MSBuild.ExtensionPack.Git

MSBuild.ExtensionPack.Loggers

MSBuild.ExtensionPack.TaskFactory.Powershell

The build seems to go just fine now. But...?

When I build the release version, however, things don't go so well. Turns out that it needs the above for the release version. Adding nuget packages:

DotNetZip

MSBuild.ExtensionPack - and the references had to be inserted by hand here as they were put in a "tools" directory. So clearly the issue is the hint path isn't right. It should be made relative.

Build also "fails" with a x86 warning - the nuget.exe is referenced. But that is probably just so that an item gets published. So we can ignore that safely. Though we should record that!

Contains an include file

Builds a dll

Contains a directory etc/content.txt that has to be in the resulting app's code.

Build a C++ program that allows you to read and write ROOT files (like histograms).

Used in a C++ .NET program

Referenced by a library built with the above, that has been packaged into a nuget library.

Some other feature requirements:

Individual packages are small enough they can be uploaded to nuget.

If the build is done right, a symbol's package is uploaded to the proper debug symbol server (wow, that would be nice!).

Client C++ and .NET programs must "just work" in debug and release mode. Limitations to code should be clearly documented.

The pain point, actually, is that it is hard to download from the internet! The code all has decent stuff to store the file locally - even copy it, but, unfortunately, it is using some code that will save to a file directly.