GSOC’17 Fifth and Sixth Week: 26 June 2017- 09 July 2017

This is a combined blog for last two weeks so this is going to be a little longer than usual. So In the past two weeks I worked on Docker Compatibility on Windows and passing conformance tests on Windows Machine. Some of the interesting things that I worked on were:

InlineJavascriptRequirement As in previous blog I mentioned that to evaluate javascript expression present in our CommandLineTool, we are creating a nodejs process(either natively or through Docker). On unix system we used select module to process multiple expression using same process using Non-blocking I/O operations, on windows it was not possible. Although threads gave some hope but they also blocked joining because they were continuously using read operation. I tried multiprocessing but again they caused problems because of pickling, as pickle library can’t pickle subprocess object. I raised this question on Stack Overflow, if you want to have a look, go here. So finally we decided to process each javascript expression in its own process and long term solution would be integrate javascript engine directly as suggested by Peter. The most interesting part was to find out that windows and Linux have quite different process definiton and implementation, while linux allows process to share I/O buffer with other process, windows doesn’t. We also added a conformance-test for multiple Javascript expression here

Default Docker Container In order to run workflows/tools on windows machine we will be using a default docker Container, in case a user doesn’t provide a docker container. It is being discussed as what kind of default container would be suitable, as of now I am using ubuntu image for default docker. I faced some issues in some this regard as in current implementation default-container was overriding container provided in hints. Another issue was that a default-container was not propogated to workflows steps. We solved this problem in this commit. Now the current order of priority and strict-checking of Docker container specification is as below:

• Requirements: Priority-Highest, Strict-checking-> Yes
• Hints: Priority-below requirements, Strict-checking-> No
• Default-Container: Priority-below Hints, Strict-checking-> Yes

Windows Newline character CRLF This is very interesting. When usually we checkout a github repository, Line ending of text files are automatically modified based on user’s machine OS.(unless you have specified in your git settings ex core.autocrlf). For most of the projects this is a very nice feature but in our case since we are using these files to generate checksum which is used to compare our results. Due to this extra Carriage Return(CR) char on windows that checksum turns out to be incorrect. If you also face such situation, you need to reset your repository, removing previous setting and create a .gitattributes file(see here) specifying the files for which you dont need auto conversion of line endings like *.txt text eol=lf

Some Docker specification were also added to conformance test, that would allow them to run in Windows OS also. see here. Few previous commits were also merged.

In the upcoming week, we would be working on runtime and input path conversion to Docker suitable path on Windows OS, removing Windows Drive letter from Docker's local path and Merging our progress to Master Branch. Also I must think about what would I do in the next half of GSOC period.