Yash Srivastava

This week I started working on two aspects:- First, adding a second flag to stop reporting to the MySQL backend. Currently, the scenario is the DIRAC project would be using both ElasticSearch and MySQL backend concurrently for storing and retrieving values from the database. This extended support to use both is kept so that the current users don't face problem from transitioning from one DB to other.                                                                                                                         The use of this second flag is to stop reporting to the MySQL backend once the project wholly starts using ES, we would just need to keep this flag as False which ...

This week had the 'first deliverable' to be submitted. But not an easy task though. Even after coding and removing bugs during the coding time, many bugs came in when my module had to be integrated with other components of the project. The week's works included:  Add a code to the Jenkins integration test to change flag in the dirac.cfg in order to activate ElasticSearch backend for storing and accessing JobParameters.  The code would be integrated into the Jenkins code for system testing. Uses CSAPI() of DIRAC service. The commit can found here .  Fix codes for generateFullIndexName() function error as it was moved into the class ElasticSearch to remove redundant imports to the same file.  Commit here . Address the requested changes on the Pull request as asked by code owners and reviewers. Along with this, started and working with Jenkins testing service. Brought efficiency to algorithms written earlier for fetching results with inputs fro...

As part of the 2nd week, I was also involved in the DIRAC's annual workshop where I attended (virtually via Vidyo) some of the reports presented by fellow mentors between 22nd to 25th May 2018 (couldn't attend all due to time difference). There was a discussion about the rise of ElasticSearch and its intended usage in the project (discusses my GSoC project as well). Presentation link here . It was quite an engaging experience as it helped to understand what differences lie between the developers and users in terms of the usage of the tools and the importance of coming to a common page to ease both the parties. More details about the workshop can be found here .

This week was quite a busy one as most of the development for the first deliverable (next week) took place during this time.  Firstly, we should be aware why are we developing an alternative database using ElasticSearch. The main reason we are having huge amounts of data coming every day which itself a large number of parameters. So, using ES is quite beneficial here since we can create indexes on a daily/monthly basis and also query the same with more flexibility than MySQL. I was majorly focussed and covered the following tasks:  Create a wrapper over the ElasticSearchDB.py for JobParameters with methods: 'get' and 'set'.  The function 'get' majorly handles the fetching of the parameters from the ElasticSearch (ES) backend using the ElasticSearch-py API 'query'. The function takes in a jobID (which is like a primary key, though no keys in ES) and an optional list of names if needed to query the ES. The function 'set' is used to upda...