Yash Srivastava

This week, I worked on separating Job Status(s) from the common Jobs table (long overdue, I think for a year). As I wrote previously, not all values linked to a JobID are not really accessed in the day to day functions of the DIRACGrid project. Last time, Job Attributes were shifted to ES backend for accessing it with similar efficiency as Job Parameters. For Job Status, a new table has been created keeping MySQL backend with the following entities: JobID (primary key) Status Minor Status Application Status So, along with shifting the values to the new table (JobsStatus), functions are modified/added to access (read/write) the table: "setJobStatus": The existing function is modified to write in the new table. "getJobStatus": The new function has been added to access values in the new table. Along with this, it keeps back support to access the values that were already written in the Jobs table. Commits related to the work can be found below: Ad...

This week, I worked on adding Job Attributes like Owner, Job Group, Running Time etc. to ElasticSearch. The main idea behind such a move is that in future we would like to see these attributes as part of Job Parameters . This would allow for easier and efficient queries as it these attributes are commonly accessed in most of the functions. The changes/additions made to the existing code are: 1. Modified "setJobParameter" to receive these parameters as keywords ,   meaning they can be sent to the function but not necessarily required. The index has been modified with additions of the following attributes: Job Group Owner Proxy Submission Time Running Time 2. Add "getJobParametersAndAttributes" function in order to access results based on JobID, containing both Parameters (Name, Value) and Attributes (all five mentioned above). Along with this, existing codes were modified to use these functions, as well as some issues, were fixed in tests, Monito...

This week I was involved with documentation and testing of the two DB backends: MySQL and ElasticSearch. Documentation: Added developer documentation for using the ES backend for Job Monitoring and Job Status Update. Includes all the functions of ES and in general full documentation of the Workload Management System (not present before in developer documentation). Performance Tests: It is very important that we are at least able to achieve similar performance while using ES backend when compared to MySQL DB. It should also be remembered that the main objective of moving to ES backend is efficient data management as currently, MySQL doesn't store all the incoming data and query processing also becomes difficult in that case. After implementing and using both backends individually, I tested both the DBs performance using multi-mechanize ( link ), a tool which helps in running concurrent process on a specified number of threads. Some of the results can be found here for ...